Statistical Rules of Thumb
Second Edition
Gerald van Belle
University of Washington
Department of Biostatistics
and Department of Environmental and Occupational Health Sciences
Seattle, WA
WILEY
AJOHN WILEY & SONS, INC., PUBLICATION
Contents
1
Preface to the Second Edition
xiv
Preface to the First Edition
xvi
Acronyms
xix
The Basics
1.1 Four Basic Questions
1.2 Observation is Selection
1.3 Replicate to Characterize Variability
1.4 Variability Occurs at Multiple Levels
1.5 Invalid Selection is the Primary Threat to Valid
Inference
1.6 There is Variation in Strength of Inference
1.7 Distinguish Randomized and Observational Studies
1.8 Beware of Linear Models
1.9 Keep Models As Simple As Possible, But Not More
Simple
1.10 Understand Omnibus Quantities
I
2
3
4
6
6
7
8
10
12
14
vii
viii
CONTENTS
1.11 Do Not Multiply Probabilities More Than Necessary
1.12 Use Two-sided p-Values
1.13 p-Values for Sample Size, Confidence Intervals for
Results
1.14 At Least Twelve Observations for a Confidence
Interval
1.15 Estimate ± Two Standard Errors is Remarkably
Robust
1.16 Know the Unit of the Variable
1.17 Be Flexible About Scale of Measurement Determining
Analysis
1.18 Be Eclectic and Ecumenical in Inference
15
16
2 Sample Size
2.1 Begin with a Basic Formula for Sample Size-Lehr's
Equation
2.2 Calculating Sample Size Using the Coefficient of
Variation
2.3 No Finite Population Correction for Survey Sample
Size
2.4 Standard Deviation and Sample Range
2.5 Do Not Formulate a Study Solely in Terms of Effect
Size
2.6 Overlapping Confidence Intervals Do Not Imply
Nonsignificance
2.7 Sample Size Calculation for the Poisson Distribution
2.8 Sample Size for Poisson With Background Rate
2.9 Sample Size Calculation for the Binomial Distribution
2.10 When Unequal Sample Sizes Matter; When They
Don't
2.11 Sample Size With Different Costs for the Two Samples
2.12 The Rule of Threes for 95% Upper Bounds When
There Are No Events
2.13 Sample Size Calculations Are Determined by the
Analysis
27
18
20
22
23
24
25
29
31
35
36
37
38
40
41
43
45
47
49
50
3
4
CONTENTS
ix
Observational Studies
3.1 The Model for an Observational Study is the Sample
Survey
3.2 Large Sample Size Does Not Guarantee Validity
3.3 Good Observational Studies Are Designed
3.4 To Establish Cause Effect Requires Longitudinal Data
3.5 Make Theories Elaborate to Establish Cause and Effect
3.6 The Hill Guidelines Are a Useful Guide to Show
Cause Effect
3.7 Sensitivity Analyses Assess Model Uncertainty and
Missing Data
53
Covariation
4.1 Assessing and Describing Covariation
4.2 Don't Summarize Regression Sampling Schemes
with Correlation
4.3 Do Not Correlate Rates or Ratios Indiscriminately
4.4 Determining Sample Size to Estimate a Correlation
4.5 Pairing Data is not Always Good
4.6 Go Beyond Correlation in Drawing Conclusions
4.7 Agreement As Accuracy, Scale Differential, and
Precision
4.8 Assess Test Reliability by Means of Agreement
4.9 Range of the Predictor Variable and Regression
4.10 Measuring Change: Width More Important than
Numbers
5 Environmental Studies
5.1 Begin with the Lognormal Distribution in
Environmental Studies
5.2 Differences Are More Symmetrical
5.3 Know the Sample Space for Statements of Risk
5.4 Beware of Pseudoreplication
5.5 Think Beyond Simple Random Sampling
5.6 The Size of the Population and Small Effects
54
55
56
57
58
60
61
65
67
68
70
71
73
75
77
80
82
84
87
88
90
92
93
95
96
CONTENTS
5.7
5.8
5.9
5.10
5.11
5.12
5.13
5.14
5.15
5.16
Models of Small Effects Are Sensitive to Assumptions
Distinguish Between Variability and Uncertainty
Description of the Database is As Important as Its Data
Always Assess the Statistical Basis for an
Environmental Standard
Measurement of a Standard and Policy
Parametric Analyses Make Maximum Use of the Data
Confidence, Prediction, and Tolerance Intervals
Statistics and Risk Assessment
Exposure Assessment is the Weak Link in Assessing
Health Effects of Pollutants
Assess the Errors in Calibration Due to Inverse
Regression
Epidemiology
6.1 Start with the Poisson to Model Incidence or
Prevalence
6.2 The Odds Ratio Approximates the Relative Risk
Assuming the Disease is Rare
6.3 The Number of Events is Crucial in Estimating
Sample Sizes
6.4 Use a Logarithmic Formulation to Calculate Sample
Size
6.5 Take No More than Four or Five Controls per Case
6.6 Obtain at Least Ten Subjects for Every Variable
Investigated
6.7 Begin with the Exponential Distribution to Model
Time to Event
6.8 Begin with Two Exponentials for Comparing
Survival Times
6.9 Be Wary of Surrogates
6.10 Prevalence Dominates in Screening Rare Diseases
6.11 Do Not Dichotomize Unless Absolutely Necessary
6.12 Additive and Multiplicative Models
97
99
100
101
102
104
106
108
110
111
113
114
115
120
122
124
125
127
129
131
134
138
139
CONTENTS
7
8
Evidence-Based Medicine
7.1 Strength of Evidence
7.2 Relevance of Information: POEM vs. DOE
7.3 Begin with Absolute Risk Reduction, then Follow
with Relative Risk
7.4 The Number Needed to Treat (NNT) is Clinically
Useful
7.5 Variability in Response to Treatment Needs to be
Considered
7.6 Safety is the Weak Component of EBM
7.7 Intent to Treat is the Default Analysis
7.8 Use Prior Information but not Priors
7.9 The Four Key Questions for Meta-analysts
Design, Conduct, and Analysis
8.1 Randomization Puts Systematic Effects into the Error
Term
8.2 Blocking is the Key to Reducing Variability
8.3 Factorial Designs and Joint Effects
8.4 High-Order Interactions Occur Rarely
8.5 Balanced Designs Allow Easy Assessment of Joint
Effects
8.6 Analysis Follows Design
8.7 Independence, Equal Variance, and Normality
8.8 Plan to Graph the Results of an Analysis
8.9 Distinguish Between Design Structure and Treatment
Structure
8.10 Make Hierarchical Analyses the Default Analysis
8.11 Distinguish Between Nested and Crossed DesignsNot Always Easy
8.12 Plan for Missing Data
8.13 Address Multiple Comparisons Before Starting
the Study
8.14 Know Properties Preserved When Transforming Units
8.15 Consider Bootstrapping for Complex Relationships
XI
143
144
148
149
151
153
155
156
157
158
163
163
165
166
168
170
171
173
177
179
181
182
184
186
188
191
X/7
CONTENTS
Words, Tables, and Graphs
9.1 Use Text for a Few Numbers, Tables for Many
Numbers, Graphs for Complex Relationships
9.2 Arrange Information in a Table to Drive Home
the Message
9.3 Always Graph the Data
9.4 Always Graph Results of An Analysis of Variance
9.5 Never Use a Pie Chart
9.6 Bar Graphs Waste Ink; They Don't Illuminate
Complex Relationships
9.7 Stacked Bar Graphs Are Worse Than Bar Graphs
9.8 Three-Dimensional Bar Graphs Constitute
Misdirected Artistry
9.9 Identify Cross-sectional and Longitudinal Patterns in
Longitudinal Data
9.10 Use Rendering, Manipulation, and Linking in
High-Dimensional Data
193
10 Consulting
10.1 Session Has Beginning, Middle, and End
10.2 Ask Questions
103 Make Distinctions
10.4 Know Yourself, Know the Investigator
10.5 Tailor Advice to the Level of the Investigator
10.6 Use Units the Investigator is Comfortable With
10.7 Agree on Assignment of Responsibilities
10.8 Any Basic Statistical Computing Package Will Do
10.9 Ethics Precedes, Guides, and Follows Consultation
10.10 Be Proactive in Statistical Consulting
10.11 Use the Web for Reference, Resource, and Education
10.12 Listen to, and Heed the Advice of Experts in the Field
217
218
219
220
222
223
224
226
227
228
230
231
233
Epilogue
236
References
239
9
193
195
198
199
203
204
206
209
210
213
CONTENTS
xiii
Author Index
255
Topic Index
261