Why we (usually) don’t worry about multiple
comparisons
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Columbia University, New York University, University of
California
30 Aug 2011
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Contents
I
What is the multiple comparisons problem?
I
Why don’t I (usually) care about it?
I
Some stories
I
Statistical framework and multilevel modeling
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
What is the multiple comparisons problem?
I
Even if nothing is going on, you can find things
I
I
I
Data snooping
Overwhelmed by data and plausible “findings”
“If not accounted for, false positive differences are very likely
to be identified”: 5% of our 95% intervals will be wrong
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Why don’t I (usually) care about multiple comparisons?
I
I
When looked at one way, multiple comparisons seem like a
major worry
But from another perspective, they don’t matter at all:
I
I
I
I don’t (usually) study study phenomena with zero effects
I don’t (usually) study comparisons with zero differences
I don’t mind being wrong 5% of the time
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Some stories
I
Medical imaging
I
SAT coaching in 8 schools
I
Effects of electromagnetic fields at 38 frequencies
I
Teacher and school effects in NYC schools
I
Grades and classroom seating
I
Beautiful parents have more daughters
I
Comparing test scores across states
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Neural activity in a dead fish
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
“Voodoo correlations” in neuroscience
I
Vul, Harris, Winkelman, Pashler:
I
I
I
Multiple comparisons corrections do not solve the problem:
I
I
I
I
Correlations reported in medical imaging studies are commonly
overstated because researchers select the highest values
These statistical problems are leading to scientific errors
Adjustment of significance levels does not stop the selected
correlations themselves from being too high
Multiple comparisons methods control the rate of false alarms
in a setting where true effects are zero
But this is not so relevant in imaging, where differences are
not in fact zero
Discussion in Perspectives in Psychological Science (2009)
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Where should be putting our technical thinking?
I
Scientific modeling
I
Mapping scientific models to data collection and statistical
modeling
I
“Building a cumulative knowledge base”
I
But not . . . discussions of p-values, cross-validation, selection
bias, etc.
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
SAT coaching in 8 schools
School
A
B
C
D
E
F
G
H
Estimated
treatment
effect, yj
28
8
−3
7
−1
1
18
12
Standard error
of effect
estimate, σj
15
10
16
11
9
11
10
18
I
Separate experiment in each school
I
Variation in treatment effects is indistinguishable from 0
Multilevel Bayes analysis
I
I
I
Overlappling confidence intervals for the 8 school effects
Statements such as Pr (effect in A > effect in C)= 0.7
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Effects of electromagnetic fields at 38 frequencies
Estimates ± standard errors
Est. treatment effect
0.0
0.2
0.4
Est. treatment effect
0.0
0.2
0.4
Estimates with statistical significance
0
100
200
300
400
500
Frequency of magnetic field (Hz)
0
100
200
300
400
500
Frequency of magnetic field (Hz)
I
Background: electromagnetic fields and cancer
I
Original article summarized using p-values
I
Confidence intervals show comparisons more clearly
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Separate estimates and hierarchical Bayes estimates
Multilevel estimates ± standard errors
Est. treatment effect
0.0
0.2
0.4
Estimated treatment effect
0.0
0.2
0.4
Estimates ± standard errors
0
100
200
300
400
500
Frequency of magnetic field (Hz)
0
100
200
300
400
500
Frequency of magnetic field (Hz)
I
Most comparisons are no longer statistically significant
I
“Multiple comparisons” is less of a concern
I
We moved the intervals together instead of widening them!
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Teacher and school effects in NYC schools
I
Goal is to estimate range of variation
(How important are teachers? Schools?)
I
Key statistic is year-to-year persistence (e.g., for teachers
ranked in top 25% one year, how well do they do the next?)
I
The “multiple comparisons” issue never arises!
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Grades and classroom seating
I
Classroom demonstration (Gelman and Nolan, 2002)
I
Assign students random numbers as “grades”
I
Ask students with “grades” 0–25 to raise one finger,
students with “grades” 75–100 to raise one hand
I
Instructor scans the room to find a statistically significant
comparison (e.g., “boys on the left side of the classroom have
higher grades than girls in the back row”)
I
This is a pure multiple comparisons problem!
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Beautiful parents have more daughters
I
S. Kanazawa (2007). Beautiful parents have more daughters:
a further implication of the generalized Trivers-Willard
hypothesis. Journal of Theoretical Biology.
I
Attractiveness was measured on a 1–5 scale
(“very unattractive” to “very attractive”)
I
I
56% of children of parents in category 5 were girls
48% of children of parents in categories 1–4 were girls
I
Statistically significant (2.44 s.e.’s from zero, p = 1.5%)
I
But the simple regression of sex ratio on attractiveness is not
significant (estimate is 1.5 with s.e. of 1.4)
I
Multiple comparisons problem: 5 natural comparisons × 4
possible time summaries!
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Comparing test scores across states
I
National Assessment of Educational Progress (NAEP)
I
Comparing states: which comparisons are statistically
significant?
I
50 × 49/2: a classic multiple comparions problem!
I
Our multilevel inferences
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Dis trict of C olumbia (DC )
G uam (G U)
Mis s is s ippi (MS )
Louis iana (LA)
Alabama (AL)
C alifornia (C A)
S outh C arolina (S C ) ‡
New Mexico (NM)
Hawaii (HI)
G eorgia (G A)
Delaware (DE )
F lorida (F L)
Arkans as (AR )
Arizona (AZ) ‡
Tennes s ee (T N)
Nevada (NV ) ‡
Kentucky (K Y )
R hode Is land (R I)
Maryland (MD)
V irginia (VA)
New York (NY ) ‡
Wyoming (WY )
DoDDS (DO)
Wes t V irginia (WV )
Alas ka (AK ) ‡
Oregon (OR )
DDE S S (DD)
North C arolina (NC )
Mis s ouri (MO)
Was hington (WA)
Vermont (V T ) ‡
C olorado (C O)
Penns ylvania (PA) ‡
Michigan (MI) ‡
New J ers ey (NJ ) ‡
Utah (UT )
Montana (MT ) ‡
Nebras ka (NE )
Texas (T X)
Iowa (IA) ‡
Mas s achus etts (MA)
Indiana (IN)
North Dakota (ND)
Wis cons in (WI)
Minnes ota (MN)
C onnecticut (C T )
Maine (ME )
Classical multiple comparisons inferences for NAEP
ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME
MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN
CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT
WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI
ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN IN
IN
IN
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA
TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX
NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE
MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT
NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ
UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT
MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI
PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA
CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO
WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA
VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT
MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO
NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC
DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD
AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK
OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR OR
WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV WV
DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO DO
WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY WY
VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA
NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY
MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD
RI RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI RI
RI
RI
RI
RI RI
RI
RI
RI
RI
RI
RI
RI
RI
RI
RI RI
RI RI
RI
RI
RI
RI RI
RI
RI
KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY KY
TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN TN
NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV NV
AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ AZ
AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR AR
FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL
GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA GA
DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE
HI HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI HI
HI
HI
HI
HI HI
HI
HI
HI
HI
HI
HI
HI
HI
HI
HI HI
HI HI
HI
HI
HI
HI HI
HI
HI
NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM NM
SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC SC
AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL AL
CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA CA
LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA LA
MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS MS
GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU GU
DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC DC
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Dis trict of C olumbia (DC )
G uam (G U)
Mis s is s ippi (MS )
Louis iana (LA)
C alifornia (C A)
Alabama (AL)
S outh C arolina (S C ) ‡
New Mexico (NM)
Hawaii (HI)
Delaware (DE )
G eorgia (G A)
F lorida (F L)
Arkans as (AR )
Arizona (AZ) ‡
Nevada (NV ) ‡
Tennes s ee (T N)
Kentucky (K Y )
R hode Is land (R I)
Maryland (MD)
New York (NY ) ‡
V irginia (VA)
Wyoming (WY )
DoDDS (DO)
Wes t V irginia (WV )
Oregon (OR )
Alas ka (AK ) ‡
DDE S S (DD)
North C arolina (NC )
Mis s ouri (MO)
Vermont (V T ) ‡
Was hington (WA)
C olorado (C O)
Penns ylvania (PA) ‡
Michigan (MI) ‡
Utah (UT )
New J ers ey (NJ ) ‡
Montana (MT ) ‡
Nebras ka (NE )
Texas (T X)
Mas s achus etts (MA)
Iowa (IA) ‡
Indiana (IN)
North Dakota (ND)
Wis cons in (WI)
C onnecticut (C T )
Minnes ota (MN)
Maine (ME )
Classical inferences for NAEP: close-up
ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME
MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN MN
CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT
WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI
ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN
IN IN
IN
IN
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
IA
MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA
TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX
NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE NE
MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT
NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ
UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT UT
MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI MI
PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA
CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO CO
WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA WA
VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT
MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO MO
NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC
DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD DD
AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK AK
OR OR OR OR
OR OR OR OR OR
OR OR OR comparisons
OR OR OR
OR OR ORJennifer
OR OR ORHill,
OR ORand
OR OR
OR OR ORYajima
OR OR OR OR ORusing
OR ORmultilevel
OR OR OR OR
OR OR OR OR OR OR OR OR
Andrew
Gelman,
Masanao
Multiple
models
Multilevel inferences for NAEP: close-up
Massachusetts
New Jersey
New Hampshire
Kansas
Minnesota
Vermont
North Dakota
Indiana
Ohio
Wisconsin
Pennsylvania
Wyoming
Montana
Virginia
Iowa
Connecticut
Washington
New York
Maine
Texas
Florida
Delaware
North Carolina
South Dakota
Idaho
Maryland
Colorado
Missouri
Utah
Nebraska
Arkansas
Michigan
Illinois
Alaska
South Carolina
Oklahoma
West Virginia
Oregon
Rhode Island
Georgia
Kentucky
Hawaii
Tennessee
Arizona
Nevada
Louisiana
California
Alabama
New Mexico
Mississippi
Comparisons of Average Mathematics Scale Schores for
Grade 4 Public Schools in Participating Jurisdictions
MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA MA
NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ NJ
NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH NH
KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS KS
MNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMNMN
VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT VT
ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND
IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN IN
OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH
WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI WI
PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA PA
WYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWYWY
MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT MT
VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA VA
IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA IA
CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT CT
WAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWAWA
NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY
ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME ME
TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX TX
FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL FL
DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE
NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC NC
SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD SD
ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID ID
MDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMDMD
CO CO
CO CO COGelman,
CO CO CO COJennifer
CO CO CO CO
CO CO
CO CO
CO CO CO COYajima
CO CO CO CO CO CO
CO CO CO CO
CO CO CO CO COusing
CO CO CO
CO CO CO COmodels
CO CO CO CO CO CO CO
Andrew
Hill,
and
Masanao
Multiple
comparisons
multilevel
NAEP: classical vs. multilevel
I
Both procedures are algorithmic (“push a button”)
I
Both procedures treat 50 states exchangeably
I
Multilevel inferences are sharper (more comparisons are
“statistically significant”)
I
How can this be?
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Something for nothing? A free lunch?
I
Classical multiple comparisons worries about
θ1 = θ2 = · · · = θ50
I
Not an issue with NAEP
I
Multilevel model estimates the group-level variance, decides
based on the data how much to adjust
I
Classical procedure does not learn from the data
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Message from the examples
I
Classical multiple comparisons corrections don’t seem so
important when we fit hierarchical models
I
But they can be crucial for classical comparisons
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Statistical framework
I
Goal is to estimate θj , for j = 1, . . . , J (for example, effects of
J schools)
I
Comparisons have the form, θj − θk .
I
For simplicity, suppose data come from J separate experiments
I
Type S errors
I
Multilevel modeling as a solution to the multiple comparisons
issue
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Type S (sign) errors
I
I’ve never made a Type 1 error in my life
I
I
I
Type 1 error is θj = θk , but I claim they’re different
I’ve never studied anything where θj = θk
I’ve never made a Type 2 error in my life
I
I
Type 2 error is θj 6= θk , but I claim they’re the same
I’ve never claimed that θj = θk
I
But I make errors all the time!
I
Type S error: θ1 > θ2 , but I claim that θ2 > θ1 (or vice versa)
I
Type S errors can occur when we make claims with confidence
(i.e., have confidence intervals for θj − θk that exclude zero)
I
We want to make fewer claims with confidence when we are
less certain about the ranking of the θj ’s
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Multilevel (hierarchical) modeling
I
Partial pooling tends to reduce the number of statistically
significant comparisons:
σθ2
(ȳj − ȳk )
σȳ2 + σθ2
.q
√
posterior sd(θj − θk ) =
2σȳ σθ
σȳ2 + σθ2
posterior E(θj − θk ) =
posterior z-score of θj − θk :
I
(ȳj − ȳk )
1
√
·q
2σȳ
1 + σȳ2 /σθ2
Posterior mean of the difference is pulled toward 0, faster than
the posterior sd decreases
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models
Conclusions
I
“Multiple comparisons” is a real concern, but . . .
I
Don’t “fix” by altering p-values or (equivalently) by making
confidence intervals wider
I
Instead, multilevel modeling does partial pooling where
necessary (especially when much of the variation in the data
can be explained by noise), so that few claims can be made
with confidence
I
“Adjustments” are a dead end; “modeling” is forward-looking
Open questions
I
I
I
Structured models (classrooms, grades, teachers, ages, . . . )
Small number of groups
Andrew Gelman, Jennifer Hill, and Masanao Yajima
Multiple comparisons using multilevel models