Multiple comparisons
http://newyorkette.com/wp-content/070430_cartoon_8_a12131_p465.gif
How do we decide which means
are truly different from one
another?
• ANOVAs only test the null hypothesis
that the treatment means were all
sampled from the same distribution…
Two general approaches
• A posteriori comparisons: unplanned, after
the fact.
• A priori comparisons: planned, before the
fact.
Effects of early
snowmelt on alpine
plant growth
Three treatment groups and 4 replicates per
treatment:
1.
Unmanipulated
2.
Heated with permanent solar-powered
heating coils that melt spring snow pack
earlier in the year than normal
3.
Controls, fitted with heating coils that are
never activated
Data
Unmanipulated
Control
Heated
10
12
12
13
9
11
11
12
Y2  10.75
12
13
15
16
Y3  14.00
Y1  11 .75
Y  12.17
ANOVA table for one-way layout
Source
df
Sum of
squares
Mean
square
Among
groups
2
22.16
11.08
Within
groups
9
19.50
2.17
11
41.67
Total
P=tail of F-distribution with (a-1) and a(n-1)
degrees of freedom
F-ratio
5.11
P-value
0.033
A posteriori comparisons
• We will use Tukey’s “honestly significant
differences” (HSD)
• It controls for the fact that we are carrying
out many simultaneous comparisons
• the P-value is adjusted downward for each
individual test to achieve an experimentwise error rate of α=0.05
The HSD
1 1
   MS residual
n n 
j 
 i
HSD  q
2
Where:
q= is the value from a statistical table of the studentized range
distribution
n = sample size
The HSD
1 1
   2.17
4 4
HSD  3.199
 2.3562
2
Y3  14.00
2.25, NS
Y1  11 .75
3.25,
P<0.05
1, NS
Y2  10.75
So we find significant differences between the control and the heated and no
significance between unmanipulated vs. control, or unmanipulated vs. heated…
Dunnett’s test
SEt  MS residual
1 1
  
n n 
j 
 i
For each treatment vs. control pair, CV= d(m,df) SEt;
Where m (number of treatments) includes the control and
d is found in tables for Dunnett’s test.
Use Dunnett’s test by comparing treatment furthest from
control first, then next furthest from control, etc.
Dunnett’s test
 2
SEt  2.17    1.0416
 4
d control_ vs _ heated
dtest 
X control  X treatment
SEt
10.75  14

 3.1201
1.0416
d control_ vs _ unmanipulated
10.75  11.72

 0.96003
1.0416
d-critical(m=3,df=9)=2.61
But…
• Occasionally posterior tests may indicate
that none of the pairs of means are
significantly different from one another,
even if the overall F-ratio led to reject the
null-hypothesis!
• This inconsistence results because the
pairwise test are not as powerful as the
overall F-ratio itself.
A priori (planned) comparisons
• They are more specific
• Usually they are more powerful
• It forces you to think clearly about which
particular treatment differences are of
interest
A priori (planned) comparisons
• The idea is to establish contrasts, or
specified comparisons between particular
sets of means that test specific hypothesis
• These test must be orthogonal or
independent of one another
• They should represent a mathematical
partitioning of the among group sum of
squares
To create a contrast
• Assign an number (positive, negative or 0)
to each treatment group
• The sum of the coefficients for a particular
contrast must equal 0 (zero)
• Groups of means that are to be averaged
together are assigned the same coefficient
• Means that are not included in the
comparison of a particular contrast are
assigned a coefficient of 0
Contrast I (Heated vs. Non-Heated)
Unmanipulated (1)
Y1  11 .75
Control (1)
Heated (-2)
Y2  10.75
Y3  14.00
a
MS contrast 
MS heated _ vs _ unheated
n( ciYi ) 2
i 1
a

2
c
i 1 i
4  ((1)(11.75)  (1)(10.75)  (2)(14.00)) 2

 20.16
2
2
2
1 1  2
F-ratio= 20.16/2.17=9.2934
With 1 df
F-critical1,9 =5.12
Contrast I (Heated vs. Non-Heated)
using formula for non equal samples
Unmanipulated (1)
Y1  11 .75
k

Control (1)
Heated (-2)
Y2  10.75
Y3  14.00

SS contrast   ni Yi  Y , k _ groups
2
i
MS heated _ vs _ unheated  8(11.25 12.167) 2  (4)(14.00 12.167) 2  20.16
F-ratio= 20.16/2.17=9.2934
With 1 df
F-critical1,9 =5.12
More information on Sokal and Rohlf (2000) Biometry
Contrast II (Control vs. Unmanipulated)
Unmanipulated (1)
Y1  11 .75
Control (-1)
Y2  10.75
Heated (0)
Y3  14.00
a
MS contrast 
n( ciYi ) 2
i 1
a

0  (1)(1)  (1)( 1)  (2)(0)
2
c
i
i 1
4  ((1)(11.75)  (1)(10.75)  (0)(14.00)) 2
MS unmanipulated _ vs _ control 
2
2
2
1 1
F-ratio= 2/2.17=0.9217
With 1 df
F-critical1,9 =5.12
In order to create additional
orthogonal contrasts
• If there are a treatment groups, at most there can
be (a-1) orthogonal contrasts created (although
there are many possible sets of such orthogonal
contrasts).
• All of the pair-wise cross products must sum to
zero. In other words, a pair of contrast Q and R is
independent if the sum of the products of their
coefficients CQi and CRi equals zero.

a
n
c
c

0
i
Qi
Ri
i 1
A priori contrasts
Source
df
SS
MS
F-ratio
P
2
22.16
11.08
2.11
0.033
Heated vs. Non-Heated
1
20.16
20.16
9.29
<0.025
Control vs. Unmanipulated
1
2.00
2
0.92
NS
9
19.50
2.17
11
41.66
Treatments
Residual