Testing Differences Among
Several Sample Means
Multiple t Tests vs. Analysis of
Variance
Several Sample Means
• What might we do if we had more than
two samples?
Sample 1 Sample 2 Sample 3
X21
X31
X11
X22
X32
X12
.
.
.
.
.
.
.
.
.
X2n
X3n
X1n
means:
X1
X2
X3
Several Sample Means
• Specifically how many t Tests could you
do?
Sample 1 Sample 2 Sample 3
X21
X31
X11
X22
X32
X12
.
.
.
.
.
.
.
.
.
X2n
X3n
X1n
means:
X1
X2
X3
Several Sample Means
• Specifically how many t Tests could you
do?
Sample 1 Sample 2 Sample 3
X21
X31
X11
X22
X32
X12
.
.
.
.
.
.
.
.
.
X2n
X3n
X1n
means:
X1
X2
X3
Several Sample Means
• Specifically how many t Tests could you
do?
Sample 1 Sample 2 Sample 3
X21
X31
X11
X22
X32
X12
.
.
.
.
.
.
.
.
.
X2n
X3n
X1n
means:
X1
X2
X3
Several Sample Means
• Specifically how many t Tests could you
do?
Sample 1 Sample 2 Sample 3
X21
X31
X11
X22
X32
X12
.
.
.
.
.
.
.
.
.
X2n
X3n
X1n
means:
X1
X2
X3
Multiple t Tests and Familywise Error Rate
• If you do all possible pair-wise
comparisons (C), what happens to the
overall probability of making a Type I
error?
Multiple t Tests and Familywise Error Rate
• If you do all possible pair-wise
comparisons (C), what happens to the
overall probability of making a Type I
error?
Family-wise Error Rate =

 (C)
Multiple t Tests and Familywise Error Rate
• How could we prevent the family-wise
error rate from exceeding .05 ?
Multiple t Tests and Familywise Error Rate
• How could we prevent the family-wise
error rate from exceeding .05 ?
• Set the alpha-level for each pair-wise t
test to be a fraction of .05; specifically:
 pair 
 family
C
Multiple t Tests and
Familywise Error Rate
• This isn’t usually done in practice because
only a few different alpha-levels appear in the
t tables
Multiple t Tests and
Familywise Error Rate
• This isn’t usually done in practice because
only a few different alpha-levels appear in the
t tables
• More importantly, consider that C increases
dramatically as more samples are added
– for 4 samples: C = 6
– for 5 samples: C = 10
– for 6 samples: C = 15
• Which leads to a precipitous drop in power
Analysis of Variance
• What is needed is a technique that
controls family-wise error rate while
looking for one or more differences
between several sample means
Analysis of Variance
• What is needed is a technique that
controls family-wise error rate while
looking for one or more differences
between several sample means
• That technique is a one-way Analysis of
Variance (ANOVA)
Analysis of Variance
• Here are three samples, each are
measurements under different treatment
conditions:
Sample 1 Sample 2 Sample 3
X21
X31
X11
X22
X32
X12
.
.
.
.
.
.
.
.
.
X2n
X3n
X1n
means:
X1
X2
X3
Each sample has a
mean and variance and
the 3 means are a
sampling distribution
of means
Analysis of Variance
Sample 1 Sample 2 Sample 3
X21
X31
X11
X22
X32
X12
.
.
.
.
.
.
.
.
.
X2n
X3n
X1n
means:
X1
X2
X3
• What would the null hypothesis be?
• What
would
hypothesis

 the alternative

be?
Analysis of Variance
• What would the null hypothesis be?
– All three samples are taken from the same population so:
1  2  3  

Analysis of Variance
• What would the null hypothesis be?
– All three samples are taken from the same
population so:
1  2  3  
• What would the alternative hypothesis
be?
– Atleast one of the samples is from a different
population and hence has a different mean
Analysis of Variance
• We can estimate the variance of the “null
hypothesis” population by averaging the j
variance estimates
Analysis of Variance
• This is called the “Mean Square Error” or
“Mean Square Within”
k
2
ˆ
j
MSerror 
j1
k
in our example:

2
2
2
ˆ
ˆ
ˆ
1  2  3
MS error 
3
Analysis of Variance
• MSerror is an estimate of the population
variance
2
2
2
MS error 

ˆ 1  
ˆ 2  
ˆ3

3
Analysis of Variance
• MSerror is an estimate of the population
variance
2
2
2
MS error 
ˆ 1  
ˆ 2  
ˆ3

3
• What’s another way we could estimate
 the population variance (hint: assume the
null hypothesis is true)?
Analysis of Variance
• Each sample has a mean and variance and the
3 means are a sampling distribution of means
Sample 1 Sample 2 Sample 3
X21
X31
X11
X22
X32
X12
.
.
.
.
.
.
.
.
.
X2n
X3n
X1n
means:
X1
X2
X3
Analysis of Variance
• Recall that we estimated the variance of a sampling distribution of
means (since we only had one sample) using the equation:
ˆ 

2
X

ˆ

2
n
Analysis of Variance
• Now we’ve got more than one sample! So we can turn this
equation around and make an estimate of the population variance
called the “Mean Square Effect” or “Mean Square Between”:
2
ˆ
X 
ˆ2

n
k
(X
ˆ  n
ˆ n
MSeffect 
2
2
X
j
 X overall )
j1
k 1
2
Analysis of Variance
• We now have two different estimates of
the population variance: MSerror and
MSeffect
• Why might these two estimates
disagree?
Analysis of Variance
• MSerror is based on deviation scores
within each sample but…
Analysis of Variance
• MSerror is based on deviation scores
within each sample but…
• MSeffect is based on deviations between
samples
Analysis of Variance
• MSerror is based on deviation scores
within each sample but…
• MSeffect is based on deviations between
samples
• MSeffect would overestimate the
population variance when…
Analysis of Variance
• MSerror is based on deviation scores
within each sample but…
• MSeffect is based on deviations between
samples
• MSeffect would overestimate the
population variance when…there is
some effect of the treatment pushing
the means of the different samples
apart
Analysis of Variance
• We compare MSeffect against MSerror by
constructing a statistic called F
Analysis of Variance
• We compare MSeffect against MSerror by
constructing a statistic called F
• If the hull hypothesis:
1  2  3  
is true then we would expect:
X1  X 2  X 3  

except for random sampling variation
Analysis of Variance
• F is the ratio of MSeffect to MSerror
Fk1,k(n1) 

MSeffect
MS error
Analysis of Variance
• F is the ratio of MSeffect to MSerror
Fk1,k(n1) 
MSeffect
MS error
• If the null hypothesis is true then F
 should equal 1.0
Analysis of Variance
• Of course there is a sampling
distribution of F - if you repeated your
experiment many times you would get a
distribution of Fs
Analysis of Variance
• Of course there is a sampling
distribution of F - if you repeated your
experiment many times you would get a
distribution of Fs
• The shape of that distribution depends
on two different degrees of freedom:
– MSeffect has k-1 degrees of freedom
– MSerror has k(n-1) degrees of freedom
Analysis of Variance
• We can look up a critical F from an F table for
any given number of degrees of freedom
Analysis of Variance
• We can look up a critical F from an F table for
any given number of degrees of freedom
• If the F statistic we’ve obtained in our
experiment exceeds Fcrit then we know that
fewer than 5% of such experiments would be
likely to obtain this F statistic if the null
hypothesis was true
Analysis of Variance
• We can look up a critical F from an F table for
any given number of degrees of freedom
• If the F statistic we’ve obtained in our
experiment exceeds Fcrit then we know that
fewer than 5% of such experiments would be
likely to obtain this F statistic if the null
hypothesis was true
• So we can reject the null and conclude that at
least one pair of means is different