EDF 802
Dr. Jeffrey Oescher
One Factor ANOVA Post Hoc Analyses
I. Description
A. An analysis of variance (ANOVA) is a procedure for examining the variance in
some dependent variable in terms of the portion of variance that can be attributed
to a specific independent variable. The independent variable is discussed as a
factor in the analysis, and since there is only one independent variable the
analysis is frequently referred to as a one factor ANOVA or a one-way ANOVA.
Very often there are more than two levels of the independent variable. That is, the
analysis is attempting to identify differences between three or more groups.
B. The ANOVA procedure determines whether or not a significance difference exists
between one or more pair of mean scores. This is known as the omnibus
ANOVA; it indicates there is a statistically different pair of means. Unfortunately
it does not provide any information as to which pair of means is different.
C. In a comparison of three groups, theses comparisons – known as pair-wise
comparisons – would involve the comparison of Group 1 with Group 2, Group 1
with Group 3, and Group 2 with Group 3. A post-hoc (i.e., after the fact) must be
used to identify the statistically significant pair or pairs of means.
II. Post hoc procedures
A. The reason for moving to an ANOVA rather than multiple t-tests was to maintain
the experiment-wise error rate (i.e., alpha). There are many post-hoc procedures,
each of which controls the experiment-wise error rate slightly differently.
1. Scheffe
a. Scheffe’s procedure accounts for all of the pair-wise comparisons as well
as all possible contrasts. In a comparison of three groups, this would
invole the comparisons of Group 1 with Group 2, Group 1 with Group 3,
and Group 2 with Group 3. In addition, it also controls for any contrasts. A
contrast is the comparison of two or more groups collectively with another
group. For example, consider the case where students in Groups 1 and 2
are receiving the same experimental treatment, while those students in
Group 3 are serving as a control group. There is no reason to compare
Group 1 with Group 2. Likewise, comparing Group 1 or Group 2 with
Group 3 does not make sense. What is appropriate is to compare the
combined effects of Groups 1 and 2 with the effects for Group 3.
Statistically, this can be represented by the following: (1 + 2) with 3. This
is known as a contrast. There are two other contrasts with three groups: (1
+ 3) with 2 as well as (2 + 3) with 1.
b. If you think of the control of alpha being dependent on the number of
“comparisons” being made, there are three one to one group comparisons
and three two groups to one group contrasts. This is six (6) “comparisons”
overall.
c. Using a common method for controlling alpha levels known as
Bonferroni’s inequality, the alpha level of .05 should be divided by six (6)
-1-
to maintain the experiment-wise error rate. The result is a comparisonwise error rate of .05/6.00 = .0083.
d. In reality, the Scheffe procedure adjusts for the inflated experiment-wise
error rate. You need only read the significance of a comparison or contrast
off of the printout.
2. Tukey
a. Tukey’s procedure accounts for all of the pair-wise comparisons. In a
three group ANOVA, this would involve the comparisons of Group 1 with
Group 2, Group 1 with Group 3, and Group 2 with Group 3.
b. Using Bonferroni to adjust the experiment-wise error rate, the alpha of .05
would be divided by three (3). The result is a comparison-wise error rate
of .05/3.00 = .0167.
c. In reality, the Tukey procedure adjusts for the inflated experiment-wise
error rate. You need only read the significance of a comparison off of the
printout.
3. There are other post-hoc procedures (e.g., Bonferroni, Duncan, LSD, Sydak)
each of which controls the experimental-wise error rate slightly differently.
B. Summarizing post-hoc procedures
1. A post-hoc procedure is need whenever you have a significant omnibus
ANOVA result.
2. An example of NOT needing a post-hoc analysis
a. The F-statistic was statistically non-significant (F2, 189 = 3.25, p = .214).
b. No further analyses were performed; there is no statistically significant
difference between any of the three pairs of group means.
3. An example of a post-hoc summary
a. The omnibus ANOVA result is reported: The F-statistic was statistically
significant (F4,26 = 11.19, p = .000).
b. The post-hoc comparisons, the mean differences between pairs, and the
significance levels are reported for all significant and non-significant
comparisons and contrasts.
i. Note the comparison between two groups can be summarized as Group
1 and 2 or Group 2 and 1. There is no need to report both.
ii. A post-hoc analysis using Scheffe’ procedure indicated statistically
significant differences between Groups 1 and 3 (MD = -3.23, p =
.041), 1 and 4 (MD = - 4.46, p = .008), 1 and 5 (MD = -6.86, p = .000),
2 and 5 (MD = -4.50, p = .010), and 3 and 5 (MD = -3.63, p = .037).
iii. Non-significant differences were found for the comparisons of Groups
1 and 2 (MD = -2.36, p = .264), 2 and 3 (MD = -0.86, p = .939), 2 and
4 (MD = -2.10, p = .482), 3 and 4 (MD = -1.23, p = .846), and 4 and 5
(MD = -2.40, p = .391).
a) SPSS reports the results in two formats, both of which are difficult
to interpret.
-2-
Table 1
SPSS Output for Post-Hoc Comparisons in the GLM Procedure
Multiple Comparisons
Trials
Scheffe
95% Confidence Interval
Mean
(I) group
(J) group
1.00
2.00
-2.3571
1.02022
.284
-5.7363
1.0220
3.00
-3.2321*
.94907
.041
-6.3756
-.0887
4.00
-4.4571*
1.07376
.008
-8.0136
-.9007
5.00
-6.8571*
1.07376
.000
-10.4136
-3.3007
1.00
2.3571
1.02022
.284
-1.0220
5.7363
3.00
-.8750
.99036
.939
-4.1552
2.4052
4.00
-2.1000
1.11041
.482
-5.7779
1.5779
5.00
-4.5000*
1.11041
.010
-8.1779
-.8221
1.00
3.2321*
.94907
.041
.0887
6.3756
2.00
.8750
.99036
.939
-2.4052
4.1552
4.00
-1.2250
1.04542
.846
-4.6876
2.2376
5.00
-3.6250*
1.04542
.037
-7.0876
-.1624
1.00
4.4571*
1.07376
.008
.9007
8.0136
2.00
2.1000
1.11041
.482
-1.5779
5.7779
3.00
1.2250
1.04542
.846
-2.2376
4.6876
5.00
-2.4000
1.15979
.391
-6.2414
1.4414
1.00
6.8571*
1.07376
.000
3.3007
10.4136
2.00
4.5000*
1.11041
.010
.8221
8.1779
3.00
3.6250*
1.04542
.037
.1624
7.0876
4.00
2.4000
1.15979
.391
-1.4414
6.2414
2.00
3.00
4.00
5.00
Difference (I-J)
Std. Error
Based on observed means.
The error term is Mean Square(Error) = 3.363.
*. The mean difference is significant at the .05 level.
-3-
Sig.
Lower Bound
Upper Bound
Table 2
SPSS Output for Post-Hoc Comparisons in the GLM Procedure
trials
Scheffea,b,c
Subset
group
N
1
2
3
1.00
7
3.1429
2.00
6
5.5000
5.5000
3.00
8
6.3750
6.3750
4.00
5
5.00
5
Sig.
7.6000
7.6000
10.0000
.083
.434
.302
Means for groups in homogeneous subsets are displayed.
Based on observed means.
The error term is Mean Square(Error) = 3.363.
a. Uses Harmonic Mean Sample Size = 5.991.
b. The group sizes are unequal. The harmonic mean of the
group sizes is used. Type I error levels are not guaranteed.
c. Alpha = .05.
III. SPSS syntax
A. An ANOVA and post-hoc analyses can be run using the COMPARE MEANS
ONEWAY or GLM UNIVARIATE procedures.
B. SPSS COMAPRE MEANS ONEWAY syntax using D3.SAV
1. ONEWAY trials BY group
/STATISTICS DESCRIPTIVES
/MISSING ANALYSIS
/POSTHOC=SCHEFFE ALPHA(0.05).
2. Note this procedure does not have options for homogeneity of variance,
power, effect size, etc.
C. SPSS GLM UNIVARIATE syntax using D3.SAV
1. UNIANOVA trials BY group
/METHOD=SSTYPE(3)
/INTERCEPT=INCLUDE
/POSTHOC=group(SCHEFFE)
/EMMEANS=TABLES(group)
/PRINT=OPOWER ETASQ HOMOGENEITY DESCRIPTIVE
/CRITERIA=ALPHA(.05)
/DESIGN=group.
2. This procedure has many options available and tis the preferred procedure for
a one factor ANOVA.
-4-