Laboratory for Simultaneous Comparisons
Purpose: In this lab you’ll gain experience in computing and evaluating null hypotheses
primarily related to post hoc comparisons (contrasts).
1. Type I Error
For any analysis, you can set the per-comparison error rate ( to a specific value
(typically .05). However, what happens if you are going to conduct a series of analytical
comparisons? We would describe the comparisons as a family of tests and define familywise
Type I error rate (FW) as “the probability of making at least one Type I error in the family
of tests when all the null hypotheses are true.”
FW = 1 – (1 – )c
For the comparisons shown below, compute FW for the two per-comparison error rates.
Number of Comparisons (c)
 = .05
 = .01
3
5
9
2. Restricted Sets of Contrasts
a. The Bonferroni Procedure
Given the Bonferroni inequality (FW < c), we can control FW. That is, we would pick a
per-comparison  that divides the FW equally among the tests. What per-comparison 
would you use for each of the families of comparisons shown below, given FW of either .05
or .10?
Number of Comparisons (c)
FW = .05
FW = .10
3
5
9
If you are dealing with a computer output for your comparison that provides you with a
usable p-value, then to determine whether or not the comparison is significant, you would
simply compare your obtained p-value to the Bonferroni per-comparison critical p-value. If it
Simultaneous Comparisons - 1
is smaller than the Bonferroni p, then you would reject H0 and conclude that the two means
were drawn from populations with different .
However, you often need to construct an FComparison (using a MSComparison from a computer
output and a MSError from a different output that includes all of your conditions), which
means that you won’t know a p-value, but only your FComparison value. How do you determine
if the FComparison is significant?
Let's assume that n = 11 for a study with a = 5. Furthermore, let’s assume homogeneity of
variance, so that you’ll always use MSError from the overall analysis. You decide to use the
Bonferroni procedure to assess four comparisons, keeping FW to .05. Thus, your percomparison  =
. The four comparisons yield Fs as seen below. In each case, assess H0.
Note that your per-comparison  does not appear in the table of critical values of F. In some
cases, that’s not a problem because you can use the table of critical values of F and some
logic to make your decision.
FComparison
Decision & Why
7.6
5.0
6.7
7.0
However, in some cases, you may need to make use of a formula that uses your percomparison  to determine the FCritical value. Well, actually the formula gives you a t value,
which you will then need to square to determine the FCritical. Here is the formula:
z3  z
t df S / A   z 
4 df S / A  2
Because you’re actually computing a two-tailed test, you're determining the z based on
splitting your -level into two tails (half). In this case, with  = 0.125, you would place
.00625 in each tail (for
a cumulative probability of .99375).
Simultaneous Comparisons - 2
b. The Sidák-Bonferroni Procedure
The Sidák-Bonferroni procedure is very similar to the Bonferroni procedure, but it is more
powerful. That is, it will result in an -level that is higher than that found with the Bonferroni
procedure. Using the formula below, compute the -level for the situations in the table (as
you had done earlier for the Bonferroni procedure).
 = 1 – (1 – FW)1/c
Number of Comparisons (c)
FW = .05
FW = .10
3
5
9
Okay, now that you’ve shown that you can do the computations, you can check your work by
using the shortcut that eliminates the need for computation.  Go to Appendix A.4 (pp. 579580 for these FW levels) and look up the -level (at the top of the table, in percentage form).
Again, let's assume that n = 11 for a study with a = 5. Furthermore, let’s assume
homogeneity of variance, so that you’ll always use MSError from the overall analysis. You
decide to use the Sidák-Bonferroni procedure to assess four comparisons, keeping FW to .05.
The four comparisons yield Fs as seen below. In each case, assess H0. For this exercise,
however, you’ll be able to make use of Appendix A.4 to determine FCritical for your
comparisons. However, the tables for Sidák-Bonferroni show critical values of t, so you need
to square the value to turn it into F.
FComparison
Decision & Why
7.6
5.0
6.7
7.0
It’s also possible to construct a critical mean difference for the Sidák-Bonferroni procedure.
Doing so would allow you to easily compare pairs of means, so it may be a preferable
Simultaneous Comparisons - 3
procedure in situations where you have a lot of simple comparisons to make. The formula
would make use of the tSidák-Bonferroni from Appendix A.4 to compute a critical difference (D).
DSidákBonferroni  t SidákBonferroni
2MSError
n
Thus, for the above study, let’s assume that MSError = 5 and two conditions you wanted to
compare (of the four total comparisons) yielded means of 5 and 8 (i.e., difference is 3).
 be significant?
Would that difference
Critical Difference
Decision
c. Dunnett’s Test
If you’ve conducted an experiment in which a control group is going to be compared to a set
of experimental groups, you’d probably make use of Dunnett’s test. This test will be a bit
more powerful than the Sidák-Bonferroni procedure. You’d use a new appendix (A.5), but
the procedure is quite similar to that used in the Sidák-Bonferroni procedure.
Again, let's assume that n = 11 for a study with a = 5. Furthermore, let’s assume
homogeneity of variance, so that you’ll always use MSError from the overall analysis. The
first condition is a control condition, to which you wish to compare the other two conditions.
Thus, you would use the Dunnett’s test to assess two comparisons (1 vs. 2 and 1 vs. 3). If you
want to keep FW to .05, determine your tCritical and then translate into FCritical. The two
comparisons yield Fs as seen below. In each case, assess H0.
Comparison
FComparison
1 vs. 2
7.6
1 vs. 3
5.0
1 vs. 4
6.7
1 vs. 5
7.0
Decision & Why
Simultaneous Comparisons - 4
You can also compute a critical mean difference (D) for Dunnett’s test:
DDunnett  t Dunnett
2MSError
n
3. Pairwise Comparisons
a. Tukey’s HSD Procedure

You need Appendix A.6 to compute Tukey’s HSD. The tables show the Studentized Range
Statistic (q) for various familywise error rates. Thus, your first step is determine the FW you
want to use. The number of conditions (number of means) and the dfError determine the value
of q you use. With q, you can compute FCritical for a Tukey HSD comparison as:
FHSD 
q2
2
Alternatively, you could compute a critical mean difference as:

DHSD  q
MSError
n
K&W51 does not exhibit heterogeneity of variance. Thus, any comparisons you would
conduct would use the pooled error term (MSError = 150.46).

Use the critical mean difference approach to make all simple pairwise comparisons. First,
compute the critical difference, then compute the actual differences below and determine
which ones are significant.
4hr (26.5)
12hr (37.75)
20hr (57.5)
28hr (61.75)
4hr
-----
12hr
20hr
28hr
-------------
b. The Fisher-Hayter Procedure
The Fisher-Hayter procedure is identical to the Tukey procedure (above) with one important
change. When looking up the q, you use the dfTreatment, rather than the number of conditions.
As a result, the Fisher-Hayter approach will be more powerful than the Tukey approach.
Compare the two procedures below:
FCritical
Tukey HSD
Fisher-Hayter
4. Using SPSS
Simultaneous Comparisons - 5
Critical Mean Difference
Okay, if you have a sense of the computations you’ve used thus far, you’ll be delighted to
learn that SPSS can make your life easier. For the K&W51 data set, compute Bonferroni,
Sidák, and Tukey HSD. Not so bad, eh? Unfortunately, you will not always be able to use the
One-Way procedure, so you need to know how to compute the comparisons as you did
above.
Simultaneous Comparisons - 6
Simultaneous Comparisons - 7