Multiple Comparisons Among Means
Author(s): Olive Jean Dunn
Source: Journal of the American Statistical Association, Vol. 56, No. 293 (Mar., 1961), pp. 5264
Published by: American Statistical Association
Stable URL: http://www.jstor.org/stable/2282330
Accessed: 06/07/2010 05:39
Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.
Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
http://www.jstor.org/action/showPublisher?publisherCode=astata.
Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
American Statistical Association is collaborating with JSTOR to digitize, preserve and extend access to Journal
of the American Statistical Association.
http://www.jstor.org
MULTIPLE
COMPARISONS
AMONG MEANS
OLIVE JEAN DUNN
University
of California,Los Angeles
Methods for constructingsimultaneousconfidenceintervalsfor all
possiblelinear contrastsamong several means of normallydistributed
variables have been given by Scheff6and Tukey. In this paper the
possibilityis consideredof pickingin advance a number (say m) of
linear contrastsamong k means, and then estimatingthese m linear
contrastsby confidenceintervalsbased on a Studentt statistic,in such
a way that the overall confidencelevel forthe m intervalsis greater
than or equal to a preassignedvalue. It is foundthat forsome values
of k, and form not too large,intervalsobtainedin thisway are shorter
or the Studentizedrange.Whenthis
thanthoseusingthe F distribution
may be willingto selectthe linearcombinations
is so, the experimenter
in advance which he wishes to estimatein order to have m shorter
intervalsinsteadof an infinitenumberoflongerintervals.
1. INTRODUCTION
THERE
simulworkdone on the problemof finding
has been considerable
taneous confidenceintervalsfor a numberof linear contrastsamong several means fornormallydistributedvariables. Scheff6[1] gives a methodfor
constructingsimultaneousconfidenceintervalsforall possible linear contrasts
among k means using the F distribution.Tukey's intervals for all possible
linear contrastsamong k means use the distributionof the Studentizedrange
[2]. Each of these methods may be extendedto give confidenceintervalsfor
all possible linear combinationsof the k means, as opposed to linear contrasts
only.
In this paper the possibilityis consideredof pickingin advance a number
(say m) of linear combinationsamong the k means, and then estimatingthese
m linear combinationsby confidenceintervalsbased on a Student t statistic,
so that the overall confidencelevel forthe m intervalsis greaterthan or equal
to a preassignedvalue, 1-ax. It is possible that forsome values of k, and for
m not too large, intervalsobtained in this way may be shorterin some sense
than those usilngthe F distributionor the Studentizedrange. If this is so, the
may be willingto selectthe linearcombinationsin advance which
experimenter
he wishesto estimatein orderto have m shorterintervalsinstead of an infinite
numberof longerintervals.
The purpose of this paper, then, is to suggestand evaluate a simple use of
the Student t statisticfor simultaneousconfidenceintervalsforlinear combinationsamong several means, and to see underwhat conditionsthese intervals
apply. The study was actually made withlinear contrastsin mind,since these
are probablyestimatedmorefrequentlythan otherlinear combinationsamong
means. The paper has been written,however,in termsof arbitrarylinear combinations,in order to stress the fact that the method is not limited to contrasts.
The method given here is so simple and so general that I am sure it must
have been used beforethis. I do not findit, however,so can only concludethat
52
53
MULTIPLE COMPARISONS AMONG MEANS
has keptstatisticians
perhapsits verysimplicity
fromrealizingthat it is a
In anycase,theusersofstatistics
verygoodmethodin somesituations.
in the
mainseemunawareofit,so I feelthatit is worthpresenting.
2. TIE
Let the k means be l,, *
METHOD
i,,and let the estimatesforthem be
1Ab *
with means ,u *,
whichare normallydistributed
,k and withvariancesatii2,i=1
* , k;let thecovariancebetween-i}and A,be a jo2 fori#j.
Here the aii and aGiare assumedto be known,but e maybe unknown.Let
02 be an estimate
of a2 whichis statistically
of k41, , fk and
independent
suchthat va2/affollowsa Chi-squaredistribution
withv degreesof freedom.
in obtaining
(Theseconditions
are exactlythoseusedby Scheff6
his intervals
[1]. The condition
thatthe dispersion
matrixofthe -i be knownexceptfora
in orderto construct
factora2 is necessary
t statistics
whicharefreeofnuisance
parameters.)
Let the m linearcombinations
ofthe meanswhichare to be estimatedbe:
Ilk,
Os =
Cl101
+
+
(I)
*,=m.
-s12t'
CksIk,
A linearcombination
is, in particular,
a linearcontrastif
=
is=1
The unbiased estimatesfor Qi, .
? -
CIAl
+
.
, amare
*+
s
Ckk,
= 1, 2, **,m.
(2)
These are m normallydistributed
variables,and the varianceofC is b2o-2,
where
Ak
b8
k
F, 1: aijxcisj,
i=l
j=-
Thisreducestheproblemto oneI discussedearlier[3], offinding
confidence
intervalsforthemeansofm normally
distributed
variables.
I The variatest1,- * *, tn,each ofwhichfollowsa t distribution
withvdegrees
of freedom,
are formed:
-
to
0,
b80.
sM1,
(3)
**m.
The variatest1,t2,
t4,,have some joint distribution
function;usinga
Bonferroni
inequality,
one can obtaina lowerlimitto theprobability
thatall
theti'sliebetween-c and +c (wherec is anypositiveconstant)without
knowing anythingabout thisjoint distribution
exceptthat all the marginalsare
Studentt distributions.
Thus
P[-c
< ti < c i
=
1,2,
inJ >
1- 2m
ff()(t)dt,
(4)
54
AMERICAN
STATISTICAL
ASSOCIATION
JOURNAL,
MARCH 1961
wheref(")(t) is the frequencyfunctionfora Studentt variable with v degreesof
freedom.
If c is selected so that the righthand memberof (4) equals 1-a, then confidenceintervalswithlevel 1- a are obtained from
P -c <
,s
,m] > 1-.(5)
,
They are
s =1, 2, * *,m.
08 ? cb8&
(6)
Here the overallconfidencelevel forthe m linearcombinationsis 1- a, where
c is definedby
00
a
f(y)(t)dt
f
--
2m
Y and f(0)(t)
is the frequencyfunctionofa Studenttvariable withvdegreesoffreedom.Some
or all of theselinearcombinationsmay of coursebe linearcontrasts.
, , are the sample means pi,
When k,
Pk
Pk, and when pi,
are statisticallyindependent,then ai = i/ni, whereni is the size of the sample
for the yi's; for i-j, aij=O. The confidenceintervals for cl.l+
+Ck8.Ak
become
(C8UP1+
+ Ck,Pk) ?
s = 1, *
C4/"Ec,,/f6,
, m.
(7)
Table 1 gives values of c for 1- a .95 and forvarious values of v and of m;
Table 2 givesc for1- a = .99. These tables as well as the othertables appearing
in thispaper have been computedfromBiometrikaTablesforStatisticians,Pearson and Hartley [4].
3. COMPARISON
WITH INTERVALS
USING
F DISTRIBUTION
Scheff6'sintervalsforany numberoflinearcontrastsamong k means are
08
? Sbs&
(8)
where S2=(k-1)F,(kI-1,
v). Here Fa,(k-1, v) is the 1-a point of the F distributionwith k-1 and v degrees of freedom,and the other symbolsare definedas in Section 2.
When intervalsfora numberoflinearcombinationsare desired(not restricting them to linear contrasts),the intervalsare as given in (8), but with S2
kFa,(k, v).
Since the t intervalsin (6) and F intervalsin (8) are seen to be of exactlythe
same form,and requireexactlythe same assumptions,it is botheasy and useful
to compare them.
the set of linear
The main difference
betweenthemis that in the t-intervals,
combinationswhichare to be estimatedmust be planned in advance, whereas
with Scheff6'sintervalsthey may be selected afterlooking at the data, since
Scheff6'smethodgives intervalsforall possiblellnearcombinationsof k means.
MULTIPLE
55
AMONG MEANS
COMPARISONS
TABLE 1. VALUES OF c 'FOR 1-a=.95
c
m\
2
3
4
5
6
7
8
9
10
15
20
25
30
35
40
45
50
100
250
*
.05
f(1)(t)dt =1f
~2m
00
5
7
10
12
15
20
24
30
40
60
120
00
3.17
3.54
3.81
4.04
4.22
4.38
4.53
4.66
4.78
5.25
5.60
5.89
6.15
6.36
6.56
6.70
6.86
8.00
9.68
2.84
3.13
3.34
3.50
3.64
3.76
3.86
3.95
4.03
4.36
4.59
4.78
4.95
5.09
5.21
5.31
5.40
6.08
7.06
2.64
2.87
3.04
3.17
3.28
3.37
3.45
3.52
3.58
3.83
4.01
4.15
4.27
4.37
4.45
4.53
4.59
5.06
5.70
2.56
2.78
2.94
3.06
3.15
3.24
3.31
3.37
3.43
3.65
3.80
3.93
4.04
4.13
4.20
4.26
4.32
4.73
5.27
2.49
2.69
2.84
2.95
3.04
3.11
3.18
3.24
3.29
3.48
3.62
3.74
3.82
3.90
3.97
4.02
4.07
4.42
4.90
2.42
2.61
2.75
2.85
2.93
3.00
3.06
3.11
3.16
3.33
3.46
3.55
3.63
3.70
3.76
3.80
3.85
4.15
4.56
2.39
2.58
2.70
2.80
2.88
2.94
3.00
3.05
3.09
3.26
3.38
3.47
3.54
3.61
3.66
3.70
3.74
4.04
4.4*
2.36
2.54
2.66
2.75
2.83
2.89
2.94
2.99
3.03
3.19
3.30
3.39
3.46
3.52
3.57
3.61
3.65
3.90
4.2*
2.33
2.50
2.62
2.71
2.78
2.84
2.89
2.93
2.97
3.12
3.23
3.31
3.38
3.43
3.48
3.51
3.55
3.79
4.1*
2.30
2.47
2.58
2.66
2.73
2.79
2.84
2.88
2.92
3.06
3.16
3.24
3.30
3.34
3.39
3.42
3.46
3.69
3.97
2.27
2.43
2.54
2.62
2.68
2.74
2.79
2.83
2.86
2.99
3.09
3.16
3.22
3.27
3.31
3.34
3.37
3.58
3.83
2.24
2.39
2.50
2.58
2.64
2.69
2.74
2.77
2.81
2.94
3.02
3.09
3.15
3.19
3.23
3 .26
3.29
3.48
3.72
Obtainedby graphicalinterpolation.
This is, ofcourse,a considerableadvantage forSchef6's method.It is, however,
possiblein usingthe t-intervalsto select as the intervalsto be estimateda very
large set of linear combinationswhichincludesall those whichmightconceivably be of interest.Then, on lookingat the data, one may decide on actually
TABLE 2. VALUES OF c FOR 1-c =.99
.01
?
r
f f(')(t)dt = 1
2m
2
3
4
5
6
7
8
9
10
15
20
25
30
35
40
45
50
100
250
*
5
7
10
12
15
20
24
30
40
60
120
4.78
5.25
5.60
5.89
6.15
6.36
6.56
6.70
6.86
7.51
8.00
8.37
8.68
8.95
9.19
9.41
9.68
11.04
13.26
4.03
4.36
4.59
4.78
4.95
5.09
5.21
5.31
5.40
5.79
6.08
6.30
6.49
6.67
6.83
6.93
7.06
7.80
8.83
3.58
3.83
4.01
4.15
4.27
4.37
4.45
4.53
4.59
4.86
5.06
5.20
5.33
5.44
5.52
5.60
5.70
6.20
6.9*
3.43
3.65
3.80
3.93
4.04
4.13
4.20
4.26
4.32
4.56
4.73
4.86
4.95
5.04
5.12
5.20
5.27
5.70
6.3*
3.29
3.48
3.62
3.74
3.82
3.90
3.97
4.02
4.07
4.29
4.42
4.53
4.61
4.71
4.78
4.84
4.90
5.20
5.8*
3.16
3.33
3.46
3.55
3.63
3.70
3.76
3.80
3.85
4.03
4.15
4.25
4.33
4.39
4.46
4.52
4.56
4.80
5.2*
3.09
3.26
3.38
3.47
3.54
3.61
3.66
3.70
3.74
3.91
4.04
4.1*
4.2*
4.3*
4.3*
4.3*
4.4*
4.7*
5.0*
3.03
3.19
3.30
3.39
3.46
3.52
3.57
3.61
3.65
3.80
3.90
3.98
4.13
4.26
4.1*
4.2*
4.2*
4.4*
4.9*
2.97
3.12
3.23
3.31
3.38
3.43
3.48
3.51
3.55
3.70
3.79
3.88
3.93
3.97
4.01
4.1*
4.1*
4.5*
4.8*
2.92
3.06
3.16
3.24
3.30
3.34
3.39
3.42
3.46
3.59
3.69
3.76
3.81
3.84
3.89
3.93
3.97
2.86
2.99
3.09
3.16
3.22
3.27
3.31
3.34
3.37
3.50
3.58
3.64
3.69
3.73
3.77
3.80
3.83
4.00
Obtainedby graphicalinterpolation.
2.81
2.94
3.02
3.09
3.15
3.19
3.23
3.26
3.29
3.40
3.48
3.54
3.59
3.63
3.66
3.69
3.72
3.89
4.11
56
AMERICAN STATISTICAL ASSOCIATION JOURNAL, MARCH 1961
computingintervalsfor only some of this set. Section 5 gives an example of
this procedure.
A second difference
betweenthe methodsis that the lengthsofthe t-intervals
depend on m, the number of linear combinations,whereas with Scheff6's
methodthe lengthsdepend on k, the numberof means. It seems reasonable to
suspect,then,that the t-ilntervals
may be shorterforsmall m and large k, and
TABLE 3. VALUES OF
\
\
C2/S2
FOR 1-a=.95,.99
1 - a - .95
k
2
5
10
1 -a = .99
15
20
2
5
10
15
20
1.33
1.87
2.38
.52
.73
.93
.27
.38
.48
.18
.26
.33
.55
.68
.88
.14
.19
.25
.42
.52
.66
v =7
2
5
10
50
100
250
1.44
2.19
2.91
5.22
6.61
8.77
.49
.74
.99
1.77
2.24
2.97
.24
.37
.49
.88
1.12
1.48
.16
.25
.33
.59
.75
.99
.12
.19
.25
.44
.56
.75
4.00
4.97
6.36
1.56
1.94
2.48
.81
1.00
1.28
1.23
1.56
1.83
2.57
.56
.71
.84
1.17
.32
.40
.48
.67
.23
.29
.33
.48
.18
.22
.26
.37
1.19
1.44
1.63
2.09
2.28
2.55
.59
.72
.82
1.04
1.14
1.27
.36
.44
.50
.64
.70
.78
.27
.33
.37
.48
.52
.58
.22
.26
.30
.38
.42
.47
v =20
2
5
10
50
100
250
1.35
1.87
2.30
3.41
3.96
4.78
.51
.71
.87
1.29
1.50
1.81
.27
.38
.46
.69
.80
.97
.19
.26
.32
.48
.55
.67
.14
.20
.25
.37
.43
.51
2.84
3.34
1.30
1.53
.74
.87
.53
.62
.41
.48
p=00
2
5
10
50
100
250
1.31
1.73
2.06
2.82
3.15
3.60
.53
.70
.83
1.14
1.28
1.46
.30
.39
.47
.64
.72
.82
.21
.28
.33
.46
.51
.58
.17
.22
.26
.36
.40
.46
and small k. This turnsout to be
that the F intervalsmay be shorterforlarge mn
true.
Perhaps the most appealing way of comparingthe two methods,fromthe
standpointof the researchworker,is on the basis oflength.To do this,Table 3
gives values of c2/S2forcertainvalues of k and m, forv=7, 20 and oo,and for
v), so that the table is
1-a-=.95 and .99. Here S2 is definedas (k-1)F,(k-1,
applicable as it stands when linear contrastsare being estimated.The square
root of C2/S2 is the ratio of the lengthof a t-intervalcomparedto the lengthof
the correspondingScheff6interval. Thus for 1- a = .95, v= o, m= 50 and
k= 10, one has c2/S2=.64. This means that if one wishesto estimate50 linear
contrastsamong 10 means,that each of the 50 t-intervalsis .8 timesas long as
MULTIPLE
COMPARISONS
57
AMONG MEANS
the correspondingintervalusing the F distribution.A second interpretationis
that about 64 per cent as many observationsare necessaryusingthe t-intervals
to obtain the same precisionas usingthe F distribution.
If one is estimatinglinear combinationsamong means ratherthan simply
linear contrasts,one entersthe table witha k value increasedby one. Thus for
50 linear combinationsamong 9 means, with 1 -a=. 95 and v= , one has
c
S-.64.
=
TABLE 4. VALUES OF ms, THE MAXIMUM NUMBER OF LINEAR
CONTRASTS OF k MEANS FOR WHICH F INTERVALS ARE
LONGER THAN t-INTERVALS, 1-a =.95, .99*
1-a-.95
\
Ic
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
7
20
0
2
5
10
16
26
37
53
71
95
123
158
190
26X10
30X10
38X10
45 X 10
0
3
7
17
33
63
110
189
30X10
56X10
89 X 10
14 X 102
54X10
1-a=.99
X
7
20
X
0
3
9
24
55
129
281
614
126X 10
27 X 102
56 X 102
108X 102
223 X 102
426 X 102
872 X 102
182X 103
329 X 103
0
3
6
12
20
32
49
71
100
0
3
10
23
46
104
19X10
38X10
0
4
13
36
99
241
59X10
14 X 102
316 X 10
696 X 10
149X102
310 X 102
694 X 102
150X 103
312 X 103
66 X 104
12X 105
26 X 105
6X106
630 X 103
132X 104
* The last significant
digitgivenin mscannotbe expectedto be exactlycorrectexceptwherems is smallerthan
100. For example,rns=26 X1IOindicatesthat at some pointin the calculationa numberwithonlytwo significant
digitswas used; in computingns=36, on the otherhand,threesignificant
digitswerecarriedthroughout.
It appears fromTable 3 that fora fairlylarge numberof means, the t-intervals are shorterforany m of reasonablesize.
In Table 4 are listed values of ms, the maximumnumberof linear combinations for which the Scheff6intervalsare longerthan the t intervals.If, for a
givenk,v, and 1 - a, one decides to estimatem linearcontrastsamong k means,
one may examine ms from Table 4. If m<ims, the Student t intervals are
shorter;if m> ms the Scheff6intervalsare shorter.Table 4 gives ms forv=7,
20, oo, 1- a =.95 and .99, and k =2, 3, * * , 20. The table indicates that for k
as large as 10, one may forma large numberof contrastsusing the t-intervals
and stillhave intervalssmallerthan ifthe F distributionhad been used.
If one wishes to estimatelinear contrastsamong 9 means with v= 20, 1- a
-.95, then enteringTable 4 at k= 9, one findsms= 189, so that if the number
58
AMERICAN
STATISTICAL
ASSOCIATION
JOURNAL,
MARCH 1961
of contraststo be estimatedis less than or equal to 189, the Student t intervals
should be used ratherthan ScheffW's
intervals. If linear combinationrather
than just contrastsare beingestimated,thenone entersthe table with10 rather
than 9 and findsms 300.
Examinationof Table 4 indicatesthat the situationbecomes morefavorable
forthe Studentt methodif,as all othervariables except one are held constant:
1) k is increased; or
2) v is increased; or
3) 1- a is increased.
4. COMPARISON
WITH TUKEY'
S INTERVALS
, h are uubiased, normaIlydistributedestimatorsof px,
if ,u<
,Sk
with Var (pi) = a11a2and Cov (ai, -uj)=0, then confidenceintervals of level
1-xa forall possible linear contrasts
k
0 = CIl
+
**+
Ck,lAk,
i=l
Ci=
O
are givenby
6 +-Z
| ci I /aiiqa,
2
(9)
whereq is the 1- a point of the Studentizedrange fora sample of size k, and
withv
a2 is an independentestimateof a2 such that a,&t/o has a x2 distribution
degrees of freedom.
For pAi-y, based on a sample ofsize n, thesebecome
6+
jE
|
csjq&/v/A/n.
(10)
It is apparent that, as formulatedhere,these intervalsare morelimitedin applicationthan the t intervalsand Scheff6'sintervals,since (1) they apply only
to linear contrastsratherthan to arbitrarylinear combinations;(2) the variances are assumed to be equal; (3) the covariancesare assumed to be all zero.
Tukey [2], states withoutproofthat they may be extended somewhatin all
three directions.
Limitation (1) may be removedby introducinga (k+l)st mean whose estimate, pk4l, is always zero. Tukey shows that there is no appreciable errorin
simply using the same intervals for linear combinations as for linear contrasts,providedk > 2. To be ultraconservative,one may use the same intervals
but enterthe tables of the Studentizedrange with k+ 1 ratherthan with k.
In usingTukey's intervalsforcombinationsas opposed to contrasts,it should
be notedthatformulas(9) and (10) mustbe alteredby replacing42EIciI by
the largerof the sum of the positive {ci} and the negative of the sum of the
negative {ct} .
In an effortto remove limitation(2), Tukey considersthe case where Var
(s) = as,2, k-1, * , Joand Cov (Ai, ,j) = 0, i#j, with the ai known con-
MULTIPLE
COMPARISONS
59
AMONG MEAN'S
stants. In other words,the covariances among the , are zero and the ratios
among theirvariances are known.
By multiplyingthe qa in (9) by a factordependingoli the particularcontrastand on the various aii, instead of by -Van, he obtained intervalswith an
overallconfidencelevel whichhe says is approximately1- a, adding that work
is beinigdone on studyingthis approximation.
Limitation (3) may also be removedwhen only contrastsare being consid, k, and Cov (-j, -j) al2q2o, i5j, whereall
ered. If Var (,aj)=a,1o-2, i=l,
and a12are known,then the confidenceintervalfor
k
=
CilAi
becomes
1k
?+-E
|
2 i=l
cif(all
-
(11)
a12)12q.
Thus the contrastsusingthe Studentizedrangehave been partiallyextended,
but not to the moregeneralsituationwherethe variancesand covariancesofthe
pAare aii-2 and atjo2,withthe aii and aij known.The extensionto unequal variances seemsto be somewhatarbitrary,and I do not know whetherit has been
put on any satisfactorybasis.
Comparison of the lengths of the Studentized range intervals with the tintervalsis complicatedby the factthat the lengthsofthe Tukey intervalsdepend on
ccI
i whereasthe lengthsofthe t intervalsdepend on
k
k
Ej=1
t=1
aijci8cj,.
Scheffe[1] comparesthe squared lengthof an F-intervalwith the squared
lengthofthe same Studentizedrangeinterval,and thenconsidersthe maximum
and minimumvalues ofthis ratio over all types of contrast.He pointsout that
thissquared ratiois a maximumforintervalsofthe type/ii-ui, and a minimum
forintervalsofthe type
2(/A +
* * * +
k/l2)/k
-
2(lk/2+1
+
+
Ak)/k
fork even, or ofthe type
(Al +
+ t1k/2-1)
/
)-(/Ak/2
+
***+
Ak) /(-+
1
fork odd.
In Table 5 are givenvalues of c2/q2. From this table, forany particularcontrast one may compute the squared ratio of lengthsfor the t-intervalsand
Studentizedrangeintervals.For variances equal to allo2 and covarianceszero,
this squared ratio is
.r;2
2
60
AMERICAN
STATISTICAL
TABLE 5. VALUES OF
k
m
\
c2/q2
ASSOCIATION
|~ 5
io10
MARCH 1961
FOR 1 -a =.95, .99
1-a.99
1-a.95
2
JOURNAL,
15 j
20
2
5
.66
.93
1.19
2.00
2.48
3.18
.33
.46
.59
1.00
1.24
1.59
|10
15
20
.23
.33
.42
.70
.87
1.11
.20
.27
.35
.59
.73
.94
.17
.25
.31
.53
.65
.84
v=7
2
5
10
50
100
250
.72
1.10
1.46
2.61
3.31
4.39
.32
.48
.63
1.14
1.44
1.91
.21
.32
.43
.77
.97
1.29
.18
.27
.36
.64
.81
1.07
.16
.24
.32
.57
.72
.95
v =20
2
5
10
50
100
250
.67
.93
1.15
1.70
1.98
2.39
.33
.45
.56
.83
.96
1.16
.23
.32
.40
.59
.69
.83
.20
.28
.34
.50
.58
.71
.18
.25
.30
.45
.53
.64
.62
.78
.92
1.29
1.43
1.67
.36
.45
.53
.74
.82
.97
.27
.34
.40
.56
.62
.73
.23
.30
.35
.49
.54
.64
.21
.27
.32
.45
.50
.58
2
5
10
50
100
250
.65
.87
1.03
1.41
1.58
1.80
.34
.45
.53
.73
.81
.93
.25
.33
.40
.54
.61
.69
.22
.29
.34
.47
.53
.60
.20
.27
.31
.43
.48
.55
.60
.72
.82
1.04
1.14
1.27
.37
.45
.51
.65
.72
.80
.30
.36
.41
.52
.57
.63
.27
.32
.36
.47
.51
.57
.25
.30
.34
.43
.47
.53
In Table 6 are givenvalues of mT, the maximumm such that everyt interval
is shorterthan the correspondingTukey interval,even forthe least favorable
(to the t interval) case, ,ui-,i. A glance at Table 6 shows that mTtends to be
rathersmall. If one's primaryinterestis in intervalslike pi-4ii, then he may
use this table to decide whichmethodto use. Otherwisea comparisonmust be
made foreach type of intervalwhichis of interest.
Again, as in the comparisonbetween Scheff6'sintervalsand the t intervals,
the t intervalsseem to become better,otherthingsbeingequal, as
1) k becomes larger,or
2) v becomes larger,or
3) 1- a becomeslarger.
In an analysis of variance situationwith a single variable of classification,
the numberof means would tend to be small and primaryinterestmightbe in
estimatingthe difference
betweenmeans. Then Tukey's intervalsare perhaps
preferable.When there are two variables of classification,then one perhaps
wishesto estimaterow differences,
and interactionsrather
columndifferences,
than the differences
betweensinglemeans. Then the t intervalsare morelikely
to be shorter.
61
MUJLTIPLE COMPARISONS AMONG MEANS
TABLE 6. VALUES OF mT, THE MAXIMUM NUMBER OF LINEAR CONTRASTS OF k MEANS FOR WHICH EVERY STUDENTIZED
RANGE INTERVAL IS LONGER THAN THE CORRESPONDING t-INTERVAL, 1-a=.95,
.99
1-a =.95
P
\
Ic
2
3
7
20
00
7
20
0
2
0
2
0
2
0
2
0
2
3
5
7
4
5
6
7
8
9
10
9
11
13
14
1
12
16
18
13
14
15
16
17
18
19
20
1-a .99
20
22
24
26
28
30
32
34
4
7
11
4
6
9
12
16
20
24
4
6
8
10
12
14
17
15
20
25
31
28
33
14
18
23
28
19
21
39
47
37
43
48
53
60
66
71
77
4
7
11
33
39
24
26
29
31
33
35
37
39
53
61
71
82
92
103
1ll
125
43
50
56
64
71
80
92
104
X
0
2
5
8
13
17
24
30
38
46
54
63
74
84
95
112
122
139
157
5. AN EXAMPLE
in whichone may wishto chooseamong
As an exampleof an experiment
modelfora two-wayclassification
with
consider
thefixed-effect
thesemethods,
in eachcell.
a rows,b columns,
andn observations
in the ith row and thejth column.Then
Let Xijk be the kthobservation
(:i.)=o-2/n fori 1,
a; j=1,
,b;
E(xijk) = i -,/+ai+bj+Iij,andVar
k= 1, * , n, with
a
and
,ai = 0,
b
Sbi
b
j11
I
a
0
,iij
O
Ii-O
i=
l, Ea.
j
l, .. * *b,
The pooledestimateofthevariance,2, has n(a -1) (b-1) degreesoffreedom.
Hereaii 1/n,ai = 0 foriij, andif
aECiijAij
oj
is anylinearcombination
oftheab means,thepointestimateforit is
O= E2Cijzi.,
.j
62
AMERICAN
STATISTICAL
ASSOCIATION
JOURNAL,
MARCH 1961
and the confidenceintervalusing the t-statisticis
?C
CijXJj.
Cij/fna.
In the firstcolumn of Table 7 are listed various linear combinationswhich
wishto estimate.For ease in comparison,theyhave been
the experimenter
nmay
multipliedwherenecessaryby a factorchoseinso that the lengthsof the Tukey
intervalsare all equal. In the second columnare listedthe usual pointestimates.
Those listed in rows 5 to 10 of the table are linear contrasts,whereasthose in
rows 1 to 4 are not. In the thirdcolumnare given the numberof each type of
linear combination;the fourthcolumngives
2
Ecij/n,
i,j
to be used in the confidenceintervalforthat type of contrast.
The last two columinsof Table 7 give
c
'
2
j
ci/n
fora= 3, b= 4, n-3 and fora = 4, b= 5, and n = 4. In these columns,c has been
computedon the assumptionthat the experimenterwishesto estimateall the
linear combinationslisted, with an overall confidencelevel of 1-xax.95. To
compare the t-intervalswith Tukey's intervals,the values in columns (5) and
(6) must be comparedwith2.99 and 2.64, respectively,since Tukey's intervals
forall these linear combinationsare cjj:. ? (q/\/n)a.
To compare the lengthsof the t-intervalswith Scheff4'sintervals,oinemust
compare the values of c, 3.97 and 4.31, with 5.11 and 5.92, the corresponding
values of S.
It is possible that the experimentermay be interestedin estimatingnot all
the linear combinations.In Table 8 are shown values of c forvarious sets of
linear combinationsestimated:rows I to 10 inclusive (all the linear combinationslisted); rows 1 to 9 inclusive(all the linearcombinationslisted exceptthe
differencesbetween means); rows 5 to 10 inclusive (all the linear contrasts
listed); and rows 5 to 9 inclusive (all the linear contrastslisted except the differencesbetweenmeans). Table 8 also gives the corresponding
values of S.
In these particularexamplesone shouldprobablypick the t-statisticover the
Studentizedrange. The exceptionto this (consideringonly lengthof interval)
would be ifone is mainlyinterestedin estimatesofAUij
-.uitj; thisdoes not seem
likely. It should be emphasized,however,that if the researchworkerwants
separately.
intervalsas shortas possible,each problemmust be exarmined
6. DISCUSSION
OF THE STUDENT
t METHOD
It is interestingto considerwhat may be the actual probabilityof coverage
usingthe Studentt method.In (4), we may let the leftside (the actual probability of coverage) be denoted by P and the rightside (the lower bound forthis
MhULTIPLE COMPARISONS
~~~~
C4~
C-
o
63
AMONG MEANS
^:|
C
C~
O
1 r--4-C Cl 1C
00
C
Cl
Cll
CC
CO
Lr
L r<-O
!It
-
Vi
H~~~~~~~~~~
i
Q
~
I
~
___
Se
_
l
,x
C)~
~
_
_
~
_
_
t
3
I
_
_
_
_
_
M
.
v]
_
_
_
c
'
u~~~~~~~~~~~~C
4
r
_
*-~~~~~~~~~~~~~~~~~~~~~~V
_
_
_
_
_
_
_
_
_
06
_
_
C; -C;
_
_
64
AMERICAN
STATISTICAL
ASSOCIATION
JOURNAL,
MARCH 1961
TABLE 8. COMPARISON BETWEEN LENGTHS OF t-INTERVALS AND
F-INTERVALS IN TWO-WAY CLASSIFICATION EXAMPLE
(1 - ce=.95)
Types
s=4, b=5, n=4
a=3, b=4, n=3
_
Estimated
Number
Estimated
c
1 to 10, incl.
1 to 9, incl.
84
48
5 to 9, incl.
28
64
5 to 10, incl.
_
_
_
.
_
_
_
_
_
_
_
_
_
_
_
S
Number
Estimated
c
S
3.97
3.73
5.11
5.11
195
75
4.31
3.93
5.92
5.92
3.53
4.93
45
3.71
5.78
3.87
4.93
165
4.26
_
5.78
probabilityof coverage) be denoted by P, and considerthe difference
between
P and P.
If all the correlationsapproachunity,thenin the limitall the linearcombinatiolnsbecome one and the same linear combination,and P attains its largest
possible value. In this case,
P
=
P(-c
< t1 < c)
= 1-2
ffv) (t)dt.
P=1-a
For v=
.95, anld n= 100,c=3.48, and P =.9994.
In [3] I conjecturedthat P attains its smallestpossible value when all the
correlationsare zero. This was established,howvever,
onlyform= 2 and 3.
Extensive tables are not available at presenitto evaluate P whenlall the correlationsare zero,thoughPillai and Ramachandran [5] give 95 per cent points
and 99 per ceintpointsof the necessarydistributionform less than or equal
to 8. For v= cc, however,the normal tables may be used. For P =1- a, P
= [1- (a/rn)
so that P is the firsttwo terms in the binomial expansion
of [I- (a//m)]r. The differeince
between them is seen to be bounded by
[(m
-
1)/2n] a2, so that foriv= cc and a snmall,P is fairly close to the actual
probabilityof coverage when all the correlationsare zero. In particular,for
l-a = .95 aiid m =100, P = (1-.0005)100 =.95 12.
Thus forv-=
and a sm-iall,the inequality (4) gives resultswhichare almost
as good as aniywhichare attainable when nothilngis knownabout the correlations.At presentI am attemptingto constructtables whichwill give some idea
ofthe situationforsmall values of v and forcorrelationsbetween0 and 1.
REFERENCES
[l] HenryScheff6,'A methodof judgingall contrastsin the analysis of variance,"Biometrika,
40 (1953), 87-104.
[2] John M. Tukey, "The problem of multiple comparisons,"mimeographednotes,
PrincetonUniversity.
[3] Olive Jean Dunn, 'Estimationof the meansof dependentvariables," Annals ofMathematicalStatistics,29 (1958), 1095-111.
[4] E. S. Pearson and H. 0. Hartley,BiometrikaTables for Statisticians,Vol. 1, Cambridge,1956.
[5] K. C. S. Pillai auidK. V. Ramachandran,"Distributionof a StudentizedOrderStatistic,"AnnalsofMathematical
Statistics,25 (1954), 565-72.