Topic 22: Inference
Outline
•
•
•
•
Review One-way ANOVA
Inference for means
Differences in cell means
Contrasts
Cell Means Model
• Yij = μi + εij
where μi is the theoretical mean or
expected value of all observations
at level i and the εij are iid N(0, σ2)
Yij ~N(μi, σ2) independent
Parameters
• The parameters of the model are
– μ1, μ2, … , μr
– σ2
• Estimate μi by the mean of the
observations at level i,
– ûi = Yi= (ΣYij)/(ni)
(level i sample mean)
– si2 = Σ(Yij- Yi)2/(ni-1)
(level i sample variance)
– s2 = Σ(ni-1)si2 / (nT-r) (pooled variance)
F test
•
•
•
•
•
•
F = MSM/MSE
H0: μ1 = μ2 = … = μr = μ
H1: not all of the μi are equal
Under H0, F ~ F(r-1, nT-r)
Reject H0 when F is large
Report the P-value
KNNL Example
• KNNL p 676, p 685
• Y is the number of cases of cereal sold
• X is the design of the cereal package
– there are 4 levels for X because there
are 4 different package designs
• i =1 to 4 levels
• j =1 to ni stores with design i
Plot the means
proc means data=a1;
var cases;
by design;
output out=a2 mean=avcases;
symbol1 v=circle i=join;
proc gplot data=a2;
plot avcases*design;
run;
The means
Confidence intervals
•
•
•
•
Yi ~ N(μi, σ2/ni)
CI for μi is Yi ± tc s n i
tc is computed from the t(α/2,nT-r)
Degrees of freedom larger than ni-1
because we’re pooling variances
together into one single estimate
• This is advantage of ANOVA if
appropriate
Using Proc Means
proc means data=a1 mean std
stderr clm maxdec=2;
class design;
This does not
var cases;
use the
run;
pooled
estimate of
variance.
Output
N
des Obs
1
5
2
5
3
4
4
5
Mean Std Dev
14.60
2.30
13.40
3.65
19.50
2.65
27.20
3.96
Std Error
1.03
1.63
1.32
1.77
We can use this information to calculate SSE
SSE = 4(2.3)2 + 4(3.65)2 + 3(2.65)2 + 4(3.96)2 = 111.37
Confidence Intervals
Lower 95%
des CL for Mean
1
11.74
2
8.87
3
15.29
4
22.28
Upper 95%
CL for Mean
17.46
17.93
23.71
32.12
There is no pooling of error in computing these CIs
Each interval assumes different variance estimate
CI’s using PROC GLM
proc glm
class
model
means
run;
data=a1;
design;
cases=design;
design/t clm;
Output
The GLM Procedure
t Confidence Intervals for cases
Alpha
0.05
Error Degrees of Freedom
15
Error Mean Square
10.54667
Critical Value of t
2.1314
CI Output
des
4
3
1
2
N
5
4
5
5
Mean
27.200
19.500
14.600
13.400
95% Confidence
Limits
24.104 30.296
16.039 22.961
11.504 17.696
10.304 16.496
These CI’s are often narrower because more
degrees of freedom (common variance)
Multiplicity Problem
• We have constructed 4 (in general, r) 95%
confidence intervals
• The overall confidence level (all intervals
contain its mean) is less that 95%
• Many different kinds of adjustments have
been proposed
• We have previously discussed the
Bonferroni (i.e., use α/r)
BON option
proc glm
class
model
means
run;
data=a1;
design;
cases=design;
design/bon clm;
Output
Bonferroni t Confidence
Intervals for cases
Alpha
0.05
Error Degrees of Freedom
15
Error Mean Square
10.54667
Critical Value of t
2.83663
Note the bigger t*
Bonferroni CIs
des N
4
3
1
2
5
4
5
5
Simultaneous 95%
Mean Confidence Limits
27.200
19.500
14.600
13.400
23.080
14.894
10.480
9.280
31.320
24.106
18.720
17.520
Using LSMEANS
• LSMEANS statement provides proper
SEs for each individual treatment
estimate AND confidence intervals as
well
lsmeans design / cl;
• However, statement does not provide
ability to adjust these single mean
CIs for multiplicity
Hypothesis tests on
individual means
• Not usually done
• Use PROC MEANS options T and PROBT
for a test of the null hypothesis H0: μi = 0
• To test H0: μi = c, where c is an arbitrary
constant, first use a DATA STEP to
subtract c from all observations and then
run PROC MEANS options T and PROBT
• Can also use GLM’s MEAN statement
with CLM option (more df)
Differences in means
• Yi.  Yk. ~ N(μi-μk, σ2/ni + σ2/nk)
• CI for μi-μk is Yi.  Yk. ± tcs( Yi.  Yk.)
where s(Yi.  Yk. ) =s
1
ni

1
nk
Determining tc
• We deal with the multiplicity problem by
adjusting tc
• Many different choices are available
– Change α level (e.g., bonferonni)
– Use different distribution
LSD
•
•
•
•
Least Significant Difference (LSD)
Simply ignores multiplicity issue
Uses t(nT-r) to determine critical value
Called T or LSD in SAS
Bonferroni
• Use the error budget idea
• There are r(r-1)/2 comparisons
among r means
• So, replace α by α/(r(r-1)/2) and use
t(nT-r) to determine critical value
• Called BON in SAS
Tukey
• Based on the studentized range distribution
(maximum minus minimum divided by the
standard deviation)
• tc = qc/ 2 where qc is detemined from SRD
• Details are in KNNL Section 17.5 (p 746)
• Called TUKEY in SAS
Scheffe
• Based on the F distribution
• tc =
(r - 1)F(1 - α; r - 1, N - r)
• Takes care of multiplicity for all
linear combinations of means
• Protects against data snooping
• Called SCHEFFE in SAS
• See KNNL Section 17.6 (page 753)
Multiple Comparisons
•
•
•
•
LSD is too liberal (get too many Type I errors)
Scheffe is too conservative (very low power)
Bonferroni is ok for small r
Tukey (HSD) is recommended
Example
proc glm
class
model
means
lsd
run;
data=a1;
design;
cases=design;
design/
tukey bon scheffe;
LSD
t Tests (LSD) for cases
NOTE: This test controls the
Type I comparisonwise error rate,
not the experimentwise error
rate.
Alpha
0.05
Error Degrees of Freedom
15
Error Mean Square
10.54667
Critical Value of t
2.13145
LSD Intervals
design
Comparison
4
4
4
3
3
3
1
1
1
2
2
2
-
3
1
2
4
1
2
4
3
2
4
3
1
Difference
Between
Means
7.700
12.600
13.800
-7.700
4.900
6.100
-12.600
-4.900
1.200
-13.800
-6.100
-1.200
Simultaneous 95%
Confidence Limits
3.057
8.222
9.422
-12.343
0.257
1.457
-16.978
-9.543
-3.178
-18.178
-10.743
-5.578
12.343
16.978
18.178
-3.057
9.543
10.743
-8.222
-0.257
5.578
-9.422
-1.457
3.178
***
***
***
***
***
***
***
***
***
***
Tukey
Tukey's Studentized Range (HSD)
Test for cases
NOTE: This test controls the
Type I experimentwise error rate.
Alpha
0.05
Error Degrees of Freedom
15
Error Mean Square
10.54667
Critical Value of Studentized
Range 4.07597
4.07/sqrt(2)= 2.88
Tukey Intervals
design
Comparison
4
4
4
3
3
3
1
1
1
2
2
2
-
3
1
2
4
1
2
4
3
2
4
3
1
Difference
Between
Means
7.700
12.600
13.800
-7.700
4.900
6.100
-12.600
-4.900
1.200
-13.800
-6.100
-1.200
Simultaneous 95%
Confidence Limits
1.421
6.680
7.880
-13.979
-1.379
-0.179
-18.520
-11.179
-4.720
-19.720
-12.379
-7.120
13.979
18.520
19.720
-1.421
11.179
12.379
-6.680
1.379
7.120
-7.880
0.179
4.720
***
***
***
***
***
***
Scheffe
Scheffe’s Test for cases
NOTE: This test controls the Type I
experimentwise error rate, but it generally
has a higher Type II error rate than Tukey's
for all pairwise comparisons.
Alpha
0.05
Error Degrees of Freedom
15
Error Mean Square
10.54667
Critical Value of F
3.28738
Sqrt(3*3.28738)= 3.14
Scheffe Intervals
design
Comparison
4
4
4
3
3
3
1
1
1
2
2
2
-
3
1
2
4
1
2
4
3
2
4
3
1
Difference
Between
Means
7.700
12.600
13.800
-7.700
4.900
6.100
-12.600
-4.900
1.200
-13.800
-6.100
-1.200
Simultaneous 95%
Confidence Limits
0.859
6.150
7.350
-14.541
-1.941
-0.741
-19.050
-11.741
-5.250
-20.250
-12.941
-7.650
14.541
19.050
20.250
-0.859
11.741
12.941
-6.150
1.941
7.650
-7.350
0.741
5.250
***
***
***
***
***
***
Output (LINES option)
Scheffe
Mean
N
design
A
27.200
5
4
B
B
B
B
B
19.500
4
3
14.600
5
1
13.400
5
2
Using LSMEANS
• LSMEANS statement provides proper
SEs AND confidence intervals based
on multiplicity adjustment
lsmeans design / cl adjust=bon;
• However, statement does not have
lines option
• Can use PROC GLIMMIX to do
this….approach shown in sasfile
Linear Combinations of
Means
• These combinations should come
from research questions, not from an
examination of the data
• L = Σiciμi
ˆ = Σici Y ~ N(L, Var( L
ˆ ))
• L
i
ˆ ) = Σici2Var(Y)
• Var( L
i
• Estimated by s2Σi(ci2/ni)
Contrasts
•
•
•
•
•
Special case of a linear combination
Requires Σici = 0
Example 1: μ1 – μ2
Example 2: μ1 – (μ2 + μ3)/2
Example 3: (μ1 + μ2)/2 - (μ3 + μ4)/2
Contrast and Estimate
proc glm data=a1;
class design;
model cases=design;
contrast '1&2 v 3&4'
design .5 .5 -.5 -.5;
estimate '1&2 v 3&4'
design .5 .5 -.5 -.5;
run;
Output
Contr
DF SS MS
F
P
1&2 v 3&4 1 411 411 39.01 <.0001
Par
Est
1&2 v 3& -9.35
St
Err
t
P
1.49 -6.25 <.0001
Contrast performs F test. Estimate
performs t-test and gives estimate.
Multiple contrasts
• We can simultaneously test a collection
of contrasts (1 df each contrast)
• Example 1, H0: μ2 = μ3 = μ4
• The F statistic for this test will have an
F(2, nT-r) distribution
• Example 2, H0: μ1 = (μ2 + μ3 + μ4)/3
• The F statistic for this test will have an
F(1, nT-r) distribution
Example
proc glm data=a1;
class design;
model cases=design;
contrast '1 v 2&3&4'
design 1 -.3333 -.3333 -.3333;
estimate '1 v 2&3&4'
design 3 -1 -1 -1 /divisor=3;
contrast '2 v 3 v 4'
design 0 1 -1 0,
design 0 0 1 -1;
Output
Con
DF
1 v 2&3&4 1
2 v 3 v 4 2
F
10.29
22.66
P
0.0059
<.0001
St
Par
Est Err
t
P
1 v 2&3&4 -5.43 1.69 -3.21 0.0059
Last slide
• We did large part of Chapter 17
• We used programs topic22.sas to
generate the output for today