One-Factor Experiments &
ANCOVA
Group 3
Jesse Colton;
Lijuan Kang;
Xin Li ;
Junyan Song;
Minqin Chen;
Yaqi Xue
Kan He;
Xiaotong Li;
Outline:
History and
Introduction
Model and
Overall F Test
A
N
O
V
A
Pairwise Test
for Group
Means
ANOVA Linear
Model and
Tests
Theoretical
Background
A
N
C
O
VA
Do ANCOVA
by Hand
Check
Assumptions
Do ANCOVA
by SAS
What Is ANCOVA?
Definition
•
ANOVA stands for Analysis Of Variance.
• ANCOVA stands for Analysis Of Covariance.
• ANCOVA uses aspects of ANOVA and Linear
Regression to compare samples to each other,
when there are outside variables involved
• “One-Factor Experiment” means we are testing
an experiment using only one single treatment
factor.
History
Like many of the important topics in statistical
analysis, elements of ANOVA/ANCOVA come from
works of R.A. Fisher, and some from Francis Galton
History
Ronald Aylmer Fisher
1890-1962
• British Statistician, Eugenicist, Evolutionary
Biologist & Geneticist
• Fisher “pioneered the principles of the design of
experiments and elaborated his studies of analysis
of variance.”(Wikipedia)
• He also developed the method of maximum
likelihood, and is known for “Fisher’s exact test”
History
Sir Francis Galton
1822-1922
• Established the concept of
correlation
• He “invented the use of the
regression line and was the
first to describe and explain
the common phenomenon
of regression toward the
mean.”(Wikipedia)
Uses
• ANOVA is used to compare the means of two or
more groups.
• ANCOVA is used in situations where another
variable effects the experiment.
• While we normally use the T-test for two group
means, there are many situations where it is not
applicable or as useful.
• More than 2 samples
• Samples with additional variables
• Other factors leading to skewed experimental results
Uses
• When conducting an experiment, there is often an
initial difference between test groups.
• ANCOVA “provides a way of measuring and
removing the effects of such initial systematic
differences between the samples.”
(http://vassarstats.net/textbook/ch17pt2.html)
• If you only compare the means, you are not
taking into account any previous advantages one
group may have
Uses
Example: Two methods of teaching a topic are
tested on two different groups (A and B). However,
in the preliminary data collected, group A is shown
to have a higher IQ than group B. The fact that
group A had a higher score after learning by one
method does not prove the method is better.
ANCOVA seeks to eliminate the difference between
the groups before the experiment in order test
which method is better.
Uses
• By merging ANOVA with Linear Regression,
ANCOVA controls for the effects that the
covariates we are not studying may have on the
outcomes
ANOVA
Linear
Regression
ANCOVA
Aims of ‘ANOVA’ Models
• Linear models with continuous response and one or
more categorical predictors
• Description:
-relation between response variable (Y) and predictor
(X) variable(s)
• Explanation:
- How much of variation in Y explained by different
sources of variation (factors or combination of
factors)
Completely Randomized
Designs
• Experimental designs where there is
no restriction on random allocation of
experimental/sampling units to
groups or treatments
- single factor and factorial designs
Single factor model
Completely randomized design
Terminology
• Factor (categorical predictor variable):
- usually designed factor A
• Number of observations within each group:
-ni
• Each observation:
-y
Data layout
Estimating Model Parameters
Estimating Model Parameters
Estimating Model Parameters
Least Square (LS) Estimate
Estimating Model Parameters
Analysis of Variance
• Test the hypothesis
H 0 : 1   2     a
H a : Not all  i are equal
Analysis of Variance
• Test the hypothesis
H 0 : 1   2     a  0
𝐻1: 𝐴𝑡 𝑙𝑒𝑎𝑠𝑡 𝑠𝑜𝑚𝑒 𝜏𝑖 ≠ 0
Analysis of Variance
Analysis of Variance
Analysis of Variance
Test Statistical
a
MSA
F0 =
=
MSE
ni
(
)
2
(
y
y
)
å å i× ×× / a -1
i=1 j=1
a ni
(
2
(
y
y
)
å å ij i× / N - a
i=1 j=1
H0
)
~ Fa-1,N -a
Analysis of Variance
Analysis of Variance
Unequal sample sizes
• Sums of squares equations provided
only work for equal sample sizes
- can be modified for unequal samples
sizes but very clumsy
-model comparison approach simpler
(and used by statistical software)
Unequal sample sizes
• F-ratio tests less reliable if sample sizes
are different, especially if variances also
different
- bigger difference in sample sizes, less
reliable tests become
• Use equal or similar sample sizes if
possible
• But don’t omit data to balance sample
sizes!
Anova— Multiple Comparisons of Means
Reject
H 0 : 1   2 
 a
, where a is the # of groups
Not all means are equal.
But which means are significantly different from
each other?
We need a more detailed comparison!
Making multiple test
Anova— Multiple Comparisons of Means
Making multiple test
Test All Pairwise equality Hypotheses
H 0 ij :  i   j
H aij :  i   j
Number of Pairs: 𝑎2 =𝑎(𝑎 − 1) 2
Using two sided t-test at level α:
y  y
Reject 𝐻0𝑖𝑗 if
i
Tij 
j
1
S

ni
a
ni
where 𝑆 2 = MSE =   ( y
i 1
1
 t ni  n j  2 , / 2
nj
 yi ) /  N  a 
2
ij
j 1
𝑛𝑖 is the number of group i, 𝑦𝑖 is the mean of the
a
observed value of group i,
N 
n
i 1
i
.
Anova— Multiple Comparisons of Means
yi  y j
Tij 
S
1
ni

1
 t ni  n j  2 , / 2  y i  y j  t ni  n j  2 , / 2 S
nj
Least Significant Difference (LSD):
The critical value,
t ni  n j  2 , / 2 S
1
ni

1
nj
that the difference |𝑦𝑖 − 𝑦𝑗 | must exceed in order to be
significant at level 𝛼 .
1
ni

1
nj
Anova— Multiple Comparisons of Means
Familywise Error Rate (FWE):
Type I error probability of declaring at least one
pairwise difference to be falsely significant.
FWE=P{Reject at least one true null hypothesis}
If each test is done at level 𝛼 , then FWE will
exceed 𝛼 .
Why?
Anova— Multiple Comparisons of Means
Let 𝐴𝑖 denote rejecting the true null hypothesis in
𝑖
𝑡ℎ
test, where total number of test is k=
P( 𝐴𝑖 ) = 𝛼 = type I error.
FWE=P( 𝐴1 ∪ ⋯ ∪ 𝐴𝑘 ) ≥ 𝛼 =P( 𝐴𝑖 )
If 𝐴𝑖 is independent to each other,
FWE=k*P( 𝐴𝑖 )= k𝛼
Our goal is to control FWE ≤ 𝛼 .
𝑎
2
.
Anova— Multiple Comparisons of Means
Two Methods:
• Bonferroni Method.
• Tukey Method.
Anova— Multiple Comparisons of Means
Bonferroni Method
• Idea: To perform k tests simultaneously, divide
the FWE α among the k tests.
If the error rate is allocated equally among
the k tests, then each test is done at level
α/k.
For example:
α=0.05 and k=10
each test: 0.05/10=0.005
Anova— Multiple Comparisons of Means
Bonferroni Method
• Test:
H 0 ij :  i   j
H aij :  i   j
At FWE= 𝛼 , we reject
yi  y j
Tij 
1
s

ni
where s2 =
H 0 ij
a
 t ni  n j  2 , / 2 k
nj
ni
MSE =  
i 1
1
if
j 1
( y ij  y i ) /  N  a 
2
Anova— Multiple Comparisons of Means
Tukey Method
H 0 ij :  i   j
H aij :  i   j
At FWE= 𝛼 , we reject
H 0 ij
yi  y j
| t ij |
s
1
ni

if

1
q a , N  a ,
2
nj
where s2 = MSE,
and the value of q a , N  a , can be found in its table.
Dummy Variable:
A Dummy Variable is an artificial variable created
to represent an attribute with two or more distinct
categories/levels.
How to create a Dummy Variable:
The number of dummy variables necessary to
represent a single attribute variable is equal to the
number of levels(categories)(k) in that variable
minus one. (k-1)
Gender: Male & Female
Categories D1
Male
1
Female
0
Yi = b 0 + b1xi + b 2D1i + e i
Rank: Assistant & Associate & Full
Categories
D1
D2
Assistant
1
0
Associate
0
1
Full
0
0
ANOVA Models(A Multiple Regression with all categorical
predictors):
General Linear Model:
0
1 1
Yi = b + b D i + b 2 D2i + b 3 D 3i + .....e i
Dummy Variables
?Relationship between these Models:
Yi = b 0 + b 1D1i + b 2 D2i + e i
m1 = m + a1 = b 0 + b 1
m 2 = m + a2 = b 0 + b 2
m3 = m = b 0
am = 0constraint
Yi = b 0 + b 1D1i + b 2 D2i + e i
m1 = m + a1 = b 0 + b 1
m 2 = m + a2 = b 0 + b 2
m 3 = m - a1- a2 = b 0 - b 1 - b 2
Note:
m is the Grand Mean, but in the last case it is the mean of Group 3.
b 0 , b 1 is different from those in the last case.
The Interpretation differs depending on which constraint we
apply.
m1 = m + a1 = b 0 + b 1
m 2 = m + a2 = b 0 + b 2
m3 = m = b 0
Group one mean-Group three
bˆ 1 :mean
bˆ 2
: Group two mean-Group three
mean
bˆ 1 - bˆ 2
:Group one mean –Group two
mean
? How do we test ANOVA in terms of General Linear Model
1. Overall F-Test
H0:
H0:
m1 = m 2 = m 3 = ...... = ma
b 1 = b 2 = b 3 = ...... = b a - 1 = 0
Recall Test for Multiple Regression Coeffcient:
Reduced Model:
Yˆi = b 0
Full Model:
ˆ
Yi = b 0 + b 1 X 1i + b 2 X 2i + ...+ b a - 1 Xa - 1i
N
F0 =
SSR / p
=
SSE / (N - p - 1)
å (Yˆ - Y )
i
2
/ (a - 1)
i=1
N
2
ˆ
(Y
i
Y
i
)
/ (N - a)
å
i=1
P: numbers of parameters in H0.
* p=a-1
Recall Test for ANOVA in terms of
a
F0 =
SSA / (a - 1)
=
SSE / (N - a)
å n (Y
i-
i
a
i=1
ni
å å (Y
ij
Model
Y )2 / (a - 1)
- Yi )2 / (N - a)
i=1 j=1
N
SSR / p
F0 =
=
SSE / (N - p - 1)
å (Yˆ - Y )
i
2
/ (a - 1)
i=1
N
2
ˆ
(Y
i
Y
i
)
/ (N - a)
å
General Linear
Model
i=1
We reject H0 when F 0 > Fa - 1, N - a, a
So the Overall Test of ANOVA for both models are consistent.
2. Test for individual regression coefficient(Pairwise Test for
Group Means)
H0 differs depending on different coding of the Dummy
Variables.
m1 = m + a1 = b 0 + b 1
For Example:
m 2 = m + a2 = b 0 + b 2
m3 = m = b 0
H0:
T test
F test: F 0 =
Full Model:
[SSE(Re duced) - SSE(F)] / (dfR - dfF )
SSE(F) / (dfF )
Yi = b 0 + b 1D1i + b 2 D2i + e i
Reduced Model:
Yi = b 0 + b c(D1i + D2i ) + e i
ANCOVA Models(A Multiple Regression with continuous
predictors and dummy coded factors)
Ymi = m + am + b 1(X1mi - X1m ) + b 2(X 2mi - X 2m) + ...+ e mi
Yi = (m - b 1 X1 - b 2 X 2...) + (b 1 X1i + b 2 X 2i + ...) + (b 1' D1i + b 2 ' D2i + ...) + e i
Yi = b 0 + å b iXij +å b i 'Dij + e i
Continuous
Variables
Dummy
Variables
Overall Test for ANCOVA in terms of Linear Model:
H0: m1 = m 2 = ... = m
H0:
b 1 ' = b 2 ' = ... = b m - 1 ' = 0
What is analysis of Covariance?
• An analysis procedure for looking at group
effects on a continuous outcome when some
other continuous explanatory variable also has an
effect on the outcome.
• Generally, ANCOVA has at least one or more
categorical independent variables, and one or
more covariates. It can be seen as multiply
regression with 1+ covariates and 1+ dummy
variable coded factors.
Why include Covariates in
ANOVA
• To reduce within-group error variance: explain
part of unexplained variance in terms of
covariates so we can reduce the error variance
and increase the statistical power.
• Elimination of Confounds: if any variables which
will have an influence on the dependent variable
can be measured, ANCOVA would be a good
choice to use to partial out such effect.
Assumptions of ANCOVA
• Normality of Residuals
• Homogeneity of Variances
• Independence of Error terms
• Linearity of Regression
• Homogeneity of Regression Slopes
• Independence of Covariates and treatment effect
Homogeneity of Regression
Test the Homogeneity of
Regression
• Run ANCOVA model including independent
variables and interaction term
• If interaction term is significant, the assumption
is invalid.
• If interaction term is not significant, then try one
more without intersection term.
General Linear Model of
ANCOVA
• Yij = GMY + αi + [βi(Ci – Mij) + …… ] + εij
Treatment
effect
Grand
Mean of
dependent
variable
A
continuous
dependent
variable
Known
Covariance
Regression
coefficient for
ith covariate
Error N(0, σ2)
General Linear Model of
ANCOVA
• Yij - [βi(Ci – Mij) + …… ] = GMY + αi + εij
Adjusted
continuous
dependent
variable
Adjusted Yij = GMY + αi + εij
Same as ANOVA model
• Adjusted dependent variable means the relationship
between dependent variable and covariates has been
partialed out of dependent variable.
How to calculate Regression
coefficient?
• The numerator is the covariance of X and Y within the group
• The denominator is the sum square of deviates within the
group
• Then we should take the summation of βi hat, which is the
regression coefficient
F test in ANCOVA
• F test in ANCOVA is same as that in ANOVA,
the only difference is that now we are using the
adjusted values of SSbg(Y) and SSwg(Y), along
with adjusted value of df.
• If it is significant, the group means statistically
differ after controlling for the effect of 1+
covariates
Abbreviation
• SS: sum square of deviates
• SC: sum of co-deviates
• SST: total sum square of deviates
• SSWG: sum square of deviates within groups
• SSBG: sum square of deviates between groups
• SCT: total sum of co-deviates
• SCWG: sum of co-deviates with group
• SCBG: sum of co-deviates between group
ANCOVA
Example:
Comparing two
methods of Hypnotic
Induction
http://vassarstats.net/textbook/ch17pt2.html
Items to calculate
For the Dependent Variable Y
𝑺𝑺𝑻(𝒀) =
2
𝑌𝑖 -
𝑺𝑺𝑾𝑮(𝒀) = (
( 𝑌𝑖 )
𝑁
2
𝑌𝑎𝑖 -
2
( 𝑌𝑎𝑖 )
𝑛𝑎
2
𝑺𝑺𝑩𝑮(𝒀) = 𝑺𝑺𝑻(𝒀) - 𝑺𝑺𝑾𝑮(𝒀)
)+(
2
𝑌𝑏𝑖 -
( 𝑌𝑏𝑖 )
𝑛𝑏
2
)
Items to calculate
For the Covariate X
𝑺𝑺𝑻(𝑿) =
2
𝑋𝑖 -
𝑺𝑺𝑾𝑮(𝑿) = (
( 𝑋𝑖 )
𝑁
2
𝑋𝑎𝑖 -
2
( 𝑋𝑎𝑖 )
𝑛𝑎
2
)+(
2
𝑋𝑏𝑖 -
( 𝑋𝑏𝑖 )
𝑛𝑏
2
)
Calculations
Items to calculate
For the Covariance of X and Y
(Sum of the co-deviates)
(𝑿𝒊 − 𝑴𝒙 )(𝒀𝒊 − 𝑴𝒚 )
𝑺𝑪 =
=
𝑺𝑪𝑻 =
𝑺𝑪𝒘𝒈 =
+
(𝑿𝒊 ) (𝒀𝒊 )
(𝑿𝒊 𝒀𝒊 ) −
(General
𝒏
(𝑿𝑻𝒊 ) (𝒀𝑻𝒊 )
(𝑿𝑻𝒊 𝒀𝑻𝒊 ) −
𝒏
(𝑿𝒂𝒊 ) (𝒀𝒂𝒊 )
(𝑿𝒂𝒊 𝒀𝒂𝒊 ) −
𝒏
(𝑿𝒃𝒊 ) (𝒀𝒃𝒊 )
(𝑿𝒃𝒊 𝒀𝒃𝒊 ) −
𝒏
form)
Calculations
4. The Final Set of Calculations
A summary of the values we obtained so far
X
Y
Covariance
SST(X) = 908.9
SSwg(X) = 788.9
SST(Y) = 668.5
SSwg(Y) = 662.5
SSbg(Y) = 6.0
SCT = 625.9
SCwg = 652.8
4a. Adjustment of SST(Y)
The overall correlation between X and Y:
The proportion of the total variability of Y
attributable to its covariance with X is accordingly
(rT)2 = (+.803)2 = .645
we adjust SST(Y) by removing from it this
proportion of covariance. Since SST(Y)=668.5
4b. Adjustment of SSwg(Y)
The overall correlation between X and Y
within the two groups:
The proportion of the within-groups variability of
Y attributable to covariance with X is therefore
(rwg)2 = (+.903)2 = .815
we adjust SSwg(Y) by removing from it this
proportion of covariance. Since SSwg(Y)=662.5
4c. Adjustment of SSbg(Y)
The adjusted value of SSbg(Y) can then be
obtained through simple subtraction as
4d Adjustment of Means of Y for
Groups A and B
Purpose:
Adjust the group means of Y to the same starting point,
using the aggregate correlation between X and Y
within the two groups.
Recall for Linear Regression:
By Least Square Method:
We can get:
An increase by 1 unit of X is associated with
an average increase of .83 units of Y.
bwg:=.83
Original:
Mx
13.1
Adjusted :
My
29.2+2.45(.83)=31.23
15.55
-2.45
18.0 28.1 -2.45(.83)=26.07
?
Linear Model for ANCOVA:
Yij     i   ( X ij  X ..)   ij
Yij - b (Xij - X) = m + ai + e ij
［adjusted Yij]
Linear Model For ANOVA
Thus, as with the corresponding one way ANOVA,
The final step in a one-way analysis of covariance
Involves the calculation of an F-ratio of the general form.
4e. Analysis of Covariance Using Adjusted Values of SS
ANCOVA
Begins
Four sets
of
calculatio
n
Get rid of
covariate
from
SS(Y)&Mean(Y)
ANAOVA
F Test,
Interpretation
ANCOVA Assumptions
ANCOVA
GLM
ANCOVA Assumptions
Full model—the model involving all x’s
Reduced model – the model involving only those x’s from the full
model whose β coefficients are not hypothesized as 0.
𝑦 = 𝛽0 + 𝛽1 𝑥1 + ⋯ + 𝛽𝑔+1 𝑥𝑔+1 + ⋯ + 𝛽𝑘 𝑥𝑘 + 𝜀 (full)
𝑦 = 𝛽0 + 𝛽1 𝑥1 + ⋯ + 𝛽𝑔 + 𝜀 (reduced)
T.S.
~𝐹𝑘−𝑔,𝑛−𝑘+1
ANCOVA Assumptions
• No interaction between the factor and the covariate.
Interaction between 2 independent variables is present when
the effect of one on the outcome depends the value of the
other.
• The slope terms for within group regression doesn’t differ
• The regression line of different groups are parallel.
• Group 1:
• Group 2:
ANCOVA Assumptions
With interaction
Group 1:
= 𝛽0 + 𝛽1 + 𝛽2 + 𝛽3 𝑐𝑖
Group2:
= 𝛽0 + 𝛽2 𝑐𝑖
Slopes not equal: 𝛽2 + 𝛽3 ≠ 𝛽2
ANCOVA Assumptions
Testing The Interaction for Signifiance
Interaction of Interest: the interaction between the covariate
and the dummy variable.
Interaction term : 𝑥1 𝑥2
FULL MODEL
REDUCED
MODEL
k= 3, g=2
ANCOVA Assumptions
Example:
ANCOVA Assumptions
𝑦 : the response, i.e. anxiety score
𝑥1 : the drug dose
Drug A: 𝑥2 = 0
Drug B: 𝑥2 = 1
ANCOVA Assumptions
Reject 𝐻0 , 𝑡ℎ𝑒𝑟𝑒 𝑖𝑠 𝑎𝑛 𝑖𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑜𝑛 𝑡𝑒𝑟𝑚. 𝑇ℎ𝑖𝑠 𝑖𝑚𝑝𝑙𝑖𝑒𝑠: 𝑡ℎ𝑒 𝑚𝑒𝑎𝑛
Anxiety level increases at different rate as the drug dose is increased
for drug A and drug B.
SAS Implementation
SAS code
1. Initial data exploration
proc contents data=Instruction;
run;
proc means data=Instruction N MEAN STD MAXDEC=2;
class method;
var prescore postscore;
run;
Proc freq data=instruction;
tables method;
Run;
proc sgplot data = instruction;
reg x = Prescore y = PostScore / group = method;
run;
SAS code
2. ANOVA Model
PROC GLM DATA=Instruction;
CLASS method ;
No class statement in PROC
MODEL PostScore=method/solution ;
MEANS method / deponly ;
RUN;
QUIT;
We can also use:
PROC ANOVA data=instruction;
class method;
model postscore=method;
means method/Tukey;
run;quit;
REG
SAS code
Another way: creating dummy variables for Method
data instruction_dummy;
set instruction;
/**create dummy variables**/
if method="A" then do; dummy1=0; dummy2=0;end;
if method="B" then do; dummy1=1; dummy2=0;end;
if method="C" then do; dummy1=0; dummy2=1;end;
run;
TITLE“ Regression model for Instruction method
dataset";
PROC GLM DATA=Instruction_dummy;
MODEL PostScore=dummy1 dummy2 /solution;
RUN;
Now we don’t need Class statement in PROC GLM
ANOVA output
Accept Null hypothesis:
H0: 𝜇1 = 𝜇2 = 𝜇3
Accept Null hypothesis:
H0:𝛽1 = 0 ;
H0:𝛽2 = 0
Missing line because there
are only two dummy
variables
SAS code
3. ANCOVA Model
ods graphic on;
Include covariate x:
PreScore
proc glm data=instruction plot=meanplot(cl);
class method;
model PostScore = method PreScore/solution;
lsmeans method / pdiff;
output out=out p=yhat r=resid stdr=eresid;
run;
quit;
ods graphic off;
ANCOVA output
𝛽1 − 𝛽2 is almost 0, we
may expect 𝜇1 = 𝜇2
Accept Alternative Hypothesis:
H1: at least one 𝜏𝑖 ≠ 0
𝛽1 ≠ 0
𝛽2 ≠ 0
𝜇1 ≠ 𝜇3
𝜇2 ≠ 𝜇3
Covariate X is significant
ANCOVA output
Adjusted Means:
𝜇1 = 𝜇2 ; 𝜇1 ≠ 𝜇3 ;
𝜇2 ≠ 𝜇3 ;
SAS code
3. Checking on the homogeneity of Slope
/**1) perform an analysis that shows the slopes of each of the
lines***/
PROC SORT DATA=instruction;
BY method;
RUN;
PROC GLM DATA=instruction;
BY method;
MODEL PostScore = PreScore / SOLUTION ;
RUN;
QUIT;
/** 2) method*prescore effect tests if the three slopes are
equal**/
PROC GLM DATA=instruction;
CLASS method;
Interaction term is not
MODEL PostScore = method PreScore method*PreScore;
significant: Assumption met
RUN;
QUIT;
Include Interaction Term
Comparing before and after adjusted means
Acknowledge:
•
•
•
•
•
http://www.ats.ucla.edu/stat/sas/library/hetreg.htm
http://www.unt.edu/rss/class/mike/6810/ANCOVA.pdf
http://www.stat.cmu.edu/~hseltman/309/Book/chapter10.pdf
SAS/STAT(R) 9.22 User's Guide
Text book: Statistics and Data Analysis from Elementary to
Intermediate