Analysis of Variance (ANOVA)
and Multivariate Analysis of
Variance (MANOVA)
Session 6
Analysis of Variance
•
•
•
•
•
•
•
•
•
•
Using Statistics.
The Hypothesis Test of Analysis of Variance.
The Theory and Computations of ANOVA.
The ANOVA Table and Examples.
Further Analysis.
Models, Factors, and Designs.
Two-Way Analysis of Variance.
Blocking Designs.
Using the Computer.
Summary and Review of Terms.
6-1 ANOVA: Using Statistics
•
ANOVA (ANalysis Of VAriance) is a statistical method for
determining the existence of differences among several
population means.
 ANOVA is designed to detect differences among means
from populations subject to different treatments.
 ANOVA is a joint test
 The equality of several population means is tested
simultaneously or jointly.
 ANOVA tests for the equality of several population
means by looking at two estimators of the population
variance (hence, analysis of variance).
Analysis of Variance: Using
Statistics (continued)
•
In an analysis of variance:
 We have r independent random samples, each one
corresponding to a population subject to a different
treatment.
 We have:
 n = n1+ n2+ n3+ ...+nr total observations.
r sample means: x1, x2 , x3 , ... , xr


These r sample means can be used to calculate an estimator
of the population variance. If the population means are equal,
we expect the variance among the sample means to be small.
r sample variances: s12, s22, s32, ...,sr2
 These sample variances can be used to find a pooled
estimator of the population variance.
Analysis of Variance: Assumptions
•
•
We assume independent random sampling from each of the r
populations
We assume that the r populations under study:
 are normally distributed,
 with means mi that may or may not be equal,
 but with equal variances, si2.
s
m1
Population 1
m2
Population 2
m3
Population 3
6-2 The Hypothesis Test of
Analysis of Variance
The hypothesis test of analysis of variance:
H0: m1 = m2 = m3 = m4 = ... mr
H1: Not all mi (i = 1, ..., r) are equal.
The test statistic of analysis of variance:
Estimate of variance based on means from r samples
F(r-1, n-r) =
Estimate of variance based on all sample observations
That is, the test statistic in an analysis of variance is
based on the ratio of two estimators of a population
variance, and is therefore based on the F distribution,
with (r-1) degrees of freedom in the numerator and (n-r)
degrees of freedom in the denominator.
When the Null Hypothesis Is
True
When the null hypothesis is true:
H0: m
x
x
= m =m
We would expect the sample means to be
nearly equal, as in this illustration. And we
would expect the variation among the
sample means (between sample) to be
small, relative to the variation found around
the individual sample means (within sample).
If the null hypothesis is true, the numerator
in the test statistic is expected to be small,
relative to the denominator:
F(r-1, n-r)=
x
Estimate of variance based on means from r samples
Estimate of variance based on all sample observations
When the Null Hypothesis Is
False
x
x
x
When the null hypothesis is false:
m is equal to m but not to m ,
m is equal to m but not to m ,
m is equal to m but not to m , or
m , m , and m are all unequal.
In any of these situations, we would not expect the sample means to all
be nearly equal. We would expect the variation among the sample
means (between sample) to be large, relative to the variation around the
individual sample means (within sample).
If the null hypothesis is false, the numerator in the test statistic is
expected to be large, relative to the denominator:
F(r-1,
n-r)=
Estimate of variance based on means from r samples
Estimate of variance based on all sample observations
The ANOVA Test Statistic for r = 4
Populations and n = 54 Total Sample
Observations
• Suppose we have 4 populations, from each of which we
draw an independent random sample, with n1 + n2 + n3 + n4
= 54. Then our test statistic is:
F(4-1, 54-4)= F(3,50) Estimate
= Estimate of variance based on means from 4 samples
of variance based on all 54 sample observations
F Distributionwith3 and 50 Degrees of Freedom
0.7
0.6
f(F)
0.5
0.4
0.3
0.2
a=0.05
0.1
0.0
0
1
2
3
2.79
4
5
F(3,50)
The nonrejection region (for a=0.05)in
this instance is F £ 2.79, and the
rejection region is F > 2.79. If the test
statistic is less than 2.79 we would not
reject the null hypothesis, and we would
conclude the 4 population means are
equal. If the test statistic is greater than
2.79, we would reject the null hypothesis
and conclude that the four population
means are not equal.
Example 6-1
Randomly chosen groups of customers were served different types of coffee and asked to
rate the coffee on a scale of 0 to 100: 21 were served pure Brazilian coffee, 20 were
served pure Colombian coffee, and 22 were served pure African-grown coffee.
The resulting test statistic was F = 2.02
H 0 : m1 = m 2 = m 3
H 1 : Not all three means equal
n 2 = 20
n 3 = 22
0.7
0.6
n = 21 + 20 + 22 = 63
r = 3
The critical point for a = 0.05 is:
F
= F
= F
= 3.15
 r -1,( n -r ) 
 31, 63 3 
 2 , 60 
F = 2 .02  F
= 3.15
 2 , 60 
H 0 cannot be rejected, and we cannot conclude that any of the
population means differs significantly from the others.
0.5
f(F)
n 1 = 21
F Distribution with 2 and 60 Degrees of Freedom
0.4
0.3
0.2
a=0.05
0.1
0.0
0
1
Test Statistic=2.02
2
3
4
F(2,60)=3.15
5
F
6-3 The Theory and
Computations of ANOVA: The
Grand Mean
The grand mean, x, is the mean of all n = n1+ n2+ n3+...+ nr
observations in all r samples.
The mean of sample i (i = 1,2,3, . . . , r):
ni
 x
j =1 ij
xi =
ni
The grand mean, the mean of all data points:
r ni
r
  x
n x
i =1 j =1 ij
i =1 i i
xi =
=
n
n
where x is the particular data point in position j within the sample from population i.
ij
The subscript i denotes the population, or treatment, and runs from 1 to r. The subscript j
denotes the data point within the sample from population i; thus, j runs from 1 to n j .
Using the Grand Mean: Table
6-1
Treatment (j)
Sample point(j)
I=1 Triangle
1
Triangle
2
Triangle
3
Triangle
4
Mean of Triangles
I=2 Square
1
Square
2
Square
3
Square
4
Mean of Squares
I=3 Circle
1
Circle
2
Circle
3
Mean of Circles
Grand mean of all data points
Value(x ij)
4
5
7
8
6
10
11
12
13
11.5
1
2
3
2
6.909
x1=6
x2=11.5
x=6.909
x3=2
0
5
10
Distance from data point to its sample mean
Distance from sample mean to grand mean
If the r population means are different (that
is, at least two of the population means are
not equal), then it is likely that the variation
of the data points about their respective
sample means (within sample variation) will
be small relative to the variation of the r
sample means about the grand mean
(between sample variation).
The Theory and Computations of
ANOVA: Error Deviation and
Treatment Deviation
We define an error deviation as the difference between a data point
and its sample mean. Errors are denoted by e, and we have:
eij = xij  xi
We define a treatment deviation as the deviation of a sample mean
from the grand mean. Treatment deviations, ti , are given by:
ti = xi  x
The ANOVA principle says:
When the population means are not equal, the “average” error
(within sample) is relatively small compared with the “average”
treatment (between sample) deviation.
The Theory and Computations of
ANOVA: The Total Deviation
The total deviation (Totij) is the difference between a data point (xij) and the grand mean (x):
Totij=xij - x
For any data point xij:
Tot = t + e
That is:
Total Deviation = Treatment Deviation + Error Deviation
Consider data point x24=13 from table 9-1. The
mean of sample 2 is 11.5, and the grand mean is
6.909, so:
e24 = x 24  x 2 = 13  11.5 = 1.5
t 2 = x 2  x = 11.5  6.909 = 4 .591
Tot 24 = t 2  e24 = 1.5  4 .591 = 6.091
or
Tot 24 = x 24  x = 13  6.909 = 6.091
Total deviation:
Tot24=x24-x=6.091
Error deviation:
e24=x24-x2=1.5
x24=13
Treatment deviation:
t2=x2-x=4.591
x2=11.5
x=6.909
0
5
10
The Theory and Computations
of ANOVA: Squared Deviations
Total Deviation = Treatment Deviation + Error Deviation
The total deviation is the sum of the treatment deviation and the error deviation:
t + e = ( x  x )  ( xij  x ) = ( xij  x ) = Tot ij
i
ij
i
i
Notice that the sample mean term ( x ) cancels out in the above addition, which
i
simplifies the equation.
Squared Deviations
2
2
2
+e
= ( x  x )  ( xij  x )
i
ij
i
i
2
2
Tot ij = ( xij  x )
t
2
The Theory and Computations of
ANOVA: The Sum of Squares
Principle
Sums of Squared Deviations
n
n
j
j
r
r
r
2
2
2

 Tot
 e
=  nt
+ 
ij
i =1j =1
i =1 ii
i = 1 j = 1 ij
n
n
j
j
r
r
r
2
2

 (x  x) =  n (x  x)  
 ( x  x )2
i
i = 1 j = 1 ij
i =1 i i
i = 1 j = 1 ij
SST =
SSTR
+
SSE
The Sum of Squares Principle
The total sum of squares (SST) is the sum of two terms: the sum of
squares for treatment (SSTR) and the sum of squares for error (SSE).
SST = SSTR + SSE
The Theory and Computations
of ANOVA: Picturing The Sum of
Squares Principle
SSTR
SSTE
SST
SST measures the total variation in the data set, the variation of all
individual data points from the grand mean.
SSTR measures the explained variation, the variation of individual sample
means from the grand mean. It is that part of the variation that is possibly
expected, or explained, because the data points are drawn from different
populations. It’s the variation between groups of data points.
SSE measures unexplained variation, the variation within each group that
cannot be explained by possible differences between the groups.
The Theory and Computations
of ANOVA: Degrees of
Freedom
The number of degrees of freedom associated with SST is (n - 1).
n total observations in all r groups, less one degree of freedom
lost with the calculation of the grand mean
The number of degrees of freedom associated with SSTR is (r - 1).
r sample means, less one degree of freedom lost with the
calculation of the grand mean
The number of degrees of freedom associated with SSE is (n-r).
n total observations in all groups, less one degree of freedom
lost with the calculation of the sample mean from each of r groups
The degrees of freedom are additive in the same way as are the sums of
squares:
df(total) = df(treatment) + df(error)
(n - 1) = (r - 1)
+ (n - r)
The Theory and Computations
of ANOVA: The Mean Squares
Recall that the calculation of the sample variance involves the division of the
sum of squared deviations from the sample mean by the number of degrees
of freedom. This principle is applied as well to find the mean squared
deviations within the analysis of variance.
Mean square treatment (MSTR):
SSTR
MSTR =
( r  1)
Mean square error (MSE):
SSE
MSE =
(n  r )
Mean square total (MST):
SST
MST =
(n  1)
(Note that the additive properties of sums of squares do not extend to the
mean squares. MST ¹ MSTR + MSE).
The Theory and Computations
of ANOVA: The Expected Mean
Squares
2
E ( MSE ) = s
and
2

m

m
n
(
)
= s 2 when the null hypothesis is true
2
i
i
E ( MSTR) = s 
r 1
> s 2 when the null hypothesis is false
where mi is the mean of population i and m is the combined mean of all r populations.
That is, the expected mean square error (MSE) is simply the common population variance
(remember the assumption of equal population variances), but the expected treatment sum of
squares (MSTR) is the common population variance plus a term related to the variation of the
individual population means around the grand population mean.
If the null hypothesis is true so that the population means are all equal, the second term in
the E(MSTR) formulation is zero, and E(MSTR) is equal to the common population variance.
Expected Mean Squares and
the
ANOVA Principle
When the null hypothesis of ANOVA is true and all r population means
are equal, MSTR and MSE are two independent, unbiased estimators
of the common population variance s2.
On the other hand, when the null hypothesis is false, then MSTR will
tend to be larger than MSE.
So the ratio of MSTR and MSE can be used as an indicator of the
equality or inequality of the r population means.
This ratio (MSTR/MSE) will tend to be near to 1 if the null hypothesis is
true, and greater than 1 if the null hypothesis is false. The ANOVA
test, finally, is a test of whether (MSTR/MSE) is equal to, or greater
than, 1.
The Theory and Computations of
ANOVA: The F Statistic
Under the assumptions of ANOVA, the ratio (MSTR/MSE) possess
an F distribution with (r-1) degrees of freedom for the numerator
and (n-r) degrees of freedom for the denominator when the null
hypothesis is true.
The test statistic in analysis of variance:
F( r -1,n -r )
=
MSTR
MSE
6-4 The ANOVA Table and
Examples
Treatment (i)
(x ij -xi ) (x ij -xi )2
i
j
Value (x ij )
Triangle
1
1
4
-2
4
Triangle
1
2
5
-1
1
Triangle
1
3
7
1
1
Triangle
1
4
8
2
4
Square
2
1
10
-1.5
2.25
Square
Square
Square
2
2
2
2
3
4
11
12
13
-0.5
0.5
1.5
0.25
0.25
2.25
Circle
3
1
1
-1
1
Circle
3
2
2
0
0
Circle
3
3
3
1
1
0
17
73
Treatment
(xi -x)
(xi -x)
2
ni (x i -x)
2
Triangle
-0.909
0.826281
3.305124
Square
4.591
21.077281
84.309124
Circle
-4.909
124.098281
72.294843
159.909091
n
j
r
 ( x  x ) 2 = 17
SSE = 
i
i = 1 j = 1 ij
r
2
SSTR =  n ( x  x ) = 159 .9
i =1 i i
SSTR
159 .9
=
= 79 .95
MSTR =


r 1
( 3 1)
SSTR 17
=
= 2 .125
MSE =

n r
8
MSTR
79 .95
=
=
= 37 .62 .
F
MSE
2 .125
( 2 ,8 )
Critical point ( a = 0.01): 8.65
H may be rejected at the 0.01 level
0
of significance.
ANOVA Table
Source of
Variation
Sum of
Squares
Degrees of
Freedom Mean Square F Ratio
Treatment SSTR=159.9 (r-1)=2
MSTR=79.95 37.62
Error
SSE=17.0
(n-r)=8
MSE=2.125
Total
SST=176.9
(n-1)=10
MST=17.69
F Distribution for 2 and 8 Degrees of Freedom
0.7
0.6
0.5
Computed test statistic=37.62
f(F)
0.4
0.3
0.2
0.01
0.1
0.0
0
10
8.65
F(2,8)
The ANOVA Table summarizes the
ANOVA calculations.
In this instance, since the test
statistic is greater than the critical
point for an a=0.01 level of
significance, the null hypothesis may
be rejected, and we may conclude
that the means for triangles, squares,
and circles are not all equal.
Using the Computer
Treat
Value
1
1
1
1
2
2
2
2
3
3
3
4
5
7
8
10
11
12
13
1
2
3
MTB > Oneway 'Value' 'Treat'.
One-Way Analysis of Variance
Analysis of Variance on Value
Source DF
SS
MS
F
Treat
2 159.91 79.95 37.63
Error
8 17.00
2.12
Total
10 176.91
p
0.000
The MINITAB output includes not only the ANOVA table and the test
statistic, but it also gives a p-value corresponding to the calculated Fratio. In this instance the p-value is approximately 0, so the null
hypothesis can be rejected at any common level of significance.
Using the Computer
Anova: Single Factor
SUMMARY
Groups
TRIANGLE
SQUARE
CIRCLE
Count
Sum
4
24
4
46
3
6
ANOVA
Source of Variation
Between Groups
Within Groups
SS
159.9090909
17
Total
176.9090909
df
Average
Variance
6 3.333333333
11.5 1.666666667
2
1
MS
F
P-value
F crit
2 79.95454545 37.62566845 8.52698E-05 8.64906724
8
2.125
10
The EXCEL output is created by selecting ANOVA: SINGLE FACTOR
option from the DATA ANALYSIS toolkit. The critical F value is based
on a = 0.01. The p-value is very small, so again the null hypothesis
can be rejected at any common level of significance.
Example 6-2: Club Med
Club Med has conducted a test to determine whether its Caribbean resorts are equally well liked by
vacationing club members. The analysis was based on a survey questionnaire (general satisfaction, on
a scale from 0 to 100) filled out by a random sample of 40 respondents from each of 5 resorts.
Resort
Guadeloupe
89
Source of
Variation
Martinique
75
Treatment
SSTR= 14208 (r-1)= 4
MSTR= 3552
Eleuthra
73
Error
SSE=98356
(n-r)= 195
MSE= 504.39
Paradise Island
91
Total
SST=112564
(n-1)= 199
MST= 565.65
St. Lucia
85
SSE=98356
Sum of
Squares
Degrees of
Freedom
Mean Square
F Ratio
7.04
F Distribution with 4 and 200 Degrees of Freedom
0.7
0.6
0.5
f(F)
SST=112564
Mean Response (x i )
Computed test statistic=7.04
0.4
0.3
0.2
0.01
0.1
0.0
0
3.41
F(4,200)
The resultant F
ratio is larger than
the critical point for
a = 0.01, so the
null hypothesis may
be rejected.
Example 6-3: Job Involvement
Source of
Variation
Sum of
Squares
Degrees of
Freedom
Mean Square
F Ratio
Treatment
SSTR= 879.3
(r-1)=3
MSTR= 293.1
8.52
Error
SSE= 18541.6
(n-r)= 539
MSE=34.4
Total
SST= 19420.9
(n-1)=542
MST= 35.83
Given the total number of observations (n = 543), the number of groups
(r = 4), the MSE (34. 4), and the F ratio (8.52), the remainder of the ANOVA
table can be completed. The critical point of the F distribution for a = 0.01
and (3, 400) degrees of freedom is 3.83. The test statistic in this example is
much larger than this critical point, so the p value associated with this test
statistic is less than 0.01, and the null hypothesis may be rejected.
Example 6-4: NBA Franchise
See text for data and information on the problem
Anova: Single Factor
SUMMARY
Groups
Michael
Damon
Allen
Count
Sum
Average
Variance
21 1979 94.23809524 8.59047619
21 1644 78.28571429 41.11428571
21 1381 65.76190476 352.8904762
ANOVA
Source of Variation
SS
Between Groups
8555.52381
Within Groups
8051.904762
Total
16607.42857
df
MS
F
P-value
F crit
2 4277.761905 31.87639718 3.69732E-10 3.150411487
60 134.1984127
62
The test statistic value is 31.8764, way over the critical point for F(2, 60) of 3.15 when a = 0.05.
The GM should do whatever it takes to sign Michael.
6-5 Further Analysis
Data
Do Not Reject H0
Stop
ANOVA
Reject H0
The sample means are unbiased estimators of the population means.
The mean square error (MSE) is an unbiased estimator of the common
population variance.
Further
Analysis
The ANOVA Diagram
Confidence Intervals
for Population Means
Tukey Pairwise
Comparisons Test
Confidence Intervals for
Population Means
A (1 - a ) 100% confidence interval for mi , the mean of population i:
MSE
xi  ta
ni
2
where t a is the value of the t distribution with n - r ) degrees of
2
freedom that cuts off a right - tailed area of
a
2
.
Confidence Intervals for
Population Means
Resort
Mean Response (x i )
Guadeloupe
89
Martinique
75
Eleuthra
73
Paradise Island
91
St. Lucia
85
SST = 112564
SSE = 98356
ni = 40
n = (5)(40) = 200
MSE = 504.39
MSE
504.39
= xi 1.96
= xi  6.96
ni
40
2
89  6.96 = [82.04, 95.96]
75  6.96 = [ 68.04,81.96]
73  6.96 = [ 66.04, 79.96]
91  6.96 = [84.04, 97.96]
85  6.96 = [ 78.04, 91.96]
xi  ta
The Tukey Pairwise Comparison
Test
The Tukey Pairwise Comparison test, or Honestly Significant Differences (MSD) test,
allows us to compare every pair of population means with a single level of significance.
It is based on the studentized range distribution, q, with r and (n-r) degrees of freedom.
The critical point in a Tukey Pairwise Comparisons test is the Tukey Criterion:
T = qa
MSE
ni
where ni is the smallest of the r sample sizes.
The test statistic is the absolute value of the difference between the appropriate sample
means, and the null hypothesis is rejected if the test statistic is greater than the
critical point of the Tukey Criterion
N o te th a t th e re a re

r
=
r!
p a irs o f p o p u la tio n m e a n s to c o m p a re . F o r e x a m p le , if r
2 !( r  2 ) !
H0: m1 = m 2
H0: m1 = m 3
H0:m2 = m3
H1: m1  m 2
H1: m1  m 3
H1: m 2  m 3
2
=
3:
The Tukey Pairwise Comparison
Test: The Club Med Example
The test statistic for each pairwise test is the absolute difference between the appropriate
sample means.
i
Resort
Mean
I. H0: m1 = m2
VI. H0: m2 = m4
1
Guadeloupe
89
H1: m1  m2
H1: m2  m4
2
Martinique
75
|89-75|=14>13.7*
|75-91|=16>13.7*
3
Eleuthra
73
II. H0: m1 = m3
VII. H0: m2 = m5
4
Paradise Is.
91
H1: m1  m3
H1: m2  m5
5
St. Lucia
85
|89-73|=16>13.7*
|75-85|=10<13.7
III. H0: m1 = m4
VIII. H0: m3 = m4
The critical point T0.05 for
H1: m1  m4
H1: m3  m4
r=5 and (n-r)=195
|89-91|=2<13.7
|73-91|=18>13.7*
degrees of freedom is:
IV. H0: m1 = m5
IX. H0: m3 = m5
H1: m1  m5
H1: m3  m5
MSE
T = qa
|89-85|=4<13.7
|73-85|=12<13.7
ni
V. H0: m2 = m3
X. H0: m4 = m5
504.4
H1: m2  m3
H1: m4  m5
= 3.86
= 13.7
|75-73|=2<13.7
|91-85|= 6<13.7
40
Reject the null hypothesis if the absolute value of the difference between the sample means
is greater than the critical value of T. (The hypotheses marked with * are rejected.)
Picturing the Results of a Tukey
Pairwise Comparisons Test: The
Club Med Example
We rejected the null hypothesis which compared the means of
populations 1 and 2, 1 and 3, 2 and 4, and 3 and 4. On the other
hand, we accepted the null hypotheses of the equality of the means of
populations 1 and 4, 1 and 5, 2 and 3, 2 and 5, 3 and 5, and 4 and 5.
m
m
m
m
m
3
2
5
1
4
The bars indicate the three groupings of populations with possibly
equal means: 2 and 3; 2, 3, and 5; and 1, 4, and 5.
6-6 Models, Factors and Designs
• A statistical model is a set of equations and assumptions that capture the
essential characteristics of a real-world situation
 The one-factor ANOVA model:
xij=mi+eij=m+ti+eij
where eij is the error associated with the jth member of the ith population. The errors are
assumed to be normally distributed with mean 0 and variance s2.
•
•
A factor is a set of populations or treatments of a single kind. For example:
 One factor models based on sets of resorts, types of airplanes, or kinds of sweaters
 Two factor models based on firm and location
 Three factor models based on color and shape and size of an ad.
Fixed-Effects and Random Effects
 A fixed-effects model is one in which the levels of the factor under study (the treatments)
are fixed in advance. Inference is valid only for the levels under study.
 A random-effects model is one in which the levels of the factor under study are
randomly chosen from an entire population of levels (treatments). Inference is valid for
the entire population of levels.
Experimental Design
•
•
A completely-randomized design is one in which the
elements are assigned to treatments completely at random.
That is, any element chosen for the study has an equal
chance of being assigned to any treatment.
In a blocking design, elements are assigned to treatments
after first being collected into homogeneous groups.
 In a completely randomized block design, all members of
each block (homogeneous group) are randomly assigned to
the treatment levels.
 In a repeated measures design, each member of each
block is assigned to all treatment levels.
6-7 Two-Way Analysis of
Variance
• In a two-way ANOVA, the effects of two factors or treatments can be investigated
•
•
simultaneously. Two-way ANOVA also permits the investigation of the effects of
either factor alone and of the two factors together.
 The effect on the population mean that can be attributed to the levels of either
factor alone is called a main effect.
 An interaction effect between two factors occurs if the total effect at some pair
of levels of the two factors or treatments differs significantly from the simple
addition of the two main effects. Factors that do not interact are called additive.
Three questions answerable by two-way ANOVA:
 Are there any factor A main effects?
 Are there any factor B main effects?
 Are there any interaction effects between factors A and B?
For example, we might investigate the effects on vacationers’ ratings of resorts by
looking at five different resorts (factor A) and four different resort attributes (factor
B). In addition to the five main factor A treatment levels and the four main factor B
treatment levels, there are (5*4=20) interaction treatment levels.3
The Two-Way ANOVA Model
•
xijk=m+ai+ bj + (abijk + eijk
 where m is the overall mean;
 ai is the effect of level i(i=1,...,a) of factor A;
 bj is the effect of level j(j=1,...,b) of factor B;
 abjj is the interaction effect of levels i and j;
 ejjk is the error associated with the kth data point from

level i of factor A and level j of factor B.
ejjk is assumed to be distributed normally with mean
zero and variance s2 for all i, j, and k.
Two-Way ANOVA Data Layout:
Club Med Example
Factor B:
Attribute
Factor A: Resort
Friendship
Sports
Culture
Excitement
Guadeloupe
n11
n12
n13
n14
Martinique
n21
n22
n23
n24
Graphical Display of Effects
Eleuthra
n31
n32
n33
n34
R a ting
St. Lucia
n51
n52
n53
n54
Eleuthra/sports interaction:
Combined effect greater than
additive main effects
Rating
Friendship
Excitement
Sports
Culture
Paradise
Island
n41
n42
n43
n44
Friendship
Attribute
Excitement
Sports
Culture
Eleuthra
St. Lucia
Paradise island
Martinique
Guadeloupe
Resort
Resort
St. Lucia
Paradise Island
Eleuthra
Guadeloupe
Martinique
Hypothesis Tests a Two-Way
ANOVA
•
Factor A main effects test:
•
Factor B main effects test:
•
Test for (AB) interactions:
 H0: ai=0 for all i=1,2,...,a
 H1: Not all ai are 0
 H0: bj=0 for all j=1,2,...,b
 H1: Not all bi are 0
 H0: abij=0 for all i=1,2,...,a and j=1,2,...,b
 H1: Not all abij are 0
Sums of Squares

In a two-way ANOVA:

xijk=m+ai+ bj + (abijk + eijk


SST = SSTR +SSE
SST = SSA + SSB +SS(AB)+SSE
SST = SSTR  SSE
  ( x  x )2 =   ( x  x )2    ( x  x )2
SSTR = SSA  SSB  SS ( AB)
=   ( x  x )2    ( x  x )2    ( x  x  x  x )2
i
j
ij i
j
The Two-Way ANOVA Table
Source of
Variation
Sum of
Squares
Degrees
of Freedom
Mean Square
F Ratio
Factor A
SSA
a-1
MSA =
SSA
a 1
MSA
F =
MSE
Factor B
SSB
b-1
MSB =
SSB
b 1
MSB
F=
MSE
Interaction SS(AB)
(a-1)(b-1)
Error
SSE
ab(n-1)
SS ( AB)
MS ( AB) = 
( a 1)(b 1)
SSE
MSE =
ab( n 1)
Total
SST
abn-1
F =
MS ( AB )
MSE
A Main Effect Test: F(a-1,ab(n-1))
B Main Effect Test: F(b-1,ab(n-1))
(AB) Interaction Effect Test: F((a-1)(b-1),ab(n-1))
Example 6-4: Two-Way ANOVA
(Location and Artist)
Source of
Variation
Sum of
Squares
Degrees
of Freedom
Location
1824
2
912
8.94
*
Artist
2230
2
1115
10.93
*
804
4
201
1.97
Error
8262
81
102
Total
13120
89
Interaction
Mean Square
F Ratio
a=0.01, F(2,81)=4.88  Both main effect null hypotheses are rejected.
a=0.05, F(2,81)=2.48  Interaction effect null hypotheses are not rejected.
Hypothesis Tests
F Distribution with 2 and 81 Degrees of Freedom
F Distribution with 4 and 81 Degrees of Freedom
0.7
0.7
Location test statistic=8.94
Artist test statistic=10.93
0.6
0.4
Interaction test statistic=1.97
0.5
f(F)
f(F)
0.5
0.6
0.4
0.3
0.3
a=0.01
0.2
a=0.05
0.2
0.1
0.1
F
0.0
0.0
0
1
2
3
4
5
F0.01=4.88
6
F
0
1
2
3
F0.05=2.48
4
5
6
Overall Significance Level and
Tukey Method for Two-Way ANOVA
Kimball’s Inequality gives an upper limit on the true probability of at
least one Type I error in the three tests of a two-way analysis:
a  1- (1-a1) (1-a2) (1-a3)
Tukey Criterion for factor A:
T = qa
MSE
bn
where the degrees of freedom of the q distribution are now a and ab(n1). Note that MSE is divided by bn.
Three-Way ANOVA Table
Source of
Variation
Sum of
Squares
Degrees
of Freedom
Mean Square
SSA
a 1
F Ratio
MSA
F=
MSE
Factor A
SSA
a-1
MSA =
Factor B
SSB
b-1
SSB
MSB = 
b 1
F =
Factor C
SSC
c-1
MSC =
SSC
c 1
F =
Interaction
(AB)
Interaction
(AC)
Interaction
(BC)
SS(AB)
(a-1)(b-1)
SS(AC)
(a-1)(c-1)
SS(BC)
(b-1)(c-1)
SS ( AB)
( a 1)(b 1)
SS ( AC)
MS ( AC) = 
(a 1)(c 1)
SS ( BC)
MS ( BC) = 
(b 1)(c 1)
Interaction
(ABC)
Error
SS(ABC)
(a-1)(b-1)(c-1)
SSE
abc(n-1)
Total
SST
abcn-1
MS ( AB) =
SS ( ABC)
(a 1)(b 1)(c 1)
SSE
MSE =
abc( n 1)
MS ( ABC) =
MSB
MSE
MSC
MSE
MS ( AB )
F =
MSE
MS ( AC )
MSE
MS ( BC)
F=
MSE
F =
F=
MS( ABC)
MSE
6-8 Blocking Designs
• A block is a homogeneous set of subjects, grouped to minimize
•
•
within-group differences.
A competely-randomized design is one in which the elements
are assigned to treatments completely at random. That is, any
element chosen for the study has an equal chance of being
assigned to any treatment.
In a blocking design, elements are assigned to treatments after
first being collected into homogeneous groups.
– In a completely randomized block design, all members of
each block (homogenous group) are randomly assigned to the
treatment levels.
– In a repeated measures design, each member of each block
is assigned to all treatment levels.
ANOVA Table for Blocking
Designs: Example 6-5
Source of Variation Sum of Squares Degress of Freedom Mean Square
Blocks
Treatments
Error
Total
SSBL
SSTR
SSE
SST
Source of Variation
Blocks
Treatments
Error
Total
n-1
r-1
(n -1)(r - 1)
nr - 1
F Ratio
MSBL = SSBL/(n-1) F = MSBL/MSE
MSTR = SSTR/(r-1) F = MSTR/MSE
MSE = SSE/(n-1)(r-1)
Sum of Squares
df
Mean Square F Ratio
2750
39
70.51
0.69
2640
2
1320
12.93
7960
78
102.05
13350 119
a = 0.01, F(2, 78) = 4.88
Using the Computer
MTB > ONEWAY C1, C2;
SUBC> TUKEY 0.05.
One-Way Analysis of Variance
Analysis of Variance on C1
Source DF
SS
MS
F
p
Method
2 1348.45 674.23 69.42 0.000
Error
63 611.91
9.71
Total
65 1960.36
Individual 95% CIs For Mean
Based on Pooled StDev
Level
N
Mean StDev ----+---------+---------+---------+-1 22 22.773 3.131
(--*--)
2 22 30.636 3.824
(---*--)
3 22 19.955 2.171 (--*--)
----+---------+---------+---------+-Pooled StDev = 3.117
20.0
24.0
28.0
32.0
Tukey's pairwise comparisons
Family error rate = 0.0500
Individual error rate = 0.0193
Critical value = 3.39
Intervals for (column level mean) - (row level mean)
1
2
2 -10.116
-5.611
3
0.566 8.429
5.071 12.934
Using the Computer:
Example 6-6 Using Excel
Anova: Single Factor
SUMMARY
Groups
METHOD A
METHOD B
METHOD C
Count
22
22
22
ANOVA
Source of Variation
SS
Between Groups
1348.454545
Within Groups
611.9090909
Total
95% Confidence
Intervals on Means
LOWER BOUND
UPPER BOUND
1960.363636
Sum
501
674
439
Average
Variance
22.77272727 9.803030303
30.63636364 14.62337662
19.95454545 4.712121212
df
MS
F
P-value
F crit
2 674.2272727 69.41606002 1.18041E-16 3.14280868
63 9.712842713
65
A
Method
B
C
21.44493
24.10052
29.309
31.964
18.62675
21.28234
6-9 Multivariate Analysis
•
•
•
•
•
•
Introduction.
The Multivariate Normal Distribution.
Discriminant Analysis.
Principal Components and Factor
Analysis.
Using the Computer.
Summary and Review of Terms.
6-10 The Multivariate Normal
Distribution
• A k-dimensional (vector) random variable X:
•
•
 X = (X1, X2, X3..., Xk)
A realization of a k-dimensional random variable
X:
 x = (x1, x2, x3..., xk)
A joint cumulative probability distribution
function of a k-dimensional random variable X:
 F(x1, x2, x3..., xk) = P(X1x1, X2x2,..., Xkxk)
The Multivariate Normal
Distribution
A multivariate normal random variable has the following
probability density function:
  1 ( X  m )  1( X  m )
e  2

f (x1, x2 , , x ) =
k
1
k
2
 2  
1
2

where X is the vector random variable, the term m = (m 1 , m 2 , , m k )
is the vector of means of the component variables X i , and  is
the variance - covariance matrix. The operations ' and -1 are
transposition and inversion of matrices, respectively, and
denotes the determinant of a matrix.
Picturing the Bivariate Normal
Distribution
f(x1,x2)
x2
x1
6-11 Discriminant Analysis
In a discriminant analysis, observations are classified into two or more
groups, depending on the value of a multivariate discriminant function.
As the figure illustrates, it
may be easier to classify
X2
observations by looking at
them from another direction.
Group 1
The groups appear more
separated when viewed
m1
from a point perpendicular
Group 2
to Line L, rather than from a
m2
point perpendicular to the X1
or X2 axis. The discriminant
Line L
function gives the direction
that maximizes the
separation between the
groups.
X1
The Discriminant Function
The form of the estimated predicted equation:
D= b0 +b1X1+b2X2+...+bkXk
where the bi are the discriminant weights. b0 is a
constant.
Group 1
Group 2
The intersection of the normal marginal
distributions of two groups gives the cutting score,
which is used to assign observations to groups.
Observations with scores less than C are assigned
to group 1, and observations with scores greater
than C are assigned to group 2. Since the
distributions may overlap, some observations may
be misclassified.
C
The model may be evaluated in terms of the
percentages of observations assigned correctly
and incorrectly.
Cutting Score
Discriminant Analysis:
Example 6-7 (Minitab)
Discriminant 'Repay' 'Assets' 'Debt' 'Famsize'.
Group
0
1
Count
14
18
Summary of Classification
Put into ....True Group....
Group
0
1
0
10
5
1
4
13
Total N
14
18
N Correct
10
13
Proport. 0.714 0.722
N = 32
N Correct = 23
Prop. Correct = 0.719
Linear Discriminant Function for Group
0
1
Constant -7.0443
-5.4077
Assets
0.0019
0.0548
Debt
0.0758
0.0113
Famsize 3.5833
2.8570
Example 6-7: Misclassified
Observations
Summary of Misclassified Observations
Observation True
Pred
Group
Group
4 **
1
0
7 **
1
0
21 **
0
1
22 **
1
0
24 **
0
1
27 **
0
1
28 **
1
0
29 **
1
0
32 **
0
1
Group Sqrd
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
Distnc
6.966
7.083
0.9790
1.7780
2.940
1.681
0.3812
2.8539
5.371
5.002
2.617
1.551
1.250
2.542
1.703
4.259
1.84529
0.03091
Probability
0.515
0.485
0.599
0.401
0.348
0.652
0.775
0.225
0.454
0.546
0.370
0.630
0.656
0.344
0.782
0.218
0.288
0.712
Example 6-7: SPSS Output (1)
1 0 set width 80
2 data list free / assets income debt famsize job repay
3 begin data
35 end data
36 discriminant groups = repay(0,1)
37 /variables assets income debt famsize job
38 /method = wilks
39 /fin = 1
40 /fout = 1
41 /plot
42 /statistics = all
Number of cases by group
Number of cases
REPAY Unweighted Weighted Label
0
14
14.0
1
18
18.0
Total
32
32.0
Example 6-7: SPSS Output (2)
-------- D IS C RI MI NANT ANALYSI S -------On groups defined by REPAY
Analysis number
1
Stepwise variable selection
Selection rule: minimize Wilks' Lambda
Maximum number of steps..................
10
Minimum tolerance level.................. .00100
Minimum F to enter....................… 1.00000
Maximum F to remove...................... 1.00000
Canonical Discriminant Functions
Maximum number of functions..............
1
Minimum cumulative percent of variance... 100.00
Maximum significance of Wilks' Lambda.... 1.0000
Prior probability for each group is .50000
Example 6-7: SPSS Output (3)
---------------- Variables not in the Analysis after Step 0 ----------------
Variable
Minimum
Tolerance
Tolerance
ASSETS
INCOME
DEBT
FAMSIZE
JOB
1.0000000
1.0000000
1.0000000
1.0000000
1.0000000
1.0000000
1.0000000
1.0000000
1.0000000
1.0000000
F to Enter
6.6151550
3.0672181
5.2263180
2.5291715
.2445652
Wilks' Lambda
.8193329
.9072429
.8516360
.9222491
. 9919137
* * * * * * * * * * * ** * * * * * * * * * * * * * * * * * * * * *
At step 1, ASSETS was included in the analysis.
Wilks' Lambda
Equivalent F
.81933
6.61516
Degrees of Freedom
1
1
30.0
1
30.0
Signif.
.0153
Between Groups
Example 6-7: SPSS Output (4)
---------------- Variables in the Analysis after Step 1 ---------------Variable Tolerance F to Remove Wilks' Lambda
ASSETS 1.0000000
6.6152
---------------- Variables not in the Analysis after Step 1 ------------
Variable
INCOME
DEBT
FAMSIZE
JOB
Tolerance
Minimum
Tolerance
F to Enter
.5784563
.9706667
.9492947
.9631433
.5784563
.9706667
.9492947
.9631433
. 0090821
6.0661878
3.9269288
.0000005
At step 2, DEBT
Wilks' Lambda
Equivalent F
Wilks' Lambda
.8190764
.6775944
.7216177
.8193329
was included in the analysis.
.67759
6.89923
Degrees of Freedom Signif. Between Groups
2
1
30.0
2
29.0 .0035
Example 6-7: SPSS Output (5)
----------------- Variables in the Analysis after Step 2 ---------------Variable
ASSETS
DEBT
Tolerance
.9706667
.9706667
F to Remove
7.4487
6.0662
Wilks' Lambda
.8516360
.8193329
-------------- Variables not in the Analysis after Step 2 -------------
Variable
INCOME
FAMSIZE
JOB
Tolerance
.5728383
.9323959
.9105435
Minimum
Tolerance
.5568120
.9308959
.9105435
F to Enter
.0175244
2.2214373
.2791429
Wilks' Lambda
.6771706
.6277876
.6709059
At step 3, FAMSIZE was included in the analysis.
Wilks' Lambda
Equivalent F
.62779
5.53369
Degrees of Freedom Signif. Between Groups
3 1
30.0
3
28.0
.0041
Example 6-7: SPSS Output (6)
------------- Variables in the Analysis after Step 3 ---------------Variable
Tolerance
F to Remove
Wilks' Lambda
ASSETS
.9308959
8.4282
.8167558
DEBT
.9533874
4.1849
.7216177
FAMSIZE
.9323959
2.2214
.6775944
------------- Variables not in the Analysis after Step 3 -----------Minimum
Variable
Tolerance Tolerance F to Enter
Wilks' Lambda
INCOME .5725772 .5410775
.0240984 .6272278
JOB
.8333526 .8333526
.0086952 .6275855
Summary Table
Action
Vars
Step Entered Removed
1 ASSETS
2 DEBT
3 FAMSIZE
1
2
3
Wilks'
in
.81933
.67759
.62779
Lambda
.0153
.0035
.0041
Sig. Label
Example 6-7: SPSS Output (7)
Classification function coefficients
(Fisher's linear discriminant functions)
REPAY =
ASSETS
DEBT
FAMSIZE
(Constant)
0
.0018509
.0758239
3.5833063
-7.7374079
1
.0547891
.0113348
2.8570101
-6.1008660
Unstandardized canonical discriminant function coefficients
Func 1
ASSETS
DEBT
FAMSIZE
(Constant)
-.0352245
.0429103
.4832695
-.9950070
Example 6-7: SPSS Output (8)
Case Mis
Actual
Highest
Probability
Number Val Sel Group
Group
P(D/G)
P(G/D)
1
1
1
.1798
.9587
2
1
1
.3357
.9293
3
1
1
.8840
.7939
4
1 **
0
.4761
.5146
5
1
1
.3368
.9291
6
1
1
.5571
.5614
7
1 **
0
.6272
.5986
8
1
1
.7236
.6452
...........................................................................
20
0
0
.1122
.9712
21
0 **
1
.7395
.6524
22
1 **
0
.9432
.7749
23
1
1
.7819
.6711
24
0 **
1
.5294
.5459
25
1
1
.5673
.8796
26
1
1
.1964
.9557
27
0 **
1
.6916
.6302
28
1 **
0
.7479
.6562
29
1 **
0
.9211
.7822
30
1
1
.4276
.9107
31
1
1
.8188
.8136
32
0 **
1
.8825
.7124
2nd
Group
0
0
0
1
0
0
1
0
Highest
P(G/D)
.0413
.0707
.2061
.4854
.0709
.4386
.4014
.3548
Discrim
Scores
-1.9990
-1.6202
-.8034
.1328
-1.6181
-.0704
.3598
-.3039
1
0
1
0
0
0
0
0
1
1
0
0
0
.0288
.3476
.2251
.3289
.4541
.1204
.0443
.3698
.3438
.2178
.0893
.1864
.2876
2.4338
-.3250
.9166
-.3807
-.0286
-1.2296
-1.9494
-.2608
.5240
.9445
-1.4509
-.8866
-.5097
Example 6-7: SPSS Output (9)
Classification results Actual Group
--------------------
No. of
Cases
------
Predicted Group Membership
0
1
---------------
Group
0
14
10
71.4%
4
28.6%
Group
1
18
5
27.8%
13
72.2%
Percent of "grouped" cases correctly classified: 71.88%
Example 6-7: SPSS Output (10)
All-groups Stacked Histogram
Canonical Discriminant Function 1
4+
|
+
|
|
|
F
r
e
q
u
e
+
n
|
c
|
y
|
1
1
1
|
3+
|
|
|
2+
|
2
2
2
2
2
1
2
2
1
2
|
2
|
1+
1
|
1
|
1
|
+
|
|
|
2
1
1
2
2
22
222
2
222
121
212112211
2
1
11
1
22
222
2
222
121
212112211
2
1
11
1
22
222
2
222
121
212112211
2
1
11
1
+
|
|
6-12 Principal Components and
Factor Analysis
y
First Component
Total
Variance
Variance
Remaining After
Extraction of
First Second Third
Second Component
Component
x
Factor Analysis
The k original Xi variables written as linear combinations of a smaller
set of m common factors and a unique component for each variable:
X1 = b11F1+ b12F2 +...+ b1mFm + U1
X1 = b21F1+ b22F2 +...+ b2mFm + U2
...
Xk = bk1F1+ bk2F2 +...+ bkmFm + Uk
The Fj are the common factors. Each Ui is the unique component
of variable Xi. The coefficients bij are called the factor loadings.
Total variance in the data is decomposed into the communality, the
common factor component, and the specific part.
Rotation of Factors
Factor 2
Orthogonal Rotation
Factor 2
Rotated Factor 2
Oblique Rotation
Rotated Factor 2
Factor 1
Factor 1
Rotated Factor 1
Rotated Factor 1
Factor Analysis of Satisfaction
Items
Satisfaction with:
Information
1
2
3
4
Variety
5
6
7
8
9
10
Closure
11
12
Pay
13
14
Factor Loadings
1
2
3
4 Communality
0.87
0.88
0.92
0.65
0.19
0.14
0.09
0.29
0.13
0.15
0.11
0.31
0.22
0.13
0.12
0.15
0.8583
0.8334
0.8810
0.6252
0.13
0.17
0.18
0.11
0.17
0.20
0.82
0.59
0.48
0.75
0.62
0.62
0.07
0.45
0.32
0.02
0.46
0.47
0.17
0.14
0.22
0.12
0.12
0.06
0.7231
0.5991
0.4136
0.5894
0.6393
0.6489
0.17
0.12
0.21
0.10
0.76
0.71
0.11
0.12
0.6627
0.5429
0.17
0.10
0.14
0.11
0.05
0.15
0.51
0.66
0.3111
0.4802