Statistical Analysis
Professor Lynne Stokes
Department of Statistical Science
Lecture 6
Solving Normal Equations and
Estimating Estimable Model
Parameters
1
Regression Models
Model
y  X  e
Residuals
r  y  Xˆ
Least Squares
Choose ˆ to Minimize ()  (y - X)(y - X)
Sum
of Squared
Residuals
Solution: Solve the Normal Equations
XXˆ  Xy
2
Regression Solution
ˆ  XX 1 Xy

Under usual assumptions, the least
squares estimator is






Unique
Unbiased
Minimum Variance
Consistent
Known sampling distribution
Universally used
3
Analysis of Completely Randomized
Designs
Fixed Factor Effects
Factor levels specifically chosen
Inferences desired only on the factor levels
included in the experiment
Systematic, repeatable changes in the mean response
4
Flow Rate Experiment
Fixed
or
Random
?
Filter
A
B
C
D
0.233
0.259
0.183
0.233
Flow Rates
0.197
0.259
0.258
0.343
0.284
0.264
0.328
0.267
0.244
0.305
0.258
0.269
MGH Fig 6.1
5
Flow Rate Experiment
0.35
Filter
A
B
C
D
0.30
Average
Flow
Rate
Effects
-0.028
0.030
-0.014
0.013
Conclusion ?
0.25
0.20
A
B
C
D
6
Filter Type
Statistical Model for Single-Factor,
Fixed Effects Experiments
Model
yij = m + ai + eij
Response
i = 1, ..., a; j = 1, ..., ri
Overall
Main
Mean
Effect
(Constant)
for
Level
i
Error
ai: Effect of Level i = change in the mean response
7
Statistical Model for Single-Factor,
Fixed Effects Experiments
Cell Means Model
yij = mi + eij
i = 1, ..., a; j = 1, ..., ri
mˆ i  yi
Effects Model
yij = m + ai + eij
i = 1, ..., a; j = 1, ..., ri
Fixed Effects Models
Connection: mi = m + ai
~  y y
mˆ  y , a
i
i

8
Solving the Normal Equations
Single-Factor, Balanced Experiment
yij = m + ai + eij
i = 1, ..., a
j = 1, ..., r
n = ar
Matrix Formulation
y = X + e
y = (y11 y12 ... y1r ... ya1 ya2 ... yar)’
1r 1r
1 0
r
X= r
... ...
1 0
r
r
0r
1r
...
0r
0r 
0r 
  [(1r  1a ) : I a  1r ]
... 
1r 
 m
 
a1 


 ... 
 
aa 
9
Solving the Normal Equations
Residuals
~
r  y  X
Least Squares
~
Choose  to Minimize ()  (y - X)(y - X)
Solution: Solve the Normal Equations
~
XX  Xy
10
Solving the Normal Equations
Normal Equations
~
X X  = X y
 n r r ...
 r r 0 ...

 r 0 r ...
... ... ... ...

 r 0 0 ...
~
r  m
  y 
~  


0  a1   y1 
 ~
0  a 2    y 2  
... ...   ... 

 ~  
r  a a   y a  
Check
11
Solving the Normal Equations
Normal Equations
~  ra
~ + ... + ra
~ y
~ + ra
nm
1
2
a

~
~ + ra
rm
 y1
1
~
~
rm
+ ra
y
2
...
~
rm
Linearly Dependent
2
~ y
+ ra
a
a
a + 1 Parameters, a Linearly Independent Equations
Infinite Number of Solutions
Check
12
Solving the Normal Equations
Normal Equations
~
nm
~
~ + ra
rm
 y
 y1
1
~
rm
~
+ ra
2
...
~
rm
One
Solution
a
ai  0
i 1
 y 2
~ y
+ ra
a
a

a
 a~ i  0
i 1
~=y
m

~ y y
a
i
i

~ y
~a
mˆ i  m
i
i13
Solving the Normal Equations
Normal Equations
~  ra
~  ...  ra
~ y
ra
1
2
a

~
ra
 y1
1
~
ra
y
2
2
...
~ y
ra
a
a
Another
Solution
~0
m0m
~ y
a
i
i
~ y
~a
mˆ i  m
i
i14
Solving the Normal Equations
Normal Equations
~  ra
~ + ... + ra
~ y
~ + ra
nm
1
2
a -1

~
~ + ra
rm
y
1
1
~
rm
...
~
rm
~
+ ra
2
 y 2
~ y
+ ra
a -1
( a 1)
Another Solution
~  0 m
~y
a
a
a
~ y y
a
i
i
a i = 1, ... , a - 1
~ y
~a
mˆ i  m
i
i15
Solving the Normal Equations
All solutions to the normal equations
produce the same estimates of “estimable functions”
of the model means


Solutions are not estimates
Estimable Functions


All solutions provide one unique
estimator
Estimators are unbiased
16
Solving the Normal Equations
Two-Factor, Balanced Experiment
yijk = mij + eijk = m + ai + j + (a)ij + eijk
Matrix Formulation
y = X + e
i = 1, ..., a
j = 1, ..., b
k = 1, ..., r
n = abr
X = [ 1 : XA : XB : XAB ]
   m , a1 , ... , aa , 1 , ... , b , a11 , ... , aab
17
Solving the Normal Equations
Two-Factor, Balanced Experiment
yijk = mij + eijk = m + ai + j + (a)ij + eijk
i = 1, ..., a
j = 1, ..., b
k = 1, ..., r
Matrix Formulation
y = X + e
n = abr
X = [ 1 : XA : XB : XAB ]
   m , a1 , ... , aa , 1 , ... , b , a11 , ... , aab
Number
of
Parameters
1 +
a
+
b
+
ab
rank( X ) < 1+a+b+ab
18
Solving the Normal Equations
Normal Equations
~
X X  = X y
 n br ... ar ...
br br ... 0 ...

 ... ... ... ... ...
 0 0 ... 0 ...

~  y 
 m





~
r a

1   y1 
0  ...   ... 
 ~   
... 1   y1 
r  ...   ... 
 a    y 

ab   ab 
Check
19
Solving the Normal Equations
Matrix
1n
XA
Linear Dependencies
None
1 : Columns of XA Sum to 1n
One Solution
aa = 0
Eliminates a column
From XA
a – 1 “degrees of freedom”
20
Solving the Normal Equations
Matrix
1n
XA
XB
Linear Dependencies
None
1 : Columns Sum of XA to 1n
1 : Columns Sum of XB to 1n
One Solution
aa = 0
b = 0
Eliminates a column
From XB
b – 1 “degrees of freedom”
21
Solving the Normal Equations
Matrix
1n
XA
XB
XAB
Linear Dependencies
None
1 : Columns sum to 1n
1 : Columns sum to 1n
1 + (a - 1) + (b - 1) :
Sum over all columns = 1n
One Solution
aa = 0
b = 0
(a)ab = 0
Eliminates a column
from XAB
22
Solving the Normal Equations
Matrix
1n
XA
XB
XAB
Linear Dependencies
One Solution
None
1 : Columns Sum to 1n
aa = 0
1 : Columns Sum to 1n
b = 0
1 + (a - 1) + (b - 1) :
Sum over all columns = 1n
(a)ab = 0
Sums of columns over
each i = 1,...,a-1 & each j = 1,...,b-1
(a)ib = 0
equal one of the remaining
i=1,...,a-1
columns of XA and XB
(a)aj = 0
j=1,...,b-1
(a – 1)(b – 1) “degrees of freedom”
23
Solving the Normal Equations
Matrix
XA
XB
XAB
Linear Dependencies
One Solution
1 : Columns sum to 1n
aa = 0
1 : Columns sum to 1n
b = 0
1 + (a - 1) + (b - 1) :
Sum over all columns = 1n
(a)ab = 0
Sums of columns over
each i = 1,...,a-1 & each j = 1,...,b-1
(a)ib = 0
equal one of the remaining
i=1,...,a-1
columns of XA and XB
(a)aj = 0
j=1,...,b-1
Constraints : 1 + 1 + {1 + (a - 1) + (b - 1)} = a + b + 1
Degrees of Freedom : (1 + a + b + ab) - (a + b + 1)
= ab
= 1 + (a - 1) + (b - 1) + (a - 1)(b - 1)
24
Solving the Normal Equations
~y
m
ab
~ y m
~
a
i
ib
~ 0
a
a
~
~
 j  yaj  m
~
b  0
i = 1, . .. , a - 1
j = 1, . . . , b - 1
~
~a
~ 
(a ) ij  yij  m
i
j
i  a ; j b
(a ) ij  0
i = a or j = b
Check
25
Solving the Normal Equations
Another Solution
~y
m

~  y y
a
i
i

~
 j  y j  y
i = 1, ... , a
j = 1, ... , b
(a )ij  yij  yi  y j  y  i, j
Check
26
Flow Rate Experiment
Fixed
or
Random
?
Filter
A
B
C
D
0.233
0.259
0.183
0.233
Flow Rates
0.197
0.259
0.258
0.343
0.284
0.264
0.328
0.267
0.244
0.305
0.258
0.269
MGH Fig 6.1
27
Quantifying Factor Effects
Effect
Change in average response
due to changes in factor levels
Factor Level
Average
1
2
3
...
k
Overall Average
y1
y 2
y3
...
y k
y
Effect of Level t : y t - y
28
Quantifying Factor Effects
Effect
Change in average response
due to changes in factor levels
Factor Level
Average
1
2
y1
y 2
3
...
k
y3
...
y k
Effect of changing
from Level s to Level t :
Overall Average
y
(y t  y ) - (ys - y )
= y t - ys
29
Quantifying Factor Effects
Main Effects for Factor A
y i  y 
Change in average response due to
changes in the levels of Factor A
Main Effects for Factor B
y  j  y 
Change in average response due to
changes in the levels of Factor B
Interaction Effects for Factors A & B
(y ij - y  j ) - (y i  y  )
Effect of Level i of
Factor A at Level j
of Factor B
Effect of Level i
of Factor A
30
Quantifying Factor Effects
Main Effects for Factor A
y i  y 
Change in average response due to
changes in the levels of Factor A
Main Effects for Factor B
y  j  y 
Change in average response due to
changes in the levels of Factor B
Interaction Effects for Factors A & B
(y ij - y  j ) - (y i  y  )
 y ij - y i - y  j  y 
Change in average response due
joint changes in Factors A & B
in excess of changes in the main
effects
31
Two-Level Factors
Effect of Level 1:
y 1  y 
Effect of Level 2:
y 2  y 
Common to Use
y 2 - y 1
Note: If r1 = r2 ,
y 1  y 
= - (y 2  y  )
32
Factors at Two Levels



Most common choice for designs involving
many factors
Many efficient fractional factorial and
screening designs available
Can use p two-level factors in place of factors
p
whose number of levels is 2
33
Calculating Two-Level Factor Effects:
Pilot Plant Study
Main Effect
Difference between the average responses at the
two levels
M(Temp) = Average @ 180o - Average @ 160o
= 75.8 - 52.8 = 23.0
M(Conc) = Average @ 40% - Average @ 20%
= 61.8 - 66.8 = -5.0
M(Catalyst) = Average @ C2 - Average @ C1
= 65.0 - 63.5 = 1.5
BHH Section 10.3
MGH Section 5.3
34
Calculating Two-Level Factor Effects
Two-Factor Interaction Effect
Half the difference between the main effects of one
factor at each level of the second factor
M(Conc @ C2)
= Average @ 40%&C2 - Average @ 20%&C2
= 62.5 - 67.5 = -5.0
M(Conc @ C1)
= Average @ 40%&C1 - Average @ 20%&C1
= 61.0 - 66.0 = -5.0
I(Conc,Cat)
= {M(Conc @ C2) - M(Conc @ C1)} / 2
=0
BHH Section 10.4
MGH Section 5.3
35
Calculating Two-Level Factor Effects
Two-Factor Interaction Effect
Half the difference between the main effects of one
factor at each level of the second factor
M(Temp @ C2)
= Average @ 180o&C2 - Average @ 160o&C2
= 81.5 - 48.5 = 33.0
M(Temp @ C1)
= Average @ 180o&C1 - Average @ 160o&C1
= 70.0 - 57.0 = 13.0
I(Temp,Cat)
= {M(Temp @ C2) - M(Temp @ C1)} / 2
= (33.0 - 13.0) / 2 = 10.0
36
Cell Means and Effects Model
Estimability
Three-Factor Balanced Experiment
yijkl = mijk + eijkl
i = 1 , ... , a ; j = 1 , ... , b ;
k = 1, ... , c ; l = 1 , ... , r
mijk = m + ai + j + gk + (a)ij + (ag)ik + (g)jk + (ag)ijk
37
Cell Means Models: Estimable Functions
All cell means are estimable
m ijk  y ijk 
38
Cell Means Models: Estimable Functions
All cell means are estimable
m ijk  y ijk 
All linear combinations of cell means are estimable
   c ijk m ijk
   c ijk yijk
Does not depend
on parameter constraints
(includes m i , m ij , etc.)
39
Cell Means Models: Estimable Functions
All cell means are estimable
m ijk  y ijk 
Some linear combinations of cell means are uninterpretable
  m1  m  23
Some linear combinations of cell means are essential
  m1  m 2
40
Cell Means and Effects Models
mi  m  ai    g   (a )i  (ag )i  (g )
 (ag )i
Imposing parameter constraints
simplifies the relationships;
makes the parameters more interpretable
41
Parameter Equivalence:
Effects Representation & Cell Means Model
Parameter constraints
 a i  . . . =  (a) ij = . . . =
i
ij
 (a) ijk 
0
ijk
Means and mean effects
m i  m  a i
m ij  m  a i   j  (a ) ij
a i  m i  m 
(a) ij  m ij  m i  m  j  m 
 ( m ij  m  j )  ( m i  m  )
42
Contrasts
Contrast
A Linear Combinatio n of Parameters whose Coefficien ts Sum to Zero
k
 a j j
j1
k
with
a j  
j1
Contrasts often eliminate
nuisance parameters; e.g., m
43
Contrasts
Main Effects
m i  m i  a i  a i
Interactions
m ij  m i  m  j  m   (a) ij
m ij  m il  m kj  m kl  (a) ij  (a) il  (a) kj  (a) kl
4cr (a) ij2   {(a) ij  (a) il  (a) kj  (a) kl }
ij
2
ijkl
Show 44