Statistical Analysis
Professor Lynne Stokes
Department of Statistical Science
Lecture 6QF
Matrix Solutions to Normal
Equations
1
Direct (Kronecker) Products
(Left) Direct Product
A n x k  Bp x q
 a11B a12 B ... a1k B 
a B a B ... a B
22
2k 
  21
...
...
... 
 ...
a B a B ... a B
 n1
n2
nk  np x kq
Some Properties
A  B  A  B
A  BC  D   AC  BD
A  B1  A 1  B1
assuming all operations are valid
2
Solving the Normal Equations
Single-Factor, Balanced Experiment
yij = m + ai + eij
i = 1, ..., a; j = 1, ..., r
Matrix Formulation
y = Xb + e
y = (y11 y12 ... y1r ... ya1 ya2 ... yar)’
1r 1r
1 0
r
X= r
... ...
1 0
r
r
0r
1r
...
0r
0r 
0r 
  [(1r  1a ) : I a  1r ]
... 
1r 
Show
m
 
 a1 
b 
...
 
 aa 
3
Properties of X’X



Symmetric
Same rank as X
Has an inverse (X’X)-1 iff X has full
column rank
4
Solving the Normal Equations
Residuals
~
r  y  Xb
Least Squares
~
Choose b to Minimize (b)  (y - Xb)(y - Xb)
Solution: Solve the Normal Equations
~
XXb  Xy
5
Solving the Normal Equations
Normal Equations
~
X  Xb = X  y
 n r r ...
 r r 0 ...

 r 0 r ...
... ... ... ...

 r 0 0 ...
~
r  m
  y 
~  


0  a1   y1 
 ~
0  a 2    y 2  
... ...   ... 

 ~  
r  a a   y a  
Problem: X’X is Singular, has no inverse
Show
6
Generalized Inverse: G or A
Definition
AGA = A
Moore-Penrose Generalized Inverse
(i) AGA = A
(ii) GAG = G
(iii) AG is symmetric
(iv) GA is symmetric
not unique
unique
Some Properties
if A has full row rank, G = A’(AA’)-1
if A has full column rank, G = (A’A)–1A’
Common Notation
A
7
Solving the Normal Equations
Normal Equations
~
XXb = Xy


Solutions
~
b = XX- Xy
Every solution to the normal equations corresponds to a
generalized inverse of X’X
Every generalized inverse of X’X solves the normal
equations
Theorem: For any (X’X) ,
X’X (X’X) X’ = X’
~

X Xb  XXXX  Xy
 Xy
8
A Solution to the Normal
Equations
 n r r ...
 r r 0 ...

XX   r 0 r ...
... ... ... ...

 r 0 0 ...
One Generalized Inverse
0
G
0 a
0a 
r 1Ia 
r
0
 n
0  
r1a


...

r
r1a 
rI a 
Verification
0a   n r1a 
 n r1a   0
XXG XX  
 0 r 1I  r1 rI 
r
1
r
I
 a
a  a
a  a
a
 0 1a   n r1a   n r1a 






9
0
I
r
1
r
I
r
1
r
I
 a
a  a
a  a
a
A Solution to the Normal
Equations
~ 
m
~ 
0
~  a1 

b     XX  Xy  
...
0a

 ~ 
 aa 
 y   0 

 

0a  y1   y1 

1  

r I a  ...
... 

 

 ya   ya 
Corresponds to the solution to the normal equations
with the constraint m = 0 imposed
10
Assignment




Find another generalized inverse for X’X in a
one-factor balanced experiment
Verify that it is a generalized inverse
Solve the normal equations using the
generalized inverse
Determine what constraint on the model
parameters correspond to the generalized
inverse
11