Constraints on Solutions to the Normal
Equations
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
1 / 41
If rank(n×p
X) = r < p, there are infinitely many solutions to the NE
X0 Xb = X0 y.
For (X0 X)− any GI of X0 X, the set of all solutions is
(X0 X)− X0 y + (I − (X0 X)− X0 X)z : z ∈ Rp .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
2 / 41
It is possible to place additional constraints on a solution to the NE so
that ∃ a unique solution that satisfies the constraints.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
3 / 41
Example:
Suppose
yij = µ + τi + εij
(i = 1, . . . , t; j = 1, . . . , ni )
where
E(εij ) = 0 ∀ i, j.
 
µ̂
 
τ̂1 
 
Consider the following constraints on a solution to the NE β̂ =  .  .
 .. 
 
τ̂2
(Note that constraints are on β̂ not β.)
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
4 / 41
Four common choices are
1. τ̂1 = 0
(set first to zero)
2. τ̂t = 0
(set last to zero)
3.
Pt
4.
Pt
i=1 τ̂i
=0
i=1 ni τ̂i
=0
(sum to zero)
(weighted sum to zero)
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
5 / 41
Any one of these constraints may be imposed by insisting that a
solution to the NE β̂ satisfies Aβ̂= 0 for some matrix A whose rows
µ̂
 
τ̂1 
 
define linear constraints on β̂ =  .  .
 .. 
 
τ̂2
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
6 / 41
For example,
1. A = [0, 1, 0, . . . , 0].
2. A = [0, 0, 0, . . . , 1].
3. A = [0, 1, 1, . . . , 1].
4. A = [0, n1 , n2 , . . . , nt ].
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
7 / 41
Note that solution β̂ satisfies both the NE and the constraint equations
Aβ̂ = 0 iff
"
X0 X
c
Copyright 2012
Dan Nettleton (Iowa State University)
A
#
β̂ =
" #
X0 y
0
.
Statistics 611
8 / 41
Furthermore, we know that
X0 Xb = X0 y ⇐⇒ Xb = PX y.
Thus, we have β̂ a solution to NE satisfying the constraints iff
" #
X
A
c
Copyright 2012
Dan Nettleton (Iowa State University)
"
β̂ =
PX y
0
#
.
Statistics 611
9 / 41
We now know that if "we#want a unique solution to NE and constraint
X
equations, we need
to be of full-column rank; that is, we need
A
" #!
X
= p.
rank
A
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
10 / 41
rank(n×p
X) = r < p. Thus, we know X has r LI rows.
If we can find s ≡ p − r
p × 1 vectors 3, when these vectors are
combined with r LI rows of X, the set of p vectors is LI, then we can
use the s vectors as rows of A to get
rank
c
Copyright 2012
Dan Nettleton (Iowa State University)
" #!
X
A
= p.
Statistics 611
11 / 41
Suppose x1 , . . . , xr denote r LI rows of X (written as column vectors)
or, equivalently, r LI columns of X0 .
We seek a1 , . . . , as (s = p − r) so that
{x1 , . . . , xr , a1 , . . . , as }
is a set of p = r + s LI vectors in Rp .
Then, with A0 = [a1 , . . . , as ],
" #
X
c
Copyright 2012
Dan Nettleton (Iowa State University)
A
will have full-column rank.
Statistics 611
12 / 41
Obviously, a1 , . . . , as must be LI and ak ∈
/ C(X0 ) ∀ k = 1, . . . , s.
Show by example that these conditions are not sufficient to guarantee
rank
c
Copyright 2012
Dan Nettleton (Iowa State University)
" #!
X
A
= p.
Statistics 611
13 / 41
Prove the following results:
Suppose x1 , · · · , xr LI in Rp .
Suppose a1 , . . . , as LI in Rp .
Suppose r + s = p. Then
x1 , . . . , xr , a1 , . . . , as LI
⇐⇒ span{x1 , . . . , xr } ∩ span{a1 , . . . , as } = {0}.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
16 / 41
Note that the condition
span{x1 , . . . , xr } ∩ span{a1 , . . . , as } = {0}
⇒ (d1 a1 + · · · + ds as )0 β is not estimable whenever
d1 , . . . , ds are not all 0.
Thus, the constraints that we add, as well as all nontrivial LCs of those
constraints, must correspond to nonestimable functions.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
19 / 41
Note that a1 , . . . , as with the desired properties always exist. Why is
that?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
20 / 41
The result that we proved can be alternatively stated as follows:
Suppose rank(n×p
X) = r < p. Suppose rank(s×p
A) = s where s = p − r. Then
C(X0 ) ∩ C(A0 ) = {0} ⇐⇒ rank
c
Copyright 2012
Dan Nettleton (Iowa State University)
" #!
X
A
= p.
Statistics 611
24 / 41
Lemma 3.1:
For s×p
A 3 rank(A) = s = p − rank(n×p
X) and C(X0 ) ∩ C(A0 ) = {0},
"
X0 X
#
b=
A
"
X0 X
A0 A
"
#
#
X0 y
is equivalent to
0
"
b=
#
X0 y
0
,
which is equivalent to
(X0 X + A0 A)b = X0 y.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
25 / 41
Result 3.6:
Suppose rank(n×p
X) = r, rank(s×p
A) = s = p − r, and C(X0 ) ∩ C(A0 ) = {0}.
Then
(i) X0 X + A0 A is nonsingular.
(ii) (X0 X + A0 A)−1 X0 y is unique solution to X0 Xb = X0 y and Ab = 0.
(iii) (X0 X + A0 A)−1 is GI of X0 X.
(iv) A(X0 X + A0 A)−1 X0 = 0.
(v) A(X0 X + A0 A)−1 A0 = I.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
30 / 41
Returning to our example,
yij = µ + τi + εij
(i = 1, . . . , t; j = 1, . . . , ni )
where
E(εij ) = 0 ∀ i, j.
 
µ
 
τ1 
 
β =  . . Let’s consider the constraint
 .. 
 
τt
t
X
ni τ̂i = 0.
i=1
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
35 / 41
Find β̂ that satisfies NE and the constraint.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
36 / 41

1n 1n1
 1
1n 0n
2
 2
X= .
..
 ..
.

1nt 0nt

n n1

n n
 1 1

0
XX=
n2 0
 ..
..
.
.

nt
c
Copyright 2012
Dan Nettleton (Iowa State University)
0
0n1
· · · 0n1


· · · 0n2 

.. 
..
. . 

0nt · · · 1nt

n2 · · · nt

0 ··· 0


n2 · · · 0 
.
.. . .
.. 
. .
.

1n2
..
.
0
· · · nt
Statistics 611
37 / 41


nµ̂ + n1 τ̂1 + · · · + nt τ̂t




n1 µ̂ + n1 τ̂1


0
X Xβ̂ = 

..


.


nt µ̂ + nt τ̂t
 
y··
 
y1· 
 
X0 y =  .  .
 .. 
 
yt·
Equating X0 Xβ̂ and X0 y leads to
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
38 / 41
nµ̂ + n1 τ̂1 + · · · + nt τ̂t = y··
n1 µ̂ + n1 τ̂1
= y1·
..
.
nt µ̂ + nt τ̂t
= yt· .
This system of equations has an infinite number of solutions.
However, if we insist that
n1 τ̂1 + · · · + nt τ̂t = 0,
the first equation becomes
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
39 / 41
nµ̂ = y·· ⇐⇒ µ̂ = ȳ·· .
Substituting µ̂ = ȳ·· in the other equations yields
ni ȳ·· + ni τ̂i = yi· ⇐⇒
ȳ·· + τ̂i = ȳi· ⇐⇒
τ̂i = ȳi· − ȳ··
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
40 / 41
For the general case, we can compute
β̂ = (X0 X + A0 A)−1 X0 y.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
41 / 41