Least Squares Estimation of the Expected
Value of the Response Vector
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
1 / 19
Consider the General Linear Model (GLM) previously introduced,
where
y = Xβ + ε,
with E(ε) = 0.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
2 / 19
Suppose we want to estimate
E(y) = Xβ
based on our observed response y.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
3 / 19
Because we know
E(y) = Xβ ∈ C(X),
a reasonable strategy may be to find the vector in C(X) that is closest
to y in terms of Euclidean distance.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
4 / 19
For example, suppose
" #
y1
=
y2
" #
1
[µ] +
1
" #
ε1
ε2
.
Then
E
" #!
y1
y2
c
Copyright 2012
Dan Nettleton (Iowa State University)
=
" #
1
1
[µ] =
" #
µ
µ
.
Statistics 611
5 / 19
If we observe y =
" #
4
2
and have to guess
E(y) = Xβ =
" #
1
1
[µ] =
" #
µ
µ
,
it may be reasonable to find the value of µ that makes
" #
4
2
" #
µ
µ
as close to
as possible.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
6 / 19
In general, we seek a vector β̂ 3 Xβ̂ is closer to y than any other
vector in C(X).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
7 / 19
If
Q(b) = ky − Xbk2
= (y − Xb)0 (y − Xb),
we seek β̂ 3 Q(β̂) ≤ Q(b) ∀ b ∈ Rp .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
8 / 19
If
Q(β̂) ≤ Q(b) ∀ b ∈ Rp ,
Xβ̂ is called the Least Squares Estimate (LSE) of E(y) = Xβ.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
9 / 19
Suppose PX is the orthogonal projection matrix onto C(X).
Show that PX y is the unique vector in C(X) that is closest to y.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
10 / 19
Because PX y ∈ C(X), we know ∃ at least one vector b 3 PX y = Xb.
We can take β̂ to be any such vector b. Then we have shown that β̂
will minimize Q(b) over b ∈ Rp .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
17 / 19
Now we know that the least square estimator of E(y) = Xβ is
Xβ̂ = PX y.
How do we find PX ?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
18 / 19
We need to find the symmetric and idempotent matrix that projects
onto C(X).
In the next set of notes we will show
PX = X(X0 X)− X0 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
19 / 19