Estimable Functions and Their Least
Squares Estimators
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
1 / 51
Consider the GLM
y =n×p
X β + ε,
n×1
p×1
n×1
where
E(ε) = 0.
Suppose we wish to estimate c0 β for some fixed and known c ∈ Rp .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
2 / 51
An estimator t(y) is an unbiased estimator of the function c0 β iff
E[t(y)] = c0 β
c
Copyright 2012
Dan Nettleton (Iowa State University)
∀ β ∈ Rp .
Statistics 611
3 / 51
An estimator t(y) is a linear estimator in y iff
t(y) = d + a0 y
for some known constants d, a1 , . . . , an .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
4 / 51
A function c0 β is linearly estimable iff ∃ a linear estimator that is an
unbiased estimator of c0 β.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
5 / 51
Henceforth, we will use estimable as a synonym for linearly estimable.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
6 / 51
A function c0 β is said to be nonestimable if there does not exist a linear
estimator that is an unbiased estimator of c0 β.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
7 / 51
Result 3.1:
Under the GLM, c0 β is estimable iff the following equivalent conditions
hold:
(i) ∃ a 3 E(a0 y) = c0 β
(ii) ∃ a 3 c0 = a0 X
∀ β ∈ Rp
(X0 a = c)
(iii) c ∈ C(X0 ).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
8 / 51
Show conditions (i), (ii), and (iii) are equivalent.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
9 / 51
(i)⇐⇒(ii)⇐⇒(iii):
∃ a 3 E(a0 y) = c0 β
∀ β ∈ Rp
⇐⇒ ∃ a 3 a0 E(y) = c0 β
⇐⇒ ∃ a 3 a0 Xβ = c0 β
∀ β ∈ Rp
∀ β ∈ Rp
⇐⇒ ∃ a 3 a0 X = c0
⇐⇒ ∃ a 3 X0 a = c
⇐⇒ c ∈ C(X0 ).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
10 / 51
Show that any of the equivalent conditions is equivalent to c0 β
estimable.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
11 / 51
c0 β estimable
⇐⇒ ∃ d, a 3 E(d + a0 y) = c0 β
⇐⇒ ∃ d, a 3 d + a0 Xβ = c0 β
⇐⇒ ∃ a 3 a0 Xβ = c0 β
∀ β ∈ Rp
∀ β ∈ Rp
∀ β ∈ Rp
⇐⇒ ∃ a 3 a0 X = c0 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
12 / 51
Example:
Suppose that when team i competes against team j, the expected
margin of victory for team i over team j is µi − µj , where µ1 , . . . , µ5 are
unknown parameters.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
13 / 51
Suppose we observe the following outcomes.
Team
1
beats Team
2
by
7
3
1
3
3
2
14
3
5
17
4
5
10
4
1
1
c
Copyright 2012
Dan Nettleton (Iowa State University)
points
Statistics 611
14 / 51
Determine y, X, β.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
15 / 51

  
1 −1 0 0
0  
7
 µ1
  
  
 3 −1
0
1
0
0
  
  
 µ2 
  
  
14  0 −1 1 0
0
, µ3 
 , 

  

0 1 0 −1 
17  0

 
  
µ
4



10  0
0
0
1
−1

  
µ5
−1
0 0 1
0
1
y
,
c
Copyright 2012
Dan Nettleton (Iowa State University)
X
, β
Statistics 611
16 / 51
Is µ1 − µ2 is estimable?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
17 / 51
Yes, µ1 − µ2 is estimable.
µ1 − µ2 = c0 β, where
c0 = [1, −1, 0, 0, 0]

1 −1 0 0

−1
0 1 0


 0 −1 1 0
= [1, 0, 0, 0, 0, 0] 

0 1 0
 0

 0
0 0 1

−1
0 0 1

0

0


0


−1

−1

0
= a0 X.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
18 / 51
Thus,
a0 y = [1, 0, 0, 0, 0, 0]y = y1
is an unbiased estimator of c0 β = µ1 − µ2 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
19 / 51
Is µ1 − µ3 is estimable?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
20 / 51
Yes, µ1 − µ3 is estimable.
µ1 − µ3 = c0 β, where
c0 = [1, 0, −1, 0, 0]

1 −1 0 0

−1
0 1 0


 0 −1 1 0
= [0, −1, 0, 0, 0, 0] 

0 1 0
 0

 0
0 0 1

−1
0 0 1

0

0


0


−1

−1

0
= a0 X.
∴ −y2 is unbiased estimator of µ1 − µ3 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
21 / 51
Is µ1 − µ5 is estimable?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
22 / 51
Yes, µ1 − µ5 is estimable.
µ1 − µ5 = c0 β, where
c0 = [1, 0, 0, 0, −1]

1 −1 0 0

−1
0 1 0


 0 −1 1 0
= [0, −1, 0, 1, 0, 0] 

0 1 0
 0

 0
0 0 1

−1
0 0 1

0

0


0


−1

−1

0
= a0 X.
∴ y4 − y2 is unbiased estimator of µ1 − µ5 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
23 / 51
Is µ1 estimable?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
24 / 51
µ1 = c0 β, where
c0 = [1, 0, 0, 0, 0].
So, does ∃ a 3 a0 X = [1, 0, 0, 0, 0]?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
25 / 51

1 −1 0 0

−1
0 1 0


 0 −1 1 0
a0 X = [a1 , a2 , a3 , a4 , a5 , a6 ] 

0 1 0
 0

 0
0 0 1

−1
0 0 1

0

0


0


−1

−1

0
= [a1 − a2 − a6 , −a1 − a3 , a2 + a3 + a4 , a5 + a6 , −a4 − a5 ]
?
= [1, 0, 0, 0, 0].
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
26 / 51
a1 − a2 − a6 = 1
(1)
−a1 − a3 = 0
(2)
a2 + a3 + a4 = 0
(3)
a5 + a6 = 0
(4)
−a4 − a5 = 0
(5)
(1) and (2) imply −a2 − a3 − a6 = 1.
(4) and (5) imply a4 = a6 , which together with (3) implies
a2 + a3 + a6 = 0.
∴ There does not exist a 3 a0 X = [1, 0, 0, 0, 0] ⇒ µ1 is nonestimable.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
27 / 51
Result 3.1 tells us that c0 β is estimable iff ∃ a 3 c0 β = a0 Xβ
∀ β ∈ Rp .
Recall that E(y) = Xβ.
Thus, c0 β is estimable iff it is a LC of the elements of E(y).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
28 / 51
This leads to Method 3.1:
LCs of expected values of observations are estimable.
c0 β is estimable iff c0 β is a LC of the elements of E(y); i.e.,
c0 β =
n
X
ai E(yi )
for some
a1 , . . . , an .
i=1
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
29 / 51
Use Method 3.1 to show that µ2 − µ4 is estimable in our previous
example.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
30 / 51


µ1 − µ2


µ3 − µ1 




µ3 − µ2 
.

E(y) = Xβ = 

µ3 − µ5 


µ4 − µ5 


µ4 − µ1
µ2 − µ4 = −(µ1 − µ2 ) − (µ4 − µ1 ) = −E(y1 ) − E(y6 ).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
31 / 51
Method 3.2:
c0 β is estimable iff c ∈ C(X0 ).
Thus, find a basis for C(X0 ), say {v1 , . . . , vr }, and determine if
c=
r
X
di vi
for some
d1 , . . . , dr .
i=1
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
32 / 51
Method 3.3:
By Result A.5, we know that C(X0 ) and N (X) are orthogonal
complements in Rp .
Thus,
c ∈ C(X0 )
iff
c0 d = 0
∀ d ∈ N (X),
which is equivalent to
Xd = 0 ⇒ c0 d = 0.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
33 / 51
Reconsider our previous example.
Use Method 3.3 to show that µ1 is nonestimable.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
34 / 51
Xd = 0 ⇒ d1 − d2 = 0
d3 − d1 = 0
d3 − d2 = 0
d3 − d5 = 0
d4 − d5 = 0
d4 − d1 = 0.
⇒ d1 = d2 = d3 = d4 = d5 .
For example, X1 = 0. Note [1, 0, 0, 0, 0]1 = 1 6= 0. Thus, µ1 is not
estimable.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
35 / 51
Now use method 3.3 to establish that
c0 β = c1 µ1 + c2 µ2 + c3 µ3 + c4 µ4 + c5 µ5
is estimable iff
5
X
ci = 0.
i=1
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
36 / 51
{d ∈ R5 : Xd = 0} = {d1 : d ∈ R}. Thus
5×1
c0 d = 0
∀ d ∈ N (X)
⇐⇒ c0 (d1) = 0 ∀ d ∈ R
⇐⇒ dc0 1 = 0 ∀ d ∈ R
⇐⇒ d
5
X
ci = 0 ∀ d ∈ R
i=1
⇐⇒
5
X
ci = 0.
i=1
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
37 / 51
The least squares estimator of an estimable function c0 β is c0 β̂, where
β̂ is any solution to the NE (X0 Xb = X0 y).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
38 / 51
Result 3.2:
If c0 β is estimable, then c0 β̂ is the same for all solutions β̂ to the NE.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
39 / 51
Proof of Result 3.2:
Suppose β̂ 1 , and β̂ 2 are any two solutions to the NE.
From Corollary 2.3, we know Xβ̂ 1 = Xβ̂ 2 .
Now c0 β is estimable ⇒ ∃ a 3 c0 = a0 X.
Thus,
c0 β̂ 1 = a0 Xβ̂ 1 = a0 Xβ̂ 2 = c0 β̂ 2 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
40 / 51
Result 3.3:
The least squares estimator of an estimable function c0 β is a linear
unbiased estimator of c0 β.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
41 / 51
Proof of Result 3.3:
From Result 3.2, we know c0 β̂ is the same ∀ solution to NE.
We know (X0 X)− X0 y is a solution to NE.
Thus, c0 β̂ = c0 (X0 X)− X0 y. This is a linear estimator.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
42 / 51
Furthermore, c0 β estimable ⇒ ∃ a 3 c0 = a0 X. Thus,
E(c0 β̂) = E(c0 (X0 X)− X0 y)
c
Copyright 2012
Dan Nettleton (Iowa State University)
= c0 (X0 X)− X0 E(y)
= c0 (X0 X)− X0 Xβ
= a0 X(X0 X)− X0 Xβ
= a0 Xβ
= c0 β.
Statistics 611
43 / 51
Consider again our previous example.
Recall that y1 is a linear unbiased estimator of µ1 − µ2 .
Is this the least squares estimator?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
44 / 51
It can be shown that the least squares estimator of µ1 − µ2 is
7
3
4
1
1
1
y1 − y2 + y3 − y4 + y5 − y6 .
11
11
11
11
11
11
Based on the observed y, the least squares estimate is 8.0.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
45 / 51
One of infinitely many solutions to the NE is

2.8


−5.2




β̂ = (X0 X)− X0 y =  7.8 .


 2.8


−8.2

Which team is best?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
46 / 51
Suppose y = Xβ + ε, where E(ε) = 0 and rank(n×p
X) = p.
Show that c0 β is estimable ∀ c ∈ Rp .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
47 / 51
Proof:
rank(n×p
X) = p ⇒ X0 has p LI columns and each column is an element of
Rp .
Thus, by Fact V4, p LI columns of X0 form a basis for Rp .
∴ C(X0 ) = Rp . This also follows from Result A.7:
C(X0 ) ⊆ Rp , dim(C(X0 )) = dim(Rp ) = p ⇒ C(X0 ) = Rp .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
48 / 51
Now from Result 3.1, we know c0 β is estimable iff c ∈ C(X0 ).
∵ C(X0 ) = Rp , c0 β is estimable ∀ c ∈ Rp .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
49 / 51
Alternatively, we can prove the result by noting the following
rank(n×p
X) = p ⇒ rank(X0 X) = p by Corollary 2.2.
Now X0 X is p × p of rank p and ∴ (X0 X)−1 exists.
Thus, β̂ = (X0 X)−1 X0 y is the unique solution to NE
c
Copyright 2012
Dan Nettleton (Iowa State University)
X0 Xb = X0 y.
Statistics 611
50 / 51
Note that ∀ c ∈ Rp
E(c0 β̂) = c0 (X0 X)−1 XE(y)
= c0 (X0 X)−1 X0 Xβ
= c0 β.
Thus, c0 β̂ = c0 (X0 X)−1 X0 y is a linear unbiased estimator of
c0 β
∀ c ∈ Rp . ∴ c0 β is estimable ∀ c ∈ Rp .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
51 / 51