B90.3302 Homework Set 10 Solutions

1. Suppose that V is a symmetric k  k matrix with inverse V -1 .
(k + 1)  (k + 1) matrix W, where
V
k k
W = 
 a

 1k
Show that W -1
Consider the
a
.
c 
11 
k 1
 1 1 1  1
V  q V a a V
= 
1

 a  V 1

q

1

 V 1a 
q
 , where q = c  a V -1 a .
1


q

SOLUTION: This can be verified by computing W W -1. (This can also be verified by
computing W -1 W, but we’ll show only the first.)
 k  k k  1
Observe first that the proposed W -1 is blocked as 
 . Thus it is possible to
 1 k 11 
compute W W -1 in the usual pattern of matrix multiplication. The upper k  k piece is
1
1
1


 1

V V 1  V 1a aV 1   a  a V 1  = I  a aV 1  a a V 1
q
q
q


 q

= I
The upper right k  1 piece of W W -1 is
1
 1

V  V 1 a  
a = 0
q
 q

The lower-left 1  k piece is the transpose of this; it’s 0.
The lower 1  1 piece of W W -1 is a scalar, namely
1
c  a V -1 a
 1

1 
= 1
a  V -1 a   c   =  a V -1 a + c  =
q
q
 q

q
Thus W W -1
=
 I
 k k
 0

 1k
0

1  , which is the (k + 1)  (k + 1) identity matrix, so the result is
11 
k 1
proved.

Page 1
gs98
B90.3302 Homework Set 10 Solutions

2. You have just performed the regression for the model Y
 X β  ε . This
n1
n1
n p p1
includes, of course, the hard work of finding  X  X  . Now…. as a surprise, your
supervisor gives you a new predictor w, and you’ve been asked to repeat the regression.
β 
That is, your model has been revised to Y   X w     ε . This means that
n1
n1
 w 
1
n p 1
you’ll now have to find the (p + 1)  (p + 1) inverse of
 p 11
 X w   X w 
 p 1  n
available  X  X 
1
Use the
n   p 1
to find the (p + 1)  (p + 1) inverse with minimal effort.
SOLUTION: Start by observing that  X w 
 X w
 p 1  n
n   p 1
 X X
 p p
= 
 w X
 1 p
.
 X 
=    X w
 w  n   p 1
 p 1  n
X w
p 1 
 . This can then be appealed to the previous problem.
w w 
11 
Symbol in problem 1
(general description of
partitioned inverse)
V
a
c
Symbol in problem 2
X X
X w
w w
w w - w X  X  X  X  w
1
q



= w I - X  X  X  X  w
Page 2
1
gs98
B90.3302 Homework Set 10 Solutions

At the start, we’ll carry the symbol q rather than its full form.
 X X
 w X

X w
w w 
Then
1
=
1
1
1
1
 

  X X   q  X  X   X  w  X  w  X  X 

1
1

  X  w   X  X 

q


1
1
 X  X   X  w  
q

1


q

This can be simplified a bit in a number of ways. Here’s one version, showing also the
substitution for q.
1



X  w w X  X  X 
1

 X X  
I +

1




w
I
X
X
X
X
w







1

w X  X  X 


1

w I - X  X  X  X  w




 X  X   X  w


1



w I - X  X X  X  w 
1



1
w I - X  X  X 
1



X  w 

This may look ugly, but it is down to the relatively simple math of matrix multiplication.
3. Suppose that Z1, Z2, Z3, Z4 are random variables with Var(Zi) = 1 and Cov(Zi , Zj) = 
(for i  j). Using matrix techniques, show that Z1 + Z2 + Z3 + Z4 is independent of
Z1 + Z2 - Z3 - Z4 . Can you show this by another technique?
This is Rice’s problem 26, page 596 (3rd edition).
the 2nd edition.)
1
 Z1 
Z 

Hint: The variance matrix of  2  = Z is

 Z3 
 

 Z4 
Y=

1 1
F
G
1 1
H
IJ
K
F
G
1
G

G
G
H
(It’s also problem 24, page 556, of


1

I
JJ
JJ
K


. Then consider

1
1 1
Z.
1 1
Page 3
gs98
B90.3302 Homework Set 10 Solutions

SOLUTION: Here Y is a linear combination of Z.
Var(Y) =
=
1 1
F
G
1 1
H
=
1  3
F
G
H1  
1 1
F
G
1 1
H
1
1
Thus
IJ
K
F
IJ
G
H
K
 I F
1 1I
 JG
1 1J
J
G
J
1 JG
1 1J
J
JG
 1K
1 1K
H
F1 1I
1  3 I G
4  12
1 1J F
= G
G
J
J
1  K
1 1J H 0
G
G
J
1 1K
H
1 1
1 1 1 1
Var Z
1 1
1 1 1 1
F1 
 1
1I G
G
J
 
1K
G
G
H 
1  3 1  3
1   1  
IJ
K
0
4  4
As the off-diagonal elements of this matrix are zero, the covariance (and hence the
correlation) must be zero.
There are other, perhaps easier, ways to do this.
First other way. Cov(Z1 + Z2 + Z3 + Z4 , Z1 + Z2 - Z3 - Z4 ) involves 16
covariances, easily laid out in a display:
Cov(Z1 , Z1)
Cov(Z1 , Z2)
- Cov(Z1 , Z3)
- Cov(Z1 , Z4)
Cov(Z2 , Z1)
Cov(Z2 , Z2)
- Cov(Z2 , Z3)
- Cov(Z2 , Z4)
Cov(Z3 , Z1)
Cov(Z3 , Z2)
- Cov(Z3 , Z3)
- Cov(Z3 , Z4)
Cov(Z4 , Z1)
Cov(Z4 , Z2)
- Cov(Z4 , Z3)
- Cov(Z4 , Z4)
The diagonal cells are either +1 or -1. There are two of each sign, so the net
contribution is zero. The off-diagonal cells are either + or -. There are six of
each, so the net contribution is also zero. The overall covariance is zero.
Second other way. Consider the problem in which G and H are independent with
equal variances and with Corr(G , H) = . It’s easy to show that
Cov(G + H, G - H) = 0. The above problem is this, with G = Z1 + Z2 and
H = Z3 + Z4 .

Page 4
gs98
B90.3302 Homework Set 10 Solutions

4. Using the model Y = X  + ,
(a)
(b)
(c)
show that the vector of residuals is orthogonal to every column of X.
if the design matrix X has a column 1 (meaning that the model has an
intercept), show that the residuals sum to zero.
show why the assumption about the column 1 is critical to (b).
This is adapted from Rice, 3rd edition, problem 19, page 595. (In the 2nd edition, thisis
problem 17 on page 556.) Observe that Rice uses e for the vector of noise terms and ê
for the residuals. (In this course we have preferred  for the vector of noise terms and e
for the residuals.)
HINTS:
-1
The residuals are found as e = Y  Yˆ = Y  X  X  X  X  Y
=
I  X  X  X 
-1

X Y =
 I  PX  Y .
Part (a) asks you to show that e X  0 .
1n n p
1 p
Part (b) asks you to show that e 1  0 .
11
1n n1
SOLUTION:
(a) Since e =  I  PX  Y , we simply check   I  PX  Y  X = Y   I  PX  X .
Observe that both I and PX are symmetric matrices.
For I, this is obvious. Then check
=  X 
  X  X   X 
1
 PX 

X X X 
1
X

= X  X  X  X  = PX . This uses the fact that
1
X  X is symmetric and thus its inverse is also symmetric.
Then   I  PX  Y  X = Y   I  PX  X = Y   X  PX X  = Y   X  X  = 0 .
It’s easy to check that PX X = X.
(b) Let’s start from e 1  0 .
1n n1
11
This is
  I  P  Y  1
X
= Y   I  PX  1 . If 1 is
one of the columns of X, then PX 1 = 1. The whole expression will reduce to 0.

Page 5
gs98
B90.3302 Homework Set 10 Solutions

(c) If 1 is not one of the columns of X , then  I  PX  1 will be the part of vector 1 that
is outside the space of the columns of X. The residuals will then sum to something that is
an exotic function of both X and Y.
It is quite interesting that none of this problem, other than the last aside in (c), has
anything to do with Y.
5. Suppose that Yi =  xi + i , where the ’s are independent normal random variables
with mean 0 and standard deviation . This is the regression-through-the-origin model.
(a)
Find the least squares estimate of .
(b)
Find the maximum likelihood estimate of .
SOLUTION:
n
 Y
(a) asks for the value of b that minimizes Q =
i 1
i
 b xi  . We find routinely that
2
n
let
dQ
=  2   xi Yi  b xi   0
db
i 1
n
x Y
The solution comes out quickly to b =
i 1
n
i
i
.
x
i 1
2
i
1
2
 2  yi   xi  
 1
2
e
(b) The likelihood for this problem is L =  

2
i 1  

n
1

1
2 2
n
  yi   xi 
2
i 1
e
. This is maximized when the sum in the exponent is
n /2
n  2  
minimized. This is exactly what we minimized in (a), so the least squares and maximum
likelihood estimate are the same.
=
6. This is the same setup as the previous problem, but now the i’s are independent
normal with mean 0 and Var(i) = (xi )2.
(a)
Find the least squares estimate of .
(b)
Find the maximum likelihood estimate of .

Page 6
gs98
B90.3302 Homework Set 10 Solutions

SOLUTION: Part (a) is exactly the same as in problem 5.
For (b), observe that the likelihood is
n
1
1
2
1



 yi   xi 2 
  yi   xi 

1
1
2 2 i 1 xi2
2 xi2 2
e
L = 
e
 =
 n  n  n /2
 xi  2
i 1 

 xi   2
 i 1 
n
The likelihood will be maximized when the sum in the exponent is minimized. This sum
is
n
1
2
y   xi  

2  i
i 1 xi
n

i 1
 yi   xi 


xi


2
1
A routine differentiation will get to ˆ MLE 
n
1
ˆ MLE 
n

n

i 1
n
=

i 1
n

i 1
 yi

 x  
 i

2
yi
. In random variable form, this is
xi
Yi
.
xi
Page 7
gs98