Lecture 30
30.1
Joint distribution of the estimates.
In our last lecture we found the maximum likelihood estimates of the unknown parameters in simple linear regression model and we found the joint distribution of β̂0
and β̂1 . Our next goal is to describe the distribution of σ̂ 2 . We will show the following:
1. σ̂ 2 is independent of βˆ0 and βˆ1 .
2. nσ̂ 2 /σ 2 has χ2n−2 distribution with n − 2 degrees of freedom.
Let us consider two vectors
1
1 √
√
a1 = (a11 , . . . , a1n ) =
,...,
n
n
and
Xi − X̄
a2 = (a21 , . . . , a2n ) where a2i = q
.
2
2
n(X − X̄ )
It is easy to check that both vectors have length 1 and they are orthogonal to each
other since their scalar product is
a1 · a2 =
n
X
i=1
n
X − X̄
1 X
q i
a1i a2i = √
= 0.
n i=1
2
2
n(X − X̄ )
Let us choose vectors a3 , . . . , an so that
result, the matrix

a11
 a12

A =  ..
 .
a1n
a1 , . . . , an is orthonormal basis and, as a

· · · an1
· · · an2 

..
.. 
.
. 
· · · ann
120
121
LECTURE 30.
is orthogonal. Let us consider vectors
Y = (Y1 , . . . , Yn ), µ =
Y = ( Y 1 , . . . , Yn )
and
Yn − Y n Y − µ Y1 − Y 1
=
,...,
σ
σ
σ
0
0
so that the random variables Y1 , . . . , Yn are i.i.d. standard normal. We proved before
that if we consider an orthogonal transformation of i.i.d. standard normal sequence:
Y 0 = (Y10 , . . . , Yn0 ) =
Z 0 = (Z10 , . . . , Zn0 ) = Y 0 A
then Z10 , . . . , Zn0 will also be i.i.d. standard normal. Since
Z 0 = Y 0A =
this implies that
Y − µ
Y A − µA
A=
σ
σ
Y A = σZ 0 + µA.
Let us define a vector
Z = (Z1 , . . . , Zn ) = Y A = σZ 0 + µA.
Each Zi is a linear combination of Yi s and, therefore, it has a normal distribution.
Since we made a specific choice of the first two columns of the matrix A we can write
down explicitely the first two coordinates Z1 and Z2 of vector Z. We have,
Z1 =
X
√
√
1 X
ai1 Yi = √
Yi = nȲ = n(β̂0 + β̂1 X̄)
n
and the second coordinate
X (Xi − X̄)Yi
q
n(X 2 − X̄ 2 )
q
q
X (Xi − X̄)Yi
2
2
= n(X 2 − X̄ 2 )β̂1 .
=
n(X − X̄ )
n(X 2 − X̄ 2 )
Z2 =
X
ai2 Yi =
Solving these two equations for β̂0 and β̂1 we can express them in terms of Z1 and Z2
as
X̄
1
1
β̂1 = q
Z2 and β̂0 = √ Z1 − q
Z2 .
n
2
2
2
2
n(X − X̄ )
n(X − X̄ )
122
LECTURE 30.
Next we will show how σ̂ 2 can also be expressed in terms of Zi s.
nσ̂
2
=
n
X
i=1
=
=
=
X
X
n
X
i=1
2
(Yi − β̂0 − β̂1 Xi ) =
2
(Yi − Ȳ ) − 2β̂1
n X
i=1
(Yi − Ȳ ) − β̂1 (Xi − X̄)
2
n(X 2
− X̄ )
P
2
{since β̂0 = Ȳ − β̂1 X̄}
X
(Yi − Ȳ )(Xi − X̄)
(Xi − X̄)2
+β̂12
n(X 2 − X̄ 2 )
|
{z
}
βˆ1
(Yi − Ȳ )2 − β̂12 n(X 2 − X̄ 2 ) =
Yi2 − Z12 − Z22 =
n
X
i=1
X
2
Yi2 − n(Ȳ )2 − βˆ1 n(X 2 − X̄ 2 )
| {z } |
{z
}
Z12
Z22
Zi2 − Z12 − Z22 = Z32 + · · · + Zn2 .
In the last line we used the fact that Z = Y A is an orthogonal transformation of Y
and since orthogonal transformation preserves the length of a vector we have,
n
X
Zi2
=
i=1
n
X
Yi2 .
i=1
If we can show that Z1 , . . . , Zn are i.i.d. with distribution N (0, σ 2 ) then we will have
shown that
Z 2
nσ̂ 2 Z3 2
n
=
+
.
.
.
+
∼ χ2n−2
2
σ
σ
σ
has χ2 distribution with n − 2 degrees of freedom, because Zi /σ ∼ N (0, 1). Since we
showed above that
Z = µA + σZ 0 ⇒ Zi = (µA)i + σZi0 ,
the fact that Z10 , . . . , Zn0 are i.i.d. standard normal implies that Zi s are independent
of each other and Zi ∼ N ((µA)i , σ). Let us compute the mean Zi = (µA)i :
(µA)i =
Zi =
n
X
aji Yj =
j=1
=
n
X
j=1
n
X
aji Yj =
j=1
n
X
aji (β0 + β1 Xj )
j=1
aji (β0 + β1 X̄ + β1 (Xj − X̄))
= (β0 + β1 X̄)
n
X
j=1
aji + β1
n
X
j=1
aji (Xj − X̄).
Since the matrix A is orthogonal its columns are orthogonal to each other. Let
ai = (a1i , . . . , ani ) be the vector in the ith column and let us consider i ≥ 3. Then the
123
LECTURE 30.
fact that ai is orthogonal to the first column gives
n
X
n
X
1
√ aji = 0
ai · a1 =
aj1 aji =
n
j=1
j=1
and the fact that ai is orthogonal to the second column gives
n
X
1
ai · a2 = q
(Xj − X̄)aji = 0.
2
2
n(X − X̄ ) j=1
This show that for i ≥ 3
n
X
j=1
aji = 0 and
n
X
j=1
aji (Xj − X̄) = 0
and this proves that Zi = 0 for i ≥ 3 and Zi ∼ N (0, σ 2 ) for i ≥ 3. As we mentioned
above this also proves that nσ̂ 2 /σ 2 ∼ χ2n−2 .
Finally, σ̂ 2 is independent of β̂0 and β̂1 because as we showed above σ̂ 2 can be
written as a function of Z3 , . . . , Zn and β̂0 and β̂1 can be written as functions of Z1
and Z2 .