Distributions of Quadratic Forms
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
1 / 31
Under the Normal Theory GMM (NTGMM),
y = Xβ + ε,
where ε ∼ N(0, σ 2 I).
By Result 5.3, the NTGMM =⇒ y ∼ N(Xβ, σ 2 I).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
2 / 31
Mean of y determined by β through Xβ.
Variance of y determined by σ 2 .
y = PX y + (I − PX )y
c
Copyright 2012
Dan Nettleton (Iowa State University)
PX y ∈ C(X), (I − PX )y ∈ C(X)⊥ .
Statistics 611
3 / 31
We use ŷ = PX y to estimate Xβ
(PX y = Xβ̂)
We use ε̂ = (I − PX )y to estimate σ 2 . σ̂ 2 =
c
Copyright 2012
Dan Nettleton (Iowa State University)
ε̂0 ε̂
n−rank(X)
.
Statistics 611
4 / 31
Also, recall that
y0 y = ŷ0 ŷ + ε̂0 ε̂
SSTO = SSR + SSE.
Under the NTGMM, what can we say about the distribution of these
sums of squares?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
5 / 31
Lemma 5.1:
A p × p symmetric matrix A is idempotent with rank s iff ∃ a p × s matrix
G with orthogonal columns such that
c
Copyright 2012
Dan Nettleton (Iowa State University)
A = GG0 .
Statistics 611
6 / 31
Result 5.14:
Let X ∼ N(µ, p×p
I ) and letp×p
A be a symmetric matrix. Then
A is idempotent with rank s =⇒ X0 AX ∼ χ2s (µ0 Aµ/2).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
11 / 31
Result 5.15:
Suppose X ∼ N(µ, Σ), with p×p
Σ of rank p. Suppose A is p × p and
symmetric. Then
AΣ is idempotent of rank s =⇒ X0 AX ∼ χ2s (µ0 Aµ/2).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
13 / 31
Proof of Result 5.15:
Let W = Σ−1/2 X. Then
W ∼ N(Σ−1/2 µ, Σ−1/2 ΣΣ−1/2 = I).
Let B = Σ1/2 AΣ1/2 . Then B is symmetric by symmetry of Σ1/2 and A.
Furthermore,
rank(B) = rank(Σ1/2 AΣ1/2 ) = rank(A)
= rank(AΣ) = s.
∵ Σ1/2 and Σ are full-rank.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
14 / 31
Finally, note that B is idempotent:
AΣAΣ = AΣ ⇐⇒ Σ1/2 AΣAΣ = Σ1/2 AΣ
⇐⇒ Σ1/2 AΣAΣΣ−1/2 = Σ1/2 AΣΣ−1/2
⇐⇒ Σ1/2 AΣAΣ1/2 = Σ1/2 AΣ1/2
⇐⇒ Σ1/2 AΣ1/2 Σ1/2 AΣ1/2 = Σ1/2 AΣ1/2
⇐⇒ BB = B.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
15 / 31
Thus, by Result 5.14,
W 0 BW ∼ χ2s ((Σ−1/2 µ)0 B(Σ−1/2 µ)/2).
Now note
W 0 BW = X0 Σ−1/2 Σ1/2 AΣ1/2 Σ−1/2 X
= X0 AX
and likewise
(Σ−1/2 µ)0 B(Σ−1/2 µ) = µ0 Aµ.
∴ X0 AX ∼ χ2s (µ0 Aµ/2).
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
16 / 31
Find the distribution of SSE.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
17 / 31
Similarly,
SSR
∼ χ2rank(X)
σ2
c
Copyright 2012
Dan Nettleton (Iowa State University)
1 0 0
β X Xβ/σ 2 .
2
Statistics 611
20 / 31
Result 5.16:
Suppose X ∼ N(µ, Σ) and A is symmetric with rank s. Then
BΣA = 0 =⇒ BX and X0 AX are independent.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
21 / 31
By the SDT, we have
A = QΛQ0 ,
p×p
where Q is square with orthonormal columns q1 , . . . , qp and
Λ = diag(λ1 , . . . , λp ) with exactly s of λ1 , . . . , λp not equal to zero.
Because
QΛQ0 =
p
X
λi qi q0i ,
i=1
we can without loss of generality (WLOG) assume
λ1 , . . . , λs 6= 0.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
22 / 31
Thus,
A=
s
X
λi qi q0i = Q1 Λ1 Q01 ,
i=1
where
Q1 = [q1 , . . . , qs ], Q01 Q1 =s×s
I
Λ1 = diag(λ1 , . . . , λs )
c
Copyright 2012
Dan Nettleton (Iowa State University)
and Λ−1
1 = diag
1
1
,...,
λ1
λs
.
Statistics 611
23 / 31
"
Now consider
BX
Q01 X
#
"
=
"
#
B
Q01
#
B
Q01
X. Then
"
X∼N
B
Q01
#
!
µ, V
,
where
"
V=
#
B
Q01
"
=
c
Copyright 2012
Dan Nettleton (Iowa State University)
h
Σ B0
BΣB0
Q1
i
BΣQ1
Q01 ΣB0 Q01 ΣQ1
#
.
Statistics 611
24 / 31
BΣA = 0 =⇒ BΣQ1 Λ1 Q01 = 0
=⇒ BΣQ1 Λ1 Q01 Q1 = 0Q1
=⇒ BΣQ1 Λ1 = 0
−1
=⇒ BΣQ1 Λ1 Λ−1
1 = 0Λ1
=⇒ BΣQ1 = 0
=⇒ BX and Q01 X independent by Result 5.4.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
25 / 31
Now BX and Q01 X are independent,
=⇒ BX and (Q01 X)0 Λ1 Q01 X independent
=⇒ BX and X0 Q1 Λ1 Q01 X independent
=⇒ BX and X0 AX independent.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
26 / 31
Corollary 5.4:
Suppose X ∼ N(µ, Σ), A symmetric with rank r, B symmetric with rank
s. Then
BΣA = 0 =⇒ X0 AX and X0 BX are independent.
Proof:
HW problem.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
27 / 31
Find the distribution of
SSR/r
,
SSE/(n − r)
c
Copyright 2012
Dan Nettleton (Iowa State University)
where r = rank(X).
Statistics 611
28 / 31