Gram-Schmidt Orthonormalization
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
1 / 20
Gram-Schmidt Orthonormalization:
Suppose x1 , . . . , xp are LI vectors in Rn .
We seek mutually orthogonal vectors u1 , . . . , up in Rn 3
span{x1 , . . . , xk } = span{u1 , . . . , uk }
c
Copyright 2012
Dan Nettleton (Iowa State University)
∀ k = 1, . . . , p.
Statistics 611
2 / 20
Define
U0 = 0
n×1
and Uk = [u1 , . . . , uk ] k = 1, . . . , p,
where
uk = (I − PUk−1 )xk
∀ k = 1, . . . , p.
We will show u1 , . . . , up have the desired properties.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
3 / 20
First note that PU0 is the orthogonal projection matrix onto
C(U0 ) = C(0), i.e.,
PU0 = 0(00 0)− 00 =n×n
0.
∴ u1 = (I − 0)x1 = x1 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
4 / 20
u2 = (I − PU1 )x2 = (I − Px1 )x2
= residual vector from the regression of x2 on x1 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
5 / 20
Likewise, uk is the residual vector from the regression of xk on
x1 , . . . , xk−1 ∀ k = 3, . . . , p.
(This will follow if we can show
C(Uk ) = span{x1 , . . . , xk }
∀ k = 1, . . . , p. )
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
6 / 20
Now can you show
span{u1 , . . . , uk } = span{x1 , . . . , xk }
c
Copyright 2012
Dan Nettleton (Iowa State University)
∀ k = 1, . . . , p?
Statistics 611
7 / 20
Because x1 = u1 , the result holds for k = 1.
Now suppose
span{u1 , . . . , ul } = span{x1 , . . . , xl }
for some l ∈ {1, . . . , p − 1}.
If we can show
span{u1 , . . . , ul+1 } = span{x1 , . . . , xl+1 },
the result will follow by induction.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
8 / 20
Recall that
ul+1 = (I − PUl )xl+1 = xl+1 − PUl xl+1
(1)
which is equivalent to
(2)
xl+1 = ul+1 + PUl xl+1 .
We know
PUl xl+1 ∈ C(Ul ) = span{u1 , . . . , ul }
= span{x1 , . . . , xl }.
Therefore,
(1) ⇒ ul+1 ∈ span{x1 , . . . , xl+1 } and
(2) ⇒ xl+1 ∈ span{u1 , . . . , ul+1 }.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
9 / 20
We have
ul+1 ∈ span{x1 , . . . , xl+1 }
⇒ span{u1 , . . . , ul+1 } ⊆ span{x1 , . . . , xl+1 }.
Likewise,
xl+1 ∈ span{u1 , . . . , ul+1 }
⇒ span{x1 , . . . , xl+1 } ⊆ span{u1 , . . . , ul+1 },
and the result follows by induction.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
10 / 20
Now can you prove mutually orthogonality of u1 , . . . , up ?
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
11 / 20
Suppose i, j ∈ {1, . . . , p} with i < j.
Then,
u0j ui = [(I − PUj−1 )xj ]0 ui
= x0j (I − PUj−1 )0 ui
= x0j (I − PUj−1 )ui
= x0j (ui − PUj−1 ui )
= x0j (ui − ui )
= 0.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
12 / 20
Now let dk = kuk k
Define qk =
1
dk uk
∀ k = 1, . . . , p.
∀ k = 1, . . . , p.
Note that
1 0
u uk
dk2 k
1
= 2 kuk k2
dk
q0k qk =
= 1 ∀ k = 1, . . . , p.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
13 / 20
Also
q0i qj =
1 0
di dj ui uj
= 0 ∀ i 6= j.
Thus, q1 , . . . , qp are mutually orthonormal.
Furthermore,
span{q1 , . . . , qk } = span{u1 , . . . , uk }
= span{x1 , . . . , xk }
c
Copyright 2012
Dan Nettleton (Iowa State University)
∀ k = 1, . . . , p.
Statistics 611
14 / 20
Show that X = QR, where
X ≡ [x1 , . . . , xp ]
Q ≡ [q1 , . . . , qp ]
and
R is an upper triangular matrix.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
15 / 20
To see the intuition behind this result, let’s look at a special case:


r11 r12 r13


.
[x1 , x2 , x3 ] = [q1 , q2 , q3 ] 
0
r
r
22
23


0
0 r33
A general proof is as follows.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
16 / 20
X = PX X = PQ X = Q(Q0 Q)− Q0 X
= QQ0 X = QR,
where
R = Q0 X = [q0i xj ].
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
17 / 20
Now, ∀ j = 1, . . . , p;
∃ c1 , . . . , cj ∈ R 3 xj = c1 q1 + · · · + cj qj .
Thus,
q0i xj = c1 q0i q1 + · · · + cj q0i qj
= 0 + ··· + 0
=0
c
Copyright 2012
Dan Nettleton (Iowa State University)
∀ i = j + 1, . . . , p.
Statistics 611
18 / 20
∴ rij = q0i xj = 0 whenever i > j. Thus, R = [q0i xj ] is upper triangular.
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
19 / 20
Note that R is unique:
Suppose QR1 = QR2 = X.
Then Q0 QR1 = Q0 QR2 ⇒ R1 = R2 .
c
Copyright 2012
Dan Nettleton (Iowa State University)
Statistics 611
20 / 20