4. Orthogonalization and Least
Squares
4.1. Orthogonality and the
Singular Value Decomposition
Reading Trefethen and Bau (1997), Lecture 2
Orthogonal vectors:
{ Orthogonal bases are better conditioned than others
Denition 1: fx1 x2 xng 2 <m is orthogonal if
xTi xj = 0
i 6= j
(1a)
{ The vectors are orthonormal if
xTi xj = ij i 6= j
(1b)
Denition 2: Two subspaces S1 S2 2 <m are orthogonal if
xT y = 0 for all x 2 S1 and all y 2 S2
{ S2 is called the orthogonal complement of S1
Theorem 1: If A 2 <m n then range(A) is the orthogonal
complement of null(AT )
{ Proof: Let y 2 range(A) and x 2 null(AT )
{ With z 2 <n, let y = Az
yT x = (Az)Tx = zTATx = zT(ATx) = 0
{ Because AT x = 0 2
1
Orthogonality
Denition 3: Vectors fx1 x2 xk g form an orthonormal basis for a subspace S 2 <m if they are orthonormal and
span S
Denition 4: A matrix Q 2 <m m is orthogonal if
QT Q = I
(2)
{ Thus, Q;1 = QT
{ The columns of Q form an orthonormal basis for <m
jj jj2 is invariant under orthogonal transformations
{ If Q is orthogonal
jjQxjj22 = xT QT Qx = xT x = jjxjj22
The matrix 2 and F norms are also invariant under orthogonal
transformations
{ If Q and z are orthogonal
jjQAZjj2 = jjAjj2
jjQAZjjF = jjAjjF
The proof follows that for a vector norm
2
(3)
The Singular Value Decomposition
Reading: Trefethen and Bau (1997), Lectures 4 and 5
Theorem 2 (Singular Value Decomposition (SVD):)
Let A 2 <m n, then there exists orthogonal matrices U 2
<m m and V 2 <n n such that
UT AV = (4a)
{ 2 <m n
2
3
1
66 2
77
66
... 7
77
= 66
p 77
64
5
(4b)
{ 1 2 p, p = min(m n)
Proof: Let
Ax = y jjxjj2 = jjyjj2 = 1 jjAjj2 = (5)
{ Dene orthogonal matrices U and V such that
U = y u2 um] = y U1]
V = x v2 vn] = x V1]
Any orthonormal set of vectors can be extended to form
an orthonormal basis
3
SVD
{ Construct
T T
T
y AV1
UT AV = Uy T A x V1 = Uy TAx
T
1
1 Ax U1 AV 1
{ Use (5)
yT Ax = UT1 Ax = UT1 y = 0
{ Thus
T
T
w := A
1
UT AV = 0 Uy TAV
:=
1
AV
0 B
1
1
{ Select zT = wT ] and compute
T 2 + wT w w
A1 z = 0 B w =
Bw
{ Then
jjA1zjj22 = ( 2 + wT w)2 + wT BT Bw ( 2 + wT w)2
{ Thus,
jjA1zjj22
2
2
T
jjA1jj2 2 +w w
jjzjj
2
4
SVD
{ Since U and V are orthogonal, use (3) to obtain
2 = jjAjj22 = jjUT AVjj22 = jjA1jj22
{ Thus, w = 0
{ An induction argument completes the proof 2
1 2 p are the singular values of A
{ Let ui and vi be the i th columns of U and V
ui and vi are the left and right singular vectors
The singular vectors satisfy
Avi = iui Aui = ivi i = 1 : p (6)
Some properties of singular values
{ If 1 2 r > r+1 = = p = 0, then
rank(A) = r
(7a)
null(A) = spanfvr+1
range(A) = spanfu1
A=
jjAjj2F
r
X
i=1
vr+2 vng
(7b)
u2 urg
(7c)
iuiviT
= 12 + 22 + + p2
jjAjj2 = 1
5
(7d)
p = min(m n) (7e)
(7f)
Orthogonal Projections
Reading: Trefethen and Bau (1997), Lecture 6
Denition 5: P 2 <n n is the orthogonal projection onto
a subspace S <n n if
range(P) = S
P 2 = P PT = P
(8)
{ If x 2 <n and Px 2 S then (I ; P)x is in the orthogonal
complement of S
{ The orthogonal projection onto a subspace is unique
Let the columns of V = v1 v2 vk ] form an orthonormal
basis for a subspace S
{ P = VVT is the unique orthogonal projection onto S
VVT is symmetric
P2 = VVT VVT = VVT by orthogonality
If x 2 <n then y = Px = V(VT x)
S = spanfv1 v2 vk g, range(P) = S
6
Orthogonal Projections
Some Properties:
{ Let rank(A) = r and partition U and V
U = Ur Um;r ] V = Vr Vn;r ]
Ur 2 <m r , Um;r 2 <m (m;r)
Vr 2 <n r , Vn;r 2 <n (n;r)
{ Ur UTr is a projection onto range(A)
{ Um;r UTm;r is a projection onto null(AT )
the orthogonal complement of range(A)
{ Vr VrT is a projection onto range(AT )
the orthogonal complement of null(A)
{ Vn;r VnT;r is a projection onto null(A)
7
(9)
Orthogonal Projections
Example 1. The singular value decomposition of
2
3
2 1 3
A=41 0 55
3 1 8
is
2
3
0:3393 ;0:7427 ;0:5774
U = 4 0:4735 0:6652 ;0:5774 5
0:8128 ;0:0775 0:5774
2
3
10:5826 0 0
= 4 0 1:4174 0 5
0
0 0
2
3
0:3393 ;0:7427 0:5774
V = 4 0:1089 ;0:5786 ;0:8083 5
0:9344 0:3371 ;0:1155
With 3 = 0 we have r = 2
Verify that UT U = I, VT V = I
8
Orthogonal Projections
Partition U and V
2
3
0:3393 ;0:7427
Ur = 4 0:4735 0:6652 5
0:8128 ;0:0775
2
0:3393
Vr = 4 0:1089
0:9344
3
;0:7427
;0:5786 5
0:3371
The projections
2
3
;0:5774
Um;r = 4 ;0:5774 5
0:5774
2
3
0:5774
Vn;r = 4 ;0:8083 5
;0:1155
2
3
0:6667 ;0:3333 0:3333
Ur UTr = 4 ;0:3333 0:6667 0:3333 5
0:3333 0:3333 0:6667
2
3
0:3333 0:3333 ;0:3333
Um;r UTm;r = 4 0:3333 0:3333 ;0:3333 5
;0:3333 ;0:3333 0:3333
2
3
0:6667 0:4667 0:0667
Vr VrT = 4 0:4667 0:3467 ;0:0933 5
0:0667 ;0:0933 0:9867
2
3
0:3333 ;0:4667 ;0:0667
Vn;r VnT;r = 4 ;0:4667 0:6533 0:0933 5
;0:0667 0:0933 0:0133
9
Orthogonal Projections
Verify that UTr Ur = I, etc
Verify that
{ Ur UTr + Un;r UTn;r = I
{ Vr VrT + Vn;r VnT;r = I
Choose x = 1 0 0]T and calculate
2
3
0:3333
T
y = Vn;r Vn;r x = 4 ;0:4667 5
;0:0667
{ Verify that Ay = 0
{ Verify that AVn;r VnT;r = 0
Similarly, verify that AT Um;r UTm;r = 0
10