Introduction to Simulation - Lecture 5
QR Factorization
Jacob White
Thanks to Deepak Ramaswamy, Michal Rewienski,
and Karen Veroy
QR Factorization
Singular Example
LU Factorization Fails
Strut
Joint
Load force
The resulting nodal matrix is SINGULAR, but a solution exists!
SMA-HPC ©2003 MIT
QR Factorization
Singular Example
LU Factorization Fails
v1
v2
1 v3
v4
1
The resulting nodal matrix is SINGULAR, but a solution exists!
SMA-HPC ©2003 MIT
QR Factorization
Singular Example
Recall weighted sum of columns view of
systems of equations
⎡↑
⎢
⎢ M1
⎢↓
⎣
↑
M2
↓
⎡ x1 ⎤ ⎡ b1 ⎤
↑ ⎤⎢ ⎥ ⎢ ⎥
⎥ ⎢ x2 ⎥ ⎢ b2 ⎥
MN ⎥
=
⎢ ⎥ ⎢ ⎥
⎥
↓ ⎦⎢ ⎥ ⎢ ⎥
⎣ xN ⎦ ⎣bN ⎦
x1M 1 + x2 M 2 +
+ xN M N = b
M is singular but b is in the span of the columns of M
SMA-HPC ©2003 MIT
QR Factorization
Orthogonalization
If M has orthogonal columns
Orthogonal columns implies:
Mi • M j = 0
i≠ j
Multiplying the weighted columns equation by ith column:
(
M i • x1M 1 + x2 M 2 +
)
+ xN M N = M i • b
Simplifying using orthogonality:
(
)
xi M i • M i = M i • b ⇒ xi =
SMA-HPC ©2003 MIT
Mi • b
(M
i
• Mi
)
QR Factorization
Orthogonalization
Orthonormal M - Picture
M is orthonormal if:
Mi • M j = 0
i ≠ j and
Mi • Mi = 1
Picture for the two-dimensional case
M1
M1
b
b
M2
Non-orthogonal Case
SMA-HPC ©2003 MIT
x1
x2
Orthogonal Case
M2
QR Factorization
Orthogonalization
QR Algorithm Key Idea
⎡ x1 ⎤ ⎡ b1 ⎤
⎡↑
↑
↑ ⎤⎢ ⎥ ⎢ ⎥
⎢
⎥ ⎢ x2 ⎥ ⎢ b2 ⎥
M
M
M
=
2
N⎥
⎢ 1
⎢ ⎥ ⎢ ⎥
⎢↓
⎥
↓
↓ ⎦⎢ ⎥ ⎢ ⎥
⎣
⎢⎣ xN ⎥⎦ ⎢⎣bN ⎥⎦
Original Matrix
y
b
⎡↑ ↑
↑ ⎤ ⎡⎢ 1 ⎤⎥ ⎡⎢ 1 ⎤⎥
⎢
⎥ y2
b2 ⎥
⎢
⎥
⎢
=
QN ⎥
⎢Q1 Q2
⎢ ⎥ ⎢ ⎥
⎢↓ ↓
⎥
↓ ⎥⎢ ⎥ ⎢ ⎥
⎢⎣
⎦ ⎢⎣ yN ⎥⎦ ⎢⎣bN ⎥⎦
Matrix with
Orthonormal
Columns
Qy = b ⇒ y = Q b
T
How to perform the conversion?
SMA-HPC ©2003 MIT
Orthogonalization
QR Factorization
Projection Formula
Given M 1 , M 2 , find Q2 =M 2 − r12 M 1 so that
(
M 1 • Q2 = M 1 • M 2 − r12 M 1
) =0
M1 • M 2
r12 =
M1 • M1
M2
Q2
SMA-HPC ©2003 MIT
r12
M1
Orthogonalization
QR Factorization
Normalization
Formulas simplify if we normalize
1
1
Q1 =
M 1 = M 1 ⇒ Q1 • Q1 = 1
r11
M1 • M1
Now find Q2 =M 2 − r12Q1 so that Q2 • Q1 = 0
r12 = Q1 • M 2
1
1
Finally Q2 =
Q2 = Q2
r
22
Q2 • Q2
SMA-HPC ©2003 MIT
QR Factorization
Orthogonalization
How was a 2x2 matrix
converted?
Since Mx should equal Qy, we can relate x to y
⎡↑ ↑ ⎤
⎡↑
↑ ⎤
⎢
⎥ ⎡ y1 ⎤
⎢
⎥ ⎡ x1 ⎤
⎢ M 1 M 2 ⎥ ⎢ x ⎥ = x1M 1 + x2 M 2 = ⎢Q1 Q2 ⎥ ⎢ y ⎥ = y1Q1 + y2Q2
⎢ ↓ ↓ ⎥⎣ 2⎦
⎢↓
⎥⎣ 2⎦
↓
⎣
⎦
⎢⎣
⎥⎦
M 1 = r11Q1
M 2 = r22 Q2 + r12Q1
⎡ r11
⎢0
⎣
SMA-HPC ©2003 MIT
r12 ⎤ ⎡ x1 ⎤ ⎡ y1 ⎤
=⎢ ⎥
⎥
⎢
⎥
r22 ⎦ ⎣ x2 ⎦ ⎣ y2 ⎦
QR Factorization
⎡↑
⎢
⎢ M1
⎢↓
⎣
↑ ⎤
⎥ ⎡ x1 ⎤
M2 ⎥ ⎢ ⎥ =
x2 ⎦
⎣
↓ ⎥⎦
Orthogonalization
The 2x2 QR Factorization
⎡↑ ↑ ⎤
⎢
⎥ ⎡ r11 r12 ⎤ ⎡ x1 ⎤ ⎡ b1 ⎤
=⎢ ⎥
⎢Q1 Q2 ⎥ ⎢
⎢
⎥
⎥
⎢ ↓ ↓ ⎥ ⎣ 0 r22 ⎦ ⎣ x2 ⎦ ⎣b2 ⎦
⎢⎣
⎥⎦
Upper
Triangular
Orthonormal
Two Step Solve Given QR
Step 1) QRx = b ⇒ Rx = QT b = b
Step 2) Backsolve Rx = b
SMA-HPC ©2003 MIT
Orthogonalization
QR Factorization
The General Case
3x3 Case
⎡↑
⎢
⎢ M1
⎢↓
⎣
↑
M2
↓
↑ ⎤ ⎡ ↑
⎥ ⎢
M 3 ⎥ ⇒ ⎢ M1
↓ ⎥⎦ ⎢⎣ ↓
↑
M 2 − r12 M 1
↓
To Insure the third column is orthogonal
(
•(M
)
M )= 0
M 1 • M 3 − r13 M 1 − r23 M 2 = 0
M2
SMA-HPC ©2003 MIT
3
− r13 M 1 − r23
2
⎤
↑
⎥
M 3 − r13 M 1 − r23 M 2 ⎥
⎥
↓
⎦
QR Factorization
(
•(M
Orthogonalization
Must Solve Equations for
Coefficients in 3x3 Case
)
M )= 0
M 1 • M 3 − r13 M 1 − r23 M 2 = 0
M2
⎡ M1 • M1
⎢
⎣ M 2 • M1
SMA-HPC ©2003 MIT
3
− r13 M 1 − r23
2
M 1 • M 2 ⎤ ⎡ r13 ⎤ ⎡ M 1 • M 3 ⎤
⎥
⎥⎢ ⎥ = ⎢
M 2 • M 2 ⎦ ⎣ r23 ⎦ ⎣ M 2 • M 3 ⎦
QR Factorization
Orthogonalization
Must Solve Equations for
Coefficients
To Orthogonalize the Nth Vector
⎡ M1 • M1
⎢
⎢
⎢ M N −1 • M 1
⎣
2
M 1 • M N −1 ⎤ ⎡ r1, N ⎤ ⎡ M 1 • M N ⎤
⎥⎢
⎥
⎥ ⎢
=
⎥⎢
⎥
⎥ ⎢
M N −1 • M N −1 ⎥⎦ ⎢⎣ rN −1, N ⎥⎦ ⎢⎣ M N −1 • M N ⎥⎦
3
N inner products requires N work
SMA-HPC ©2003 MIT
QR Factorization
⎡↑
⎢
⎢ M1
⎢↓
⎣
Orthogonalization
3x3 Case
↑
M2
↓
↑ ⎤ ⎡ ↑
⎥ ⎢
M 3 ⎥ ⇒ ⎢ M1
↓ ⎥⎦ ⎢⎢ ↓
⎣
Use previously
orthogonalized vectors
↑
M 2 − r12Q1
↓
⎤
⎥
M 3 − r13Q1 − r23Q2 ⎥
⎥
↓
⎥⎦
↑
To Insure the third column is orthogonal
(
)
(
)
Q1 • M 3 − Q1r13 − Q2 r23 = 0 ⇒ r13 = Q1 • M 3
Q2 • M 3 − Q1r13 − Q2 r23 = 0 ⇒ r23 = Q2 • M 3
SMA-HPC ©2003 MIT
Basic Algorithm
QR Factorization
For i = 1 to N
rii = M i • M i
1
Qi = M i
rii
“For each Source Column”
N
∑ 2N ≈ 2N
Normalize
For j = i+1 to N {
rij ← M j • Qi
M j ⇐ M j − rij Qi
SMA-HPC ©2003 MIT
“Modified Gram-Schmidt”
2
operations
i =1
“For each target Column right of source”
N
∑ ( N − i)2 N ≈ N
i =1
3
operations
QR Factorization
Basic Algorithm
“By Picture”
⎡↑
⎢
⎢Q1
⎢↓
⎢⎣
↑
↑
Q2
Q3
↓
↓
SMA-HPC ©2003 MIT
↑⎤
⎥
QN ⎥
↓ ⎥⎥
⎦
⎡ r11
⎢0
⎢
⎢0
⎢
⎢
⎢⎣ 0
r12
r13
r22
0
r23
r33
0
0
r1N ⎤
⎥
r2 N ⎥
r3 N ⎥
⎥
⎥
rNN ⎥⎦
QR Factorization
Basic Algorithm
“By Picture”
↑↑
↑
M1 1
Q
↓↓
M
Q22
↓
SMA-HPC ©2003 MIT
↑
↑
Q33
M
↑↑
Q
M44
↓
↓↓
r11 r12
r13
r14
r22
r23
r24
r33
r34
r44
QR Factorization
Basic Algorithm
Zero Column
What if a Column becomes Zero?
⎡↑
⎢
⎢Q1
⎢
⎢⎣ ↓
↑ ⎤
⎡ r11 r12 r13
⎥
⎢0 0 0
MN ⎥
0 M3
⎢
⎥
⎢⎣ 0 0 0
↓ ⎥
0 ↓
⎦
Matrix MUST BE Singular!
0
↑
r1N ⎤
0 ⎥⎥
0 ⎥⎦
1) Do not try to normalize the column.
2) Do not use the column as a source for orthogonalization.
3) Perform backward substitution as well as possible
SMA-HPC ©2003 MIT
Basic Algorithm
QR Factorization
Zero Column Continued
Resulting QR Factorization
⎡↑
⎢
⎢Q1
⎢↓
⎢⎣
0
↑
0 Q3
0 ↓
SMA-HPC ©2003 MIT
↑⎤
⎥
QN ⎥
↓ ⎥⎥
⎦
⎡ r11
⎢0
⎢
⎢0
⎢
⎢0
⎢⎣ 0
r12
0
r13
0
0
r33
0
0
0
0
r1N ⎤
0 ⎥⎥
r3 N ⎥
⎥
⎥
rNN ⎥⎦
QR Factorization
Singular Example
Recall weighted sum of columns view of
systems of equations
⎡↑
⎢
⎢ M1
⎢↓
⎣
↑
M2
↓
⎡ x ⎤ ⎡b ⎤
↑ ⎤⎢ 1 ⎥ ⎢ 1 ⎥
⎥ ⎢ x2 ⎥ ⎢ b2 ⎥
=
MN ⎥
⎢ ⎥ ⎢ ⎥
↓ ⎥⎦ ⎢ ⎥ ⎢ ⎥
⎣ xN ⎦ ⎣bN ⎦
x1M 1 + x2 M 2 +
+ xN M N = b
Two Cases when M is singular
Case 1) b ∈ span{M 1 ,.., M N } ⇒ b ∈ span{Q1 ,.., QN }
Case 2) b ∉ span{M 1 ,.., M N }, How accurate is x ?
SMA-HPC ©2003 MIT
QR Factorization
Minimization View
Alternative Formulations
Definition of the Residual R: R ( x ) ≡ b − Mx
Find x which satisfies
Mx = b
Minimize over all x
N
R ( x ) R ( x ) = ∑ ( Ri ( x ) )
T
2
i =1
Equivalent if b ∈ span {cols ( M )}
T
⇒ Mx = b and min x R ( x ) R ( x ) = 0
Minimization extends to non-singular or nonsquare case!
SMA-HPC ©2003 MIT
Minimization View
QR Factorization
One-dimensional
Minimization
Suppose x = x1e1 and therefore Mx = x1Me1 = x1M 1
One dimensional Minimization
R ( x ) R ( x ) = ( b − x1Me1 ) ( b − x1Me1 )
T
T
= b b − 2 x1b Me1 + x ( Me1 )
T
T
2
1
T
( Me1 )
d
T
T
T
R ( x ) R ( x ) = −2b Me1 + 2 x1 ( Me1 ) ( Me1 ) = 0
dx
T
b Me1
x1 = T T
e1 M Me1 Normalization
SMA-HPC ©2003 MIT
Minimization View
QR Factorization
One-dimensional
Minimization, Picture
Me1 = M 1
b
x1
T
b Me1
x1 = T T
e1 M Me1
e1
One dimensional minimization yields same result as
projection on the column!
SMA-HPC ©2003 MIT
Minimization View
QR Factorization
Two-dimensional
Minimization
Now x = x1e1 + x2 e2 and Mx = x1Me1 + x2 Me2
Residual Minimization
R ( x ) R ( x ) = ( b − x1Me1 − x2 Me2 ) ( b − x1Me1 − x2 Me2 )
T
T
= b b − 2 x1b Me1 + x ( Me1 )
( Me1 )
T
T
2
−2 x2b Me2 + x2 ( Me2 ) ( Me2 )
T
T
Coupling
Term
SMA-HPC ©2003 MIT
2
1
T
+2 x1 x2 ( Me1 )
T
( Me2 )
Minimization View
QR Factorization
Two-dimensional
Minimization Continued
More General Search Directions
x = v1 p1 + v2 p2 and Mx = v1Mp1 + v2 Mp2
span { p1 , p2 } = span {e1 , e2 }
R ( x ) R ( x ) = b b − 2v1b Mp1 + v ( Mp1 )
T
T
T
2
1
T
( Mp1 )
−2v2b Mp2 + v ( Mp2 )
T
Coupling
Term
T
1
( Mp2 )
T
+2v1v2 ( Mp1 ) ( Mp2 )
2
2
T
If p M T Mp2 = 0 Minimizations Decouple!!
SMA-HPC ©2003 MIT
Minimization View
QR Factorization
Forming MTM orthogonal
Minimization Directions
ith search direction equals MTM orthogonalized unit vector
i −1
pi = ei − ∑ rji p j
T
pi M T Mp j = 0
j =1
Use previous orthogonalized
Search directions
Mp ) ( Me )
(
=
( Mp ) ( Mp )
T
⇒ rji
j
T
j
SMA-HPC ©2003 MIT
i
j
Minimization View
QR Factorization
Minimizing in the Search
Direction
Decoupled minimizations done individually
Minimize: v ( Mpi )
2
i
Differentiating:
T
T
Mp
−
2
v
b
( i ) i Mpi
2vi ( Mpi )
T
⇒ vi =
SMA-HPC ©2003 MIT
( Mpi ) − 2bT Mpi = 0
bT Mpi
( Mpi ) ( Mpi )
T
QR Factorization
For i = 1 to N
pi = ei
“For each Source Column left of target”
rij ← pTj M T Mpi
pi ⇐ pi − rij p j
rii = Mpi • Mpi
1
pi ⇐
pi
rii
SMA-HPC ©2003 MIT
Minimization Algorithm
“For each Target Column”
For j = 1 to i-1
x = x + vi pi
Minimization View
Orthogonalize Search Direction
Normalize search direction
QR Factorization
Q1
Minimization and QR
Comparison
QN
Q2
M
1
e1
r11
p1
M
1
( e2 − r12e1 )
r22
SMA-HPC ©2003 MIT
p2
Orthonormal
M
1
e2 − ∑ riN ei )
(
MTM
rNN
Orthonormal
pN
QR Factorization
Search Direction
Orthogonalized unit vectors Æ search directions
{ p1 ,… , pN }
{e1 , e2 ,… , eN }
Unit Vectors
MTM
Orthogonalization
Search Directions
Could use other sets of starting vectors
2
b
,
Mb
,
M
b,…}
{
MTM
Krylov-Subspace Orthogonalization
Why?
SMA-HPC ©2003 MIT
{ p1 ,… , pN }
Search Directions
Summary
• QR Algorithm
– Projection Formulas
– Orthonormalizing the columns as you go
– Modified Gram-Schmidt Algorithm
• QR and Singular Matrices
– Matrix is singular, column of Q is zero.
• Minimization View of QR
– Basic Minimization approach
– Orthogonalized Search Directions
– QR and Length minimization produce identical results
• Mentioned changing the search directions
SMA-HPC ©2003 MIT