Gram-Schmidt as Triangular Orthogonalization
• Gram-Schmidt multiplies with triangular matrices to make columns
orthogonal, for example at the first step:
Lecture 6
Householder Reflectors and Givens Rotations
"
MIT 18.335J / 6.337J
Introduction to Numerical Methods
Per-Olof Persson (persson@mit.edu)
v1 v2 · · ·

#


vn 


1
r11
−r12
r11
−r13
r11
···
1
1
..
.

 " 

(2)  = q1 v 2 · · ·


#
(2)
vn
• After all the steps we get a product of triangular matrices
September 26, 2007
A R1 R2 · · · Rn = Q̂
|
{z
}
R̂−1
• “Triangular orthogonalization”
1
2
Householder Triangularization
Introducing Zeros
• The Householder method multiplies by unitary matrices to make columns
• Qk introduces zeros below the diagonal in column k
triangular, for example at the first step:





Q1 A = 




r11 ×
× ···
0
×
×
···
0
..
.
×
×
..
.
···
0
×
×
···
..
.
×
×


×
×


×
×

.. 
.

×
×
• After all the steps we get a product of orthogonal matrices
• Preserves all the zeros previously introduced

×

 ×


 ×


 ×

×
×
×
×
×
×
A
×


× 


× 


× 

×

Q1
−→
×
×

 0


 0


 0

0
×
×
×
×
×
×
×
×
×
×
Q1 A
×
×


×
× 


×
× 


×
× 

×
×

Q2
−→
×









×
×
×
0
0
0
×


×
× 


×
× 


×
× 

×
×

Q3
−→
Q2 Q1 A
×









×
×
×

× 


×
× 


0 

0
Q3 Q2 Q1 A
Qn · · · Q 2 Q1 A = R
{z
}
|
Q∗
• “Orthogonal triangularization”
3
4
Householder Reflectors
Householder Reflectors
• Idea: Reflect across hyperplane H orthogonal to v = kxke1 − x, by the
• Let Qk be of the form

where I is (k
Qk = 
I
0
0 F


− 1) × (k − 1) and F is (m − k + 1) × (m − k + 1)
• Create Householder reflector F that introduces zeros:


 
kxk
×


 
 0 
×


 
x =  .  F x =  .  = kxke1
 .. 
 .. 


 
0
×
5
unitary matrix
F =I −2
• Compare with projector
P⊥v
vv ∗
v∗v
x
vv ∗
=I− ∗
v v
v
F x = kxke1
H
6

The Householder Algorithm
Choice of Reflector
• We can choose to reflect to any multiple z of kxke1 with |z| = 1
• Compute the factor R of a QR factorization of m × n matrix A (m ≥ n)
• Better numerical properties with large kvk, for example
• Leave result in place of A, store reflection vectors vk for later use
v = sign(x1 )kxke1 + x
Algorithm: Householder QR Factorization
x
• Note: sign(0) = 1, but in
for k
MATLAB, sign(0)==0
−kxke1 − x
+kxke1 − x
−kxke1
+kxke1
H
+
H
−
= 1 to n
x = Ak:m,k
vk = sign(x1 )kxk2 e1 + x
vk = vk /kvk k2
Ak:m,k:n = Ak:m,k:n − 2vk (vk∗ Ak:m,k:n )
7
8
Applying or Forming Q
Operation Count - Householder QR
• Compute Q∗ b = Qn · · · Q2 Q1 b and Qx = Q1 Q2 · · · Qn x implicitly
• Most work done by
Ak:m,k:n = Ak:m,k:n − 2vk (vk∗ Ak:m,k:n )
• To create Q explicitly, apply to x = I
Algorithm: Implicit Calculation of Q∗ b
for k
= 1 to n
bk:m = bk:m − 2vk (vk∗ bk:m )
Algorithm: Implicit Calculation of Qx
• Operations per iteration:
–
2(m − k)(n − k) for the dot products vk∗ Ak:m,k:n
–
(m − k)(n − k) for the outer product 2vk (· · · )
–
(m − k)(n − k) for the subtraction Ak:m,k:n − · · ·
–
4(m − k)(n − k) total
• Including the outer loop, the total becomes
for k
= n downto 1
xk:m = xk:m − 2vk (vk∗ xk:m )
n
X
4(m − k)(n − k) = 4
k=1
n
X
(mn − k(m + n) + k 2 )
k=1
∼ 4mn2 − 4(m + n)n2 /2 + 4n3 /3 = 2mn2 − 2n3 /3
9
10
Givens Rotations
Givens QR
• Alternative to Householder reflectors


cos θ − sin θ
 rotates x ∈ R2 by θ
• A Givens rotation R = 
sin θ cos θ
• To set an element to zero, choose cos θ and sin θ so that

   q

x2i + x2j
cos θ − sin θ
xi

  = 

sin θ cos θ
xj
0
or
cos θ = q
xi
x2i + x2j
,
11
−xj
sin θ = q
x2i + x2j
• Introduce zeros in column from bottom and up

×

 ×


 ×

×

×
×

 0




×
×
×
×
×
×
×
×
×
×
×





× 
 (3,4) 
 −→ 


× 

×


×
×



×
× 
 (3,4) 
 −→ 

× 


×
×
×
×
×
×
×
×
×
0
×
×
×
×
×
×
×
0
×


×


 ×
× 
 (2,3)  ×
 −→ 

 0
×
× 

×
×


×
×



× 
(2,3)


 −→ 

×
× 


×
×
×
×
×
×
×
×
×
×
×
0
×

×
× 
 (1,2)
 −→
×
× 

×

×

×
× 
 (3,4)
 −→ R
×
× 

×
• Flop count 3mn2 − n3 (or 50% more than Householder QR)
12
