Introduction to Simulation - Lecture 6
Krylov-Subspace Matrix Solution Methods
Jacob White
Thanks to Deepak Ramaswamy, Michal Rewienski,
and Karen Veroy
Outline
• General Subspace Minimization Algorithm
– Review orthogonalization and projection formulas
• Generalized Conjugate Residual Algorithm
– Krylov-subspace
– Simplification in the symmetric case.
– Convergence properties
• Eigenvalue and Eigenvector Review
– Norms and Spectral Radius
– Spectral Mapping Theorem
Arbitrary Subspace
methods
Approach to Approximately Solving Mx=b
⎧ ⎡ w01 ⎤
⎡ wk −11 ⎤ ⎫
⎪⎢
⎥
⎢
⎥⎪
⎨⎢
⎥ ,..., ⎢
⎥ ⎬ ≡ {w0 ,..., wk −1}
⎪⎢ w ⎥
⎪
⎢
⎥
w
⎣ k −1N ⎦ ⎭
⎩ ⎣ 0N ⎦
Pick a kdimensional
Subspace
Approximate x as a weighted sum of {w0 ,..., wk −1}
k
⇒ x =
k
k −1
∑α w
i =0
SMA-HPC ©2003 MIT
i
i
Residual Minimization
Arbitrary Subspace
methods
The residual is defined as r ≡ b − Mx
k
If x =
k
k
k −1
∑α w
i
i =0
i
k −1
⇒ r = b − Mx = b − ∑ α i Mwi
k
k
i =0
Residual Minimizing idea: pick α i ' s to minimize
r
k 2
2
( ) (r )
≡ r
k
SMA-HPC ©2003 MIT
T
k
T
k −1
⎛
⎞ ⎛
⎞
= ⎜ b − ∑ α i Mwi ⎟ ⎜ b − ∑ α i Mwi ⎟
i =0
i =0
⎝
⎠ ⎝
⎠
k −1
Arbitrary Subspace
methods
Minimizing r
k 2
2
= b−
Residual Minimization
Computational Approach
k −1
∑ α Mw
i =0
i
2
is easy if
i
2
( Mw ) = 0, i ≠ j or ( Mw ) orthogonal to ( Mw )
Create a set of vectors { p0 ,..., pk −1} such that
( Mwi )
T
j
i
span { p0 ,..., pk −1} = span {w0 ,..., wk −1}
and ( Mpi ) ( Mp j ) = 0, i ≠ j
T
SMA-HPC ©2003 MIT
j
Residual Minimization
Arbitrary Subspace
methods
Algorithm Steps
Given M , b and a set of search directions {w0 ,..., wk −1}
1) Generate p j 's by orthogonalizing Mw j ' s
j −1
For j = 0 to k − 1 p j = w j − ∑
i =0
( Mw ) ( Mp ) p
T
j
i
( Mpi ) ( Mpi )
T
2) compute the r minimizing solution x k
k −1
( r ) ( Mp )
i =0
( Mpi ) ( Mpi )
x =∑
k
SMA-HPC ©2003 MIT
0 T
i
T
k −1
( r ) ( Mp )
i =0
( Mpi ) ( Mpi )
pi = ∑
i T
i
T
pi
i
Arbitrary Subspace
methods
1) orthogonalize the Mwi ' s
w
p00
w11
p
Residual Minimization
Algorithm Steps by Picture
w
p22
w
p33
2) compute the r minimizing solution x k
M p1
r0
SMA-HPC ©2003 MIT
M p0
Minimization Algorithm
Arbitrary Subspace
Solution Algorithm
r 0 = b − Ax 0
For j = 0 to k-1
p j = wj
For i = 0 to j-1
T
p j ← p j − ( Mp j ) ( Mpi ) pi
pj ←
1
( Mp ) ( Mp )
T
j
x
j +1
r
j +1
j
Normalize
j
= x + (r
j
=r
pj
) ( Mp ) p
− ( r ) ( Mp ) M p
SMA-HPC ©2003 MIT
Orthogonalize
Search Direction
j T
j
Update Solution
j
j T
j
j
Update Residual
Arbitrary Subspace
methods
Subspace Selection
Criteria
Criteria for selecting w0 ,..., wk −1
All that matters is the span {w0 ,..., wk −1}
∃ α i ' s such that b − Mx k = b −
k −1
∑ α Mw
i =0
i
A b ≈ in the span {w0 ,..., wk −1} for k
−1
i
is small
N
One choice, unit vectors, x k ∈ span {e1 ,..., ek }
Generates the QR algorithm if k=N
Can be terrible if k < N
SMA-HPC ©2003 MIT
Arbitrary Subspace
methods
Subspace Selection
Historical Development
1 T
T
Consider minimizing f ( x ) = x Mx − x b
2
T
T
Assume M = M (symmetric) and x Mx > 0 (pos. def)
∇ x f ( x ) = Mx − b ⇒ x = M −1b minimizes f
{
( )
(
Pick span {w0 ,..., wk −1} = span ∇ x f x ,..., ∇ x f x
0
Steepest descent directions for f, but f is not residual
Does not extend to nonsymmetric, non pos def case
SMA-HPC ©2003 MIT
k −1
)}
Arbitrary Subspace
methods
{
Subspace Selection
Krylov Subspace
Note: span ∇ x f x 0 ,..., ∇ x f x k −1
)}
{
0
( )
(
{
= span r 0 ,..., r k −1
If: span {w0 ,..., wk −1} = span r ,..., r
then r k = r 0 −
{
}
k −1
i
α
Mr
∑ i
i =0
0
k −1
and span r ,..., r
k −1
} = span {r , Mr ,..., M
0
0
k −1 0
Krylov Subspace
SMA-HPC ©2003 MIT
r
}
}
The Generalized Conjugate
Residual Algorithm
Krylov Methods
The kth step of GCR
( r ) ( Mp )
k T
αk =
k
( Mpk ) ( Mpk )
T
Determine optimal stepsize in
kth search direction
x k +1 = x k + α k pk
r
k +1
Update the solution
and the residual
= r − α k Mpk
pk +1 = r
k
k +1
Mr ) ( Mp )
(
−∑
p
( Mp ) ( Mp )
SMA-HPC ©2003 MIT
k
j =0
k +1 T
j
T
j
j
j
Compute the new
orthogonalized
search direction
The Generalized Conjugate
Residual Algorithm
Krylov Methods
Algorithm Cost for iter k
( r ) ( Mp )
k T
αk =
Vector inner products, O(n)
Matrix-vector product, O(n) if sparse
k
( Mpk ) ( Mpk )
T
x k +1 = x k + α k pk
r
k +1
Vector Adds, O(n)
= r − α k Mpk
pk +1 = r
k
k +1
Mr ) ( Mp )
(
−∑
p
( Mp ) ( Mp )
k
j =0
k +1 T
j
T
j
j
j
O(k) inner products,
total cost O(nk)
If M is sparse, as k (# of iters) approaches n,
3
total cost = O (n ) + O (2n ) + .... + O ( kn ) = O (n )
SMA-HPC ©2003 MIT
Better Converge Fast!
The Generalized Conjugate
Residual Algorithm
Krylov Methods
Symmetric Case
An Amazing fact that will not be derived
T
k +1
j
If M = M then r ⊥ Mp j < k
Mr ) ( Mp )
Mr ) ( Mp )
(
(
p = r −∑
p ⇒ p =r −
p
( Mp ) ( Mp )
( Mp ) ( Mp )
Orthogonalization in one step
If k (# of iters ) Æ n, then symmetric,
sparse, GCR is O(n2 )
Better Converge Fast!
k +1
k +1
k
j =0
SMA-HPC ©2003 MIT
k +1 T
k +1 T
j
j
T
j
j
k +1
k
k +1
T
k
k
k
Krylov Methods
Nodal Formulation
“No-leak Example”
Insulated bar and Matrix
Incoming Heat
T (1)
T (0)
Near End
Temperature
Discretization
m
SMA-HPC ©2003 MIT
⎡ 2 −1
⎤
⎢−1 2
⎥ Nodal
⎢
⎥
⎢
−1⎥ Equation
Form
⎢
⎥
−1 2 ⎦
⎣
M
Far End
Temperature
Krylov Methods
Nodal Formulation
1
2
m
SMA-HPC ©2003 MIT
3
“No-leak Example”
Circuit and Matrix
4
⎡ 2 −1
⎤
⎢−1 2
⎥
⎢
⎥
⎢
−1⎥
⎢
⎥
−1 2 ⎦
⎣
M
m −1
Nodal
Equation
Form
m
Krylov Methods
Nodal Formulation
“leaky” Example
Conducting bar and Matrix
T (1)
T (0)
Near End
Temperature
Discretization
⎡2.01 −1
⎤
⎢ −1 2.01
⎥ Nodal
⎢
⎥ Equation
⎢
−1 ⎥ Form
⎢
⎥
−1 2.01⎦
⎣
M
m
SMA-HPC ©2003 MIT
Far End
Temperature
“leaky” Example
Krylov Methods
Nodal Formulation
1
3
2
m
SMA-HPC ©2003 MIT
Circuit and Matrix
4
m −1
⎡2.01 −1
⎤
⎢ −1 2.01
⎥ Nodal
⎢
⎥ Equation
⎢
−1 ⎥
Form
⎢
⎥
−1 2.01⎦
⎣
M
m
GCR Performance(Random Rhs)
10
R
E
S
I
D
U
A
L
10
10
10
10
0
Insulating
-1
Leaky
-2
-3
-4
0
10
20
Iteration
30
40
Plot of log(residual) versus Iteration
SMA-HPC ©2003 MIT
50
60
GCR Performance(Rhs = -1,+1,-1,+1….)
0
10
R
E
S
I
D
U
A
L
-1
10
-2
10
-3
10
Insulating
-4
10
Leaky
-5
10
0
5
10
15
20
25
Iteration
30
35
Plot of log(residual) versus Iteration
SMA-HPC ©2003 MIT
40
45
50
Convergence Analysis
Krylov Subspace
Methods
Polynomial Approach
{
If span {w0 ,..., wk } = span r 0 , Mr 0 ,..., M k r 0
x
k +1
=
k
∑α M r
i =0
r
k +1
i 0
i
k
= ξk ( M ) r
}
0
kth order polynomial
= r − ∑αi M r = ( I − M ξk ( M )) r
0
i +1 0
i =0
Note: for any α 0 ≠ 0
0
1
0
0
span r , r = r − α 0 Mr
{
SMA-HPC ©2003 MIT
}=
{
0
span r , Mr
0
}
0
Krylov Methods
Convergence Analysis
Basic Properties
If α j ≠ 0 for all j ≤ k in GCR, then
0
0
k 0
1) span { p0 , p1 ,..., pk } = span r , Mr , ..., M r
2) x
{
= ξ k ( M )r , ξ k is the k order
k +1
polynomial which minimizes r
k +1
0
}
th
2
2
= b − Mx = r − M ξ k ( M )r
0
0
= ( I − M ξ k ( M ) ) r ≡℘k +1 ( M ) r
th
0
where ℘k +1 ( M ) r is the ( k + 1) order poly
k +1 2
minimizing r
subject to ℘k +1 ( 0 ) =1
3) r
k +1
k +1
0
0
2
SMA-HPC ©2003 MIT
Convergence Analysis
Krylov Methods
Optimality of GCR poly
GCR Optimality Property
r
k +1 2
2
≤ ℘k+1 ( M )r
0 2
2
where ℘k+1 is any k order
th
polynomial such that ℘k+1 ( 0 ) =1
Therefore
Any polynomial which satisfies
the zero constraint can be used
to get an upper bound on
SMA-HPC ©2003 MIT
r
k +1 2
2
Eigenvalues and
Vectors Review
Basic Definitions
Eigenvalues and eigenvectors of a matrix M satisfy
eigenvalue
Mui = λi ui
eigenvector
Or, λi is an eigenvalue of M if
M − λi I is singular
ui is an eigenvector of M if
( M − λi I ) ui
SMA-HPC ©2003 MIT
=0
Eigenvalues and
Vectors Review
⎡1.1 −1⎤
⎢ −1 1.1⎥
⎣
⎦
⎡ M 11
⎢M
⎢ 21
⎢
⎢
⎢⎣ M N 1
Examples
⎡ 1 −1 0 0 ⎤
⎢ −1 1 0 0 ⎥
⎢
⎥
⎢ 0 0 1 −1⎥
⎢
⎥
⎣ 0 0 −1 2 ⎦
0
M 22
SMA-HPC ©2003 MIT
Basic Definitions
M NN −1
0 ⎤
⎥
⎥
0 ⎥
⎥
M NN ⎥⎦
Eigenvalues?
Eigenvectors?
What about a lower
triangular matrix
A Simplifying
Assumption
Eigenvalues and
Vectors Review
Almost all NxN matrices have N linearly
independent Eigenvectors
M
⎡↑
⎢
⎢u1
⎢↓
⎣
↑
u2
↓
↑
u3
↓
↑⎤
⎥
uN ⎥
↓ ⎥⎦
⎡ ↑
⎢
= ⎢λ1u1
⎢ ↓
⎣
↑
λ2u2
↓
↑
λ3u3
↓
↑ ⎤
⎥
λN u N ⎥
↓ ⎥⎦
The set of all eigenvalues of M is known as
the Spectrum of M
SMA-HPC ©2003 MIT
Eigenvalues and
Vectors Review
A Simplifying
Assumption Continued
Almost all NxN matrices have N linearly
independent Eigenvectors
MU =U
⎡λ1
⎢0
⎢
⎢⎣ 0
0
0
0⎤
0 ⎥⎥
λN ⎥⎦
U MU = λ or M = U λU
−1
−1
Does NOT imply distinct eigenvalues, λi can equal λ j
Does NOT imply M is nonsingular
SMA-HPC ©2003 MIT
Eigenvalues and
Vectors Review
Im ( λ )
λi
Spectral Radius
Re ( λ )
The spectral Radius of M is the radius of the smallest
circle, centered at the origin, which encloses all of
M’s eigenvalues
SMA-HPC ©2003 MIT
Eigenvalues and
Vectors Review
Heat Flow Example
Incoming Heat
T (1)
T (0)
Unit Length Rod
T1
+
-
vs = T(0)
SMA-HPC ©2003 MIT
TN
+
-
vs = T(1)
Eigenvalues and
Vectors Review
Heat Flow Example
Continued
⎡ 2 −1 0 0 ⎤
⎢ −1 2
⎥
0
⎢
⎥
⎢0
−1⎥
⎢
⎥
0
0
1
2
−
⎣
⎦
Eigenvalues N=20
SMA-HPC ©2003 MIT
Eigenvalues and
Vectors Review
Heat Flow Example
Continued
Four Eigenvectors – Which ones?
SMA-HPC ©2003 MIT
Useful
Eigenproperties
Spectral Mapping
Theorem
Given a polynomial
f ( x ) = a0 + a1 x + … + a p x
p
Apply the polynomial to a matrix
f ( M ) = a0 + a1M + … + a p M
Then
p
spectrum ( f ( M ) ) = f ( spectrum ( M ) )
SMA-HPC ©2003 MIT
Useful
Eigenproperties
Spectral Mapping
Theorem Proof
Note a property of matrix powers
MM = U λU U λU = U λ U
p
p −1
⇒ M = Uλ U
−1
−1
2
−1
Apply to the polynomial of the matrix
−1
−1
p −1
f ( M ) = a0UU + a1U λU + … + a pU λ U
Factoring
f ( M ) = U ( a0 I + a1λ + … + a p λ
Diagonal
f ( M ) U = U ( a0 I + a1λ + … + a p λ
SMA-HPC ©2003 MIT
p
p
)
)U
−1
Useful
Eigenproperties
Spectral
Decomposition
Decompose arbitrary x in eigencomponents
x = α1u1 + α 2u2 + … + α N u N
⎡ α1 ⎤
Compute by solving U ⎢ ⎥ = x ⇒ α = U −1 x
⎢ ⎥
⎢⎣α N ⎥⎦
Applying M to x yeilds
Mx = M (α1u1 + α 2u2 + … + α N u N )
= α1λ1u1 + α 2 λ2u2 + … + α N λN u N
SMA-HPC ©2003 MIT
Krylov Methods
Convergence Analysis
Important Observations
1) The GCR Algorithm converges to the exact solution
in at most n steps
Proof: Let ℘n ( x ) = ( x − λ1 )( x − λ2 ) ... ( x − λn )
where λi ∈λ ( M ) .
⇒ ℘n ( M ) r 0 = 0 and therefore r n = 0
2) If M has only q distinct eigenvalues, the GCR
Algorithm converges in at most q steps
Proof: Let ℘q ( x ) = ( x − λ1 )( x − λ2 ) ... ( x − λq )
SMA-HPC ©2003 MIT
Summary
• Arbitrary Subspace Algorithm
– Orthogonalization of Search Directions
• Generalized Conjugate Residual Algorithm
– Krylov-subspace
– Simplification in the symmetric case.
– Leaky and insulating examples
• Eigenvalue and Eigenvector Review
– Spectral Mapping Theorem
• GCR limiting Cases
– Q-step guaranteed convergence