Introduction to Simulation - Lecture 7
Krylov-Subspace Matrix Solution Methods
Part II
Jacob White
Thanks to Deepak Ramaswamy, Michal Rewienski,
and Karen Veroy
1
Outline
• Reminder about GCR
– Residual minimizing solution
– Krylov Subspace
– Polynomial Connection
• Review Eigenvalues and Norms
– Induced Norms
– Spectral mapping theorem
• Estimating Convergence Rate
– Chebychev Polynomials
• Preconditioners
– Diagonal Preconditioners
– Approximate LU preconditioners
2
With Normalization
Generalized
Conjugate Residual
Algorithm
r 0 = b − Ax 0
For j = 0 to k-1
pj = r j
Residual is next search direction
For i = 0 to j-1
Orthogonalize
T
p j ← p j − ( Mp j ) ( Mpi ) pi Search Direction
pj ←
1
pj
( Mp ) ( Mp )
T
j
j
x j +1 = x j + ( r
r j +1 = r j
Normalize
) ( Mp ) p
− ( r ) ( Mp ) M p
j T
j
Update Solution
j
j T
j
j
Update Residual
SMA-HPC ©2003 MIT
3
Generalized
Conjugate Residual
Algorithm
1) orthogonalize the Mr i ' s
rp00
rp11
With Normalization
Algorithm Steps by Picture
pr 22
rp33
2) compute the r minimizing solution x k
r k+1
SMA-HPC ©2003 MIT
rk
M pk
4
First Few Steps
Generalized
Conjugate Residual
Algorithm
r0
• First search direction r = b − Mx = b, p0 =
Mr 0
0
• Residual minimizing x1 =
solution
• Second Search
Direction
0
(
)
( r 0 ) Mp0 p0
T
r1 = b − Mx1 = r 0 − γ 1Mr 0
p1 =
r1 − β1,0 p0
(
M r1 − β1,0 p0
)
SMA-HPC ©2003 MIT
5
Generalized
Conjugate Residual
Algorithm
• Residual minimizing
solution
First few steps
Continued…
(
)
x 2 = x1 + ( r1 ) Mp1 p1
T
• Third Search Direction
r 2 = b − Mx 2 = r 0 − γ 2,1Mr 0 − γ 2,0 M 2 r 0
p2 =
r1 − β 2,0 p0 − β 2,1 p1
(
M r1 − β 2,0 p0 − β 2,1 p1
)
SMA-HPC ©2003 MIT
6
The kth step of GCR
Generalized
Conjugate Residual
Algorithm
k −1
pk = r − ∑ ( Mr k )
k
j =0
pk =
pk
Mpk
α k = ( r k ) ( Mpk )
T
x k +1 = x k + α k pk
r k +1 = r k − α k Mpk
T
( Mp ) p
j
j
Orthogonalize and
normalize search
direction
Determine optimal stepsize in
kth search direction
Update the solution
and the residual
SMA-HPC ©2003 MIT
7
Polynomial view
Generalized
Conjugate Residual
Algorithm
If α j ≠ 0 for all j ≤ k in GCR, then
1) span { p0 , p1 ,..., pk } = span r 0 , Mr 0 ,..., Mr k
{
}
2) x k +1 = ξ k ( M ) r 0 , ξ k is the k th order poly
2
minimizing r k +1
2
k +1
k +1
0
3) r = b − Mx = r − M ξ k ( M ) r 0
= ( I − M ξ k ( M ) ) r 0 ≡℘k +1 ( M ) r 0
th
where ℘k +1 ( M ) r 0 is the ( k + 1) order poly
2
minimizing r k +1 subject to ℘k +1 ( 0 ) =1
SMA-HPC ©2003 MIT
2
8
Residual Minimization
Krylov Methods
Polynomial View
{
If x k +1 ∈ span r 0 , Mr 0 ,..., Mr k
}
minimizes r k +1
2
2
= ξ k ( M )r , ξ k is the k order poly
2
minimizing r k +1
2
k +1
k +1
2) r = b − Mx = ( I − M ξ k ( M ) ) r 0 =℘k +1 ( M ) r 0
th
where ℘k +1 ( M ) r 0 is the ( k + 1) order poly
k +1 2
minimizing r
subject to ℘k +1 ( 0 ) =1
1) x
k +1
0
th
2
Polynomial Property only a function of
solution space and residual minimization
SMA-HPC ©2003 MIT
9
Krylov Methods
Nodal Formulation
“No-leak Example”
Insulated bar and Matrix
Incoming Heat
T (1)
T (0)
Near End
Temperature
Discretization
m
SMA-HPC ©2003 MIT
Far End
Temperature
⎡ 2 −1
⎤
⎢−1 2
⎥ Nodal
⎢
⎥
⎢
−1⎥ Equation
Form
⎢
⎥
−
1
2
⎣
⎦
M
10
Krylov Methods
Nodal Formulation
“leaky” Example
Conducting bar and Matrix
T (1)
T (0)
Near End
Temperature
Discretization
Far End
Temperature
⎡2.01 −1
⎤
⎢ −1 2.01
⎥ Nodal
⎢
⎥ Equation
⎢
−1 ⎥ Form
⎢
⎥
−1 2.01⎦
⎣
M
m
SMA-HPC ©2003 MIT
11
GCR Performance(Random Rhs)
10
R
E
S
I
D
U
A
L
10
10
10
10
0
Insulating
-1
Leaky
-2
-3
-4
0
10
20
Iteration
30
40
50
60
Plot of log(residual) versus Iteration
SMA-HPC ©2003 MIT
12
GCR Performance(Rhs = -1,+1,-1,+1….)
0
10
R
E
S
I
D
U
A
L
-1
10
-2
10
-3
10
Insulating
-4
10
Leaky
-5
10
0
5
10
15
20
25
Iteration
30
35
40
45
50
Plot of log(residual) versus Iteration
SMA-HPC ©2003 MIT
13
Krylov Methods
Residual Minimization
Optimality of poly
Residual Minimizing Optimality Property
r k +1 ≤ ℘k+1 ( M )r 0 ≤ ℘k+1 ( M ) r 0
℘k+1 is any k th order poly such that ℘k+1 ( 0 ) =1
Therefore
Any polynomial which satisfies
the constraints can be used to
get an upper bound on
r k +1
r0
SMA-HPC ©2003 MIT
14
Induced Norms
Matrix Magnification
Question
Suppose y = Mx
How much larger is y than x?
OR
How much does M magnify x?
SMA-HPC ©2003 MIT
15
Vector Norm
Review
Induced Norms
L2 (Euclidean) norm :
x
2
=
1
∑
i=1
L1 norm :
x
n
=
xi
n
∑
x
x
xi
i=1
2
2
1
< 1
< 1
L∞ norm :
x
∞
= max
i
xi
x
∞
< 1
SMA-HPC ©2003 MIT
16
Standard Induced
l-norms
Induced Matrix
Norms
Definition:
M l ≡ max
Mx
x
x
Examples
M
M
SMA-HPC ©2003 MIT
≡ max
1
∞
= max
l
l
N
i∑
j =1
≡ max
M ij
N
M
j∑
i =1
ij
x l =1
Mx
l
Max Column
Sum
Max Row
Sum
17
Standard Induced
l-norms continued
Induced Matrix
Norms
M
1
= m ax
j
N
∑
i =1
Why? Let x =
M
∞
= m ax
i
Why? Let
M
2
M
[1
N
∑
j =1
x =
M
= max abs column sum
ij
0
ij
[± 1
0
]
T
= max abs column sum
±1
± 1]
T
Not So easy to compute
SMA-HPC ©2003 MIT
As the algebra on the slide shows the relative changes in the solution x is bounded
by an A-dependent factor times the relative changes in A. The factor
|| A−1 || || A ||
was historically referred to as the condition number of A, but that definition has
been abandoned as then the condition number is norm-dependent. Instead the
condition number of A is the ratio of singular values of A.
cond ( A) =
σ max( A)
σ min( A)
Singular values are outside the scope of this course, consider consulting Trefethen
& Bau.
18
Useful
Eigenproperties
Spectral Mapping
Theorem
Given a polynomial
f ( x ) = a0 + a1 x + … + a p x p
Apply the polynomial to a matrix
f ( M ) = a0 + a1M + … + a p M p
Then
spectrum ( f ( M ) ) = f ( spectrum ( M ) )
SMA-HPC ©2003 MIT
19
Krylov Methods
⎡
⎤
⎢
u N ⎥⎥
=
⎢u1
⎢⎣
⎥⎦
eigenvectors of M
℘k (M )
≤
⎡
⎢u
⎢ 1
⎢⎣
⎤
u N ⎥⎥
⎥⎦
⎡
⎢u
⎢ 1
⎢⎣
Convergence Analysis
Norm of matrix polynomials
⎡℘k ( λ1 )
⎢
⎢
⎢⎣
⎤
u N ⎥⎥
⎥⎦
⎤
⎥
⎥
℘k ( λN ) ⎥⎦
−1
Cond(U)
⎡
⎢u
⎢ 1
⎢⎣
⎡℘k ( λ1 )
⎢
⎢
⎢⎣
⎤
u N ⎥⎥
⎥⎦
−1
⎤
⎥
⎥
℘k ( λN ) ⎥⎦
condition number of
M's eigenspace
SMA-HPC ©2003 MIT
20
Krylov Methods
⎡℘k ( λ1 )
⎢
⎢
⎢⎣
Convergence Analysis
Norm of matrix polynomials
⎤
⎥
⎥ = max x =1
℘k ( λN ) ⎥⎦
2
∑ ℘ (λ ) x
k
i
2
i
i
= max i ℘k ( λi )
℘k ( M )
≤ cond (V ) max i ℘k ( λi )
SMA-HPC ©2003 MIT
21
Krylov Methods
Convergence Analysis
Important Observations
1) A residual minimizing Krylov subspace algorithm
converges to the exact solution in at most n steps
Proof: Let ℘n ( x ) = ( x − λ1 )( x − λ2 ) ... ( x − λn )
where λi ∈λ ( M ) . Then, max i ℘n ( λi ) = 0,
⇒ ℘n ( M ) = 0 and therefore r n = 0
2) If M has only q distinct e-values, the residual
minimizing Krylov subspace algorithm converges
in at most q steps
Proof: Let ℘q ( x ) = ( x − λ1 )( x − λ2 ) ... ( x − λq )
SMA-HPC ©2003 MIT
22
Convergence for M =MT
Krylov Methods
Residual Polynomial
If M = MT then
1) M has orthonormal eigenvectors
⇒ cond (V ) =
⇒
⎡
⎢u
⎢ 1
⎢⎣
⎤
u N ⎥⎥
⎥⎦
⎡
⎢u
⎢ 1
⎢⎣
⎤
u N ⎥⎥
⎥⎦
−1
=1
℘k ( M ) = max i ℘k ( λi )
2) M has real eigenvalues
If M is postive definite, then λ ( M ) > 0
SMA-HPC ©2003 MIT
23
Residual Poly Picture for Heat Conducting Bar Matrix
No loss to air (n=10)
1
* = evals(M)
- = 5th order poly
- = 8th order poly
SMA-HPC ©2003 MIT
24
Residual Poly Picture for Heat Conducting Bar Matrix
No loss to air (n=10)
Keep ℘k ( λi ) as small as possible:
Strategically place zeros of the poly
SMA-HPC ©2003 MIT
25
Convergence for M =MT
Krylov Methods
Polynomial Min-Max Problem
Consider λ ( M ) ∈ [ λmin , λmax ] , λmin > 0
Then a good polynomial ( pk ( M ) is small)
can be found by solving the min-max problem
min kth order max x∈ [λmin ,λmax ] pk ( x )
polys s . t .
pk ( 0 ) =1
The min-max problem is exactly
solved by Chebyshev Polynomials
SMA-HPC ©2003 MIT
26
Convergence for M =MT
Krylov Methods
Chebyshev Solves Min-Max
The Chebyshev Polynomial
Ck ( x ) ≡ cos ( k cos −1 ( x ) ) x ∈ [ −1,1]
min kth order max x∈ [λmin ,λmax ] ℘k ( x )
polys s .t .
℘k ( 0 ) =1
=
max x∈ [λmin ,λmax ]
⎛
λ −x
Ck ⎜ 1 + 2 min
λmax − λmin
⎝
⎛
λmin
Ck ⎜ 1 + 2
λmax − λmin
⎝
⎞
⎟
⎠
⎞
⎟
⎠
SMA-HPC ©2003 MIT
27
Chebyshev Polynomials minimizing over [1,10]
SMA-HPC ©2003 MIT
28
Convergence for M =MT
Krylov Methods
Chebychev Bounds
min kth order max x∈ [λmin ,λmax ] ℘k ( x )
polys s .t .
℘k ( 0 ) =1
=
1
⎛
λmax ⎞
Ck ⎜ 1 − 2
λmax − λmin ⎟⎠
⎝
≤
⎛ λmax
⎞
−1⎟
⎜
λmin
⎟
2⎜
⎜ λmax
⎟
+ 1 ⎟⎟
⎜⎜
⎝ λmin
⎠
k
SMA-HPC ©2003 MIT
29
Convergence for M =MT
Krylov Methods
Chebychev Result
If λ ( M ) ∈ [ λmin , λmax ] , λmin > 0
k
rk
⎛ λmax
⎞
−1⎟
⎜
λmin
⎜
⎟ r0
≤2
⎜ λmax
⎟
+ 1 ⎟⎟
⎜⎜
⎝ λmin
⎠
SMA-HPC ©2003 MIT
30
Preconditioning
Krylov Methods
⎡1
⎢0
⎢
⎢
⎢
⎣0
0
1
0
0⎤
⎥
⎥
0⎥
⎥
1⎦
Diagonal Example
⎡1
⎢0
⎢
⎢
⎢
⎣0
0
2
0
0⎤
⎥
⎥
0⎥
⎥
N⎦
For which problem will GCR Converge Faster?
SMA-HPC ©2003 MIT
31
Preconditioning
Krylov Methods
Diagonal Preconditioners
Let M = D + M nd
(
)
(
)
Apply GCR to D −1M x = I + D −1M nd x = D −1b
• The Inverse of a diagonal is cheap to compute
• Usually improves convergence
SMA-HPC ©2003 MIT
32
Heat Conducting
Bar example
∆x
x1
x2
∆x
100
xi
xi +1
Discretized system
one small ∆x
xn
⎡2 +γ −1
⎤ ⎡uˆ1 ⎤ ⎡ f (x1) ⎤
⎥
⎢ −1 2 +γ
⎥⎢ ⎥ ⎢
⎥
⎢
⎥⎢ ⎥ ⎢
⎥
⎢
⎥⎢ ⎥ ⎢
⎥
⎢
⎥⎢ ⎥ ⎢
−1 1+γ +100
−100
⎥
⎢
⎥⎢ ⎥ = ⎢
⎥
⎢
⎥⎢ ⎥ ⎢
−100 1+γ +100 1
⎥
⎢
⎥⎢ ⎥ ⎢
−1
−1 ⎥ ⎢ ⎥ ⎢
⎥
⎢
⎢
⎥ ⎢uˆ ⎥ ⎢ f (x )⎥
1
2
γ
−
+
⎣
⎦⎣ n ⎦ ⎣ n ⎦
λmax
> 100
λmin
SMA-HPC ©2003 MIT
33
Which Convergence Curve is GCR?
rk
r0
Iteration
SMA-HPC ©2003 MIT
34
Heat Conducting
Bar example
Preconditioned Matrix
Eigenvalues
Residual Minimizing
Krylov-subspace
Algorithm can
eliminate outlying
eigenvalues by
placing polynomial
zeros directly on
them.
SMA-HPC ©2003 MIT
35
The World According
to Krylov
Heat Flow Comparison Example
Dimension Dense GE Sparse GE
GCR
1
O ( m3 )
O (m)
O ( m2 )
2
O ( m6 )
O ( m3 )
O ( m3 )
3
O ( m9 )
O ( m6 )
O ( m4 )
GCR faster than banded GE in 2 and 3 dimensions
Could be faster, 3-D matrix only m3 nonzeros.
GCR converges too slowly!
SMA-HPC ©2003 MIT
36
Preconditioning
Krylov Methods
Approximate LU
Preconditioners
Let M ≈ L U
Applying GCR to
((
LU
)
−1
)
( )
M x = LU
−1
b
Use an Implicit matrix representation!
Forming y =
(( LU )
−1
)
M x is equivalent to
solving LUy = Mx
SMA-HPC ©2003 MIT
37
Preconditioning
Krylov Methods
Approximate LU
Preconditioners Continued
Nonzeros in an exact LU Factorization
Filled-in LU factorization
Too expensive.
Ignore the fillin!
SMA-HPC ©2003 MIT
38
Factoring 2-D Grid Matrices
Generated Fill-in Makes Factorization Expensive
SMA-HPC ©2003 MIT
39
Preconditioning
Krylov Methods
Approximate LU
Preconditioners Continued
THROW AWAY FILL-INS!
Throw away all fill-ins
Throw away only fill-ins with small values
Throw away fill-ins produced by other fill-ins
Throw away fill-ins produced by fill-ins of
other fill-ins, etc.
40
Summary
• Reminder about GCR
– Residual minimizing solution
– Krylov Subspace
– Polynomial Connection
• Review Norms and Eigenvalues
– Induced Norms
– Spectral mapping theorem
• Estimating Convergence Rate
– Chebychev Polynomials
• Preconditioners
– Diagonal Preconditioners
– Approximate LU preconditioners
SMA-HPC ©2003 MIT
41