Administrivia: November 9, 2009
• Final project proposals are due this Wednesday
– please come discuss with me.
• Reading in Saad (either edition):
• Sections 10.1 – 10.5: Preconditioning
• Recommended:
• Sections 6.5.1 – 6.5.4, 6.5.6: GMRES
• Section 6.7.1: Conjugate gradient
• Section 7.4.2: BiCGSTAB
• Chapters 6 & 7 of Saad are excellent references on
other Krylov subspace methods, but we won’t cover
most of them in class.
Conjugate gradient iteration
x0 = 0, r0 = b, d0 = r0
for k = 1, 2, 3, . . .
αk = (rTk-1rk-1) / (dTk-1Adk-1)
xk = xk-1 + αk dk-1
rk = rk-1 – αk Adk-1
βk = (rTk rk) / (rTk-1rk-1)
dk = rk + βk dk-1
step length
approx solution
residual
improvement
search direction
• One matrix-vector multiplication per iteration
• Two vector dot products per iteration
• Four n-vectors of working storage
Conjugate gradient: Orthogonal sequences
• Krylov subspace: Ki (A, b) = span (b, Ab, A2b, . . ., Ai-1b)
• Conjugate gradient algorithm:
for i = 1, 2, 3, . . .
find xi  Ki (A, b)
such that ri = (b – Axi)  Ki (A, b)
• Notice ri  Ki+1 (A, b), so ri  rj for all j < i
• Similarly, the “directions” are A-orthogonal:
(xi – xi-1 )T·A· (xj – xj-1 ) = 0
• The magic: Short recurrences. . .
A is symmetric => can get next residual and direction
from the previous one, without saving them all.
Conjugate gradient: Convergence
• In exact arithmetic, CG converges in n steps
(completely unrealistic!!)
• Accuracy after k steps of CG is related to:
• consider polynomials of degree k that are equal to 1 at 0.
• how small can such a polynomial be at all the eigenvalues of A?
• Thus, eigenvalues close together are good.
• Condition number: κ(A) = ||A||2 ||A-1||2 = λmax(A) / λmin(A)
• Residual is reduced by a constant factor by
O(κ1/2(A)) iterations of CG.
Complexity of linear solvers
Time to solve
model problem
(Poisson’s
equation) on
regular mesh
n1/2
n1/3
2D
3D
Dense Cholesky:
O(n3 )
O(n3 )
Sparse Cholesky:
O(n1.5 )
O(n2 )
CG, exact arithmetic:
O(n2 )
O(n2 )
CG, no precond:
O(n1.5 )
O(n1.33 )
CG, modified IC0:
O(n1.25 )
O(n1.17 )
CG, support trees:
Multigrid:
O(n1.20 ) -> O(n1+ ) O(n1.75 ) -> O(n1.31 )
O(n)
O(n)