Iterative methods
Tom Lyche
University of Oslo
Norway
Iterative methods – p. 1/2
Plan for the day
The classic iterative methods
Richardson’s method
Jacobi’s method
Gauss Seidel’s method
SOR
numerical examples
matrix splitting, convergence, and rate of convergence
numerical examples
Iterative methods – p. 2/2
Fix Point Iteration
Consider solving a linear system Ax = b, where A ∈ Cn,n is
nonsingular and b ∈ Cn .
In an iterative method we start with an approximation x0 to the exact
solution x and then compute a sequence {xk } such that hopefully
xk → x.
We consider methods where each xk+1 is computed using only A, b
and the previous iterate xk .
In general such an iteration can be written xk+1 = gk (xk ), where
{gk } is a sequence of functions.
The method is stationary if each gk = g is independent of k and
non-stationary otherwise.
Iterative methods – p. 3/2
Jacobi’s and Gauss-Seidel’s method
4x1
−x1
−x1
−x2
+4x2
−x3
+4x3
−x2
−x3
= b1
x1
= 41 x2 + 14 x3 + 41 b1
−x4
= b2
x2
= 14 x1 + 14 x4 + 41 b2
+4x4
= b4 .
−x4
= b3
→
x3
=
x4
=
1
4 x1
1
4 x2
+
+
1
4 x4
1
4 x3
+
+
1
4 b3
1
4 b4
(1)
xk+1 (1) = 14 xk (2) + 14 xk (3) + 14 b1
xk+1 (1) = 14 xk (2) + 14 xk (3) + 41 b1
xk+1 (2) = 14 xk (1) + 14 xk (4) + 14 b2
xk+1 (2) = 14 xk+1 (1) + 14 xk (4) + 14 b2
xk+1 (3) = 14 xk (1) + 14 xk (4) + 14 b3
xk+1 (3) = 14 xk+1 (1) + 14 xk (4) + 14 b3
xk+1 (4) = 14 xk (2) + 14 xk (3) + 14 b4
xk+1 (4) = 14 xk+1 (2) + 14 xk+1 (3) + 14 b
Iterative methods – p. 4/2
Jacobi’s method
k
0
1
2
3
4
···
10
xk (1)
0
0.411
0.617
0.720
0.771
0.822
xk (2)
···
0
0.411
0.617
0.720
0.771
xk (3)
0
0.411
0.617
0.720
0.771
xk (4)
0
0.411
0.617
0.720
0.771
xk
···
···
···
0.822
0.822
0.822
The exact solution is xi = π 2 /12 ≈ 0.822. It appears that xki → xi for
i = 1, 2, 3, 4 when k → ∞.
Iterative methods – p. 5/2
Gauss-Seidel’s method
k
0
1
2
3
4
···
6
xk (1)
0
0.411
0.668
0.784
0.813
0.822
xk (2)
···
0
0.514
0.745
0.803
0.818
xk (3)
0
0.514
0.745
0.803
0.818
xk (4)
0
0.668
0.784
0.813
0.820
xk
···
···
···
0.822
0.822
0.822
Iterative methods – p. 6/2
Why iterative methods?
Iterative methods are useful since:
1. With some effort they work well for large sparse systems where a
direct P LU -factorization requires O(nr ) flops for some r ≥ 2.
2. The bulk of the work in most iterative methods is the matric vector
multiplication Axk . This can normally be carried out without storing
the matrix A thus resulting in considerably savings in storage if A is
sparse.
3. Often we only need a crude approximation to the exact solution x of
Ax = b. For example in Newton’s method for solving a nonlinear
system of equations we need to solve a linear system in each
iteration so x is only an approximation to the underlying problem.
Differential equations present another example where the solution of
the linear system is only an approximation to the exact solution of the
differential equation.
Iterative methods – p. 7/2
A splitting of A

0
a21
0
A =  .. . . . .
.
.
.
=
an,1 ··· an,n−1 0
L

+
+
" a11
a22
...
D
ann
#

0 a12 ···
. .
+  .. ..
+
a1n
..
.
0 an−1,n
0
U


L := tril(A, −1), D := diag(A), and U := triu(A, 1).
Iterative methods – p. 8/2
Jacobi’s method
Assume that D is nonsingular.
Component form:
xk+1 (i) = bi −
i−1
X
j=1
aij xk (j)−
n
X
j=i+1
aij xk (j) /aii for i = 1, 2, . . . , n.
Matrix form:
xk+1 = D
−1
b − Lxk − U xk = D
−1
b − Axk + Dxk ,
xk+1 = xk + D−1 rk , where r k := b − Axk .
Iterative methods – p. 9/2
Gauss-Seidel
Assume that D is nonsingular. Component form:
aii xk+1 (i) = bi −
i−1
X
aij xk+1 (j)−
j=1
n
X
aij xk (j) for i = 1, 2, . . . , n.
j=i+1
Dxk+1 = b − Lxk+1 − U xk ,
(L + D)xk+1 = b − U xk
= b − (L + D + U )xk + (L + D)xk
= (L + D)xk + r k .
xk+1 = xk + (L + D)−1 r k , where r k := b − Axk .
Iterative methods – p. 10/2
Splitting matrix
Given Ax = b, where A ∈ Cn,n is nonsingular.
Choose a nonsingular splitting matrix C ∈ Cn,n
Ax = b ⇐⇒ x = x + C −1 r , where r := b − Ax.
xk+1 = xk + C −1 r k , where r k = b − Axk .
Iterative methods – p. 11/2
Splitting matrices
Method
Richardson
Jacobi
Gauss Seidel
SOR
SSOR
Splitting matrix
I
D
L+D
L + ω −1 D
(D + ωL)D−1 (D + ωU )/(ω(2 − ω))
Iterative methods – p. 12/2
Convergence? Fixed Point Form
Ax = b ⇐⇒ x = x + C −1 (b − Ax) ⇐⇒ x = Bx + C −1 b,
xk+1 = Bxk + C −1 b, where B = I − C −1 A.
If {xk } converges then it must converge to the solution x of
Ax = b. This follows since x = Bx + C −1 b ⇐⇒ Ax = b and
x = lim xk+1 = lim(Bxk + C −1 b) = Bx + C −1 b.
k
k
Iterative methods – p. 13/2
Convergence
For k ≥ 0 we define ǫk := x − xk .
x = Bx + C −1 b
xk+1 = Bxk + C −1 b
Subtraction gives ǫk+1 = Bǫk .
By induction ǫk = B k ǫ0 for k = 0, 1, 2, . . ..
Now ǫk → 0 for all ǫ0 if and only if B k → 0.
Iterative methods – p. 14/2
Spectral radius ρ(A) := maxλ∈σ(A)|λ|
Theorem 2. For any submultiplicative matrix norm k·k on Cn,n and any
A ∈ Cn,n we have ρ(A) ≤ kAk.
Theorem 3. Let A ∈ Cn,n and ǫ > 0 be given. There is a
submultiplicative matrix norm k·k′ on Cn,n such that
ρ(A) ≤ kAk′ ≤ ρ(A) + ǫ.
Theorem 4. For any A ∈ Cn,n we have
lim Ak = 0 ⇐⇒ ρ(A) < 1.
k→∞
Theorem 5. For any submultiplicative matrix norm k·k on Cn,n and any
A ∈ Cn,n we have
lim kAk k1/k = ρ(A).
k→∞
(2)
Iterative methods – p. 15/2
Convergence Theorem
Theorem 6. For given B ∈ Cn,n the iteration xk+1 = Bxk + C −1 b
converges for all starting values x0 ∈ Cn and all right-hand sides
b ∈ Cn if and only if ρ(B) < 1.
Iterative methods – p. 16/2
Rate of convergence xk+1 = Bxk + C −1b
R := − log10
ρ(B) , rate of convergence
limk→∞ kB k k1/k = ρ(B) =⇒ kB k k ≈ ρ(B)k for k large.
It follows that if the matrix norm is subordinate to some
vector norm k·k then kǫk k ≤ kB k kkǫ0 k ≈ ρ(B)k kǫ0 k at
least for k sufficiently large.
To obtain kǫk k/kǫ0 k ≤ 10−s we require ρ(B)k ≤ 10−s or
s
k≥
for s correct digits.
R
(3)
Iterative methods – p. 17/2
Good Splitting matrix
We are looking for a splitting matrix C with the following
properties:
(i) The linear system Cz = r should be solved in O(n)
flops.
(ii) C −1 should be a good approximation to A−1 so that
ρ(B) = ρ(I − C −1 A) is small.
These are conflicting requirements. Indeed, A−1 is often a
full matrix even if A is a band matrix. So at the same time
designing a splitting matrix which both makes Cz = y easy
to solve and C −1 a good approximation to A−1 is not easy.
Iterative methods – p. 18/2
SOR
Gauss-Seidel: xk+1 = xk + (L + D)−1 r k
The Successive Over Relaxation method (SOR) is
obtained by introducing an acceleration parameter ω in
Gauss-Seidel’s method.
xk+1 = xk + (L + ω −1 D)−1 r k
It reduces to the Gauss-Seidel method for ω = 1.
Iterative methods – p. 19/2
Some SOR theory
If SOR converges then 0 < ω < 2
If A ∈ Rn,n is positive definite then SOR converges if and only if
0 < ω < 2.
Iterative methods – p. 20/2
Algorithms
[Iterative Methods]
n × n linear system Ax = b, a
−1
splitting matrix C ∈ Cn,n and a starting vector x. We iterate xk+1 = xk + C r k
with r k = b − Axk until kr k k2 /kr 0 k2 < tol or k > itmax. it is the number of
Given a nonsingular
iterations used.
function [ x , i t ] = s p l i t i t ( A , b , C, x , t o l , m a x it )
n r=norm ( b−A∗x , 2 ) ;
f o r k =1: m a x it
r =b−A∗ x ;
x=x+C\ r ;
i f norm ( r , 2 ) / nr<t o l
i t =k ; re turn ;
end
end
i t = m a x it +1;
Iterative methods – p. 21/2
The classical methods
[x,it]=splitit(A,b,C,x,tol,maxit)
D=diag(diag(A))
Richardson: [x,it]=
splitit(A,b,I,x,tol,maxit);
Jacobi: [x,it]=
splitit(A,b,D,x,tol,maxit);
GS: [x,it]=
splitit(A,b,tril(A),x,tol,maxit);
SOR: [x,it]=
splitit(A,b,tril(A,-1)+D/w),x,tol,maxit);
Iterative methods – p. 22/2
A family of test problems
We can test the methods on the Kronecker sum matrix

 
C1
cI

 

  bI
C1

 

 
.

 +
..
A = C 1 ⊗I+I⊗C 2 = 
 

 

 
C1

 
C1
bI
cI
..
.
bI
..
.
..
bI
cI
bI
bI
cI
.
where C 1 = tridiagm (a, c, a) and C 2 = tridiagm (b, c, b).
Positive definite if c > 0 and c ≥ |a| + |b|.
Iterative methods – p. 23/2
m = 3, n = 9









A=







2c a 0 b 0 0 0 0 0
a 2c a 0 b 0 0 0 0
0 a 2c 0 0 b 0 0 0
b 0 0 2c a 0 b 0 0
0 b 0 a 2c a 0 b 0
0 0 b 0 a 2c 0 0 b
0 0 0 b 0 0 2c a 0
0 0 0 0 b 0 a 2c a
0 0 0 0 0 b 0 a 2c
b = a = −1, c = 2: Poisson matrix

















b = a = 1/9, c = 5/18: Averaging matrix
Iterative methods – p. 24/2
Averaging problem
λjk = 2c + 2a cos (jπh) + 2b cos (kπh), j, k = 1, 2, . . . , m.
a = b = 1/9, c = 5/18
λmax =
5
9
+ 49 cos (πh), λmin =
5
9
− 49 cos (πh)
Richardson: ρ(I − A) = 1 − λmin = 49 (1 + cos(πh))
Jacobi: ρ(I − D−1 A) = 1 − 59 λmin = 45 cos(πh)
Gauss-Seidel: ρ(I − (L + D)−1 A) =
16
25
cos2 (πh)
Iterative methods – p. 25/2
Averaging problem, results
m=10 m=50
ρ(B)
Richardson
31
33
≈ 0.89
Jacobi
71
83
≈ 0.8
Gauss-Seidel
24
21
≈ 0.64
SOR,ω = ω ∗
13
13 ω ∗ − 1 ≈ 0.25
tol = 10−8 , h = 1/(m + 1),
x0 = 0,
ω∗ =
1+
2
1−ρ(B J )2
√
=
5+
10
9+16 sin2 (πh)
√
All methods perform quite well.
Iterative methods – p. 26/2
Poisson problem
λjk = 2c + 2a cos (jπh) + 2b cos (kπh), j, k = 1, 2, . . . , m.
a = b = −1, c = 2
λmax = 4 + 4 cos (πh), λmin = 4 − 4 cos (πh)
Richardson: ρ(I − A) = λmax − 1 = 3 + 4 cos(πh))
Jacobi: ρ(I − D−1 A) = 1 − 41 λmin = cos(πh)
Gauss-Seidel: ρ(I − (L + D)−1 A) = cos2 (πh)
SOR: ω = ωb =
2
1+sin(πh) ,
and ρ(Bωb ) = ωb − 1,
Iterative methods – p. 27/2
Poisson problem results
m=10
m=50
ρ(B)
1/R
8/R, m=50
Richardson
diverges
diverges
3 + 4 cos(πh)
Jacobi
385
8386
cos(πh)
0.47n
9400
Gauss-Seidel
194
4194
cos2 (πh)
4800
168
1−sin(πh)
1+sin(πh)
0.24n
√
0.37 n
SOR, ω = ωb
35
148
The number of iteration for m = 10(n = 100) and m = 50(n = 2500)
are shown in the first two columns
We see that Gauss-Seidel requires about half of the number of
iterations as the Jacobi method
The number of flops is O(n2 ) for Jacobi and Gauss-Seidel and
O(n3/2 ) for SOR.
Iterative methods – p. 28/2
Estimating ρ(B) and R for Jacobi
B = I − D −1 A = I − 14 A
ρ(B) = 1 − 14 λmin (A) = 1 − 2 sin2 (πh/2) = cos(πh)
R = − log10 (cos(πh) = − log(cos(πh)/ log 10
lim −
x→0
log(cos x)
sin x
cos x
1
=
lim
=
lim
=
x→0 2x cos x
x→0 2 cos x − 2x sin x
x2
2
2 log 10
≈ 0.4666n
1/R ≈
π 2 h2
Iterative methods – p. 29/2