Solving Linear Systems Ax=b
Bruce Maggs
Maverick Woo
6/28/2016
Linear Systems
Ax = b
known
unknown
n by n matrix
known
n by 1 vector
2
Linear Systems in The Real World
3
Circuit Voltage Problem
Given a resistive network and the net current flow at each
terminal, find the voltage at each node.
2A
A node with an external
connection is a terminal.
Otherwise it’s a junction.
Conductance is the
reciprocal of resistance.
Its unit is siemens (S).
2A
3S
2S
1S
1S
4A
4
Kirchoff’s Law of Current
At each node,
net current flow = 0.
Consider v1. We have
2A
2 + 3(v2 ¡ v1 ) + (v3 ¡ v1 ) = 0;
which after regrouping yields
2A
3S
v1
2S
v2
3(v1 ¡ v2 ) + (v1 ¡ v3 ) = 2:
v4
1S
1S
I=VC
v3
4A
5
Summing Up
2A
Here A represents
conductances, v
represents voltages at
nodes, and Av is the net
current flow at each node.
2A
3S
v1
2S
v2
v4
1S
1S
v3
4A
6
Solving
2A
Here A represents
conductances, v
represents voltages at
nodes, and Av is the net
current flow at each node.
2A
3S
2V
2S
2V
3V
1S
1S
0V
4A
7
Sparse Matrix
An n by n matrix is sparse when there are O(n) nonzeros.
A reasonably-sized power grid has way more junctions and
each junction has only a couple of neighbors.
3
v1
2
v2
v4
1
1
v3
8
Sparse Matrix Representation
A simple scheme
An array of columns, where
each column Aj is a linked-list of tuples (i, x).
9
Modeling a Physical System
Let G(x, y) and g(x, y) be continuous functions defined in
R and S respectively, where R and S are respectively the
region and the boundary of the unit square (as in the figure).
1
R
0
S
1
10
Modeling a Physical System
We seek a function u(x, y) that satisfies
2u
2u
@
@
Poisson’s equation in R
+
=
G(x;
y)
@x2
@y 2
and the boundary condition in S. u(x; y) = g(x; y)
1
R
0
S
1
Example: g(x, y) specifies
fixed temperature at each point
on the boundary, G(x, y)=0,
u(x, y) is the temperature at
each interior point.
11
Example
Temperature on three
sides of a square plate is 0
and on one side is .
http://www.math.uiuc.edu/~jlebl/286-dirich.pdf
12
Discretization
Imagine a uniform grid with a small spacing h.
h
1
0
1
13
Finite Difference Method
Replace the partial derivatives by difference quotients
@ 2 u=@x2 s [u(x + h; y) + u(x ¡ h; y) ¡ 2u(x; y)]=h2
@ 2 u=@y 2 s [u(x; y + h) + u(x; y ¡ h) ¡ 2u(x; y)]=h2
Poisson’s equation now becomes
4u(x; y) ¡ u(x + h; y) ¡ u(x ¡ h; y)
¡ u(x; y + h) ¡ u(x; y ¡ h) = ¡h2 G(x; y)
Exercise:
Derive the 5-pt diff. eqt.
from first principle (limit).
14
For each point in R
4u(x; y) ¡ u(x + h; y) ¡ u(x ¡ h; y)
¡ u(x; y + h) ¡ u(x; y ¡ h) = ¡h2 G(x; y)
1
The total number of equations is (h ¡ 1)2
Now write them in the matrix form, we’ve got one BIG linear
system to solve!
4u(x; y) ¡ u(x + h; y) ¡ u(x ¡ h; y)
¡ u(x; y + h) ¡ u(x; y ¡ h) = ¡h2 G(x; y)
15
An Example
4u(x; y) ¡ u(x + h; y) ¡ u(x ¡ h; y)
¡ u(x; y + h) ¡ u(x; y ¡ h) = ¡h2 G(x; y)
Consider u3,1, we have
4
4u(3; 1) ¡ u(4; 1) ¡ u(2; 1)
¡ u(3; 2) ¡ u(3; 0) = ¡G(3; 1)
u32
u21
u31
which can be rearranged as
4u(3; 1) ¡ u(2; 1) ¡ u(3; 2)
= ¡G(3; 1) + u(4; 1) + u(3; 0)
u41
u30
0
4
16
An Example
4u(x; y) ¡ u(x + h; y) ¡ u(x ¡ h; y)
¡ u(x; y + h) ¡ u(x; y ¡ h) = ¡h2 G(x; y)
Each row and column can have a maximum of 5 nonzeros.
17
How to Solve?
Ax = b
Find A-1
Direct Method: Gaussian Elimination
Iterative Method: Guess x repeatedly until we guess a
solution
18
Inverse Of Sparse Matrices
…are not necessarily sparse!
A
A-1
19
Direct Methods
20
Solution by LU Decomposition
21
Transform A to U through Gaussian Elimination
22
L Specified by Multiples of pivots
23
Pivoting Creates Fill
24
25
Reordering rows and columns doesn’t change solution
26
… but might reduce fill
27
Graph Model of Matrix
28
Graph View of Partitioning
29
Disclaimer
30
Min-Degree Heuristic
Alternatively, eliminate vertex with smallest fill.
31
Separators
32
Nested Dissection
33
Nested Dissection Example
34
Analysis of Nested Dissection
35
Results Concerning Fill
Finding an elimination order with minimal fill is hopeless
Garey and Johnson-GT46, Yannakakis SIAM JADM 1981
O(log n) Approximation
Sudipto Guha, FOCS 2000
Nested Graph Dissection and Approximation Algorithms
(n log n) lower bound on fill
(Maverick still has not dug up the paper…)
36
Iterative Methods
37
Algorithms Apply to Laplacians
Given a positively weighted, undirected graph G = (V, E),
we can represent it as a Laplacian matrix.
3
v1
2
v2
v4
1
1
v3
38
Laplacians
Laplacians have many interesting properties, such as
Diagonals ¸ 0 denotes total incident weights
Off-diagonals < 0 denotes individual edge weights
Row sum = 0, column sum = 0
Symmetric
v
3
2
2
v1
v4
1
1
v3
39
Laplacian???
Earlier I showed you a system that is not quite Laplacian.
Boundary points don’t have 4 neighbors
40
Making It Laplacian
We add a dummy variable and force it to zero.
41
The Basic Idea
Start off with a guess x(0).
Use x(i) to compute x(i+1) until convergence.
We hope
the process converges in a small number of iterations
each iteration is efficient
42
The RF Method [Richardson, 1910]
Residual
x(i+1) = x(i) ¡ (Ax(i) ¡ b)
Domain
A
Range
x
Test
x(i+1)
x(i)
Correct
b
Ax(i)
Units are all wrong! x(i) is voltage, residual is current!
43
Why should it converge at all?
x(i+1) = x(i) ¡ (Ax(i) ¡ b)
Domain
Range
x
Test
x(i+1)
x(i)
Correct
b
Ax(i)
44
It only converges when…
x(i+1) = x(i) ¡ (Ax(i) ¡ b)
Theorem
A first-order stationary iterative method
x(i+1) = Gx(i) + k
converges iff
½(G) < 1.
(A) is the maximum
absolute eigenvalue of G
45
Fate?
Ax = b
Once we are given the system, we do not have any control on
A and b.
How do we guarantee even convergence?
46
Preconditioning
B -1Ax = B -1b
Instead of dealing with A and b,
we now deal with B-1A and B-1b.
The word “preconditioning”
originated with Turing in 1948,
but his idea was slightly different.
47
Preconditioned RF
x(i+1) = x(i) ¡ (B -1Ax(i) ¡ B -1b)
Since we may precompute B-1b by solving By = b,
each iteration is dominated by computing B-1Ax(i), which is
a multiplication step Ax(i) and
a direct-solve step Bz = Ax(i).
Hence a preconditioned iterative method is in fact a hybrid.
48
The Art of Preconditioning
We have a lot of flexibility in choosing B.
Solving Bz = Ax(i) must be fast
B should approximate A well for a low iteration count
I
Trivial
B
A
What’s
the point?
49
Classics
x(i+1) = x(i) ¡ (B -1Ax(i) ¡ B -1b)
Jacobi
Let D be the diagonal sub-matrix of A.
Pick B = D.
Gauss-Seidel
Let L be the lower triangular part of A w/ zero diagonals
Pick B = L + D.
Conjugate gradient
Move in an orthogonal direction in each iteration
50