Numerical Methods / Scientific Computing
Lecture: Systems of Equations
Dr. Marian Gidea
Department of Mathematical Sciences
Outline
Solving equations
Gaussian Elimination
LU Factorization
Sources of Error
Iterative Methods
Jacobi method
Gauss-Seidel method
Nonlinear Systems of Equations
Systems of equations
General question
Ÿ Solve linear systems of equations Ax b where A is a
n n matrix, b P Rn
Ÿ Methods to solve systems of equations:
Ÿ Gaussian elimination
Ÿ Iterative methods
Ÿ Multivariate Newton’s method
Gaussian elimination
Ÿ Consider a square linear system of equations (n equations
in n unknowns)
a11 x1
a21 x1
a12 x2
a22 x2
an1 x1
an2 x2
a1n xn
a2n xn
b1
b2
ann xn
bn
..
.
,
abbreviated as
Ax
b,
A P R n n ,
b
P Rn .
Ÿ The method known as Gaussian elimination is an efficient
way to solve the system
Gaussian elimination
Ÿ Transform the system to an equivalent one using 3
elementary row operations:
1. Add or subtract a multiple of one equation to another
equation
2. Swap two equations
3. Multiply an equation by a nonzero constant
Ÿ Naive Gaussian elimination uses only row operations of
type 1:
1. Elimination: Put zeros in the elements below the diagonal
to obtain a triangular system.
2. Back substitution: Solve for the variables xn , xn1 , . . . , x1 in
reverse order.
Ÿ Often, these operations are done in tableau form (i.e.
omitting the variables x1 , x2 , ..., xn ).
Gaussian elimination
Exercise Solve this linear system using elementary row
operations of type 1
1x1 2x2 1x3 5
3x1 2x2 4x3 17 .
4x1 4x2 3x3 26
Gaussian elimination
Elimination step: Pseudo-code
Ÿ Consider a general tableau of size n n
aa11
21..
.
a12
a22
an1 an2
a1n
a2n
b1
b2
..
.
ann
bn
Ÿ Goal: Set zeros below the diagonal
Algorithm 1 Elimination step
for j=1:n-1 do
for i=j+1:n do
eliminate entry a(i,j)
end for
end for
.
Gaussian elimination
Elimination step: Pseudo-code
aa11
21..
.
a12
a22
an1 an2
a1n
a2n
b1
b2
..
.
ann
bn
.
Ÿ To eliminate entry a21 , we can do
a21
eq.p2q eq.p2q eq.p1q
a11
Ÿ To eliminate entry ai1 ,
if a11
0.
eq.pi q eq.pi q if a11
0.
ai1
eq.p1q
a11
Ÿ In general, to eliminate entry aij in eq.(i),
eq.pi q eq.pi q aij
eq.pj q
ajj
if ajj
0.
Gaussian elimination
Elimination step: Pseudo-code
Algorithm 2 Elimination step
for j=1:n-1 do
if abs(a(j,j))<eps then error(’zero pivot encountered’)
end if
for i=j+1:n do
™ eliminate entry a(i,j)
mult = a(i,j)/a(j,j);
a(i, j+1:n) = a(i, j+1:n) - mult.*a(j, j+1:n);
b(i) = b(i) - mult*b(j);
end for
end for
Gaussian elimination
Naive Gaussian elimination: Limitations
eq.pi q eq.pi q Ÿ The numbers mij
aa
aij
eq.pj q
ajj
ij
jj
if ajj
0.
are called multipliers.
Ÿ The numbers ajj are called pivots.
Ÿ Note: aij , ajj refer to the revised, not original,
entries.
Ÿ If a pivot is zero the algorithm stops
Theorem
Let Ak be the kth leading principal submatrix of A. If Ak
are nonsingular for k 1, . . . , n, then the pivots are all
nonzero, and Naive Gaussian elimination works.
In particular, Ap An q is itself nonsingular, so the
system Ax b has a unique solution.
Gaussian elimination
Operation count for the elimination step
A simple count of the number of operations
(add/substract/multiply/divide) gives:
Theorem
The elimination step for a system of n equations in n variables
takes
2 3 1 2 7
n
n n
3
2
6
operations.
Ÿ The elimination step is Opn3 q.
Gaussian elimination
Key concept: Complexity
Definition
We say that f px q Opg px qq as x Ñ 8 if there exists a positive
real number M and a real number x0 such that
|f px q| ¤ M|g px q|
for all x
¥ x0 .
Gaussian elimination
Back-substitution
Ÿ After elimination, we obtain an upper-triangular system
a11 x1
a12 x2
a22 x2
b1
b2
a1n xn
a2n xn
..
.
bn
ann xn
Ux
b,
U
PR ,
n n
b
ô
P Rn .
Ÿ Back substitution: Solve for the variables, starting from the last
equation.
xn
abn
nn
..
.
x2
x1
b2 a23 x3 a a2n xn
22
b1 a12 x2 a1n xn
.
a11
Gaussian elimination
Back-substitution: Pseudo-code
Algorithm 3 Back-substitution step
for i=n:-1:1 do
b(i) = b(i) - a(i, i+1:n)*transpose(x(i+1:n));
x(i) = b(i)/a(i,i);
end for
™ dot product
Theorem
The back-substitution step for a system of n equations in n
variables takes n2 operations.
Ÿ Note that elimination is relatively expensive, while
back-substitution is cheap.
Gaussian elimination
Operation count for Naive Gaussian elimination
Combining the operation count for the elimination step and the
back-substitution step, we get
Theorem
Overall, naive Gaussian elimination for a system of n equations
in n variables takes
2 3
n
3
3 2 7
n n
2
6
operations.
Ÿ Naive Gaussian elimination is Opn3 q.
2 3
n
3
LU Factorization
Matrix form
Ÿ We write Gaussian elimination in matrix form. This will
simplify the algorithms and their analysis.
Ÿ Further, we will see that in some cases, it leads to
computational savings.
Ÿ The system
a11 x1
a21 x1
a12 x2
a22 x2
an1 x1
an2 x2
a1n xn
a2n xn
b1
b2
ann xn
bn
..
.
can be written in matrix form
Ax
b,
x
P Rn ,
where A P Rnn is the coefficient matrix, and b
right-hand-side vector.
P Rn is the
LU Factorization
LU factorization: main idea
Ÿ Gaussian elimination
AùU
upper triangular
can be viewed as a process of decomposing A into a
product
A LU
where
Ÿ L is lower triangular, contains all the elimination steps (the
multipliers),
Ÿ U is upper triangular, contains the outcome of the
elimination step.
LU Factorization
Ÿ Exercise Find the LU factorization of the matrix
A
1
3
1
4
Ÿ Exercise Find the LU factorization of the matrix
2
A 2
1
2
4 1
1
1
6
Why is LU factorization equivalent to Gaussian
elimination?
1
1
L pc q c pi, j q
1. Let
..
.
.
1
Then Lpc qA is equivalent to “eq.(i) = eq.(i) - c eq(j)”.
2. The following product helps understand why all steps can
be recorded in just one matrix:
1
c1
1
1
1
c2
1
1
1
1
c3 1
1
c1
c2
1
.
c3 1
LU Factorization
How to use LU factorization to solve a system of equations
Ÿ We want to solve
b
LUx b.
Ax
Ux. Then,
#
Ly b
LUx b ðñ
.
Ux y
Ÿ Introduce an auxiliary vector y
Thus it amounts to solve two triangular systems.
Ÿ Exercise: Solve the system of equations Ax b using LU
factorization
2
A 2
1
2
4 1
1
1 ,
6
9
9
16
LU Factorization
Complexity of the LU Factorization
Combining the operation count for the LU factorization step and
the two back-substitution steps, we get
Theorem
Overall, using LU factorization to solve a system of n equations
in n variables takes
2 3
n
3
3 2 7
n n
2
6
2 3
n
3
operations.
Ÿ This is exactly the same number of operations as
Gaussian elimination!
Ÿ The second back-substitution step Ux y takes the same
number of operations as modifying the right-hand-side
vector b along Gaussian elimination.
LU Factorization
Why use LU factorization
Ÿ If we need to solve a set of k problems
b1
Ax b2
Ax
..
.
Ax
bk ,
then we have
Gaussian elimination
k (elimination + back-substitution)
LU factorization
1 factorization + 2k back-substitu
2
3
3 kn
Ÿ If n is large, then LU factorization is more efficient.
2 3
3n
2kn2
Sources of Error
In Gaussian elimination, there are two potential sources of
error:
1. Ill-conditioned problem Ax
2. “Swamping” (fixable)
b (not easily fixable)
Sources of Error
Swamping
Ÿ Exercise Solve the linear system of equations
1020 x1
x1
1
2x2 4
x2
1. Solve it exactly.
2. Solve it in double precision.
3. Exchange rows and solve it again in double precision.
Ÿ How to fix it: Keep multipliers small!
Sources of Error
Partial pivoting
Ÿ ‘Naive’ Gaussian elimination has two problems:
1. Encountering a zero pivot
2. Swamping
Ÿ For a non-singular matrix, both can be avoided by
exchanging rows of the coefficient matrix A.
Ÿ Idea of partial pivoting: Locate the largest entry of the first
column, and swap its row with the pivot row (first row), etc.
Iterative methods
Introduction
Ÿ Gaussian elimination is a direct method: If A is nonsingular,
it finds the solution in a finite Opn3 q number of steps.
Ÿ In contrast, iterative methods start with an initial guess for
the solution, and refine it at each step, converging towards
the solution (under some hypothesis).
Ÿ There are two well-known iterative methods for systems of
equations:
1. Jacobi method,
2. Gauss-Seidel method (a modification of Jacobi).
Iterative methods
Remember FPI
Example
Solve the nonlinear equation x 3
1. Solve for the unknown:
1.
?
1 x f px q.
Iterate the function f px q starting from an initial guess x0 :
xi 1 f pxi q.
x
2.
x
3
Jacobi method
Example
Solve the system of equations
3u
u
5
2v 5.
v
1. Solve the ith equation for the ith unknown, in order:
5 3 v f pu, v q
5u
v
g pu, v q.
u
2
2. Iterate the two functions f , g starting from an initial guess
pu0, v0q:
5vi ui 1
53ui
vi 1
2
Jacobi method
Convergence of Jacobi method
Definition
A strictly diagonally dominant matrix A paij q is a matrix whose
diagonal entries dominate the rest of entries in the same row:
¸
|aii | ¡ |aij |
@i 1, ..., n.
j i
Theorem
Let the system of equations Ax b. If A is strictly diagonally
dominant, then:
Ÿ A is nonsingular, and thus the system has a unique
solution
Ÿ Given any vector b and any initial guess x0 , the Jacobi
method converges to the solution
Jacobi method
Jacobi as FPI
Ÿ Write A D
L
U, where
Ÿ D= diagonal of A,
Ÿ L= lower triangle of A,
Ÿ U= upper triangle of A.
Note: This use of notation “L" and “U" differs from the use
in the LU factorization.
Ÿ The system to solve is
pD
b ðñ
U qx b ðñ
Dx b pL U qx ðñ
x D 1 pb p L U q x q.
Ax
L
Ÿ Jacobi method is just FPI using the function defined above.
Jacobi method
Jacobi as FPI
Algorithm 4 Jacobi method
initial guess x0
for i 0 : n do
xi 1 D 1 pb pL U qxi q
end for
Jacobi method
Convergence examples
Ÿ Exercises: Does Jacobi method converge for the following
systems?
1.
u
5
2v 5.
u
2v
3u
v
2.
3u
5
v 5.
Jacobi method
Sparse matrices A situation where iterative methods are used
is sparse matrices.
Definition
An n n matrix A is called sparse if “many” of the matrix entries
are zero.
In contrast, a full matrix has “very few” matrix entries equal to
zero.
Ÿ When A is sparse, solving the system Ax b using
Gaussian elimination produces fill-in of the matrix due to
row operations.
Ÿ Jacobi method only uses the non-zero entries!
Gauss-Seidel method
Ÿ Exercise Solve the system of equations
3u
u
5
2v 5.
v
1. Solve the ith equation for the ith unknown, in order:
5 3 v f pu, v q
5u
v
g pu, v q.
u
2
2. Iterate the two functions f , g starting from an initial guess
pu0 , v0 q:
5vi ui 1
53ui 1 .
vi 1
2
Gauss-Seidel method
Ÿ Gauss-Seidel is identical to Jacobi, except that the most
recent updated values are used at each step.
Ÿ Typically, Gauss-Seidel converges faster than Jacobi.
Ÿ As Jacobi, Gauss-Seidel converges to the solution if the
matrix is strictly diagonally dominant.
Ÿ As Jacobi, Gauss-Seidel can be seen as a fixed point
iteration.
Algorithm 5 Gauss-Seidel method
initial guess x0
for i 0 : n do
xi 1 D 1 pb Uxi Lxi 1 q
end for
Nonlinear Systems of Equations
Introduction
Ÿ In Chapter 1 we solved both linear and nonlinear 1-dimensional
equations
f px q 0
using Bisection, FPI, Newton.
Ÿ In Chapter 2 we have solved only linear systems
Ax
b
using Gaussian elimination, LU, Jacobi.
Ÿ Now we show how to solve nonlinear systems of equations (n
equations in n unknowns)
F1 px1 , x2 , . . . , xn q 0
F2 px1 , x2 , . . . , xn q 0
,
..
.
Fn px1 , x2 , . . . , xn q 0
abbreviated as
F px q 0,
F : Rn
Ñ Rn ,
x
P Rn .
Nonlinear Systems of Equations
Multivariate Newton’s Method
Ÿ Recall the usual Newton’s method. The linear approximation to
the function f at x0 P R is given by Taylor’s theorem:
f px q f px0 q
f 1 px0 qpx
x0 q
Oppx
x0 q2 q.
Discarding the remainder, we obtain Newton’s iteration:
xi
1
xi ff1ppxxi qq .
i
Ÿ The multidimensional analogue follows. The linear
approximation to the function F at x0 P Rn is given by
(multivariate) Taylor’s theorem:
F px q F px0 q
DF px0 qpx
x0 q
Oppx
x0 q2 q.
Discarding the remainder, we obtain Newton’s iteration:
xi
1
xi DF pxi q1 F pxi q.
Nonlinear Systems of Equations
Jacobian Matrix of Partial Derivatives
Ÿ For multivariate functions F : Rn Ñ Rn , the analogue of
the derivative is the Jacobian matrix of partial derivatives
BF
BBFx
Bx
DF px q B F1
Bx
B F22
Bx2
B Fn B Fn
Bx1 Bx2
1
1
2
1
..
.
B F1 Bxn
B F2 Bxn BF
Bxn
n
Nonlinear Systems of Equations
Multivariate Newton’s Method
Algorithm 6 Multivariate Newton’s Method
initial vector x0
for i 0 : n do
xi 1 xi DF pxi q1 F pxi q
end for
Ÿ In numerical linear algebra we avoid computing the inverse
matrix DF pxi q1 .
Ÿ Instead, solve an equivalent linear system. Let
DF pxi q1 F pxi q s,
so that
DF pxi qs
F px i q .
Ÿ We can solve for s using Gaussian elimination at the cost
Opn3 {3q.
Nonlinear Systems of Equations
Example:
Ÿ Solve the nonlinear system: F px, y q pF1 px, y q, F2 px, y qq
F1 px, y q x 2 y
F2 px, y q yx
x cospπx q 0
ey
x 1 0
Ÿ Note: px, y q p1, 0q is an exact solution
Ÿ The Jacobian is
DF
2x
cospπx q πx sinpπx q
y x 2
x
1
e y
Ÿ We can use the symbolic computation library SymPy to
compute the Jacobian