BEKG 2452 Numerical Methods
Chapter 3: Solution of Linear
Systems
- Gauss Elimination
- LU Decomposition
- Gauss-Seidel Method
BEKG 2452 Numerical Methods
Solution of Linear Systems, 𝐴𝐱 = 𝐛
LU
Decomposition
Gauss
Seidel
Gauss
Elimination
- A=LU
- Apply forward
and backward
substitution
- Strictly diagonal
dominant matrix
- Run iteration
-Row Reduction
- GE
with/without
partial pivoting
May be Exact or Approximated solution
1
BEKG 2452 Numerical Methods
3.1 Gauss Elimination
3.1.1 Elementary Row Operation
Aim to reduce a matrix into an upper triangular
matrix.
I.e.
𝑿 𝑋 𝑋
0 𝑿 𝑋
0 0 𝑿
X can be any different
values
Pivot
BEKG 2452 Numerical Methods
3.1.1 Elementary Row Operation
1) Interchange any two rows
- Interchange two rows 𝑖 and 𝑗: 𝑟+ ↔ 𝑟E.g.
0 1 12 ↔13 2 3
2 3
0 1
2
BEKG 2452 Numerical Methods
3.1.1 Elementary Row Operation
2) Row replacement operation:
- Replace row 𝑗 by 𝑚𝑟+ + 𝑟- :
𝑚𝑟+ + 𝑟- → 𝑟3
= −3
1
−2
𝑚=−
=2
1
𝑚=−
𝟏 0 3
3 5 −2
−2 1 −3
A
: 13 <1>
B
𝑚=−
1
5
𝟏
0
0
0
𝟓
0
:;12 <13
=12 <1>
𝟏
0
0
0
𝟓
1
3
−11
3
3
−11
26/5
BEKG 2452 Numerical Methods
Example:
Reduce the matrix to an upper triangular matrix.
−1 2 −5
𝐴 = 2 −1 6
1
1
3
3
BEKG 2452 Numerical Methods
Solution:
−1 2 −5
2 −1 6
1
1
3
:13 <1>
=12 <13
12 <1>
−1 2 −5
0 3 −4
0 3 −2
−1 2 −5
0 3 −4
0 0 2
BEKG 2452 Numerical Methods
Example 3.1:
Reduce the following matrix to an upper triangular
matrix.
2 2 3
𝐴 = −2 1 5
1 3 4
4
BEKG 2452 Numerical Methods
3.1.2 Gauss Elimination Algorithm
Basic idea:
Solve the equivalent system which is in a simpler form
compared to the original linear system.
Flow:
𝑎AA
𝑎=A
𝑎;A
𝑎AA 𝑥A + 𝑎A= 𝑥= + 𝑎A; 𝑥; = 𝑏A
𝑎=A 𝑥A + 𝑎== 𝑥= + 𝑎=; 𝑥; = 𝑏=
𝑎;A 𝑥A + 𝑎;= 𝑥= + 𝑎;; 𝑥; = 𝑏;
𝑎A;
𝑎=;
𝑎;;
𝑥A
𝑏A
𝑥= = 𝑏=
𝑥;
𝑏;
Matrix form
Linear System
Backward
Substitution
𝑎A=
𝑎==
𝑎;=
Row
Reduction
𝑎AA
𝑎=A
𝑎;A
𝑎A=
𝑎==
𝑎;=
𝑎A;
𝑎=;
𝑎;;
𝑏A
𝑏=
𝑏;
Augmented Matrix
BEKG 2452 Numerical Methods
Example:
Solve the following linear system by using
Gauss Elimination.
2𝑥A + 3𝑥= − 𝑥; = 2
4𝑥A + 4𝑥= − 𝑥; = −1
−2𝑥A − 3𝑥= + 4𝑥; = 1
Solution:
Transform the linear system into matrix form:
2
3 −1 𝑥A
2
4
4 −1 𝑥= = −1
−2 −3 4 𝑥;
1
5
BEKG 2452 Numerical Methods
Solution:
Augmented Matrix:
2
3 −1 2
4
4 −1 −1
−2 −3 4
1
Row reduction:
2
3 −1 2
4
4 −1 −1
−2 −3 4
1
:=12 <13
12 <1>
2 3 −1 2
0 −2 1 −5
0 0
3
3
Backward substitution:
𝑥A
−3
∴ 𝑥= = 3
𝑥;
1
BEKG 2452 Numerical Methods
Example 3.2:
Solve the following linear system by using Gauss
Elimination.
𝑥A + 2𝑥= + 3𝑥; = 9
2𝑥A − 𝑥= + 𝑥; = 8
3𝑥A
− 𝑥; = 3
6
BEKG 2452 Numerical Methods
3.1.3 Gauss Elimination with Partial Pivoting
Basic idea:
Avoid to have pivot element which is near to zero (it may
cause to division by zero after normalization)
Flow:
𝑎AA 𝑥A + 𝑎A= 𝑥= + 𝑎A; 𝑥; = 𝑏A
𝑎=A 𝑥A + 𝑎== 𝑥= + 𝑎=; 𝑥; = 𝑏=
𝑎;A 𝑥A + 𝑎;= 𝑥= + 𝑎;; 𝑥; = 𝑏;
Linear System
𝑎AA
𝑎=A
𝑎;A
𝑎A=
𝑎==
𝑎;=
𝑎A;
𝑎=;
𝑎;;
𝑏A
𝑏=
𝑏;
Augmented Matrix
Row Reduction
- Pivot element is the largest absolute
value among entries underneath it
- Use only 𝑟+ ↔ 𝑟- and 𝑚𝑟+ + 𝑟- → 𝑟-
Backward
Substitution
BEKG 2452 Numerical Methods
Example:
Solve the following linear system by using Gauss
Elimination with partial pivoting.
0.3𝑥A − 0.2𝑥= + 10𝑥; = 71.4
3𝑥A − 0.1𝑥= − 0.2𝑥; = 7.85
0.1𝑥A + 7𝑥= − 0.3𝑥; = −19.3
Solution:
Augmented matrix:
0.3 −0.2 10
71.4
3 −0.1 −0.2 7.85
0.1
7
−0.3 −19.3
7
BEKG 2452 Numerical Methods
Solution (cont.):
Row reduction:
Max 0.3 , 3 , 0.1 = 3
12 ↔13
:O.A12 <13
:O.O;;;12 <1>
0.3 −0.2 10
71.4
3 −0.1 −0.2 7.85
0.1
7
−0.3 −19.3
𝟑 −0.1 −0.2 7.85
0.3 −0.2 10
71.4
0.1
7
−0.3 −19.3
3 −0.1
−0.2
7.85
0 −0.19
10.02
70.615
0 7.0033 −0.2933 −19.5617
Max −0.19 , 7.0033
= 7.0033
BEKG 2452 Numerical Methods
Solution (cont.):
13 ↔1>
3
−0.1
0 𝟕. 𝟎𝟎𝟑𝟑
0 −0.19
O.O=SA13 <1>
−0.2
7.85
−0.2933 −19.5617
10.02
70.615
3 −0.1
−0.2
7.85
0 7.0033 −0.2933 −19.5617
0
0
10.1921 71.1451
Backward substitution:
𝑥A
2.9987
∴ 𝑥= = −2.5007
𝑥;
6.9804
8
BEKG 2452 Numerical Methods
Example 3.3:
Solve the following linear system by using
Gauss Elimination with partial pivoting.
2𝑥A + 3𝑥= − 𝑥; = 2
4𝑥A + 4𝑥= − 𝑥; = −1
−2𝑥A − 3𝑥= + 4𝑥; = 1
Solution:
Augmented matrix:
2
3 −1 2
4
4 −1 −1
−2 −3 4
1
BEKG 2452 Numerical Methods
3.2 LU Decomposition
Given a linear system,
𝐴𝐱 = 𝐛
Decompose 𝐴 = 𝐿𝑈
𝐿𝑈 𝐱 = 𝐛
𝐿 𝑈𝐱 = 𝐛
Let 𝑈𝐱 = 𝐲, ∴ 𝐿𝐲 = 𝐛
solve 𝐿𝐲 = 𝐛 for 𝐲
(forward substitution)
solve 𝑈𝐱 = 𝐲 for 𝐱
(backward substitution)
9
BEKG 2452 Numerical Methods
3.2 LU Decomposition
An 𝑛×𝑛 nonsingular matrix can be decomposed into
a lower triangular matrix (L) and an upper triangular
matrix (U) as follows:
𝑎AA
𝑎=A
𝑎;A
𝑎A=
𝑎==
𝑎;=
𝑎A;
1
𝑎=; = 𝑙=A
𝑎;;
𝑙;A
𝑙;=
𝑨
=
𝑳
0
1
0 𝑢AA
0 0
1 0
𝑢A=
𝑢==
0
𝑢A;
𝑢=;
𝑢;;
𝑼
BEKG 2452 Numerical Methods
3.2.1 Steps of LU Decomposition
Step 1: Construct L and U
1
𝑙=A
𝑙;A
0
1
𝑙;=
0 𝑢AA
0 0
1 0
Multiply L and U:
𝑢AA
𝑢A=
𝑙=A 𝑢AA
𝑙=A 𝑢A= + 𝑢==
𝑙;A 𝑢AA 𝑙;A 𝑢A= + 𝑙;= 𝑢==
LU
𝑢A=
𝑢==
0
= A
𝑢A;
𝑎AA
𝑢=; = 𝑎=A
𝑎;A
𝑢;;
𝑎A=
𝑎==
𝑎;=
𝑎A;
𝑎=;
𝑎;;
𝑢A;
𝑎AA
𝑙=A 𝑢A; + 𝑢=;
= 𝑎=A
𝑎;A
𝑙;A 𝑢A; + 𝑙;= 𝑢=; + 𝑢;;
𝑎A=
𝑎==
𝑎;=
𝑎A;
𝑎=;
𝑎;;
Compare each of the entries to obtain the values for u and l.
i.e. 𝑢AA = 𝑎AA
10
BEKG 2452 Numerical Methods
3.2.1 Steps of LU Decomposition
Step 2: Solve 𝐿𝐲 = 𝐛 for 𝐲 by using forward substitution
𝐴𝐱 = 𝐛
When 𝐴 = 𝐿𝑈,
𝐿(𝑈𝐱) = 𝐛
Let 𝑈𝐱 = 𝐲,
𝐿𝐲 = 𝐛
1
𝑙=A
𝑙;A
0
1
𝑙;=
𝑏A
0 𝑦A
0 𝑦= = 𝑏=
1 𝑦;
𝑏;
𝑦A = 𝑏A
𝑙=A 𝑦A + 𝑦= = 𝑏=
𝑙;A 𝑦A + 𝑙;= 𝑦= + 𝑦; = 𝑏;
Forward
substitution
BEKG 2452 Numerical Methods
3.2.1 Steps of LU Decomposition
Step 3: Solve 𝑈𝐱 = 𝐲 for 𝐱 by using backward substitution
𝑈𝐱 = 𝐲
𝑢AA
0
0
𝑢A=
𝑢==
0
𝑢A;
𝑢=;
𝑢;;
𝑦A
𝑥A
𝑥= = 𝑦=
𝑦;
𝑥;
𝑢;; 𝑥; = 𝑦;
𝑢== 𝑥= + 𝑢=; 𝑥; = 𝑦=
𝑢AA 𝑥A + 𝑢A= 𝑥= + 𝑢A; 𝑥; = 𝑦A
Backward
substitution
11
BEKG 2452 Numerical Methods
Example:
Solve the following linear system by using
LU decomposition.
2𝑥A + 3𝑥= − 𝑥; = 2
4𝑥A + 4𝑥= − 𝑥; = −1
−2𝑥A − 3𝑥= + 4𝑥; = 1
Solution:
Transform the linear system into matrix form:
2
3 −1 𝑥A
2
4
4 −1 𝑥= = −1
−2 −3 4 𝑥;
1
BEKG 2452 Numerical Methods
Solution (cont.):
Step 1: Construct L and U
1
𝑙=A
𝑙;A
0
1
𝑙;=
Multiply L and U:
𝑢AA
𝑙=A 𝑢AA
𝑙;A 𝑢AA
0 𝑢AA
0 0
1 0
𝑢A=
𝑙=A 𝑢A= + 𝑢==
𝑙;A 𝑢A= + 𝑙;= 𝑢==
LU = A
𝑢A=
𝑢==
0
𝑢A;
2
𝑢=; = 4
𝑢;;
−2
3
4
−3
−1
−1
4
𝑢A;
2
𝑙=A 𝑢A; + 𝑢=;
= 4
𝑙;A 𝑢A; + 𝑙;= 𝑢=; + 𝑢;;
−2
3
4
−3
−1
−1
4
Compare each of the entries to obtain the values for u and l.
∴
𝐿=
1 0 0
2 1 0
−1 0 1
2 3 −1
𝑈 = 0 −2 1
0 0
3
12
BEKG 2452 Numerical Methods
Solution (cont.):
Step 2: Solve 𝐿𝐲 = 𝐛 for 𝐲 by using forward substitution
𝐴𝐱 = 𝐛,
𝐿𝑈 𝐱 = 𝐛 ⇒
when 𝐴 = 𝐿𝑈,
Let 𝑈𝐱 = 𝐲, we have
1
2
−1
By forward substitution:
𝑦A = 2,
𝐿 𝑈𝐱 = 𝐛.
𝐿𝐲 = 𝐛
2
0 0 𝑦A
𝑦
1 0 = = −1
1
0 1 𝑦;
2𝑦A + 𝑦= = −1,
2(2) + 𝑦= = −1,
𝑦= = −5
−𝑦A + 𝑦; = 1,
−(2) + 𝑦; = 1,
𝑦; = 3
𝑦A
2
∴ 𝑦= = −5
𝑦;
3
BEKG 2452 Numerical Methods
Solution (cont.):
Step 3: Solve 𝑈𝐱 = 𝐲 for 𝐱 by using backward substitution
𝑈𝐱 = 𝐲
2 3 −1 𝑥A
2
0 −2 1 𝑥= = −5
0 0
3 𝑥;
3
By Backward substitution:
3𝑥; = 3, −2𝑥= + 𝑥; = −5,
𝑥; = 1 −2𝑥= + (1) = −5,
𝑥= = 3
2𝑥A + 3𝑥= − 𝑥; = 2
2𝑥A + 3(3) − (1) = 2
𝑥A = −3
𝑥A
−3
𝑥
∴ = = 3
𝑥;
1
13
BEKG 2452 Numerical Methods
Alternate way to decompose A into LU
Row reduction:
Example 1:
𝟏
𝐴= 2
3
2
−1
0
3
1
−1
:=12 <13
:;12 <1>
1
0
0
÷1
2
−𝟓
−6
3
−5
−10
e
: 13 <1>
B
1
0
0
2
−5
0
÷ −4
÷ −5
1
∴𝐿= 2
3
3
−5 = 𝑈
−𝟒
0
1
6/5
0
0
1
BEKG 2452 Numerical Methods
Alternate way to decompose A into LU
Row reduction:
Example 2:
1
3
𝐴=
−1
−3
:h13 <1i
1
0
0
0
−2
−9
2
−6
−2
−3
0
0
−2
0
4
26
−2
6
2
−4
−3
−9
7
2
−3
0
4
−7
1
3
∴𝐿=
−1
−3
:;12 <13
12 <1>
;12 <1i
=1> <1i
0
1
0
4
1
0
0
0
−2
−3
0
−12
1
0
0
0
−2
−3
0
0
0
0
1
−2
0
0
0
1
−2
6
2
0
−2
6
2
20
−3
0
4
−7
−3
0
=𝑈
4
1
14
BEKG 2452 Numerical Methods
Example 3.4:
Solve the following linear system by using LU
decomposition.
𝑥A + 2𝑥= + 3𝑥; = 9
2𝑥A − 𝑥= + 𝑥; = 8
3𝑥A
− 𝑥; = 3
BEKG 2452 Numerical Methods
Example 3.5:
Solve the following linear system by using LU
decomposition.
1.012𝑥A − 2.132𝑥= + 3.104𝑥; = 1.984
−2.132𝑥A + 4.096𝑥= − 7.013𝑥; = −5.049
3.104𝑥A − 7.013𝑥= + 0.014𝑥; = −3.895
15
BEKG 2452 Numerical Methods
3.3 Gauss Seidel
Strictly Diagonal Dominant Matrix
- The magnitude of diagonal entry in a row is
larger than the sum of the magnitudes of all the
other entries in that row:
𝑎jj > ∑m
-nj 𝑎j- for 𝑘 = 1,2, … , 𝑁)
6 > 2 + 3
e.g.
6
−1
3
2
−9
0
3
4
−7
−9 > −1 + 4
−7 > 3 + 0
BEKG 2452 Numerical Methods
3.3 Gauss Seidel
Basic Idea:
Solve unknown variable of a linear system iteratively by using
previously computed results as soon as they are available.
Flow:
𝑎AA 𝑥A + 𝑎A= 𝑥= + 𝑎A; 𝑥; = 𝑏A
𝑎=A 𝑥A + 𝑎== 𝑥= + 𝑎=; 𝑥; = 𝑏=
𝑎;A 𝑥A + 𝑎;= 𝑥= + 𝑎;; 𝑥; = 𝑏;
Strictly diagonal dominant
matrix arrangement
Linear System
Stop iteration when
Form the iterative sequence
𝐱 (j) − 𝐱 (j:A)
=
(j)
max 𝑥+
Av+vw
−
r
(j:A)
𝑥+
(j<A)
<𝜀
(known as infinity norm)
𝑥A
(j<A)
, 𝑥=
(j<A)
, 𝑥;
(O)
(O)
(O)
Initial guess: (𝑥A , 𝑥= , 𝑥; )
16
BEKG 2452 Numerical Methods
3.3.1 Gauss Seidel Iterative Sequence
From 𝐴𝐱 = 𝐛 (A is a strictly diagonal dominant matrix),
𝑎AA 𝑥A + 𝑎A= 𝑥= + 𝑎A; 𝑥; = 𝑏A
𝑎=A 𝑥A + 𝑎== 𝑥= + 𝑎=; 𝑥; = 𝑏=
𝑎;A 𝑥A + 𝑎;= 𝑥= + 𝑎;; 𝑥; = 𝑏;
1
(j)
j
𝑏 − 𝑎A= 𝑥= − 𝑎A; 𝑥;
𝑎AA A
1
(j)
j<A
=
𝑏= − 𝑎=A 𝑥A
− 𝑎=; 𝑥;
𝑎==
(j<A)
𝑥A
(j<A)
𝑥=
(j<A)
𝑥;
=
=
1
(j<A)
j<A
𝑏; − 𝑎;A 𝑥A
− 𝑎;= 𝑥=
𝑎;;
BEKG 2452 Numerical Methods
Example:
Solve the following linear system by using
Gauss Seidel.
12𝑥A + 3𝑥= − 𝑥; = 15
2𝑥A − 𝑥= + 10𝑥; = 30
𝑥A + 8𝑥= + 𝑥; = 20
Start the initial guess with 𝐱 (O) = 𝟎 and stop the
iteration when 𝐱 (j) − 𝐱 (j:A)
< 0.001
r
17
BEKG 2452 Numerical Methods
Solution:
Check for strictly diagonal dominant:
12𝑥A + 3𝑥= − 𝑥; = 15
2𝑥A − 𝑥= + 10𝑥; = 30
𝑥A + 8𝑥= + 𝑥; = 20
12𝑥A + 3𝑥= − 𝑥; = 15
𝑥A + 8𝑥= + 𝑥; = 20
2𝑥A − 𝑥= + 10𝑥; = 30
𝑟= ↔ 𝑟;
Iterative Sequence:
1
(j)
(j)
15 − 3𝑥= + 𝑥;
12
1
(j<A)
(j)
j<A
𝑥=
= 20 − 𝑥A
− 𝑥;
8
1
(j<A)
(j<A)
(j<A)
𝑥;
=
30 − 2𝑥A
+ 𝑥=
10
(j<A)
𝑥A
=
BEKG 2452 Numerical Methods
Solution:
(j)
max 𝑥+
Av+vw
(j)
(j)
(j)
𝐱 (j) − 𝐱 (j:A)
𝑘
𝑥A
𝑥𝟐
𝑥𝟑
0
0
0
0
1
1.2500
2.3438
2.9844
2.9844
2
0.9128
2.0129
3.0187
0.0343
3
0.9983
1.9979
3.0001
0.0855
4
1.0005
1.9999
2.9999
0.0022
5
1.0000
2.0000
3.0000
0.0005
(j:A)
− 𝑥+
r
𝑥A
1
∴ 𝑥= = 2
𝑥;
3
18
BEKG 2452 Numerical Methods
Example 3.6:
Solve the following linear system by using Gauss Seidel.
1)
0.3𝑥A − 0.2𝑥= + 10𝑥; = 71.4
3𝑥A − 0.1𝑥= − 0.2𝑥; = 7.85
0.1𝑥A + 7𝑥= − 0.3𝑥; = −19.3
2)
−𝑥A + 𝑥= + 7𝑥; = −6
4𝑥A − 𝑥= − 𝑥; = 3
−2𝑥A + 6𝑥= + 𝑥; = 9
Start the initial guess with 𝐱 (O) = 𝟎 and stop the iteration when
𝐱 (j) − 𝐱 (j:A) r < 0.0005
19