Chapter 5 Simultaneous Linear
Equations
• Many engineering and scientific problems
can be formulated in terms of systems of
simultaneous linear equations.
• In a system consisting of only a few
equations, a solution can be found
analytically using the standard methods
from algebra, such as substitution.
5-1
• Example: Electrical Circuit Analysis
I1  I 2  I 3  0
Kirchhoff’s law
V1  R1 I 1  R 3 I 3
I: current
V1  V 2  R1 I 1  R 2 I 2
V: voltage
R: resistance
5-2
Assume R1=2, R2=4, R3=5, V1=6, and V2=2.
We get the following system of three linear
simultaneous equations.
I1  I 2  I 3  0
2 I1
 5I3  6
2 I1  4 I 2
4
The solution to these three equations produces the
current flows in the network.
5-3
General Form for a System of Equations
a 11 X 1  a 12 X 2  ...  a 1 n X n  C 1
a 21 X 1  a 22 X 2  ...  a 2 n X n  C 2

a n 1 X 1  a n 2 X 2  ...  a nn X n  C n
aij : known coefficient
Xj : unknown variable
Ci : known contant
• Assume # of unknowns = # of equations
• Assume the equations are linearly independent; that is, any
one equation is not a linear combination of any of the other
equations.
5-4
The linear system can be written in a matrix-vector
form:
 a 11

a 21

 

 a n1
a 12

a 22



an2

a1n   X 1 
 C1 




a2n
X2
C2

  

   
  




a nn   X n 
C n 
AX  C
Combining A and C, it can be expressed as:
 a 11

a
 21
 

 a n1
a 12

a 22



an2

a1n C 1 

a2n C2


 

a nn C n 
5-5
2 X1  3X 2  1
 4X1  X 2  5
can be expressed as:
 2

 4
3
1
1

5
5-6
Solution of Two Equations
a 11 X 1  a 12 X
2
 C1
a 21 X 1  a 22 X
2
 C2
It can be solved by substitution.
X1 
C 1  a 12 X
2
a 11
With substitution, we get
X1 
X
2

a 22 C 1  a 12 C 2
a 11 a 22  a 21 a 12
a 11 C 2  a 21 C 1
a 11 a 22  a 21 a 12
5-7
Classification of Systems of Equations
•
Systems that have unique solutions
2 X 1  3 X 2 6
2 X 1  9 X 2  12
5-8
•
Systems without solutions (parallel lines)
3X1  9X 2  5
X1  3X 2  6
5-9
•
Systems with an infinite number of solutions
(same line)
2X1  3X 2  4
4X1  6X 2  8
5-10
• A system that has a solution, but has illconditioned parameters.
2 X 1  2 .2 X 2  5 .7
2 X 1  2 X 2  5 .5
5-11
Permissible Operations
2 X1  3X 2  1
 4X1  X 2  5
• Rule 1: The solution is not changed if the
order of the equations is changed.
 4X1  X 2  5
2 X1  3X 2  1
5-12
• Rule 2: Any one of the equations can be multiplied
or divided by a nonzero constant without changing
the solution.
4X1  6X 2  2
 4X1  X 2  5
• Rule3: The solution is not changed if two
equations are added together and the resulting
equation replaces either of the two original
equations.
2 X1  3X 2  1
 2X1  4X 2  6
5-13
Gaussian Elimination
• Gaussian elimination procedure:
Phase 1: forward pass
Phase 2: back substitution
• The the forward pass is to apply the three
permissible operations to transform the original
matrix to an upper-triangular matrix:
1

0

0




0
d 12
d 13

d 1n
1
d 23

d 2n
0
1

d 3n




0
0

1
e1 

e2

e3 



en 

5-14
Written in terms of individual equations:
X 1  d 12 X 2  d 13 X 3    d 1 , n  2 X n  2  d 1 , n  1 X n  1  d 1 , n X n  e1
X 2  d 23 X 3    d 2 , n  2 X n  2  d 2 , n  1 X n  1  d 2 , n X n  e 2
X 3    d 3 ,n  2 X n  2  d 3 , n 1 X n 1  d 3 ,n X n  e 3

X n  2  d n  2 ,n 1 X n 1  d n  2 ,n X n  e n  2
X n 1  d n 1, n X n  e n 1
X n  en
• Back substitution:
X n  en
X n 1  e n 1  d n 1 , n X n

5-15
Example: Gaussian Elimination Procedure
2 X 1  3 X 2  2 X 3  X 4  2
2 X1  5X 2  3X 3  X 4  7
 2X1  X
2
 3X 3  2 X 4  1
 5X1  2 X 2  X 3  3X 4  8
Represented in the matrix form:






2
3
2
1
2
5
3
1
2
1
3
2
5
2
1
3
2 

7

1 

8 
5-16
• As the step 1 in the forward pass, we will convert the
element a11 (a11 is called the pivot for row 1) to 1 and
eliminate, that is set to zero, all the other elements in the
first column.
1

0

0

0
original
 2

2

 2

 5
d' 12
d' 13
d' 14
d' 22
d' 23
d' 24
d' 32
d' 33
d' 34
d' 42
d' 43
d' 44
e' 1 

e' 2

e' 3 

e' 4 
resultant
operation
matrix
3
2
1
5
3
1
1
3
2
2
1
3
2

7

1

8
R '1 
R1
2
R ' 2  R 2  2 R '1
R ' 3  R 3  2 R '1
R ' 4  R 4  5 R '1

1

0

0

0

3
2
2
4
19
2
matrix
1

1
1
2
2
1
3
1
6
2


9

 1

3

1
5-17
• Step 2:
original

1

0

0

0

3
matrix
1
2
2
4
19

1
1
2
2
1
3
1
6
2
resultant
operation
2

1

9

 1

3

R '1  R 1
R '2 
R2
2
R '3  R 3  4 R '2
R '4  R 4 
19
2
R '2

1


0

0
0

3
matrix
1

2
1
2

1
0

0
1
1
2
3
5
7
9
4


9

2
 19 
159 

4 
1
• Step 3:
original

1


0

0
0

3
1
2
1

1
2

0
0
resultant
matrix

1
2
3
5
4
1
7
9
operation


9

2
 19 
159 

4 
1
R '1  R 1
R '2  R 2
R '3 
R3
3
R '4  R 4

1


0

0

5

 R '3 0

4

3
matrix
1

2
1
1
2

1
1
2
0
0

1
0

7
3
143
12


9

2
19 

3 
572 


12  5-18
1
• Step 4:
original

1


0

0


0

3
matrix
1

2
1
1
2

1
1
2
0
0

1
0

7
3
143
12



9

2

19 

3 
572 


12 
1
original
operation

1


0

0

 0
R '1  R 1
R '2  R 2
R '3  R 3
R '4 
R4
 143 / 12
3
matrix
1

2
1
1
2

1
1
2
0
0
1
0

7
3
1


9

2
19 

3 
4 
1
It represents:
X1 
3
2
X2  X3 
X2 
1
2
1
2
X 4  1
X3  X4 
X3 
7
3
9
2
X4  
X4  4
19
3
5-19
• Step 1 of backward substitution:

1


0

0

 0


2
2
1
9
1 
1

2
2
7
19 
0
1 

3
3
0
0
1
4 
3
1 
1
1
R '1  R1 
R '4
2
R '2  R 2  R '4
R '3  R 3 
7 R '4
3
R '4  R 4

1


0

0
0

3


1

2
3
4 
1 0
1
2
1 
1
0
0
2
1 0
0
0
1
• Step 2:

1


0

0
0

3
1 0
2
1 
0
0
1
0
2
1 0
0
1


1

2
3
4 
1
R '1  R 1  R ' 3
R '2  R 2 
R '3
2
R '3  R 3
R '4  R 4

1

0

0
0

3
0
0
2
1
0
0
0
1
0
0
0
1


2

3
4 
4
5-20
• Step 3:

1

0

0
0

3
0
0
2
1
0
0
0
1
0
0
0
1

4

2

3
4 
R '1  R1 
3 R '2
R '2  R 2
R '3  R 3
R '4  R 4
2
1

0

0

0
0
0
0
1
0
0
0
1
0
0
0
1
1

2

3

4
• The solution is:
X1 = 1, X2 = 2, X3 = 3, X4 = 4
5-21
Gauss-Jordan Elimination
• The Gaussian elimination procedure requires a forward
pass to transform the coefficient matrix into an uppertriangle form.
• In Gauss-Jordan elimination, all coefficients in a column
except for the pivot element are eliminated.
• In Gauss-Jordan elimination, the solution is obtained
directly after the forward pass; there is no back substitution
phase.
• The Gauss-Jordan method needs more computational effort
than Gaussian elimination.
5-22
Example: Gauss-Jordan Elimination
• Step1:
original
 2

2

 2

 5
matrix
3
2
1
5
3
1
1
3
2
2
1
3
2

7

1

8
• Step 2:

1

0

0

0

3
2
2
4
19
2
1
1
1
6
resultant
operation

1
2
2
3
1
2

1

9

 1

3

R '1 
R '2 
R '3 
R '4 

1

2
0
R 2  2 R '1 
R 3  2 R '1  0

R 4  5 R '1 0

3
R1
R '1  R 1 
R '2 
3 R '2
2
R2
2
R '3  R 3  4 R ' 2
R '4  R 4 
19
2
R '2
matrix
1
2
2
4
19

1
1
3
1
6
2
0
1


0
0

1
4
1
2
3
5
4


9

 1

3

1
2
2
2

1


0

0
0

1
2
1
7
9
31 
4 
9

2
 19 
159 

4 

5-23
• Step 3:

1


0

0
0

0
1


0
0

1
2
4
1
1
2
3
5
7
9
4
31 

4 
9

2
 19 
159 

4 
R '1  R 1 
R '3
R '2  R 2 
1
R '3 
4
2
R '3
R3
3
R '4  R 4 
5
4
R '3

1


0

0


0

0
0
1
0
0
1
0
0

112 
4 
4

3
19 

3 
572 


12 
31


12
1

6
7

3
143
12
• Step 4:

1


0

0


0

0
1
0

0
0
1
0
0

31
12
1

6
7

3
143
12
112 

4 
4

3
19 

3 
572 


12 
R '1  R1 
31
12
R '2  R 2 
R '3 
R '4 
R3
3

1
6
7
3
R '4
R '4
R '4
R4
 143 12
1

0

0

0
0
0
1
0
0
1
0
0
1

0 2

0 3

1 4
5-24
0
Accumulated Round-off Errors
• Problems with round-off and truncation are most
likely to occur when the coefficients in the
equations differ by several orders of magnitude.
• Round-off problems can be reduced by
rearranging the equations such that the largest
coefficient in each equation is placed on the
principal diagonal of the matrix.
• The equations should be ordered such that the
equation having the largest pivot is reduced first,
followed by the equation having the next largest,
and so on.
5-25
Programming Gaussian Elimination
•
Forward pass:
1.
Loop over each row i, making each row i in turn the
pivot row.
2.
Normalize the elements of the pivot row (row i) by
dividing each element in the row by aii as follows:
a ij 
Ci 
a ij
for j  ( i  1 ) , ( i  2 ) , , ( n  1 )
a ii
Ci
C ii
a ii  1
5-26
3.
Loop over rows( i + 1) to n below the pivot row and
reduce the elements in each row as follows:
a kj  a kj  a ki a ij
for j  i,...,n
C k  C k  a ki C i
for k  ( i  1) , ,n
• Back substitution
1. For the last row n:
X n  Cn
2. For rows (n-1) through 1,
n
X i  Ci 
a
ij
X
j
for j  ( i  1),  , n and i  ( n  1),  ,1
j  i 1
5-27
LU Decomposition
• A matrix A can be decomposed into L and
U, where L is a lower-triangular matrix and
U is a upper-triangular matrix.
LU=A
 l11

l 21

 l 31



 l n 1
0
0

l 22
0

l 32
l 33




ln 2
ln 3

0

0

0



l nn 
1

0

0



 0
u12
u13

1
u 23

0
1




0
0

u1 n   a 11
 
a
u2n
  21
u 3 n    a 31
 


 
1   a n 1
a 12
a 13

a 22
a 23

a 32
a 33




an2
an3

a1n 

a2n

a 3n 



a nn 
5-28
• L and U can be determined as follows:
li1  a i1
u1 j 
for i  1, 2 ,  , n
a1 j
for
j  2 , 3,  , n
l11
j 1
l ij   a ij 
l
ik
u kj
for j  2 , 3 ,  , n
k 1
and i  j, j  1 , , n
j 1
a ji 
u ji  
l
k 1
jk
u ki
for j  2 , 3 ,  , n  1
l jj
and i  j  1, j  2 , , n
5-29
AX=C
LUX=C, we have LE=C and UX=E
• To calculate LE=C (forward substitution)
e1 
C1
l11
j 1
Ci 
ei  
l
ij
ej
j 1
for i  2 , 3 ,  , n
l ii
• To calculate UX=E (back substitution)
X n  en
n
X i  ei 
u
ij
X
j
for i  n  1, n  2 ,  ,1
j  i 1
5-30
Relation between LU Decomposition
and Gaussian Elimination
• In the LU decomposition, matrix U is
equivalent to the upper triangular matrix
obtained in the forward pass in Gaussian
elimination.
• The calculation of UX=E is equivalent to
the back substitution in Gaussian
elimination.
5-31
Example: LU Decompostion
X 1  3 X 2  2 X 3  15
2 X 1  4 X 2  3 X 3  22
3 X 1  4 X 2  7 X 3  39
1

A  2

 3
3
4
4
2

3

7 
Applying LU decomposition:
l11  a 11  1
l 21  a 21  2
l 31  a 31  3
5-32
u 12 
a 12
3
l11
1
u 13 
a 13
2


l11
 3
 2
1
2 1
l 22  a 22 
l
2k
u k 2  4  2 ( 3)   2
3k
u k 3  4  3( 3 )   5
k 1
2 1
l 32  a 32 
l
k 1
2 1
a 23 
u 23 
l
k 1
l 22
2k
uk 3

3  2(2)
2
 0 .5
3 1
l 33

a 33 
l
k 1
3k
u k 3  7  ( 3 )( 2 )  (  5 )( 0 . 5 )  3 . 5
5-33
Thus, the L and U matrices are
1

L  2

 3


0

3 .5 
0
1

U  0

 0
0
 2
5
3
1
0


0 .5

1 
2
Forward substitution:
e1 
C1

l11
15
 15
1
2 1
C2 
l
2 j
ej
j 1
e2 

l 22
22  2 (15 )
2
 4
3 1
C3 
e3 
l
j 1
l 33
3 j
ej

39  3 (15 )  (  5 )( 4 )
3 .5
 4
5-34
Back substitution:
X 3  e3  4
3
X 2  e2 
u
2 j
X
j
 4  0 .5 ( 4 )  2
j  2 1
3
X 1  e1 
u
1j
X
j
 15  3 ( 2 )  ( 2 )( 4 )  1
j 1 1
5-35
Cholesky Decomposition for
Symmetric Matrices
• A symmetric matrix A:
A  A
T
• Cholesky decomposition for a symmetric matrix A
A  LL
T
5-36
Matrix L can be computed as follows:
l11 
a 11
i 1
l ii 
a ii 

2
l ik
for i  2 , 3 ,  , n
k 1
i 1
a ij 
l ij 
l
k 1
ik
l jk
for i  2 , 3 ,  , i- 1, and j  i
l jj
5-37
Example: Cholesky Decomposition
1

A  2


3
3 

10

22 

2
8
10
We can obtain the following:
l11 
l 21 
a 11 
a 21

l11
2
1  1
 2
1
2 1
l 22 
a 22 

l2 k 
2
82
2
 2
k 1
5-38
a 31
l 31 
l11

3
3
1
2 1
a 32 
l 32 
l
l
3k 2 k
k 1

10  ( 3 )( 2 )
l11
2
2
3 1
l 33 
a 33 

l3 k 
2
22  3  ( 2 )
2
2
 3
k 1
5-39
Therefore, the L matrix is
1

L  2

 3
0
2
2
0

0

3 
The validity can be verified as
A  LL
T
1

 2

 3
0
2
2
0  1
 
0  0
 
3   0
2
2
0
3
1


2  2


3 
 3
2
8
10
3 

10

22 
5-40
Iterative Methods
• Elimination methods like the Gaussian elimination
procedure are often called direct equation-solving methods.
An iterative method is a trial-and error procedure.
• In iterative methods, we can assume a solution, that is, a
set of estimates for the unknowns, and successively refine
our estimate of the solution through some set of rules.
• A major advantage of iterative methods is that they can be
used to solve nonlinear simultaneous equations, a task that
is not possible using direct elimination methods.
5-41
Jacobi Iteration
• Each equation is rearranged to produce an
expression for a single unknown.
X1 
X2 
C 1  a 12 X
2
 a 13 X 3    a 1 n X
n
a 11
C 2  a 21 X 1  a 23 X 3    a 2 n X
n
a 22

Xn 
C n  a n1 X 1  a n 2 X
2
   a n 1,n X
n 1
a nn
5-42
Example: Jacobi Iteration
3X1  X
2
 2X3  9
 X1  4X
2
 3 X 3  8
X1  X
2
 4X3 1
Rearrange each equation as follows:
X1 
X
2

X3 
9 X
2
 2X3
3
9  X1  3X 3
4
1 X1  X
2
4
5-43
Assume an initial estimate for the solution: X1=X2=X3=1.
First iteration:
X1 
X
X
2
3

9  (  1)  2 (1)

10
3
3
 8  1  3 (1)
 1
4

111
4

1
4
Second iteration:
9  (  1)  2 (

3

10
10
 3(
1
)
4

4
 
47
48
 (  1)
3
7
2
3
4
1
X

3
8
2
)
4
X1 
X
1
The solution is shown in
the next table.
5
6
5-44
Table: Example of Jacobi Iteration
|△ X1|
X2
|△
X2|
X1
0
1
1
3.333
2.333
-1.000
2.000
0.250
0.750
2
3.500
0.167
-0.979
0.021
-0.833
1.803
3
2.771
0.729
-1.750
0.771
-0.870
0.036
4
3.003
0.233
-1.960
0.210
-0.880
0.010
5
3.006
0.063
-1.960
0.210
-0.991
0.111
6
2.996
0.090
-1.976
0.067
-0.994
0.003
7
2.996
0.020
-2.001
0.025
-0.988
0.006
8
3.008
0.012
-1.992
0.009
-0.999
0.011
9
2.998
0.011
-1.997
0.005
-1.000
0.001
10
2.999
0.001
-2.001
0.003
-0.999
0.001
11
3.001
0.002
-1.999
0.001
-1.000
0.001
12
3.000
0.001
-2.000
0.000
-1.000
0.001
1
X3
|△ X3|
Iteration
1
5-45
Gauss-Seidel Iteration
• In the Jacobi iteration procedure, we always
complete a full iteration cycle over all the
equations before updating our solution estimates.
In the Gauss-Seidel iteration procedure, we update
each unknown as soon as a new estimate of that
unknown is computed.
• Example: Gauss-Seidel Iteration
3X1  X 2  2X 3  9
 X 1  4 X 2  3 X 3  8
X1  X 2  4X3  1
5-46
Assume an initial solution estimate of X1=X2=X3=1.
First iteration:
X1 
X
2

9  (  1)  2 (1)
 3 . 333
3
 8  3 . 333  3 (1)
  0 . 417
4
X3 
1  3 . 333  (  0 . 417 )
  0 . 688
4
Second iteration:
X1 
X
2

X3 
9  (  0 . 417 )  2 (  0 . 068 )
 2 . 680
3
 8  2 . 680  3 (  0 . 068 )
  1 . 845
4
1  2 . 680  (  1 . 845 )
  0 . 882
4
5-47
Table: Example of Gauss-Seidel Iteration
|△ X1|
X2
|△ X2|
X1
0
1
1
3.333
2.333
-0.417
1.417
-0.688
1.688
2
2.680
0.348
-1.845
1.428
-0.882
0.194
3
3.027
0.346
-1.904
0.059
-0.0983
0.101
4
2.979
0.048
-1.992
0.088
-0.993
0.010
5
3.002
0.023
-1.994
0.002
-0.999
0.006
6
2.999
0.003
-2.000
0.006
-1.000
0.001
7
3.000
0.001
-2.000
0.000
-1.000
0.000
8
3.000
0.000
-2.000
0.000
-1.000
0.000
1
X3
|△ X3|
Iteration
1
Jocobi iteration method requires 13 iterations to reach
the accuracy of 3 decimal places. Gauss-Seidel iteration
method needs 7 iterations.
5-48
Convergence Considerations of the
Iterative Methods
• Both the Jacobi and Gauss-Seidel iterative methods
may diverge.
• Interchange the order of equations in the above
example, and solve it by the Gauss-Seidel method:
X1  8  4X 2  3X 3
X 2  9  3X1  2X 3
X3 
1 X1  X 2
4
5-49
Table: Divergence of Gauss-Seidel Iteration
|△ X1|
X2
|△ X2|
X1
0
1
1
9
8
-16
17
-6
7
2
-38
47
111
127
37.5
43.5
3
33935
377.5
-943.5
1045.5
-318.25
355.75
1
X3
|△ X3|
Iteration
1
• The solution in the above table would not
converge. That is, we can not get a solution.
• The divergence of the iterative calculation does
not imply there is no solution, since it is a
permissible operation to interchange the order in a
set of equations.
5-50
Convergence and Divergence of
Gauss-Seidel Iteration
a11 X 1  a12 X 2  C 1
a 21 X 1  a 22 X 2  C 2
If we solve both equations individually for X1
we get
X1 
X1 
C1

a12
a 11
a 11
C2
a 22
a 21

a 21
X2
X2
5-51
5-52
• It will converge when the absolute value of the
slope of f1 is less than the absolute value the slope
of f2. Thus the equations should be arranged such
that X1 is expressed in terms of X2 with the
following conditions:
a 12
a 11

a 22
a 21
• For a system with more than 2 equations, we
should select the equation with the largest
coefficient as the first equation, the equation with
the largest coefficient in the remaining equations
as the second equation, and so on.
5-53
Cramer’s rule
 a 11

a 21

A 
 

 a n1
a 12

a 22



an2

a1n 

a2n

 

a nn 
 C1 


C2

C  
  


C
 n
Cramer’s rule for obtaining Xi :
Xi 
Ai
A
An example of |Ai| :
A2
 a 11

a 21


 

 a n1
C1
a 13

C2
a 23




Cn
an3

a1n 

a2n

 

a nn 
5-54
Example: Cramer’s Rule
3X1  X
2
 2X3  9
 X1  4X
2
 3 X 3  8
X1  X
2
 4X3 1
A 
A1 
1
4
3
3
1
 2   46
1
1
4
8
4
3
9
1
 2   138
1
1
4
5-55
A2 
1
8
3
3
9
1
1
 2  92
4
A3 
1
4
3
1
1
1
8
9  46
1
The solution is
X1 
X2 
X3 
A1
 138

A
 46
A2
92

A
A3
A

 46
46
 46
3
 2
 1
5-56
Matrix Inversion
• The inverse of a matrix P is defined by the
following equation
1
P PI
.
in which I is the identity or unit matrix and both P
and I are square matrices.
• The values of P-1 can be computed by solving a
set of n2 simultaneous equations.
5-57
Let the elements of P-1 and P be denoted as qij and pij.
 q11

q 21

 q 31
q12
q 22
q 32
q13 

q 23 

q 33 
 p11

p 21

 p 31
p12
p 22
p 32
p13   1
 
p 23  0
 
p 33   0
0
1
0
0

0

1 
p 11 q11
 p 21 q12
 p 31 q13
1
p 12 q11
 p 22 q12
 p 32 q13
0
p 13 q11
 p 23 q12
 p 33 q13
0
p 11 q 21
 p 21 q 22
 p 31 q 23
0
p 12 q 22
 p 22 q 22
 p 32 q 23
1
p 13 q 23
 p 23 q 22
 p 33 q 23
0
p 11 q 31
 p 21 q 32
p 12 q 31
 p 22 q 32
p 13 q 31
 p 23 q 32
 p 31 q 33  0
 p 32 q 33  0
 p 33 q 33
1
5-58