3. SYSTEMS OF EQUATIONS
Many engineering and scientific problems can be modelled in terms of systems of
simultaneous linear and non-linear equations.
3.1. LINEAR SYSTEMS OF EQUATIONS
Consider the electric circuit below with two separate networks, loop acdb and loop
aefb :
V2
V1
c
a
I1
R1
e
R3
I3
b
R2
d
I2
f
Applying Kirchhoff’s first law at junction c yields the equation
I1  I 2  I 3  0,
applying Kirchhoff’s second law at network loop acdb yields the equation
V1  R1I1  R3 I 3
and applying Kirchhoff’s second law at network loop aefb yields the equation
V1  V2  R1I1  R2 I 2 .
Assuming that R1  2, R2  4, R3  5, V1  6 and V2  2, we get the following system
of three linear simultaneous equations:
I1  I 2  I 3  0
2 I1
 5I3  6
2 I1  4 I 2  4.
The solution to these three equations produces the current flow in the network.
In general,
a11 x1  a12 x2  ...  a1n xn  b1
a21 x1  a22 x2  ...  a2 n xn  b2
(3.1)
an1 x1  an 2 x2  ...  ann xn  bn
We will use two methods to solve linear systems of equations. These are direct and
iterative methods.
3.1.1. Direct Methods
Notice that system (3.1) can be written as
AX  B,
(3.2)
where
 a11

a
A   21


 an1
a12
a22
an 2
a1n 
 b1 

 
a2 n 
b
, B 2

 

 
ann 
 bn 
 x1 
 
x
and X   2  .
 
 
 xn 
Then we can write the augmented matrix for system (3.1) as
 a11

a
A | B   21


 an1
a12
a22
an 2
a1n b1 

a2 n b2 


ann bn 
and apply row operations to it to find the solution. This procedure is called Gaussian
elimination with backward substitution. The three row operations we use are:
1) Multiply (or divide) the sides of an equation by a non-zero number
2) Add a multiple of one equation to another equation in the system
3) Interchange two equations in the system.
When row operations are applied, the augmented matrix is left in the form with the
following characteristics:
(a) The first no-zero number (starting from the left) in any row is called the pivot
(b) Any row (if any) consisting of zero entries appears at the bottom of the matrix
(c) If two successive rows do not consist of entirely zeros, then the pivot in the lower row
occurs farther to the right than the first pivot in the highest row.
Example 3.1.1
Represent the linear system
x1  x2  2 x3  x4  8
2 x1  2 x2  3x3  3x4  20
x1  x2  x3  2
x1  x2  4 x3  3x4  4
as an augmented matrix and use Gaussian elimination with backward substitution to find its
solution.
Solution:
The augmented matrix is
 1 1

 2 2
1 1

 1 1
2 1 8 

3 3 20 
1 0 2 

4 3 4 
Applying row operations, we get
 1 1

 2 2
1 1

 1 1
 1 1 2 1 8 
 1 1 2 1 8 
2 1 8 



 r2  r2  2r1 
0 0 1 1 4  r2  r3  0 2 1 1 6 
3 3 20 

r3  r3  r1
 0 2 1 1 6  r4  12 r4  0 0 1 1 4 
1 0 2 



 r4  r4  r1 
4 3 4 
 0 0 2 4 12 
0 0 1 2 6 
 1 1 2 1 8 


0 2 1 1 6 

r4  r3  r4
 0 0 1 1 4 


0 0 0 1 2 
 x4  2   x3  x4  4
 x3  2
2 x2  x3  x4  6
 x2  3
x1  x2  2 x3  x4  8  x1  7
 X   x1 , x2 , x3 , x4    7,3, 2, 2  .
t
t
Pivoting Strategies
Note that applying Gaussian elimination method to the augmented matrix involves multiplying
a number
m jk 
To one row to “reduce” it. For example, m21 
a jk ( k )
akk
(k )
a21(1)
a11
(1)

a21
so that r2  r2  ( aa1121 )r1. If akk ( k ) is
a11
small in magnitude, any round-off error in the numerator will be increased. The next example
illustrates how a wrong choice of the pivot can affect the final answer.
Example 3.1.2
The exact solution to the system
0.003000 x1  59.14 x2  59.17
5.291x1  6.130 x2  46.78
using four-digit rounding-off is x1  10.00 and x2  1.000. Apply Gaussian elimination
method to find the solution.
Solution:
m21 
a21(1)
5.291

 1764.
(1)
a11
0.003000
 0.003000
 0.003000 59.14 59.17 
59.14
59.17 

 r2  r2  m21r1 

0
104300 104400 
6.130 46.78 

 5.291
104300
 1.001
104400
59.17  59.14(1.001)
 x1 
 10.00.
0.003000
 x2 
Comparing with the exact solution, we notice serious errors.
Example 3.1.2 shows that serious errors may arise when pivoting element akk ( k ) is small
relative to the entries aij ( k ) , for k  i  n and k  j  n. To avoid this problem, we use scaled
partial pivoting. The first step in this procedure is to define a scale factor si for each row as
si 1Max
 j  n aij .
Assuming that si  0, the appropriate row interchange to place zeros in the first column is
determined by choosing the least integer p with
a p1
sp
 ak1 
 1Max

 j n
 sk 
and then interchange row 1 and row p. In a similar manner, before eliminating the variable
xi , we select the smallest integer p  i with
a pi
sp
 aki 
1Max

 j n 
 sk 
and perform row interchange. The scale factors s1 , s2, ,..., sn are computed only once at the
start of the procedure. They are row dependent, so they must be interchanged when row
interchanges are performed.
Example 3.1.3
Apply Gaussian elimination with scaled partial pivoting to find the solution to the system
2.11x1  4.21x2  0.921x3  2.01
4.01x1  10.2 x2  1.12 x3  3.09
1.09 x1  0.987 x2  0.832 x3  4.21
using three-digit rounding.
Solution:
s1  Max{| 2.11|,| 4.21|,| 0.921|}  4.21
s2  Max{| 4.01|,|10.2 |,| 1.12 |}  10.2
s3  Max{|1.09 |,| 0.987 |,| 0.832 |}  1.09

| a11 | 2.11

 0.501
s1
4.21
| a21 | 4.01

 0.393
s2
10.2
| a31 |
 1.09  1.
s3
 | a11 | | a21 | | a31 | 
| a31 |
Since Max 
,
,
, we interchange row 1
  Max{0.501, 0.393,1}  1 
s2
s3 
s3
 s1
and row 3. Thus, we have the augmented matrix
 1.09 0.987 0.832 4.21 
4.01
2.11


 4.01 10.2 1.12 3.09  m21  1.09  3.68, m31  1.09  1.94
 2.11 4.21 0.921 2.01 


1.09 0.987 0.832 4.21 
 1.09 0.987 0.832 4.21 

 r2  r2  m21r1 

6.57 4.18 18.6 
 4.01 10.2 1.12 3.09  r  r  m r  0
3
31 1 
 2.11 4.21 0.921 2.01  3
6.21 0.693 6.16 
 0


| a22 | 6.57

 0.644
s2
10.2
| a32 | 6.12

 1.45
s1
4.21
 | a22 | | a32 | 
| a32 |
Since Max 
,
, we interchange row 2 and
  Max{0.644, 1.45}  1.45 
s2
s1 
s1

row 3.
1.09 0.987 0.832 4.21 
6.57


6.12 0.693 6.16  m32 
 1.07
 0

6.12
 0
6.57 4.18 18.6 

1.09 0.987 0.832 4.21 
1.09 0.987 0.832 4.21 




6.12 0.693 6.16  r3  r3  m32 r2  0
6.12 0.693 6.16 
 0
 0
 0
6.57 4.18 18.6 
0
4.92 25.2 


 4.92 x3  25.2
 6.12 x2  0.693 x3  6.16
 x3  5.12
 x2  0.426
1.09 x1  0.987 x2  0.832 x3  4.21  x1  0.431
 X   x1 , x2 , x3    0.431, 0.426, 5.12 
t
t
Exercise: Use scaled partial pivoting to solve the system in Example 3.1.2.
3.1.2. Iterative Methods
Iterative or indirect methods use trial-and-error procedure to solve large systems of linear
equations with high percentage of zero entries. An iterative method to solve the (n  n) linear
system (3.2) starts with an initial approximation X (0) to the solution X and generate a
sequence of vectors  X ( k ) 

k 0
that converges to X . We will discuss two of the common
methods, the Jacobi and Gauss-Siedel iteration techniques. To determine the difference
between the approximations and the exact solution, we will define the norm of a vector which
will help us to determine the distance between n  dimensional column vectors.
Definition 3.1.1
A vector norm on
(i)
n
X  0 X 
is a function . , from
n
to
with the following properties:
n
(ii) X  0  X  0
(iii)  X |  | X ,  
and X 
(iv) X  Y  X  Y , X , Y 
n
n
Definition 3.1.2
The l2 and l norms for the vector X   x1 , x2 ,..., xn  are defined by
t
 n

X 2    xi 2 
 i 1 
1
2
and
X   1Max
i  n xi .
Example 3.1.4
The l2 and l norms for the vector X   1,1,5, 20  are
t
1
 n

X 2    xi 2   (1) 2  12  52  (20) 2  427
 i 1 
2
X   1Max
i  n xi  Max | 1|, |1|, | 5 |, | 20 |  20
Definition 3.1.3
If X   x1 , x2 ,..., xn  and Y   y1 , y2 ,..., yn  are vectors in
t
t
n
, then l2 and l distances
between X and Y are defined by
 n

X  Y 2    ( xi  yi ) 2 
 i 1

1
2
and
X  Y   1Max
i  n xi  yi .
Jacobi’s Method
The Jacobi’s iterative method is obtained by solving the i th equation in the system (3.2) for
xi to obtain
n
  aij x j  bi
xi   
  , for i  1, 2,..., n
aii  aii
j 1 
j i
provided that aii  0. For each k  1, generate the components xi ( k 1) of X ( k ) from the
components of X ( k 1) by
xi
(k )


1  n
( k 1)

  aij x j   bi  , for i  1, 2,..., n.
aii  j 1
 j i

Example 3.1.5
The following system
10 x1  x2  2 x3
6
 x1  11x2  x3  3x4  25
2 x1  x2  10 x3  x4  11
3x2  x3  8 x4  15
has the unique solution X  (1, 2, 1,1)t . Use Jacobi’s iterative method with four-decimal
places rounding-off to find approximations X ( k ) to X with X (0)  (0, 0, 0, 0)t until
X ( k )  X ( k 1)
X
(k )


 103.
Solution:
We first check that aii  0 for i  1, 2,3, 4. a11  10, a22  11, a33  10, a44  8. Thus,
1
( x2 ( k 1)  2 x3( k 1)  6)
10
1
x2 ( k )  ( x1( k 1)  x3( k 1)  3x4 ( k 1)  25)
11
1
x3( k )  (2 x1( k 1)  x2 ( k 1)  x4 ( k 1)  11)
10
1
x4 ( k )  (3x2 ( k 1)  x3( k 1)  15).
8
x1( k ) 
Starting with X (0)  (0, 0, 0, 0)t , we obtain
1
3
( x2 (0)  2 x3(0)  6)   0.6000
10
5
1
25
x2 (1)  ( x1(0)  x3(0)  3x4 (0)  25) 
 2.2727
11
11
1
11
x3(1)  (2 x1(0)  x2 (0)  x4(0)  11) 
 1.1000
10
10
1
15
x4 (1)  (3x2 (0)  x3(0)  15)   1.8750.
8
8
x1(1) 
Thus, X (1)  (0.6000, 2.2727,  1.1000,1.8750)t

X (1)  X (0)
X

(1)


Max{| 0.6000 |,| 2.2727 |,| 1.1000 |,|1.8750 |}
 1  103.
Max{| 0.6000 |,| 2.2727 |,| 1.1000 |,|1.8750 |}
Using X (1) , we get
1
1
( x2 (1)  2 x3(1)  6)  (2.2727  2(1.1000)  6)  1.0473
10
10
1
1
x2 (2)  ( x1(1)  x3(1)  3x4 (1)  25)  (0.6000  1.1000  3(1.8750)  25)  1.7159
11
11
1
1
x3(2)  (2 x1(1)  x2 (1)  x4 (1)  11)  (2(0.6000)  2.2727  1.8750)  0.8052
10
10
1
1
x4 (2)  (3x2 (1)  x3(1)  15)  (3(2.2727)  1.1000  15)  0.8852.
8
8
x1(2) 
Thus, X (2)  (1.0473,1.7159,  0.8052, 0.8852)t


X (2)  X (1)
X
(2)



Max{|1.0473  0.6000 |, |1.7159  2.2727 |, | 0.8052  1.1000 |, | 0.8852  1.8750 |}
Max{|1.0473 |,|1.7159 |,| 0.8052 |,| 0.8852 |}
0.9898
 0.5768  103.
1.7159
Continuing in this manner, we get the following values:
0
k
(k )
0
x1
1
2
0.6000 1.0473
6
7
10
8
3
5
9
4
0.9326 1.0152 0.9890 1.0032 0.9981 1.0006 0.9997 1.0001
x2 ( k ) 0
x3( k ) 0
2.2727 1.7159
2.0530 1.9537 2.0114 1.9922
2.0023 1.9987 2.0004 1.9998
1.1000 0.8052 1.0493 0.9681 1.0103 0.9945 1.0020 0.9990 1.0004 0.9998
x4 ( k ) 0 1.8750


0.8852 1.1309 0.9739 1.0214 0.9944 1.0036 0.9989 1.0006 0.9998
X (10)  X (9)
X

(10)


Max{|1.0001  0.9997 |, |1.9998  2.0004 |, | 0.9998  1.0004 |, | 0.9998  1.0006 |}
Max{|1.0001|,|1.9998 |,| 0.9998 |,| 0.9998 |}
0.0008
 0.0004  103.
1.9998
 we end at k  10
Gauss-Seidel Method
The Gauss-Seidel method is an improvement of the Jacobi’s iterative method where we use the
first approximation in the first iteration to approximate other values. For example, in Example
3.1.5, we can use the first approximation x1(1)  0.6000 to approximate x2 (1) and so on. Thus,
xi ( k ) 
i 1
n

1 
(k )
b

a
x

aij x j ( k 1)   , for i  1, 2,..., n.



 i  ij j

aii 
j 1
j i 1

Example 3.1.6
Solve the system in Example 3.1.5 using Gauss-Seidel method.
Solution:
For X (0)  (0, 0, 0, 0)t , we got x1(1)  0.6000. We now use this value to approximate x2 (1) .
1 (1)
1
( x1  x3(0)  3x4 (0)  25)   0.6000  0  3(0)  25   2.3273
11
11
1
1
x3(1)  (2 x1(1)  x2 (1)  x4 (0)  11)   2(0.6000)  2.3273  0  11  0.9873
10
10
1
1
x4 (1)  (3x2 (1)  x3(1)  15)   3(2.3273)  0.9873  15   0.8789.
8
8
x2 (1) 
Exercise: Show that five iterations are required to achieve the required stopping criterion.
3.2. NON-LINEAR SYSTEMS
We consider a system of non-linear equations of the form
f1 ( x1 , x2 ,..., xn )  0
f 2 ( x1 , x2 ,..., xn )  0
f n ( x1 , x2 ,..., xn )  0.
The number of equations should be equal to the number of variables.
3.2.1. Fixed Point Method for Non-linear systems
Theorem 3.2.1
Let D   X  ( x1 , x2 ,..., xn )t : ai  xi  bi , i  1, 2,..., n be some collection of constants ai and
bi . Suppose G( X )   g1 ( x1 , x2 ,..., xn ), g2 ( x1 , x2 ,..., xn ),..., g n ( x1, x2 ,..., xn )  is a continuous
t
function from D 
n
into
n
with the property that G ( X )  D whenever X  D. Then, G
has a fixed point in D.
Moreover, suppose that each g i has continuous partial derivatives and a constant M  1 exists
with
  gi ( X )  M
 , whenever X  D
x j
n
for each j  1, 2,..., n. Then the sequence  X ( k ) 

k 0
defined by an arbitrarily selected X (0) in
D and generated by
X ( k )  G( X ( k 1) ), k  1
converges to the unique fixed point p  D and
X (k )  p


Mk
X (1)  X (0) .

1 M
Example 3.2.1
Show that the system
1
0
2
x12  81( x2  0.1) 2  sin x3  1.06  0,
3x1  cos( x2 x3 ) 
e  x1x2  20 x3 
10  3
0
3
has a unique solution on D   X  ( x1 , x2 , x3 )t : 1  xi  1, i  1, 2,3 and iterate starting with
X (0)  (0.1, 0.1, 0.1)t until
X ( k )  X ( k 1)
X
(k )

 105.

Solution:
For each equation, we solve for xi
1
1
x1  cos x2 x3 
3
6
1
x2 
x12  sin x3  1.06  0.1
9
1
10  3
x3   e  x1x2 
20
60
Then, G( X )   g1 ( X ), g2 ( X ), g3 ( X )  , where
t
1
1
g1 ( X )  cos x2 x3 
3
6
1
g2 ( X ) 
x12  sin x3  1.06  0.1
9
1
10  3
g3 ( X )   e  x1x2 
.
20
60
If each gi  D, then G  D.
g1 ( X ) 
1
1 1
cos x2 x3  
3
6 2
g2 ( X ) 
1
1
x12  sin x3  1.06  0.1 
1  sin1  1.06  0.1  0.09
9
9
g3 ( X )  
1  x1x2 10  3 1 1( 1) 10  3
e

 e

 0.61.
20
60
20
60
Thus, 1  gi  1, for i  1, 2,3 and G  D implying that G has a fixed point.
Also,
g1
 0,
x1
g1
1
1
  x3 sin x2 x3  .1.sin1  0.281
x2
3
3
g1
1
1
  x2 sin x2 x3  .1.sin1  0.281
x3
3
3
g 2
x1
1
1
 .

 0.1006
2
x1
9 x1  sin x3  1.06 9. 1  sin1  1.06
g 2
0
x2
cos x3
g 2
1


 0.050
2
x3
18. x1  sin x3  1.06 18. 1.219
g3
x
1
 2 e  x1x2  e1  0.1359
x1
20
20
g3
x
1
 1 e  x1x2  e1  0.1359
x2
20
20
g3
0
x3

gi
 0.281, for each i, j  1, 2,3.
x j

M
 0.281  M  3(0.281)  0.843  1.
n
Therefore, G has a unique fixed point in D.
Starting with X (0)  (0.1, 0.1, 0.1)t , we have that
1
1 1
1
x1(1)  cos x2 (0) x3(0)   cos  0.1 (0.1)    0.49998333
3
6 3
6
(0)
1
1
2
x2 (1) 
x12   sin x3(0)  1.06  0.1 
 0.1  sin 0.1  1.06  0.1  0.00944115

9
9
(
0)
(
0)
1
10  3
1
10  3
x3(1)   e  x1 x2 
  e  0.10.1 
 0.52310127
20
60
20
60

X (1)  X (0)
X
(1)



0.42310127
 0.8088  105.
0.52310127
Other values are given the table below:
k
0
1
2
3
4
5
x1( k )
x3( k )
x2 ( k )
0.10000000
0.49998333
0.49999593
0.50000000
0.50000000
0.50000000
0.10000000
0.00944115
0.00002557
0.00001234
0.00000003
0.00000002
0.10000000
0.52310127
0.52336331
0.52359814
0.52359847
0.52359877
X ( k )  X ( k 1)

X (k )

0.8088
0.018
0.00044
0.000023
0.00000059
3.2.1. Newton’s Method for Non-linear systems
Newton’s method for non-linear systems involves selecting the initial point X (0) in
n
and
generating
X ( k )  G ( X ( k 1) )  X ( k 1)   J  X ( k 1)   F  X ( k 1)  ,
1
(3.3)
where J ( X ) is the Jacobian matrix defined by
 f1 ( X )
 x
1

 f 2 ( X )

J ( X )   x1


 f n ( X )
 x
1

f1 ( X )
x2
f 2 ( X )
x2
f n ( X )
x2
f1 ( X ) 
xn 

f 2 ( X ) 
xn  .


f n ( X ) 
xn 
Computing  J  X ( k 1)   at each stage can be avoided by finding a vector Y ( k 1) that satisfies
1
 J  X ( k 1)   Y ( k 1)   F ( X ( k 1) )


so that the new approximation is given by
X ( k )  X ( k 1)  Y ( k 1)
Example 3.2.2
Use Newton’s method to solve the system in Example 3.2.1
(3.4)
Solution:
Here X  ( x1 , x2 , x3 )t and F ( X )   f1 ( X ), f 2 ( X ), f3 ( X )  , where
t
1
2
2
f 2 ( X )  x1  81( x2  0.1) 2  sin x3  1.06
f1 ( X )  3x1  cos x2 x3 
f3 ( X )  e  x1x2  20 x3 
10  3
.
3
The Jacobian matrix is
3


J ( X )   2 x1
  x e  x1x2
 2
x3 sin x2 x3
162( x2  0.1)
 x1e  x1x2
x2 sin x2 x3 

cos x3 

20

and using the initial point X (0)   0.1,0.1, 0.1 , we get
t
F ( X (0) )   0.199995,  2.269833417,8.462025346
t
J (X
(0)
3
0.0009999833334 0.0009999833334 



)
0.2
32.4
0.9950041653  .
 0.9900498337 0.9900498337

20


1
We can find  J ( X (0) )  and use (3.3) to approximate X (1) . If we want to use (3.4), then we
let Y (0)   y1(0) (0), y2 (0) (0), y3(0) (0)  so that  J  X (0)   Y (0)   F ( X (0) ) gives the linear
t
system
3 y1(0)  0.0009999833334 y2 (0)  0.0009999833334 y3(0)  0.199995
0.2 y1(0)  32.4 y2 (0)  0.9950041653 y3(0)  2.269833417
0.9900498337 y1(0)  0.9900498337 y2 (0)  20 y3(0)  8.462025346
Solving this system, we get Y (0)   0.3998696728,  0.08053315147,  0.4215204718 .
t
 X (1)  X (0)  Y (0)  (0.4998696728, 0.01946684853,  0.5215204718)t
Other values for k  2,3,..., are given in the table below:
k
0
1
2
3
4
5
x1( k )
0.1
0.4998696728
0.5000142403
0.5000000113
0.5
0.5
x3( k )
x2 ( k )
0.1
0.0194668485
0.0015885914
0.0000124448
8.516 1010
1.375 1011
0.1
0.5215204718
0.5235569638
0.5235984500
0.5235987755
0.5235987756
THE END!
X ( k )  X ( k 1)

X (k )
0.8083
0.0342
3.0099 103
2.376 105
1.653  109
