Gaussian Elimination
Major: All Engineering Majors
Author(s): Autar Kaw
http://numericalmethods.eng.usf.edu
Transforming Numerical Methods Education for STEM
Undergraduates
Naïve Gauss Elimination
http://numericalmethods.eng.usf.edu
Naïve Gaussian Elimination
A method to solve simultaneous linear
equations of the form [A][X]=[C]
Two steps
1. Forward Elimination
2. Back Substitution
Forward Elimination
The goal of forward elimination is to transform the
coefficient matrix into an upper triangular matrix
 25 5 1  x1  106.8 
 64 8 1  x   177.2 

  2 

144 12 1  x3  279.2
5
1   x1   106.8 
25
 0  4.8  1.56  x    96.21

  2 

0
0.7   x3   0.735 
 0
Forward Elimination
A set of n equations and n unknowns
a11x1  a12 x2  a13 x3  ...  a1n xn  b1
a21x1  a22 x2  a23 x3  ...  a2 n xn  b2
.
.
.
.
.
.
an1x1  an 2 x2  an3 x3  ...  ann xn  bn
(n-1) steps of forward elimination
Forward Elimination
Step 1
For Equation 2, divide Equation 1 by a11 and
multiply by a21 .
 a21 
 a (a11x1  a12 x2  a13 x3  ...  a1n xn  b1 )
 11 
a21
a21
a21
a21x1 
a12 x2  ... 
a1n xn 
b1
a11
a11
a11
Forward Elimination
Subtract the result from Equation 2.
a21x1  a22 x2  a23 x3  ...  a2 n xn  b2
a21
a21
a21
− a21x1  a a12 x2  ...  a a1n xn  a b1
11
11
11
_________________________________________________


a21 
a21 
a21
 a22 
a12  x2  ...   a2 n 
a1n  xn  b2 
b1
a11 
a11 
a11


or
a x  ...  a x  b
'
22 2
'
2n n
'
2
Forward Elimination
Repeat this procedure for the remaining
equations to reduce the set of equations as
a11x1  a12 x2  a13 x3  ...  a1n xn  b1
'
'
a22
x2  a23
x3  ...  a2' n xn  b2'
'
'
a32
x2  a33
x3  ...  a3' n xn  b3'
.
.
.
.
.
.
.
.
.
'
an' 2 x2  an' 3 x3  ...  ann
xn  bn'
End of Step 1
Forward Elimination
Step 2
Repeat the same procedure for the 3rd term of
Equation 3.
a11x1  a12 x2  a13 x3  ...  a1n xn  b1
'
'
a22
x2  a23
x3  ...  a2' n xn  b2'
"
a33
x3  ...  a3" n xn  b3"
.
.
.
.
.
.
"
an" 3 x3  ...  ann
xn  bn"
End of Step 2
Forward Elimination
At the end of (n-1) Forward Elimination steps, the
system of equations will look like
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
'
a22
x2  a23
x3  ...  a2' n xn  b2'
"
a33
x3  ...  a3"n xn  b3"
.
.
.
.
.
.
n 1 
 n 1
ann
xn  bn
End of Step (n-1)
Matrix Form at End of Forward
Elimination
a11 a12
 0 a'
22

0
0




 0
0
a13
'
a23
"
a33

0
 a1n   x1   b1 
'
' 




 a2 n x2
b2
  

"
"
 a3n   x3    b3 
  


      
(n1 )
(n-1 )




0 ann   xn  bn 
Back Substitution
Solve each equation starting from the last equation
5
1   x1   106.8 
25
 0  4.8  1.56  x    96.21

  2 

0
0.7   x3   0.735 
 0
Example of a system of 3 equations
Back Substitution Starting Eqns
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
'
a22
x2  a23
x3  ...  a2' n xn  b2'
"
a33
x3  ...  an" xn  b3"
.
.
.
 n 1
.
.
.
n 1 
ann xn  bn
Back Substitution
Start with the last equation because it has only one unknown
( n 1)
n
( n 1)
nn
b
xn 
a
Back Substitution
( n 1)
n
( n 1)
nn
b
xn 
a
xi 
bii 1  ai,ii11 xi 1  ai,ii12 xi  2  ...  ai,in1 xn
i 1
aii
i 1
xi 
bi
  aiji 1 x j
n
j i 1
i 1
ii
a
for i  n  1,...,1
for i  n  1,...,1
THE END
http://numericalmethods.eng.usf.edu
Additional Resources
For all resources on this topic such as digital audiovisual
lectures, primers, textbook chapters, multiple-choice tests,
worksheets in MATLAB, MATHEMATICA, MathCad and
MAPLE, blogs, related physical problems, please visit
http://numericalmethods.eng.usf.edu/topics/gaussian_elimi
nation.html
Naïve Gauss Elimination
Example
http://numericalmethods.eng.usf.edu
Example 1
The upward velocity of a rocket is given at three
different times
Table 1 Velocity vs. time data.
Time, t s 
5
Velocity, v m/s 
106.8
8
12
177.2
279.2
The velocity data is approximated by a polynomial as:
vt   a1t 2  a 2 t  a3 ,
Find the velocity at t=6 seconds .
5  t  12.
Example 1 Cont.
Assume
vt   a1t 2  a2t  a3 , 5  t  12.
Results in a matrix template of the form:
t12
 2
t 2
t32

t1
t2
t3
1

1
1

 a1   v1 
 a   v 
 2  2

 a3 
 
 v3 

Using data from Table 1, the matrix becomes:
 25 5 1  a1  106.8 
 64 8 1 a   177.2 

  2 

144 12 1 a3  279.2
Example 1 Cont.
 25 5 1  a1  106.8 
 25 5 1  106.8 
 64
8 1 a2   177.2    64
8 1  177.2 

   



144 12 1  a3  279.2
144 12 1  279.2
1. Forward Elimination
2. Back Substitution
Forward Elimination
Number of Steps of Forward
Elimination
Number of steps of forward elimination is
(n1)(31)2
Forward Elimination: Step 1
 25 5 1  106.8 
 64 8 1  177.2 


144 12 1  279.2
25
Divide Equation 1 by 25 and
64
 2.56 .
multiply it by 64,
25
5 1  106.8 2.56  64 12.8 2.56  273.408
.
Subtract the result from
Equation 2
Substitute new equation for
Equation 2
64
 64
0
1 
2.56 
177.2
273.408
 4.8  1.56 
 96.208
8
12.8
5
1
 106.8 
 25
 0  4.8  1.56   96.208


144 12
1
 279.2 
Forward Elimination: Step 1 (cont.)
5
1
 106.8 
 25
Divide Equation 1 by 25 and
 0  4.8  1.56   96.208
144


144 12
1
 279.2  multiply it by 144, 25  5.76 .
25
5 1  106.8 5.76  144 28.8 5.76  615.168
12
1 
Subtract the result from 144
 144
28.8 5.76 
Equation 3
0  16.8  4.76 
.
279.2
615.168
 335.968
5
1

106.8 
25
Substitute new equation for 

0

4
.
8

1
.
56


96
.
208
Equation 3


 0  16.8  4.76   335.968
Forward Elimination: Step 2
5
1

106.8 
25
 0  4.8  1.56   96.208 


 0  16.8  4.76   335.968
0
Divide Equation 2 by −4.8
and multiply it by −16.8,
 16.8
 3.5 .
 4.8
 4.8 1.56   96.208 3.5  0 16.8  5.46   336.728
Subtract the result from
Equation 3
Substitute new equation for
Equation 3
0
 0
0
 16.8  4.76 
335.968
 16.8  5.46   336.728
0
0.7 
0.76
5
1
 106.8 
25
 0  4.8  1.56   96.208


 0
0
0.7 
0.76 
Back Substitution
Back Substitution
5
1
 106.8  25
5
1   a1   106.8 
25
 0  4.8  1.56   96.2   0  4.8  1.56 a    96.208

 
  2 

 0
0
0.7  0.7   0
0
0.7   a3   0.76 
Solving for a3
0.7a3  0.76
0.76
a3 
0.7
a3  1.08571
Back Substitution (cont.)
5
1   a1   106.8 
25
 0  4.8  1.56 a    96.208

  2 

 0
0
0.7   a3   0.76 
Solving for a2
 4.8a2  1.56a3  96.208
 96.208  1.56a3
a2 
 4.8
 96.208  1.56 1.08571
a2 
 4.8
a2  19.6905
Back Substitution (cont.)
5
1   a1   106.8 
25
 0  4.8  1.56 a    96.2

  2 

 0
0
0.7   a3   0.76 
Solving for a1
25a1  5a2  a3  106.8
106.8  5a2  a3
a1 
25
106.8  5 19.6905  1.08571

25
 0.290472
Naïve Gaussian Elimination Solution
 25 5 1  a1  106.8 
 64
8 1 a2   177.2 

   

144 12 1  a3  279.2
 a1  0.290472
a    19.6905 
 2 

 a3   1.08571 
Example 1 Cont.
 a1  0.290472
a    19.6905 
 2 

 a3   1.08571 
Solution
The solution vector is
The polynomial that passes through the three data points is then:
vt   a1t 2  a2t  a3
 0.290472t 2  19.6905t  1.08571, 5  t  12
v6   0.2904726   19.69056   1.08571
 129.686 m/s .
2
THE END
http://numericalmethods.eng.usf.edu
Naïve Gauss Elimination
Pitfalls
http://numericalmethods.eng.usf.edu
Pitfall#1. Division by zero
10 x2  7 x3  3
6 x1  2 x2  3x3  11
5 x1  x2  5 x3  9
0 10  7  x1   3 
6 2





3 x2  11

   
5  1 5   x3   9 
Is division by zero an issue here?
12 x1  10 x2  7 x3  15
6 x1  5 x2  3x3  14
5 x1  x2  5 x3  9
12 10  7  x1  15
6 5
3   x2   14

   
 5  1 5   x3   9 
Is division by zero an issue here?
YES
12 x1  10 x2  7 x3  15
6 x1  5 x2  3x3  14
24 x1  x2  5 x3  28
12 10  7  x1  15
6 5
3   x2   14 

   
24  1 5   x3  28
12 10  7  x1   15 
0
0
6.5  x2   6.5

   
12  21 19   x3   2
Division by zero is a possibility at any step
of forward elimination
Pitfall#2. Large Round-off Errors
15
10  x1   45 
 20
 3  2.249 7   x   1.751

  2 

1
3   x3   9 
 5
Exact Solution
 x1  1
 x   1
 2  
 x3  1
Pitfall#2. Large Round-off Errors
15
10  x1   45 
 20
 3  2.249 7   x   1.751

  2 

1
3   x3   9 
 5
Solve it on a computer using
6 significant digits with chopping
 x1   0.9625 
 x    1.05 
 2 

 x3  0.999995
Pitfall#2. Large Round-off Errors
15
10  x1   45 
 20
 3  2.249 7   x   1.751

  2 

1
3   x3   9 
 5
Solve it on a computer using
5 significant digits with chopping
 x1   0.625 
 x    1.5 
 2 

 x3  0.99995
Is there a way to reduce the round off error?
Avoiding Pitfalls
Increase the number of significant digits
• Decreases round-off error
• Does not avoid division by zero
Avoiding Pitfalls
Gaussian Elimination with Partial Pivoting
• Avoids division by zero
• Reduces round off error
THE END
http://numericalmethods.eng.usf.edu
Gauss Elimination with
Partial Pivoting
http://numericalmethods.eng.usf.edu
Pitfalls of Naïve Gauss Elimination
• Possible division by zero
• Large round-off errors
Avoiding Pitfalls
Increase the number of significant digits
•
Decreases round-off error
• Does not avoid division by zero
Avoiding Pitfalls
Gaussian Elimination with Partial Pivoting
• Avoids division by zero
• Reduces round off error
What is Different About Partial
Pivoting?
At the beginning of the kth step of forward elimination,
find the maximum of
akk , ak 1,k ,................, ank
If the maximum of the values is a pk
in the p
th
row, k  p  n, then switch rows p and k.
Matrix Form at Beginning of 2nd
Step of Forward Elimination
a11 a12
 0 a'
22

'
 0 a32




 0 a'n 2
a13 
'
a23 
'
a33 
 
'
'
an 3 an 4
a1n   x1   b1 
' 
' 


a 2 n x2
b2
   
'
'
a3n   x3   b3 
   
      
'
'




ann   xn  bn 
Example (2nd step of FE)
6 14 5.1 3.7 6   x1   5 
0  7 6
1
2   x2   6

   
12 1 11  x3    8 
0 4

   
x
0
9
23
6
8
9
4

   
0  17 12 11 43  x5   3 
Which two rows would you switch?
Example (2nd step of FE)
6 14 5.1 3.7 6   x1   5 
0  17 12 11 43  x   3 

 2   
12 1 11  x3    8 
0 4

   
x
0
9
23
6
8
9
4

   
0  7 6
1
2   x5   6
Switched Rows
Gaussian Elimination
with Partial Pivoting
A method to solve simultaneous linear
equations of the form [A][X]=[C]
Two steps
1. Forward Elimination
2. Back Substitution
Forward Elimination
Same as naïve Gauss elimination method
except that we switch rows before each
of the (n-1) steps of forward elimination.
Example: Matrix Form at Beginning
of 2nd Step of Forward Elimination
a11 a12
 0 a'
22

'
 0 a32



 0 a'n 2
a13 
'
a23 
'
a33 
 
'
'
an 3 an 4
a1n   x1   b1 
' 
' 


a 2 n x2
b2
   
'
'
a3n   x3   b3 
   
      
'
'




ann   xn  bn 
Matrix Form at End of Forward
Elimination
a11 a12 a13
 0 a' a'
22
23

"
0 a33
0




 0
0
0
 a1n   x1   b1 
'
' 




 a2 n x2
b2
  

"
"
 a3n   x3    b3 
  


      
(n1 )
(n-1 )




0 ann   xn  bn 
Back Substitution Starting Eqns
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
'
a22
x2  a23
x3  ...  a2' n xn  b2'
"
a33
x3  ...  an" xn  b3"
.
.
.
 n 1
.
.
.
n 1 
ann xn  bn
Back Substitution
( n 1)
n
( n 1)
nn
b
xn 
a
i 1
xi 
bi
n
i 1
  aij x j
j i 1
i 1
ii
a
for i  n  1,...,1
THE END
http://numericalmethods.eng.usf.edu
Gauss Elimination with
Partial Pivoting
Example
http://numericalmethods.eng.usf.edu
Example 2
Solve the following set of equations
by Gaussian elimination with partial
pivoting
5 1  a1  106.8 
 25
 64





8 1 a 2  177.2

   

144 12 1  a3  279.2
Example 2 Cont.
 25 5 1  a1  106.8 
 25 5 1  106.8 
 64 8 1 a   177.2   

64
8
1

177
.
2
2

   



144 12 1  a3  279.2
144 12 1  279.2
1. Forward Elimination
2. Back Substitution
Forward Elimination
Number of Steps of Forward
Elimination
Number of steps of forward elimination is
(n1)=(31)=2
Forward Elimination: Step 1
• Examine absolute values of first column, first row
and below.
25 , 64 , 144
• Largest absolute value is 144 and exists in row 3.
• Switch row 1 and row 3.
 25 5 1  106.8 
144 12 1  279.2
 64 8 1  177.2    64 8 1  177.2 




144 12 1  279.2
 25 5 1  106.8 
Forward Elimination: Step 1 (cont.)
144 12 1  279.2
 64 8 1  177.2 


 25 5 1  106.8 
144
Divide Equation 1 by 144 and
64
 0.4444 .
multiply it by 64,
144
12 1  279.2 0.4444  63.99 5.333 0.4444  124.1
.
Subtract the result from
Equation 2
64
 63.99
0
8
1  177.2
5.333 0.4444 
124.1
2.667 0.5556  53.10
1
 279.2
Substitute new equation for 144 12
 0 2.667 0.5556  53.10
Equation 2


 25
5
1
 106.8 
Forward Elimination: Step 1 (cont.)
12
1
 279.2 Divide Equation 1 by 144 and
144
 0 2.667 0.5556  53.10
25

 multiply it by 25,
 0.1736.
144
 25
5
1
 106.8 
144
12 1  279.2 0.1736  25.00 2.083 0.1736  48.47
.
Subtract the result from
Equation 3
Substitute new equation for
Equation 3
25
 25
0
5
1  106.8
2.917
0.8264  58.33
2.083 0.1736  48.47
12
1
 279.2
144
 0 2.667 0.5556  53.10


 0 2.917 0.8264  58.33
Forward Elimination: Step 2
• Examine absolute values of second column, second row
and below.
2.667 , 2.917
• Largest absolute value is 2.917 and exists in row 3.
• Switch row 2 and row 3.
1
 279.2
1
 279.2
144 12
144 12
 0 2.667 0.5556  53.10   0 2.917 0.8264  58.33




 0 2.917 0.8264  58.33
 0 2.667 0.5556  53.10
Forward Elimination: Step 2 (cont.)
12
1
 279.2
144
 0 2.917 0.8264  58.33


 0 2.667 0.5556  53.10
0
Divide Equation 2 by 2.917 and
multiply it by 2.667,
2.667
 0.9143.
2.917
2.917 0.8264  58.33 0.9143  0 2.667 0.7556  53.33
Subtract the result from
Equation 3
0
 0
0
Substitute new equation for
Equation 3
12
1
 279.2 
144
 0 2.917 0.8264  58.33 


 0
0
 0.2   0.23
.
2.667 0.5556 
2.667 0.7556 
0
53.10
53.33
 0.2   0.23
Back Substitution
Back Substitution
1
 279.2 
1 
144 12
144 12
 0 2.917 0.8264  58.33    0 2.917 0.8264




 0
 0
0
 0.2   0.23
0
 0.2 
Solving for a3
 0.2a3  0.23
 0.23
a3 
 0.2
 1.15
 a1   279.2 
a    58.33 
 2 

 a3   0.23
Back Substitution (cont.)
1 
144 12
 0 2.917 0.8264


0
 0.2 
 0
 a1   279.2 
a    58.33 
 2 

 a3   0.23
Solving for a2
2.917a2  0.8264a3  58.33
58.33  0.8264a3
a2 
2.917
58.33  0.8264 1.15

2.917
 19.67
Back Substitution (cont.)
1 
144 12
 0 2.917 0.8264


0
 0.2 
 0
 a1   279.2 
a    58.33 
 2 

 a3   0.23
Solving for a1
144a1  12a2  a3  279.2
279.2  12a2  a3
a1 
144
279.2  12 19.67  1.15

144
 0.2917
Gaussian Elimination with Partial
Pivoting Solution
 25 5 1  a1  106.8 
 64 8 1 a   177.2 

  2 

144 12 1  a3  279.2
 a1  0.2917
a    19.67 
 2 

 a3   1.15 
Gauss Elimination with
Partial Pivoting
Another Example
http://numericalmethods.eng.usf.edu
Partial Pivoting: Example
Consider the system of equations
10 x1  7 x2  7
 3x1  2.099 x2  6 x3  3.901
5 x1  x2  5 x3  6
In matrix form
 7 0  x1 
 7 
 10
3.901
 3 2.099 6  x 


  2 = 
 6 
 5
 1 5  x 3 
Solve using Gaussian Elimination with Partial Pivoting using five
significant digits with chopping
Partial Pivoting: Example
Forward Elimination: Step 1
Examining the values of the first column
|10|, |-3|, and |5| or 10, 3, and 5
The largest absolute value is 10, which means, to
follow the rules of Partial Pivoting, we switch
row1 with row1.
Performing Forward Elimination
 7 0  x1   7 
 10
 3 2.099 6  x   3.901

 2  

 5
 1 5  x3   6 

7
0  x1   7 
10
 0  0.001 6  x   6.001

 2  

 0
2.5
5  x3   2.5 
Partial Pivoting: Example
Forward Elimination: Step 2
Examining the values of the first column
|-0.001| and |2.5| or 0.0001 and 2.5
The largest absolute value is 2.5, so row 2 is
switched with row 3
Performing the row swap
7
0  x1   7 
10
 0  0.001 6  x   6.001

 2  

 0
2.5
5  x3   2.5 

7
0  x1   7 
10
0
  x    2.5 
2
.
5
5

 2  

 0  0.001 6  x3  6.001
Partial Pivoting: Example
Forward Elimination: Step 2
Performing the Forward Elimination results in:
0   x1   7 
10  7
 0 2.5
  x    2.5 
5

 2  

 0 0 6.002  x3  6.002
Partial Pivoting: Example
Back Substitution
Solving the equations through back substitution
0   x1   7 
10  7
 0 2.5
  x    2.5 
5

 2  

 0 0 6.002  x3  6.002
6.002
x3 
1
6.002
2.5  5 x3
x2 
 1
2.5
7  7 x 2  0 x3
x1 
0
10
Partial Pivoting: Example
Compare the calculated and exact solution
The fact that they are equal is coincidence, but it
does illustrate the advantage of Partial Pivoting
 x1   0 
X  calculated   x2    1
 x3   1 
X  exact
 x1   0 
  x 2    1
 x3   1 
THE END
http://numericalmethods.eng.usf.edu
Determinant of a Square Matrix
Using Naïve Gauss Elimination
Example
http://numericalmethods.eng.usf.edu
Theorem of Determinants
If a multiple of one row of [A]nxn is added or
subtracted to another row of [A]nxn to result in
[B]nxn then det(A)=det(B)
Theorem of Determinants
The determinant of an upper triangular matrix
[A]nxn is given by
det A   a11  a22  ...  aii  ...  ann
n
  a ii
i 1
Forward Elimination of a
Square Matrix
Using forward elimination to transform [A]nxn to an
upper triangular matrix, [U]nxn.
Ann  U  nn
det  A  det U 
Example
Using naïve Gaussian elimination find the
determinant of the following square
matrix.
 25 5 1
 64 8 1


144 12 1
Forward Elimination
Forward Elimination: Step 1
 25 5 1
 64 8 1


144 12 1
25
Divide Equation 1 by 25 and
64
 2.56 .
multiply it by 64,
25
5 1 2.56  64 12.8 2.56
Subtract the result from
Equation 2
64
 64
0
Substitute new equation for
Equation 2
5
1 
 25
 0  4.8  1.56


144 12
1 
.
8
12.8
1
2.56
 4.8  1.56
Forward Elimination: Step 1 (cont.)
5
1 
 25
 0  4.8  1.56 Divide Equation 1 by 25 and

 multiply it by 144, 144  5.76 .
144 12
1 
25
25 5 1 5.76  144 28.8 5.76
.
Subtract the result from
Equation 3
Substitute new equation for
Equation 3
144 12
 144 28.8
0  16.8
1
5.76
 4.76
5
1 
25
 0  4.8  1.56 


 0  16.8  4.76
Forward Elimination: Step 2
5
1 
25
 0  4.8  1.56 


 0  16.8  4.76
0
Divide Equation 2 by −4.8
and multiply it by −16.8,
 16.8
 3.5 .
 4.8
 4.8  1.56  3.5  0  16.8  5.46
.
Subtract the result from
Equation 3
Substitute new equation for
Equation 3
0
 0
0
 16.8  4.76
 16.8  5.46
0
0.7
5
1 
25
 0  4.8  1.56


 0
0
0.7 
Finding the Determinant
After forward elimination
5
1 
 25 5 1
25
 64 8 1   0  4.8  1.56




0
0.7 
144 12 1
 0
.
det A   u11  u22  u33
 25   4.8 0.7
 84.00
Summary
-Forward Elimination
-Back Substitution
-Pitfalls
-Improvements
-Partial Pivoting
-Determinant of a Matrix
Additional Resources
For all resources on this topic such as digital audiovisual
lectures, primers, textbook chapters, multiple-choice tests,
worksheets in MATLAB, MATHEMATICA, MathCad and
MAPLE, blogs, related physical problems, please visit
http://numericalmethods.eng.usf.edu/topics/gaussian_elimi
nation.html
THE END
http://numericalmethods.eng.usf.edu