Chapter I
Computational Linear Algebra
1. Introduction
Many of the problems of the numerical analysis can be reduced to the problem
of solving linear systems of equations. Among the problems, which could be treated
are the solution of ordinary and/or partial differential equations by using finite
difference methods, the solution of systems of equations, the eigen-value problems of
mathematical physics, least squares fitting of data and polynomial approximation.
In this course will restrict us to solve linear system of equations, including:

To solve a linear system A x  b , where A is a given non-singular matrix of
order n (real or complex) b is a given column vector of n components, and x
is an unknown column vector of n components

In the above problem there are many several different right hand sides b - for
example, k of them – and therefore k unknown vectors x to be found. If we let
B be the n-rowed, k-column matrix of right-hand sided, and let X be the
corresponding n–rowed, k –column matrix of solution vectors, then we are to
solve the linear system AX = B.

To find the inverse A-1of a non-singular matrix A.
2. Direct Methods For Solving Linear System of Equations
By a direct method, we mean a method, which after a certain finite number of steps
gives the exact solution, disregarding rounding errors. For systems
Axb
where the matrix A is full (that is, most of the elements of A are non-zero), direct
elimination methods are almost always the most efficient, but when A is sparse (that
is, a large proportion of the elements of A are zeros), iterative methods offer certain
advantages. Iterative methods give a sequence of approximate solutions, converging
when the number of steps tends to infinity. Thus the choice between direct and
iterative methods depends on the proportion and distribution as well as sign and size
of non-zero elements of A.
2.1. Triangular Systems
Linear system of equations where the matrix is triangular are particularly simple to
solve. A system
Uxb,
where matrix U is upper triangular, has the form
u 11 x 1  u 12 x 2  ...
 u 1n x n  b1
u 22 x 2  ...
 u 2n x n  b 2
....
u n -1,n -1 x n -1  u n -1,n x n  b n -1
u n,n x n  b n
If we assume that ui,i  0, for i=1 (1) n, then the unknowns could be computed in the
order xn, xn-1, …, x1 (backward order) as follows
xn 
bn
u n,n
x n -1 
b n -1  u n -1,n x n
u n -1,n -1
....
bi 
xi 
n
u
j i 1
i, j
xj
u i,i
...
n
x1 
b1   u i, j x j
j 2
u 1,1
Now we are ready to write a simple algorithm to find xi, i=n (-1) 1
x (n)  b(n)/u(n, n);
for i  (n - 1) : (-1) : 1
{s  0;
for j  i  1 : n
{s  s  u(i, j) * x(j); }
x(i)  b(i) - s)/u(i, i) ;
}
It is very simply to convert the above algorithm to MATLAB code (m file to solve the
a linear system of equations with an upper triangle matrix form - save the file as
uppersys.m)
function [x]= uppersys(n,u,b)
x(n)=b(n)/u(n,n);
for i=n-1:-1:1
s=0;
for j=i+1:n
s=s+u(i,j)*x(j);
end;
x(i)=(b(i)-s)/u(i,i);
end;
x=x';
the following is a MATLAB session to solve such systems for n=3;
» n=3;
» b=[1;2;3]
b =
1
2
3
» u=[ 12 2.4 3; 0 20 3;0 0 9]
u =
12.0000
2.4000
3.0000
0
20.0000
3.0000
0
0
9.0000
» x=uppersys(n,u,b)
x =
-0.0100
0.0500
0.3333
» check = u*x-b
check =
1.0e-015 *
-0.1110
0
0
It is easy to show that the number of division and multiplication is 0.5*(n2 + n), we
left the proof as an exercise.
2.2. Full Matrix and Gaussian Elimination
The most important among the direct methods for solving a general linear system of
equations is Gaussian elimination. The idea behind this method is to eliminate the
unknowns in a systematic way, in such a way that we end up with a triangular system,
which we know how to solve. Consider the system,
A n,n x n,1  b n,1 ,
We assume in the following that the matrix A is non-singular (det(A)  0), so the
system A x  b has a unique solution (only one solution).
Now we assume a1,1 0.Then we could eliminate x1 from the last (n-1) equations by
subtracting from the ith equation the multiple
m i,1 
a i,1
a 1,1
,
i  2 (1) n
of the first equation. So the pattern for the last (n-1) equations become
(2)
(2)
(2)
a (2)
2,2 x 2  a 2,3 x 3  ...  a 2,n x n  b 1
.
.
.
.
a
.
.
(2)
n,2
(2)
x n  a x 3  ...  a (2)
n,n x n  b n
(2)
n,3
where the new coefficient are given by
a i,(2)j  a i, j  mi,1a 1, j
,
b i(2)  b i  mi,1b1 , i  2 (1) n .
This is a system of (n-1) equations in the (n-1) unknowns xi , i =2 (1) n. If the a 2( 2, 2) 0,
in a similar way, as in the elimination of x1from equations 2 (1) n, one could eliminate
x2 from the last (n-2) of these equations (n-1 equations). We get a system of (n-2)
equations in the unknowns xi , i= 3 (1) n, if we let
m i,2 
(2)
a i,2
a (2)
2,2
i  3 (1) n
,
The coefficients of this system are given by
a i,(3)j  a i,(2)j  mi,2 a (2)
2, j
,
b i(3)  b i(2)  mi,2 b i(2) , i  3 (1) n .
(n 1)
The elements a 1,1 , a (2)
2,2 ,..., a n 1,n 1 , which appear during the elimination are called
pivotal elements. If all these are non-zero, we can continue until after (n-1) steps, we
get a single equation in one unknown, which is xn
(n)
a (n)
n,n x n  b n
Now, we gather the first equation from each step to get the following new system
(1)
(1)
a 1,1
x 1  a 1,2
x 2  ...  a 1,(1)n x n  b1(1)
(2)
(2)
a (2)
2,2 x 2  ...  a 2,n x n  b 2
...
(n)
a (n)
n,n x n  b n
Thus we have reduced the original system to triangular system, which as we have
seen could be easily solved.
The algorithm of the above-mentioned method (the triangularization of the system)
could be written as follows
for k  1 : (n - 1)
for i  k  1 : n
{m i,k 
a i,(k)k
a (k)
k,k
;
b i(k 1)  b i(k) - m i,k b (k)
k ;
for j  k  1 : n
a i,(kj1)  a i,(k)j  m i,k a (k)
k, j ;
}
The triangular decomposition algorithm could be simplified some what if we assume
that we are dealing with a copy of the given system of equations and therefore can use
the memory occupied by the ai,j for the storage of mi,k and a i,(k)j . The mi,k (called
multipliers) could be stored where the zeros occur, and the a i,(k)j could be written over
the old ai,j . The modified algorithm (minimizing the storage used) could be written as
follows
for k  1 : (n - 1)
for i  k  1 : n
{a i,k 
a i,k
a k,k
;
b i  b i - a i,k b k ;
for j  k  1 : n
a i, j  a i, j  a i,k a k, j ;
}
the MATLAB code for the modified algorithm is as follows (save with fmatsys.m)
function [a,b]=fmatsys (n,a,b)
for k=1:n-1
for i=k+1:n
a(i,k)=a(i,k)/a(k,k);
b(i)=b(i)-a(i,k)*b(k);
for j=k+1:n
a(i,j)=a(i,j)-a(i,k)*a(k,j);
end;
end;
end;
the MATLAB session to test the code is:
» n=3;
» a=magic(n)
a=
8
3
4
1
5
9
» b=[1;2;3]
b=
1
2
6
7
2
3
» [a,b]=fmatsys(n,a,b)
a=
8.0000
0.3750
0.5000
1.0000 6.0000
4.6250 4.7500
1.8378 -9.7297
b=
1.0000
1.6250
-0.4865
to find the unknown xi , i = 1 (1) n; we combine the code of triangularization and the
code of solving the upper triangular system, and the MATLAB code (save it as
gausseli.m) is as follows
function [x]=gausseli(n,a,b)
for k=1:n-1
for i=k+1:n
a(i,k)=a(i,k)/a(k,k);
b(i)=b(i)-a(i,k)*b(k);
for j=k+1:n
a(i,j)=a(i,j)-a(i,k)*a(k,j);
end;
end;
end;
x(n)=b(n)/a(n,n);
for i=n-1:-1:1
s=0;
for j=i+1:n
s=s+a(i,j)*x(j);
end;
x(i)=(b(i)-s)/a(i,i);
end;
x=x';
and the MATLAB session for testing the code is:
» n=3;
» a=magic(n)
a=
8
3
4
1
5
9
» b=[1;2;3]
6
7
2
b=
1
2
3
» x=gausseli(n,a,b)
x=
0.0500
0.3000
0.0500
» check=a*x-b
check =
1.0e-015 *
0
-0.2220
-0.4441
The estimate number of arithmetic operations required to reduce a system to
triangular form using Gauss elimination method is of order n3, and is denoted
mathematically as O(n3) and approximately equal to 1/3 n3, and we already know that
the of operations to solve upper triangular system, so we can calculate (or estimate )
the total number of operation to solve a system of linear equation with full matrix
form as a polynomial of n.
2.3. Pivoting Strategies
We have seen that, if in Gaussian elimination the pivotal elements a (k)
k,k = 0, for some
k, then the method breaks down. As a simple example for such case, let consider the
following
x1  x 2  x 3  1 


x 1  x 2  2 x 3  2


x 1  2 x 2  2 x 3  1
(*)
This system is non-singular and has the unique solution x1 = - x2 = x3 = 1. However,
after the first step in the elimination processes, we get
x2
x3 1 


 x 3  0
Which means that a (2)
2,2 = 0, and we could not proceed as usual. The way to proceed
obviously is to interchange equations 2 and 3 before the next step, which in this case
actually gives us the triangular system directly.
Now we consider the general case that in step k we have a (k)
k,k = 0, but for some other
element a i,(k)k , i = k+1 (1) n in column k must be non-zero, otherwise the matrix is
singular (det(A) = 0), assume that a (k)
r,k , so we could interchange rows k and r and
proceed with the elimination. Using the above technique, we can conclude that any
non-singular matrix could be reduced to triangular form using Gaussian elimination
combined with row interchange.
To ensure numerical stability, it will be necessary to perform row interchange not
only a pivotal element is exactly zero, but also when it nearly zero. Suppose that in
the system of equations (*) the element a2,2 is changed to 0.0001 and Gaussian
elimination carried through without row interchanging. The resulting triangular
system becomes
x1  x 2  x 3  1



0.0001 x 2  2 x 3  1



9999 x 3  10000
(**)
If we perform the back-substitution using three-figure floating-point arithmetic, the
computed solution becomes
x1  0 , x 2  0
, x 3  1.000
where the true solution rounded to four decimals is
x 1  1.0000 , x 2  - 1.0001
, x 3  1.0001
It is left to the reader to verify that if we interchange rows 2 and 3, then using same
precision the computed solution becomes
x 1  1.000 , x 2  - 1.000
, x 3  1.000
which is correct to the three decimals.
To prevent possible catastrophic errors as illustrated by the example mentioned above,
it is usually necessary to choose pivotal element in the step k, by the following
strategy:
Partial Pivoting: Choose r as the smallest integer for which
(k)
a (k)
r, k  max a i,k , k  i  n .
and interchange rows k and r, or interchange the order of reading
p(i)
k
k
or
r
r
In practice, partial pivoting is generally satisfactory with Gaussian elimination, to
convert any full matrix system to upper triangular system.
There are two important cases when Gaussian elimination could be carried without
partial pivoting (row interchanging). These are systems where the matrix A is of one
of the following types:
Diagonally Dominant Matrix
n
a i,i   a i, j , i  1 (1) n
j 1
j i
Symmetric and positive definite
T
A T  A and x A x  0,  x  0
Now, we are ready to write the algorithm for Gaussian elimination (with partial
pivoting – permutation vector) and back-substitution
for k  1 : n
{p(k)  k; }
for k  1 : n - 1
{max0  a k,k ;
locmax  p(k);
for i  k  1 : n

{ if a i,k  max

{max0  a i,k ;
locmax  i; }
}
if p(k)  k 
{r  p(k); p(k)  p(locmax); p(locmax)  r; }
for i  k  1 : n
{ a p(i),k 
a p(i),k
a p(k),k
;
for j  k  1 : n  1
a p(i), j  a p(i), j  a p(i),k * a p(k), j ;
}
}
xn 
a p(n),n 1
a p(n),n
;
for k  n - 1 : -1 : 1
{s  0;
for j  k  1 : n
s  s  a p(k), j * x j ;
xk 
a
p(k), j
 s
a p(k),k
;
}
Now, it is easy to write the MATLAB code for the general direct algorithm to solve
system of linear equations, a possible code for the above algorithm is as follows (save
the file with gepartpb.m, we augment the Matrix A to include b as the (n+1)th column
function [x] = gepartpb (n,a)
% initialization of permutation vector
for k=1:n
p(k)=k;
end;
%apply partial pivoting associated with gaussian elimination
for k=1:n-1
max0=abs(a(k,k));
locmax=p(k);
for i=k+1:n
if abs(a(i,k)) > max0
max0=abs(a(i,k));
locmax=i;
end;
end;
if k ~= locmax
r=p(k);
p(k)=p(locmax);
p(locmax)=r;
end;
for i=k+1:n
a(p(i),k)=a(p(i),k)/a(p(k),k);
for j=k+1:n+1
a(p(i),j)=a(p(i),j)-a(p(i),k)*a(p(k),j);
end;
end;
end;
% back substitution to get the solution
x(n)=a(p(n),n+1)/a(p(n),n);
for k=(n-1):-1:1
s=0;
for j=k+1:n
s=s+a(p(k),j)*x(j);
end;
x(k)=(a(p(k),n+1)-s)/a(p(k),k);
end;
x=x’;
The MATLAB session for testing the code is provided in the following
» n=3;
» a=[1 1 1 1; 1 1 2 2; 1 2 2 1]
a=
1
1
1
1
1
2
1
2
2
1
2
1
» x=gepartpb(n,a)
locmax =
3
x=
1
-1
1
» check=a(:,1:3)*x-a(:,4)
check =
0
0
0
Now we test the code for, when we change the value of a(2,2) to 1.0001)
» a(2,2)=1.0001
a=
1.0000
1.0000
1.0000
1.0000 1.0000 1.0000
1.0001 2.0000 2.0000
2.0000 2.0000 1.0000
» x=gepartpb(n,a)
locmax =
3
x=
1.0000
-1.0001
1.0001
» check=a(:,1:3)*x-a(:,4)
check =
0
0
0
2.3. Tri-diagonal Matrix and Thomas Algorithm
We consider very important special case of system of linear equation, in which the
elements distributed on the diagonal and above and below that diagonal, and the
system has the following form ( for a system of n = 4)
 b1
a
 2
0

0
c1
0
b2
a3
0
c2
b3
a4
0
0 
c3 

b4 
 x1 
x 
 2 
x 3 
 
x 4 
 d1 
d 
 2
d 3 
 
d 4 
Instead of storing the above system in matrix form with a very large number of zero
elements (especially in case of large n), we store the system in vector format: the
diagonal elements in vector b of order n, and the elements below the diagonal a vector
a also of order n and the elements above the diagonal in a vector c of order n with a(1)
= c(n) = 0, and we apply the elimination process of one row below the diagonal. The
benefits for solving such systems using this technique (Thomas algorithm) instead of
using the general elimination (Gauss elimination on a full matrix) is to reduce the
number of arithmetic operation form O(n3) to O(n), and we sketch the algorithm in the
following steps
1. in row 1: divide the elements by b1: so b1=1; c1=c1/b1; d1=d1/b1.
2. start the elimination process to the second row: so a2 = 0; b2=b2- c1*a2; d2=d2a2*d1;(c2 is unchanged);
3. before we continue elimination process to the third row; we divide the second
row by b2, and so on till we reach to the nth row.
Now we are ready to develop the algorithm to solve the tri-diagonal system (Thomas
algorithm), as follows
c1  c1 /b 1 ; d 1  d 1 /b 1 ;
for i  2 : n
{ b i  b i  c i 1 * a i ;
c i  c i /b i ;
d i  d i  d i 1 * a i  /b i ;
}
xn  d n ;
for k  n - 1 : -1 : 1
{x k  d k  c k * x k 1 ; }
the MATLAB code:
function x=thomas(n,a,b,c,d)
c(1)=c(1)/b(1);
d(1)=d(1)/b(1);
for i=2:n
b(i)=b(i)-a(i)*c(i-1);
c(i)=c(i)/b(i);
d(i)=(d(i)-d(i-1)*a(i))/b(i);
end;
x(n)=d(n);
for k=n-1:-1:1
x(k)=d(k)-c(k)*x(k+1);
end;
x=x';
the MATLAB session to test the code
» n=5;
» a=[0;1;1;1;1]
a =
0
1
1
1
1
» b=[-4;-4;-4;-4;-4]
b =
-4
-4
-4
-4
-4
» c=[1;1;1;1;0]
c =
1
1
1
1
0
» d=[1;1;1;1;1]
d =
1
1
1
1
1
» x=thomas(n,a,b,c,d)
x =
-0.3654
-0.4615
-0.4808
-0.4615
-0.3654
» check=a(2)*x(1)+b(2)*x(2)+c(2)*x(3)-d(2)
check =
-1.1102e-016