Lecture 2. Contents
•
Solution of Linear Systems of Equations
• Consistency
• Rank
• Geometric Interpretation
• Gaussian Elimination
Math for CS
Lecture 2
1
Systems of linear equations
In Linear equations the unknowns appear raised to the power
one.
If the variables in the equation are ordered as x1, … xn and the
missing variables
xi are written as … + 0·xi + … , then the linear equation can be
written in the
matrix notation, according to the matrix multiplication rules.
Solution of linear
equations was one of the reasons, for matrix notation invention.
Math for CS
Lecture 2
2
Linear Systems in Matrix Form
a11x1  a12 x2    a1n xn  b1
a21x1  a22 x2    a2 n xn  b2

an1 x1  an 2 x2    ann xn  bn
Math for CS
 a11 a12
a a
21
22

 



an1 an 2
Lecture 2
 a1n   x1   b1 
 a2 n   x2  b2 

       
   
 ann   xn  bn 
(1)
3
Solution of Linear Systems
Each side of the equation
A x  b
(2)
Can be multiplied by A-1 :
1
1
A A x  A b
Due to the definition of
A-1:
1
A A x  I  x  x
Therefore the solution of (2) is:
1
Math for CS
xA b
Lecture 2
4
Consistency (Solvability)
• A-1 does not exist for every A
• The linear system of equations A·x=b has
a solution, or said to be consistent IFF
Rank{A}=Rank{A|b}
• A system is inconsistent when
Rank{A}<Rank{A|b}
Rank{A} is the maximum number of linearly independent columns
or rows of A. Rank can be found by using ERO (Elementary Row
Oparations) or ECO (Elementary column operations).
Math for CS
Lecture 2
5
Elementary row and column operations
• The following operations applied to the
augmented matrix [A|b], yield an equivalent
linear system
– Interchanges: The order of two rows/columns can be
changed
– Scaling: Multiplying a row/column by a nonzero
constant
– Sum: The row can be replaced by the sum of that row
and a nonzero multiple of any other row.
One can use ERO and ECO to find the Rank as follows:
EROminimum # of rows with at least one nonzero entry
or
ECOminimum
# of columns Lecture
with at
Math
for CS
2 least one nonzero entry
6
An inconsistent example:
Geometric interpretation
1 2  x1  4
 2 4  x    5 

 2   
ERO:Multiply the first row with
-2 and add to the second row
1 2
0 0


Rank{A}=1
1 2 4 
0 0  3


Rank{A|b}=2 > Rank{A}
Math for CS
Lecture 2
7
Uniqueness of solutions
• The system has a unique solution IFF
Rank{A}=Rank{A|b}=n
n is the order of the system
• Such systems are called full-rank systems
Math for CS
Lecture 2
8
Full-rank systems
• If Rank{A}=n
Det{A}  0  A-1 exists  Unique solution
1 2   x1  4
1  1  x   2

 2   
Math for CS
Lecture 2
9
Rank deficient matrices
• If Rank{A}=m<n
Det{A} = 0  A is singular so not invertible
infinite number of solutions (n-m free
variables)
under-determined system
1 2  x1  4
2 4  x   8

 2   
Rank{A}=Rank{A|b}=1
Math for CS Consistent so solvable
Lecture 2
10
Ill-conditioned system of
equations
• A small deviation in the entries of A matrix,
causes a large deviation in the solution.
2   x1   3 
 x1  1
 1
0.48 0.99  x   1.47   x   1

 2  

 2  
 x1  3
2   x1   3 
 1
0.49 0.99  x   1.47   x   0
 2  

 2  

Math for CS
Lecture 2
11
Ill-conditioned continued.....
• A linear system
of equations is
said to be “illconditioned” if
the coefficient
matrix tends to
be
singular
Math for CS
Lecture 2
12
Gaussian Elimination
 a11
 

 ai1

 
an1
 a1i
 
 aii

 ani
Math for CS
 a1n   x1   b1 
       
 ain   xi    bi 
   
       
 ann   xn  bn 
a11


 0

ERO  
 0
Lecture 2
 a1i
 
 a~ii

 0
 a1n   x1   b1 
       
~
 a~in   xi    bi 
   
       
~
 a~nn   xn  bn 
Back substitution
– By using ERO, matrix A is transformed into an
upper triangular matrix (all elements below
diagonal 0)
– Back substitution is used to solve the uppertriangular system
13
Pivotal Element
(1)
Pivotal element  a11
 (1)
a21
(1)
a31

 
a (1)
 n1
(1)
a12
(1)
a22
(1)
a32

an(12)
(1)
a13
(1)
a23
(1)
a33

an(13)





b1(1) 
a1(1n)   x1 

 (1) 

a2(1n)   x2 
b2 
a3(1n)   x3   b3(1) 




   
  
(1)  
b (1) 

ann
  xn 
 n 
The first coefficient of the first row (pivot) is used to zero out first coefficients of
Other rows.
Math for CS
Lecture 2
14
First step of elimination
(1)
(1)
m2,1  a21
/ a11
(1)
(1)
m3,1  a31
/ a11

(1)
mn ,1  an(11) / a11
(1)
a11

 0
 0

 
 0

(1)
a12
( 2)
a22
( 2)
a32

an( 22)
(1)
a13
( 2)
a23
( 2)
a33

an( 23)





 b1(1) 
a1(1n)   x1 
 
 ( 2) 
a2( 2n)   x2 
b2 
a3( 2n )   x3   b3( 2 ) 
 



  
  
( 2)  
( 2) 


x
ann
b
 n 
 n 
First row, multiplied by appropriate factor is subtracted from other rows.
Math for CS
Lecture 2
15
Second step of elimination
(1)
a11

Pivotal element  0
 0

 
 0

(1)
a12
( 2)
a22
( 2)
a32

an( 22)
(1)
a13
( 2)
a23
( 2)
a33

an( 23)





a1(1n)   x1   b1(1) 
  ( 2) 
( 2)  
x
a2 n   2  b2 
a3( 2n )   x3   b3( 2 ) 
  

      
( 2)  
b ( 2 ) 

x
ann
n



 n 
The second coefficient (pivot) of the second row is used to zero out second coefficients of
other rows.
Math for CS
Lecture 2
16
Second step of elimination
( 2)
( 2)
m3, 2  a32
/ a22

( 2)
mn , 2  an( 22) / a22
(1)
a11

 0
 0

 
 0

(1)
a12
( 2)
a22
0

0
(1)
a13
( 2)
a23
( 3)
a33

an( 33)





a1(1n)   x1   b1(1) 
  

a2( 2n)   x2  b2( 2 ) 
a3( 3n)   x3   b3( 3) 
  


     
( 3)  
( 3) 


x
ann
b
 n   n 
Second row, multiplied by appropriate factor
is subtracted from other rows (ERO).
Math for CS
Lecture 2
17
Gaussion elimination algorithm
mr , p  arp( p ) / a (ppp )
arp( p )  0
br( p1)  br( p )  mr , p  bp( p )
For c=p+1 to n
( p 1)
rc
a
Math for CS
Lecture 2
a
( p)
rc
 mr , p  a
( p)
pc
18
Back substitution algorithm
Finally, the following system is obtained:
(1)
a11

 0
 0

 
 0


 0
Math for CS
an( n1) n 1
0
  x1 
 b1(1) 

 ( 2) 

x
 2 
 b2 
  x3 
 b3( 3) 




 
  

b ( n 1) 
an( n1) n   xn 1 
n 1





(n)
(n)
ann 

 xn 


 bn 

Lecture 2
19
(1)
a12
( 2)
a22
0
(1)
a13
( 2)
a23
( 3)
a33




0
0

0
0

a1(1n)
a2( 2n)
a3( 3n)
Back substitution algorithm
The answer is obtained as following:
bn( n )
xn  ( n )
a nn
x n 1 
1
a
1
xi  ( i )
a ii
Math for CS
( n 1 )
n 1 n 1
b
( n 1 )
n 1
 a nn11n x n
n
 (i )

(i )
bi   a ik x k 
k  i 1


Lecture 2

i  n  1, n  2,  ,1
20