Systems of Linear Equations
Linear equation:
a1 x1 + a2 x2 + . . . . +an xn = b
a1, a2, . . . an, b - constants
x1, x2, . . . xn - variables
no x2, x3, sqrt(x),. . . ,
no cross-terms like xi xj
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
Applications:
1. Reaction stoichiometry (balancing equations)
2. Electronic circuit analysis (current flow in networks)
3. Structural analysis (linear deformations of various
constructions)
4. Statistics (least squares analysis)
5. Economics: optimization problems (Nobel prize in
economics in 70s for “Linear Programming”).
6. System of non-linear equations – approximate
solutions.
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
Examples of linear equations:
7x=2
ax=b
Solution:
point in 1D
3x+4y=1
a1 x + a2 y = b
line in 2D
2 x + 5 y - 2= -3
a1 x + a2 y + a3z=b
plane in 3D
What if we have several equations (system)?
How many solutions we will have?
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
Example: What is the stoichiometry of the
complete combustion of propane?
C3H8 + x O2 Å y CO2 + z H2O
atom balances:
oxygen
2x=2y+z
carbon
3 = y
=>
y=3
hydrogen
8=2z
=>
z=4
substitute: 2 x = 10 => x = 5
C3H8 + 5 O2 Å 3 CO2 + 4 H2O
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
In 2D (2 variables ) to solve an SLE is to find an intersection of
several lines.
1 equation: " solutions.
2 equations: a) no solutions (parallel lines)
b) one solution
c) " solutions
a11x1 + a12x2 = b1
a21x1 + a22x2 = b2
to have one solution we need the determinant a11a22 - a21 a12 " 0,
in cases (a) and (c) a11/a21=a12/a22.
>= 3 equations:
a) no solutions (most likely)
b) one solution: all equations except 2 are
“linear combinations” of the others and may be scrapped if we
look at solution only.
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
In general:
If the number of variables m is less than the number of
equations n the system is said to be “overdefined”: too
many constraints. If the solution still exists, n-m equations
may be thrown away.
If m is greater than n the system is “underdefined” and
often has many solutions.
We consider only m = n cases.
November 2002
10.001 Introduction to Computer
Methods
Matrix Formulation of SLE
Any system of linear equations can be fromulated in the matrix
form:
a11x1 + a12x2 + …+ a1nxn = b1
a21x1 + a22x2 + ….+ a2nxn = b2
…..
an1x1 + an1x2 +….+ annxn = bn
aij -elements of the coefficient matrix A, b - load vector
 a 11 a 12 … a 1n

 a 21 a 22 … a 2n
 … ..

a a … a
nn
 n1 n2
November 2002
  x1   b1 
    
  x 2   b2 
 ⋅  ...  =  ... 
    
  x  b 
  n  
10.001 Introduction to Computer
Methods
A⋅x = b
Matrix Formulation of SLE
SLE in a compact matrix form: A · x = b
Inverse matrix A-1 : A · A-1 = I = A-1 ·A
A-1 · A · x = A-1 · b Å x = A-1 · b
Thus, to solve SLE we need to invert the
matrix.
In Matlab:
>> x = A\b
Just one line!!!
“ \ “ is a black box. What is inside?
Do we always need A-1 to solve the SLE?
November 2002
10.001 Introduction to Computer
Methods
Matrix Formulation of SLE
For n=2:
 a 11 a 12

 a 21 a 22
The solution is:
a22b1 − a12b2
x1 =
a11a22 − a12 a21
  x1   b1 
 ⋅   =  
  x 2   b2 
a11b2 − a21b2
x2 =
a11a22 − a12 a21
a11a22-a12a21 = det(A) is the determinant of matrix A.
It should be non-zero for the unique solution to exist.
a11a22 - a12a21 = 0 ÄÅ a11 /a21 = a12 /a22
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
 a1 
A =   , where a1 = (a11 a12), a2 = (a21 a22) , so
 a2  det(A) ? 0 is equivalent to α1 a1 + α2 a2 ? 0
for any α1,2 ? 0.
In this case a1 and a2 are called linearly independent.
This is true for any number of equations.
NB: The SLE has a single solution if the coefficient
matrix has a non-zero determinant
or
if the vectors a1, a2,… an are linearly independent.
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
To solve the SLE without using A-1:
1. eliminate x from equation 2: eq.2 - 2 x eq.1.
2. solve equation 2 for y.
3. substitute y into equation 1.
4. solve equation 1.
Example:
x + 2 y = -1
2x+2y= 0
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
Or more generally:
Form the equations.
Eliminate variable until eq-s n, n-1, . . . , 1
have 1, 2, 3, . . . , n variables left.
A is now an upper triangular matrix.
Backsubstitute solution of eq. n to eq. n-1,
n-1 to n-2, . . . , 2 to 1 to solve the system
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
To solve SLE we perform invariant operations, which
do not change the solutions:
1. add/subtract the same value to/from both sides of
the equation
2. multiply/divide both sides of the equation by the
same value
3. add/subtract some equation from another one
4. rearrange equations
5. rearrange columns in the coefficients matrix
November 2002
10.001 Introduction to Computer
Methods
Systems of Linear Equations
Example:
x+2y+z=0
2x+2y+3z=3
-x + 3 y = -4
1. Eliminate z from eq. 2.
2. Eliminate y from equation 2.
3. Solve eq. 3 for x and backsubstitute to eqs 1&2
4. Solve eq. 2 for y and backsubstitute to eq. 1.
5. Solve eq. 1 for z.
November 2002
10.001 Introduction to Computer
Methods
Gaussian Elimination using Matrix Algebra
1.
 a 11 a 12 a 13
a a a
pivot
22
23
element,  21
 a 31 a 32 a 33
row
2.
  x1   b1 
  x  = b  = b
 2   2 
  x 3   b3 
m21 = -a21/a11, add row 1 x m21 to row1
m31 = -a31/a11, add row 1 x m31 to row3
 a 11 a 12 a 13

(2)
(2)
0
a
a
22
23

0 a (2) a (2)
32
33

November 2002
  x1   b1 
    (2) 
  x 2  = b2  = b
  x 3   b ( 2 ) 
 3 

10.001 Introduction to Computer
Methods
Gaussian Elimination using Matrix Algebra
3.
m32 = -a(2)32/a(2)22, add row 2 x m32 to row1
 a 11 a 12 a 13
pivot

(2)
(2)
0
a
a
element, 
22
23
row
0 0 a (3)
33

  x1   b1 
    (2) 
  x 2  = b2  = b
  x 3   b ( 3 ) 
 3 

Upper triangular matrix
November 2002
10.001 Introduction to Computer
Methods
Gaussian Elimination using Matrix Algebra
4.
Solve for x1, x2, x3 by backsubstitution:
b 3( 3 )
x 3 = (3) ;
a 33
x2 =
b
(2)
2
−a
a
(2)
23
(2)
22
x3
;
b 1 − a (222 ) x 2 − a 13 x 3
;
x2 =
a 11
November 2002
10.001 Introduction to Computer
Methods
Summary
1. Recognizing systems of linear equations.
2. Matrix representation of systems of linear
equations.
3. Gaussian elimination to get an upper
triangular matrix.
4. Backsubstitution.
November 2002
10.001 Introduction to Computer
Methods
Key Concepts from Previous Lecture
• Solution of linear equations
– Matrix formulation of equation system
– Decomposition to upper triangular from
– Back substitution to solve in reverse order
• Gaussian Elimination algorithm
November 2002
10.001 Introduction to Computer
Methods
Essence of the Gaussian Elimination Algorithm
• Form the equations
• Successively eliminate variables until the
upper triangular form is reached
(ELIMINATION STEP)
• Once the elimination has been completed
perform a back substitution in the reverse
order to obtain solution for each of the
variables (BACK SUBSTITUTION STEP)
Extremely valuable algorithm -- Gaussian Elimination
November 2002
10.001 Introduction to Computer
Methods
Numerical Problems
1. Scalability - how big a problem can be solved?
Physical memory
Disk storage
Processor time
2. What is the “fastest” algorithm?
3. What is the most “robust” algorithm?
Numerical stability: what happens if aij=0 etc.?
4. What are the effects of finite precision
arithmetic?
November 2002
10.001 Introduction to Computer
Methods
Typical Times (Microseconds)
• Multiplication = 1
• Division = 3
• Addition = 0.5
• Subtraction = 0.5
How long to solve A x = b, when
n = 100, 1,000, 1,000,000?
November 2002
10.001 Introduction to Computer
Methods
Scalability
Code for Gaussian elimination contains 3 loops:
1. it makes n-1 runs to eliminate variables
2. k-th run goes through n-k rows (k = 1,. . ., n-1)
3. in i-th row we calculate aij(k) = aij - m akj n-k+1 times
n −1
1 Å
∑
2 Å (n-k)
k =1
3 Å (n-k+1)
n −1
Overall about
November 2002
∑ ( n − k )( n − k + 1)
k =1
10.001 Introduction to Computer
Methods
operations.
Gaussian Elimination (How many operations)
/*
Gaussian Decomposition
*/
n-1 runs, eliminating n-1 variables
for ( k=0; k < n-1; k++){
n-k runs, k-th variable is eliminated from n-k rows
for ( i=k+1; i < n; i++){
m = A[i][k]/A[k][k];
n-k+1 runs, eliminating k-th variable form i-th row
for ( j=k; j < n; j++) {
A[i][j] += -m*A[k][j];
}
b[i] += -m*b[k];
}
}
November 2002
10.001 Introduction to Computer
Methods
Gaussian Elimination (How many operations)
Useful Identities
n(n + 1)
1 + 2 + 3+L+ n =
2
2
2
2
2 n( n + 1)( 2n + 1)
+ 2 + 3 +L+ n =
6
Gauss reduction
Multiplication
Addition
n3 n 2 n
− + ≈ O(n 3 )
3 2 6
n3 n 2 n
− + ≈ O(n 3 )
3 2 6
Time scales as n3 ! A rather poor scalability.
November 2002
10.001 Introduction to Computer
Methods
Numerical Stability
What if one of the diagonal elements is a small
number r, close to zero?
r x 1 + x2 = 1
x1 + x2 = 2
Possible problems caused by dividing by r:
1. Overflow: 1/r is too big.
2. Numerical instability.
November 2002
10.001 Introduction to Computer
Methods
Numerical Stability
After elimination
r x1 + x2
0
After substitution
= 1
+ (1-1/r) x2 = 2-1/r
x2 = (2 - 1/r)/(1-1/r)
x1 = (1 - x2)/r
if 1/r >> 2, then x2 = 1 and x1 =0.
x2 = (large number)/(large number)
x1 = (small number)/(large number) is a problem.
November 2002
10.001 Introduction to Computer
Methods
Numerical Stability
Solution - remove small numbers from the
diagonal by exchanging rows or columns.
May be done by pivoting:
Exchanging rows just “renumbers”
equations.
Exchanging columns “reindexes” variables.
November 2002
10.001 Introduction to Computer
Methods
Numerical Stability
Let’s exchange rows in the previous example.
a) x1 + x2 = 2
r x 1 + x2 = 1
b) x1 + x2 = 2
c) x1 = 2 - x2
(1-r) x2 = 1-2r
The correct answer:
if r << 1,
November 2002
x2 = 1 and x1 = 1
10.001 Introduction to Computer
Methods
x2 = (1-2r)/(1-r)
Numerical Stability
Pivoting algorithm:
Searches for the largest aik in each row below
the current one to use for the next elimination
step, and rearranges the rows so that mik is
always less than one.
November 2002
10.001 Introduction to Computer
Methods
Numerical Stability
Example:
Augmented matrix: n x (n+1)
0.0001 0.5 | 0.5
A=
= [A, b]

 0.4 − 0.3 | 0.1
~
Use 4 digit arithmetic:
9.9999… Å9.9998
0.0001 0.5 | 0.5 
A=

0
−
2000
|
−
2000


~
0.1-(4000)(0.5) =
-1999.9 = -2000
-0.3-(4000)(0.5) = -2000.3 = -2000
− 2000
x2 =
=1
− 2000
November 2002
1
x1 =
(0.5 − (0.5)(1)) = 0
0.0001
10.001 Introduction to Computer
Methods
Numerical Stability
 0.4 − 0.3
A=
0.0001 0.5
~
 0.4 − 0.3
A=
0
0.5

~
| 0.1

| 0.5
| 0.1
| 0.5
0.5+(0.3)0.000025=0.5
0.5-(0.1)0.000025=0.5
1
0.5
x2 = =1 x1 = (0.1+(0.3)(1))=1
0.4
0.5
November 2002
10.001 Introduction to Computer
Methods
“1” instead of “0”
Quite a difference!
LU factorization.
Linear system Ax = b is solved by Gaussian
elimination.
Matrix A is fixed, but we have a set of b-s: b1, b2, b3…
How to avoid repeating the solution for A and do it
only for b-s??
Answer: Express Gaussian elimination as a matrix.
Each step of elimination is represented by some
elementary matrix acting on A and on b. Overall GE
will be represented by the product of these matrices.
November 2002
10.001 Introduction to Computer
Methods
LU factorization.
A = LU,
U - upper diagonal, L – lower diagonal
If A – nonsingular,
1. Ax = b Å LUx = b Å L(Ux) = b Å Ly = b
y is found by forward substitution.
2.
Ux = y
x is found by backsubstitution
Gaussian elimination == L-1.
LU factorization is a standard way to solve
SLE in case A is a non-singular matrix.
November 2002
10.001 Introduction to Computer
Methods
LU factorization.
Elementary matrix: A matrix obtained from identity
matrix by the following “elementary row operations”
is called elementary matrix.
Elementary row operations:
1. multiply a non-zero constant throughout a row
2. interchange two rows
3. add a constant multiple of another row
(remember invariant operations ?)
November 2002
10.001 Introduction to Computer
Methods
LU factorization.
Examples of elementary matrices:
1 0 
0 −5


1
0

0

0
multiply the second
row of I2 by -5
0
0
0
0
0
1
1
0
November 2002
1 0 3 
 0 1 0  add 3 times the 3rd
row
to
the
1st
row
of
I


3
0
 0 0 1 

1
0  interchange the 2nd

0  & the 4th rows of I4
10.001 Introduction to Computer
Methods
How do elementary matrices work?
 1
E =  − 2
 0
0
1
0
0
0 
1 
“adding -2 x first row to
the second row”
 1 0 0 b1   b1 






E ⋅ b = − 2 1 0 b2  = b2 − 2b1 
 0 0 1 b3   b3 
November 2002
10.001 Introduction to Computer
Methods
How do elementary matrices work?
0 0 1 


P = 0 1 0 
1 0 0
“Interchanging the first
and the third rows”:
permutation matrix
0 0 1  b1  b3 






P ⋅ b = 0 1 0 b2  = b2 
1 0 0 b3   b1 
November 2002
10.001 Introduction to Computer
Methods
Elementary Operations.
 2 1 1   x1   b1 
Ax =  4 1 0   x 2  = b2  = b
 − 2 2 1   x3   b3 
Elementary matrices for Gaussian elimination:
2 row - 2x1st row
3 row + 1st row
3 row + 3x2 row
1 0 0
1 0 0
 1 0 0






E1 = − 2 1 0 , E2 = 0 1 0 , E3 = 0 1 0
0 3 1
1 0 1
 0 0 1
November 2002
10.001 Introduction to Computer
Methods
Elementary Operations.
Inverting elementary matrices:
Ek-1-? (Ek-1)ii= (Ek)ii, (Ek-1)ij= -(Ek)ij for i? j
 1 0 0
1 0 0
−1




E1 = − 2 1 0, E1 = 2 1 0
 0 0 1
0 0 1
0 0 1 


−1
P = 0 1 0  = P
1 0 0
November 2002
10.001 Introduction to Computer
Methods
LU factorization.
Ek Ek-1 … E1 (A x) = Ek Ek-1 … E1 b
Ek Ek-1 … E1 = L-1
L-1 A x = L-1 b,
On the other hand: L= E1-1 E2-1 … Ek -1
Also:
0 0
0 0
0
0
 1
 1
1
E1 = − m21 1 0, E2 =  0
1 0, E3 = 0
1
0
 0
− m31 0 1
0 − m32 1
0 1
m-s – coefficients form Gaussian elimination.
0
0
 1
L-1 = E3 E2 E1 = − m21
1
0
− m31 − m32 1
November 2002
U is obtained after
The elimination:
U=L-1A.
10.001 Introduction to Computer
Methods
Gains due to LU factorization?
We care only for L-1: it allows for calculation of load
vectors without repeating Gaussian elimination.
For every bj we need to calculate L-1bj instead of
performing a complete Gaussian elimination.
n times (row · column) ~ n2 operations instead of n3!
Quite a difference.
November 2002
10.001 Introduction to Computer
Methods
Lecture Summary
• Solution of linear equations
– Matrix formulation of equation system
– Decomposition to upper triangular from
– Back substitution to solve in reverse order
• Gaussian Elimination algorithm
• LU decomposition if many b-s for the same A.
November 2002
10.001 Introduction to Computer
Methods
Gaussian Elimination Algorithm
Forward Reduction:
for k=1,…,n-1
for i=k+1,…n
lik = aik/akk
for j=k+1,…,n
aij = aij - likakj
end loop j
bi = bi -likbk
end loop i
end loop k
November 2002
Back substitution:
for k = n,…,1
xk = b k
for i=k+1,…n
xk = xk-akixi
end loop i
xk = xk/akk
end of loop k
10.001 Introduction to Computer
Methods
Gaussian Elimination II
Back substitution:
for k = n,…,1
xk = bk-A(k,:)*x
xk = xk/akk
end of loop k
Forward Reduction:
for k=1,…,n-1
for i=k+1,…n
lik = aik/akk
Rowi = Rowi - lik*Rowk
bi = bi -likbk
end loop i
end loop k
November 2002
10.001 Introduction to Computer
Methods