Direct Methods for Linear
Systems
Lecture 3
Alessandra Nardi
Thanks to Prof. Jacob White, Suvranu De, Deepak Ramaswamy,
Michal Rewienski, and Karen Veroy
Last lecture review
• Formulation of circuit equations
– Conservation laws (KCL, KVL)
– Branch constitutive equations (BCE)
– KCL, KVL, BCE combined in different ways:
• STA
• MNA
Outline
• Systems of linear equations
– Existence and uniqueness review
– Gaussian Elimination basics
• LU factorization
• Pivoting
Systems of linear equations
• Problem to solve: M x = b
• Given M x = b :
– Is there a solution?
– Is the solution unique?
Systems of linear equations


M
 1



M2

 x1   b1 
    
  x2   b2 
MN 

   
     
 xN  bN 
x1M 1  x2 M 2 
 xN M N  b
Find a set of weights x so that the weighted sum of the
columns of the matrix M is equal to the right hand side b
Systems of linear equations - Existence
A solution exists if:
There exist weights, x1, …., xN, such that:
x1 M1  x2 M 2  ...  xN M N  b
A solution exists when b is in the span of the
columns of M
Systems of linear equations - Uniqueness
Suppose there exist weights, y1, …., yN, not all zero, such that:
y1 M 1  y2 M 2  ...  y N M N  0
Then: Mx = b  Mx + My= b  M(x+y) = b
A solution is unique only if the
columns of M are linearly
independent.
Systems of linear equations
Square matrices
• Given Mx = b, where M is square
– If a solution exists for any b, then the solution
for a specific b is unique.
For a solution to exist for any b, the columns of M must
span all N-length vectors. Since there are only N
columns of the matrix M to span this space, these vectors
must be linearly independent.
A square matrix with linearly independent
columns is said to be nonsingular.
Application Problems
Y e S
 
M x
b
• Matrix is n x n
• Often symmetric and diagonally dominant
• Nonsingular of real numbers
Methods for solving linear equations
• Direct methods: find the exact solution in a
finite number of steps
• Iterative methods: produce a sequence a
sequence of approximate solutions hopefully
converging to the exact solution
Gaussian Elimination Basics
Gaussian Elimination Method for Solving M x = b
• A “Direct” Method
Finite Termination for exact result (ignoring roundoff)
• Produces accurate results for a broad range of
matrices
• Computationally Expensive
Gaussian Elimination Basics
Reminder by 3x3 example
 M 11 M 12 M 13   x1 
b1 

 


 M 21 M 22 M 23   x2   b2 
 M 31 M 32 M 33   x3 


b
3




M11 x1  M12 x2  M13 x3  b1
M 21 x1  M 22 x2  M 23 x3  b2
M 31 x1  M 32 x2  M 33 x3  b3
Gaussian Elimination Basics – Key idea
Use Eqn 1 to Eliminate x1 from Eqn 2 and 3
M11 x1  M12 x2  M13 x3  b1




M 21
M 21
M 21
M12  x2   M 23 
M13  x3  b2 
b1
 M 22 
M11
M11
M11








M 31
M 31
M 31
M12  x2   M 33 
M 13  x3  b3 
b1
 M 32 
M11
M11
M11




GE Basics – Key idea in the matrix
Remove x1 from eqn 2 and eqn 3
Pivot


 M 11



0




0

MULTIPLIERS
  x1 
 
M 12
M 13
 
 
 

 
  
 M 21 
 M 21 
 M 22  
 M 12   M 23  
 M 12    x2 
 M 11 
 M 11 

 
  
 
 

 
   
 M 31 
 M 31 
 M 32  
 M 12   M 33  
 M 12    
 M 11 
 M 11 

 
   x3 




b1











M
 b2   21  b1 

 M 11  









 M 31  
b3  
 b1 
 M 11  

GE Basics – Key idea in the matrix
Remove x2 from eqn 3
Pivot
M 12
 M 11










 M 21 
 M 22  
 M 12 
0
 M 11 







0
0



M 13


 M 21 
 M 23  
 M 12 
 M 11 




 M   M 31  M 
 33  M  12
 11 





 M 31 
 M 32  
 M 12 
 M 11 




 M 21 
 M 22  
 M 12 
 M 11 






 M 21 
 M 23  
 M 12  
 M 11 




Multiplier
  x1  b1

  

  

  

  

  

  



M
   b2   21  b1

  
M 11 


  x2   

  

  

  

  


 M 31 

 M 32  
  
 M 12 

M


 11 
 b   M 21  b 
   b   M 31  b  
 2 
 1
    3  M 11  1 

 M 21 
 M 11   
  
M

M

 22 
 12 
 
 M 11 


  x3  

GE Basics – Simplify the notation
Remove x1 from eqn 2 and eqn 3


 M 11



0




0

  x1 
 
M 12
M 13
 
 
 

 
  
 M 21 
 M 21 
 M 22  
 M 12   M 23  
 M 12    x2 
 M 1122

 M23
11 

 
  
 
 

 
   
 M 31 
 M 31 
 M 32  
 M 12   M 33  
 M 12    
 M 1132

 M33
11 

 
   x3 
M
M
M
M




b1











M
 b2   21  b1 

 M211  









 M 31  
b3   3  b1 

 M 11  
b
b
GE Basics – Simplify the notation
Remove x2 from eqn 3


M
 11
 0

 0








b
M 13
x

1
 1 






M 23
x

b
2
2
  

 x3  
M 32
M 32 

M 33 
M 23
b3 
b2



M 22
M


22

Pivot
M 12
M 22
0
Multiplier
GE Basics – GE yields triangular system
 M11 M12 M13   x1   b1 
 0 M M   x   b 
22
23   2   2 

~
~
 0
0 M 33   x3  b3 
U11 U12 U13   x1   y1 
 0 U U  x    y 
22
23   2   2 

 0 0 U 33   x3   y3 
Altered
During
GE
GE Basics – Backward substitution
U11 U12 U13   x1   y1 
 0 U U  x    y 
22
23   2   2 

 0 0 U 33   x3   y3 
y3
x3 
U 33
y2  U 23 x3
x2 
U 22
y1  U12 x2  U13 x3
x1 
U11
GE Basics – RHS updates
b1

 y1  

  

  

 M 21 
 y2   b2  

 b1
  

 M 11 
  

 y3  b   M 31  b  M 32 b 
 3  M 11  1 M 22 2 




0 0
 1
 y1   b1 
 M 21
   
1 0 y2  b2





M
 11
  y  b 
 3  3
M

M 32
31
1

 M 11 M 22 
1
L
 21
 L31
0
1
L32
0  y1   b1 





0  y2   b2 
1  y3  b3 
GE basics: summary
(1)
Mx=b
GE
Ux=y
(2) Noticed that:
Ly = b
Equivalent system
U: upper trg
L: unit lower trg
(3) U x = y
LU x = b  M x = b
 Efficient way of implementing GE: LU factorization
Gaussian Elimination Basics
Solve M x = b
Step 1
M=LU
=
Step 2 Forward Elimination
Solve L y = b
Step 3 Backward Substitution
Solve U x = y
Note: Changing RHS does not imply to recompute LU factorization
GE Basics – Fitting the pieces together
 M 11 M 12
M
 21 M 22
 M 31 M 32


1

M 21

L
 M 11

 M 31
M
 11
0
1
~
M 32
~
M 22


0

0



1

M 13 
M 23 
M 33 
M 11 M 12
~

U   0 M 22
 0
0

M 13 
~

M 23 

M 33 
GE Basics – Fitting the pieces together


1

M
L   21
 M 11

 M 31
M
 11
0
1
~
M 32
~
M 22


0

0



1

M 11 M 12
~

U   0 M 22
 0
0



 M 11

 M 21
 M 11

 M 31
M
 11
M 21
~
M 22
~
M 32
~
M 22


M 31 

~
M 23 




M 33 

M 13 
~

M 23 

M 33 
LU factorization Basics – Picture
 M 11
M
M
21
M

M
M
M 31
M
M
M 41
21
11
31
11
41
11
M 12 M 13 M 14 

M 2222 M 2323 M 2424

M
M

M
M
M
34
33
33
34
M 32

M
M
M
M
M
M
M
44
44 
42 M 43
43
44
43
M 42
32
22
42
43
22
33
LU Basics
Source-row oriented approach algorithm
“For each source row”
“For each target row below the source”
M ji
Pivot
M ji 
M ii
For k = i+1 to n { “For each row element beyond Pivot”
For i = 1 to n-1 {
For j = i+1 to n {
M jk  M jk  M ji M ik
}
}
}
Multiplier
LU Basics
Target-row oriented approach algorithm
“For each target row”
“For each source row above the target”
M ij
Pivot
M ij 
M jj
For k = j+1 to n { “For each row element beyond Pivot”
For i = 2 to n {
For j = 1 to i-1 {
M ik  M ik  M ij M jk
}
}
}
Multiplier
LU – Source-row and Target-row
k
k
Factored Portion
k
Factored Portion
k
Mult
Multipliers
Active Set
Source-Row
oriented approach
Active Set
Target-Row
oriented approach
LU Basics – Computational Complexity
For i = 1 to n-1 {
For j = i+1 to n {
M ji 
M ji
M ii
“For each Row”
“For each target Row below the source”
Pivot
n 1
n2
(n  i ) 

2
i 1
multipliers
For k = i+1 to n { “For each Row element beyond Pivot”
M jk  M jk  M ji M ik
}
}
}
Multiplier
n 1
2 3
(n  i)  n

3
i 1
2
Multiply-adds
LU Basics – Limitations of the naïve approach
• Zero Pivots
• Small Pivots (Round-off error)
 both can be solved with partial pivoting
LU Basics – Partial pivoting for zero pivots
At Step i
 Factored Portion 
Multipliers M

 (L)

M


ii
ji
What if
M ii  0 ? Cannot form
Simple Fix (Partial Pivoting)
If M ii  0
Find M ji  0 j  i
Swap Row j with i
Row i
Row j
M ji
M ii
LU Basics – Partial pivoting for zero pivots
Two Important Theorems
1) Partial pivoting (swapping rows) always succeeds
if M is non singular
2) LU factorization applied to a diagonally
dominant matrix will never produce a zero pivot
LU Basics – Partial pivoting for small pivots
1.25 104 1.25  x1  6.25

   

x
75
12
.
5
12
.
5


 2  
GE
6.25
1.25 104
  x1  
1.25




5
5 
x
75

6
.
25

10
0
12
.
5

1
.
25

10


 2  
 x1 
1.0001 

x 

4
.
9999

 2  5 digits 
LU Basics – Partial pivoting for small pivots
1.25 104 1.25  x1  6.25

   

x
75
12
.
5
12
.
5


 2  
GE
6.25
1.25 104
  x1  
1.25




5
5 
x
75

6
.
25

10
0
12
.
5

1
.
25

10


 2  
Rounded to 3 digits
1.25 104
1.25   x1   6.25 



5
5 
x

6
.
25

10
0
 1.25 10   2  


 x1 
0 
 
x 
 2  3digits 5
 x1 
1.0001 

x 

4
.
9999

 2  5 digits 
LU Basics – Partial pivoting for small pivots
An Aside on Floating Point Arithmetic
Double precision number
64 bits
sign
11 bits
52 bits
Basic Problem
Avoid sum and subtraction of large and tiny numbers
 Avoid big multipliers
LU Basics – Partial pivoting for small pivots
Partial Pivoting for Roundoff reduction
If | M ii | < max | M ji |
j i
Swap row i with arg (max | M ij |)
j i
Small Multipliers
LU Basics – Partial pivoting for small pivots
swap
1.25 104 1.25  x1  6.25

   

12
.
5
12
.
5

  x2   75 
12.5  x1   75 
 12.5
1.25 104 1.25  x   6.25

 2  

GE
12.5
75
12.5
  x1  

 0 1.25  12.5 105   x   6.25  75 105 

 2  

Rounded to 3 digits
12.5 12.5  x1   75 
 0 1.25  x   6.25

 2  

 x1 
1
 
x 
 2  3digits 5
 x1 
1.0001 

x 

4
.
9999

 2  5 digits 
Pivoting strategies
k
• Partial Pivoting:
– Only row interchange
k
k
• Complete Pivoting
– Row and Column interchange
k
• Threshold Pivoting
– Only if prospective pivot is found to be smaller
than a certain threshold
Summary
• Existence and uniqueness review
• Gaussian elimination basics
– GE basics
– LU factorization
– Pivoting