Chapter 11
Gaussian Elimination (I)
Speaker: Lung-Sheng Chien
Reference book: David Kincaid, Numerical Analysis
OutLine
• Basic operation of matrix
- representation
- three elementary matrices
• Example of Gaussian Elimination (GE)
• Formal description of GE
• MATLAB usage
Matrix notation in MATLAB
A
6
-2
2
4
12
-8
6
10
3
-13
9
3
-6
4
1
-18
A1,:  A1,1: 4  first row of A   6 2 2 4
 2 



8

A :, 2   A 1: 4, 2   second column of A  
 13 


4


Matrix-vector product
橫直
Inner-product based
6
-2
2
4
x1
12
-8
6
10
x2
3
-13
9
3
x3
-6
4
1
-18
x4
6 x1 + (-2) x2 + 2 x3 + 4 x4

12 x1 + (-8) x2 + 6 x3 + 10 x4
3 x1 + (-13) x2 + 9 x3 + 3 x4
(-6) x1 + 4 x2 + 1 x3 + (-18) x4
直橫
outer-product based
6
-2
2
4
x1
12
-8
6
10
x2
3
-13
9
3
x3
-6
4
1
-18
x4
-2
6

x1
12
3
-6

x2
-8
-13
4
2

x3
6
9
1
4

x4
10
3
-18
Matrix-vector product: MATLAB implementation
matvec.m
compute b  Ax
Inner-product based
for i  1: 4
for
j  1: 4
b  i   A  i, j  * x  j 
end
end
outer-product based
for j  1: 4
for i  1: 4
b  i   A  i, j  * x  j 
end
end
Question: which one is better
Column-major nature in MATLAB
physical index : 1D
Logical index : 2D
6
1
12
2
3
3
-6
4
-2
5
-8
6
6
-2
2
4
-13
7
12
-8
6
10
4
8
3
-13
9
3
2
9
-6
4
1
-18
6
10
9
11
1
12
4
13
10 14
3
15
-18 16
Question: how does column-major affect
inner-product based matrix-vector product
and outer-product based matrix-vector
product?
Matrix representation: outer-product
 a11 a12

 a21 a22
a
 31 a32
where
 a11

 a21
a
 31
a12
a22
a32
a11e1e1T  a12e1e2T  a13e1e3T  
a13  3



T
T
T
T
a23    aij ei e j  a21e2e1  a22e2e2  a23e2e3  
i , j 1


T
T
T
a33 
a
e
e

a
e
e

a
e
e
32 3 2
33 3 3
 31 3 1

0
0 0 0
 


e3e2T   0   0 1 0    0 0 0  is outer-product representation
1
0 1 0
 


 A 1,: x 
a13  x1  3
3
3

 inner  product based 

 
a23  x2    aij ei eTj x   ei   aij x j  
A
2,:
x




i , j 1
i 1
j 1




 A  3,: x 
a33  x3 


 
Elementary matrix
[1]
(1) The interchange of two rows in A: A s,:  At,:
 1 0 0   a11


0
0
1

  a21
0 1 0 a

  31
a12
a22
a32
a13 

a23 
a33 

 a11

 a31
a
 21
a12
a32
a22
a13 

a33 
a23 
 e1T   1 0 0 
  

Define permutation matrix P  1,3, 2    e3T    0 0 1 
 e2T   0 1 0 

  
 e1T 
 e1T x   x1 
 
 T   
inner  product based
Px   e3T  x 
 e3 x    x3 
 e2T 
 e2T x   x2 
 

  
1 0 0
Question 1: why P 1  PT   0 0 1 


0 1 0


How to explain?
1
Question 2: how to easily obtain P
Concatenation of permutation matrices
 e1T   1 0 0 
  

P1  1,3, 2    e3T    0 0 1 
 e2T   0 1 0 

  
 e3T   0 0 1 
  

P2   3, 2,1   e2T    0 1 0 
 e1T   1 0 0 

  
 x1 
 
such that P1 x   x3 
x 
 2
 x3 
 
such that P2 x   x2 
x 
 1
Question: P2 P1  3,2,11,3,2  ?
 x1 
 
P2  P1 x   P2  x3 
x 
 2

 x2 
 
 x3 
x 
 1
implies P2 P1   2,3,1
0 0 11 0 0 0 1 0


 

Direct calculation P2 P1   0 1 0   0 0 1    0 0 1 
1 0 00 1 0 1 0 0


 

 x1 
 
 x2 
x 
 3
 x1 
 
 x3 
x 
 2
 x1 
 
 x2 
x 
 3
 x3 
 
 x2 
x 
 1
Elementary matrix
[2]
(2) Multiplying one row by a nonzero constant:  A  s,:  A  s,:
 1 0 0  a11


0

0

 a21
 0 0 1  a

 31
a12
a22
a32
a13 

a23 
a33 

a12
 a11

  a21  a22
 a
a32
 31
a13 
 a23 
a33 
1 0 0
Define scaling matrix S2      0  0 


0 0 1


 x1 


inner  product based
S 2    x 

x
 2
 x 
 3 
 
S2     S2  1
1
1 0

1
 0


0 0

0

0

1 
How to explain?
 x1 
 x1 
 x1 


1




S2   
S2 1/  
since x2 
   x2      x2  
 


x 
 x 
 x

 3
 3 
3


Elementary matrix
[3]
(3) Adding to one row a multiple of another: A s,:   At,:  A s,:
 1 0 0   a11


0
1
0

  a21
0  1 a

  31
a12
a22
a32
a13 

a23 
a33 

a12
 a11

a22
 a21
 a  a
 21 31  a22  a32


a23

 a23  a33 
a13
1 0 0


Define GE (Gaussian Elimination) matrix L32      0 1 0 
0  1


How to explain?
 1 0 0   x1 
 x1 

   inner  product based 

L32    x   0 1 0   x2  
x
2


0  1 x 
x  x 

 3 
 2 3
1 0 0 0 0 0

 

L32      0 1 0    0 0 0   I   e3e2T
0 0 1 0  0

 

outer-product representation

L32    x  I   e3e2T

 x1   0 
  

x  x   e3 e2T x  x    x2  e3   x2    0 
 x  x 
 3  2
 
L32    x : multiply x2 by  and add  x2 to x3
L32    A : multiply A 2,: by  and add  A 2,: to A3,:
for i  j
Lij    x : multiply x j by  and add  x j to xi
Lij    A : multiply A j,: by  and add  A j,: to Ai,:
Use MATLAB notation


A 1,:


L32    A  
A  2,:

  A  2,:  A  3,: 



 A 1,:  

A 1,:

 

A
2,:

A
2,:





 

 A  3,:    A  2,:  A  3,: 

 

Concatenation of GE matrices
Suppose
1



L21       1 

1 



A 1,:


such that L21    A    A 1,:  A  2,: 


A  3,:


1



L31     
1 

1 



A 1,:


A  2,:
such that L31    A  

  A 1,:  A  3,: 


Question: L31    L21     ?

 

A 1,:
A 1,:

 

L31    L21    A  L31      A 1,:  A  2,:     A 1,:  A  2,: 

   A 1,:  A  3,: 
A  3,:

 

1



L31    L21       1   L21    L31   

1 

OutLine
• Basic operation of matrix
• Example of Gaussian Elimination (GE)
- forward elimination to upper triangle form
- backward substitution
• Formal description of GE
• MATLAB usage

12
6
12

)
)
12

6
3
 
6
6
6
-2
2
4
x1
12
-8
6
10
x2
3
-13
9
3
x3
-6
4
1
-18
x4
-8
-2
6
2
10
4
12

34
27
-38
34
3

6
12
6
-2
2
4
x1
0
-4
2
2
x2
3
-13
9
3
x3
12

10
27
0
-4
2
2
10
-6
4
1
-18
x4
-38
3
-13
9
3
27
6
-2
2
4
x1
12
6
-2
2
4
12
0
-4
2
2
x2
0
-12
8
1
x3
-6
4
1
-18
x4
0
-12
8
1
21

6
6

10
21
-38
-6
)

6

6
6
0

12
4
)

12

4
4
-2
2
1
2
3
-18
4
-14
-38

12
-26
6
-2
2
4
x1
0
-4
2
2
x2
0
-12
8
1
x3
0
2
3
-14
x4
-12
8
1
21
-4
2
2
10
0
2
-5
12
4
-9
6
-2
2
4
x1
0
-4
2
2
x2
0
-12
8
1
x3
0
2
3
-14
x4
12

12

10
21
-26
First row does not change thereafter
10
21
-26
2

4
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
2
3
-14
x4
12

10
-9
-26
2
)

2

4
4

2
)
4
 
2
3
-14
-26
-4
2
2
10
0
4
-13
-21
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
0
4
-13
x4
4
-13
-21
2
-5
-9
0
-3
-3
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
0
4
-13
x4
12

10
-9
-21
12

10
-9
-21
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
0
0
-3
x4
12

10
-9
-3
Backward substitution: inner-product-based
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
0
0
-3
x4

12
6
-2
2
4
x1
10
0
-4
2
2
x2
-9
0
0
2
-5
x3
-3
-2
2
4
x1
0
-4
2
2
x2
x3
-9
2x3  5x4  9  x3  2
12


10
x4
3x4  3  x4  1
6
12
10
x4
4x2  2x3  2x4  10  x2  3
6
-2
2
4
x1
x2
x3
12

x4
6x1  2x2  2x3  4x4  12  x1  1
Backward substitution: outer-product-based
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
0
0
-3
x4
6
-2
2
x1
0
-4
2
x2
0
0
2
x3
6
-2
x1
0
-4
x2
6
x1

12

10
3x4  3  x4  1
-9
-3
12

4

10
x4

12
8
8


2
-9
8
8
-5

x2
2
x3
-2
2

2x3  4  x3  2
-4

6
12
12
4 x2  12  x2  3
6x1  6  x1  1
OutLine
• Basic operation of matrix
• Example of Gaussian Elimination (GE)
• Formal description of GE
- component-wise and column-wise representation
- recursive structure
• MATLAB usage
12
3 

6
6
6

6
6
-2
2
4
x1
12
-8
6
10
x2
3
-13
9
3
x3
-6
4
1
-18
x4
12

34
27
-38
 3
 3
L31    Ax  L31    b
 6
 6
 6 
 6 
L41    Ax  L41    b
 6 
 6 
6
-2
2
4
x1
12
-8
6
10
x2
3
-13
9
3
x3
0
2
3
-14
x4

 12 
 12 
L21    Ax  L21    b
 6
 6
6
-2
2
4
x1
0
-4
2
2
x2
3
-13
9
3
x3
-6
4
1
-18
x4
6
-2
2
4
x1
12
12
-8
6
10
x2
34
0
-12
8
1
x3
27
-6
4
1
-18
x4
-26
12

27
-38
12

10
34
21
-38
Eliminate first column: component-wise
L41 1 L31  0.5 L21  2 Ax  L41 1 L31  0.5 L21  2 b
6
-2
2
4
x1
12
-8
6
10
x2
3
-13
9
3
x3
-6
4
1
-18
x4
where

12
6
-2
2
4
x1
34
0
-4
2
2
x2
27
0
-12
8
1
x3
-38
0
2
3
-14
x4
 1




2
1

L41 1 L31  0.5  L21  2   
 0.5
1 


1
 1
Exercise: check
 1
 6 2


 2 1
 12 8
 0.5
1  3 13


1  6 4
 1
2 4   6 2
 
6 10   0 4

9 3   0 12
 
1 18   0 2
2 4 

2 2 
8 1 

3 14 
by MATLAB
12

10
21
-26
Eliminate first column: column-wise
 6 2

8
1  12
define A  A 
 3 13

 6 4
then
 1
L1 A   C
 a
 11
 a11
BT  

A   0

0 
  a11
I 33   C

Exercise: check A 2
 a111
 1
 a21
1
Notation: A    1
 a31
 1
 a41
2 4 

6 10   a11

9 3  C

1 18 
 1




2
1
 1
1
 

L


and

 0.5
1   C a11


1
 1
BT 

A
 6 2
 
  0 4
T
CB   
0 12
A
a11  
0 2
BT
2 4 

2 2 
8 1 

3 14 
 4 2 2 
CB


 A
  12 8 1  by MATLAB
a11 

 2 3 14 
T
a121
a131
1
a22
1
a23
1
a32
1
a33
1
a42
1
a43
 a111
a141 


1
a24  L1
 0
1


L
A


1 
a34

 0

1 
a44

 0
a121
a131
 2
a22
 2
a23
 2
a32
 2
a33
 2
a42
 2
a43
a141 

 2
a24   a111

 2  
a34   0
 2 
a44

 

 2 
A 
0 

I 33 
Eliminate second column: component-wise

12
4
6
-2
2
4
x1
0
-4
2
2
x2
0
-12
8
1
x3
0
2
3
-14
x4
12

-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
0
4
-13
x4

 12  1
1
L32  
L
A

L
b


4


21
-26

6
10
2
4
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
2
3
-14
x4

12

10
-9
-26
12

10
-9
-21

 2   12  1
1
L42    L32  
L
A

L
b


4

4

 


Eliminate second column: column-wise
1



1
2

define L   L42  0.5  L32  3  

3 1 


0.5
1


Exercise: check L 2

1
 6 2


1
1
 0 4
L A 

3 1  0 12


1  0 2
 0.5

2 4   6 2
 
2 2   0 4

8 1  0 0
 
3 14   0 0
2 4 

2 2 
2 5 

4 13 
 1




2
1
  L1 L 2  , why? Give a simple explanation.
Exercise: check L 2 L1  
 0.5 3 1 


1
0.5
1


 a111

 0
1
Notation: L  A  
 0

 0
a121
a131
 2
 2
a22
a23
 2
a32
 2
a33
 2
 2
a42
a43
 a111
a141 


 2
a24  L 2
 0
 2  1

L
L
A


 2 
a34

 0

 2 
a44

 0
a121
a131
 2
 2
a22
a23
0
 3
a33
0
 3
a43
a141 
 a111

 2
a24
 
 0
 3 
a34  
 0
 3  
a44 

 2
a22
0
 

 

A 3 
Eliminate third column: component-wise
4

2
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
0
4
-13
x4
1



1

define L 3  L43  2   


1



2
1


12

10
-9

 4
L43    L 2 L1 A  L 2 L1b
 2
-21
6
-2
2
4
x1
0
-4
2
2
x2
0
0
2
-5
x3
0
0
0
-3
x4
 1




2
1
3  2  1

  L 2 L3 L1  L 2  L1 L3
then L L L  
 0.5 3 1



0.5 2 1 
 1

12

10
-9
-3
LU-decomposition
Notation:
 a111

 0
L 2 L1 A  
 0

 0
a121
a131
 2
a22
 2
a23
0
 3
a33
0
 3
a43
 a111
a141 


 2
a24  L3
 0
 3  2  1

L
L
L
A


 3 
a34

 0

 3 
a44
 0

 6 2 2 4 


4 2 2 
 3  2  1

L L L A U 

2 5 



3



Exercise: check L  L
1
 2
1
3
a131
 2
a22
 2
a23
0
 3
a33
0
0

U  LU
 3  2 1
A L L L
   L   L  
1
a121
1
a141   a111
 
 2
a24   0

 3  
a34   0
 4  
a44
  0
 1



1
2
1


 0.5
3
1 


 1 0.5 2 1 
Question: Do you have any effective method to write down matrix L
Question: why don’t we care about right hand side vector b


 2
a22

0
 3
a33
0
0
LU-decomposition
 

 

 

A 4 
Problems about LU-decomposition
A  LU
 6 2

 12 8
 3 13

 6 4
2 4   1
 6 2 2 4 
 


6 10   2
1

4
2
2



9 3   0.5
3
1 
2 5 
 


1 18   1 0.5 2 1 
3 
Question 1: what condition does LU-decomposition fail?
Question 2: does any invertible matrix has LU-decomposition?
1 0 0


A  0 0 1
0 1 0


A  LU ?
Question 3: How to measure “goodness of LU-decomposition“?
Question 4: what is order of performance of LU-decomposition (operation count)?
Question 5: How to parallelize LU-decomposition?
Question 6: How to implement LU-decomposition with help of GPU?
OutLine
•
•
•
•
Basic operation of matrix
Example of Gaussian Elimination (GE)
Formal description of GE
MATLAB usage
- website resource
- “help” command
- M-file
MATLAB website
http://www.mathworks.com/access/helpdesk/help/techdoc/matlab.html
MATLAB: create matrix
MATLAB: LU-factorization
Start MATLAB 2008
Command “help” : documentation
!= in C
Loop in MATLAB
Decision-making in MATLAB
IO in MATLAB
LU-factorization in MATLAB
A  LU
PA  LU
P : permutation
PAQ  LU P, Q : permutation
M-file in MATLAB
matvec.m
Description of function matvec, write
specificaiton of input parameter
Consistency check
compute b  Ax
Inner-product based
for i  1: 4
for
j  1: 4
b  i   A  i, j  * x  j 
end
end
Move to working directory
working directory
M-file
Execute M-file
Construct matrix
Execute M-file, like function call in
C-language
Exercise: forward / backward substitution
 a11

a
A   21
 a31

 a41
a12
a13
a22
a23
a32
a33
a42
a43
Ax  b
a14 
 l11


a24 
l
l
 LU   21 22
 l31 l32
a34 


a44 
 l41 l42
LUx  b
l33
l43
  u11 u12

u22



l44  
u23
u33
u14 

u24 
u34 

u44 
y  L1b
x  U 1 y
y  L1b
since
y1  y2  y3  y4
backward substitution x  U 1 y
since
x4  x3  x2  x1
Forward substitution
u13
Write MATLAB code to do forward substitution and backward substitution
Exercise: LU-decomposition
 6 2

12 8
A
 3 13

 6 4
2 4 

6 10 
9 3 

1 18 
 1




2
1
1

L   
 0.5
1 


1
 1
 6 2 2 4 



4
2
2
 3  2  1

L L L A U  

2 5 


3 

1



1
 2 

L 

3 1 


1
 0.5

A  L3 L 2 L1

1
U  LU
1



1
 3 

L 


1


2 1 

LU-decomposition
You should find recursive structure of the decomposition