Linear Algebraic Equations
Part 3
• An equation of the form ax+by+c=0 or equivalently ax+by=c is called a linear equation in x and y variables.
• ax+by+cz=d is a linear equation in three variables, x, y, and
z.
• Thus, a linear equation in n variables is
a1x1+a2x2+ … +anxn = b
• A solution of such an equation consists of real numbers c1, c2,
c3, … , cn. If you need to work more than one linear
equations, a system of linear equations must be solved
simultaneously.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
1
Noncomputer Methods for Solving
Systems of Equations
• For small number of equations (n ≤ 3) linear
equations can be solved readily by simple
techniques such as “method of elimination.”
• Linear algebra provides the tools to solve such
systems of linear equations.
• Nowadays, easy access to computers makes
the solution of large sets of linear algebraic
equations possible and practical.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
2
Gauss Elimination
Chapter 9
Solving Small Numbers of Equations
• There are many ways to solve a system of
linear equations:
–
–
–
–
Graphical method
Cramer’s rule
Method of elimination
Computer methods
by Lale Yurttas, Texas
A&M University
For n ≤ 3
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
3
Graphical Method
• For two equations:
a11 x1  a12 x2  b1
a21 x1  a22 x2  b2
• Solve both equations for x2:
 a11 
b1


x2  
x1 

a12
 a12 

x2  (slope) x1  intercept
 a21 
b2


x2  
x1 

a22
 a22 
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
4
• Plot x2 vs. x1
on rectilinear
paper, the
intersection of
the lines
present the
solution.
by Lale Yurttas, Texas
A&M University
Fig. 9.1
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
5
Figure 9.2
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
6
Determinants and Cramer’s Rule
• Determinant can be illustrated for a set of three
equations:
Ax  B
• Where [A] is the coefficient matrix:
a11 a12 a13 


A  a21 a22 a23 
a31 a32 a33 
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
7
• Assuming all matrices are square matrices,
there is a number associated with each square
matrix [A] called the determinant, D, of [A].
If [A] is order 1, then [A] has one element:
[A]=[a11]
D=a11
• For a square matrix of order 3, the minor of
an element aij is the determinant of the matrix
of order 2 by deleting row i and column j of
[A].
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
8
a11 a12 a13
D  a21 a22 a23
a31 a32 a33
D11 
a22 a23
D12 
a21 a23
D13 
a21 a22
a32 a33
a31 a33
a31 a32
by Lale Yurttas, Texas
A&M University
 a22 a33  a32 a23
 a21 a33  a31 a23
 a21 a32  a31 a22
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
9
D  a11
a22 a23
a32 a33
 a12
a21 a23
a31 a33
 a13
a21 a22
a31 a32
• Cramer’s rule expresses the solution of a
systems of linear equations in terms of ratios
of determinants of the array of coefficients of
the equations. For example, x1 would be
computed as:
b1 a12 a13
b2 a22 a23
x1 
by Lale Yurttas, Texas
A&M University
b3 a32 a33
D
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
10
Method of Elimination
• The basic strategy is to successively solve one
of the equations of the set for one of the
unknowns and to eliminate that variable from
the remaining equations by substitution.
• The elimination of unknowns can be extended
to systems with more than two or three
equations; however, the method becomes
extremely tedious to solve by hand.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
11
Naive Gauss Elimination
• Extension of method of elimination to large
sets of equations by developing a systematic
scheme or algorithm to eliminate unknowns
and to back substitute.
• As in the case of the solution of two equations,
the technique for n equations consists of two
phases:
– Forward elimination of unknowns
– Back substitution
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
12
Fig. 9.3
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
13
Pitfalls of Elimination Methods
• Division by zero. It is possible that during both
elimination and back-substitution phases a division
by zero can occur.
• Round-off errors.
• Ill-conditioned systems. Systems where small changes
in coefficients result in large changes in the solution.
Alternatively, it happens when two or more equations
are nearly identical, resulting a wide ranges of
answers to approximately satisfy the equations. Since
round off errors can induce small changes in the
coefficients, these changes can lead to large solution
errors.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
14
• Singular systems. When two equations are
identical, we would loose one degree of
freedom and be dealing with the impossible
case of n-1 equations for n unknowns. For
large sets of equations, it may not be obvious
however. The fact that the determinant of a
singular system is zero can be used and tested
by computer algorithm after the elimination
stage. If a zero diagonal element is created,
calculation is terminated.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
15
Techniques for Improving Solutions
• Use of more significant figures.
• Pivoting. If a pivot element is zero,
normalization step leads to division by zero.
The same problem may arise, when the pivot
element is close to zero. Problem can be
avoided:
– Partial pivoting. Switching the rows so that the
largest element is the pivot element.
– Complete pivoting. Searching for the largest
element in all rows and columns then switching.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
16
Gauss-Jordan
• It is a variation of Gauss elimination. The
major differences are:
– When an unknown is eliminated, it is eliminated
from all other equations rather than just the
subsequent ones.
– All rows are normalized by dividing them by their
pivot elements.
– Elimination step results in an identity matrix.
– Consequently, it is not necessary to employ back
substitution to obtain solution.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
17
LU Decomposition and Matrix Inversion
Chapter 10
• Provides an efficient way to compute matrix
inverse by separating the time consuming
elimination of the Matrix [A] from
manipulations of the right-hand side {B}.
• Gauss elimination, in which the forward
elimination comprises the bulk of the
computational effort, can be implemented as
an LU decomposition.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
18
If
L- lower triangular matrix
U- upper triangular matrix
Then,
[A]{X}={B} can be decomposed into two matrices [L] and
[U] such that
[L][U]=[A]
[L][U]{X}={B}
Similar to first phase of Gauss elimination, consider
[U]{X}={D}
[L]{D}={B}
– [L]{D}={B} is used to generate an intermediate vector {D}
by forward substitution
– Then, [U]{X}={D} is used to get {X} by back substitution.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
19
Fig.10.1
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
20
LU decomposition
• requires the same total FLOPS as for Gauss
elimination.
• Saves computing time by separating timeconsuming elimination step from the
manipulations of the right hand side.
• Provides efficient means to compute the matrix
inverse
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
21
Error Analysis and System Condition
•
Inverse of a matrix provides a means to test
whether systems are ill-conditioned.
Vector and Matrix Norms
• Norm is a real-valued function that provides a
measure of size or “length” of vectors and
matrices. Norms are useful in studying the error
behavior of algorithms.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
22
A simple example is a vector in three - dimensiona l Euclidean space
that can be represente d as
F   a b c 
where a, b, and c are the distances along x, y, and z axes, repectivel y.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
23
Figure 10.6
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
24
• The length of this vector can be simply computed as
F e  a b c
2
2
2
Length or Euclidean norm of [F]
• For an n dimensional vector
 X   x1 x2  xn 
a Euclidean norm is computed as
Xe
n
2
x
 i
i 1
For a matrix [A]
Ae
n
Frobenius norm
n

by Lale Yurttas, Texas
i 1 j 1
A&M University
ai2, j
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
25
• Frobenius norm provides a single value to quantify the
“size” of [A].
Matrix Condition Number
• Defined as
Cond A  A  A
1
•For a matrix [A], this number will be greater
than or equal to 1.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
26
X
X
 Cond A
A
A
• That is, the relative error of the norm of the
computed solution can be as large as the relative
error of the norm of the coefficients of [A]
multiplied by the condition number.
• For example, if the coefficients of [A] are
known to t-digit precision (rounding errors~10-t)
and Cond [A]=10c, the solution [X] may be valid
to only t-c digits (rounding errors~10c-t).
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
27
Special Matrices and Gauss-Seidel
Chapter 11
• Certain matrices have particular structures that can be
exploited to develop efficient solution schemes.
– A banded matrix is a square matrix that has all elements equal to zero,
with the exception of a band centered on the main diagonal. These
matrices typically occur in solution of differential equations.
– The dimensions of a banded system can be quantified by two
parameters: the band width BW and half-bandwidth HBW. These two
values are related by BW=2HBW+1.
• Gauss elimination or conventional LU decomposition methods
are inefficient in solving banded equations because pivoting
becomes unnecessary.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
28
Figure 11.1
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
29
Tridiagonal Systems
• A tridiagonal system has a bandwidth of 3:
 f1
e
 2



g1
f2
e3
g2
f3
e4
  x1   r1 
  x  r 
  2    2 
g 3   x3  r3 
   
f 4   x4  r4 
• An efficient LU decomposition method, called Thomas
algorithm, can be used to solve such an equation. The
algorithm consists of three steps: decomposition, forward
and back substitution, and has all the advantages of LU
decomposition.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
30
Gauss-Seidel
• Iterative or approximate methods provide an
alternative to the elimination methods. The
Gauss-Seidel method is the most commonly
used iterative method.
• The system [A]{X}={B} is reshaped by solving
the first equation for x1, the second equation
for x2, and the third for x3, …and nth equation
for xn. For conciseness, we will limit ourselves
to a 3x3 set of equations.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
31
b1  a12 x2  a13 x3
x1 
a11
b2  a21x1  a23 x3
x2 
a22
b3  a31x1  a32 x2
x1 
a33
•Now we can start the solution process by choosing
guesses for the x’s. A simple way to obtain initial
guesses is to assume that they are zero. These zeros
can be substituted into x1equation to calculate a new
x1=b1/a11.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
32
• New x1 is substituted to calculate x2 and x3. The
procedure is repeated until the convergence criterion
is satisfied:
 a ,i
xij  xij 1

100%   s
j
xi
For all i, where j and j-1 are the present and previous
iterations.
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
33
Fig. 11.4
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
34
Convergence Criterion for GaussSeidel Method
• The Gauss-Seidel method has two fundamental
problems as any iterative method:
– It is sometimes nonconvergent, and
– If it converges, converges very slowly.
• Recalling that sufficient conditions for convergence
of two linear equations, u(x,y) and v(x,y) are
u u

1
x y
v v

1
x y
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
35
• Similarly, in case of two simultaneous equations,
the Gauss-Seidel algorithm can be expressed as
b1 a12
u ( x1 , x2 ) 

x2
a11 a11
b2 a21
v( x1 , x2 ) 

x1
a22 a22
u
0
x1
u
a
  12
x2
a11
v
a21

x1
a22
v
0
x2
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
36
• Substitution into convergence criterion of two linear
equations yield:
a12
1
a11
a21
1
a22
• In other words, the absolute values of the slopes
must be less than unity for convergence:
a11  a12
a22  a21
For n equations :
n
aii   ai , j
by Lale Yurttas, Texas
A&M University
j 1
j  iPart 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
37
Figure 11.5
by Lale Yurttas, Texas
A&M University
Part 3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
38