- No category
EMTH211 Week 1: Gaussian Elimination & Floating Point Arithmetic
advertisement
EMTH211 Week 1: Computation of Linear Systems
1
EMTH211-24S2 Engineering
Linear Algebra and Statistics
Lecture notes written by Mark Hickman, updated by Michael Langton and Nic Lam.
EMTH211 Week 1: Computation of Linear Systems
1.1
2
Gaussian Elimination
1.1.1
Linear Systems
Problem: Solve a linear system of equations
A x = b.
From last year, Gaussian elimination is an algorithm that will solve a system of
linear equations.
The description of this algorithm given last year assumed that calculations were
performed using exact arithmetic (e.g. working by hand). However this algorithm
needs to be modified when the calculations are performed using finite precision
floating point arithmetic (for example, by NumPy).
In EMTH211 we will be more systematic and careful about how we do Gaussian
elimination.
Algorithm: a process or set of rules to be followed in calculations or other problemsolving operations, especially by a computer. Programs implement algorithms.
EMTH211 Week 1: Computation of Linear Systems
1.1.2
3
Gaussian elimination – a refresher
Given a system of linear equations A x = b, Gaussian elimination solves this system
in two steps:
1. Row Reduction:
Row operations are used to row reduce the augmented
matrix A b to row echelon form (see below).
2. Back Substitution is then used on the row reduced matrix to solve the
system of equations.
A matrix is in row echelon form (REF) when
• All rows consisting entirely of zeros are at the bottom.
• In each non-zero row, the first non-zero entry is in a column to the left of any
lower row’s first non-zero entry.
In a matrix in row echelon form, the first non-zero entry in each row is called a
pivot. These matrices are in row echelon form (pivots in red boxes):
1 2
0 1 2
1 2 3
1 2 3
1 2
0 0
0 0 4
0 4 5
0 3
0 0 3
An important subset of matrices in row echelon form is upper triangular matrices, which are square matrices in REF – all elements below the main diagonal are
zero.
EMTH211 Week 1: Computation of Linear Systems
1.1.3
4
Gaussian elimination – Row Reduction
It is essential that the row operations we do to row reduce leave the solution
unchanged. Since we will be analysing how a computer performs this algorithm,
we will stick to what are called elementary row operations (other safe row
operations are just combinations of these).
There are three types of elementary row operation:
• Add a multiple of one row to another: R ← R + α Rj , where = j.
• Swap two rows: R ↔ Rj , where = j.
• Multiply a row by a non-zero scalar β: R ← β R .
It is mostly the first two operations that we use in row reduction.
5
EMTH211 Week 1: Computation of Linear Systems
Gaussian elimination proceeds column by column, left to right, and within each
column from just below that column’s pivot down to the bottom. We may need to
swap rows to get an appropriate pivot into the right position.
11 12 13 · · · 1n b1
21 22 32 · · · 2n b2
A b =
31 32 33 · · · 3n b3
..
..
.. . .
..
..
.
.
.
. .
.
We are starting with the first column, so the first element we need to eliminate is
21 . We can do this by
21
R2 ← R2 −
R1 .
11
This will update the entire second row (including the RHS part).
We now wish to eliminate 31 , which we do with
R3 ← R3 −
31
11
R1 ,
and so on down the first column.
6
EMTH211 Week 1: Computation of Linear Systems
After the first column is done, we have
11 12 13 · · · 1n b1
0
̃22 ̃23 · · · ̃2n b̃2
.
A b −→
̃32 ̃33 · · · ̃3n b̃3
0
..
..
..
..
.. . .
.
. .
.
.
.
We now wish to eliminate ̃32 . We do this by
R3 ← R3 −
̃32
̃22
R2
and so on down the second column.
Repeating this procedure on the remaining columns of A, we eventually obtain the
row reduced form of the augmented matrix.
7
EMTH211 Week 1: Computation of Linear Systems
Notes on row reduction:
• At each step an entire row of the augmented matrix is updated.
• When we have finished our row reduction, our matrix will be in row echelon
form: each row’s first non-zero element will be further to the right than that
of the rows above.
• When working on a column, we add multiples of the row containing the column’s pivot to eliminate the elements below it.
• The variable corresponding to a column containing a pivot is called a pivot
variable, and its value can be solved for. The variable corresponding to a
column without a pivot is called a free variable, and its value is arbitrary.
• If our system of equations has n unknown variables, and all of them are pivot
variables, and the row-reduced augmented matrix has no pivots in the RHS
section, then the system has a unique solution. If the row-reduced augmented
matrix has a pivot in the RHS section, the system has no solutions.
• We will look more deeply at the question of whether a system of equations
has a solution, and how many solutions there are, later in the course.
8
EMTH211 Week 1: Computation of Linear Systems
1.1.4
Gaussian elimination – Back Substitution
After row reduction, we can find all solutions by back substitution. If all our
unknown variables are pivot variables and we have no RHS pivots, we have a
unique solution, and it is easy to tell a computer how to find it.
After row reduction, the linear system has the form
11 12 · · · 1,n−1
n
1
β1
0
22 · · · 2,n−1
2n 2 β2
.
. .
.
.
.
.
.. .. = .. .
..
..
..
.
Ux =
.
0 · · · n−1,n−1 n−1,n n−1 βn−1
0
n
βn
0
0 ···
0
nn
The last row gives
n =
βn
nn
.
The second last row gives
n−1 =
and so on.
1
n−1,n−1
βn−1 − n−1,n n =
1
n−1,n−1
βn−1 − n−1,n
βn
nn
EMTH211 Week 1: Computation of Linear Systems
1.2
9
Computer representations of numbers
Computer (and phone, etc.) CPUs, and also GPUs, have dedicated circuitry for
working with numbers at high speed. These numbers necessarily have limited
precision.
Integer types
CPUs can represent signed1 or unsigned finite-precision subsets of the integers.
Common integer types nowadays are:
• 8-bit bytes. Unsigned: 0 to 28 − 1 = 255, byte. Signed: −27 = −128 to
27 − 1 = +127, e.g. ASCII character (C char).
• 32-bit. Unsigned: 0 to 232 − 1, e.g. IP address, pointer on 32-bit CPU. Signed:
−231 = −2,147,483,648 to 231 − 1 = +2,147,483,647, e.g. traditional Unix
time (seconds since 1970-01-01 00:00, subject to the Year 2038 problem2 ).
• 64-bit. Unsigned: 0 to 264 − 1, e.g. pointer on 64-bit CPU. Signed: −263 to
263 − 1, e.g. updated Unix time (still seconds since 1970-01-01 00:00).
The Python int type is arbitrary precision (emulated in software) and not one of
the above, which means integer operations with Python’s native int are quite
slow, but don’t really have an upper or lower limit.
1 using “two’s complement”: https://en.wikipedia.org/wiki/Two%27s_complement
2 https://en.wikipedia.org/wiki/Year_2038_problem
EMTH211 Week 1: Computation of Linear Systems
10
Any of these integer types can be used to represent fixed point numbers, which
have fixed whole number precision and fixed fractional precision. These have limited practical use, however; one current example is representing dollar values in
cents, and historically some computer games used them for speed.
Floating point types
For most applications, floating point numbers are essential, and 32-bit and 64-bit
floating point numbers have been in widespread use for many decades.
Floating point numbers are represented by a sign, a significand, and an exponent, and can represent a much larger range of values than fixed point, mostly to
higher precision. The significand holds all the significant digits and the exponent
changes the scale, moving the floating decimal point wherever it needs to go.
• 16-bit (half precision). Relatively new, but supported on some GPUs as it’s
good for some machine learning tasks. 1 sign bit, 5 exponent bits, 11 significand bits (10 explicitly stored).
• 32-bit (single precision). Available in NumPy as np.float32. 1 sign bit, 8
exponent bits, 24 significand bits (23 explicitly stored).
• 64-bit (double precision). Python float, NumPy’s default np.float64.
≈ ±10−308 to ≈ ±10308 , with 15-17 decimal digits of precision. 1 sign bit,
11 exponent bits, 53 significand bits (52 explicitly stored).
11
EMTH211 Week 1: Computation of Linear Systems
1.2.1
Consequences of limited precision
• Most real numbers cannot be represented exactly:
>>> f’{0.1:.100g}’
’0.1000000000000000055511151231257827021181583404541015625’
>>> f’{1/3:.100g}’
’0.333333333333333314829616256247390992939472198486328125’
>>> np.sin(np.pi)
1.2246467991473532e-16
>>> np.pi == 884279719003555/281474976710656
True
>>> (1e308, 1e309)
(1e+308, inf)
• Round off error causes information
loss in almost every operation:
• Errors can be magnified:
>>> .2
0.2
>>> .1
0.1
>>> .2+.1
0.30000000000000004
>>> np.sin(1000000*np.pi)
-2.231912181360871e-10
>>> x = 1.
for i in range(10):
x -= .1
x
1.3877787807814457e-16
>>> x = 10.
for i in range(100):
x -= .1
x
1.8790524691780774e-14
>>> 1.+1e-16 == 1.
True
• Comparing numbers for exact
equality is not sensible:
>>> .3 == .1+.2
False
>>> np.allclose(.3, .1+.2)
True
12
EMTH211 Week 1: Computation of Linear Systems
1.3
Gaussian elimination – What can go wrong?
When examining errors by hand we will work in base 10, to a limited number of
decimal significant figures (unlike the computer’s base 2).
Consider the system
0.011 + 2 = 1
1 − 2 = 0.
Note that the coefficients and right hand side values can be represented exactly
with two-digit numbers (for example, 0.01 is represented as 1.0 × 10−2 )
Exact Arithmetic:
• Row Reduction:
0.01 1 1 R2 ←R2 −100R1 0.01
1
1
−−−−−−−−→
1
−1 0
0
−101 −100
exact
• Back Substitution:
100
x=
101
100
101
=
0.990099 . . .
0.990099 . . .
.
The best we could expect to obtain from a two-digit computer would be the closest
two-digit numbers to the exact solution; namely
0.99
x≈
.
0.99
13
EMTH211 Week 1: Computation of Linear Systems
So what happens on our hypothetical two-digit computer?
Finite Precision Arithmetic: (on a two-digit computer)
• Row Reduction:
0.01 1 1 R2 ←R2 −100R1 0.01
1
1
−−−−−−−−→
1
−1 0
0
−100 −100
two-digit
since, to two digits,
−1 − 100 = − 1.0 × 100 − 1.0 × 102 = −1.0 × 102 = −100.
• Back Substitution:
0
.
x≈
1
While we still have a good approximation for 2 , 1 = 0 which is a poor (to say the
least) approximation for 0.99.
However if we do a row swap first, then the computation becomes:
• Row Reduction:
• Back Substitution:
1
−1 0 R2 ←R2 −0.01R1 1 −1 0
−−−−−−−−→
0.01 1 1
0 1 1
two-digit
1
x≈
1
(!)
EMTH211 Week 1: Computation of Linear Systems
1.3.1
14
Gaussian elimination – The Moral
• This example demonstrates that, with finite precision arithmetic, different
pivots may lead to dramatically different answers through Gaussian elimination. Some of these answers may be acceptable but some can be totally
unacceptable.
• We need to develop a pivot selection strategy to avoid (if possible) bad
answers!
• But first we need to understand what is happening in this example.
15
EMTH211 Week 1: Computation of Linear Systems
1.3.2
Gaussian elimination – Backward Error Analysis
What system of equations do we need to solve with exact arithmetic in order to
obtain the same row reduced matrix as we computed with finite precision arithmetic?
Note that the first row of a matrix is unchanged by Gaussian elimination. In the
second (that is, good) case, we have
1
−1 0 R2 ←R2 −0.01R1 1 −1 0 R2 ←R2 −0.01R1 1
−1 0
−−−−−−−−→
←−−−−−−−−
0.01 1 1
0 1 1
0.01 0.99 1
exact
two-digit
We see that the two systems are close:
1 − 2 = 0
0.011 + 2 = 1
two-digit
1 − 2 = 0
0.011 + 0.992 = 1
exact
Just to reiterate, the second system is the system that we would need to solve
using exact arithmetic to obtain the same solution as solving the first system by
two-digit arithmetic.
Therefore the effect of using a two-digit computer to perform Gaussian elimination
is to solve a slightly different (that is perturbed) system.
16
EMTH211 Week 1: Computation of Linear Systems
What about the first case which gave us such a bad answer?
R2 ←R2 −100R1 0.01 1 1
0.01 1 1 R2 ←R2 −100R1 0.01
1
1
−−−−−−−−→
←−−−−−−−−
1
−1 0
0
−100 −100
1
0 0
exact
two-digit
In this case the two systems are not close:
0.011 + 2 = 1
1 − 2 = 0
two-digit
0.011 + 2 = 1
1 = 0
exact
• In both cases, the computed solution may be viewed as the exact solution to
a perturbed system.
• This approach is called backward error analysis. The error in the computation is pushed back from the computation and placed on the data.
• It is an useful approach since frequently data is already inaccurate through
experimental or modelling errors.
EMTH211 Week 1: Computation of Linear Systems
17
Ideally we would like that errors created by the computer solution would be no
larger than the errors inherent in the data.
• Thus we would like the computer solution to be an exact solution to a slightly
perturbed problem.
• However this example demonstrates that Gaussian elimination, unmodified,
may result in the solution to a greatly perturbed problem.
In both row reduction and backwards substitution we are dividing by pivots. Therefore if a pivot is small, any errors will be magnified.
• In the bad case, the pivot was 0.01. Therefore, each time we divide by the
pivot, errors could be magnified by a factor of 1/ 0.01 = 100 (and this does
not allow for errors in the pivot itself).
• In the good case, the pivot was 1. Now errors are not magnified.
This suggests the strategy avoid small pivots.
EMTH211 Week 1: Computation of Linear Systems
1.3.3
Gaussian elimination – Small Pivots
But (and there is always a but!) what about the rescaled system
101 + 10002 = 1000
1 − 2 = 0?
In this case we obtain
10 1000 1000
10 1000 1000
−−−−→
1
−1
0
0 −100 −100
two-digit
the same bad result.
Here the pivot is 10 so the bad result is not a consequence of a small pivot.
If we do a row swap and use the smaller pivot, we obtain
1
−1
0
1 −1
0
−−−−→
10 1000 1000 two-digit 0 1000 1000
and thus the good result.
18
EMTH211 Week 1: Computation of Linear Systems
1.3.4
19
Gaussian elimination – Scaling
However, in a properly scaled system, we should avoid small pivots.
Ideally, the system should be scaled before it is given to the computer to solve.
• We can scale an equation by multiplying by a non-zero constant. This is equivalent to an elementary row operation.
• We can also scale the variables of a problem
−→ α
where α are non-zero constants. This does not correspond to an elementary
row operation (in fact, it is an elementary column operation). This is like
changing units.
There is no strategy that will work in all situations. However one should use scaling,
if possible, to avoid “large” and very small non-zero numbers in the system. The
errors in the example above arose from adding a large number to a small number
and the consequent loss of precision when using finite precision arithmetic.
EMTH211 Week 1: Computation of Linear Systems
20
For example, consider
1 − 0.00012 = 0
1001 + 2 = 100.
One might
• scale the second equation
1 − 0.00012 = 0
1 + 0.012 = 1
• and scale 2 −→ 0.012 = ̃2 say
1 − 0.01̃2 = 0
1 + ̃2 = 1.
In the original system, the coefficients varied by six orders of magnitude; in the
scaled system, they varied by only two orders of magnitude.
21
EMTH211 Week 1: Computation of Linear Systems
1.3.5
Gaussian elimination – Partial Pivoting
Again there is no pivoting strategy that will work in all situations.
From a practical point of view, partial pivoting is the strategy most commonly
used (NumPy uses it by default). It balances low computational overhead with
accuracy of the final answer.
With partial pivoting, we swap rows so that the pivot always has the largest
absolute value of all remaining elements in that column.
For example, in the third column, we search the boxed area for the element with
largest absolute value. We then use a row operation to move this element to the
pivot position.
∗ ∗ ∗ ∗ ··· ∗ ∗
0 ∗ ∗ ∗ ··· ∗ ∗
0 0 ∗ ∗
··· ∗ ∗
0 0 ∗ ∗
··· ∗ ∗
..
..
..
.. . .
..
..
. .
.
.
.
.
.
·
·
·
∗ ∗
0 0 ∗ ∗
22
EMTH211 Week 1: Computation of Linear Systems
1.3.6
Gaussian elimination – Complete Pivoting
An alternative is complete pivoting.
In this case we search the submatrix
∗ ∗ ∗ ∗ ··· ∗
0 ∗ ∗ ∗ ··· ∗
0 0 ∗ ∗ ··· ∗
0 0 ∗ ∗ ··· ∗
..
..
.. .. . . ..
.
. .
.
. .
0 0 ∗ ∗ ··· ∗
∗
∗
∗
∗
..
.
∗
for the largest (in absolute value) entry. We then use row and column swaps to
bring this element to the pivot position. Note that row swaps just reorder equations, so nothing really changes, but column swaps rename variables.
While this strategy is better than partial pivoting, its use is rarely justified since
the computational overhead is significantly higher.
EMTH211 Week 1: Computation of Linear Systems
1.3.7
23
Gaussian elimination – Example of Partial Pivoting
Consider the system whose augmented matrix is
1 0 3 −1 1
2 1 0 0
1
.
−4
0 −2 0 1
−1 2 2 0 −1
• Column 1:
– We want −4 moved to the pivot position so do R1 ↔ R3 .
– Gaussian elimination row operations
−4
0 −2 0 1
2
2 1 0 0
R ← R2 − −4
R1
1
2
1
1 0 3 −1
1 R3 ← R3 − −4 R1
−1 2 2 0 −1 R4 ← R4 − −1 R1 .
−4
−4 0 −2 0
1
0 1 −1 0
1.5
0 0 2.5 −1 1.25
0 2 2.5 0 −1.25
EMTH211 Week 1: Computation of Linear Systems
• Column 2:
– We want 2 moved to the pivot position so do R2 ↔ R4 .
– Gaussian elimination row operation
−4 0 −2 0
1
0 2 2.5 0 −1.25
0 0 2.5 −1 1.25
0 1 −1 0
1.5
R4 ← R4 − 12 R2
which completes the second column:
−4 0
−2
0
1
0 2
2.5
0 −1.25
.
0 0
2.5 −1 1.25
0 0 −2.25 0 2.125
24
25
EMTH211 Week 1: Computation of Linear Systems
• Column 3:
– No row swaps required.
– Gaussian elimination row operation
R4 ← R4 −
−2.25
2.5
R3
which gives the row reduced form
−4 0 −2
0
1
0 2 2.5
0
−1.25
.
0 0 2.5
−1
1.25
0 0 0 −0.9 3.25
26
EMTH211 Week 1: Computation of Linear Systems
1.3.8
Gaussian elimination – Another Example
Solve the system whose augmented matrix is
1
1 2
3
2 4.01
2
3
using a three-digit computer.
1
The exact solution is x =
3
0
.
With three-digit arithmetic, the augmented matrix becomes
1 2
0.333
2 4.01 0.667
To obtain a three-digit representation of this system, we are forced to introduce
inaccuracies (albeit small) in the vector b.
So what happens?
27
EMTH211 Week 1: Computation of Linear Systems
Gaussian elimination with partial pivoting gives
1
1 2
0.333 R1 ↔R2 2 4.01 0.667 R2 ←R2 − 2 R1 2
4.01
0.667
−−−−→
−−−−−−−→
2 4.01 0.667
1 2
0.333
0 −0.0100 −0.00100
three-digit
Note that the outcome of the row operation R2 ← R2 − 12 R1 depends on the order
in which the arithmetic operations are performed.
• If we compute 12 (2R2 − R1 ) we obtain
4 − 4.01
2
=
−0.0100
2
= −0.00500,
• but if we compute R2 − 12 R1 we obtain
2−
4.01
2
= 2 − 2.01 = −0.0100.
Finally the computed solution is
x≈
0.133
.
0.100
EMTH211 Week 1: Computation of Linear Systems
28
Notes:
• Partial pivoting does not help in this case. The inaccuracies in the data are
less than 0.1%. However
0.133
0.333
0.100
0.000
computed solution
exact solution
(to 3 significant figures)
• This is an example of an ill-conditioned system.
• Geometrically, this system represents two lines that are almost parallel.
• We will revisit ill-conditioned systems later in this course.
29
EMTH211 Week 1: Computation of Linear Systems
1.3.9
Gaussian elimination – Ill-conditioned systems
Here is an exaggerated view of what’s going on. The solid lines are the exact
equations, and their intersection is the exact solution. The dashed lines are the
equations with rounded RHS, and their intersection is the computed solution.
2
sol. computed
with
round-off
0.133
0.100
11 1 + 12 2 = b1
21 1 + 22 2 = b2
11 1 + 12 2 = b̃1
21 1 + 22 2 = b̃2
1/ 3
exact sol.
0
1
The lines are nearly parallel, so a small error can shift the intersection considerably.
EMTH211 Week 1: Computation of Linear Systems
1.4
30
How fast is fast?
We have discussed accuracy issues for Gaussian elimination but what about speed?
Is it a fast algorithm or do we need another algorithm?
First we need a measure of how fast an algorithm is.
Given that an algorithm is used on a computer (rather than by us), we measure the
speed of a algorithm in terms of the number of slow operations that a computer
has to perform.
The slowest operations are the floating point operations; multiplication/division
and addition/subtraction.
Floating point operations are measured in terms of flops.
Flops per second is one measure that is used to benchmark computer performance.
• The fastest supercomputer today (Frontier) has performance in the 1 exaflop/sec (1018 flop/sec) range.
• The fastest Intel or AMD processors have performance around 1 teraflop (1012
flop)/sec.
• Graphic processing units (GPUs) used on dedicated graphics cards are, for
double precision floating point, in the 1-60 teraflop range (consumer vs. datacentre).
So what is the flop count for Gaussian elimination and should it concern us?
31
EMTH211 Week 1: Computation of Linear Systems
1.4.1
Gaussian elimination – flop count
Two formulas that will help us in the flop count are
n
∑
=
=1
1 2
n +n
2
n
∑
and
2 =
=1
1
3
n3 +
1
2
n2 +
1
6
n.
Row Reduction:
• First Column: We use row operations of the form
R ← R − 1 ×
R1
11
.
• We need to divide (that is multiply!) each element in R1 by 11 . However we
do not need to do this for the first element in this row and so there are n
multiplications to perform (remember though there are only n − 1 remaining
elements in the coefficient matrix, we also have the element b1 in the augmented matrix). We update the first row so that the pivot element is now
1.
• Each remaining row we need to perform one addition and one multiplication
per element. Again we do not need to do this for the first element since we
know that this element is 0 by construction. Thus there are n multiplications
and n additions to perform.
32
EMTH211 Week 1: Computation of Linear Systems
• Since there are n − 1 rows we need to update we have a total count
(n − 1)n + n = n2 multiplication
and
(n − 1)n addition
operations to perform. Thus 2n2 − n flops are required.
• Remaining Columns: The count is the same for remaining columns except
that, for the jth column, we are performing the operations on a (n−j)×(n−j+1)
submatrix. Thus the flop count is
for j = 2, . . . , n.
2(n − j + 1)2 − (n − j + 1)
The total flop count for row reduction is
n
∑
j=1
n
∑
2
2(n − j + 1)2 − (n − j + 1) =
2 −
( = n − j + 1)
=1
= 23 n3 + 12 n2 − 16 n
≈ 23 n3
for large n. We are only interested in large n, so we write
Row Reduction = O 23 n3 flops, as n → ∞.
33
EMTH211 Week 1: Computation of Linear Systems
After the above row reduction, the linear system has the form
1 12 13 · · · 1,n−1 1n
1
β1
0 1 23 · · · 2,n−1 2n 2 β2
. .
. .
.. . .
..
..
. .
. .
Ux =
.
.
.
.
. .
. = .
0 0
β
0 ···
1
0
0
0
···
0
n−1,n
1
n−1
n
n−1
βn
The last row gives
n = βn .
The second to last row gives
and so on.
n−1 = βn−1 − n−1,n n
EMTH211 Week 1: Computation of Linear Systems
34
Back Substitution: In the (n − j + 1)th row (that is, the jth row from the bottom)
we need (j − 1) multiplications and (j − 1) additions (check; note after the above
row reduction, the pivot elements are all 1). Therefore the flop count is
n
∑
j=1
We write
2(j − 1) = n2 − n
Back Substitution = O n2 flops.
The total flop count for Gaussian elimination without pivoting is
Gaussian Elimination = O 23 n3 flops,
since n2 n3 for large n.
EMTH211 Week 1: Computation of Linear Systems
35
What about pivoting? It takes one addition to compare two numbers (computers
compute the difference and then check the sign).
Partial Pivoting: In the jth column we need to compare n − j + 1 elements to
obtain the pivot. This involves n − j comparisons. Therefore the flop count is
n
∑
j=1
or
∑
n−1
n−j =
= 12 n2 − 12 n
=0
Partial Pivoting = O 12 n2 flops.
Complete Pivoting: In the jth column we need to compare (n − j + 1)2 elements
to obtain the pivot. This involves (n − j + 1)2 − 1 comparisons. Therefore the flop
count is
n
n
∑
∑
2
(n − j + 1)2 − 1 =
− 1 = 13 n3 + 12 n2 − 56 n
j=1
or
=1
Complete Pivoting = O 13 n3 flops.
While partial pivoting does not increase the flop count significantly, complete pivoting does.
Gaussian Elimination with partial pivoting = O 23 n3 flops.
Gaussian Elimination with complete pivoting = O n3 flops.
EMTH211 Week 1: Computation of Linear Systems
36
Notes:
• As a comparison we have (check if you like)
Gauss-Jordan Elimination = O n3
Computing A−1 = O 2n3
Multiplying two n × n matrices = O 2n3
• A generic Gaussian elimination routine would take O 83 n3 to compute the
inverse. The algorithm has to use the fact that there are many zeros in the
identity matrix to obtain the above speed. Computing the inverse is the least
efficient method of solving a system of linear equations.
• The fact that row reduction grows as cube of the number of variables is a very
big issue. If we double the size of the problem, then the problem will take at
least 8 times longer to solve.
37
EMTH211 Week 1: Computation of Linear Systems
1.4.2
Is speed an issue?
• Problems with a million data points (equations) are pretty common in the real
world. Problems with a billion data points are not hard to find either.
Say our problem has 106 equations, with 106 unknown variables. If we used
Gaussian elimination, how long would it take?
• The flop count would be
0.67 × 1018 .
• That is, on a
supercomputer
state of the art graphics processor
state of the art PC
0.67 sec
67,000 sec (18 hours)
670,000 sec (8 days)
... if you had a GPU or PC with (106 )2 floats×8 bytes/float / 230 bytes/gigabyte ≈
7500 gigabytes of RAM!
• Therefore Gaussian elimination is NOT a fast algorithm.
• More precisely, generic Gaussian elimination is not fast. When exploiting
knowledge of matrix structure (e.g. storing tridiagonal systems as sparse matrices), it can be fast.
EMTH211 Week 1: Computation of Linear Systems
1.5
38
Matrix Algebra
1.5.1
Matrix terminology and notation
Definition 1: Matrix
A matrix is a rectangular array of numbers (real or complex) called the elements
or entries of the matrix. A matrix with m rows and n columns is called a m × n
matrix. A 1 × n matrix is called a row vector and a m × 1 matrix is called a
column vector. A 1 × 1 matrix is called a scalar.
For a m × n matrix, A we write
11 12 · · · 1n
21 22 · · · 2n
A=
. = j
.. . .
...
. ..
.
m1 m2 · · · mn
where j are the elements of A.
Convention: We will use the convention that a vector written x will be a column
vector. A row vector will be written as the transpose of a column vector; i.e. xT .
39
EMTH211 Week 1: Computation of Linear Systems
The columns of A are denoted by the column vectors 1 , 2 , . . . , n with
1j
2j
j =
... .
mj
We write
A = 1 2 · · · n
in order to emphasise the column structure of A.
The rows of A are denoted by the row vectors AT1 , AT2 , . . . , ATm with
AT = 1 2 · · · n .
We write
AT1
T
A
2
A=
..
.
ATm
in order to emphasise the row structure of A.
EMTH211 Week 1: Computation of Linear Systems
40
We see that the transpose of A is given by
T
1T
2
AT =
.. = A1 A2 · · · Am = j .
.
Tn
In EMTH118/9 (and MATH199), the focus was on the elements of a matrix. In this
course we will focus more on the structure of a matrix; that is, on the columns and
rows of a matrix. The basic matrix operations can be interpreted in terms of the
column and row structure.
41
EMTH211 Week 1: Computation of Linear Systems
Let ej be the jth standard basis vector
0
..
.
0
ej = 1 ← jth.
0
.
..
0
Then eT will be the (row) vector eT =
th
0 ··· 0 1 0 ··· 0 .
The identity matrix can be written
eT1
T
e2
=
.. = e1 e2 · · · en .
.
eTn
42
EMTH211 Week 1: Computation of Linear Systems
1.5.2
Matrix Addition and multiplication by scalars
Let A and B be both m × n matrices. Then
A + B = j + bj
= 1 + b1 2 + b2 · · · n + bn
AT1 + BT1
T
A + BT
2
2
=
..
.
T
T
Am + Bm
Let A be a m × n matrix and c be a scalar.
cA = cj
= c1 c2 · · · cn
cAT1
cAT
2
= .
..
cATm
elements
columns
rows
elements
columns
rows
43
EMTH211 Week 1: Computation of Linear Systems
1.5.3
Matrix Multiplication
Let A be a m × n matrix and B be a n × p matrix. The product C = AB will be a m × p
matrix with
n
∑
cj = 1 b1j + 2 b2j + · · · + n bnj =
k bkj
k=1
In terms of the structure of A and B, the entry cj is given by
bj
11 · · · 1n
↓
..
.. . .
. . b
.
·
·
·
b
11
1j · · · b1p
T
A → 1 · · · n .. . .
. ..
.
. .
.
. ..
. ..
. . ...
..
bn1 · · · bnj · · · bnp
m1 · · · mn
and so
cj = AT bj .
This is the “dot product” of the th row of A and the jth column of B. For (column)
vectors x and y, the dot product is given by
x · y ≡ xT y = 1 y1 + · · · + n yn .
44
EMTH211 Week 1: Computation of Linear Systems
Therefore
AB = AT bj
= Ab1 Ab2 · · · Abn
AT1 B
T
A B
2
=
..
.
ATm B
elements
columns
rows
45
EMTH211 Week 1: Computation of Linear Systems
Note
0
..
.
Aej = 1 2 · · · n ej = 1 2 · · · n 1
.
..
0
= 01 + · · · + 1j + · · · + 0n = j .
Therefore Aej is the jth column of A.
Also,
1
2
Ax = 1 2 · · · n
... = 1 1 + 2 2 + · · · + n n .
n
Therefore Ax is a linear combination of the columns of A.
Since AB = Ab1 Ab2 · · · Abn , every column of AB is a linear combination of the
columns of A.
46
EMTH211 Week 1: Computation of Linear Systems
Similarly,
BT1
BT1
T
T
B2
B2
.
0
·
·
·
1
·
·
·
0
=
eT B = eT
..
.
.
.
T
Bn
BTn
= 0BT1 + · · · + 1BT + · · · + 0BTn = BT .
Therefore eT B is the th row of B.
Also,
BT1
T
B2
T
T
T
xT B = 1 2 · · · n
.. = 1 B1 + 2 B2 + · · · + n Bn .
.
BTn
Therefore xT B is a linear combination of the rows of B.
AT1 B
T
A B
2
Since AB =
.. , every row of AB is a linear combination of the rows of B.
.
ATm B
EMTH211 Week 1: Computation of Linear Systems
1.5.4
47
Outer Products
Combining these two ideas, we have
T
B1
T
B2
T
T
T
A B = 1 2 · · · n
.. = 1 B1 + 2 B2 + · · · + n Bn .
.
BTn
The products BT are matrices which are called outer products. This representation of the product is called the outer product expansion of A B.
A and B need not be square matrices. If A is m × n and B is n × p then the outer
products are m × p matrices.
EMTH211 Week 1: Computation of Linear Systems
Example 2. Evaluate A B by the outer product expansion where
4 6 −1
1 3 0
B = 1 0 −1
A=
2 1 4
2 5 1
48
EMTH211 Week 1: Computation of Linear Systems
1.6
49
Partitioned Matrices
• Matrix multiplication is a costly operation.
• While you might wonder about the use of outer products to compute a matrix
product (we will see a use for them later with the spectral theorem and SVD),
if a matrix has structure then we may be able to reduce the cost of multiplication by partitioning the matrix (for an artificial example, see Poole Example
3.12).
Block diagonal matrices occur frequently in applications. For example
2 3 5 0 0
1 1 4 0 0
A=
7 2 5 0 0
0 0 0 1 3
0 0 0 2 2
is a block diagonal matrix.
EMTH211 Week 1: Computation of Linear Systems
50
We partition this matrix
2 3 5 0 0
1 1 4 0 0
7 2 5 0 0 = A1T O
A=
O A2
0 0 0 1 3
0 0 0 2 2
where O represents a 3× 2 zero matrix. (Often mathematicians use the same letter
O to represent zero matrices of different sizes in the same equation; be careful.)
In partitioned form A “looks like” a 2 × 2 matrix (albeit with matrix elements).
The blocks of 3 columns and 2 columns can differ in how their rows are partitioned
(but they don’t here).
In order to compute AB using this structure, we need to partition B into compatible
blocks; that is blocks that can be multiplied by the appropriate blocks in A.
EMTH211 Week 1: Computation of Linear Systems
51
Suppose B is a 5 × 8 matrix (and so the product will be 5 × 8 matrix). Since A1 is
3 × 3 we must partition
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
B
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ = 1
B=
B2
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
which looks like a 2 × 1 matrix.
The product becomes
A1 O B1
A1
O
A1 B1 + O B2
A1 B1
AB =
=
B
+
B
=
=
.
OT A2 B2
A2 2
OT 1
OT B1 + A2 B2
A2 B2
Rather than multiplying a 5 × 5 matrix by a 5 × 8 matrix, we have managed to find
the product by multiplying a 3 × 3 matrix by a 3 × 8 matrix and a 2 × 2 matrix by a
2 × 8 matrix. Note that A and B are structured as a kind of row vector and column
vector, respectively; we are treating full-column blocks of A as its elements, not
the smaller submatrices of A.
EMTH211 Week 1: Computation of Linear Systems
52
If B has structure then we can partition it vertically to take advantage of this structure.
Suppose
∗ ∗ ∗ ∗ ∗ 1 0 0
∗ ∗ ∗ ∗ ∗ 0 1 0
∗ ∗ ∗ ∗ ∗ 0 0 1 .
B=
1 0 ∗ ∗ ∗ ∗ ∗ ∗
0 1 ∗ ∗ ∗ ∗ ∗ ∗
We would then partition B to take advantage of these identity matrices.
B3 3
B=
2 B4
The blocks of 3 rows and 2 rows can differ in how their columns are partitioned.
The product now becomes
A1 O B3 3
A1
O
A1 B3 3 + O 2 B4
B
B
AB =
=
+
=
3 3
2
4
OT A2 2 B4
A2
OT
OT B3 3 + A2 2 B4
A1 B3 A1 3
A 1 B 3 3
A1 B3 A1
=
=
=
.
A2 A2 B4
A2 2 A2 B4
A2 2 B4
Always keep in mind that these are not scalar elements, but matrices with particular sizes, so not every operation that “looks okay” is valid.
53
EMTH211 Week 1: Computation of Linear Systems
1.7
Elementary Matrices
Elementary row operations can be performed by multiplying on the left by particular matrices, called elementary matrices.
There are three types of elementary row operations, each of which will lead to a
type of elementary matrix:
• Add a multiple of one row to another: R ← R + α Rj , where = j.
• Swap two rows: R ↔ Rj , where = j.
• Multiply a row by a non-zero scalar β: R ← β R .
1.7.1
Elementary matrix for R ← R + α Rj
Consider the row operation R2 ← R2 + α R1 . We want to find a matrix E such that
EA is A with R2 ← R2 + α R1 done to it, where A is any appropriately sized matrix.
Suppose we let A = :
with the row op. done to it = E = E,
so we can find E just by doing the row operation we want to . Therefore,
1 0 0 ··· 0
1 0 0 ··· 0
0 1 0 · · · 0
α 1 0 · · · 0
R2 ←R2 +α R1
0 0 1 · · · 0 = E
0
0
1
·
·
·
0
=
−−−−−−−→ . . . .
.
.
.
.
.
.
.. .. .. . . ..
.. .. .. . . ..
0 0 0 ··· 1
0 0 0 ··· 1
54
EMTH211 Week 1: Computation of Linear Systems
• Note that
eT1
eT1 A
AT1
α eT + eT
α e T + e T A α A T + A T
1
2
1
2
1
2
EA =
A=
=
eT3
AT3
eT3 A
.
.
.
..
..
..
eTn
ATn
eTn A
• Therefore E A is the result of performing the elementary row operation R2 ←
R2 + α R1 on A.
Any other elementary row operation of this type R ← R + α Rj is obtained by replacing row in the identity matrix by α eTj + eT . We will call this matrix Ej (α).
55
EMTH211 Week 1: Computation of Linear Systems
Therefore
eT1
eT2
..
.
Ej (α) =
.
α eT + eT
j
..
.
eTn
This matrix has 1s along the main diagonal and α in the (, j) position. Note that
for Gaussian elimination only row operations with j < are used and so Ej (α) is a
lower triangular matrix.
In order to “undo” this row operation we must add −α Rj to R ; that is apply Ej (−α).
Therefore Ej (α) is invertible with
E−1
(α) = Ej (−α).
j
EMTH211 Week 1: Computation of Linear Systems
1.7.2
Elementary matrix for R ↔ Rj
Interchanging rows and j is accomplished by
eT1
.
..
T
e−1
eT
j
eT
+1
.
E↔j = .. .
eT
j−1
eT
T
ej+1
.
.
.
eTn
Clearly to undo the row swap we just swap the same rows back:
= E↔j .
E−1
↔j
56
57
EMTH211 Week 1: Computation of Linear Systems
1.7.3
Elementary matrix for R ← βR
Finally, multiplying row by a scalar β is given by
eT1
T
e
.2
.
.
E (β) =
.
β eT
..
.
eTn
Note that
E−1
(β) = E
1
β
.
EMTH211 Week 1: Computation of Linear Systems
1.8
58
An Aside: Elementary Column Operations
In the discussion on pivoting, it was mentioned that complete pivoting required
column swaps. We can extend the concept of elementary row operations to column
operations. However these operations change the variables in the associated
system of linear equations.
Recall that Ax = 1 1 + 2 2 + · · · + n n .
• A column swap C ↔ Cj will interchange the variables and j in the associated linear system.
• Let à be obtained from A by multiplying a column by a non-zero constant
We have
C ← β C .
à x̃ = ̃1 1 + · · · + ̃ β + · · · + ̃n n .
This is equivalent to the original system Ax = b if we identify β ̃ = (with
̃k = k for all k = ). Thus this column operation scales the variable ;
← 1β .
EMTH211 Week 1: Computation of Linear Systems
59
• Let à be obtained from A by adding a multiple of one column to another
We have
C ← C + α Cj .
à x̃ = ̃1 1 + · · · + ̃ + α j + · · · + ̃n n .
This is equivalent to the original system if we identify ̃j = j − α (with ̃k = k
for all k = j).
An elementary column operation on A is equivalent to an elementary row operation
on the transpose AT .
Therefore a column swap on A will be given by
T
E↔j AT = A ET↔j = A E↔j
since ET↔j = E↔j .
EMTH211 Week 1: Computation of Linear Systems
60
Notes:
• A row swap results from pre-multiplying by E↔j . A column swap results
from post-multiplying by E↔j .
• Note E (β)T = E (β). Pre-multiplying by E (β) scales row . Post-multiplying
by E (β) scales column .
• Finally, pre-multiplying by Ej (α) results in R ← R +α Rj while post-multiplying
by Ej (α)T results in C ← C + α Cj .
Let E be any elementary matrix. The effect of a elementary column operation on
a system of equations is
A ET x̃ = A ET x̃
and so we identify
or, since E is easily invertible,
ET x̃ = x
x̃ = E−T x,
and we have equivalent systems of equations.
61
EMTH211 Week 1: Computation of Linear Systems
1.9
LU decomposition
In Gaussian elimination we convert the coefficient matrix A to an upper triangular matrix U, the so-called row echelon form (REF) of A. The solution of the
original system A x = b is found by deriving U x = b̃ with
A b −→ U b̃
using elementary row operations. This second system is solved by back substitution.
There are two problems with this (at least!):
• As we have seen, Gaussian elimination is not fast.
• If we want to solve another system involving the same matrix (common in
applications) we have to repeat quite a bit of work.
Maybe you’re thinking the second point might justify the very expensive leastefficient solution method: finding the inverse? Still ‘no’. There’s a better way.
If we keep track of the row operations we do using products of elementary matrices, we can find a matrix factorisation of A, the LU decomposition.
EMTH211 Week 1: Computation of Linear Systems
62
1 1 2
8
Example 3. Solve the system Ax = b, where A = −1 −2 3 and b = 1, and
3 −7 4
10
factorise A = LU by writing the row operations in terms of elementary matrices.
EMTH211 Week 1: Computation of Linear Systems
63
EMTH211 Week 1: Computation of Linear Systems
64
1.9.1
Formalising LU decomposition
In order to process the first column of A
11 12 13 · · · 1n
0
∗ ∗ ··· ∗
(1)
A −→
0
∗ ∗ ··· ∗ = U
..
..
.. . .
.
.
.
.
. ..
we perform the elementary row operations
R ← R −
1
11
R1 .
for = 2, 3, . . .. These row operations are equivalent to multiplication by elementary matrices of the form E1 (α ). Therefore
with
U(1) = En1 (αn ) · · · E31 (α3 ) E21 (α2 ) A
α = −
1
11
.
65
EMTH211 Week 1: Computation of Linear Systems
Note that
eT1
eT1 E21 (α2 )
eT2 E21 (α2 )
eT2
α eT + eT
α eT + eT E (α )
E31 (α3 ) E21 (α2 ) = 3 1
3 1
21
2
3 E21 (α2 ) =
3
..
..
.
.
T
eTn
e E21 (α2 )
n
and so
eT1 E21 (α2 )
eT1
α2 eT + eT
eT2 E21 (α2 )
1
2
E31 (α3 ) E21 (α2 ) = α3 eT1 + eT3 E21 (α2 ) = α3 eT + eT .
1
3
..
..
.
.
eTn
eTn E21 (α2 )
66
EMTH211 Week 1: Computation of Linear Systems
Note
eT1 E31 (α3 )
eT1
α2 eT + eT E31 (α3 ) α2 eT + eT
1
2
1
2
E21 (α2 ) E31 (α3 ) =
= α3 eT + eT = E31 (α3 ) E21 (α2 ).
eT3 E31 (α3 )
1
3
.
.
..
..
eTn
eTn E31 (α3 )
Thus
1 0 0 ··· 0
α 2 e T + e T α 1 0 · · · 0
2
1
2
α 0 1 · · · 0
= L1 .
3
En1 (αn ) · · · E31 (α3 ) E21 (α2 ) = α3 eT + eT =
1
3
.. .. .. . . ..
.
..
. . .
. .
αn 0 0 · · · 1
αn eT1 + eTn
eT1
We perform the same row operations on b. Therefore, after the first stage of row
reduction, we have
U(1) = L1 A,
b̃ = L1 b
where L1 is a lower triangular matrix.
67
EMTH211 Week 1: Computation of Linear Systems
Note:
• L1 is invertible since
L−1
= E−1
(α2 ) E−1
(α3 ) · · · E−1
(αn ) = E21 (−α2 ) E31 (−α3 ) · · · En1 (−αn )
1
21
31
n1
1 0 0 ··· 0
−α2 1 0 · · · 0
−α 0 1 · · · 0
=
.3 . . .
..
.. .. . . ...
−αn 0 0 · · · 1
• Our original linear system A x = b is transformed to L1 A x = L1 b; that is
U(1) x = b̃.
68
EMTH211 Week 1: Computation of Linear Systems
We now repeat the procedure on the second column of U
with
(1)
; that is, let
L2 = En2 (α2n ) · · · E42 (α24 ) E32 (α23 )
(1)
α2j = −
U2j
(1)
U22
(note that the elements in U(1) are not the same as the equivalent elements in A).
Define
11 12 13 · · · 1n
(1)
(1)
(1)
0 U22 U23 · · · U2n
(2)
(1)
0
∗ ··· ∗
U = L2 U = 0
0
∗ ··· ∗
0
..
.. . .
..
..
. .
.
.
.
We continue inductively. In order to obtain the kth column of the REF of A, we have
U(k) = Lk U(k−1)
with
and
Lk = Enk (αkn ) · · · E(k+2)k (αk(k+2) ) Ek(k+1) (αk(k+1) )
(k−1)
αkj = −
Ukj
(k−1)
Ukk
.
69
EMTH211 Week 1: Computation of Linear Systems
Therefore the REF form of A (assuming that no row swaps are required) is
U ≡ U(n) = Ln−1 · · · L2 L1 A;
that is,
L−1
· · · L−1
U.
A = (Ln−1 · · · L2 L1 )−1 U = L−1
1
2
n−1
Let
L−1
· · · L−1
.
L = L−1
1
2
n−1
Note that
eT1
..
.
eTk
−1
Lk =
−αk(k+1) eT + eT
k
k+1
..
.
−αkn eTk + eTn
Therefore
eT1
eT1
eT1
−α2 eT + eT
eT2
−α2 eT1 + eT2
1
2
−1 −1
T
T
T
T
T
T
T
L1 L2 = −α3 e + e −α32 e + e = −α3 e − α32 e + e
1
3
2
3
1
2
3
.
.
.
..
..
..
−αn eT1 + eTn
−αn2 eT2 + eTn
−αn eT1 − αn2 eT2 + eTn
70
EMTH211 Week 1: Computation of Linear Systems
Note that
eT1
eT1
eT1
α2 eT + eT
eT2
α2 eT1 + eT2
1
2
=
L2 L1 =
T
T
T
T
T
T
T
α32 e2 + e3 α3 e1 + e3 (α2 α32 + α3 ) e1 + α32 e2 + e3
..
..
..
.
.
.
Continuing this process we see that
L=
eT1
−α2 eT1 + eT2
−α3 eT1 − α32 eT2 + eT3
−α4 eT1 − α42 eT2 − α43 eT3 + eT4
..
.
−αn eT1 − αn2 eT2 − αn3 eT3 − · · · − αn(n−1) eTn−1 + eTn
1
0
0
0
··· 0
−α2
1
0
0
· · · 0
−α3 −α32
1
0
· · · 0
= −α −α
1
· · · 0
42 −α43
.4
. . ..
..
..
..
..
. .
.
.
.
−αn −αn2 −αn3 −αn4 · · · 1
0
advertisement
Related documents
Download
advertisement
Add this document to collection(s)
You can add this document to your study collection(s)
Sign in Available only to authorized usersAdd this document to saved
You can add this document to your saved list
Sign in Available only to authorized users