Jim Lambers MAT 610 Summer Session 2009-10 Lecture 4 Notes These notes correspond to Sections 3.1 and 3.2 in the text. Gaussian Elimination and Back Substitution The basic idea behind methods for solving a system of linear equations is to reduce them to linear equations involving a single unknown, because such equations are trivial to solve. Such a reduction is achieved by manipulating the equations in the system in such a way that the solution does not change, but unknowns are eliminated from selected equations until, ο¬nally, we obtain an equation involving only a single unknown. These manipulations are called elementary row operations, and they are deο¬ned as follows: β Multiplying both sides of an equation by a scalar β Reordering the equations by interchanging both sides of the πth and πth equation in the system β Replacing equation π by the sum of equation π and a multiple of both sides of equation π The third operation is by far the most useful. We will now demonstrate how it can be used to reduce a system of equations to a form in which it can easily be solved. Example Consider the system of linear equations π₯1 + 2π₯2 + π₯3 = 5, 3π₯1 + 2π₯2 + 4π₯3 = 17, 4π₯1 + 4π₯2 + 3π₯3 = 26. First, we eliminate π₯1 from the second equation by subtracting 3 times the ο¬rst equation from the second. This yields the equivalent system π₯1 + 2π₯2 + π₯3 = 5, −4π₯2 + π₯3 = 2, 4π₯1 + 4π₯2 + 3π₯3 = 26. Next, we subtract 4 times the ο¬rst equation from the third, to eliminate π₯1 from the third equation as well: π₯2 + 2π₯2 + π₯3 = 5, 1 −4π₯2 + π₯3 = 2, −4π₯2 − π₯3 = 6. Then, we eliminate π₯2 from the third equation by subtracting the second equation from it, which yields the system π₯1 + 2π₯2 + π₯3 = 5, −4π₯2 + π₯3 = 2, −2π₯3 = 4. This system is in upper-triangular form, because the third equation depends only on π₯3 , and the second equation depends on π₯2 and π₯3 . Because the third equation is a linear equation in π₯3 , it can easily be solved to obtain π₯3 = −2. Then, we can substitute this value into the second equation, which yields −4π₯2 = 4. This equation only depends on π₯2 , so we can easily solve it to obtain π₯2 = −1. Finally, we substitute the values of π₯2 and π₯3 into the ο¬rst equation to obtain π₯1 = 9. This process of computing the unknowns from a system that is in upper-triangular form is called back substitution. β‘ In general, a system of π linear equations in π unknowns is in upper-triangular form if the πth equation depends only on the unknowns π₯π , π₯π+1 , . . . , π₯π , for π = 1, 2, . . . , π. Now, performing row operations on the system π΄x = b can be accomplished by performing them on the augmented matrix β‘ β€ π11 π12 ⋅ ⋅ ⋅ π1π β£ π1 β₯ [ ] β’ β’ π21 π22 ⋅ ⋅ ⋅ π2π β£ π2 β₯ π΄ b =β’ . β₯. . . .. β£ .. β£ .. β¦ ππ1 ππ2 ⋅ ⋅ ⋅ πππ β£ ππ By working with the augmented matrix instead of the original system, there is no need to continually rewrite the unknowns or arithmetic operators. Once the augmented matrix is reduced to upper triangular form, the corresponding system of linear equations can be solved by back substitution, as before. The process of eliminating variables from the equations, or, equivalently, zeroing entries of the corresponding matrix, in order to reduce the system to upper-triangular form is called Gaussian elimination. The algorithm is as follows: for π = 1, 2, . . . , π − 1 do for π = π + 1, π + 2, . . . , π do πππ = πππ /πππ for π = π + 1, π + 2, . . . , π do πππ = πππ − πππ πππ 2 end ππ = ππ − πππ ππ end end This algorithm requires approximately 23 π3 arithmetic operations, so it can be quite expensive if π is large. Later, we will discuss alternative approaches that are more eο¬cient for certain kinds of systems, but Gaussian elimination remains the most generally applicable method of solving systems of linear equations. The number πππ is called a multiplier. It is the number by which row π is multiplied before adding it to row π, in order to eliminate the unknown π₯π from the πth equation. Note that this algorithm is applied to the augmented matrix, as the elements of the vector b are updated by the row operations as well. It should be noted that in the above description of Gaussian elimination, each entry below the main diagonal is never explicitly zeroed, because that computation is unnecessary. It is only necessary to update entries of the matrix that are involved in subsequent row operations or the solution of the resulting upper triangular system. This system is solved by the following algorithm for back substitution. In the algorithm, we assume that π is the upper triangular matrix containing the coeο¬cients of the system, and y is the vector containing the right-hand sides of the equations. for π = π, π − 1, . . . , 1 do π₯π = π¦π for π = π + 1, π + 2, . . . , π do π₯π = π₯π − π’ππ π₯π end π₯π = π₯π /π’ππ end This algorithm requires approximately π2 arithmetic operations. We will see that when solving systems of equations in which the right-hand side vector b is changing, but the coeο¬cient matrix π΄ remains ο¬xed, it is quite practical to apply Gaussian elimination to π΄ only once, and then repeatedly apply it to each b, along with back substitution, because the latter two steps are much less expensive. We now illustrate the use of both these algorithms with an example. Example Consider the system of linear equations π₯1 + 2π₯2 + π₯3 − π₯4 = 5 3π₯1 + 2π₯2 + 4π₯3 + 4π₯4 = 16 4π₯1 + 4π₯2 + 3π₯3 + 4π₯4 = 22 2π₯1 + π₯3 + 5π₯4 = 15. 3 This system can be represented by the coeο¬cient matrix π΄ and right-hand side vector b, as follows: β€ β€ β‘ β‘ 5 1 2 1 −1 β₯ β’ β’ 3 2 4 4 β₯ β₯ , b = β’ 16 β₯ . π΄=β’ β¦ β£ β£ 4 4 3 22 β¦ 4 15 2 0 1 5 To perform row operations to reduce this system to upper triangular form, we deο¬ne the augmented matrix β‘ β€ 1 2 1 −1 5 [ ] β’ 3 2 4 4 16 β₯ β₯. π΄˜ = π΄ b = β’ β£ 4 4 3 4 22 β¦ 2 0 1 5 15 We ο¬rst deο¬ne π΄˜(1) = π΄˜ to be the original augmented matrix. Then, we denote by π΄˜(2) the result of the ο¬rst elementary row operation, which entails subtracting 3 times the ο¬rst row from the second in order to eliminate π₯1 from the second equation: β‘ β€ 1 2 1 −1 5 β’ 0 −4 1 7 1 β₯ β₯. π΄˜(2) = β’ β£ 4 4 3 4 22 β¦ 2 0 1 5 15 Next, we eliminate π₯1 from the third equation by subtracting 4 times the ο¬rst row from the third: β€ β‘ 1 2 1 −1 5 β’ 0 −4 1 7 1 β₯ β₯. π΄˜(3) = β’ β£ 0 −4 −1 8 2 β¦ 2 0 1 5 15 Then, we complete the elimination of π₯1 by β‘ 1 β’ 0 π΄˜(4) = β’ β£ 0 0 subtracting 2 times the ο¬rst row from the fourth: β€ 2 1 −1 5 −4 1 7 1 β₯ β₯. −4 −1 8 2 β¦ −4 −1 7 5 We now need to eliminate π₯2 from the third and fourth equations. This is accomplished by subtracting the second row from the third, which yields β‘ β€ 1 2 1 −1 5 β’ 0 −4 1 7 1 β₯ β₯, π΄˜(5) = β’ β£ 0 0 −2 1 1 β¦ 0 −4 −1 7 5 4 and the fourth, which yields β€ 1 2 1 −1 5 β’ 0 −4 1 7 1 β₯ β₯. =β’ β£ 0 0 −2 1 1 β¦ 0 0 −2 0 4 β‘ π΄˜(6) Finally, we subtract the third row from the fourth to obtain the augmented matrix of an uppertriangular system, β‘ β€ 1 2 1 −1 5 β’ 0 −4 1 7 1 β₯ β₯. π΄˜(7) = β’ β£ 0 0 −2 1 1 β¦ 0 0 0 −1 3 Note that in a matrix for such a system, all entries below the main diagonal (the entries where the row index is equal to the column index) are equal to zero. That is, πππ = 0 for π > π. Now, we can perform back substitution on the corresponding system, π₯1 + 2π₯2 + π₯3 − π₯4 = 5, −4π₯2 + π₯3 + 7π₯4 = 1, −2π₯3 + π₯4 = 1, −π₯4 = 3, to obtain the solution, which yields π₯4 = −3, π₯3 = −2, π₯2 = −6, and π₯1 = 16. β‘ The πΏπ Factorization We have learned how to solve a system of] linear equations π΄x = b by applying Gaussian elimination [ ˜ to the augmented matrix π΄ = π΄ b , and then performing back substitution on the resulting upper-triangular matrix. However, this approach is not practical if the right-hand side b of the system is changed, while π΄ is not. This is due to the fact that the choice of b has no eο¬ect on the row operations needed to reduce π΄ to upper-triangular form. Therefore, it is desirable to instead apply these row operations to π΄ only once, and then “store” them in some way in order to apply them to any number of right-hand sides. To accomplish this, we ο¬rst note that subtracting πππ times row π from row π to eliminate πππ 5 is equivalent to multiplying π΄ by the matrix β‘ 1 0 ⋅⋅⋅ β’ 0 1 0 β’ β’ . . . .. β’ .. . . β’ β’ . .. β’ .. . β’ πππ = β’ . β’ .. −πππ β’ β’ .. β’ . β’ β’ .. β£ . 0 ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ 0 0 .. .. . . .. .. .. . . . .. .. .. .. . . . . . . . . . . .. . . . . 0 ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ 1 0 β€ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯, β₯ β₯ β₯ β₯ β₯ β₯ 0 β¦ 1 where the entry −πππ is in row π, column π. More generally, if we let π΄(1) = π΄ and let π΄(π+1) be the matrix obtained by eliminating elements of column π in π΄(π) , then we have, for π = 1, 2, . . . , π − 1, π΄(π+1) = π (π) π΄(π) where β‘ π (π) β’ β’ β’ β’ β’ β’ β’ β’ β’ =β’ β’ β’ β’ β’ β’ β’ β’ β’ β£ 1 0 ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ 0 1 0 .. . . . . .. . . . . .. .. .. . . . 0 .. .. .. .. . . . . −ππ+1,π .. .. .. .. . . . . 0 .. .. .. .. . . . . . . . .. .. .. .. . . . . 0 ⋅⋅⋅ 0 −πππ 0 ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ 0 0 .. . .. . .. . .. .. . . . . . . .. . . . .. . 1 0 ⋅⋅⋅ 0 1 β€ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯, β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β¦ with the elements −ππ+1,π , . . . , −πππ occupying column π. It follows that the matrix π = π΄(π) = π (π−1) π΄(π−1) = π (π−1) π (π−2) ⋅ ⋅ ⋅ π (1) π΄ is upper triangular, and the vector y = π (π−1) π (π−2) ⋅ ⋅ ⋅ π (1) b, being the result of applying the same row operations to b, is the right-hand side for the uppertriangular system that is to be solved by back subtitution. 6 Unit Lower Triangular Matrices We have previously learned about upper triangular matrices that result from Gaussian elimination. Recall that an π × π matrix π΄ is upper triangular if πππ = 0 whenever π > π. This means that all entries below the main diagonal, which consists of the entries π11 , π22 , . . ., are equal to zero. A system of linear equations of the form π x = y, where π is an π × π nonsingular upper triangular matrix, can be solved by back substitution. Such a matrix is nonsingular if and only if all of its diagonal entries are nonzero. Similarly, a matrix πΏ is lower triangular if all of its entries above the main diagonal, that is, entries βππ for which π < π, are equal to zero. We will see that a system of equations of the form πΏy = b, where πΏ is an π × π nonsingular lower triangular matrix, can be solved using a process similar to back substitution, called forward substitution. As with upper triangular matrices, a lower triangular matrix is nonsingular if and only if all of its diagonal entries are nonzero. Triangular matrices have the following useful properties: β The product of two upper (lower) triangular matrices is upper (lower) triangular. β The inverse of a nonsingular upper (lower) triangular matrix is upper (lower) triangular. That is, matrix multiplication and inversion preserve triangularity. Now, we note that each matrix π (π) , π = 1, 2, . . . , π − 1, is not only a lower-triangular matrix, but a unit lower triangular matrix, because all of its diagonal entries are equal to 1. Next, we note two important properties of unit lower (or upper) triangular matrices: β The product of two unit lower (upper) triangular matrices is unit lower (upper) triangular. β A unit lower (upper) triangular matrix is nonsingular, and its inverse is unit lower (upper) triangular. In fact, the inverse of each π (π) is easily computed. We have β‘ 1 0 ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ β’ 0 1 0 β’ β’ . . .. . . . β’ . . . .. β’ β’ . . .. .. β’ .. . 0 β’ β’ . . . . .. π .. πΏ(π) = [π (π) ]−1 = β’ π+1,π β’ . β’ .. .. .. β’ . . . 0 β’ β’ .. .. .. .. β’ . . . . β’ .. .. .. β’ .. β£ . . . . 0 ⋅⋅⋅ 0 πππ 0 7 ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ 0 0 .. . .. . .. .. . . .. .. .. . . . . . . . . . .. . . . . .. . 1 0 ⋅⋅⋅ ⋅⋅⋅ 0 1 β€ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯. β₯ β₯ β₯ β₯ β₯ β₯ β₯ β₯ β¦ It follows that if we deο¬ne π = π (π−1) ⋅ ⋅ ⋅ π (1) , then π is unit lower triangular, and π π΄ = π , where π is upper triangular. It follows that π΄ = π −1 π = πΏπ , where πΏ = πΏ(1) ⋅ ⋅ ⋅ πΏ(π−1) = [π (1) ]−1 ⋅ ⋅ ⋅ [π (π−1) ]−1 is also unit lower triangular. Furthermore, from be determined that β‘ 1 0 β’ β’ π21 1 β’ β’ .. πΏ=β’ . π32 β’ β’ .. .. β£ . . ππ1 ππ2 the structure of each matrix πΏ(π) , it can readily β€ ⋅⋅⋅ ⋅⋅⋅ 0 .. β₯ 0 . β₯ β₯ .. β₯ . .. .. . . . β₯ β₯ β₯ .. . 1 0 β¦ ⋅ ⋅ ⋅ ππ,π−1 1 That is, πΏ stores all of the multipliers used during Gaussian elimination. The factorization of π΄ that we have obtained, π΄ = πΏπ, is called the πΏπ decomposition, or πΏπ factorization, of π΄. Solution of π΄x = b Once the πΏπ decomposition π΄ = πΏπ has been computed, we can solve the system π΄x = b by ο¬rst noting that if x is the solution, then π΄x = πΏπ x = b. Therefore, we can obtain x by ο¬rst solving the system πΏy = b, and then solving π x = y. Then, if b should change, then only these last two systems need to be solved in order to obtain the new solution; the πΏπ decomposition does not need to be recomputed. The system π x = y can be solved by back substitution, since π is upper-triangular. To solve πΏy = b, we can use forward substitution, since πΏ is unit lower triangular. for π = 1, 2, . . . , π do π¦π = ππ for π = 1, 2, . . . , π − 1 do π¦π = π¦π − βππ π¦π end end 8 Like back substitution, this algorithm requires π(π2 ) ο¬oating-point operations. Unlike back substitution, there is no division of the πth component of the solution by a diagonal element of the matrix, but this is only because in this context, πΏ is unit lower triangular, so βππ = 1. When applying forward substitution to a general lower triangular matrix, such a division is required. Implementation Details Because both forward and back substituion require only π(π2 ) operations, whereas Gaussian elimination requires π(π3 ) operations, changes in the right-hand side b can be handled quite eο¬ciently by computing the factors πΏ and π once, and storing them. This can be accomplished quite eο¬ciently, because πΏ is unit lower triangular. It follows from this that πΏ and π can be stored in a single π × π matrix by storing π in the upper triangular part, and the multipliers πππ in the lower triangular part. Existence and Uniqueness Not every nonsingular π × π matrix π΄ has an πΏπ decomposition. For example, if π11 = 0, then the multipliers ππ1 , for π = 2, 3, . . . , π, are not deο¬ned, so no multiple of the ο¬rst row can be added to the other rows to eliminate subdiagonal elements in the ο¬rst column. That is, Gaussian elimination can break down. Even if π11 β= 0, it can happen that the (π, π) element of π΄(π) is zero, in which case a similar breakdown occurs. When this is the case, the πΏπ decomposition of π΄ does not exist. This will be addressed by pivoting, resulting in a modiο¬cation of the πΏπ decomposition. It can be shown that the πΏπ decomposition of an π × π matrix π΄ does exist if the leading principal minors of π΄, deο¬ned by β‘ β€ π11 ⋅ ⋅ ⋅ π1π β’ .. β₯ , π = 1, 2, . . . , π, .. [π΄]1:π,1:π = β£ ... . . β¦ ππ1 ⋅ ⋅ ⋅ πππ are all nonsingular. Furthermore, when the πΏπ decomposition exists, it is unique. To see this, suppose that π΄ has two distinct πΏπ decompositions, π΄ = πΏ1 π1 = πΏ2 π2 . Then, because the factors of each decomposition are nonsingular, it follows that −1 πΏ−1 2 πΏ1 = π2 π1 . The left side is lower-triangular, while right side is upper-triangular, because the triangularity of a matrix is preserved by inversion and multiplication with matrices of the same triangularity. It follows that both matrices must be diagonal. However, the left side is also unit lower triangular, and so both sides must equal the identity matrix πΌ. Therefore, πΏ2 = πΏ1 and π1 = π2 , contradicting the assumption that the decompositions are distinct. We conclude that the πΏπ decomposition is unique. 9 Practical Computation of Determinants Computing the determinant of an π × π matrix π΄ using its deο¬nition requires a number of arithmetic operations that is exponential in π. However, more practical methods for computing the determinant can be obtained by using its properties: ˜ = β If π΄˜ is obtained from π΄ by adding a multiple of a row of π΄ to another row, then det(π΄) det(π΄). β If π΅ is an π × π matrix, then det(π΄π΅) = det(π΄) det(π΅). β If π΄ is a triangular matrix (either upper or lower), then det(π΄) = ∏π π=1 πππ . It follows from these properties that if Gaussian elimination is used to reduce π΄ to an uppertriangular matrix π , then det(π΄) = det(π ), where π is the resulting upper-triangular matrix, because the elementary row operations needed to reduce π΄ to π do not change the determinant. Because π is upper triangular, det(π ), being the product of its diagonal entries, can be computed in π − 1 multiplications. It follows that the determinant of any matrix can be computed in π(π3 ) operations. It can also be seen that det(π΄) = det(π ) by noting that if π΄ = πΏπ , then det(π΄) = det(πΏ) det(π ), by one of the abovementioned properties, but det(πΏ) = 1, because πΏ is a unit lower triangular matrix. It follows from the fact that πΏ is lower triangular that det(πΏ) is the product of its diagonal entries, and it follows from the fact that πΏ is unit lower triangular that all of its diagonal entries are equal to 1. 10