Table of Contents 1 1 2 Introduction to Vectors 1.1 Vectors and Linear Combinations. 1.2 Lengths and Dot Products. 1.3 Matrices ......... 22 2 Solving Linear Equations 2.1 Vectors and Linear Equations . 2.2 The Idea of Elimination ... 2.3 Elimination Using Matrices . 2.4 Rules for Matrix Operations 2.5 Inverse Matrices ....... 2.6 Elimination= Factorization: A= LU 2.7 Transposes and Permutations . . . . . . 31 31 46 58 70 83 97 109 3 Vector Spaces and Subspaces 3.1 Spaces of Vectors ....... . . . . . . . . . . . . 3.2 The Nullspace of A: Solving Ax = 0 and Rx = 0 3.3 The Complete Solution to Ax = b 3.4 Independence, Basis and Dimension 3.5 Dimensions of the Four Subspaces 123 123 135 150 164 181 4 Orthogonality 4.1 Orthogonality of the Four Subspaces 4.2 Projections . . . . . . . . . . . . . 4.3 Least Squares Approximations ... 4.4 Orthonormal Bases and Gram-Schmidt. 194 194 206 219 233 5 Determinants 5.1 The Properties of Determinants . 5.2 Permutations and Cofactors ... 5.3 Cramer's Rule, Inverses, and Volumes 247 247 258 273 iii 11 iv Table of Contents 6 Eigenvalues and Eigenvectors 6.1 Introduction to Eigenvalues.. 6.2 Diagonalizing a Matrix .... 6.3 Systems of Differential Equations 6.4 Symmetric Matrices........ 6.5 Positive Definite Matrices..... 288 2 88 30 4 319 338 350 7 The Singular Value Decomposition (SVD) 7.1 Image Processing by Linear Algebra .... 7.2 Bases and Matrices in the SVD ....... 7.3 Principal Component Analysis (PCA by the SVD). 7.4 The Geometry of the SVD ............. 364 36 4 371 382 39 2 8 Linear Transformations 8.1 The Idea of a Linear Transformation 8.2 The Matrix of a Linear Transformation. 8.3 The Search for a Good Basis ...... 401 40 1 411 42 1 9 Complex Vectors and Matrices 9.1 Complex Numbers ...... 9.2 Hermitian and Unitary Matrices 9.3 The Fast Fourier Transform. 430 431 438 445 10 Applications 10.1 Graphs and Networks ............. 10.2 Matrices in Engineering............ 10.3 Markov Matrices, Population, and Economics 10.4 Linear Programming ............. 10.5 Fourier Series: Linear Algebra for Functions. 10.6 Computer Graphics ........ 10.7 Linear Algebra for Cryptography....... 452 452 46 2 474 483 49 0 49 6 50 2 11 Numerical Linear Algebra 11.1 Gaussian Elimination in Practice 11.2 Norms and Condition Numbers. 11.3 Iterative Methods and Preconditioners 508 50 8 518 52 4 12 Linear Algebra in Probability & Statistics 12.1 Mean, Variance, and Probability ...... 12.2 Covariance Matrices and Joint Probabilities 12.3 Multivariate Gaussian and Weighted Least Squares 535 535 546 555 Matrix Factorizations 563 Index 565 Six Great Theorems/ Linear Algebra in a Nutshell 574 m+4m- UJ = m· 2 5 angle with v less than 90 ° ! u. i = cos 0 , v+w u- [-i l X1 [ - ~ 0 v- l + X2 [ Ul ~ -1 w - [ l ~l + X3 [ 1 n- = [ :~ X3 - X1 X2 l· (1) = b1 X1 - x1+x2 =b2 -x2 +x3 = b3 b ~ m gives x ~ m Cx = 1 01 [ -1 0 -1 - 10 1 i[ l X1 x2 X3 = rn V 1 right weights :r1, .'.C2, x3, can produce weights A- 1 b. 1 0 ~1 0 0 ~1 1 0 0l [ 1 1 0 1 1 1 [ Y2 Y1 l = Y3 [ C1C2 l • C3 - 1 1 ol [ 321 74c [11 1 o 0cl 011 [c2 3 · -- -- The column picture combines the column vectors on the left side to produce the vector b on the right side. '' 3[ t ]+ [ -~ ] = [ 1~ ] · = o [ ~ ] +o [ j] +2 [ ; ] = [ i]. 1 ol o0 0 0 I = [ 0 1 l After ~ ~ =l =8 Before elimination 8y =8 2 3 X l X X X X X X X X X X rn rn = 1 = ~] ~l m~ m. P23 1 o ol 0 1 0 = [0 0 1 . 0 0 0 -4 1 3 1 E = [a ol ol0 b O l o ol 1 F= [ 0 1 0 . 0 C 1 0 0 1 0 2 1 3 3 1 [3 X X X] 4 X X X 31 2 21 21 3 1 1 1 2 l 2 0 0 2 0 0 0 0 AB = [~ : :] [~ : :] = [ 0 00x 00x 00 l· Multiply Ax = b by A - 1 . Then x = A - 1 Ax = A - 1 b. FE= [-t 20 0 1 -4 (6) 2 2 2 ~ 1 -~ ~] ➔ [~ ➔ [~ -~J 3 1 1 1 1 3 A= [ 1 3 1 l 2 1 1 1 1 3 B= [ 1 2 1 l C = [ 11 11 11 1 1 3 l 6 [ 6 G] () ,-1 C [ 6.· 6 ] 6 6 1 [ () 6 = :36 2 0 3 0 0 6 0 7 ll 2 1 A= [ 1 2 1 1 1 2 1 o ol 0 0 A= [ 2 1 3 1 and ll A= [ 11 21 2 . 1 2 3 -~i 1 -11 0 1 -1 0 0 1 1 T = [ -l 0 -1 2 -1 ol -1 2 r- 1 = ll 3 [2 2 2 1 1 1 1 1 <Al'J' ~ r (uJ' j ( IJ,) 97 1 2 1 0 0 1 2 1 1 dn U12/d1 U13/d1 1 U23/d2 ol 2 1 A= [ O 4 2 . 6 3 5 ol [e1 1 1 [ 112= 121 1 c ol 3 5 1 A= [ 2 4 1 . 1 mnl l [d gl ef ~ i A= [~ a a ~ ~ b b c c ~] . c d 3by 3 2 -1 - O l [3 2 ll [4 4 l . 1 2 4 2 case [ The determinant (K)(4K- 1 ) = - 1 2 of this K is 4 0 -1 2 1 2 3 4 S [~ ;] = [~ ~] u;] = [~ ~] [1 2] = 2 5 = S T and D [l 0] = O IO = D T . Ps2P21 = [ P21Ps2 1 1 1] = [1 1 l P = [0o1 010 10ol -.'• ·:. PS= PS= [451 46 6]5 2 3 Q o o [451 46 6]5 2 3 1] = [1 0 0 0 1 0 S = LDLT = [ 41 5 o1 ol0 1 1 l [10 [1 -14 -8 0 41 0 5l1 . 1 A= o o [1 2 0 4 6]3 . 5 = and and S ol . 2 -1 2 -1 0 -1 2 = [ -1 1 b S= [b d C e cl e j . o 1 1] 2 3 A= [ 1 0 1 1 2 ol 1 1 1 A= [ 2 4 1 . and 4 o 1 2] 2 1 A= [ O 3 8 . 1 00 0 ~j - 1 1 0 '•ft Plane = = 1 [0 0 ll [X1] b2 x2 = [b1] 11 1 0 1 X3 and and b3 [1O 11 0 0 ll [X1] b2 X2 = [b1] 1 0 ll [X1] b2 . x2 = [b1] 1 1 [O O 1 0 0 1 X3 b3 X3 b3 !] 1 becomes U = [~ ~ t t 1 3 R= [ 0 O 0 0 C ;t: 4 (b) B = [02 44 42] . 0 8 8 A= [o0 oO 3ol 2 4 6 A= [12 2 4] 4 B = 2 ]. [1-0 2-c c ll 1 42] [X [011 330 O 1 6 X2 X3 X4 = [ll 6 7 A = [ ~ ~] and -2 -3 b= [ ~~] · b3 S1 - ~ [ -21 S2 - l [ 2 : 1 3 2 4 l A= [ 3 8 2 . 0 0 0 62 ~] 0 0 6 0 2 0 0 and = 1 A= [ l o1 ol0 1 1 1 Vj Vj = = mv, mv, m m = h] H] [j] v, = V3 = = V4 = [-I] V4 = V5 = u] [j] V6 = A= [101 33l 212] and U = [100 30l 02]1 . B - [c cl cl] . C 5 0 0 1 5 1 :;;t J .• ·- J and B-[J A, A= [0o0 001 ol01 and I +A = 1 [0 0 1 1 0 ol1 . 1 V W w I C(A) _l_ N(AT) dimension all vectors orthogonal to the columns dimm-r 3rows row S\lace 33 Suppose I give you eight vectors r1, r2, n1, n2, c1, c2, l 1, l2 in R 4. o o ol Pi= [0 0 0 0 0 1 I b I I a p =xa and e = b- p = ( ig ' - !g ' - !9 ) . = 2 column 8Pace h1 :::::: 6 Pi :::::: 5 e b - 5 - Jr best lin = = = l ~ ""'[>,,"/ / / / / ' '' a ~ H] and b ~ U] and c ~ H] . 1 - 1 1 1 -~] 1 - 1 - 1 -1 -1 1 = = AT A= [99 189] = [11 10] [9 9] [10 11] . v12 -v12 -1 -1] -1 -1 1 -1 -1 1 . A= o o [1 0 1] A= l [1 o 1 1] 1 1 B = [1 0 0 0 1 0 1 2 ll 2 1 2 3 1 0 A= C= l [1 1 [l 2 3] 2 2 3 3 3 3 1 1 1 . ll 1 . 1 <let [l n+l n ----= 21 1 [1 1 2 1 2 1 1 1 1 2 j B = [14 42 3]4 5 6 7 B= [41 25 63] . 7 0 0 [-BI M= [-B0 A]I = [ABO A] I OJ . I 1 0 0 1 0 0 0 1 1 0 0 0 Find x = A - 1b Check [ detB2 x2 = -d-et_A_ ~ !][_~ ]= [ ~ ] L Ya ---- 1 2 (0,0) X1 Y1 X2 Y2 X3 Y3 - I I 1 4 A= [ 2 3 97 2 2 8 l 1 1 A= [ 1 1 21 1 1 1 l Ci 1 2 (a) A= [ O 3 0 7 1 1 4] A= [l 2 2 1 2 5 ol O 1 (b) A = [- 2 -1 ol 0 -1 2 l 2 - 1 6 -3 C= [ . . .' • • ~. ·• I . '\ - . . for >-1 = 0 - ::::- - .:;;.;:;;;; - A = [01 21 ] and A- 1 = [-1112/2 10 ]. l A= [ 31 62 3 4 8 4 P= and B = P APT = [L 0 0 [682 431 43]1 . .... Ax1 \ I = x1 ,\1 x - 1 Ax = A = [ [~~] [~i] A X = >-n [1 5] = [1 l][1 ][1-lJ= [1 0 6 k 0 1 6k O 1 0 6k 6k ] =XA. l]= Ak . 1 ------- - Forward (llF) Yn+l = Yn + flt Zn Zn+I = Zn - flt Yn [ 1 becomes U n+l = - flt -Yn-1 - Yn - Yn + l flt][z: 1 Y, ] = AUn · (12) Backward Yn+ l Zn+l = Yn + .6.t Zn+ l = Zn - .6.t Yn+ l -- -- al [o1 0o0o0ol [alb [ dt ~ = ~ ~ ~ ~ ~ db d [ y] dt y' [ 0 1] [y'y] . = -9 6 u = [t] B o = [o -4] 0 . Trapezoidal [ 1 6;.t/ 2 -b;.t/2 ] [ Yn+l ] _ [ 1 1 Zn+l - b;.t/2 6;.t/2 ] [ Yn ] 1 Zn · A3 = 2 [ -1 0 -1 2 - 1 o -1 2 l 1 ... 9 B = [ 12 1 1 ~1 -1 0 S 2 = [- 1 0 ol - 1 2 - 1 -1 2 T bl 2 -1 2 - 1 . b - 1 2 = [-1 ol [- O11 - o11 o]01 O -l O O - 1 X x+y =X J2 S4 10] . = [ 101 101 2 2 o] 0 3 8 S= [ 2 5 3 . 5 = C [ 1 1 1 1] C 1 1 C T= [1 d2 3] 2 3 4 4 5 . ' ' t 3 ol T= [ 3 t 4 . 0 4 t S = 9 [0 0 o1 ol 2 2 8 1 S= [ 1 21 2ll . 1 2 7 101 - • • 10-1 < - - - - - ~_ _ _ _ ___,____ _ _ 0 10 20 30 _ _ _ J L __ _ ____, 40 10-20 < - - - - - - ' - - - - - - ~ - - - - - " - - - - - ' 0 10 20 30 40 AT A= [ 20 10 10 ] 5 ~ [1] Av2 - ~ [ -f ] xTATAx xTx xTSx xTx · 2 (/1 · (J/ . 0/ . 2 2 + ··· + (JR2 1s 99.10 or 99.9.10 of <T1 + ••· + <T.,,. '-.__/ V '. n J5 [~ = -~] ~] . Then A = Q S. o o0 l [2 0 3o o 0l = 1/3 0 0 000 [1 o ol [i ~]. 0 1 0 000 = AA+p= ~~ r A+ Ax+= The input is v = (v1 , v2 , V3). v,~~v) \ -~ ,;. -), ~ w ) T(v3) ➔ v, lnp~t basis [vi v = [ 33 6] 8 2] [ w 1 w 2 ] [ B ] -_ [v l v 2 ] 1S • [ 3 1 0 2 ] [ 11 23 ] -_ [ 3 3 6 8] . 0 1 0 Ac= [ 0 0 2 0 0 0 2 ? = o o [0 0 0 0 1 0 0 l 0l ~ ll l .\ 1 ~: = ,\ 2 ll .\ 1 ~: 2 = ,\ x 4 when A = 1. J 1 [ 02 21 0] = 0 J= l 0 2 O 01 00 00 0 1 0 J K= l O 01 01 00 0 0 0 J .' -. ,, I, = iz l2 = 12 rcos0 + 12 = 2 + irsin0 = v'2 ( ~ ) + iv'2 ( ~ ) = 1 + i . becomes 1 [ ~ - ~] [ ~] = [ ] . [ 1 e-Li/3 e -21ri/ 3 l . S = [. 0 i +1 11 i] 2n \8 ~3] [~ w6 w9 1 l 1 w w2 w3 1 i i2 i3 1 1 ] [1 1 ] [1 ] [ = 1 -1 -i ~ i\ 1 1. i F4 F 1024 = [ h12 I 1 i2 D512] [Fs12 512 - D 512 1 F,512 ] [o00 001 o01 ol01 1 0 0 0 [11 1i 1 i2 1 i3 i12 i4 i6 i13 i6 i9 l = [11 1i 1 i2 1 i3 i12 i4 i6 i13 i6 i9 l [>11 A2 A3 ol 0 0 -+ [-1 0 0 = s Xz a<==-- - - - ------ - - - - - - - - ""9 X3 Y3 = [U1] U2 U3 1[1 =- 16 1 l l 0(.6.x) 2 C = A[-~] = .75 [-n . Uo = [:~~] = [:~] + .18 [-~J. A=[l0 .8.2 ] A= [·.82 10] A=[t 4 1 4 f 2 l 4 !] 4 1 2 · 481 1 0 0 0 0 1 0 0 The dot product of x ::: 0 and s =c - A '1' y ::: 0 gave x'I' s 2 0. This is xT AT y ~ xT c. Algebra (b) Graphics by [ 2 0 2 1 ] -1 =2 - 1 [ 1 - 0 2 -2 ] _ 1 = 2 [ -2 -0 2 ] _ = [ 2 0 2 1 ] (mod 3) · o A= [ 1l o 1 O 1 1 1 l B = [ 01 1l o 1 1 0 1 l 1 1 1 1 0 0 C= [ O O 1 l 1 -11 -1 1 - ~ . = - -- 3 2 3 2 2 2 1 1 ForA- 1 2 ll 0 2 ------+ [1 1 ll 0 0 3 2 2 4 3 a32 =0 a32 .6 Q,,A = r -.8 0 = 2 Q,1Q21A =[ 1 0 ll A= [ 2 2 0 . 0 2 0 PKPT 2I = [ DT '• XT A Ax _ _ _ _ _ _ _ _ _ _ _ _ _ __ T - x'T"x - ------ --· ... F = l . a ✓ - - 2N.,,,..;., _ ....__ _ _ _ _ _ ___,__ --I-----+---+----+---+-- ( 1 1) 4 2+ 2 1 4 6 4 1 = 16 + 16 + 16 + 16 + 16 = 1. p = [ Pu P21 P 12 ] = P22 [ ¾ ¾ .! 4 .! 4 l n i i i i - I (x - m1) 2 (y - m2) 2 2o-f 2o-~ -I X X X = b" = b2 = b3 1][ 9 4 J[::] I v-1 = [ (72 1 a12 C