Table of Contents
1
1
2
Introduction to Vectors
1.1 Vectors and Linear Combinations.
1.2 Lengths and Dot Products.
1.3 Matrices .........
22
2 Solving Linear Equations
2.1 Vectors and Linear Equations .
2.2 The Idea of Elimination ...
2.3 Elimination Using Matrices .
2.4 Rules for Matrix Operations
2.5 Inverse Matrices .......
2.6 Elimination= Factorization: A= LU
2.7 Transposes and Permutations . . . . . .
31
31
46
58
70
83
97
109
3 Vector Spaces and Subspaces
3.1 Spaces of Vectors ....... . . . . . . . . . . . .
3.2 The Nullspace of A: Solving Ax = 0 and Rx = 0
3.3 The Complete Solution to Ax = b
3.4 Independence, Basis and Dimension
3.5 Dimensions of the Four Subspaces
123
123
135
150
164
181
4 Orthogonality
4.1 Orthogonality of the Four Subspaces
4.2 Projections . . . . . . . . . . . . .
4.3 Least Squares Approximations ...
4.4 Orthonormal Bases and Gram-Schmidt.
194
194
206
219
233
5 Determinants
5.1 The Properties of Determinants .
5.2 Permutations and Cofactors ...
5.3 Cramer's Rule, Inverses, and Volumes
247
247
258
273
iii
11
iv
Table of Contents
6 Eigenvalues and Eigenvectors
6.1 Introduction to Eigenvalues..
6.2 Diagonalizing a Matrix ....
6.3 Systems of Differential Equations
6.4 Symmetric Matrices........
6.5 Positive Definite Matrices.....
288
2 88
30 4
319
338
350
7 The Singular Value Decomposition (SVD)
7.1 Image Processing by Linear Algebra ....
7.2 Bases and Matrices in the SVD .......
7.3 Principal Component Analysis (PCA by the SVD).
7.4 The Geometry of the SVD .............
364
36 4
371
382
39 2
8 Linear Transformations
8.1 The Idea of a Linear Transformation
8.2 The Matrix of a Linear Transformation.
8.3 The Search for a Good Basis ......
401
40 1
411
42 1
9 Complex Vectors and Matrices
9.1 Complex Numbers ......
9.2 Hermitian and Unitary Matrices
9.3 The Fast Fourier Transform.
430
431
438
445
10 Applications
10.1 Graphs and Networks .............
10.2 Matrices in Engineering............
10.3 Markov Matrices, Population, and Economics
10.4 Linear Programming .............
10.5 Fourier Series: Linear Algebra for Functions.
10.6 Computer Graphics ........
10.7 Linear Algebra for Cryptography.......
452
452
46 2
474
483
49 0
49 6
50 2
11 Numerical Linear Algebra
11.1 Gaussian Elimination in Practice
11.2 Norms and Condition Numbers.
11.3 Iterative Methods and Preconditioners
508
50 8
518
52 4
12 Linear Algebra in Probability & Statistics
12.1 Mean, Variance, and Probability ......
12.2 Covariance Matrices and Joint Probabilities
12.3 Multivariate Gaussian and Weighted Least Squares
535
535
546
555
Matrix Factorizations
563
Index
565
Six Great Theorems/ Linear Algebra in a Nutshell
574
m+4m- UJ = m·
2
5
angle with v
less than 90 °
!
u. i
= cos 0
, v+w
u-
[-i l
X1 [ -
~
0
v-
l
+ X2
[
Ul
~
-1
w - [
l ~l
+
X3 [
1
n-
= [ :~ X3 -
X1
X2
l·
(1)
= b1
X1
- x1+x2
=b2
-x2 +x3 = b3
b
~
m
gives x
~
m
Cx
=
1
01
[ -1
0 -1
- 10
1
i[ l
X1
x2
X3
=
rn
V
1
right weights :r1, .'.C2, x3, can produce
weights
A- 1 b.
1
0
~1
0
0 ~1
1 0 0l
[ 1 1 0
1
1
1
[ Y2
Y1 l
=
Y3
[ C1C2 l
•
C3
-
1 1 ol
[ 321
74c
[11 1
o 0cl
011
[c2
3
·
--
--
The column picture combines the column vectors on the left side to
produce the vector b on the right side.
''
3[
t ]+ [ -~ ] = [
1~ ] ·
=
o [ ~ ] +o [
j]
+2 [ ; ] = [
i].
1
ol o0
0
0
I = [ 0
1
l
After ~
~
=l
=8
Before elimination
8y
=8
2
3
X
l
X
X
X
X
X
X
X
X
X
X
rn rn
=
1
=
~]
~l m~ m.
P23
1
o ol
0
1 0
= [0 0 1 .
0
0 0
-4
1
3
1
E = [a
ol ol0
b O l
o ol
1
F= [ 0 1 0 .
0
C
1
0 0
1 0
2 1
3
3
1
[3
X
X
X]
4
X
X
X
31
2
21 21
3 1
1
1
2
l
2
0
0
2
0 0
0 0
AB
= [~ : :] [~ : :] = [ 0
00x
00x
00
l·
Multiply
Ax = b by A - 1 .
Then
x = A - 1 Ax = A - 1 b.
FE=
[-t
20
0
1
-4
(6)
2
2
2
~
1 -~
~]
➔ [~
➔ [~
-~J
3
1
1
1
1 3
A= [ 1 3
1
l
2
1
1
1
1 3
B= [ 1 2
1
l
C = [ 11 11 11
1
1
3
l
6
[ 6
G]
()
,-1
C
[ 6.·
6 ]
6 6
1 [ ()
6
= :36
2
0
3
0
0 6
0
7
ll
2 1
A= [ 1 2 1
1
1
2
1
o ol
0
0
A= [ 2 1 3
1
and
ll
A= [ 11 21 2 .
1
2
3
-~i
1 -11
0
1 -1
0 0 1
1
T = [ -l
0
-1
2
-1
ol
-1
2
r- 1 =
ll
3
[2
2
2
1
1
1
1
1
<Al'J' ~ r
(uJ' j ( IJ,)
97
1
2
1
0
0
1
2
1
1
dn
U12/d1
U13/d1
1
U23/d2
ol
2 1
A= [ O 4 2 .
6
3
5
ol [e1
1 1
[ 112=
121
1
c
ol
3
5
1
A= [ 2 4 1 .
1
mnl
l [d
gl
ef ~ i
A= [~
a
a
~ ~
b
b
c
c
~] .
c
d
3by 3
2 -1 - O
l [3
2 ll [4 4 l .
1
2 4 2
case
[
The determinant (K)(4K- 1 ) = - 1
2
of this K is 4
0 -1
2
1
2
3
4
S
[~ ;] = [~ ~]
u;]
= [~ ~]
[1 2]
= 2 5 = S T and
D
[l 0]
= O IO = D T .
Ps2P21 = [
P21Ps2
1 1 1]
= [1
1
l
P =
[0o1 010 10ol
-.'• ·:.
PS=
PS=
[451 46 6]5
2
3
Q
o o
[451 46 6]5
2
3
1]
= [1 0 0
0 1 0
S = LDLT = [ 41
5
o1 ol0
1
1
l [10
[1 -14
-8
0
41
0
5l1 .
1
A=
o o
[1 2
0
4
6]3 .
5
=
and
and
S
ol .
2 -1
2 -1
0 -1
2
= [ -1
1 b
S= [b d
C
e
cl
e
j
.
o
1 1]
2
3
A= [ 1 0 1
1
2
ol
1
1
1
A= [ 2 4 1 .
and
4
o
1 2]
2
1
A= [ O 3 8 .
1
00
0
~j
- 1
1
0
'•ft
Plane
=
=
1
[0
0
ll [X1]
b2
x2 = [b1]
11 1
0 1
X3
and
and
b3
[1O 11
0 0
ll [X1]
b2
X2 = [b1]
1
0
ll [X1]
b2 .
x2 = [b1]
1 1
[O O 1
0 0 1
X3
b3
X3
b3
!]
1
becomes U
= [~
~
t t
1
3
R= [ 0 O
0 0
C
;t: 4
(b)
B =
[02 44 42] .
0
8
8
A=
[o0 oO 3ol
2
4
6
A=
[12 2 4]
4
B =
2 ].
[1-0 2-c
c
ll
1 42] [X
[011 330 O
1 6
X2
X3
X4
=
[ll
6
7
A
= [
~
~] and
-2 -3
b= [
~~] ·
b3
S1 -
~
[ -21
S2
-
l
[ 2
:
1
3
2
4
l
A= [ 3 8 2 .
0
0
0 62 ~]
0 0
6
0 2
0 0
and
=
1
A= [ l
o1 ol0
1 1 1
Vj
Vj
=
=
mv, mv, m m
=
h] H] [j]
v, =
V3
=
=
V4
=
[-I]
V4
=
V5
=
u] [j]
V6
=
A=
[101 33l 212]
and
U
=
[100 30l 02]1 .
B - [c
cl
cl] .
C
5
0
0
1
5 1
:;;t
J .• ·-
J
and
B-[J
A,
A=
[0o0 001 ol01
and
I
+A =
1
[0
0
1
1
0
ol1 .
1
V
W
w
I
C(A)
_l_
N(AT)
dimension
all vectors
orthogonal to
the columns
dimm-r
3rows
row S\lace
33
Suppose I give you eight vectors r1, r2, n1, n2, c1, c2, l 1, l2 in R 4.
o o ol
Pi= [0 0 0
0 0 1
I
b
I
I
a
p
=xa
and
e
= b-
p
= ( ig ' - !g ' - !9 )
.
=
2
column 8Pace
h1 :::::: 6
Pi :::::: 5
e b - 5 - Jr
best lin
=
=
=
l
~ ""'[>,,"/
/
/
/
/
'
''
a
~
H]
and
b
~
U]
and
c
~
H] .
1
- 1
1
1
-~]
1 - 1 - 1
-1 -1
1
=
=
AT A=
[99 189] = [11 10] [9 9] [10 11] .
v12
-v12
-1 -1]
-1 -1
1 -1
-1
1
.
A=
o o
[1 0
1]
A=
l
[1
o
1 1]
1
1
B = [1 0
0
0 1 0
1
2
ll
2
1 2 3
1
0
A=
C=
l
[1
1
[l 2 3]
2
2
3
3
3
3
1
1
1
.
ll
1 .
1
<let
[l
n+l
n
----=
21 1
[1 1 2 1
2
1
1
1 1
2
j
B
=
[14 42 3]4
5 6
7
B=
[41 25 63] .
7 0
0
[-BI
M= [-B0 A]I = [ABO A]
I
OJ .
I
1 0
0 1
0
0
0
1
1
0
0
0
Find x
=
A - 1b
Check [
detB2
x2 = -d-et_A_
~
!][_~ ]= [ ~ ]
L
Ya
----
1
2
(0,0)
X1
Y1
X2
Y2
X3
Y3
-
I
I
1 4
A= [ 2 3 97
2
2
8
l
1 1
A= [ 1 1 21
1
1
1
l
Ci
1
2
(a) A= [ O 3
0 7
1 1 4]
A= [l 2 2
1 2 5
ol
O
1
(b)
A
= [-
2 -1
ol
0 -1
2
l
2 - 1
6 -3
C= [ .
.
.'
•
•
~.
·•
I
. '\ -
.
.
for
>-1 = 0
-
::::-
-
.:;;.;:;;;;
-
A
=
[01 21 ]
and
A-
1
=
[-1112/2 10 ].
l
A= [ 31 62 3
4 8 4
P=
and
B = P APT =
[L
0 0
[682 431 43]1 .
....
Ax1
\
I
=
x1
,\1
x - 1 Ax
= A = [
[~~] [~i] A
X
=
>-n
[1 5] = [1 l][1 ][1-lJ= [1
0
6
k
0
1
6k
O
1
0
6k 6k
] =XA.
l]=
Ak
.
1
------- -
Forward
(llF)
Yn+l = Yn + flt Zn
Zn+I = Zn - flt Yn
[ 1
becomes U n+l = - flt
-Yn-1
- Yn
- Yn + l
flt][z:
1
Y, ]
= AUn ·
(12)
Backward
Yn+ l
Zn+l
= Yn + .6.t Zn+ l
= Zn - .6.t Yn+ l
--
--
al [o1 0o0o0ol [alb
[
dt ~ = ~ ~ ~ ~
~
db
d [ y]
dt y'
[
0 1] [y'y] .
= -9 6
u
=
[t]
B
o
= [o
-4]
0
.
Trapezoidal
[
1
6;.t/ 2
-b;.t/2 ] [ Yn+l ] _ [
1
1
Zn+l
- b;.t/2
6;.t/2 ] [ Yn ]
1
Zn ·
A3
=
2
[ -1
0
-1
2
- 1
o
-1
2
l
1
...
9
B = [ 12
1
1
~1
-1
0
S
2
= [- 1
0
ol
- 1
2 - 1
-1
2
T
bl
2 -1
2 - 1 .
b - 1
2
= [-1
ol [- O11 - o11 o]01
O
-l
O
O - 1
X
x+y =X
J2
S4
10] .
= [ 101 101
2
2
o]
0
3
8
S= [ 2 5 3 .
5
=
C
[
1
1
1 1]
C
1
1
C
T=
[1 d2 3]
2
3
4
4
5
.
'
'
t 3 ol
T= [ 3 t 4 .
0 4 t
S
=
9
[0
0
o1 ol
2
2
8
1
S= [ 1 21 2ll .
1
2
7
101 - •
•
10-1 < - - - - - ~_ _ _ _ ___,____ _ _
0
10
20
30
_ _ _ J L __ _ ____,
40
10-20 < - - - - - - ' - - - - - - ~ - - - - - " - - - - - '
0
10
20
30
40
AT A= [ 20
10
10 ]
5
~ [1]
Av2
-
~
[
-f ]
xTATAx
xTx
xTSx
xTx ·
2
(/1
·
(J/ .
0/ .
2
2
+ ··· + (JR2 1s
99.10
or 99.9.10
of <T1
+ ••· + <T.,,.
'-.__/
V
'.
n J5 [~
=
-~]
~] . Then A = Q S.
o o0 l [2
0 3o o
0l =
1/3
0
0
000
[1 o ol [i ~].
0 1 0
000
=
AA+p= ~~
r
A+ Ax+=
The input is
v
= (v1 , v2 , V3).
v,~~v)
\ -~
,;.
-), ~ w )
T(v3)
➔
v,
lnp~t
basis
[vi v = [ 33 6]
8
2]
[ w 1 w 2 ] [ B ] -_
[v
l
v 2 ] 1S
• [ 3
1
0
2 ] [ 11
23 ] -_ [ 3
3
6
8] .
0 1 0
Ac= [ 0 0 2
0 0 0
2
?
=
o o
[0 0
0
0
1
0
0
l
0l
~
ll l
.\
1
~:
= ,\
2
ll
.\
1
~:
2
= ,\ x
4
when A
=
1.
J
1
[ 02 21 0]
=
0
J=
l
0
2
O 01 00 00
0
1
0
J
K=
l
O 01 01 00
0
0
0
J
.'
-.
,,
I,
=
iz l2
=
12
rcos0
+ 12 = 2
+ irsin0 = v'2 ( ~ ) + iv'2 ( ~ ) = 1 + i .
becomes
1
[ ~ - ~] [ ~] = [ ] .
[
1
e-Li/3
e -21ri/ 3
l
.
S
= [. 0
i
+1
11
i]
2n
\8
~3] [~
w6
w9
1
l
1
w
w2
w3
1
i
i2
i3
1
1
]
[1
1
]
[1
]
[
= 1
-1 -i
~ i\
1 1.
i
F4
F 1024
=
[ h12
I
1 i2
D512] [Fs12
512 - D 512
1
F,512
]
[o00 001 o01 ol01
1 0
0
0
[11 1i
1 i2
1 i3
i12
i4
i6
i13
i6
i9
l
=
[11 1i
1 i2
1 i3
i12
i4
i6
i13
i6
i9
l [>11
A2
A3
ol
0
0
-+
[-1
0
0
=
s
Xz a<==-- - - - ------ - - - - - - - - ""9 X3
Y3
=
[U1]
U2
U3
1[1
=-
16
1
l
l
0(.6.x) 2
C
=
A[-~] = .75 [-n .
Uo = [:~~] = [:~] + .18
[-~J.
A=[l0 .8.2 ] A= [·.82 10] A=[t
4
1
4
f
2
l
4
!]
4
1
2
·
481
1
0
0
0
0
1
0
0
The dot product of x ::: 0 and s
=c
- A '1' y ::: 0 gave x'I' s 2 0. This is
xT AT y
~
xT c.
Algebra
(b)
Graphics
by
[
2 0
2 1
] -1
=2
- 1
[
1 - 0
2
-2
]
_
1
= 2 [ -2
-0
2
]
_
=
[
2 0
2 1
]
(mod 3) ·
o
A= [ 1l o
1 O
1
1
1
l
B
= [ 01 1l o
1
1
0
1
l
1
1
1
1
0
0
C= [ O O 1
l
1
-11 -1
1 - ~ .
=
- --
3 2
3
2
2
2
1
1
ForA-
1
2
ll
0
2
------+
[1 1 ll
0
0
3
2
2
4
3
a32
=0
a32
.6
Q,,A
=
r -.8
0
=
2
Q,1Q21A
=[
1 0
ll
A= [ 2 2 0 .
0 2 0
PKPT
2I
= [ DT
'•
XT A Ax _ _ _ _ _ _ _ _ _ _ _ _ _ __
T
- x'T"x
-
------
--·
...
F = l
.
a
✓
- - 2N.,,,..;.,
_
....__ _ _ _ _ _ ___,__
--I-----+---+----+---+--
(
1
1) 4
2+ 2
1
4
6
4
1
= 16 + 16 + 16 + 16 + 16 =
1.
p = [ Pu
P21
P 12 ] =
P22
[
¾ ¾
.!
4
.!
4
l
n
i
i
i
i
-
I
(x - m1) 2
(y - m2) 2
2o-f
2o-~
-I
X
X
X
= b"
= b2
= b3
1][ 9 4
J[::]
I
v-1
=
[
(72
1
a12
C