Elementary Linear Algebra
Kuttler
June 14, 2016
2
CONTENTS
1 Some Prerequisite Topics
1.1 Sets And Set Notation . . . . . . . . . .
1.2 Well Ordering And Induction . . . . . .
1.3 The Complex Numbers . . . . . . . . . .
1.4 Polar Form Of Complex Numbers . . . .
1.5 Roots Of Complex Numbers . . . . . . .
1.6 The Quadratic Formula . . . . . . . . .
1.7 The Complex Exponential . . . . . . . .
1.8 The Fundamental Theorem Of Algebra .
1.9 Exercises . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
1
2
4
6
6
8
9
9
11
Algebra in Fn . . . . . . . . . . . . . . . . . . . . . .
Geometric Meaning Of Vectors . . . . . . . . . . . .
Geometric Meaning Of Vector Addition . . . . . . .
Distance Between Points In Rn Length Of A Vector
Geometric Meaning Of Scalar Multiplication . . . .
Parametric Lines . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . .
Vectors And Physics . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
13
14
15
16
17
20
20
21
22
23
3 Vector Products
3.1 The Dot Product . . . . . . . . . . . . . . . . . . . .
3.2 The Geometric Significance Of The Dot Product . .
3.2.1 The Angle Between Two Vectors . . . . . . .
3.2.2 Work And Projections . . . . . . . . . . . . .
3.2.3 The Inner Product And Distance In Cn . . .
3.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . .
3.4 The Cross Product . . . . . . . . . . . . . . . . . . .
3.4.1 The Distributive Law For The Cross Product
3.4.2 The Box Product . . . . . . . . . . . . . . . .
3.4.3 Another Proof Of The Distributive Law . . .
3.5 The Vector Identity Machine . . . . . . . . . . . . .
3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
25
25
27
27
28
30
33
34
37
38
39
39
41
4 Systems Of Equations
4.1 Systems Of Equations, Geometry . . . . . . .
4.2 Systems Of Equations, Algebraic Procedures
4.2.1 Elementary Operations . . . . . . . .
4.2.2 Gauss Elimination . . . . . . . . . . .
4.2.3 Balancing Chemical Reactions . . . .
4.2.4 Dimensionless Variables∗ . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
43
43
45
45
47
56
57
2 Fn
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
CONTENTS
4
4.3
Exercises
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Matrices
5.1 Matrix Arithmetic . . . . . . . . . . . . . . .
5.1.1 Addition And Scalar Multiplication Of
5.1.2 Multiplication Of Matrices . . . . . .
5.1.3 The ij th Entry Of A Product . . . . .
5.1.4 Properties Of Matrix Multiplication .
5.1.5 The Transpose . . . . . . . . . . . . .
5.1.6 The Identity And Inverses . . . . . . .
5.1.7 Finding The Inverse Of A Matrix . . .
5.2 Exercises . . . . . . . . . . . . . . . . . . . .
. . . . . .
Matrices
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
6 Determinants
6.1 Basic Techniques And Properties . . . . . . . . . . .
6.1.1 Cofactors And 2 × 2 Determinants . . . . . .
6.1.2 The Determinant Of A Triangular Matrix . .
6.1.3 Properties Of Determinants . . . . . . . . . .
6.1.4 Finding Determinants Using Row Operations
6.2 Applications . . . . . . . . . . . . . . . . . . . . . . .
6.2.1 A Formula For The Inverse . . . . . . . . . .
6.2.2 Cramer’s Rule . . . . . . . . . . . . . . . . .
6.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . .
60
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
65
65
67
70
72
73
74
76
80
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
87
87
87
90
91
92
94
94
97
99
7 The Mathematical Theory Of Determinants∗
7.0.1 The Function sgn . . . . . . . . . . . . . . . . .
7.1 The Determinant . . . . . . . . . . . . . . . . . . . . .
7.1.1 The Definition . . . . . . . . . . . . . . . . . .
7.1.2 Permuting Rows Or Columns . . . . . . . . . .
7.1.3 A Symmetric Definition . . . . . . . . . . . . .
7.1.4 The Alternating Property Of The Determinant
7.1.5 Linear Combinations And Determinants . . . .
7.1.6 The Determinant Of A Product . . . . . . . . .
7.1.7 Cofactor Expansions . . . . . . . . . . . . . . .
7.1.8 Formula For The Inverse . . . . . . . . . . . . .
7.1.9 Cramer’s Rule . . . . . . . . . . . . . . . . . .
7.1.10 Upper Triangular Matrices . . . . . . . . . . .
7.2 The Cayley Hamilton Theorem∗ . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
105
105
107
107
107
108
109
110
110
111
112
113
114
114
8 Rank Of A Matrix
8.1 Elementary Matrices . . . . . . . . . . . . . . . . . . . . . .
8.2 THE Row Reduced Echelon Form Of A Matrix . . . . . . .
8.3 The Rank Of A Matrix . . . . . . . . . . . . . . . . . . . .
8.3.1 The Definition Of Rank . . . . . . . . . . . . . . . .
8.3.2 Finding The Row And Column Space Of A Matrix .
8.4 A Short Application To Chemistry . . . . . . . . . . . . . .
8.5 Linear Independence And Bases . . . . . . . . . . . . . . . .
8.5.1 Linear Independence And Dependence . . . . . . . .
8.5.2 Subspaces . . . . . . . . . . . . . . . . . . . . . . . .
8.5.3 Basis Of A Subspace . . . . . . . . . . . . . . . . . .
8.5.4 Extending An Independent Set To Form A Basis . .
8.5.5 Finding The Null Space Or Kernel Of A Matrix . .
8.5.6 Rank And Existence Of Solutions To Linear Systems
8.6 Fredholm Alternative . . . . . . . . . . . . . . . . . . . . . .
8.6.1 Row, Column, And Determinant Rank . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
117
123
127
127
129
131
132
132
135
137
140
141
143
143
144
CONTENTS
8.7
Exercises
5
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
9 Linear Transformations
9.1 Linear Transformations . . . . . . . . . . . . . . . . .
9.2 Constructing The Matrix Of A Linear Transformation
9.2.1 Rotations in R2 . . . . . . . . . . . . . . . . . .
9.2.2 Rotations About A Particular Vector . . . . . .
9.2.3 Projections . . . . . . . . . . . . . . . . . . . .
9.2.4 Matrices Which Are One To One Or Onto . . .
9.2.5 The General Solution Of A Linear System . . .
9.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
153
153
155
156
157
159
160
161
164
10 A Few Factorizations
10.1 Definition Of An LU factorization . . . . . . .
10.2 Finding An LU Factorization By Inspection . .
10.3 Using Multipliers To Find An LU Factorization
10.4 Solving Systems Using An LU Factorization . .
10.5 Justification For The Multiplier Method . . . .
10.6 The P LU Factorization . . . . . . . . . . . . .
10.7 The QR Factorization . . . . . . . . . . . . . .
10.8 Exercises . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
171
171
171
172
173
174
176
178
181
11 Linear Programming
11.1 Simple Geometric Considerations .
11.2 The Simplex Tableau . . . . . . . .
11.3 The Simplex Algorithm . . . . . .
11.3.1 Maximums . . . . . . . . .
11.3.2 Minimums . . . . . . . . . .
11.4 Finding A Basic Feasible Solution .
11.5 Duality . . . . . . . . . . . . . . .
11.6 Exercises . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
185
185
186
190
190
192
199
201
205
12 Spectral Theory
12.1 Eigenvalues And Eigenvectors Of A Matrix . . . . .
12.1.1 Definition Of Eigenvectors And Eigenvalues .
12.1.2 Finding Eigenvectors And Eigenvalues . . . .
12.1.3 A Warning . . . . . . . . . . . . . . . . . . .
12.1.4 Triangular Matrices . . . . . . . . . . . . . .
12.1.5 Defective And Nondefective Matrices . . . . .
12.1.6 Diagonalization . . . . . . . . . . . . . . . . .
12.1.7 The Matrix Exponential . . . . . . . . . . . .
12.1.8 Complex Eigenvalues . . . . . . . . . . . . . .
12.2 Some Applications Of Eigenvalues And Eigenvectors
12.2.1 Principal Directions . . . . . . . . . . . . . .
12.2.2 Migration Matrices . . . . . . . . . . . . . . .
12.2.3 Discrete Dynamical Systems . . . . . . . . . .
12.3 The Estimation Of Eigenvalues . . . . . . . . . . . .
12.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
207
207
207
208
211
213
214
217
221
222
223
223
225
228
232
234
13 Matrices And The Inner Product
13.1 Symmetric And Orthogonal Matrices . . . . . . . .
13.1.1 Orthogonal Matrices . . . . . . . . . . . . .
13.1.2 Symmetric And Skew Symmetric Matrices
13.1.3 Diagonalizing A Symmetric Matrix . . . . .
13.2 Fundamental Theory And Generalizations . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
241
241
241
243
248
251
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
CONTENTS
6
. . . . .
Process
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
251
255
256
259
261
262
263
266
268
270
271
14 Numerical Methods For Solving Linear Systems
14.1 Iterative Methods For Linear Systems . . . . . . .
14.1.1 The Jacobi Method . . . . . . . . . . . . .
14.1.2 The Gauss Seidel Method . . . . . . . . . .
14.2 The Operator Norm∗ . . . . . . . . . . . . . . . . .
14.3 The Condition Number∗ . . . . . . . . . . . . . . .
14.4 Exercises . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
279
279
280
282
286
288
290
Eigenvalue Problem
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
293
293
295
302
303
307
307
311
315
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
Vector
. . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Spaces .
. . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
319
319
321
321
328
328
332
337
339
340
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
345
345
346
348
351
354
355
357
13.3
13.4
13.5
13.6
13.7
13.8
13.2.1 Block Multiplication Of Matrices .
13.2.2 Orthonormal Bases, Gram Schmidt
13.2.3 Schur’s Theorem . . . . . . . . . .
Least Square Approximation . . . . . . .
13.3.1 The Least Squares Regression Line
13.3.2 The Fredholm Alternative . . . . .
The Right Polar Factorization∗ . . . . . .
The Singular Value Decomposition . . . .
Approximation In The Frobenius Norm∗ .
Moore Penrose Inverse∗ . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . .
15 Numerical Methods For Solving The
15.1 The Power Method For Eigenvalues .
15.2 The Shifted Inverse Power Method .
15.2.1 Complex Eigenvalues . . . . .
15.3 The Rayleigh Quotient . . . . . . . .
15.4 The QR Algorithm . . . . . . . . . .
15.4.1 Basic Considerations . . . . .
15.4.2 The Upper Hessenberg Form
15.5 Exercises . . . . . . . . . . . . . . .
16 Vector Spaces
16.1 Algebraic Considerations . . . . . . . . . . . .
16.2 Exercises . . . . . . . . . . . . . . . . . . . .
16.3 Linear Independence And Bases . . . . . . . .
16.4 Vector Spaces And Fields∗ . . . . . . . . . . .
16.4.1 Irreducible Polynomials . . . . . . . .
16.4.2 Polynomials And Fields . . . . . . . .
16.4.3 The Algebraic Numbers . . . . . . . .
16.4.4 The Lindemannn Weierstrass Theorem
16.5 Exercises . . . . . . . . . . . . . . . . . . . .
17 Inner Product Spaces
17.1 Basic Definitions And Examples . . .
17.1.1 The Cauchy Schwarz Inequality
17.2 The Gram Schmidt Process . . . . . .
17.3 Approximation And Least Squares . .
17.4 Fourier Series . . . . . . . . . . . . . .
17.5 The Discreet Fourier Transform . . . .
17.6 Exercises . . . . . . . . . . . . . . . .
. . .
And
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
And
. . .
. . . .
Norms
. . . .
. . . .
. . . .
. . . .
. . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
CONTENTS
7
18 Linear Transformations
18.1 Matrix Multiplication As A Linear Transformation . . . . .
18.2 L (V, W ) As A Vector Space . . . . . . . . . . . . . . . . . .
18.3 Eigenvalues And Eigenvectors Of Linear Transformations .
18.4 Block Diagonal Matrices . . . . . . . . . . . . . . . . . . . .
18.5 The Matrix Of A Linear Transformation . . . . . . . . . . .
18.5.1 Some Geometrically Defined Linear Transformations
18.5.2 Rotations About A Given Vector . . . . . . . . . . .
18.6 The Matrix Exponential, Differential Equations ∗ . . . . . .
18.6.1 Computing A Fundamental Matrix . . . . . . . . . .
18.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
A The Jordan Canonical Form*
363
363
363
365
369
373
381
381
383
388
391
397
B Directions For Computer Algebra Systems
B.1 Finding Inverses . . . . . . . . . . . . . . .
B.2 Finding Row Reduced Echelon Form . . . .
B.3 Finding P LU Factorizations . . . . . . . . .
B.4 Finding QR Factorizations . . . . . . . . . .
B.5 Finding The Singular Value Decomposition
B.6 Use Of Matrix Calculator On Web . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
405
405
405
405
405
405
405
C Answers To Selected Exercises
C.1 Exercises 11 . . . . . . . . . .
C.2 Exercises 23 . . . . . . . . . .
C.3 Exercises 33 . . . . . . . . . .
C.4 Exercises 41 . . . . . . . . . .
C.5 Exercises 60 . . . . . . . . . .
C.6 Exercises 80 . . . . . . . . . .
C.7 Exercises 99 . . . . . . . . . .
C.8 Exercises 147 . . . . . . . . .
C.9 Exercises 164 . . . . . . . . .
C.10 Exercises 181 . . . . . . . . .
C.11 Exercises 205 . . . . . . . . .
C.12 Exercises 234 . . . . . . . . .
C.13 Exercises 271 . . . . . . . . .
C.14 Exercises 290 . . . . . . . . .
C.15 Exercises 315 . . . . . . . . .
C.16 Exercises 321 . . . . . . . . .
C.17 Exercises 340 . . . . . . . . .
C.18 Exercises 357 . . . . . . . . .
C.19 Exercises 391 . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
409
409
411
412
412
412
413
415
416
418
420
421
422
424
426
426
427
428
428
434
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
CONTENTS
Preface
This is an introduction to linear algebra. The main part of the book features row operations and
everything is done in terms of the row reduced echelon form and specific algorithms. At the end, the
more abstract notions of vector spaces and linear transformations on vector spaces are presented.
However, this is intended to be a first course in linear algebra for students who are sophomores
or juniors who have had a course in one variable calculus and a reasonable background in college
algebra. I have given complete proofs of all the fundamental ideas, but some topics such as Markov
matrices are not complete in this book but receive a plausible introduction. The book contains a
complete treatment of determinants and a simple proof of the Cayley Hamilton theorem although
these are optional topics. The Jordan form is presented as an appendix. I see this theorem as the
beginning of more advanced topics in linear algebra and not really part of a beginning linear algebra
course. There are extensions of many of the topics of this book in my on line book [13]. I have also
not emphasized that linear algebra can be carried out with any field although there is an optional
section on this topic, most of the book being devoted to either the real numbers or the complex
numbers. It seems to me this is a reasonable specialization for a first course in linear algebra.
Linear algebra is a wonderful interesting subject. It is a shame when it degenerates into nothing
more than a challenge to do the arithmetic correctly. It seems to me that the use of a computer
algebra system can be a great help in avoiding this sort of tedium. I don’t want to over emphasize
the use of technology, which is easy to do if you are not careful, but there are certain standard
things which are best done by the computer. Some of these include the row reduced echelon form,
P LU factorization, and QR factorization. It is much more fun to let the machine do the tedious
calculations than to suffer with them yourself. However, it is not good when the use of the computer
algebra system degenerates into simply asking it for the answer without understanding what the
oracular software is doing. With this in mind, there are a few interactive links which explain how
to use a computer algebra system to accomplish some of these more tedious standard tasks. These
are obtained by clicking on the symbol I. I have included how to do it using maple and scientific
notebook because these are the two systems I am familiar with and have on my computer. Also, I
have included the very easy to use matrix calculator which is available on the web. Other systems
could be featured as well. It is expected that people will use such computer algebra systems to do
the exercises in this book whenever it would be helpful to do so, rather than wasting huge amounts
of time doing computations by hand. However, this is not a book on numerical analysis so no effort
is made to consider many important numerical analysis issues.
I appreciate those who have found errors and needed corrections over the years that this has
been available.
There is a pdf file of this book on my web page http://www.math.byu.edu/klkuttle/ along with
some other materials soon to include another set of exercises, and a more advanced linear algebra
book. This book, as well as the more advanced text, is also available as an electronic version at
http://www.saylor.org/archivedcourses/ma211/ where it is used as an open access textbook. In
addition, it is available for free at BookBoon under their linear algebra offerings.
c
Elementary Linear Algebra ⃝2012
by Kenneth Kuttler, used under a Creative Commons Attribution(CCBY) license made possible by funding The Saylor Foundation’s Open Textbook Challenge
in order to be incorporated into Saylor.org’s collection of open courses available at
http://www.Saylor.org. Full license terms may be viewed at:
http://creativecommons.org/licenses/by/3.0/.
i
ii
CONTENTS
Chapter 1
Some Prerequisite Topics
The reader should be familiar with most of the topics in this chapter. However, it is often the case
that set notation is not familiar and so a short discussion of this is included first. Complex numbers
are then considered in somewhat more detail. Many of the applications of linear algebra require the
use of complex numbers, so this is the reason for this introduction.
1.1
Sets And Set Notation
A set is just a collection of things called elements. Often these are also referred to as points in
calculus. For example {1, 2, 3, 8} would be a set consisting of the elements 1,2,3, and 8. To indicate
that 3 is an element of {1, 2, 3, 8} , it is customary to write 3 ∈ {1, 2, 3, 8} . 9 ∈
/ {1, 2, 3, 8} means 9 is
not an element of {1, 2, 3, 8} . Sometimes a rule specifies a set. For example you could specify a set
as all integers larger than 2. This would be written as S = {x ∈ Z : x > 2} . This notation says: the
set of all integers, x, such that x > 2.
If A and B are sets with the property that every element of A is an element of B, then A is a subset
of B. For example, {1, 2, 3, 8} is a subset of {1, 2, 3, 4, 5, 8} , in symbols, {1, 2, 3, 8} ⊆ {1, 2, 3, 4, 5, 8} .
It is sometimes said that “A is contained in B” or even “B contains A”. The same statement about
the two sets may also be written as {1, 2, 3, 4, 5, 8} ⊇ {1, 2, 3, 8}.
The union of two sets is the set consisting of everything which is an element of at least one of
the sets, A or B. As an example of the union of two sets {1, 2, 3, 8} ∪ {3, 4, 7, 8} = {1, 2, 3, 4, 7, 8}
because these numbers are those which are in at least one of the two sets. In general
A ∪ B ≡ {x : x ∈ A or x ∈ B} .
Be sure you understand that something which is in both A and B is in the union. It is not an
exclusive or.
The intersection of two sets, A and B consists of everything which is in both of the sets. Thus
{1, 2, 3, 8} ∩ {3, 4, 7, 8} = {3, 8} because 3 and 8 are those elements the two sets have in common. In
general,
A ∩ B ≡ {x : x ∈ A and x ∈ B} .
The symbol [a, b] where a and b are real numbers, denotes the set of real numbers x, such that
a ≤ x ≤ b and [a, b) denotes the set of real numbers such that a ≤ x < b. (a, b) consists of the set of
real numbers x such that a < x < b and (a, b] indicates the set of numbers x such that a < x ≤ b.
[a, ∞) means the set of all numbers x such that x ≥ a and (−∞, a] means the set of all real numbers
which are less than or equal to a. These sorts of sets of real numbers are called intervals. The two
points a and b are called endpoints of the interval. Other intervals such as (−∞, b) are defined by
analogy to what was just explained. In general, the curved parenthesis indicates the end point it
sits next to is not included while the square parenthesis indicates this end point is included. The
reason that there will always be a curved parenthesis next to ∞ or −∞ is that these are not real
numbers. Therefore, they cannot be included in any set of real numbers.
1
2
CHAPTER 1. SOME PREREQUISITE TOPICS
A special set which needs to be given a name is the empty set also called the null set, denoted
by ∅. Thus ∅ is defined as the set which has no elements in it. Mathematicians like to say the empty
set is a subset of every set. The reason they say this is that if it were not so, there would have to
exist a set A, such that ∅ has something in it which is not in A. However, ∅ has nothing in it and so
the least intellectual discomfort is achieved by saying ∅ ⊆ A.
If A and B are two sets, A \ B denotes the set of things which are in A but not in B. Thus
A \ B ≡ {x ∈ A : x ∈
/ B} .
Set notation is used whenever convenient.
To illustrate the use of this notation relative to intervals consider three examples of inequalities.
Their solutions will be written in the notation just described.
Example 1.1.1 Solve the inequality 2x + 4 ≤ x − 8
x ≤ −12 is the answer. This is written in terms of an interval as (−∞, −12].
Example 1.1.2 Solve the inequality (x + 1) (2x − 3) ≥ 0.
The solution is x ≤ −1 or x ≥
3
3
. In terms of set notation this is denoted by (−∞, −1] ∪ [ , ∞).
2
2
Example 1.1.3 Solve the inequality x (x + 2) ≥ −4.
This is true for any value of x. It is written as R or (−∞, ∞) .
1.2
Well Ordering And Induction
Mathematical induction and well ordering are two extremely important principles in math. They
are often used to prove significant things which would be hard to prove otherwise.
Definition 1.2.1 A set is well ordered if every nonempty subset S, contains a smallest element z
having the property that z ≤ x for all x ∈ S.
Axiom 1.2.2 Any set of integers larger than a given number is well ordered.
In particular, the natural numbers defined as
N ≡ {1, 2, · · · }
is well ordered.
The above axiom implies the principle of mathematical induction. The symbol Z denotes the set
of all integers. Note that if a is an integer, then there are no integers between a and a + 1.
Theorem 1.2.3 (Mathematical induction) A set S ⊆ Z, having the property that a ∈ S and n+1 ∈ S
whenever n ∈ S contains all integers x ∈ Z such that x ≥ a.
Proof: Let T consist of all integers larger than or equal to a which are not in S. The theorem
will be proved if T = ∅. If T ̸= ∅ then by the well ordering principle, there would have to exist a
smallest element of T, denoted as b. It must be the case that b > a since by definition, a ∈
/ T. Thus
b ≥ a + 1, and so b − 1 ≥ a and b − 1 ∈
/ S because if b − 1 ∈ S, then b − 1 + 1 = b ∈ S by the assumed
property of S. Therefore, b − 1 ∈ T which contradicts the choice of b as the smallest element of T.
(b − 1 is smaller.) Since a contradiction is obtained by assuming T ̸= ∅, it must be the case that
T = ∅ and this says that every integer at least as large as a is also in S. Mathematical induction is a very useful device for proving theorems about the integers.
Example 1.2.4 Prove by induction that
∑n
k=1
k2 =
n (n + 1) (2n + 1)
.
6
1.2. WELL ORDERING AND INDUCTION
3
I
By inspection, if n = 1 then the formula is true. The sum yields 1 and so does the formula on
the right. Suppose this formula is valid for some n ≥ 1 where n is an integer. Then
n+1
∑
k=1
k2 =
n
∑
2
k 2 + (n + 1) =
k=1
n (n + 1) (2n + 1)
2
+ (n + 1) .
6
The step going from the first to the second line is based on the assumption that the formula is true
for n. This is called the induction hypothesis. Now simplify the expression in the second line,
n (n + 1) (2n + 1)
2
+ (n + 1) .
6
This equals
(
(n + 1)
and
n (2n + 1)
+ (n + 1)
6
)
n (2n + 1)
6 (n + 1) + 2n2 + n
(n + 2) (2n + 3)
+ (n + 1) =
=
6
6
6
Therefore,
n+1
∑
k=1
k2 =
(n + 1) (n + 2) (2n + 3)
(n + 1) ((n + 1) + 1) (2 (n + 1) + 1)
=
,
6
6
showing the formula holds for n+1 whenever it holds for n. This proves the formula by mathematical
induction.
Example 1.2.5 Show that for all n ∈ N,
1 3
2n − 1
1
· ···
<√
.
2 4
2n
2n + 1
If n = 1 this reduces to the statement that
1
1
< √ which is obviously true. Suppose then that
2
3
the inequality holds for n. Then
√
1 3
2n + 1
2n + 1
2n − 1 2n + 1
1
· ···
·
<√
=
.
2 4
2n
2n + 2
2n + 2
2n + 1 2n + 2
1
The theorem will be proved if this last expression is less than √
. This happens if and only if
2n + 3
(
1
√
2n + 3
2
)2
=
1
2n + 1
>
2
2n + 3
(2n + 2)
which occurs if and only if (2n + 2) > (2n + 3) (2n + 1) and this is clearly true which may be seen
from expanding both sides. This proves the inequality.
Lets review the process just used. If S is the set of integers at least as large as 1 for which the
formula holds, the first step was to show 1 ∈ S and then that whenever n ∈ S, it follows n + 1 ∈ S.
Therefore, by the principle of mathematical induction, S contains [1, ∞) ∩ Z, all positive integers.
In doing an inductive proof of this sort, the set S is normally not mentioned. One just verifies the
steps above. First show the thing is true for some a ∈ Z and then verify that whenever it is true for
m it follows it is also true for m + 1. When this has been done, the theorem has been proved for all
m ≥ a.
4
CHAPTER 1. SOME PREREQUISITE TOPICS
1.3
The Complex Numbers
Recall that a real number is a point on the real number line. Just as a real number should be
considered as a point on the line, a complex number is considered a point in the plane which can
be identified in the usual way using the Cartesian coordinates of the point. Thus (a, b) identifies a point whose x coordinate is a and whose y coordinate is b. In dealing with complex numbers, such a point is written as a + ib. For example, in the following picture, I have graphed
the point 3 + 2i. You see it corresponds to the point in the plane whose coordinates are (3, 2) .
Multiplication and addition are defined in the most obvious way subject to
the convention that i2 = −1. Thus,
3 + 2i
(a + ib) + (c + id) = (a + c) + i (b + d)
and
(a + ib) (c + id)
=
ac + iad + ibc + i2 bd
= (ac − bd) + i (bc + ad) .
Every non zero complex number a + ib, with a2 + b2 ̸= 0, has a unique multiplicative inverse.
1
a − ib
a
b
= 2
= 2
−i 2
.
2
2
a + ib
a +b
a +b
a + b2
You should prove the following theorem.
Theorem 1.3.1 The complex numbers with multiplication and addition defined as above form a
field satisfying all the field axioms. These are the following list of properties.
1. x + y = y + x, (commutative law for addition)
2. x + 0 = x, (additive identity).
3. For each x ∈ R, there exists −x ∈ R such that x + (−x) = 0, (existence of additive inverse).
4. (x + y) + z = x + (y + z) , (associative law for addition).
5. xy = yx, (commutative law for multiplication). You could write this as x × y = y × x.
6. (xy) z = x (yz) , (associative law for multiplication).
7. 1x = x, (multiplicative identity).
8. For each x ̸= 0, there exists x−1 such that xx−1 = 1.(existence of multiplicative inverse).
9. x (y + z) = xy + xz.(distributive law).
Something which satisfies these axioms is called a field. Linear algebra is all about fields, although
in this book, the field of most interest will be the field of complex numbers or the field of real numbers.
You have seen in earlier courses that the real numbers also satisfies the above axioms. The field
of complex numbers is denoted as C and the field of real numbers is denoted as R. An important
construction regarding complex numbers is the complex conjugate denoted by a horizontal line above
the number. It is defined as follows.
a + ib ≡ a − ib.
What it does is reflect a given complex number across the x axis. Algebraically, the following formula
is easy to obtain.
)
(
a + ib (a + ib) = (a − ib) (a + ib)
= a2 + b2 − i (ab − ab) = a2 + b2 .
1.3. THE COMPLEX NUMBERS
5
Definition 1.3.2 Define the absolute value of a complex number as follows.
√
|a + ib| ≡ a2 + b2 .
Thus, denoting by z the complex number z = a + ib,
|z| = (zz)
1/2
.
Also from the definition, if z = x+iy and w = u+iv are two complex numbers, then |zw| = |z| |w| .
You should verify this. I
Notation 1.3.3 Recall the following notation.
n
∑
aj ≡ a1 + · · · + an
j=1
There is also a notation which is used to denote a product.
n
∏
aj ≡ a1 a2 · · · an
j=1
The triangle inequality holds for the absolute value for complex numbers just as it does for the
ordinary absolute value.
Proposition 1.3.4 Let z, w be complex numbers. Then the triangle inequality holds.
|z + w| ≤ |z| + |w| , ||z| − |w|| ≤ |z − w| .
Proof: Let z = x + iy and w = u + iv. First note that
zw = (x + iy) (u − iv) = xu + yv + i (yu − xv)
and so |xu + yv| ≤ |zw| = |z| |w| .
2
|z + w| = (x + u + i (y + v)) (x + u − i (y + v))
2
2
= (x + u) + (y + v) = x2 + u2 + 2xu + 2yv + y 2 + v 2
2
2
2
≤ |z| + |w| + 2 |z| |w| = (|z| + |w|) ,
so this shows the first version of the triangle inequality. To get the second,
z = z − w + w, w = w − z + z
and so by the first form of the inequality
|z| ≤ |z − w| + |w| , |w| ≤ |z − w| + |z|
and so both |z| − |w| and |w| − |z| are no larger than |z − w| and this proves the second version
because ||z| − |w|| is one of |z| − |w| or |w| − |z|. With this definition, it is important to note the following. Be sure to verify this. It is not too
hard but you need to do it.
√
2
2
Remark 1.3.5 : Let z = a + ib and w = c + id. Then |z − w| = (a − c) + (b − d) . Thus the
distance between the point in the plane determined by the ordered pair (a, b) and the ordered pair
(c, d) equals |z − w| where z and w are as just described.
For example,√consider the distance between (2, 5) and (1, 8) . From the distance formula this
√
2
2
distance equals (2 − 1) + (5 − 8) = 10. On the other hand, letting z = 2 + i5 and w = 1 + i8,
√
z − w = 1 − i3 and so (z − w) (z − w) = (1 − i3) (1 + i3) = 10 so |z − w| = 10, the same thing
obtained with the distance formula.
6
CHAPTER 1. SOME PREREQUISITE TOPICS
1.4
Polar Form Of Complex Numbers
Complex numbers, are often written in the so called polar form which is described next. Suppose
z = x + iy is a complex number. Then
(
)
√
x
y
x + iy = x2 + y 2 √
+ i√
.
x2 + y 2
x2 + y 2
Now note that
(
√
)2
x
+
x2 + y 2
(
and so
(
x
y
)2
√
x2 + y 2
y
=1
)
√
,√
x2 + y 2
x2 + y 2
is a point on the unit circle. Therefore, there exists a unique angle θ ∈ [0, 2π) such that
x
y
cos θ = √
, sin θ = √
.
x2 + y 2
x2 + y 2
The polar
√form of the complex number is then r (cos θ + i sin θ) where θ is this angle just described
and r = x2 + y 2 ≡ |z|.
r=
√
x2 + y 2
r * x + iy = r(cos(θ) + i sin(θ))
θ
1.5
Roots Of Complex Numbers
A fundamental identity is the formula of De Moivre which follows.
Theorem 1.5.1 Let r > 0 be given. Then if n is a positive integer,
n
[r (cos t + i sin t)] = rn (cos nt + i sin nt) .
Proof: It is clear the formula holds if n = 1. Suppose it is true for n.
n+1
[r (cos t + i sin t)]
n
= [r (cos t + i sin t)] [r (cos t + i sin t)]
which by induction equals
= rn+1 (cos nt + i sin nt) (cos t + i sin t)
= rn+1 ((cos nt cos t − sin nt sin t) + i (sin nt cos t + cos nt sin t))
= rn+1 (cos (n + 1) t + i sin (n + 1) t)
by the formulas for the cosine and sine of the sum of two angles. Corollary 1.5.2 Let z be a non zero complex number. Then there are always exactly k k th roots of
z in C.
1.5. ROOTS OF COMPLEX NUMBERS
7
Proof: Let z = x + iy and let z = |z| (cos t + i sin t) be the polar form of the complex number.
By De Moivre’s theorem, a complex number
r (cos α + i sin α) ,
is a k th root of z if and only if
rk (cos kα + i sin kα) = |z| (cos t + i sin t) .
1/k
This requires rk = |z| and so r = |z|
only happen if
and also both cos (kα) = cos t and sin (kα) = sin t. This can
kα = t + 2lπ
for l an integer. Thus
α=
t + 2lπ
,l ∈ Z
k
and so the k th roots of z are of the form
(
(
)
(
))
t + 2lπ
t + 2lπ
1/k
|z|
+ i sin
, l ∈ Z.
cos
k
k
Since the cosine and sine are periodic of period 2π, there are exactly k distinct numbers which result
from this formula. Example 1.5.3 Find the three cube roots of i.
( ))
(
( )
First note that i = 1 cos π2 + i sin π2 . Using the formula in the proof of the above corollary,
the cube roots of i are
(
(
)
(
))
(π/2) + 2lπ
(π/2) + 2lπ
1 cos
+ i sin
3
3
where l = 0, 1, 2. Therefore, the roots are
( )
( )
( )
( )
(π )
(π )
5
5
3
3
cos
+ i sin
, cos
π + i sin
π , cos
π + i sin
π .
6
6
6
6
2
2
√
√
( )
( )
3
1
− 3
1
Thus the cube roots of i are
+i
,
+i
, and −i.
2
2
2
2
The ability to find k th roots can also be used to factor some polynomials.
Example 1.5.4 Factor the polynomial x3 − 27.
First find (
the cube roots
of 27. (
By the above)procedure using De Moivre’s theorem, these cube
√ )
√
−1
3
−1
3
, and 3
. Therefore, x3 − 27 =
roots are 3, 3
+i
−i
2
2
2
2
(
(
(x − 3) x − 3
(
√ )) (
√ ))
−1
3
−1
3
+i
x−3
−i
.
2
2
2
2
(
(
(
√ )) (
√ ))
3
3
−1
Note also x − 3 −1
x
−
3
= x2 + 3x + 9 and so
+
i
−
i
2
2
2
2
(
)
x3 − 27 = (x − 3) x2 + 3x + 9
where the quadratic polynomial x2 + 3x + 9 cannot be factored without using complex numbers.
8
CHAPTER 1. SOME PREREQUISITE TOPICS
3
Note
√that even though
√ the polynomial x − 27 has all real coefficients, it has some complex zeros,
−1
3
−1
3
+i
and
−i
. These zeros are complex conjugates of each other. It is always this way.
2
2
2
2
You should show this is the case. To see how to do this, see Problems 17 and 18 below.
Another fact for your information is the fundamental theorem of algebra. This theorem says
that any polynomial of degree at least 1 having any complex coefficients always has a root in C.
This is sometimes referred to by saying C is algebraically complete. Gauss is usually credited with
giving a proof of this theorem in 1797 but many others worked on it and the first completely correct
proof was due to Argand in 1806. For more on this theorem, you can google fundamental theorem of
algebra and look at the interesting Wikipedia article on it. Proofs of this theorem usually involve the
use of techniques from calculus even though it is really a result in algebra. A proof and plausibility
explanation is given later.
1.6
The Quadratic Formula
The quadratic formula
x=
−b ±
√
b2 − 4ac
2a
gives the solutions x to
ax2 + bx + c = 0
where a, b, c are real numbers. It holds even if b2 − 4ac < 0. This is easy to show from the above.
There are exactly two square roots to this number b2 −4ac from the above methods using De Moivre’s
theorem. These roots are of the form
(
(π )
( π ))
√
√
4ac − b2 cos
+ i sin
= i 4ac − b2
2
2
and
(
( )
( ))
√
√
3π
3π
4ac − b2 cos
+ i sin
= −i 4ac − b2
2
2
Thus the solutions, according to the quadratic formula are still given correctly by the above formula.
Do these solutions predicted by the quadratic formula continue to solve the quadratic equation?
Yes, they do. You only need to observe that when you square a square root of a complex number z,
you recover z. Thus
(
)2
(
)
√
√
−b + b2 − 4ac
−b + b2 − 4ac
+c
a
+b
2a
2a
)
(
√
(
)
1 2 1
1 √ 2
−b + b2 − 4ac
=a
b − c − 2 b b − 4ac + b
+c
2a2
a
2a
2a
(
))
)
1 ( √ 2
1 ( √ 2
= −
b b − 4ac + 2ac − b2
+
b b − 4ac − b2 + c = 0
2a
2a
√
b −4ac
Similar reasoning shows directly that −b− 2a
also solves the quadratic equation.
What if the coefficients of the quadratic equation are actually complex numbers? Does the
formula hold even in this case? The answer is yes. This is a hint on how to do Problem 27 below, a
special case of the fundamental theorem of algebra, and an ingredient in the proof of some versions
of this theorem.
2
Example 1.6.1 Find the solutions to x2 − 2ix − 5 = 0.
Formally, from the quadratic formula, these solutions are
√
2i ± −4 + 20
2i ± 4
x=
=
= i ± 2.
2
2
Now you can check that these really do solve the equation. In general, this will be the case. See
Problem 27 below.
1.7. THE COMPLEX EXPONENTIAL
1.7
9
The Complex Exponential
It was shown above that every complex number can be written in the form r (cos θ + i sin θ) where
r ≥ 0. Laying aside the zero complex number, this shows that every non zero complex number is of
the form eα (cos β + i sin β) . We write this in the form eα+iβ . Having done so, does it follow that
the expression preserves the most important property of the function t → e(α+iβ)t for t real, that
(
)′
e(α+iβ)t = (α + iβ) e(α+iβ)t ?
By the definition just given which does not contradict the usual definition in case β = 0 and the
usual rules of differentiation in calculus,
(
)′
(
)′
e(α+iβ)t
= eαt (cos (βt) + i sin (βt))
= eαt [α (cos (βt) + i sin (βt)) + (−β sin (βt) + iβ cos (βt))]
Now consider the other side. From the definition it equals
(
)
(α + iβ) eαt (cos (βt) + i sin (βt)) = eαt [(α + iβ) (cos (βt) + i sin (βt))]
= eαt [α (cos (βt) + i sin (βt)) + (−β sin (βt) + iβ cos (βt))]
which is the same thing. This is of fundamental importance in differential equations. It shows that
there is no change in going from real to complex numbers for ω in the consideration of the problem
y ′ = ωy, y (0) = 1. The solution is always eωt . The formula just discussed, that
eα (cos β + i sin β) = eα+iβ
is Euler’s formula.
1.8
The Fundamental Theorem Of Algebra
The fundamental theorem of algebra states that every non constant polynomial having coefficients
in C has a zero in C. If C is replaced by R, this is not true because of the example, x2 + 1 = 0.
This theorem is a very remarkable result and notwithstanding its title, all the most straightforward
proofs depend on either analysis or topology. It was first mostly proved by Gauss in 1797. The first
complete proof was given by Argand in 1806. The proof given here follows Rudin [15]. See also
Hardy [9] for a similar proof, more discussion and references. The shortest proof is found in the
theory of complex analysis. First I will give an informal explanation of this theorem which shows
why it is is reasonable to believe in the fundamental theorem of algebra.
Theorem 1.8.1 Let p (z) = an z n + an−1 z n−1 + · · · + a1 z + a0 where each ak is a complex number
and an ̸= 0, n ≥ 1. Then there exists w ∈ C such that p (w) = 0.
To begin with, here is the informal explanation. Dividing by the leading coefficient an , there is
no loss of generality in assuming that the polynomial is of the form
p (z) = z n + an−1 z n−1 + · · · + a1 z + a0
If a0 = 0, there is nothing to prove because p (0) = 0. Therefore, assume a0 ̸= 0. From the polar
form of a complex number z, it can be written as |z| (cos θ + i sin θ). Thus, by DeMoivre’s theorem,
n
z n = |z| (cos (nθ) + i sin (nθ))
n
It follows that z n is some point on the circle of radius |z|
Denote by Cr the circle of radius r in the complex plane which is centered at 0. Then if r is
sufficiently large and |z| = r, the term z n is far larger than the rest of the polynomial. It is on
10
CHAPTER 1. SOME PREREQUISITE TOPICS
n
k
the circle of radius |z| while the other terms are on circles of fixed multiples of |z| for k ≤ n − 1.
Thus, for r large enough, Ar = {p (z) : z ∈ Cr } describes a closed curve which misses the inside of
some circle having 0 as its center. It won’t be as simple as suggested in the following picture, but it
will be a closed curve thanks to De Moivre’s theorem and the observation that the cosine and sine
are periodic. Now shrink r. Eventually, for r small enough, the non constant terms are negligible
and so Ar is a curve which is contained in some circle centered at a0 which has 0 on the outside.
Thus it is reasonable to believe that for some r during this
Ar
shrinking process, the set Ar must hit 0. It follows that
a0
Ar r large
p (z) = 0 for some z.
0
For example, consider the polynomial x3 + x + 1 + i.
r small
It has no real zeros. However, you could let z =
r (cos t + i sin t) and insert this into the polynomial. Thus you would want to find a point where
3
(r (cos t + i sin t)) + r (cos t + i sin t) + 1 + i = 0 + 0i
Expanding this expression on the left to write it in terms of real and imaginary parts, you get on
the left
(
)
r3 cos3 t − 3r3 cos t sin2 t + r cos t + 1 + i 3r3 cos2 t sin t − r3 sin3 t + r sin t + 1
Thus you need to have both the real and imaginary parts equal to 0. In other words, you need to
have
( 3
)
r cos3 t − 3r3 cos t sin2 t + r cos t + 1, 3r3 cos2 t sin t − r3 sin3 t + r sin t + 1 = (0, 0)
for some value of r and t. First here is a graph of this parametric function of t for t ∈ [0, 2π] on the
left, when r = 2. Note how the graph misses the origin 0 + i0. In fact, the closed curve contains a
small circle which has the point 0 + i0 on its inside.
y
y
y
x
x
x
Next is the graph when r = .5. Note how the closed curve is included in a circle which has 0 + i0
on its outside. As you shrink r you get closed curves. At first, these closed curves enclose 0 + i0 and
later, they exclude 0 + i0. Thus one of them should pass through this point. In fact, consider the
curve which results when r = 1. 386 2 which is the graph on the right. Note how for this value of r
the curve passes through the point 0 + i0. Thus for some t,
1.3862 (cos t + i sin t)
is a solution of the equation p (z) = 0.
Now here is a rigorous proof for those who have studied analysis.
Proof. Suppose the nonconstant polynomial p (z) = a0 + a1 z + · · · + an z n , an ̸= 0, has no zero
in C. Since lim|z|→∞ |p (z)| = ∞, there is a z0 with
|p (z0 )| = min |p (z)| > 0
z∈C
0)
Then let q (z) = p(z+z
p(z0 ) . This is also a polynomial which has no zeros and the minimum of |q (z)|
is 1 and occurs at z = 0. Since q (0) = 1, it follows q (z) = 1 + ak z k + r (z) where r (z) consists of
higher order terms. Here ak is the first coefficient which is nonzero. Choose a sequence, zn → 0,
such that ak znk < 0. For example, let −ak znk = (1/n). Then |q (zn )| ≤ 1 − ak znk + |r (zn )| < 1 for
all n large enough because the higher order terms in r (zn ) converge to 0 faster than znk . This is a
contradiction. 1.9. EXERCISES
1.9
11
Exercises
∑n
1 4 1 3 1 2
n + n + n .
4
2
4
√
∑n
1
2. Prove by induction that whenever n ≥ 2, k=1 √ > n.
k
∑n
3. Prove by induction that 1 + i=1 i (i!) = (n + 1)!.
∑n ( )
n
4. The binomial theorem states (x + y) = k=0 nk xn−k y k where
(
) ( ) (
)
( )
( )
n+1
n
n
n
n
=
+
if k ∈ [1, n] ,
≡1≡
k
k
k−1
0
n
1. Prove by induction that
k=1
k3 =
Prove the binomial theorem by induction. Next show that
( )
n
n!
=
, 0! ≡ 1
k
(n − k)!k!
I
5. Let z = 5 + i9. Find z −1 .
6. Let z = 2 + i7 and let w = 3 − i8. Find zw, z + w, z 2 , and w/z.
7. Give the complete solution to x4 + 16 = 0.
8. Graph the complex cube roots of 8 in the complex plane. Do the same for the four fourth
roots of 16. I
9. If z is a complex number, show there exists ω a complex number with |ω| = 1 and ωz = |z| .
n
10. De Moivre’s theorem says [r (cos t + i sin t)] = rn (cos nt + i sin nt) for n a positive integer.
Does this formula continue to hold for all integers n, even negative integers? Explain. I
11. You already know formulas for cos (x + y) and sin (x + y) and these were used to prove De
Moivre’s theorem. Now using De Moivre’s theorem, derive a formula for sin (5x) and one for
cos (5x). I
12. If z and w are two complex numbers and the polar form of z involves the angle θ while the
polar form of w involves the angle ϕ, show that in the polar form for zw the angle involved is
θ + ϕ. Also, show that in the polar form of a complex number z, r = |z| .
13. Factor x3 + 8 as a product of linear factors.
(
)
14. Write x3 + 27 in the form (x + 3) x2 + ax + b where x2 + ax + b cannot be factored any more
using only real numbers.
15. Completely factor x4 + 16 as a product of linear factors.
16. Factor x4 + 16 as the product of two quadratic polynomials each of which cannot be factored
further without using complex numbers.
∏n
17. If z, w are complex numbers prove zw = zw and then show by induction that j=1 zj =
∏n
∑m
∑m
j=1 zj . Also verify that
k=1 zk =
k=1 zk . In words this says the conjugate of a product equals the product of the conjugates and the conjugate of a sum equals the sum of the
conjugates.
18. Suppose p (x) = an xn + an−1 xn−1 + · · · + a1 x + a0 where all the ak are real numbers. Suppose
also that p (z) = 0 for some z ∈ C. Show it follows that p (z) = 0 also.
12
CHAPTER 1. SOME PREREQUISITE TOPICS
19. Show that 1 + i, 2 + i are the only two zeros to
p (x) = x2 − (3 + 2i) x + (1 + 3i)
so the zeros do not necessarily come in conjugate pairs if the coefficients are not real.
20. I claim that 1 = −1. Here is why.
−1 = i =
2
√
√
√
√
2
−1 −1 = (−1) = 1 = 1.
This is clearly a remarkable result but is there something wrong with it? If so, what is wrong?
21. De Moivre’s theorem is really a grand thing. I plan to use it now for rational exponents, not
just integers.
1/4
1 = 1(1/4) = (cos 2π + i sin 2π)
= cos (π/2) + i sin (π/2) = i.
Therefore, squaring both sides it follows 1 = −1 as in the previous problem. What does this
tell you about De Moivre’s theorem? Is there a profound difference between raising numbers
to integer powers and raising numbers to non integer powers?
22. Review Problem 10 at this point. Now here is another question: If n is an integer, is it always
n
true that (cos θ − i sin θ) = cos (nθ) − i sin (nθ)? Explain.
23. Suppose
any polynomial in cos θ and sin θ. By this I mean an expression of the
∑myou∑have
n
form α=0 β=0 aαβ cosα θ sinβ θ where aαβ ∈ C. Can this always be written in the form
∑n+m
∑m+n
τ =−(n+m) cτ sin τ θ? Explain.
γ=−(n+m) bγ cos γθ +
24. Suppose p (x) = an xn + an−1 xn−1 + · · · + a1 x + a0 is a polynomial and it has n zeros,
z1 , z2 , · · · , zn
m
listed according to multiplicity. (z is a root of multiplicity m if the polynomial f (x) = (x − z)
divides p (x) but (x − z) f (x) does not.) Show that
p (x) = an (x − z1 ) (x − z2 ) · · · (x − zn ) .
25. Give the solutions to the following quadratic equations having real coefficients.
(a)
(b)
(c)
(d)
(e)
x2 − 2x + 2 = 0
3x2 + x + 3 = 0
x2 − 6x + 13 = 0
x2 + 4x + 9 = 0
4x2 + 4x + 5 = 0
26. Give the solutions to the following quadratic equations having complex coefficients. Note how
the solutions do not come in conjugate pairs as they do when the equation has real coefficients.
(a)
(b)
(c)
(d)
(e)
x2 + 2x + 1 + i = 0
4x2 + 4ix − 5 = 0
4x2 + (4 + 4i) x + 1 + 2i = 0
x2 − 4ix − 5 = 0
3x2 + (1 − i) x + 3i = 0
27. Prove the fundamental theorem of algebra for quadratic polynomials having coefficients in C.
That is, show that an equation of the form ax2 + bx + c = 0 where a, b, c are complex numbers,
a ̸= 0 has a complex solution. Hint: Consider the fact, noted earlier that the expressions
given from the quadratic formula do in fact serve as solutions.
Chapter 2
Fn
The notation, Cn refers to the collection of ordered lists of n complex numbers. Since every real
number is also a complex number, this simply generalizes the usual notion of Rn , the collection of
all ordered lists of n real numbers. In order to avoid worrying about whether it is real or complex
numbers which are being referred to, the symbol F will be used. If it is not clear, always pick C.
Definition 2.0.1 Define Fn ≡ {(x1 , · · · , xn ) : xj ∈ F for j = 1, · · · , n} .
(x1 , · · · , xn ) = (y1 , · · · , yn )
if and only if for all j = 1, · · · , n, xj = yj . When (x1 , · · · , xn ) ∈ Fn , it is conventional to denote
(x1 , · · · , xn ) by the single bold face letter, x. The numbers, xj are called the coordinates. Elements
in Fn are called vectors. The set
{(0, · · · , 0, t, 0, · · · , 0) : t ∈ R}
for t in the ith slot is called the ith coordinate axis in the case of Rn . The point 0 ≡ (0, · · · , 0) is
called the origin.
Thus (1, 2, 4i) ∈ F3 and (2, 1, 4i) ∈ F3 but (1, 2, 4i) ̸= (2, 1, 4i) because, even though the same
numbers are involved, they don’t match up. In particular, the first entries are not equal.
The geometric significance of Rn for n ≤ 3 has been encountered already in calculus or in precalculus. Here is a short review. First consider the case when n = 1. Then from the definition,
R1 = R. Recall that R is identified with the points of a line. Look at the number line again.
Observe that this amounts to identifying a point on this line with a real number. In other words a
real number determines where you are on this line. Now suppose n = 2 and consider two lines which
intersect each other at right angles as shown in the following picture.
(2, 6)
6
(−8, 3)
3
2
−8
Notice how you can identify a point shown in the plane with the ordered pair, (2, 6) . You go to
the right a distance of 2 and then up a distance of 6. Similarly, you can identify another point in the
plane with the ordered pair (−8, 3) . Starting at 0, go to the left a distance of 8 on the horizontal
line and then up a distance of 3. The reason you go to the left is that there is a − sign on the
13
CHAPTER 2. FN
14
eight. From this reasoning, every ordered pair determines a unique point in the plane. Conversely,
taking a point in the plane, you could draw two lines through the point, one vertical and the other
horizontal and determine unique points, x1 on the horizontal line in the above picture and x2 on
the vertical line in the above picture, such that the point of interest is identified with the ordered
pair, (x1 , x2 ) . In short, points in the plane can be identified with ordered pairs similar to the way
that points on the real line are identified with real numbers. Now suppose n = 3. As just explained,
the first two coordinates determine a point in a plane. Letting the third component determine how
far up or down you go, depending on whether this number is positive or negative, this determines
a point in space. Thus, (1, 4, −5) would mean to determine the point in the plane that goes with
(1, 4) and then to go below this plane a distance of 5 to obtain a unique point in space. You see
that the ordered triples correspond to points in space just as the ordered pairs correspond to points
in a plane and single real numbers correspond to points on a line.
You can’t stop here and say that you are only interested in n ≤ 3. What if you were interested
in the motion of two objects? You would need three coordinates to describe where the first object
is and you would need another three coordinates to describe where the other object is located.
Therefore, you would need to be considering R6 . If the two objects moved around, you would need
a time coordinate as well. As another example, consider a hot object which is cooling and suppose
you want the temperature of this object. How many coordinates would be needed? You would need
one for the temperature, three for the position of the point in the object and one more for the time.
Thus you would need to be considering R5 . Many other examples can be given. Sometimes n is very
large. This is often the case in applications to business when they are trying to maximize profit
subject to constraints. It also occurs in numerical analysis when people try to solve hard problems
on a computer.
There are other ways to identify points in space with three numbers but the one presented is
the most basic. In this case, the coordinates are known as Cartesian coordinates after Descartes1
who invented this idea in the first half of the seventeenth century. I will often not bother to draw a
distinction between the point in space and its Cartesian coordinates.
The geometric significance of Cn for n > 1 is not available because each copy of C corresponds
to the plane or R2 .
2.1
Algebra in Fn
There are two algebraic operations done with elements of Fn . One is addition and the other is
multiplication by numbers, called scalars. In the case of Cn the scalars are complex numbers while
in the case of Rn the only allowed scalars are real numbers. Thus, the scalars always come from F
in either case.
Definition 2.1.1 If x ∈ Fn and a ∈ F, also called a scalar, then ax ∈ Fn is defined by
ax = a (x1 , · · · , xn ) ≡ (ax1 , · · · , axn ) .
(2.1)
This is known as scalar multiplication. If x, y ∈ Fn then x + y ∈ Fn and is defined by
x + y = (x1 , · · · , xn ) + (y1 , · · · , yn )
≡ (x1 + y1 , · · · , xn + yn )
(2.2)
With this definition, vector addition and scalar multiplication satisfy the conclusions of the
following theorem. More generally, these properties are called the vector space axioms.
Theorem 2.1.2 For v, w ∈ Fn and α, β scalars, (real numbers), the following hold.
v + w = w + v,
1 René
(2.3)
Descartes 1596-1650 is often credited with inventing analytic geometry although it seems the ideas were
actually known much earlier. He was interested in many different subjects, physiology, chemistry, and physics being
some of them. He also wrote a large book in which he tried to explain the book of Genesis scientifically. Descartes
ended up dying in Sweden.
2.2. GEOMETRIC MEANING OF VECTORS
15
the commutative law of addition,
(v + w) + z = v+ (w + z) ,
(2.4)
v + 0 = v,
(2.5)
v+ (−v) = 0,
(2.6)
α (v + w) = αv+αw,
(2.7)
(α + β) v =αv+βv,
(2.8)
α (βv) = αβ (v) ,
(2.9)
1v = v.
(2.10)
the associative law for addition,
the existence of an additive identity,
the existence of an additive inverse, Also
In the above 0 = (0, · · · , 0).
You should verify these properties all hold. For example, consider 2.7
α (v + w) = α (v1 + w1 , · · · , vn + wn )
= (α (v1 + w1 ) , · · · , α (vn + wn )) = (αv1 + αw1 , · · · , αvn + αwn )
= (αv1 , · · · , αvn ) + (αw1 , · · · , αwn ) = αv + αw.
As usual subtraction is defined as x − y ≡ x+ (−y) .
2.2
Geometric Meaning Of Vectors
The geometric meaning is especially significant in the case of Rn for n = 2, 3. Here is a short
discussion of this topic.
Definition 2.2.1 Let x = (x1 , · · · , xn ) be the coordinates of a point in Rn . Imagine an arrow (line
segment with a point) with its tail at 0 = (0, · · · , 0) and its point at x as shown in the following
picture in the case of R3 .
(x1 , x2 , x3 ) = x
3
Then this arrow is called the position vector of the point x. Given two points, P, Q whose
coordinates are (p1 , · · · , pn ) and (q1 , · · · , qn ) respectively, one can also determine the position vector
from P to Q defined as follows.
−−→
P Q ≡ (q1 − p1 , · · · , qn − pn )
Thus every point in Rn determines a vector and conversely, every such position vector (arrow)
which has its tail at 0 determines a point of Rn , namely the point of Rn which coincides with the
point of the positioin vector. Also two different points determine a position vector going from one
to the other as just explained.
Imagine taking the above position vector and moving it around, always keeping it pointing
in the same direction as shown in the following picture. After moving it around, it is regarded
CHAPTER 2. FN
16
as the same vector because it points in the same direction and has the same length.2 Thus each of the arrows
(x
,
x
,
x
)
=
x
in
the above picture is regarded as the same vector. The
1
2
3
3
components
of this vector are the numbers, x1 , · · · , xn
3
3
3
obtained by placing the initial point of an arrow representing the vector at the origin. You should think of
these numbers as directions for obtaining such a vector
illustrated above. Starting at some point (a1 , a2 , · · · , an )
in Rn , you move to the point (a1 + x1 , · · · , an ) and from
there to the point (a1 + x1 , a2 + x2 , a3 · · · , an ) and then
to (a1 + x1 , a2 + x2 , a3 + x3 , · · · , an ) and continue this way until you obtain the point
(a1 + x1 , a2 + x2 , · · · , an + xn ) .
The arrow having its tail at (a1 , a2 , · · · , an ) and its point at
(a1 + x1 , a2 + x2 , · · · , an + xn )
looks just like (same length and direction) the arrow which has its tail at 0 and its point at
(x1 , · · · , xn ) so it is regarded as representing the same vector.
2.3
Geometric Meaning Of Vector Addition
It was explained earlier that an element of Rn is an ordered list of numbers and it was also shown
that this can be used to determine a point in three dimensional space in the case where n = 3 and in
two dimensional space, in the case where n = 2. This point was specified relative to some coordinate
axes.
Consider the case where n = 3 for now. If you draw an arrow from the point in three dimensional
space determined by (0, 0, 0) to the point (a, b, c) with its tail sitting at the point (0, 0, 0) and its point
at the point (a, b, c) , it is obtained by starting at (0, 0, 0), moving parallel to the x1 axis to (a, 0, 0)
and then from here, moveing parallel to the x2 axis to (a, b, 0) and finally parallel to the x3 axis to
(a, b, c) . It is evident that the same vector would result if you began at the point v ≡ (d, e, f ) , moved
parallel to the x1 axis to (d + a, e, f ) , then parallel to the x2 axis to (d + a, e + b, f ) , and finally
parallel to the x3 axis to (d + a, e + b, f + c) only this time, the arrow representing the vector would
have its tail sitting at the point determined by v ≡ (d, e, f ) and its point at (d + a, e + b, f + c) . It
is the same vector because it will point in the same direction and have the same length. It is like
you took an actual arrow, the sort of thing you shoot with a bow, and moved it from one location
to another keeping it pointing the same direction. This is illustrated in the following picture in
which v + u is illustrated. Note the parallelogram determined in the picture by the vectors u and v.
Thus the geometric significance of (d, e, f ) + (a, b, c) =
u
(d + a, e + b, f + c) is this. You start with the position
vector of the point (d, e, f ) and at its point, you place the
vector determined by (a, b, c) with its tail at (d, e, f ) . Then
the point of this last vector will be (d + a, e + b, f + c) .
I
This is the geometric significance of vector addition. Also,
v
u+v
x3
as shown in the picture, u + v is the directed diagonal
of the parallelogram determined by the two vectors u and
v. A similar interpretation holds in Rn , n > 3 but I can’t
u
draw a picture in this case.
x2
Since the convention is that identical arrows pointing
in the same direction represent the same vector, the geox1
metric significance of vector addition is as follows in any
number of dimensions.
2 I will discuss how to define length later. For now, it is only necessary to observe that the length should be defined
in such a way that it does not change when such motion takes place.
2.4. DISTANCE BETWEEN POINTS IN RN LENGTH OF A VECTOR
17
Procedure 2.3.1 Let u and v be two vectors. Slide v so that the tail of v is on the point of u.
Then draw the arrow which goes from the tail of u to the point of the slid vector v. This arrow
represents the vector u + v.
*
u+v
u
v
-
−−→
Note that P +P Q = Q.
2.4
Distance Between Points In Rn Length Of A Vector
How is distance between two points in Rn defined?
Definition 2.4.1 Let x = (x1 , · · · , xn ) and y = (y1 , · · · , yn ) be two points in Rn . Then |x − y| to
indicates the distance between these points and is defined as
(
distance between x and y ≡ |x − y| ≡
n
∑
)1/2
|xk − yk |
2
.
k=1
This is called the distance formula. Thus |x| ≡ |x − 0| . The symbol, B (a, r) is defined by
B (a, r) ≡ {x ∈ Rn : |x − a| < r} .
This is called an open ball of radius r centered at a. It means all points in Rn which are closer to
a than r. The length of a vector x is the distance between x and 0.
First of all, note this is a generalization of the notion of distance in R. There the distance
between two points, x and y was given by the absolute value of their difference. Thus |x − y| is
(
)1/2
2
equal to the distance between these two points on R. Now |x − y| = (x − y)
where the square
root is always the positive square root. Thus it is the same formula as the above definition except
there is only one term in the sum. Geometrically, this is the right way to define distance which is
seen from the Pythagorean theorem. This is known as the Euclidean norm. Often people use two
lines to denote this distance ||x − y||. However, I want to emphasize that this is really just like the
absolute value, so when the norm is defined in this way, I will usually write |·|.
Consider the following picture in the case that n = 2.
(y1 , y2 )
(x1 , x2 )
(y1 , x2 )
There are two points in the plane whose Cartesian coordinates are (x1 , x2 ) and (y1 , y2 ) respectively. Then the solid line joining these two points is the hypotenuse of a right triangle which is
half of the rectangle shown in dotted lines. What is its length? Note the lengths of the sides of this
triangle are |y1 − x1 | and |y2 − x2 | . Therefore, the Pythagorean theorem implies the length of the
hypotenuse equals
(
2
2
|y1 − x1 | + |y2 − x2 |
)1/2
(
)1/2
2
2
= (y1 − x1 ) + (y2 − x2 )
CHAPTER 2. FN
18
which is just the formula for the distance given above. In other words, this distance defined above
is the same as the distance of plane geometry in which the Pythagorean theorem holds.
Now suppose n = 3 and let (x1 , x2 , x3 ) and (y1 , y2 , y3 ) be two points in R3 . Consider the following
picture in which one of the solid lines joins the two points and a dotted line joins the points (x1 , x2 , x3 )
and (y1 , y2 , x3 ) .
(y1 , y2 , y3 )
(y1 , y2 , x3 )
(x1 , x2 , x3 )
(y1 , x2 , x3 )
By the Pythagorean theorem, the length of the dotted line joining (x1 , x2 , x3 ) and (y1 , y2 , x3 )
equals
(
)1/2
2
2
(y1 − x1 ) + (y2 − x2 )
while the length of the line joining (y1 , y2 , x3 ) to (y1 , y2 , y3 ) is just |y3 − x3 | . Therefore, by the
Pythagorean theorem again, the length of the line joining the points (x1 , x2 , x3 ) and (y1 , y2 , y3 )
equals
{[
(
2
2
(y1 − x1 ) + (y2 − x2 )
)1/2 ]2
}1/2
+ (y3 − x3 )
2
(
)1/2
2
2
2
= (y1 − x1 ) + (y2 − x2 ) + (y3 − x3 )
,
which is again just the distance formula above.
This completes the argument that the above definition is reasonable. Of course you cannot
continue drawing pictures in ever higher dimensions but there is no problem with the formula for
distance in any number of dimensions. Here is an example.
Example 2.4.2 Find the distance between the points in R4 ,
a = (1, 2, −4, 6)
and
b = (2, 3, −1, 0)
Use the distance formula and write
2
2
2
2
2
|a − b| = (1 − 2) + (2 − 3) + (−4 − (−1)) + (6 − 0) = 47
√
Therefore, |a − b| = 47.
All this amounts to defining the distance between two points as the length of a straight line
joining these two points. However, there is nothing sacred about using straight lines. One could
define the distance to be the length of some other sort of line joining these points. It won’t be done
very much in this book but sometimes this sort of thing is done.
Another convention which is usually followed, especially in R2 and R3 is to denote the first
component of a point in R2 by x and the second component by y. In R3 it is customary to denote
the first and second components as just described while the third component is called z.
Example 2.4.3 Describe the points which are at the same distance between (1, 2, 3) and (0, 1, 2) .
2.4. DISTANCE BETWEEN POINTS IN RN LENGTH OF A VECTOR
19
Let (x, y, z) be such a point. Then
√
√
2
2
2
2
2
(x − 1) + (y − 2) + (z − 3) = x2 + (y − 1) + (z − 2) .
Squaring both sides
2
2
2
2
(x − 1) + (y − 2) + (z − 3) = x2 + (y − 1) + (z − 2)
2
and so
x2 − 2x + 14 + y 2 − 4y + z 2 − 6z = x2 + y 2 − 2y + 5 + z 2 − 4z
which implies
−2x + 14 − 4y − 6z = −2y + 5 − 4z
and so
2x + 2y + 2z = −9.
(2.11)
Since these steps are reversible, the set of points which is at the same distance from the two given
points consists of the points (x, y, z) such that 2.11 holds.
There are certain properties of the distance which are obvious. Two of them which follow directly
from the definition are
|x − y| = |y − x| ,
|x − y| ≥ 0 and equals 0 only if y = x.
The third fundamental property of distance is known as the triangle inequality. Recall that in any
triangle the sum of the lengths of two sides is always at least as large as the third side. I will show
you a proof of this later. This is usually stated as
|x + y| ≤ |x| + |y| .
Here is a picture which illustrates the statement of this inequality in terms of geometry. Later,
this is proved, but for now, the geometric motivation will suffice. When you have a vector u,
its additive inverse −u will be the vector which has the same magnitude as u
3
x+y
but the opposite direction. When one writes u − v, the meaning is u+ (−v)
y
as with real numbers. The following example is art which illustrates these
definitions and conventions.
x
Example 2.4.4 Here is a picture of two vectors, u and v.
u
v j
Sketch a picture of u + v, u − v.
First here is a picture of u + v. You first draw u and then at the point of u you place the tail of
v as shown. Then u + v is the vector which results which is drawn in the following pretty picture.
u
v j
:
u+v
CHAPTER 2. FN
20
Next consider u − v. This means u+ (−v) . From the above geometric description of vector
addition, −v is the vector which has the same length but which points in the opposite direction to
v. Here is a picture.
Y −v
6
u + (−v)
u
2.5
Geometric Meaning Of Scalar Multiplication
As discussed earlier, x = (x1 , x2 , x3 ) determines a vector. You draw the line from 0 to x placing the
point of the vector on x. What is the length of this vector? The
√ length of this vector is defined to
equal |x| as in Definition 2.4.1. Thus the length of x equals x21 + x22 + x23 . When you multiply x
by a scalar α, you get (αx1 , αx2 , αx3 ) and the length of this vector is defined as
√(
√
)
2
2
(αx1 ) + (αx2 ) + (αx3 )
2
= |α|
x21 + x22 + x23 .
Thus the following holds.
|αx| = |α| |x| .
In other words, multiplication by a scalar magnifies or shrinks the length of the vector. What about
the direction? You should convince yourself by drawing a picture that if α is negative, it causes the
resulting vector to point in the opposite direction while if α > 0 it preserves the direction the vector
points.
Exercise 2.5.1 Here is a picture of two vectors, u and v.
u
v j
Sketch a picture of u+2v, u− 12 v.
The two vectors are shown below.
− 12 v
Y
u
2v
u + 2v
2.6
u − 12 v
u
j
Parametric Lines
To begin with, suppose you have a typical equation for a line in the plane. For example,
y = 2x + 1
2.7. EXERCISES
21
A typical point on this line is of the form (x, 2x + 1) where x ∈ R. You could just as well write it as
(t, 2t + 1) , t ∈ R. That is, as t changes, the ordered pair traces out the points of the line. In terms
of ordered pairs, this line can be written as
(x, y) = (0, 1) + t (1, 2) , t ∈ R.
It is the same in Rn . A parametric line is of the form x = a + tv, t ∈ R. You can see this deserves
to be called a line because if you find the vector determined by two points a + t1 v and a + t2 v, this
vector is
a + t2 v− (a + t1 v) = (t2 − t1 ) v
which is parallel to the vector v. Thus the vector between any two points on this line is always
parallel to v which is called the direction vector.
There are two things you need for a line. A point and a direction vector. Here is an example.
Example 2.6.1 Find a parametric equation for the line between the points (1, 2, 3) and (2, −3, 1) .
A direction vector is (1, −5, −2) because this is the vector from the first to the second of these.
Then an equation of the line is
(x, y, z) = (1, 2, 3) + t (1, −5, −2) , t ∈ R
The example shows how to do this in general. If you have two points in Rn , a, b, then a parametric
equation for the line containing these points is of the form
x = a + t (b − a) .
Note that when t = 0 you get the point a and when t = 1, you get the point b.
Example 2.6.2 Find a parametric equation for the line which contains the point (1, 2, 0) and has
direction vector (1, 2, 1) .
From the above this is just
(x, y, z) = (1, 2, 0) + t (1, 2, 1) , t ∈ R.
2.7
(2.12)
Exercises
1. Verify all the properties 2.3-2.10.
2. Compute 5 (1, 2 + 3i, 3, −2) + 6 (2 − i, 1, −2, 7) .
3. Draw a picture of the points in R2 which are determined by the following ordered pairs.
(a) (1, 2)
(b) (−2, −2)
(c) (−2, 3)
(d) (2, −5)
4. Does it make sense to write (1, 2) + (2, 3, 1)? Explain.
5. Draw a picture of the points in R3 which are determined by the following ordered triples.
(a) (1, 2, 0)
(b) (−2, −2, 1)
(c) (−2, 3, −2)
CHAPTER 2. FN
22
2.8
Vectors And Physics
Suppose you push on something. What is important? There are really two things which are important, how hard you push and the direction you push. This illustrates the concept of force.
Definition 2.8.1 Force is a vector. The magnitude of this vector is a measure of how hard it is
pushing. It is measured in units such as Newtons or pounds or tons. Its direction is the direction in
which the push is taking place.
Vectors are used to model force and other physical vectors like velocity. What was just described
would be called a force vector. It has two essential ingredients, its magnitude and its direction.
Note there are n special vectors which point along the coordinate axes. These are
ei ≡ (0, · · · , 0, 1, 0, · · · , 0)
th
where the 1 is in the i
of R3 .
slot and there are zeros in all the other spaces. See the picture in the case
z
e3 6
e1
e2
y
x
The direction of ei is referred to as the ith direction. Given a vector v = (a1 , · · · , an ) , it follows
that
n
∑
v = a1 e1 + · · · + an en =
ai e i .
k=1
What does addition of vectors mean physically? Suppose two forces are applied to some object.
Each of these would be represented by a force vector and the two forces acting together would yield an
overall force acting on the
which would
be a force vector known as the resultant. Suppose
∑also
∑object
n
n
the two vectors are a = k=1 ai ei and b = k=1 bi ei . Then the vector a involves a component in
the ith direction, ai ei while the component in the ith direction of b is bi ei . Then it seems physically
reasonable that the resultant vector should have a component in the ith direction equal to (ai + bi ) ei .
This is exactly what is obtained when the vectors, a and b are added.
a + b = (a1 + b1 , · · · , an + bn ) =
n
∑
(ai + bi ) ei .
i=1
Thus the addition of vectors according to the rules of addition in Rn which were presented earlier,
yields the appropriate vector which duplicates the cumulative effect of all the vectors in the sum.
An item of notation should be mentioned here. In the case of Rn where n ≤ 3, it is standard
notation to use i for e1 , j for e2 , and k for e3 . Now here are some applications of vector addition to
some problems.
Example 2.8.2 There are three ropes attached to a car and three people pull on these ropes. The
first exerts a force of 2i+3j−2k Newtons, the second exerts a force of 3i+5j + k Newtons and the
third exerts a force of 5i − j+2k. Newtons. Find the total force in the direction of i.
To find the total force add the vectors as described above. This gives 10i+7j + k Newtons.
Therefore, the force in the i direction is 10 Newtons.
As mentioned earlier, the Newton is a unit of force like pounds.
Example 2.8.3 An airplane flies North East at 100 miles per hour. Write this as a vector.
A picture of this situation follows. The vector has length 100. Now using that vector as
2.9. EXERCISES
23
the hypotenuse√of a right triangle having equal sides, the
√ sides should
√ be each
of length 100/ 2. Therefore, the vector would be 100/ 2i + 100/ 2j.
This example also motivates the concept of velocity.
Definition 2.8.4 The speed of an object is a measure of how fast it is going.
It is measured in units of length per unit time. For example, miles per hour,
kilometers per minute, feet per second. The velocity is a vector having the
speed as the magnitude but also specifying the direction.
√
√
Thus the velocity vector in the above example is 100/ 2i + 100/ 2j.
Example 2.8.5 The velocity of an airplane is 100i + j + k measured in kilometers per hour and
at a certain instant of time its position is (1, 2, 1) . Here imagine a Cartesian coordinate system in
which the third component is altitude and the first and second components are measured on a line
from West to East and a line from South to North. Find the position of this airplane one minute
later.
Consider the vector (1, 2, 1) , is the initial position vector of the airplane. As it moves, the
position vector changes. After one minute the airplane has moved in the i direction a distance of
1
1
100 × 60
= 53 kilometer. In the j direction it has moved 60
kilometer during this same time, while
1
it moves 60 kilometer in the k direction. Therefore, the new displacement vector for the airplane is
(
) (
)
5 1 1
8 121 121
(1, 2, 1) +
, ,
=
,
,
3 60 60
3 60 60
Example 2.8.6 A certain river is one half mile wide with a current flowing at 4 miles per hour
from East to West. A man swims directly toward the opposite shore from the South bank of the
river at a speed of 3 miles per hour. How far down the river does he find himself when he has swam
across? How far does he end up swimming?
Consider the following picture. You should write these vectors in terms of components.
The velocity of the swimmer in still water would be 3j while the
velocity of the river would be −4i. Therefore, the velocity of the
swimmer is −4i+3j. Since the component of velocity in the direction
6
across the river is 3, it follows the√trip takes 1/6 hour or 10 minutes.
3
The speed at which he travels is 42 + 32 = 5 miles per hour and so
4
he travels 5 × 61 = 56 miles. Now to find the distance downstream he
finds himself, note that if x is this distance, x and 1/2 are two legs
of a right triangle whose hypotenuse equals 5/6 miles. Therefore,
by the Pythagorean theorem the distance downstream is
√
2
2
2
(5/6) − (1/2) = miles.
3
2.9
Exercises
1. The wind blows from West to East at a speed of 50 miles per hour and an airplane which
travels at 300 miles per hour in still air is heading North West. What is the velocity of the
airplane relative to the ground? What is the component of this velocity in the direction North?
2. In the situation of Problem 1 how many degrees to the West of North should the airplane head
in order to fly exactly North. What will be the speed of the airplane relative to the ground?
3. In the situation of 2 suppose the airplane uses 34 gallons of fuel every hour at that air speed
and that it needs to fly North a distance of 600 miles. Will the airplane have enough fuel to
arrive at its destination given that it has 63 gallons of fuel?
24
CHAPTER 2. FN
4. An airplane is flying due north at 150 miles per hour. A wind is pushing the airplane due east
at 40 miles per hour. After 1 hour, the plane starts flying 30◦ East of North. Assuming the
plane starts at (0, 0) , where is it after 2 hours? Let North be the direction of the positive y
axis and let East be the direction of the positive x axis.
5. City A is located at the origin while city B is located at (300, 500) where distances are in miles.
An airplane flies at 250 miles per hour in still air. This airplane wants to fly from city A to
city B but the wind is blowing in the direction of the positive y axis at a speed of 50 miles per
hour. Find a unit vector such that if the plane heads in this direction, it will end up at city B
having flown the shortest possible distance. How long will it take to get there?
6. A certain river is one half mile wide with a current flowing at 2 miles per hour from East to
West. A man swims directly toward the opposite shore from the South bank of the river at
a speed of 3 miles per hour. How far down the river does he find himself when he has swam
across? How far does he end up swimming?
7. A certain river is one half mile wide with a current flowing at 2 miles per hour from East to
West. A man can swim at 3 miles per hour in still water. In what direction should he swim
in order to travel directly across the river? What would the answer to this problem be if the
river flowed at 3 miles per hour and the man could swim only at the rate of 2 miles per hour?
8. Three forces are applied to a point which does not move. Two of the forces are 2i + j + 3k
Newtons and i − 3j + 2k Newtons. Find the third force.
9. The total force acting on an object is to be 2i + j + k Newtons. A force of −i + j + k Newtons
is being applied. What other force should be applied to achieve the desired total force?
10. A bird flies from its nest 5 km. in the direction 60◦ north of east where it stops to rest on a
tree. It then flies 10 km. in the direction due southeast and lands atop a telephone pole. Place
an xy coordinate system so that the origin is the bird’s nest, and the positive x axis points
east and the positive y axis points north. Find the displacement vector from the nest to the
telephone pole.
11. A car is stuck in the mud. There is a cable stretched tightly from this car to a tree which is
20 feet long. A person grasps the cable in the middle and pulls with a force of 100 pounds
perpendicular to the stretched cable. The center of the cable moves two feet and remains still.
What is the tension in the cable? The tension in the cable is the force exerted on this point
by the part of the cable nearer the car as well as the force exerted on this point by the part of
the cable nearer the tree.
Chapter 3
Vector Products
3.1
The Dot Product
There are two ways of multiplying vectors which are of great importance in applications. The first of
these is called the dot product, also called the scalar product and sometimes the inner product.
Definition 3.1.1 Let a, b be two vectors in Rn define a · b as
a·b≡
n
∑
ak bk .
k=1
The dot product a · b is sometimes denoted as (a, b) of ⟨a, b⟩ where a comma replaces ·.
With this definition, there are several important properties satisfied by the dot product. In the
statement of these properties, α and β will denote scalars and a, b, c will denote vectors.
Proposition 3.1.2 The dot product satisfies the following properties.
a·b=b·a
(3.1)
a · a ≥ 0 and equals zero if and only if a = 0
(3.2)
(αa + βb) · c =α (a · c) + β (b · c)
(3.3)
c · (αa + βb) = α (c · a) + β (c · b)
(3.4)
2
|a| = a · a
(3.5)
You should verify these properties. Also be sure you understand that 3.4 follows from the first
three and is therefore redundant. It is listed here for the sake of convenience.
Example 3.1.3 Find (1, 2, 0, −1) · (0, 1, 2, 3) .
This equals 0 + 2 + 0 + −3 = −1.
Example 3.1.4 Find the magnitude of a = (2, 1, 4, 2) . That is, find |a| .
√
This is (2, 1, 4, 2) · (2, 1, 4, 2) = 5.
The dot product satisfies a fundamental inequality known as the Cauchy Schwarz inequality.
Theorem 3.1.5 The dot product satisfies the inequality
|a · b| ≤ |a| |b| .
Furthermore equality is obtained if and only if one of a or b is a scalar multiple of the other.
25
(3.6)
26
CHAPTER 3. VECTOR PRODUCTS
Proof: First note that if b = 0 both sides of 3.6 equal zero and so the inequality holds in this
case. Therefore, it will be assumed in what follows that b ̸= 0.
Define a function of t ∈ R
f (t) = (a + tb) · (a + tb) .
Then by 3.2, f (t) ≥ 0 for all t ∈ R. Also from 3.3,3.4,3.1, and 3.5
f (t) = a · (a + tb) + tb · (a + tb)
= a · a + t (a · b) + tb · a + t2 b · b
2
2
= |a| + 2t (a · b) + |b| t2 .
Now this means the graph, y = f (t) is a polynomial which opens up and either its vertex touches
the t axis or else the entire graph is above the t axis. In the first case, there exists some t where
f (t) = 0 and this requires a + tb = 0 so one vector is a multiple of the other. Then clearly equality
holds in 3.6. In the case where b is not a multiple of a, it follows f (t) > 0 for all t which says f (t)
has no real zeros and so from the quadratic formula,
2
2
2
(2 (a · b)) − 4 |a| |b| < 0
which is equivalent to |(a · b)| < |a| |b|. t
t
You should note that the entire argument was based only on the properties of the dot product
listed in 3.1 - 3.5. This means that whenever something satisfies these properties, the Cauchy
Schwarz inequality holds. There are many other instances of these properties besides vectors in Rn .
The Cauchy Schwarz inequality allows a proof of the triangle inequality for distances in Rn
in much the same way as the triangle inequality for the absolute value.
Theorem 3.1.6 (Triangle inequality) For a, b ∈ Rn
|a + b| ≤ |a| + |b|
(3.7)
and equality holds if and only if one of the vectors is a nonnegative scalar multiple of the other. Also
||a| − |b|| ≤ |a − b|
(3.8)
Proof : By properties of the dot product and the Cauchy Schwarz inequality,
2
|a + b| = (a + b) · (a + b) = (a · a) + (a · b) + (b · a) + (b · b)
2
2
2
= |a| + 2 (a · b) + |b| ≤ |a| + 2 |a · b| + |b|
2
2
2
2
≤ |a| + 2 |a| |b| + |b| = (|a| + |b|) .
Taking square roots of both sides you obtain 3.7.
It remains to consider when equality occurs. If either vector equals zero, then that vector equals
zero times the other vector and the claim about when equality occurs is verified. Therefore, it can
be assumed both vectors are nonzero. To get equality in the second inequality above, Theorem 3.1.5
implies one of the vectors must be a multiple of the other. Say b = αa. If α < 0 then equality
cannot occur in the first inequality because in this case
2
2
(a · b) = α |a| < 0 < |α| |a| = |a · b|
Therefore, α ≥ 0.
To get the other form of the triangle inequality,
a=a−b+b
3.2. THE GEOMETRIC SIGNIFICANCE OF THE DOT PRODUCT
27
so
|a| = |a − b + b| ≤ |a − b| + |b| .
Therefore,
|a| − |b| ≤ |a − b|
(3.9)
|b| − |a| ≤ |b − a| = |a − b| .
(3.10)
Similarly,
It follows from 3.9 and 3.10 that 3.8 holds. This is because ||a| − |b|| equals the left side of either
3.9 or 3.10 and either way, ||a| − |b|| ≤ |a − b|. 3.2
3.2.1
The Geometric Significance Of The Dot Product
The Angle Between Two Vectors
Given two vectors, a and b, the included angle is the angle between these two vectors which is less
than or equal to 180 degrees. The dot product can be used to determine the included angle between
two vectors. To see how to do this, consider the following picture.
b
θ
*
a
a−b
q
U
U
By the law of cosines,
2
2
2
|a − b| = |a| + |b| − 2 |a| |b| cos θ.
Also from the properties of the dot product,
2
2
2
|a − b| = (a − b) · (a − b) = |a| + |b| − 2a · b
and so comparing the above two formulas,
a · b = |a| |b| cos θ.
(3.11)
In words, the dot product of two vectors equals the product of the magnitude of the two vectors
multiplied by the cosine of the included angle. Note this gives a geometric description of the dot
product which does not depend explicitly on the coordinates of the vectors.
Example 3.2.1 Find the angle between the vectors 2i + j − k and 3i + 4j + k.
√
√
The
√ dot product√of these two vectors equals 6 + 4 − 1 = 9 and the norms are 4 + 1 + 1 = 6
and 9 + 16 + 1 = 26. Therefore, from 3.11 the cosine of the included angle equals
9
cos θ = √ √ = . 720 58
26 6
Now the cosine is known, the angle can be determines by solving the equation, cos θ = . 720 58.
This will involve using a calculator or a table of trigonometric functions. The answer is θ = . 766 16
◦
radians or in terms of degrees, θ = . 766 16× 360
2π = 43. 898 . Recall how this last computation is done.
x
360
◦
Set up a proportion, .76616 = 2π because 360 corresponds to 2π radians. However, in calculus, you
should get used to thinking in terms of radians and not degrees. This is because all the important
calculus formulas are defined in terms of radians.
28
CHAPTER 3. VECTOR PRODUCTS
Example 3.2.2 Let u, v be two vectors whose magnitudes are equal to 3 and 4 respectively and such
that if they are placed in standard position with their tails at the origin, the angle between u and the
positive x axis equals 30◦ and the angle between v and the positive x axis is −30◦ . Find u · v.
From the geometric description of the dot product in 3.11
u · v = 3 × 4 × cos (60◦ ) = 3 × 4 × 1/2 = 6.
Observation 3.2.3 Two vectors are said to be perpendicular if the included angle is π/2 radians
(90◦ ). You can tell if two nonzero vectors are perpendicular by simply taking their dot product. If
the answer is zero, this means they are perpendicular because cos θ = 0.
Example 3.2.4 Determine whether the two vectors, 2i + j − k and 1i + 3j + 5k are perpendicular.
When you take this dot product you get 2 + 3 − 5 = 0 and so these two are indeed perpendicular.
Definition 3.2.5 When two lines intersect, the angle between the two lines is the smaller of the two
angles determined.
Example 3.2.6 Find the angle between the two lines, (1, 2, 0)+t (1, 2, 3) and (0, 4, −3)+t (−1, 2, −3) .
These two lines intersect, when t = 0 in the first and t = −1 in the second. It is only a matter
of finding the angle between the direction vectors. One angle determined is given by
cos θ =
−6
−3
=
.
14
7
(3.12)
We don’t want this angle because it is obtuse. The angle desired is the acute angle given by cos θ = 37 .
It is obtained by replacing one of the direction vectors with −1 times it.
3.2.2
Work And Projections
Our first application will be to the concept of work. The physical concept of work does not in any
way correspond to the notion of work employed in ordinary conversation. For example, if you were
to slide a 150 pound weight off a table which is three feet high and shuffle along the floor for 50
yards, sweating profusely and exerting all your strength to keep the weight from falling on your
feet, keeping the height always three feet and then deposit this weight on another three foot high
table, the physical concept of work would indicate that the force exerted by your arms did no work
during this project even though the muscles in your hands and arms would likely be very tired. The
reason for such an unusual definition is that even though your arms exerted considerable force on
the weight, enough to keep it from falling, the direction of motion was at right angles to the force
they exerted. The only part of a force which does work in the sense of physics is the component
of the force in the direction of motion (This is made more precise below.). The work is defined to
be the magnitude of the component of this force times the distance over which it acts in the case
where this component of force points in the direction of motion and (−1) times the magnitude of
this component times the distance in case the force tends to impede the motion. Thus the work
done by a force on an object as the object moves from one point to another is a measure of the
extent to which the force contributes to the motion. This is illustrated in the following picture in
the case where the given force contributes to the motion.
F⊥O
p1
F
θ :
F||
p2
In this picture the force, F is applied to an object which moves on the straight line from p1 to p2 .
There are two vectors shown, F|| and F⊥ and the picture is intended to indicate that when you add
these two vectors you get F while F|| acts in the direction of motion and F⊥ acts perpendicular to
3.2. THE GEOMETRIC SIGNIFICANCE OF THE DOT PRODUCT
29
the direction of motion. Only F|| contributes to the work done by F on the object as it moves from
p1 to p2 . F|| is called the component of the force in the direction of motion. From trigonometry,
you see the magnitude of F|| should equal |F| |cos θ| . Thus, since F|| points in the direction of the
vector from p1 to p2 , the total work done should equal
→ cos θ = |F| |p − p | cos θ
|F| −
p−1−
p
2
2
1
If the included angle had been obtuse, then the work done by the force, F on the object would have
been negative because in this case, the force tends to impede the motion from p1 to p2 but in this
case, cos θ would also be negative and so it is still the case that the work done would be given by
the above formula. Thus from the geometric description of the dot product given above, the work
equals
|F| |p2 − p1 | cos θ = F· (p2 −p1 ) .
This explains the following definition.
Definition 3.2.7 Let F be a force acting on an object which moves from the point p1 to the point
p2 . Then the work done on the object by the given force equals F· (p2 − p1 ) .
The concept of writing a given vector F in terms of two vectors, one which is parallel to a
given vector D and the other which is perpendicular can also be explained with no reliance on
trigonometry, completely in terms of the algebraic properties of the dot product. As before, this is
mathematically more significant than any approach involving geometry or trigonometry because it
extends to more interesting situations. This is done next.
Theorem 3.2.8 Let F and D be nonzero vectors. Then there exist unique vectors F|| and F⊥ such
that
F = F|| + F⊥
(3.13)
where F|| is a scalar multiple of D, also referred to as
projD (F) ,
and F⊥ · D = 0. The vector projD (F) is called the projection of F onto D.
Proof: Suppose 3.13 and F|| = αD. Taking the dot product of both sides with D and using
F⊥ · D = 0, this yields
2
F · D = α |D|
2
which requires α = F · D/ |D| . Thus there can be no more than one vector F|| . It follows F⊥ must
equal F − F|| . This verifies there can be no more than one choice for both F|| and F⊥ .
Now let
F·D
F|| ≡
2 D
|D|
and let
F⊥ = F − F|| = F−
Then F|| = α D where α =
F·D
.
|D|2
F·D
|D|
2
D
It only remains to verify F⊥ · D = 0. But
F⊥ · D = F · D−
F·D
2 D·D
|D|
= F · D − F · D = 0. Example 3.2.9 Let F = 2i+7j − 3k Newtons. Find the work done by this force in moving from
the point (1, 2, 3) to the point (−9, −3, 4) along the straight line segment joining these points where
distances are measured in meters.
30
CHAPTER 3. VECTOR PRODUCTS
According to the definition, this work is
(2i+7j − 3k) · (−10i − 5j + k) = −20 + (−35) + (−3)
= −58 Newton meters.
Note that if the force had been given in pounds and the distance had been given in feet, the
units on the work would have been foot pounds. In general, work has units equal to units of a force
times units of a length. Instead of writing Newton meter, people write joule because a joule is by
definition a Newton meter. That word is pronounced “jewel” and it is the unit of work in the metric
system of units. Also be sure you observe that the work done by the force can be negative as in
the above example. In fact, work can be either positive, negative, or zero. You just have to do the
computations to find out.
Example 3.2.10 Find proju (v) if u = 2i + 3j − 4k and v = i − 2j + k.
From the above discussion in Theorem 3.2.8, this is just
1
(i − 2j + k) · (2i + 3j − 4k) (2i + 3j − 4k)
4 + 9 + 16
−8
16
24
32
(2i + 3j − 4k) = − i − j + k.
29
29
29
29
=
Example 3.2.11 Suppose a, and b are vectors and b⊥ = b − proja (b) . What is the magnitude of
b⊥ in terms of the included angle?
(
2
|b⊥ | = (b − proja (b)) · (b − proja (b)) =
2
= |b| − 2
= |b|
2
(b · a)
|a|
2
(
2
+
b·a
2
|a|
b−
|a|
(
)2
2
|a| = |b|
b·a
2
(
)
2
1 − cos2 θ = |b| sin2 (θ)
2
1−
) (
a
·
b−
(b · a)
2
2
2
b·a
)
2
|a|
)
a
|a| |b|
where θ is the included angle between a and b which is less than π radians. Therefore, taking
square roots, |b⊥ | = |b| sin θ.
bM b⊥
θ 1
0 proja (b)
3.2.3
The Inner Product And Distance In Cn
It is necessary to give a generalization of the dot product for vectors in Cn . This is often called the
inner product. It reduces to the definition of the dot product in the case the components of the
vector are real.
Definition 3.2.12 Let x, y ∈ Cn . Thus x = (x1 , · · · , xn ) where each xk ∈ C and a similar formula
holding for y. Then the inner product of these two vectors is defined to be
∑
x·y ≡
xj yj ≡ x1 y1 + · · · + xn yn .
j
The inner product is often denoted as (x, y) or ⟨x, y⟩ .
3.2. THE GEOMETRIC SIGNIFICANCE OF THE DOT PRODUCT
31
Notice how you put the conjugate on the entries of the vector y. It makes no difference if the
vectors happen to be real vectors but with complex vectors you must do it this way. The reason for
this is that when you take the inner product of a vector with itself, you want to get the square of
the length of the vector, a positive number. Placing the conjugate on the components of y in the
above definition assures this will take place. Thus
∑
∑
2
x·x=
xj xj =
|xj | ≥ 0.
j
j
If you didn’t place a conjugate as in the above definition, things wouldn’t work out correctly. For
example,
2
(1 + i) + 22 = 4 + 2i
and this is not a positive number.
The following properties of the inner product follow immediately from the definition and you
should verify each of them.
Properties of the inner product:
1. u · v = v · u.
2. If a, b are numbers and u, v, z are vectors then (au + bv) · z = a (u · z) + b (v · z) .
3. u · u ≥ 0 and it equals 0 if and only if u = 0.
Note this implies (x·αy) = α (x · y) because
(x·αy) = (αy · x) = α (y · x) = α (x · y)
The norm is defined in the usual way.
Definition 3.2.13 For x ∈ Cn ,
(
|x| ≡
n
∑
)1/2
|xk |
2
= (x · x)
1/2
k=1
Here is a fundamental inequality called the Cauchy Schwarz inequality which is stated here
in Cn . First here is a simple lemma.
Lemma 3.2.14 If z ∈ C there exists θ ∈ C such that θz = |z| and |θ| = 1.
Proof: Let θ = 1 if z = 0 and otherwise, let θ =
z
. Recall that for z = x + iy, z = x − iy and
|z|
2
zz = |z| .
I will give a proof of this important inequality which depends only on the above list of properties
of the inner product. It will be slightly different than the earlier proof.
Theorem 3.2.15 (Cauchy Schwarz)The following inequality holds for x and y ∈ Cn .
|(x · y)| ≤ (x · x)
1/2
(y · y)
1/2
Equality holds in this inequality if and only if one vector is a multiple of the other.
Proof: Let θ ∈ C such that |θ| = 1 and
θ (x · y) = |(x · y)|
(3.14)
32
CHAPTER 3. VECTOR PRODUCTS
(
)
Consider p (t) ≡ x + θty, x + tθy where t ∈ R. Then from the above list of properties of the dot
product,
0 ≤ p (t) = (x · x) + tθ (x · y) + tθ (y · x) + t2 (y · y)
= (x · x) + tθ (x · y) + tθ(x · y) + t2 (y · y)
= (x · x) + 2t Re (θ (x · y)) + t2 (y · y)
= (x · x) + 2t |(x · y)| + t2 (y · y)
(3.15)
and this must hold for all t ∈ R. Therefore, if (y · y) = 0 it must be the case that |(x · y)| = 0 also
since otherwise the above inequality would be violated. Therefore, in this case,
1/2
|(x · y)| ≤ (x · x)
1/2
(y · y)
.
On the other hand, if (y · y) ̸= 0, then p (t) ≥ 0 for all t means the graph of y = p (t) is a parabola
which opens up and it either has exactly one real zero in the case its vertex touches the t axis or it
has no real zeros.
t
t
From the quadratic formula this happens exactly when
2
4 |(x · y)| − 4 (x · x) (y · y) ≤ 0
which is equivalent to 3.14.
It is clear from a computation that if one vector is a scalar multiple of the other that equality
2
holds in 3.14. Conversely, suppose equality does hold. Then this is equivalent to saying 4 |(x · y)| −
4 (x · x) (y · y) = 0 and so from the quadratic formula, there exists one real zero to p (t) = 0. Call it
t0 . Then
((
) (
))
2
p (t0 ) ≡ x + θt0 y · x + t0 θy = x + θty = 0
and so x = −θt0 y. Note that I only used part of the above properties of the inner product. It was not necessary to
use the one which says that if (x · x) = 0 then x = 0.
By analogy to the case of Rn , length or magnitude of vectors in Cn can be defined.
1/2
Definition 3.2.16 Let z ∈ Cn . Then |z| ≡ (z · z)
.
The conclusions of the following theorem are also called the axioms for a norm.
Theorem 3.2.17 For length defined in Definition 3.2.16, the following hold.
|z| ≥ 0 and |z| = 0 if and only if z = 0
(3.16)
If α is a scalar, |αz| = |α| |z|
(3.17)
|z + w| ≤ |z| + |w| .
(3.18)
Proof: The first two claims are left as exercises. To establish the third, you use the same
argument which was used in Rn .
|z + w|
2
=
(z + w, z + w)
= z·z+w·w+w·z+z·w
2
2
= |z| + |w| + 2 Re w · z
≤
2
2
2
2
|z| + |w| + 2 |w · z|
2
≤ |z| + |w| + 2 |w| |z| = (|z| + |w|) . Occasionally, I may refer to the inner product in Cn as the dot product. They are the same thing
for Rn . However, it is convenient to draw a distinction when discussing matrix multiplication a little
later.
3.3. EXERCISES
3.3
33
Exercises
1. Use formula 3.11 to verify the Cauchy Schwarz inequality and to show that equality occurs if
and only if one of the vectors is a scalar multiple of the other.
2. For u, v vectors in R3 , define the product, u ∗ v ≡ u1 v1 + 2u2 v2 + 3u3 v3 . Show the axioms for
a dot product all hold for this funny product. Prove
|u ∗ v| ≤ (u ∗ u)
1/2
(v ∗ v)
1/2
.
Hint: Do not try to do this with methods from trigonometry.
3. Find the angle between the vectors 3i − j − k and i + 4j + 2k.
4. Find the angle between the vectors i − 2j + k and i + 2j − 7k.
5. Find proju (v) where v = (1, 0, −2) and u = (1, 2, 3) .
6. Find proju (v) where v = (1, 2, −2) and u = (1, 0, 3) .
7. Find proju (v) where v = (1, 2, −2, 1) and u = (1, 2, 3, 0) .
8. Does it make sense to speak of proj0 (v)?
9. If F is a force and D is a vector, show projD (F) = (|F| cos θ) u where u is the unit vector in
the direction of D, u = D/ |D| and θ is the included angle between the two vectors, F and D.
|F| cos θ is sometimes called the component of the force, F in the direction, D.
10. Prove the Cauchy Schwarz inequality in Rn as follows. For u, v vectors, consider
(u − projv u) · (u − projv u) ≥ 0
Now simplify using the axioms of the dot product and then put in the formula for the projection.
Of course this expression equals 0 and you get equality in the Cauchy Schwarz inequality if
and only if u = projv u. What is the geometric meaning of u = projv u?
11. A boy drags a sled for 100 feet along the ground by pulling on a rope which is 20 degrees from
the horizontal with a force of 40 pounds. How much work does this force do?
12. A girl drags a sled for 200 feet along the ground by pulling on a rope which is 30 degrees from
the horizontal with a force of 20 pounds. How much work does this force do?
13. A large dog drags a sled for 300 feet along the ground by pulling on a rope which is 45 degrees
from the horizontal with a force of 20 pounds. How much work does this force do?
14. How much work in Newton meters does it take to slide a crate 20 meters along a loading dock
by pulling on it with a 200 Newton force at an angle of 30◦ from the horizontal?
15. An object moves 10 meters in the direction of j. There are two forces acting on this object,
F1 = i + j + 2k, and F2 = −5i + 2j−6k. Find the total work done on the object by the two
forces. Hint: You can take the work done by the resultant of the two forces or you can add
the work done by each force. Why?
16. An object moves 10 meters in the direction of j + i. There are two forces acting on this object,
F1 = i + 2j + 2k, and F2 = 5i + 2j−6k. Find the total work done on the object by the two
forces. Hint: You can take the work done by the resultant of the two forces or you can add
the work done by each force. Why?
17. An object moves 20 meters in the direction of k + j. There are two forces acting on this object,
F1 = i + j + 2k, and F2 = i + 2j−6k. Find the total work done on the object by the two forces.
Hint: You can take the work done by the resultant of the two forces or you can add the work
done by each force.
34
CHAPTER 3. VECTOR PRODUCTS
18. If a, b, c are vectors. Show that (b + c)⊥ = b⊥ + c⊥ where b⊥ = b− proja (b) .
19. Find (1, 2, 3, 4) · (2, 0, 1, 3) .
[
]
2
2
20. Show that (a · b) = 14 |a + b| − |a − b| .
2
2
21. Prove from the axioms of the dot product the parallelogram identity, |a + b| + |a − b| =
2
2
2 |a| + 2 |b| .
22. Recall that the open ball having center at a and radius r is given by
B (a,r) ≡ {x : |x − a| < r}
Show that if y ∈ B (a, r) , then there exists a positive number δ such that B (y, δ) ⊆ B (a, r) .
(The symbol ⊆ means that every point in B (y, δ) is also in B (a, r) . In words, it states that
B (y, δ) is contained in B (a, r) . The statement y ∈ B (a, r) says that y is one of the points of
B (a, r) .) When you have done this, you will have shown that an open ball is open. This is a
fantastically important observation although its major implications will not be explored very
much in this book.
3.4
The Cross Product
The cross product is the other way of multiplying two vectors in R3 . It is very different from the
dot product in many ways. First the geometric meaning is discussed and then a description in
terms of coordinates is given. Both descriptions of the cross product are important. The geometric
description is essential in order to understand the applications to physics and geometry while the
coordinate description is the only way to practically compute the cross product.
Definition 3.4.1 Three vectors, a, b, c form a right handed system if when you extend the fingers
of your right hand along the vector a and close them in the direction of b, the thumb points roughly
in the direction of c.
For an example of a right handed system of vectors, see the following picture.
c
y
a
b
In this picture the vector c points upwards from the plane determined by the other two vectors.
You should consider how a right hand system would differ from a left hand system. Try using your
left hand and you will see that the vector c would need to point in the opposite direction as it would
for a right hand system.
From now on, the vectors, i, j, k will always form a right handed system. To repeat,
if you extend the fingers of your right hand along i and close them in
k
the direction j, the thumb points in the direction of k. The following
6
is the geometric description of the cross product. It gives both the
direction and the magnitude and therefore specifies the vector.
j
Definition 3.4.2 Let a and b be two vectors in R3 . Then a × b is
defined by the following two rules.
i
3.4. THE CROSS PRODUCT
35
1. |a × b| = |a| |b| sin θ where θ is the included angle.
2. a × b · a = 0, a × b · b = 0, and a, b, a × b forms a right hand
system.
Note that |a × b| is the area of the parallelogram determined by a and b.
3
b
θ
a
|b| sin(θ)
-
The cross product satisfies the following properties.
a × b = − (b × a) , a × a = 0,
(3.19)
(αa) ×b = α (a × b) = a× (αb) ,
(3.20)
For α a scalar,
For a, b, and c vectors, one obtains the distributive laws,
a× (b + c) = a × b + a × c,
(3.21)
(b + c) × a = b × a + c × a.
(3.22)
Formula 3.19 follows immediately from the definition. The vectors a × b and b × a have the
same magnitude, |a| |b| sin θ, and an application of the right hand rule shows they have opposite
direction. Formula 3.20 is also fairly clear. If α is a nonnegative scalar, the direction of (αa) ×b is
the same as the direction of a × b,α (a × b) and a× (αb) while the magnitude is just α times the
magnitude of a × b which is the same as the magnitude of α (a × b) and a× (αb) . Using this yields
equality in 3.20. In the case where α < 0, everything works the same way except the vectors are all
pointing in the opposite direction and you must multiply by |α| when comparing their magnitudes.
The distributive laws are much harder to establish but the second follows from the first quite easily.
Thus, assuming the first, and using 3.19,
(b + c) × a = −a× (b + c) = − (a × b + a × c)
= b × a + c × a.
A proof of the distributive law is given in a later section for those who are interested.
Now from the definition of the cross product,
i × j = k j × i = −k
k × i = j i × k = −j
j × k = i k × j = −i
With this information, the following gives the coordinate description of the cross product.
Proposition 3.4.3 Let a = a1 i + a2 j + a3 k and b = b1 i + b2 j + b3 k be two vectors. Then
a × b = (a2 b3 − a3 b2 ) i+ (a3 b1 − a1 b3 ) j+
+ (a1 b2 − a2 b1 ) k.
Proof: From the above table and the properties of the cross product listed,
(a1 i + a2 j + a3 k) × (b1 i + b2 j + b3 k) =
(3.23)
36
CHAPTER 3. VECTOR PRODUCTS
a1 b2 i × j + a1 b3 i × k + a2 b1 j × i + a2 b3 j × k+
+a3 b1 k × i + a3 b2 k × j
= a1 b2 k − a1 b3 j − a2 b1 k + a2 b3 i + a3 b1 j − a3 b2 i
= (a2 b3 − a3 b2 ) i+ (a3 b1 − a1 b3 ) j+ (a1 b2 − a2 b1 ) k
(3.24)
It is probably impossible for most people to remember 3.23. Fortunately, there is a somewhat
easier way to remember it. Define the determinant of a 2 × 2 matrix as follows
a
c
b
d
≡ ad − bc
Then
i
a × b = a1
b1
j
a2
b2
k
a3
b3
(3.25)
where you expand the determinant along the top row. This yields
i (−1)
a2
b2
1+1
=i
a3
b3
2+1
+ j (−1)
a2
b2
a3
b3
−j
a1
b1
a1
b1
a3
b3
a3
b3
+ k (−1)
+k
a1
b1
3+1
a1
b1
a2
b2
a2
b2
Note that to get the scalar which multiplies i you take the determinant of what is left after deleting
1+1
the first row and the first column and multiply by (−1)
because i is in the first row and the first
column. Then you do the same thing for the j and k. In the case of the j there is a minus sign
1+2
because j is in the first row and the second column and so(−1)
= −1 while the k is multiplied
3+1
by (−1)
= 1. The above equals
(a2 b3 − a3 b2 ) i− (a1 b3 − a3 b1 ) j+ (a1 b2 − a2 b1 ) k
(3.26)
which is the same as 3.24. There will be much more presented on determinants later. For now,
consider this an introduction if you have not seen this topic.
Example 3.4.4 Find (i − j + 2k) × (3i − 2j + k) .
Use 3.25 to compute this.
i
1
3
j k
−1 2
−2 1
=
−1
−2
2
1
i−
1
3
2
1
j+
1
3
−1
k = 3i + 5j + k.
−2
Example 3.4.5 Find the area of the parallelogram determined by the vectors,
(i − j + 2k) , (3i − 2j + k) .
These are the same two vectors in Example 3.4.4.
From Example 3.4.4 and the geometric description of the
√ cross product,√the area is just the norm
of the vector obtained in Example 3.4.4. Thus the area is 9 + 25 + 1 = 35.
Example 3.4.6 Find the area of the triangle determined by (1, 2, 3) , (0, 2, 5) , (5, 1, 2) .
3.4. THE CROSS PRODUCT
37
This triangle is obtained by connecting the three points with lines. Picking (1, 2, 3) as a starting
point, there are two displacement vectors, (−1, 0, 2) and (4, −1, −1) such that the given vector added
to these displacement vectors gives the other two vectors. The area of the triangle is half the area of
the parallelogram determined by (−1,
√ −1) . Thus (−1, 0, 2) × (4, −1, −1) = (2, 7, 1)
√ 0, 2) and (4, −1,
and so the area of the triangle is 21 4 + 49 + 1 = 32 6.
Observation 3.4.7 In general, if you have three points (vectors) in R3 , P, Q, R the area of the
triangle is given by
1
|(Q − P) × (R − P)| .
2
Q
P
3.4.1
-
R
The Distributive Law For The Cross Product
This section gives a proof for 3.21, a fairly difficult topic. It is included here for the interested
student. If you are satisfied with taking the distributive law on faith, it is not necessary to read this
section. The proof given here is quite clever and follows the one given in [3]. Another approach,
based on volumes of parallelepipeds is found in [16] and is discussed a little later.
Lemma 3.4.8 Let b and c be two vectors. Then b × c = b × c⊥ where c|| + c⊥ = c and c⊥ · b = 0.
Proof:
c⊥ 6c
θ
b
b b
Consider the following picture.
Now c⊥ = c − c· |b|
|b| and so c⊥ is in
the plane determined by c and b. Therefore, from the geometric definition of
the cross product, b × c and b × c⊥ have the same direction. Now, referring
to the picture,
|b × c⊥ | = |b| |c⊥ | = |b| |c| sin θ = |b × c| .
Therefore, b × c and b × c⊥ also have the same magnitude and so they are the same vector. With this, the proof of the distributive law is in the following theorem.
Theorem 3.4.9 Let a, b, and c be vectors in R3 . Then
a× (b + c) = a × b + a × c
(3.27)
Proof: Suppose first that a · b = a · c = 0. Now imagine a is a vector coming out of the page and
let b, c and b + c be as shown in the following picture. Then a × b, a× (b + c) , are each vectors in
the same plane, and a × c perpendicular to a as shown. Thus
a × (b + c)
a × c · c = 0, a× (b + c) · (b + c) = 0, and a × b · b = 0. This imM
plies that to get a × b you move counterclockwise through an angle
of π/2 radians from the vector b. Similar relationships exist between
the vectors a× (b + c) and b + c and the vectors a × c and c. Thus
6
the angle between a × b and a× (b + c) is the same as the angle
a×b
between b + c and b and the angle between a × c and a× (b + c)
a × cI
is
the same as the angle between c and b + c. In addition to this,
c
1
since
a is perpendicular to these vectors,
b
+
c
b
|a × b| = |a| |b| , |a× (b + c)| = |a| |b + c| , and
|a × c| = |a| |c| .
38
CHAPTER 3. VECTOR PRODUCTS
Therefore,
|a× (b + c)|
|a × c|
|a × b|
=
=
= |a|
|b + c|
|c|
|b|
and so
|a× (b + c)|
|b + c| |a× (b + c)|
|b + c|
=
,
=
|a × c|
|c|
|a × b|
|b|
showing the triangles making up the parallelogram on the right and the four sided figure on the left
in the above picture are similar. It follows the four sided figure on the left is in fact a parallelogram
and this implies the diagonal is the vector sum of the vectors on the sides, yielding 3.27.
Now suppose it is not necessarily the case that a · b = a · c = 0. Then write b = b|| + b⊥ where
b⊥ · a = 0. Similarly c = c|| + c⊥ . By the above lemma and what was just shown,
a× (b + c) = a× (b + c)⊥ = a× (b⊥ + c⊥ )
= a × b⊥ + a × c⊥ = a × b + a × c. The result of Problem 18 of the exercises 3.3 is used to go from the first to the second line.
3.4.2
The Box Product
Definition 3.4.10 A parallelepiped determined by the three vectors, a, b, and c consists of
{ra+sb + tc : r, s, t ∈ [0, 1]} .
That is, if you pick three numbers, r, s, and t each in [0, 1] and form ra+sb + tc, then the collection
of all such points is what is meant by the parallelepiped determined by these three vectors.
The following is a picture of such a thing.You notice the area of the base of the parallelepiped,
the parallelogram determined by the vectors, a and b
has area equal to |a × b| while the altitude of the parallelepiped is |c| cos θ where θ is the angle shown in the
6
picture between c and a × b. Therefore, the volume of
a×b
this parallelepiped is the area of the base times the altitude which is just
|a × b| |c| cos θ = a × b · c.
c
3
b
a
This expression is known as the box product and is sometimes written as [a, b, c] . You should consider what happens if you interchange the b with the c or the a with the
c. You can see geometrically from drawing pictures that
this merely introduces a minus sign. In any case the box product of three vectors always equals
either the volume of the parallelepiped determined by the three vectors or else minus this volume.
θ
-
Example 3.4.11 Find the volume of the parallelepiped determined by the vectors, i + 2j − 5k, i +
3j − 6k,3i + 2j + 3k.
According to the above discussion, pick any two of these, take the cross product and then take
the dot product of this with the third of these vectors. The result will be either the desired volume
or minus the desired volume.
(i + 2j − 5k) × (i + 3j − 6k) =
i j
1 2
1 3
k
−5
−6
= 3i + j + k
Now take the dot product of this vector with the third which yields
(3i + j + k) · (3i + 2j + 3k) = 9 + 2 + 3 = 14.
3.5. THE VECTOR IDENTITY MACHINE
39
This shows the volume of this parallelepiped is 14 cubic units.
There is a fundamental observation which comes directly from the geometric definitions of the
cross product and the dot product.
Lemma 3.4.12 Let a, b, and c be vectors. Then (a × b) ·c = a· (b × c) .
Proof: This follows from observing that either (a × b) ·c and a· (b × c) both give the volume of
the parallelepiped or they both give −1 times the volume. Notation 3.4.13 The box product a × b · c = a · b × c is denoted more compactly as [a, b, c].
3.4.3
Another Proof Of The Distributive Law
Here is another proof of the distributive law for the cross product. Let x be a vector. From the
above observation,
x · a× (b + c) = (x × a) · (b + c) = (x × a) · b+ (x × a) · c
= x · a × b + x · a × c = x· (a × b + a × c) .
Therefore,
x· [a× (b + c) − (a × b + a × c)] = 0
for all x. In particular, this holds for x = a× (b + c) − (a × b + a × c) showing that a× (b + c) =
a × b + a × c and this proves the distributive law for the cross product another way.
Observation 3.4.14 Suppose you have three vectors, u = (a, b, c) , v = (d, e, f ) , and w = (g, h, i) .
Then u · v × w is given by the following.
u · v × w = (a, b, c) ·
i j k
d e f
g h i
=

=a
e f
h i
−b
d
g
f
i
+c
d e
g h
a

≡ det  d
g

b c

e f .
h i
The message is that to take the box product, you can simply take the determinant of the matrix which
results by letting the rows be the rectangular components of the given vectors in the order in which
they occur in the box product. More will be presented on determinants later.
3.5
The Vector Identity Machine
In practice, you often have to deal with combinations of several cross products mixed in with dot
products. It is extremely useful to have a technique which will allow you to discover vector identities
and simplify expressions involving cross and dot products in three dimensions. This involves two
special symbols, δ ij and εijk which are very useful in dealing with vector identities. To begin with,
here is the definition of these symbols.
Definition 3.5.1 The symbol δ ij , called the Kronecker delta symbol is defined as follows.
{
1 if i = j
δ ij ≡
.
0 if i ̸= j
With the Kronecker symbol i and j can equal any integer in {1, 2, · · · , n} for any n ∈ N.
40
CHAPTER 3. VECTOR PRODUCTS
Definition 3.5.2 For i, j, and k integers in the set, {1, 2, 3} , εijk is defined as follows.


 1 if (i, j, k) = (1, 2, 3) , (2, 3, 1) , or (3, 1, 2)
εijk ≡
−1 if (i, j, k) = (2, 1, 3) , (1, 3, 2) , or (3, 2, 1) .


0 if there are any repeated integers
The subscripts ijk and ij in the above are called indices. A single one is called an index. This
symbol εijk is also called the permutation symbol.
The way to think of εijk is that ε123 = 1 and if you switch any two of the numbers in the list
i, j, k, it changes the sign. Thus εijk = −εjik and εijk = −εkji etc. You should check that this
rule reduces to the above definition. For example, it immediately implies that if there is a repeated
index, the answer is zero. This follows because εiij = −εiij and so εiij = 0.
It is useful to use the Einstein summation convention when dealing with these
∑ symbols. Simply
stated, ∑
the convention is that you sum over the repeated index. Thus ai bi means i ai bi . Also, δ ij xj
means j δ ij xj = xi . Thus δ ij xj = xi , δ ii = 3, δ ij xjkl = xikl . When you use this convention, there
is one very important thing to never forget. It is this: Never have an index be repeated more
than once. Thus ai bi is all right but aii bi is not. ∑
The reason for this is that you end up getting
confused about what is meant. If you want to write i ai bi ci it is best to simply use the summation
notation. There is a very important reduction identity connecting these two symbols.
Lemma 3.5.3 The following holds.
εijk εirs = (δ jr δ ks − δ kr δ js ) .
Proof: If {j, k} ̸= {r, s} then every term in the sum on the left must have either εijk or εirs
contains a repeated index. Therefore, the left side equals zero. The right side also equals zero in
this case. To see this, note that if the two sets are not equal, then there is one of the indices in one
of the sets which is not in the other set. For example, it could be that j is not equal to either r or
s. Then the right side equals zero.
Therefore, it can be assumed {j, k} = {r, s} . If i = r and j = s for s ̸= r, then there is exactly
one term in the sum on the left and it equals 1. The right also reduces to 1 in this case. If i = s
and j = r, there is exactly one term in the sum on the left which is nonzero and it must equal −1.
The right side also reduces to −1 in this case. If there is a repeated index in {j, k} , then every
term in the sum on the left equals zero. The right also reduces to zero in this case because then
j = k = r = s and so the right side becomes (1) (1) − (−1) (−1) = 0. Proposition 3.5.4 Let u, v be vectors in Rn where the Cartesian coordinates of u are (u1 , · · · , un )
and the Cartesian coordinates of v are (v1 , · · · , vn ). Then u · v = ui vi . If u, v are vectors in R3 ,
then
(u × v)i = εijk uj vk .
Also, δ ik ak = ai .
Proof: The first claim is obvious from the definition of the dot product. The second is verified
by simply checking that it works. For example,
i
u × v ≡ u1
v1
j
u2
v2
k
u3
v3
and so
(u × v)1 = (u2 v3 − u3 v2 ) .
From the above formula in the proposition,
ε1jk uj vk ≡ u2 v3 − u3 v2 ,
3.6. EXERCISES
41
the same thing. The cases for (u × v)2 and (u × v)3 are verified similarly. The last claim follows
directly from the definition. With this notation, you can easily discover vector identities and simplify expressions which
involve the cross product.
Example 3.5.5 Discover a formula which simplifies (u × v) · (z × w) , u, v ∈ R3 .
From the above description of the cross product and dot product, along with the reduction
identity,
(u × v) · (z × w) =
εijk uj vk εirs zr ws
= (δ jr δ ks − δ js δ kr ) uj vk zr ws
= uj vk zj wk − uj vk zk wj
= (u · z) (v · w) − (u · w) (v · z)
Example 3.5.6 Simplify u× (u × v) .
The ith component is
εijk uj (u × v)k
= εijk uj εkrs ur vs = εkij εkrs uj ur vs
= (δ ir δ js − δ jr δ is ) uj ur vs
= uj ui vj − uj uj vi
2
= (u · v) ui − |u| vi
Hence
2
u× (u × v) = (u · v) u − |u| v
because the ith components of the two sides are equal for any i.
3.6
Exercises
1. Show that if a × u = 0 for all unit vectors, u, then a = 0.
2. Find the area of the triangle determined by the three points, (1, 2, 3) , (4, 2, 0) and (−3, 2, 1) .
3. Find the area of the triangle determined by the three points, (1, 0, 3) , (4, 1, 0) and (−3, 1, 1) .
4. Find the area of the triangle determined by the three points, (1, 2, 3) , (2, 3, 4) and (3, 4, 5) .
Did something interesting happen here? What does it mean geometrically?
5. Find the area of the parallelogram determined by the vectors, (1, 2, 3), (3, −2, 1) .
6. Find the area of the parallelogram determined by the vectors, (1, 0, 3), (4, −2, 1) .
7. Find the volume of the parallelepiped determined by the vectors, i−7j−5k, i−2j−6k,3i+2j+3k.
8. Suppose a, b, and c are three vectors whose components are all integers. Can you conclude the
volume of the parallelepiped determined from these three vectors will always be an integer?
9. What does it mean geometrically if the box product of three vectors gives zero?
10. Using Problem 9, find an equation of a plane containing the two position vectors, a and b
and the point 0. Hint: If (x, y, z) is a point on this plane the volume of the parallelepiped
determined by (x, y, z) and the vectors a, b equals 0.
42
CHAPTER 3. VECTOR PRODUCTS
11. Using the notion of the box product yielding either plus or minus the volume of the parallelepiped determined by the given three vectors, show that
(a × b) ·c = a· (b × c)
In other words, the dot and the cross can be switched as long as the order of the vectors
remains the same. Hint: There are two ways to do this, by the coordinate description of the
dot and cross product and by geometric reasoning. It is better if you use geometric reasoning.
12. Is a× (b × c) = (a × b)×c? What is the meaning of a × b × c? Explain. Hint: Try (i × j) ×j.
13. Discover a vector identity for (u × v) ×w and one for u× (v × w).
14. Discover a vector identity for (u × v) × (z × w).
15. Simplify (u × v) · (v × w) × (w × z) .
2
2
2
2
16. Simplify |u × v| + (u · v) − |u| |v| .
17. For u, v, w functions of t, u′ (t) is defined as the limit of the difference quotient as in calculus,
(limh→0 w (h))i ≡ limh→0 wi (h) . Show the following
′
′
(u × v) = u′ × v + u × v′ , (u · v) = u′ · v + u · v′
18. If u is a function of t, and the magnitude |u (t)| is a constant, show from the above problem
that the velocity u′ is perpendicular to u.
19. When you have a rotating rigid body with angular velocity vector Ω, then the velocity vector
v ≡ u′ is given by
v =Ω×u
where u is a position vector. The acceleration is the derivative of the velocity. Show that if Ω
is a constant vector, then the acceleration vector a = v′ is given by the formula
a = Ω× (Ω × u) .
Now simplify the expression. It turns out this is centripetal acceleration.
20. Verify directly that the coordinate description of the cross product, a × b has the property
that it is perpendicular to both a and b. Then show by direct computation that this coordinate
description satisfies
2
2
2
2
|a × b| = |a| |b| − (a · b)
)
2
2(
= |a| |b| 1 − cos2 (θ)
where θ is the angle included between the two vectors. Explain why |a × b| has the correct
magnitude. All that is missing is the material about the right hand rule. Verify directly from
the coordinate description of the cross product that the right thing happens with regards to
the vectors i, j, k. Next verify that the distributive law holds for the coordinate description of
the cross product. This gives another way to approach the cross product. First define it in
terms of coordinates and then get the geometric properties from this. However, this approach
does not yield the right hand rule property very easily.
Chapter 4
Systems Of Equations
4.1
Systems Of Equations, Geometry
As you know, equations like 2x + 3y = 6 can be graphed as straight lines in R2 . To find the solution
to two such equations, you could graph the two straight lines and the ordered pairs identifying the
point (or points) of intersection would give the x and y values of the solution to the two equations
because such an ordered pair satisfies both equations. The following picture illustrates what can
occur with two equations involving two variables.
y
y
y
two parallel lines
no solutions
x
one solution
x
infinitely
many solutions
x
In the first example of the above picture, there is a unique point of intersection. In the second,
there are no points of intersection. The other thing which can occur is that the two lines are really
the same line. For example, x + y = 1 and 2x + 2y = 2 are relations which when graphed yield
the same line. In this case there are infinitely many points in the simultaneous solution of these
two equations, every ordered pair which is on the graph of the line. It is always this way when
considering linear systems of equations. There is either no solution, exactly one or infinitely many
although the reasons for this are not completely comprehended by considering a simple picture in
two dimensions, R2 .
Example 4.1.1 Find the solution to the system x + y = 3, y − x = 5.
You can verify the solution is (x, y) = (−1, 4) . You can see this geometrically by graphing the
equations of the two lines. If you do so correctly, you should obtain a graph which looks something
like the following in which the point of intersection represents the solution of the two equations.
(x, y) = (−1, 4)
x
Example 4.1.2 You can also imagine other situations
such as the case of three intersecting lines having no common point of intersection or three intersecting lines which
do intersect at a single point as illustrated in the following
picture.
43
44
CHAPTER 4. SYSTEMS OF EQUATIONS
y
y
x
x
In the case of the first picture above, there would be no solution to the three equations whose
graphs are the given lines. In the case of the second picture there is a solution to the three equations
whose graphs are the given lines.
The points, (x, y, z) satisfying an equation in three variables like 2x + 4y − 5z = 8 form a plane
1
and geometrically, when you solve systems of equations involving three variables, you are taking
intersections of planes.
Consider the following picture involving two planes. Notice how these two planes intersect in
a line. It could also happen the two planes could fail to intersect.
Now imagine a third plane. One thing that could happen is
this third plane could have an intersection with one of the first
planes which results in a line which fails to intersect the first line as
illustrated in the following picture.
Thus there is no point which lies in all three planes. The picture
illustrates the situation in which the line of intersection of the new
plane with one of the original planes forms a line parallel to the line
of intersection of the first two planes. However, in three dimensions,
New Plane it is possible for two lines to fail to intersect even though they are
not parallel. Such lines are called skew lines. You might consider
whether there exist two skew lines, each of which is the intersection
of a pair of planes selected from a set of exactly three planes such
that there is no point of intersection between the three planes. You
can also see that if you tilt one of the planes you could obtain every
pair of planes having a nonempty intersection in a line and yet there
may be no point in the intersection of all three.
It could happen also that the three planes could intersect in a single point as shown in the
following picture.
In this case, the three planes have a single point of intersection.
The three planes could also intersect in a line. Thus in the case
New Plane
of three equations having three variables, the planes determined by
these equations could intersect in a single point, a line, or even fail
to intersect at all. You see that in three dimensions there are many
possibilities. If you want to waste some time, you can try to imagine
all the things which could happen but this will not help for more
variables than 3 which is where many of the important applications
lie.
Relations like x+y−2z+4w = 8 are often called hyper-planes.2
However, it is impossible to draw pictures of such things. The only
rational and useful way to deal with this subject is through the use
of algebra not art. Mathematics exists partly to free us from having
to always draw pictures in order to draw conclusions.
1 Don’t worry about why this is at this time. It is not important. The discussion is intended to show you that
geometric considerations like this don’t take you anywhere. It is the algebraic procedures which are important and
lead to important applications.
2 The evocative semi word, “hyper” conveys absolutely no meaning but is traditional usage which makes the
terminology sound more impressive than something like long wide flat thing.Later we will discuss some terms which
are not just evocative but yield real understanding.
4.2. SYSTEMS OF EQUATIONS, ALGEBRAIC PROCEDURES
4.2
4.2.1
45
Systems Of Equations, Algebraic Procedures
Elementary Operations
Consider the following example.
Example 4.2.1 Find x and y such that
x + y = 7 and 2x − y = 8.
(4.1)
The set of ordered pairs, (x, y) which solve both equations is called the solution set.
You can verify that (x, y) = (5, 2) is a solution to the above system. The interesting question is
this: If you were not given this information to verify, how could you determine the solution? You
can do this by using the following basic operations on the equations, none of which change the set
of solutions of the system of equations.
Definition 4.2.2 Elementary operations are those operations consisting of the following.
1. Interchange the order in which the equations are listed.
2. Multiply any equation by a nonzero number.
3. Replace any equation with itself added to a multiple of another equation.
Example 4.2.3 To illustrate the third of these operations on this particular system, consider the
following.
x+y =7
2x − y = 8
The system has the same solution set as the system
x+y =7
.
−3y = −6
To obtain the second system, take the second equation of the first system and add −2 times the first
equation to obtain
−3y = −6.
Now, this clearly shows that y = 2 and so it follows from the other equation that x + 2 = 7 and so
x = 5.
Of course a linear system may involve many equations and many variables. The solution set is
still the collection of solutions to the equations. In every case, the above operations of Definition
4.2.2 do not change the set of solutions to the system of linear equations.
Theorem 4.2.4 Suppose you have two equations, involving the variables,
(x1 , · · · , xn )
E 1 = f1 , E 2 = f2
(4.2)
where E1 and E2 are expressions involving the variables and f1 and f2 are constants. (In the above
example there are only two variables, x and y and E1 = x + y while E2 = 2x − y.) Then the system
E1 = f1 , E2 = f2 has the same solution set as
E1 = f1 , E2 + aE1 = f2 + af1 .
(4.3)
Also the system E1 = f1 , E2 = f2 has the same solutions as the system, E2 = f2 , E1 = f1 . The
system E1 = f1 , E2 = f2 has the same solution as the system E1 = f1 , aE2 = af2 provided a ̸= 0.
46
CHAPTER 4. SYSTEMS OF EQUATIONS
Proof: If (x1 , · · · , xn ) solves E1 = f1 , E2 = f2 then it solves the first equation in E1 = f1 , E2 +
aE1 = f2 + af1 . Also, it satisfies aE1 = af1 and so, since it also solves E2 = f2 it must solve
E2 +aE1 = f2 +af1 . Therefore, if (x1 , · · · , xn ) solves E1 = f1 , E2 = f2 it must also solve E2 +aE1 =
f2 +af1 . On the other hand, if it solves the system E1 = f1 and E2 +aE1 = f2 +af1 , then aE1 = af1
and so you can subtract these equal quantities from both sides of E2 + aE1 = f2 + af1 to obtain
E2 = f2 showing that it satisfies E1 = f1 , E2 = f2 .
The second assertion of the theorem which says that the system E1 = f1 , E2 = f2 has the same
solution as the system, E2 = f2 , E1 = f1 is seen to be true because it involves nothing more than
listing the two equations in a different order. They are the same equations.
The third assertion of the theorem which says E1 = f1 , E2 = f2 has the same solution as the
system E1 = f1 , aE2 = af2 provided a ̸= 0 is verified as follows: If (x1 , · · · , xn ) is a solution of
E1 = f1 , E2 = f2 , then it is a solution to E1 = f1 , aE2 = af2 because the second system only
involves multiplying the equation, E2 = f2 by a. If (x1 , · · · , xn ) is a solution of E1 = f1 , aE2 = af2 ,
then upon multiplying aE2 = af2 by the number 1/a, you find that E2 = f2 . Stated simply, the above theorem shows that the elementary operations do not change the solution set of a system of equations.
Here is an example in which there are three equations and three variables. You want to find
values for x, y, z such that each of the given equations are satisfied when these values are plugged in
to the equations.
Example 4.2.5 Find the solutions to the system,
x + 3y + 6z = 25
2x + 7y + 14z = 58
2y + 5z = 19
(4.4)
To solve this system replace the second equation by (−2) times the first equation added to the
second. This yields the system
x + 3y + 6z = 25
(4.5)
y + 2z = 8
2y + 5z = 19
Now take (−2) times the second and add to the third. More precisely, replace the third equation
with (−2) times the second added to the third. This yields the system
x + 3y + 6z = 25
y + 2z = 8
z=3
(4.6)
At this point, you can tell what the solution is. This system has the same solution as the original
system and in the above, z = 3. Then using this in the second equation, it follows y + 6 = 8 and so
y = 2. Now using this in the top equation yields x + 6 + 18 = 25 and so x = 1. This process is called
back substitution.
Alternatively, in 4.6 you could have continued as follows. Add (−2) times the bottom equation
to the middle and then add (−6) times the bottom to the top. This yields
x + 3y = 7
y=2
z=3
Now add (−3) times the second to the top. This yields
x=1
y=2 ,
z=3
4.2. SYSTEMS OF EQUATIONS, ALGEBRAIC PROCEDURES
47
a system which has the same solution set as the original system. This avoided back substitution and
led to the same solution set.
4.2.2
Gauss Elimination
A less cumbersome way to represent a linear system is to write it as an augmented matrix. For
example the linear system, 4.4 can be written as


1 3 6 | 25


 2 7 14 | 58  .
0 2 5 | 19
It has exactly
 it is understood there is an
 same information
 asthe original systembut here
 the
6
3
1


 
 
x column,  2  , a y column,  7  and a z column,  14  . The rows correspond to the
5
0
2
equations in the system. Thus the top row in the augmented matrix corresponds to the equation,
x + 3y + 6z = 25.
Now when you replace an equation with a multiple of another equation added to itself, you are just
taking a row of this augmented matrix and replacing it with a multiple of another row added to it.
Thus the first step in solving 4.4 would be to take (−2) times the first row of the augmented matrix
above and add it to the second row,


1 3 6 | 25


 0 1 2 | 8 .
0 2 5 | 19
Note how this corresponds to 4.5. Next take

1

 0
0
(−2) times the second row and add to the third,

3 6 | 25

1 2 |8 
0 1 |3
This augmented matrix corresponds to the system
x + 3y + 6z = 25
y + 2z = 8
z=3
which is the same as 4.6. By back substitution you obtain the solution x = 1, y = 6, and z = 3.
In general a linear system is of the form
a11 x1 + · · · + a1n xn = b1
..
,
.
am1 x1 + · · · + amn xn = bm
(4.7)
where the xi are variables and the aij and bi are constants. This system can be represented by the
augmented matrix


a11 · · · a1n | b1
 .
..
. 
 .

(4.8)
.
| ..  .
 .
am1 · · · amn | bm
48
CHAPTER 4. SYSTEMS OF EQUATIONS
Changes to the system of equations in 4.7 as a result of an elementary operations translate into
changes of the augmented matrix resulting from a row operation. Note that Theorem 4.2.4 implies
that the row operations deliver an augmented matrix for a system of equations which has the same
solution set as the original system.
Definition 4.2.6 The row operations consist of the following
1. Switch two rows.
2. Multiply a row by a nonzero number.
3. Replace a row by a multiple of another row added to it.
Gauss elimination is a systematic procedure to simplify an augmented matrix to a reduced
form. In the following definition, the term “leading entry” refers to the first nonzero entry of a
row when scanning the row from left to right.
Definition 4.2.7 An augmented matrix is in echelon form if
1. All nonzero rows are above any rows of zeros.
2. Each leading entry of a row is in a column to the right of the leading entries of any rows above
it.
How do you know when to stop doing row operations? You might stop when you have obtained
an echelon form as described above, but you certainly should stop doing row operations if you have
gotten a matrix in row reduced echelon form described next.
Definition 4.2.8 An augmented matrix is in row reduced echelon form if
1. All nonzero rows are above any rows of zeros.
2. Each leading entry of a row is in a column to the right of the leading entries of any rows above
it.
3. All entries in a column above and below a leading entry are zero.
4. Each leading entry is a 1, the only nonzero entry in its column.
Example 4.2.9 Here are some matrices which are in row reduced echelon form.

1 0
 0 0


 0 0
0 0
0
1
0
0
5
2
0
0
8
7
0
0
0
0
1
0



 
 
,
 


1
0
0
0
0
Example 4.2.10 Here are matrices in echelon form which
which are in echelon form.



1
1 0 6 5 8 2

0
 0 0 2 2 7 3  

 

0

,
 0 0 0 0 0 1  

 0
0 0 0 0 0 0
0
0
1
0
0
0
0
0
1
0
0
0
0
0
1
0




.



are not in row reduced echelon form but
3
2
0
0
0
5
0
3
0
0
4
7
0
1
0








4.2. SYSTEMS OF EQUATIONS, ALGEBRAIC PROCEDURES
Example 4.2.11 Here are

0

 1

 0


 0
0
some matrices which are not in echelon


0 0 0

0 2
 
2 3 3 
1 2 3
 1 5
 
 
1 0 2 
 ,  2 4 −6  , 
 7 5

0 0 1 
4 0 7
0 0
0 0 0
49
form.
3
0
0
1
3
2
1
0



.

Definition 4.2.12 A pivot position in a matrix is the location of a leading entry in an echelon
form resulting from the application of row operations to the matrix. A pivot column is a column
that contains a pivot position.
For example consider the following.
Example 4.2.13 Suppose

1

A= 3
4
2
2
4
3
1
4

4

6 
10
Where are the pivot positions and pivot columns?
Replace the second row by −3 times the first

1 2

 0 −4
4 4
added to the second. This yields

3
4

−8 −6  .
4
10
This is not in reduced echelon form so replace the bottom row by −4 times the top row added to
the bottom. This yields


1 2
3
4


 0 −4 −8 −6  .
0 −4 −8 −6
This is still not in reduced echelon form. Replace the bottom row by −1 times the middle row added
to the bottom. This yields


1 2
3
4


 0 −4 −8 −6 
0 0
0
0
which is in echelon form, although not in reduced echelon form. Therefore, the pivot positions in
the original matrix are the locations corresponding to the first row and first column and the second
row and second columns as shown in the following:


1
2 3 4


2 1 6 
 3
4
4 4 10
Thus the pivot columns in the matrix are the first two columns.
The following is the algorithm for obtaining a matrix which is in row reduced echelon form.
Algorithm 4.2.14
This algorithm tells how to start with a matrix and do row operations on it in such a way as to
end up with a matrix in row reduced echelon form.
50
CHAPTER 4. SYSTEMS OF EQUATIONS
1. Find the first nonzero column from the left. This is the first pivot column. The position at
the top of the first pivot column is the first pivot position. Switch rows if necessary to place
a nonzero number in the first pivot position.
2. Use row operations to zero out the entries below the first pivot position.
3. Ignore the row containing the most recent pivot position identified and the rows above it.
Repeat steps 1 and 2 to the remaining sub-matrix, the rectangular array of numbers obtained
from the original matrix by deleting the rows you just ignored. Repeat the process until there
are no more rows to modify. The matrix will then be in echelon form.
4. Moving from right to left, use the nonzero elements in the pivot positions to zero out the
elements in the pivot columns which are above the pivots.
5. Divide each nonzero row by the value of the leading entry. The result will be a matrix in row
reduced echelon form.
This row reduction procedure applies to both augmented matrices and non augmented matrices.
There is nothing special about the augmented column with respect to the row reduction procedure.
Example 4.2.15 Here is a matrix.








0
0
0
0
0
0
1
0
0
0
2
1
1
0
0
3
4
2
0
2
2
3
2
0
1








Do row reductions till you obtain a matrix in echelon form. Then complete the process by producing
one in row reduced echelon form.
The pivot column is the second. Hence the pivot position is the one in the first row and second
column. Switch the first two rows to obtain a nonzero entry in this pivot position.


0 1 1 4 3


 0 0 2 3 2 


 0 0 1 2 2 




 0 0 0 0 0 
0 0 0 2 1
Step two is not necessary because all the entries below the first pivot position in the resulting matrix
are zero. Now ignore the top row and the columns to the left of this first pivot position. Thus you
apply the same operations to the smaller matrix





2
1
0
0
3
2
0
2
2
2
0
1



.

The next pivot column is the third corresponding to the first in this smaller matrix and the second
pivot position is therefore, the one which is in the second row and third column. In this case it is
not necessary to switch any rows to place a nonzero entry in this position because there is already a
4.2. SYSTEMS OF EQUATIONS, ALGEBRAIC PROCEDURES
51
nonzero entry there. Multiply the third row of the original matrix by −2 and then add the second
row to it. This yields


0 1 1 4
3


 0 0 2 3
2 


 0 0 0 −1 −2  .




0 
 0 0 0 0
0 0 0 2
1
The next matrix the steps in the algorithm are applied to is


−1 −2


0 .
 0
2
1
The first pivot column is the first column in this case and no switching of rows is necessary because
there is a nonzero entry in the first pivot position. Therefore, the algorithm yields for the next step


0 1 1 4
3


 0 0 2 3
2 


 0 0 0 −1 −2  .




0 
 0 0 0 0
0 0 0 0 −3
Now the algorithm will be applied to the matrix
(
0
−3
)
There is only one column and it is nonzero so this single column is the pivot column. Therefore, the
algorithm yields the following matrix for the echelon form.


0 1 1 4
3


 0 0 2 3
2 


 0 0 0 −1 −2  .




 0 0 0 0 −3 
0 0 0 0
0
To complete placing the matrix in reduced echelon form, multiply the third row by 3 and add −2
times the fourth row to it. This yields


0 1 1 4
3


 0 0 2 3
2 


 0 0 0 −3 0 




 0 0 0 0 −3 
0 0 0 0
0
Next multiply the second row by 3 and take
fourth row to the first.

0 1

 0 0

 0 0


 0 0
0 0
2 times the fourth row and add to it. Then add the

1 4
0

6 9
0 

0 −3 0 
.

0 0 −3 
0 0
0
52
CHAPTER 4. SYSTEMS OF EQUATIONS
Next work on the fourth column in the same way.

0 3 3

 0 0 6

 0 0 0


 0 0 0
0 0 0
Take −1/2 times the second row and add

0 3

 0 0

 0 0


 0 0
0 0
0
0
−3
0
0
0
0
0
−3
0








to the first.
0
6
0
0
0
0
0
0
0
−3 0
0 −3
0
0




.



Finally, divide by the value of the leading entries in the nonzero rows.


0 1 0 0 0


 0 0 1 0 0 


 0 0 0 1 0 .




 0 0 0 0 1 
0 0 0 0 0
The above algorithm is the way a computer would obtain a reduced echelon form for a given
matrix. It is not necessary for you to pretend you are a computer but if you like to do so, the
algorithm described above will work. The main idea is to do row operations in such a way as to end
up with a matrix in echelon form or row reduced echelon form because when this has been done, the
resulting augmented matrix will allow you to describe the solutions to the linear system of equations
in a meaningful way. When you do row operations until you obtain row reduced echelon form, the
process is called the Gauss Jordan method. Otherwise, it is called Gauss elimination.
Example 4.2.16 Give the complete solution to the system of equations, 5x + 10y − 7z = −2,
2x + 4y − 3z = −1, and 3x + 6y + 5z = 9.
The augmented matrix for this system is

2 4 −3

 5 10 −7
3 6
5

−1

−2 
9
Multiply the second row by 2, the first row by 5, and then take (−1) times the first row and add to
the second. Then multiply the first row by 1/5. This yields


2 4 −3 −1


1 
 0 0 1
3 6 5
9
Now, combining some row operations, take (−3) times the first row and add this to 2 times the last
row and replace the last row with this. This yields.


2 4 −3 −1


1 .
 0 0 1
0 0 1
21
4.2. SYSTEMS OF EQUATIONS, ALGEBRAIC PROCEDURES
One more row operation, taking (−1) times

2

 0
0
53
the second row and adding to the bottom yields.

4 −3 −1

0 1
1 .
0 0
20
This is impossible because the last row indicates the need for a solution to the equation
0x + 0y + 0z = 20
and there is no such thing because 0 ̸= 20. This shows there is no solution to the three given
equations. When this happens, the system is called inconsistent. In this case it is very easy to
describe the solution set. The system has no solution.
Here is another example based on the use of row operations.
Example 4.2.17 Give the complete solution to the system of equations, 3x−y −5z = 9, y −10z = 0,
and −2x + y = −6.
The augmented matrix of this system is

3 −1 −5

1 −10
 0
−2 1
0
Replace the last row with 2 times the top row added to
operations. This gives

3 −1 −5

 0 1 −10
0 1 −10

9

0 
−6
3 times the bottom row combining two row

9

0 .
0
The entry, 3 in this sequence of row operations is called the pivot. It is used to create zeros in the
other places of the column. Next take −1 times the middle row and add to the bottom. Here the 1
in the second row is the pivot.


3 −1 −5 9


 0 1 −10 0 
0 0
0
0
Take the middle row and add to the top and then divide the top row which results by 3.


1 0 −5 3


 0 1 −10 0  .
0 0
0
0
This is in reduced echelon form. The equations corresponding to this reduced echelon form are
y = 10z and x = 3 + 5z. Apparently z can equal any number. Lets call this number t. 3 Therefore,
the solution set of this system is x = 3 + 5t, y = 10t, and z = t where t is completely arbitrary. The
system has an infinite set of solutions which are given in the above simple way. This is what it is
all about, finding the solutions to the system.
There is some terminology connected to this which is useful. Recall how each column corresponds
to a variable in the original system of equations. The variables corresponding to a pivot column are
called basic variables. The other variables are called free variables. In Example 4.2.17 there
was one free variable, z, and two basic variables, x and y. In describing the solution to the system
of equations, the free variables are assigned a parameter. In Example 4.2.17 this parameter was t.
Sometimes there are many free variables and in these cases, you need to use many parameters. Here
is another example.
3 In
this context t is called a parameter.
54
CHAPTER 4. SYSTEMS OF EQUATIONS
Example 4.2.18 Find the solution to the system
x + 2y − z + w = 3
x+y−z+w =1
x + 3y − z + w = 5
The augmented matrix is

1

 1
1
2
1
3
−1
−1
−1
Take −1 times the first row and add to the second.
the third. This yields

1 2 −1

 0 −1 0
0 1
0
Now add the second row to the bottom row

1 2

 0 −1
0 0
1
1
1

3

1 .
5
Then take −1 times the first row and add to

1 3

0 −2 
0 2

−1 1 3

0 0 −2 
0 0 0
(4.9)
This matrix is in echelon form and you see the basic variables are x and y while the free variables
are z and w. Assign s to z and t to w. Then the second row yields the equation, y = 2 while the
top equation yields the equation, x + 2y − s + t = 3 and so since y = 2, this gives x + 4 − s + t = 3
showing that x = −1 + s − t, y = 2, z = s, and w = t. It is customary to write this in the form





x
y
z
w


 
 
=
 
−1 + s − t
2
s
t



.

(4.10)
This is another example of a system which has an infinite solution set but this time the solution
set depends on two parameters, not one. Most people find it less confusing in the case of an infinite
solution set to first place the augmented matrix in row reduced echelon form rather than just echelon
form before seeking to write down the description of the solution. In the above, this means we don’t
stop with the echelon form 4.9. Instead we first place it in reduced echelon form as follows.


1 0 −1 1 −1


 0 1 0 0 2 .
0 0 0 0 0
Then the solution is y = 2 from the second row and x = −1 + z − w from the first. Thus letting
z = s and w = t, the solution is given in 4.10.
The number of free variables is always equal to the number of different parameters used to
describe the solution. If there are no free variables, then either there is no solution as in the case
where row operations yield an echelon form like


1 2 3


 0 4 −2 
0 0 1
4.2. SYSTEMS OF EQUATIONS, ALGEBRAIC PROCEDURES
55
or there is a unique solution as in the case where row operations yield an echelon form like


1 2 2 3


 0 4 3 −2  .
0 0 4 1
Also, sometimes there are free variables and

1

 0
0
no solution as in the following:

2 2 3

4 3 −2  .
0 0 1
There are a lot of cases to consider but it is not necessary to make a major production of this.
Do row operations till you obtain a matrix in echelon form or reduced echelon form and determine
whether there is a solution. If there is, see if there are free variables. In this case, there will be
infinitely many solutions. Find them by assigning different parameters to the free variables and
obtain the solution. If there are no free variables, then there will be a unique solution which is easily
determined once the augmented matrix is in echelon or row reduced echelon form. In every case,
the process yields a straightforward way to describe the solutions to the linear system. As indicated
above, you are probably less likely to become confused if you place the augmented matrix in row
reduced echelon form rather than just echelon form.
In summary,
Definition 4.2.19 A system of linear equations is a list of equations,
a11 x1 + a12 x2 + · · · + a1n xn = b1
a21 x1 + a22 x2 + · · · + a2n xn = b2
..
.
am1 x1 + am2 x2 + · · · + amn xn = bm
where aij are numbers, and bj is a number. The above is a system of m equations in the n variables,
x1 , x2 · · · , xn . Nothing is said about the relative size of m and n. Written more simply in terms of
summation notation, the above can be written in the form
n
∑
aij xj = fi , i = 1, 2, 3, · · · , m
j=1
It is desired to find (x1 , · · · , xn ) solving each of the equations listed.
As illustrated above, such a system of linear equations may have a unique solution, no solution,
or infinitely many solutions and these are the only three cases which can occur for any linear system.
Furthermore, you do exactly the same things to solve any linear system. You write the augmented
matrix and do row operations until you get a simpler system in which it is possible to see the solution,
usually obtaining a matrix in echelon or reduced echelon form. All is based on the observation that
the row operations do not change the solution set. You can have more equations than variables,
fewer equations than variables, etc. It doesn’t matter. You always set up the augmented matrix and
go to work on it.
Definition 4.2.20 A system of linear equations is called consistent if there exists a solution. It
is called inconsistent if there is no solution.
These are reasonable words to describe the situations of having or not having a solution. If you
think of each equation as a condition which must be satisfied by the variables, consistent would
mean there is some choice of variables which can satisfy all the conditions. Inconsistent would mean
there is no choice of the variables which can satisfy each of the conditions.
56
CHAPTER 4. SYSTEMS OF EQUATIONS
4.2.3
Balancing Chemical Reactions
Consider the chemical reaction
SnO2 + H2 → Sn + H2 O
Here the elements involved are tin Sn oxygen O and Hydrogen H. Some chemical reaction happens
and you end up with some tin and some water. The question is, how much do you start with and
how much do you end up with.
The balance of mass requires that you have the same number of oxygen, tin, and hydrogen on
both sides of the reaction. However, this does not happen in the above. For example, there are two
oxygen atoms on the left and only one on the right. The problem is to find numbers x, y, z, w such
that
xSnO2 + yH2 → zSn + wH2 O
and both sides have the same number of atoms of the various substances. You can do this in a
systematic way by setting up a system of equations which will require that this take place. Thus
you need
Sn :
x=z
O : 2x = w
H : 2y = 2w
The augmented matrix for this system of equations

1 0 −1

 2 0 0
0 2 0
Row reducing this yields

1 0 0

 0 1 0
0 0 1
is then

0 0

−1 0 
−2 0
− 21
−1
− 12

0

0 
0
Thus you could let w = 2 and this would yield x = 1, y = 2, and z = 1. Hence, the description of
the reaction which has the same numbers of atoms on both sides would be
SnO2 + 2H2 → Sn + 2H2 O
You see that this preserves the total number of atoms and so the chemical equation is balanced.
Here is another example
Example 4.2.21 Potassium is denoted by K, oxygen by O, phosphorus by P and hydrogen by H.
The reaction is
KOH + H3 P O4 → K3 P O4 + H2 O
balance this equation.
You need to have
xKOH + yH3 P O4 → zK3 P O4 + wH2 O
Equations which preserve the total number of atoms of each element on both sides of the equation
are
K:
x = 3z
O : x + 4y = 4z + w
H:
x + 3y = 2w
P :
y=z
4.2. SYSTEMS OF EQUATIONS, ALGEBRAIC PROCEDURES
57
The augmented matrix for this system is





1
1
1
0
0
4
3
1
−3 0 0
−4 −1 0
0 −2 0
−1 0 0





Then the row reduced echelon form is





1
0
0
0
0
1
0
0
0
0
1
0
−1
− 13
− 13
0
0
0
0
0





You could let w = 3 and this yields x = 3, y = 1, z = 1. Then the balanced equation is
3KOH + 1H3 P O4 → 1K3 P O4 + 3H2 O
Note that this results in the same number of atoms on both sides.
Of course these numbers you are finding would typically be the number of moles of the molecules
on each side. Thus three moles of KOH added to one mole of H3 P O4 yields one mole of K3 P O4
and three moles of H2 O, water.
Note that in this example, you have a row of zeros. This means that some of the information in
computing the appropriate numbers was redundant. If this can happen with a single reaction, think
how much more it could happen if you were dealing with hundreds of reactions. This aspect of the
problem can be understood later in terms of the rank of a matrix.
For an introduction to the chemical considerations mentioned here, there is a nice site on the web
http://chemistry.about.com/od/chemicalreactions/a/reactiontypes.htm where there is a sample test
and examples of chemical reactions. For names of the various elements symbolized by the various
letters, you can go to the site http://chemistry.about.com/od/elementfacts/a/elementlist.htm. Of
course these things are in standard chemistry books, but if you have not seen much chemistry, these
sites give a nice introduction to these concepts.
4.2.4
Dimensionless Variables∗
This section shows how solving systems of equations can be used to determine appropriate dimensionless variables. It is only an introduction to this topic. I got this example from [7]. This considers
a specific example of a simple airplane wing shown below. We assume for simplicity that it is just
a flat plane at an angle to the wind which is blowing against it with speed V as shown.
V
θ
A
B
The angle is called the angle of incidence, B is the span of the wing and A is called the chord.
Denote by l the lift. Then this should depend on various quantities like θ, V, B, A and so forth. Here
58
CHAPTER 4. SYSTEMS OF EQUATIONS
is a table which indicates various quantities on which it is reasonable to expect l to depend.
Variable
chord
span
angle incidence
speed of wind
speed of sound
density of air
viscosity
lift
Symbol
A
B
θ
V
V0
ρ
µ
l
Units
m
m
m0 kg 0 sec0
m sec−1
m sec−1
kgm−3
kg sec−1 m−1
kg sec−2 m
Here m denotes meters, sec refers to seconds and kg refers to kilograms. All of these are likely
familiar except for µ. One can simply decree that these are the dimensions of something called
viscosity but it might be better to consider this a little more.
Viscosity is a measure of how much internal friction is experienced when the fluid moves. It is
roughly a measure of how “sticky” the fluid is. Consider a piece of area parallel to the direction of
motion of the fluid. To say that the viscosity is large is to say that the tangential force applied to
this area must be large in order to achieve a given change in speed of the fluid in a direction normal
to the tangential force. Thus
µ (area) (velocity gradient) = tangential force.
Hence
(units on µ) m2
Thus the units on µ are
( m )
= kg sec−2 m
sec m
kg sec−1 m−1
as claimed above.
Then one would think that you would want
l = f (A, B, θ, V, V0 , ρ, µ)
However, this is very cumbersome because it depends on seven variables. Also, it doesn’t make very
good sense. It is likely that without much care, a change in the units such as going from meters to
feet would result in an incorrect value for l. The way to get around this problem is to look for l as
a function of dimensionless variables multiplied by something which has units of force. It is helpful
because first of all, you will likely have fewer independent variables and secondly, you could expect
the formula to hold independent of the way of specifying length, mass and so forth. One looks for
l = f (g1 , · · · , gk ) ρV 2 AB
where the units on ρV 2 AB are
kg × m
kg ( m )2 2
m =
3
m sec
sec2
which are the units of force. Each of these gi is of the form
Ax1 B x2 θx3 V x4 V0x5 ρx6 µx7
(4.11)
and each gi is independent of the dimensions. That is, this expression must not depend on meters,
kilograms, seconds, etc. Thus, placing in the units for each of these quantities, one needs
(
)(
)(
)x6 (
)x7
= m0 kg 0 sec0
mx1 mx2 mx4 sec−x4 mx5 sec−x5 kgm−3
kg sec−1 m−1
4.2. SYSTEMS OF EQUATIONS, ALGEBRAIC PROCEDURES
59
Notice that there are no units on θ because it is just the radian measure of an angle. Hence its
dimensions consist of length divided by length, thus it is dimensionless. Then this leads to the
following equations for the xi .
m:
sec :
kg :
x1 + x2 + x4 + x5 − 3x6 − x7 = 0
−x4 − x5 − x7 = 0
x6 + x7 = 0
Then the augmented matrix for this system

1 1 0

0
0 0

0 0 0
The row reduced echelon form is then

1 1

 0 0
0 0
of equations is
1
1
0
0 0
0 1
0 0
1
1
0

−3 −1 0

0
1 0 
1
1 0
0 0
1 0
0 1
1
1
1

0

0 
0
and so the solutions are of the form
x1 = −x2 − x7 , x3 = x3 , x4 = −x5 − x7 , x6 = −x7
Thus, in terms of vectors, the solution is

 

x1
−x2 − x7
 x  

x2
 2  


 

 x3  

x3

 

 x4  =  −x5 − x7 

 


 

x5
 x5  


 

 x6  

−x7
x7
x7
Thus the free variables are x2 , x3 , x5 , x7 . By assigning values to these, we can obtain dimensionless
variables by placing the values obtained for the xi in the formula 4.11. For example, let x2 = 1 and
all the rest of the free variables are 0. This yields
x1 = −1, x2 = 1, x3 = 0, x4 = 0, x5 = 0, x6 = 0, x7 = 0.
The dimensionless variable is then A−1 B 1 . This is the ratio between the span and the chord. It is
called the aspect ratio, denoted as AR. Next let x3 = 1 and all others equal zero. This gives for a
dimensionless quantity the angle θ. Next let x5 = 1 and all others equal zero. This gives
x1 = 0, x2 = 0, x3 = 0, x4 = −1, x5 = 1, x6 = 0, x7 = 0.
Then the dimensionless variable is V −1 V01 . However, it is written as V /V0 . This is called the Mach
number M. Finally, let x7 = 1 and all the other free variables equal 0. Then
x1 = −1, x2 = 0, x3 = 0, x4 = −1, x5 = 0, x6 = −1, x7 = 1
then the dimensionless variable which results from this is A−1 V −1 ρ−1 µ. It is customary to write it
as Re = (AV ρ) /µ. This one is called the Reynolds number. It is the one which involves viscosity.
Thus we would look for
l = f (Re, AR, θ, M) kg × m/ sec2
This is quite interesting because it is easy to vary Re by simply adusting the velocity or A but it is
hard to vary things like µ or ρ. Note that all the quantities are easy to adjust. Now this could be
used, along with wind tunnel experiments to get a formula for the lift which would be reasonable.
Obviously, you could consider more variables and more complicated situations in the same way.
60
CHAPTER 4. SYSTEMS OF EQUATIONS
4.3
Exercises
1. Find the point (x1 , y1 ) which lies on both lines, x + 3y = 1 and 4x − y = 3.
2. Solve Problem 1 graphically. That is, graph each line and see where they intersect.
3. Find the point of intersection of the two lines 3x + y = 3 and x + 2y = 1.
4. Solve Problem 3 graphically. That is, graph each line and see where they intersect.
5. Do the three lines, x+2y = 1, 2x−y = 1, and 4x+3y = 3 have a common point of intersection?
If so, find the point and if not, tell why they don’t have such a common point of intersection.
6. Do the three planes, x + y − 3z = 2, 2x + y + z = 1, and 3x + 2y − 2z = 0 have a common
point of intersection? If so, find one and if not, tell why there is no such point.
7. You have a system of k equations in two variables, k ≥ 2. Explain the geometric significance
of
(a) No solution.
(b) A unique solution.
(c) An infinite number of solutions.
8. Here is an augmented matrix in which ∗ denotes an arbitrary number and denotes a nonzero
number. Determine whether the given augmented matrix is consistent. If consistent, is the
solution unique?


∗ ∗ ∗ ∗ | ∗
 0 ∗ ∗ 0 | ∗ 




 0 0 ∗ ∗ | ∗ 
0
0
0
0 |
∗
9. Here is an augmented matrix in which ∗ denotes an arbitrary number and denotes a nonzero
number. Determine whether the given augmented matrix is consistent. If consistent, is the
solution unique?


∗ ∗ | ∗


 0 ∗ | ∗ 
0 0 | ∗
10. Here is an augmented matrix in which ∗ denotes an arbitrary number and denotes a nonzero
number. Determine whether the given augmented matrix is consistent. If consistent, is the
solution unique?


∗ ∗ ∗ ∗ | ∗
 0 0 ∗ 0 | ∗ 




 0 0 0 ∗ | ∗ 
0 0 0 0 | ∗
11. Here is an augmented matrix in which ∗ denotes an arbitrary number and denotes a nonzero
number. Determine whether the given augmented matrix is consistent. If consistent, is the
solution unique?


∗ ∗ ∗ ∗ | ∗
 0 ∗ ∗ 0 | ∗ 




 0 0 0 0 | 0 
0
0
0
0
∗
|
4.3. EXERCISES
61
12. Suppose a system of equations has fewer equations than variables. Must such a system be
consistent? If so, explain why and if not, give an example which is not consistent.
13. If a system of equations has more equations than variables, can it have a solution? If so, give
an example and if not, tell why not.
14. Find h such that
(
2 h | 4
3 6 | 7
)
is the augmented matrix of an inconsistent matrix.
15. Find h such that
(
1 h | 3
2 4 | 6
)
is the augmented matrix of a consistent matrix.
16. Find h such that
(
1
3
1 | 4
h | 12
)
is the augmented matrix of a consistent matrix.
17. Choose h and k such that the augmented matrix shown has one solution. Then choose h and
k such that the system has no solutions. Finally, choose h and k such that the system has
infinitely many solutions.
(
)
1 h | 2
.
2 4 | k
18. Choose h and k such that the augmented matrix shown has one solution. Then choose h and
k such that the system has no solutions. Finally, choose h and k such that the system has
infinitely many solutions.
(
)
1 2 | 2
.
2 h | k
19. Determine if the system is consistent. If so, is the solution unique?
x + 2y + z − w = 2
x−y+z+w =1
2x + y − z = 1
4x + 2y + z = 5
20. Determine if the system is consistent. If so, is the solution unique?
x + 2y + z − w = 2
x−y+z+w =0
2x + y − z = 1
4x + 2y + z = 3
21. Find the general solution of the system whose augmented matrix is


1 2 0 | 2


 1 3 4 | 2 .
1 0 2 | 1
62
CHAPTER 4. SYSTEMS OF EQUATIONS
22. Find the general solution of the system whose augmented matrix is


1 2 0 | 2


 2 0 1 | 1 .
3 2 1 | 3
23. Find the general solution of the system whose augmented matrix is
(
)
1 1 0 | 1
.
1 0 4 | 2
24. Find the general solution of the system whose augmented matrix is





1
0
1
1
0
1
2
0
2
0
0
1
1
1
0
0
1
2
1
2
|
|
|
|
2
1
3
2



.

25. Find the general solution of the system whose augmented matrix is





1 0 2 1 1
0 1 0 1 2
0 2 0 0 1
1 −1 2 2 2
|
|
|
|
2
1
3
0



.

26. Give the complete solution to the system of equations, 7x + 14y + 15z = 22, 2x + 4y + 3z = 5,
and 3x + 6y + 10z = 13.
27. Give the complete solution to the system of equations, 3x − y + 4z = 6, y + 8z = 0, and
−2x + y = −4.
28. Give the complete solution to the system of equations, 9x−2y+4z = −17, 13x−3y+6z = −25,
and −2x − z = 3.
29. Give the complete solution to the system of equations, 65x+84y+16z = 546, 81x+105y+20z =
682, and 84x + 110y + 21z = 713.
30. Give the complete solution to the system of equations, 8x + 2y + 3z = −3, 8x + 3y + 3z = −1,
and 4x + y + 3z = −9.
31. Give the complete solution to the system of equations, −8x + 2y + 5z = 18, −8x + 3y + 5z = 13,
and −4x + y + 5z = 19.
32. Give the complete solution to the system of equations, 3x − y − 2z = 3, y − 4z = 0, and
−2x + y = −2.
33. Give the complete solution to the system of equations, −9x + 15y = 66, −11x + 18y = 79
,−x + y = 4, and z = 3.
34. Give the complete solution to the system of equations, −19x+8y = −108, −71x+30y = −404,
−2x + y = −12, 4x + z = 14.
35. Consider the system −5x + 2y − z = 0 and −5x − 2y − z = 0. Both equations equal zero and so
−5x + 2y − z = −5x − 2y − z which is equivalent to y = 0. Thus x and z can equal anything.
But when x = 1, z = −4, and y = 0 are plugged in to the equations, it doesn’t work. Why?
4.3. EXERCISES
63
36. Four times the weight of Gaston is 150 pounds more than the weight of Ichabod. Four times
the weight of Ichabod is 660 pounds less than seventeen times the weight of Gaston. Four
times the weight of Gaston plus the weight of Siegfried equals 290 pounds. Brunhilde would
balance all three of the others. Find the weights of the four sisters.
37. The steady state temperature, u in a plate solves Laplace’s equation, ∆u = 0. One way to
approximate the solution which is often used is to divide the plate into a square mesh and
require the temperature at each node to equal the average of the temperature at the four
adjacent nodes. This procedure is justified by the mean value property of harmonic functions.
In the following picture, the numbers represent the observed temperature at the indicated
nodes. Your task is to find the temperature at the interior nodes, indicated by x, y, z, and w.
One of the equations is z = 41 (10 + 0 + w + x).
30 30
20
y
w
0
20
x
z
0
10 10
38. Consider the following diagram of four circuits.
3Ω
20 volts 1 Ω
I2
I3
5Ω
5 volts
2Ω
1Ω
6Ω
I1
10 volts
I4
1Ω
4Ω
3Ω
2Ω
Those jagged places denote resistors and the numbers next to them give their resistance in
ohms, written as Ω. The breaks in the lines having one short line and one long line denote a
voltage source which causes the current to flow in the direction which goes from the longer of
the two lines toward the shorter along the unbroken part of the circuit. The current in amps
in the four circuits is denoted by I1 , I2 , I3 , I4 and it is understood that the motion is in the
counter clockwise direction. If Ik ends up being negative, then it just means the current flows
in the clockwise direction. Then Kirchhoff’s law states that
The sum of the resistance times the amps in the counter clockwise direction around a loop
equals the sum of the voltage sources in the same direction around the loop.
In the above diagram, the top left circuit should give the equation
2I2 − 2I1 + 5I2 − 5I3 + 3I2 = 5
For the circuit on the lower left, you should have
4I1 + I1 − I4 + 2I1 − 2I2 = −10
Write equations for each of the other two circuits and then give a solution to the resulting
system of equations. You might use a computer algebra system to find the solution. It might
be more convenient than doing it by hand.
64
CHAPTER 4. SYSTEMS OF EQUATIONS
39. Consider the following diagram of three circuits.
3Ω
12 volts 7 Ω
I1
10 volts
I2
5Ω
2Ω
3Ω
1Ω
I3
2Ω
4Ω
4Ω
Those jagged places denote resistors and the numbers next to them give their resistance in
ohms, written as Ω. The breaks in the lines having one short line and one long line denote a
voltage source which causes the current to flow in the direction which goes from the longer of
the two lines toward the shorter along the unbroken part of the circuit. The current in amps in
the four circuits is denoted by I1 , I2 , I3 and it is understood that the motion is in the counter
clockwise direction. If Ik ends up being negative, then it just means the current flows in the
clockwise direction. Then Kirchhoff’s law states that
The sum of the resistance times the amps in the counter clockwise direction around a loop
equals the sum of the voltage sources in the same direction around the loop. Find I1 , I2 , I3 .
40. Here are some chemical reactions. Balance them.
(a) KN O3 + H2 CO3 → K2 CO3 + HN O3
(b) Ba3 N2 + H2 O → Ba (OH)2 + N H3
(c) CaCl2 + N a3 P O4 → Ca3 (P O4 )2 + N aCl
41. In the section on dimensionless variables 57 it was observed that ρV 2 AB has the units of
force. Describe a systematic way to obtain such combinations of the variables which will yield
something which has the units of force.
Chapter 5
Matrices
5.1
5.1.1
Matrix Arithmetic
Addition And Scalar Multiplication Of Matrices
You have now solved systems of equations by writing them in terms of an augmented matrix and then
doing row operations on this augmented matrix. It turns out such rectangular arrays of numbers
are important from many other different points of view. Numbers are also called scalars. In this
book, numbers will generally be either real or complex numbers. I will refer to the set of numbers as
F sometimes when it is not important to worry about whether the number is real or complex. Thus
F can be either the real numbers, R or the complex numbers C. However, most of the algebraic
considerations hold for more general fields of scalars.
A matrix is a rectangular array of numbers. Several of them are referred to as matrices. For
example, here is a matrix.


1 2 3 4


 5 2 8 7 
6 −9 1 2
The size or dimension of a matrix is defined as m × n where m is the number of rows and n is
the number of columns. The above matrix is a 3 × 4 matrix because there are three rows and four
columns.
  The first row is (1 2 3 4) , the second row is (5 2 8 7) and so forth. The first column is
1
 
 5  . When specifying the size of a matrix, you always list the number of rows before the number
6
of columns. Also, you can remember the columns are like columns in a Greek temple. They stand
upright while the rows just lie there like rows made by a tractor in a plowed field. Elements of the
matrix are identified according to position in the matrix. For example, 8 is in position 2, 3 because
it is in the second row and the third column. You might remember that you always list the rows
before the columns by using the phrase Rowman Catholic. The symbol, (aij ) refers to a matrix.
The entry in the ith row and the j th column of this matrix is denoted by aij . Using this notation
on the above matrix, a23 = 8, a32 = −9, a12 = 2, etc.
There are various operations which are done on matrices. Matrices can be added multiplied by
a scalar, and multiplied by other matrices. To illustrate scalar multiplication, consider the following
example in which a matrix is being multiplied by the scalar 3.

 

1 2 3 4
3
6
9 12

 

3  5 2 8 7  =  15
6
24 21  .
6 −9 1 2
18 −27 3
6
65
66
CHAPTER 5. MATRICES
The new matrix is obtained by multiplying every entry of the original matrix by the given scalar. If
A is an m × n matrix, −A is defined to equal (−1) A.
Two matrices must be the same size to be added. The sum of two matrices is a matrix which is
obtained by adding the corresponding entries. Thus

 
 

0
6
−1 4
1 2

 
 

8  =  5 12  .
 3 4 + 2
11 −2
6 −4
5 2
Two matrices are equal exactly when they are the same size and the corresponding entries are
identical. Thus


(
)
0 0
0 0


 0 0  ̸=
0 0
0 0
because they are different sizes. As noted above, you write (cij ) for the matrix C whose ij th entry is
cij . In doing arithmetic with matrices you must define what happens in terms of the cij sometimes
called the entries of the matrix or the components of the matrix.
The above discussion stated for general matrices is given in the following definition.
Definition 5.1.1 (Scalar Multiplication) If A = (aij ) and k is a scalar, then kA = (kaij ) .
(
) (
)
2 0
14
0
Example 5.1.2 7
=
.
1 −4
7 −28
Definition 5.1.3 (Addition) If A = (aij ) and B = (bij ) are two m × n matrices. Then A + B = C
where
C = (cij )
for cij = aij + bij .
Example 5.1.4
(
1
1
2
0
3
4
)
(
+
5 2 3
−6 2 1
)
(
=
6 4 6
−5 2 5
)
To save on notation, we will often use Aij to refer to the ij th entry of the matrix A.
Definition 5.1.5 (The zero matrix) The m × n zero matrix is the m × n matrix having every entry
equal to zero. It is denoted by 0.
(
)
0 0 0
Example 5.1.6 The 2 × 3 zero matrix is
.
0 0 0
Note there are 2 × 3 zero matrices, 3 × 4 zero matrices, etc. In fact there is a zero matrix for
every size.
Definition 5.1.7 (Equality of matrices) Let A and B be two matrices. Then A = B means that
the two matrices are of the same size and for A = (aij ) and B = (bij ) , aij = bij for all 1 ≤ i ≤ m
and 1 ≤ j ≤ n.
The following properties of matrices can be easily verified. You should do so. These properties
are called the vector space axioms.
• Commutative Law Of Addition.
A + B = B + A,
(5.1)
5.1. MATRIX ARITHMETIC
67
• Associative Law for Addition.
(A + B) + C = A + (B + C) ,
(5.2)
• Existence of an Additive Identity
A + 0 = A,
(5.3)
A + (−A) = 0,
(5.4)
• Existence of an Additive Inverse
Also for α, β scalars, the following additional properties hold.
• Distributive law over Matrix Addition.
α (A + B) = αA + αB,
(5.5)
• Distributive law over Scalar Addition
(α + β) A = αA + βA,
(5.6)
• Associative law for Scalar Multiplication
α (βA) = αβ (A) ,
(5.7)
1A = A.
(5.8)
• Rule for Multiplication by 1.
As an example, consider the Commutative Law of Addition. Let A + B = C and B + A = D.
Why is D = C?
Cij = Aij + Bij = Bij + Aij = Dij .
Therefore, C = D because the ij th entries are the same. Note that the conclusion follows from the
commutative law of addition of numbers.
5.1.2
Multiplication Of Matrices
Definition 5.1.8 Matrices which are n × 1 or 1 × n
bold letter. Thus the n × 1 matrix

x1
 .
.
x =
 .
xn
is also called a column vector. The 1 × n matrix
(
x1 · · ·
are called vectors and are often denoted by a




)
xn
is called a row vector.
Although the following description of matrix multiplication may seem strange, it is in fact the
most important and useful of the matrix operations. To begin with consider the case where a matrix
is multiplied by a column vector. First consider a special case.


(
)
7
1 2 3 

 8  =?
4 5 6
9
68
CHAPTER 5. MATRICES
By definition, this equals
(
7
In more general terms,
(
a11
a21
a12
a22
a13
a23
)
1
4
)
(
+8
2
5
)
(
+9
3
6
)
(
=
50
122
)

(
)
)
(
)
(
x1
a13
a12
a11


+ x3
+ x2
 x2  = x1
a23
a22
a21
x3
(
)
a11 x1 + a12 x2 + a13 x3
=
.
a21 x1 + a22 x2 + a23 x3

Thus you take x1 times the first column, add to x2 times the second column, and finally x3 times the
third column. The above sum is called a linear combination of the given column vectors. These
will be discussed more later. In general, a linear combination of vectors is just a sum consisting of
scalars times vectors. When you multiply a matrix on the left by a vector on the right, the numbers
making up the vector are just the scalars to be used in the linear combination of the columns as
illustrated above.
More generally, here is the definition of how to multiply an (m × n) matrix times a (n × 1) matrix
(column vector).
Definition 5.1.9 Let A = Aij be an m × n matrix and let v be an n × 1 matrix,


v1
 . 
. 
v =
 .  , A = (a1 , · · · , an )
vn
where ai is an m × 1 vector. Then Av, written as
(
a1
···


v1
)

 ..  ,
 . 
vn
an
is the m × 1 column vector which equals the following linear combination of the columns.
v1 a1 + v2 a2 + · · · + vn an ≡
n
∑
v j aj
(5.9)
j=1
If the j th column of A is






A1j
A2j
..
.
Amj






then 5.9 takes the form



A11
A12



A
 21 
 A22
 + v2  .
v1 
.
 . 
 .
 . 
 .
Am1
Am2
∑
n
Thus the ith entry of Av is j=1 Aij vj . Note that
matrix, and produces an m × 1 matrix (vector).






 + · · · + vn 




A1n
A2n
..
.
Amn






multiplication by an m × n matrix takes an n × 1
5.1. MATRIX ARITHMETIC
69
Here is another example.
Example 5.1.10 Compute

1 2 1

 0 2 1
2 1 4


3


−2  

1
1
2
0
1



.

First of all this is of the form (3 × 4) (4 × 1) and so the result should be a (3 × 1) . Note how the
inside numbers cancel. To get the element in the second row and first and only column, compute
4
∑
a2k vk
= a21 v1 + a22 v2 + a23 v3 + a24 v4
k=1
= 0 × 1 + 2 × 2 + 1 × 0 + (−2) × 1 = 2.
You should do the rest of the problem and verify

1 2 1

 0 2 1
2 1 4


1
3
 2

−2  
 0
1
1

 
8

  
 =  2 .

5
The next task is to multiply an m×n matrix times an n×p matrix. Before doing so, the following
may be helpful.
For A and B matrices, in order to form the product, AB the number of columns of A must equal
the number of rows of B.
these must match!
[
n)
(n × p
(m ×
)=m×p
Note the two outside numbers give the size of the product. Remember:
If the two middle numbers don’t match, you can’t multiply the matrices!
Definition 5.1.11 When the number of columns of A equals the number of rows of B the two
matrices are said to be conformable and the product AB is obtained as follows. Let A be an m × n
matrix and let B be an n × p matrix. Then B is of the form
B = (b1 , · · · , bp )
where bk is an n × 1 matrix or column vector. Then the m × p matrix AB is defined as follows:
AB ≡ (Ab1 , · · · , Abp )
where Abk is an m × 1 matrix or column vector which gives the k th column of AB.
Example 5.1.12 Multiply the following.
(
1 2 1
0 2 1
)


1 2 0


 0 3 1 
−2 1 1
(5.10)
70
CHAPTER 5. MATRICES
The first thing you need to check before doing anything else is whether it is possible to do the
multiplication. The first matrix is a 2 × 3 and the second matrix is a 3 × 3. Therefore, is it possible
to multiply these matrices. According to the above discussion it should be a 2 × 3 matrix of the
form


Second column
Third column
First column
z
z
}| 
{
}|
{
}|
{
z

 
 
(
)
(
)
(
)

1
2
0 
 1 2 1 
1 2 1  
1 2 1  


,
,
 3 
 1 

 0 2 1  0 
0 2 1
0 2 1


−2
1
1 

You know how to multiply a matrix times a
columns. Thus

(
)
1
1 2 1 
 0
0 2 1
−2
vector and so you do so to obtain each of the three

(
)
2 0
−1 9 3

.
3 1 =
−2 7 3
1 1
Example 5.1.13 Multiply the following.


(
1 2 0

 1
 0 3 1 
0
−2 1 1
2
2
1
1
)
First check if it is possible. This is of the form (3 × 3) (2 × 3) . The inside numbers do not
match and so you can’t do this multiplication. This means that anything you write will be absolute
nonsense because it is impossible to multiply these matrices in this order. Aren’t they the same
two matrices considered in the previous example? Yes they are. It is just that here they are in a
different order. This shows something you must always remember about matrix multiplication.
Order Matters!
Matrix Multiplication Is Not Commutative!
This is very different than multiplication of numbers!
5.1.3
The ij th Entry Of A Product
It is important to describe matrix multiplication in terms of entries of the matrices. What is the
ij th entry of AB? It would be the ith entry of the j th column of AB. Thus it would be the ith entry
of Abj . Now


B1j
 . 
. 
bj = 
 . 
Bnj
and from the above definition, the ith entry is
n
∑
Aik Bkj .
(5.11)
k=1
In terms of pictures of the matrix, you are doing


A11 A12 · · · A1n
B11


 A21 A22 · · · A2n   B21
 .

..
.. 
 .
  ..
 .
.
.  .
Am1 Am2 · · · Amn
Bn1
B12
B22
..
.
Bn2
···
···
···
B1p
B2p
..
.
Bnp






5.1. MATRIX ARITHMETIC
71
Then as explained above, the j th column is of the

A11 A12 · · ·

 A21 A22 · · ·
 .
..
 .
 .
.
Am1 Am2 · · ·
form
A1n
A2n
..
.
Amn






B1j
B2j
..
.
Bnj






which is a m × 1 matrix or column vector which equals





A1n
A12
A11





A
A
 A2n
 22 
 21 
 .  B1j +  .  B2j + · · · +  .
 .
 . 
 . 
 .
 . 
 . 
Amn
Am2
Am1



 Bnj .


The second entry of this m × 1 matrix is
A21 B1j + A22 B2j + · · · + A2n Bnj =
m
∑
A2k Bkj .
k=1
Similarly, the ith entry of this m × 1 matrix is
Ai1 B1j + Ai2 B2j + · · · + Ain Bnj =
m
∑
Aik Bkj .
k=1
This shows the following definition for matrix multiplication in terms of the ij th entries of the
product coincides with Definition 5.1.11.
Definition 5.1.14 Let A = (Aij ) be an m × n matrix and let B = (Bij ) be an n × p matrix. Then
AB is an m × p matrix and
n
∑
(AB)ij =
Aik Bkj .
(5.12)
k=1
Another way to write this is

(
(AB)ij =
Ai1
Ai2
···
Ain
)




B1j
B2j
..
.
Bnj






Note that to get (AB)ij you involve the ith row of A and the j th column of B. Specifically, the ij th
entry of AB is the dot product of the ith row of A with the j th column of B. This is what the formula
in 5.12 says. (Note that here the dot product does not involve taking conjugates.)


(
)
1 2

 2 3 1
Example 5.1.15 Multiply if possible  3 1 
.
7 6 2
2 6
First check to see if this is possible. It is of the form (3 × 2) (2 × 3) and since the inside numbers
match, the two matrices are conformable and it is possible to do the multiplication. The result
should be a 3 × 3 matrix. The answer is of the form







( )
( )
( )
1 2
1 2
1 2

 2

 3

 1 
, 3 1 
, 3 1 
 3 1 

7
6
2
2 6
2 6
2 6
72
CHAPTER 5. MATRICES
where the commas separate the columns in the

16

 13
46
resulting product. Thus the above product equals

15 5

15 5  ,
42 14
a 3 × 3 matrix as desired. In terms of the ij th entries and the above definition, the entry in the third
row and second column of the product should equal
∑
a3k bk2 = a31 b12 + a32 b22
j
= 2 × 3 + 6 × 6 = 42.
You should try a few more such examples to verify the
works for other entries.


2
1 2


Example 5.1.16 Multiply if possible  3 1   7
2 6
0
above definition in terms of the ij th entries
3
6
0

1

2 .
0
This is not possible because it is of the form (3 × 2) (3 × 3) and the middle numbers don’t match.
In other words the two matrices are not conformable in the indicated order.



2 3 1
1 2



Example 5.1.17 Multiply if possible  7 6 2   3 1  .
0 0 0
2 6
This is possible because in this case it is of the form (3 × 3) (3 × 2) and the middle numbers do
match so the matrices are conformable. When the multiplication is done it equals


13 13


 29 32  .
0
0
Check this and be sure you come up with the same answer.
 
1
(
)
 
Example 5.1.18 Multiply if possible  2  1 2 1 0 .
1
In this case you are trying to do (3 × 1) (1 × 4) . The inside numbers match so you can do it.
Verify
 


1
1 2 1 0
(
)
 


 2  1 2 1 0 = 2 4 2 0 
1
1 2 1 0
5.1.4
Properties Of Matrix Multiplication
As pointed out above, sometimes it is possible to multiply matrices in one order but not in the other
order. What if it makes sense to multiply them in either order? Will the two products be equal
then?
(
)(
)
(
)(
)
1 2
0 1
0 1
1 2
Example 5.1.19 Compare
and
.
3 4
1 0
1 0
3 4
5.1. MATRIX ARITHMETIC
The first product is
73
(
The second product is
(
1
3
2
4
0
1
1
0
)(
)(
0
1
1
0
1 2
3 4
)
(
=
)
(
=
2 1
4 3
3 4
1 2
)
.
)
.
You see these are not equal. Again you cannot conclude that AB = BA for matrix multiplication
even when multiplication is defined in both orders. However, there are some properties which do
hold.
Proposition 5.1.20 If all multiplications and additions make sense, the following hold for matrices,
A, B, C and a, b scalars.
A (aB + bC) = a (AB) + b (AC)
(5.13)
(B + C) A = BA + CA
(5.14)
A (BC) = (AB) C
(5.15)
Proof: Using Definition 5.1.14,
(A (aB + bC))ij =
∑
Aik (aB + bC)kj =
k
=a
∑
∑
Aik (aBkj + bCkj )
k
Aik Bkj + b
k
∑
Aik Ckj = a (AB)ij + b (AC)ij
k
= (a (AB) + b (AC))ij .
Thus A (B + C) = AB + AC as claimed. Formula 5.14 is entirely similar.
Formula 5.15 is the associative law of multiplication. Using Definition 5.1.14,
∑
∑
∑
(A (BC))ij =
Aik (BC)kj =
Aik
Bkl Clj
k
=
∑
k
l
(AB)il Clj = ((AB) C)ij .
l
This proves 5.15. 5.1.5
The Transpose
Another important operation on matrices is that of taking the transpose. The following example
shows what is meant by this operation, denoted by placing a T as an exponent on the matrix.

T
(
1 4
1 3


 3 1  =
4 1
2 6
2
6
)
What happened? The first column became the first row and the second column became the second
row. Thus the 3 × 2 matrix became a 2 × 3 matrix. The number 3 was in the second row and the
first column and it ended up in the first row and second column. Here is the definition.
Definition 5.1.21 Let A be an m × n matrix. Then AT denotes the n × m matrix which is defined
as follows.
( T)
A ij = Aji
74
CHAPTER 5. MATRICES
Example 5.1.22
(
1 2
3 5
−6
4
)T


1 3


=  2 5 .
−6 4
The transpose of a matrix has the following important properties.
Lemma 5.1.23 Let A be an m × n matrix and let B be a n × p matrix. Then
T
(5.16)
T
(5.17)
(AB) = B T AT
and if α and β are scalars,
(αA + βB) = αAT + βB T
Proof: From the definition,
(
)
∑
∑( ) ( )
(
)
T
B T ik AT kj = B T AT ij
(AB)
= (AB)ji =
Ajk Bki =
ij
k
k
The proof of Formula 5.17 is left as an exercise. Definition 5.1.24 An n × n matrix A is said to be symmetric if A = AT . It is said to be skew
symmetric if A = −AT .
Example 5.1.25 Let

2

A= 1
3
1
5
−3

3

−3  .
7
Then A is symmetric.
Example 5.1.26 Let

0

A =  −1
−3

1 3

0 2 
−2 0
Then A is skew symmetric.
5.1.6
The Identity And Inverses
There is a special matrix called I and referred to as the identity matrix. It is always a square matrix,
meaning the number of rows equals the number of columns and it has the property that there are
ones down the main diagonal and zeroes elsewhere. Here are some identity matrices of various sizes.




1 0 0 0
(
)
1 0 0


1 0

  0 1 0 0 
(1) ,
, 0 1 0 ,
.
 0 0 1 0 
0 1
0 0 1
0 0 0 1
The first is the 1 × 1 identity matrix, the second is the 2 × 2 identity matrix, the third is the 3 × 3
identity matrix, and the fourth is the 4 × 4 identity matrix. By extension, you can likely see what
the n × n identity matrix would be. It is so important that there is a special symbol to denote the
ij th entry of the identity matrix Iij = δ ij where δ ij is the Kronecker symbol defined by
{
1 if i = j
δ ij =
0 if i ̸= j
It is called the identity matrix because it is a multiplicative identity in the following sense.
5.1. MATRIX ARITHMETIC
75
Lemma 5.1.27 Suppose A is an m × n matrix and In is the n × n identity matrix. Then AIn = A.
If Im is the m × m identity matrix, it also follows that Im A = A.
Proof:
(AIn )ij =
∑
Aik δ kj = Aij
k
and so AIn = A. The other case is left as an exercise for you. Definition 5.1.28 An n × n matrix A has an inverse, A−1 if and only if AA−1 = A−1 A = I.
Such a matrix is called invertible.
It is very important to observe that the inverse of a matrix, if it exists, is unique. Another way
to think of this is that if it acts like the inverse, then it is the inverse.
Theorem 5.1.29 Suppose A−1 exists and AB = BA = I. Then B = A−1 .
Proof:
(
)
A−1 = A−1 I = A−1 (AB) = A−1 A B = IB = B. Unlike ordinary multiplication of numbers, it can happen that A ̸= 0 but A may fail to have an
inverse. This is illustrated in the following example.
(
)
1 1
Example 5.1.30 Let A =
. Does A have an inverse?
1 1
One might think A would have an inverse because it does not equal zero. However,
(
)(
) ( )
1 1
−1
0
=
1 1
1
0
and if A−1 existed, this could not happen because you could write
( )
((
))
( (
))
0
0
−1
−1
−1
=A
=A
A
=
0
0
1
(
−1
= A
)
A
(
−1
1
)
(
=I
−1
1
)
(
=
−1
1
)
,
a contradiction. Thus the answer is that A does not have an inverse.
(
Example 5.1.31 Let A =
1 1
1 2
)
(
. Show
2
−1
−1
1
)
is the inverse of A.
To check this, multiply
(
and
(
1 1
1 2
2
−1
)(
−1
1
2 −1
−1 1
)(
1 1
1 2
showing that this matrix is indeed the inverse of A.
)
(
=
)
(
=
1
0
0
1
1
0
0
1
)
)
76
5.1.7
CHAPTER 5. MATRICES
Finding The Inverse Of A Matrix
−1
In the last example, how would you find A
(
1
1
1
2
)(
(
? You wish to find a matrix
x z
y w
)
(
=
1
0
0
1
x z
y w
)
such that
)
.
This requires the solution of the systems of equations,
x + y = 1, x + 2y = 0
and
z + w = 0, z + 2w = 1.
Writing the augmented matrix for these two systems gives
(
)
1 1 | 1
1 2 | 0
for the first system and
(
1 1
1 2
| 0
| 1
(5.18)
)
(5.19)
for the second. Lets solve the first system. Take (−1) times the first row and add to the second to
get
(
)
1 1 | 1
0 1 | −1
Now take (−1) times the second row and add to the first to get
(
)
1 0 | 2
.
0 1 | −1
Putting in the variables, this says x = 2 and y = −1.
Now solve the second system, 5.19 to find z and w. Take (−1) times the first row and add to the
second to get
(
)
1 1 | 0
.
0 1 | 1
Now take (−1) times the second row and add to the first to get
(
)
1 0 | −1
.
0 1 | 1
Putting in the variables, this says z = −1 and w = 1. Therefore, the inverse is
(
)
2 −1
.
−1 1
Didn’t the above seem rather repetitive? Note that exactly the same row operations were used
in both systems. In each case, the end result was something
of the form (I|v) where I is the identity
(
)
x
and v gave a column of the inverse. In the above,
, the first column of the inverse was
y
(
)
z
obtained first and then the second column
.
w
5.1. MATRIX ARITHMETIC
77
To simplify this procedure, you could have written
(
)
1 1 | 1 0
1 2 | 0 1
and row reduced till you obtained
(
1
0
0
1
| 2 −1
| −1 1
)
and read off the inverse as the 2 × 2 matrix on the right side.
This is the reason for the following simple procedure for finding the inverse of a matrix. This
procedure is called the Gauss-Jordan procedure.
Procedure 5.1.32 Suppose A is an n × n matrix. To find A−1 if it exists, form the augmented
n × 2n matrix
(A|I)
and then, if possible do row operations until you obtain an n × 2n matrix of the form
(I|B) .
(5.20)
When this has been done, B = A−1 . If it is impossible to row reduce to a matrix of the form (I|B) ,
then A has no inverse.
Actually, all this shows is how to find a right inverse if it exists. What has been shown from
the above discussion is that AB = I. Later, I will show that this right inverse is the inverse. See
Corollary 7.1.15 or Theorem 8.2.11 presented later. However, it is not hard to see that this should
be the case as follows.
The row operations are all reversible. If the row operation involves switching two rows, the
reverse row operation involves switching them again to get back to where you started. If the row
operation involves multiplying a row by a ̸= 0, then you would get back to where you began by
multiplying the row by 1/a. The third row operation involving addition of c times row i to row j
can be reversed by adding −c times row i to row j.
In the above procedure, a sequence of row operations applied to I yields B while the same
sequence of operations applied to A yields I. Therefore, the sequence of reverse row operations in
the opposite order applied to B will yield I and applied to I will yield A. That is, there are row
operations which provide
(B|I) → (I|A)
and as just explained, A must be a right inverse for B. Therefore, BA = I. Hence B is both a right
and a left inverse for A because AB = BA = I.
If it is impossible to row reduce (A|I) to get (I|B) , then in particular, it is impossible to row
reduce A to I and consequently impossible to do a sequence of row operations to I and get A. Later
it will be made clear
( that)the only way this can happen is that it is possible to row reduce A to a
C
matrix of the form
where 0 is a row of zeros. Then there will be no solution to the system
0
of equations represented by the augmented matrix
(
)
C 0
|
0 1
Using the reverse row operations in the opposite order on both matrices in the above, it follows that
there must exist a such that there is no solution to the system of equations represented by (A|a).
Hence A fails to have an inverse, because if it did, then there would be a solution x to the equation
Ax = a given by A−1 a.
78
CHAPTER 5. MATRICES

1

Example 5.1.33 Let A =  1
3
2
0
1

2

2 . Find A−1 if it exists.
−1
Set up the augmented matrix (A|I)

1

 1
3
2 | 1
2 | 0
−1 | 0
2
0
1

0

0 
1
0
1
0
Next take (−1) times the first row and add to the second followed by (−3) times the first row added
to the last. This yields


1 2
2 | 1 0 0


 0 −2 0 | −1 1 0  .
0 −5 −7 | −3 0 1
Then take 5 times the second row and add to -2

1
2
2

 0 −10 0
0
0
14
times the last row.

| 1 0 0

| −5 5 0 
| 1 5 −2
Next take the last row and add to (−7) times the top row. This yields


−7 −14 0 | −6 5 −2


 0 −10 0 | −5 5 0  .
0
0
14 | 1 5 −2
Now take (−7/5) times the second row and

−7
0

 0 −10
0
0
Finally divide the top row by -7, the

1



 0


 0
Therefore, the inverse is







1

Example 5.1.34 Let A =  1
2
2
0
2
0
0
14
| 1 −2
| −5 5
| 1
5

−2

0 .
−2
second row by -10 and the bottom row by 14 which yields

2
2
0 0 | − 17
7
7



1
1
1 0 |
−
0
.
2
2


1
5
1 
0 1 |
−


add to the top.
14
14
− 17
2
7
2
7
1
2
− 12
1
14
5
14
7




0 


1 
−7

2

2 . Find A−1 if it exists.
4
5.1. MATRIX ARITHMETIC
79
Write the augmented matrix (A|I)


| 1 0 0

| 0 1 0 
| 0 0 1
(
)
and proceed to do row operations attempting to obtain I|A−1 . Take (−1) times the top row and
add to the second. Then take (−2) times the top row and add to the bottom.


1 2 2 | 1 0 0


 0 −2 0 | −1 1 0 
0 −2 0 | −2 0 1
1 2

 1 0
2 2
Next add (−1) times the second row to the

1 2

 0 −2
0 0
2
2
4
bottom row.
2
0
0

0 0

1 0 
−1 1
| 1
| −1
| −1
At this point, you can see there will be no inverse because you have obtained a row of zeros in the
left half of the augmented matrix (A|I) . Thus there will be no way to obtain I on the left.


1 0
1


Example 5.1.35 Let A =  1 −1 1 . Find A−1 if it exists.
1 1 −1
II
Form the augmented matrix

1

 1
1
Now do row operations until the n × n
after some computations,

1


 0

0

0
1 | 1 0 0

−1 1 | 0 1 0  .
1 −1 | 0 0 1
matrix on the left becomes the identity matrix. This yields
0
0
1 0
0 1
| 0
1
2
1
2


0 

− 12
| 1 −1
| 1 − 12
and so the inverse of A is the matrix on the right,

1
1
0
2
2


0
 1 −1

1 − 12 − 12
Checking the answer is easy. Just multiply the matrices

1
1


0
2
2
1 0
1



0
 1 −1 1   1 −1

1 − 12 − 12
1 1 −1




.

and see if it works.



1 0 0

 

 =  0 1 0 .

0 0 1
Always check your answer because if you are like some of us, you will usually have made a mistake.
80
CHAPTER 5. MATRICES
Example 5.1.36 In this example, it is shown how to use the inverse of a matrix to find the solution
to a system of equations. Consider the following system of equations. Use the inverse of a suitable
matrix to give the solutions to this system.


x+z =1


 x − y + z = 3 .
x+y−z =2
The system of equations can be written in

1 0
1

 1 −1 1
1 1 −1
terms of matrices as

  
1
x

  
 y  =  3 .
2
z
(5.21)
More simply, this is of the form Ax = b. Suppose you find the inverse of the matrix A−1 . Then you
could multiply both sides of this equation by A−1 to obtain
(
)
x = A−1 A x = A−1 (Ax) = A−1 b.
This gives the solution as x = A−1 b. Note that once you have found the inverse, you can easily get
the solution for different right hand sides without any effort. It is always just A−1 b. In the given
example, the inverse of the matrix is


1
1
0
2
2


0 
 1 −1
1 − 12 − 12
This was shown in Example 5.1.35. Therefore, from what was just explained, the solution to the
given system is

 
  

1
1
5
x
1
0
2
2
2

 
  

0   3  =  −2  .
 y  =  1 −1
z
2
1 − 12 − 12
− 32
(
)T
What if the right side of 5.21 had been 0 1 3
? What would be the solution to


  
1 0
1
x
0


  
=
1
−1
1
y


  1 ?
1 1 −1
z
3
By the above discussion, it is just

 
1
x
0
2

 
 y  =  1 −1
z
1 − 12
 

0
2
  

0   1  =  −1  .
− 12
3
−2
1
2

This illustrates why once you have found the inverse of a given matrix, you can use it to solve many
different systems easily.
5.2
Exercises
1. Here are some matrices:
(
A =
(
C
=
1
2
1
3
)
(
)
2 3
3 −1 2
,B =
,
1 7
−3 2 1
)
(
)
(
)
2
−1 2
2
,D =
,E =
.
1
2 −3
3
5.2. EXERCISES
81
Find if possible −3A, 3B − A, AC, CB, AE, EA. If it is not possible explain why.
2. Here are some matrices:

A =
C
=

)
(
1 2
2 −5 2


,
 3 2 ,B =
−3 2 1
1 −1
(
)
(
)
(
)
1 2
−1 1
1
,D =
,E =
.
5 0
4 −3
3
Find if possible −3A, 3B − A, AC, CA, AE, EA, BE, DE. If it is not possible explain why.
3. Here are some matrices:

(
)
1 2
2 −5 2


,
 3 2 ,B =
−3 2 1
1 −1
(
)
(
)
(
)
1 2
−1 1
1
,D =
,E =
.
5 0
4 −3
3

A =
C
=
Find if possible −3AT , 3B − AT , AC, CA, AE, E T B, BE, DE, EE T , E T E. If it is not possible
explain why.
4. Here are some matrices:


(
)
1 2
2 −5 2


A =  3 2 ,B =
,
−3 2 1
1 −1
(
)
(
)
( )
1 2
−1
1
C =
,D =
,E =
.
5 0
4
3
Find the following if possible and explain why it is not possible if this is the case.
AD, DA, DT B, DT BE, E T D, DE T .


(
1
1
1


5. Let A =  −2 −1 , B =
2
1
2
ble.
−1 −2
1 −2
)

1

, and C =  −1
−3

1 −3

2
0  . Find if possi−1 0
(a) AB
(b) BA
(c) AC
(d) CA
(e) CB
(f) BC
6. Suppose A and B are square matrices of the same size. Which of the following are correct?
2
(a) (A − B) = A2 − 2AB + B 2
2
(b) (AB) = A2 B 2
82
CHAPTER 5. MATRICES
2
(c) (A + B) = A2 + 2AB + B 2
2
(d) (A + B) = A2 + AB + BA + B 2
(e) A2 B 2 = A (AB) B
3
(f) (A + B) = A3 + 3A2 B + 3AB 2 + B 3
(g) (A + B) (A − B) = A2 − B 2
(
)
−1 −1
7. Let A =
. Find all 2 × 2 matrices, B such that AB = 0.
3
3
8. Let x = (−1, −1, 1) and y = (0, 1, 2) . Find xT y and xyT if possible.
(
)
(
)
1 2
1 2
9. Let A =
,B =
. Is it possible to choose k such that AB = BA? If so,
3 4
3 k
what should k equal?
(
)
(
)
1 2
1 2
10. Let A =
,B =
. Is it possible to choose k such that AB = BA? If so,
3 4
1 k
what should k equal?
11. In 5.1 - 5.8 describe −A and 0.
12. Let A be an n × n matrix. Show A equals the sum of a symmetric and a skew symmetric
T
T
matrix.
( T (M) is skew symmetric if M = −M . M is symmetric if M = M .) Hint: Show that
1
2 A + A is symmetric and then consider using this as one of the matrices.
13. Show every skew symmetric matrix has all zeros down the main diagonal. The main diagonal
consists of every entry of the matrix which is of the form aii . It runs from the upper left down
to the lower right.
14. Suppose M is a 3 × 3 skew symmetric matrix. Show there exists a vector Ω such that for all
u ∈ R3
Mu = Ω × u
Hint: Explain why, since M is skew symmetric it is of the form


0
−ω 3 ω 2


M =  ω3
0
−ω 1 
−ω 2 ω 1
0
where the ω i are numbers. Then consider ω 1 i + ω 2 j + ω 3 k.
15. Using only the properties 5.1 - 5.8 show −A is unique.
16. Using only the properties 5.1 - 5.8 show 0 is unique.
17. Using only the properties 5.1 - 5.8 show 0A = 0. Here the 0 on the left is the scalar 0 and the
0 on the right is the zero for m × n matrices.
18. Using only the properties 5.1 - 5.8 and previous problems show (−1) A = −A.
19. Prove 5.17.
20. Prove that Im A = A where A is an m × n matrix.
21. Give an example of matrices, A, B, C such that B ̸= C, A ̸= 0, and yet AB = AC.
22. Suppose AB = AC and A is an invertible n × n matrix. Does it follow that B = C? Explain
why or why not. What if A were a non invertible n × n matrix?
5.2. EXERCISES
83
23. Find your own examples:
(a) 2 × 2 matrices, A and B such that A ̸= 0, B ̸= 0 with AB ̸= BA.
(b) 2 × 2 matrices, A and B such that A ̸= 0, B ̸= 0, but AB = 0.
(c) 2 × 2 matrices, A, D, and C such that A ̸= 0, C ̸= D, but AC = AD.
24. Explain why if AB = AC and A−1 exists, then B = C.
25. Give an example of a matrix A such that A2 = I and yet A ̸= I and A ̸= −I.
26. Give an example of matrices, A, B such that neither A nor B equals zero and yet AB = 0.
27. Give another example other than the one given in this section of two square matrices, A and
B such that AB ̸= BA.
(
28. Let
A=
2 1
−1 3
)
.
Find A−1 if possible. If A−1 does not exist, determine why.
(
29. Let
A=
0 1
5 3
)
.
Find A−1 if possible. If A−1 does not exist, determine why.
(
30. Let
A=
2 1
3 0
)
.
Find A−1 if possible. If A−1 does not exist, determine why.
(
31. Let
A=
2 1
4 2
)
.
Find A−1 if possible. If A−1 does not exist, determine why.
(
32. Let A be a 2 × 2 matrix which has an inverse. Say A =
in terms of a, b, c, d.
33. Let

1

A= 2
1
2
1
0

3

4 .
2
Find A−1 if possible. If A−1 does not exist, determine why.
34. Let

1

A= 2
1
0
3
0

3

4 .
2
Find A−1 if possible. If A−1 does not exist, determine why.
a b
c d
)
. Find a formula for A−1
84
CHAPTER 5. MATRICES

35. Let
1

A= 2
4

3

4 .
10
2
1
5
Find A−1 if possible. If A−1 does not exist, determine why.
36. Let



A=

1
1
2
1
2
1
1
2
0
2
−3
1
2
0
2
2





Find A−1 if possible. If A−1 does not exist, determine why.




x1
x1 − x2 + 2x3
 x 


2x3 + x1


 2 
37. Write 
 where A is an appropriate matrix.
 in the form A 
 x3 


3x3
3x4 + 3x2 + x1



38. Write 




39. Write 

x1 + 3x2 + 2x3
2x3 + x1
6x3
x4 + 3x2 + x1
x1 + x2 + x3
2x3 + x1 + x2
x3 − x1
3x4 + x1
x4






 in the form A 








 in the form A 


x1
x2
x3
x4
x1
x2
x3
x4



 where A is an appropriate matrix.




 where A is an appropriate matrix.

40. Using the inverse of the matrix, find the solution to the systems



 


  
1 0 3
x
1
1 0 3
x
2



  

 

 2 3 4  y  =  2 ,  2 3 4  y  =  1 
1 0 2
z
3
1 0 2
z
0



 


  
1 0 3
x
1
1 0 3
x
3



  

 

 2 3 4   y  =  0  ,  2 3 4   y  =  −1  .
1 0 2
z
1
1 0 2
z
−2
Now give the solution in terms of a, b,

1 0

 2 3
1 0
and c to

 

3
x
a

 

4  y  =  b .
2
z
c
41. Using the inverse of the matrix, find the solution to the systems



  

 

1 0 3
x
1
1 0 3
x
2



  

 

 2 3 4  y  =  2 ,  2 3 4  y  =  1 
1 0 2
z
3
1 0 2
z
0



  

 

1 0 3
x
1
1 0 3
x
3



  

 

 2 3 4   y  =  0  ,  2 3 4   y  =  −1  .
1 0 2
z
1
1 0 2
z
−2
5.2. EXERCISES
85
Now give the solution in terms of a, b,

1 0

 2 3
1 0
and c to

 

3
x
a

 

4  y  =  b .
2
z
c
42. Using the inverse of the matrix, find the solution to the system





1
−1
2
1
3
2
−1 0
−2 − 34
1
2
− 12
1
2
− 52

0



1 
1
4
9
4
x
y
z
w


 
 
=
 
a
b
c
d



.

43. Show that if A is an n × n invertible matrix and x is a n × 1 matrix such that Ax = b for b
an n × 1 matrix, then x = A−1 b.
44. Prove that if A−1 exists and Ax = 0 then x = 0.
45. Show that if A−1 exists for an n × n matrix, then it is unique. That is, if BA = I and AB = I,
then B = A−1 .
( )−1 ( −1 )T
46. Show that if A is an invertible n × n matrix, then so is AT and AT
= A
.
(
)
−1
47. Show (AB) = B −1 A−1 by verifying that AB B −1 A−1 = I and
B −1 A−1 (AB) = I.
Hint: Use Problem 45.
−1
48. Show that (ABC)
= C −1 B −1 A−1 by verifying that
(
)
(ABC) C −1 B −1 A−1 = I
(
)
and C −1 B −1 A−1 (ABC) = I. Hint: Use Problem 45.
( )−1 ( −1 )2
49. If A is invertible, show A2
= A
. Hint: Use Problem 45.
(
)−1
50. If A is invertible, show A−1
= A. Hint: Use Problem 45.
(
)
51. Let A and be a real m × n matrix and let x ∈ Rn and y ∈ Rm . Show (Ax, y)Rm = x,AT y Rn
where (·, ·)Rk denotes the dot product in Rk . In the notation above, Ax · y = x·AT y. Use the
definition of matrix multiplication to do this.
T
52. Use the result of Problem 51 to verify directly that (AB)
reference to subscripts.
= B T AT without making any
53. Suppose A is an n × n matrix and for each j,
n
∑
|Aij | < 1
i=1
∑∞
Ak converges in the sense that the ij th entry of the partial
( )
∑n
sums converge for each ij. Hint: Let R ≡ maxj i=1 |Aij | . Thus R < 1. Show that A2 ij ≤
Show that the infinite series
k=0
R2 . Then generalize to show that (Am )ij ≤ Rm . Use this to show that the ij th entry of the
partial sums is a Cauchy sequence. From calculus, these converge by completeness of the real
86
CHAPTER 5. MATRICES
−1
or complex numbers. Next show that (I − A) =
involves solving an equation for x of the form
∑∞
k=0
Ak . The Leontief model in economics
x = Ax + b, or (I − A) x = b
The vector Ax is called the intermediate demand and the vectors Ak x have economic meaning.
From the above,
x = Ib + Ab + A2 b + · · ·
The series is also called the Neuman series. It is important in functional analysis.
54. An elementary matrix is one which results from doing a row operation to the identity matrix.
Thus the elementary matrix E which results from adding a times the ith row to the j th row
would have aδ ik + δ jk as the jk th entry and all other rows would be unchanged. That is δ rs
provided r ̸= j. Show that multiplying this matrix on the left of an appropriate sized matrix
A results in doing the row operation to the matrix A. You might also want to verify that the
other elementary matrices have the same effect, doing the row operation which resulted in the
elementary matrix to A.
Chapter 6
Determinants
6.1
6.1.1
Basic Techniques And Properties
Cofactors And 2 × 2 Determinants
Let A be an n × n matrix. The determinant of A, denoted as det (A) is a number. If the matrix
is a 2×2 matrix, this number is very easy to find.
(
)
a b
Definition 6.1.1 Let A =
. Then
c d
det (A) ≡ ad − cb.
The determinant is also often denoted by enclosing the matrix with two vertical lines. Thus
(
)
a b
a b
det
=
.
c d
c d
(
)
2 4
Example 6.1.2 Find det
.
−1 6
From the definition this is just (2) (6) − (−1) (4) = 16.
Having defined what is meant by the determinant of a 2 × 2 matrix, what about a 3 × 3 matrix?
Definition 6.1.3 Suppose A is a 3 × 3 matrix. The ij th minor, denoted as minor(A)ij , is the
determinant of the 2 × 2 matrix which results from deleting the ith row and the j th column.
Example 6.1.4 Consider the matrix

1

 4
3

2 3

3 2 .
2 1
The (1, 2) minor is the determinant of the 2 × 2 matrix which results when you delete the first row
and the second column. This minor is therefore
(
)
4 2
det
= −2.
3 1
The (2, 3) minor is the determinant of the 2 × 2 matrix which results when you delete the second row
and the third column. This minor is therefore
(
)
1 2
det
= −4.
3 2
87
88
CHAPTER 6. DETERMINANTS
i+j
Definition 6.1.5 Suppose A is a 3 × 3 matrix. The ij th cofactor is defined to be (−1)
×
( th
)
i+j
th
th
ij minor . In words, you multiply (−1)
times the ij minor to get the ij cofactor. The
cofactors of a matrix are so important that special notation is appropriate when referring to them.
The ij th cofactor of a matrix A will be denoted by cof (A)ij . It is also convenient to refer to the
cofactor of an entry of a matrix as follows. For aij an entry of the matrix, its cofactor is just
cof (A)ij . Thus the cofactor of the ij th entry is just the ij th cofactor.
Example 6.1.6 Consider the matrix

1

A= 4
3

3

2 .
1
2
3
2
The (1, 2) minor is the determinant of the 2 × 2 matrix which results when you delete the first row
and the second column. This minor is therefore
(
)
4 2
det
= −2.
3 1
It follows
(
cof (A)12 = (−1)
1+2
4
3
det
2
1
)
= (−1)
1+2
(−2) = 2
The (2, 3) minor is the determinant of the 2 × 2 matrix which results when you delete the second row
and the third column. This minor is therefore
(
)
1 2
det
= −4.
3 2
Therefore,
(
2+3
cof (A)23 = (−1)
det
1 2
3 2
Similarly,
)
2+3
= (−1)
(
cof (A)22 = (−1)
2+2
det
1
3
3
1
(−4) = 4.
)
= −8.
Definition 6.1.7 The determinant of a 3 × 3 matrix A, is obtained by picking a row (column) and
taking the product of each entry in that row (column) with its cofactor and adding these up. This
process when applied to the ith row (column) is known as expanding the determinant along the ith
row (column).
Example 6.1.8 Find the determinant of

1

A= 4
3

3

2 .
1
2
3
2
Here is how it is done by “expanding along the first column”.
z
1(−1)
cof(A)11
}|
1+1
3 2
2 1
{
cof(A)21
}|
z
2+1
+ 4(−1)
2
2
{
3
1
z
+ 3(−1)
cof(A)31
}|
3+1
2
3
{
3
2
= 0.
6.1. BASIC TECHNIQUES AND PROPERTIES
89
You see, we just followed the rule in the above definition. We took the 1 in the first column and
multiplied it by its cofactor, the 4 in the first column and multiplied it by its cofactor, and the 3 in
the first column and multiplied it by its cofactor. Then we added these numbers together.
You could also expand the determinant along the second row as follows.
cof(A)21
z
}|
4(−1)
2+1
{
2 3
2 1
cof(A)22
}|
z
2+2
+ 3(−1)
1
3
{
3
1
cof(A)23
}|
z
1
3
2+3
+ 2(−1)
{
2
2
= 0.
Observe this gives the same number. You should try expanding along other rows and columns. If
you don’t make any mistakes, you will always get the same answer.
What about a 4 × 4 matrix? You know now how to find the determinant of a 3 × 3 matrix. The
pattern is the same.
Definition 6.1.9 Suppose A is a 4 × 4 matrix. The ij th minor is the determinant of the 3 × 3
matrix you obtain when you delete the ith row and the j th column. The ij th cofactor, cof (A)ij is
(
)
i+j
i+j
defined to be (−1) × ij th minor . In words, you multiply (−1)
times the ij th minor to get the
th
ij cofactor.
Definition 6.1.10 The determinant of a 4 × 4 matrix A, is obtained by picking a row (column) and
taking the product of each entry in that row (column) with its cofactor and adding these together.
This process when applied to the ith row (column) is known as expanding the determinant along the
ith row (column).
Example 6.1.11 Find det (A) where



A=

1
5
1
3
2
4
3
4
3
2
4
3
4
3
5
2





As in the case of a 3 × 3 matrix, you can expand this along any row or column. Lets pick the
third column. det (A) =
3 (−1)
1+3
5 4
1 3
3 4
3
5
2
+ 2 (−1)
2+3
1 2
1 3
3 4
4
5
2
+ 4 (−1)
3+3
1
5
3
2
4
4
4
3
2
4+3
+ 3 (−1)
1
5
1
2
4
3
4
3 .
5
Now you know how to expand each of these 3 × 3 matrices along a row or a column. If you do so, you
will get −12 assuming you make no mistakes. You could expand this matrix along any row or any
column and assuming you make no mistakes, you will always get the same thing which is defined to
be the determinant of the matrix A. This method of evaluating a determinant by expanding along
a row or a column is called the method of Laplace expansion.
Note that each of the four terms above involves three terms consisting of determinants of 2 × 2
matrices and each of these will need 2 terms. Therefore, there will be 4 × 3 × 2 = 24 terms to
evaluate in order to find the determinant using the method of Laplace expansion. Suppose now you
have a 10 × 10 matrix and you follow the above pattern for evaluating determinants. By analogy to
the above, there will be 10! = 3, 628 , 800 terms involved in the evaluation of such a determinant by
Laplace expansion along a row or column. This is a lot of terms.
In addition to the difficulties just discussed, you should regard the above claim that you always
get the same answer by picking any row or column with considerable skepticism. It is incredible and
not at all obvious. However, it requires a little effort to establish it. This is done in the section on
the theory of the determinant.
90
CHAPTER 6. DETERMINANTS
Definition 6.1.12 Let A = (aij ) be an n × n matrix and suppose the determinant of a (n − 1) ×
(n − 1) matrix has been defined. Then a new matrix called the cofactor matrix, cof (A) is defined by
cof (A) = (cij ) where to obtain cij delete the ith row and the j th column of A, take the determinant
of the (n − 1) × (n − 1) matrix which results, (This is called the ij th minor of A. ) and then
(
)
i+j
i+j
multiply this number by (−1) . Thus (−1) × the ij th minor equals the ij th cofactor. To make
the formulas easier to remember, cof (A)ij will denote the ij th entry of the cofactor matrix.
With this definition of the cofactor matrix, here is how to define the determinant of an n × n
matrix.
Definition 6.1.13 Let A be an n × n matrix where n ≥ 2 and suppose the determinant of an
(n − 1) × (n − 1) has been defined. Then
det (A) =
n
∑
aij cof (A)ij =
j=1
n
∑
aij cof (A)ij .
(6.1)
i=1
The first formula consists of expanding the determinant along the ith row and the second expands
the determinant along the j th column.
Theorem 6.1.14 Expanding the n × n matrix along any row or column always gives the same
answer so the above definition is a good definition.
6.1.2
The Determinant Of A Triangular Matrix
Notwithstanding the difficulties involved in using the method of Laplace expansion, certain types of
matrices are very easy to deal with.
Definition 6.1.15 A matrix M , is upper triangular if
equals zero below the main diagonal, the entries of the

∗ ∗ ···

..
 0 ∗
.

 .
..
 . ..
.
.
 .
0
Mij = 0 whenever i > j. Thus such a matrix
form Mii , as shown.

∗
.. 
. 



∗ 
0 ∗
···
A lower triangular matrix is defined similarly as a matrix for which all entries above the main
diagonal are equal to zero.
You should verify the following using the above theorem on Laplace expansion.
Corollary 6.1.16 Let M be an upper (lower) triangular matrix. Then det (M ) is obtained by taking
the product of the entries on the main diagonal.
Example 6.1.17 Let



A=

Find det (A) .
1
0
0
0
2
2
0
0
3
6
3
0
77
7
33.7
−1





6.1. BASIC TECHNIQUES AND PROPERTIES
91
From the above corollary, it suffices to take the product of the diagonal elements. Thus det (A) =
1 × 2 × 3 × (−1) = −6. Without using the corollary, you could expand along the first column. This
gives
2 6
0 3
0 0
7
33.7
−1
2
0
0
3
3
0
77
33.7
−1
2
2
0
3
6
0
77
7
−1
Now expand this along the first column to obtain
(
6 7
3 33.7
2+1
3+1
+ 0 (−1)
+ 0 (−1)
1× 2×
0 −1
0 −1
6
3
7
33.7
1
2+1
+ 0 (−1)
3+1
+ 0 (−1)
+ 0 (−1)
4+1
2 3
2 6
0 3
77
7
33.7
and the only nonzero term in the expansion is
1
2 6
0 3
0 0
7
33.7 .
−1
)
=1×2×
3
0
33.7
−1
Next expand this last determinant along the first column to obtain the above equals
1 × 2 × 3 × (−1) = −6
which is just the product of the entries down the main diagonal of the original matrix. It works this
way in general.
6.1.3
Properties Of Determinants
There are many properties satisfied by determinants. Some of these properties have to do with row
operations. Recall the row operations.
Definition 6.1.18 The row operations consist of the following
1. Switch two rows.
2. Multiply a row by a nonzero number.
3. Replace a row by a multiple of another row added to itself.
Theorem 6.1.19 Let A be an n × n matrix and let A1 be a matrix which results from multiplying
some row of A by a scalar c. Then c det (A) = det (A1 ).
(
)
(
)
1 2
2 4
Example 6.1.20 Let A =
, A1 =
. det (A) = −2, det (A1 ) = −4.
3 4
3 4
Theorem 6.1.21 Let A be an n × n matrix and let A1 be a matrix which results from switching two
rows of A. Then det (A) = − det (A1 ) . Also, if one row of A is a multiple of another row of A, then
det (A) = 0.
(
)
(
)
1 2
3 4
Example 6.1.22 Let A =
and let A1 =
. det A = −2, det (A1 ) = 2.
3 4
1 2
Theorem 6.1.23 Let A be an n × n matrix and let A1 be a matrix which results from applying
row operation 3. That is you replace some row by a multiple of another row added to itself. Then
det (A) = det (A1 ).
92
CHAPTER 6. DETERMINANTS
(
)
(
)
1 2
1 2
Example 6.1.24 Let A =
and let A1 =
. Thus the second row of A1 is one
3 4
4 6
times the first row added to the second row. det (A) = −2 and det (A1 ) = −2.
Theorem 6.1.25 In Theorems 6.1.19 - 6.1.23 you can replace the word, “row” with the word “column”.
There are two other major properties of determinants which do not involve row operations.
Theorem 6.1.26 Let A and B be two n × n matrices. Then
det (AB) = det (A) det (B).
Also,
( )
det (A) = det AT .
Example 6.1.27 Compare det (AB) and det (A) det (B) for
(
)
(
)
1 2
3 2
A=
,B =
.
−3 2
4 1
(
First
AB =
1 2
−3 2
)(
3 2
4 1
(
and so
det (AB) = det
det (A) = det
(
and
(
det (B) = det
1 2
−3 2
3 2
4 1
11
−1
=
11
4
−1 −4
(
Now
)
4
−4
)
)
= −40.
)
=8
)
= −5.
Thus det (A) det (B) = 8 × (−5) = −40.
6.1.4
Finding Determinants Using Row Operations
Theorems 6.1.23 - 6.1.25 can be used to find determinants using row operations. As pointed out
above, the method of Laplace expansion will not be practical for any matrix of large size. Here is
an example in which all the row operations are used.
Example 6.1.28 Find the determinant of the matrix



A=

1
5
4
2
2
1
5
2
3
2
4
−4
4
3
3
5





6.1. BASIC TECHNIQUES AND PROPERTIES
93
Replace the second row by (−5) times the first row added to it. Then replace the third row by
(−4) times the first row added to it. Finally, replace the fourth row by (−2) times the first row
added to it. This yields the matrix



B=

1 2
0 −9
0 −3
0 −2

3
4
−13 −17 


−8 −13 
−10 −3
and from (Theorem
6.1.23, it has the same determinant as
)
det (B) = −1
det
(C)
where
3

1 2
3
4
 0 0
11
22

C=
 0 −3 −8 −13
0 6
30
9
A. Now using other row operations,



.

The second row was replaced by (−3) times the third row added to the second row. By Theorem
6.1.23 this didn’t change the value of the determinant. Then the last row was multiplied by (−3) .
By Theorem 6.1.19 the resulting matrix has a determinant which is (−3) times the determinant of
the un-multiplied matrix. Therefore, we multiplied by −1/3 to retain the correct value. Now replace
the last row with 2 times the third added to it. This does not change the value of the determinant
by Theorem 6.1.23. Finally switch the third and second rows. This causes the determinant to be
multiplied by (−1) . Thus det (C) = − det (D) where



D=

1
0
0
0

2
3
4
−3 −8 −13
0
11
22
0
14 −17




You could do more row operations or you could note that this can be easily expanded along the first
column followed by expanding the 3 × 3 matrix which results along its first column. Thus
det (D) = 1 (−3)
and so det (C) = −1485 and det (A) = det (B) =
11 22
14 −17
( −1 )
3
= 1485
(−1485) = 495.
Example 6.1.29 Find the determinant of the matrix






1 2 3 2
1 −3 2 1 


2 1 2 5 
3 −4 1 2
Replace the second row by (−1) times the first row added to it. Next take −2 times the first row
and add to the third and finally take −3 times the first row and add to the last row. This yields





1
2
3
0 −5 −1
0 −3 −4
0 −10 −8
2
−1
1
−4



.

94
CHAPTER 6. DETERMINANTS
By Theorem 6.1.23 this matrix has the same determinant as the original matrix. Remember you can
work with the columns also. Take −5 times the last column and add to the second column. This
yields


1 −8 3
2
 0 0 −1 −1 




 0 −8 −4 1 
0
10
−8 −4
By Theorem 6.1.25 this matrix has the same determinant as the original matrix. Now take (−1)
times the third row and add to the top row. This gives.





1
0
0
0
0
0
−8
10

7
1
−1 −1 


−4 1 
−8 −4
which by Theorem 6.1.23 has the same determinant as the original matrix. Lets expand it now along
the first column. This yields the following for the determinant of the original matrix.


0 −1 −1


det  −8 −4 1 
10 −8 −4
which equals
(
8 det
−1
−8
−1
−4
)
(
+ 10 det
−1 −1
−4 1
)
= −82
We suggest you do not try to be fancy in using row operations. That is, stick mostly to the one
which replaces a row or column with a multiple of another row or column added to it. Also note
there is no way to check your answer other than working the problem more than one way. To be
sure you have gotten it right you must do this.
6.2
6.2.1
Applications
A Formula For The Inverse
The definition of the determinant in terms of Laplace expansion along a row or column also provides
a way to give a formula for the inverse of a matrix. Recall the definition of the inverse of a matrix
in Definition 5.1.28 on Page 75. Also recall the definition of the cofactor matrix given in Definition
6.1.12 on Page 90. This cofactor matrix was just the matrix which results from replacing the ij th
entry of the matrix with the ij th cofactor.
The following theorem says that to find the inverse, take the transpose of the cofactor matrix
and divide by the determinant. The transpose of the cofactor matrix is called the adjugate or
sometimes the classical adjoint of the matrix A. In other words, A−1 is equal to one divided by
the determinant of A times the adjugate matrix of A. This is what the following theorem says with
more precision.
)
(
where
Theorem 6.2.1 A−1 exists if and only if det(A) ̸= 0. If det(A) ̸= 0, then A−1 = a−1
ij
−1
a−1
cof (A)ji
ij = det(A)
for cof (A)ij the ij th cofactor of A.
6.2. APPLICATIONS
95
Example 6.2.2 Find the inverse of the matrix

1

A= 3
1

2 3

0 1 
2 1
First find the determinant of this matrix. Using Theorems 6.1.23 - 6.1.25 on Page 92, the
determinant of this matrix equals the determinant of the matrix


1 2
3


 0 −6 −8 
0 0 −2
which equals 12. The cofactor matrix of A is


−2 −2 6


 4 −2 0  .
2
8 −6
Each entry of A was replaced by
should equal

−2
1 
 4
12
2
its cofactor. Therefore, from the above theorem, the inverse of A

T 
−1/6 1/3
1/6
−2 6



−2 0  =  −1/6 −1/6 2/3  .
1/2
0
−1/2
8 −6
Does it work? You should check to see if it does.


−1/6 1/3
1/6


 −1/6 −1/6 2/3  
1/2
0
−1/2
When the matrices
 
1 2 3
1
 
=
3 0 1   0
1 2 1
0
are multiplied

0 0

1 0 
0 1
and so it is correct.
Example 6.2.3 Find the inverse of the matrix

1
2

 1
 −
6
A=

 5
 −
6
0
1
3
2
3
1
2



− 12 



1 
−2
First find its determinant. This determinant is 16 . The inverse is therefore equal to

1/3
2/3
−1/2
−1/2




0
1/2

6 −

2/3 −1/2



0
1/2
1/3 −1/2
−1/6
−
−5/6
−1/2
−1/2
1/2
1/2
−5/6 −1/2
−
1/2
−1/6
1/2
−1/2
−1/6
1/3
−5/6 2/3
1/2
0
−
−5/6 2/3
1/2
−1/6
0
1/3
T





 .




96
CHAPTER 6. DETERMINANTS
Expanding all the 2 × 2 determinants this yields

T 
1/6 1/3 1/6
1



6  1/3 1/6 −1/3  =  2
−1/6 1/6 1/6
1

2 −1

1
1 
−2 1
Always check your work.


 
1 2 −1
1/2
0
1/2
1 0


 
1   −1/6 1/3 −1/2  =  0 1
 2 1
1 −2 1
−5/6 2/3 −1/2
0 0

0

0 
1
and so we got it right. If the result of multiplying these matrices had been something other than
the identity matrix, you would know there was an error. When this happens, you need to search for
the mistake if you are interested in getting the right answer. A common mistake is to forget to take
the transpose of the cofactor matrix.
Proof of Theorem 6.2.1: From the definition of the determinant in terms of expansion along
a column, and letting (air ) = A, if det (A) ̸= 0,
n
∑
air cof (A)ir det(A)−1 = det(A) det(A)−1 = 1.
i=1
Now consider
n
∑
air cof (A)ik det(A)−1
i=1
when k ̸= r. Replace the k th column with the rth column to obtain a matrix Bk whose determinant
equals zero by Theorem 6.1.21. However, expanding this matrix Bk along the k th column yields
0 = det (Bk ) det (A)
−1
=
n
∑
air cof (A)ik det (A)
−1
i=1
Summarizing,
n
∑
{
air cof (A)ik det (A)
−1
= δ rk ≡
i=1
Now
n
∑
air cof (A)ik =
i=1
n
∑
1 if r = k
.
0 if r ̸= k
T
air cof (A)ki
i=1
T
which is the krth entry of cof (A) A. Therefore,
T
cof (A)
A = I.
det (A)
(6.2)
Using the other formula in Definition 6.1.13, and similar reasoning,
n
∑
−1
arj cof (A)kj det (A)
= δ rk
j=1
Now
n
∑
j=1
arj cof (A)kj =
n
∑
j=1
T
arj cof (A)jk
6.2. APPLICATIONS
97
T
which is the rk th entry of A cof (A) . Therefore,
T
A
cof (A)
= I,
det (A)
(6.3)
(
)
and it follows from 6.2 and 6.3 that A−1 = a−1
ij , where
−1
a−1
ij = cof (A)ji det (A)
.
In other words,
A−1 =
T
cof (A)
.
det (A)
Now suppose A−1 exists. Then by Theorem 6.1.26,
(
)
(
)
1 = det (I) = det AA−1 = det (A) det A−1
so det (A) ̸= 0. This way of finding inverses is especially useful in the case where it is desired to find the inverse
of a matrix whose entries are functions.
Example 6.2.4 Suppose

et

A (t) =  0
0
Show that A (t)
−1

0
0

cos t sin t 
− sin t cos t
exists and then find it.
−1
First note det (A (t)) = et ̸= 0 so A (t)

exists. The cofactor matrix is

1
0
0


C (t) =  0 et cos t et sin t 
0 −et sin t et cos t
and so the inverse is

1
1 
0

et
0
6.2.2
0
et cos t
−et sin t
T 
0
e−t


t
e sin t  =  0
et cos t
0

0
0

cos t − sin t  .
sin t cos t
Cramer’s Rule
This formula for the inverse also implies a famous procedure known as Cramer’s rule. Cramer’s
rule gives a formula for the solutions, x, to a system of equations, Ax = y in the special case that A
is a square matrix. Note this rule does not apply if you have a system of equations in which there
is a different number of equations than variables.
In case you are solving a system of equations, Ax = y for x, it follows that if A−1 exists,
(
)
x = A−1 A x = A−1 (Ax) = A−1 y
thus solving the system. Now in the case that A−1 exists, there is a formula for A−1 given above.
Using this formula,
n
n
∑
∑
1
−1
xi =
aij yj =
cof (A)ji yj .
det
(A)
j=1
j=1
98
CHAPTER 6. DETERMINANTS
By the formula for the expansion of a determinant along a column,


∗ · · · y1 · · · ∗
 .
1
..
.. 

..
xi =
det 
.
. ,

det (A)
∗ · · · yn · · · ∗
T
where here the ith column of A is replaced with the column vector (y1 · · · ·, yn ) , and the determinant
of this modified matrix is taken and divided by det (A). This formula is known as Cramer’s rule.
Procedure 6.2.5 Suppose A is an n × n matrix and it is desired to solve the system Ax = y, y =
T
T
(y1 , · · · , yn ) for x = (x1 , · · · , xn ) . Then Cramer’s rule says
xi =
det Ai
det A
where Ai is obtained from A by replacing the ith column of A with the column
T
(y1 , · · · , yn ) .
Find x, y if

  
2 1
x
1

  
2 1  y  =  2 .
−3 2
z
3


1 2 1


The determinant of the matrix of coefficients,  3 2 1  is −14. From Cramer’s rule, to get
2 −3 2
x, you replace the first column of A with the right side of the equation and take its determinant and
divide by the determinant of A. Thus

1

 3
2
1
2
3
x=
2 1
2 1
−3 2
−14
=
1
2
Now to find y, z, you do something similar.
y=
1 1
3 2
2 3
−14
1
1
2
1
=− , z=
7
1
3
2
2 1
2 2
−3 3
−14
=
11
14
You see the pattern. For large systems Cramer’s rule is less than useful if you want to find an
answer. This is because to use it you must evaluate determinants. However, you have no practical
way to evaluate determinants for large matrices other than row operations and if you are using row
operations, you might just as well use them to solve the system to begin with. It will be a lot less
trouble. Nevertheless, there are situations in which Cramer’s rule is useful.
Example 6.2.6 Solve for z if

1

 0
0

 

0
0
x
1

 

et cos t et sin t   y  =  t 
−et sin t et cos t
z
t2
6.3. EXERCISES
99
You could do it by row operations but it might be easier in this case to use Cramer’s rule because
the matrix of coefficients does not consist of numbers but of functions. Thus
z=
1
0
1
0 et cos t
t
t
0 −e sin t t2
1
0
t
0 e cos t
0 −et sin t
= t ((cos t) t + sin t) e−t .
0
t
e sin t
et cos t
You end up doing this sort of thing sometimes in ordinary differential equations in the method of
variation of parameters.
6.3
Exercises
1. Find the determinants of the following matrices.


1 2 3


(a)  3 2 2  (The answer is 31.)
(c)
0 9 8


4 3 2


(b)  1 7 8 (The answer is 375.)
3 −9 3





1
1
4
1
2
3
1
2
3
2
5
1
2
3
0
2



, (The answer is −2.)

2. Find the following determinant by expanding along the first row and second column.
1 2
2 1
2 1
1
3
1
3. Find the following determinant by expanding along the first column and third row.
1 2
1 0
2 1
1
1
1
4. Find the following determinant by expanding along the second row and first column.
1 2
2 1
2 1
1
3
1
5. Compute the determinant by cofactor expansion. Pick the easiest row or column to use.
1
2
0
2
0
1
0
1
0
1
0
3
1
0
2
1
100
CHAPTER 6. DETERMINANTS
6. Find the determinant using row operations.
1 2 1
2 3 2
−4 1 2
7. Find the determinant using row operations.
2 1
2 4
1 4
3
2
−5
8. Find the determinant using row operations.
1 2
3 1
−1 0
2 3
1
2
−2 3
3
1
2 −2
9. Find the determinant using row operations.
1 4
3 2
−1 0
2 1
1
2
−2 3
3
3
2 −2
10. Verify an example of each property of determinants found in Theorems 6.1.23 - 6.1.25 for 2 × 2
matrices.
11. An operation is done to get from the first matrix to the second. Identify what was done and
tell how it will affect the value of the determinant.
(
) (
)
a b
a c
,
c d
b d
12. An operation is done to get from the first matrix to the second. Identify what was done and
tell how it will affect the value of the determinant.
(
) (
)
a b
c d
,
c d
a b
13. An operation is done to get from the first matrix to the second. Identify what was done and
tell how it will affect the value of the determinant.
(
) (
)
a b
a
b
,
c d
a+c b+d
14. An operation is done to get from the first matrix to the second. Identify what was done and
tell how it will affect the value of the determinant.
(
) (
)
a b
a b
,
c d
2c 2d
6.3. EXERCISES
101
15. An operation is done to get from the first matrix to the second. Identify what was done and
tell how it will affect the value of the determinant.
(
) (
)
a b
b a
,
c d
d c
16. Let A be an r × r matrix and suppose there are r − 1 rows (columns) such that all rows
(columns) are linear combinations of these r − 1 rows (columns). Show det (A) = 0.
17. Show det (aA) = an det (A) where here A is an n × n matrix and a is a scalar.
18. Illustrate with an example of 2 × 2 matrices that the determinant of a product equals the
product of the determinants.
19. Is it true that det (A + B) = det (A) + det (B)? If this is so, explain why it is so and if it is
not so, give a counter example.
20. An n × n matrix is called nilpotent if for some positive integer, k it follows Ak = 0. If A is a
nilpotent matrix and k is the smallest possible integer such that Ak = 0, what are the possible
values of det (A)?
21. A matrix is said to be orthogonal if AT A = I. Thus the inverse of an orthogonal matrix is
just its transpose. What are the possible values of det (A) if A is an orthogonal matrix?
22. Fill in the missing entries to make the matrix orthogonal as in Problem 21.






−1
√
2
√1
2
√1
6
√
6
3
√
12
6



.


23. Let A and B be two n×n matrices. A ∼ B (A is similar to B) means there exists an invertible
matrix S such that A = S −1 BS. Show that if A ∼ B, then B ∼ A. Show also that A ∼ A and
that if A ∼ B and B ∼ C, then A ∼ C.
24. In the context of Problem 23 show that if A ∼ B, then det (A) = det (B) .
25. Two n × n matrices, A and B, are similar if B = S −1 AS for some invertible n × n matrix
S. Show that if two matrices are similar, they have the same characteristic polynomials. The
characteristic polynomial of an n × n matrix M is the polynomial, det (λI − M ) .
26. Tell whether the statement is true or false.
(a) If A is a 3 × 3 matrix with a zero determinant, then one column must be a multiple of
some other column.
(b) If any two columns of a square matrix are equal, then the determinant of the matrix
equals zero.
(c) For A and B two n × n matrices, det (A + B) = det (A) + det (B) .
(d) For A an n × n matrix, det (3A) = 3 det (A)
(
)
−1
(e) If A−1 exists then det A−1 = det (A) .
(f) If B is obtained by multiplying a single row of A by 4 then det (B) = 4 det (A) .
n
(g) For A an n × n matrix, det (−A) = (−1) det (A) .
(
)
(h) If A is a real n × n matrix, then det AT A ≥ 0.
102
CHAPTER 6. DETERMINANTS
(i) Cramer’s rule is useful for finding solutions to systems of linear equations in which there
is an infinite set of solutions.
(j) If Ak = 0 for some positive integer, k, then det (A) = 0.
(k) If Ax = 0 for some x ̸= 0, then det (A) = 0.
27. Use Cramer’s rule to find the solution to x + 2y = 1, 2x − y = 2.
28. Use Cramer’s rule to find the solution to x + 2y + z = 1, 2x − y − z = 2, x + z = 1.
29. Here is a matrix,

1 2

 0 2
3 1

3

1 
0
Determine whether the matrix has an inverse by finding whether the determinant is non zero.
If the determinant is nonzero, find the inverse using the formula for the inverse which involves
the cofactor matrix.
30. Here is a matrix,

1 2

 0 2
3 1

0

1 
1
Determine whether the matrix has an inverse by finding whether the determinant is non zero.
If the determinant is nonzero, find the inverse using the formula for the inverse which involves
the cofactor matrix.
31. Here is a matrix,

1 3

 2 4
0 1

3

1 
1
Determine whether the matrix has an inverse by finding whether the determinant is non zero.
If the determinant is nonzero, find the inverse using the formula for the inverse which involves
the cofactor matrix.
32. Here is a matrix,

1 2

 0 2
2 6

3

1 
7
Determine whether the matrix has an inverse by finding whether the determinant is non zero.
If the determinant is nonzero, find the inverse using the formula for the inverse which involves
the cofactor matrix.
33. Here is a matrix,

1 0

 1 0
3 1

3

1 
0
Determine whether the matrix has an inverse by finding whether the determinant is non zero.
If the determinant is nonzero, find the inverse using the formula for the inverse which involves
the cofactor matrix.
6.3. EXERCISES
103
34. Use the formula for the inverse in terms of
of the matrices

(
)
1
1 1

, 0
1 2
4
the cofactor matrix to find if possible the inverses
 

2 3
1 2 1
 

2 1 , 2 3 0 .
1 1
0 1 2
If the inverse does not exist, explain why.
35. Here is a matrix,

1

 0
0

0
0

cos t − sin t 
sin t cos t
Does there exist a value of t for which this matrix fails to have an inverse? Explain.
36. Here is a matrix,

1 t

 0 1
t 0

t2

2t 
2
Does there exist a value of t for which this matrix fails to have an inverse? Explain.
37. Here is a matrix,

et
 t
 e
et

cosh t sinh t

sinh t cosh t 
cosh t sinh t
Does there exist a value of t for which this matrix fails to have an inverse? Explain.
38. Show that if det (A) ̸= 0 for A an n × n matrix, it follows that if Ax = 0, then x = 0.
39. Suppose A, B are n × n matrices and that AB = I. Show that then BA = I. Hint: You might
do something like this: First explain why det (A) , det (B) are both nonzero. Then (AB) A = A
and then show BA (BA − I) = 0. From this use what is given to conclude A (BA − I) = 0.
Then use Problem 38.
40. Use the formula for the inverse in terms of the cofactor matrix to find the inverse of the matrix


et
0
0


A= 0
et cos t
et sin t
.
t
t
t
t
0 e cos t − e sin t e cos t + e sin t
41. Find the inverse if it exists of the matrix


et
cos t
sin t
 t

 e − sin t cos t  .
et − cos t − sin t
42. Here is a matrix,

et
 t
 e
et

e−t cos t
e−t sin t

−e−t cos t − e−t sin t −e−t sin t + e−t cos t 
2e−t sin t
−2e−t cos t
Does there exist a value of t for which this matrix fails to have an inverse? Explain.
104
CHAPTER 6. DETERMINANTS
43. Suppose A is an upper triangular matrix. Show that A−1 exists if and only if all elements of
the main diagonal are non zero. Is it true that A−1 will also be upper triangular? Explain. Is
everything the same for lower triangular matrices?
44. If A, B, and C are each n × n matrices and ABC is invertible, why are each of A, B, and C
invertible.
(
)
(
)
(
)
a (t) b (t)
a′ (t) b′ (t)
a (t) b (t)
′
. Verify F (t) = det
+det
.
45. Let F (t) = det
c (t) d (t)
c (t) d (t)
c′ (t) d′ (t)
Now suppose


a (t) b (t) c (t)


F (t) = det  d (t) e (t) f (t)  .
g (t) h (t) i (t)
Use Laplace expansion and the first part to verify F ′ (t) =





a (t) b (t) c (t)
a (t) b (t)
a′ (t) b′ (t) c′ (t)

 ′



′
′
det  d (t) e (t) f (t)  + det  d (t) e (t) f (t)  + det  d (t) e (t)
g ′ (t) h′ (t)
g (t) h (t) i (t)
g (t) h (t) i (t)

c (t)

f (t)  .
i′ (t)
Conjecture a general result valid for n × n matrices and explain why it will be true. Can a
similar thing be done with the columns?
46. Let Ly = y (n) + an−1 (x) y (n−1) + · · · + a1 (x) y ′ + a0 (x) y where the ai are given continuous
functions defined on a closed interval, (a, b) and y is some function which has n derivatives so
it makes sense to write Ly. Suppose Lyk = 0 for k = 1, 2, · · · , n. The Wronskian of these
functions, yi is defined as


y1 (x)
···
yn (x)


···
yn′ (x) 
 y1′ (x)

W (y1 , · · · , yn ) (x) ≡ det 
..
..


.
.


(n−1)
(n−1)
(x)
(x) · · · yn
y1
Show that for W (x) = W (y1 , · · · , yn ) (x) to save space,



W (x) = det 


′
y1 (x) · · ·
y1′ (x) · · ·
..
.
(n)
y1 (x) · · ·
yn (x)
yn′ (x)
..
.
(n)
yn (x)



.


Now use the differential equation, Ly = 0 which is satisfied by each of these functions, yi
and properties of determinants presented above to verify that W ′ + an−1 (x) W = 0. Give an
explicit solution of this linear differential equation, Abel’s formula, and use your answer to
verify that the Wronskian of these solutions to the equation, Ly = 0 either vanishes identically
on (a, b) or never. Hint: To solve the differential equation, let A′ (x) = an−1 (x) and multiply
both sides of the differential equation by eA(x) and then argue the left side is the derivative of
something.
47. Find the following determinants.

2

(a) det  2 − 2i
3 + 3i
2 + 2i
5
1 + 7i

3 − 3i

1 − 7i 
16

10
2 + 6i

(b) det  2 − 6i
9
8 + 6i 1 + 7i

8 − 6i

1 − 7i 
17
Chapter 7
The Mathematical Theory Of
Determinants∗
7.0.1
The Function sgn
The following Lemma will be essential in the definition of the determinant.
Lemma 7.0.1 There exists a function, sgnn which maps each ordered list of numbers from {1, · · · , n}
to one of the three numbers, 0, 1, or −1 which also has the following properties.
sgnn (1, · · · , n) = 1
(7.1)
sgnn (i1 , · · · , p, · · · , q, · · · , in ) = − sgnn (i1 , · · · , q, · · · , p, · · · , in )
(7.2)
In words, the second property states that if two of the numbers are switched, the value of the function
is multiplied by −1. Also, in the case where n > 1 and {i1 , · · · , in } = {1, · · · , n} so that every number
from {1, · · · , n} appears in the ordered list, (i1 , · · · , in ) ,
sgnn (i1 , · · · , iθ−1 , n, iθ+1 , · · · , in ) ≡
(−1)
n−θ
sgnn−1 (i1 , · · · , iθ−1 , iθ+1 , · · · , in )
(7.3)
where n = iθ in the ordered list, (i1 , · · · , in ) .
Proof: Define sign (x) = 1 if x > 0, −1 if x < 0 and 0 if x = 0. If n = 1, there is only one list
and it is just the number 1. Thus one can define sgn1 (1) ≡ 1. For the general case where n > 1,
simply define
(
)
∏
sgnn (i1 , · · · , in ) ≡ sign
(is − ir )
r<s
This delivers either −1, 1, or 0 by definition. What about the other claims? Suppose you switch ip
with iq where p < q so two numbers in the ordered list (i1 , · · · , in ) are switched. Denote the new
ordered list of numbers as (j1 , · · · , jn ) . Thus jp = iq and jq = ip and if r ∈
/ {p, q} , jr = ir . See the
following illustration
105
CHAPTER 7. THE MATHEMATICAL THEORY OF DETERMINANTS∗
106
i1
1
i2
2
···
ip
p
···
iq
q
···
in
n
i1
1
i2
2
···
iq
p
···
ip
q
···
in
n
j1
1
j2
2
···
jp
p
···
jq
q
···
jn
n
(
Then
sgnn (j1 , · · · , jn ) ≡ sign
∏
)
(js − jr )
r<s

 both p,q
= sign 
(ip − iq )
z∏
p<j<q
one of p,q
}| ∏
{
(ij − iq )
(ip − ij )

neither
∏ p nor q
r<s,r,s∈{p,q}
/
p<j<q

(is − ir )

∏
The last product consists of the product of terms which were in the un-switched product r<s (is − ir )
so produces no change in sign, while the two products in the middle both introduce q − p − 1 minus
signs. Thus their product produces no change in sign. The first factor is of opposite sign to the
iq − ip which occured in sgnn (i1 , · · · , in ) . Therefore, this switch introduced a minus sign and
sgnn (j1 , · · · , jn ) = − sgnn (i1 , · · · , in )
Now consider the last claim. In computing sgnn (i1 , · · · , iθ−1 , n, iθ+1 , · · · , in ) there will be the
product of n − θ negative terms
(iθ+1 − n) · · · (in − n)
and the other terms in the product for computing sgnn (i1 , · · · , iθ−1 , n, iθ+1 , · · · , in ) are those which
are required to compute sgnn−1 (i1 , · · · , iθ−1 , iθ+1 , · · · , in ) multiplied by terms of the form (n − ij )
which are nonnegative. It follows that
sgnn (i1 , · · · , iθ−1 , n, iθ+1 , · · · , in ) = (−1)
n−θ
sgnn−1 (i1 , · · · , iθ−1 , iθ+1 , · · · , in )
It is obvious that if there are repeats in the list the function gives 0. Lemma 7.0.2 Every ordered list of distinct numbers from {1, 2, · · · , n} can be obtained from every
other such ordered list by a finite number of switches. Also, sgnn is unique.
Proof: This is obvious if n = 1 or 2. Suppose then that it is true for sets of n − 1 elements. Take
two ordered lists of numbers, P1 , P2 . Make one switch in both to place n at the end. Call the result
P1n and P2n . Then using induction, there are finitely many switches in P1n so that it will coincide
with P2n . Now switch the n in what results to where it was in P2 .
To see sgnn is unique, if there exist two functions, f and g both satisfying 7.1 and 7.2, you could
start with f (1, · · · , n) = g (1, · · · , n) = 1 and applying the same sequence of switches, eventually
arrive at f (i1 , · · · , in ) = g (i1 , · · · , in ) . If any numbers are repeated, then 7.2 gives both functions
are equal to zero for that ordered list. Definition 7.0.3 When you have an ordered list of distinct numbers from {1, 2, · · · , n} , say
(i1 , · · · , in ) ,
this ordered list is called a permutation. The symbol for all such permutations is Sn . The number
sgnn (i1 , · · · , in ) is called the sign of the permutation.
7.1. THE DETERMINANT
107
A permutation can also be considered as a function from the set
{1, 2, · · · , n} to {1, 2, · · · , n}
as follows. Let f (k) = ik . Permutations are of fundamental importance in certain areas of math.
For example, it was by considering permutations that Galois was able to give a criterion for solution
of polynomial equations by radicals, but this is a different direction than what is being attempted
here.
In what follows sgn will often be used rather than sgnn because the context supplies the appropriate n.
7.1
The Determinant
Definition 7.1.1 Let f be a function which has the set of ordered lists of numbers from {1, · · · , n}
as its domain. Define
∑
f (k1 · · · kn )
(k1 ,··· ,kn )
to be the sum of all the f (k1 · · · kn ) for all possible choices of ordered lists (k1 , · · · , kn ) of numbers
of {1, · · · , n} . For example,
∑
f (k1 , k2 ) = f (1, 2) + f (2, 1) + f (1, 1) + f (2, 2) .
(k1 ,k2 )
7.1.1
The Definition
Definition 7.1.2 Let (aij ) = A denote an n × n matrix. The determinant of A, denoted by det (A)
is defined by
∑
sgn (k1 , · · · , kn ) a1k1 · · · ankn
det (A) ≡
(k1 ,··· ,kn )
where the sum is taken over all ordered lists of numbers from {1, · · · , n}. Note it suffices to take the
sum over only those ordered lists in which there are no repeats because if there are, sgn (k1 , · · · , kn ) =
0 and so that term contributes 0 to the sum.
7.1.2
Permuting Rows Or Columns
Let A be an n × n matrix, A = (aij ) and let (r1 , · · · , rn ) denote an ordered list of n numbers from
{1, · · · , n}. Let A (r1 , · · · , rn ) denote the matrix whose k th row is the rk row of the matrix A. Thus
∑
det (A (r1 , · · · , rn )) =
sgn (k1 , · · · , kn ) ar1 k1 · · · arn kn
(7.4)
(k1 ,··· ,kn )
and
A (1, · · · , n) = A.
Proposition 7.1.3 Let
(r1 , · · · , rn )
be an ordered list of numbers from {1, · · · , n}. Then
sgn (r1 , · · · , rn ) det (A)
=
∑
sgn (k1 , · · · , kn ) ar1 k1 · · · arn kn
(7.5)
(k1 ,··· ,kn )
=
det (A (r1 , · · · , rn )) .
(7.6)
CHAPTER 7. THE MATHEMATICAL THEORY OF DETERMINANTS∗
108
Proof: Let (1, · · · , n) = (1, · · · , r, · · · s, · · · , n) so r < s.
det (A (1, · · · , r, · · · , s, · · · , n)) =
∑
(7.7)
sgn (k1 , · · · , kr , · · · , ks , · · · , kn ) a1k1 · · · arkr · · · asks · · · ankn ,
(k1 ,··· ,kn )
and renaming the variables, calling ks , kr and kr , ks , this equals
∑
=
sgn (k1 , · · · , ks , · · · , kr , · · · , kn ) a1k1 · · · arks · · · askr · · · ankn
(k1 ,··· ,kn )

∑
=

These got switched
− sgn k1 , · · · ,
z }| {
kr , · · · , ks
, · · · , kn  a1k1 · · · askr · · · arks · · · ankn
(k1 ,··· ,kn )
= − det (A (1, · · · , s, · · · , r, · · · , n)) .
(7.8)
Consequently,
det (A (1, · · · , s, · · · , r, · · · , n)) =
− det (A (1, · · · , r, · · · , s, · · · , n)) = − det (A)
Now letting A (1, · · · , s, · · · , r, · · · , n) play the role of A, and continuing in this way, switching pairs
of numbers,
p
det (A (r1 , · · · , rn )) = (−1) det (A)
where it took p switches to obtain(r1 , · · · , rn ) from (1, · · · , n). By Lemma 7.0.1, this implies
p
det (A (r1 , · · · , rn )) = (−1) det (A) = sgn (r1 , · · · , rn ) det (A)
and proves the proposition in the case when there are no repeated numbers in the ordered list,
(r1 , · · · , rn ). However, if there is a repeat, say the rth row equals the sth row, then the reasoning of
7.7 -7.8 shows that det A (r1 , · · · , rn ) = 0 and also sgn (r1 , · · · , rn ) = 0 so the formula holds in this
case also. Observation 7.1.4 There are n! ordered lists of distinct numbers from {1, · · · , n} .
To see this, consider n slots placed in order. There are n choices for the first slot. For each of
these choices, there are n − 1 choices for the second. Thus there are n (n − 1) ways to fill the first
two slots. Then for each of these ways there are n − 2 choices left for the third slot. Continuing this
way, there are n! ordered lists of distinct numbers from {1, · · · , n} as stated in the observation.
7.1.3
A Symmetric Definition
With the above, it is possible to( give) a more symmetric description of the determinant from which
it will follow that det (A) = det AT .
Corollary 7.1.5 The following formula for det (A) is valid.
det (A) =
∑
∑
(r1 ,··· ,rn ) (k1 ,··· ,kn )
1
·
n!
sgn (r1 , · · · , rn ) sgn (k1 , · · · , kn ) ar1 k1 · · · arn kn .
(7.9)
( )
( )
And also det AT = det (A) where AT is the transpose of A. (Recall that for AT = aTij , aTij = aji .)
7.1. THE DETERMINANT
109
Proof: From Proposition 7.1.3, if the ri are distinct,
∑
det (A) =
sgn (r1 , · · · , rn ) sgn (k1 , · · · , kn ) ar1 k1 · · · arn kn .
(k1 ,··· ,kn )
Summing over all ordered lists, (r1 , · · · , rn ) where the ri are distinct, (If the ri are not distinct,
sgn (r1 , · · · , rn ) = 0 and so there is no contribution to the sum.)
n! det (A) =
∑
∑
sgn (r1 , · · · , rn ) sgn (k1 , · · · , kn ) ar1 k1 · · · arn kn .
(r1 ,··· ,rn ) (k1 ,··· ,kn )
This proves the corollary since the formula gives the same number for A as it does for AT . 7.1.4
The Alternating Property Of The Determinant
Corollary 7.1.6 If two rows or two columns in an n × n matrix A, are switched, the determinant
of the resulting matrix equals (−1) times the determinant of the original matrix. If A is an n × n
matrix in which two rows are equal or two columns are equal then det (A) = 0. Suppose the ith row
of A equals (xa1 + yb1 , · · · , xan + ybn ). Then
det (A) = x det (A1 ) + y det (A2 )
where the ith row of A1 is (a1 , · · · , an ) and the ith row of A2 is (b1 , · · · , bn ) , all other rows of A1
and A2 coinciding with those of A. In other words, det is a linear function of each row A. The same
is true with the word “row” replaced with the word “column”.
Proof: By Proposition 7.1.3 when two rows are switched, the determinant of the resulting matrix
is (−1) times the determinant of the original matrix. By Corollary 7.1.5 the same holds for columns
because the columns of the matrix equal the rows of the transposed matrix. Thus if A1 is the matrix
obtained from A by switching two columns,
( )
( )
det (A) = det AT = − det AT1 = − det (A1 ) .
If A has two equal columns or two equal rows, then switching them results in the same matrix.
Therefore, det (A) = − det (A) and so det (A) = 0.
It remains to verify the last assertion.
∑
det (A) ≡
sgn (k1 , · · · , kn ) a1k1 · · · (xaki + ybki ) · · · ankn
(k1 ,··· ,kn )
=x
∑
sgn (k1 , · · · , kn ) a1k1 · · · aki · · · ankn
(k1 ,··· ,kn )
+y
∑
sgn (k1 , · · · , kn ) a1k1 · · · bki · · · ankn
(k1 ,··· ,kn )
≡ x det (A1 ) + y det (A2 ) .
( )
The same is true of columns because det AT = det (A) and the rows of AT are the columns of A.
CHAPTER 7. THE MATHEMATICAL THEORY OF DETERMINANTS∗
110
7.1.5
Linear Combinations And Determinants
Linear combinations have been discussed already. However, here is a review and some new terminology.
Definition 7.1.7 A vector w, ∑
is a linear combination of the vectors {v1 , · · · , vr } if there exists
r
scalars, c1 , · · · cr such that w = k=1 ck vk . This is the same as saying
w ∈ span (v1 , · · · , vr ) .
The following corollary is also of great use.
Corollary 7.1.8 Suppose A is an n × n matrix and some column (row) is a linear combination of
r other columns (rows). Then det (A) = 0.
(
)
Proof: Let A =
be the columns of A and suppose the condition that one
a1 · · · an
column is a linear combination of r of the others is satisfied. Then by using Corollary 7.1.6 the
determinant of A is zero if and only if the determinant of∑the matrix B, which has this special
r
column placed in the last position, equals zero. Thus an = k=1 ck ak and so
(
)
∑r
det (B) = det a1 · · · ar · · · an−1
.
c
a
k
k
k=1
By Corollary 7.1.6
r
∑
det (B) =
(
ck det
···
a1
ar
···
)
an−1
ak
= 0.
k=1
because
( )there are two equal columns. The case for rows follows from the fact that det (A) =
det AT . 7.1.6
The Determinant Of A Product
Recall the following definition of matrix multiplication.
Definition 7.1.9 If A and B are n × n matrices, A = (aij ) and B = (bij ), AB = (cij ) where
cij ≡
n
∑
aik bkj .
k=1
One of the most important rules about determinants is that the determinant of a product equals
the product of the determinants.
Theorem 7.1.10 Let A and B be n × n matrices. Then
det (AB) = det (A) det (B) .
Proof: Let cij be the ij th entry of AB. Then by Proposition 7.1.3,
det (AB) =
∑
sgn (k1 , · · · , kn ) c1k1 · · · cnkn
(k1 ,··· ,kn )
=
∑
sgn (k1 , · · · , kn )
(k1 ,··· ,kn )
=
∑
∑
(
∑
r1
)
a1r1 br1 k1
(
···
∑
)
anrn brn kn
rn
sgn (k1 , · · · , kn ) br1 k1 · · · brn kn (a1r1 · · · anrn )
(r1 ··· ,rn ) (k1 ,··· ,kn )
=
∑
(r1 ··· ,rn )
sgn (r1 · · · rn ) a1r1 · · · anrn det (B) = det (A) det (B) . 7.1. THE DETERMINANT
7.1.7
111
Cofactor Expansions
Lemma 7.1.11 Suppose a matrix is of the form
(
M=
(
or
M=
A ∗
0 a
A 0
∗ a
)
(7.10)
)
(7.11)
where a is a number and A is an (n − 1) × (n − 1) matrix and ∗ denotes either a column or a row
having length n − 1 and the 0 denotes either a column or a row of length n − 1 consisting entirely
of zeros. Then det (M ) = a det (A) .
Proof: Denote M by (mij ) . Thus in the first case, mnn = a and mni = 0 if i ̸= n while in the
second case, mnn = a and min = 0 if i ̸= n. From the definition of the determinant,
∑
det (M ) ≡
sgnn (k1 , · · · , kn ) m1k1 · · · mnkn
(k1 ,··· ,kn )
Letting θ denote the position of n in the ordered list, (k1 , · · · , kn ) then using Lemma 7.0.1, det (M )
equals
(
)
∑
θ
n−1
n−θ
(−1)
sgnn−1 k1 , · · · , kθ−1 , kθ+1 , · · · , kn m1k1 · · · mnkn
(k1 ,··· ,kn )
Now suppose 7.11. Then if kn ̸= n, the term involving mnkn in the above expression equals zero.
Therefore, the only terms which survive are those for which θ = n or in other words, those for which
kn = n. Therefore, the above expression reduces to
∑
a
sgnn−1 (k1 , · · · kn−1 ) m1k1 · · · m(n−1)kn−1 = a det (A) .
(k1 ,··· ,kn−1 )
To get the assertion in the situation of 7.10 use Corollary 7.1.5 and 7.11 to write
((
))
( T)
( )
AT 0
det (M ) = det M = det
= a det AT = a det (A) .
∗ a
In terms of the theory of determinants, arguably the most important idea is that of Laplace
expansion along a row or a column. This will follow from the above definition of a determinant.
Definition 7.1.12 Let A = (aij ) be an n × n matrix. Then a new matrix called the cofactor matrix,
cof (A) is defined by cof (A) = (cij ) where to obtain cij delete the ith row and the j th column of A,
take the determinant of the (n − 1) × (n − 1) matrix which results, (This is called the ij th minor of
i+j
A. ) and then multiply this number by (−1) . To make the formulas easier to remember, cof (A)ij
will denote the ij th entry of the cofactor matrix.
The following is the main result. Earlier this was given as a definition and the outrageous
totally unjustified assertion was made that the same number would be obtained by expanding the
determinant along any row or column. The following theorem proves this assertion.
Theorem 7.1.13 Let A be an n × n matrix where n ≥ 2. Then
det (A) =
n
∑
j=1
aij cof (A)ij =
n
∑
aij cof (A)ij .
(7.12)
i=1
The first formula consists of expanding the determinant along the ith row and the second expands
the determinant along the j th column.
CHAPTER 7. THE MATHEMATICAL THEORY OF DETERMINANTS∗
112
Proof: Let (ai1 , · · · , ain ) be the ith row of A. Let Bj be the matrix obtained from A by leaving
every row the same except the ith row which in Bj equals
(0, · · · , 0, aij , 0, · · · , 0) .
Then by Corollary 7.1.6,
det (A) =
n
∑
det (Bj )
j=1
Denote by Aij the (n − 1) × (n − 1) matrix obtained by deleting the ith row and the j th column of
( )
i+j
A. Thus cof (A)ij ≡ (−1) det Aij . At this point, recall that from Proposition 7.1.3, when two
rows or two columns in a matrix M, are switched, this results in multiplying the determinant of the
old matrix by −1 to get the determinant of the new matrix. Therefore, by Lemma 7.1.11,
((
))
Aij ∗
n−j
n−i
det (Bj ) = (−1)
(−1)
det
0 aij
((
))
Aij ∗
i+j
= (−1) det
= aij cof (A)ij .
0 aij
Therefore,
det (A) =
n
∑
aij cof (A)ij
j=1
which is the formula for expanding det (A) along the ith row. Also,
det (A) =
n
( )
( ) ∑
det AT =
aTij cof AT ij
j=1
=
n
∑
aji cof (A)ji
j=1
which is the formula for expanding det (A) along the ith column. 7.1.8
Formula For The Inverse
Note that this gives an easy way to write a formula for the inverse of an n × n matrix.
(
)
Theorem 7.1.14 A−1 exists if and only if det(A) ̸= 0. If det(A) ̸= 0, then A−1 = a−1
where
ij
−1
a−1
cof (A)ji
ij = det(A)
for cof (A)ij the ij th cofactor of A.
Proof: By Theorem 7.1.13 and letting (air ) = A, if det (A) ̸= 0,
n
∑
air cof (A)ir det(A)−1 = det(A) det(A)−1 = 1.
i=1
Now consider
n
∑
air cof (A)ik det(A)−1
i=1
when k ̸= r. Replace the k th column with the rth column to obtain a matrix Bk whose determinant
equals zero by Corollary 7.1.6. However, expanding this matrix along the k th column yields
0 = det (Bk ) det (A)
−1
=
n
∑
i=1
air cof (A)ik det (A)
−1
7.1. THE DETERMINANT
Summarizing,
113
n
∑
air cof (A)ik det (A)
−1
= δ rk .
i=1
Using the other formula in Theorem 7.1.13, and similar reasoning,
n
∑
−1
arj cof (A)kj det (A)
= δ rk
j=1
(
)
This proves that if det (A) ̸= 0, then A−1 exists with A−1 = a−1
ij , where
−1
a−1
ij = cof (A)ji det (A)
.
Now suppose A−1 exists. Then by Theorem 7.1.10,
(
)
(
)
1 = det (I) = det AA−1 = det (A) det A−1
so det (A) ̸= 0. The next corollary points out that if an n × n matrix A has a right or a left inverse, then it has
an inverse.
Corollary 7.1.15 Let A be an n × n matrix and suppose there exists an n × n matrix B such that
BA = I. Then A−1 exists and A−1 = B. Also, if there exists C an n × n matrix such that AC = I,
then A−1 exists and A−1 = C.
Proof: Since BA = I, Theorem 7.1.10 implies
det B det A = 1
and so det A ̸= 0. Therefore from Theorem 7.1.14, A−1 exists. Therefore,
(
)
A−1 = (BA) A−1 = B AA−1 = BI = B.
The case where CA = I is handled similarly. The conclusion of this corollary is that left inverses, right inverses and inverses are all the same
in the context of n × n matrices.
Theorem 7.1.14 says that to find the inverse, take the transpose of the cofactor matrix and
divide by the determinant. The transpose of the cofactor matrix is called the adjugate or sometimes
the classical adjoint of the matrix A. It is an abomination to call it the adjoint although you do
sometimes see it referred to in this way. In words, A−1 is equal to one over the determinant of A
times the adjugate matrix of A.
7.1.9
Cramer’s Rule
In case you are solving a system of equations, Ax = y for x, it follows that if A−1 exists,
(
)
x = A−1 A x = A−1 (Ax) = A−1 y
thus solving the system. Now in the case that A−1 exists, there is a formula for A−1 given above.
Using this formula,
n
n
∑
∑
1
cof (A)ji yj .
xi =
a−1
y
=
ij j
det (A)
j=1
j=1
By the formula for the expansion of a determinant along a column,


∗ · · · y1 · · · ∗
 .
1
..
.. 

..
xi =
det 
.
. ,

det (A)
∗ · · · yn · · · ∗
T
where here the ith column of A is replaced with the column vector (y1 · · · ·, yn ) , and the determinant
of this modified matrix is taken and divided by det (A). This formula is known as Cramer’s rule.
CHAPTER 7. THE MATHEMATICAL THEORY OF DETERMINANTS∗
114
7.1.10
Upper Triangular Matrices
Definition 7.1.16 A matrix M , is upper triangular if
equals zero below the main diagonal, the entries of the

∗ ∗ ···

..
 0 ∗
.

 .
..
 . ..
.
.
 .
0 ···
0
Mij = 0 whenever i > j. Thus such a matrix
form Mii as shown.

∗
.. 
. 



∗ 
∗
A lower triangular matrix is defined similarly as a matrix for which all entries above the main
diagonal are equal to zero.
With this definition, here is a simple corollary of Theorem 7.1.13.
Corollary 7.1.17 Let M be an upper (lower) triangular matrix. Then det (M ) is obtained by taking
the product of the entries on the main diagonal.
7.2
The Cayley Hamilton Theorem∗
Definition 7.2.1 Let A be an n × n matrix. The characteristic polynomial is defined as
qA (t) ≡ det (tI − A)
and the solutions to pA (t) = 0 are called eigenvalues. For A a matrix and p (t) = tn + an−1 tn−1 +
· · · + a1 t + a0 , denote by p (A) the matrix defined by
p (A) ≡ An + an−1 An−1 + · · · + a1 A + a0 I.
The explanation for the last term is that A0 is interpreted as I, the identity matrix.
The Cayley Hamilton theorem states that every matrix satisfies its characteristic equation, that
equation defined by pA (t) = 0. It is one of the most important theorems in linear algebra1 . The
proof in this section is not the most general proof, but works well when the field of scalars is R or
C. The following lemma will help with its proof.
Lemma 7.2.2 Suppose for all |λ| large enough,
A0 + A1 λ + · · · + Am λm = 0,
where the Ai are n × n matrices. Then each Ai = 0.
Proof: Multiply by λ−m to obtain
A0 λ−m + A1 λ−m+1 + · · · + Am−1 λ−1 + Am = 0.
Now let |λ| → ∞ to obtain Am = 0. With this, multiply by λ to obtain
A0 λ−m+1 + A1 λ−m+2 + · · · + Am−1 = 0.
Now let |λ| → ∞ to obtain Am−1 = 0. Continue multiplying by λ and letting λ → ∞ to obtain that
all the Ai = 0. With the lemma, here is a simple corollary.
1 A special case was first proved by Hamilton in 1853. The general case was announced by Cayley some time later
and a proof was given by Frobenius in 1878.
7.2. THE CAYLEY HAMILTON THEOREM∗
115
Corollary 7.2.3 Let Ai and Bi be n × n matrices and suppose
A0 + A1 λ + · · · + Am λm = B0 + B1 λ + · · · + Bm λm
for all |λ| large enough. Then Ai = Bi for all i. If Ai = Bi for each Ai , Bi then one can substitute
an n × n matrix M for λ and the identity will continue to hold.
Proof: Subtract and use the result of the lemma. The last claim is obvious by matching terms.
With this preparation, here is a relatively easy proof of the Cayley Hamilton theorem.
Theorem 7.2.4 Let A be an n × n matrix and let q (λ) ≡ det (λI − A) be the characteristic polynomial. Then q (A) = 0.
Proof: Let C (λ) equal the transpose of the cofactor matrix of (λI − A) for |λ| large. (If |λ| is
−1
large enough, then λ cannot be in the finite list of eigenvalues of A and so for such λ, (λI − A)
exists.) Therefore, by Theorem 7.1.14
C (λ) = q (λ) (λI − A)
−1
.
Say
q (λ) = a0 + a1 λ + · · · + λn
Note that each entry in C (λ) is a polynomial in λ having degree no more than n − 1. For example,
you might have something like
 2

λ − 6λ + 9
3−λ
0


C (λ) = 
2λ − 6
λ2 − 3λ
0

λ−1
λ−1
λ2 − 3λ + 2

9

=  −6
−1


3 0
−6


0 0  + λ 2
−1 2
1


−1 0
1 0

2
−3 0  + λ  0 1
1 −3
0 0

0

0 
1
Therefore, collecting the terms in the general case,
C (λ) = C0 + C1 λ + · · · + Cn−1 λn−1
for Cj some n × n matrix. Then
(
)
C (λ) (λI − A) = C0 + C1 λ + · · · + Cn−1 λn−1 (λI − A) = q (λ) I
Then multiplying out the middle term, it follows that for all |λ| sufficiently large,
a0 I + a1 Iλ + · · · + Iλn = C0 λ + C1 λ2 + · · · + Cn−1 λn
[
]
− C0 A + C1 Aλ + · · · + Cn−1 Aλn−1
= −C0 A + (C0 − C1 A) λ + (C1 − C2 A) λ2 + · · · + (Cn−2 − Cn−1 A) λn−1 + Cn−1 λn
Then, using Corollary 7.2.3, one can replace λ on both sides with A. Then the right side is seen to
equal 0. Hence the left side, q (A) I is also equal to 0. It is good to keep in mind the following example when considering the above proof of the Cayley
Hamilton theorem. It was shown to me by Marc van Leeuwen. If p (λ) = q (λ) for all λ or for all λ
large enough where p (λ) , q (λ) are polynomials having matrix coefficients, then it is not necessarily
the case that p (A) = q (A) for A a matrix of an appropriate size. Let
(
)
(
)
(
)
1 0
0 0
0 1
E1 =
, E2 =
,N =
0 0
0 1
0 0
116
CHAPTER 7. THE MATHEMATICAL THEORY OF DETERMINANTS∗
Then a short computation shows that for all complex λ,
(
)
(λI + E1 ) (λI + E2 ) = λ2 + λ I = (λI + E2 ) (λI + E1 )
However,
(N I + E1 ) (N I + E2 ) ̸= (N I + E2 ) (N I + E1 )
The reason this can take place is that N fails to commute with Ei . Of course a scalar commutes
with any matrix so there was no difficulty in obtaining that the matrix equation held for arbitrary λ,
but this factored equation does not continue to hold if λ is replaced by a matrix. In the above proof
of the Cayley Hamilton theorem, this issue was avoided by considering only polynomials which are
of the form C0 + C1 λ + · · · in which the polynomial identity held because the corresponding matrix
coefficients were equal. However, you can also argue that in the above proof, the Ci each commute
with A. Nevertheless, an earlier proof of the Cayley Hamilton theorem using this approach was
misleading because this issue was not made clear.
Chapter 8
Rank Of A Matrix
8.1
Elementary Matrices
The elementary matrices result from doing a row operation to the identity matrix.
Definition 8.1.1 The row operations consist of the following
1. Switch two rows.
2. Multiply a row by a nonzero number.
3. Replace a row by a multiple of another row added to it.
The elementary matrices are given in the following definition.
Definition 8.1.2 The elementary matrices consist of those matrices which result by applying a
row operation to an identity matrix. Those which involve switching rows of the identity are called
permutation matrices1 .
As an example of why these elementary matrices


0 1 0
a b c d


 1 0 0  x y z w
0 0 1
f g h i
are interesting, consider the following.
 

x y z w
 

= a b c d 
f g h i
A 3 × 4 matrix was multiplied on the left by an elementary matrix which was obtained from row
operation 1 applied to the identity matrix. This resulted in applying the operation 1 to the given
matrix. This is what happens in general.
Now consider what these elementary matrices look like. First consider the one which involves
switching row i and row j where i < j. This matrix is of the form


1
0


..


.






0
1




.
..






1
0




..


.


0
1
1 More generally, a permutation matrix is a matrix which comes by permuting the rows of the identity matrix, not
just switching two rows.
117
118
CHAPTER 8. RANK OF A MATRIX
Note how the ith and j th rows are switched in the identity matrix and there are thus all ones on the
main diagonal except for those two positions indicated. The two exceptional rows are shown. The
ith row was the j th and the j th row was the ith in the identity matrix. Now consider what this does
to a column vector.


 

1
0
x1
x1

 .   . 
..

  ..   .. 
.


 



 


  xi   xj 
0
1


 


  ..   .. 
..
=






.

 .   . 






1
0

  xj   xi 

 .   . 
..

 .   . 
.

 .   . 
0
1
xn
xn
Now denote by P ij the elementary matrix which comes from the identity from switching rows i
and j. From what was just explained consider multiplication on the left by this elementary matrix.


a11 a12 · · · a1p
 .
..
.. 
 ..
.
. 




 ai1 ai2 · · · aip 


 .
..
.. 
P ij  ..
.
. 


 a

 j1 aj2 · · · ajp 
 .
..
.. 
 .

.
. 
 .
an1 an2 · · · anp
From the way you multiply matrices this is a matrix which has the indicated columns.






 
 


a11
a12
a1p
a11
a12

 . 
 . 
 .   .   . 


 .. 
 .. 
 ..   ..   .. 







 
 








 
 



 ai1 
 ai2 
 aip   aj1   aj2 







 
 


 ij  ..  ij  .. 

..   ..   .. 
ij 
P  .  , P  .  , · · · , P  .  =  .  ,  .  , · · · , 






 
 



 a 
 a 
 a   a   a 


 j1 
 j2 
 jp   i1   i2 


 . 
 . 
 .   .   . 


 . 
 . 
 .   .   . 


 . 
 . 
 .   .   . 

an1
an2
anp
an1
an2


a11 a12 · · · a1p
 .
..
.. 
 ..
.
. 




 aj1 aj2 · · · ajp 


 .
..
.. 
=  ..
.
. 


 a

 i1 ai2 · · · aip 
 .
..
.. 
 .

.
. 
 .
an1 an2 · · · anp
a1p
..
.
ajp
..
.
aip
..
.
anp















This has established the following lemma.
Lemma 8.1.3 Let P ij denote the elementary matrix which involves switching the ith and the j th
rows. Then
P ij A = B
where B is obtained from A by switching the ith and the j th rows.
8.1. ELEMENTARY MATRICES
119
Example 8.1.4 Consider the following.


 
0 1 0
a b
g


 
=
 1 0 0  g d   a
0 0 1
e f
e

d

b 
f
Next consider the row operation which involves multiplying the ith row by a nonzero constant,
c. The elementary matrix which results from applying this operation to the ith row of the identity
matrix is of the form


1
0


..


.






1




c




1




..


.


0
Now consider what this does to a column vector.


1
0
v1

 .
.

  ..
..





  vi−1
1



  vi
c




1

  vi+1

 .
..

 .
.

 .
0
1
vn
1


 
 
 
 
 
 
=
 
 
 
 
 
 
v1
..
.
vi−1
cvi
vi+1
..
.
vn














Denote by E (c, i) this elementary matrix which multiplies the ith row of the identity by the nonzero
constant, c. Then from what was just discussed and the way matrices are multiplied,


a11 a12 · · · a1p
 .
..
.. 
 ..
.
. 




 ai1 ai2 · · · aip 


 .
..
.. 
E (c, i)  ..

.
.


 a

 j2 aj2 · · · ajp 
 .
..
.. 
 .

.
. 
 .
an1 an2 · · · anp
equals a matrix having the columns


a11

 .

 ..





 ai1



 .
= E (c, i)  ..



 a

 j1

 .

 .

 .
an1
indicated below.




a12

 . 


 .. 










 ai2 






 .. 

 , E (c, i)  .  , · · · , E (c, i) 





 a 


 j2 


 . 


 . 


 . 

an2
a1p
..
.
aip
..
.
ajp
..
.
anp















120
CHAPTER 8. RANK OF A MATRIX








=






a11
..
.
cai1
..
.
aj2
..
.
an1
a12
..
.
cai2
..
.
aj2
..
.
an2
···
···
···
···
a1p
..
.
caip
..
.
ajp
..
.
anp















This proves the following lemma.
Lemma 8.1.5 Let E (c, i) denote the elementary matrix corresponding to the row operation in which
the ith row is multiplied by the nonzero constant, c. Thus E (c, i) involves multiplying the ith row of
the identity matrix by c. Then
E (c, i) A = B
where B is obtained from A by multiplying the ith row of A by c.
Example 8.1.6 Consider this.


1 0 0
a


 0 5 0  c
0 0 1
e
 
b
a
 
d  =  5c
f
e

b

5d 
f
Finally consider the third of these row operations. Denote by E (c × i + j) the elementary matrix
which replaces the j th row with itself added to c times the ith row added to it. In case i < j this
will be of the form


1
0


..


.






1




..


.




c
1




..


.


0
0 1
Now consider what this does to a column vector.


 
1
0
v1
v1

 .  
..
.

  ..  
..
.


 


 

  vi  
1
vi


 

  ..  
..
..

 .  = 
.
.


 





c
1

  vj   cvi + vj

 .  
..
..

 .  
.
.

 .  
0
0 1
vn
vn















8.1. ELEMENTARY MATRICES
121
Now from this and the way matrices are multiplied,

a11 a12
 .
..
 ..
.


 ai1 ai2

 .
..
E (c × i + j)  ..
.

 a
 j2 aj2
 .
..
 .
.
 .
an1 an2
···
···
···
···
a1p
..
.
aip
..
.
ajp
..
.
anp















equals a matrix of the following form having the indicated columns.






a12
a11

 . 
 . 


 .. 
 .. 














 ai2 
 ai1 








 .. 

 .. 
E (c × i + j)  .  , E (c × i + j)  .  , · · · E (c × i + j) 







 a 
 a 


 j2 

 j2 

 . 
 . 


 . 
 . 


 . 
 . 

an2
an1








=






a11
..
.
ai1
..
.
aj2 + cai1
..
.
an1
a12
..
.
ai2
..
.
aj2 + cai2
..
.
an2
···
···
···
···
a1p
..
.
aip
..
.
ajp + caip
..
.
anp
a1p
..
.
aip
..
.
ajp
..
.
anp






























The case where i > j is handled similarly. This proves the following lemma.
Lemma 8.1.7 Let E (c × i + j) denote the elementary matrix obtained from I by replacing the j th
row with c times the ith row added to it. Then
E (c × i + j) A = B
where B is obtained from A by replacing the j th row of A with itself added to c times the ith row of
A.
Example 8.1.8 Consider the

1

 0
2
third row operation.

 

0 0
a b
a
b

 

1 0  c d  = 
c
d

0 1
e f
2a + e 2b + f
The next theorem is the main result.
Theorem 8.1.9 To perform any of the three row operations on a matrix A it suffices to do the row
operation on the identity matrix obtaining an elementary matrix E and then take the product, EA.
Furthermore, each elementary matrix is invertible and its inverse is an elementary matrix.
122
CHAPTER 8. RANK OF A MATRIX
Proof: The first part of this theorem has been proved in Lemmas 8.1.3 - 8.1.7. It only remains
to verify the claim about the inverses. Consider first the elementary matrices corresponding to row
operation of type three.
E (−c × i + j) E (c × i + j) = I
This follows because the first matrix takes c times row i in the identity and adds it to row j. When
multiplied on the left by E (−c × i + j) it follows from the first part of this theorem that you take
the ith row of E (c × i + j) which coincides with the ith row of I since that row was not changed,
multiply it by −c and add to the j th row of E (c × i + j) which was the j th row of I added to c
times the ith row of I. Thus E (−c × i + j) multiplied on the left, undoes the row operation which
resulted in E (c × i + j). The same argument applied to the product
E (c × i + j) E (−c × i + j)
replacing c with −c in the argument yields that this product is also equal to I. Therefore,
−1
E (c × i + j)
= E (−c × i + j) .
Similar reasoning shows that for E (c, i) the elementary matrix which comes from multiplying
the ith row by the nonzero constant, c,
(
)
−1
E (c, i) = E c−1 , i .
Finally, consider P ij which involves switching the ith and the j th rows.
P ij P ij = I
because by the first part of this theorem, multiplying on the left by P ij switches the ith and j th rows
of P ij which was obtained from switching the ith and j th rows of the identity. First you switch them
to get P ij and then you multiply on the left by P ij which switches these rows again and restores
( )−1
the identity matrix. Thus P ij
= P ij . The geometric significance
( of these
) elementary operations is interesting. The following picture
shows the effect of doing E 12 × 3 + 1 on a box. You will see that it shears the box in one direction.
Of course there would be corresponding shears in the other directions also. Note that this does not
change the volume.
z
6
z
6
-y
x
-y
x
The other elementary matrices have similar simple geometric interpretations. For example,
E (c, i) merely multiplies the ith variable by c. It stretches or contracts the box in that direction. If
c is negative, it also causes the box to be reflected in this direction. The following picture illustrates
the effect of P 13 on a box in three dimensions. It changes the x and the z values.
z
6
z
6
-y
x
-y
x
8.2. THE ROW REDUCED ECHELON FORM OF A MATRIX
8.2
123
THE Row Reduced Echelon Form Of A Matrix
Recall that putting a matrix in row reduced echelon form involves doing row operations as described
on Page 49. In this section we review the description of the row reduced echelon form and prove the
row reduced echelon form for a given matrix is unique. That is, every matrix can be row reduced to
a unique row reduced echelon form. Of course this is not true of the echelon form. The significance
of this is that it becomes possible to use the definite article in referring to the row reduced echelon
form and hence important conclusions about the original matrix may be logically deduced from an
examination of its unique row reduced echelon form. First we need the following definition of some
terminology.
Definition 8.2.1 Let v1 , · · · , vk , u be vectors. Then u is said to be a linear combination of the
vectors {v1 , · · · , vk } if there exist scalars, c1 , · · · , ck such that
u=
k
∑
ci vi .
i=1
The collection of all linear combinations of the vectors, {v1 , · · · , vk } is known as the span of these
vectors and is written as span (v1 , · · · , vk ).
Another way to say the same thing as expressed in the earlier definition of row reduced echelon
form found on Page 48 is the following which is a more useful description when proving the major
assertions about the row reduced echelon form.
Definition 8.2.2 Let ei denote the column vector which has all zero entries except for the ith slot
which is one. An m × n matrix is said to be in row reduced echelon form if, in viewing successive
columns from left to right, the first nonzero column encountered is e1 and if you have encountered
e1 , e2 , · · · , ek , the next column is either ek+1 or is a linear combination of the vectors, e1 , e2 , · · · , ek .
Theorem 8.2.3 Let A be an m × n matrix. Then A has a row reduced echelon form determined by
a simple process.
Proof: Viewing the columns of A from left to right take the first nonzero column. Pick a nonzero
entry in this column and switch the row containing this entry with the top row of A. Now divide
this new top row by the value of this nonzero entry to get a 1 in this position and then use row
operations to make all entries below this equal to zero. Thus the first nonzero column is now e1 .
Denote the resulting matrix by A1 . Consider the sub-matrix of A1 to the right of this column and
below the first row. Do exactly the same thing for this sub-matrix that was done for A. This time
the e1 will refer to Fm−1 . Use the first 1 obtained by the above process which is in the top row of
this sub-matrix and row operations to zero out every entry above it in the rows of A1 . Call the
resulting matrix A2 . Thus A2 satisfies the conditions of the above definition up to the column just
encountered. Continue this way till every column has been dealt with and the result must be in row
reduced echelon form. The following diagram illustrates the above procedure. Say the matrix looked something like the
following.


0 ∗ ∗ ∗ ∗ ∗ ∗


 0 ∗ ∗ ∗ ∗ ∗ ∗ 
 . . . . . . . 
 . . . . . . . 
 . . . . . . . 
0 ∗ ∗ ∗ ∗ ∗ ∗
First step would yield something like






0 1
0 0
.. ..
. .
0 0
∗ ∗
∗ ∗
.. ..
. .
∗ ∗
∗ ∗
∗ ∗
.. ..
. .
∗ ∗
∗
∗
..
.
∗






124
CHAPTER 8. RANK OF A MATRIX
For the second step you look at the lower right

∗ ∗
 . .
 . .
 . .
∗ ∗
corner as described,

∗ ∗ ∗
.. .. .. 

. . . 
∗ ∗ ∗
and if the first column consists of all zeros but the next one is not all zeros, you would get something
like this.


0 1 ∗ ∗ ∗
 . . . . . 
 . . . . . 
 . . . . . 
0 0 ∗ ∗ ∗
Thus, after zeroing out the term in the top row above the 1, you get the following for the next step
in the computation of the row reduced echelon form for the original matrix.


0 1 ∗ 0 ∗ ∗ ∗


 0 0 0 1 ∗ ∗ ∗ 
 . . . . . . . .
 . . . . . . . 
 . . . . . . . 
0 0 0 0 ∗ ∗ ∗
Next you look at the lower right matrix below the top two rows and to the right of the first four
columns and repeat the process.
Recall the following definition which was discussed earlier.
Definition 8.2.4 The first pivot column of A is the first nonzero column of A. The next pivot
column is the first column after this which becomes e2 in the row reduced echelon form. The third is
the next column which becomes e3 in the row reduced echelon form and so forth.
There are three choices for row operations at each step in the above theorem. A natural question
is whether the same row reduced echelon matrix always results in the end from following the above
algorithm applied in any way. The next corollary says this is the case but first, here is a fundamental
lemma.
In rough terms, the following lemma states that linear relationships between columns in a
matrix are preserved by row operations. This simple lemma is the main result in understanding all
the major questions related to the row reduced echelon form as well as many other topics.
Lemma 8.2.5 Let A and B be two m × n matrices and suppose B results from a row operation
applied to A. Then the k th column of B is a linear combination of the i1 , · · · , ir columns of B if and
only if the k th column of A is a linear combination of the i1 , · · · , ir columns of A. Furthermore, the
scalars in the linear combination are the same. (The linear relationship between the k th column of
A and the i1 , · · · , ir columns of A is the same as the linear relationship between the k th column of
B and the i1 , · · · , ir columns of B.)
Proof: Let A equal the following matrix in which the ak are the columns
(
)
a1 a2 · · · an
and let B equal the following matrix in which the columns are given by the bk
(
)
b1 b2 · · · bn
Then by Theorem 8.1.9 on Page 121 bk = Eak where E is an elementary matrix. Suppose then
that one of the columns of A is a linear combination of some other columns of A. Say
∑
ak =
cr ar .
r∈S
8.2. THE ROW REDUCED ECHELON FORM OF A MATRIX
Then multiplying by E,
bk = Eak =
∑
r∈S
cr Ear =
∑
125
cr br . r∈S
Definition 8.2.6 Two matrices are said to be row equivalent if one can be obtained from the
other by a sequence of row operations.
It has been shown above that every
echelon form. Note

x1
 .
 .
 .
xn
matrix is row equivalent to one which is in row reduced


 = x1 e1 + · · · + xn en

so to say two column vectors are equal is to say they are the same linear combination of the special
vectors ej .
Thus the row reduced echelon form is completely determined by the positions of columns which
are not linear combinations of preceding columns (These become the ei vectors in the row reduced
echelon form.) and the scalars which are used in the linear combinations of these special pivot
columns to obtain the other columns. All of these considerations pertain only to linear relations
between the columns of the matrix, which by Lemma 8.2.5 are all preserved. Therefore, there is
only one row reduced echelon form for any given matrix. The proof of the following corollary is just
a more careful exposition of this simple idea.
Corollary 8.2.7 The row reduced echelon form is unique. That is if B, C are two matrices in row
reduced echelon form and both are row equivalent to A, then B = C.
Proof: Suppose B and C are both row reduced echelon forms for the matrix A. Then they
clearly have the same zero columns since row operations leave zero columns unchanged. In reading from left to right in B, suppose e1 , · · · , er occur first in positions i1 , · · · , ir respectively. The
description of the row reduced echelon form means that each of these columns is not a linear combination of the preceding columns. Therefore, by Lemma 8.2.5, the same is true of the columns in
positions i1 , i2 , · · · , ir for C. It follows from the description of the row reduced echelon form that
in C, e1 , · · · , er occur first in positions i1 , i2 , · · · , ir . Therefore, both B and C have the sequence
e1 , e2 , · · · , er occurring first in the positions, i1 , i2 , · · · , ir . By Lemma 8.2.5, the columns between
the ik and ik+1 position in the two matrices are linear combinations involving the same scalars of the
columns in the i1 , · · · , ik position. Also the columns after the ir position are linear combinations of
the columns in the i1 , · · · , ir positions involving the same scalars in both matrices. This is equivalent
to the assertion that each of these columns is identical and this proves the corollary. The above corollary shows that you can determine whether two matrices are row equivalent by
simply checking their row reduced echelon forms. The matrices are row equivalent if and only if
they have the same row reduced echelon form.
Now with the above corollary, here is a very fundamental observation. It concerns a matrix which
looks like this: (More columns than rows.)
Corollary 8.2.8 Suppose A is an m × n matrix and that m < n. That is, the number of rows is less
than the number of columns. Then one of the columns of A is a linear combination of the preceding
columns of A. Also, there exists a nonzero solution x to the equation Ax = 0.
Proof: Since m < n, not all the columns of A can be pivot columns. In reading from left to
right, pick the first one which is not a pivot column. Then from the description of the row reduced
126
CHAPTER 8. RANK OF A MATRIX
echelon form, this column is a linear combination of the preceding columns. Denote the j th column
of A by aj . Thus for some k > 1,
ak =
k−1
∑
xj aj , so
j=1
k−1
∑
xj aj + (−1) ak = 0
j=1
T
Let x = (x1 , · · · , xk−1 , −1, 0, · · · , 0) . Then Ax = 0. Example 8.2.9 Find the row reduced echelon

0

 0
0
form of the matrix

0 2 3

2 0 1 
1 1 5
The first nonzero column is the second in
obtain

0

 0
0
the matrix. We switch the third and first rows to

1 1 5

2 0 1 
0 2 3
Now we multiply the top row by −2 and add to

0 1

 0 0
0 0
Next, add the second row to the bottom

0

 0
0
the second.

1
5

−2 −9 
2
3
and then divide the bottom row by −6

1 1
5

0 −2 −9 
0 0
1
Next use the bottom row to obtain zeros in the last column above the 1 and divide the second row
by −2


0 1 1 0


 0 0 1 0 
0 0 0 1
Finally, add −1 times the middle row to the top.

0 1 0

 0 0 1
0 0 0

0

0 .
1
This is in row reduced echelon form.
Example 8.2.10 Find the row reduced echelon form

1 2 0

 −1 3 4
0 5 4
for the matrix

2

3 
5
8.3.
THE RANK OF A MATRIX
127
II
You should verify that the row reduced echelon form

1 0 − 85 0

4
1
 0 1
5
0 0 0 0
is


.
Having developed the row reduced echelon form, it is now easy to verify that the right inverse
found earlier using the Gauss Jordan procedure is the inverse.
Theorem 8.2.11 Suppose A, B are n × n matrices and AB = I. Then it follows that BA = I
also, and so B = A−1 . For n × n matrices, the left inverse, right inverse and inverse are all the
same thing. Furthermore, if A−1 exists, then it can be found using the technique of row operations
described earlier.
Proof. If AB = I for A, B n × n matrices, is BA = I? If AB = I, there exists at most one
solution x to the equation
Bx = y
for any choice of y. In fact,
x = A (Bx) = Ay.
This means the row reduced echelon form of B must be I. Thus every column is a pivot column.
Otherwise, there exists a free variable and the solution, if it exists, would not be unique, contrary
to what was just shown must happen if AB = I. It follows that a right inverse B −1 for B exists.
The Gauss Jordan procedure for finding the inverse yields
(
)
(
)
B I 7→ I B −1 .
Now multiply both sides of the equation AB = I on the right by B −1 . Then
(
)
A = A BB −1 = (AB) B −1 = B −1 .
Thus A is the right inverse of B, and so BA = I. This shows that if AB = I, then BA = I also.
Exchanging roles of A and B, we see that if BA = I, then AB = I.
Now suppose A−1 exists. Then there exists a unique solution x to the system Ax = b given by
x = A−1 b. It follows that each column of A is a pivot column. Hence one can row reduce and
obtain
(
)
(
)
A I → I A−1
8.3
The Rank Of A Matrix
8.3.1
The Definition Of Rank
To begin, here is a definition to introduce some terminology.
Definition 8.3.1 Let A be an m × n matrix. The column space of A is the span of the columns.
The row space is the span of the rows.
There are three definitions of the rank of a matrix which are useful. These are given in the
following definition. It turns out that the concept of determinant rank is often important but is
virtually impossible to find directly. The other two concepts of rank are very easily determined and
it is a happy fact that all three yield the same number. This is shown later.
128
CHAPTER 8. RANK OF A MATRIX
Definition 8.3.2 A sub-matrix of a matrix A is a rectangular array of numbers obtained by
deleting some rows and columns of A. Let A be an m × n matrix. The determinant rank of the
matrix equals r where r is the largest number such that some r × r sub-matrix of A has a non zero
determinant. The row space of a matrix is the span of the rows and the column space of a matrix
is the span of the columns. The row rank of a matrix is the number of nonzero rows in the row
reduced echelon form and the column rank is the number columns in the row reduced echelon form
which are one of the ek vectors. Thus the column rank equals the number of pivot columns. It follows
the row rank equals the column rank. This is also called the rank of the matrix. The rank of a matrix
A is denoted by rank (A) .
Example 8.3.3 Consider the matrix
(
1 2
2 4
3
6
)
What is its rank?
You could look at all the 2 × 2 submatrices
(
) (
1 2
1
,
2 4
2
3
6
) (
,
2
4
3
6
)
.
Each has determinant equal to 0. Therefore, the rank is less than 2. Now look at the 1 × 1 submatrices. There exists one of these which has nonzero determinant. For example (1) has determinant
equal to 1 and so the rank of this matrix equals 1.
Of course this example was pretty easy but what if you had a 4 × 7 matrix? You would have to
consider all the 4 × 4 submatrices and then all the 3 × 3 submatrices and then all the 2 × 2 matrices
and finally all the 1 × 1 matrices in order to compute the rank. Clearly this is not practical. The
following theorem will remove the difficulties just indicated.
The following theorem is proved later.
Theorem 8.3.4 Let A be an m × n matrix. Then the row rank, column rank and determinant rank
are all the same.
Example 8.3.5 Find the rank of the matrix





1
−4
3
4
2
1
3
2
2
1
−3 −2

3 0
1 2 

.
6 5 
1 7
From the above definition, all you have to do is find the row reduced echelon form and then
count up the number of nonzero rows. But the row reduced echelon form of this matrix is





and so the rank of this matrix is 4.
Find the rank of the matrix





1
0
0
0
1
−4
3
0
0
1
0
0
0
0
1
0
2 1
3 2
2 1
7 4
0
0
0
1
− 17
4
1
− 45
4





9
2

3 0
1 2 


6 5 
10 7
8.3.
THE RANK OF A MATRIX
The row reduced echelon form is
129





1
0
0
0
0 0 32
1 0 −4
0 1 19
2
0 0 0
5
2

−17 

63 

2
0
and so this time the rank is 3.
8.3.2
Finding The Row And Column Space Of A Matrix
The row reduced echelon form also can be used to obtain an efficient description of the row and
column space of a matrix. Of course you can get the column space by simply saying that it equals
the span of all the columns but often you can get the column space as the span of fewer columns than
this. This is what we mean by an “efficient description”. This is illustrated in the next example.
Example 8.3.6 Find the rank of the following matrix and describe the column and row spaces
efficiently.


1 2 1 3 2


(8.1)
 1 3 6 0 2 
3 7 8 6 6
The row reduced echelon form is

1 0

 0 1
0 0

−9 9 2

5 −3 0  .
0
0 0
Therefore, the rank of this matrix equals 2. All columns of this row reduced echelon form are in

  
1
0

  
span  0  ,  1  .
0
0
For example,


 
 
−9
1
0


 
 
 5  = −9  0  + 5  1  .
0
0
0
By Lemma 8.2.5, all columns of the original matrix, are similarly contained in the span of the first
two columns of that matrix. For example, consider the third column of the original matrix.
 
 
 
1
1
2
 
 
 
 6  = −9  1  + 5  3  .
8
3
7
How did I know to use −9 and 5 for the coefficients? This is what Lemma 8.2.5 says! It says linear
relationships are all preserved. Therefore, the column space of the original matrix equals the span
of the first two columns. This is the desired efficient description of the column space.
What about an efficient description of the row space? When row operations are used, the resulting
vectors remain in the row space. Thus the rows in the row reduced echelon form are in the row space
of the original matrix. Furthermore, by reversing the row operations, each row of the original matrix
can be obtained as a linear combination of the rows in the row reduced echelon form. It follows that
the span of the nonzero rows in the row reduced echelon matrix equals the(span of the original)rows.
In the above example, the row space equals the span of the two vectors, 1 0 −9 9 2 and
(
)
0 1 5 −3 0 .
130
CHAPTER 8. RANK OF A MATRIX
Example 8.3.7 Find the rank of the following matrix and describe the column and row spaces
efficiently.


1 2 1 3 2
 1 3 6 0 2 


(8.2)


 1 2 1 3 2 
1 3 2 4 0
The row reduced echelon form is





1
0
0
0
0 0
1 0
0 1
0 0
13
0
2
2 − 52
1
−1
2
0
0



.

and so the rank is 3, the row space is the span of the vectors,
(
) (
0 0 1 −1 12 , 0 1 0
(
)
,
1 0 0 0 13
2
2
)
− 52
,
and the column space is the span of the first three columns in the original matrix,
     
1
2
1
 1   3   6 
     
span   ,   ,   .
 1   2   1 
1
3
2
Example 8.3.8 Find the rank of the following matrix and describe the column and row spaces
efficiently.


1 2 3 0 1


 2 1 3 2 4 .
−1 2 1 3 1
The row reduced echelon form is

1 0

 0 1
0 0
1
1
0
0
0
1
21
17
2
− 17
14
17


.
It follows the rank is three and the column space is the span of
of the original matrix.

   
1
2
0

   
span  2  ,  1  ,  2
−1
2
3
while the row space is the span of the vectors
(
) (
, 0 1 1
0 0 0 1 14
17
0
2
− 17
the first, second and fourth columns



) (
, 1 0
1
0
21
17
)
.
Procedure 8.3.9 To find the rank of a matrix, obtain the row reduced echelon form for the matrix.
Then count the number of nonzero rows or equivalently the number of pivot columns. This is the
rank. The row space is the span of the nonzero rows in the row reduced echelon form and the column
space is the span of the pivot columns of the original matrix.
8.4. A SHORT APPLICATION TO CHEMISTRY
8.4
131
A Short Application To Chemistry
The following example is to chemistry. Sometimes there are numerous chemical reactions and some
are in a sense redundant. This example is discussed in the book by Greenberg [7]. Suppose you have
the folowing chemical reactions.
CO + 12 O2 → CO2
H2 + 12 O2 → H2 O
CH4 + 32 O2 → CO + 2H2 O
CH4 + 2O2 → CO2 + 2H2 O
There are four chemical reactions here but they are not independent reactions. There is some
redundancy. What are the independent reactions? Is there a way to consider a shorter list of
reactions? To analyze this situation, you can write as a matrix


CO O2 CO2 H2 H2 O CH4


 1
1/2 −1
0
0
0 


 0
1/2
0
1
−1
0 




0
0
−2
1 
 −1 3/2
0
2
−1
0
−2
1
The top row of numbers comes from CO + 12 O2 − CO2 = 0 which represents the first of the chemical
reactions. The entries of the matrix


1 1/2 −1 0 0 0
 0 1/2 0 1 −1 0 




 −1 3/2 0 0 −2 1 
0
2
−1
0
−2
1
are called Stoichiometric coefficients. Rather than listing all of the reactions as above, it would be
more efficient to only list those which are in a sense independent by throwing out that which is
redundant. This is easy to do. Just take the row reduced echelon from of the above matrix.





1
0
0
0
0 0
1 0
0 1
0 0
3
2
4
0

−1 −1
−2 0 


−2 −1 
0
0
The top three rows represent “independent” reactions which come from the original four reactions.
One can obtain each of the original four rows of the stoichiometric matrix given above by taking a
suitable linear combination of rows of this row reduced matrix.
Thus one could consider the simplified reactions
CO + 3H2 − 1H2 O − 1CH4 = 0
O2 + 2H2 − 2H2 O = 0
CO2 + 4H2 − 2H2 O − 1CH4 = 0
In terms of the original notation, these are the reactions
CO + 3H2 → H2 O + CH4
O2 + 2H2 → 2H2 O
CO2 + 4H2 → 2H2 O + CH4
132
CHAPTER 8. RANK OF A MATRIX
Instead of the four you started with, you could consider the simpler list given above. The idea
is that, in terms of what happens chemically, you obtain the same information with the shorter
list of reactions and have gotten rid of the redundancy which was present in the original list. You
can probably imagine that if you had a very large list of reactions made up from some sort of
experimental evidence, such a simplification could be a considerable improvement.
This is motivation for the general notion of a basis for a vector space which is discussed in the
next section. The idea of a basis is similar to what was just done, reducing a list of reactions to
a shorter list. With vectors, you have the span of some vectors and you want to get the shortest
possible list of vectors which will leave the span unchanged.
8.5
8.5.1
Linear Independence And Bases
Linear Independence And Dependence
First we consider the concept of linear independence. We define what it means for vectors in Fn
to be linearly independent and then give equivalent descriptions. In the following definition, the
symbol,
(
)
v1 v2 · · · vk
denotes the matrix which has the vector v1 as the first column, v2 as the second column and so
forth until vk is the k th column.
Definition 8.5.1 Let {v1 , · · · , vk } be vectors in Fn . Then this collection of vectors is said to be
linearly independent if each of the columns of the n × k matrix
(
)
v1 v2 · · · vk
is a pivot column. Thus the row reduced echelon form for this matrix is
(
)
e1 e2 · · · ek
and you cannot delete any of these vectors without diminishing the span of the resulting list.
The question whether any vector in the first k columns in a matrix is a pivot column is independent of the presence of later columns. Thus each of {v1 , · · · , vk } is a pivot column in
(
)
v1 v2 · · · vk
if and only if these vectors are each pivot columns in
(
v1 v2 · · · vk w1
···
)
wr
Here is what the linear independence means in terms of linear relationships.
Corollary 8.5.2 The collection of vectors, {v1 , · · · , vk } is linearly independent if and only if none
of these vectors is a linear combination of the others.
Proof: If {v1 , · · · , vk } is linearly independent, then every column in
(
)
v1 v2 · · · vk
is a pivot column which requires that the row reduced echelon form is
(
)
e1 e2 · · · ek .
8.5. LINEAR INDEPENDENCE AND BASES
133
Now none of the ei vectors is a linear combination of the others. By Lemma 8.2.5 on Page 124 none
of the vi is a linear combination of the others. Recall this lemma says linear relationships between
the columns are preserved under row operations.
Next suppose none of the vectors {v1 , · · · , vk } is a linear combination of the others. Then none
of the columns in
)
(
v1 v2 · · · vk
is a linear combination of the others. By Lemma 8.2.5 the same is true of the row reduced echelon form for this matrix. From the description of the row reduced echelon form, it follows that
the ith column of the row reduced echelon form must be ei since otherwise, it would be a linear
combination of the first i − 1 vectors e1 ,· · · , ei−1 and by Lemma 8.2.5, it follows vi would be the
same
linear combination
(
) of v1 , · · · , vi−1 contrary to the assumption that none of the columns in
is a linear combination of the others. Therefore, each of the k columns in
v1 v2 · · · vk
(
)
v1 v2 · · · vk is a pivot column and so {v1 , · · · , vk } is linearly independent. Corollary 8.5.3 The collection of vectors, {v1 , · · · , vk } is linearly independent if and only if whenever
n
∑
c i vi = 0
i=1
it follows each ci = 0.
Proof: Suppose first {v1 , · · · , vk } is linearly independent. Then by Corollary 8.5.2, none of the
vectors is a linear combination of the others. Now suppose
n
∑
c i vi = 0
i=1
and not all the ci = 0. Then pick ci which is not zero, divide by it and solve for vi in terms of the
other vj , contradicting the fact that none of the vi equals a linear combination of the others.
Now suppose the condition about the sum holds. If vi is a linear combination of the other vectors
in the list, then you could obtain an equation of the form
∑
cj vj
vi =
j̸=i
and so
0=
∑
cj vj + (−1) vi ,
j̸=i
contradicting the condition about the sum. Sometimes we refer to this last condition about sums as follows: The set of vectors, {v1 , · · · , vk }
is linearly independent if and only if there is no nontrivial linear combination which equals zero. (A
nontrivial linear combination is one in which not all the scalars equal zero.)
We give the following equivalent definition of linear independence which follows from the above
corollaries.
Definition 8.5.4 A set of vectors, {v1 , · · · , vk } is linearly independent if and only if none of the
vectors is a linear combination of the others or equivalently if there is no nontrivial linear combination
of the vectors which equals 0. It is said to be linearly dependent if at least one of the vectors is
a linear combination of the others or equivalently there exists a nontrivial linear combination which
equals zero.
Note the meaning of the words. To say a set of vectors is linearly dependent means at least one
is a linear combination of the others. In other words, it is in a sense “dependent” on these other
134
CHAPTER 8. RANK OF A MATRIX
vectors. At this time, the vectors are in Fn but the above definition makes sense without knowing
any description of the vectors. This will be considered later in the book.
The following corollary follows right away from the row reduced echelon form. It concerns a
matrix which looks like this: (More columns than rows.)
Corollary 8.5.5 Let {v1 , · · · , vk } be a set of vectors in Fn . Then if k > n, it must be the case that
{v1 , · · · , vk } is not linearly independent. In other words, if k > n, then {v1 , · · · , vk } is dependent.
(
)
Proof: If k > n, then the columns of
cannot each be a pivot column
v1 v2 · · · vk
because there are at most n pivot columns due to the fact the matrix has only n rows. In reading
from left to right, pick the first column which is not a pivot column. Then from the description of
row reduced echelon form, this column is a linear combination of the preceding columns and so the
given vectors are dependent by Corollary 8.5.2.       

3
0
2
1





      


1
2
2
1

   
Example 8.5.6 Determine whether the vectors       
 are linearly

 3   0   1   2 






−1
2
1
0
independent. If they are linearly dependent, exhibit one of the vectors as a linear combination of the
others.
Form the matrix mentioned above.





1
2
3
0
2
1
0
1
0 3
1 2
1 2
2 −1
Then the row reduced echelon form of this matrix is

1 0 0 1
 0 1 0 1


 0 0 1 −1
0 0 0 0








.

Thus not all the columns are pivot columns and so the vectors are not linear independent. Note the
fourth column is of the form
 


 
1
0
0
 0 
 1 
 0 
 


 
1  + 1
 + (−1)  
 0 
 0 
 1 
0
0
0
From Lemma 8.2.5, the same linear relationship exists between the columns of the original matrix.
Thus
 


  

1
2
0
3
 2 
 1 
 1   2 
 


  

1  + 1
 + (−1)   = 
.
 3 
 0 
 1   2 
0
1
2
−1
8.5. LINEAR INDEPENDENCE AND BASES
135
Note the usefulness of the row reduced echelon form in discovering hidden linear relationships in
collections of vectors.
       
1
2
0
3 




 2   1   1   2 

    
Example 8.5.7 Determine whether the vectors         are linearly in
 3   0   1   2 






0
1
2
0
dependent. If they are linearly dependent, exhibit one of the vectors as a linear combination of the
others.
The matrix used to find this is





The row reduced echelon form is





1
2
3
0
2
1
0
1
0
1
1
2
3
2
2
0
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1










and so every column is a pivot column. Therefore, these vectors are linearly independent and there
is no way to obtain one of the vectors as a linear combination of the others.
8.5.2
Subspaces
A subspace is a set of vectors with the property that linear combinations of these vectors remain in
the set. Geometrically, subspaces are like lines and planes which contain the origin. More precisely,
the following definition is the right way to think of this.
Definition 8.5.8 Let V be a nonempty collection of vectors in Fn . Then V is called a subspace if
whenever α, β are scalars and u, v are vectors in V, the linear combination αu + βv is also in V .
There is no substitute for the above definition or equivalent algebraic definition! However, it is
sometimes helpful to look at pictures at least initially. The following are four subsets of R2 . The
first is the shaded area between two lines which intersect at the origin, the second is a line through
the origin, the third is the union of two lines through the origin, and the last is the region between
two rays from the origin. Note that in the last, multiplication of a vector in the set by a nonnegative
scalar results in a vector in the set as does the sum of two vectors in the set. However, multiplication
by a negative scalar does not take a vector in the set to another in the set.
not subspace
subspace
not subspace
I
not subspace
R
Observe how the above definition indicates that the claims posted on the picture are valid.
Subspaces are exactly those subsets of Fn which are themselves vector spaces. Recall that a
vector space is something which satisfies the vector space axioms on Page 14.
Proposition 8.5.9 Let V be a nonempty collection of vectors in Fn . Then V is a subspace if and
only if V is itself a vector space having the same operations as those defined on Fn .
136
CHAPTER 8. RANK OF A MATRIX
Proof: Suppose first that V is a subspace. It is obvious all the algebraic laws hold on V because
it is a subset of Fn and they hold on Fn . Thus u + v = v + u along with the other axioms. Does
V contain 0? Yes because it contains 0u = 0. Are the operations defined on V ? That is, when
you add vectors of V do you get a vector in V ? When you multiply a vector in V by a scalar,
do you get a vector in V ? Yes. This is contained in the definition. Does every vector in V have
an additive inverse? Yes because −v = (−1) v which is given to be in V provided v ∈ V . (
(−1) v + v = ((−1) + 1) v = 0v = 0. )
Next suppose V is a vector space. Then by definition, it is closed with respect to linear combinations. Hence it is a subspace. It turns out that every subspace equals the span of some vectors. This is the content of the next
theorem.
Theorem 8.5.10 V is a subspace of Fn if and only if there exist vectors of V
{u1 , · · · , uk }
such that V = span (u1 , · · · , uk ) .
Proof: Pick a vector of V, u1 . If V = span {u1 } , then stop. You have found your list of vectors.
If V ̸= span (u1 ) , then there exists u2 a vector of V which is not a vector in span (u1 ) . Consider
span (u1 , u2 ) . If V = span (u1 , u2 ) , stop. Otherwise, pick u3 ∈
/ span (u1 , u2 ) . Continue this way.
Note that since V is a subspace, these spans are each contained in V . The process must stop with
uk for some k ≤ n since otherwise, the matrix
(
)
u1 · · · uk
having these vectors as columns would have n rows and k > n columns. Consequently, it can have
no more than n pivot columns and so the first column which is not a pivot column would be a linear
combination of the preceding columns contrary to the construction.
∑k
∑k
For the other half, suppose V = span (u1 , · · · , uk ) and let
i=1 di ui be two
i=1 ci ui and
vectors in V. Now let α and β be two scalars. Then
α
k
∑
i=1
ci ui + β
k
∑
i=1
di ui =
k
∑
(αci + βdi ) ui
i=1
which is one of the things in span (u1 , · · · , uk ) showing that span (u1 , · · · , uk ) has the properties of
a subspace. The following corollary also follows easily.
Corollary 8.5.11 If V is a subspace of Fn , then there exist vectors of V, {u1 , · · · , uk } such that
V = span (u1 , · · · , uk ) and {u1 , · · · , uk } is linearly independent.
Proof: Let V = span (u1 , · · · , uk ) . Then let the vectors {u1 , · · · , uk } be the columns of the
following matrix.
(
)
u1 · · · uk
Retain only the pivot columns. That is, determine the pivot columns from the row reduced echelon
form and these are a basis for span (u1 , · · · , uk ). The message is that subspaces of Fn consist of spans of finite, linearly independent collections of
vectors of Fn .
The following fundamental lemma is very useful. It is called the exchange theorem.
Lemma 8.5.12 Suppose {x1 , · · · , xr } is linearly independent and each xk is contained in
span (y1 , · · · , ys ) .
Then s ≥ r. In words, spanning sets have at least as many vectors as linearly independent sets.
8.5. LINEAR INDEPENDENCE AND BASES
137
Proof: Since {y1 , · · · , ys } is a spanning set, there exist scalars aij such that
xj =
s
∑
aij yi
i=1
Suppose s < r. Then the matrix A whose ij th entry is aij has fewer rows, s than columns, r. By
Corollary 8.2.8 there exists d such that d ̸= 0 but Ad = 0. In other words,
r
∑
aij dj = 0, i = 1, 2, · · · , s
j=1
Therefore,
r
∑
dj xj
=
j=1
r
∑
j=1
=
s
∑
i=1
dj


s
∑
aij yi
i=1
r
∑

aij dj  yi =
j=1
s
∑
0yi = 0
i=1
which contradicts {x1 , · · · , xr } is linearly independent, because not all the dj = 0. Thus s ≥ r. Note how this lemma was totally dependent on algebraic considerations and was independent of
context. This will be considered more later in the chapter on abstract vector spaces. I didn’t need
to know what the xk , yk were, only that the {x1 , · · · , xr } were independent and contained in the
span of the yk .
8.5.3
Basis Of A Subspace
It was just shown in Corollary 8.5.11 that every subspace of Fn is equal to the span of a linearly
independent collection of vectors of Fn . Such a collection of vectors is called a basis.
Definition 8.5.13 Let V be a subspace of Fn . Then {u1 , · · · , uk } is a basis for V if the following
two conditions hold.
1. span (u1 , · · · , uk ) = V.
2. {u1 , · · · , uk } is linearly independent.
The plural of basis is bases.
The main theorem about bases is the following.
Theorem 8.5.14 Let V be a subspace of Fn and suppose {u1 , · · · , uk }, {v1 , · · · , vm } are two bases
for V . Then k = m.
Proof: This follows right away from Lemma 8.5.12. {u1 , · · · , uk } is a spanning set while
{v1 , · · · , vm } is linearly independent so k ≥ m. Also {v1 , · · · , vm } is a spanning set while {u1 , · · · , uk }
is linearly independent so m ≥ k.
Now here is another proof. Suppose k < m. Then since {u1 , · · · , uk } is a basis for V, each vi is
a linear combination of the vectors of {u1 , · · · , uk } . Consider the matrix
(
)
u1 · · · uk v1 · · · vm
in which each of the ui is a pivot column because the {u1 , · · · , uk } are linearly independent. Therefore, the row reduced echelon form of this matrix is
(
)
(8.3)
e1 · · · ek w1 · · · wm
138
CHAPTER 8. RANK OF A MATRIX
where each wj has zeroes below the k th row. This is because of Lemma 8.2.5 which implies each wi
is a linear combination of the e1 , · · · , ek . Discarding the bottom n − k rows of zeroes in the above,
yields the matrix
(
)
′
e′1 · · · e′k w1′ · · · wm
in which all vectors are in Fk . Since m > k, it follows from Corollary 8.5.5 that the vectors,
′
{w1′ , · · · , wm
} are dependent. Therefore, some wj′ is a linear combination of the other wi′ . Therefore, wj is a linear combination of the other wi in 8.3. By Lemma 8.2.5 again, the same linear
relationship exists between the {v1 , · · · , vm } showing that {v1 , · · · , vm } is not linearly independent
and contradicting the assumption that {v1 , · · · , vm } is a basis. It follows m ≤ k. Similarly, k ≤ m.
This is a very important theorem so here is yet another proof of it.
Theorem 8.5.15 Let V be a subspace and suppose {u1 , · · · , uk } and {v1 , · · · , vm } are two bases
for V . Then k = m.
Proof: Suppose k > m. Then since the vectors, {u1 , · · · , uk } span V, there exist scalars, cij
such that
m
∑
cij vi = uj .
i=1
Therefore,
k
∑
dj uj = 0 if and only if
j=1
k ∑
m
∑
cij dj vi = 0
j=1 i=1
if and only if
m
∑
i=1


k
∑

cij dj  vi = 0
j=1
Now since{v1 , · · · , vn } is independent, this happens if and only if
k
∑
cij dj = 0, i = 1, 2, · · · , m.
j=1
However, this is a system of m equations in k variables, d1 , · · · , dk and m < k. Therefore, there
exists a solution to this system of equations in which not all (
the dj are) equal to zero. Recall why
this is so. The augmented matrix for the system is of the form C 0 where C is a matrix which
has more columns than rows. Therefore, there are free variables and hence nonzero solutions to the
system of equations. However, this contradicts the linear independence of {u1 , · · · , uk } because, as
∑k
explained above, j=1 dj uj = 0. Similarly it cannot happen that m > k. The following definition can now be stated.
Definition 8.5.16 Let V be a subspace of Fn . Then the dimension of V is defined to be the
number of vectors in a basis.
Corollary 8.5.17 The dimension of Fn is n. The dimension of the space of m × n matrices is mn.
Proof: You only need to exhibit a basis for Fn which has n vectors. Such a basis is {e1 , · · · , en }.
As to the vector space of m × n matrices, a basis consists of the matrices Eij which has a 1 in the
ij th position and 0 elsewhere. Corollary 8.5.18 Suppose {v1 , · · · , vn } is linearly independent and each vi is a vector in Fn . Then
{v1 , · · · , vn } is a basis for Fn . Suppose {v1 , · · · , vm } spans Fn . Then m ≥ n. If {v1 , · · · , vn } spans
Fn , then {v1 , · · · , vn } is linearly independent.
8.5. LINEAR INDEPENDENCE AND BASES
139
Proof: Let u be a vector of Fn and consider the matrix
(
)
v1 · · · vn u .
Since each vi is a pivot column, the row reduced echelon form is
(
)
e1 · · · en w
and so, since w is in span (e1 , · · · , en ) , it follows from Lemma 8.2.5 that u is one of the vectors in
span (v1 , · · · , vn ) . Therefore, {v1 , · · · , vn } is a basis as claimed.
To establish the second claim, suppose that m < n. Then letting vi1 , · · · , vik be the pivot columns
of the matrix
(
)
v1 · · · vm
it follows k ≤ m < n and these k pivot columns would be a basis for Fn having fewer than n vectors,
contrary to Theorem 8.5.14 which states every two bases have the same number of vectors in them.
Finally consider the third claim. If {v1 , · · · , vn } is not linearly independent, then replace this
list with {vi1 , · · · , vik } where these are the pivot columns of the matrix
(
)
v1 · · · vn
Then {vi1 , · · · , vik } spans Fn and is linearly independent so it is a basis having less than n vectors
contrary to Theorem 8.5.14 which states every two bases have the same number of vectors in them.
Example 8.5.19 Find the rank of the following matrix. If the rank is r, identify r columns in
the original matrix which have the property that every other column may be written as a linear
combination of these. Also find a basis for the row and column spaces of the matrices.


1 2 3
2


 1 5 −4 −1 
−2 3 1
0
The row reduced echelon form is

1 0

 0 1
0 0
0
0
1
27
70
1
10
33
70



and so the rank of the matrix is 3. A basis for the column space is the first three columns of the
original matrix. I know they span because the first three columns of the row reduced echelon
form above span the column space of that matrix. They are linearly independent because the first
three columns of the row reduced echelon form are linearly independent. By Lemma 8.2.5 all linear
relationships are preserved and so these first three vectors form a basis for the column space. The
four rows of the row reduced echelon form form a basis for the row space of the original matrix.
Example 8.5.20 Find the rank of the following matrix. If the rank is r, identify r columns in
the original matrix which have the property that every other column may be written as a linear
combination of these. Also find a basis for the row and column spaces of the matrices.


1 2 3 0 1


 1 1 2 −6 2 
−2 3 1 0 2
140
CHAPTER 8. RANK OF A MATRIX
The row reduced echelon form is

1 0 1

 0 1 1
0 0 0
0
0
1

− 17
4
7
− 11
42

.
A basis for the column space of this row reduced echelon form is the first second and fourth columns.
Therefore, a basis for the column space in the original matrix is the first second and fourth columns.
The rank of the matrix is 3. A basis for the row space of the original matrix is the columns of the
row reduced echelon form.
8.5.4
Extending An Independent Set To Form A Basis
Suppose {v1 , · · · , vm } is a linearly independent set of vectors in Fn . It turns out there is a larger
set of vectors, {v1 , · · · , vm , vm+1 , · · · , vn } which is a basis for Fn . It is easy to do this using the
row reduced echelon form. Consider the following matrix having rank n in which the columns are
shown.
(
)
v1 · · · vm e1 e2 · · · en .
Since the {v1 , · · · , vm } are linearly independent, the row reduced echelon form of this matrix is of
the form
(
)
e1 · · · em u1 u2 · · · un
Now the pivot columns can be identified and this leads to a basis for the column space of the original
matrix which is of the form
{
}
v1 , · · · , vm , ei1 , · · · , ein−m .
This proves the following theorem.
Theorem 8.5.21 Let {v1 , · · · , vm } be a linearly independent set of vectors in Fn . Then there is a
larger set of vectors, {v1 , · · · , vm , vm+1 , · · · , vn } which is a basis for Fn .
  

1
1 




 1   0 

  

Example 8.5.22 The vectors,   , 
 are linearly independent. Enlarge this set of

 0   1 






0
0
vectors to form a basis for R4 .
Using the above technique, consider the following

1 1 1 0
 1 0 0 1


 0 1 0 0
0 0 0 0
whose row reduced echelon form is





1
0
0
0
0
1
0
0
0 1
0 0
1 −1
0 0
matrix.

0 0
0 0 


1 0 
0
0
1
−1
0
1
0
0
0
1





The pivot columns are numbers 1,2,3, and 6. Therefore, a basis
      
1
1
1
0



 1   0   0   0
      
 , , ,

   1   0   0

 0

0
0
0
1
is











8.5. LINEAR INDEPENDENCE AND BASES
8.5.5
141
Finding The Null Space Or Kernel Of A Matrix
Let A be an m × n matrix.
Definition 8.5.23 ker (A), also referred to as the null space of A is defined as follows.
ker (A) = {x : Ax = 0}
and to find ker (A) one must solve the system of equations Ax = 0.
This is not new! There is just some new terminology being used. To repeat, ker (A) is the
solution to the system Ax = 0.
Example 8.5.24 Let

1 2 1


A =  0 −1 1  .
2 3 3

Find ker (A).
You need to solve the equation Ax = 0. To do this you write the augmented matrix and then
obtain the row reduced echelon form and the solution. The augmented matrix is


1 2 1 | 0


 0 −1 1 | 0 
2 3 3 | 0
Next place this matrix in row reduced echelon form,


1 0 3 | 0


 0 1 −1 | 0 
0 0 0 | 0
Note that x1 and x2 are basic variables while x3 is a free variable. Therefore, the solution to this
system of equations, Ax = 0 is given by


3t


 t  : t ∈ R.
t
Example 8.5.25 Let



A=


1 2 1 0 1
2 −1 1 3 0 


3 1 2 3 1 
4 −2 2 6 0
Find the null space of A.
You need to solve the equation, Ax = 0. The augmented matrix is





1 2 1 0 1
2 −1 1 3 0
3 1 2 3 1
4 −2 2 6 0
|
|
|
|
0
0
0
0





142
CHAPTER 8. RANK OF A MATRIX
Its row reduced echelon form is





1
0
0
0
0
1
0
0
3
5
1
5
6
5
− 35
1
5
2
5
0
0
0
0
0
0
|
|
|
|
0
0
0
0





It follows x1 and x2 are basic variables and x3 , x4 , x5 are free variables. Therefore, ker (A) is given
by
 ( )

( )
( )
− 53 s1 + −6
s2 + 15 s3
5
( )
( )
 ( 1)

 − 5 s1 + 35 s2 + − 25 s3 



 : s1 , s2 , s3 ∈ R.
s1




s2


s3
We write this in the form




s1 



− 35
− 15
1
0
0


−6
5
3
5


1
5
− 25
















 + s2  0  + s3  0  : s1 , s2 , s3 ∈ R.











 1 
 0 
0
1
In other words, the null space of this matrix equals the span of the three vectors above. Thus

 
 

−6
1
− 35
5
5
 1   3   2 
 − 5   5   − 5 

 
 

 
 


ker (A) = span 
 1  ,  0  ,  0  .

 
 

 0   1   0 
0
0
1
This is the same as

3
5
1
5
 
6
5
−3
5

 

 

 



ker (A) = span  −1 
, 0

 
 0   −1
0
0
 
−1
5
2
5

 

 

 

 ,  0  .
 

 

  0 
−1
Notice also that the three vectors above are linearly independent and so the dimension of ker (A)
is 3. This is generally the way it works. The number of free variables equals the dimension of the
null space while the number of basic variables equals the number of pivot columns which equals the
rank. We state this in the following theorem.
Definition 8.5.26 The dimension of the null space of a matrix is called the nullity2 and written
as null (A) .
Theorem 8.5.27 Let A be an m × n matrix. Then rank (A) + null (A) = n.
2 Isn’t
it amazing how many different words are available for use in linear algebra?
8.6. FREDHOLM ALTERNATIVE
8.5.6
143
Rank And Existence Of Solutions To Linear Systems
Consider the linear system of equations,
Ax = b
(8.4)
where A is an m × n matrix, x is a n × 1 column vector, and b is an m × 1 column vector. Suppose
(
)
A = a1 · · · an
where the ak denote the columns of A. Then x = (x1 , · · · , xn )
and only if
x1 a1 + · · · + xn an = b
T
is a solution of the system 8.4, if
which says that b is a vector in span (a1 , · · · , an ) . This shows that there exists a solution to the
system, 8.4 if and only if b is contained in span (a1 , · · · , an ) . In words, there is a solution to 8.4 if
and only if b is in the column space of A. In terms of rank, the following proposition describes the
situation.
Proposition 8.5.28 Let A be an m × n matrix and let b be an m × 1 column vector. Then there
exists a solution to 8.4 if and only if
(
)
rank A | b = rank (A) .
(8.5)
(
)
Proof: Place
and A in row reduced echelon form, respectively B and C. If the
A | b
above condition on rank is true, then both B and C have the same number of nonzero rows. In
particular, you cannot have a row of the form
(
)
0 ··· 0 where ̸= 0 in B. Therefore, there will exist a solution to the system 8.4.
Conversely, suppose there exists a solution. This means there cannot be such a row in B described
above. Therefore, B and C must have the same number of zero rows and so they have the same
number of nonzero rows. Therefore, the rank of the two matrices in 8.5 is the same. 8.6
Fredholm Alternative
There is a very useful version of Proposition 8.5.28 known as the Fredholm alternative. I will
only present this for the case of real matrices here. Later a much more elegant and general approach
is presented which allows for the general case of complex matrices.
The following definition is used to state the Fredholm alternative.
Definition 8.6.1 Let S ⊆ Rm . Then S ⊥ ≡ {z ∈ Rm : z · s = 0 for every s ∈ S} . The funny exponent, ⊥ is called “perp”.
Now note
(
) {
}
ker AT ≡ z : AT z = 0 =
{
z:
m
∑
}
z k ak = 0
k=1
Here the ak are the rows of A because they are the columns of AT .
Lemma 8.6.2 Let A be a real m × n matrix, let x ∈ Rn and y ∈ Rm . Then
(
)
(Ax · y) = x·AT y
144
CHAPTER 8. RANK OF A MATRIX
Proof: This follows right away from the definition of the dot product and matrix multiplication.
∑
∑( )
(
)
(Ax · y) =
Akl xl yk =
AT lk xl yk = x · AT y . k,l
k,l
Now it is time to state the Fredholm alternative. The first version of this is the following theorem.
Theorem 8.6.3 Let A be a real m × n matrix and let b ∈ Rm . There exists a solution, x to the
( )⊥
equation Ax = b if and only if b ∈ ker AT .
( )⊥
Proof: First suppose b ∈ ker AT . Then this says that if AT x = 0, it follows that
b · x = xT b = 0.
In other words, taking the transpose, if
xT A = 0, then xT b = 0.
Thus, if P is a product of elementary matrices such that P A is in row reduced echelon form, then
if P A has a row of zeros, in the k th position, obtained from the k th row of P times A, then there is
th
also a zero in the k th position of P b. This is because the k(th position in
) P b is just the k row of
P times b. Thus the row reduced echelon forms of A and A | b have the same number of
(
)
zero rows. Thus rank A | b = rank (A). By Proposition 8.5.28, there exists a solution x to
the system Ax (= b.) It remains to prove the converse.
Let z ∈ ker AT and suppose Ax = b. I need to verify b · z = 0. By Lemma 8.6.2,
b · z = Ax · z = x · AT z = x · 0 = 0 This implies the following corollary which is also called the Fredholm alternative. The “alternative” becomes more clear in this corollary.
Corollary 8.6.4 Let A be an m × n matrix. Then A maps Rn onto Rm if and only if the only
solution to AT x = 0 is x = 0.
( )
( )⊥
Proof: If the only solution to AT x = 0 is x = 0, then ker AT = {0} and so ker AT
= Rm
m
because every b ∈ R has the property that b · 0 = 0. Therefore, Ax = b has a solution for any
( )⊥
b ∈ Rm because the b for which there is a solution are those in ker AT
by Theorem 8.6.3. In
other words, A maps Rn onto Rm .
Conversely if A is onto, then if AT x = 0, there exists y such that x = Ay and then AT Ay = 0
2
and so |Ay| = Ay · Ay = AT Ay · y = 0 · y = 0 and so x = Ay = 0. Here is an amusing example.
Example 8.6.5 Let A be an m × n matrix in which m > n. Then A cannot map onto Rm .
The reason for this is that AT is an n × m where m > n and so in the augmented matrix
( T )
A |0
there must be some free variables. Thus there exists a nonzero vector x such that AT x = 0.
8.6.1
Row, Column, And Determinant Rank
I will now present a review of earlier topics and prove Theorem 8.3.4.
8.6. FREDHOLM ALTERNATIVE
145
Definition 8.6.6 A sub-matrix of a matrix A is the rectangular array of numbers obtained by deleting some rows and columns of A. Let A be an m × n matrix. The determinant rank of the matrix
equals r where r is the largest number such that some r × r sub-matrix of A has a non zero determinant. The row rank is defined to be the dimension of the span of the rows. The column rank
is defined to be the dimension of the span of the columns.
Theorem 8.6.7 If A, an m × n matrix has determinant rank, r, then there exist r rows of the
matrix such that every other row is a linear combination of these r rows.
Proof: Suppose the determinant rank of A = (aij ) equals r. Thus some r × r submatrix has non
zero determinant and there is no larger square submatrix which has non zero determinant. Suppose
such a submatrix is determined by the r columns whose indices are
j1 < · · · < jr
and the r rows whose indices are
i1 < · · · < ir
I want to show that every row is a linear combination of these rows. Consider the lth row and let p
be an index between 1 and n. Form the following (r + 1) × (r + 1) matrix


ai1 j1 · · · ai1 jr ai1 p

 .
..
..

 ..
.
.




 air j1 · · · air jr air p 
alj1
· · · aljr
alp
Of course you can assume l ∈
/ {i1 , · · · , ir } because there is nothing to prove if the lth row is one of
the chosen ones. The above matrix has determinant 0. This is because if p ∈
/ {j1 , · · · , jr } then the
above would be a submatrix of A which is too large to have non zero determinant. On the other
hand, if p ∈ {j1 , · · · , jr } then the above matrix has two columns which are equal so its determinant
is still 0.
Expand the determinant of the above matrix along the last column. Let Ck denote the cofactor
associated with the entry aik p . This is not dependent on the choice of p. Remember, you delete the
column and the row the entry is in and take the determinant of what is left and multiply by −1
raised to an appropriate power. Let C denote the cofactor associated with alp . This is given to be
nonzero, it being the determinant of the matrix


ai1 j1 · · · ai1 jr
 .

..
 .

.
 .

air j1 · · · air jr
∑r
∑r
∑r
k
Thus 0 = alp C + k=1 Ck aik p which implies alp = k=1 −C
k=1 mk aik p . Since this is true
C aik p ≡
for every p and since mk does not depend on p, this has shown the lth row is a linear combination
of the i1 , i2 , · · · , ir rows. Corollary 8.6.8 The determinant rank equals the row rank.
Proof: From Theorem 8.6.7, the row rank is no larger than the determinant rank. Could the row
rank be smaller than the determinant rank? If so, there exist p rows for p < r such that the span of
these p rows equals the row space. But this implies that the r × r sub-matrix whose determinant is
nonzero also has row rank no larger than p which is impossible if its determinant is to be nonzero
because at least one row is a linear combination of the others. Corollary 8.6.9 If A has determinant rank r, then there exist r columns of the matrix such that
every other column is a linear combination of these r columns. Also the column rank equals the
determinant rank.
146
CHAPTER 8. RANK OF A MATRIX
Proof: This follows from the above by considering AT . The rows of AT are the columns of A
and the determinant rank of AT and A are the same. Therefore, from Corollary 8.6.8, column rank
of A = row rank of AT = determinant rank of AT = determinant rank of A. The following theorem is of fundamental importance and ties together many of the ideas presented
above.
Theorem 8.6.10 Let A be an n × n matrix. Then the following are equivalent.
1. det (A) = 0.
2. A, AT are not one to one.
3. A is not onto.
Proof: Suppose det (A) = 0. Then the determinant rank of A = r < n. Therefore, there exist r
columns such that every other column is a linear combination of these columns by Theorem 8.6.7.
In particular, it follows
that for some m, the )mth column is a linear combination of all the others.
(
Thus letting A =
where the columns are denoted by ai , there exists
a1 · · · am · · · an
scalars, αi such that
∑
am =
α k ak .
k̸=m
Now consider the column vector x ≡
(
α1
···
Ax = −am +
−1
∑
···
)T
αn
. Then
αk ak = 0.
k̸=m
Since also A0 = 0, it follows A is not one to one. Similarly, AT is not one to one by the same
argument applied to AT . This verifies that 1.) implies 2.).
Now suppose 2.). Then since AT is not one to one, it follows there exists x ̸= 0 such that
AT x = 0.
Taking the transpose of both sides yields
xT A = 0
where the 0 is a 1 × n matrix or row vector. Now if Ay = x, then
(
)
2
|x| = xT (Ay) = xT A y = 0y = 0
contrary to x ̸= 0. Consequently there can be no y such that Ay = x and so A is not onto. This
shows that 2.) implies 3.).
Finally, suppose 3.). If 1.) does not hold, then det (A) ̸= 0 but then from Theorem 7.1.14 A−1
exists and so for every y ∈ Fn there exists a unique x ∈ Fn such that Ax = y. In fact x = A−1 y.
Thus A would be onto contrary to 3.). This shows 3.) implies 1.) Corollary 8.6.11 Let A be an n × n matrix. Then the following are equivalent.
1. det(A) ̸= 0.
2. A and AT are one to one.
3. A is onto.
Proof: This follows immediately from the above theorem. Corollary 8.6.12 Let A be an invertible n×n matrix. Then A equals a finite product of elementary
matrices.
8.7. EXERCISES
147
Proof: Since A−1 is given to exist, det (A) ̸= 0 and it follows A must have rank n and so the
row reduced echelon form of A is I. Therefore, by Theorem 8.1.9 there is a sequence of elementary
matrices, E1 , · · · , Ep which accomplish successive row operations such that
(Ep Ep−1 · · · E1 ) A = I.
−1
−1
But now multiply on the left on both sides by Ep−1 then by Ep−1
and then by Ep−2
etc. until you
get
−1
A = E1−1 E2−1 · · · Ep−1
Ep−1
and by Theorem 8.1.9 each of these in this product is an elementary matrix. 8.7
Exercises
1. Let {u1 , · · · , un } be vectors in Rn . The parallelepiped determined by these vectors
P (u1 , · · · , un )
{
is defined as
P (u1 , · · · , un ) ≡
n
∑
}
tk uk : tk ∈ [0, 1] for all k
.
k=1
Now let A be an n × n matrix. Show that
{Ax : x ∈ P (u1 , · · · , un )}
is also a parallelepiped.
2. In the context of Problem 1, draw P (e1 , e2 ) where e1 , e2 are the standard basis vectors for
R2 . Thus e1 = (1, 0) , e2 = (0, 1) . Now suppose
(
)
1 1
E=
0 1
where E is the elementary matrix which takes the third row and adds to the first. Draw
{Ex : x ∈ P (e1 , e2 )} .
In other words, draw the result of doing E to the vectors in P (e1 , e2 ). Next draw the results
of doing the other elementary matrices to P (e1 , e2 ).
3. In the context of Problem 1, either draw or describe the result of doing elementary matrices
to P (e1 , e2 , e3 ). Describe geometrically the conclusion of Corollary 8.6.12.
4. Determine
(
1
(a)
0

1

(b)  0
0
which matrices are in row reduced echelon form.
)

1 1
2 0

(c)
 0 0
1 7

0 0
0 0 0

0 1 2 
0 0 0
0
1
0
0
2
0
0
0
1

5

4 
3
5. Row reduce the following matrices to obtain the row reduced echelon form. List the pivot
columns in the original matrix.
148
CHAPTER 8. RANK OF A MATRIX

1 2

(a)  2 1
1 1

1 2
 2 1

(b) 
 3 0
3 2
0
2
0
3
−2
0
1

3

2 
3


1 2 1 3


(c)  −3 2 1 0 
3 2 1 1





6. Find the rank of the following matrices. If the rank is r, identify r columns in the original matrix which have the property that every other column may be written as a linear
combination of these. Also find a basis for the row and column spaces of the matrices.



(a) 




(b) 




(c) 

1
3
2
0
2
2
1
2
0
1
0
1
1
4
2
0
0
1
1
2
0
1
0
0
0
0
0
0
1
3
1
2
0
2
1
1








(d) 









(e) 

2
12
5
7
1 2
1 6
0 2
0 3
2
8
3
4

0
0
0
0
1
3
1
2
0
2
1
1
2
6
2
4
0
0
0
0
1
5
2
3
0
4
2
2
0
0
0
0
1
3
1
2
0
2
1
1
2
6
2
4
1
1
0
0
1
5
2
3
2
1
1
1














7. Suppose A is an m × n matrix. Explain why the rank of A is always no larger than min (m, n) .
8. A matrix A is called a projection if A2 = A.

2

 1
−1
Here is a matrix.

0 2

1 2 
0 −1
Show that this is a projection. Show that a vector in the column space of a projection matrix
is left unchanged by multiplication by A.
((
) (
) ( ))
1
2
1
9. Let H denote span
,
,
. Find the dimension of H and determine a
2
4
3
basis.
  
    
1
2
1
0
  
    
10. Let H denote span  2  ,  4  ,  3  ,  1  . Find the dimension of H and de0
0
1
1
termine a basis.
  
    
1
1
1
0
  
    
11. Let H denote span  2  ,  4  ,  3  ,  1  . Find the dimension of H and de0
0
1
1
termine a basis.
8.7. EXERCISES
149
{
}
12. Let M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : u3 = u1 = 0 . Is M a subspace? Explain.
{
}
13. Let M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : u3 ≥ u1 . Is M a subspace? Explain.
{
}
14. Let w ∈ R4 and let M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : w · u = 0 . Is M a subspace? Explain.
{
}
15. Let M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : ui ≥ 0 for each i = 1, 2, 3, 4 . Is M a subspace? Explain.
16. Let w, w1 be given vectors in R4 and define
{
}
M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : w · u = 0 and w1 · u = 0 .
Is M a subspace? Explain.
{
}
17. Let M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : |u1 | ≤ 4 . Is M a subspace? Explain.
{
}
18. Let M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : sin (u1 ) = 1 . Is M a subspace? Explain.
19. Study the definition of span. Explain what is meant by the span of a set of vectors. Include
pictures.
20. Suppose {x1 , · · · , xk } is a set of vectors from Fn . Show that span (x1 , · · · , xk ) contains 0.
21. Study the definition of linear independence. Explain in your own words what is meant by
linear independence and linear dependence. Illustrate with pictures.
22. Use Corollary 8.5.18 to prove the following theorem: If A, B are n × n matrices and if AB = I,
then BA = I and B = A−1 . Hint: First note that if AB = I, then it must be the case that
A is onto. Explain why this requires span (columns of A) = Fn . Now explain why, using the
corollary that this requires A to be one to one. Next explain why A (BA − I) = 0 and why
the fact that A is one to one implies BA = I.
23. Here are three vectors. Determine whether they are linearly independent or linearly dependent.
(
)T (
)T (
)T
, 2 0 1
, 3 0 0
1 2 0
24. Here are three vectors. Determine whether they are linearly independent or linearly dependent.
(
)T (
)T (
)T
, 2 2 1
, 0 2 2
4 2 0
25. Here are three vectors. Determine whether they are linearly independent or linearly dependent.
(
)T (
)T (
)T
, 4 5 1
, 3 1 0
1 2 3
26. Here are four vectors. Determine whether they span R3 . Are these vectors linearly independent?
(
)T (
)T (
)T (
)T
, 4 3 3
, 3 1 0
, 2 4 6
1 2 3
27. Here are four vectors. Determine whether they span R3 . Are these vectors linearly independent?
(
)T (
)T (
)T (
)T
, 4 3 3
, 3 2 0
, 2 4 6
1 2 3
28. Determine whether the following vectors are a basis for R3 . If they are, explain why they are
and if they are not, give a reason and tell whether they span R3 .
(
)T (
)T (
)T (
)T
, 4 3 3
, 1 2 0
, 2 4 0
1 0 3
150
CHAPTER 8. RANK OF A MATRIX
29. Determine whether the following vectors are a basis for R3 . If they are, explain why they are
and if they are not, give a reason and tell whether they span R3 .
(
1 0
3
)T (
, 0
1
0
)T (
, 1 2
)T
0
30. Determine whether the following vectors are a basis for R3 . If they are, explain why they are
and if they are not, give a reason and tell whether they span R3 .
(
1 0
3
)T (
, 0 1
0
)T (
, 1 2
0
)T (
, 0 0
)T
0
31. Determine whether the following vectors are a basis for R3 . If they are, explain why they are
and if they are not, give a reason and tell whether they span R3 .
(
1 0
3
)T (
, 0 1
32. Consider the vectors of the form
0
)T (
, 1 1
3
)T (
, 0 0
)T
0





 2t + 3s



:
s,
t
∈
R
.
 s−t 




t+s
Is this set of vectors a subspace of R3 ? If so, explain why, give a basis for the subspace and
find its dimension.
33. Consider the vectors of the form

2t + 3s + u




s−t




t+s



u







 : s, t, u ∈ R .






Is this set of vectors a subspace of R4 ? If so, explain why, give a basis for the subspace and
find its dimension.
34. Consider the vectors of the form

2t + u



 t + 3u



 t+s+v



u








 : s, t, u, v ∈ R .





Is this set of vectors a subspace of R4 ? If so, explain why, give a basis for the subspace and
find its dimension.
35. If you have 5 vectors in F5 and the vectors are linearly independent, can it always be concluded
they span F5 ? Explain.
36. If you have 6 vectors in F5 , is it possible they are linearly independent? Explain.
37. Suppose A is an m × n matrix and {w1 , · · · , wk } is a linearly independent set of vectors in
A (Fn ) ⊆ Fm . Now suppose A (zi ) = wi . Show {z1 , · · · , zk } is also independent.
38. Suppose V, W are subspaces of Fn . Show V ∩ W defined to be all vectors which are in both V
and W is a subspace also.
8.7. EXERCISES
151
39. Suppose V and W both have dimension equal to 7 and they are subspaces of F10 . What are
the possibilities for the dimension of V ∩ W ? Hint: Remember that a linear independent set
can be extended to form a basis.
40. Suppose V has dimension p and W has dimension q and they are each contained in a subspace,
U which has dimension equal to n where n > max (p, q) . What are the possibilities for the
dimension of V ∩ W ? Hint: Remember that a linear independent set can be extended to form
a basis.
41. If b ̸= 0, can the solution set of Ax = b be a plane through the origin? Explain.
42. Suppose a system of equations has fewer equations than variables and you have found a solution
to this system of equations. Is it possible that your solution is the only one? Explain.
43. Suppose a system of linear equations has a 2 × 4 augmented matrix and the last column is a
pivot column. Could the system of linear equations be consistent? Explain.
44. Suppose the coefficient matrix of a system of n equations with n variables has the property
that every column is a pivot column. Does it follow that the system of equations must have a
solution? If so, must the solution be unique? Explain.
45. Suppose there is a unique solution to a system of linear equations. What must be true of the
pivot columns in the augmented matrix.
46. State whether each of the following sets of data are possible for the matrix equation Ax = b.
If possible, describe the solution set. That is, tell whether there exists a unique solution no
solution or infinitely many solutions.
(a) A is a 5 × 6 matrix, rank (A) = 4 and rank (A|b) = 4. Hint: This says b is in the span
of four of the columns. Thus the columns are not independent.
(b) A is a 3 × 4 matrix, rank (A) = 3 and rank (A|b) = 2.
(c) A is a 4 × 2 matrix, rank (A) = 4 and rank (A|b) = 4. Hint: This says b is in the span
of the columns and the columns must be independent.
(d) A is a 5 × 5 matrix, rank (A) = 4 and rank (A|b) = 5. Hint: This says b is not in the
span of the columns.
(e) A is a 4 × 2 matrix, rank (A) = 2 and rank (A|b) = 2.
47. Suppose A is an m × n matrix in which m ≤ n. Suppose also that the rank of A equals m.
Show that A maps Fn onto Fm . Hint: The vectors e1 , · · · , em occur as columns in the row
reduced echelon form for A.
48. Suppose A is an m × n matrix in which m ≥ n. Suppose also that the rank of A equals n.
Show that A is one to one. Hint: If not, there exists a vector x such that Ax = 0, and this
implies at least one column of A is a linear combination of the others. Show this would require
the column rank to be less than n.
49. Explain why an n × n matrix A is both one to one and onto if and only if its rank is n.
50. Suppose A is an m × n matrix and B is an n × p matrix. Show that
dim (ker (AB)) ≤ dim (ker (A)) + dim (ker (B)) .
Hint: Consider the subspace, B (Fp ) ∩ ker (A) and suppose a basis for this subspace is
{w1 , · · · , wk } .
Now suppose {u1 , · · · , ur } is a basis for ker (B) . Let {z1 , · · · , zk } be such that Bzi = wi and
argue that
ker (AB) ⊆ span (u1 , · · · , ur , z1 , · · · , zk ) .
152
CHAPTER 8. RANK OF A MATRIX
Here is how you do this. Suppose ABx = 0. Then Bx ∈ ker (A)∩B (Fp ) and so Bx =
showing that
k
∑
x−
zi ∈ ker (B) .
∑k
i=1
Bzi
i=1
51. Explain why Ax = 0 always has a solution even when A−1 does not exist.
(a) What can you conclude about A if the solution is unique?
(b) What can you conclude about A if the solution is not unique?
52. Suppose det (A − λI) = 0. Show using Theorem 9.2.9 there exists x ̸= 0 such that
(A − λI) x = 0.
53. Let A be an n × n matrix and let x be a nonzero vector such that Ax = λx for some scalar
λ. When this occurs, the vector x is called an eigenvector and the scalar λ is called an
eigenvalue. It turns out that not every number is an eigenvalue. Only certain ones are.
Why? Hint: Show that if Ax = λx, then (A − λI) x = 0. Explain why this shows that
(A − λI) is not one to one and not onto. Now use Theorem 9.2.9 to argue det (A − λI) = 0.
What sort of equation is this? How many solutions does it have?
54. Let m < n and let A be an m × n matrix. Show that A is not one to one. Hint: Consider
the n × n matrix A1 which is of the form
(
)
A
A1 ≡
0
where the 0 denotes an (n − m) × n matrix of zeros. Thus det A1 = 0 and so A1 is not one to
one. Now observe that A1 x is the vector
(
)
Ax
A1 x =
0
which equals zero if and only if Ax = 0. Do this using the Fredholm alternative.
55. Let A be an m × n real matrix and let b ∈ Rm . Show there exists a solution, x to the system
AT Ax = AT b
(
)T
Next show that if x, x1 are(two solutions,
then Ax = Ax1 . Hint: First show that AT A =
)
AT A. Next show if x ∈ ker AT A , then Ax = 0. Finally apply the Fredholm alternative. This
will give existence of a solution.
56. Show that in the context of Problem 55 that if x is the solution there, then |b − Ax| ≤ |b − Ay|
for every y. Thus Ax is the point of A (Rn ) which is closest to b of every point in A (Rn ).
This gives an approach to least squares based on rank considerations. In Problem 55, x is the
least squares solution to Ax = b, which may not even have an actual solution.
{
}
2
57. Let A be an n × n matrix and consider the matrices I, A, A2 , · · · , An . Explain why there
exist scalars, ci not all zero such that
n
∑
2
ci Ai = 0.
i=1
Then argue there exists a polynomial, p (λ) of the form
λm + dm−1 λm−1 + · · · + d1 λ + d0
such that p (A) = 0 and if q (λ) is another polynomial such that q (A) = 0, then q (λ) is of the
form p (λ) l (λ) for some polynomial, l (λ) . This extra special polynomial, p (λ) is called the
2
minimal polynomial. Hint: You might consider an n × n matrix as a vector in Fn .
Chapter 9
Linear Transformations
9.1
Linear Transformations
An m × n matrix can be used to transform vectors in Fn to vectors in Fm through the use of matrix
multiplication.
(
)
1 2 0
Example 9.1.1 Consider the matrix
. Think of it as a function which takes vectors
2 1 0


x


in F3 and makes them in to vectors in F2 as follows. For  y  a vector in F3 , multiply on the
z
left by the given matrix to obtain the vector in F2 . Here are some numerical examples.

 

(
)
( ) (
)
(
)
1
1
1 2 0  
5
1 2 0 
−3

,
,
 2 =
 −2  =
2 1 0
4
2 1 0
0
3
3




(
)
(
)
(
)
(
)
10
0
1 2 0 
20
1 2 0 
14


,
,
 5 =
 7 =
2 1 0
25
2 1 0
7
−3
3
More generally,
(
1
2
2
1
0
0
)


(
)
x
x + 2y


 y =
2x + y
z
The idea is to define a function which takes vectors in F3 and delivers new vectors in F2 .
This is an example of something called a linear transformation.
Definition 9.1.2 Let T : Fn 7→ Fm be a function. Thus for each x ∈ Fn , T x ∈ Fm . Then T is a
linear transformation if whenever α, β are scalars and x1 and x2 are vectors in Fn ,
T (αx1 + βx2 ) = α1 T x1 + βT x2 .
A linear transformation is also called a homomorphism. In the case that T is in addition to this
one to one and onto, it is sometimes called an isomorphism.
The last two terms are typically used more in abstract algebra than in linear algebra so in this
book, such mappings will be referred to as linear transformations. In sloppy language, it distributes
across vector addition and you can factor out the scalars.
153
154
CHAPTER 9. LINEAR TRANSFORMATIONS
In words, linear transformations distribute across + and allow you to factor out scalars. At this
point, recall the properties of matrix multiplication. The pertinent property is 5.14 on Page 73.
Recall it states that for a and b scalars,
A (aB + bC) = aAB + bAC
In particular, for A an m×n matrix and B and C, n×1 matrices (column vectors) the above formula
holds which is nothing more than the statement that matrix multiplication gives an example of a
linear transformation.
The reason this concept is so important is there are many examples of things which are linear
transformations. You might remember from calculus that the operator which consists of taking the
derivative is a linear transformation. That is, if f, g are functions (vectors) and α, β are numbers
(scalars)
d
d
d
(αf + βg) = α f + β g
dx
dx
dx
Another example of a linear transformation is that of rotation through an angle. For example,
I may want to rotate every vector through an angle of 45 degrees. Such a rotation would achieve
something like the following if applied to each vector corresponding to points on the picture which
is standing upright.
T (a + b)
More generally, denote a rotation by T . Why is such a transformation linear? Consider the
following picture which illustrates a rotation.
T (a)
T (b)
b
a+
b
b
a
To get T (a + b) , you can add T a and T b. Here is why. If you add T a to T b you get the
diagonal of the parallelogram determined by T a and T b. This diagonal also results from rotating
the diagonal of the parallelogram determined by a and b. This is because the rotation preserves
all angles between the vectors as well as their lengths. In particular, it preserves the shape of this
parallelogram. Thus both T a + T b and T (a + b) give the same directed line segment. Thus T
distributes across + where + refers to vector addition. Similarly, if k is a number T ka = kT a
(draw a picture) and so you can factor out scalars also. Thus rotations are an example of a linear
transformation.
9.2. CONSTRUCTING THE MATRIX OF A LINEAR TRANSFORMATION
155
Definition 9.1.3 A linear transformation is called one to one (often written as 1 − 1) if it never
takes two different vectors to the same vector. Thus T is one to one if whenever x ̸= y
T x ̸= T y.
Equivalently, if T (x) = T (y) , then x = y.
In the case that a linear transformation comes from matrix multiplication, it is common usage
to refer to the matrix as a one to one matrix when the linear transformation it determines is one to
one.
Definition 9.1.4 A linear transformation mapping Fn to Fm is called onto if whenever y ∈ Fm
there exists x ∈ Fn such that T (x) = y.
Thus T is onto if everything in Fm gets hit. In the case that a linear transformation comes from
matrix multiplication, it is common to refer to the matrix as onto when the linear transformation it
determines is onto. Also it is common usage to write T Fn , T (Fn ) ,or Im (T ) as the set of vectors of
Fm which are of the form T x for some x ∈ Fn . In the case that T is obtained from multiplication by
an m × n matrix A, it is standard to simply write A (Fn ), AFn , or Im (A) to denote those vectors in
Fm which are obtained in the form Ax for some x ∈ Fn .
9.2
Constructing The Matrix Of A Linear Transformation
It turns out that if T is any linear transformation which maps Fn to Fm , there is always an m × n
matrix A with the property that
Ax = T x
(9.1)
for all x ∈ Fn . Here is why. Suppose T : Fn 7→ Fm is a linear transformation and you want to find
the matrix defined by this linear transformation as described in 9.1. Then if x ∈ Fn it follows
x=
n
∑
xi ei
i=1
where ei is the vector which has zeros in every slot but the ith and a 1 in this slot. Then since T is
linear,
n
∑
Tx =
xi T (ei )
i=1

|

=  T (e1 )
|
···



x1
|

  .. 


T (en )  
 .  ≡ A
|
xn

x1
.. 

. 
xn
and so you see that the matrix desired is obtained from letting the ith column equal T (ei ) . We state
this as the following theorem.
Theorem 9.2.1 Let T be a linear transformation from Fn to Fm . Then the matrix A satisfying 9.1
is given by


|
|


 T (e1 ) · · · T (en ) 
|
|
where T ei is the ith column of A.
156
9.2.1
CHAPTER 9. LINEAR TRANSFORMATIONS
Rotations in R2
Sometimes you need to find a matrix which represents a given linear transformation which is described in geometrical terms. The idea is to produce a matrix which you can multiply a vector by
to get the same thing as some geometrical description. A good example of this is the problem of
rotation of vectors discussed above. Consider the problem of rotating through an angle of θ.
Example 9.2.2 Determine the matrix which represents the linear transformation defined by rotating
every vector through an angle of θ.
(
)
( )
1
0
Let e1 ≡
and e2 ≡
. These identify the geometric vectors which point along the
0
1
positive x axis and positive y axis as shown.
e2
6
(− sin(θ), cos(θ))
I
T (e2 )
T (e1 )
(cos(θ), sin(θ))
θ
θ
- e1
From the above, you only need to find T e1 and T e2 , the first being the first column of the
desired matrix A and the second being the second column. From the definition of the cos, sin the
coordinates of T (e1 ) are as shown in the picture. The coordinates of T (e2 ) also follow from simple
trigonometry. Thus
(
)
(
)
cos θ
− sin θ
T e1 =
, T e2 =
.
sin θ
cos θ
Therefore, from Theorem 9.2.1,
(
A=
cos θ
sin θ
− sin θ
cos θ
)
For those who prefer a more algebraic approach, the definition of (cos (θ) , sin (θ)) is as the x and
y coordinates of the point (1, 0) . Now the point of the vector from (0, 0) to (0, 1), e2 is exactly π/2
further along along the unit circle. Therefore, when it is rotated through an angle of θ the x and y
coordinates are given by
(x, y) = (cos (θ + π/2) , sin (θ + π/2)) = (− sin θ, cos θ) .
Example 9.2.3 Find the matrix of the linear transformation which is obtained by first rotating all
vectors through an angle of ϕ and then through an angle θ. Thus you want the linear transformation
which rotates all angles through an angle of θ + ϕ.
Let Tθ+ϕ denote the linear transformation which rotates every vector through an angle of θ + ϕ.
Then to get Tθ+ϕ , you could first do Tϕ and then do Tθ where Tϕ is the linear transformation which
rotates through an angle of ϕ and Tθ is the linear transformation which rotates through an angle of
θ. Denoting the corresponding matrices by Aθ+ϕ , Aϕ , and Aθ , you must have for every x
Aθ+ϕ x = Tθ+ϕ x = Tθ Tϕ x = Aθ Aϕ x.
9.2. CONSTRUCTING THE MATRIX OF A LINEAR TRANSFORMATION
157
Consequently, you must have
(
Aθ+ϕ
=
(
=
)
− sin (θ + ϕ)
= Aθ Aϕ
cos (θ + ϕ)
)(
)
− sin θ
cos ϕ − sin ϕ
.
cos θ
sin ϕ cos ϕ
cos (θ + ϕ)
sin (θ + ϕ)
cos θ
sin θ
You know how to multiply matrices. Do so to the pair on the right. This yields
(
)
cos (θ + ϕ) − sin (θ + ϕ)
sin (θ + ϕ) cos (θ + ϕ)
(
)
cos θ cos ϕ − sin θ sin ϕ − cos θ sin ϕ − sin θ cos ϕ
=
.
sin θ cos ϕ + cos θ sin ϕ cos θ cos ϕ − sin θ sin ϕ
Don’t these look familiar? They are the usual trig. identities for the sum of two angles derived here
using linear algebra concepts.
You do not have to stop with two dimensions. You can consider rotations and other geometric
concepts in any number of dimensions. This is one of the major advantages of linear algebra. You can
break down a difficult geometrical procedure into small steps, each corresponding to multiplication by
an appropriate matrix. Then by multiplying the matrices, you can obtain a single matrix which can
give you numerical information on the results of applying the given sequence of simple procedures.
That which you could never visualize can still be understood to the extent of finding exact numerical
answers. Another example follows.
Example 9.2.4 Find the matrix of the linear transformation which is obtained by first rotating all
vectors through an angle of π/6 and then reflecting through the x axis.
As shown in Example 9.2.3, the matrix of the transformation which involves rotating through an
angle of π/6 is
(
) ( √
)
1
1
cos (π/6) − sin (π/6)
3
−
2
√2
=
1
1
sin (π/6) cos (π/6)
2
2 3
The matrix for the transformation which reflects all vectors through the x axis is
(
)
1 0
.
0 −1
Therefore, the matrix of the linear transformation which first rotates through π/6 and then reflects
through the x axis is
(
)( √
) ( √
)
1
1
1 0
− 12
3
− 12
2 3
2
√
√
=
.
1
1
0 −1
− 12 − 12 3
2
2 3
9.2.2
Rotations About A Particular Vector
The problem is to find the matrix of the linear transformation which rotates all vectors about a
given unit vector u which is possibly not one of the coordinate vectors i, j, or k. Suppose for |c| ̸= 1
√
u = (a, b, c) , a2 + b2 + c2 = 1.
First I will produce a matrix which maps u to k such that the right handed rotation about k
corresponds to the right handed rotation about u. Then I will rotate about k and finally, I will
multiply by the inverse of the first matrix to get the desired result.
158
CHAPTER 9. LINEAR TRANSFORMATIONS
To begin, find vectors w, v such that w × v = u. Let
(
)
b
a
w = −√
,√
,0 .
a2 + b2
a2 + b2
wI
u
This vector is clearly perpendicular to u. Then v = (a, b, c) × w ≡ u × w. Thus from the
geometric description of the cross product, w × v = u. Computing the cross product gives
(
)
b
a
v = (a, b, c) × − √
,√
,0
a2 + b2
a2 + b2
)
(
b
a2
b2
a
, −c √
,√
+√
= −c √
(a2 + b2 )
(a2 + b2 )
(a2 + b2 )
(a2 + b2 )
Now I want to have T w = i, T v = j, T u = k. What does this? It is the inverse of the matrix
which takes i to w, j to v, and k to u. This matrix is

− √a2b+b2



√ a
a2 +b2
0
Its inverse is

−√



−√
c
a
(a2 +b2 )
c
−√ 2 2 b
(a +b )
2
2
√a +b
a2 +b2
√
1
b
(a2 +b2 )
c
−√ 2 2 a
(a +b )
1
a
(a2 +b2 )
c
−√ 2 2 b
(a +b )
a
b
a


b 
.
c
0


√
(a2 + b2 ) 

c
Therefore, the matrix which does the rotating is




−√
− √a2b+b2
c
a
(a2 +b2 )
− √ 2c 2 b
(a +b )
2
2
√a +b
a2 +b2
√ a
a2 +b2
0

−√
1
(a2 +b2 )
b

 −√ c
a

(a2 +b2 )
a
a

− sin θ
cos θ
0
cos θ


b   sin θ
0
c
√
1
(a2 +b2 )
−√
a
c
b
(a2 +b2 )
0


√
(a2 + b2 ) 

b
c
This yields a matrix whose columns are

2



0

0 ·
1
b cos θ+c2 a2 cos θ+a4 +a2 b2
a2 +b2
−ba cos θ+cb2 sin θ+ca2 sin θ+c2 ab cos θ+ba3 +b3 a
a2 +b2


,
− (sin θ) b − (cos θ) ca + ca



−ba cos θ−ca2 sin θ−cb2 sin θ+c2 ab cos θ+ba3 +b3 a
a2 +b2
a2 cos θ+c2 b2 cos θ+a2 b2 +b4
a2 +b2

(sin θ) a − (cos θ) cb + cb

(sin θ) b − (cos θ) ca + ca


 − (sin θ) a − (cos θ) cb + cb 
( 2
)
a + b2 cos θ + c2


,
9.2. CONSTRUCTING THE MATRIX OF A LINEAR TRANSFORMATION
159
Using the assumption that u is a unit vector so that a2 + b2 + c2 = 1, it follows the desired matrix
is

cos θ − a2 cos θ + a2
−ba cos θ + ba − c sin θ
(sin θ) b − (cos θ) ca + ca


−b2 cos θ + b2 + cos θ
− (sin θ) a − (cos θ) cb + cb 
 −ba cos θ + ba + c sin θ
)
(
− (sin θ) b − (cos θ) ca + ca (sin θ) a − (cos θ) cb + cb
1 − c2 cos θ + c2

This was done under the assumption that |c| ̸= 1. However, if this condition does not hold, you
can verify directly that the above still gives the correct answer.
9.2.3
Projections
In Physics it is important to consider the work done by a force field on an object. This involves the
concept of projection onto a vector. Suppose you want to find the projection of a vector v onto the
given vector u, denoted by proju (v) This is done using the dot product as follows.
(v · u)
proju (v) =
u
u·u
Because of properties of the dot product, the map v7→proju (v) is linear,
(
)
(v · u)
(w · u)
αv+βw · u
proju (αv+βw) =
u=α
u+β
u
u·u
u·u
u·u
= α proju (v) + β proju (w) .
T
Example 9.2.5 Let the projection map be defined above and let u = (1, 2, 3) . Does this linear
transformation come from multiplication by a matrix? If so, what is the matrix?
You can find this matrix in the same way as in the previous example. Let ei denote the vector in Rn which has a 1 in the ith position and a zero everywhere else. Thus a typical vector
T
x = (x1 , · · · , xn ) can be written in a unique way as
x=
n
∑
xj ej .
j=1
From the way you multiply a matrix by a vector, it follows that proju (ei ) gives the ith column of
the desired matrix. Therefore, it is only necessary to find
( e ·u )
i
proju (ei ) ≡
u
u·u
For the given vector in the example, this implies the columns
 



1
1
1   2 
 3 
 2 ,
 2 ,

14
14
14
3
3
Hence the matrix is

1 2
1 
 2 4
14
3 6

3

6 .
9
of the desired matrix are

1

2 .
3
160
CHAPTER 9. LINEAR TRANSFORMATIONS
9.2.4
Matrices Which Are One To One Or Onto
Lemma 9.2.6 Let A be an m×n matrix. Then A (Fn ) = span (a1 , · · · , an ) where a1 , · · · , an denote
T
the columns of A. In fact, for x = (x1 , · · · , xn ) ,
Ax =
n
∑
xk ak .
k=1
Proof: This follows from the definition of matrix multiplication in Definition 5.1.9 on Page 68.
The following is a theorem of major significance. First here is an interesting observation.
Observation 9.2.7 Let A be an m × n matrix. Then A is one to one if and only if Ax = 0 implies
x = 0.
Here is why: A0 = A (0 + 0) = A0 + A0 and so A0 = 0.
Now suppose A is one to one and Ax = 0. Then since A0 = 0, it follows x = 0. Thus if A is one
to one and Ax = 0, then x = 0.
Next suppose the condition that Ax = 0 implies x = 0 is valid. Then if Ax = Ay, then
A (x − y) = 0 and so from the condition, x − y = 0 so that x = y. Thus A is one to one.
Theorem 9.2.8 Suppose A is an n × n matrix. Then A is one to one if and only if A is onto. Also,
if B is an n × n matrix and AB = I, then it follows BA = I.
Proof: First suppose A is one to one. Consider the vectors, {Ae1 , · · · , Aen } where ek is the
column vector which is all zeros except for a 1 in the k th position. This set of vectors is linearly
independent because if
n
∑
ck Aek = 0,
k=1
then since A is linear,
(
A
n
∑
)
ck ek
=0
k=1
and since A is one to one, it follows
n
∑
ck ek = 0
k=1
which implies each ck = 0. Therefore, {Ae1 , · · · , Aen } must be a basis for Fn by Corollary 8.5.18 on
Page 138. It follows that for y ∈ Fn there exist constants, ci such that
( n
)
n
∑
∑
y=
ck Aek = A
ck ek
k=1
k=1
showing that, since y was arbitrary, A is onto.
Next suppose A is onto. This implies the span of the columns of A equals Fn and by Corollary
T
8.5.18 this implies the columns of A are independent. If Ax = 0, then letting x = (x1 , · · · , xn ) , it
follows
n
∑
x i ai = 0
i=1
and so each xi = 0. If Ax = Ay, then A (x − y) = 0 and so x = y. This shows A is one to one.
Now suppose AB = I. Why is BA = I? Since AB = I it follows B is one to one since otherwise,
there would exist, x ̸= 0 such that Bx = 0 and then ABx = A0 = 0 ̸= Ix. Therefore, from what
was just shown, B is also onto. In addition to this, A must be one to one because if Ay = 0, then
9.2. CONSTRUCTING THE MATRIX OF A LINEAR TRANSFORMATION
161
y = Bx for some x and then x = ABx = Ay = 0 showing y = 0. Now from what is given to be so,
it follows (AB) A = A and so using the associative law for matrix multiplication,
A (BA) − A = A (BA − I) = 0.
But this means (BA − I) x = 0 for all x since otherwise, A would not be one to one. Hence BA = I
as claimed. This theorem shows that if an n × n matrix B acts like an inverse when multiplied on one side
of A it follows that B = A−1 and it will act like an inverse on both sides of A.
The conclusion of this theorem pertains to square matrices only. For example, let


(
)
1 0
1 0 0


A =  0 1 , B =
(9.2)
1 1 −1
1 0
(
Then
BA =

but
1
0
1 0

AB =  1 1
1 0
0
1
)

0

−1  .
0
There is also an important characterization in terms of determinants. This is proved completely
in the section on the mathematical theory of the determinant.
Theorem 9.2.9 Let A be an n × n matrix and let TA denote the linear transformation determined
by A. Then the following are equivalent.
1. TA is one to one.
2. TA is onto.
3. det (A) ̸= 0.
9.2.5
The General Solution Of A Linear System
Recall the following definition which was discussed above.
Definition 9.2.10 T is a linear transformation if whenever x, y are vectors and a, b scalars,
T (ax + by) = aT x + bT y.
(9.3)
Thus linear transformations distribute across addition and pass scalars to the outside. A linear
system is one which is of the form
T x = b.
If T xp = b, then xp is called a particular solution to the linear system.
For example, if A is an m × n matrix and TA is determined by
TA (x) = Ax,
then from the properties of matrix multiplication, TA is a linear transformation. In this setting,
we will usually write A for the linear transformation as well as the matrix. There are many other
examples of linear transformations other than this. In differential equations, you will encounter
linear transformations which act on functions to give new functions. In this case, the functions are
considered as vectors. Don’t worry too much about this at this time. It will happen later. The
fundamental idea is that something is linear if 9.3 holds and if whenever a, b are scalars and x, y are
vectors ax + by is a vector. That is you can add vectors and multiply by scalars.
162
CHAPTER 9. LINEAR TRANSFORMATIONS
Definition 9.2.11 Let T be a linear transformation. Define
ker (T ) ≡ {x : T x = 0} .
In words, ker (T ) is called the kernel of T . As just described, ker (T ) consists of the set of all vectors
which T sends to 0. This is also called the null space of T . It is also called the solution space
of the equation T x = 0.
The above definition states that ker (T ) is the set of solutions to the equation,
T x = 0.
In the case where T is really a matrix, you have been solving such equations for quite some time.
However, sometimes linear transformations act on vectors which are not in Fn . There is more on this
in Chapter 16 on Page 16 and this is discussed more carefully then. However, consider the following
familiar example.
d
Example 9.2.12 Let dx
denote the linear transformation defined on X, the functions which are
(d)
defined on R and have a continuous derivative. Find ker dx
.
df
The example asks for functions, f which the property that dx
= 0. As you know from calculus,
(d)
these functions are the constant functions. Thus ker dx = constant functions.
When T is a linear transformation, systems of the form T x = 0 are called homogeneous systems. Thus the solution to the homogeneous system is known as ker (T ) .
Systems of the form T x = b where b ̸= 0 are called nonhomogeneous systems. It turns out
there is a very interesting and important relation between the solutions to the homogeneous systems
and the solutions to the nonhomogeneous systems.
Theorem 9.2.13 Suppose xp is a solution to the linear system,
Tx = b
Then if y is any other solution, there exists x ∈ ker (T ) such that
y = xp + x.
(
)
Proof: Consider y − xp ≡ y+ (−1) xp . Then T y − xp = T y − T xp = b − b = 0. Let
x ≡ y − xp .
Sometimes people remember the above theorem in the following form. The solutions to the
nonhomogeneous system, T x = b are given by xp + ker (T ) where xp is a particular solution to
T x = b.
I have been vague about what T is and what x is on purpose. This theorem is completely
algebraic in nature and will work whenever you have linear transformations. In particular, it will be
important in differential equations. For now, here is a familiar example.
Example 9.2.14 Let

1

A= 2
4
2
1
5
3
1
7

0

2 
2
Find ker (A). Equivalently, find the solution space to the system of equations Ax = 0.
9.2. CONSTRUCTING THE MATRIX OF A LINEAR TRANSFORMATION
163
This asks you to find {x : Ax = 0} . In other words you are asked to solve the system, Ax = 0.
T
Let x = (x, y, z, w) . Then this amounts to solving



 

x
0
1 2 3 0
 y 

  

=
  0 
 2 1 1 2 
 z 
4 5 7 2
0
w
This is the linear system
x + 2y + 3z = 0
2x + y + z + 2w = 0
4x + 5y + 7z + 2w = 0
and you know how to solve this using row
matrix

1

 2
4
operations, (Gauss Elimination). Set up the augmented

2 3 0 | 0

1 1 2 | 0 
5 7 2 | 0
Then row reduce to obtain the row reduced echelon form,


4
1 0 − 13
| 0
3




5
− 23 | 0  .
 0 1
3


0
0
0
0
| 0
This yields x = 13 z − 43 w and y = 23 w − 53 z. Thus ker (A) consists of vectors of the form,









 1
3


 5
5
2


3w − 3z  = z  −3



 1
z

0
w
1
3z
− 43 w


− 43

 2


+w 3



 0
1



.


Example 9.2.15 The general solution of a linear system of equations is just the set of all solutions. Find the general solution to the linear system,






x
1 2 3 0
9
 y 


 

= 7 
 2 1 1 2 
 z 
4 5 7 2
25
w
(
given that
)T
1 1
2
1
(
=
)T
x
y
z
w
is one solution.
Note the matrix on the left is the same as the matrix in Example 9.2.14. Therefore, from Theorem
9.2.13, you will obtain all solutions to the above linear system in the form
 1 
 4   
−3
3
1
 5 
 2   
 − 

  1 
3 +w 3 +
z



  .
 1 
 0   2 
0
1
1
164
9.3
CHAPTER 9. LINEAR TRANSFORMATIONS
Exercises
1. Study the definition of a linear transformation. State it from memory.
2. Show the map T : Rn 7→ Rm defined by T (x) = Ax where A is an m × n matrix and x is an
m × 1 column vector is a linear transformation.
3. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of π/3.
4. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of π/4.
5. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of −π/3.
6. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of 2π/3.
7. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of π/12. Hint: Note that π/12 = π/3 − π/4.
8. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of 2π/3 and then reflects across the x axis.
9. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of π/3 and then reflects across the x axis.
10. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of π/4 and then reflects across the x axis.
11. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of π/6 and then reflects across the x axis followed by a reflection across the y axis.
12. Find the matrix for the linear transformation which reflects every vector in R2 across the x
axis and then rotates every vector through an angle of π/4.
13. Find the matrix for the linear transformation which reflects every vector in R2 across the y
axis and then rotates every vector through an angle of π/4.
14. Find the matrix for the linear transformation which reflects every vector in R2 across the x
axis and then rotates every vector through an angle of π/6.
15. Find the matrix for the linear transformation which reflects every vector in R2 across the y
axis and then rotates every vector through an angle of π/6.
16. Find the matrix for the linear transformation which rotates every vector in R2 through an
angle of 5π/12. Hint: Note that 5π/12 = 2π/3 − π/4.
17. Find the matrix of the linear transformation which rotates every vector in R3 counter clockwise
about the z axis when viewed from the positive z axis through an angle of 30◦ and then reflects
through the xy plane.
z
6
-y
x
T
18. Find the matrix for proju (v) where u = (1, −2, 3) .
T
19. Find the matrix for proju (v) where u = (1, 5, 3) .
9.3. EXERCISES
165
T
20. Find the matrix for proju (v) where u = (1, 0, 3) .
21. Show that the function Tu defined by Tu (v) ≡ v − proju (v) is also a linear transformation.
22. Show that
⟨v − proju (v) , u⟩ ≡ (v − proju (v) , u) ≡ (v − proju (v)) · u = 0
and conclude every vector in Rn can be written as the sum of two vectors, one which is
perpendicular and one which is parallel to the given vector.
23. Here are some descriptions of functions mapping Rn to Rn .
(a) T multiplies the j th component of x by a nonzero number b.
(b) T replaces the ith component of x with b times the j th component added to the ith
component.
(c) T switches two components.
Show these functions are linear and describe their matrices.
24. In Problem 23, sketch the effects of the linear transformations on the unit square in R2 . Give
a geometric description of an arbitrary invertible matrix in terms of products of matrices of
these special matrices in Problem 23.
25. Let u = (a, b) be a unit vector in R2 . Find the matrix which reflects all vectors across this
vector.
u
1
Hint: You might want to notice that (a, b) = (cos θ, sin θ) for some θ. First rotate through −θ.
Next reflect through the x axis which is easy. Finally rotate through θ.
26. Let u be a unit vector. Show the linear transformation of the matrix I − 2uuT preserves all
distances and satisfies
(
)T (
)
I − 2uuT
I − 2uuT = I.
This matrix is called a Householder reflection. More generally, any matrix Q which satisfies
QT Q = QQT is called an orthogonal matrix. Show the linear transformation determined by
an orthogonal matrix always preserves the length of a vector in Rn . Hint: First either recall,
depending on whether you have done Problem 51 on Page 85, or show that for any matrix A,
⟨
⟩
⟨Ax, y⟩ = x,AT y
27. Suppose |x| = |y| for x, y ∈ Rn . The problem is to find an orthogonal transformation Q, (see
Problem 26) which has the property that Qx = y and Qy = x. Show
Q≡I −2
x−y
2
|x − y|
(x − y)
T
does what is desired.
28. Let a be a fixed vector. The function Ta defined by Ta v = a + v has the effect of translating
all vectors by adding a. Show this is not a linear transformation. Explain why it is not possible
to realize Ta in R3 by multiplying by a 3 × 3 matrix.
166
CHAPTER 9. LINEAR TRANSFORMATIONS
29. In spite of Problem 28 we can represent both translations and rotations by matrix multiplication at the expense of using higher dimensions. This is done by the homogeneous coordinates.
T
I will illustrate in R3 where most interest in this is found. For each vector v = (v1 , v2 , v3 ) ,
T
consider the vector in R4 (v1 , v2 , v3 , 1) . What happens when you do



1 0 0 a1
v1
 0 1 0 a  v 

2  2 


?
 0 0 1 a3   v3 
0 0
0
1
1
Describe how to consider both rotations and translations all at once by forming appropriate
4 × 4 matrices.
(
)
30. You want to add 1, 2, 3 to every point in R3 and then rotate about the z axis counter
(
)
clockwise through an angle of 30◦ . Find what happens to the point 1, 1, 1 .
31. Write the solution set of the following
solution space of the following system.

1 −1

 1 −2
3 −4
32. Using Problem 31 find the general

1

 1
3
system as the span of vectors and find a basis for the
  

x
0
2

  
1  y  =  0 .
0
z
5
solution to the following linear system.

  
−1 2
x
1

  
−2 1   y  =  2  .
−4 5
z
4
33. Write the solution set of the following
solution space of the following system.

0 −1

 1 −2
1 −4
system as the span of vectors and find a basis for the

  
2
x
0

  
1  y  =  0 .
5
z
0
34. Using Problem 33 find the general solution to the following linear system.


 

0 −1 2
x
1


 

 1 −2 1   y  =  −1  .
1 −4 5
z
1
35. Write the solution set of the following
solution space of the following system.

1 −1

 1 −2
3 −4
36. Using Problem 35 find the general

1

 1
3
system as the span of vectors and find a basis for the

  
2
x
0

  
0  y  =  0 .
4
z
0
solution to the following linear system.

  
−1 2
x
1

  
−2 0   y  =  2  .
−4 4
z
4
9.3. EXERCISES
167
37. Write the solution set of the following
solution space of the following system.

0 −1

 1 0
1 −2
system as the span of vectors and find a basis for the

  
2
x
0

  
1  y  =  0 .
5
z
0
38. Using Problem 37 find the general solution to the following linear system.

 


1
x
0 −1 2

 


 1 0 1   y  =  −1  .
1
z
1 −2 5
39. Write the solution set of the following system as
solution space of the following system.


1 0 1 1
 1 −1 1 0  




 3 −1 3 2  
3
3
0
3
the span of vectors and find a basis for the
x
y
z
w


 
 
=
 
0
0
0
0



.

40. Using Problem 39 find the general solution to the following linear system.


  
1 0 1 1
x
1
 1 −1 1 0   y   2 


  


 =  .
 3 −1 3 2   z   4 
3 3 0 3
w
3
41. Write the solution set of the following system as the span of vectors and find a basis for the
solution space of the following system.


  
1 1 0 1
x
0
 2 1 1 2  y   0 


  


 =  .
 1 0 1 1  z   0 
0 0 0 0
w
0
42. Using Problem 41 find the general solution

1 1 0 1
 2 1 1 2


 1 0 1 1
0 −1 1 1
to the following

 
x
 y  

 

=
 z  
w
linear system.

2
−1 

.
−3 
0
43. Give an example of a 3 × 2 matrix with the property that the linear transformation determined
by this matrix is one to one but not onto.
44. Write the solution set of the following system as
solution space of the following system.


1 1 0 1
 1 −1 1 0  




 3 1 1 2 
3
3
0
3
the span of vectors and find a basis for the
x
y
z
w


 
 
=
 
0
0
0
0



.

168
CHAPTER 9. LINEAR TRANSFORMATIONS
45. Using Problem 44 find the general solution to the following linear system.

  

1 1 0 1
x
1
 1 −1 1 0   y   2 


  


 =  .
 3 1 1 2  z   4 
3 3 0 3
w
3
46. Write the solution set of the following system as
solution space of the following system.


1 1 0 1
 2 1 1 2 




 1 0 1 1 
0 −1
1
1
47. Using Problem 46 find the general solution

1 1 0 1
 2 1 1 2


 1 0 1 1
0 −1 1 1
48. Find ker (A) for



A=

1
0
1
0
the span of vectors and find a basis for the
x
y
z
w


0
0
0
0
 
 
=
 
to the following

 
x
 y  

 

=
 z  
3
1
4
1


.

linear system.

2
−1 

.
−3 
w
2
2
4
2

1
2
1
3
1
1
2
3
2



.

Recall ker (A) is just the set of solutions to Ax = 0. It is the solution space to the system
Ax = 0.
49. Using Problem 48, find the general solution to the following linear system.






x1
1 2 3 2 1
11




x
 0 2 1 1 2  2   7 




=

  x3 


 1 4 4 3 3 
  18 
x
 4 
0 2 1 1 2
7
x5
50. Using Problem 48, find the general solution to the following linear system.






x1
1 2 3 2 1
6


x2  
 0 2 1 1 2 
  7 




x



3 =
 1 4 4 3 3 


13


 x4 
0 2 1 1 2
7
x5
51. Suppose Ax = b has a solution. Explain why the solution is unique precisely when Ax = 0
has only the trivial (zero) solution.
52. Show that if A is an m × n matrix, then ker (A) is a subspace.
9.3. EXERCISES
169
53. Verify the linear transformation determined by the matrix of 9.2 maps R3 onto R2 but the
linear transformation determined by this matrix is not one to one.
54. You are given a linear transformation T : Rn → Rm and you know that
T ai = b i
(
)−1
exists. Show that the matrix A of T with respect to the usual basis
a1 · · · an
vectors (T x = Ax) must be of the form
where
(
b1
···
)(
bn
a1
···
)−1
an
55. You have a linear transformation T and

 

  


 

5
0
1
−1
5
1

 

  


 

T  2  =  1  , T  −1  =  1  , T  −1  =  3 
2
−2
3
5
5
−6
Find the matrix of T . That is find A such that T x = Ax.
56. You have a linear transformation T and

  

  

 

1
1
−1
2
0
6

  

  

 

T  1  =  3  , T  0  =  4  , T  −1  =  1 
−8
1
6
1
3
−1
Find the matrix of T . That is find A such that T x = Ax.
57. You have a linear transformation T and

 


 


 

1
−3
−1
1
0
5

 


 


 

T  3  =  1  , T  −2  =  3  , T  −1  =  3 
−7
3
6
−3
2
−3
Find the matrix of T . That is find A such that T x = Ax.
58. You have a linear transformation T and

 


  

 

1
3
−1
1
0
1

 


  

 

T  1  =  3  , T  0  =  2  , T  −1  =  3 
−7
3
6
3
2
−1
Find the matrix of T . That is find A such that T x = Ax.
59. You have a linear transformation T and

  

  

 

1
5
−1
3
0
2

  

  

 

T  2  =  2  , T  −1  =  3  , T  −1  =  5 
−18
5
15
5
4
−2
Find the matrix of T . That is find A such that T x = Ax.
(
)−1
60. Suppose a1 · · · an
exists where each aj ∈ Rn and let vectors {b1 , · · · , bn } in Rm
be given. Show that there always exists a linear transformation T such that T ai = bi .
170
CHAPTER 9. LINEAR TRANSFORMATIONS
61. Let V ̸= {0} be a subspace of Fn . Show directly that there exists a subset of V {v1 , · · · , vr }
such that span (v1 , · · · , vr ) = V . Out of all such spanning sets, let the dimension of V be
the smallest number of vectors in any spanning set. The spanning set having this smallest
number of vectors will be called a minimal spanning set. Thus it is automatically the case
that any two of these minimal spanning sets contain the same number of vectors. Now show
that {v1 , · · · , vr } is a minimal spanning set if and only if {v1 , · · · , vr } is a basis. This gives
a little different way to define a basis which has the advantage of making it obvious that any
two bases have the same number of vectors. Hence the dimension is well defined.
62. In general, if V is a subspace of Fn and W is a subspace of Fm , a function T : V → W is called
a linear transformation if whenever x, y are vectors in V and a, b are scalars,
T (ax + by) = aT x + bT y
this is more general than having T be defined only on all of Fn . Why must V be a subspace
in order for this to make sense?
Chapter 10
A Few Factorizations
10.1
Definition Of An LU factorization
An LU factorization of a matrix involves writing the given matrix as the product of a lower triangular
matrix which has the main diagonal consisting entirely of ones L, and an upper triangular matrix U
in the indicated order. This is the version discussed here but it is sometimes the case that the L has
numbers other than 1 down the main diagonal. It is still a useful concept. The L goes with “lower”
and the U with “upper”. It turns out many matrices can be written in this way and when this is
possible, people get excited about slick ways of solving the system of equations, Ax = y. It is for
this reason that you want to study the LU factorization. It allows you to work only with triangular
matrices. It turns out that it takes about half as many operations to obtain an LU factorization as
it does to find the row reduced echelon form.
First it should be noted not all matrices have an LU factorization and so we will emphasize the
techniques for achieving it rather than formal proofs.
(
)
0 1
in the form LU as just described?
Example 10.1.1 Can you write
1 0
To do so you would need
(
)(
1 0
a
x 1
0
b
c
)
(
=
a
b
xa xb + c
)
(
=
0
1
1
0
)
.
Therefore, b = 1 and a = 0. Also, from the bottom rows, xa = 1 which can’t happen and have a = 0.
Therefore, you can’t write this matrix in the form LU. It has no LU factorization. This is what we
mean above by saying the method lacks generality.
10.2
Finding An LU Factorization By Inspection
Which matrices have an LU factorization? It turns out it is those whose row reduced echelon form
can be achieved without switching rows and which only involve row operations of type 3 in which
row j is replaced with a multiple of row i added to row j for i < j.


1 2 0 2


Example 10.2.1 Find an LU factorization of A =  1 3 2 1  .
2 3 4 0
One way to find the LU

1

 1
2
factorization is to simply look for it directly.
 

2 0 2
1 0 0
a d h j
 

=
3 2 1   x 1 0  0 b e i
3 4 0
y z 1
0 0 c f
171
You need


.
172
CHAPTER 10. A FEW FACTORIZATIONS
Then multiplying these you get

a
d
h

xh + e
 xa xd + b
ya yd + zb yh + ze + c

j

xj + i

yj + iz + f
and so you can now tell what the various quantities equal. From the first column, you need a = 1, x =
1, y = 2. Now go to the second column. You need d = 2, xd + b = 3 so b = 1, yd + zb = 3 so z = −1.
From the third column, h = 0, e = 2, c = 6. Now from the fourth column, j = 2, i = −1, f = −5.
Therefore, an LU factorization is



1 2 0 2
1 0 0



 1 1 0   0 1 2 −1  .
0 0 6 −5
2 −1 1
You can check whether you got it right by simply multiplying these two.
10.3
Using Multipliers To Find An LU Factorization
There is also a convenient procedure for finding an LU factorization. It turns out that it is only
necessary to keep track of the multipliers which are used to row reduce to upper triangular form.
This procedure is described in the following examples.


1 2 3


Example 10.3.1 Find an LU factorization for A =  2 1 −4 
1 5 2
Write the matrix next to the identity matrix as shown.



1 0 0
1 2 3



 0 1 0   2 1 −4  .
0 0 1
1 5 2
The process involves doing row operations to the matrix on the right while simultaneously updating
successive columns of the matrix on the left. First take −2 times the first row and add to the second
in the matrix on the right.



1 0 0
1 2
3



 2 1 0   0 −3 −10 
0 0 1
1 5
2
Note the way we updated the matrix on the left. We put a 2 in the second entry of the first column
because we used −2 times the first row added to the second row. Now replace the third row in the
matrix on the right by −1 times the first row added to the third. Notice that the product of the
two matrices is unchanged and equals the original matrix. This is because a row operation was done
on the original matrix to get the matrix on the right and then on the left, it was multiplied by an
elementary matrix which “undid” the row operation which was done.
The next step is



1 0 0
1 2
3



 2 1 0   0 −3 −10 
1 0 1
0 3
−1
10.4. SOLVING SYSTEMS USING AN LU FACTORIZATION
173
Again, the product is unchanged because we just did and then undid a row operation. Finally, we
will add the second row to the bottom row and make the following changes



1 0 0
1 2
3



 2 1 0   0 −3 −10  .
1 −1 1
0 0 −11
At this point, we stop because the matrix on the right is upper triangular. An LU factorization is
the above.
The justification for this gimmick will be given later in a more general context.



Example 10.3.2 Find an LU factorization for A = 

1
2
2
1
2
0
3
0
1
2
1
1
2
1
3
1
1
1
2
2



.

II
We will use the same procedure as above. However, this time we will do everything for one
column at a time. First multiply the first row by (−1) and then add to the last row. Next take (−2)
times the first and add to the second and then (−2) times the first and add to the third.





1
2
2
1
0
1
0
0
0
0
1
0
0
0
0
1





1 2
1
0 −4 0
0 −1 −1
0 −2 0
2
−3
−1
−1
1
−1
0
1



.

This finishes the first column of L and the first column of U. As in the above, what happened was
this. Lots of row operations were done and then these were undone by multiplying by the matrix
on the left. Thus the above product equals the original matrix. Now take − (1/4) times the second
row in the matrix on the right and add to the third followed by − (1/2) times the second added to
the last.



1 0
0 0
1 2
1
2
1
 2 1

0 0 
−3
−1 

  0 −4 0




 2 1/4 1 0   0 0 −1 −1/4 1/4 
1 1/2 0 1
0 0
0
1/2 3/2
This finishes the second column of L as well as the second column of U . Since the matrix on the
right is upper triangular, stop. The LU factorization has now been obtained. This technique is
called Dolittle’s method.
This process is entirely typical of the general case. The matrix U is just the first upper triangular
matrix you come to in your quest for the row reduced echelon form using only the row operation
which involves replacing a row by itself added to a multiple of another row. The matrix L is what
you get by updating the identity matrix as illustrated above.
You should note that for a square matrix, the number of row operations necessary to reduce to
LU form is about half the number needed to place the matrix in row reduced echelon form. This is
why an LU factorization is of interest in solving systems of equations.
10.4
Solving Systems Using An LU Factorization
One reason people care about the LU factorization is it allows the quick solution of systems of
equations. Here is an example.
174
CHAPTER 10. A FEW FACTORIZATIONS
Example 10.4.1 Suppose you want to find the solutions to






x
1 2 3 2
1
 y 


 

 =  2 .
 4 3 1 1 
 z 
1 2 3 0
3
w
Of course one way is to write the augmented matrix and grind away. However, this involves
more row operations than the computation of the LU factorization and it turns out that the LU
factorization can give the solution quickly. Here is how. The following is an LU factorization for
the matrix.



 
1 2
3
2
1 0 0
1 2 3 2


 

 4 3 1 1  =  4 1 0   0 −5 −11 −7  .
0 0
0
−2
1 0 1
1 2 3 0
T
Let U x = y and consider Ly = b where in this case, b = (1, 2, 3) . Thus


  
1 0 0
y1
1


  
 4 1 0   y2  =  2 
1 0 1
y3
3


1


which yields very quickly that y =  −2  . Now you can find x by solving U x = y. Thus in this
2
case,






x
1 2
3
2
1
 y 


 

 =  −2 
 0 −5 −11 −7  
 z 
0 0
0
−2
2
w
which yields




x =



10.5
− 35 + 57 t
9
5
−
11
5 t
t
−1




 , t ∈ R.



Justification For The Multiplier Method
Why does the multiplier method work for finding the LU factorization? Suppose A is a matrix
which has the property that the row reduced echelon form for A may be achieved using only the
row operations which involve replacing a row with itself added to a multiple of another row. It is
not ever necessary to switch rows. Thus every row which is replaced using this row operation in
obtaining the echelon form may be modified by using a row which is above it.
Lemma 10.5.1 Let L be a lower (upper) triangular matrix m × m which has ones down the main
diagonal. Then L−1 also is a lower (upper) triangular matrix which has ones down the main diagonal.
In the case that L is of the form


1


 a1 1



L= .
(10.1)
.

.
.
.
 .

an
1
10.5. JUSTIFICATION FOR THE MULTIPLIER METHOD
175
where all entries are zero except for the left column and main diagonal, it is also the case that L−1
is obtained from L by simply multiplying each entry below the main diagonal in L with −1. The
same is true if the single nonzero column is in another position.
(
)
Proof: Consider the usual setup for finding the inverse
L I . Then each row operation
done to L to reduce to row reduced echelon form results in changing only the entries in I below
the main diagonal. In the special case of L given in 10.1 or the single nonzero column is in another
position, multiplication by −1 as described in the lemma clearly results in L−1 . For a simple illustration of the last claim,




1 0 0 1 0 0
1 0 0 1 0 0




 0 1 0 0 1 0 → 0 1 0 0 1 0 
0 a 1 0 0 1
0 0 1 0 −a 1
Now let A be an m × n matrix, say



A=


a11
a21
..
.
am1
a12
a22
..
.
am2
···
···
···
a1n
a2n
..
.
amn






and assume A can be row reduced to an upper triangular form using only row operation 3. Thus,
in particular, a11 ̸= 0. Multiply on the left by E1 =


1
0 ··· 0


a
1 ··· 0 
 − a21
11

..
.. . .
.. 


.

.
.
. 
0 ··· 1
− aam1
11
This is the product of elementary matrices which make modifications in the first column only. It is
equivalent to taking −a21 /a11 times the first row and adding to the second. Then taking −a31 /a11
times the first row and adding to the third and so forth. The quotients in the first column of the
above matrix are the multipliers. Thus the result is of the form


a11 a12 · · · a′1n


a′22 · · · a′2n 
 0

E1 A =  .
..
.. 

 ..
.
. 
0 a′m2 · · · a′mn
below it in
By assumption, a′22 ̸= 0 and so it is possible to use this entry to zero out(all the entries
)
1 0
the matrix on the right by multiplication by a matrix of the form E2 =
where E is an
0 E
(m − 1) × (m − 1) matrix of the form


1
0 ··· 0
′


 − aa′32 1 · · · 0 


22
E=
..
.. . .
.. 

. . 
.
.


′
a
0 ··· 1
− am2
′
22
Again, the entries in the first column below the 1 are the multipliers. Continuing this way, zeroing
out the entries below the diagonal entries, finally leads to
Em−1 En−2 · · · E1 A = U
176
CHAPTER 10. A FEW FACTORIZATIONS
where U is upper triangular. Each Ej has all ones down the main diagonal and is lower triangular.
Now multiply both sides by the inverses of the Ej in the reverse order. This yields
−1
A = E1−1 E2−1 · · · Em−1
U
By Lemma 10.5.1, this implies that the product of those Ej−1 is a lower triangular matrix having all
ones down the main diagonal.
The above discussion and lemma gives the justification for the multiplier method. The expressions
−a21 /a11 , −a31 /a11 , · · · − am1 /a11
denoted respectively by M21 , · · · , Mm1 to save notation which were obtained in building E1 are the
multipliers. Then according to the lemma, to find E1−1 you simply write






1
−M21
..
.
−Mm1
0
1
..
.
0
···
···
..
.
···
0
0
..
.
1






Similar considerations apply to the other Ej−1 . Thus L is a product of the form






1
−M21
..
.
−Mm1
0 ···
1 ···
.. . .
.
.
0 ···
0
0
..
.
1


 
 
···
 
 
1 0
0 1
..
.
0
0 ···
···
···
..
.
−Mm(m−1)
0
0
..
.
1






each factor having at most one nonzero column, the position of which moves from left to right in
scanning the above product of matrices from left to right. It follows from Theorem 8.1.9 about the
effect of multiplying on the left by an elementary matrix that the above product is of the form

1
−M21
..
.







 −M(M −1)1
−MM 1
0
1
−M32
..
.
−MM 2
···
···
..
.
0
0
..
.
···
···
1
−MM M −1
0
0
..
.








0 
1
In words, beginning at the left column and moving toward the right, you simply insert, into the
corresponding position in the identity matrix, −1 times the multiplier which was used to zero out
an entry in that position below the main diagonal in A, while retaining the main diagonal which
consists entirely of ones. This is L.
10.6
The P LU Factorization
As indicated above, some matrices don’t have an

1

M = 1
4
LU factorization. Here is an example.

2 3 2

2 3 0 
3 1 1
(10.2)
In this case, there is another factorization which is useful called a P LU factorization. Here P is a
permutation matrix.
10.6. THE P LU FACTORIZATION
177
Example 10.6.1 Find a P LU factorization for the above matrix in 10.2.
Proceed as before trying to find the row echelon form of the matrix. First add −1 times the first
row to the second row and then add −4 times the first to the third. This yields



1 0 0
1 2
3
2



0
−2 
 1 1 0  0 0
4 0 1
0 −5 −11 −7
There is no way to do only row operations involving replacing a row with itself added to a multiple
of another row to the matrix on the right in such a way as to obtain an upper triangular matrix.
Therefore, consider the original matrix with the bottom two rows switched.


 

1 2 3 2
1 0 0
1 2 3 2


 

M′ =  4 3 1 1  =  0 0 1   1 2 3 0 
1 2 3 0
0 1 0
4 3 1 1
=
PM
Now try again with this matrix. First take −1 times the first row and add to the bottom row and
then take −4 times the first row and add to the second row. This yields



1 0 0
1 2
3
2



 4 1 0   0 −5 −11 −7 
1 0 1
0 0
0
−2
The matrix on the right is upper triangular and so the LU factorization of the matrix M ′ has been
obtained above.
Thus M ′ = P M = LU where L and U are given above. Notice that P 2 = I and therefore,
M = P 2 M = P LU and so

 



1 2 3 2
1 0 0
1 0 0
1 2
3
2

 



 1 2 3 0  =  0 0 1   4 1 0   0 −5 −11 −7 
4 3 1 1
0 1 0
1 0 1
0 0
0
−2
This process can always be followed and so there always exists a
matrix even though there isn’t always an LU factorization.

1 2 3 2

Example 10.6.2 Use the P LU factorization of M ≡  1 2 3 0
4 3 1 1
P LU factorization of a given


 to solve the system M x = b
T
where b = (1, 2, 3) .
Let U x = y and consider P Ly = b. In other


1 0 0
1 0


 0 0 1  4 1
0 1 0
1 0
Multiplying both sides by P gives

1 0

 4 1
1 0
words, solve,

  
0
y1
1

  
0   y2  =  2  .
1
y3
3

  
0
y1
1

  
=
0   y2   3 
1
y3
2
178
CHAPTER 10. A FEW FACTORIZATIONS

 

y1
1

 

y =  y2  =  −1  .
y3
1
and so
Now U x = y and so it only remains to solve

1

 0
0
2
3
−5 −11
0
0
which yields




x1
2
 x
 2
−7  
 x3
−2
x4

1
5
x1

 x  
 2  

=
 x3  


x4
10.7
9
10


1

 

 =  −1 

1

+ 75 t
−

11
5 t
t
− 12



 : t ∈ R.



The QR Factorization
As pointed out above, the LU factorization is not a mathematically respectable thing because it
does not always exist. There is another factorization which does always exist. Much more can be
said about it than I will say here. At this time, I will only deal with real matrices and so the inner
product will be the usual real dot product. Letting A be an m × n real matrix and letting (·, ·)
denote the usual real inner product,
∑
∑∑
∑∑( )
(Ax, y) =
(Ax)i yi =
Aij xj yi =
AT ji yi xj
i
=
∑(
T
A y
i
)
j
(
j
T
xj = x,A y
)
j
i
j
Thus, when you take the matrix across the comma, you replace with a transpose.
Definition 10.7.1 An n × n real matrix Q is called an orthogonal matrix if
QQT = QT Q = I.
Thus an orthogonal matrix is one whose inverse is equal to its transpose.
From the above observation,
(
)
2
2
|Qx| = (Qx, Qx) = x,QT Qx = (x,Ix) = (x, x) = |x|
This shows that orthogonal transformations preserve distances. Conversely you can also show that
if you have a matrix which does preserve distances, then it must be orthogonal.
Example 10.7.2 One of the most important examples of an orthogonal matrix is the so called
Householder matrix. You have v a unit vector and you form the matrix
I − 2vvT
This is an orthogonal matrix which is also symmetric. To see this, you use the rules of matrix
operations.
(
)T
(
)T
= I T − 2vvT
I − 2vvT
= I − 2vvT
10.7. THE QR FACTORIZATION
179
so it is symmetric. Now to show it is orthogonal,
(
)(
)
I − 2vvT I − 2vvT
= I − 2vvT − 2vvT + 4vvT vvT
= I − 4vvT + 4vvT = I
2
because vT v = v · v = |v| = 1. Therefore, this is an example of an orthogonal matrix.
Next consider the problem illustrated in the following picture.
x
y
Find an orthogonal matrix Q which switches the two vectors taking x to y and y to x.
Procedure 10.7.3 Given two vectors x, y such that |x| = |y| ̸= 0 but x ̸= y and you want an
orthogonal matrix Q such that Qx = y and Qy = x. The thing which works is the Householder
matrix
x−y
T
Q≡I −2
2 (x − y)
|x − y|
Here is why this works.
Q (x − y)
=
(x − y) − 2
= (x − y) − 2
Q (x + y)
=
(x + y) − 2
x−y
T
2
(x − y) (x − y)
2
|x − y| = y − x
|x − y|
x−y
|x − y|
x−y
2
T
2
|x − y|
x−y
(x − y) (x + y)
2 ((x − y) · (x + y))
|x − y|
)
x−y ( 2
2
= (x + y) − 2
|x|
−
|y|
=x+y
2
|x − y|
= (x + y) − 2
Hence
Qx + Qy
= x+y
Qx − Qy
= y−x
Adding these equations, 2Qx = 2y and subtracting them yields 2Qy = 2x.
Definition 10.7.4 Let A be an m×n matrix. Then a QR factorization of A consists of two matrices,
Q orthogonal and R upper triangular or in other words equal to zero below the main diagonal such
that A = QR.
With the solution to this simple problem, here is how to obtain a QR factorization for any matrix
A. Let
A = (a1 , a2 , · · · , an )
180
CHAPTER 10. A FEW FACTORIZATIONS
where the ai are the columns. If a1 = 0, let Q1 = I. If a1 ̸= 0, let


|a1 |


 0 

b ≡ . 

 .. 
0
and form the Householder matrix
(a1 − b)
Q1 ≡ I − 2
|a1 − b|
As in the above problem Q1 a1 = b and so
(
Q1 A =
2
T
(a1 − b)
|a1 | ∗
0 A2
)
where A2 is a m − 1 × n − 1 matrix. Now find in the same way as was just done a n − 1 × n − 1
b 2 such that
matrix Q
(
)
∗
∗
b 2 A2 =
Q
0 A3
(
Let
Q2 ≡
(
Then
Q2 Q1 A =


=

1 0
b2
0 Q
1 0
b2
0 Q
)(
)
.
|a1 | ∗
0 A2

)
|a1 | ∗ ∗

..

.
∗ ∗ 
0
0 A3
Continuing this way until the result is upper triangular, you get a sequence of orthogonal matrices
Qp Qp−1 · · · Q1 such that
Qp Qp−1 · · · Q1 A = R
(10.3)
where R is upper triangular.
Now if Q1 and Q2 are orthogonal, then from properties of matrix multiplication,
T
Q1 Q2 (Q1 Q2 ) = Q1 Q2 QT2 QT1 = Q1 IQT1 = I
and similarly
T
(Q1 Q2 ) Q1 Q2 = I.
Thus the product of orthogonal matrices is orthogonal. Also the transpose of an orthogonal matrix
is orthogonal directly from the definition. Therefore, from 10.3
T
A = (Qp Qp−1 · · · Q1 ) R ≡ QR,
where Q is orthogonal. This suggests the proof of the following theorem.
Theorem 10.7.5 Let A be any real m × n matrix. Then there exists an orthogonal matrix Q and
an upper triangular matrix R having nonnegative entries down the main diagonal such that
A = QR
and this factorization can be accomplished in a systematic manner.
10.8. EXERCISES
181
Proof: The theorem is clearly true if A is a 1 × m matrix. Suppose it is true for A an n × m
matrix. Thus, if A is any n × m matrix, there exists an orthogonal matrix Q such that
QA = R
where R is upper triangular. Suppose A is an (n + 1) × m matrix. Then, as indicated above, there
exists an orthogonal (n + 1) × (n + 1) matrix Q1 such that
(
)
a b
Q1 A =
0 A1
where A1 is n × m − 1 or else, in case m = 1, the right side is of the form
(
)
a
0
and in this case, the conclusion of the theorem follows. If m − 1 ≥ 1, then by induction, there exists
Q2 an orthogonal n × n matrix such that Q2 A1 = R1 . Then
(
)
(
)(
) (
)
1 0
1 0
a b
a b
Q1 A =
=
≡R
0 Q2
0 Q2
0 A1
0 R1
(
1 0
Since
0 Q2
follows. II
10.8
)
Q1 is orthogonal, being the product of two orthogonal matrices, the conclusion
Exercises

1 2

1. Find an LU factorization of  2 1
1 2

1 2

2. Find an LU factorization of  1 3
5 0

0

3 .
3

3 2

2 1 .
1 3

1 −2

3. Find an LU factorization of the matrix  −2 5
3 −6

1 −1

4. Find an LU factorization of the matrix  −1 2
2 −3

1 −3

5. Find an LU factorization of the matrix  −3 10
1 −6

−5 0

11 3  .
−15 1

−3 −1

4
3 .
−7 −3

−4 −3

10 10  .
2 −5
182
CHAPTER 10. A FEW FACTORIZATIONS

1

6. Find an LU factorization of the matrix  3
2



7. Find an LU factorization of the matrix 




8. Find an LU factorization of the matrix 




9. Find an LU factorization of the matrix 


3
1 −1

10 8 −1  .
5 −3 −3

3 −2
9 −8
−6 2
3
2
1
6
2
−7
−3 −1
9
9
3
19
12 40
3
−12
−16
−26
−1 −3
1
3
3
9
4
12
−1
0
0
16


.




.




.

10. Find the LU factorization of the coefficient matrix using Dolittle’s method and use it to solve
the system of equations.
x + 2y = 5
2x + 3y = 6
11. Find the LU factorization of the coefficient matrix using Dolittle’s method and use it to solve
the system of equations.
x + 2y + z = 1
y + 3z = 2
2x + 3y = 6
12. Find the LU factorization of the coefficient matrix using Dolittle’s method and use it to solve
the system of equations.
x + 2y + 3z = 5
2x + 3y + z = 6
x−y+z =2
13. Find the LU factorization of the coefficient matrix using Dolittle’s method and use it to solve
the system of equations.
x + 2y + 3z = 5
2x + 3y + z = 6
3x + 5y + 4z = 11
14. Is there only one LU factorization for a given matrix? Hint: Consider the equation
(
) (
)(
)
0 1
1 0
0 1
=
.
0 1
1 1
0 0
Look for all possible LU factorizations.


1 2 1


15. Find a P LU factorization of  1 2 2  .
2 1 1
10.8. EXERCISES
183

1

16. Find a P LU factorization of  2
1



17. Find a P LU factorization of 

1
1
2
3



18. Find a P LU factorization of 

1
2
1
2



1
x


 2 


  
a. 
 y  =  
 1 


z
1

0

19. Find a P LU factorization of  2
2





x
0 2 1 2
 y 


 
a.  2 1 −2 0  
=
 z 
2 3 −1 2
w

1
2
1
2
2
4
0
2
1
1
2
1


2
4
2
1
2
1
2
2
4
2
1
2
1
1
2
4
3

1

1 .
2



.


2 1
4 1 

 and use it to solve the systems
0 2 
2 1






a
1 2 1
x


 2 4 1 


  b 
b.

 y  = 
 c 
 1 0 2 
z
d
2 2 1

2 1 2

1 −2 0  and use it to solve the systems
3 −1 2


 

 
x
1
0 2 1 2
2
 y 
 

  
1  b. 2 1 −2 0  
= 1 
 z 
2
2 3 −1 2
3
w
20. Find a QR factorization for the matrix


1 2 1


 3 −2 1 
1 0 2
21. Find a QR factorization for the matrix

1 2

 3 0
1 0

1 0

1 1 
2 1
22. If you had a QR factorization, A = QR, describe how you could use it to solve the equation
Ax = b. This is not usually the way people solve this equation. However, the QR factorization is of great importance in certain other problems, especially in finding eigenvalues and
eigenvectors.
23. In this problem, is another explanation of the

1

A= 2
1
LU factorization. Let

2 3

0 −2 
3 1
184
CHAPTER 10. A FEW FACTORIZATIONS
Review how to take the inverse

1

A =  2
0

1

=  2
0

Next
1

A= 2
0
Next

1

A= 2
1
0
1
0
0
1
0
of an elementary matrix. Then


0 0
1 0 0
1


1 0   −2 1 0   2
0 1
0 0 1
1


0 0
1 2
3


1 0   0 −4 −8 
0 1
1 3
1

1 0 0
0

0  0 1 0
1 0 1
1

1 0 0

= 2 1 0
1 0 1

we can do the following.

2 3

0 −2 
3 1

1
1 0 0


 0 1 0  0
1
−1 0 1


1 2
3


  0 −4 −8 
0 1 −2


1
1
0
0
0


1
0  0
0  0
0
0 −1/4 1
1


1 0 0
1


=  2 1 0  0
1 − 14 1
0

2
3

−4 −8 
3
1


1 2
3
0
0


1
0   0 −4 −8 
0 1 −2
1/4 1

2
3

−4 −8 
0 −4
Using this example, describe why, when a matrix can be reduced to echelon form using only
row operation 3, then it has an LU factorization.
Chapter 11
Linear Programming
11.1
Simple Geometric Considerations
One of the most important uses of row operations is in solving linear program problems which involve
maximizing a linear function subject to inequality constraints determined from linear equations.
Here is an example. A certain hamburger store has 9000 hamburger patties to use in one week
and a limitless supply of special sauce, lettuce, tomatoes, onions, and buns. They sell two types of
hamburgers, the big stack and the basic burger. It has also been determined that the employees
cannot prepare more than 9000 of either type in one week. The big stack, popular with the teenagers
from the local high school, involves two patties, lots of delicious sauce, condiments galore, and a
divider between the two patties. The basic burger, very popular with children, involves only one
patty and some pickles and ketchup. Demand for the basic burger is twice what it is for the big
stack. What is the maximum number of hamburgers which could be sold in one week given the
above limitations?
Let x be the number of basic burgers and y the number of big stacks which could be sold
in a week. Thus it is desired to maximize z = x + y subject to the above constraints. The
total number of patties is 9000 and so the number of patty used is x + 2y. This number must
satisfy x + 2y ≤ 9000 because there are only 9000 patty available. Because of the limitation on
the number the employees can prepare and the demand, it follows 2x + y ≤ 9000. You never
sell a negative number of hamburgers and so x, y ≥ 0. In simpler terms the problem reduces to
maximizing z = x + y subject to the two constraints, x + 2y ≤ 9000 and 2x + y ≤ 9000. This
problem is pretty easy to solve geometrically. Consider the following picture in which R labels the
region described by the above inequalities and the line z = x + y is shown for a particular value of z.
As you make z larger this line moves away from the origin,
always having the same slope and the desired solution
would consist of a point in the region, R which makes z
x+y =z
2x + y = 4
as large as possible or equivalently one for which the line
is as far as possible from the origin. Clearly this point
is the point of intersection of the two lines, (3000, 3000)
and so the maximum value of the given function is 6000.
R
x + 2y = 4
Of course this type of procedure is fine for a situation
in which there are only two variables but what about a
similar problem in which there are very many variables.
In reality, this hamburger store makes many more types
of burgers than those two and there are many considerations other than demand and available patty.
Each will likely give you a constraint which must be considered in order to solve a more realistic
problem and the end result will likely be a problem in many dimensions, probably many more than
three so your ability to draw a picture will get you nowhere for such a problem. Another method is
needed. This method is the topic of this section. I will illustrate with this particular problem. Let
185
186
CHAPTER 11. LINEAR PROGRAMMING
x1 = x and y = x2 . Also let x3 and x4 be nonnegative variables such that
x1 + 2x2 + x3 = 9000, 2x1 + x2 + x4 = 9000.
To say that x3 and x4 are nonnegative is the same as saying x1 + 2x2 ≤ 9000 and 2x1 + x2 ≤ 9000
and these variables are called slack variables at this point. They are called this because they “take
up the slack”. I will discuss these more later. First a general situation is considered.
11.2
The Simplex Tableau
Here is some notation.
Definition 11.2.1 Let x, y be vectors in Rq . Then x ≤ y means for each i, xi ≤ yi .
The problem is as follows:
Let A be an m × (m + n) real matrix of rank m. It is desired to find x ∈ Rn+m such that x
satisfies the constraints,
x ≥ 0, Ax = b
(11.1)
and out of all such x,
z≡
m+n
∑
c i xi
i=1
is as large (or small) as possible. This is usually referred to as maximizing or minimizing z subject
to the above (constraints. First I)will consider the constraints.
Let A = a1 · · · an+m . First you find a vector x0 ≥ 0, Ax0 = b such that n of the components of this vector equal 0. Letting i1 , · · · , in be the positions of x0 for which x0i = 0, suppose
also that {aj1 , · · · , ajm } is linearly independent for ji the other positions of x0 . Geometrically, this
means that x0 is a corner of the feasible region, those x which satisfy the constraints. This is called
a basic feasible solution. Also define
cB
xB
and
≡
≡
(cj1 . · · · , cjm ) , cF ≡ (ci1 , · · · , cin )
(xj1 , · · · , xjm ) , xF ≡ (xi1 , · · · , xin ) .
(
z ≡z x
0
0
)
(
=
)
cB
cF
(
x0B
x0F
)
= cB x0B
since x0F = 0. The variables which are the components of the vector xB are called the basic
variables and the variables which are the entries of xF are called the free variables. You set
(
)T
xF = 0. Now x0 , z 0 is a solution to
(
)(
) (
)
A 0
x
b
=
−c 1
z
0
along with the constraints x ≥ 0. Writing the above in augmented matrix form yields
(
)
A 0 b
−c 1 0
Permute the columns and variables on the left if necessary to write the above in the form


(
)
(
)
xB
B
F
0 
b

 xF  =
−cB −cF 1
0
z
(11.2)
(11.3)
11.2. THE SIMPLEX TABLEAU
187
or equivalently in the augmented matrix form

B

−c

B
xB
keeping track of the variables on the bottom as

F
0 b

(11.4)
−cF 1 0  .
xF
0 0
Here B pertains to the variables xi1 , · · · , xjm and is an m × m matrix with linearly independent
columns, {aj1 , · · · , ajm } , and F is an m × n matrix. Now it is assumed that
(
)
)
(
(
) x0
(
) x0
B
B
= Bx0B = b
= B F
B F
0
x0F
and since B is assumed to have rank m, it follows
x0B = B −1 b ≥ 0.
This is very important to observe. B −1 b ≥ 0! This is by the assumption that x0 ≥ 0.
Do row operations on the top part of the matrix
(
)
B
F
0 b
−cB −cF 1 0
(11.5)
(11.6)
and obtain its row reduced echelon form. Then after these row operations the above becomes
(
)
I
B −1 F 0 B −1 b
.
(11.7)
−cB −cF
1
0
where B −1 b ≥ 0. Next do another row
(
I
0
(
I
=
0
(
I
=
0
operation in order to get a 0 where you see a −cB . Thus
)
B −1 F
0
B −1 b
(11.8)
cB B −1 F ′ − cF 1 cB B −1 b
)
B −1 F
0 B −1 b
cB B −1 F ′ − cF 1 cB x0B
)
B −1 F
0 B −1 b
(11.9)
cB B −1 F − cF 1
z0
(
)T
The reason there is a z 0 on the bottom right corner is that xF = 0 and x0B , x0F , z 0 is a solution of
the system of equations represented by the above augmented matrix because it is a solution to the
system of equations corresponding to the system of equations represented by 11.6 and row operations
leave solution sets unchanged. Note how attractive this is. The z0 is the value of z at the point x0 .
The augmented matrix of 11.9 is called the simplex tableau and it is the beginning point for the
simplex algorithm to be described a little later. It is very convenient to express the
tableau
( simplex
)
I
in the above form in which the variables are possibly permuted in order to have
on the left
0
side. However, as far as the simplex algorithm is concerned it is not necessary to be permuting the
variables in this manner. Starting with 11.9 you could permute the variables and columns to obtain
an augmented matrix in which the variables are in their original order. What is really required for
the simplex tableau?
It is an augmented m + 1 × m + n + 2 matrix which represents a system of equations which has
T
the same set of solutions, (x,z) as the system whose augmented matrix is
(
)
A 0 b
−c 1 0
188
CHAPTER 11. LINEAR PROGRAMMING
(Possibly the variables for x are taken in another order.) There are m linearly independent columns
in the first m + n columns for which there is only one nonzero entry, a 1 in one of the first m
rows, the “simple columns”, the other first m + n columns being the “nonsimple columns”. As in
the above, the variables corresponding to the simple columns are xB , the basic variables and those
corresponding to the nonsimple columns are xF , the free variables. Also, the top m entries of the
last column on the right are nonnegative. This is the description of a simplex tableau.
In a simplex tableau it is easy to spot a basic feasible solution. You can see one quickly by
setting the variables, xF corresponding to the nonsimple columns equal to zero. Then the other
variables, corresponding to the simple columns are each equal to a nonnegative entry in the far right
column. Lets call this an “obvious basic feasible solution”. If a solution is obtained by setting
the variables corresponding to the nonsimple columns equal to zero and the variables corresponding
to the simple columns equal to zero this will be referred to as an “obvious” solution. Lets also call
the first m + n entries in the bottom row the “bottom left row”. In a simplex tableau, the entry
in the bottom right corner gives the value of the variable being maximized or minimized when the
obvious basic feasible solution is chosen.
The following is a special case of the general theory presented above and shows how such a
special case can be fit into the above framework. The following example is rather typical of the sorts
of problems considered. It involves inequality constraints instead of Ax = b. This is handled by
adding in “slack variables” as explained below.
The idea is to obtain an augmented matrix for the constraints such that obvious solutions are
also feasible. Then there is an algorithm, to be presented later, which takes you from one obvious
feasible solution to another until you obtain the maximum.
Example 11.2.2 Consider z = x1 − x2 subject to the constraints, x1 + 2x2 ≤ 10, x1 + 2x2 ≥ 2,
and 2x1 + x2 ≤ 6, xi ≥ 0. Find a simplex tableau for a problem of the form x ≥ 0,Ax = b which is
equivalent to the above problem.
You add in slack variables. These are positive variables, one for each of the first three constraints,
which change the first three inequalities into equations. Thus the first three inequalities become
x1 + 2x2 + x3 = 10, x1 + 2x2 − x4 = 2, and 2x1 + x2 + x5 = 6, x1 , x2 , x3 , x4 , x5 ≥ 0. Now it
is necessary to find a basic feasible solution. You mainly need to find a positive solution to the
equations,
x1 + 2x2 + x3 = 10
x1 + 2x2 − x4 = 2 .
2x1 + x2 + x5 = 6
the solution set for the above system is given by
x2 =
2
2 1
1
10 2
x 4 − + x 5 , x1 = − x 4 +
− x5 , x3 = −x4 + 8.
3
3 3
3
3
3
An easy way to get a basic feasible solution is to let x4 = 8 and x5 = 1. Then a feasible solution is
(x1 , x2 , x3 , x4 , x5 ) = (0, 5, 0, 8, 1) .
(
)
A 0 b
0
It follows z = −5 and the matrix 11.2,
with the variables kept track of on the
−c 1 0
bottom is


1
2
1
0
0 0 10


 1
2
0 −1 0 0 2 


 2
1
0
0
1 0 6 




0
0
0 1 0 
 −1 1
x1 x2 x3 x4 x5 0 0
11.2. THE SIMPLEX TABLEAU
189
and the first thing to do is to permute the columns so that the list of variables on the bottom will
have x1 and x3 at the end.


2
0
0
1
1 0 10


 2 −1 0
1
0 0 2 


 1
0
1
2
0 0 6 




0
0 −1 0 1 0 
 1
x2 x4 x5 x1 x3 0 0
Next, as described above, take the row reduced echelon form of the top three lines of the above
matrix. This yields


1
0 5
1 0 0 12
2


1 0 8 .
 0 1 0 0
0 0 1 23 − 12 0 1
Now do row operations to





to finally obtain





1
0
0
1
1
0
0
0
0
1
0
0
0
1
0
0
0
0
1
0
0
0
1
0
1
2
1
2
0
1
1
3
−
2
2
−1 0
1
2
1
2
0
1
− 12
− 12
3
2
− 32
0
0
0
1
0
0
0
1
5
8
1
0
5
8
1
−5










and this is a simplex tableau. The variables are x2 , x4 , x5 , x1 , x3 , z.
It isn’t as hard as it may appear from the above. Lets not permute the variables and simply find
an acceptable simplex tableau as described above.
Example 11.2.3 Consider z = x1 − x2 subject to the constraints, x1 + 2x2 ≤ 10, x1 + 2x2 ≥ 2, and
2x1 + x2 ≤ 6, xi ≥ 0. Find a simplex tableau.
Adding in slack variables, an augmented

1 2

 1 2
2 1
matrix which is descriptive of the constraints is

1 0 0 10

0 −1 0 6 
0 0 1 6
The obvious solution is not feasible because of that -1 in the fourth column. When you let x1 , x2 = 0,
you end up having x4 = −6 which is negative. Consider the second column and select the 2 as a
pivot to zero out that which is above and below the 2.


0 0 1 1 0 4
 1

 2 1 0 − 12 0 3 
3
1
0 0
1 3
2
2
This one is good. When you let x1 = x4 = 0, you find that x2 = 3, x3 = 4, x5 = 3. The obvious
solution is now feasible. You can now assemble the simplex tableau. The first step is to include a
column and row for z. This yields


0 0 1 1 0 0 4
 1 1 0 −1 0 0 3 
 2

2
 3

1
 2 0 0
1 0 3 
2
−1 0 1 0 0 1 0
190
CHAPTER 11. LINEAR PROGRAMMING
Now you need to get zeros in the right places so the simple columns will be preserved as simple
columns in this larger matrix. This means you need to zero out the 1 in the third column on the
bottom. A simplex tableau is now


0 0 1 1 0 0 4
 1 1 0 −1 0 0 3 
 2

2
 3
.
1
 2 0 0

1
0
3
2
−1 0 0 −1 0 1 −4
Note it is not the same one obtained earlier. There is no reason a simplex tableau should be unique.
In fact, it follows from the above general description that you have one for each basic feasible point
of the region determined by the constraints.
11.3
The Simplex Algorithm
11.3.1
Maximums
The simplex algorithm takes you from one basic feasible solution to another while maximizing or
minimizing the function you are trying to maximize or minimize. Algebraically, it takes you from one
simplex tableau to another in which the lower right corner either increases in the case of maximization
or decreases in the case of minimization.
I will continue writing the simplex tableau in such a way that the simple columns having only
one entry nonzero are on the left. As explained above, this amounts to permuting the variables. I
will do this because it is possible to describe what is going on without onerous notation. However,
in the examples, I won’t worry so much about it. Thus, from a basic feasible solution, a simplex
tableau of the following form has been obtained in which the columns for the basic variables, xB are
listed first and b ≥ 0.
(
)
I F 0 b
(11.10)
0 c 1 z0
(
)
Let x0i = bi for i = 1, · · · , m( and x)0i = 0 for i > m. Then x0 , z 0 is a solution to the above system
and since b ≥ 0, it follows x0 , z 0 is a basic feasible solution.
(
)
F
If ci < 0 for some i, and if Fji ≤ 0 so that a whole column of
is ≤ 0 with the bottom
c
entry < 0, then letting xi be the variable corresponding to that column, you could leave all the other
entries of xF equal to zero but change xi to be positive. Let the new vector be denoted by x′F and
letting x′B = b − F x′F it follows
∑
(x′B )k = bk −
Fkj (xF )j
j
= bk − Fki xi ≥ 0
Now this shows (x′B , x′F ) is feasible whenever xi > 0 and so you could let xi become arbitrarily large
and positive and conclude there is no maximum for z because
z = (−ci ) xi + z 0
(11.11)
If this happens in a simplex tableau, you can say there is no maximum and stop.
What if c ≥ 0? Then z = z 0 − cxF and to satisfy the constraints, you need xF ≥ 0. Therefore, in
this case, z 0 is the largest possible value of z and so the maximum has been found. You stop when
this occurs. Next I explain what to do if neither of the above stopping conditions hold.
(
)
F
The only case which remains is that some ci < 0 and some Fji > 0. You pick a column in
c
in which ci < 0, usually the one for which ci is the largest in absolute value. You pick Fji > 0 as
11.3. THE SIMPLEX ALGORITHM
191
a pivot element, divide the j th row by Fji and then use to obtain zeros above Fji and below Fji ,
thus obtaining a new simple column. This row operation also makes exactly one of the other simple
columns into a nonsimple column. (In terms of variables, it is said that a free variable becomes
a basic variable and a basic variable becomes a free variable.) Now permuting the columns and
variables, yields
(
)
I F ′ 0 b′
0 c′ 1 z 0′
( )
bj
where z 0′ ≥ z 0 because z 0′ = z 0 − ci Fji
and ci < 0. If b′ ≥ 0, you are in the same position
you were at the beginning but now z 0 is larger. Now here is the important thing. You don’t pick
just any Fji when you do these row operations. You pick the positive one for which the row
operation results in b′ ≥ 0. Otherwise the obvious basic feasible solution obtained by letting
x′F = 0 will fail to satisfy the constraint that x ≥ 0.
How is this done? You need
Fki bj
b′k ≡ bk −
≥0
(11.12)
Fji
for each k = 1, · · · , m or equivalently,
Fki bj
.
(11.13)
bk ≥
Fji
Now if Fki ≤ 0 the above holds. Therefore, you only need to check Fpi for Fpi > 0. The pivot,
Fji is the one which makes the quotients of the form
bp
Fpi
for all positive Fpi the smallest. This will work because for Fki > 0,
bp
bk
Fki bp
≤
⇒ bk ≥
Fpi
Fki
Fpi
Having gotten a new simplex tableau, you do the same thing to it which was just done and continue.
As long as b > 0, so you don’t encounter the degenerate case, the values for z associated with setting
xF = 0 keep getting strictly larger every time the process is repeated. You keep going until you find
c ≥ 0. Then you stop. You are at a maximum. Problems can occur in the process in the so called
degenerate case when at some stage of the process some bj = 0. In this case you can cycle through
different values for x with no improvement in z. This case will not be discussed here.
Example 11.3.1 Maximize 2x1 +3x2 subject to the constraints x1 +x2 ≥ 1, 2x1 +x2 ≤ 6, x1 +2x2 ≤
6, x1 , x2 ≥ 0.
The constraints are of the form
x1 + x2 − x3
= 1
2x1 + x2 + x4
x1 + 2x2 + x5
= 6
= 6
where the x3 , x4 , x5 are the slack variables. An augmented matrix for these equations is of the form


1 1 −1 0 0 1


 2 1 0 1 0 6 
1 2 0 0 1 6
Obviously the obvious solution is not feasible. It
variables. Lets just try something.

1 1 −1

 0 −1 2
0 1
1
results in x3 < 0. We need to exchange basic

0 0 1

1 0 4 
0 1 5
192
CHAPTER 11. LINEAR PROGRAMMING
Now this one is all right because the obvious solution is feasible. Letting x2 = x3 = 0, it follows
that the obvious solution is feasible. Now we add in the objective function as described above.


1
1 −1 0 0 0 1
 0 −1 2 1 0 0 4 




 0
1
1 0 1 0 5 
−2 −3 0 0 0 1 0
Then do row operations to leave the simple columns the

1 1 −1 0 0
 0 −1 2 1 0


 0 1
1 0 1
0 −1 −2 0 0
same. Then

0 1
0 4 


0 5 
1
2
Now there are negative numbers on the bottom row to the left of the 1. Lets pick the first. (It would
be more sensible to pick the second.) The ratios to look at are 5/1, 1/1 so pick for the pivot the 1
in the second column and first row. This will leave the right column above the lower right corner
nonnegative. Thus the next tableau is


1 1 −1 0 0 0 1
 1 0 1 1 0 0 5 




 −1 0 2 0 1 0 4 
1 0 −3 0 0 1 3
There is still a negative number there to the left of the 1 in the bottom row. The new ratios are
4/2, 5/1 so the new pivot is the 2 in the third column. Thus the next tableau is
 1

1
1 0 0
0 3
2
2
 3
0 0 1 − 12 0 3 
 2



 −1 0 2 0 1 0 4 
3
− 12 0 0 0
1 9
2
Still, there is a negative number in the bottom row to the left of the 1 so the process does not stop
yet. The ratios are 3/ (3/2) and 3/ (1/2) and so the new pivot is that 3/2 in the first column. Thus
the new tableau is


2
0 2
0 1 0 − 13
3
 3 0 0 1 −1 0 3 
 2

2


2
2
 0 0 2
0 6 
3
3
4
1
1 10
0 0 0
3
3
Now stop. The maximum value is 10. This is an easy enough problem to do geometrically and so you
can easily verify that this is the right answer. It occurs when x4 = x5 = 0, x1 = 2, x2 = 2, x3 = 3.
11.3.2
Minimums
How does it differ if you are finding a minimum? From a basic feasible solution, a simplex tableau
of the following form has been obtained in which the simple columns for the basic variables, xB are
listed first and b ≥ 0.
(
)
I F 0 b
(11.14)
0 c 1 z0
(
)
Let x0i = bi for i = 1, · · · , m( and x)0i = 0 for i > m. Then x0 , z 0 is a solution to the above system
and since b ≥ 0, it follows x0 , z 0 is a basic feasible solution. So far, there is no change.
11.3. THE SIMPLEX ALGORITHM
193
Suppose first that some ci > 0 and Fji ≤ 0 for each j. Then let x′F consist of changing xi by
making it positive but leaving the other entries of xF equal to 0. Then from the bottom row,
z = −ci xi + z 0
and you let x′B = b − F x′F ≥ 0. Thus the constraints continue to hold when xi is made increasingly
positive and it follows from the above equation that there is no minimum for z. You stop when this
happens.
Next suppose c ≤ 0. Then in this case, z = z 0 − cxF and from the constraints, xF ≥ 0 and so
−cxF ≥ 0 and so z 0 is the minimum value and you stop since this is what you are looking for.
What do you do in the case where some ci > 0 and some Fji > 0? In this case, you use the simplex
algorithm as in the case of maximums to obtain a new simplex tableau in which z 0′ is smaller. You
choose Fji the same way to be the positive entry of the ith column such that bp /Fpi ≥ bj /Fji for all
positive entries, Fpi and do the same row operations. Now this time,
(
)
bj
z 0′ = z 0 − ci
< z0
Fji
As in the case of maximums no problem can occur and the process will converge unless you have the
degenerate case in which some bj = 0. As in the earlier case, this is most unfortunate when it occurs.
You see what happens of course. z 0 does not change and the algorithm just delivers different values
of the variables forever with no improvement.
To summarize the geometrical significance of the simplex algorithm, it takes you from one corner
of the feasible region to another. You go in one direction to find the maximum and in another to
find the minimum. For the maximum you try to get rid of negative entries of c and for minimums
you try to eliminate positive entries of c, where the method of elimination involves the auspicious
use of an appropriate pivot element and row operations.
Now return to Example 11.2.2. It will be modified to be a maximization problem.
Example 11.3.2 Maximize z = x1 − x2 subject to the constraints,
x1 + 2x2 ≤ 10, x1 + 2x2 ≥ 2,
and 2x1 + x2 ≤ 6, xi ≥ 0.
Recall this is the same as maximizing z = x1 − x2 subject to


x1



 
 x2 
1 2 1 0 0
10
 




 1 2 0 −1 0  
 x3  =  2  , x ≥ 0,


2 1 0 0 1
6
 x4 
x5
the variables, x3 , x4 , x5 being slack variables. Recall the simplex tableau was





1
0
0
0
0
1
0
0
0
0
1
0
1
2
1
2
0
1
− 12
− 12
3
2
− 32
0
0
0
1
5
8
1
−5





with the variables ordered as x2 , x4 , x5 , x1 , x3 and so xB = (x2 , x4 , x5 ) and
xF = (x1 , x3 ) .
194
CHAPTER 11. LINEAR PROGRAMMING
Apply the simplex algorithm to the fourth column because − 32 < 0 and this is the most negative
entry in the bottom row. The pivot is 3/2 because 1/(3/2) = 2/3 < 5/ (1/2) . Dividing this row by
3/2 and then using this to zero out the other elements in that column, the new simplex tableau is

1 0
0 1
0 0
0 0




− 13
0
0
0
1
0
2
3
1
2
3
1
− 31
−1

0 14
3
0 8
0 23
1 −4


.

Now there is still a negative number in the bottom left row. Therefore, the process should be
continued. This time the pivot is the 2/3 in the top of the column. Dividing the top row by 2/3
and then using this to zero out the entries below it,





3
2
− 23
1
2
3
2
0 − 12
1
1
2
1
0
2
1
0
2
0
0
1
0
1
0
0
0
0
0
0
1
7
1
3
3



.

Now all the numbers on the bottom left row are nonnegative so the process stops. Now recall the
variables and columns were ordered as x2 , x4 , x5 , x1 , x3 . The solution in terms of x1 and x2 is x2 = 0
and x1 = 3 and z = 3. Note that in the above, I did not worry about permuting the columns to
keep those which go with the basic variables on the left.
Here is a bucolic example.
Example 11.3.3 Consider the following table.
iron
protein
folic acid
copper
calcium
F1
1
5
1
2
1
F2
2
3
2
1
1
F3
1
2
2
1
1
F4
3
1
1
1
1
This information is available to a pig farmer and Fi denotes a particular feed. The numbers in the
table contain the number of units of a particular nutrient contained in one pound of the given feed.
Thus F2 has 2 units of iron in one pound. Now suppose the cost of each feed in cents per pound is
given in the following table.
F1 F2 F3 F4
2
3
2
3
A typical pig needs 5 units of iron, 8 of protein, 6 of folic acid, 7 of copper and 4 of calcium. (The
units may change from nutrient to nutrient.) How many pounds of each feed per pig should the pig
farmer use in order to minimize his cost?
His problem is to minimize C ≡ 2x1 + 3x2 + 2x3 + 3x4 subject to the constraints
x1 + 2x2 + x3 + 3x4
5x1 + 3x2 + 2x3 + x4
x1 + 2x2 + 2x3 + x4
≥ 5,
≥ 8,
≥ 6,
2x1 + x2 + x3 + x4
x1 + x2 + x3 + x4
≥ 7,
≥ 4.
11.3. THE SIMPLEX ALGORITHM
195
where each xi ≥ 0. Add in the slack variables,
The augmented matrix for this

1

 5

 1


 2
1
x1 + 2x2 + x3 + 3x4 − x5
5x1 + 3x2 + 2x3 + x4 − x6
x1 + 2x2 + 2x3 + x4 − x7
= 5
= 8
= 6
2x1 + x2 + x3 + x4 − x8
x1 + x2 + x3 + x4 − x9
= 7
= 4
system is
2
3
2
1
1
1
2
2
1
1
3
1
1
1
1
−1 0
0
0
0 −1 0
0
0
0 −1 0
0
0
0 −1
0
0
0
0
0
0
0
0
−1
5
8
6
7
4








How in the world can you find a basic feasible solution? Remember the simplex algorithm is designed
to keep the entries in the right column nonnegative so you use this algorithm a few times till the
obvious solution is a basic feasible solution.
Consider the first column. The pivot is the 5. Using the row operations described in the
algorithm, you get


3
14
1
17
7
−1
0
0
0
0
5
5
5
5
5

3
2
1
8 
 1

0 − 15
0
0
0
5
5
5
5 

8
4
1
22 
7
 0
0
−1 0
0

5
5
5
5
5 

1
1
3
2
19 
0
−
0
0
−1
0

5
5
5
5
5 
2
3
4
1
12
0
0
0
0
−1
5
5
5
5
5
Now go to the second column. The pivot in this column is the 7/5. This is in a different row than
the pivot in the first column so I will use it to zero out everything below it. This will get rid of the
zeros in the fifth column and introduce zeros in the second. This yields


1
17
0 1 73
2 − 57
0
0
0
7
7

3
1 
 1 0 17 −1

− 27
0
0
0
7
7 

 0 0 1 −2 1
0 −1 0
0
1 



2
1
3
30 
1 −7
0
−1
0
 0 0 7
7
7 
3
2
1
10
0 0 7
0
0
0 −1 7
7
7
Now consider another column, this time the fourth. I will pick this one because it has some negative
numbers in it so there are fewer entries to check in looking for a pivot. Unfortunately, the pivot is
the top 2 and I don’t want to pivot on this because it would destroy the zeros in the second column.
Consider the fifth column. It is also not a good choice because the pivot is the second element from
the top and this would destroy the zeros in the first column. Consider the sixth column. I can use
either of the two bottom entries as the pivot. The matrix is


0 1 0
2 −1 0 0
0
1
1


 1 0 1 −1 1 0 0
0 −2 3 


 0 0 1 −2 1 0 −1 0
0
1 




0 
 0 0 −1 1 −1 0 0 −1 3
0 0 3
0
2 1 0
0 −7 10
196
CHAPTER 11. LINEAR PROGRAMMING
Next consider the third column. The pivot is the 1 in the third row. This yields


0 1 0 2 −1 0 0
0
1 1


 1 0 0 1
0 0 1
0 −2 2 


 0 0 1 −2 1 0 −1 0
0 1 

.


 0 0 0 −1 0 0 −1 −1 3 1 
0 0 0 6 −1 1 3
0 −7 7
There are still 5 columns which consist entirely of zeros except for one entry. Four of them have
that entry equal to 1 but one still has a -1 in it, the -1 being in the fourth column. I need to do the
row operations on a nonsimple column which has the pivot in the fourth row. Such a column is the
second to the last. The pivot is the 3. The new matrix is


7
1
1
0 1 0
−1 0
0 23
3
3
3


1
1
 1 0 0
0 0
− 23 0 83 
3
3


 0 0 1 −2 1 0 −1
(11.15)
0 0 1 

.

1
1
1
1 
0
0
0
−
0
0
−
−
1

3
3
3
3 
2
7
28
0 0 0 11
−1
1
−
0
3
3
3
3
Now the obvious basic solution is feasible. You let x4 = 0 = x5 = x7 = x8 and x1 = 8/3, x2 =
2/3, x3 = 1, and x6 = 28/3. You don’t need to worry too much about this. It is the above matrix
which is desired. Now you can assemble the simplex tableau and begin the algorithm. Remember
C ≡ 2x1 + 3x2 + 2x3 + 3x4 . First add the row and column which deal with C. This yields


1
1
7
−1 0
0 0 32
0
1
0
3
3
3


1
1
0
0
0 0
− 23 0 0 83 
 1
3
3


 0
0
1 −2 1 0 −1
0 0 0 1 


(11.16)
 0
0 0 − 13 − 31 1 0 13 
0
0 − 13




11
2
 0

0
0
−1 1
− 73 0 0 28
3
3
3
−2 −3 −2 −3 0 0 0
0 0 1 0
Now you do row operations to keep the simple columns of 11.15 simple in 11.16. Of course you could
permute the columns if you wanted but this is not necessary.
This yields the following for a simplex tableau. Now it is a matter of getting rid of the positive
entries in the bottom row because you are trying to minimize.


1
1
7
−1 0
0 0 23
0 1 0
3
3
3


1
1
0 0
− 23 0 0 83 
 1 0 0
3
3


 0 0 1 −2 1 0 −1
0 0 0 1 


 0 0 0 −1
0 0 − 13 − 13 1 0 13 


3

2
7
28 
 0 0 0 11

−1
1
−
0
0
3
3
3
3
2
1
1
28
0 0 0
−1 0 − 3 − 3 0 1 3
3
The most positive of them is the 2/3 and so I will apply the algorithm to this one first. The pivot
is the 7/3. After doing the row operation the next tableau is


1
1
3
0 1 − 37 0
0 0 27
0
7
7
7


1
2
0
− 57 0 0 18
 1 − 17 0 0
7
7
7 


1
2
6
 0

1 0
0 − 57
0 0 11
7
7
7
7 

 0
1
1
2
2
3 
0 0 −7 0 −7 −7 1 0 7 

7


11
4
1
 0 −7 0 0

1
− 20
0 0 58
7
7
7
7
0 − 27 0 0 − 57 0 − 37 − 37 0 1 64
7
11.3. THE SIMPLEX ALGORITHM
197
and you see that all the entries are negative and so the minimum is 64/7 and it occurs when
x1 = 18/7, x2 = 0, x3 = 11/7, x4 = 2/7.
There is no maximum for the above problem. However, I will pretend I don’t know this and
attempt to use the simplex algorithm. You set up the simiplex tableau the same way. Recall it is


1
1
7
0 1 0
−1 0
0 0 23
3
3
3


1
1
0 0
− 23 0 0 83 
 1 0 0
3
3


 0 0 1 −2 1 0 −1
0 0 0 1 


 0 0 0 −1
0 0 − 13 − 13 1 0 13 


3

2
7
28 
 0 0 0 11

−1
1
−
0
0
3
3
3
3
2
1
1
28
0 0 0
−1 0 − 3 − 3 0 1 3
3
Now to maximize, you try to get rid of the negative entries in the bottom left row. The most
negative entry is the -1 in the fifth column. The pivot is the 1 in the third row of this column. The
new tableau is


1
1
0 1 1
0 0 − 23
0 0 53
3
3


1
1
0 0
− 23 0 0 83 
 1 0 0
3
3


 0 0 1 −2 1 0 −1
0 0 0 1 


 0 0 0 −1 0 0 −1 −1 1 0 1  .

3
3
3
3 


5

 0 0 1
0 1 − 13 − 73 0 0 31
3
3
0 0 1 − 43 0 0 − 43 − 13 0 1 31
3
Consider the fourth column. The pivot is the top 1/3. The new

0 3
3 1 0 0 −2 1 0

 1 −1 −1 0 0 0 1 −1 0

 0 6
7 0 1 0 −5 2 0

 0 1
1 0 0 0 −1 0 1


 0 −5 −4 0 0 1 3 −4 0
0 4
5 0 0 0 −4 1 0
There is still a negative in the bottom,
yields

1
0 − 13
3

1
2
 1
3
3

1
 0 −7
3
3

 0 −2 −1

3
3

 0 − 53 − 43
0 − 83 − 13
tableau is

0 5

0 1 

0 11 

0 2 


0 2 
1 17
the -4. The pivot in that column is the 3. The algorithm
1 0
0 0
0 1
0 0
0 0
0 0
2
3
− 13
5
3
1
3
1
3
4
3
0
0
0
0
1
0
− 53
1
3
− 14
3
− 43
− 43
− 13
3
0
0
0
1
0
0
0
0
0
0
0
1
19
3
1
3
43
3
8
3
2
3
59
3










Note how z keeps getting larger. Consider the column having the −13/3 in it. The pivot is the
single positive entry, 1/3. The next tableau is


5 3 2 1 0 −1 0 0 0 0 8


 3 2 1 0 0 −1 0 1 0 0 1 


 14 7 5 0 1 −3 0 0 0 0 19 


 4 2 1 0 0 −1 0 0 1 0 4  .




 4 1 0 0 0 −1 1 0 0 0 2 
13 6 4 0 0 −3 0 0 0 1 24
There is a column consisting of all negative entries. There is therefore, no maximum. Note also how
there is no way to pick the pivot in that column.
198
CHAPTER 11. LINEAR PROGRAMMING
Example 11.3.4 Minimize z = x1 − 3x2 + x3 subject to the constraints x1 + x2 + x3 ≤ 10, x1 +
x2 + x3 ≥ 2, x1 + x2 + 3x3 ≤ 8 and x1 + 2x2 + x3 ≤ 7 with all variables nonnegative.
There exists an answer because the region defined by the constraints is closed and bounded.
Adding in slack variables you get the following augmented matrix corresponding to the constraints.





1
1
1
1
1
1
1
2
1
1
3
1
1
0
0
0
0
−1
0
0

0 0 10
0 0 2 


1 0 8 
0 1 7
Of course there is a problem with the obvious solution obtained by setting to zero all variables
corresponding to a nonsimple column because of the simple column which has the −1 in it. Therefore,
I will use the simplex algorithm to make this column non simple. The third column has the 1 in the
second row as the pivot so I will use this column. This yields





0
1
−2
0
0
1
−2
1
0 1
1 0
0 0
0 0
1
−1
3
1

0 0 8
0 0 2 


1 0 2 
0 1 5
(11.17)
and the obvious solution is feasible. Now it is time to assemble the simplex tableau. First add in
the bottom row and second to last column corresponding to the equation for z. This yields


0
0
0 1 1 0 0 0 8


 1
1
1 0 −1 0 0 0 2 


 −2 −2 0 0 3 1 0 0 2 




1
0 0 1 0 1 0 5 
 0
−1 3 −1 0 0 0 0 1 0
Next you need to zero out the entries in the bottom row
in 11.17. This yields the simplex tableau

0
0 0 1 1 0

 1
1 1 0 −1 0

 −2 −2 0 0 3 1


1 0 0 1 0
 0
0
4 0 0 −1 0
which are below one of the simple columns
0
0
0
1
0
0
0
0
0
1
8
2
2
5
2




.



The desire is to minimize this so you need to get rid of the positive entries in the left bottom row.
There is only one such entry, the 4. In that column the pivot is the 1 in the second row of this
column. Thus the next tableau is


0 0 0 1 1 0 0 0 8


 1 1 1 0 −1 0 0 0 2 


 0 0 2 0 1 1 0 0 6 




 −1 0 −1 0 2 0 1 0 3 
−4 0 −4 0 3 0 0 1 −6
There is still a positive number there, the 3. The pivot in this column is the 2. Apply the algorithm
11.4. FINDING A BASIC FEASIBLE SOLUTION
again. This yields








1
2
1
2
1
2
− 12
− 52
1
0
2
1
1
2
5
0
2
0 − 12
0 − 52
1
0
0
0
0
0
0
0
1
0
0
0
1
0
0
199
− 12
1
2
− 12
1
2
− 32
13
2
7
2
9
2
3
2
− 21
2
0
0
0
0
1




.



Now all the entries in the left bottom row are nonpositive so the process has stopped. The minimum
is −21/2. It occurs when x1 = 0, x2 = 7/2, x3 = 0.
Now consider the same problem but change the word, minimize to the word, maximize.
Example 11.3.5 Maximize z = x1 − 3x2 + x3 subject to the constraints x1 + x2 + x3 ≤ 10, x1 +
x2 + x3 ≥ 2, x1 + x2 + 3x3 ≤ 8 and x1 + 2x2 + x3 ≤ 7 with all variables nonnegative.
The first part of it is the same. You wind

0
0 0

 1
1 1

 −2 −2 0


1 0
 0
0
4 0
up with the same simplex tableau,

1 1 0 0 0 8

0 −1 0 0 0 2 

0 3 1 0 0 2 


0 1 0 1 0 5 
0 −1 0 0 1 2
but this time, you apply the algorithm to get rid of the negative entries in the left bottom row.
There is a −1. Use this column. The pivot is the 3. The next tableau is


2
2
0 1 0 − 13 0 0 22
3
3
3
 1

1
1
 3
1 0 0
0 0 38 
3
3
 2

2
1
 −
0 0 32 
 3 −3 0 0 1

3
 2
5
1
13 
0
0
0
−
1
0
 3
3
3
3 
10
1
2
0 0 0
0 1 38
−3
3
3
There is still a negative entry, the −2/3. This
on the fourth row. This yields

0 −1 0 1

 0 − 12 1 0

 0 1 0 0


5
0 0
 1
2
0 5 0 0
will be the new pivot column. The pivot is the 2/3
0 0
1
0
2
1 0
0 − 12
0 0
−1
− 12
1
3
2
1
0
0
0
0
1
3




5 

13 
2 
7
1
2
and the process stops. The maximum for z is 7 and it occurs when x1 = 13/2, x2 = 0, x3 = 1/2.
11.4
Finding A Basic Feasible Solution
By now it should be fairly clear that finding a basic feasible solution can create considerable difficulty. Indeed, given a system of linear inequalities along with the requirement that each variable
be nonnegative, do there even exist points satisfying all these inequalities? If you have many variables, you can’t answer this by drawing a picture. Is there some other way to do this which is more
systematic than what was presented above? The answer is yes. It is called the method of artificial
variables. I will illustrate this method with an example.
Example 11.4.1 Find a basic feasible solution to the system 2x1 + x2 − x3 ≥ 3, x1 + x2 + x3 ≥
2, x1 + x2 + x3 ≤ 7 and x ≥ 0.
200
CHAPTER 11. LINEAR PROGRAMMING
If you write the appropriate augmented matrix with the slack variables,


2 1 −1 −1 0 0 3


0 −1 0 2 
 1 1 1
1 1 1
0
0 1 7
(11.18)
The obvious solution is not feasible. This is why it would be hard to get started with the simplex
method. What is the problem? It is those −1 entries in the fourth and fifth columns. To get around
this, you add in artificial variables to get an augmented matrix of the form


2 1 −1 −1 0 0 1 0 3


(11.19)
0 −1 0 0 1 2 
 1 1 1
1 1 1
0
0 1 0 0 7
Thus the variables are x1 , x2 , x3 , x4 , x5 , x6 , x7 , x8 . Suppose you can find a feasible solution to the
system of equations represented by the above augmented matrix. Thus all variables are nonnegative.
Suppose also that it can be done in such a way that x8 and x7 happen to be 0. Then it will follow
that x1 , · · · , x6 is a feasible solution for 11.18. Conversely, if you can find a feasible solution for
11.18, then letting x7 and x8 both equal zero, you have obtained a feasible solution to 11.19. Since
all variables are nonnegative, x7 and x8 both equalling zero is equivalent to saying the minimum of
z = x7 + x8 subject to the constraints represented by the above augmented matrix equals zero. This
has proved the following simple observation.
Observation 11.4.2 There exists a feasible solution to the constraints represented by the augmented
matrix of 11.18 and x ≥ 0 if and only if the minimum of x7 + x8 subject to the constraints of 11.19
and x ≥ 0 exists and equals 0.
Of course a similar observation would hold in other similar situations. Now the point of all this
is that it is trivial to see a feasible solution to 11.19, namely x6 = 7, x7 = 3, x8 = 2 and all the other
variables may be set to equal zero. Therefore, it is easy to find an initial simplex tableau for the
minimization problem just described. First add the column and row for z






−1 −1 0 0 1
0 0 3
1
0 −1 0 0
1 0 2 


1
0
0 1 0
0 0 7 
0
0
0 0 −1 −1 1 0
2 1
1 1
1 1
0 0
Next it is necessary to make the last two columns on the bottom left row into simple columns.
Performing the row operation, this yields an initial simplex tableau,





2
1
1
3
1
1
1
2

−1 −1 0 0 1 0 0 3
1
0 −1 0 0 1 0 2 


1
0
0 1 0 0 0 7 
0 −1 −1 0 0 0 1 5
Now the algorithm involves getting rid of the positive entries on the left bottom row. Begin with
the first column. The pivot is the 2. An application of the simplex algorithm yields the new tableau





1
0
0
0
1
2
1
2
1
2
1
2
− 12
− 12
3
2
3
2
3
2
1
2
1
2
1
2
0 0
−1 0
0 1
−1 0
1
2
− 21
− 21
− 23
0 0
1 0
0 0
0 1
3
2
1
2
11
2
1
2





11.5. DUALITY
Now go to the third column. The pivot
algorithm yields

1 23 0
 0 1 1

3

 0 0 0
0 0 0
201
is the 3/2 in the second row. An application of the simplex
− 13
1
3
0
0
− 13
− 23
1
0
0
0
1
0
1
3
− 13
0
−1
1
3
2
3
0
0
−1 0
−1 1
5
3
1
3




5 
0
(11.20)
and you see there are only nonpositive numbers on the bottom left column so the process stops and
yields 0 for the minimum of z = x7 + x8 . As for the other variables, x1 = 5/3, x2 = 0, x3 = 1/3, x4 =
0, x5 = 0, x6 = 5. Now as explained in the above observation, this is a basic feasible solution for the
original system 11.18.
Now consider a maximization problem associated with the above constraints.
Example 11.4.3 Maximize x1 −x2 +2x3 subject to the constraints, 2x1 +x2 −x3 ≥ 3, x1 +x2 +x3 ≥
2, x1 + x2 + x3 ≤ 7 and x ≥ 0.
From 11.20 you can immediately assemble an initial simplex tableau. You begin with the first 6
columns and top 3 rows in 11.20. Then add in the column and row for z. This yields


2
1
0 − 13 − 13 0 0 53
3
 0
1
1
1
− 23 0 0 13 


3
3


 0 0 0
0
1 1 0 5 
−1 1 −2 0
0 0 1 0
and you first do row operations to make
simplex tableau is

1 23
 0 1

3

 0 0
0 73
the first and third columns simple columns. Thus the next
0
1
0
0
− 13
1
3
0
1
3
− 13
− 23
1
− 53
0
0
1
0
0
0
0
1
5
3
1
3




5 
7
3
You are trying to get rid of negative entries in the bottom left row. There is only one, the −5/3.
The pivot is the 1. The next simplex tableau is then


1 32 0 − 13 0 13 0 10
3
 0 1 1

1
0 23 0 11

3
3
3 


 0 0 0 0 1 1 0 5 
1
0 53 1 32
0 73 0
3
3
and so the maximum value of z is 32/3 and it occurs when x1 = 10/3, x2 = 0 and x3 = 11/3.
11.5
Duality
You can solve minimization problems by solving maximization problems. You can also go the other
direction and solve maximization problems by minimization problems. Sometimes this makes things
much easier. To be more specific, the two problems to be considered are
A.) Minimize z = cx subject to x ≥ 0 and Ax ≥ b and
B.) Maximize w = yb such that y ≥ 0 and yA ≤ c,
(
)
equivalently AT yT ≥ cT and w = bT yT .
In these problems it is assumed A is an m × p matrix.
I will show how a solution of the first yields a solution of the second and then show how a solution
of the second yields a solution of the first. The problems, A.) and B.) are called dual problems.
202
CHAPTER 11. LINEAR PROGRAMMING
Lemma 11.5.1 Let x be a solution of the inequalities of A.) and let y be a solution of the inequalities
of B.). Then
cx ≥ yb.
and if equality holds in the above, then x is the solution to A.) and y is a solution to B.).
Proof: This follows immediately. Since c ≥ yA,
cx ≥ yAx ≥ yb.
It follows from this lemma that if y satisfies the inequalities of B.) and x satisfies the inequalities
of A.) then if equality holds in the above lemma, it must be that x is a solution of A.) and y is a
solution of B.). Now recall that to solve either of these problems using the simplex method, you first add in slack
variables. Denote by x′ and y′ the enlarged list of variables. Thus x′ has at least m entries and so
does y′ and the inequalities involving A were replaced by equalities whose augmented matrices were
of the form
(
)
(
)
A −I b , and
AT I cT
Then you included the row and column for z and w to obtain
(
)
(
A −I 0 b
AT
I 0
and
T
−c 0 1 0
−b
0 1
cT
0
)
.
(11.21)
Then the problems have basic feasible solutions if it is possible to permute the first p + m columns
in the above two matrices and obtain matrices of the form
(
)
(
)
B
F
0 b
B1
F1
0 cT
and
(11.22)
−cB −cF 1 0
−bTB1 −bTF1 1 0
where B, B1 are invertible m × m and p × p matrices and denoting the variables associated with
these columns by xB , yB and those variables associated with F or F1 by xF and(yF , it follows
) that
′
′
letting BxB = b and xF = 0, the resulting vector x is a solution to x ≥ 0 and A −I x′ = b
with similar constraints holding for y′ . In other words, it is possible to obtain simplex tableaus,
(
)
) (
I
B −1 F
0
B −1 b
I
B1−1 F1
0
B1−1 cT
(11.23)
,
0 cB B −1 F − cF 1 cB B −1 b
0 bTB1 B1−1 F − bTF1 1 bTB1 B1−1 cT
Similar considerations apply to the second problem. Thus as just described, a basic feasible solution
is one which determines a simplex tableau like the above in which you get a feasible solution by
setting all but the first m variables equal to zero. The simplex algorithm takes you from one basic
feasible solution to another till eventually, if there is no degeneracy, you obtain a basic feasible
solution which yields the solution of the problem of interest.
Theorem 11.5.2 Suppose there exists a solution, x to A.) where x is a basic feasible solution of
the inequalities of A.). Then there exists a solution, y to B.) and cx = by. It is also possible to
find y from x using a simple formula.
Proof: Since the solution to A.) is basic and feasible, there exists a simplex tableau like 11.23
such that x′ can be split into xB and xF such that xF = 0 and xB = B −1 b. Now since it is
a minimizer, it follows cB B −1 F − cF ≤ 0 and the minimum value for cx is cB B −1 b. Stating
this again, cx = cB B −1 b. Is it possible you can take y = cB B −1 ? From Lemma 11.5.1 this will be
−1
−1
so if cB B −1 solves the constraints of problem( B.). Is cB
) B ( ≥ 0? Is) cB B A ≤ c? These two
−1
≤
conditions are satisfied if and only if cB B
A −I
c 0 . Referring to the process
of permuting the columns of the first augmented matrix of 11.21 to get 11.22 and doing the same
11.5. DUALITY
203
(
)
(
)
permutations on the columns of A −I and c 0 , the desired inequality holds if and only if
(
) (
)
(
) (
)
cB B −1 B F
≤ cB cF
which is equivalent to saying cB cB B −1 F
≤ cB cF
and this is true because cB B −1 F − cF ≤ 0 due to the assumption that x is a minimizer. The simple
formula is just
y = cB B −1 . The proof of the following corollary is similar.
Corollary 11.5.3 Suppose there exists a solution, y to B.) where y is a basic feasible solution of
the inequalities of B.). Then there exists a solution, x to A.) and cx = by. It is also possible to
find x from y using a simple formula. In this case, and referring to 11.23, the simple formula is
x = B1−T bB1 .
As an example, consider the pig farmers problem. The main difficulty in this problem was finding
an initial simplex tableau. Now consider the following example and marvel at how all the difficulties
disappear.
Example 11.5.4 minimize C ≡ 2x1 + 3x2 + 2x3 + 3x4 subject to the constraints
x1 + 2x2 + x3 + 3x4
≥ 5,
5x1 + 3x2 + 2x3 + x4
x1 + 2x2 + 2x3 + x4
≥ 8,
≥ 6,
2x1 + x2 + x3 + x4
x1 + x2 + x3 + x4
≥ 7,
≥ 4.
where each xi ≥ 0.
Here the dual problem is to maximize w = 5y1 + 8y2 + 6y3 + 7y4 + 4y5 subject to the constraints




 
y1
1 5 1 2 1
2


y2   
 2 3 2 1 1 




 3 
y3 


.
≤
 1 2 2 1 1 

  2 
 y4 
3 1 1 1 1
3
y5
Adding in slack variables, these
augmented matrix is

1
 2


 1
3
inequalities are equivalent to the system of equations whose
5
3
2
1
1
2
2
1
2
1
1
1
1
1
1
1
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
2
3
2
3





Now the obvious solution is feasible so there is no hunting for an initial obvious feasible solution
required. Now add in the row and column for w. This yields


1
5
1
2
1 1 0 0 0 0 2


 2
3
2
1
1 0 1 0 0 0 3 


 1
2
2
1
1 0 0 1 0 0 2 

.


3
1
1
1
1
0
0
0
1
0
3


−5 −8 −6 −7 −4 0 0 0 0 1 0
204
CHAPTER 11. LINEAR PROGRAMMING
It is a maximization problem so you want to eliminate the negatives in the bottom left row. Pick
the column having the one which is most negative, the −8. The pivot is the top 5. Then apply the
simplex algorithm to obtain


1
2
1
1
1
1
0 0 0 0 25
5
5
5
5
5

 7
7
2
 5
0
− 15
− 35 1 0 0 0 95 
5
5

 3
8
1
3
2
6 

.
0
−
0
1
0
0
 5
5
5
5
5
5 
 14
4
3
4
1
13 
0
−
0
0
1
0
 5
5
5
5
5
5 
22
19
12
8
16
− 17
0
−
−
−
0
0
0
1
5
5
5
5
5
5
There are still negative entries in the bottom left row.
8
which has the − 22
5 . The pivot is the 5 . This yields

1
3
1
1
1 0
0
8
8
4
 78
3
1
1
 8
0 0 −8 −8 −4 1
 3
1
3

0 1
− 41 0
 8
8
8
 5
1
1
0 0
0 0
 2
2
2
3
1
− 74 0 0 − 13
−
0
4
4
2
Do the simplex algorithm to the column
− 18
− 78
5
8
− 12
11
4
0
0
0
1
0
0
0
0
0
1
1
4
3
4
3
4







2 
13
2
and there are still negative numbers. Pick the column which has the −13/4. The pivot is the 3/8 in
the top. This yields


8
2
1
0 1 13
0 − 13 0 0 23
3
3
3


 1
1 0 0 0
0 1 −1 0 0 1 
 1

2

− 13 1 0 13 − 13 0
0 0 23 
 3

3
 7
4
1
1
1
5 
−
0
0
−
0
−
1
0
 3
3
3
3
3
3 
1
8
5
26
0
0
0
0
1
− 32 26
3
3
3
3
3
which has only one negative entry on the bottom left. The pivot for this first column is the 73 . The
next tableau is


2
5
2
1
3
0
1
0
−
−
0
0 20
7
7
7
7
7
7


1
 0 11
0 0 − 17
1 − 67 − 37 0 27 
7
7


2
5
 0 −1 1 0
− 27 0
− 17 0 37 


7
7
7

1
1
1
3
5 
4
−
0
−
0
 1 −7 0 0
7
7
7
7
7 
3
18
11
2
64
0 58
0
0
0
1
7
7
7
7
7
7
and all the entries in the left bottom row are nonnegative so the answer is 64/7. This is the same
as obtained before. So what values for x are needed? Here the basic variables are y1 , y3 , y4 , y7 .
Consider the original augmented matrix, one step before the simplex tableau.


1
5
1
2
1 1 0 0 0 0 2


 2
3
2
1
1 0 1 0 0 0 3 


 1
2
2
1
1 0 0 1 0 0 2 

.


3
1
1
1
1
0
0
0
1
0
3


−5 −8 −6 −7 −4 0 0 0 0 1 0
Permute the columns to put the columns

1
1
2

 2
2
1

 1
2
1


1
1
 3
−5 −6 −7
associated with these basic variables first. Thus

0 5
1 1 0 0 0 2

1 3
1 0 0 0 0 3 

0 2
1 0 1 0 0 2 


0 1
1 0 0 1 0 3 
0 −8 −4 0 0 0 1 0
11.6. EXERCISES
205
The matrix B is





and so B −T equals





Also bTB =
(
− 17
0
− 17
3
7
)
5 6
7
0
1
2
1
3
1
2
2
1
− 27
0
5
7
− 17
2
1
1
1
0
1
0
0






5
7
1
7
0
− 27
− 17
1
− 67
− 37




and so from Corollary 11.5.3,



x=

− 17
0
− 17
3
7
− 27
0
5
7
− 17
5
7
1
7
0
− 27
− 17
1
− 67
− 37





5
6
7
0


18
7

  0 
 

 =  11 
  7 
2
7
which agrees with the original way of doing the problem.
Two good books which give more discussion of linear programming are Strang [17] and Nobel
and Daniels [14]. Also listed in these books are other references which may prove useful if you are
interested in seeing more on these topics. There is a great deal more which can be said about linear
programming.
11.6
Exercises
1. Maximize and minimize z = x1 − 2x2 + x3 subject to the constraints x1 + x2 + x3 ≤ 10, x1 +
x2 + x3 ≥ 2, and x1 + 2x2 + x3 ≤ 7 if possible. All variables are nonnegative.
2. Maximize and minimize the following if possible. All variables are nonnegative.
(a) z = x1 − 2x2 subject to the constraints x1 + x2 + x3 ≤ 10, x1 + x2 + x3 ≥ 1, and
x1 + 2x2 + x3 ≤ 7
(b) z = x1 − 2x2 − 3x3 subject to the constraints x1 + x2 + x3 ≤ 8, x1 + x2 + 3x3 ≥ 1, and
x1 + x2 + x3 ≤ 7
(c) z = 2x1 + x2 subject to the constraints x1 − x2 + x3 ≤ 10, x1 + x2 + x3 ≥ 1, and
x1 + 2x2 + x3 ≤ 7
(d) z = x1 + 2x2 subject to the constraints x1 − x2 + x3 ≤ 10, x1 + x2 + x3 ≥ 1, and
x1 + 2x2 + x3 ≤ 7
3. Consider contradictory constraints, x1 + x2 ≥ 12 and x1 + 2x2 ≤ 5, x1 ≥ 0, x2 ≥ 0. You know
these two contradict but show they contradict using the simplex algorithm.
4. Find a solution to the following inequalities for x, y ≥ 0 if it is possible to do so. If it is not
possible, prove it is not possible.
(a)
6x + 3y ≥ 4
8x + 4y ≤ 5
(b)
6x1 + 4x3 ≤ 11
5x1 + 4x2 + 4x3 ≥ 8
6x1 + 6x2 + 5x3 ≤ 11
206
CHAPTER 11. LINEAR PROGRAMMING
(c)
6x1 + 4x3 ≤ 11
5x1 + 4x2 + 4x3 ≥ 9
6x1 + 6x2 + 5x3 ≤ 9
(d)
x1 − x2 + x3 ≤ 2
x1 + 2x2 ≥ 4
3x1 + 2x3 ≤ 7
(e)
5x1 − 2x2 + 4x3 ≤ 1
6x1 − 3x2 + 5x3 ≥ 2
5x1 − 2x2 + 4x3 ≤ 5
5. Minimize z = x1 + x2 subject to x1 + x2 ≥ 2, x1 + 3x2 ≤ 20, x1 + x2 ≤ 18. Change to a
maximization problem and solve as follows: Let yi = M − xi . Formulate in terms of y1 , y2 .
Chapter 12
Spectral Theory
12.1
Eigenvalues And Eigenvectors Of A Matrix
Spectral Theory refers to the study of eigenvalues and eigenvectors of a matrix. It is of fundamental
importance in many areas. Row operations will no longer be such a useful tool in this subject.
12.1.1
Definition Of Eigenvectors And Eigenvalues
In this section, F = C.
To illustrate the idea behind what will be discussed, consider the following example.
Example 12.1.1 Here is a matrix.

0

 0
0

5 −10

22
16  .
−9 −2
(
)T
Multiply this matrix by the vector
and see what happens. Then multiply it by
5 −4 3
(
)T
and see what happens. Does this matrix act this way for some other vector?
1 0 0
First
Next

 



0 5 −10
−5
−50
−5


 



16   −4  =  −40  = 10  −4  .
 0 22
0 −9 −2
3
30
3


0 5

 0 22
0 −9
   
 
−10
1
0
1
   
 
16   0  =  0  = 0  0  .
−2
0
0
0
When you multiply the first vector by the given matrix, it stretched the vector, multiplying it by
10. When you multiplied the matrix by the second vector it sent it to the zero vector. Now consider


 

0 5 −10
1
−5


 

16   1  =  38  .
 0 22
0 −9 −2
1
−11
In this case, multiplication by the matrix did not result in merely multiplying the vector by a number.
In the above example, the first two vectors were called eigenvectors and the numbers, 10 and 0
are called eigenvalues. Not every number is an eigenvalue and not every vector is an eigenvector.
207
208
CHAPTER 12. SPECTRAL THEORY
When you have a nonzero vector which, when multiplied by a matrix results in another vector
which is parallel to the first or equal to 0, this vector is called an eigenvector of the matrix. This is
the meaning when the vectors are in Rn . Things are less apparent geometrically when the vectors
are in Cn . The precise definition in all cases follows.
Definition 12.1.2 Let M be an n × n matrix and let x ∈ Cn be a nonzero vector for which
M x = λx
(12.1)
for some scalar λ. Then x is called an eigenvector and λ is called an eigenvalue (characteristic
value) of the matrix M.
Note: Eigenvectors are never equal to zero!
The set of all eigenvalues of an n × n matrix M, is denoted by σ (M ) and is referred to as the
spectrum of M.
The eigenvectors of a matrix M are those vectors, x for which multiplication by M results in a
vector in the same direction or opposite direction to x. Since the zero vector 0 has no direction this
would make no sense for the zero vector. As noted above, 0 is never allowed to be an eigenvector.
How can eigenvectors be identified? Suppose x satisfies 12.1. Then
(M − λI) x = 0
for some x ̸= 0. (Equivalently, you could write (λI − M ) x = 0.) Sometimes we will use
(λI − M ) x = 0
and sometimes (M − λI) x = 0. It makes absolutely no difference and you should use whichever you
like better. Therefore, the matrix M − λI cannot have an inverse because if it did, the equation
could be solved,
(
)
−1
−1
−1
x = (M − λI) (M − λI) x = (M − λI) ((M − λI) x) = (M − λI) 0 = 0,
and this would require x = 0, contrary to the requirement that x ̸= 0. By Theorem 6.2.1 on Page
94,
det (M − λI) = 0.
(12.2)
(Equivalently you could write det (λI − M ) = 0.) The expression, det (λI − M ) or equivalently,
det (M − λI) is a polynomial called the characteristic polynomial and the above equation is
called the characteristic equation. For M an n × n matrix, it follows from the theorem on expanding
a matrix by its cofactor that det (M − λI) is a polynomial of degree n. As such, the equation 12.2
has a solution, λ ∈ C by the fundamental theorem of algebra. Is it actually an eigenvalue? The
answer is yes, and this follows from Observation 9.2.7 on Page 160 along with Theorem 6.2.1 on
Page 94. Since det (M − λI) = 0 the matrix det (M − λI) cannot be one to one and so there exists
a nonzero vector x such that (M − λI) x = 0. This proves the following corollary.
Corollary 12.1.3 Let M be an n × n matrix and det (M − λI) = 0. Then there exists a nonzero
vector x ∈ Cn such that (M − λI) x = 0.
12.1.2
Finding Eigenvectors And Eigenvalues
As an example, consider the following.
Example 12.1.4 Find the eigenvalues and eigenvectors for the matrix


5 −10 −5


A= 2
14
2 .
−4 −8
6
12.1. EIGENVALUES AND EIGENVECTORS OF A MATRIX
You first need to identify the eigenvalues.
det (A − λI) = 0. In this case this equation is

5 −10 −5

det  2
14
2
−4 −8
6
209
Recall this requires the solution of the equation


1


 − λ 0
0
0
1
0

0

0  = 0
1
When you expand this determinant and simplify, you find the equation you need to solve is
(
)
(λ − 5) λ2 − 20λ + 100 = 0
and so the eigenvalues are 5, 10, 10. We have listed 10 twice because it is a zero of multiplicity two
due to
2
λ2 − 20λ + 100 = (λ − 10) .
Having found the eigenvalues, it only remains to find the eigenvectors. First find the eigenvectors
for λ = 5. As explained above, this requires you to solve the equation,
  
 



0
5 −10 −5
1 0 0
x
  



 
14
2  − 5  0 1 0   y  =  0  .
 2
z
0
0 0 1
−4 −8
6
That is you need to find the solution to


  
0 −10 −5
x
0


  
9
2  y  =  0 
 2
−4 −8
1
z
0
By now this is an old problem. You set up the augmented matrix and row reduce to get the solution.
Thus the matrix you must row reduce is


0 −10 −5 | 0


(12.3)
9
2 | 0 .
 2
−4 −8
1 | 0
The row reduced echelon form is

1 0

 0

0
− 54
1
1
2
0
0
| 0


| 0 

| 0
and so the solution is any vector of the form
 5 
 5
4t
4
 −1 


 = t  −1
t
 2 
 2
t
1




where t ∈ F. You would obtain the same collection of vectors if you replaced t with 4t. Thus a
simpler description for the solutions to this system of equations whose augmented matrix is in 12.3
is


5


t  −2 
(12.4)
4
210
CHAPTER 12. SPECTRAL THEORY
where t ∈ F. Now you need to remember that you can’t take t = 0 because this would result in the
zero vector and
Eigenvectors are never equal to zero!
Other than this value, every other choice of z in 12.4 results in an eigenvector. It is a good idea to
check your work! To do so, we will take the original matrix and multiply by this vector and see if
we get 5 times this vector.





 
5
25
5 −10 −5
5





 
14
2   −2  =  −10  = 5  −2 
 2
4
20
−4 −8
6
4
so it appears this is correct. Always check your work on these problems if you care about getting
the answer right.
The parameter, t is sometimes called a free variable. The set of vectors in 12.4 is called the
eigenspace and it equals ker (A − λI) . You should observe that in this case the eigenspace has
dimension 1 because the eigenspace is the span of a single vector. In general, you obtain the solution
from the row echelon form and the number of different free variables gives you the dimension of the
eigenspace. Just remember that not every vector in the eigenspace is an eigenvector. The vector 0
is not an eigenvector although it is in the eigenspace because
Eigenvectors are never equal to zero!
Next consider the eigenvectors

5 −10

14
 2
−4 −8
for λ = 10. These


−5
1


2  − 10  0
6
0
vectors are solutions to
 
 
0 0
x
 
 
1 0   y  = 
0 1
z
the equation,

0

0 
0
That is you must find the solutions to


  
−5 −10 −5
x
0


  
=
2
4
2
y


  0 
−4 −8 −4
z
0
which reduces to consideration of the augmented matrix


−5 −10 −5 | 0


4
2 | 0 
 2
−4 −8 −4 | 0
The row reduced echelon form for this matrix

1

 0
0
is
2
0
0
1
0
0

0

0 
0
and so the eigenvectors are of the form






−2s − t
−2
−1






s

 = s 1  + t 0 .
t
0
1
You can’t pick t and s both equal to zero because this would result in the zero vector and
Eigenvectors are never equal to zero!
12.1. EIGENVALUES AND EIGENVECTORS OF A MATRIX
211
However, every other choice of t and s does result in an eigenvector for the eigenvalue λ = 10. As in
the case for λ = 5 you should check your work if you care about getting it right.

 




5 −10 −5
−1
−10
−1

 




14
2   0  =  0  = 10  0 
 2
−4 −8
6
1
10
1
so it worked. The other vector will also work. Check it.
12.1.3
A Warning
The above example shows how to find eigenvectors and eigenvalues algebraically. You may have
noticed it is a bit long. Sometimes students try to first row reduce the matrix before looking
for eigenvalues. This is a terrible idea because row operations destroy the eigenvalues. The
eigenvalue problem is really not about row operations.
The general eigenvalue problem is the hardest problem in algebra and people still do research
on ways to find eigenvalues and their eigenvectors. If you are doing anything which would yield a
way to find eigenvalues and eigenvectors for general matrices without too much trouble, the thing
you are doing will certainly be wrong. The problems you will see in this book are not too hard
because they are cooked up to be easy. General methods to compute eigenvalues and eigenvectors
numerically are presented later. These methods work even when the problem is not cooked up to
be easy.
Notwithstanding the above discouraging observations, one can sometimes simplify the matrix
first before searching for the eigenvalues in other ways.
Lemma 12.1.5 Let A = S −1 BS where A, B are n × n matrices. Then A, B have the same eigenvalues.
Proof: Say Ax = λx, x ̸= 0. Then
S −1 BSx = λx and so BSx = λSx.
Since S is one to one, Sx ̸= 0. Thus if λ is an eigenvalue for A then it is also an eigenvalue for B.
The other direction is similar. Note that from the proof of the lemma, the eigenvectors for A, B are different.
One can now attempt to simplify a matrix before looking for the eigenvalues by using elementary
matrices for S. This is illustrated in the following example.
Example 12.1.6 Find the eigenvalues for the

33

 10
−20
matrix
105
28
−60

105

30 
−62
It has big numbers. Use the row operation of adding two times the second row to the bottom
and multiply by the inverse of the elementary matrix which does this row operation as illustrated.



 

1 0 0
33
105 105
1 0 0
33 −105 105



 

28
30   0 1 0  =  10 −32
30 
 0 1 0   10
0 2 1
−20 −60 −62
0 −2 1
0
0
−2
This one has the same eigenvalues as the first matrix but
Next, do the same sort of thing



1 −3 0
33 −105 105
1 3



30   0 1
 0 1 0   10 −32
0 0 1
0
0
−2
0 0
is of course much easier to work with.
 
0
3
 
=
0   10
1
0
0
−2
0

15

30 
−2
(12.5)
212
CHAPTER 12. SPECTRAL THEORY
At this point, you can get the eigenvalues easily. This last matrix has the same eigenvalues as the
first. Thus the eigenvalues are obtained by solving
(−2 − λ) (−2 − λ) (3 − λ) = 0,
and so the eigenvalues of the original matrix are −2, −2, 3.
At this point, you go back to the original matrix A, form A − λI, and then the problem from
here on does reduce to row operations. In general, if you are so fortunate as to find the eigenvalues
as in the above example, then finding the eigenvectors does reduce to row operations and this part
of the problem is easy.
However, finding the eigenvalues along with the eigenvectors is anything but easy because for
an n × n matrix A, it involves solving a polynomial equation of degree n. If you only find a good
approximation to the eigenvalue, it won’t work. It either is or is not an eigenvalue and if it is not,
the only solution to the equation, (A − λI) x = 0 will be the zero solution as explained above and
Eigenvectors are never equal to zero!
Another thing worth noting is that when you multiply on the right by an elementary operation,
you are merely doing the column operation defined by the elementary matrix. In 12.5 multiplication
by the elementary matrix on the right merely involves taking three times the first column and adding
to the second. Thus, without referring to the elementary matrices, the transition to the new matrix
in 12.5 can be illustrated by






33 −105 105
3 −9 15
3
0
15






30  →  10 −32 30  →  10 −2 30 
 10 −32
0
0
−2
0
0
−2
0
0 −2
Here is another example.
Example 12.1.7 Let

2 2

A= 1 3
−1 1

−2

−1 
1
First find the eigenvalues. If you like, you could use the above technique to simplify the matrix,
obtaining one which has the same eigenvalues, but since the numbers are not large, it is probably
better to just expand the determinant without any tricks.




2 2 −2
1 0 0




det  1 3 −1  − λ  0 1 0  = 0
−1 1 1
0 0 1
This reduces to λ3 − 6λ2 + 8λ = 0 and the solutions are 0, 2, and 4.
0 Can be an Eigenvalue!
Now find the eigenvectors. For λ = 0 the augmented

2 2 −2

 1 3 −1
−1 1 1
matrix for finding the solutions is

| 0

| 0 
| 0
12.1. EIGENVALUES AND EIGENVECTORS OF A MATRIX
213
and the row reduced echelon form is

1 0

 0 1
0 0

−1 0

0 0 
0 0
(
Therefore, the eigenvectors are of the form t
Next find the eigenvectors for λ = 2. The
to find these eigenvectors is

0

 1
−1
)T
where t ̸= 0.
1 0 1
augmented matrix for the system of equations needed

−2 | 0

−1 | 0 
−1 | 0
2
1
1
and the row reduced echelon form is

1 0

 0 1
0 0

0 0

−1 0 
0 0
(
)T
and so the eigenvectors are of the form t 0 1 1
where t ̸= 0.
Finally find the eigenvectors for λ = 4. The augmented matrix for the system of equations needed
to find these eigenvectors is


−2 2 −2 | 0


 1 −1 −1 | 0 
−1 1 −3 | 0
and the row reduced echelon form is


1 −1 0 0


 0 0 1 0 .
0 0 0 0
(
Therefore, the eigenvectors are of the form t
12.1.4
)T
1
1
0
where t ̸= 0.
Triangular Matrices
Although it is usually hard to solve the eigenvalue problem, there is a kind of matrix for which this
is not the case. These are the upper or lower triangular matrices. I will illustrate by examples.


1 2 4


Example 12.1.8 Let A =  0 4 7  . Find its eigenvalues.
0 0 6
You need to solve




1 2 4
1 0 0




0 = det  0 4 7  − λ  0 1 0 
0 0 6
0 0 1


1−λ
2
4


= det  0
4−λ
7  = (1 − λ) (4 − λ) (6 − λ) .
0
0
6−λ
214
CHAPTER 12. SPECTRAL THEORY
Thus the eigenvalues are just the diagonal entries of the original matrix. You can see it would work
this way with any such matrix. These matrices are called upper triangular. Stated precisely, a
matrix A is upper triangular if Aij = 0 for all i > j. Similarly, it is easy to find the eigenvalues for
a lower triangular matrix, on which has all zeros above the main diagonal.
12.1.5
Defective And Nondefective Matrices
Definition 12.1.9 By the fundamental theorem of algebra, it is possible to write the characteristic
equation in the form
r
r
r
(λ − λ1 ) 1 (λ − λ2 ) 2 · · · (λ − λm ) m = 0
where ri is some integer no smaller than 1. Thus the eigenvalues are λ1 , λ2 , · · · , λm . The algebraic
multiplicity of λj is defined to be rj .
Example 12.1.10 Consider the matrix

1

A= 0
0
1
1
0

0

1 
1
(12.6)
What is the algebraic multiplicity of the eigenvalue λ = 1?
In this case the characteristic equation is
3
det (A − λI) = (1 − λ) = 0
or equivalently,
3
det (λI − A) = (λ − 1) = 0.
Therefore, λ is of algebraic multiplicity 3.
Definition 12.1.11 The geometric multiplicity of an eigenvalue is the dimension of the eigenspace,
ker (A − λI) .
Example 12.1.12 Find the geometric multiplicity of λ = 1 for the matrix in 12.6.
We need to solve

  
0 1 0
x
0


  
 0 0 1  y  =  0 .
0 0 0
z
0

The augmented matrix which must be row reduced

0 1 0

 0 0 1
0 0 0
to get this solution is therefore,

| 0

| 0 
| 0
(
This requires z = y = 0 and x is arbitrary. Thus the eigenspace is t
the geometric multiplicity of λ = 1 is 1.
)T
1 0
0
, t ∈ F. It follows
Definition 12.1.13 An n × n matrix is called defective if the geometric multiplicity is not equal to
the algebraic multiplicity for some eigenvalue. Sometimes such an eigenvalue for which the geometric
multiplicity is not equal to the algebraic multiplicity is called a defective eigenvalue. If the geometric
multiplicity for an eigenvalue equals the algebraic multiplicity, the eigenvalue is sometimes referred
to as nondefective.
Here is another more interesting example of a defective matrix.
12.1. EIGENVALUES AND EIGENVECTORS OF A MATRIX
Example 12.1.14 Let
215


2 −2 −1


A =  −2 −1 −2  .
14 25 14
Find the eigenvectors and eigenvalues.
In this case the eigenvalues are 3, 6, 6 where we have listed 6 twice because it is a zero of algebraic
multiplicity two, the characteristic equation being
2
(λ − 3) (λ − 6) = 0.
It remains to find the eigenvectors for these eigenvalues. First consider the eigenvectors for λ = 3.
You must solve
  
 



0
x
1 0 0
2 −2 −1
  
 



 −2 −1 −2  − 3  0 1 0   y  =  0  .
0
z
14 25 14
0 0 1
The augmented matrix is


−2 −1 | 0

−4 −2 | 0 
25 11 | 0
−1

 −2
14
and the row reduced echelon form is

1 0

 0 1
0 0

−1 0

1 0 
0 0
(
)T
(
so the eigenvectors are nonzero vectors of the form t −t t
=t 1
Next consider the eigenvectors for λ = 6. This requires you to solve



 
 
2 −2 −1
1 0 0
x



 
 
 −2 −1 −2  − 6  0 1 0   y  = 
14 25 14
0 0 1
z
−1
)T
1
.

0

0 
0
and the augmented matrix for this system of equations is


−4 −2 −1 | 0


 −2 −7 −2 | 0 
14 25
8 | 0
The row reduced echelon form is

1
1
8
0

 0 1

0 0
1
4
0
(
0


0 

0
)T
(
)T
and so the eigenvectors for λ = 6 are of the form t − 18 − 14 1
or simply as t −1 −2 8
where t ∈ F.
Note that in this example the eigenspace for the eigenvalue λ = 6 is of dimension 1 because there
is only one parameter. However, this eigenvalue is of multiplicity two as a root to the characteristic
216
CHAPTER 12. SPECTRAL THEORY
equation. Thus this eigenvalue is a defective eigenvalue. However, the eigenvalue 3 is nondefective.
The matrix is defective because it has a defective eigenvalue.
The word, defective, seems to suggest there is something wrong with the matrix. This is in fact
the case. Defective matrices are a lot of trouble in applications and we may wish they never occurred.
However, they do occur as the above example shows. When you study linear systems of differential
equations, you will have to deal with the case of defective matrices and you will see how awful they
are. The reason these matrices are so horrible to work with is that it is impossible to obtain a basis
of eigenvectors. When you study differential equations, solutions to first order systems are expressed
in terms of eigenvectors of a certain matrix times eλt where λ is an eigenvalue. In order to obtain a
general solution of this sort, you must have a basis of eigenvectors. For a defective matrix, such a basis
does not exist and so you have to go to something called generalized eigenvectors. Unfortunately,
it is never explained in beginning differential equations courses why there are enough generalized
eigenvectors and eigenvectors to represent the general solution. In fact, this reduces to a difficult
question in linear algebra equivalent to the existence of something called the Jordan Canonical form
which is much more difficult than everything discussed in the entire differential equations course. If
you become interested in this, see Appendix A.
Ultimately, the algebraic issues which will occur in differential equations are a red herring anyway.
The real issues relative to existence of solutions to systems of ordinary differential equations are
analytical, having much more to do with calculus than with linear algebra although this will likely
not be made clear when you take a beginning differential equations class.
In terms of algebra, this lack of a basis of eigenvectors says that it is impossible to obtain a
diagonal matrix which is similar to the given matrix.
Although there may be repeated roots to the characteristic equation, 12.2 and it is not known
whether the matrix is defective in this case, there is an important theorem which holds when considering eigenvectors which correspond to distinct eigenvalues.
Theorem 12.1.15 Suppose M vi = λi vi , i = 1, · · · , r , vi ̸= 0, and that if i ̸= j, then λi ̸= λj .
Then the set of eigenvectors, {v1 , · · · , vr } is linearly independent.
Proof. Suppose the claim of the lemma is not true. Then there exists a subset of this set of
vectors
{w1 , · · · , wr } ⊆ {v1 , · · · , vk }
such that
r
∑
cj wj = 0
(12.7)
j=1
where each cj ̸= 0. Say M wj = µj wj where
{µ1 , · · · , µr } ⊆ {λ1 , · · · , λk } ,
the µj being distinct eigenvalues of M . Out of all such subsets, let this one be such that r is as
small as possible. Then necessarily, r > 1 because otherwise, c1 w1 = 0 which would imply w1 = 0,
which is not allowed for eigenvectors.
Now apply M to both sides of 12.7.
r
∑
cj µj wj = 0.
(12.8)
j=1
Next pick µk ̸= 0 and multiply both sides of 12.7 by µk . Such a µk exists because r > 1. Thus
r
∑
cj µk wj = 0
j=1
Subtract the sum in 12.9 from the sum in 12.8 to obtain
r
∑
(
)
cj µk − µj wj = 0
j=1
(12.9)
12.1. EIGENVALUES AND EIGENVECTORS OF A MATRIX
217
(
)
Now one of the constants cj µk − µj equals 0, when j = k. Therefore, r was not as small as
possible after all. Here is another proof in case you did not follow the above.
Theorem 12.1.16 Suppose M vi = λi vi , i = 1, · · · , r , vi ̸= 0, and that if i ̸= j, then λi ̸= λj .
Then the set of eigenvectors, {v1 , · · · , vr } is linearly independent.
Proof: Suppose the conclusion is not true. Then in the matrix
(
)
v1 v2 · · · vr
not every column is a pivot column. Let the pivot columns be {w1 , · · · , wk }, k < r. Then there
exists v ∈ {v1 , · · · , vr } , M v =λv v, v ∈
/ {w1 , · · · , wk } , such that
v=
k
∑
ci wi .
(12.10)
ci λwi wi
(12.11)
i=1
Then doing M to both sides yields
λv v =
k
∑
i=1
But also you could multiply both sides of 12.10 by λv to get
λv v =
k
∑
ci λv wi .
i=1
And now subtracting this from 12.11 yields
0=
k
∑
ci (λv − λwi ) wi
i=1
and by independence of the {w1 , · · · , wk } , this requires ci (λv − λwi ) = 0 for each i. Since the
eigenvalues are distinct, λv − λwi ̸= 0 and so each ci = 0. But from 12.10, this requires v = 0 which
is impossible because v is an eigenvector and
Eigenvectors are never equal to zero!
12.1.6
Diagonalization
First of all, here is what it means for two matrices to be similar.
Definition 12.1.17 Let A, B be two n × n matrices. Then they are similar if and only if there
exists an invertible matrix S such that
A = S −1 BS
Proposition 12.1.18 Define for n × n matrices A ∼ B if A is similar to B. Then
A ∼ A,
If A ∼ B then B ∼ A
If A ∼ B and B ∼ C then A ∼ C
218
CHAPTER 12. SPECTRAL THEORY
Proof: It is clear that A ∼ A because you could just take S = I. If A ∼ B, then for some S
invertible,
A = S −1 BS and so SAS −1 = B
But then
(
S −1
)−1
AS −1 = B
which shows that B ∼ A.
Now suppose A ∼ B and B ∼ C. Then there exist invertible matrices S, T such that
A = S −1 BS, B = T −1 CT.
Therefore,
A = S −1 T −1 CT S = (T S)
−1
C (T S)
showing that A is similar to C. For your information, when ∼ satisfies the above conditions, it is called a similarity relation.
Similarity relations are very significant in mathematics.
When a matrix is similar to a diagonal matrix, the matrix is said to be diagonalizable. I think
this is one of the worst monstrosities for a word that I have ever seen. Nevertheless, it is commonly
used in linear algebra. It turns out to be the same as nondefective which will follow easily from later
material. The following is the precise definition.
Definition 12.1.19 Let A be an n×n matrix. Then A is diagonalizable if there exists an invertible
matrix S such that
S −1 AS = D
where D is a diagonal matrix. This means D has a zero as every entry except for the main diagonal.
More precisely, Dij = 0 unless i = j. Such matrices look like the following.


∗
0


..


.


0
∗
where ∗ might not be zero.
The most important theorem about diagonalizability1 is the following major result. First here is
a simple observation.
(
)
Observation 12.1.20 Let S =
where S is n × n. Then here is the result of
s1 · · · sn
multiplying on the right by a diagonal matrix.


λ1
(
)
)
 (
..
 = λ1 s1 · · · λn sn
s1 · · · sn 
.


λn
This follows from the way we multiply matrices. The ith entry of the j th column of the product on
the left is of the form si λj . Thus the j th column of the matrix on the left is just λj sj .
Theorem 12.1.21 An n × n matrix is diagonalizable if and only if Fn has a basis of eigenvectors
of A. Furthermore, you can take the matrix S described above, to be given as
(
)
S = s1 s2 · · · sn
where here the sk are the eigenvectors in the basis for Fn . If A is diagonalizable, the eigenvalues of
A are the diagonal entries of the diagonal matrix.
1 This
word has 9 syllables
12.1. EIGENVALUES AND EIGENVECTORS OF A MATRIX
Proof: To say that A is diagonalizable, is to say that

λ1

..
−1
S AS = 
.

the λi being elements of F. This is to say that for S =
(




λn
A
s1
···
)
sn
(
=
···
s1
sn
)
···
s1

(
219
sn
λ1
)


..
, sk being the k th column,




.
λn
which is equivalent, from the way we multiply matrices, that
(
) (
As1 · · · Asn = λ1 s1 · · ·
)
λn sn
which is equivalent to saying that the columns of S are eigenvectors and the diagonal matrix has
the eigenvectors down the main diagonal. Since S −1 is invertible, these eigenvectors are a basis.
Similarly, if there is a basis of eigenvectors, one can take them as the columns of S and reverse the
above steps, finally concluding that A is diagonalizable. 

2
0
0


Example 12.1.22 Let A =  1
4 −1  . Find a matrix, S such that S −1 AS = D, a diag−2 −4 4
onal matrix.
Solving det (λI − A) = 0 yields the eigenvalues are 2 and 6 with 2 an eigenvalue of multiplicity
two. Solving (2I − A) x = 0 to find the eigenvectors, you find that the eigenvectors are




−2
1




a 1  + b 0 
0
1


0


where a, b are scalars. An eigenvector for λ = 6 is  1  . Let the matrix S be
−2


−2 1 0


S= 1 0 1 
0 1 −2
That is, the columns are the eigenvectors. Then

− 14
 1
−1
S = 2
1
4
Then S −1 AS =

− 14
 1
 2
1
4
1
2
1
1
2
1
4
1
2
− 14

2

 1
−2
1
2
1
1
2
1
4
1
2
− 14


.

0
0
−2 1

4 −1   1 0
−4 4
0 1
 
0
2 0
 
1 = 0 2
−2
0 0

0

0 .
6
We know the result from the above theorem, but it is nice to see it work in a specific example just
the same. You may wonder if there is a need to find S −1 . The following is an example of a situation
where this is needed. It is one of the major applications of diagonalizability.
220
CHAPTER 12. SPECTRAL THEORY

2

Example 12.1.23 Here is a matrix. A =  0
−1
Sometimes this sort of problem can
are eigenvectors,

0

 0
1

1 0

1 0  Find A50 .
−1 1
be made easy by using diagonalization. In this case there

 
−1
−1

 
 
, 1 , 0 ,
1
0
 
the first two corresponding to λ = 1 and the last
be the columns of the matrix, S. Thus

0

S= 0
1
corresponding to λ = 2. Then let the eigenvectors
−1
1
0

−1

0 
1

Then also
S −1
and S −1 AS =

1

 0
−1


1 1
2
1 0
0


1 0  0
1 0  0
−1 0
−1 −1 1
1
Now it follows

A = SDS −1
0

= 0
1
−1
1
0

1

0 
0
1
1

= 0
1
−1 −1
 
−1 −1
1
 
1
0 = 0
0
1
0

−1
1 0

0  0 1
1
0 0

0
1

0  0
2
−1
0
1
0
1
1
−1

0

0 =D
2

1

0 .
0
(
)2
(
)3
Note that SDS −1 = SDS −1 SDS −1 = SD2 S −1 and SDS −1 = SDS −1 SDS −1 SDS −1 =
SD3 S −1 , etc. In general, you can see that
(
)n
SDS −1 = SDn S −1
In other words, An = SDn S −1 . Therefore,


0 −1 −1
1 0


50
50 −1
A = SD S =  0 1
0  0 1
1 0
1
0 0
50 
0
1
 
0   0
2
−1
It is easy to raise a diagonal matrix

1

 0
0

0

0 .
250
It follows A50 =

0 −1

 0 1
1 0

−1
1

0  0
1
0
0
1
0
to a power.
50 
1 0
0 0


1 0  = 0 1
0 0
0 2

0
1
1

0  0
1
250
−1 −1
 
1
250
 
0 =
0
0
1 − 250

1 1

1 0 .
−1 0
−1 + 250
1
1 − 250

0

0 .
1
12.1. EIGENVALUES AND EIGENVECTORS OF A MATRIX
221
That isn’t too hard. However, this would have been horrendous if you had tried to multiply A50 by
hand.
This technique of diagonalization is also important in solving the differential equations resulting
from vibrations. Sometimes you have systems of differential equation and when you diagonalize an
appropriate matrix, you “decouple” the equations. This is very nice. It makes hard problems trivial.
The above example is entirely typical. If A = SDS −1 then Am = SDm S −1 and it is easy to
compute Dm . More generally, you can define functions of the matrix using power series in this way.
12.1.7
The Matrix Exponential
When A is diagonalizable, one can easily define what is meant by eA . Here is how. You know
S −1 AS = D
where D is a diagonal matrix. You also know that if D is of the form


λ1
0


..


.


0
λn

then

Dm = 

λm
1
0
..
.




λm
n
0
and that
(12.12)
Am = SDm S −1
as shown above. Recall why this was.
A = SDS −1
and so
n times
m
A
z
}|
{
= SDS −1 SDS −1 SDS −1 · · · SDS −1 = SDm S −1
Now formally write the following power series for eA
eA ≡
∞
∑
Ak
k=0
k!
=
∞
∑
SDk S −1
k!
k=0
=S
∞
∑
Dk
k=0
If D is given above in 12.12, the above sum is of the form

 ∑∞

1 k
1 k
λ
0
1
k=0 k! λ1
∞
∑


 k!
..
 S −1 = S 

S
.



k=0
1 k
λ
0
0
k! n


=S

eλ1
0
..
0
.

 −1
S

eλn
and this last thing is the definition of what is meant by eA .
k!
S −1
0
..
.
∑∞
1 k
k=0 k! λn

 −1
S

222
CHAPTER 12. SPECTRAL THEORY
Example 12.1.24 Let

2 −1

A= 1
2
−1 1

−1

1 
2
Find eA .
The eigenvalues happen to be 1, 2, 3 and eigenvectors associated with these eigenvalues are






−1
0
−1






 −1  ↔ 2,  −1  ↔ 1,  0  ↔ 3
1
1
1

−1 0 −1


S =  −1 −1 0 
1
1
1

Then let

and so
S −1
−1

= 1
0

and
2

D= 0
0
Then the matrix exponential is

−1 0

 −1 −1
1
1

−1
e2

0  0
1
0
0
e1
0

−1 −1

0
1 
1
1

0

0 
3
0
1
0

0
−1

0  1
e3
0

e2
e2 − e3
 2
e2
 e −e
−e2 + e −e2 + e3

−1 −1

0
1 
1
1

e2 − e3

e2 − e

2
3
−e + e + e
Isn’t that nice? You could also talk about sin (A) or cos (A) etc. You would just have to use a
different power series.
This matrix exponential is actually a useful idea when solving autonomous systems of first order
linear differential equations. These are equations which are of the form
x′ = Ax
where x is a vector in Rn or Cn and A is an n × n matrix. Then it turns out that the solution to
the above system of equations is x (t) = eAt c where c is a constant vector.
12.1.8
Complex Eigenvalues
Sometimes you have to consider eigenvalues which are complex numbers. This occurs in differential
equations for example. You do these problems exactly the same way as you do the ones in which
the eigenvalues are real. Here is an example.
12.2. SOME APPLICATIONS OF EIGENVALUES AND EIGENVECTORS
223
Example 12.1.25 Find the eigenvalues and eigenvectors of the matrix


1 0 0


A =  0 2 −1  .
0 1 2
You need to find the eigenvalues. Solve




1 0 0
1 0 0




det  0 2 −1  − λ  0 1 0  = 0.
0 0 1
0 1 2
( 2
)
This reduces to (λ − 1) λ − 4λ + 5 = 0. The solutions are λ = 1, λ = 2 + i, λ = 2 − i.
There is nothing new about finding the eigenvectors for λ = 1 so consider the eigenvalue λ = 2+i.
You need to solve


 
 
  
1 0 0
1 0 0
x
0


 
 
  
(2 + i)  0 1 0  −  0 2 −1   y  =  0 
0 0 1
0 1 2
z
0
In other words, you must consider the augmented matrix


1+i 0 0 | 0


i 1 | 0 
 0
0
−1 i | 0
for the solution. Divide the top row by (1 + i)
the bottom. This yields

1 0

 0 i
0 0
Now multiply the second row by −i to obtain

1 0

 0 1
0 0
and then take −i times the second row and add to

0 | 0

1 | 0 
0 | 0

0 | 0

−i | 0 
0 | 0
(
)T
Therefore, the eigenvectors are of the form t 0 i 1
. You should find the eigenvectors for
(
)T
λ = 2 − i. These are t 0 −i 1
. As usual, if you want to get it right you had better check it.

1 0

 0 2
0 1

 



0
0
0
0

 



−1   −i  =  −1 − 2i  = (2 − i)  −i 
2
1
2−i
1
so it worked.
12.2
Some Applications Of Eigenvalues And Eigenvectors
12.2.1
Principal Directions
Recall that n × n matrices can be considered as linear transformations. If F is a 3 × 3 real matrix
having positive determinant, it can be shown that F = RU where R is a rotation matrix and U is a
224
CHAPTER 12. SPECTRAL THEORY
symmetric real matrix having positive eigenvalues. An application of this wonderful result, known
to mathematicians as the right polar factorization, is to continuum mechanics where a chunk of
material is identified with a set of points in three dimensional space.
The linear transformation, F in this context is called the deformation gradient and it describes
the local deformation of the material. Thus it is possible to consider this deformation in terms of
two processes, one which distorts the material and the other which just rotates it. It is the matrix
U which is responsible for stretching and compressing. This is why in elasticity, the stress is often
taken to depend on U which is known in this context as the right Cauchy Green strain tensor.
In this context, the eigenvalues will always be positive. The symmetry of U allows the proof of a
theorem which says that if λM is the largest eigenvalue, then in every other direction other than
the one corresponding to the eigenvector for λM the material is stretched less than λM and if λm
is the smallest eigenvalue, then in every other direction other than the one corresponding to an
eigenvector of λm the material is stretched more than λm . This process of writing a matrix as a
product of two such matrices, one of which preserves distance and the other which distorts is also
important in applications to geometric measure theory an interesting field of study in mathematics
and to the study of quadratic forms which occur in many applications such as statistics. Here we
are emphasizing the application to mechanics in which the eigenvectors of the symmetric matrix U
determine the principal directions, those directions in which the material is stretched the most
or the least.
Example 12.2.1 Find the principal directions determined by the matrix
 29 6

6







11
11
11
6
11
41
44
19
44
6
11
19
44
41
44







The eigenvalues are 3, 1, and 12 .
It is nice to be given the eigenvalues. The largest eigenvalue is 3 which means that in the
direction determined by the eigenvector associated with 3 the stretch is three times as large. The
smallest eigenvalue is 1/2 and so in the direction determined by the eigenvector for 1/2 the material
is stretched by a factor of 1/2, becoming locally half as long. It remains to find these directions.
First consider the eigenvector for 3. It is necessary to solve


 29 6
6
 

1
 
3  0


0

0
1
0



0
 
0 −


1

11
11
11
6
11
41
44
19
44
6
11
19
44
41
44
  
 

x
0
 
 
  y 
=
0 




z
0

Thus the augmented matrix for this system of equations is

4
11


 −6
 11


 −6
11
The row reduced echelon form is

1

 0
0
6
− 11
6
− 11
91
44
19
− 44
− 19
44
91
44
0
1
0
| 0



| 0 



| 0 

−3 0

−1 0 
0 0
12.2. SOME APPLICATIONS OF EIGENVALUES AND EIGENVECTORS
225
and so the principal direction for the eigenvalue, 3 in which the material is stretched to the maximum
extent is
 
3
 
 1 .
1
A direction vector (or unit vector) in this direction is

√ 
3/ 11
√ 

 1/ 11  .
√
1/ 11
You should show that the direction in which the material is compressed the most is in the direction


0
√ 

 −1/ 2 
√
1/ 2
Note this is meaningful information which you would have a hard time finding without the theory
of eigenvectors and eigenvalues.
12.2.2
Migration Matrices
There are applications which are of great importance which feature only one eigenvalue.
Definition 12.2.2 Let n locations be denoted by the numbers 1, 2, · · · , n. Also suppose it is the case
that each year aij denotes the proportion of residents in location j which move to location i. Also
suppose
no one escapes or emigrates from without these n locations. This last assumption requires
∑
a
=
1. Such matrices in which the columns are nonnegative numbers which sum to one are
i ij
called Markov matrices. In this context describing migration, they are also called migration
matrices.
Example 12.2.3 Here is an example of one of these matrices.
(
)
.4 .2
.6 .8
Thus if it is considered as a migration matrix, .4 is the proportion of residents in location 1 which
stay in location one in a given time period while .6 is the proportion of residents in location 1 which
move to location 2 and .2 is the proportion of residents in location 2 which move to location 1.
Considered as a Markov matrix, these numbers are usually identified with probabilities.
T
If v = (x1 , · · · , xn ) where xi is the population of∑
location i at a given instant, you obtain the
population of location i one year later by computing j aij xj = (Av)i . Therefore, the population
(
)
of location i after k years is Ak v i . An obvious application of this would be to a situation in which
you rent trailers which can go to various parts of a city and you observe through experiments the
proportion of trailers which go from point i to point j in a single day. Then you might want to find
how many trailers would be in all the locations after 8 days.
Proposition 12.2.4 Let A = (aij ) be a migration matrix. Then 1 is always an eigenvalue for A.
( )
Proof: Remember that det B T = det (B) . Therefore,
(
)
(
)
T
det (A − λI) = det (A − λI)
= det AT − λI
226
CHAPTER 12. SPECTRAL THEORY
because I T = I. Thus the characteristic equation for A is the same as the characteristic equation
for AT and so A and AT have the same eigenvalues. We will show that 1 is an eigenvalue for AT
and then it will follow that 1 is an eigenvalue∑
for A.
T
Remember that for a migration matrix,
= (bij ) so bij = aji , it
i aij = 1. Therefore, if A
follows that
∑
∑
bij =
aji = 1.
j
j
Therefore, from matrix multiplication,
   ∑
  
1
1
j bij






.
.
 =  .. 
..  = 
..
AT 
.
  
  
∑
1
b
1
ij
j
 
1
 . 
T

which shows that  .. 
 is an eigenvector for A corresponding to the eigenvalue, λ = 1. As
1
explained above, this shows that λ = 1 is an eigenvalue for A because A and AT have the same
eigenvalues. 

.6 0 .1


Example 12.2.5 Consider the migration matrix  .2 .8 0  for locations 1, 2, and 3. Suppose
.2 .2 .9
initially there are 100 residents in location 1, 200 in location 2 and 400 in location 4. Find the
population in the three locations after 10 units of time.
From the above, it suffices

.6

 .2
.2
to consider
10 
 

0 .1
100
115. 085 829 22
 
 

.8 0   200  =  120. 130 672 44 
.2 .9
400
464. 783 498 34
Of course you would need to round these numbers off.
A related problem asks for how many there will be in the various locations after a long time. It
turns out that if some power of the migration matrix has all positive entries, then there is a limiting
vector x = limk→∞ Ak x0 where x0 is the initial vector describing the number of inhabitants in the
various locations initially. This vector will be an eigenvector for the eigenvalue 1 because
x = lim Ak x0 = lim Ak+1 x0 = A lim Ak x = Ax,
k→∞
k→∞
k→∞
and the sum of its entries will equal the sum of the entries of the initial vector x0 because this sum
is preserved for every multiplication by A since
(
)
∑∑
∑
∑
∑
aij xj =
xj
aij =
xj .
i
j
j
i
j
Here is an example. It is the same example as the one above but here it will involve the long time
limit.


.6 0 .1


Example 12.2.6 Consider the migration matrix  .2 .8 0  for locations 1, 2, and 3. Suppose
.2 .2 .9
initially there are 100 residents in location 1, 200 in location 2 and 400 in location 4. Find the
population in the three locations after a long time.
12.2. SOME APPLICATIONS OF EIGENVALUES AND EIGENVECTORS
227
You just need to find the eigenvector which goes with the eigenvalue 1 and then normalize it so
the sum of its entries equals the sum of the entries of the initial vector. Thus you need to find a
solution to

 
 
  
1 0 0
.6 0 .1
x
0

 
 
  
 0 1 0  −  .2 .8 0   y  =  0 
0 0 1
.2 .2 .9
z
0
The augmented matrix is


| 0

| 0 
| 0
−. 1
0
.1
.4
0

 −. 2 . 2
−. 2 −. 2
and its row reduced echelon form is

1 0

 0 1
0 0
Therefore, the eigenvectors are

−. 25 0

−. 25 0 
0
0

(1/4)


s  (1/4) 
1

and all that remains is to choose the value of s such that
1
1
s + s + s = 100 + 200 + 400
4
4
This yields s =
1400
3
and so the long time limit would equal
 

(1/4)
116. 666 666 666 666 7
1400 
 

 (1/4)  =  116. 666 666 666 666 7  .
3
1
466. 666 666 666 666 7

You would of course need to round these numbers off. You see that you are not far off after just 10
units of time. Therefore, you might consider this as a useful procedure because it is probably easier
to solve a simple system of equations than it is to raise a matrix to a large power.




Example 12.2.7 Suppose a migration matrix is 



1
5
1
2
1
5
1
4
1
4
1
2
11
20
1
4
3
10




 . Find the comparison between



the populations in the three locations after a long time.
This amounts to nothing more than finding the eigenvector for λ = 1. Solve


 1

 0


 0

0
1
0
 

0
 
−
0  


1

1
5
1
2
1
5
1
4
1
4
1
2
11
20
1
4
3
10

  
 

x
0
 
 
  y 
=
  0 .


z
0

228
CHAPTER 12. SPECTRAL THEORY
The augmented matrix is

4
5


 −1

4


 − 11
20
The row echelon form is

Thus there will be
18 th
16
− 15
3
4
− 21
− 14
7
10
1 0
− 16
19
1
− 18
19
0
0

 0

0
and so an eigenvector is
− 12
| 0



| 0 



| 0 
0


0 

0

16


 18  .
19

more in location 2 than in location 1. There will be
19 th
18
more in location 3
than in location 2.
You see the eigenvalue problem makes these sorts of determinations fairly simple.
There are many other things which can be said about these sorts of migration problems.
They include things like the gambler’s ruin problem which asks for the probability that a compulsive
gambler will eventually lose all his money. However those problems are not so easy although they
still involve eigenvalues and eigenvectors.
12.2.3
Discrete Dynamical Systems
The migration matrices discussed above give an example of a discrete dynamical system. They are
discrete, not because they are somehow tactful and polite but because they involve discrete values
taken at a sequence of points rather than on a whole interval of time. An example of a situation
which can be studied in this way is a predator prey model. Consider the following model where x is
the number of prey and y the number of predators. These are functions of k ∈ N where 1, 2, · · · are
the ends of intervals of time which may be of interest in the problem.
(
) (
)(
)
x (n + 1)
A −B
x (n)
=
y (n + 1)
C D
y (n)
This says that x increases if there are more x and decreases as there are more y. As for y, it
increases if there are more y and also if there are more x.
Example 12.2.8 Suppose a dynamical system is of the form
(
) (
)(
)
x (n + 1)
1. 5 −0.5
x (n)
=
y (n + 1)
1.0
0
y (n)
Find solutions to the dynamical system for given initial conditions.
In this case, the eigenvalues of the matrix are 1, and .5. The matrix is of the form
(
)(
)(
)
1 1
1 0
2 −1
1 2
0 .5
−1 1
12.2. SOME APPLICATIONS OF EIGENVALUES AND EIGENVECTORS
and so given an initial condition
(
x0
y0
229
)
the solution to the dynamical system is
(
)
(
)(
)n (
)(
)
x (n)
1 1
1 0
2 −1
x0
=
y (n)
1 2
0 .5
−1 1
y0
(
)(
)(
)(
)
1 1
1
0
2 −1
x0
=
n
1 2
0 (.5)
−1 1
y0
(
)
n
n
y0 ((.5) − 1) − x0 ((.5) − 2)
=
n
n
y0 (2 (.5) − 1) − x0 (2 (.5) − 2)
In the limit as n → ∞, you get
Thus for large n,
Letting the initial condition be
(
(
x (n)
y (n)
2x0 − y0
2x0 − y0
)
(
≈
(
20
10
)
2x0 − y0
2x0 − y0
)
)
one can graph these solutions for various values of n. Here are the solutions for values of n between
1 and 5
(
)(
)(
)(
)(
)
25.0
27. 5
28. 75
29. 375
29. 688
20.0
25.0
27. 5
28. 75
29. 375
Another very different kind of behavior is also observed. It is possible for the ordered pairs to
spiral around the origin.
Example 12.2.9 Suppose a dynamical system is of the form
(
) (
)(
)
x (n + 1)
0.7 0.7
x (n)
=
y (n + 1)
−0.7 0.7
y (n)
Find solutions to the dynamical system for given initial conditions.
In this case, the eigenvalues are complex, .7 + .7i and .7 − .7i. Suppose the initial condition is
(
)
x0
y0
230
CHAPTER 12. SPECTRAL THEORY
what is a formula for the solutions to the dynamical system? Some computations show that the
eigen pairs are
(
)
( )
1
1
←→ .7 + .7i,
←→ .7 − .7i
i
−i
Thus the matrix is of the form
(
)(
)(
1 1
.7 + .7i
0
i −i
0
.7 − .7i
and so,
(
x (n)
y (n)
(
)
=
1 1
i −i
)(
(.7 + .7i)
0
n
0
n
(.7 − .7i)
1
2
1
2
− 12 i
1
2i
)(
1
2
1
2
)
− 12 i
1
2i
)(
x0
y0
)
The(explicit solution is given by
)
(
(
n
n)
n
n)
x0 12 ((0.7 − 0.7i)) + 12 ((0.7 + 0.7i)) + y0 12 i ((0.7 − 0.7i)) − 12 i ((0.7 + 0.7i))
(
(
n
n)
n
n)
y0 21 ((0.7 − 0.7i)) + 12 ((0.7 + 0.7i)) − x0 12 i ((0.7 − 0.7i)) − 12 i ((0.7 + 0.7i))
Suppose the initial condition is
(
) (
)
x0
10
=
y0
10
Then one obtains the following sequence of values which are graphed below by letting n = 1, 2, · · · , 20
In this picture, the dots are the values and the dashed line is to
help to picture what is happening.
These points are getting gradually closer to the origin, but they
are circling the origin in the clockwise direction as they do so. Also,
since both eigenvalues are slightly smaller than 1 in absolute value,
(
) ( )
x (n)
0
lim
=
n→∞
y (n)
0
This type of behavior along with complex eigenvalues is typical
of the deviations from an equilibrium point in the Lotka Volterra
system of differential equations which is a famous model for predator
prey interactions. These differential equations are given by
X′
Y′
= X (a − bY )
= −Y (c − dX)
where a, b, c, d are positive constants. For example, you might have X be the population of moose
and Y the population of wolves on an island.
Note how reasonable these equations are. The top says that the rate at which the moose population increases would be aX if there were no predators Y . However, this is modified by multiplying
instead by (a − bY ) because if there are predators, these will militate against the population of
moose. By definition, the wolves eat the moose and when the moose is eaten, it is not around
anymore to make new moose. The more predators there are, the more pronounced is this effect. As
to the predator equation, you can see that the equations predict that if there are many prey around,
then the rate of growth of the predators would seem to be high. However, this is modified by the
term −cY because if there are many predators, there would be competition for the available food
supply and this would tend to decrease Y ′ .
The behavior near an equilibrium point, which is a point where the right side of the differential
equations equals zero, is of great interest. In this case, the equilibrium point is
c
a
Y = ,X =
b
d
12.2. SOME APPLICATIONS OF EIGENVALUES AND EIGENVECTORS
231
Then one defines new variables according to the formula
x+
c
a
= X, Y = y +
d
b
In terms of these new variables, the differential equations become
(
(
a ))
c)(
x′ =
x+
a−b y+
( d a) (
( b c ))
′
y = − y+
c−d x+
b
d
Multiplying out the right sides yields
x′
y′
c
= −bxy − b y
d
a
= dxy + dx
b
The interest is for x, y small and so these equations are essentially equal to
c
a
x′ = −b y, y ′ = dx
d
b
Replace x′ with the difference quotient x(t+h)−x(t)
where h is a small positive number and y ′ with a
h
similar difference quotient. For example one could have h correspond to one day or even one hour.
Thus, for h small enough, the following would seem to be a good approximation to the differential
equations.
x (t + h) =
y (t + h) =
c
x (t) − hb y
d
a
y (t) + h dx
b
Let 1, 2, 3, · · · denote the ends of discrete intervals of time having length h chosen above. Then the
above equations take the form
(
) (
)(
)
−hbc
x (n + 1)
1
x
(n)
d
=
had
y (n + 1)
1
y (n)
b
Note that the eigenvalues of this matrix are always complex.
You are not interested in time intervals of length h for h very small. Instead, you are interested
in much longer lengths of time. Thus, replacing the time interval with mh,
(
) (
)m (
)
−hbc
x (n + m)
1
x (n)
d
=
had
1
y (n + m)
y (n)
b
For example, if m = 2, you would have
(
) (
x (n + 2)
1 − ach2
=
y (n + 2)
2 ab dh
−2b dc h
1 − ach2
)(
x (n)
y (n)
)
Note that the eigenvalues of the new matrix will likely still be complex. You can also notice that
the upper right corner will be negative by considering higher powers of the matrix. Thus letting
1, 2, 3, · · · denote the ends of discrete intervals of time, the desired discrete dynamical system is of
the form
(
) (
)(
)
x (n + 1)
A −B
x (n)
=
y (n + 1)
C D
y (n)
where A, B, C, D are positive constants and the matrix will likely have complex eigenvalues because
it is a power of a matrix which has complex eigenvalues.
232
CHAPTER 12. SPECTRAL THEORY
You can see from the above discussion that if the eigenvalues of the matrix used to define the
dynamical system are less than 1 in absolute value, then the origin is stable in the sense that as
n → ∞, the solution converges to the origin. If either eigenvalue is larger than 1 in absolute value,
then the solutions to the dynamical system will usually be unbounded, unless the initial condition
is chosen very carefully. The next example exhibits the case where one eigenvalue is larger than 1
and the other is smaller than 1.
Example 12.2.10 The Fibonacci sequence is the sequence which is defined recursively in the form
x (0) = 1 = x (1) , x (n + 2) = x (n + 1) + x (n)
This sequence is extremely important in the study of reproducing rabbits. It can be considered
as a dynamical system as follows. Let y (n) = x (n + 1) . Then the above recurrence relation can be
written as
(
) (
)(
) (
) ( )
x (n + 1)
0 1
x (n)
x (0)
1
=
,
=
y (n + 1)
1 1
y (n)
y (0)
1
The eigenvectors and eigenvalues of the matrix are
(
)
(
√
− 21 5 − 12
1 1√
↔ −
5,
2 2
1
1
2
√
5−
1
1
2
)
↔
1√
1
5+
2
2
You can see from a short computation that one of these is smaller than 1 in absolute value while
the other is larger than 1 in absolute value.
( √
)−1 (
)( √
)
√
√
1
1
1
1
− 12 5 − 12
− 12 5 − 12
0 1
2 5− 2
2 5− 2
1
1
1 1
1
1
( √
)
1
1
0
2 5+ 2
√
=
1
1
0
2 − 2 5
Then it follows that for the given initial condition the solution to this dynamical system is of the
form
(
)
( √
)( ( √
)
√
)n
1
1
x (n)
5 − 12 − 12 5 − 12
5 + 21
0
2
2
( 1 1 √ )n
=
·
y (n)
0
1
1
2 − 2 5
)(
(
)
√
√
1
1
1
5
5
+
1
5 √
√10( √ 2 )
− 15 5 15 5 12 5 − 12
1
It follows that
(
x (n) =
1√
1
5+
2
2
)n (
1√
1
5+
10
2
)
(
+
)n (
)
1 1√
1
1√
−
5
−
5
2 2
2 10
This might not be the first thing you would think of. Here is a picture of the ordered pairs
(x (n) , y (n)) for n = 0, 1, · · · , n. There is so much more that can be said about dynamical systems.
It is a major topic of study in differential equations and what is
given above is just an introduction.
12.3
The Estimation Of Eigenvalues
There are many other important applications of eigenvalue problems. We have just given a few such applications here. As pointed
out, this is a very hard problem but sometimes you don’t need to
find the eigenvalues exactly. There are ways to estimate the eigenvalues for matrices from just looking at the matrix. The most famous is known as Gerschgorin’s
theorem. This theorem gives a rough idea where the eigenvalues are just from looking at the
matrix.
12.3. THE ESTIMATION OF EIGENVALUES
233
Theorem 12.3.1 Let A be an n × n matrix. Consider the n Gerschgorin discs defined as




∑
|aij | .
Di ≡ λ ∈ C : |λ − aii | ≤


j̸=i
Then every eigenvalue is contained in some Gerschgorin disc.
This theorem says to add up the absolute values of the entries of the ith row which are off the
main diagonal and form the disc centered at aii having this radius. The union of these discs contains
σ (A) , the spectrum of A.
Theorem 12.3.2 Let A be an n × n matrix. Consider the n Gerschgorin discs defined as




∑
Di ≡ λ ∈ C : |λ − aii | ≤
|aij | .


j̸=i
Then every eigenvalue is contained in some Gerschgorin disc.
This theorem says to add up the absolute values of the entries of the ith row which are off the
main diagonal and form the disc centered at aii having this radius. The union of these discs contains
σ (A) .
Proof: Suppose Ax = λx where x ̸= 0. Then for A = (aij ) , let |xk | ≥ |xj | for all xj . Thus
|xk | ̸= 0.
∑
akj xj = (λ − akk ) xk .
j̸=k
Then
|xk |
∑
j̸=k
|akj | ≥
∑
|akj | |xj | ≥
j̸=k
∑
akj xj = |λ − aii | |xk | .
j̸=k
Now dividing by |xk |, it follows λ is contained in the k th Gerschgorin disc. Example 12.3.3 Suppose the matrix is

21

A =  14
7

−16 −6

60
12 
8
38
Estimate the eigenvalues.
The exact eigenvalues are 35, 56, and 28. The Gerschgorin disks are
D1 = {λ ∈ C : |λ − 21| ≤ 22} ,
D2 = {λ ∈ C : |λ − 60| ≤ 26} ,
and
D3 = {λ ∈ C : |λ − 38| ≤ 15} .
Gerschgorin’s theorem says these three disks contain the eigenvalues. Now 35 is in D3 , 56 is in D2
and 28 is in D1 .
More can be said when the Gerschgorin disks are disjoint but this is an advanced topic which
requires the theory of functions of a complex variable. If you are interested and have a background
in complex variable techniques, this is in [13]
234
CHAPTER 12. SPECTRAL THEORY
12.4
Exercises
1. State the eigenvalue problem from an algebraic perspective.
2. State the eigenvalue problem from a geometric perspective.
3. Consider the linear transformation which projects all vectors in R2 onto the span of the vector
(1, 2) . Show that the matrix of this linear transformation is
(
)
1/5 2/5
2/5 4/5
Now based on geometric considerations only, show that 1 is an eigenvalue and that an eigenT
vector is (1, 2) . Also explain why 0 will also be an eigenvalue.
4. If A is the matrix of a linear transformation which rotates all vectors in R2 through 30◦ , explain
why A cannot have any real eigenvalues.
5. If A is an n × n matrix and c is a nonzero constant, compare the eigenvalues of A and cA.
6. If A is an invertible n × n matrix, compare the eigenvalues of A and A−1 . More generally, for
m an arbitrary integer, compare the eigenvalues of A and Am .
7. Let A, B be invertible n × n matrices which commute. That is, AB = BA. Suppose x is an
eigenvector of B. Show that then Ax must also be an eigenvector for B.
8. Suppose A is an n × n matrix and it satisfies Am = A for some m a positive integer larger
than 1. Show that if λ is an eigenvalue of A then |λ| equals either 0 or 1.
9. Show that if Ax = λx and Ay = λy, then whenever a, b are scalars,
A (ax + by) = λ (ax + by) .
Does this imply that ax + by is an eigenvector? Explain.
10. Find the eigenvalues and eigenvectors of the matrix


−1 −1 7


 −1 0 4  .
−1 −1 5
Determine whether the matrix is defective.
11. Find the eigenvalues and eigenvectors of the

−3

 −2
−2
matrix

−7 19

−1 8  .
−3 10
Determine whether the matrix is defective.
12. Find the eigenvalues and eigenvectors of the matrix


−7 −12 30


 −3 −7 15  .
−3 −6 14
Determine whether the matrix is defective.
12.4. EXERCISES
13. Find the eigenvalues and eigenvectors of the matrix


7 −2 0


 8 −1 0  .
−2 4 6
Determine whether the matrix is defective.
14. Find the eigenvalues and eigenvectors of the matrix


3 −2 −1


1 .
 0 5
0 2
4
Determine whether the matrix is defective.
15. Find the eigenvalues and eigenvectors of the matrix


6 8 −23


 4 5 −16 
3 4 −12
Determine whether the matrix is defective.
16. Find the eigenvalues and eigenvectors of the matrix


5 2 −5


 12 3 −10  .
12 4 −11
Determine whether the matrix is defective.
17. Find the eigenvalues and eigenvectors of the matrix


20 9 −18


−6  .
 6 5
30 14 −27
Determine whether the matrix is defective.
18. Find the eigenvalues and eigenvectors of the matrix


1
26 −17


−4
4 .
 4
−9 −18
9
Determine whether the matrix is defective.
19. Find the eigenvalues and eigenvectors of the matrix


3 −1 −2


 11 3 −9  .
8
0 −6
Determine whether the matrix is defective.
235
236
CHAPTER 12. SPECTRAL THEORY
20. Find the eigenvalues and eigenvectors of the matrix


−2
1 2


 −11 −2 9  .
−8
0 7
Determine whether the matrix is defective.
21. Find the eigenvalues and eigenvectors of the matrix


2 1 −1


 2 3 −2  .
2 2 −1
Determine whether the matrix is defective.
22. Find the complex eigenvalues and eigenvectors of the matrix


4 −2 −2


 0 2 −2  .
2 0
2
23. Find the eigenvalues and eigenvectors of the matrix


9
6 −3


6
0 .
 0
−3 −6 9
Determine whether the matrix is defective.
24.
25.
26.
27.


4 −2 −2


Find the complex eigenvalues and eigenvectors of the matrix  0 2 −2  . Determine
2 0
2
whether the matrix is defective.


−4 2
0


Find the complex eigenvalues and eigenvectors of the matrix  2 −4 0  . Determine
−2 2 −2
whether the matrix is defective.


1
1 −6


Find the complex eigenvalues and eigenvectors of the matrix  7 −5 −6  . Determine
−1 7
2
whether the matrix is defective.


4 2 0


Find the complex eigenvalues and eigenvectors of the matrix  −2 4 0  . Determine
−2 2 6
whether the matrix is defective.
28. Let A be a real 3 × 3 matrix which has a complex eigenvalue of the form a + ib where b ̸= 0.
Could A be defective? Explain. Either give a proof or an example.
29. Let T be the linear transformation which reflects vectors about the x axis. Find a matrix for
T and then find its eigenvalues and eigenvectors.
12.4. EXERCISES
237
30. Let T be the linear transformation which rotates all vectors in R2 counterclockwise through
an angle of π/2. Find a matrix of T and then find eigenvalues and eigenvectors.
31. Let A be the 2 × 2 matrix of the linear transformation which rotates all vectors in R2 through
an angle of θ. For which values of θ does A have a real eigenvalue?
32. Let T be the linear transformation which reflects all vectors in R3 through the xy plane. Find
a matrix for T and then obtain its eigenvalues and eigenvectors.
33. Find the principal direction for stretching for the matrix
√
√ 
 13
2
8
9
15 5
45 5




6
4

 2 √5
.
 15
5
15





 8 √5
4
61
45
15
45
The eigenvalues are 2 and 1.
34. Find the principal directions for the matrix

5
2

 1
 −2

0
− 12
5
2
0
0



0 

1
35. Suppose the migration matrix for three locations is


.5 0 .3


 .3 .8 0  .
.2 .2 .7
Find a comparison for the populations in the three locations after a long time.
36. Suppose the migration matrix for three locations is


.1 .1 .3


 .3 .7 0  .
.6 .2 .7
Find a comparison for the populations in the three locations after a long time.
37. You own a trailer rental company in a large city and you have four locations, one in the South
East, one in the North East, one in the North West, and one in the South West. Denote these
locations by SE,NE,NW, and SW respectively. Suppose you observe that in a typical day, .8
of the trailers starting in SE stay in SE, .1 of the trailers in NE go to SE, .1 of the trailers
in NW end up in SE, .2 of the trailers in SW end up in SE, .1 of the trailers in SE end up
in NE,.7 of the trailers in NE end up in NE,.2 of the trailers in NW end up in NE,.1 of the
trailers in SW end up in NE, .1 of the trailers in SE end up in NW, .1 of the trailers in NE
end up in NW, .6 of the trailers in NW end up in NW, .2 of the trailers in SW end up in NW,
0 of the trailers in SE end up in SW, .1 of the trailers in NE end up in SW, .1 of the trailers in
NW end up in SW, .5 of the trailers in SW end up in SW. You begin with 20 trailers in each
location. Approximately how many will you have in each location after a long time? Will any
location ever run out of trailers?
238
CHAPTER 12. SPECTRAL THEORY
38. Let A be the n×n, n > 1, matrix of the linear transformation which comes from the projection
v7→projw (v). Show that A cannot be invertible. Also show that A has an eigenvalue equal to
1 and that for λ an eigenvalue, |λ| ≤ 1.
39. Let v be a unit vector in Rn and let A = I − 2vvT . Show that A has an eigenvalue equal to
−1.
40. Let M be an n × n matrix and suppose x1 , · · · , xn are n eigenvectors which form a linearly
independent set. Form the matrix S by making the columns these vectors. Show that S −1
exists and that S −1 M S is a diagonal matrix (one having zeros everywhere except on the
main diagonal) having the eigenvalues of M on the main diagonal. When this can be done the
matrix is diagonalizable. This is presented in the text. You should write it down in your
own words filling in the details without looking at the text.
41. Show that a matrix M is diagonalizable if and only if it has a basis of eigenvectors. Hint:
The first part is done in Problem 40. It only remains to show that if the matrix can be
diagonalized by some matrix S giving D = S −1 M S for D a diagonal matrix, then it has a
basis of eigenvectors. Try using the columns of the matrix S. Like the last problem, you should
try to do this yourself without consulting the text. These problems are a nice review of the
meaning of matrix multiplication.
42. Suppose A is an n × n matrix which is diagonally dominant. This means
∑
|aii | >
|aij | .
j̸=i
Show that A−1 must exist.
43. Is it possible for a nonzero matrix to have only 0 as an eigenvalue?
44. Let M be an n × n matrix. Then define the adjoint of M,denoted by M ∗ to be the transpose
of the conjugate of M. For example,
(
)∗ (
)
2
i
2 1−i
=
.
1+i 3
−i
3
A matrix M, is self adjoint if M ∗ = M. Show the eigenvalues of a self adjoint matrix are all
real. If the self adjoint matrix has all real entries, it is called symmetric.
45. Suppose A is an n×n matrix consisting entirely of real entries but a+ib is a complex eigenvalue
having the eigenvector x + iy. Here x and y are real vectors. Show that then a − ib is also an
eigenvalue with the eigenvector x − iy. Hint: You should remember that the conjugate of a
product of complex numbers equals the product of the conjugates. Here a + ib is a complex
number whose conjugate equals a − ib.
46. Recall an n × n matrix is said to be symmetric if it has all real entries and if A = AT . Show
the eigenvectors and eigenvalues of a real symmetric matrix are real.
47. Recall an n × n matrix is said to be skew symmetric if it has all real entries and if A = −AT .
Show that any nonzero eigenvalues must be of the form ib where i2 = −1. In words, the
eigenvalues are either 0 or pure imaginary.
48. A discreet dynamical system is of the form
x (k + 1) = Ax (k) , x (0) = x0
where A is an n × n matrix and x (k) is a vector in Rn . Show first that
x (k) = Ak x0
12.4. EXERCISES
239
for all k ≥ 1. If A is nondefective so that it has a basis of eigenvectors, {v1 , · · · , vn } where
Avj = λj vj
you can write the initial condition x0 in a unique way as a linear combination of these eigenvectors. Thus
n
∑
aj vj
x0 =
j=1
Now explain why
x (k) =
n
∑
aj Ak vj =
j=1
n
∑
aj λkj vj
j=1
which gives a formula for x (k) , the solution of the dynamical system.
49. Suppose A is an n × n matrix and let v be an eigenvector such that Av = λv. Also suppose
the characteristic polynomial of A is
det (λI − A) = λn + an−1 λn−1 + · · · + a1 λ + a0
Explain why
(
)
An + an−1 An−1 + · · · + a1 A + a0 I v = 0
If A is nondefective, give a very easy proof of the Cayley Hamilton theorem based on this.
Recall this theorem says A satisfies its characteristic equation,
An + an−1 An−1 + · · · + a1 A + a0 I = 0.
50. Suppose an n × n nondefective matrix A has only 1 and −1 as eigenvalues. Find A12 .
51. Suppose the characteristic polynomial of an n × n matrix A is 1 − λn . Find Amn where m is
an integer. Hint: Note first that A is nondefective. Why?
52. Sometimes sequences come in terms of a recursion formula. An example is the Fibonacci
sequence.
x0 = 1 = x1 , xn+1 = xn + xn−1
Show this can be considered as a discreet dynamical system as follows.
(
) (
)(
) (
) ( )
xn+1
1 1
xn
x1
1
=
,
=
xn
1 0
xn−1
x0
1
Now use the technique of Problem 48 to find a formula for xn . This was done in the chapter.
Next change the initial conditions to x0 = 0, x1 = 1 and find the solution.
53. Let A be an n × n matrix having characteristic polynomial
det (λI − A) = λn + an−1 λn−1 + · · · + a1 λ + a0
n
Show that a0 = (−1) det (A).
(
)35
3
1
2
54. Find
. Next find
− 12 0
(
lim
n→∞
(
A
55. Find e where A is the matrix
3
2
− 12
1
0
3
2
− 12
1
0
)n
)
in the above problem.
240
CHAPTER 12. SPECTRAL THEORY
(
56. Consider the dynamical system
x (n + 1)
y (n + 1)
(
values and eigenvectors are 0.8 + 0.8i ←→
)
(
)
x (n)
=
. Show the eigeny (n)
)
( )
−i
i
, 0.8 − 0.8i ←→
. Find a formula for
1
1
.8 .8
−.8 .8
)(
T
the solution to the dynamical system for given initial condition (x0 , y0 ) . Show that the magT
nitude of (x (n) , y (n)) must diverge provided the initial condition is not zero. Next graph
the vector field for
)(
) (
)
(
x
x
.8 .8
−
−.8 .8
y
y
Note that this vector field seems to indicate a conclusion different than what you just obtained.
Therefore, in this context of discreet dynamical systems the consideration of such a picture is
not all that reliable.
Chapter 13
Matrices And The Inner Product
13.1
Symmetric And Orthogonal Matrices
13.1.1
Orthogonal Matrices
Remember that to find the inverse of a matrix was often a long process. However, it was very easy
to take the transpose of a matrix. For some matrices, the transpose equals the inverse and when the
matrix has all real entries, and this is true, it is called an orthogonal matrix. Recall the following
definition given earlier.
Definition 13.1.1 A real n × n matrix U is called an Orthogonal matrix if U U T = U T U = I.
Example 13.1.2 Show the matrix

U =

√1
2
√1
2
√1
2
− √12

is orthogonal.

UU
T
=
√1
2
√1
2
√1
2
− √12


√1
2
√1
2
√1
2
− √12

=
(
1 0
0 1
)
.


1 0
0


Example 13.1.3 Let U =  0 0 −1  . Is U orthogonal?
0 −1 0
The answer is yes. This is because the columns form an orthonormal set of vectors as well as the
rows. As discussed above this is equivalent to U T U = I.

1 0

T
U U = 0 0
0 −1
T 
1 0
0
 
−1   0 0
0 −1
0
 
0
1 0
 
−1  =  0 1
0
0 0
When you say that U is orthogonal, you are saying that
∑
∑
T
Uij Ujk
=
Uij Ukj = δ ik .
j
j
241

0

0 
1
242
CHAPTER 13. MATRICES AND THE INNER PRODUCT
In words, the dot product of the ith row of U with the k th row gives 1 if i = k and 0 if i ̸= k. The
same is true of the columns because U T U = I also. Therefore,
∑
∑
T
Uij
Ujk =
Uji Ujk = δ ik
j
j
which says that the one column dotted with another column gives 1 if the two columns are the same
and 0 if the two columns are different.
More succinctly, this states that if u1 , · · · , un are the columns of U, an orthogonal matrix, then
{
1 if i = j
ui · uj = δ ij ≡
.
(13.1)
0 if i ̸= j
Definition 13.1.4 A set of vectors, {u1 , · · · , un } is said to be an orthonormal set if 13.1.
Theorem 13.1.5 If {u1 , · · · , um } is an orthonormal set of vectors then it is linearly independent.
Proof: Using the properties of the dot product,
0 · u = (0 + 0) · u = 0 · u + 0 · u
∑
and so, subtracting 0 · u from both sides yields 0 · u = 0. Now suppose j cj uj = 0. Then from
the properties of the dot product,


∑
∑
∑
ck =
cj δ jk =
cj (uj · uk ) = 
cj uj  · uk = 0 · uk = 0.
j
j
j
∑
Since k was arbitrary, this shows that each ck = 0 and this has shown that if j cj uj = 0, then
each cj = 0. This is what it means for the set of vectors to be linearly independent. 
 1
1
1



Example 13.1.6 Let U = 



√
2
√
6
√1
3
−1
√
2
√1
6
√1
3
0
3
√
−
√
6
3



 . Is U an orthogonal matrix?



The answer is yes. This is because the columns (rows) form an orthonormal set of vectors.
The importance of orthogonal matrices is that they change components of vectors relative to
different Cartesian coordinate systems. Geometrically, the orthogonal matrices are exactly those
which preserve all distances in the sense that if x ∈ Rn and U is orthogonal, then ||U x|| = ||x||
because
2
T
2
||U x|| = (U x) U x = xT U T U x = xT Ix = ||x|| .
Observation 13.1.7 Suppose U is an orthogonal matrix. Then det (U ) = ±1.
This is easy to see from the properties of determinants. Thus
( )
(
)
2
det (U ) = det U T det (U ) = det U T U = det (I) = 1.
Orthogonal matrices are divided into two classes, proper and improper. The proper orthogonal
matrices are those whose determinant equals 1 and the improper ones are those whose determinants
equal −1. The reason for the distinction is that the improper orthogonal matrices are sometimes
considered to have no physical significance since they cause a change in orientation which would
correspond to material passing through itself in a non physical manner. Thus in considering which
coordinate systems must be considered in certain applications, you only need to consider those
which are related by a proper orthogonal transformation. Geometrically, the linear transformations
determined by the proper orthogonal matrices correspond to the composition of rotations.
13.1. SYMMETRIC AND ORTHOGONAL MATRICES
13.1.2
243
Symmetric And Skew Symmetric Matrices
Definition 13.1.8 A real n × n matrix A, is symmetric if AT = A. If A = −AT , then A is called
skew symmetric.
Theorem 13.1.9 The eigenvalues of a real symmetric matrix are real. The eigenvalues of a real
skew symmetric matrix are 0 or pure imaginary.
Proof: The proof of this theorem is in [13]. It is best understood as a special case of more
general considerations. However, here is a proof in this special case.
Recall that for a complex number a + ib, the complex conjugate, denoted by a + ib is given by
the formula a + ib = a − ib. The notation, x will denote the vector which has every entry replaced
by its complex conjugate.
Suppose A is a real symmetric matrix and Ax = λx. Then
( )T
λxT x = Ax x = xT AT x = xT Ax = λxT x.
Dividing by xT x on both sides yields λ = λ which says λ is real. (Why?)
Next suppose A = −AT so A is skew symmetric and Ax = λx. Then
( )T
λxT x = Ax x = xT AT x = −xT Ax = −λxT x
and so, dividing by xT x as before, λ = −λ. Letting λ = a + ib, this means a − ib = −a − ib and so
a = 0. Thus λ is pure imaginary. (
)
0 −1
Example 13.1.10 Let A =
. This is a skew symmetric matrix. Find its eigenvalues.
1 0
(
Its eigenvalues are obtained by solving the equation det
−λ −1
1 −λ
)
= λ2 + 1 = 0. You see the
eigenvalues are ±i, pure imaginary.
(
)
1 2
Example 13.1.11 Let A =
. This is a symmetric matrix. Find its eigenvalues.
2 3
(
Its eigenvalues are obtained by solving the equation, det
√
√
and the solution is λ = 2 + 5 and λ = 2 − 5.
1−λ
2
2
3−λ
)
= −1 − 4λ + λ2 = 0
Definition 13.1.12 An n × n matrix A = (aij ) is called a diagonal matrix if aij = 0 whenever
i ̸= j. For example, a diagonal matrix is of the form indicated below where ∗ denotes a number.


∗
0


..


.


0
∗
Theorem 13.1.13 Let A be a real symmetric matrix. Then there exists an orthogonal matrix U
such that U T AU is a diagonal matrix. Moreover, the diagonal entries are the eigenvalues of A.
Proof: The proof is given later.
Corollary 13.1.14 If A is a real n × n symmetric matrix, then there exists an orthonormal set of
eigenvectors, {u1 , · · · , un } .
244
CHAPTER 13. MATRICES AND THE INNER PRODUCT
Proof: Since A is symmetric, then by Theorem 13.1.13, there exists an orthogonal matrix U
such that U T AU = D, a diagonal matrix whose diagonal entries are the eigenvalues of A. Therefore,
since A is symmetric and all the matrices are real,
D = DT = U T AT U = U T AT U = U T AU = D
showing D is real because each entry of D equals its complex conjugate.1
Finally, let
(
)
U = u1 u2 · · · un
where the ui denote the columns of U and


D=

λ1
0
..
.
0




λn
The equation, U T AU = D implies
(
AU
=
Au1
(
= UD =
···
Au2
λ1 u1
)
Aun
···
λ2 u2
)
λn un
where the entries denote the columns of AU and U D respectively. Therefore, Aui = λi ui and since
the matrix is orthogonal, the ij th entry of U T U equals δ ij and so
δ ij = uTi uj = ui · uj .
This proves the corollary because it shows the vectors {ui } form an orthonormal basis. Example 13.1.15 Find the eigenvalues and an orthonormal basis of eigenvectors for the matrix
√
√ 

8
2
19
− 15
5 45
5
9




1
16 
 − 8 √5
−
−
 15
5
15 




 2 √5

94
16
−
45
15
45
given that the eigenvalues are 3, −1, and 2.
The augmented matrix which needs to be row reduced to find
√
√
 19
8
2
− 15
5 45
5 | 0
9 −3


 − 8 √5 − 1 − 3 − 16
| 0
 15
5
15


 2 √5
94
− 16
45
15
45 − 3 | 0
and the row reduced echelon form for this is

1 0

 0 1

0
1 Recall
0
√
− 12 5
3
4
0
| 0
the eigenvectors for λ = 3 is










| 0 

| 0
that for a complex number, x + iy, the complex conjugate, denoted by x + iy is defined as x − iy.
13.1. SYMMETRIC AND ORTHOGONAL MATRICES
245
( √
)T
Therefore, eigenvectors for λ = 3 are z 12 5 − 34 1
where z ̸= 0.
The augmented matrix, which must be row reduced to find the eigenvectors for λ = −1, is
√
√
 19

8
2
− 15
5 45
5 | 0
9 +1




 − 8 √5 − 1 + 1 − 16
| 0 
 15

5
15




√
 2 5
16
94
−
+1 | 0 
45
15
and the row reduced echelon form is

45

√
− 12 5 | 0


 0 1
−3
| 0 
.

0 0
0
| 0
( √
)T
Therefore, the eigenvectors for λ = −1 are z 12 5 3 1
, z ̸= 0
The augmented matrix which must be row reduced to find the eigenvectors for λ = 2 is
√
√
 19

8
2
− 15
5 45
5 | 0
9 −2




 − 8 √5 − 1 − 2 − 16
| 0 
 15

5
15




√
 2 5
94
− 16
−2 | 0 
1 0
45
and its row reduced echelon form is
15
45

√
5 | 0


 0 1
0
| 0 


0 0
0
| 0
(
)T
√
so the eigenvectors for λ = 2 are z − 25 5 0 1
, z ̸= 0.
It remains to find an orthonormal basis. You can check that the dot product of any of these
vectors with another of them gives zero and so it suffices choose z in each case such that the
resulting vector has length 1. First consider the vectors for λ = 3. It is required to choose z such
( √
)T
that z 12 5 − 34 1
is a unit vector. In other words, you need


1
2
1 0
√
5

2
5

1
2
√
5


 

3 
3 

z
 − 4  · z  − 4  = 1.
1
1
But the above dot product equals
(
√
2
which is desired is
− 15 5
3
45 2
16 z
and this equals 1 when z =
√ )T
4
.
15 5
4
15
√
5. Therefore, the eigenvector
2
Next find the eigenvector for λ = −1. The same process requires that 1 = 45
4 z which happens
√
2
5. Therefore, an eigenvector for λ = −1 which has unit length is
when z = 15
 √   1 
1
3
2 5
√ 
 
2√ 

2

5 
5
.
 3 =
15
 5√ 
2
1
15 5
246
CHAPTER 13. MATRICES AND THE INNER PRODUCT
Finally, consider λ = 2. This time you need 1 = 95 z 2 which occurs when z =
eigenvector is
√  
− 32
− 25 5
 
1√ 
= 0
5
0

  √
3
1
1
3 5

1
3
√
5. Therefore, the


.

Now recall that the vectors form an orthonormal set of vectors if the matrix having them as
columns is orthogonal. That matrix is


2
1
− 23
3
3




√
 1√

2
0 .
 −5 5 5 5


 4√
√
√ 
2
1


15 5
15 5
3 5
Is this orthogonal? To find out, multiply by its transpose. Thus


√
√ 
2
1
4
2
− 15 5 15
5
− 23
3
3
3
 





1 0
√
√
√
√
 1
 1
 
2
2
2
5
5
−
5
5
0
= 0 1
 3


5
15
5
5




 4 √
√
√ 
√
2
1
0 0
 2



1
15 5
15 5
3 5
−3
0
3 5

0

0 .
1
Since the identity was obtained this shows the above matrix is orthogonal and that therefore, the
columns form an orthonormal set of vectors. The problem asks for you to find an orthonormal basis.
However, you will show in Problem 23 that an orthonormal set of n vectors in Rn is always a basis.
Therefore, since there are three of these vectors, they must constitute a basis.
Example 13.1.16 Find an orthonormal set of three eigenvectors for the matrix
√
√ 
 13
2
8
9
15 5
45 5




6
4
 2 √5

 15

5
15




√
 8 5

4
61
45
15
45
given the eigenvalues are 2, and 1.
The eigenvectors which go with λ = 2 are obtained from row
√
√
 13
2
8
| 0
9 −2
15 5
45 5


4
 2 √5 6 − 2
| 0
 15
5
15


 8 √5
4
61
−2 | 0
45
and its row reduced echelon form is

15
1
0

 0 1

0 0
45
√
− 12 5
− 34
0
| 0


| 0 

| 0
reducing the matrix








13.1. SYMMETRIC AND ORTHOGONAL MATRICES
247
( √
)T
which shows the eigenvectors for λ = 2 are z 12 5 34 1
and a choice for z which will produce
(
√
√
√ )T
4
2
1
4
a unit vector is z = 15
5. Therefore, the vector we want is
.
5
3
5
15 5
Next consider the eigenvectors for λ = 1. The matrix which must be row reduced is
√
√

 13
2
8
| 0
9 −1
15 5
45 5




4
 2 √5 6 − 1
| 0 
 15

5
15




√
 8 5
4
61
−1 | 0 
45
and its row reduced echelon form is

15
1


 0
0
3
10
√
5
0
0
45
2
5
√
5
0
0
| 0



| 0 .
| 0
(
)T
√
√
3
Therefore, the eigenvectors are of the form − 10
, y, z arbitrary. This is a
5y − 25 5z y z
two dimensional eigenspace.
Before going further, we want to point out that no matter how we choose y and z the resulting
vector will be orthogonal to the eigenvector for λ = 2. This is a special case of a general result
which states that eigenvectors for distinct eigenvalues of a symmetric matrix are orthogonal. This
is explained in Problem 15. For this case you need to show the following dot product equals zero.

 

√
√
2
3
5y − 25 5z
− 10
3
 √  

 1 5  

(13.2)
 5
·

y


√
4
z
15 5
This is left for you to do.
Continuing with the task of finding an orthonormal basis, Let y = 0 first. This results in
)T
(
√
√
and letting z = 31 5 you obtain a unit vector. Thus
eigenvectors of the form − 52 5z 0 z
the second vector will be
√ ( √ )  
− 52 5 13 5
− 23

 

= 0
0
  √

√
1
1
3 5
3 5



.

It remains to find the third vector in the orthonormal basis. This merely involves choosing y and
z in 13.2 in such a way that the resulting vector has dot product with the two given vectors equal
to zero. Thus you need
 


√
√
3
− 23
− 10
5y − 25 5z
√
√
 


 ·  0  = 1 5y + 3 5z = 0.

  √  5

y
5
1
3 5
z
The dot product with the eigenvector for λ = 2 is automatically equal to zero and so all that you
need is the above equation. This is satisfied when z = − 13 y. Therefore, the vector we want is of the
248
form
CHAPTER 13. MATRICES AND THE INNER PRODUCT

√
√ (
)   1√
3
− 6 5y
− 10
5y − 25 5 − 13 y

 

=
y

 
y
( 1 )
1
−
3y
−3y




and it√only remains to choose y in such a way that this vector has unit length. This occurs when
y = 52 5. Therefore, the vector we want is
√  
− 13
− 16 5

√
√


2
2
=
5
5
1

5

 
5
√
1
2
−3
− 15
5




.

The three eigenvectors which constitute an orthonormal basis are

 



2
− 31
− 23
3


 √


 2 √5  
 1 5 


 5
 ,  0  , and  5
.
√


√ 
√ 
1
2
4
5
− 15 5
3
15 5
To check the work and see if this is really an orthonormal set of vectors, make them the columns
of a matrix and see if the resulting matrix is orthogonal. The matrix is


2
− 23
− 13
3



√ 
 2√

1
0
 5 5
5 5 

.


√
√
√
 −2 5 1 5 4 5 
15
3
15
This matrix times its transpose equals


2
− 23
− 13
−1
3

 3


√ 
 2√
 2
1
0
5
 5 5

 −
5

 3



√
√
√
 − 2 5 1 5 4 5  2
15
3
15
3
√ 
2
5 − 15
5
 

1 0
√
 
1
0
5

3
= 0 1
√
√ 
1
4
0 0
5
5 
2
5
√
5

0

0 
1
15
and so this is indeed an orthonormal basis.
Because of the repeated eigenvalue, there would have been many other orthonormal bases which
could have been obtained. It was pretty arbitrary for to take y = 0 in the above argument. We
could just as easily have taken z = 0 or even y = z = 1. Any such change would have resulted in a
different orthonormal basis. Geometrically, what is happening is the eigenspace for λ = 1 was two
dimensional. It can be visualized as a plane in three dimensional space which passes through the
origin. There are infinitely many different pairs of perpendicular unit vectors in this plane.
13.1.3
Diagonalizing A Symmetric Matrix
Recall the following definition:
Definition 13.1.17 An n × n matrix A = (aij ) is called a diagonal matrix if aij = 0 whenever
13.1. SYMMETRIC AND ORTHOGONAL MATRICES
249
i ̸= j. For example, a diagonal matrix is of the form indicated below where ∗ denotes a number.


∗ 0 ··· 0

.. 
 0 ∗
. 


 .

..
 .

. 0 
 .
0 ···
0 ∗
Definition 13.1.18 An n × n matrix A is said to be non defective or diagonalizable if there
exists an invertible matrix S such that S −1 AS = D where D is a diagonal matrix as described above.
Some matrices are non defective and some are not. As indicated in Theorem 13.1.13 if A is a real
symmetric matrix, there exists an orthogonal matrix U such that U T AU = D a diagonal matrix.
Therefore, every symmetric matrix is non defective because if U is an orthogonal matrix, its inverse
is U T . In the following example, this orthogonal matrix will be found.


1 0 0
 0 3 1 

2
2  . Find an orthogonal matrix U such that U T AU is a
Example 13.1.19 Let A = 



1
3
0 2 2
diagonal matrix.
In this case, a tedious computation shows the eigenvalues are 2 and 1. First we will find an
eigenvector for the eigenvalue 2. This involves row reducing the following augmented matrix.


1
0
0
| 0
 0 2− 3
− 21
| 0 


2




0 − 12
2 − 32 | 0

The row reduced echelon form is
1

 0
0
(
and so an eigenvector is
)T
0
1
0
0
−1
0

| 0

| 0 
| 0
. However, it is desired that the eigenvectors obtained all be
(
√
√ )T
unit vectors and so dividing this vector by its length gives 0 12 2 12 2
. Next consider the
case of the eigenvalue, 1. The matrix which needs to be row reduced in this case is


0
0
0
| 0
 0 1− 3
− 21
| 0 


2




1
3
0 −2
1− 2 | 0
0 1
1

0 1 1 | 0


 0 0 0 | 0 .
0 0 0 | 0
(
)T
Therefore, the eigenvectors are of the form s −t t
. Two of these which are orthonormal
are




1
0
√ 



 0  and  −1/ 2  .
√
0
1/ 2
The row reduced echelon form is

250
CHAPTER 13. MATRICES AND THE INNER PRODUCT
An orthogonal matrix which works in the process is then obtained by letting these vectors be the
columns.


0
1
0
√
√ 

 −1/ 2 0 1/ 2  .
√
√
1/ 2 0 1/ 2
It remains to verify this works. U T AU is of the form
√
√ 


 

0 − 21 2 12 2
1 0 0
1
0
1
0
 0 3 1 

√
√

 

2
2 
0
0 
 1
  −1/ 2 0 1/ 2  =  0
√
√ 
√
√


1
1
0
0
0 12 32
1/ 2 0 1/ 2
2 2
2 2
0
1
0

0

0 ,
2
the desired diagonal matrix.
One of the applications for this technique has to do with rotation of axes so that with respect
to the new axes, the graph the level curve of a quadratic form is oriented parallel to the coordinate
axes. This makes it much easier to understand. This is discussed more in the exercises. However,
here is a simple example.
Example 13.1.20 Consider the following level curve.
5x2 − 6xy + 5y 2 = 8
Its graph is given in the following picture.
y
x
You can write this in terms of a symmetric matrix as follows.
(
)(
)
(
)
5 −3
x
=8
x y
−3 5
y
Change the variables as follows.
(
and so
(
)
x
Let
√
√
1
2 √2
1
2 2
y
1
2
2 √
1
−2 2
(
)T (
√
√
1
2 √2
1
2 2
2 0
0 8
1
2
2 √
1
−2 2
(
√
1
2 √2
1
2 2
)(
)T (
√
1
2
2 √
1
−2 2
√
√
1
2 √2
1
2 2
2 0
0 8
)(
1
2
2 √
1
−2 2
)(
x
y
√
1
2 √2
1
2 2
)
(
=
)
(
=
5 −3
−3 5
√
)(
1
2
2 √
1
−2 2
x̂
ŷ
x
y
)
)
)
Then in terms of these new variables, you get
2x̂2 + 8ŷ 2 = 8
This is an ellipse which is parallel to the coordinate axes. Its graph is of the form
=8
13.2. FUNDAMENTAL THEORY AND GENERALIZATIONS
251
ŷ
x̂
Thus this change of variables chooses new axes such that with respect to these new axes, the ellipse
is oriented parallel to the coordinate axes. These new axes are called the principal axes.
In general a quadratic form is an expression of the form
xT Ax
where A is a symmetric matrix. When you write something like
xT Ax = c
you are considering a level surface or level curve of some sort. By diagonalizing the matrix as shown
above, you can choose new variables such that in the new variables, there are no “mixed” terms like
xy or yz. Geometrically this has the effect of choosing new coordinate axes such that with respect
to these new axes, the various axes of symmetry of the level surfaces or curves are parallel to the
coordinate axes. Therefore, this is a desirable simplification. Other quadratic forms in two variables
lead to parabolas or hyperbolas. In three dimensions there are also names associated with these
quadratic surfaces usually involving the semi word “oid”. They are typically discussed in calculus
courses where they are invariably oriented parallel to the coordinate axes. However, the process of
diagonalization just explained will allow one to start with one which is not oriented this way and
reduce it to one which is.
13.2
Fundamental Theory And Generalizations
13.2.1
Block Multiplication Of Matrices
Consider the following problem
(
A
C
B
D
)(
E
G
F
H
)
You know how to do this. You get
(
AE + BG AF + BH
CE + DG CF + DH
)
.
Now what if instead of numbers, the entries, A, B, C, D, E, F, G are matrices of a size such that the
multiplications and additions needed in the above formula all make sense. Would the formula be
true in this case? I will show below that this is true.
Suppose A is a matrix of the form


A11 · · · A1m
 .
.. 
..

.
A=
(13.3)
.
. 
 .
Ar1 · · · Arm
where Aij is a si × pj matrix where si is constant for j = 1, · · · , m for each i = 1, · · · , r. Such a
matrix is called a block matrix, also a partitioned matrix. How do you get the block Aij ? Here
252
CHAPTER 13. MATRICES AND THE INNER PRODUCT
is how for A an m × n matrix:
si ×m
z(
}|
0 Isi ×si
z

n×pj
}| {
{
0
)


A
0
 Ipj ×pj .
0
(13.4)
In the block column matrix on the right, you need to have cj −1 rows of zeros above the small pj ×pj
identity matrix where the columns of A involved in Aij are cj , · · · , cj + pj − 1 and in the block row
matrix on the left, you need to have ri − 1 columns of zeros to the left of the si × si identity matrix
where the rows of A involved in Aij are ri , · · · , ri + si . An important observation to make is that
the matrix on the right specifies columns to use in the block and the one on the left specifies the
rows used. Thus the block Aij in this case is a matrix of size si × pj . There is no overlap between
the blocks of A. Thus the identity n × n identity matrix corresponding to multiplication on the right
of A is of the form


Ip1 ×p1
0


..


.


0
Ipm ×pm
these little identity matrices don’t overlap. A similar conclusion follows from consideration of the
matrices Isi ×si .
Next consider the question of multiplication of two block matrices. Let B be a block matrix of
the form


B11 · · · B1p
 .
.. 
..
 .

(13.5)
.
. 
 .
Br1 · · · Brp
and A is a block matrix of the form

A11
 .
 .
 .
Ap1
···
..
.
···

A1m
.. 

. 
Apm
(13.6)
and that for all i, j, it makes sense to multiply Bis Asj for all s ∈ {1, · · · , p}. (That is the two
matrices, Bis and Asj are conformable.)
∑ and that for fixed ij, it follows Bis Asj is the same size for
each s so that it makes sense to write s Bis Asj .
The following theorem says essentially that when you take the product of two matrices, you can
do it two ways. One way is to simply multiply them forming BA. The other way is to partition
both matrices, formally multiply the blocks to get another block matrix and this one will be BA
partitioned. Before presenting this theorem, here is a simple lemma which is really a special case of
the theorem.
Lemma 13.2.1 Consider the following product.


0
(


 I  0 I
0
)
0
where the first is n × r and the second is r × n. The small identity matrix I is an r × r matrix and
there are l zero rows above I and l zero columns to the left of I in the right matrix. Then the product
of these matrices is a block matrix of the form


0 0 0


 0 I 0 
0 0 0
13.2. FUNDAMENTAL THEORY AND GENERALIZATIONS
253
Proof: From the definition of the way you multiply matrices, the product is



 






 
0
0
0
0
0
0



 






 
 I  0 · · ·  I  0  I  e1 · · ·  I  er  I  0 · · ·  I  0
0
0
0
0
0
0
which yields the claimed result. In the formula ej refers to the column vector of length r which has
a 1 in the j th position. Theorem 13.2.2 Let B be a q × p block matrix as in 13.5 and let A be a p × n block matrix as in
13.6 such that Bis is conformable with Asj and each product, Bis Asj for s = 1, · · · , p is of the same
size so they can be added. Then BA can be obtained as a block matrix such that the ij th block is of
the form
∑
Bis Asj .
(13.7)
s
Proof: From 13.4
(
Bis Asj =
)
0
Iri ×ri
0


0


B  Ips ×ps 
0
(
)
0 Ips ×ps
0


0


A  Iqj ×qj 
0
where here it is assumed Bis is ri × ps and Asj is ps × qj . The product involves the sth block in the
ith row of blocks for B and the sth block in the j th column of A. Thus there are the same number of
rows above the Ips ×ps as there are columns to the left of Ips ×ps in those two inside matrices. Then
from Lemma 13.2.1




0
0
0
0
(
)




 Ips ×ps  0 Ips ×ps 0 =  0 Ips ×ps 0 
0
0
0
0
Since the blocks of small identity matrices do not overlap,

 
Ip1 ×p1
0
0
0
∑
 
..
 0 Ips ×ps 0  = 
.

s
0
0
0
0
and so
∑
s

0

=I

Ipp ×pp
Bis Asj =
∑(
)
0 Iri ×ri
0
s
(
=
0



B  Ips ×ps 
0
(
)
0 Ips ×ps
0

0



A  Iqj ×qj 
0




0
0
(
)
∑



0 B
 Ips ×ps  0 Ips ×ps 0 A  Iqj ×qj 
s
0
0




0
0
)
(
)




0 BIA  Iqj ×qj  = 0 Iri ×ri 0 BA  Iqj ×qj 
0
0
)
0 Iri ×ri
(
=

0 Iri ×ri
which equals the ij th block of BA.∑Hence the ij th block of BA equals the formal multiplication
according to matrix multiplication, s Bis Asj . 254
CHAPTER 13. MATRICES AND THE INNER PRODUCT
Example 13.2.3 Let an n × n matrix have the form
)
(
a b
A=
c P
where P is n − 1 × n − 1. Multiply it by
(
B=
p q
r Q
)
where B is also an n × n matrix and Q is n − 1 × n − 1.
You use block multiplication
(
)(
a b
p
c P
r
q
Q
)
(
=
ap + br
pc + P r
aq + bQ
cq + P Q
)
Note that this all makes sense. For example, b = 1 × n − 1 and r = n − 1 × 1 so br is a 1 × 1. Similar
considerations apply to the other blocks.
Here is an interesting and significant application of block multiplication. In this theorem, pM (t)
denotes the characteristic polynomial, det (tI − M ) . Thus the zeros of this polynomial are the eigenvalues of the matrix M .
Theorem 13.2.4 Let A be an m × n matrix and let B be an n × m matrix for m ≤ n. Then
pBA (t) = tn−m pAB (t) ,
so the eigenvalues of BA and AB are the same including multiplicities except that BA has n − m
extra zero eigenvalues.
Proof: Use block multiplication to write
(
)(
AB 0
I
B 0
0
(
Therefore,
(
I
0
I
0
A
I
A
I
)(
)−1 (
0
B
AB
B
A
I
0
BA
0
0
)
(
=
)
)(
(
=
I
0
A
I
AB
B
ABA
BA
AB
B
ABA
BA
)
(
=
(
0
B
)
)
.
0
BA
)
)
(
)
0
0
AB 0
Since the two matrices above are similar it follows that
and
have the
B BA
B 0
same characteristic polynomials. Therefore, noting that BA is an n × n matrix and AB is an m × m
matrix,
tm det (tI − BA) = tn det (tI − AB)
and so det (tI − BA) = pBA (t) = tn−m det (tI − AB) = tn−m pAB (t). 13.2. FUNDAMENTAL THEORY AND GENERALIZATIONS
13.2.2
255
Orthonormal Bases, Gram Schmidt Process
Not all bases for Fn are created equal. Recall F equals either C or R and the dot product is given
by
∑
x · y ≡ (x, y) ≡ ⟨x, y⟩ =
xj yj .
j
The best bases are orthonormal. Much of what follows will be for Fn in the interest of generality.
Definition 13.2.5 Suppose {v1 , · · · , vk } is a set of vectors in Fn . It is an orthonormal set if
{
1 if i = j
vi · vj = δ ij =
0 if i ̸= j
Every orthonormal set of vectors is automatically linearly independent.
Proposition 13.2.6 Suppose {v1 , · · · , vk } is an orthonormal set of vectors. Then it is linearly
independent.
∑k
Proof: Suppose i=1 ci vi = 0. Then taking dot products with vj ,
∑
∑
0 = 0 · vj =
ci vi · vj =
ci δ ij = cj .
i
i
Since j is arbitrary, this shows the set is linearly independent as claimed. It turns out that if X is any subspace of Fm , then there exists an orthonormal basis for X. This
follows from the use of the next lemma applied to a basis for X.
Lemma 13.2.7 Let {x1 , · · · , xn } be a linearly independent subset of Fp , p ≥ n. Then there exist
orthonormal vectors {u1 , · · · , un } which have the property that for each k ≤ n, span (x1 , · · · , xk ) =
span (u1 , · · · , uk ) .
Proof: Let u1 ≡ x1 / |x1 | . Thus for k = 1, span (u1 ) = span (x1 ) and {u1 } is an orthonormal
set. Now suppose for some k < n, u1 , · · · , uk have been chosen such that (uj , ul ) = δ jl and
span (x1 , · · · , xk ) = span (u1 , · · · , uk ). Then define
∑k
xk+1 − j=1 (xk+1 · uj ) uj
uk+1 ≡
,
(13.8)
∑k
xk+1 − j=1 (xk+1 · uj ) uj
where the denominator is not equal to zero because the xj form a basis, and so
xk+1 ∈
/ span (x1 , · · · , xk ) = span (u1 , · · · , uk )
Thus by induction,
uk+1 ∈ span (u1 , · · · , uk , xk+1 ) = span (x1 , · · · , xk , xk+1 ) .
Also, xk+1 ∈ span (u1 , · · · , uk , uk+1 ) which is seen easily by solving 13.8 for xk+1 and it follows
span (x1 , · · · , xk , xk+1 ) = span (u1 , · · · , uk , uk+1 ) .
If l ≤ k,

(uk+1 · ul ) = C (xk+1 · ul ) −

C (xk+1 · ul ) −
k
∑
j=1

k
∑

(xk+1 · uj ) (uj · ul ) =
j=1
(xk+1 · uj ) δ lj  = C ((xk+1 · ul ) − (xk+1 · ul )) = 0.
256
CHAPTER 13. MATRICES AND THE INNER PRODUCT
n
The vectors, {uj }j=1 , generated in this way are therefore orthonormal because each vector has unit
length. The process by which these vectors were generated is called the Gram Schmidt process. Note
that from the construction, each xk is in the span of {u1 , · · · , uk }. In terms of matrices, this says
(x1 · · · xn ) = (u1 · · · un ) R
where R is an upper triangular matrix. This is closely related to the QR factorization discussed
earlier. It is called the thin QR factorization. If the Gram Schmidt process is used to enlarge
{u1 · · · un } to an orthonormal basis for Fm , {u1 · · · un , un+1 , · · · , um } then if Q is the matrix which
has these vectors as columns and if R is also enlarged to R′ by adding in rows of zeros, if necessary,
to form an m × n matrix, then the above would be of the form
(x1 · · · xn ) = (u1 · · · um ) R′
and you could read off the orthonormal basis for span (x1 · · · xn ) by simply taking the first n columns
of Q = (u1 · · · um ). This is convenient because computer algebra systems are set up to find QR
factorizations.
  

2
1

  
Example 13.2.8 Find an orthonormal basis for span  3  ,  0 .
1
1
This is really

1

 3
1
easy to do using a computer algebra system.
 
 √
√
√ √
√
1
19
3
2
11 46
46
11
11 √11
506 √
46
√
√
  3

9
1
=
0   11 11 − 506 11 46
46   0
46 √
√
√ √
1
4
3
1
11
11
46
−
0
11
253
23 46
1
11

√
11
√

11 46 
0
3
11
√
and so the desired orthonormal basis is

 

√
√ √
1
19
11 46
11 √11
506 √
√
 3
 

9
11 46 
 11 11  ,  − 506
√
√
√
1
4
11 11
253 11 46
I
13.2.3
Schur’s Theorem
Every matrix is related to an upper triangular matrix in a particularly significant way. This is Schur’s
theorem and it is the most important theorem in the spectral theory of matrices. The important
result which makes this theorem possible is the Gram Schmidt procedure of Lemma 13.2.7.
Definition 13.2.9 An n × n matrix U, is unitary if U U ∗ = I = U ∗ U where U ∗ is defined to be the
∗
transpose of the conjugate of U. Thus Uij = Uji
. Note that every real orthogonal matrix is unitary.
∗
For A any matrix A , just defined as the conjugate of the transpose, is called the adjoint.
(
)
Note that if U = v1 · · · vn where the vk are orthonormal vectors in Cn , then U is unitary.
This follows because the ij th entry of U ∗ U is viT vj = δ ij since the vi are assumed orthonormal.
∗
Lemma 13.2.10 The following holds. (AB) = B ∗ A∗ .
Proof: From the definition and remembering the properties of complex conjugation,
∑
∑
(
∗)
Aik Bkj =
Aik Bkj
(AB) ji = (AB)ij =
=
∑
k
k
∗
Bjk
A∗ki
k
∗
∗
= (B A )ji 13.2. FUNDAMENTAL THEORY AND GENERALIZATIONS
257
Theorem 13.2.11 Let A be an n × n matrix. Then there exists a unitary matrix U such that
U ∗ AU = T,
(13.9)
where T is an upper triangular matrix having the eigenvalues of A on the main diagonal listed
according to multiplicity as roots of the characteristic equation. If A is a real matrix having all real
eigenvalues, then U can be chosen to be an orthogonal real matrix.
Proof: The theorem is clearly true if A is a 1 × 1 matrix. Just let U = 1, the 1 × 1 matrix which
has entry 1. Suppose it is true for (n − 1) × (n − 1) matrices, n ≥ 2 and let A be an n × n matrix.
Then let v1 be a unit eigenvector for A. Then there exists λ1 such that
Av1 = λ1 v1 , |v1 | = 1.
Extend {v1 } to a basis and then use the Gram - Schmidt process to obtain {v1 , · · · , vn }, an orthonormal basis of Cn . Let U0 be a matrix whose ith column is vi so that U0 is unitary. Consider
U0∗ AU0




v1∗
v1∗
) 
)
 . (
(
 .. 
. 
U0∗ AU0 = 
 .  Av1 · · · Avn =  .  λ1 v1 · · · Avn
vn∗
vn∗
Thus U0∗ AU0 is of the form
(
λ1
0
)
a
A1
where A1 is an n − 1 × n − 1 matrix. Now by induction, there exists an (n − 1) × (n − 1) unitary
e1 = Tn−1 , an upper triangular matrix. Consider
e1 such that U
e ∗ A1 U
matrix U
1
(
)
1 0
U1 ≡
.
e1
0 U
(
Then
U1∗ U1
=
1 0
e∗
0 U
1
Also
(
U1∗ U0∗ AU0 U1
=
(
=
)(
1 0
e1
0 U
1 0
e∗
0 U
1
λ1
0
)(
∗
Tn−1
)
)
(
=
λ1
0
∗
A1
1
0
0 In−1
)(
)
1 0
e1
0 U
)
≡T
where T is upper triangular. Then let U = U0 U1 . It is clear that this is unitary because both
matrices preserve distance. Therefore, so does the product and hence U . Alternatively,
I = U0 U1 U1∗ U0∗ = (U0 U1 ) (U0 U1 )
∗
and so, it follows that A is similar to T and that U0 U1 is unitary. Hence A and T have the same
characteristic polynomials, and since the eigenvalues of T (A) are the diagonal entries listed with
multiplicity, this proves the main conclusion of the theorem. In case A is real with all real eigenvalues,
the above argument can be repeated word for word using only the real dot product to show that U
can be taken to be real and orthogonal. As a simple consequence of the above theorem, here is an interesting lemma.
258
CHAPTER 13. MATRICES AND THE INNER PRODUCT
Lemma 13.2.12 Let A be of the form


∗
.. 

. 
Ps
···
..
.
P1
 .
.
A=
 .
0
···
where Pk is an mk × mk matrix. Then
det (A) =
∏
det (Pk ) .
k
Proof: Let Uk be an mk × mk unitary matrix such that
Uk∗ Pk Uk = Tk
where Tk is upper triangular. Then letting U denote the
the blocks on the diagonal,



U1 · · · 0
 .

.. 
..
∗


.
U =
.
. , U = 
 .
0 · · · Us
and

U1∗
 .
 .
 .
0
···
..
.
···

0
P1

.. 
  ..
.  .
Us∗
0

∗
U1

.. 
  ..
.  .
Ps
0
···
..
.
···
and so
det (A) =
∏
det (Tk ) =
U1∗
..
.
0
···
..
.
···
∏
k
block diagonal matrix, having the Ui as
···
..
.
···

0
.. 

. 
Us∗
 
0
T1

.. 
 =  ..
.   .
Us
0
···
..
.
···

∗
.. 

. 
Ts
det (Pk ) . k
Definition 13.2.13 An n × n matrix A is called Hermitian if A = A∗ . Thus a real symmetric
(A = AT ) matrix is Hermitian.
Recall that from Theorem 13.2.14, the eigenvalues of a real symmetric matrix are all real.
Theorem 13.2.14 If A is an n × n Hermitian matrix, there exists a unitary matrix U such that
U ∗ AU = D
(13.10)
where D is a real diagonal matrix. That is, D has nonzero entries only on the main diagonal and
these are real. Furthermore, the columns of U are an orthonormal basis of eigenvectors for Cn . If
A is real and symmetric, then U can be assumed to be a real orthogonal matrix and the columns of
U form an orthonormal basis for Rn .
Proof: From Schur’s theorem above, there exists U unitary (real and orthogonal if A is real)
such that
U ∗ AU = T
where T is an upper triangular matrix. Then from Lemma 13.2.10
∗
T ∗ = (U ∗ AU ) = U ∗ A∗ U = U ∗ AU = T.
Thus T = T ∗ and T is upper triangular. This can only happen if T is really a diagonal matrix
having real entries on the main diagonal. (If i ̸= j, one of Tij or Tji equals zero. But Tij = Tji and
so they are both zero. Also Tii = Tii .)
13.3. LEAST SQUARE APPROXIMATION
Finally, let
259
(
U=
u1
where the ui denote the columns of U and


D=

)
···
u2
λ1
un
0
..
.
0




λn
The equation, U ∗ AU = D implies
(
AU
=
Au1
=
UD =
(
Au2
λ1 u1
···
)
Aun
λ2 u2
···
)
λn un
where the entries denote the columns of AU and U D respectively. Therefore, Aui = λi ui and since
the matrix is unitary, the ij th entry of U ∗ U equals δ ij and so
δ ij = uTi uj = uTi uj = ui · uj .
This proves the corollary because it shows the vectors {ui } form an orthonormal basis. In case A is
real and symmetric, simply ignore all complex conjugations in the above argument. 13.3
Least Square Approximation
A very important technique is that of the least square approximation.
Lemma 13.3.1 Let A be an m × n matrix and let A (Fn ) denote the set of vectors in Fm which are
of the form Ax for some x ∈ Fn . Then A (Fn ) is a subspace of Fm .
Proof: Let Ax and Ay be two points of A (Fn ) . It suffices to verify that if a, b are scalars, then
aAx + bAy is also in A (Fn ) . But aAx + bAy = A (ax + by) because A is linear. Lemma 13.3.2 Suppose b ≥ 0 and c ∈ R such that a + bt2 + ct ≥ a for all t, then c = 0.
Proof: You need bt2 + ct ≥ 0 for all t. Thus
(
( c )2 ( c )2
c )2
c
t+
= t2 + t +
≥
2b
b
2b
2b
c
which is impossible unless c = 0. Otherwise, you could take t = − 2b
. The following theorem gives the equivalence of an orthogonality condition with a minimization
condition. The following picture illustrates the geometric meaning of this theorem
y
Au AFn
Ax
Theorem 13.3.3 Let y ∈ Fm and let A be an m × n matrix. Then there exists x ∈ Fn minimizing
2
the function x7→ |y−Ax| . Furthermore, x minimizes this function if and only if
((y−Ax) , Au) = 0
for all u ∈ Fn .
260
CHAPTER 13. MATRICES AND THE INNER PRODUCT
Proof: First consider the characterization of the minimizer. Let u ∈ Fn . Let |θ| = 1,
θ̄ (y − Ax, Au) = |(y − Ax, Au)|
Now consider the function of t ∈ R
2
p (t) ≡ |y− (Ax+tθAu)| = ((y − Ax) − tθAu, (y − Ax) − tθAu)
2
2
= |y − Ax| + t2 |Au| − 2t Re (y − Ax, θAu) ≥ |y − Ax|
2
2
2
= |y − Ax| + t2 |Au| − 2t Re θ̄ (y − Ax, Au)
2
2
= |y − Ax| + t2 |Au| − 2t |(y − Ax, Au)|
Then if |y − Ax| is as small as possible, this will occur when t = 0 and so p′ (0) = 0. But this says
|(y − Ax, Au)| = 0
You could also use Lemma 13.3.2 to see this is 0. Since u was arbitrary, this proves one direction.
Conversely, if this quantity equals 0,
2
|y− (Ax+Au)|
2
2
2
2
= |y − Ax| + |Ax − Au| + 2 Re (y − Ax,Au)
= |y − Ax| + |Ax − Au|
and so the minimum occurs at any point z such that Ax = Az.
Does there exist an x which minimizes this function? From what was just shown, it suffices to
show that there exists x such that ((y−Ax) , Au) for all u. By the Gramm Schmidt process there
exists an orthonormal basis {Axk } for AFn . Then for a given y,
(
y−
r
∑
)
= (y,Axj ) −
(y, Axk ) Axk , Axj
k=1
In particular,
r
∑
δ kj
}|
{
z
(y, Axk ) (Axk , Axj ) = 0.
k=1
(
(
y−A
r
∑
)
(y,Axk ) xk
)
,w
=0
k=1
for all w ∈ AFn since {Axk } is a basis. Therefore,
x=
r
∑
(y,Axk ) xk
k=1
is a minimizer. So is any z such that Az = Ax. Recall the definition of the adjoint of a matrix.
Definition 13.3.4 Let A be an m × n matrix. Then
A∗ ≡ (AT ).
This means you take the transpose of A and then replace each entry by its conjugate. This matrix
is called the adjoint. Thus in the case of real matrices having only real entries, the adjoint is just
the transpose.
Lemma 13.3.5 Let A be an m × n matrix. Then
Ax · y = x·A∗ y
13.3. LEAST SQUARE APPROXIMATION
261
Proof: This follows from the definition.
∑
∑
xj A∗ji yi = x·A∗ y. Ax · y =
Aij xj yi =
i,j
i,j
The next corollary gives the technique of least squares.
Corollary 13.3.6 A value of x which solves the problem of Theorem 13.3.3 is obtained by solving
the equation
A∗ Ax = A∗ y
and furthermore, there exists a solution to this system of equations.
Proof: For x the minimizer of Theorem 13.3.3, (y−Ax) · Aw = 0 for all w ∈ Fn and from
Lemma 13.3.5, this is the same as saying
A∗ (y−Ax) · w = 0
for all w ∈ Fn . This implies
A∗ y − A∗ Ax = 0.
Therefore, there is a solution to the equation of this corollary, and it solves the minimization problem
of Theorem 13.3.3. Note that x might not be unique but Ax, the closest point of AFn to y is unique. This was
shown in the above argument. Sometimes people like to consider the x such that Ax is as close as
possible to y and also |x| is as small as possible. It turns out that there exists a unique such x and
it is denoted as A+ y. However, this is as far as I will go with this in this part of the book.
There is also a useful observation about orthonormal sets of vectors which is stated in the next
lemma.
Lemma 13.3.7 Suppose {x1 , x2 , · · · , xr } is an orthonormal set of vectors. Then if c1 , · · · , cr are
scalars,
2
r
r
∑
∑
2
ck xk =
|ck | .
k=1
k=1
Proof: This follows from the definition. From the properties of the dot product and using the
fact that the given set of vectors is orthonormal,


2
r
r
r
r
∑
∑
∑
∑
∑
2
|ck | . ck xk = 
ck xk ,
cj xj  =
ck cj (xk , xj ) =
k=1
13.3.1
k=1
j=1
k,j
k=1
The Least Squares Regression Line
For the situation of the least squares regression line discussed here I will specialize to the case of Rn
rather than Fn because it seems this case is by far the most interesting and the extra details are not
justified by an increase in utility. Thus, everywhere you see A∗ it suffices to place AT .
An important application of Corollary 13.3.6 is the problem of finding the least squares regression
line in statistics. Suppose you are given points in xy plane
n
{(xi , yi )}i=1
and you would like to find constants m and b such that the line y = mx + b goes through all these
points. Of course this will be impossible in general. Therefore, try to find m, b to get as close as
possible. The desired system is

 

(
)
(
)
y1
x1 1
 .   .
m
.. 
 . = .
 m
≡A
. 
 .   .
b
b
yn
xn 1
262
CHAPTER 13. MATRICES AND THE INNER PRODUCT
which is of the form y = Ax and it is desired to choose m and b to make
(
A
m
b
)

y1
 . 
. 
−
 . 
yn
as small as possible. According to Theorem 13.3.3
b occur as the solution to

(
)
y1

m
T
T  ..
A A
=A  .
b
yn

2
and Corollary 13.3.6, the best values for m and


,


x1
 .

A =  ..
xn

1
.. 

. .
1
Thus, computing AT A,
( ∑
n
x2i
∑i=1
n
i=1 xi
∑n
i=1
xi
n
)(
m
b
)
( ∑
)
n
i=1 xi yi
=
∑n
i=1 yi
Solving this system of equations for m and b,
∑n
∑n
∑n
− ( i=1 xi ) ( i=1 yi ) + ( i=1 xi yi ) n
m=
∑n
∑n
2
( i=1 x2i ) n − ( i=1 xi )
and
∑n
∑n
∑n
∑n
− ( i=1 xi ) i=1 xi yi + ( i=1 yi ) i=1 x2i
.
b=
∑n
∑n
2
( i=1 x2i ) n − ( i=1 xi )
One could clearly do a least squares fit for curves of the form y = ax2 + bx + c in the same way.
In this case you want to solve as well as possible for a, b, and c the system



 
x21 x1 1
y1
a
 .

 
..
.. 
 .

 . 
.
.   b  =  .. 
 .
c
x2n xn 1
yn
and one would use the same technique as above. Many other similar problems are important,
including many in higher dimensions and they are all solved the same way.
13.3.2
The Fredholm Alternative
The next major result is called the Fredholm alternative. It comes from Theorem 13.3.3 and Lemma
13.3.5.
Theorem 13.3.8 Let A be an m × n matrix. Then there exists x ∈ Fn such that Ax = y if and
only if whenever A∗ z = 0 it follows that z · y = 0.
Proof: First suppose that for some x ∈ Fn , Ax = y. Then letting A∗ z = 0 and using Lemma
13.3.5
y · z = Ax · z = x · A∗ z = x · 0 = 0.
This proves half the theorem.
To do the other half, suppose that whenever, A∗ z = 0 it follows that z · y = 0. It is necessary
to show there exists x ∈ Fn such that y = Ax. From Theorem 13.3.3 there exists x minimizing
2
|y − Ax| which therefore satisfies
(y − Ax) · Aw = 0
(13.11)
13.4. THE RIGHT POLAR FACTORIZATION∗
263
for all w ∈ Fn . Therefore, for all w ∈ Fn ,
A∗ (y − Ax) · w = 0
which shows that A∗ (y − Ax) = 0. (Why?) Therefore, by assumption,
(y − Ax) · y = 0.
Now by 13.11 with w = x,
(y − Ax) · (y−Ax) = (y − Ax) · y− (y − Ax) · Ax = 0
showing that y = Ax. The following corollary is also called the Fredholm alternative.
Corollary 13.3.9 Let A be an m × n matrix. Then A is onto if and only if A∗ is one to one.
Proof: Suppose first A is onto. Then by Theorem 13.3.8, it follows that for all y ∈ Fm , y · z = 0
whenever A∗ z = 0. Therefore, let y = z where A∗ z = 0 and conclude that z · z = 0 whenever A∗ z =
0. If A∗ x = A∗ y, then A∗ (x − y) = 0 and so x − y = 0. Thus A∗ is one to one.
Now let y ∈ Fm be given. y · z = 0 whenever A∗ z = 0 because, since A∗ is assumed to be one
to one, and 0 is a solution to this equation, it must be the only solution. Therefore, by Theorem
13.3.8 there exists x such that Ax = y therefore, A is onto. 13.4
The Right Polar Factorization∗
The right polar factorization involves writing a matrix as a product of two other matrices, one which
preserves distances and the other which stretches and distorts. First here are some lemmas which
review and add to many of the topics discussed so far about adjoints and orthonormal sets and
such things. This is of fundamental significance in geometric measure theory and also in continuum
mechanics. Not surprisingly the stress should depend on the part which stretches and distorts. See
[8].
Lemma 13.4.1 Let A be a Hermitian matrix such that all its eigenvalues are nonnegative. Then
(
)2
there exists a Hermitian matrix A1/2 such that A1/2 has all nonnegative eigenvalues and A1/2 =
A.
Proof: Since A is Hermitian, there exists a diagonal matrix D having all real nonnegative entries
and a unitary matrix U such that A = U ∗ DU. Then denote by D1/2 the matrix which is obtained
by replacing each diagonal entry of D with its square root. Thus D1/2 D1/2 = D. Then define
A1/2 ≡ U ∗ D1/2 U.
Then
Since D1/2 is real,
(
(
A1/2
)2
= U ∗ D1/2 U U ∗ D1/2 U = U ∗ DU = A.
U ∗ D1/2 U
)∗
)∗
(
∗
= U ∗ D1/2 (U ∗ ) = U ∗ D1/2 U
so A1/2 is Hermitian. Next it is helpful to recall the Gram Schmidt algorithm and observe a certain property stated in
the next lemma.
Lemma 13.4.2 Suppose {w1 , · · · , wr , vr+1 , · · · , vp } is a linearly independent set of vectors such
that {w1 , · · · , wr } is an orthonormal set of vectors. Then when the Gram Schmidt process is applied
to the vectors in the given order, it will not change any of the w1 , · · · , wr .
264
CHAPTER 13. MATRICES AND THE INNER PRODUCT
Proof: Let {u1 , · · · , up } be the orthonormal set delivered by the Gram Schmidt process. Then
u1 = w1 because by definition, u1 ≡ w1 / |w1 | = w1 . Now suppose uj = wj for all j ≤ k ≤ r. Then
if k < r, consider the definition of uk+1 .
uk+1 ≡
wk+1 −
wk+1 −
∑k+1
j=1
(wk+1 , uj ) uj
j=1
(wk+1 , uj ) uj
∑k+1
By induction, uj = wj and so this reduces to wk+1 / |wk+1 | = wk+1 . This lemma immediately implies the following lemma.
Lemma 13.4.3 Let V be a subspace of dimension p and let {w1 , · · · , wr } be an orthonormal set of
vectors in V . Then this orthonormal set of vectors may be extended to an orthonormal basis for V,
{w1 , · · · , wr , yr+1 , · · · , yp }
Proof: First extend the given linearly independent set {w1 , · · · , wr } to a basis for V and then
apply the Gram Schmidt theorem to the resulting basis. Since {w1 , · · · , wr } is orthonormal it follows
from Lemma 13.4.2 the result is of the desired form, an orthonormal basis extending {w1 , · · · , wr }.
Here is another lemma about preserving distance.
Lemma 13.4.4 Suppose R is an m × n matrix with m > n and R preserves distances. Then
R∗ R = I.
Proof: Since R preserves distances, |Rx| = |x| for every x. Therefore from the axioms of the dot
product,
2
2
2
=
|x| + |y| + (x, y) + (y, x) = |x + y| = (R (x + y) , R (x + y))
(Rx,Rx) + (Ry,Ry) + (Rx, Ry) + (Ry, Rx)
=
|x| + |y| + (R∗ Rx, y) + (y, R∗ Rx)
and so for all x, y,
Hence for all x, y,
2
2
(R∗ Rx − x, y) + (y,R∗ Rx − x) = 0
Re (R∗ Rx − x, y) = 0
Now for a x, y given, choose α ∈ C such that
α (R∗ Rx − x, y) = |(R∗ Rx − x, y)|
Then
0 = Re (R∗ Rx − x,αy) = Re α (R∗ Rx − x, y) = |(R∗ Rx − x, y)|
Thus |(R∗ Rx − x, y)| = 0 for all x, y because the given x, y were arbitrary. Let y = R∗ Rx − x to
conclude that for all x,
R∗ Rx − x = 0
which says R∗ R = I since x is arbitrary. With this preparation, here is the big theorem about the right polar factorization.
Theorem 13.4.5 Let F be an m × n matrix where m ≥ n. Then there exists a Hermitian n × n
matrix U which has all nonnegative eigenvalues and an m × n matrix R which preserves distances
and satisfies R∗ R = I such that
F = RU.
13.4. THE RIGHT POLAR FACTORIZATION∗
265
Proof: Consider F ∗ F. This is a Hermitian matrix because
∗
∗
(F ∗ F ) = F ∗ (F ∗ ) = F ∗ F
Also the eigenvalues of the n×n matrix F ∗ F are all nonnegative. This is because if x is an eigenvalue,
λ (x, x) = (F ∗ F x, x) = (F x,F x) ≥ 0.
Therefore, by Lemma 13.4.1, there exists an n × n Hermitian matrix U having all nonnegative
eigenvalues such that
U 2 = F ∗ F.
Consider the subspace U (Fn ). Let {U x1 , · · · , U xr } be an orthonormal basis for
U (Fn ) ⊆ Fn .
Note that U (Fn ) might not be all of Fn . Using Lemma 13.4.3, extend to an orthonormal basis for
all of Fn ,
{U x1 , · · · , U xr , yr+1 , · · · , yn } .
Next observe that {F x1 , · · · , F xr } is also an orthonormal set of vectors in Fm . This is because
(
)
(F xk , F xj ) = (F ∗ F xk , xj ) = U 2 xk , xj
= (U xk , U ∗ xj ) = (U xk , U xj ) = δ jk
Therefore, from Lemma 13.4.3 again, this orthonormal set of vectors can be extended to an orthonormal basis for Fm ,
{F x1 , · · · , F xr , zr+1 , · · · , zm }
Thus there are at least as many zk as there are yj . Now for x ∈ Fn , since
{U x1 , · · · , U xr , yr+1 , · · · , yn }
is an orthonormal basis for Fn , there exist unique scalars,
c1 · · · , cr , dr+1 , · · · , dn
such that
x=
r
∑
k=1
Define
Rx ≡
n
∑
ck U xk +
dk yk
k=r+1
r
∑
n
∑
ck F xk +
k=1
dk zk
(13.12)
k=r+1
Then also there exist scalars bk such that
Ux =
r
∑
bk U xk
k=1
and so from 13.12,
RU x =
∑r
Is F ( k=1 bk xk ) = F (x)?
( (
F
r
∑
(
bk F xk = F
k=1
r
∑
k=1
)
bk xk
)
bk xk
k=1
(
− F (x) , F
r
∑
r
∑
k=1
)
bk xk
)
− F (x)
266
CHAPTER 13. MATRICES AND THE INNER PRODUCT
(
=
(
∗
(F F )
( (
=
U
r
∑
) (
bk xk − x ,
k=1
r
∑
)
(
bk xk − x , U
k=1
r
∑
))
bk xk − x
k=1
r
∑
=
))
bk xk − x
(
(
=
(
U
2
r
∑
) (
bk xk − x ,
k=1
r
∑
bk U xk − U x,
))
bk xk − x
k=1
r
∑
)
bk U xk − U x
=0
k=1
k=1
k=1
r
∑
∑r
Therefore, F ( k=1 bk xk ) = F (x) and this shows RU x = F x. From 13.12 and Lemma 13.3.7 R
preserves distances. Therefore, by Lemma 13.4.4 R∗ R = I. 13.5
The Singular Value Decomposition
In this section, A will be an m × n matrix. To begin with, here is a simple lemma.
Lemma 13.5.1 Let A be an m × n matrix. Then A∗ A is self adjoint and all its eigenvalues are
nonnegative.
Proof: It is obvious that A∗ A is self adjoint. Suppose A∗ Ax = λx. Then λ |x| = (λx, x) =
(A Ax, x) = (Ax,Ax) ≥ 0. 2
∗
Definition 13.5.2 Let A be an m × n matrix. The singular values of A are the square roots of the
positive eigenvalues of A∗ A.
With this definition and lemma here is the main theorem on the singular value decomposition.
Theorem 13.5.3 Let A be an m × n matrix. Then there exist unitary matrices, U and V of the
appropriate size such that
(
)
σ 0
∗
U AV =
0 0
where σ is of the form


σ=

σ1
0
..
0
.




σk
for the σ i the singular values of A.
n
Proof: By the above lemma and Theorem 13.2.14 there exists an orthonormal basis, {vi }i=1
such that A∗ Avi = σ 2i vi where σ 2i > 0 for i = 1, · · · , k, (σ i > 0) and equals zero if i > k. Thus for
i > k, Avi = 0 because
(Avi , Avi ) = (A∗ Avi , vi ) = (0, vi ) = 0.
For i = 1, · · · , k, define ui ∈ Fm by
ui ≡ σ −1
i Avi .
Thus Avi = σ i ui . Now
(ui , uj )
(
) ( −1
)
−1
−1 ∗
σ −1
i Avi , σ j Avj = σ i vi , σ j A Avj
(
) σj
−1 2
(vi , vj ) = δ ij .
= σ −1
i vi , σ j σ j vj =
σi
=
k
Thus {ui }i=1 is an orthonormal set of vectors in Fm . Also,
−1
−1
∗
2
2
AA∗ ui = AA∗ σ −1
i Avi = σ i AA Avi = σ i Aσ i vi = σ i ui .
k
m
Now extend {ui }i=1 to an orthonormal basis for all of Fm , {ui }i=1 and let
U ≡ (u1 · · · um )
13.5. THE SINGULAR VALUE DECOMPOSITION
267
while V ≡ (v1 · · · vn ) . Thus U is the matrix which has the ui as columns and V is defined as the
matrix which has the vi as columns. Then
 ∗ 
 ∗ 
u1
u1
 . 
 . 
 .. 
 .. 
(
)




σ 0
 ∗ 
 ∗ 
∗
U AV =  uk  A (v1 · · · vn ) =  uk  (σ 1 u1 · · · σ k uk , 0 · · · 0) =




0 0
 .. 
 .. 
 . 
 . 
u∗m
u∗m
where σ is given in the statement of the theorem. The singular value decomposition has as an immediate corollary the following interesting result.
Corollary 13.5.4 Let A be an m × n matrix. Then the rank of A and A∗ equals the number of
singular values.
Proof: Since V and U are unitary, it follows that
(
rank (A) =
=
rank (U ∗ AV ) = rank
σ
0
0
0
)
number of singular values.
Also since U, V are unitary,
(
∗)
rank (A∗ ) = rank (V ∗ A∗ U ) = rank (U ∗ AV )
((
)∗ )
σ 0
= rank
= number of singular values. 0 0
How could you go about computing the singular value decomposition? The proof of existence
indicates how to do it. Here is an informal method. You have from the singular value decompositon,
(
)
(
)
σ 0
σ 0
∗
∗
A=U
V , A =V
U∗
0 0
0 0
Then it follows that
(
∗
A A=V
σ
0
0
0
)
(
∗
U U
σ
0
0
0
)
(
∗
V =V
σ2
0
(
0
0
)
V∗
)
(
)
σ2 0
σ2 0
∗
and so A AV = V
. Similarly, AA U = U
. Therefore, you would find
0 0
0 0
an orthonormal basis of eigenvectors for AA∗ make them the columns of a matrix such that the
corresponding eigenvalues are decreasing. This gives U. You could then do the same for A∗ A to get
V.
∗
Example 13.5.5 Find a singular value decomposition for the matrix
( √ √
)
√ √
2
2 5 54 2 5 0
5
√ √
√ √
A≡
2
4
5 2 5
5 2 5 0
First consider A∗ A



16
5
32
5
32
5
64
5
0
0

0

0 
0
268
CHAPTER 13. MATRICES AND THE INNER PRODUCT
What are some eigenvalues and eigenvectors? Some computing shows these are
  
  √ 
√ 
1


− 25 5 
 0


 5 √5 
   1√ 
 2

 0  ,  5 5  ↔ 0,  5 5  ↔ 16








1
0
0
Thus the matrix V is given by


V =

√
− 25 5 0
√

1
0 
5 5
0
1
√
1
5 √5
2
5 5
0
(
Next consider AA∗ =
)
8 8
. Eigenvectors and eigenvalues are
8 8
{( √ )}
{(
√ )}
1
− 12 2
2 √2
√
↔ 0,
↔ 16
1
1
2
2
2 2
In this case you can let U be given by
(
U=
√ )
− 12 2
√
1
2 2
√
1
2 √2
1
2 2
Lets check this. U ∗ AV =
(
√
1
2
2 √
1
−2 2
√ )(
1
2 √2
1
2 2
√ √
2
5 √2√5
2
5 2 5
(
=
√ √
4
5 √2√5
4
5 2 5
4
0
0
0
0
0
0
0
)



√
1
5 √5
2
5 5
0

√
− 25 5 0
√

1
0 
5 5
0
1
)
This illustrates that if you have a good way to find the eigenvectors and eigenvalues for a Hermitian matrix which has nonnegative eigenvalues, then you also have a good way to find the singular
value decomposition of an arbitrary matrix.
13.6
Approximation In The Frobenius Norm∗
The Frobenius norm is one of many norms for a matrix. It is arguably the most obvious of all norms.
First here is a short discussion of the trace.
Definition 13.6.1 Let A be an n × n matrix. Then
∑
trace (A) ≡
Aii
i
just the sum of the entries on the main diagonal.
The fundamental property of the trace is in the next lemma.
Lemma 13.6.2 Let A = S −1 BS. Then trace (A) = trace (B). Also, for any two n × n matrices
A, B
trace (AB) = trace (BA)
13.6. APPROXIMATION IN THE FROBENIUS NORM∗
269
Proof: Consider the displayed formula.
trace (AB)
=
trace (BA)
=
∑∑
i
j
j
i
∑∑
Aij Bji
Bji Aij
they are the same thing. Thus if A = S −1 BS,
(
)
(
)
trace (A) = trace S −1 (BS) = trace BSS −1 = trace (B) . Here is the definition of the Frobenius norm.
Definition 13.6.3 Let A be a complex m × n matrix. Then
||A||F ≡ (trace (AA∗ ))
1/2
Also this norm comes from the inner product
(A, B)F ≡ trace (AB ∗ )
2
Thus ||A||F is easily seen to equal
∑
ij
2
|aij | so essentially, it treats the matrix as a vector in Fm×n .
Lemma 13.6.4 Let A be an m × n complex matrix with singular matrix
(
)
σ 0
Σ=
0 0
with σ as defined above. Then
2
2
||Σ||F = ||A||F
(13.13)
and the following hold for the Frobenius norm. If U, V are unitary and of the right size,
||U A||F = ||A||F , ||U AV ||F = ||A||F .
(13.14)
Proof: From the definition and letting U, V be unitary and of the right size,
||U A||F ≡ trace (U AA∗ U ∗ ) = trace (AA∗ ) = ||A||F
2
Also,
2
||AV ||F ≡ trace (AV V ∗ A∗ ) = trace (AA∗ ) = ||A||F .
2
2
It follows
2
2
2
||U AV ||F = ||AV ||F = ||A||F .
Now consider 13.13. From what was just shown,
||A||F = ||U ΣV ∗ ||F = ||Σ||F . 2
2
Of course, this shows that
2
||A||F =
∑
2
σ 2i ,
i
the sum of the squares of the singular values of A.
Why is the singular value decomposition important? It implies
(
)
σ 0
A=U
V∗
0 0
270
CHAPTER 13. MATRICES AND THE INNER PRODUCT
where σ is the diagonal matrix having the singular values down the diagonal. Now sometimes A is
a huge matrix, 1000×2000 or something like that. This happens in applications to situations where
the entries of A describe a picture. What also happens is that most of the singular values are very
small. What if you deleted those which were very small, say for all i ≥ l and got a new matrix,
)
(
σ′ 0
′
V ∗?
A ≡U
0 0
Then the entries of A′ would end up being close to the entries of A but there is much less information
to keep track of. This turns out to be very useful. More precisely, letting


(
)
σ1
0


σ
0
..
 , U ∗ AV =
σ=
,
.


0 0
0
σr
(
||A −
2
A′ ||F
′
σ − σ′
0
= U
0
0
)
2
V
∗
=
F
r
∑
σ 2k
k=l+1
′
Thus A is approximated by A where A has rank l < r. In fact, it is also true that out of all
matrices of rank l, this A′ is the one which is closest to A in the Frobenius norm. Here is why.
Let B be a matrix which has rank l. Then from Lemma 13.6.4
2
||A − B||F
= ||U ∗ (A − B) V ||F = ||U ∗ AV − U ∗ BV ||F
(
)
2
σ 0
∗
=
− U BV
0 0
2
2
F
and since the singular values of A decrease from the upper left to the lower right, it follows that for
B to be closest as possible to A in the Frobenius norm,
(
)
σ′ 0
∗
U BV =
0 0
which implies B = A′ above. This is really obvious if

(
)
3
σ 0

= 0
0 0
0
you look at a simple example. Say

0 0 0

2 0 0 
0 0 0
for example. Then what rank 1 matrix would be closest to this one in the Frobenius norm? Obviously


3 0 0 0


 0 0 0 0 
0 0 0 0
13.7
Moore Penrose Inverse∗
The singular value decomposition also has a very interesting connection to the problem of least
squares solutions. Recall that it was desired to find x such that |Ax − y| is as small as possible.
Lemma 13.3.3 shows that there is a solution to this problem which can be found by solving the system
A∗ Ax = A∗ y. Each x which solves this system, solves the minimization problem as was shown in the
13.8. EXERCISES
271
lemma just mentioned. Now consider this equation for the solutions of the minimization problem in
terms of the singular value decomposition.
z (
V
A∗
A∗
A
z ( }| ) {
}| ) {z ( }| ) {
σ 0
σ
0
σ 0
U ∗U
V ∗x = V
U ∗ y.
0 0
0 0
0 0
Therefore, this yields the following upon using block multiplication and multiplying on the left by
V ∗.
)
(
)
(
σ 0
σ2 0
∗
V x=
U ∗ y.
(13.15)
0 0
0 0
One solution to this equation which is very easy to spot is
(
)
σ −1 0
x=V
U ∗ y.
0
0
(13.16)
(
)
−1
σ
0
This special x is denoted by A+ y. The matrix V
U ∗ is denoted by A+ . Thus x just
0
0
defined is a solution to the least squares problem of finding the x such that Ax is as close as possible
to y. Suppose now that z is some other solution to this least squares problem. Thus from the above,
(
)
(
)
σ2 0
σ 0
∗
V z=
U ∗y
0 0
0 0
(
and so, multiplying both sides by
(
σ −2
0
Ir×r
0
0
0
0
0
)
,
)
(
V ∗z =
σ −1
0
0
0
)
U ∗y
To make V ∗ z as small as possible, you would have only the first r entries of V ∗ z be nonzero since
the later ones will be zeroed out anyway so they are unnecessary. Hence
(
)
−1
σ
0
V ∗z =
U ∗y
0
0
and consequently
(
z=V
σ −1
0
0
0
)
U ∗ y ≡ A+ y
However, minimizing |V ∗ z| is the same as minimizing |z| because V is unitary. Hence A+ y is the
solution to the least squares problem which has smallest norm.
13.8
Exercises
1. Here are some matrices. Label according to whether they are symmetric, skew symmetric, or
orthogonal. If the matrix is orthogonal, determine whether it is proper or improper.
272
CHAPTER 13. MATRICES AND THE INNER PRODUCT

1
0
√

(a)  0 1/ 2
√
0 1/ 2

0
√ 
−1/ 2 
√
1/ 2

1 2

(b)  2 1
−3 4

−3

4 
7

0

(c)  2
3
−2
0
4

−3

−4 
0
2. Show that every real matrix may be written as the sum of( a skew symmetric
and a symmetric
)
matrix. Hint: If A is an n × n matrix, show that B ≡ 12 A − AT is skew symmetric.
T
3. Let x be a vector in Rn and consider the matrix I − 2xx
. Show this matrix is both symmetric
||x||2
and orthogonal.
4. For U an orthogonal matrix, explain why ||U x|| = ||x|| for any vector x. Next explain why if
U is an n × n matrix with the property that ||U x|| = ||x|| for all vectors, x, then U must be
orthogonal. Thus the orthogonal matrices are exactly those which preserve distance.
5. A quadratic form in three variables is an expression of the form a1 x2 + a2 y 2 + a3 z 2 + a4 xy +
a5 xz + a6 yz. Show that every such quadratic form may be written as


x
(
)


x y z A y 
z
where A is a symmetric matrix.
6. Given a quadratic form in three variables, x, y, and
U and variables x′ , y ′ , z ′ such that



x



 y =U
z
z, show there exists an orthogonal matrix

x′

y′ 
z′
with the property that in terms of the new variables, the quadratic form is
λ1 (x′ ) + λ2 (y ′ ) + λ3 (z ′ )
2
2
2
where the numbers, λ1 , λ2 , and λ3 are the eigenvalues of the matrix A in Problem 5.
7. If A is a symmetric invertible matrix, is it always the case that A−1 must be symmetric also?
How about Ak for k a positive integer? Explain.
8. If A, B are symmetric matrices, does it follow that AB is also symmetric?
9. Suppose A, B are symmetric and AB = BA. Does it follow that AB is symmetric?
10. Here are some matrices. What can you say about the eigenvalues of these matrices just by
looking at them?


0 0 0


(a)  0 0 −1 
0 1 0


1 2 −3


(b)  2 1 4 
−3 4 7

0

(c)  2
3

1

(d)  0
0
−2
0
4
2
2
0

−3

−4 
0

3

3 
2
13.8. EXERCISES
273

0 0

0 −b  . Here b, c are real numbers.
b 0


c 0 0


12. Find the eigenvalues and eigenvectors of the matrix  0 a −b . Here a, b, c are real
0 b a
numbers.

c

11. Find the eigenvalues and eigenvectors of the matrix  0
0
13. Find the eigenvalues and an orthonormal basis of eigenvectors for A.


11 −1 −4


A =  −1 11 −4  .
−4 −4 14
Hint: Two eigenvalues are 12 and 18.
14. Find the eigenvalues and an orthonormal basis of eigenvectors for A.


4
1 −2


A= 1
4 −2  .
−2 −2 7
Hint: One eigenvalue is 3.
15. Show that if A is a real symmetric matrix and λ and µ are two different eigenvalues, then if x
is an eigenvector for λ and y is an eigenvector for µ, then x · y = 0. Also all eigenvalues are
real. Supply reasons for each step in the following argument. First
T
λxT x = (Ax) x = xT Ax = xT Ax = xT λx = λxT x
and so λ = λ. This shows that all eigenvalues are real. It follows all the eigenvectors are real.
Why? Now let x, y, µ and λ be given as above.
λ (x · y) = λx · y = Ax · y = x · Ay = x·µy = µ (x · y) = µ (x · y)
and so
(λ − µ) x · y = 0.
Since λ ̸= µ, it follows x · y = 0.
16. Suppose U is an orthogonal n × n matrix. Explain why rank (U ) = n.
17. Show that if A is an Hermitian matrix and λ and µ are two different eigenvalues, then if x is
an eigenvector for λ and y is an eigenvector for µ, then x · y = 0. Also all eigenvalues are real.
Supply reasons for each step in the following argument. First
λx · x = Ax · x = x·Ax = x·λx = λx · x
and so λ = λ. This shows that all eigenvalues are real. Now let x, y, µ and λ be given as
above.
λ (x · y) = λx · y = Ax · y = x · Ay = x·µy = µ (x · y) = µ (x · y)
and so
(λ − µ) x · y = 0.
Since λ ̸= µ, it follows x · y = 0.
274
CHAPTER 13. MATRICES AND THE INNER PRODUCT
18. Show that the eigenvalues and eigenvectors of a real matrix occur in conjugate pairs.
19. If a real matrix A has all real eigenvalues, does it follow that A must be symmetric. If so,
explain why and if not, give an example to the contrary.
20. Suppose A is a 3 × 3 symmetric matrix and you have found two eigenvectors which form an
orthonormal set. Explain why their cross product is also an eigenvector.
21. Study the definition of an orthonormal set of vectors. Write it from memory.
22. Determine which of the following sets of vectors are orthonormal sets. Justify your answer.
{( 1 2 2 ) ( −2 −1 2 ) ( 2 −2 1 )}
(a) {(1, 1) , (1, −1)}
(c)
3, 3, 3 ,
3 , 3 , 3 , 3, 3 , 3
)
}
{(
1
−1
√ , √
(b)
, (1, 0)
2
2
23. Show that if {u1 , · · · , un } is an orthonormal set of vectors in Fn , then it is a basis. Hint: It
was shown earlier that this is a linearly independent set. If you wish, replace Fn with Rn . Do
this version if you do not know the dot product for vectors in Cn .
24. Fill in the missing entries to make the matrix orthogonal.
 −1 −1

1





√
2
√1
2
√
√
6
6
3
√
3


.


25. Fill in the missing entries to make the matrix orthogonal.

√
√ 
2
1
2
2
6 2
 3

 2

 3



0
26. Fill in the missing entries to make the matrix orthogonal.


1
− √25
3


 2

0
 3



√
4
15 5
27. Find the eigenvalues and an orthonormal basis of eigenvectors for A. Diagonalize A by finding
an orthogonal matrix U and a diagonal matrix D such that U T AU = D.


−1 1
1


A =  1 −1 1  .
1
1 −1
Hint: One eigenvalue is -2.
28. Find the eigenvalues and an orthonormal basis of eigenvectors for A. Diagonalize A by finding
an orthogonal matrix U and a diagonal matrix D such that U T AU = D.


17 −7 −4


A =  −7 17 −4  .
−4 −4 14
Hint: Two eigenvalues are 18 and 24.
13.8. EXERCISES
275
29. Find the eigenvalues and an orthonormal basis of eigenvectors for A. Diagonalize A by finding
an orthogonal matrix U and a diagonal matrix D such that U T AU = D.


13 1
4


A =  1 13 4  .
4
4 10
Hint: Two eigenvalues are 12 and 18.
30. Find the eigenvalues and an orthonormal basis of eigenvectors for A. Diagonalize A by finding
an orthogonal matrix U and a diagonal matrix D such that U T AU = D.




A=



− 53
1
15
1
15
√ √
6 5
√ √
6 5
− 14
5
√
5
√
1
− 15
6
8
15
√ 
5

√ 
1
− 15 6 




7
8
15
15
Hint: The eigenvalues are −3, −2, 1.
31. Find the eigenvalues and an orthonormal basis of eigenvectors for A. Diagonalize A by finding
an orthogonal matrix U and a diagonal matrix D such that U T AU = D.

3
 0

A=

0
0
0
3
2
1
2
1
2
3
2



.

32. Find the eigenvalues and an orthonormal basis of eigenvectors for A. Diagonalize A by finding
an orthogonal matrix U and a diagonal matrix D such that U T AU = D.


2 0 0


A =  0 5 1 .
0 1 5
33. Find the eigenvalues and an orthonormal basis of eigenvectors for A. Diagonalize A by finding
an orthogonal matrix U and a diagonal matrix D such that U T AU = D.




A=



4
3
1
3
√ √
3 2
√
1
3 2
1
3
√ √
3 2
1
√
1
−3 3
1
3
√
2


√ 

3 


5

− 13
3
Hint: The eigenvalues are 0, 2, 2 where 2 is listed twice because it is a root of multiplicity 2.
34. Find the eigenvalues and an orthonormal basis of eigenvectors for A. Diagonalize A by finding
an orthogonal matrix U and a diagonal matrix D such that U T AU = D.
276
CHAPTER 13. MATRICES AND THE INNER PRODUCT

1
6
1
√ √
1
6 3 2



A=



1
6
√ √
3 2
3
2
√ √
3 6
1
12
√ √
3 6
√ √
1
12 2 6
1
6
√ √
2 6
1
2








Hint: The eigenvalues are 2, 1, 0.
35. Find the eigenvalues and an orthonormal basis of eigenvectors for the matrix








√ √ 
7
− 18
3 6

√ √
√ √ 
1
3
1

3
2
−
2
6

6
2
12


√
√
√
√

7
1
3 6 − 12
2 6
− 56
− 18
1
3
1
6
√ √
3 2
Hint: The eigenvalues are 1, 2, −2.
36. Find the eigenvalues and an orthonormal basis of eigenvectors for the matrix

− 12


 − 1 √6√5
 5


√

1
5
10
√ √
− 15 6 5
7
5
√
− 15 6
√ 
5

√ 
1
−5 6 




9
−
1
10
10
Hint: The eigenvalues are −1, 2, −1 where −1 is listed twice because it has multiplicity 2 as
a zero of the characteristic equation.
37. Explain why a matrix A is symmetric if and only if there exists an orthogonal matrix U such
that A = U T DU for D a diagonal matrix.
38. The proof of Theorem 13.3.3 concluded with the following observation. If −ta + t2 b ≥ 0 for
all t ∈ R and b ≥ 0, then a = 0. Why is this so?
39. Using Schur’s theorem, show that whenever A is an n × n matrix, det (A) equals the product
of the eigenvalues of A.
40. In the proof of Theorem 13.3.8 the following argument was used. If x · w = 0 for all w ∈ Rn ,
then x = 0. Why is this so?
41. Using Corollary 13.3.9 show that a real m × n matrix is onto if and only if its transpose is one
to one.
42. Suppose A is a 3 × 2 matrix. Is it possible that AT is one to one? What does this say about
A being onto? Prove your answer.
43. Find the least squares solution to the system x + 2y = 1, 2x + 3y = 2, 3x + 5y = 4.
44. You are doing experiments and have obtained the ordered pairs,
(0, 1) , (1, 2) , (2, 3.5) , (3, 4)
Find m and b such that y = mx + b approximates these four points as well as possible. Now
do the same thing for y = ax2 + bx + c, finding a, b, and c to give the best approximation.
13.8. EXERCISES
277
45. Suppose you have several ordered triples, (xi , yi , zi ) . Describe how to find a polynomial,
z = a + bx + cy + dxy + ex2 + f y 2
for example giving the best fit to the given ordered triples. Is there any reason you have to
use a polynomial? Would similar approaches work for other combinations of functions just as
well?
46. Find an orthonormal basis for the spans of the following sets of vectors.
(a) (3, −4, 0) , (7, −1, 0) , (1, 7, 1).
(b) (3, 0, −4) , (11, 0, 2) , (1, 1, 7)
(c) (3, 0, −4) , (5, 0, 10) , (−7, 1, 1)
47. Using the Gram Schmidt process or the QR factorization, find an orthonormal basis for the
span of the vectors, (1, 2, 1) , (2, −1, 3) , and (1, 0, 0) .
48. Using the Gram Schmidt process or the QR factorization, find an orthonormal basis for the
span of the vectors, (1, 2, 1, 0) , (2, −1, 3, 1) , and (1, 0, 0, 1) .
49. The set, V ≡ {(x, y, z) : 2x + 3y − z = 0} is a subspace of R3 . Find an orthonormal basis for
this subspace.
50. The two level surfaces, 2x + 3y − z + w = 0 and 3x − y + z + 2w = 0 intersect in a subspace
of R4 , find a basis for this subspace. Next find an orthonormal basis for this subspace.
51. Let A, B be a m × n matrices. Define an inner product on the set of m × n matrices by
(A, B)F ≡ trace (AB ∗ ) .
Show this is an inner∑
product satisfying all the inner product axioms. Recall for M an n × n
n
matrix, trace (M ) ≡ i=1 Mii . The resulting norm, ||·||F is called the Frobenius norm and it
can be used to measure the distance between two matrices.
∑
2
52. Let A be an m × n matrix. Show ||A||F ≡ (A, A)F = j σ 2j where the σ j are the singular
values of A.
∑
53. The trace of an n × n matrix M is defined as i Mii . In other words it is the sum of the entries
on the main diagonal. If A, B are n × n matrices, show trace (AB) = trace (BA). Now explain
why if A = S −1 BS it follows trace (A) = trace (B). Hint: For the first part, write these in
terms of components of the matrices and it just falls out.
54. Using Problem 53 and Schur’s theorem, show that the trace of an n × n matrix equals the sum
of the eigenvalues.
55. If A is a general n × n matrix having possibly repeated eigenvalues, show there is a sequence
{Ak } of n × n matrices having distinct eigenvalues which has the property that the ij th entry
of Ak converges to the ij th entry of A for all ij. Hint: Use Schur’s theorem.
56. Prove the Cayley Hamilton theorem as follows. First suppose A has a basis of eigenvectors
n
{vk }k=1 , Avk = λk vk . Let p (λ) be the characteristic polynomial. Show p (A) vk = p (λk ) vk =
0. Then since {vk } is a basis, it follows p (A) x = 0 for all x and so p (A) = 0. Next in the general
case, use Problem 55 to obtain a sequence {Ak } of matrices whose entries converge to the entries
of A such that Ak has n distinct eigenvalues and therefore by Theorem 12.1.15 Ak has a basis
of eigenvectors. Therefore, from the first part and for pk (λ) the characteristic polynomial for
Ak , it follows pk (Ak ) = 0. Now explain why and the sense in which limk→∞ pk (Ak ) = p (A) .
278
CHAPTER 13. MATRICES AND THE INNER PRODUCT
57. Show that the Moore Penrose inverse A+ satisfies the following conditions.
AA+ A = A, A+ AA+ = A+ , A+ A, AA+ are Hermitian.
Next show that if A0 satisfies the above conditions, then it must be the Moore Penrose inverse
and that if A is an n × n invertible matrix, then A−1 satisfies the above conditions. Thus
the Moore Penrose inverse generalizes the usual notion of inverse but does not contradict it.
Hint: Let
(
)
σ 0
∗
U AV = Σ ≡
0 0
and suppose
(
+
V A0 U =
P
R
Q
S
)
where P is the same size as σ. Now use the conditions to identify P = σ, Q = 0 etc.
58. Find the least squares solution to




(
)
1
1
a

 x


= b 
1 
 1
y
c
1 1+ε
Next suppose ε is so small that all ε2 terms are ignored by the computer but the terms of
order ε are not ignored. Show the least squares equations in this case reduce to
(
)(
) (
)
3
3+ε
x
a+b+c
=
.
3 + ε 3 + 2ε
y
a + b + (1 + ε) c
Find the solution to this and compare the y values of the two solutions. Show that one of these
is −2 times the other. This illustrates a problem with the technique for finding least squares
solutions presented as the solutions to A∗ Ax = A∗ y. One way of dealing with this problem
is to use the QR factorization. This is illustrated in the next problem. It turns out that this
helps alleviate some of the round off difficulties of the above.
59. Show that the equations A∗ Ax = A∗ y can be written as R∗ Rx = R∗ Q∗ y where R is upper
triangular and R∗ is lower triangular. Explain how to solve this system efficiently. Hint: You
first find Rx and then you find x which will not be hard because R is upper triangular.
60. Show that A+ = (A∗ A) A∗ . Hint: You might use the description of A+ in terms of the
singular value decomposition.
(
)
1 −3 0
61. Let A =
. Then
3 −1 0
+

T
 √
√
√
√
2/2 0
2/2 0
− 2/2
− 2/2
√
√
√
√



2/2
2/2 0  AT A 
2/2
2/2 0

0
0
1
0
0
1
(
)
(
10 6
16
AAT =
. A matrix U with U T AAT U =
6 10
0
However,
 √
√
)T (
)
( √
√
− 2/2
2/2
√
1 −3 0  √
2/2 − 2/2
√
√
2/2
2/2

2/2
2/2
3 −1 0
0
0
(
)
4 0 0
How can this be fixed so that you get
?
0 2 0



16 0 0
 

= 0 4 0 
0 0 0
)
)
( √
√
2/2 − 2/2
0
√
√
.
is
2/2
2/2
4

(
)
0
−4 0 0

.
0 =
0 2 0
1
Chapter 14
Numerical Methods For Solving
Linear Systems
14.1
Iterative Methods For Linear Systems
Consider the problem of solving the equation
Ax = b
(14.1)
where A is an n × n matrix. In many applications, the matrix A is huge and composed mainly of
zeros. For such matrices, the method of Gauss elimination (row operations) is not a good way to
solve the system because the row operations can destroy the zeros and storing all those zeros takes
a lot of room in a computer. These systems are called sparse. To solve them it is common to use
an iterative technique. The idea is to obtain a sequence of approximate solutions which get close to
the true solution after a sufficient number of iterations.
∞
Definition 14.1.1 Let {xk }k=1 be a sequence of vectors in Fn . Say
(
)
xk = xk1 , · · · , xkn .
Then this sequence is said to converge to the vector x = (x1 , · · · , xn ) ∈ Fn , written as
lim xk = x
k→∞
if for each j = 1, 2, · · · , n,
lim xkj = xj .
k→∞
In words, the sequence converges if the entries of the vectors in the sequence converge to the corresponding entries of x.
(
(
))
k2
1+k2
Example 14.1.2 Consider xk = sin (1/k) , 1+k
. Find limk→∞ xk .
2 , ln
k2
From the above definition, this limit is the vector (0, 1, 0) because
(
)
k2
1 + k2
lim sin (1/k) = 0, lim
= 1, and lim ln
= 0.
k→∞
k→∞ 1 + k 2
k→∞
k2
A more complete mathematical explanation is given in Linear Algebra. Linear Algebra
279
280
CHAPTER 14. NUMERICAL METHODS FOR SOLVING LINEAR SYSTEMS
14.1.1
The Jacobi Method
The first technique to be discussed here is the Jacobi method
is described in the following
{ which
}
definition. In this technique, you have a sequence of vectors, xk which converge to the solution to
the linear system of equations and to get the ith component of the xk+1 , you use all the components
of xk except for the ith . The precise description follows.
Definition 14.1.3 The Jacobi iterative technique, also called the method of simultaneous corrections, is defined as follows. Let x1 be an initial vector, say the zero vector or some other vector.
The method generates a succession of vectors, x2 , x3 , x4 , · · · and hopefully this sequence of vectors
will converge to the solution to 14.1. The vectors in this list are called iterates and they are obtained
according to the following procedure. Letting A = (aij ) ,
∑
(14.2)
aii xr+1
=−
aij xrj + bi .
i
j̸=i
In terms of matrices, letting

a11
 .

A =  ..
an1
The iterates are defined as

a11

 0

 .
 .
 .
0

a22
..
.
0
..
.
···
0
ann
0

 a
 21
= − .
 .
 .
an1
0
..
.
···
0
···
..
.
..
.
a12
···

···
..
.
..
.
0

a1n
.. 

. 
ann
···
..
.
ann−1

xr+1
1
  r+1 
  x2


 . 
  .. 


xr+1
n

a1n
xr
  1r
..
  x2
.

 .
.

an−1n   .
xrn
0


 
 
+
 
 
b1
b2
..
.
bn






(14.3)
The matrix on the left in 14.3 is obtained by retaining the main diagonal of A and setting every
other entry equal to zero. The matrix on the right in 14.3 is obtained from A by setting every
diagonal entry equal to zero and retaining all the other entries unchanged.
Example 14.1.4 Use the Jacobi method to solve the system





3
1
0
0
1
4
2
0
0
1
5
2
0
0
1
4





x1
x2
x3
x4


 
 
=
 
1
2
3
4





In terms of the matrices, the Jacobi iteration is of the form





3
0
0
0
0
4
0
0
0
0
5
0
0
0
0
4





Now iterate this starting with
xr+1
1
xr+1
2
xr+1
3
xr+1
4






 = −


0
1
0
0
1
0
2
0
0
1
0
2
0
0
1
0





xr1
xr2
xr3
xr4


 
 
+
 
1
2
3
4



.

14.1. ITERATIVE METHODS FOR LINEAR SYSTEMS



x1 ≡ 






3
0
0
0
0
4
0
0
0
0
5
0
0
0
0
4





x21
x22
x23
x24






 = −


0
1
0
0


Solving this system yields


x2 = 

Then you use x2 to find x3 =





3
0
0
0
0
4
0
0
0
0
5
0
0
0
0
4





(
x31
x31
x32
x33
x34
x32
x21
x22
x23
x24


x34





 = −



0
1
0
0


.

0
1
0
2
 
 
=
 
x33

0
0
0
0
1
0
2
0
0
0
1
0



x3 = 

Now use this as the new data to find x4





3
0
0
0
0
4
0
0
0
0
5
0
0
0
0
4





x41
x42
x43
x44




0
0
0
0


 
 
+
 
1
2
3
4


 
 
=
 
1.0
2.0
3.0
4.0










)T
1
0
2
0
0
1
0
2

0
. 333 333 33


0 
.5



1
.6
0
1.0



 
 
+
 
1
2
3
4









 

. 166 666 67
x31


x32 
  . 266 666 68 
=




.2
x33  
3
x4
.7
)T
(
= x41 x42 x43 x44






 = −



0
1
0
0
1
0
2
0
0
1
0
2
. 733 333 32
 1. 633 333 3

= 
 1. 766 666 6
3. 6
Thus you find

. 333 333 33
.5
.6
1.0
.5
 1. 066 666 7

= 

1.0
2. 8
The solution is
281



x4 = 


0
. 166 666 67


0   . 266 666 68

1 
.2
0
.7



.

. 244 444 44
. 408 333 33
. 353 333 32
.9







 
 
+
 
1
2
3
4





282
CHAPTER 14. NUMERICAL METHODS FOR SOLVING LINEAR SYSTEMS
Then another iteration for x5 gives





3
0
0
0
0
4
0
0
0
0
5
0
0
0
0
4





x51
x52
x53
x54


0
1
0
0




 = −



1
0
2
0
0
1
0
2
. 591 666 67
 1. 402 222 2

= 
 1. 283 333 3
3. 293 333 4
and so



x5 = 


0
. 244 444 44


0   . 408 333 33

1   . 353 333 32
0
.9



 
 
+
 
1
2
3
4









. 197 222 22
. 350 555 55
. 256 666 66
. 823 333 35



.

The solution to the system of equations obtained by row operations is





so already after only five
it work?

3 1
 1 4


 0 2
0 0
x1
x2
x3
x4


 
 
=
 
. 206
. 379
. 275
. 862





iterations the iterates are pretty close to the true solution. How well does
0
1
5
2
0
0
1
4

. 197 222 22
. 350 555 55
. 256 666 66
. 823 333 35






 
 
=
 
. 942 222 21
1. 856 111 1
2. 807 777 8
3. 806 666 7


 
 
≈
 
1
2
3
4





A few more iterates will yield a better solution.
14.1.2
The Gauss Seidel Method
The Gauss Seidel method differs from the Jacobi method in using xk+1
for all j < i in going from
j
xk to xk+1 . This is why it is called the method of successive corrections. The precise description of
this method is in the following definition.
Definition 14.1.5 The Gauss Seidel method, also called the method of successive corrections
is given as follows. For A = (aij ) , the iterates for the problem Ax = b are obtained according to the
formula
i
n
∑
∑
aij xr+1
=−
aij xrj + bi .
(14.4)
j
j=1
In terms of matrices, letting
j=i+1

a11
 .

A =  ..
an1
···
..
.
···

a1n
.. 

. 
ann
14.1. ITERATIVE METHODS FOR LINEAR SYSTEMS
283
The iterates are defined as

a11

 a
 21
 .
 .
 .
an1

0

 0

− .
 .
 .
0
=
a22
..
.
···
a12
0
..


xr+1
  1r+1 
  x2


 . 
  .. 

0 
r+1
xn
ann−1 ann

 
···
a1n
xr1

..
 
..

xr2  
.
.


+
 . 
..
.  

. an−1n   .  
xrn
0
0
···
..
.
..
.
0
.
···
0
..
.
b1
b2
..
.
bn






(14.5)
In words, you set every entry in the original matrix which is strictly above the main diagonal
equal to zero to obtain the matrix on the left. To get the matrix on the right, you set every entry
of A which is on or below the main diagonal equal to zero. Using the iteration procedure of 14.4
directly, the Gauss Seidel method makes use of the very latest information which is available at that
stage of the computation.
The following example is the same as the example used to illustrate the Jacobi method.
Example 14.1.6 Use the Gauss Seidel method to solve the system





3
1
0
0
1
4
2
0
0
1
5
2

0
0
1
4




x1
x2
x3
x4


 
 
=
 
1
2
3
4





In terms of matrices, this procedure is





3
1
0
0
0
4
2
0
0
0
5
2
0
0
0
4





xr+1
1
xr+1
2
xr+1
3
xr+1
4






 = −


0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
0





xr1
xr2
xr3
xr4


 
 
+
 
1
2
3
4



.

As before, let x1 be the zero vector. Thus the first iteration is to solve





Hence
3
1
0
0
0
4
2
0
0
0
5
2
0
0
0
4





x21
x22
x23
x24






 = −


0
0
0
0



x2 = 

1
0
0
0
0
1
0
0
0
0
1
0
. 333 333 33
. 416 666 67
. 433 333 33
. 783 333 33










0
0
0
0


 
 
+
 
1
2
3
4


 
 
=
 
1
2
3
4





284
CHAPTER 14. NUMERICAL METHODS FOR SOLVING LINEAR SYSTEMS
Thus x3 =





(
3
1
0
0
x31
0
4
2
0
x32
x33
0
0
5
2
0
0
0
4
x34





)T
is given by
x31
x32
x33
x34






 = −



0
0
0
0
1
0
0
0
0
1
0
0
. 583 333 33
 1. 566 666 7

= 
 2. 216 666 7
4.0
And so



x3 = 


0

0 


1 
0

. 333 333 33
. 416 666 67
. 433 333 33
. 783 333 33


 
 
+
 
1
2
3
4










. 194 444 44
. 343 055 56
. 306 111 11
. 846 944 44


.

Another iteration for x4 involves solving





3
1
0
0
0
4
2
0
0
0
5
2
0
0
0
4





x41
x42
x43
x44






 = −



0
0
0
0
1
0
0
0
0
1
0
0
. 656 944 44
 1. 693 888 9

= 
 2. 153 055 6
4.0
and so



x4 = 

Recall the answer is






0

0 


1 
0



 
 
+
 
1
2
3
4









. 218 981 48
. 368 726 86
. 283 120 38
. 858 439 81
. 206
. 379
. 275
. 862
. 194 444 44
. 343 055 56
. 306 111 11
. 846 944 44










so the iterates are already pretty close to the answer. You could continue doing these iterates and
it appears they converge to the solution. Now consider the following example.
Example 14.1.7 Use the Gauss Seidel method to solve the system





1
1
0
0
4
4
2
0
0
1
5
2
0
0
1
4





x1
x2
x3
x4


 
 
=
 
1
2
3
4





14.1. ITERATIVE METHODS FOR LINEAR SYSTEMS
285
The exact solution is given by doing row operations on the augmented matrix. When this is done
the row echelon form is


1 0 0 0 6
 0 1 0 0 −5 

4 


 0 0 1 0 1 


1
0 0 0 1
2
and so the solution is
 
6
6.0
 −5  
 4   −1. 25


 1 =
  1.0

1
.5
2

The Gauss Seidel iterations are of the form


  r+1 
1 0 0 0
x1
0 4
 0 0
 1 4 0 0   xr+1 



 2


 = −
 0 2 5 0   xr+1

 0 0
3
r+1
x4
0 0
0 0 2 4
and so, multiplying by the inverse of the
terms of matrix multiplication.

0
 0



xr+1 = −  0


 0





0
1
0
0
0
0
1
0





xr1
xr2
xr3
xr4


 
 
+
 
1
2
3
4





matrix on the left, the iteration reduces to the following in
4
−1
0
1
4
0
0
2
5
1
− 10
1
5
− 15
1
20
1
− 10



1


 1 



 r  4 
x +  1 .

 2 




3
4
This time, we will pick an initial vector close to the answer. Let


6


 −1 

x1 = 
 1 


1
2
This is very close to the answer. Now lets see what the Gauss Seidel iteration does to it.



0 4
0
0
 



1
 0 −1

1
6
5.0
0 



1
4


  −1 
 
 


 4   −1.0 
2
1
1

−
x2 = −  0
+
=





5
10
5
1   1   .9 



 1   2 
 0 −1
1
3
1 
. 55
2
− 10
5
20
4
You can’t expect to be real

0
 0



x3 = −  0


 0
close after only one iteration. Lets do another.



4
0
0



1

1
5.0
5.0
−1
0



4
  −1.0   14   −. 975

 
 
1
1
2
− 10

=
+
5
5
  . 9   12   . 88



3
1
. 55
. 56
−1
−1 
5
20
10
4





286
CHAPTER 14. NUMERICAL METHODS FOR SOLVING LINEAR SYSTEMS

0 4
 0 −1



2
4
x = − 0
5


 0 −1
5
0

1
4
0
0
1
− 10
1
5
1
20
1
− 10


5.0

  −. 975


  . 88


. 56



 
 
+
 

1
1
4
1
2
3
4


4. 9

  −. 945
 
=
  . 866

. 567





The iterates seem to be getting farther from the actual solution. Why is the process which worked so
well in the other examples not working here? A better question might be: Why does either process
ever work at all? A complete answer to this question is given in more advanced linear algebra books.
You can also see it in Linear Algebra.
Both iterative procedures for solving
Ax = b
(14.6)
are of the form
Bxr+1 = −Cxr + b
where A = B + C. In the Jacobi procedure, the matrix C was obtained by setting the diagonal of
A equal to zero and leaving all other entries the same while the matrix B was obtained by making
every entry of A equal to zero other than the diagonal entries which are left unchanged. In the
Gauss Seidel procedure, the matrix B was obtained from A by making every entry strictly above
the main diagonal equal to zero and leaving the others unchanged, and C was obtained from A by
making every entry on or below the main diagonal equal to zero and leaving the others unchanged.
Thus in the Jacobi procedure, B is a diagonal matrix while in the Gauss Seidel procedure, B is lower
triangular. Using matrices to explicitly solve for the iterates, yields
xr+1 = −B −1 Cxr + B −1 b.
(14.7)
This is what you would never have the computer do, but this is what will allow the statement of a
theorem which gives the condition for convergence of these and all other similar methods.
Theorem 14.1.8 Let A = B + C and suppose all eigenvalues of B −1 C have absolute value less
than 1 where A = B + C. Then the iterates in 14.7 converge to the unique solution of 14.6.
A complete explanation of this important result is found in more advanced linear algebra books.
You can also see it in Linear Algebra. It depends on a theorem of Gelfand which is completely
proved in this reference. Theorem 14.1.8 is very remarkable because it gives an algebraic condition
for convergence, which is essentially an analytical question.
14.2
The Operator Norm∗
Recall that for x ∈ Cn ,
|x| ≡
√
⟨x, x⟩
Also recall Theorem 3.2.17 which says that
|z| ≥ 0 and |z| = 0 if and only if z = 0
(14.8)
If α is a scalar, |αz| = |α| |z|
(14.9)
|z + w| ≤ |z| + |w| .
(14.10)
If you have the above axioms holding for ∥·∥ replacing |·| , then ∥·∥ is called a norm. For example,
you can easily verify that
∥x∥ ≡ max {|xi | , i = 1, · · · , n : x = (x1 , · · · , xn )}
is a norm. However, there are many other norms.
14.2. THE OPERATOR NORM∗
287
One important observation is that x7→∥x∥ is a continuous function. This follows from the observation that from the triangle inequality,
∥x − y∥ + ∥y∥ ≥ ∥x∥
∥x − y∥ + ∥x∥ = ∥y − x∥ + ∥x∥ ≥ ∥y∥
Hence
∥x∥ − ∥y∥ ≤ ∥x − y∥
∥y∥ − ∥x∥ ≤ ∥x − y∥
and so
|∥x∥ − ∥y∥| ≤ ∥x − y∥
This section will involve some analysis. If you want to talk about norms, this is inevitable. It
will need some of the theorems of calculus which are usually neglected. In particular, it needs the
following result which is a case of the Heine Borel theorem. To see this proved, see any good calculus
book.
∞
Theorem 14.2.1 Let S denote the points x ∈ Fn such that |x| = 1. Then if {xk }k=1 is any sequence
of points of S, there exists a subsequence which converges to a point of S.
Definition 14.2.2 Let A be an m × n matrix. Let ∥·∥k denote a norm on Ck . Then the operator
norm is defined as follows.
∥A∥ ≡ max {∥Ax∥m : ∥x∥n ≤ 1}
Lemma 14.2.3 The operator norm is well defined and is in fact a norm on the vector space of
m × n matrices.
Proof: It has already been observed that the m × n matrices form a vector space starting on
Page 66. Why is ∥A∥ < ∞?
claim: There exists c > 0 such that whenever ∥x∥ ≤ 1, it follows that |x| ≤ c.
Proof of the claim: If not, then there exists {xk } such that ∥xk ∥ ≤ 1 but |xk | > k for
k = 1, 2, · · · . Then |xk / |xk || = 1 and so by the Heine Borel theorem from calculus, there exists a
further subsequence, still denoted by k such that
xk
− y → 0, |y| = 1.
|xk |
Letting
n
n
∑
∑
xk
=
aki ei , y =
ai ei ,
|xk |
i=1
i=1
It follows that ak → a in Fn . Hence
∑
xk
−y ≤
aki − ai ∥ei ∥
|xk |
i=1
n
which converges to 0. However,
xk
1
≤
|xk |
k
and so, by continuity of ∥·∥ mentioned above,
∥y∥ = lim
k→∞
xk
=0
|xk |
Therefore, y = 0 but also |y| = 1, a contradiction. This proves the claim.
288
CHAPTER 14. NUMERICAL METHODS FOR SOLVING LINEAR SYSTEMS
Now consider why ∥A∥ < ∞. Let c be as just described in the claim.
sup {∥Ax∥m : ∥x∥n ≤ 1} ≤ sup {∥Ax∥m : |x| ≤ c}
Consider for x, y with |x| , |y| ≤ c
∥Ax − Ay∥ =
≤
∑
∑
Aij (xj − yj ) ei
i
|Aij | |xj − yj | ∥ei ∥ ≤ C |x − y|
i
for some constant C. So x 7→ Ax is continuous. Since the norm ∥·∥m is continuous also, it follows
from the extreme value theorem of calculus that ∥Ax∥m achieves its maximum on the compact set
{x : |x| ≤ c} . Thus ∥A∥ is well defined. The only other issue of significance is the triangle inequality.
However,
∥A + B∥ ≡
max {∥(A + B) x∥m : ∥x∥n ≤ 1}
≤ max {∥Ax∥m + ∥Bx∥m : ∥x∥n ≤ 1}
≤ max {∥Ax∥m : ∥x∥n ≤ 1} + max {∥Bx∥m : ∥x∥n ≤ 1}
= ∥A∥ + ∥B∥
Obviously ∥A∥ = 0 if and only if A = 0. The rule for scalars is also immediate. The operator norm is one way to describe the magnitude of a matrix. Earlier the Frobenius
norm was discussed. The Frobenius norm is actually not used as much as the operator norm. Recall
that the Frobenius norm involved considering the m × n matrix as a vector in Fmn and using the
usual Euclidean norm. It can be shown that it really doesn’t matter which norm you use in terms
of estimates because they are all equivalent. This is discussed in Problem 25 below for those who
have had a legitimate calculus course, not just the usual undergraduate offering.
14.3
The Condition Number∗
Let A be an m × n matrix and consider the problem Ax = b where it is assumed there is a unique
solution to this problem. How does the solution change if A is changed a little bit and if b is changed
a little bit? This is clearly an interesting question because you often do not know A and b exactly.
If a small change in these quantities results in a large change in the solution x, then it seems clear
this would be undesirable. In what follows ||·|| when applied to a matrix will always refer to the
operator norm.
Lemma 14.3.1 Let A, B be m × n matrices. Then for ||·|| denoting the operator norm,
||AB|| ≤ ||A|| ||B|| .
Proof: This follows from the definition. Letting ||x|| ≤ 1, it follows from the definition of the
operator norm that
||ABx|| ≤ ||A|| ||Bx|| ≤ ||A|| ||B|| ||x|| ≤ ||A|| ||B||
and so
||AB|| ≡ sup ||ABx|| ≤ ||A|| ||B|| . ||x||≤1
Lemma 14.3.2 Let A, B be m × n matrices such that A−1 exists, and suppose ||B|| < 1/ A−1 .
−1
Then (A + B) exists and
(A + B)
−1
The above formula makes sense because
≤ A−1
A−1 B < 1.
1
.
1 − ||A−1 B||
14.3. THE CONDITION NUMBER∗
289
Proof: By Lemma 14.3.1,
1
=1
||A−1 ||
A−1 B ≤ A−1 ||B|| < A−1
(
)
(
)
Suppose (A + B) x = 0. Then 0 = A I + A−1 B x and so since A is one to one, I + A−1 B x =
0. Therefore,
(
)
0 =
I + A−1 B x ≥ ||x|| − A−1 Bx
(
)
≥ ||x|| − A−1 B ||x|| = 1 − A−1 B ||x|| > 0
(
)
a contradiction. This also shows I + A−1 B is one to one.
(
)
−1
I + A−1 B
exist. Hence
(A + B)
−1
−1
Therefore, both (A + B)
and
( (
))−1 (
)−1 −1
= A I + A−1 B
= I + A−1 B
A
Now if
(
)−1
x = I + A−1 B
y
for ||y|| ≤ 1, then
(
)
I + A−1 B x = y
and so
(
)
||x|| 1 − A−1 B ≤ x + A−1 Bx ≤ ||y|| = 1
and so
(
||x|| =
I + A−1 B
)−1
y ≤
1
1 − ||A−1 B||
Since ||y|| ≤ 1 is arbitrary, this shows
(
I + A−1 B
)−1
≤
1
1 − ||A−1 B||
Therefore,
(A + B)
−1
=
≤
(
I + A−1 B
A−1
(
)−1
A−1
I + A−1 B
)−1
≤ A−1
1
1 − ||A−1 B||
Proposition 14.3.3 Suppose A is invertible, b ̸= 0, Ax = b, and A1 x1 = b1 where ||A − A1 || <
1/ A−1 . Then
(
)
1
||A1 − A|| ||b − b1 ||
||x1 − x||
−1
≤
||A|| A
+
.
(14.11)
||x||
(1 − ||A−1 (A1 − A)||)
||A||
||b||
Proof: It follows from the assumptions that
Ax − A1 x + A1 x − A1 x1 = b − b1 .
Hence
A1 (x − x1 ) = (A1 − A) x + b − b1 .
Now A1 = (A + (A1 − A)) and so by the above lemma, A−1
1 exists and so
−1
(x − x1 ) = A−1
1 (A1 − A) x + A1 (b − b1 )
= (A + (A1 − A))
−1
(A1 − A) x + (A + (A1 − A))
−1
(b − b1 ) .
290
CHAPTER 14. NUMERICAL METHODS FOR SOLVING LINEAR SYSTEMS
By the estimate in Lemma 14.3.2,
||x − x1 || ≤
A−1
(||A1 − A|| ||x|| + ||b − b1 ||) .
1 − ||A−1 (A1 − A)||
Dividing by ||x|| ,
(
)
A−1
||x − x1 ||
||b − b1 ||
≤
||A1 − A|| +
||x||
1 − ||A−1 (A1 − A)||
||x||
( −1 )
−1
Now b = Ax = A A b and so ||b|| ≤ ||A|| A b and so
(14.12)
||x|| = A−1 b ≥ ||b|| / ||A|| .
Therefore, from 14.12,
||x − x1 ||
||x||
≤
≤
A−1
1 − ||A−1 (A1 − A)||
(
||A|| ||A1 − A|| ||A|| ||b − b1 ||
+
||A||
||b||
(
)
−1
A
||A||
||A1 − A|| ||b − b1 ||
+
−1
1 − ||A (A1 − A)||
||A||
||b||
)
This shows that the number, A−1 ||A|| , controls how sensitive the relative change in the
solution of Ax = b is to small changes in A and b. This number is called the condition number. It
is bad when it is large because a small relative change in b, for example, could yield a large relative
change in x.
14.4
Exercises
1. Solve the system

4

 1
0
1
5
2

  
1
x
2

  
=
2  y   1 
6
z
3
using the Gauss Seidel method and the Jacobi method. Check your answer by also solving it
using row operations.
2. Solve the system

4

 1
0
1
7
2

  
1
x
1

  
=
2  y   2 
4
z
3
using the Gauss Seidel method and the Jacobi method. Check your answer by also solving it
using row operations.
3. Solve the system

5

 1
0
1
7
2
  

3
x
1

  
2  y  =  0 
1
z
4
using the Gauss Seidel method and the Jacobi method. Check your answer by also solving it
using row operations.
4. Solve the system

7 1

 1 5
0 2

 

0
x
1

 

2  y  =  1 
6
z
−1
using the Gauss Seidel method and the Jacobi method. Check your answer by also solving it
using row operations.
14.4. EXERCISES
291
5. Solve the system

5 0

 1 7
0 2

  
1
x
1

  
1  y  =  7 
4
z
3
using the Gauss Seidel method and the Jacobi method. Check your answer by also solving it
using row operations.
6. Solve the system

5 0

 1 7
0 2

  
1
x
1

  
1  y  =  1 
9
z
0
using the Gauss Seidel method and the Jacobi method. Check your answer by also solving it
using row operations.
7. If you are considering a system of the form Ax = b and A−1 does not exist, will either the
Gauss Seidel or Jacobi methods work? Explain. What does this indicate about using either of
these methods for finding eigenvectors for a given eigenvalue?
8. Verify that
∥x∥∞ ≡ max {|xi | , i = 1, · · · , n : x = (x1 , · · · , xn )}
is a norm. Next verify that
∥x∥1 ≡
n
∑
|xi | , x = (x1 , · · · , xn )
i=1
is also a norm on Fn .
9. Let A be an n × n matrix. Denote by ∥A∥2 the operator norm taken with respect to the usual
norm on Fn . Show that
∥A∥2 = σ 1
where σ 1 is the largest singular value. Next explain why A−1 2 = 1/σ n where σ n is the
smallest singular value of A. Explain why the condition number reduces to σ 1 /σ n if the
)1/2
(∑
n
2
.
operator norm is defined in terms of the usual norm, |x| =
j=1 |xj |
10. Let p, q > 1 and 1/p + 1/q = 1. Consider the following picture.
x
b
x = tp−1
t = xq−1
t
a
Using elementary calculus, verify that for a, b > 0,
bq
ap
+
≥ ab.
p
q
11. ↑For p > 1, the p norm on Fn is defined by
(
∥x∥p ≡
n
∑
k=1
)1/p
|xk |
p
292
CHAPTER 14. NUMERICAL METHODS FOR SOLVING LINEAR SYSTEMS
In fact, this is a norm and this will be shown in this and the next problem. Using the above
problem in the context stated there where p, q > 1 and 1/p+1/q = 1, verify Holder’s inequality
n
∑
|xk | |yk | ≤ ∥x∥p ∥y∥q
k=1
Hint: You ought to consider the following.
n
∑
|xk | |yk |
∥x∥p ∥y∥q
k=1
Now use the result of the above problem.
12. ↑ Now for p > 1, verify that ∥x + y∥p ≤ ∥x∥p + ∥y∥p . Then verify the other axioms of a norm.
This will give an infinite collection of norms for Fn . Hint: You might do the following.
p
∥x + y∥p
≤
=
n
∑
k=1
n
∑
p−1
(|xk | + |yk |)
p−1
|xk | +
|xk + yk |
|xk + yk |
k=1
n
∑
|xk + yk |
p−1
|yk |
k=1
Now explain why p − 1 = p/q and use the Holder inequality.
13. This problem will reveal the best kept secret in undergraduate mathematics, the definition of
the derivative of a function of n variables. Let ∥·∥ be a norm on Fn and also denote by ∥·∥ a
norm on Fm . If you like, just use the standard norm on both Fn and Fm . It can be shown that
this doesn’t matter at all (See Problem 25 on 361.) but to avoid possible confusion, you can
be specific about the norm. A set U ⊆ Fn is said to be open if for every x ∈ U, there exists
some rx > 0 such that B (x, rx ) ⊆ U where
B (x, r) ≡ {y ∈ Fn : ∥y − x∥ < r}
This just says that if U contains a point x then it contains all the other points sufficiently near
to x. Let f : U 7→ Fm be a function defined on U having values in Fm . Then f is differentiable
at x ∈ U means that there exists an m × n matrix A such that for every ε > 0, there exists a
δ > 0 such that whenever 0 < ∥v∥ < δ, it follows that
∥f (x + v) − f (x) − Av∥
<ε
∥v∥
Stated more simply,
∥f (x + v) − f (x) − Av∥
=0
∥v∥
∥v∥→0
lim
Show that A is unique and verify that the ith column of A is
∂f
(x)
∂xi
so in particular, all partial derivatives exist. This unique m × n matrix is called the derivative
of f . It is written as Df (x) = A.
Chapter 15
Numerical Methods For Solving
The Eigenvalue Problem
15.1
The Power Method For Eigenvalues
This chapter presents some simple ways to find eigenvalues and eigenvectors. It is only an introduction to this important subject. However, I hope to convey some of the ideas which are used.
As indicated earlier, the eigenvalue eigenvector problem is extremely difficult. Consider for example
what happens if you find an eigenvalue approximately. Then you can’t find an approximate eigenvector by the straight forward approach because A − λI is invertible whenever λ is not exactly equal
to an eigenvalue.
The power method allows you to approximate the largest eigenvalue and also the eigenvector
which goes with it. By considering the inverse of the matrix, you can also find the smallest eigenvalue.
The method works in the situation of a nondefective matrix A which has a real eigenvalue of algebraic
multiplicity 1, λn which has the property that |λk | < |λn | for all k ̸= n. Such an eigenvalue is called
a dominant eigenvalue.
Let {x1 , · · · , xn } be a basis of eigenvectors for Fn such that Axn = λn xn . Now let u1 be some
nonzero vector. Since {x1 , · · · , xn } is a basis, there exists unique scalars, ci such that
u1 =
n
∑
ck xk .
k=1
Assume you have not been so unlucky as to pick u1 in such a way that cn = 0. Then let Auk = uk+1
so that
n−1
∑
m
um = Am u1 =
ck λm
(15.1)
k xk + λn cn xn .
k=1
λm
n cn xn ,
For large m the last term,
determines quite well the direction of the vector on the right.
∑n−1
This is because |λn | is larger than |λk | for k < n and so for a large m, the sum, k=1 ck λm
k xk , on
the right is fairly insignificant. Therefore, for large m, um is essentially a multiple of the eigenvector
xn , the one which goes with λn . The only problem is that there is no control of the size of the
vectors um . You can fix this by scaling. Let S2 denote the entry of Au1 which is largest in absolute
value. We call this a scaling factor. Then u2 will not be just Au1 but Au1 /S2 . Next let S3 denote
the entry of Au2 which has largest absolute value and define u3 ≡ Au2 /S3 . Continue this way. The
scaling just described does not destroy the relative insignificance of the term involving a sum in
15.1. Indeed it amounts to nothing more than changing the units of length. Also note that from this
scaling procedure, the absolute value of the largest element of uk is always equal to 1. Therefore,
for large m,
λm
n cn xn
+ (relatively insignificant term) .
um =
S2 S3 · · · Sm
293
294 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
Therefore, the entry of Aum which has the largest absolute value is essentially equal to the entry
having largest absolute value of
( m
)
λn cn xn
λm+1 cn xn
A
= n
≈ λn um
S2 S3 · · · Sm
S2 S3 · · · Sm
and so for large m, it must be the case that λn ≈ Sm+1 . This suggests the following procedure.
Finding the largest eigenvalue with its eigenvector.
1. Start with a vector u1 which you hope has a component in the direction of xn . The vector
T
(1, · · · , 1) is usually a pretty good choice.
2. If uk is known,
uk+1 =
Auk
Sk+1
where Sk+1 is the entry of Auk which has largest absolute value.
3. When the scaling factors, Sk are not changing much, Sk+1 will be close to the eigenvalue and
uk+1 will be close to an eigenvector.
4. Check your answer to see if it worked well.
In finding an initial vector, it is clear that if you start with a vector which isn’t too far from an
eigenvector, the process will work faster. Also, the computer is able to raise the matrix to a power
quite easily. You might start with Ap x for large p. As explained above, this will point in roughly
the right direction. Then normalize it by dividing by the largest entry and use the resulting vector
as your initial approximation. This ought to be close to an eigenvector and so the process would be
expected to converge rapidly for this as an initial choice.


5 −14 11


Example 15.1.1 Find the largest eigenvalue of A =  −4
4
−4  .
3
6
−3
I
I will use the above suggestion.

5 −14

4
 −4
3
6
15   

11
1
1. 027 1 × 1016
   

−4   1  =  −5. 135 7 × 1015 
−3
1
4. 701 8 × 1011
Now divide by the largest entry to get the initial approximation for an eigenvector




1. 027 1 × 1016
1.0
1




=
−0.500 02
 −5. 135 7 × 1015 
 = u1
16
1.
027
1
×
10
4. 701 8 × 1011
4. 577 7 × 10−5
The power method will now be applied to find the largest eigenvalue for
beginning with this vector.


 
5 −14 11
1.0
12. 001


 
4
−4  
−0.500 02
−6. 000 3
 −4
=
3
6
−3
4. 577 7 × 10−5
−2. 573 3 × 10−4
the above matrix


.
15.2. THE SHIFTED INVERSE POWER METHOD
295
Scaling this vector by dividing by the largest entry gives




12. 001
1.0



 1
=
−0.499 98
−6. 000 3
 ≡ u2


12.001
−2. 573 3 × 10−4
−2. 144 2 × 10−5
Now lets do it again.

5 −14

4
 −4
3
6

 

11
1.0
11. 999

 

−4  
−0.499 98
−5. 999 8
=

−3
−2. 144 2 × 10−5
1. 843 3 × 10−4
The new scaling factor is very close to the one just encountered. Therefore, it seems this is a
good place to stop. The eigenvalue is approximately 11. 999 and the eigenvector is close to the one
obtained above. How well does it work? With the above equation, consider

 

1.0
11. 999


 
11. 999 
−5. 999 3
−0.499 98

=
−2. 572 8 × 10−4
−2. 144 2 × 10−5
These are clearly very close so this is a good approximation. In fact, the exact eigenvalue is 12 and
an eigenvector is


1.0


 −0.5 
0
15.2
The Shifted Inverse Power Method
This method can find various eigenvalues and eigenvectors. It is a significant generalization of the
above simple procedure and yields very good results. The situation is this: You have a number α
which is close to λ, some eigenvalue of an n × n matrix A. You don’t know λ but you know that
α is closer to λ than to any other eigenvalue. Your problem is to find both λ and an eigenvector
which goes with λ. Another way to look at this is to start with α and seek the eigenvalue λ, which
is closest to α along with an eigenvector associated with λ. If α is an eigenvalue of A, then you have
−1
what you want. Therefore, we will always assume α is not an eigenvalue of A and so (A − αI)
exists. When using this method it is nice to choose α fairly close to an eigenvalue. Otherwise, the
method will converge slowly. In order to get some idea where to start, you could use Gerschgorin’s
theorem to get a rough idea where to look. The method is based on the following lemma.
n
Lemma 15.2.1 Let {λk }k=1 be the eigenvalues of A, α not an eigenvalue. Then xk is an eigenvector
−1
of A for the eigenvalue λk , if and only if xk is an eigenvector for (A − αI) corresponding to the
1
eigenvalue λk −α .
Proof: Let λk and xk be as described in the statement of the lemma. Then
(A − αI) xk = (λk − α) xk
if and only if
1
−1
xk = (A − αI) xk . λk − α
In explaining why the method works, we will assume A is nondefective. This is not necessary!
One can use Gelfand’s theorem on the spectral radius which is presented in [13] and invariance of
296 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
−1
(A − αI) on generalized eigenspaces to prove more general results. It suffices to assume that the
eigenspace for λk has dimension equal to the multiplicity of the eigenvalue λk but even this is not
necessary to obtain convergence of the method. This method is better than might be supposed from
the following explanation.
Pick u1 , an initial vector and let Axk = λk xk , where {x1 , · · · , xn } is a basis of eigenvectors
which exists from the assumption that A is nondefective. Assume α is closer to λn than to any other
eigenvalue. Since A is nondefective, there exist constants, ak such that
u1 =
n
∑
ak xk .
k=1
Possibly λn is a repeated eigenvalue. Then combining the terms in the sum which involve eigenvectors
for λn , a simpler description of u1 is
u1 =
m
∑
aj xj + y
j=1
where y is an eigenvector for λn which is assumed not equal to 0. (If you are unlucky in your choice
for u1 , this might not happen and things won’t work.) Now the iteration procedure is defined as
uk+1 ≡
where Sk+1 is the element of (A − αI)
−1
uk+1
=
j=1
uk
uk which has largest absolute value. From Lemma 15.2.1,
(
∑m
−1
(A − αI)
Sk+1
aj
1
λj −α
)k
(
xj +
1
λn −α
)k
y
S2 · · · Sk+1
)k 

1
(
)k
m
∑
λn −α
λn − α

aj
xj + y  .
S2 · · · Sk+1 j=1
λj − α
(
=
Now it is being assumed that λn is the eigenvalue which is closest to α and so for large k, the term,
m
∑
(
aj
j=1
λn − α
λj − α
)k
x j ≡ Ek
is very small, while for every k ≥ 1, uk is a moderate sized vector because every entry has absolute
value less than or equal to 1. Thus
(
uk+1 =
1
λn −α
)k
S2 · · · Sk+1
(Ek + y) ≡ Ck (Ek + y)
where Ek → 0, y is some eigenvector for λn , and Ck is of moderate size, remaining bounded as
k → ∞. Therefore, for large k,
uk+1 − Ck y = Ck Ek ≈ 0
−1
and multiplying by (A − αI)
(A − αI)
−1
yields
−1
uk+1 − (A − αI)
Ck y
)
1
= (A − αI) uk+1 − Ck
y
λn − α
(
)
1
−1
≈ (A − αI) uk+1 −
uk+1 ≈ 0.
λn − α
−1
(
15.2. THE SHIFTED INVERSE POWER METHOD
297
Therefore, for large k, uk is approximately equal to an eigenvector of (A − αI)
−1
(A − αI)
uk ≈
−1
. Therefore,
1
uk
λn − α
and so you could take the dot product of both sides with uk and approximate λn by solving the
following for λn .
−1
(A − αI) uk · uk
1
=
2
λ
n−α
|uk |
How else can you find the eigenvalue from this? Suppose uk = (w1 , · · · , wn )
construction |wi | ≤ 1 and wk = 1 for some k. Then
Sk+1 uk+1 = (A − αI)
−1
−1
uk ≈ (A − αI)
(Ck−1 y) =
T
1
1
(Ck−1 y) ≈
uk .
λn − α
λn − α
−1
Hence the entry of (A − αI) uk which has largest absolute value is approximately
is likely that you can estimate λn using the formula
Sk+1 =
and from the
1
λn −α
and so it
1
.
λn − α
−1
Of course this would fail if (A − αI) uk had consistently more than one entry having equal absolute
value, but this is unlikely.
Here is how you use the shifted inverse power method to find the eigenvalue and
eigenvector closest to α.
1. Find (A − αI)
−1
.
∑m
2. Pick u1 . It is important that u1 = j=1 aj xj + y where y is an eigenvector which goes with
the eigenvalue closest to α and the sum is in an “invariant subspace corresponding to the other
eigenvalues”. Of course you have no way of knowing whether this is so but it typically is so. If
things don’t work out, just start with a different u1 . You were phenomenally unlucky in your
choice.
3. If uk has been obtained,
−1
uk+1 =
−1
where Sk+1 is the element of (A − αI)
(A − αI)
Sk+1
uk
uk which has largest absolute value.
4. When the scaling factors, Sk+1 are not changing much and the uk are not changing much, find
the approximation to the eigenvalue by solving
Sk+1 =
1
λ−α
for λ. The eigenvector is approximated by uk+1 .
5. Check your work by multiplying by the original matrix to see how well what you have found
works.
−1
Also note that this is just the power method applied to (A − λI) . The eigenvalue you want is
1
the one which makes λ−α
as large as possible for all λ ∈ σ (A). This is because making λ − α small
−1
is the same as making (λ − α) large.
298 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM

5 −14

Example 15.2.2 Find the eigenvalue of A =  −4
4
3
6
find an eigenvector which goes with this eigenvalue.

11

−4  which is closest to −7. Also
−3
In this case the eigenvalues are −6, 0, and 12 so the correct answer is −6 for the eigenvalue.



−1
5 −14 11
1 0 0




4
−4  + 7  0 1 0 
 −4
3
6
−3
0 0 1


0.511 28
0.917 29
−0.488 72


=  3. 007 5 × 10−2 0.112 78 3. 007 5 × 10−2 
−0.428 57
−0.857 14
0.571 43
To get a good initial vector, follow the shortcut described earlier which works for the power method.

0.511 28

 3. 007 5 × 10−2
−0.428 57

0.999 99

=  4. 871 3 × 10−20
−0.999 99
23  
1
0.917 29
−0.488 72

−2  
0.112 78 3. 007 5 × 10
  1 
−0.857 14
0.571 43
1



That middle term looks a lot like 0 so for an initial guess I will simply take it equal to 0. Also the
other term is very close to 1. Therefore, I will take it to equal 1. Then





= 
0.511 28
3. 007 5 × 10−2
−0.428 57

1.0

0 
−1.0


1
−0.488 72


3. 007 5 × 10−2   0 
0.571 43
−1
0.917 29
0.112 78
−0.857 14
It looks like we have found an eigenvector for (A + 7I)
−1
. Then to find the eigenvalue desired, solve
1
=1
λ+7
This yields λ = −6 which is actually the exact eigenvalue closest

1 2

Example 15.2.3 Consider the symmetric matrix A =  2 1
3 4
and an eigenvector which goes with it.
Since A is symmetric, it follows it has three
 
1
 
p (λ) = det λ  0
0
real eigenvalues
 
0 0
1
 
1 0 − 2
0 1
3
= λ3 − 4λ2 − 24λ − 17 = 0
to −7.

3

4  . Find the middle eigenvalue
2
which are solutions to

2 3

1 4 
4 2
15.2. THE SHIFTED INVERSE POWER METHOD
299
If you use your graphing calculator to graph this polynomial, you find there is an eigenvalue somewhere between −.9 and −.8 and that this is the middle eigenvalue. Of course you could zoom in
and find it very accurately without much trouble but what about the eigenvector which goes with
it? If you try to solve

 
 
 


0
x
1 2 3
1 0 0
 

 
 


(−.8)  0 1 0  −  2 1 4   y  =  0 
0
z
3 4 2
0 0 1
there will be only the zero solution because the matrix on the left will be invertible and the same
will be true if you replace −.8 with a better approximation like −.86 or −.855. This is because all
these are only approximations to the eigenvalue and so the matrix in the above is nonsingular for
all of these. Therefore, you will only get the zero solution and
Eigenvectors are never equal to zero!
However, there exists such an eigenvector and you can find it using the shifted inverse power method.
Pick α = −.855. Then
−1 




−367. 5 215. 96
83. 601
1 2 3
1 0 0






 2 1 4  + .855  0 1 0  =  215. 96 −127. 17 −48. 753 
0 0 1
83. 601 −48. 753 −19. 191
3 4 2
Next use the power method for this matrix.

−367. 5 215. 96
83. 601

 215. 96 −127. 17 −48. 753
83. 601 −48. 753 −19. 191

−2. 282 2 × 1034

 1. 341 5 × 1034
5. 183 7 × 1033
13 



1
 
 1 =
1



Now divide by the largest entry to get a good first approximation to the eigenvector.




−2. 282 2 × 1034
1.0
1




=  −0.587 81 
 1. 341 5 × 1034 
34
−2.
282
2
×
10
5. 183 7 × 1033
−0.227 14

−367. 5

 215. 96
83. 601

 

215. 96
83. 601
1.0
−513. 43

 

−127. 17 −48. 753   −0.587 81  =  301. 79 
−48. 753 −19. 191
−0.227 14
116. 62
Now divide by the largest entry to get the next approximation




−513. 43
1.0
1




=  −0.587 79 
 301. 79 
−513. 43
116. 62
−0.227 14
Clearly this vector has not changed much and so the next scaling factor will be very close to the one
just considered. Hence the eigenvalue will be determined by solving
1
= −513. 43
λ + .855
300 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
It follows the approximate eigenvalue is about
λ = −0.856 95
The approximate eigenvector is
How well does it work?


1.0


 −0.587 81 
−0.227 14


 

−0.857 04
1.0
2 3

 

1 4   −0.587 81  =  0.503 63 
0.194 48
−0.227 14
4 2

 

−0.856 95
1.0

 

−0.856 95  −0.587 81  =  0.503 72 
−0.227 14
0.194 65
1

 2
3
This is pretty close. If you wanted to get closer, you could simply do more iterations. For practical
purposes, the eigenvalue and eigenvector have been obtained.
There is an easy to use trick which will eliminate some of the fuss and bother in using the shifted
inverse power method. If you have
−1
(A − αI) x = µx
then multiplying through by (A − αI) , one finds that x will be an eigenvector for A with eigenvalue
−1
α + µ−1 . Hence you could simply take (A − αI) to a high power and multiply by a vector to get
a vector which points in the direction of an eigenvalue of A. Then divide by the largest entry and
identify the eigenvalue directly by multiplying the eigenvector by A. This is illustrated in the next
example.
Example 15.2.4 Find the eigenvalues and eigenvectors of the matrix


2 1 3


A =  2 1 1 .
3 2 1
This is only a 3×3 matrix and so it is not hard to estimate the eigenvalues. Just get the
characteristic equation, graph it using a calculator and zoom in to find the eigenvalues. If you do
this, you find there is an eigenvalue near −1.2, one near −.4, and one near 5.5. (The characteristic
equation is 2 + 8λ + 4λ2 − λ3 = 0.) Of course we have no idea what the eigenvectors are.
Lets first try to find the eigenvector and a better approximation for the eigenvalue near −1.2. In
this case, let α = −1.2. Then


−25. 357 143 −33. 928 571 50.0


−1
(A − αI) = 
12. 5
17. 5
−25.0  .
23. 214 286
30. 357 143 −45.0
Then

−25. 357 143 −33. 928 571

12. 5
17. 5

23. 214 286
30. 357 143


−4. 943 2 × 1028


=  2. 431 2 × 1028 
4. 492 8 × 1028
17  
1
50.0
  
−25.0   1 
1
−45.0
15.2. THE SHIFTED INVERSE POWER METHOD
301
The initial approximation for an eigenvector will then be the above divided



−4. 943 2 × 1028
1.0
1


28 
=  −0.491 83

 2. 431 2 × 10
−4. 943 2 × 1028
28
4. 492 8 × 10
−0.908 88
by its largest entry.



How close is this to being an eigenvector?

 


−1. 218 5
1.0
2 1 3

 


 2 1 1   −0.491 83  =  0.599 29 
1. 107 5
−0.908 88
3 2 1

 

1.0
−1. 218 5

 

−1. 218 5  −0.491 83  =  0.599 29 
−0.908 88
1. 107 5
For all practical purposes, this has found the eigenvector and eigenvalue of −1. 218 5.
Next we shall find the eigenvector and a more precise value for the eigenvalue near −.4. In this
case,


8. 064 516 1 × 10−2 −9. 274 193 5 6. 451 612 9


−1
(A − αI) = 
−. 403 225 81
11. 370 968 −7. 258 064 5  .
. 403 225 81
3. 629 032 3 −2. 741 935 5
The first approximation to an eigenvector can be obtained as before.





= 
8. 064 516 1 × 10−2 −9. 274 193 5
−. 403 225 81
11. 370 968
. 403 225 81
3. 629 032 3

−1. 853 5 × 1016

2. 372 4 × 1016 
6. 287 4 × 1015
17  
6. 451 612 9
1
  
−7. 258 064 5   1 
−2. 741 935 5
1
The first choice for an approximate eigenvector is




−1. 853 5 × 1016
−0.781 28
1




=
1.0
 2. 372 4 × 1016 

16
2.
372
4
×
10
6. 287 4 × 1015
0.265 02
Lets see how well this works as an eigenvector.

 


0.232 5
−0.781 28
2 1 3

 


1.0
 =  −0.297 54 
 2 1 1 
−0.078 82
0.265 02
3 2 1

 

−0.781 28
0.232 46

 

(−0.297 54) 
1.0
−0.297 54
=

−2
0.265 02
−7. 885 4 × 10
Thus this works as an eigenvector with the eigenvalue (−0.297 54).
Next we will find the eigenvalue and eigenvector for the eigenvalue near 5.5. In this case,


29. 2 16. 8 23. 2


−1
(A − αI) =  19. 2 10. 8 15. 2  .
28.0 16.0 22.0
302 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
As before, I have no idea what the eigenvector is but to avoid giving the impression that you always
T
T
need to start with the vector (1, 1, 1) , let u1 = (1, 2, 3) . I will use the same shortcut to get this
eigenvector as in the above case.

16 
 

1. 098 7 × 1029
1
29. 2 16. 8 23. 2


 
 
 19. 2 10. 8 15. 2   2  =  7. 186 8 × 1028 
1. 048 2 × 1029
28.0 16.0 22.0
3
Then dividing by the largest entry, a good guess for the eigenvector is


1.0


 0.654 12 
0.954 04
To see if more iteration would be needed, check this.

 


5. 516 2
2 1 3
1.0

 


 2 1 1   0.654 12  =  3. 608 2 
0.954 04
5. 262 3
3 2 1

 

1.0
5. 516 2

 

5. 516 2  0.654 12  =  3. 608 3 
0.954 04
5. 262 7
and
Thus this is essentially an eigenvector with eigenvalue equal to 5. 516 2.
15.2.1
Complex Eigenvalues
What about complex eigenvalues? If your matrix is real, you won’t see these by graphing the characteristic equation on your calculator. Will the shifted inverse power method find these eigenvalues
and their associated eigenvectors? The answer is yes. However, for a real matrix, you must pick α to
be complex. This is because the eigenvalues occur in conjugate pairs, so if you don’t pick it complex,
it will be the same distance between any conjugate pair of complex numbers and so nothing in the
above argument for convergence implies you will get convergence to a complex number. Also, the
process of iteration will yield only real vectors and scalars.
Example 15.2.5 Find the complex eigenvalues and corresponding eigenvectors for the matrix


5 −8 6


 1 0 0 .
0 1 0
Here the characteristic equation is λ3 − 5λ2 + 8λ − 6 = 0. One solution is λ = 3. The other two
are 1 + i and 1 − i. Apply the process to α = i so we will find the eigenvalue closest to i.


−.0 2 − . 14i 1. 24 + . 68i −. 84 + . 12i


−1
(A − αI) =  −. 14 + .0 2i . 68 − . 24i
. 12 + . 84i 
.0 2 + . 14i −. 24 − . 68i . 84 + . 88i
I shall use the trick of raising to a high power to begin with.
15  
1
−.0 2 − . 14i 1. 24 + . 68i −. 84 + . 12i

  
. 12 + . 84i   1 
 −. 14 + .0 2i . 68 − . 24i
1
.0 2 + . 14i −. 24 − . 68i . 84 + . 88i

15.3. THE RAYLEIGH QUOTIENT
303

−0.4 + 0.8i


=  0.2 + 0.6i 
0.4 + 0.2i

Now normalize by dividing by the largest entry to get a good initial vector.




−0.4 + 0.8i
1.0
1




=  0.5 − 0.5i 
 0.2 + 0.6i 
−0.4 + 0.8i
0.4 + 0.2i
−0.5i
Then

 

1.0
1.0
−.0 2 − . 14i 1. 24 + . 68i −. 84 + . 12i

 


. 12 + . 84i   0.5 − 0.5i  =  0.5 − 0.5i 
 −. 14 + .0 2i . 68 − . 24i
−0.5i
−0.5i
.0 2 + . 14i −. 24 − . 68i . 84 + . 88i

It looks like this has found the eigenvector exactly. Then to get the eigenvalue, you need to solve
1
=1
λ−i
Thus λ = 1 + i. How well does the

5 −8

 1 0
0 1
above vector work?

 

6
1.0
1.0 + 1.0i

 

0   0.5 − 0.5i  = 
1.0

0
−0.5i
0.5 − 0.5i
 


1.0
1.0 + 1.0i

 

(1 + i)  0.5 − 0.5i  = 
1.0

−0.5i
0.5 − 0.5i
It appears that this has found the exact answer.
This illustrates an interesting topic which leads to many related topics. If you have a polynomial,
x4 + ax3 + bx2 + cx + d, you can consider it as the characteristic polynomial of a certain matrix,
called a companion matrix. In this case,


−a −b −c −d
 1
0
0
0 



.
 0
1
0
0 
0
0
1
0
The above example was just a companion matrix for λ3 − 5λ2 + 8λ − 6. You can see the pattern
which will enable you to obtain a companion matrix for any polynomial of the form λn + a1 λn−1 +
· · · + an−1 λ + an . This illustrates that one way to find the complex zeros of a polynomial is to use
the shifted inverse power method on a companion matrix for the polynomial. Doubtless there are
better ways but this does illustrate how impressive this procedure is. Do you have a better way?
15.3
The Rayleigh Quotient
There are many specialized results concerning the eigenvalues and eigenvectors for Hermitian matrices. A matrix A is Hermitian if A = A∗ where A∗ means to take the transpose of the conjugate
of A. In the case of a real matrix, Hermitian reduces to symmetric. Recall also that for x ∈ Fn ,
|x| = x∗ x =
2
n
∑
j=1
2
|xj | .
304 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
The following corollary gives the theoretical foundation for the spectral theory of Hermitian
matrices. This is a corollary of a theorem which is proved Corollary 13.2.14 and Theorem 13.2.14
on Page 258.
Corollary 15.3.1 If A is Hermitian, then all the eigenvalues of A are real and there exists an
orthonormal basis of eigenvectors.
n
Thus for {xk }k=1 this orthonormal basis,
{
x∗i xj = δ ij ≡
1 if i = j
0 if i ̸= j
For x ∈ Fn , x ̸= 0, the Rayleigh quotient is defined by
x∗ Ax
2
|x|
.
n
Now let the eigenvalues of A be λ1 ≤ λ2 ≤ · · · ≤ λn and Axk = λk xk where {xk }k=1 is the above
orthonormal basis of eigenvectors mentioned in the corollary. Then if x is an arbitrary vector, there
exist constants, ai such that
n
∑
x=
ai xi .
i=1
Also,
|x|
2
=
n
∑
ai x∗i
i=1
=
∑
n
∑
aj xj
j=1
ai aj x∗i xj
=
ij
∑
ai aj δ ij =
ij
n
∑
2
|ai | .
i=1
Therefore,
x Ax
|x|
2
)
(∑
n
∗
a
λ
x
a
x
)
j=1 j j j
i=1 i i
∑n
2
i=1 |ai |
∑
∑
∗
ij ai aj λj xi xj
ij ai aj λj δ ij
=
∑n
∑n
2
2
i=1 |ai |
i=1 |ai |
∑n
2
i=1 |ai | λi
∑n
2 ∈ [λ1 , λn ] .
i=1 |ai |
(
∗
=
=
=
∑n
In other words, the Rayleigh quotient is always between the largest and the smallest eigenvalues of
A. When x = xn , the Rayleigh quotient equals the largest eigenvalue and when x = x1 the Rayleigh
quotient equals the smallest eigenvalue. Suppose you calculate a Rayleigh quotient. How close is it
to some eigenvalue?
Theorem 15.3.2 Let x ̸= 0 and form the Rayleigh quotient,
x∗ Ax
|x|
2
≡ q.
Then there exists an eigenvalue of A, denoted here by λq such that
|λq − q| ≤
|Ax − qx|
.
|x|
(15.2)
15.3. THE RAYLEIGH QUOTIENT
Proof: Let x =
∑n
305
n
k=1
|Ax − qx|
ak xk where {xk }k=1 is the orthonormal basis of eigenvectors.
2
∗
= (Ax − qx) (Ax − qx)
( n
)∗ ( n
)
∑
∑
=
ak λk xk − qak xk
ak λk xk − qak xk
k=1

= 
n
∑
(
(λj −
q) aj x∗j 
j=1
=
∑
k=1
n
∑
)
(λk − q) ak xk
k=1
(λj − q) aj (λk − q) ak x∗j xk
j,k
=
n
∑
2
|ak | (λk − q)
2
k=1
Now pick the eigenvalue, λq which is closest to q. Then
2
|Ax − qx| =
n
∑
2
2
2
|ak | (λk − q) ≥ (λq − q)
k=1
n
∑
2
2
|ak | = (λq − q) |x|
2
k=1
which implies 15.2. 
1 2

Example 15.3.3 Consider the symmetric matrix A =  2 2
3 1

3

1  . Let
4
T
x = (1, 1, 1) .
How close is the Rayleigh quotient to some eigenvalue of A? Find the eigenvector and eigenvalue to
several decimal places.
Everything is real and so there is no need
Rayleigh quotient is

1
(
)

1 1 1  2
3
to worry about taking conjugates. Therefore, the
2
2
1
 
3
1
 
1  1 
4
1
3
=
19
3
According to the above theorem, there is some eigenvalue of this matrix, λq such that
 

 
1
1 2 3
1

   19  
 2 2 1  1  − 3  1 
1
3 1 4
1
19
√
λq −
≤
3
3


1
−3

1 
 4 
= √  −3 

3
5
3
√
=
1
9
+
( 4 )2
3
√
3
+
( 5 )2
3
= 1. 247 2
Could you find this eigenvalue and associated eigenvector? Of course you could. This is what
the inverse shifted power method is all about.
306 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
Solve

1

 2
3


3
1 0
 19 
1 −
 0 1
3
4
0 0
2
2
1

In other words solve




− 16
3
2
2
− 13
3
3
1
 
  
0
x
1
 
  
0   y  =  1 
1
z
1

  

1
x


  
1 
 y  =  1 

1
z
−7
3
3
and divide by the entry which is largest, 3. 870 7, to get


. 699 25


u2 =  . 493 89 
1.0
Now solve





− 16
3
2
2
− 13
3
3
1

 


. 699 25
x


 

1 
  y  =  . 493 89 

z
1.0
− 73
3
and divide by the entry with largest absolute value, 2. 997 9 to get


. 714 73


u3 =  . 522 63 
1. 0
Now solve





− 16
3
2
2
− 13
3
3
1


 

x
. 714 73


 

1 
  y  =  . 522 63 

z
1. 0
− 73
3
and divide by the entry with largest absolute value, 3. 045 4, to get


. 713 7


u4 =  . 520 56 
1.0
Solve





− 16
3
2
2
− 13
3
3
1


 

x
. 713 7


 

1 
  y  =  . 520 56 

z
1.0
−7
3
3
and divide by the largest entry, 3. 042 1 to get


. 713 78


u5 =  . 520 73 
1.0
You can see these scaling factors are not changing much. The predicted eigenvalue is obtained by
solving
1
= 3. 042 1
λ − 19
3
15.4. THE QR ALGORITHM
307
to obtain λ = 6. 6621. How close

1

 2
3
is this?

 

2 3
. 713 78
4. 755 2

 

2 1   . 520 73  =  3. 469 
1 4
1.0
6. 662 1

 
4. 755 3
. 713 78

 

6. 662 1  . 520 73  =  3. 469 2  .
6. 662 1
1.0

while
You see that for practical purposes, this has found the eigenvalue and an eigenvector.
15.4
The QR Algorithm
15.4.1
Basic Considerations
The QR algorithm is one of the most remarkable techniques for finding eigenvalues. In this section,
I will discuss this method. To see more on this algorithm, consult Golub and Van Loan [6]. For an
explanation of why the algorithm works see Wilkinson [18]. There is also more discussion in Linear
Algebra. This will only discuss real matrices for the sake of simplicity. Also, there is a lot more to
this algorithm than will be presented here. First here is an introductory lemma.
Lemma 15.4.1 Suppose A is a block upper triangular matrix,


B1
∗


..

A=
.


0
Br
This means that the Bi are ri × ri matrices whose diagonals are subsets of the main diagonal of A.
Then σ (A) = ∪ri=1 σ (Bi ).
Proof: Say Q∗i Bi Qi = Ti where Ti is upper triangular. Such unitary matrices exist by Schur’s
theorem. Then consider the similarity transformation,




B1
∗
Q1
0
0
Q∗1




..
..
..




.
.
.




0
Q∗r
0
Br
0
Qr
By block multiplication this equals

Q∗1

..

.

0


=

0
Q∗r
Q∗1 B1 Q1



..
.
Q∗r Br Qr



.
0


∗
B1 Q1
∗
..
0

Br Qr

 
=
 
∗
T1
..
0
.




Tr
Now this is a real upper triangular matrix and the eigenvalues of A consist of the union of the
eigenvalues of the Ti which is the same as the union of the eigenvalues of the Bi . Here is the description of the great and glorious QR algorithm.
308 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
The QR Algorithm
Let A be an n × n real matrix. Let A0 = A. Suppose that Ak−1 has been found. To find Ak let
Ak−1 = Qk Rk , Ak = Rk Qk ,
where Qk Rk is a QR factorization of Ak−1 . Thus R is upper triangular with nonnegative entries on
the main diagonal and Q is real and unitary (orthogonal).
The main significance of this algorithm is in the following easy but important theorem.
Theorem 15.4.2 Let A be any n × n complex matrix and let {Ak } be the sequence of matrices
described above by the QR algorithm. Then each of these matrices is unitarily similar to A.
Proof: Clearly A0 is orthogonally similar to A because they are the same thing. Suppose then
that
Ak−1 = Q∗ AQ
Then from the algorithm,
Ak−1 = Qk Rk , Rk = Q∗k Ak−1
Therefore, from the algorithm,
∗
Ak ≡ Rk Qk = Q∗k Ak−1 Qk = Q∗k Q∗ AQQk = (QQk ) AQQk ,
and so Ak is unitarily similar to A also. Although the sequence {Ak } may fail to converge, it is nevertheless often the case that for large
k, Ak is of the form


Bk
∗


..

Ak = 
.


e
Br
where the Bi are blocks which run down the diagonal of the matrix, and all of the entries below
this block diagonal are very small. Then letting TB denote the matrix obtained by setting all of
these small entries equal to zero, one can argue, using methods of analysis, that the eigenvalues of
Ak are close to the eigenvalues of TB . From Lemma 15.4.1 the eigenvalues of TB are the eigenvalues
of the blocks Bi . Thus, the eigenvalues of A are the same as those of Ak and these are close to the
eigenvalues of TB .
In proving things about this algorithm and also for the sake of convenience, here is a technical
result.
Corollary 15.4.3 For Qk , Rk , Ak given in the QR algorithm,
A = Q1 · · · Qk Ak Q∗k · · · Q∗1
For Q(k) ≡ Q1 · · · Qk and R(k) ≡ Rk · · · R1 , it follows that
Ak = Q(k) R(k)
Here Ak is the usual thing, A raised to the k th power.
Proof: From the algorithm,
A = A0 = Q1 R1 , Q∗1 A0 = R1 , A1 ≡ R1 Q1 = Q∗1 AQ1
Hence
Q1 A1 Q∗1 = A
Suppose the formula 15.3 holds for k. Then from the algorithm,
Ak = Qk+1 Rk+1 , Rk+1 = Q∗k+1 Ak , Ak+1 = Rk+1 Qk+1 = Q∗k+1 Ak Qk+1
(15.3)
15.4. THE QR ALGORITHM
309
Hence Qk+1 Ak+1 Q∗k+1 = Ak and so
A = Q1 · · · Qk Ak Q∗k · · · Q∗1 = Q1 · · · Qk Qk+1 Ak+1 Q∗k+1 Q∗k · · · Q∗1
This shows the first part.
The second part is clearly true from the algorithm if k = 1. Then from the first part and the
algorithm,
I
A=
Q1 · · · Qk Qk+1 Ak+1 Q∗k+1 Q∗k
· · · Q∗1
z }| {
= Q1 · · · Qk Qk+1 Rk+1 Qk+1 Q∗k+1 Q∗k · · · Q∗1
It follows that
Ak+1
= AAk = Q1 · · · Qk Qk+1 Rk+1 Q∗k · · · Q∗1 Q(k) R(k)
(
)∗
= Q(k+1) Rk+1 Q(k) Q(k) R(k)
Hence
Ak+1 = Q(k+1) Rk+1 R(k) = Q(k+1) R(k+1) Now suppose that A−1 exists. How do two QR factorizations compare? Since A−1 exists, it
would require that if A = QR, then R−1 must exist. Now an upper triangular matrix has inverse
which is also upper triangular. This follows right away from the algorithm presented early in the
book for finding the inverse. If A = Q1 R1 = Q2 R2 , then Q∗1 Q2 = R1 R2−1 and so R1 R2−1 is an upper
triangular matrix which is also unitary and in addition has all positive entries down the diagonal.
For simplicity, call it R. Thus R is upper triangular and RR∗ = R∗ R = I. It follows easily that R
must equal I and so R1 = R2 which requires Q1 = Q2 .
Now in the above corollary, you know that
(
)∗
A = Q1 · · · Qk Ak Q∗k · · · Q∗1 = Q(k) Ak Q(k)
Also, from this corollary, you know that
Ak = Q(k) R(k)
You could also simply take the QR factorization of Ak to obtain Ak = QR. Then from what was
just pointed out, if A−1 exists,
Q(k) = Q
Thus from the above corollary,
(
)∗
Ak = Q(k) AQ(k) = Q∗ AQ
Therefore, in using the QR algorithm in the case where A has an inverse, it suffices to take
Ak = QR
and then consider the matrix
Q∗ AQ = Ak .
This is so theoretically. In practice it might not work out all that well because of round off errors.
There is also an interesting relation to the power method. Let
(
)
A = a1 · · · an
Then from the way we multiply matrices,
Ak+1 =
(
Ak a1
···
Ak an
)
310 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
and for large k, Ak ai would be expected to point roughly in the direction of the eigenvector corresponding to the largest eigenvalue. Then if you form the QR factorization,
Ak+1 = QR
the columns of Q are an orthonormal basis obtained essentially from the Gram Schmidt procedure.
Thus the first column of Q has roughly the direction of an eigenvector associated with the largest
eigenvalue of A. It follows that the first column of Q∗ AQ is approximately equal to λ1 q1 and so the
top entry will be close to λ1 q∗1 q1 = λ1 and the entries below it are close to 0. Thus the eigenvalues
of the matrix should be close to this top entry of the first column along with the eigenvalues of the
(n − 1) × (n − 1) matrix in the lower right corner. If this is a 2 × 2 you can find the eigenvalues
using the quadratic formula. If it is larger, you could just use the same procedure for finding its
eigenvalues but now you are dealing with a smaller matrix.
Example 15.4.4 Find the eigenvalues of the matrix


5
4
3


3
2 
 2
−8 −9 −6
First use the computer to raise this matrix to a large power.
27 

2. 684 4 × 108
1. 342 2 × 108
5
4 3.0



3
2  =
−2.0
−3.0
 2
8
−2. 684 4 × 10 −1. 342 2 × 108
−8 −9 −6

1. 342 2 × 108

−2.0

−1. 342 2 × 108
Now find the QR factorization of this last matrix. The Q equals


0.707 11
−3. 725 2 × 10−9
−0.707 11


−1. 000 0
1. 441 6 × 10−21 
 −5. 268 3 × 10−9
−0.707 11
3. 725 2 × 10−9
−0.707 11
Next examine

5

 2
−8
T
0.707 11
−3. 725 2 × 10−9
−0.707 11


−1. 000 0
1. 441 6 × 10−21  ·
 −5. 268 3 × 10−9
−0.707 11
3. 725 2 × 10−9
−0.707 11


4 3.0
0.707 11
−3. 725 2 × 10−9
−0.707 11


3
2   −5. 268 3 × 10−9
−1. 000 0
1. 441 6 × 10−21  =
−9 −6
−0.707 11
3. 725 2 × 10−9
−0.707 11


2.0
−9. 192 4 −11.0


3.0
2. 828 4 
 5. 268 4 × 10−9
−1. 862 6 × 10−8 −3. 535 6
−3.0

You see now that this is essentially equal to the block upper triangular matrix


2.0 −9. 192 4 −11.0


3.0
2. 828 4 
 0
0 −3. 535 6
−3.0
and the eigenvalues of this matrix can be easily identified. They are 2 and the eigenvalues of the
block
(
)
3.0
2. 828 4
−3. 535 6
−3.0
But these can be found easily using the quadratic formula. This yields i, −i. In fact the exact
eigenvalues for this matrix are 2, i, −i.
15.4. THE QR ALGORITHM
15.4.2
311
The Upper Hessenberg Form
Actually, when using the QR algorithm, contrary to what I have done above, you should always deal
with a matrix which is similar to the given matrix which is in upper Hessenberg form. This means
all the entries below the sub diagonal equal 0. Here is an easy lemma.
Lemma 15.4.5 Let A be an n × n matrix. Then it is unitarily similar to a matrix in upper Hessenberg form and this similarity can be computed.
Proof: Let A be an n × n matrix. Suppose n > 2. There is nothing to show otherwise.
(
)
a b
A=
d A1
where A1 is n − 1 × n − 1. Consider the n − 1 × 1 matrix d. Then let Q be a Householder reflection
such that
(
)
c
Qb =
≡c
0
(
Then
1 0
0 Q
)(
(
=
a b
d A1
a
c
)(
bQ∗
QA1 Q∗
1 0
0 Q∗
)
)
By similar reasoning, there exists an n − 1 × n − 1 matrix
(
)
1 0
U=
0 Q1
such that
Thus
(
1 0
0 Q1
(
(
)
QA1 Q∗
1 0
0 U
)(
a
c
1 0
0 Q∗1
bQ∗
QA1 Q∗
)

∗
···
..
.

=
 ∗
0 ···
)(
1 0
0 U∗

∗
.. 

. 
∗
)
will have all zeros below the first two entries on the sub diagonal. Continuing this way shows the
result. The reason you should use a matrix which is upper Hessenberg and similar to A in the QR
algorithm is that the algorithm keeps returning a matrix in upper Hessenberg form and if you are
looking for block upper triangular matrices, this will force the size of the blocks to be no larger than
2 × 2 which are easy to handle using the quadratic formula. This is in the following lemma.
Lemma 15.4.6 Let {Ak } be the sequence of iterates from the QR algorithm, A−1 exists. Then if
Ak is upper Hessenberg, so is Ak+1 .
Proof: The matrix is upper Hessenberg means that Aij = 0 whenever i − j ≥ 2.
Ak+1 = Rk Qk
where Ak = Qk Rk . Therefore Ak Rk−1 = Qk and so
Ak+1 = Rk Qk = Rk Ak Rk−1
312 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
Let the ij th entry of Ak be akij . Then if i − j ≥ 2
ak+1
=
ij
j
n ∑
∑
−1
rip akpq rqj
p=i q=1
It is given that akpq = 0 whenever p − q ≥ 2. However, from the above sum,
p − q ≥ i − j ≥ 2,
and so the sum equals 0. Example 15.4.7 Find the solutions to the equation x4 − 4x3 + 8x2 − 8x + 4 = 0 using the QR
algorithm.
This is the characteristic equation of the matrix

4 −8 8
1 0 0
0 1 0
0 0 1




−4
0
0
0





Since the constant term in the equation is not 0, it follows that the matrix has an inverse. It is
already in upper Hessenberg form. Lets apply the algorithm.










−7. 516 2 × 109
−7. 516 2 × 109
−3. 758 1 × 109
−6. 710 9 × 107
4 −8 8
1 0 0
0 1.0 0
0 0 1
3. 033 3 × 1010
2. 254 9 × 1010
7. 516 2 × 109
−3. 489 7 × 109
−4
0
0
0
55




=
−4. 509 7 × 1010
−2. 979 6 × 1010
−7. 516 2 × 109
6. 979 3 × 109
3. 006 5 × 1010
1. 503 2 × 1010
2. 684 4 × 108
−6. 979 3 × 109





Then when you take the QR factorization of this, you find Q =



Q=

−0.666 65
−0.666 65
−0.333 33
−5. 952 3 × 10−3
Then you look at
0.605 55
−0.305 4
−0.592 53
−0.434 68



QT 

which yields





0.652 78
0.757 89
−9. 699 × 10−6
1. 635 5 × 10−5
4
1
0
0
−8
0
1.0
0
−0.407 45
0.446 60
−6. 411 2 × 10−2
−0.793 99
0.151 21
−0.512 69
0.730 54
−0.424 96

8 −4
0 0 

Q
0 0 
1 0
−1. 540 9
1. 382 3
1. 043 4 × 10−3
7. 249 8 × 10−5
1. 816 1
−1. 650 1
0.875 04
0.178 40
−8. 101 1
7. 358 4
−5. 471 1
1. 089 9










15.4. THE QR ALGORITHM
313
Of course the entries in the bottom left should all be 0. They aren’t because of round off error. The
only other entry of importance is 1. 043 4 × 10−3 which is small. Hence the eigenvalues are close to
the eigenvalues of the two blocks
(
) (
)
0.875 04 −5. 471 1
0.652 78 −1. 540 9
,
0.757 89 1. 382 3
0.178 40 1. 089 9
This yields
0.982 47 + 0.982 09i, 0.982 47 − 0.982 09i
1. 017 5 + 1. 017 2i,
1. 017 5 − 1. 017 2i
The real solutions are 1 + i, 1 + i, 1 − i, and 1 − i. You could of course use the shifted inverse power
method to get closer and to also obtain eigenvectors for the matrix.
Example 15.4.8 Find the eigenvalues for the symmetric matrix


1 2 3 −1
 2 0 1 3 


A=

 3 1 3 2 
−1 3 2 1
Also find an eigenvector.
I should work with a matrix which is upper Hessenberg which is similar to the given matrix.
However, I don’t feel like it. It is easier to just raise to a power and see if things work. This is what
I will do.










1. 209 1 × 1028
1. 118 8 × 1028
1. 876 9 × 1028
1. 045 8 × 1028
1
2
3
−1
2 3
0 1
1 3
3 2
1. 118 8 × 1028
1. 035 3 × 1028
1. 736 9 × 1028
9. 677 4 × 1027
Now take the QR factorization of

0.446 59
 0.413 24


 0.693 24
0.386 27
−1
3
2
1
35




=
1. 876 9 × 1028
1. 736 9 × 1028
2. 913 7 × 1028
1. 623 5 × 1028
1. 045 8 × 1028
9. 677 4 × 1027
1. 623 5 × 1028
9. 045 5 × 1027





this. When you do, Q =
−0.737 57
−0.106 17
0.640 97
−0.184 03
−0.435 04 0.259 39
0.825 07
0.370 43
−0.312 12 0.105 57
0.180 47 −0.885 64





Thus you look at QT AQ, a matrix which is similar to A and which equals





6. 642 9
1. 637 9 × 10−4
1. 195 5 × 10−4
1. 335 6 × 10−4
1. 637 9 × 10−4
−1. 475 1
−1. 134 9
−1. 637 5
1. 195 5 × 10−4
−1. 134 9
0.203 11
−2. 507 7
1. 335 6 × 10−4
−1. 637 5
−2. 507 7
−0.371 01





It follows that the eigenvalues are approximately 6. 642 9 and the eigenvalues of the 3 × 3 matrix in
the lower right corner. I shall use the same technique to find its eigenvalues.
314 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM

B 37

−1. 475 1

=  −1. 134 9
−1. 637 5
−1. 738 6 × 1022

 −1. 483 9 × 1022
−1. 760 5 × 1022
−1. 134 9
0.203 11
−2. 507 7
37
−1. 637 5

−2. 507 7  =
−0.371 01

−1. 760 5 × 1022

−1. 502 6 × 1022 
−1. 782 7 × 1022
−1. 483 9 × 1022
−1. 266 5 × 1022
−1. 502 6 × 1022
Then take the QR factorization of this to get Q =

−0.602 6 −6. 115 8 × 10−2

0.792 13
 −0.514 32
−0.610 20
−0.607 28
Then you look at QT BQ which equals

−4. 101 8

 −3. 848 5 × 10−5
2. 023 4 × 10−5

0.795 69

−0.328 63 
−0.508 80
−3. 848 5 × 10−5
2. 386 1
0.416 67

2. 023 4 × 10−5

0.416 67

−2
7. 275 9 × 10
Thus the eigenvalues are approximately 6. 642 9, −4. 101 8, and the eigenvalues of the matrix in the
lower right corner in the above. These are 2. 458 9, −1. 480 0 × 10−6 . In fact, 0 is an eigenvalue of the
original matrix and it is being approximated by −1. 480 0 × 10−6 . To summarize, the eigenvalues are
approximately
6. 642 9, −4. 101 8, 2. 458 9, 0
Of course you could now use the shifted inverse power method to find these more exactly if
desired and also to find the eigenvectors.
If you wanted to find an eigenvector, you could start with one of these approximate eigenvalues
and use the shifted inverse power method to get an eigenvector. For example, pick 6. 642 9.





2 3 −1
0 1 3
1 3 2
3 2 1

3189. 3
 2951. 5

=
 4951. 3
2758. 8
1
2
3
−1








= 







 − 6. 642 9 


2951. 5
2731. 1
4581. 9
2552. 9
3189. 3
2951. 5
4951. 3
2758. 8
2951. 5
2731. 1
4581. 9
2552. 9

9. 061 9 × 1020
8. 385 7 × 1020 


1. 406 8 × 1021 
7. 838 1 × 1020
1
0
0
0
0
1
0
0
0
0
1
0
−1
0

0 


0 
1

4951. 3 2758. 8
4581. 9 2552. 9 


7686. 2 4282. 7 
4282. 7 2386.0
5 
4951. 3 2758. 8
1


4581. 9 2552. 9   1
 
7686. 2 4282. 7   1
4282. 7 2386.0
1





15.5. EXERCISES
315
So try for the next approximation

9. 061 9 × 1020
8. 385 7 × 1020
1. 406 8 × 1021
7. 838 1 × 1020












= 





1


=

21
 1. 406 8 × 10

0.644 15
0.596 08
1.0
0.557 16






3189. 3 2951. 5 4951. 3 2758. 8
0.644 15


2951. 5 2731. 1 4581. 9 2552. 9   0.596 08

4951. 3 4581. 9 7686. 2 4282. 7  
1.0
2758. 8 2552. 9 4282. 7 2386.0
0.557 16

10302.
9533. 4 


15993. 
8910. 9
Next one is

1 


15993 
10302.
9533. 4
15993.
8910. 9


 
 
=
 
0.644 16
0.596 10
1.0
0.557 18










This isn’t changing by much from the above vector and so the scaling factor will be about 15993.
Now solve
1
= 15993
λ − 6. 642 9
The solution is 6. 643 0. The approximate eigenvector is what I just got. Lets check it.





1
2
3
−1

 
−1
0.644 16
4. 279 2




3   0.596 10   3. 959 9

=
  6. 642 9
2 
1.0
1
0.557 18
3. 701 3

 

0.644 16
4. 279 2
 0.596 10   3. 959 9 

 

6. 643 0 
=


  6. 643 
1.0
2 3
0 1
1 3
3 2
0.557 18





3. 701 3
This is clearly very close. This has essentially found the eigenvalues and an eigenvector for the
largest eigenvalue.
15.5
Exercises
1. Using the power method, find the
 eigenvalue correct
 to one decimal place having largest abso0 −4 −4


lute value for the matrix A =  7
10
5  along with an eigenvector associated with
−2 0
6
this eigenvalue.
316 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
2. Using the power method, find the
correct to one decimal place having largest abso eigenvalue
15 6 1


lute value for the matrix A =  −5 2 1  along with an eigenvector associated with this
1 2 7
eigenvalue.
3. Using the power method, find the
 eigenvalue correct
 to one decimal place having largest ab10 4 2


solute value for the matrix A =  −3 2 −1  along with an eigenvector associated with
0 0 4
this eigenvalue.
4. Using the power method, find
the eigenvalue
15
14

lute value for the matrix A =  −13 −18
5
10
this eigenvalue.
correct
 to one decimal place having largest abso−3

9  along with an eigenvector associated with
−1
5. In Example 15.3.3 an eigenvalue was found correct to several decimal places along with an
eigenvector. Find the other eigenvalues along with their eigenvectors.


3 2 1


6. Find the eigenvalues and eigenvectors of the matrix A =  2 1 3  numerically. In this
1 3 2
√
case the exact eigenvalues are ± 3, 6. Compare with the exact answers.


3 2 1


7. Find the eigenvalues and eigenvectors of the matrix A =  2 5 3  numerically. The exact
1 3 2
√
√
eigenvalues are 2, 4 + 15, 4 − 15. Compare your numerical results with the exact values. Is
it much fun to compute the exact eigenvectors?


0 2 1


8. Find the eigenvalues and eigenvectors of the matrix A =  2 5 3  numerically. We don’t
1 3 2
know the exact eigenvalues in this case. Check your answers by multiplying your numerically
computed eigenvectors by the matrix.


0 2 1


9. Find the eigenvalues and eigenvectors of the matrix A =  2 0 3  numerically. We don’t
1 3 2
know the exact eigenvalues in this case. Check your answers by multiplying your numerically
computed eigenvectors by the matrix.


3 2 3


T
10. Consider the matrix A =  2 1 4  and the vector (1, 1, 1) . Estimate the distance be3 4 0
tween the Rayleigh quotient determined by this vector and some eigenvalue of A.


1 2 1


T
11. Consider the matrix A =  2 1 4  and the vector (1, 1, 1) . Estimate the distance be1 4 5
tween the Rayleigh quotient determined by this vector and some eigenvalue of A.
15.5. EXERCISES
317
12. Using Gerschgorin’s theorem, find upper and

3

A= 2
3
lower bounds for the eigenvalues of

2 3

6 4 .
4 −3
13. The QR algorithm works very well on general matrices. Try the QR algorithm on the following
matrix which happens to have some complex eigenvalues.


1
2
3


A= 1
2 −1 
−1 −1 1
Use the QR algorithm to get approximate eigenvalues and then use the shifted inverse power
method on one of these to get an approximate eigenvector for one of the complex eigenvalues.
14. Use the QR algorithm to approximate the eigenvalues of the symmetric matrix


1 2 3


 2 −8 1 
3 1 0


3
3
1


15. Try to find the eigenvalues of the matrix  −2 −2 −1  using the QR algorithm. It has
0
1
0
eigenvalues 1, i, −i. You will see the algorithm won’t work well. I
16. Let q (λ) = a0 + a1 λ + · · · + an−1 λn−1 + λn . Now consider the companion matrix,

0 ···

 1 0
C≡
..

.

0
0
..
.
1
−a0
−a1
..
.
−an−1






Show that q (λ) is the characteristic equation for C. Thus the roots of q (λ) are the eigenvalues
of C. You can prove something similar for


−an−1 −an−2 · · · −a0


1




C=
..

.


1
Hint: The characteristic equation is

λ ···

 −1 λ
det 
..

.

0
0
a0
a1
..
..
.
.
−1 λ + an−1






318 CHAPTER 15. NUMERICAL METHODS FOR SOLVING THE EIGENVALUE PROBLEM
Expand along the first column. Thus
λ
−1
λ
···
λ
..
.
0
0
a1
a2
..
..
.
.
−1 λ + an−1
0
−1
..
.
0
+
0
λ
..
.
···
···
−1
a0
a2
..
.
λ + a3
Now use induction on the first term and for the second, note that you can expand along the
top row to get
n−2
n
(−1)
a0 (−1) = a0 .
17. Suppose A is a real symmetric, invertible, matrix, or more generally one which has real eigenvalues. Then as described above, it is typically the case that
Ap = Q1 R
(
and
QT1 AQ1
=
a1
e1
bT1
A1
)
where e is very small. Then you can do the same thing with A1 to obtain another smaller
orthogonal matrix Q2 such that
(
)
a2 bT2
T
Q2 A1 Q2 =
e2 A2
Explain why
(
1 0T
0 Q2
)T
(
QT1 AQ1
1 0T
0 Q2
)

a1
 .

=  ..
e1
∗
a2
e2




A3
where the ei are very small. Explain why one can construct an orthogonal matrix Q such that
QT AQ = (T + E)
where T is an upper triangular matrix and E is very small. In case A is symmetric, explain
why T is actually a diagonal matrix. Next explain why, in the case of A symmetric, that the
columns of Q are an orthonormal basis of vectors, each of which is close to an eigenvector.
Thus this will compute, not just the eigenvalues but also the eigenvectors.
18. Explain how one could use the QR algorithm or the above procedure to compute the singular
value decomposition of an arbitrary real m × n matrix. In fact, there are other algorithms
which will work better.
Chapter 16
Vector Spaces
16.1
Algebraic Considerations
It is time to consider the idea of an abstract Vector space which is something which has two operations
satisfying the following vector space axioms.
Definition 16.1.1 A vector space is an Abelian group of “vectors” denoted here by bold face letters,
satisfying the axioms of an Abelian group,
v + w = w + v,
the commutative law of addition,
(v + w) + z = v + (w + z) ,
the associative law for addition,
v + 0 = v,
the existence of an additive identity,
v + (−v) = 0,
the existence of an additive inverse, along with a field of “scalars” F which are allowed to multiply
the vectors according to the following rules. (The Greek letters denote scalars.)
α (v + w) = αv + αv,
(16.1)
(α + β) v = αv + βv,
(16.2)
α (βv) = αβ (v) ,
(16.3)
1v = v.
(16.4)
The field of scalars is usually R or C and the vector space will be called real or complex depending
on whether the field is R or C. However, other fields are also possible. For example, one could use
the field of rational numbers or even the field of the integers mod p for p a prime. A vector space
is also called a linear space. These axioms do not tell us anything about what is being considered.
Nevertheless, one can prove some fundamental properties just based on these vector space axioms.
Proposition 16.1.2 In any vector space, 0 is unique, −x is unique, 0x = 0, and (−1) x = −x.
Proof: Suppose 0′ is also an additive identity. Then for 0 the additive identity in the axioms,
0′ = 0′ + 0 = 0
319
320
CHAPTER 16. VECTOR SPACES
Next suppose x + y = 0. Then add −x to both sides.
−x = −x+ (x + y) = (−x + x) + y = 0 + y = y
Thus if y acts like the additive inverse, it is the additive inverse.
0x = (0 + 0) x = 0x + 0x
Now add −0x to both sides. This gives 0 = 0x. Finally,
(−1) x + x = (−1) x + 1x = (−1 + 1) x = 0x = 0
By the uniqueness of the additive inverse shown earlier, (−1) x = −x. If you are interested in considering other fields, you should have some examples other than C,
R, Q. Some of these are discussed in the following exercises. If you are happy with only considering
R and C, skip these exercises. Here is an important example which gives the typical vector space.
Example 16.1.3 Let Ω be a nonempty set and define V to be the set of functions defined on Ω.
Letting a, b, c be scalars and f, g, h functions, the vector operations are defined as
(f + g) (x)
(af ) (x)
≡ f (x) + g (x)
≡ a (f (x))
Then this is an example of a vector space.
To verify this, we check the axioms.
(f + g) (x) = f (x) + g (x) = g (x) + f (x) = (g + f ) (x)
Since x is arbitrary, f + g = g + f .
((f + g) + h) (x) ≡ (f + g) (x) + h (x) = (f (x) + g (x)) + h (x)
= f (x) + (g (x) + h (x)) = (f (x) + (g + h) (x)) = (f + (g + h)) (x)
and so (f + g) + h = f + (g + h) . Let 0 denote the function which is given by 0 (x) = 0. Then this
is an additive identity because
(f + 0) (x) = f (x) + 0 (x) = f (x)
and so f + 0 = f . Let −f be the function which satisfies (−f ) (x) ≡ −f (x) . Then
(f + (−f )) (x) ≡ f (x) + (−f ) (x) ≡ f (x) + −f (x) = 0
Hence f + (−f ) = 0.
((a + b) f ) (x) ≡ (a + b) f (x) = af (x) + bf (x) ≡ (af + bf ) (x)
and so (a + b) f = af + bf .
(a (f + g)) (x) ≡ a (f + g) (x) ≡ a (f (x) + g (x))
= af (x) + bg (x) ≡ (af + bg) (x)
and so a (f + g) = af + bg.
((ab) f ) (x) ≡ (ab) f (x) = a (bf (x)) ≡ (a (bf )) (x)
so (abf ) = a (bf ). Finally (1f ) (x) ≡ 1f (x) = f (x) so 1f = f .
Note that Rn can be considered the set of real valued functions defined on (1, 2, · · · , n). Thus
everything up till now was just a special case of this more general situation.
16.2. EXERCISES
16.2
321
Exercises
1. Prove the Euclidean algorithm: If m, n are positive integers, then there exist integers q, r ≥ 0
such that r < m and
n = qm + r
Hint: You might try considering
S ≡ {n − km : k ∈ N and n − km < 0}
and picking the smallest integer in S or something like this.
2. ↑The greatest common divisor of two positive integers m, n, denoted as q is a positive number
which divides both m and n and if p is any other positive number which divides both m, n,
then p divides q. Recall what it means for p to divide q. It means that q = pk for some integer
k. Show that the greatest common divisor of m, n is the smallest positive integer in the set S
S ≡ {xm + yn : x, y ∈ Z and xm + yn > 0}
Two positive integers are called relatively prime if their greatest common divisor is 1.
3. ↑A positive integer larger than 1 is called a prime number if the only positive numbers which
divide it are 1 and itself. Thus 2,3,5,7, etc. are prime numbers. If m is a positive integer and
p does not divide m where p is a prime number, show that p and m are relatively prime.
4. ↑There are lots of fields. This will give an example of a finite field. Let Z denote the set of
integers. Thus Z = {· · · , −3, −2, −1, 0, 1, 2, 3, · · · }. Also let p be a prime number. We will say
that two integers, a, b are equivalent and write a ∼ b if a − b is divisible by p. Thus they are
equivalent if a − b = px for some integer x. First show that a ∼ a. Next show that if a ∼ b
then b ∼ a. Finally show that if a ∼ b and b ∼ c then a ∼ c. For a an integer, denote by [a] the
set of all integers which is equivalent to a, the equivalence class of a. Show first that is suffices
to consider only [a] for a = 0, 1, 2, · · · , p − 1 and that for 0 ≤ a < b ≤ p − 1, [a] ̸= [b]. That is,
[a] = [r] where r ∈ {0, 1, 2, · · · , p − 1}. Thus there are exactly p of these equivalence classes.
Hint: Recall the Euclidean algorithm. For a > 0, a = mp + r where r < p. Next define the
following operations.
[a] + [b] ≡ [a + b]
[a] [b] ≡ [ab]
Show these operations are well defined. That is, if [a] = [a′ ] and [b] = [b′ ] , then [a] + [b] =
[a′ ] + [b′ ] with a similar conclusion holding for multiplication. Thus for addition you need to
verify [a + b] = [a′ + b′ ] and for multiplication you need to verify [ab] = [a′ b′ ]. For example,
if p = 5 you have [3] = [8] and [2] = [7] . Is [2 × 3] = [8 × 7]? Is [2 + 3] = [8 + 7]? Clearly so
in this example because when you subtract, the result is divisible by 5. So why is this so in
general? Now verify that {[0] , [1] , · · · , [p − 1]} with these operations is a Field. This is called
the integers modulo a prime and is written Zp . Since there are infinitely many primes p, it
follows there are infinitely many of these finite fields. Hint: Most of the axioms are easy once
you have shown the operations are well defined. The only two which are tricky are the ones
which give the existence of the additive inverse and the multiplicative inverse. Of these, the
first is not hard. − [x] = [−x]. Since p is prime, there exist integers x, y such that 1 = px + ky
and so 1 − ky = px which says 1 ∼ ky and so [1] = [ky] . Now you finish the argument. What
is the multiplicative identity in this collection of equivalence classes?
16.3
Linear Independence And Bases
Just as in the case of Fn one has a concept of subspace, linear independence, and bases.
322
CHAPTER 16. VECTOR SPACES
Definition 16.3.1 If {v1 , · · · , vn } ⊆ V, a vector space, then
}
{ n
∑
α i vi : α i ∈ F .
span (v1 , · · · , vn ) ≡
i=1
A non empty subset, W ⊆ V is said to be a subspace if ax + by ∈ W whenever a, b ∈ F and x, y ∈ W.
The span of a set of vectors as just described is an example of a subspace.
Then the following fundamental result says that subspaces are subsets of a vector space which
are themselves vector spaces.
Proposition 16.3.2 Let W be a nonempty collection of vectors in V, a vector space. Then W is a
subspace if and only if W is itself a vector space having the same operations as those defined on V .
Proof: Suppose first that W is a subspace. It is obvious that all the algebraic laws hold on W
because it is a subset of V and they hold on V . Thus u + v = v + u along with the other axioms.
Does W contain 0? Yes because it contains 0u = 0. See Proposition 16.1.2. Are the operations
defined on W ? That is, when you add vectors of W do you get a vector in W ? When you multiply
a vector in W by a scalar, do you get a vector in W ? Yes. This is contained in the definition. Does
every vector in W have an additive inverse? Yes by Proposition 16.1.2 because −v = (−1) v which
is given to be in W provided v ∈ W .
Next suppose W is a vector space. Then by definition, it is closed with respect to linear combinations. Hence it is a subspace. Next is the definition of linear independence.
Definition 16.3.3 If {v1 , · · · , vn } ⊆ V, the set of vectors is linearly independent if
n
∑
α i vi = 0
i=1
implies
α1 = · · · = αn = 0
and {v1 , · · · , vn } is called a basis for V if
span (v1 , · · · , vn ) = V
and {v1 , · · · , vn } is linearly independent. The set of vectors is linearly dependent if it is not linearly
independent.
The next theorem is called the exchange theorem. It is very important that you understand this
theorem. There are two kinds of people who go further in linear algebra, those who understand
this theorem and its corollary presented later and those who don’t. Those who do understand these
theorems are able to proceed and learn more linear algebra while those who don’t are doomed to
wander in the wilderness of confusion and sink into the swamp of despair. Therefore, I am giving
multiple proofs. Try to understand at least one of them. Several amount to the same thing, just
worded differently. Before giving the proof, here is some important notation.
Notation 16.3.4 Let wij ∈ V, a vector space and let 1 ≤ i ≤ r while 1 ≤ j ≤ s. Thus these vectors
can be listed in a rectangular array.
w11
w21
..
.
wr1
w12
w22
..
.
wr2
···
···
···
w1s
w2s
..
.
wrs
16.3. LINEAR INDEPENDENCE AND BASES
323
∑s ∑r
Then j=1 i=1 wij means to sum the vectors in each column and then to add the s sums which
∑r ∑s
result while i=1 j=1 wij means to sum the vectors in each row and then to add the r sums which
result. Either way you simply get the sum of all the vectors in the above array. This is because, from
the vector space axioms, you can add vectors in any order and you get the same answer.
Theorem 16.3.5 Let {x1 , · · · , xr } be a linearly independent set of vectors such that each xi is in
the span{y1 , · · · , ys } . Then r ≤ s.
Proof 1: Let
xk =
s
∑
ajk yj
j=1
If r > s, then the matrix A = (ajk ) has more columns than rows. By Corollary 8.2.8 one of these
columns is a linear combination of the others. This implies there exist scalars c1 , · · · , cr such that
r
∑
ajk ck = 0, j = 1, · · · , r
k=1
Then


=0
z }| {
r
r
r
s
s ∑

∑
∑
∑
∑


ck xk =
ck
ck ajk  yj = 0
ajk yj =



j=1
j=1
k=1
k=1
k=1
which contradicts the assumption that {x1 , · · · , xr } is linearly independent. Hence r ≤ s. Proof 2: Define span(y1 , · · · , ys ) ≡ V, it follows there exist scalars, c1 , · · · , cs such that
x1 =
s
∑
ci yi .
(16.5)
i=1
Not all of these scalars can equal zero because if this were the case, it would∑
follow that x1 = 0 and
r
so {x1 , · · · , xr } would not be linearly independent. Indeed, if x1 = 0, 1x1 + i=2 0xi = x1 = 0 and
so there would exist a nontrivial linear combination of the vectors {x1 , · · · , xr } which equals zero.
Say ck ̸= 0. Then solve (16.5) for y k and obtain


s-1 vectors here
z
}|
{
yk ∈ span x1 , y1 , · · · , yk−1 , yk+1 , · · · , ys  .
Define {z1 , · · · , zs−1 } by
{z1 , · · · , zs−1 } ≡ {y1 , · · · , yk−1 , yk+1 , · · · , ys }
Therefore, span (x1 , z1 , · · · , zs−1 ) = V because if v ∈ V, there exist constants c1 , · · · , cs such that
v=
s−1
∑
ci zi + cs yk .
i=1
Replace the yk in the above with a linear combination of the vectors,
{x1 , z1 , · · · , zs−1 }
to obtain v ∈ span (x1 , z1 , · · · , zs−1 ) . The vector yk , in the list {y1 , · · · , ys } , has now been replaced
with the vector x1 and the resulting modified list of vectors has the same span as the original list of
vectors, {y1 , · · · , ys } .
324
CHAPTER 16. VECTOR SPACES
Now suppose that r > s and that span (x1 , · · · , xl , z1 , · · · , zp ) = V, where the vectors, z1 , · · · , zp
are each taken from the set, {y1 , · · · , ys } and l + p = s. This has now been done for l = 1 above.
Then since r > s, it follows that l ≤ s < r and so l + 1 ≤ r. Therefore, xl+1 is a vector not in the
list, {x1 , · · · , xl } and since span (x1 , · · · , xl , z1 , · · · , zp ) = V there exist scalars, ci and dj such that
xl+1 =
l
∑
c i xi +
i=1
p
∑
dj zj .
(16.6)
j=1
Not all the dj can equal zero because if this were so, it would follow that {x1 , · · · , xr } would be a
linearly dependent set because one of the vectors would equal a linear combination of the others.
Therefore, (16.6) can be solved for one of the zi , say zk , in terms of xl+1 and the other zi and just
as in the above argument, replace that zi with xl+1 to obtain


p-1 vectors here
}|
{
z
span x1 , · · · xl , xl+1 , z1 , · · · zk−1 , zk+1 , · · · , zp  = V.
Continue this way, eventually obtaining
span (x1 , · · · , xs ) = V.
But then xr ∈ span (x1 , · · · , xs ) contrary to the assumption that {x1 , · · · , xr } is linearly independent. Therefore, r ≤ s as claimed. Proof 3: Let V ≡ span (y1 , · · · , ys ) and suppose r > s. Let Al ≡ {x1 , · · · , xl } , A0 = ∅, and let
Bs−l denote a subset of the vectors, {y1 , · · · , ys } which contains s − l vectors and has the property
that span (Al , Bs−l ) = V. Note that the assumption of the theorem says span (A0 , Bs ) = V.
Now an exchange operation is given for span (Al , Bs−l ) = V . Since r > s, it follows l < r. Letting
Bs−l ≡ {z1 , · · · , zs−l } ⊆ {y1 , · · · , ys } ,
it follows there exist constants, ci and di such that
xl+1 =
l
∑
i=1
ci xi +
s−l
∑
di zi ,
i=1
and not all the di can equal zero. (If they were all equal to zero, it would follow that the set,
{x1 , · · · , xr } would be dependent since one of the vectors in it would be a linear combination of the
others.)
Let dk ̸= 0. Then z k can be solved for as follows.
l
∑
∑ di
1
ci
zk =
xl+1 −
xi −
zi .
dk
d
dk
i=1 k
i̸=k
This implies V = span (Al+1 , Bs−l−1 ), where Bs−l−1 ≡ Bs−l \ {zk } , a set obtained by deleting
zk from Bk−l . You see, the process exchanged a vector in Bs−l with one from {x1 , · · · , xr } and
kept the span the same. Starting with V = span (A0 , Bs ) , do the exchange operation until V =
span (As−1 , z) where z ∈ {y1 , · · · , ys } . Then one more application of the exchange operation yields
V = span (As ) . But this implies xr ∈ span (As ) = span (x1 , · · · , xs ) , contradicting the linear
independence of {x1 , · · · , xr } . It follows that r ≤ s as claimed. Proof 4: Suppose r > s. Let zk denote a vector of {y1 , · · · , ys } . Thus there exists j as small
as possible such that
span (y1 , · · · , ys ) = span (x1 , · · · , xm , z1 , · · · , zj )
16.3. LINEAR INDEPENDENCE AND BASES
325
where m + j = s. It is given that m = 0, corresponding to no vectors of {x1 , · · · , xm } and j = s,
corresponding to all the yk results in the above equation holding. If j > 0 then m < s and so
xm+1 =
m
∑
ak xk +
j
∑
bi zi
i=1
k=1
Not all the bi can equal 0 and so you can solve for one of the zi in terms of
xm+1 , xm , · · · , x1 ,
and the other z k . Therefore, there exists
{z1 , · · · , zj−1 } ⊆ {y1 , · · · , ys }
such that
span (y1 , · · · , ys ) = span (x1 , · · · , xm+1 , z1 , · · · , zj−1 )
contradicting the choice of j. Hence j = 0 and
span (y1 , · · · , ys ) = span (x1 , · · · , xs )
It follows that
xs+1 ∈ span (x1 , · · · , xs )
contrary to the assumption the xk are linearly independent. Therefore, r ≤ s as claimed. Corollary 16.3.6 If {u1 , · · · , um } and {v1 , · · · , vn } are two bases for V, then m = n.
Proof: By Theorem 16.3.5, m ≤ n and n ≤ m. This corollary is very important so here is another proof of it given independent of the exchange
theorem above.
Theorem 16.3.7 Let V be a vector space and suppose {u1 , · · · , uk } and {v1 , · · · , vm } are two bases
for V . Then k = m.
Proof: Suppose k > m. Then since the vectors, {u1 , · · · , uk } span V, there exist scalars, cij
such that
m
∑
cij vi = uj .
i=1
Therefore,
k
∑
dj uj = 0 if and only if
j=1
k ∑
m
∑
cij dj vi = 0
j=1 i=1
if and only if
m
∑
i=1


k
∑

cij dj  vi = 0
j=1
Now since{v1 , · · · , vn } is independent, this happens if and only if
k
∑
cij dj = 0, i = 1, 2, · · · , m.
j=1
However, this is a system of m equations in k variables, d1 , · · · , dk and m < k. Therefore, there
exists a solution to this system of equations in which not all (
the dj are) equal to zero. Recall why
this is so. The augmented matrix for the system is of the form C 0 where C is a matrix which
has more columns than rows. Therefore, there are free variables and hence nonzero solutions to the
system of equations. However, this contradicts the linear independence of {u1 , · · · , uk } because, as
∑k
explained above, j=1 dj uj = 0. Similarly it cannot happen that m > k. 326
CHAPTER 16. VECTOR SPACES
Definition 16.3.8 A vector space V is of dimension n if it has a basis consisting of n vectors. This
is well defined thanks to Corollary 16.3.6. It is always assumed here that n < ∞ and in this case,
such a vector space is said to be finite dimensional.
The following says that if you add a vector which is not in the span of a linearly independent set
of vectors to this set of vectors, then the resulting list is linearly independent.
Lemma 16.3.9 Suppose v ∈
/ span (u1 , · · · , uk ) and {u1 , · · · , uk } is linearly independent. Then
{u1 , · · · , uk , v} is also linearly independent.
∑k
Proof: Suppose i=1 ci ui + dv = 0. It is required to verify that each ci = 0 and that d = 0.
But if d ̸= 0, then you can solve for v as a linear combination of the vectors, {u1 , · · · , uk },
v=−
k ( )
∑
ci
d
i=1
contrary to assumption. Therefore, d = 0. But then
{u1 , · · · , uk } implies each ci = 0 also. ui
∑k
i=1 ci ui
= 0 and the linear independence of
Theorem 16.3.10 If V = span (u1 , · · · , un ) then some subset of {u1 , · · · , un } is a basis for V.
Also, if {u1 , · · · , uk } ⊆ V is linearly independent and the vector space is finite dimensional, then
the set {u1 , · · · , uk }, can be enlarged to obtain a basis of V.
Proof: Let
S = {E ⊆ {u1 , · · · , un } such that span (E) = V }.
For E ∈ S, let |E| denote the number of elements of E. Let
m ≡ min{|E| such that E ∈ S}.
Thus there exist vectors
{v1 , · · · , vm } ⊆ {u1 , · · · , un }
such that
span (v1 , · · · , vm ) = V
and m is as small as possible for this to happen. If this set is linearly independent, it follows it is
a basis for V and the theorem is proved. On the other hand, if the set is not linearly independent,
then there exist scalars, c1 , · · · , cm such that
0=
m
∑
ci vi
i=1
and not all the ci are equal to zero. Suppose ck ̸= 0. Then the vector vk may be solved for in terms
of the other vectors. Consequently,
V = span (v1 , · · · , vk−1 , vk+1 , · · · , vm )
contradicting the definition of m. This proves the first part of the theorem.
To obtain the second part, begin with {u1 , · · · , uk } and suppose a basis for V is {v1 , · · · , vn } .
If
span (u1 , · · · , uk ) = V,
then k = n. If not, there exists a vector
uk+1 ∈
/ span (u1 , · · · , uk ) .
16.3. LINEAR INDEPENDENCE AND BASES
327
Then from Lemma 16.3.9, {u1 , · · · , uk , uk+1 } is also linearly independent. Continue adding vectors
in this way until n linearly independent vectors have been obtained. Then
span (u1 , · · · , un ) = V
because if it did not do so, there would exist un+1 as just described and {u1 , · · · , un+1 } would be a
linearly independent set of vectors having n + 1 elements even though {v1 , · · · , vn } is a basis. This
would contradict Theorem 16.3.5. Therefore, this list is a basis. Theorem 16.3.11 Let V be a nonzero subspace of a finite dimensional vector space W of dimension
n. Then V has a basis with no more than n vectors.
Proof: Let v1 ∈ V where v1 ̸= 0. If span (v1 ) = V, stop. {v1 } is a basis for V . Otherwise, there
exists v2 ∈ V which is not in span (v1 ) . By Lemma 16.3.9 {v1 , v2 } is a linearly independent set of
vectors. If span (v1 , v2 ) = V stop, {v1 , v2 } is a basis for V. If span (v1 , v2 ) ̸= V, then there exists
v3 ∈
/ span (v1 , v2 ) and {v1 , v2 , v3 } is a larger linearly independent set of vectors. Continuing this
way, the process must stop before n + 1 steps because if not, it would be possible to obtain n + 1
linearly independent vectors contrary to the exchange theorem, Theorem 16.3.5. Example(16.3.12
) Let {V be the
} polynomials of degree no more than 2. Thus, abusing notation,
V = span 1, x, x2 . Is 1, x, x2 a basis for V ?
It will be a basis if it is linearly independent. Suppose then that
a + bx + cx2 = 0
First let x = 0 and this shows that a = 0. Therefore, bx + cx2 = 0. Take a derivative of both sides.
Then
b + 2cx = 0
Let x = 0. Then b = 0. Hence 2cx = 0 and this must hold for all x. Therefore, c = 0. This shows
that these functions are linearly independent. Since they also span, these must yield a basis.
{
}
Example 16.3.13 Let V be the polynomials of degree no more than 2. Is x2 + x + 1, 2x + 1, 3x2 + 1
a basis for V ?
If these vectors are linearly independent, then they must be a basis since otherwise, you could
obtain four functions in V which are linearly
which is impossible because there is a
{ independent
}
spanning set of only three vectors, namely 1, x, x2 . Suppose then that
(
)
(
)
a x2 + x + 1 + b (2x + 1) + c 3x2 + 1 = 0
Then
(a + 3c) x2 + (a + 2b) x + (a + b + c) = 0
From the above example, you know that
a + 3c = 0, a + 2b = 0, a + b + c = 0
and there is only one solution to this system of equations, a = b = c = 0. Therefore, these are
linearly independent and hence are a basis for this vector space.
328
CHAPTER 16. VECTOR SPACES
16.4
Vector Spaces And Fields∗
16.4.1
Irreducible Polynomials
I mentioned earlier that most things hold for arbitrary fields. However, I have not bothered to give
any examples of other fields. This is the point of this section. It also turns out that showing the
algebraic numbers are a field can be understood using vector space concepts and it gives a very
convincing application of the abstract theory presented earlier in this chapter.
Here I will give some basic algebra relating to polynomials. This is interesting for its own sake
but also provides the basis for constructing many different kinds of fields. The first is the Euclidean
algorithm for polynomials.
∑n
Definition 16.4.1 A polynomial is an expression of the form p (λ) = k=0 ak λk where as usual λ0
is defined to equal 1. Two polynomials are said to be equal if their corresponding coefficients are the
same. Thus, in particular, p (λ) = 0 means each of the ak = 0. An element of the field λ is said to
be a root of the polynomial if p (λ) = 0 in the sense that when you plug in λ into the formula and
do the indicated operations, you get 0. The degree of a nonzero polynomial is the highest exponent
appearing on λ. The degree of the zero polynomial p (λ) = 0 is not defined.
Example 16.4.2 Consider the polynomial p (λ) = λ2 + λ where the coefficients are in Z2 . Is this
polynomial equal to 0? Not according to the above definition, because its coefficients are not all equal
to 0. However, p (1) = p (0) = 0 so it sends every element of Z2 to 0. Note the distinction between
saying it sends everything in the field to 0 with having the polynomial be the zero polynomial.
The fundamental result is the division theorem for polynomials.
Lemma 16.4.3 Let f (λ) and g (λ) ̸= 0 be polynomials. Then there exists a polynomial, q (λ) such
that
f (λ) = q (λ) g (λ) + r (λ)
where the degree of r (λ) is less than the degree of g (λ) or r (λ) = 0. These polynomials q (λ) and
r (λ) are unique.
Proof: Suppose that f (λ)−q (λ) g (λ) is never equal to 0 for any q (λ). If it is, then the conclusion
follows. Now suppose
r (λ) = f (λ) − q (λ) g (λ)
and the degree of r (λ) is m ≥ n where n is the degree of g (λ). Say the leading term of r (λ) is bλm
while the leading term of g (λ) is b̂λn . Then letting a = b/b̂ , aλm−n g (λ) has the same leading term
as r (λ). Thus the degree of r1 (λ) ≡ r (λ) − aλm−n g (λ) is no more than m − 1. Then

q1 (λ)
}|
{
z
(
)


r1 (λ) = f (λ) − q (λ) g (λ) + aλm−n g (λ) = f (λ) − q (λ) + aλm−n  g (λ)

Denote by S the set of polynomials f (λ) − g (λ) l (λ) . Out of all these polynomials, there exists
one which has smallest degree r (λ). Let this take place when l (λ) = q (λ). Then by the above
argument, the degree of r (λ) is less than the degree of g (λ). Otherwise, there is one which has
smaller degree. Thus f (λ) = g (λ) q (λ) + r (λ).
As to uniqueness, if you have r (λ) , r̂ (λ) , q (λ) , q̂ (λ) which work, then you would have
(q̂ (λ) − q (λ)) g (λ) = r (λ) − r̂ (λ)
Now if the polynomial on the right is not zero, then neither is the one on the left. Hence this would
involve two polynomials which are equal although their degrees are different. This is impossible.
Hence r (λ) = r̂ (λ) and so, matching coefficients implies that q̂ (λ) = q (λ). 16.4. VECTOR SPACES AND FIELDS∗
329
Now with this lemma, here is another one which is very fundamental. First here is a definition.
A polynomial is monic means it is of the form
λn + cn−1 λn−1 + · · · + c1 λ + c0 .
That is, the leading coefficient is 1. In what follows, the coefficients of polynomials are in F, a field
of scalars which is completely arbitrary. Think R if you need an example.
Definition 16.4.4 A polynomial f is said to divide a polynomial g if g (λ) = f (λ) r (λ) for some
polynomial r (λ). Let {ϕi (λ)} be a finite set of polynomials. The greatest common divisor will be
the monic polynomial q (λ) such that q (λ) divides each ϕi (λ) and if p (λ) divides each ϕi (λ) , then
p (λ) divides q (λ) . The finite set of polynomials {ϕi } is said to be relatively prime if their greatest
common divisor is 1. A polynomial f (λ) is irreducible if there is no polynomial with coefficients in
F which divides it except nonzero scalar multiples of f (λ) and constants.
Proposition 16.4.5 The greatest common divisor is unique.
Proof: Suppose both q (λ) and q ′ (λ) work. Then q (λ) divides q ′ (λ) and the other way around
and so
q ′ (λ) = q (λ) l (λ) , q (λ) = l′ (λ) q ′ (λ)
Therefore, the two must have the same degree. Hence l′ (λ) , l (λ) are both constants. However, this
constant must be 1 because both q (λ) and q ′ (λ) are monic. Theorem 16.4.6 Let ψ (λ) be the greatest common divisor of {ϕi (λ)} , not all of which are zero
polynomials. Then there exist polynomials ri (λ) such that
ψ (λ) =
p
∑
ri (λ) ϕi (λ) .
i=1
Furthermore, ψ (λ) is the monic polynomial of smallest degree which can be written in the above
form.
Proof: Let S denote the set of monic polynomials which are of the form
p
∑
ri (λ) ϕi (λ)
i=1
where ri (λ) is a polynomial. Then
∑Sp ̸= ∅ because some ϕi (λ) ̸= 0. Then let the ri be chosen such
that the degree of the expression i=1 ri (λ) ϕi (λ) is as small as possible. Letting ψ (λ) equal this
sum, it remains to verify it is the greatest common divisor. First, does it divide each ϕi (λ)? Suppose
it fails to divide ϕ1 (λ) . Then by Lemma 16.4.3
ϕ1 (λ) = ψ (λ) l (λ) + r (λ)
where degree of r (λ) is less than that of ψ (λ). Then dividing r (λ) by the leading coefficient if
necessary and denoting the result by ψ 1 (λ) , it follows the degree of ψ 1 (λ) is less than the degree
of ψ (λ) and ψ 1 (λ) equals
(
)
p
∑
ψ 1 (λ) = (ϕ1 (λ) − ψ (λ) l (λ)) a = ϕ1 (λ) −
ri (λ) ϕi (λ) l (λ) a
i=1
(
=
(1 − r1 (λ)) ϕ1 (λ) +
p
∑
)
(−ri (λ) l (λ)) ϕi (λ) a
i=2
for a suitable a ∈ F. This is one of the polynomials in S. Therefore, ψ (λ) does not have the smallest
degree after all because the degree of ψ 1 (λ) is smaller. This is a contradiction. Therefore, ψ (λ)
divides ϕ1 (λ) . Similarly it divides all the other ϕi (λ).
∑p If p (λ) divides all the ϕi (λ) , then it divides ψ (λ) because of the formula for ψ (λ) which equals
i=1 ri (λ) ϕi (λ) . 330
CHAPTER 16. VECTOR SPACES
Lemma 16.4.7 Suppose ϕ (λ) and ψ (λ) are monic polynomials which are irreducible and not equal.
Then they are relatively prime.
Proof: Suppose η (λ) is a nonconstant polynomial. If η (λ) divides ϕ (λ) , then since ϕ (λ) is
irreducible, η (λ) equals aϕ (λ) for some a ∈ F. If η (λ) divides ψ (λ) then it must be of the form
bψ (λ) for some b ∈ F and so it follows
ψ (λ) =
a
ϕ (λ)
b
but both ψ (λ) and ϕ (λ) are monic polynomials which implies a = b and so ψ (λ) = ϕ (λ). This
is assumed not to happen. It follows the only polynomials which divide both ψ (λ) and ϕ (λ) are
constants and so the two polynomials are relatively prime. Thus a polynomial which divides them
both must be a constant, and if it is monic, then it must be 1. Thus 1 is the greatest common
divisor. Lemma 16.4.8 Let ψ (λ) be an irreducible monic polynomial not equal to 1 which divides
p
∏
k
ϕi (λ) i , ki a positive integer,
i=1
where each ϕi (λ) is an irreducible monic polynomial not equal to 1. Then ψ (λ) equals some ϕi (λ) .
∏p
k
Proof : Say ψ (λ) l (λ) = i=1 ϕi (λ) i . Suppose ψ (λ) ̸= ϕi (λ) for all i. Then by Lemma 16.4.7,
there exist polynomials mi (λ) , ni (λ) such that
1 =
ψ (λ) mi (λ) + ϕi (λ) ni (λ)
= 1 − ψ (λ) mi (λ)
ϕi (λ) ni (λ)
Hence,
ψ (λ) n (λ)
≡ ψ (λ) l (λ)
p
∏
ni (λ)
ki
=
i=1
=
p
∏
p
∏
(ni (λ) ϕi (λ))
ki
i=1
(1 − ψ (λ) mi (λ))
ki
= 1 + g (λ) ψ (λ)
i=1
for a polynomial g (λ). Thus
1 = ψ (λ) (n (λ) − g (λ))
which is impossible because ψ (λ) is not equal to 1. Now here is a simple lemma about canceling monic polynomials.
Lemma 16.4.9 Suppose p (λ) is a monic polynomial and q (λ) is a polynomial such that
p (λ) q (λ) = 0.
Then q (λ) = 0. Also if
p (λ) q1 (λ) = p (λ) q2 (λ)
then q1 (λ) = q2 (λ) .
Proof: Let
p (λ) =
k
∑
pj λj , q (λ) =
j=1
n
∑
qi λi , pk = 1.
i=1
Then the product equals
k ∑
n
∑
j=1 i=1
pj qi λi+j .
16.4. VECTOR SPACES AND FIELDS∗
331
Then look at those terms involving λk+n . This is pk qn λk+n and is given to be 0. Since pk = 1, it
follows qn = 0. Thus
k n−1
∑
∑
pj qi λi+j = 0.
j=1 i=1
n−1+k
Then consider the term involving λ
and conclude that since pk = 1, it follows qn−1 = 0.
Continuing this way, each qi = 0. This proves the first part. The second follows from
p (λ) (q1 (λ) − q2 (λ)) = 0. The following is the analog of the fundamental theorem of arithmetic for polynomials.
Theorem 16.4.10 Let f (λ)
∏nbe a nonconstant polynomial with coefficients in F. Then there is some
a ∈ F such that f (λ) = a i=1 ϕi (λ) where ϕi (λ) is an irreducible nonconstant monic polynomial
and repeats are allowed. Furthermore, this factorization is unique in the sense that any two of these
factorizations have the same nonconstant factors in the product, possibly in different order and the
same constant a.
Proof: That such a factorization exists is obvious. If f (λ) is irreducible, you are done. Factor
out the leading coefficient. If not, then f (λ) = aϕ1 (λ) ϕ2 (λ) where these are monic polynomials.
Continue doing this with the ϕi and eventually arrive at a factorization of the desired form.
It remains to argue the factorization is unique except for order of the factors. Suppose
n
∏
a
ϕi (λ) = b
i=1
m
∏
ψ i (λ)
i=1
where the ϕi (λ) and the ψ i (λ) are all irreducible monic nonconstant polynomials and a, b ∈ F. If
n > m, then by Lemma 16.4.8, each ψ i (λ) equals one of the ϕj (λ) . By the above cancellation lemma,
Lemma 16.4.9, you can cancel all these ψ i (λ) with appropriate ϕj (λ) and obtain a contradiction
because the resulting polynomials on either side would have different degrees. Similarly, it cannot
happen that n < m. It follows n = m and the two products consist of the same polynomials. Then
it follows a = b. The following corollary will be well used. This corollary seems rather believable but does require
a proof.
∏p
k
Corollary 16.4.11 Let q (λ) = i=1 ϕi (λ) i where the ki are positive integers and the ϕi (λ) are
irreducible monic polynomials. Suppose also that p (λ) is a monic polynomial which divides q (λ) .
Then
p
∏
r
p (λ) =
ϕi (λ) i
i=1
where ri is a nonnegative integer no larger than ki .
∏s
r
Proof: Using Theorem 16.4.10, let p (λ) = b i=1 ψ i (λ) i where the ψ i (λ) are each irreducible
and monic and b ∈ F. Since p (λ) is monic, b = 1. Then there exists a polynomial g (λ) such that
p (λ) g (λ) = g (λ)
s
∏
ri
ψ i (λ)
i=1
=
p
∏
ϕi (λ)
ki
i=1
Hence g (λ) must be monic. Therefore,
p(λ)
z }| {
p
s
l
∏
∏
∏
r
k
p (λ) g (λ) =
ψ i (λ) i
η j (λ) =
ϕi (λ) i
i=1
j=1
i=1
for η j monic and irreducible. By uniqueness, each ψ i equals one of the ϕj (λ) and the same holding
true of the η i (λ). Therefore, p (λ) is of the desired form. 332
CHAPTER 16. VECTOR SPACES
16.4.2
Polynomials And Fields
When you have a polynomial like x2 − 3 which has no rational roots, it turns out you can enlarge
the field of rational numbers to obtain a larger field such that this polynomial does have roots in
this larger field. I am going to discuss a systematic way to do this. It will turn out that for any
polynomial with coefficients in any field, there always exists a possibly larger field such that the
polynomial has roots in this larger field. This book has mainly featured the field of real or complex
numbers but this procedure will show how to obtain many other fields which could be used in most
of what was presented earlier in the book. Here is an important idea concerning equivalence relations
which I hope is familiar.
Definition 16.4.12 Let S be a set. The symbol, ∼ is called an equivalence relation on S if it
satisfies the following axioms.
1. x ∼ x
for all x ∈ S. (Reflexive)
2. If x ∼ y then y ∼ x. (Symmetric)
3. If x ∼ y and y ∼ z, then x ∼ z. (Transitive)
Definition 16.4.13 [x] denotes the set of all elements of S which are equivalent to x and [x] is
called the equivalence class determined by x or just the equivalence class of x.
Also recall the notion of equivalence classes.
Theorem 16.4.14 Let ∼ be an equivalence class defined on a set, S and let H denote the set of
equivalence classes. Then if [x] and [y] are two of these equivalence classes, either x ∼ y and [x] = [y]
or it is not true that x ∼ y and [x] ∩ [y] = ∅.
Definition 16.4.15 Let F be a field, for example the rational numbers, and denote by F [x] the
polynomials having coefficients in F. Suppose p (x) is a polynomial. Let a (x) ∼ b (x) (a (x) is
similar to b (x)) when
a (x) − b (x) = k (x) p (x)
for some polynomial k (x) .
Proposition 16.4.16 In the above definition, ∼ is an equivalence relation.
Proof: First of all, note that a (x) ∼ a (x) because their difference equals 0p (x) . If a (x) ∼ b (x) ,
then a (x)−b (x) = k (x) p (x) for some k (x) . But then b (x)−a (x) = −k (x) p (x) and so b (x) ∼ a (x).
Next suppose a (x) ∼ b (x) and b (x) ∼ c (x) . Then a (x) − b (x) = k (x) p (x) for some polynomial
k (x) and also b (x) − c (x) = l (x) p (x) for some polynomial l (x) . Then
a (x) − c (x) = a (x) − b (x) + b (x) − c (x)
= k (x) p (x) + l (x) p (x) = (l (x) + k (x)) p (x)
and so a (x) ∼ c (x) and this shows the transitive law. With this proposition, here is another definition which essentially describes the elements of the
new field. It will eventually be necessary to assume the polynomial p (x) in the above definition is
irreducible so I will begin assuming this.
Definition 16.4.17 Let F be a field and let p (x) ∈ F [x] be a monic irreducible polynomial of degree
greater than 0. Thus there is no polynomial having coefficients in F which divides p (x) except for
itself and constants. For the similarity relation defined in Definition 16.4.15, define the following
operations on the equivalence classes. [a (x)] is an equivalence class means that it is the set of all
polynomials which are similar to a (x).
[a (x)] + [b (x)] ≡
[a (x)] [b (x)] ≡
[a (x) + b (x)]
[a (x) b (x)]
This collection of equivalence classes is sometimes denoted by F [x] / (p (x)).
16.4. VECTOR SPACES AND FIELDS∗
333
Proposition 16.4.18 In the situation of Definition 16.4.17 where p (x) is a monic irreducible polynomial, the following are valid.
1. p (x) and q (x) are relatively prime for any q (x) ∈ F [x] which is not a multiple of p (x).
2. The definitions of addition and multiplication are well defined.
3. If a, b ∈ F and [a] = [b] , then a = b. Thus F can be considered a subset of F [x] / (p (x)) .
4. F [x] / (p (x)) is a field in which the polynomial p (x) has a root.
5. F [x] / (p (x)) is a vector space with field of scalars F and its dimension is m where m is the
degree of the irreducible polynomial p (x).
Proof: First consider the claim about p (x) , q (x) being relatively prime. If ψ (x) is the greatest
common divisor, it follows ψ (x) is either equal to p (x) or 1. If it is p (x) , then q (x) is a multiple of
p (x) which does not happen. If it is 1, then by definition, the two polynomials are relatively prime.
To show the operations are well defined, suppose
[a (x)] = [a′ (x)] , [b (x)] = [b′ (x)]
It is necessary to show
[a (x) + b (x)] = [a′ (x) + b′ (x)]
[a (x) b (x)] = [a′ (x) b′ (x)]
Consider the second of the two.
a′ (x) b′ (x) − a (x) b (x)
=
=
a′ (x) b′ (x) − a (x) b′ (x) + a (x) b′ (x) − a (x) b (x)
b′ (x) (a′ (x) − a (x)) + a (x) (b′ (x) − b (x))
Now by assumption (a′ (x) − a (x)) is a multiple of p (x) as is (b′ (x) − b (x)) , so the above is a
multiple of p (x) and by definition this shows [a (x) b (x)] = [a′ (x) b′ (x)]. The case for addition is
similar.
Now suppose [a] = [b] . This means a − b = k (x) p (x) for some polynomial k (x) . Then k (x)
must equal 0 since otherwise the two polynomials a − b and k (x) p (x) could not be equal because
they would have different degree.
It is clear that the axioms of a field are satisfied except for the one which says that non zero
elements of the field have a multiplicative inverse. Let [q (x)] ∈ F [x] / (p (x)) where [q (x)] ̸= [0] .
Then q (x) is not a multiple of p (x) and so by the first part, q (x) , p (x) are relatively prime. Thus
there exist n (x) , m (x) such that
1 = n (x) q (x) + m (x) p (x)
Hence
[1] = [1 − n (x) p (x)] = [n (x) q (x)] = [n (x)] [q (x)]
−1
which shows that [q (x)]
because if
= [n (x)] . Thus this is a field. The polynomial has a root in this field
p (x) = xm + am−1 xm−1 + · · · + a1 x + a0 ,
m
[0] = [p (x)] = [x] + [am−1 ] [x]
m−1
+ · · · + [a1 ] [x] + [a0 ]
Thus [x] is a root of this polynomial in the field F [x] / (p (x)).
Consider the last claim. Let f (x) ∈ F [x] / (p (x)) . Thus [f (x)] is a typical thing in F [x] / (p (x)).
Then from the division algorithm,
f (x) = p (x) q (x) + r (x)
334
CHAPTER 16. VECTOR SPACES
where r (x) is either 0 or has degree less than the degree of p (x) . Thus
[r (x)] = [f (x) − p (x) q (x)] = [f (x)]
)
(
)
m−1
m−1
but clearly [r (x)] ∈ span [1] , · · · , [x]
. Thus span [1] , · · · , [x]
= F [x] / (p (x)). Then
{
}
m−1
[1] , · · · , [x]
is a basis if these vectors are linearly independent. Suppose then that
(
m−1
∑
i
ci [x] =
i=0
[m−1
∑
]
ci x
i
=0
i=0
∑m−1
Then you would need to have p (x) / i=0 ci xi which is impossible unless each ci = 0 because p (x)
has degree m. The last assertion in the proof follows from the definition of addition and multiplication in
F [x] / (p (x)) and math induction. If each ai ∈ F,
[
]
n
n−1
an xn + an−1 xn−1 + · · · + a1 x + a0 = [an ] [x] + [an−1 ] [x]
+ · · · [a1 ] [x] + [a0 ]
(16.7)
Remark 16.4.19 The polynomials consisting of all polynomial multiples of p (x) , denoted by (p (x))
is called an ideal. An ideal I is a subset of the commutative ring (Here the ring is F [x] .) with
unity consisting of all polynomials which is itself a ring and which has the property that whenever
f (x) ∈ F [x] , and g (x) ∈ I, f (x) g (x) ∈ I. In this case, you could argue that (p (x)) is an ideal and
that the only ideal containing it is itself or the entire ring F [x]. This is called a maximal ideal.
Example 16.4.20 The polynomial x2 −2 is irreducible in Q [x] . This is because if x2 −2 = p (x) q (x)
where p (x) , q (x) both have degree less than 2, then they both have degree 1. Hence you would have
x2 −2 = (x + a) (x + b) which requires
that a+b = 0 so( this factorization
is of the form (x − a) (x + a)
√
)
and now you need to have a = 2 ∈
/ Q. Now Q [x] / x2 − 2 is of the form a + b [x] where a, b ∈ Q
√
√
(
)
2
and [x] − 2 = 0. Thus one can regard [x] as 2. Q [x] / x2 − 2 is of the form a + b 2.
Thus the above is an illustration of something general described in the following definition.
Definition 16.4.21 Let F ⊆ K be two fields. Then clearly K is also a vector space over F. Then
also, K is called a finite field extension of F if the dimension of this vector space, denoted by [K : F ]
is finite.
There are some easy things to observe about this.
Proposition 16.4.22 Let F ⊆ K ⊆ L be fields. Then [L : F ] = [L : K] [K : F ].
n
m
Proof: Let {li }i=1 be a basis for L over K and let {kj }j=1 be a basis of K over F . Then if
l ∈ L, there exist unique scalars xi in K such that
l=
n
∑
xi li
i=1
Now xi ∈ K so there exist fji such that
xi =
m
∑
fji kj
j=1
Then it follows that
l=
n ∑
m
∑
i=1 j=1
fji kj li
16.4. VECTOR SPACES AND FIELDS∗
335
It follows that {kj li } is a spanning set. If
n ∑
m
∑
fji kj li = 0
i=1 j=1
Then, since the li are independent, it follows that
m
∑
fji kj = 0
j=1
and since {kj } is independent, each fji = 0 for each j for a given arbitrary i. Therefore, {kj li } is a
basis. Note that if p (x) were not irreducible, then you could find a field extension G such that [G : F] ≤
n. You could do this by working with an irreducible factor of p (x).
Usually, people simply write b rather than [b] if b ∈ F. Then with this convention,
[bϕ (x)] = [b] [ϕ (x)] = b [ϕ (x)] .
This shows how to enlarge a field to get a new one in which the polynomial has a root. By using
a succession of such enlargements, called field extensions, there will exist a field in which the given
polynomial can be factored into a product of polynomials having degree one. The field you obtain
in this process of enlarging in which the given polynomial factors in terms of linear factors is called
a splitting field.
Theorem 16.4.23 Let p (x) = xn + an−1 xn−1 + · · · + a1 x + a0 be a polynomial with coefficients in
a field of scalars F. There exists a larger field G such that there exist {z1 , · · · , zn } listed according
to multiplicity such that
n
∏
p (x) =
(x − zi )
i=1
This larger field is called a splitting field. Furthermore,
[G : F] ≤ n!
Proof: From Proposition 16.4.18, there exists a field F1 such that p (x) has a root, z1 (= [x] if
p is irreducible.) Then by the Euclidean algorithm
p (x) = (x − z1 ) q1 (x) + r
where r ∈ F1 . Since p (z1 ) = 0, this requires r = 0. Now do the same for q1 (x) that was done for
p (x) , enlarging the field to F2 if necessary, such that in this new field
q1 (x) = (x − z2 ) q2 (x) .
and so
p (x) = (x − z1 ) (x − z2 ) q2 (x)
After n such extensions, you will have obtained the necessary field G.
Finally consider the claim about dimension. By Proposition 16.4.18, there is a larger field G1
such that p (x) has a root a1 in G1 and [G : F] ≤ n. Then
p (x) = (x − a1 ) q (x)
Continue this way until the polynomial equals the product of linear factors. Then by Proposition
16.4.22 applied multiple times, [G : F] ≤ n!. 336
CHAPTER 16. VECTOR SPACES
Example 16.4.24 The polynomial x2 +1 is irreducible in R (x) , polynomials having real coefficients.
To see this is the case, suppose ψ (x) divides x2 + 1. Then
x2 + 1 = ψ (x) q (x)
If the degree of ψ (x) is less than 2, then it must be either a constant or of the form ax + b. In the
latter case, −b/a must be a zero of the right side, hence of the left but x2 + 1 has no real zeros.
Therefore, the degree of ψ (x) must be two and q (x) must be a constant. Thus the only polynomial
2
2
which divides x2 + 1 [are constants
] and multiples of x + 1. Therefore, this( shows
)x + 1 is irreducible.
2
2
Find the inverse of x + x + 1 in the space of equivalence classes, R/ x + 1 .
You can solve this with partial fractions.
x
x+1
1
=− 2
+
(x2 + 1) (x2 + x + 1)
x + 1 x2 + x + 1
and so
(
)
(
)
1 = (−x) x2 + x + 1 + (x + 1) x2 + 1
which implies
(
)
1 ∼ (−x) x2 + x + 1
and so the inverse is [−x] .
The following proposition is interesting. It was essentially proved above but to emphasize it,
here it is again.
Proposition 16.4.25 Suppose p (x) ∈ F [x] is irreducible and has degree n. Then every element of
G = F [x] / (p (x)) is of the form [0] or [r (x)] where the degree of r (x) is less than n.
Proof: This follows right away from the Euclidean algorithm for polynomials. If k (x) has degree
larger than n − 1, then
k (x) = q (x) p (x) + r (x)
where r (x) is either equal to 0 or has degree less than n. Hence
[k (x)] = [r (x)] . Example 16.4.26 In the situation of the above example, find [ax + b]
this includes all cases of interest thanks to the above proposition.
−1
assuming a2 +b2 ̸= 0. Note
You can do it with partial fractions as above.
(x2
1
b − ax
a2
= 2
+ 2
2
2
2
+ 1) (ax + b)
(a + b ) (x + 1) (a + b ) (ax + b)
and so
1=
Thus
( 2
)
1
a2
(b
−
ax)
(ax
+
b)
+
x +1
2
2
2
2
a +b
(a + b )
1
(b − ax) (ax + b) ∼ 1
a2 + b2
and so
[ax + b]
−1
=
b − a [x]
[(b − ax)]
= 2
a2 + b2
a + b2
You might find it interesting to recall that (ai + b)
−1
=
b−ai
a2 +b2 .
16.4. VECTOR SPACES AND FIELDS∗
16.4.3
337
The Algebraic Numbers
Each polynomial having coefficients in a field F has a splitting field. Consider the case of all
polynomials p (x) having coefficients in a field F ⊆ G and consider all roots which are also in G. The
theory of vector spaces is very useful in the study of these algebraic numbers. Here is a definition.
Definition 16.4.27 The algebraic numbers A are those numbers which are in G and also roots of
some polynomial p (x) having coefficients in F. The minimal polynomial of a ∈ A is defined to be
the monic polynomial p (x) having smallest degree such that p (a) = 0.
Theorem 16.4.28 Let a ∈ A. Then there exists a unique monic irreducible polynomial p (x) having
coefficients in F such that p (a) = 0. This polynomial is the minimal polynomial.
Proof: Let p (x) be the monic polynomial having smallest degree such that p (a) = 0. Then p (x)
is irreducible because if not, there would exist a polynomial having smaller degree which has a as a
root. Now suppose q (x) is monic and irreducible such that q (a) = 0.
q (x) = p (x) l (x) + r (x)
where if r (x) ̸= 0, then it has smaller degree than p (x). But in this case, the equation implies
r (a) = 0 which contradicts the choice of p (x). Hence r (x) = 0 and so, since q (x) is irreducible,
l (x) = 1 showing that p (x) = q (x). Definition 16.4.29 For a an algebraic number, let deg (a) denote the degree of the minimal polynomial of a.
Also, here is another definition.
Definition 16.4.30 Let a1 , · · · , am be in A. A polynomial in {a1 , · · · , am } will be an expression of
the form
∑
ak1 ···kn ak11 · · · aknn
k1 ···kn
where the ak1 ···kn are in F, each kj is a nonnegative integer, and all but finitely many of the ak1 ···kn
equal zero. The collection of such polynomials will be denoted by
F [a1 , · · · , am ] .
Now notice that for a an algebraic number, F [a] is a vector space with field of scalars F. Similarly,
for {a1 , · · · , am } algebraic numbers, F [a1 , · · · , am ] is a vector space with field of scalars F. The
following fundamental proposition is important.
Proposition 16.4.31 Let {a1 , · · · , am } be algebraic numbers. Then
dim F [a1 , · · · , am ] ≤
m
∏
deg (aj )
j=1
and for an algebraic number a,
dim F [a] = deg (a)
Every element of F [a1 , · · · , am ] is in A and F [a1 , · · · , am ] is a field.
Proof: Let the minimal polynomial be
p (x) = xn + an−1 xn−1 + · · · + a1 x + a0 .
If q (a) ∈ F [a] , then
q (x) = p (x) l (x) + r (x)
338
CHAPTER 16. VECTOR SPACES
where r (x) has degree less than the degree of p (x) if it is not zero. Thus F [a] is spanned by
{
}
1, a, a2 , · · · , an−1
Since p (x) has smallest degree of all polynomial which have a as a root, the above set is also linearly
independent. This proves the second claim.
Now consider
} definition, F [a1 , · · · , am ] is obtained from all linear combinations
{ the first claim. By
of products of ak11 , ak22 , · · · , aknn where the ki are nonnegative integers. From the first part, it
suffices to consider only kj ≤ deg (aj ). Therefore, there exists a spanning set for F [a1 , · · · , am ]
which has
m
∏
deg (ai )
i=1
entries. By Theorem 16.3.5 this proves the first claim.
Finally consider the last claim. Let g (a1 , · · · , am ) be a polynomial in {a1 , · · · , am } in
F [a1 , · · · , am ]
Since
m
∏
dim F [a1 , · · · , am ] ≡ p ≤
deg (aj ) < ∞,
j=1
it follows
2
p
1, g (a1 , · · · , am ) , g (a1 , · · · , am ) , · · · , g (a1 , · · · , am )
are dependent. It follows g (a1 , · · · , am ) is the root of some polynomial having coefficients in F.
Thus everything in F [a1 , · · · , am ] is algebraic. Why is F [a1 , · · · , am ] a field? Let g (a1 , · · · , am ) be
as just mentioned. Then it has a minimal polynomial,
p (x) = xq + aq−1 xq−1 + · · · + a1 x + a0
where the ai ∈ F. Then a0 ̸= 0 or else the polynomial would not be minimal. Therefore,
(
)
q−1
q−2
g (a1 , · · · , am ) g (a1 , · · · , am )
+ aq−1 g (a1 , · · · , am )
+ · · · + a1 = −a0
and so the multiplicative inverse for g (a1 , · · · , am ) is
g (a1 , · · · , am )
q−1
q−2
+ aq−1 g (a1 , · · · , am )
−a0
+ · · · + a1
∈ F [a1 , · · · , am ] .
The other axioms of a field are obvious. Now from this proposition, it is easy to obtain the following interesting result about the algebraic
numbers.
Theorem 16.4.32 The algebraic numbers A, those roots of polynomials in F [x] which are in G,
are a field.
Proof: By definition, each a ∈ A has a minimal polynomial. Let a ̸= 0 be an algebraic number
and let p (x) be its minimal polynomial. Then p (x) is of the form
xn + an−1 xn−1 + · · · + a1 x + a0
where a0 ̸= 0. Otherwise p(x) would not have minimal degree. Then plugging in a yields
( n−1
)
a
+ an−1 an−2 + · · · + a1 (−1)
a
= 1.
a0
16.4. VECTOR SPACES AND FIELDS∗
339
(an−1 +an−1 an−2 +···+a1 )(−1)
and so a−1 =
∈ F [a]. By the proposition, every element of F [a] is in A
a0
and this shows that for every nonzero element of A, its inverse is also in A. What about products
and sums of things in A? Are they still in A? Yes. If a, b ∈ A, then both a + b and ab ∈ F [a, b] and
from the proposition, each element of F [a, b] is in A. A typical example of what is of interest here is when the field F of scalars is Q, the rational
numbers and the field G is R. However, you can certainly conceive of many other examples by
considering the integers mod a prime, for example (See Problem 4 on Page 321 for example.) or any
of the fields which occur as field extensions in the above.
There is a very interesting thing about F [a1 · · · an ] in the case where F is infinite which says that
there exists a single algebraic γ such that F [a1 · · · an ] = F [γ]. In other words, every field extension
of this sort is a simple field extension. I found this fact in an early version of [5].
Proposition 16.4.33 There exists γ such that F [a1 · · · an ] = F [γ].
Proof: To begin with, consider F [α, β]. Let γ = α + λβ. Then by Proposition 16.4.31 γ is an
algebraic number and it is also clear
F [γ] ⊆ F [α, β]
I need to show the other inclusion. This will be done for a suitable choice of λ. To do this, it suffices
to verify that both α and β are in F [γ].
Let the minimal polynomials of α and β be f (x) and g (x) respectively. Let the distinct roots
of f (x) and g (x) be {α1 , α2 , · · · , αn } and {β 1 , β 2 , · · · , β m } respectively. These roots are in a field
which contains splitting fields of both f (x) and g (x). Let α = α1 and β = β 1 . Now define
h (x) ≡ f (α + λβ − λx) ≡ f (γ − λx)
so that h (β) = f (α) = 0. It follows (x − β) divides (both h (x)
) and g (x). If (x − η) is a different
linear factor of both g (x) and h (x) then it must be x − β j for some β j for some j > 1 because
these are the only factors of g (x) . Therefore, this would require
( )
(
)
0 = h β j = f α1 + λβ 1 − λβ j
and so it would be the case that α1 + λβ 1 − λβ j = αk for some k. Hence
λ=
αk − α1
β1 − βj
Now there are finitely many quotients of the above form and if λ is chosen to not be any of them,
then the above cannot happen and so in this case, the only linear factor of both g (x) and h (x) will
be (x − β). Choose such a λ.
Let ϕ (x) be the minimal polynomial of β with respect to the field F [γ]. Then this minimal
polynomial must divide both h (x) and g (x) because h (β) = g (β) = 0. However, the only factor
these two have in common is x−β and so ϕ (x) = x−β which requires β ∈ F [γ] . Now also α = γ −λβ
and so α ∈ F [γ] also. Therefore, both α, β ∈ F [γ] which forces F [α, β] ⊆ F [γ] . This proves the
proposition in the case that n = 2. The general result follows right away by observing that
F [a1 · · · an ] = F [a1 · · · an−1 ] [an ]
and using induction. When you have a field F, F (a) denotes the smallest field which contains both F and a. When a
is algebraic over F, it follows that F (a) = F [a] . The latter is easier to think about because it just
involves polynomials.
16.4.4
The Lindemannn Weierstrass Theorem And Vector Spaces
As another application of the abstract concept of vector spaces, there is an amazing theorem due to
Weierstrass and Lindemannn.
340
CHAPTER 16. VECTOR SPACES
Theorem 16.4.34 Suppose a1 , · · · , an are algebraic numbers, roots of a polynomial with rational
coefficients, and suppose α1 , · · · , αn are distinct algebraic numbers. Then
n
∑
ai eαi ̸= 0
i=1
In other words, the {eα1 , · · · , eαn } are independent as vectors with field of scalars equal to the
algebraic numbers.
A number is transcendental, as opposed to algebraic, if it is not a root of a polynomial which
has integer (rational) coefficients. Most numbers are this way but it is hard to verify that specific
numbers are transcendental. That π is transcendental follows from
e0 + eiπ = 0.
By the above theorem, this could not happen if π were algebraic because then iπ would also be
algebraic. Recall these algebraic numbers form a field and i is clearly algebraic, being a root of
x2 + 1. This fact about π was first proved by Lindemannn in 1882 and then the general theorem
above was proved by Weierstrass in 1885. This fact that π is transcendental solved an old problem
called squaring the circle which was to construct a square with the same area as a circle using a
straight edge and compass. It can be shown that the fact π is transcendental implies this problem
is impossible.1
16.5
Exercises
{
}
1. Let M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : |u1 | ≤ 4 . Is M a subspace? Explain.
{
}
2. Let M = u = (u1 , u2 , u3 , u4 ) ∈ R4 : sin (u1 ) = 1 . Is M a subspace? Explain.
3. If you have 5 vectors in F5 and the vectors are linearly independent, can it always be concluded
they span F5 ? Here F is an arbitrary field. Explain.
4. If you have 6 vectors in F5 , is it possible they are linearly independent? Here F is an arbitrary
field. Explain.
5. Show in any vector space, 0 is unique. This is done in the book. You do it yourself.
6. ↑In any vector space, show that if x + y = 0, then y = −x.This is done in the book. You do
it yourself.
7. ↑Show that in any vector space, 0x = 0. That is, the scalar 0 times the vector x gives the
vector 0. This is done in the book. You do it yourself.
8. ↑Show that in any vector space, (−1) x = −x.This is done in the book. You do it yourself.
9. Let X be a vector space and suppose {x1 , · · · , xk } is a set of vectors from X. Show that 0 is
in span (x1 , · · · , xk ) .This is done in the book. You do it yourself.
10. Let X consist of the real valued functions which are defined on an interval [a, b] . For f, g ∈
X, f + g is the name of the function which satisfies (f + g) (x) = f (x) + g (x) and for α a
real number, (αf ) (x) ≡ α (f (x)). Show this is a vector space with field of scalars equal to R.
Also explain why it cannot possibly be finite dimensional.
1 Gilbert, the librettist of the Savoy operas, may have heard about this great achievement. In Princess Ida which
opened in 1884 he has the following lines. “As for fashion they forswear it, so the say - so they say; and the circle they will square it some fine day some fine day.” Of course it had been proved impossible to do this a couple of years
before.
16.5. EXERCISES
341
11. Let S be a nonempty set and let V denote the set of all functions which are defined on S and
have values in W a vector space having field of scalars F. Also define vector addition according
to the usual rule, (f + g) (s) ≡ f (s) + g (s) and scalar multiplication by (αf ) (s) ≡ αf (s).
Show that V is a vector space with field of scalars F.
12. Verify that any field F is a vector space with field of scalars F. However, show that R is a
vector space with field of scalars Q.
13. Let F be a field and consider functions defined on {1, 2, · · · , n} having values in F. Explain
how, if V is the set of all such functions, V can be considered as Fn .
14. Let V be the set of all functions defined on N ≡ {1, 2, · · · } having values in a field F such
that vector addition and scalar multiplication are defined by (f + g) (s) ≡ f (s) + g (s) and
(αf ) (s) ≡ αf (s) respectively, for f , g ∈ V and α ∈ F. Explain how this is a vector space and
show that for ei given by
{
1 if i = k
ei (k) ≡
,
0 if i ̸= k
∞
the vectors {ek }k=1 are linearly independent.
15. Suppose, in the context of Problem 10 you have smooth functions {y1 , y2 , · · · , yn } (all derivatives exist) defined on an interval [a, b] . Then the Wronskian of these functions is the determinant


···
yn (x)
y1 (x)


···
yn′ (x) 
 y1′ (x)

W (y1 , · · · , yn ) (x) = det 
..
..


.
.


(n−1)
(n−1)
(x)
y1
(x) · · · yn
Show that if W (y1 , · · · , yn ) (x) ̸= 0 for some x, then the functions are linearly independent.
16. Give an example of two functions, y1 , y2 defined on [−1, 1] such that
W (y1 , y2 ) (x) = 0
for all x ∈ [−1, 1] and yet {y1 , y2 } is linearly independent.
17. Let the vectors be polynomials of degree no more than 3. Show that with the usual definitions
of scalar multiplication and addition wherein, for p (x) a polynomial, (αp) (x) = αp (x) and for
p, q polynomials (p + q) (x) ≡ p (x) + q (x) , this is a vector space.
{
}
18. In the previous problem show that a basis for the vector space is 1, x, x2 , x3 .
19. Let V be the polynomials of degree no more than 3. Determine which of the following are
bases for this vector space.
{
}
(a) x + 1, x3 + x2 + 2x, x2 + x, x3 + x2 + x
{
}
(b) x3 + 1, x2 + x, 2x3 + x2 , 2x3 − x2 − 3x + 1
20. In the context of the above problem, consider polynomials
{ 3
}
ai x + bi x2 + ci x + di , i = 1, 2, 3, 4
Show that this collection of polynomials is linearly independent on an interval [a, b] if and only
if


a1 b1 c1 d1
 a b c d 
 2 2 2
2 


 a3 b3 c3 d3 
a4
is an invertible matrix.
b4
c4
d4
342
CHAPTER 16. VECTOR SPACES
√
21. Let the field of scalars be Q, the rational numbers and let the vectors be of the form a + b 2
where a, b are rational numbers. Show that this collection of vectors is a vector space with
field of scalars Q and give a basis for this vector space.
22. Suppose V is a finite dimensional vector space. Based on the exchange theorem above, it was
shown that any two bases have the same number of vectors in them. Give a different proof of
this fact using the earlier material in the book. Hint: Suppose {x1 , · · · , xn } and {y1 , · · · , ym }
are two bases with m < n. Then define
ϕ : Fn 7→ V, ψ : Fm 7→ V
by
ϕ (a) ≡
n
∑
ak xk , ψ (b) ≡
m
∑
bj yj
j=1
k=1
Consider the linear transformation, ψ −1 ◦ ϕ. Argue it is a one to one and onto mapping from
Fn to Fm . Now consider a matrix of this linear transformation and its row reduced echelon
form.
23. This and the following problems will present most of a differential equations course. To begin
with, consider the scalar initial value problem
y ′ = ay, y (t0 ) = y0
When a is real, show the unique solution to this problem is y = y0 ea(t−t0 ) . Next suppose
y ′ = (a + ib) y, y (t0 ) = y0
(16.8)
where y (t) = u (t) + iv (t) . Show there exists a unique solution and it is
y (t) = y0 ea(t−t0 ) (cos b (t − t0 ) + i sin b (t − t0 )) ≡ e(a+ib)(t−t0 ) y0 .
(16.9)
Next show that for a real or complex there exists a unique solution to the initial value problem
y ′ = ay + f, y (t0 ) = y0
and it is given by
∫
t
y (t) = ea(t−t0 ) y0 + eat
e−as f (s) ds.
t0
Hint: For the first part write as y ′ − ay = 0 and multiply both sides by e−at . Then explain
why you get
)
d ( −at
e y (t) = 0, y (t0 ) = 0.
dt
Now you finish the argument. To show uniqueness in the second part, suppose
y ′ = (a + ib) y, y (0) = 0
and verify this requires y (t) = 0. To do this, note
y ′ = (a − ib) y, y (0) = 0
and that
d
2
|y (t)|
dt
= y ′ (t) y (t) + y ′ (t) y (t) = (a + ib) y (t) y (t) + (a − ib) y (t) y (t)
2
2
= 2a |y (t)| , |y| (t0 ) = 0
Thus from the first part |y (t)| = 0e−2at = 0. Finally observe by a simple computation that
16.8 is solved by 16.9. For the last part, write the equation as
2
y ′ − ay = f
and multiply both sides by e−at and then integrate from t0 to t using the initial condition.
16.5. EXERCISES
343
24. ↑Now consider A an n × n matrix. By Schur’s theorem there exists unitary Q such that
Q−1 AQ = T
where T is upper triangular. Now consider the first order initial value problem
x′ = Ax, x (t0 ) = x0 .
Show there exists a unique solution to this first order system. Hint: Let y = Q−1 x and so
the system becomes
y′ = T y, y (t0 ) = Q−1 x0
(16.10)
T
Now letting y = (y1 , · · · , yn ) , the bottom equation becomes
(
)
yn′ = tnn yn , yn (t0 ) = Q−1 x0 n .
Then use the solution you get in this to get the solution to the initial value problem which
occurs one level up, namely
(
)
′
yn−1
= t(n−1)(n−1) yn−1 + t(n−1)n yn , yn−1 (t0 ) = Q−1 x0 n−1
Continue doing this to obtain a unique solution to 16.10.
25. ↑Now suppose Φ (t) is an n × n matrix of the form
(
Φ (t) = x1 (t) · · ·
)
(16.11)
xn (t)
where x′k (t) = Axk (t) . Explain why
Φ′ (t) = AΦ (t)
if and only if Φ (t) is given in the form of 16.11. Also explain why if c ∈ Fn , y (t) ≡ Φ (t) c
solves the equation
y′ (t) = Ay (t) .
26. ↑In the above problem, consider the question whether all solutions to
x′ = Ax
(16.12)
are obtained in the form Φ (t) c for some choice of c ∈ Fn . In other words, is the general
solution to this equation Φ (t) c for c ∈ Fn ? Prove the following theorem using linear algebra.
Theorem 16.5.1 Suppose Φ (t) is an n × n matrix which satisfies
Φ′ (t) = AΦ (t) .
−1
Then the general solution to 16.12 is Φ (t) c if and only if Φ (t) exists for some t. Further−1
−1
more, if Φ′ (t) = AΦ (t) , then either Φ (t)
exists for all t or Φ (t)
never exists for any
t.
(det (Φ (t)) is called the Wronskian and this theorem is sometimes called the Wronskian alternative.)
Hint: Suppose first the general solution is of the form Φ (t) c where c is an arbitrary constant
−1
−1
vector in Fn . You need to verify Φ (t) exists for some t. In fact, show Φ (t) exists for every
−1
t. Suppose then that Φ (t0 ) does not exist. Explain why there exists c ∈ Fn such that there
is no solution x to c = Φ (t0 ) x. By the existence part of Problem 24 there exists a solution to
−1
x′ = Ax, x (t0 ) = c, but this cannot be in the form Φ (t) c. Thus for every t, Φ (t) exists.
344
CHAPTER 16. VECTOR SPACES
−1
Next suppose for some t0 , Φ (t0 )
Then both z (t) , Φ (t) c solve
exists. Let z′ = Az and choose c such that z (t0 ) = Φ (t0 ) c.
x′ = Ax, x (t0 ) = z (t0 )
Apply uniqueness to conclude z = Φ (t) c. Finally, consider that Φ (t) c for c ∈ Fn either is the
−1
general solution or it is not the general solution. If it is, then Φ (t) exists for all t. If it is
−1
not, then Φ (t) cannot exist for any t from what was just shown.
27. ↑Let Φ′ (t) = AΦ (t) . Then Φ (t) is called a fundamental matrix if Φ (t)
there exists a unique solution to the equation
−1
exists for all t. Show
x′ = Ax + f , x (t0 ) = x0
(16.13)
and it is given by the formula
−1
x (t) = Φ (t) Φ (t0 )
∫
t
x0 + Φ (t)
Φ (s)
−1
f (s) ds
t0
Now these few problems have done virtually everything of significance in an entire undergraduate differential equations course, illustrating the superiority of linear algebra. The above
formula is called the variation of constants formula.
Hint: Uniqueness is easy. If x1 , x2 are two solutions then let u (t) = x1 (t) − x2 (t) and argue
u′ = Au, u (t0 ) = 0. Then use Problem 24. To verify there exists a solution, you could just
differentiate the above formula using the fundamental theorem of calculus and verify it works.
Another way is to assume the solution in the form
x (t) = Φ (t) c (t)
and find c (t) to make it all work out. This is called the method of variation of parameters.
−1
28. ↑Show there exists a special Φ such that Φ′ (t) = AΦ (t) , Φ (0) = I, and Φ (t)
t. Show using uniqueness that
−1
Φ (−t) = Φ (t)
exists for all
and that for all t, s ∈ R
Φ (t + s) = Φ (t) Φ (s)
Explain why with this special Φ, the solution to 16.13 can be written as
∫ t
x (t) = Φ (t − t0 ) x0 +
Φ (t − s) f (s) ds.
t0
Hint: Let Φ (t) be such that the j th column is xj (t) where
x′j = Axj , xj (0) = ej .
Use uniqueness as required.
29. ∗ Using the Lindemann Weierstrass theorem show that if σ is an algebraic number sin σ, cos σ, ln σ,
and e are all transcendental. Hint: Observe, that
ee−1 + (−1) e0 = 0, 1eln(σ) + (−1) σe0 = 0,
1
1 iσ
e − e−iσ + (−1) sin (σ) e0 = 0.
2i
2i
Chapter 17
Inner Product Spaces
17.1
Basic Definitions And Examples
An inner product space V is a vector space which also has an inner product. It is usually assumed,
when considering inner product spaces that the field of scalars is either F = R or C. This terminology
has already been considered in the context of Fn . In this section, it will be assumed that the field
of scalars is C, the complex numbers, unless specified to be something else. An inner product is a
mapping ⟨·, ·⟩ : V × V 7→ C which satisfies the following axioms.
Axioms For Inner Product
1. ⟨u, v⟩ ∈ C, ⟨u, v⟩ = ⟨v, u⟩.
2. If a, b are numbers and u, v, z are vectors then ⟨(au + bv) , z⟩ = a ⟨u, z⟩ + b ⟨v, z⟩ .
3. ⟨u, u⟩ ≥ 0 and it equals 0 if and only if u = 0.
Note this implies ⟨x,αy⟩ = α ⟨x, y⟩ because
⟨x,αy⟩ = ⟨αy, x⟩ = α ⟨y, x⟩ = α ⟨x, y⟩
Example 17.1.1 Let V be the continuous complex valued functions defined on a finite closed interval
I. Define an inner product as follows.
∫
⟨f, g⟩ ≡ f (x) g (x)p (x) dx
I
where p (x) some function which is strictly positive on the closed interval I. It is understood in
writing this that
∫
∫
∫
f (x) + ig (x) dx ≡ f (x) dx + i g (x) dx
I
I
I
Then with this convention, the usual calculus theorems hold about evaluating integrals using the
fundamental theorem of calculus and so forth. You simply apply these theorems to the real and
imaginary parts of a complex valued function.
Example 17.1.2 Let V be the polynomials of degree at most n which are defined on a closed interval
I and let {x0 , x1 , · · · , xn } be n + 1 distinct points in I. Then define
⟨f, g⟩ ≡
n
∑
f (xk ) g (xk )
k=0
345
346
CHAPTER 17. INNER PRODUCT SPACES
This last example clearly satisfies all the axioms for an inner product except for the one which
says that ⟨u, u⟩ = 0 if and only if u = 0. Suppose then that ⟨f, f ⟩ = 0. Then f must vanish at
n + 1 distinct points but f is a polynomial of degree n. Therefore, it has at most n zeros unless it
is identically equal to 0. Hence the second case holds and so f equals 0.
Example 17.1.3 Let V be any complex vector space and let {v1 , · · · , vn } be a basis. Decree that
⟨vi , vj ⟩ = δ ij .
Then define
⟨ n
∑
j=1
c j vj ,
n
∑
⟩
dk vk
≡
k=1
∑
cj dk ⟨vj , vk ⟩ =
j,k
n
∑
ck dk
k=1
This makes the complex vector space into an inner product space.
∞
Example 17.1.4 Let V consist of sequences a = {ak }k=1 , ak ∈ C, with the property that
∞
∑
2
|ak | < ∞
k=1
and the inner product is then defined as
⟨a, b⟩ ≡
∞
∑
ak bk
k=1
All of the axioms of the inner product are obvious for this example except the most basic one
which says that the inner product has values in C. Why does the above sum even converge? It
converges from a comparison test.
2
2
|bk |
|ak |
ak bk ≤
+
2
2
and by assumption,
)
(
∞
2
2
∑
|bk |
|ak |
<∞
+
2
2
k=1
and therefore, the given sum which defines the inner product is absolutely convergent. Therefore,
thanks to completeness of C this sum also converges. This fact should be familiar to anyone who
has had a calculus class in the context that the sequences are real valued. The case where they are
complex valued follows right away from a consideration of real and imaginary parts.
By far the most important example of an inner product space is L2 (Ω), the space of Lebesgue
measurable square integrable functions defined on Ω. However, this is a book on algebra, not
analysis, so this example will be ignored.
17.1.1
The Cauchy Schwarz Inequality And Norms
The most fundamental theorem relative to inner products is the Cauchy Schwarz inequality.
Theorem 17.1.5 (Cauchy Schwarz)The following inequality holds for x and y ∈ V, an inner product space.
1/2
1/2
|⟨x, y⟩| ≤ ⟨x, x⟩ ⟨y, y⟩
(17.1)
Equality holds in this inequality if and only if one vector is a multiple of the other.
Proof: Let θ ∈ C such that |θ| = 1 and
θ ⟨x, y⟩ = |⟨x, y⟩|
17.1. BASIC DEFINITIONS AND EXAMPLES
347
⟨
⟩
Consider p (t) ≡ x + θty, x + tθy where t ∈ R. Then from the above list of properties of the dot
product,
0
≤ p (t) = ⟨x, x⟩ + tθ ⟨x, y⟩ + tθ ⟨y, x⟩ + t2 ⟨y, y⟩
= ⟨x, x⟩ + tθ ⟨x, y⟩ + tθ⟨x, y⟩ + t2 (y, y)
= ⟨x, x⟩ + 2t Re θ ⟨x, y⟩ + t2 ⟨y, y⟩
= ⟨x, x⟩ + 2t |⟨x, y⟩| + t2 ⟨y, y⟩
(17.2)
and this must hold for all t ∈ R. Therefore, if ⟨y, y⟩ = 0 it must be the case that |⟨x, y⟩| = 0 also
since otherwise the above inequality would be violated for large negative t. Therefore, in this case,
|⟨x, y⟩| ≤ ⟨x, x⟩
1/2
1/2
⟨y, y⟩
.
In the other case, if ⟨y, y⟩ =
̸ 0, then p (t) ≥ 0 for all t means the graph of y = p (t) is a parabola
which opens up and it either has exactly one real zero in the case its vertex touches the t axis or it
has no real zeros.
t
t
From the quadratic formula this happens exactly when
2
4 |⟨x, y⟩| − 4 ⟨x, x⟩ ⟨y, y⟩ ≤ 0
which is equivalent to 17.1.
It is clear from a computation that if one vector is a scalar multiple of the other that equality
2
holds in 17.1. Conversely, suppose equality does hold. Then this is equivalent to saying 4 |⟨x, y⟩| −
4 ⟨x, x⟩ ⟨y, y⟩ = 0 and so from the quadratic formula, there exists one real zero to p (t) = 0. Call it
t0 . Then
⟨
⟩
2
p (t0 ) ≡ x + θt0 y, x + t0 θy = x + θt0 y = 0
and so x = −θt0 y. So what does the Cauchy Schwarz inequality say in the above examples? In Example 17.1.1 it
says that
(∫
∫
)1/2 (∫
f (x) g (x)p (x) dx ≤
I
)1/2
2
2
|f (x)| p (x) dx
I
|g (x)| p (x) dx
I
With the Cauchy Schwarz inequality, it is possible to obtain the triangle inequality. This is the
inequality in the next theorem. First it is necessary to define the norm or length of a vector. This
is what is in the next definition.
Definition 17.1.6 Let V be an inner product space and let z ∈ V. Then |z| ≡ ⟨z, z⟩
the norm of z and also the length of z.
1/2
. |z| is called
With the definition of length of a vector, here are the main properties of length.
Theorem 17.1.7 For length defined in Definition 17.1.6, the following hold.
|z| ≥ 0 and |z| = 0 if and only if z = 0
(17.3)
If α is a scalar, |αz| = |α| |z|
(17.4)
|z + w| ≤ |z| + |w| .
(17.5)
348
CHAPTER 17. INNER PRODUCT SPACES
Proof: The first two claims are left as exercises. To establish the third,
|z + w|
2
= ⟨z + w, z + w⟩
= ⟨z, z⟩ + ⟨w, w⟩ + ⟨w, z⟩ + ⟨z, w⟩
2
2
2
2
2
2
= |z| + |w| + 2 Re ⟨w, z⟩
≤ |z| + |w| + 2 |⟨w, z⟩|
≤
2
|z| + |w| + 2 |w| |z| = (|z| + |w|) . The properties 17.3 - 17.5 are the axioms for a norm. A vector space which has a norm is called
a normed linear space or a normed vector space.
17.2
The Gram Schmidt Process
The Gram Schmidt process is also valid in an inner product space. If you have a linearly independent
set of vectors, there is an orthonormal set of vectors which has the same span. Recall the definition
of an orthonormal set. It is the same as before.
Definition 17.2.1 Let V be an inner product space and let {ui } be a collection of vectors. It is an
orthonormal set if
⟨uk , uj ⟩ = δ jk .
As before, every orthonormal set of vectors is linearly independent. If
n
∑
ck uk = 0
k=1
n
where {uk }k=1 is an orthonormal set of vectors, why is each ck = 0?
This is true because you can take the inner product of both sides with uj . Then
⟩
⟨ n
∑
⟨
⟩
ck uk , uj = 0, uj .
k=1
The right side equals 0 because
⟨0, u⟩ = ⟨0 + 0, u⟩ = ⟨0, u⟩ + ⟨0, u⟩
Subtracting ⟨0, u⟩ from both sides shows that ⟨0, u⟩ = 0. Therefore, from the properties of the inner
product,
⟨ n
⟩
n
n
∑
∑
∑
⟨
⟩
0 = 0, uj =
ck uk , uj =
ck ⟨uk , uj ⟩ =
ck δ kj = cj .
k=1
k=1
k=1
Since cj was arbitrary, this verifies that an orthonormal set of vectors is linearly independent.
Now consider the Gram Schmidt process.
Lemma 17.2.2 Let {x1 , · · · , xn } be a linearly independent subset of an inner product space V. Then
there exists an orthonormal set of vectors {u1 , · · · , un } which has the property that for each k ≤ n,
span(x1 , · · · , xk ) = span (u1 , · · · , uk ) .
Proof: Let u1 ≡ x1 / |x1 | . Thus for k = 1, span (u1 ) = span (x1 ) and {u1 } is an orthonormal
set. Now suppose for some k < n, u1 , · · · , uk have been chosen such that (uj , ul ) = δ jl and
span (x1 , · · · , xk ) = span (u1 , · · · , uk ). Then define
∑k
xk+1 − j=1 ⟨xk+1 , uj ⟩ uj
uk+1 ≡
,
(17.6)
∑k
xk+1 − j=1 ⟨xk+1 , uj ⟩ uj
17.2. THE GRAM SCHMIDT PROCESS
349
where the denominator is not equal to zero because the xj form a basis, and so
xk+1 ∈
/ span (x1 , · · · , xk ) = span (u1 , · · · , uk )
Thus by induction,
uk+1 ∈ span (u1 , · · · , uk , xk+1 ) = span (x1 , · · · , xk , xk+1 ) .
Also, xk+1 ∈ span (u1 , · · · , uk , uk+1 ) which is seen easily by solving 17.6 for xk+1 and it follows
span (x1 , · · · , xk , xk+1 ) = span (u1 , · · · , uk , uk+1 ) .
If l ≤ k,

⟨uk+1 , ul ⟩
= C ⟨xk+1 , ul ⟩ −
k
∑

⟨xk+1 , uj ⟩ ⟨uj , ul ⟩
j=1

= C ⟨xk+1 , ul ⟩ −
k
∑

⟨xk+1 , uj ⟩ δ lj 
j=1
= C (⟨xk+1 , ul ⟩ − ⟨xk+1 , ul ⟩) = 0.
n
The vectors, {uj }j=1 , generated in this way are therefore orthonormal because each vector has unit
length. As in the case of Fn , if you have a finite dimensional subspace of an inner product space, you
can begin with a basis and then apply the Gram Schmidt process above to obtain an orthonormal
basis.
There is nothing wrong with the above algorithm, but when you use it, it tends to get pretty
intricate and it is easy to get lost in the details. There is a way to simplify it to produce fewer
steps using matrices. I will illustrate in the case of three vectors. Say {u1 , u2 , u3 } is linearly
independent and you wish to find an orthonormal set {v1 , v2 , v3 } which has the same span such
that span (u1 , · · · , uk ) = span (v1 , · · · , vk ) for each k = 1, 2, 3. Then you would have
(
) (
)
v1 v2 v3 = u1 u2 u3 R
where R is an upper triangular matrix. Then
⟨
δ jk = ⟨vj , vk ⟩ =
n
∑
ur Rrj ,
r=1
=
∑
n
∑
⟩
us Rsk
s=1
Rrj ⟨ur , us ⟩ Rsk
r,s
Let G be the matrix whose rs entry is ⟨ur , us ⟩ . This is called the Grammian matrix. Then the
above reduces to the following matrix equation.
I = RT GR
Taking inverses of both sides yields
( )−1
I = R−1 G−1 RT
Then it follows that
RRT = G−1 .
(17.7)
350
CHAPTER 17. INNER PRODUCT SPACES
Example 17.2.3 Let the real inner product space V consist of the continuous real functions defined
on [0, 1] with the inner product given by
∫ 1
⟨f, g⟩ ≡
f (x) g (x) dx
0
{
}
and consider the functions (vectors) 1, x, x2 , x3 . Show this is a linearly independent set of vectors
and obtain an orthonormal set of vectors having the same span.
First, why is this a linearly independent set
Page 341. You consider the Wronskian of these

1 x
 0 1

det 
 0 0
0 0
of vectors? This follows easily from Problem 15 on
functions.

x2 x 3
2x 3x2 

 ̸= 0.
2
6x 
0
6
Therefore, the vectors are linearly independent. Now following the above procedure with matrices,
let


a1 a2 a3 a4
 0 a a
a7 


5
6
R=

 0 0 a8 a9 
0 0 0 a10
Also it is necessary to compute the Grammian.
∫1
∫1 2
 ∫1
dx
xdx
x dx
0
0
0
 ∫ 1 xdx ∫ 1 x2 dx ∫ 1 x3 dx
 0
∫0
∫0
 ∫1 2
 0 x dx 01 x3 dx 01 x4 dx
∫1 3
∫1
∫1
x dx 0 x4 dx 0 x5 dx
0
∫1 3
x dx
∫01 4
x dx
∫01 5
x dx
∫01 6
x dx
0


 
 
=
 
1
1
2
1
3
1
4
1
2
1
3
1
4
1
5
1
3
1
4
1
5
1
6
1
4
1
5
1
6
1
7





You also need the inverse of this Grammian. However, a computer algebra system can provide this
right away.


16
−120
240
−140
 −120 1200 −2700 1680 


G−1 = 

 240 −2700 6480 −4200 
−140 1680 −4200 2800
Now it just remains to find the ai .










a1
0
0
0
a2
a5
0
0
a3
a6
a8
0
a4
a7
a9
a10





a1
0
0
0
a2
a5
0
0
a21 + a22 + a23 + a24 a2 a5 + a3 a6 + a4 a7
a2 a5 + a3 a6 + a4 a7
a25 + a26 + a27
a3 a8 + a4 a9
a6 a8 + a7 a9
a4 a10
a7 a10

16
−120
240
 −120 1200 −2700


 240 −2700 6480
−140 1680 −4200
a3
a6
a8
0
a4
a7
a9
a10
T


 =

a3 a8 + a4 a9
a6 a8 + a7 a9
a28 + a29
a9 a10

−140
1680 


−4200 
2800
a4 a10
a7 a10
a9 a10
a210



=

17.3. APPROXIMATION AND LEAST SQUARES
351
Thus you can take (There is more than one solution.)
√
√
√
√
√
√
a10 = 2800 = 20 7, a9 = −30 7, a8 = 6 5, a7 = 12 7, a6 = −6 5
√
√
√
√
a5 = 2 3, a4 = − 7, a3 = 5, a2 = − 3, a1 = 1
Thus the desired orthonormal basis is given by

(
1
x x2
)



x3
√
√
− 3
5
√
√
2 3 −6 5
√
0
6 5
0
0
1
0
0
0
√
− 7
√
12 7
√
−30 7
√
20 7





which yields
√
√
√
√
√
√
√
√
√
1, 2 3x − 3, 6 5x2 − 6 5x + 5, 20 7x3 − 30 7x2 + 12 7x − 7
17.3
Approximation And Least Squares
Let V be an inner product space and let U be a finite dimensional subspace. Given y ∈ V, how can
you find the vector of U which is closest to y out of all such vectors in U ? Does there even exist
such a closest vector? The following picture is suggestive of the conclusion of the following lemma.
It turns out that pictures like this do not mislead when you are dealing with inner product spaces
in any number of dimensions.
y
w
z
U
Note that in the picture, z is a point in U and also w is a point of U . The following lemma states
that for z to be closest to y out of all vectors in U, the vector from z to y should be perpendicular
to any vector w ∈ U. Since U is a subspace, this is the same as saying that the vector y − z is
perpendicular to the vector w − z which is the situation illustrated by the above picture.
Lemma 17.3.1 Suppose y ∈ V, an inner product space and U is a subspace of V. Then
|y − z| ≤ |y − w|
for all w ∈ U if and only if, for all w ∈ U,
⟨y − z, w⟩ = 0.
(17.8)
Furthermore, there is at most one z which minimizes |y − w| for w ∈ U .
Proof: First suppose condition 17.8. Letting w ∈ U, and using the properties of the inner
product and the definition of the norm,
|y − w|
2
2
2
2
= |y − z + z − w| = |y − z| + |z − w| + 2 Re ⟨y − z, z − w⟩
2
= |y − z| + |z − w|
2
It follows then that |y − w| is minimized when w = z.Next suppose z is a minimizer. Then pick
w ∈ U and let t ∈ R. Let θ ∈ C be such that |θ| = 1 and θ ⟨y − z, w⟩ = |⟨y − z, w⟩|. Then
t 7→ (y − z) +tθw
2
352
CHAPTER 17. INNER PRODUCT SPACES
has a minimum when t = 0. But from the axioms of the inner product and definition of the norm,
this function of t equals
⟨
⟩
2
2
|y − z| + |tθw| + 2t Re y − z,θw
2
2
2
2
= |y − z| + |tθw| + 2t Re θ ⟨y − z, w⟩
= |y − z| + t2 |w| + 2t |⟨y − z, w⟩|
Hence its derivative when t = 0 which is 2 |⟨y − z, w⟩| equals 0.
Suppose now that zi , i = 1, 2 both are minimizers. Then, as above,
2
2
|y − z1 | = |y − z2 | + |z1 − z2 |
2
2
2
and this is a contradiction unless z1 = z2 because |y − z1 | = |y − z2 | . This z described above is called the orthogonal projection of y onto U . The picture suggests
why it is called this. The vector y − z is perpendicular to the vectors in U .
With the above lemma, here is a theorem about existence, uniqueness and properties of a minimizer. The following theorem shows that the orthogonal projection is obtained by the linear transformation given by the formula
n
∑
Ty ≡
⟨y, uk ⟩ uk
k=1
Note that the formula as well as the geometric interpretation suggested in the above picture shows
that T 2 = T .
Theorem 17.3.2 Let V be an inner product space and let U be an n dimensional subspace of V .
Then if y ∈ V is given, there exists a unique x ∈ U such that
|y − x| ≤ |y − w|
for all w ∈ U and in addition, there is a formula for x in terms of any orthonormal basis for
U, {u1 , · · · , un },
n
∑
x=
⟨y, uk ⟩ uk
k=1
Proof: By Lemma 17.3.1 there is at most one minimizer and it is characterized by the condition
⟨y − x, w⟩ = 0
n
for all w ∈ U . Let {uk }k=1 be an orthonormal basis for U . By the Gram Schmidt process, Lemma
17.2.2, there exists such an orthonormal basis. Now it only remains to verify that
⟨
⟩
n
∑
y−
⟨y, uk ⟩ uk , w = 0
k=1
for all w. Since
n
{uk }k=1
is a basis, it suffices to verify that
⟩
⟨
n
∑
⟨y, uk ⟩ uk , ul = 0, all l = 1, 2, · · · , n
y−
k=1
However, from the properties of the inner product,
⟨
⟩
n
n
∑
∑
y−
⟨y, uk ⟩ uk , ul = ⟨y, ul ⟩ −
⟨y, uk ⟩ ⟨uk , ul ⟩
k=1
= ⟨y, ul ⟩ −
k=1
n
∑
k=1
⟨y, uk ⟩ δ kl = ⟨y, ul ⟩ − ⟨y, ul ⟩ = 0. 17.3. APPROXIMATION AND LEAST SQUARES
353
n
Note it follows that for any orthonormal basis {uk }k=1 , the same unique vector x is obtained as
n
∑
⟨y, uk ⟩ uk
k=1
and it is the unique minimizer of w7→|y − w|. This is stated in the following corollary for the sake
of emphasis.
Corollary 17.3.3 Let V be an inner product space and let U be an n dimensional subspace of V.
n
n
Then for y given in V, and {uk }k=1 , {vk }k=1 two orthonormal bases for U,
n
∑
⟨y, uk ⟩ uk =
k=1
n
∑
⟨y, vk ⟩ vk
k=1
The scalars ⟨y, uk ⟩ are called the Fourier coefficients.
Example 17.3.4 Let V denote the real inner product space consisting of continuous functions defined on [0, 1] with the inner product
∫
1
⟨f, g⟩ ≡
f (x) g (x) dx.
0
(
)
Let U = span 1, x, x2 . It is desired to find the vector (function) in U which is closest to sin in the
norm determined by this inner product. Thus it is desired to minimize
(∫
1
)1/2
2
|sin (x) − p (x)| dx
0
out of all functions p contained in U .
By Example 17.2.3, an orthonormal basis for U is
{
√
√
√
√
√ }
1, 2 3x − 3, 6 5x2 − 6 5x + 5
Then by Theorem 17.3.2, the closest vector (function) in U to sin can be computed as follows. First
determine the Fourier coefficients.
∫ 1
1 sin (x) dx = 1 − cos (1)
0
∫
∫
1
( √
√ )
√
2 3x − 3 sin (x) dx = 3 (− cos 1 + 2 sin 1 − 1)
0
1
( √
√
√ )
√
6 5x2 − 6 5x + 5 sin (x) dx = 5 (11 cos 1 + 6 sin 1 − 11)
0
Next, from Theorem 17.3.2, the closest point to sin is
(√
)( √
√ )
(1 − cos (1)) +
3 (− cos 1 + 2 sin 1 − 1) 2 3x − 3
(√
)( √
√
√ )
+
5 (11 cos 1 + 6 sin 1 − 11) 6 5x2 − 6 5x + 5
Simplifying and approximating things like sin 1, this yields the following for the approximation to
sin x.
−0.235 46x2 + 1. 091 3x − 7. 464 9 × 10−3
If this is graphed along with sin x for x ∈ [0, 1] the result is as follows. One of the functions is
354
CHAPTER 17. INNER PRODUCT SPACES
represented by the solid line and the other by the dashed line.There are two
graphs. The first is the least squares approximation of the given function and the
second is the result of using the Taylor series, both up to degree 2. You see the
difference. The approximation using the inner product norm, called mean square
approximation, attempts to approximate the given function on the whole interval
while the Taylor series approximation is only good for small x.
17.4
Fourier Series
One of the most important applications of these ideas about approximation is to
Fourier series. Much more can be said about these than will be presented here. However, Theorem
17.3.2 is a very useful framework for discussing these series.
For x ∈ R, define eix by the following formula
eix ≡ cos x + i sin x
( )′
The reason for defining it this way is that ei0 = 1, and eix = ieix if you use this definition. Also
it follows from the trigonometry identities that ei(x+y) = eix eiy . This is because
eix eiy = (cos x + i sin x) (cos y + i sin y)
= cos x cos y − sin x sin y + i (sin x cos y + cos x sin y)
= cos (x + y) + i sin (x + y) = ei(x+y)
In addition, eix = e−ix because
eix = cos x − i sin x = cos (−x) + i sin (−x) = e−ix
It follows that the functions
√1 eikx
2π
for
k ∈ {−n, − (n − 1) , · · · , −1, 0, 1, · · · , (n − 1) , n} ≡ In
form an orthonormal set in V, the inner product space of continuous functions defined on [−π, π]
with the inner product given by
∫ π
f (x) g (x)dx ≡ ⟨f, g⟩ .
−π
I will verify this now. Let k, l be two integers in In , k ̸= l
∫ π
∫ π
ei(k−l)x π
eikx eilx dx =
ei(k−l)x dx =
|
i (k − l) −π
−π
−π
= cos (k − l) π − cos (k − l) (−π) = 0
Also
∫
π
−π
1
1
1
√ eikx √ eikx dx =
2π
2π
2π
∫
π
e0x dx = 1.
−π
Example 17.4.1 Let V be the inner product space of piecewise
functions defined on
{
}continuous
n
[−π, π] and let U denote the span of the functions (vectors) eikx k=−n . Let
{
f (x) =
1 if x ≥ 0
−1 if x < 0
Find the vector of U which is closest to f in the mean square sense (In the norm defined by this
inner product).
17.5. THE DISCREET FOURIER TRANSFORM
355
First of all, you need to find the Fourier coefficients. Since x 7→ cos (x) is even and x 7→ sin (x)
is odd,
∫ π
∫
−i2 π
1 −ikx
f (x) √ e
dx = √
sin (−kx) dx
2π
2π 0
−π
√
∫ π
−i 2 1 − cos (kπ)
= √
,
f (x) dx = 0.
k
π
−π
Therefore, the best approximation is
) √
n (
∑
−i 1 − cos (kπ)
2
√
√ eikx
k
π
2π
k=−n
The term for k can be combined with the term for −k to yield
(
(
) √
) √
)
2 ( ikx
2
−i 1 − cos (kπ)
−i 1 − cos (kπ)
−ikx
√
√
√ (2i sin kx)
e −e
= √
k
k
π
π
2π
2π
)
(
2 1 − cos (kπ)
=
sin kx
π
k
The terms when k is even are all 0. Therefore, the above reduces to
)
n (
4∑
1
sin (2k − 1) x
π
2k − 1
k=1
In the case where n = 4, the graph of the function f being approximated along with the above function which is approximating it are as shown in the following picture.
This sum which
delivers the closest point in U will be denoted by Sn f.
Note how the approximate function, closest in the mean square
norm, is not equal to the given function at very many points but is
trying to be close to it across the entire interval [−π, π], except for a
small interval centered at 0. You might try doing similar graphs on
a calculator or computer in which you take larger and larger values
of n. What will happen is that there will be a little bump near the point of discontinuity which
won’t go away, but this little bump will get thinner and thinner. The reason this must happen is
roughly because the functions in the sum are continuous and the function being approximated is not.
Therefore, convergence cannot take place uniformly. This is all I will say about these considerations
because this is not an analysis book. See Problem 20 below for a discussion of where the Fourier
series does converge at jumps.
17.5
The Discreet Fourier Transform
Everything done above for the Fourier series on [−π, π] could have been done just as easily on [0, 2π]
because all the functions are periodic of period 2π. Thus, for f a function defined on [0, 2π] , you
could consider the partial sums of the Fourier series on [0, 2π]
n
∑
ak eikx
k=−n
where
ak =
1
2π
∫
2π
f (y) e−iky dy
0
and all the results of the last section continue to apply except now it is on the new interval. This is
done to make the presentation of what follows easier to write.
356
CHAPTER 17. INNER PRODUCT SPACES
The idea is that maybe you don’t know what the function f is at all points, only at certain points
xj = j
2π
, j = 0, 1, · · · , N − 1
N
Then instead of the integral given above, you could write a Riemann sum which approximates it.
I will simply write the left Riemann sum. This yields the approximation bk for ak . Assuming f is
continuous, the approximation would improve as N → ∞.
bk =
)
N −1 (
N −1
2π 2π
1 ∑
2π
1 ∑ ( −i 2π )kj
yj
f j
e N
e−ik(j N )
=
2π j=0
N
N
N j=0
where yj is defined to be the value of the function at xj . This is called the discreet Fourier transform.
2π
In terms of matrix multiplication, let ω N = e−i N . Then



N −1
(N −1)(N −1) 
1 (ω N )
· · · (ω N )
b−(N −1)



 .
..
..
..



 ..

.
.
.




y
0




(N −1)


 b−1
 1

ωN
···
(ω N )

  y1 


1






=
1
···
1
b0
 N 1

  .. 



 . 
(N −1)
b1
ωN
···
ωN



 1



 .
yN −1
..
..
..


 .



 .

.
.
.
bN −1
1
−1
ωN
N
···
(N −1)(N −1)
ωN
Thus you can find these approximate values by matrix multiplication.
Example 17.5.1 Suppose you have the following table of values for the function f.


0
1


 π/2
2 


 π
−1 




 3π/2 1 
2π
2
Note that the above only uses the first four values.
In this case, N = 4 and so ω N = e−i(π/2) = −i. Then the approximate Fourier coefficients are
given by




1
i3
i6
i9
b−3
( 2 )2 ( 2 )3 

 b

2
 1


i
i
i
 −2 


1




2
3
 b−1 
i
i
i
 1
 2 




 b0  = 1 


1
1
1 

 4 1



−1




1
−i
−1
i
 b1 




1
 1 (−i)2 (−i)4 (−i)6 
 b2 


3
6
9
b3
1 (−i) (−i) (−i)




1 −i −1 i
2−i
 1 −1 1 −1  
 −3 





1




 1 i −1 −i  
 2+i 





1
 2  1
= 
1 1
1
1 

=  3 



4
  −1  4 

 1 −i −1 i 
 2−i 




1
 1 −1 1 −1 
 −3 
1 i −1 −i
2+i
17.6. EXERCISES
357
It follows that the approximate Fourier series for the given function is
)
1(
(2 − i) e−3ix + (−3) e−2ix + (2 + i) e−ix + 3 + (2 − i) eix + (−3) e2ix + (2 + i) e3ix
4
This simplifies to
3
+2
4
(
)
(
)
1
1
3
1
1
cos x − sin x − cos (2x) + 2
cos 3x − sin 3x
2
4
2
2
4
If you graph this, it will not do all that well in approximating some functions which have the given
values at the given points. This is not surprising since only four points were considered. This is
why in practice, people like to use a large number of points and when you do, the computations
become sufficiently long that a special algorithm was developed for doing them. It is called the fast
Fourier transform. So when you see this mentioned, this is what it is about, efficiently computing the
discreet Fourier transform which can be thought of as a way to approximate the Fourier coefficients
based on incomplete information for a given function.
17.6
Exercises
1. Verify that Examples 17.1.1 - 17.1.4 are each inner product spaces.
2. In each of the examples 17.1.1 - 17.1.4 write the Cauchy Schwarz inequality.
3. Verify 17.3 and 17.4.
4. Consider the Cauchy Schwarz inequality. Show that it still holds under the assumptions
⟨u, v⟩ = ⟨v, u⟩, ⟨(au + bv) , z⟩ = a ⟨u, z⟩ + b ⟨v, z⟩ , and ⟨u, u⟩ ≥ 0. Thus it is not necessary to
say that ⟨u, u⟩ = 0 only if u = 0. It is enough to simply state that ⟨u, u⟩ ≥ 0.
5. Consider the integers modulo a prime, Zp . This is a field of scalars. Now let the vector space
n
be (Zp ) where n ≥ p. Define now
⟨z, w⟩ ≡
n
∑
zi wi
i=1
Does this satisfy the axioms of an inner product? Does the Cauchy Schwarz inequality hold
for this ⟨⟩? Does the Cauchy Schwarz inequality even make any sense?
6. If you only know that ⟨u, u⟩ ≥ 0 along with the other axioms of the inner product and if you
define |z| the same way, how do the conclusions of Theorem 17.1.7 change?
7. In an inner product space, an open ball is the set
B (x, r) ≡ {y : |y − x| < r} .
If z ∈ B (x, r) , show there exists δ > 0 such that B (z, δ) ⊆ B (x, r). In words, this says that
an open ball is open. Hint: This depends on the triangle inequality.
8. Let V be the real inner product space consisting of continuous functions defined on [−1, 1]
with the inner product given by
∫ 1
f (x) g (x) dx
−1
{
}
Show that 1, x, x2 are linearly independent and find an orthonormal basis for the span of
these vectors.
358
CHAPTER 17. INNER PRODUCT SPACES
9. A regular Sturm Liouville problem involves the differential equation for an unknown
function of x which is denoted here by y,
′
(p (x) y ′ ) + (λq (x) + r (x)) y = 0, x ∈ [a, b]
and it is assumed that p (t) , q (t) > 0 for any t along with boundary conditions,
C1 y (a) + C2 y ′ (a) =
′
C3 y (b) + C4 y (b) =
0
0
where
C12 + C22 > 0, and C32 + C42 > 0.
There is an immense theory connected to these important problems. The constant λ is called
an eigenvalue. Show that if y is a solution to the above problem corresponding to λ = λ1 and
if z is a solution corresponding to λ = λ2 ̸= λ1 , then
∫ b
q (x) y (x) z (x) dx = 0.
(17.9)
a
Hint: Do something like this:
′
(p (x) y ′ ) z + (λ1 q (x) + r (x)) yz = 0,
′
(p (x) z ′ ) y + (λ2 q (x) + r (x)) zy = 0.
Now subtract and either use integration by parts or show
′
′
′
(p (x) y ′ ) z − (p (x) z ′ ) y = ((p (x) y ′ ) z − (p (x) z ′ ) y)
and then integrate. Use the boundary conditions to show that y ′ (a) z (a) − z ′ (a) y (a) = 0 and
y ′ (b) z (b) − z ′ (b) y (b) = 0.
10. Using the above problem or standard techniques of calculus, show that
}∞
{√
2
√ sin (nx)
π
n=1
are orthonormal with respect to the inner product
∫ π
⟨f, g⟩ =
f (x) g (x) dx
0
Hint: If you want to use the above problem, show that sin (nx) is a solution to the boundary
value problem
y ′′ + n2 y = 0, y (0) = y (π) = 0
11. Find S5 f (x) where f (x) = x on [−π, π] . Then graph both S5 f (x) and f (x) if you have access
to a system which will do a good job of it.
12. Find S5 f (x) where f (x) = |x| on [−π, π] . Then graph both S5 f (x) and f (x) if you have
access to a system which will do a good job of it.
13. Find S5 f (x) where f (x) = x2 on [−π, π] . Then graph both S5 f (x) and f (x) if you have
access to a system which will do a good job of it.
14. Let V be the set of real polynomials defined on [0, 1] which have degree at most 2. Make this
into a real inner product space by defining
⟨f, g⟩ ≡ f (0) g (0) + f (1/2) g (1/2) + f (1) g (1)
Find an orthonormal basis and explain why this is an inner product.
17.6. EXERCISES
359
15. Consider Rn with the following definition.
⟨x, y⟩ ≡
n
∑
xi yi i
i=1
Does this define an inner product? If so, explain why and state the Cauchy Schwarz inequality
in terms of sums.
16. From the above, for f a piecewise continuous function,
(∫ π
)
n
1 ∑ ikx
−iky
Sn f (x) =
e
f (y) e
dy .
2π
−π
k=−n
Show this can be written in the form
∫
Sn f (x) =
π
−π
f (y) Dn (x − y) dy
where
Dn (t) =
n
1 ∑ ikt
e
2π
k=−n
This is called the Dirichlet kernel. Show that
Dn (t) =
1 sin (n + (1/2)) t
2π
sin (t/2)
For V the vector space of piecewise continuous functions, define Sn : V 7→ V by
∫ π
Sn f (x) =
f (y) Dn (x − y) dy.
−π
Show that Sn is a linear transformation. (In∫ fact, Sn f is not just piecewise continuous but
π
infinitely differentiable. Why?) Explain why −π Dn (t) dt = 1. Hint: To obtain the formula,
do the following.
ei(t/2) Dn (t) =
ei(−t/2) Dn (t)
=
n
1 ∑ i(k+(1/2))t
e
2π
1
2π
k=−n
n
∑
ei(k−(1/2))t
k=−n
Change the variable of summation in the bottom sum and then subtract and solve for Dn (t).
17. ↑Let V be an inner product space and let U be a finite dimensional subspace with an orthonorn
mal basis {ui }i=1 . If y ∈ V, show
2
|y| ≥
n
∑
2
|⟨y, uk ⟩|
k=1
Now suppose that
∞
{uk }k=1
is an orthonormal set of vectors of V . Explain why
lim ⟨y, uk ⟩ = 0.
k→∞
When applied to functions, this is a special case of the Riemann Lebesgue lemma.
360
CHAPTER 17. INNER PRODUCT SPACES
18. ↑Let f be any piecewise continuous function which is bounded on [−π, π] . Show, using the
above problem, that
∫ π
∫ π
lim
f (t) sin (nt) dt = lim
f (t) cos (nt) dt = 0
n→∞
n→∞
−π
−π
19. ↑∗ Let f be a function which is defined on (−π, π]. The 2π periodic extension is given by
the formula f (x + 2π) = f (x) . In the rest of this problem, f will refer to this 2π periodic
extension. Assume that f is piecewise continuous, bounded, and also that the following limits
exist
f (x + y) − f (x+)
f (x − y) − f (x+)
lim
, lim
y→0+
y→0+
y
y
Here it is assumed that
f (x+) ≡ lim f (x + h) , f (x−) ≡ lim f (x − h)
h→0+
h→0+
both exist at every point. The above conditions rule out functions where the slope taken
from either side becomes infinite. Justify the following assertions and eventually conclude that
under these very reasonable conditions
lim Sn f (x) = (f (x+) + f (x−)) /2
n→∞
the mid point of the jump. In words, the Fourier series converges to the midpoint of the jump
of the function.
∫ π
Sn f (x) =
f (x − y) Dn (y) dy
f (x+) + f (x−)
=
Sn f (x) −
2
∫
∫
−π
)
(
f (x+) + f (x−)
Dn (y) dy
f (x − y) −
2
−π
π
∫
π
π
f (x − y) Dn (y) dy +
=
f (x + y) Dn (y) dy
0
∫
−
0
π
(f (x+) + f (x−)) Dn (y) dy
0
∫
∫
π
≤
π
(f (x − y) − f (x−)) Dn (y) dy +
(f (x + y) − f (x+)) Dn (y) dy
0
0
Now apply some trig. identities and use the result of Problem 18 to conclude that both of
these terms must converge to 0.
20. ↑Using the Fourier series obtained in Problem 11 and the result of Problem 19 above, find an
interesting formula by examining where the Fourier series converges when x = π/2. Of course
you can get many other interesting formulas in the same way. Hint: You should get
Sn f (x) =
n
k+1
∑
2 (−1)
k=1
k
sin (kx)
21. Let V be an inner product space and let K be a convex subset of V . This means that if
x, z ∈ K, then the line segment x + t (z − x) = (1 − t) x + tz is contained in K for all t ∈ [0, 1] .
Note that every subspace is a convex set. Let y ∈ V and let x ∈ K. Show that x is the closest
point to y out of all points in K if and only if for all w ∈ K,
Re ⟨y − x, w − x⟩ ≤ 0.
In Rn , a picture of the above situation where x is the closest point to y is as follows.
17.6. EXERCISES
361
K
wy
θ
-y
x
The condition of the above variational inequality is that the angle θ shown in the picture is
larger than 90 degrees. Recall the geometric description of the dot product presented earlier.
See Page 27.
22. Show that in any inner product space the parallelogram identity holds.
2
2
2
|x + y| + |x − y| = 2 |x| + 2 |y|
2
Next show that in a real inner product space, the polarization identity holds.
)
1(
2
2
⟨x, y⟩ =
|x + y| − |x − y| .
4
23. ∗ This problem is for those who know about Cauchy sequences and completeness of Fp and
about closed sets. Suppose K is a closed nonempty convex subset of a finite dimensional
subspace U of an inner product space V . Let y ∈ V. Then show there exists a unique point
x ∈ K which is closest to y. Hint: Let
λ = inf {|y − z| : z ∈ K}
Let {xn } be a minimizing sequence,
|y − xn | → λ.
Use the parallelogram identity in the above problem to show that {xn } is a Cauchy sequence.
p
Now let {uk }k=1 be an orthonormal basis for U . Say
xn =
p
∑
cnk uk
k=1
)
Verify that for cn ≡ cn1 , · · · , cnp ∈ Fp
(
|xn − xm | = |cn − cm |Fp .
Now use completeness of Fp and the assumption that K is closed to get the existence of x ∈ K
such that |x − y| = λ.
24. ∗ Let K be a closed nonempty convex subset of a finite dimensional subspace U of a real inner
product space V . (It is true for complex ones also.) For x ∈ V, denote by P x the unique
closest point to x in K. Verify that P is Lipschitz continuous with Lipschitz constant 1,
|P x − P y| ≤ |x − y| .
Hint: Use Problem 21.
25.
∗
This problem is for people who know about compactness. It is an analysis problem. If
you have only had the usual undergraduate calculus course, don’t waste your time with this
problem. Suppose V is a finite dimensional normed linear space. Recall this means that there
exists a norm ∥·∥ defined on V as described above,
∥v∥ ≥ 0 equals 0 if and only if v = 0
∥v + u∥ ≤ ∥u∥ + ∥v∥ , ∥αv∥ = |α| ∥v∥ .
362
CHAPTER 17. INNER PRODUCT SPACES
Let |·| denote the norm which comes from Example 17.1.3, the inner product by decree. Show
|·| and ∥·∥ are equivalent. That is, there exist constants δ, ∆ > 0 such that for all x ∈ V,
δ |x| ≤ ∥x∥ ≤ ∆ |x| .
Explain why every two norms on a finite dimensional vector space must be equivalent in the
above sense.
26. Let A be an n × n matrix such that A = A∗ . Verify that ⟨Ax, x⟩ is always real. Then A is said
to be nonnegative if ⟨Ax, x⟩ ≥ 0 for every x. Verify that [·, ·] given by [x, y] ≡ ⟨Ax, y⟩ satisfies
all the axioms of an inner product except for the one which says that [x, x] = 0 if and only if
x = 0. Also verify that the Cauchy Schwarz inequality holds for [·, ·].
27. Verify that for V equal to the space of continuous complex valued functions defined on an
interval [a, b] , an inner product is
∫
b
⟨f, g⟩ =
f (x) ḡ (x) dx
a
If the functions are only assumed to be Riemann integrable, why is this no longer an inner
product? In this last case, does the Cauchy Schwarz inequality still hold?
Chapter 18
Linear Transformations
18.1
Matrix Multiplication As A Linear Transformation
Definition 18.1.1 Let V and W be two finite dimensional vector spaces. A function, L which maps
V to W is called a linear transformation and L ∈ L (V, W ) if for all scalars α and β, and vectors
v, w,
L (αv+βw) = αL (v) + βL (w) .
These linear transformations are also called homomorphisms. If one of them is one to one, it is
called injective and if it is onto, it is called surjective. When a linear transformation is both one to
one and onto, it is called bijective. ,
An example of a linear transformation is familiar matrix multiplication. Let A = (aij ) be an
m × n matrix. Then an example of a linear transformation L : Fn 7→ Fm is given by
(Lv)i ≡
n
∑
aij vj .
j=1
Here
18.2


v1
 . 
n
. 
v ≡
 . ∈F .
vn
L (V, W ) As A Vector Space
In what follows I will often continue the practice of denoting vectors in bold face to distinguish them
from scalars. However, this does not mean they are in Fn .
Definition 18.2.1 Given L, M ∈ L (V, W ) define a new element of L (V, W ) , denoted by L + M
according to the rule
(L + M ) v ≡ Lv + M v.
For α a scalar and L ∈ L (V, W ) , define αL ∈ L (V, W ) by
αL (v) ≡ α (Lv) .
You should verify that all the axioms of a vector space hold for L (V, W ) with the above definitions
of vector addition and scalar multiplication. What about the dimension of L (V, W )?
363
364
CHAPTER 18. LINEAR TRANSFORMATIONS
Lemma 18.2.2 Let V and W be vector spaces and suppose {v1 , · · · , vn } is a basis for V. Then if
L : V → W is given by Lvk = wk ∈ W and
( n
)
n
n
∑
∑
∑
L
a k vk ≡
ak Lvk =
ak wk
k=1
k=1
k=1
then L is well defined and is in L (V, W ) . Also, if L, M are two linear transformations such that
Lvk = M vk for all k, then M = L.
Proof: L is well defined on V because, since {v1 , · · · , vn } is a basis, there is exactly one way to
write a given vector of V as a linear combination. Next, observe that L is obviously linear
∑nfrom the
definition. If L, M are equal on the basis, then an arbitrary vector in V is of the form k=1 ak vk .
Therefore,
( n
)
( n
)
n
n
∑
∑
∑
∑
L
ak vk =
ak Lvk =
ak M vk = M
ak vk
k=1
k=1
k=1
k=1
and so L = M because they give the same result for every vector in V . The message is that when you define a linear transformation, it suffices to tell what it does to a
basis.
Lemma 18.2.3 Suppose θ ∈ L (X, Y ) where X, Y are
{ vector spaces and} θ is one to one and onto.
Then if {y1 , · · · , yn } is a basis for Y, it follows that θ−1 y1 , · · · , θ−1 yn is a basis for X.
∑
Proof: Let xk = θ−1 yk . If k ck xk = 0, then
(
)
∑
∑
∑
ck xk =
ck θxk =
ck yk = 0
θ
k
k
k
and so, each ck = 0. Hence {x1 , · · · , xn } is independent. If x ∈ X, then θx ∈ Y so it equals an
expression of the form
n
∑
∑
ck yk =
ck θxk .
k=1
k
(
Hence
θ x−
and so, since θ is one to one, x−
therefore, a basis. ∑
)
ck xk
=0
k
∑
k ck xk
= 0 which shows that {x1 , · · · , xn } also spans and is
Theorem 18.2.4 Let V and W be finite dimensional linear spaces of dimension n and m respectively Then dim (L (V, W )) = mn.
Proof: Let the two sets of bases be
{v1 , · · · , vn } and {w1 , · · · , wm }
for X and Y respectively. Let L be a linear transformation. Then there are unique (since the wj
form a basis) scalars cij such that
m
∑
Lvk =
cik wi
i=1
Let C denote the m×n matrix whose ij entry is cij . Let θ be the mapping which takes L ∈ L (V, W )
to this matrix which is just defined. Then θ is one to one because if θL = θM, then both L and
M are equal on the basis for V . Therefore, L = M . Given such an m × n matrix C = (cij ) , use
the above formula to define L. Thus θ is a one to one and onto map from L (V, W ) to the space of
th
18.3. EIGENVALUES AND EIGENVECTORS OF LINEAR TRANSFORMATIONS
365
m × n matrices Mmn . It is also clear that θ is a linear map because if θL = C and θM = D and
a, b scalars,
m
m
m
∑
∑
∑
(aL + bM ) vk =
acik vi +
bdik vi =
(acik + bdik ) vi
i=1
i=1
i=1
and so θ (aL + bM ) = aC + bD.
Obviously this space of matrices is of dimension mn, a basis
of Eij the matrix which
{ consisting
}
has a 1 in the ij th position and zeros elsewhere. Therefore, θ−1 Eij i.j is a basis for L (V, W ) by
the above lemma. 18.3
Eigenvalues And Eigenvectors Of Linear Transformations
Here is a very useful theorem due to Sylvester.
Theorem 18.3.1 Let A ∈ L (V, W ) and B ∈ L (W, U ) where V, W, U are all vector spaces over a
field F. Suppose also that ker (A) and A (ker (BA)) are finite dimensional subspaces. Then
dim (ker (BA)) ≤ dim (ker (B)) + dim (ker (A)) .
Proof: If x ∈ ker (BA) , then Ax ∈ ker (B) and so A (ker (BA)) ⊆ ker (B) . The following picture
may help.
ker(BA)
ker(B)
A
ker(A)
-
A(ker(BA))
Now let {x1 , · · · , xn } be a basis of
ker (A) and let {Ay1 , · · · , Aym } be a basis for A (ker (BA)) .
∑m
Take any z ∈ ker (BA) . Then Az = i=1 ai Ayi and so
(
)
m
∑
A z−
ai yi = 0
which means z −
∑m
i=1
i=1
ai yi ∈ ker (A) and so there are scalars bi such that
z−
m
∑
i=1
ai yi =
n
∑
bi xi .
j=1
It follows span (x1 , · · · , xn , y1 , · · · , ym ) ⊇ ker (BA) and so by the first part, (See the picture.)
dim (ker (BA)) ≤ n + m ≤ dim (ker (A)) + dim (ker (B)) Of course this result holds for any finite product of linear transformations by induction. One
∏l way
this is quite useful is in the case where you have a finite product of linear transformations i=1 Li
all in L (V, V ) . Then
(
)
l
l
∏
∑
dim ker
Li ≤
dim (ker Li )
i=1
i=1
and so if you can find a linearly independent set of vectors in ker
l
∑
then it must be a basis for ker
(∏
i=1
l
i=1
)
Li .
dim (ker Li ) ,
(∏
l
i=1
)
Li of size
366
CHAPTER 18. LINEAR TRANSFORMATIONS
r
Definition 18.3.2 Let {Vi }i=1 be subspaces of V. Then
r
∑
denotes all sums of the form
Vi
i=1
∑r
i=1
vi where vi ∈ Vi . If whenever
r
∑
vi = 0, vi ∈ Vi ,
i=1
it follows that vi = 0 for each i, then a special notation is used to denote
(18.1)
∑r
i=1
Vi . This notation is
V1 ⊕ · · · ⊕ Vr
and it is called a direct sum of subspaces.
}
{
i
is a basis for Vi , then a basis for
Lemma 18.3.3 If V = V1 ⊕ · · · ⊕ Vr and if β i = v1i , · · · , vm
i
V is {β 1 , · · · , β r }.
∑ r ∑ mi
Proof: Suppose i=1 j=1
cij vji = 0. then since it is a direct sum, it follows for each i,
mi
∑
cij vji = 0
j=1
}
{
i
is a basis, each cij = 0. and now since v1i , · · · , vm
i
Here is a useful lemma.
Lemma 18.3.4 Let Li be in L (V, V ) and suppose for i ̸= j, Li Lj = Lj Li and also Li is one to one
on ker (Lj ) whenever i ̸= j. Then
( p
)
∏
ker
Li = ker (L1 ) ⊕ + · · · + ⊕ ker (Lp )
i=1
∏p
Here i=1 Li is the product of all the linear transformations. A symbol like
of all of them but Li .
∏
j̸=i
Lj is the product
Proof: Note that since the operators commute, Lj : ker (Li ) 7→ ker (Li ). Here is why. If Li y = 0
so that y ∈ ker (Li ) , then
Li Lj y = Lj Li y = Lj 0 = 0
and so Lj : ker (Li ) 7→ ker (Li ). Next observe that it is obvious that, since the operators commute,
( p
)
p
∑
∏
ker (Lp ) ⊆ ker
Li
i=1
Suppose
but some vi ̸= 0. Then do
results in
p
∑
i=1
vi = 0, vi ∈ ker (Li ) ,
i=1
∏
j̸=i
Lj to both sides. Since the linear transformations commute, this
∏
Lj vi = 0
j̸=i
which contradicts the assumption that these Lj are one to one and the observation that they map
ker (Li ) to ker (Li ). Thus if
∑
vi = 0, vi ∈ ker (Li )
i
18.3. EIGENVALUES AND EIGENVECTORS OF LINEAR TRANSFORMATIONS
then each vi = 0. It follows that
(
ker (L1 ) ⊕ + · · · + ⊕ ker (Lp ) ⊆ ker
p
∏
367
)
Li
(*)
i=1
From Sylvester’s theorem and the observation about direct sums in Lemma 18.3.3,
p
∑
dim (ker (Li )) =
i=1
dim (ker (L1 ) ⊕ + · · · + ⊕ ker (Lp ))
(
(
≤ dim ker
p
∏
))
≤
Li
p
∑
i=1
dim (ker (Li ))
i=1
which implies all these are equal. Now in general, if W is a subspace of V, a finite dimensional vector
space and the two have the same dimension, then W = V . This is because W has a basis and if v is
not in the span of this basis, then v adjoined to the basis of W would be a linearly independent set
so the dimension of V would then be strictly larger than the dimension of W . It follows from * that
( p
)
∏
ker (L1 ) ⊕ + · · · + ⊕ ker (Lp ) = ker
Li i=1
r
Here is a situation in which the above holds. ker (A − λi I) is sometimes called a generalized
eigenspace. The following is an important result on generalized eigenspaces.
Theorem 18.3.5 Let V be a vector space of dimension n and A a linear transformation and suppose
{λ1 , · · · , λk } are distinct scalars. Define for ri ∈ N
Vi = ker (A − λi I)
(
Then
ker
p
∏
ri
(18.2)
)
ri
(A − λi I)
= Vi ⊕ · · · ⊕ Vp .
(18.3)
i=1
r
Proof: It is obvious the linear transformations (A − λi I) i commute. Now here is a claim.
m
Claim : Let µ ̸= λi , Then (A − µI) : Vi 7→ Vi and is one to one and onto for every m ∈ N.
m
r
Proof: It is clear (A − µI) maps Vi to Vi because if v ∈ Vi then (A − λi I) i v = 0. Consequently,
r
m
m
r
m
(A − λi I) i (A − µI) v = (A − µI) (A − λi I) i v = (A − µI) 0 = 0
m
which shows that (A − µI) v ∈ Vi .
m
It remains to verify (A − µI) is one to one. This will be done by showing that (A − µI) is one to
m
one. Let w ∈ Vi and suppose (A − µI) w = 0 so that Aw = µw. Then for m ≡ ri , (A − λi I) w = 0
and so by the binomial theorem,
m ( )
∑
m
m
m−l l
(µ − λi ) w =
(−λi )
µw
l
l=0
m ( )
∑
m
m−l l
m
(−λi )
A w = (A − λi I) w = 0.
l
l=0
m
Therefore, since µ ̸= λi , it follows{ w = 0 and}this verifies (A − µI) is one to one. Thus (A − µI)
is also one to one on Vi . Letting ui1 , · · · , uirk be a basis for Vi , it follows
{
}
m
m
(A − µI) ui1 , · · · , (A − µI) uirk
m
is also a basis and so (A − µI) is also onto. The desired result now follows from Lemma 18.3.4. Let V be a finite dimensional vector space with field of scalars C. For example, it could be a
subspace of Cn . Also suppose A ∈ L (V, V ) . Does A have eigenvalues and eigenvectors just like the
case where A is a n × n matrix?
368
CHAPTER 18. LINEAR TRANSFORMATIONS
Theorem 18.3.6 Let V be a nonzero finite dimensional vector space of dimension n. Suppose also
the field of scalars equals C.1 Suppose A ∈ L (V, V ) . Then there exists v ̸= 0 and λ ∈ C such that
Av = λv.
2
Proof: Consider the linear transformations, I, A, A2 , · · · , An . There are n2 + 1 of these transformations and so by Theorem 18.2.4 the set is linearly dependent. Thus there exist constants,
ci ∈ C such that
n2
∑
ck Ak = 0.
c0 I +
k=1
This implies there exists a polynomial, q (λ) which has the property that q (A) = 0. In fact, q (λ) ≡
∑n2
c0 + k=1 ck λk . Dividing by the leading term, it can be assumed this polynomial is of the form
λm + cm−1 λm−1 + · · · + c1 λ + c0 , a monic polynomial. Now consider all such monic polynomials
q such that q (A) = 0 and pick one which has the smallest degree. This is called the minimal
polynomial and will be denoted here by p (λ) . By the fundamental theorem of algebra, p (λ) is of
the form
m
∏
p (λ) =
(λ − λk ) .
k=1
where some of the λk might be repeated. Thus, since p has minimal degree,
m
∏
(A − λk I) = 0, but
k=1
m−1
∏
(A − λk I) ̸= 0.
k=1
Therefore, there exists u ̸= 0 such that
)
(m−1
∏
(A − λk I) (u) ̸= 0.
v≡
k=1
But then
(A − λm I) v = (A − λm I)
(m−1
∏
)
(A − λk I) (u) = 0. k=1
As a corollary, it is good to mention that the minimal polynomial just discussed is unique.
Corollary 18.3.7 Let A ∈ L (V, V ) where V is an n dimensional vector space, the field of scalars
being F. Then there exists a polynomial q (λ) having coefficients in F such that q (A) = 0. Letting
p (λ) be the monic polynomial having smallest degree such that p (A) = 0, it follows that p (λ) is
unique.
Proof: The existence of p (λ) follows from the above theorem. Suppose then that p1 (λ) is
another one. That is, it has minimal degree of all polynomials q (λ) satisfying q (A) = 0 and is
monic. Then by Lemma 16.4.3 there exists r (λ) which is either equal to 0 or has degree smaller
than that of p (λ) and a polynomial l (λ) such that
p1 (λ) = p (λ) l (λ) + r (λ)
By assumption, r (A) = 0. Therefore, r (λ) = 0. Also by assumption, p1 (λ) and p (λ) have the same
degree and so l (λ) is a scalar. Since p1 (λ) and p (λ) are both monic, it follows this scalar must
equal 1. This shows uniqueness. Corollary 18.3.8 In the above theorem, each of the scalars λk has the property that there exists a
nonzero v such that (A − λi I) v = 0. Furthermore the λi are the only scalars with this property.
1 All that is really needed is that the minimal polynomial can be completely factored in the given field. The complex
numbers have this property from the fundamental theorem of algebra.
18.4. BLOCK DIAGONAL MATRICES
369
Proof: For the first claim, just factor out (A − λi I) instead of (A − λm I) . Next suppose
(A − µI) v = 0
for some µ and v ̸= 0. Then
0 =
m
∏
(A − λk I) v =
k=1
= (µ − λm )
(m−1
∏
m−1
∏
 =µv

z}|{
(A − λk I)  Av − λm v
k=1
)
(A − λk I) v
k=1
= (µ − λm )
(m−2
∏
)
(A − λk I) (Av − λm−1 v)
k=1
= (µ − λm ) (µ − λm−1 )
(m−2
∏
)
(A − λk I)
k=1
continuing this way yields
=
m
∏
(µ − λk ) v,
k=1
a contradiction unless µ = λk for some k. Therefore, these are eigenvectors and eigenvalues with the usual meaning. This leads to the
following definition.
Definition 18.3.9 For A ∈ L (V, V ) where dim (V ) = n, the scalars, λk in the minimal polynomial,
p (λ) =
m
∏
(λ − λk ) ≡
p
∏
(λ − λk )
rk
k=1
k=1
are called the eigenvalues of A. In the last expression, λk is a repeated root which occurs rk times.
The collection of eigenvalues of A is denoted by σ (A). The generalized eigenspaces are
rk
ker (A − λk I)
≡ Vk .
Theorem 18.3.10 In the situation of the above definition,
V = V1 ⊕ · · · ⊕ Vp
That is, the vector space equals the direct sum of its generalized eigenspaces.
∏p
r
Proof: Since V = ker ( k=1 (A − λk I) k ) , the conclusion follows from Theorem 18.3.5. 18.4
Block Diagonal Matrices
In this section the vector space will be Cn and the linear transformations will be those which result
by multiplication by n × n matrices.
Definition 18.4.1 Let A and B be two n × n matrices. Then A is similar to B, written as A ∼ B
when there exists an invertible matrix S such that A = S −1 BS.
370
CHAPTER 18. LINEAR TRANSFORMATIONS
Theorem 18.4.2 Let A be an n × n matrix. Letting λ1 , λ2 , · · · , λr be the distinct eigenvalues of
A,arranged in some order, there exist square matrices P1 , · · · , Pr such that A is similar to the block
diagonal matrix


P1 · · · 0
 . .

.
. . ... 
P =
 .

0 · · · Pr
in which Pk has the single eigenvalue λk . Denoting by rk the size of Pk it follows that rk equals the
dimension of the generalized eigenspace for λk . Furthermore, if S is the matrix satisfying
S −1 AS = P,
then S is of the form
(
B1
(
where Bk =
Vλk .
uk1
···
ukrk
···
)
Br
)
in which the columns,
{ k
}
u1 , · · · , ukrk = Dk constitute a basis for
Proof: By Theorem 18.3.9 and Lemma 18.3.3,
Cn = Vλ1 ⊕ · · · ⊕ Vλk
and a basis for Cn is {D1 , · · · , Dr } where Dk is a basis for Vλk , ker (A − λk I)
Let
(
)
S = B1 · · · Br
rk
.
where the Bi are the matrices described in the statement of the theorem. Then S −1 must be of the
form


C1
 . 
. 
S −1 = 
 . 
Cr
where Ci Bi = Iri ×ri . Also, if i ̸= j, then Ci ABj = 0 the last claim holding because A : Vλj 7→ Vλj
so the columns of ABj are linear combinations of the columns of Bj and each of these columns is
orthogonal to the rows of Ci since Ci Bj = 0 if i ̸= j. Therefore,




C1
C1
(
)
)
 . 

(
..  A B1 · · · Br =  ...  AB1 · · · ABr
S −1 AS = 




Cr
Cr


C1 AB1
0
···
0


0
C2 AB2 · · ·
0




= 
..
..

.


.
0
0
0
···
0 Cr ABr
and Crk ABrk is an rk × rk matrix.
What about the eigenvalues of Crk ABrk ? The only eigenvalue of A restricted to Vλk is λk because
if Ax = µx for some x ∈ Vλk and µ ̸= λk , then
r
r
(A − λk I) k x = (A − µI + (µ − λk ) I) k x
(
)
rk
∑
rk
r −j
j
r
=
(µ − λk ) k (A − µI) x = (µ − λk ) k x ̸= 0
j
j=0
18.4. BLOCK DIAGONAL MATRICES
371
contrary to the assumption that x ∈ Vλk . Suppose then that Crk ABrk x = λx where x ̸= 0. Why is
λ = λk ? Let y = Brk x so y ∈ Vλk . Then
 




0
0
0
 .  

 . 
..
 ..  

 .. 
.

 









S −1 Ay = S −1 AS  x  =  Crk ABrk x  = λ  x 

 



..
 ..  

 .. 
 .  

 . 
.
0
0
and so

0
..
.
x
..
.
0




Ay = λS 



0





 = λy.



Therefore, λ = λk because, as noted above, λk is the only eigenvalue of A restricted to Vλk . Now
let Pk = Crk ABrk . The above theorem contains a result which is of sufficient importance to state as a corollary.
Corollary 18.4.3 Let A be an n × n matrix and let Dk denote a basis for the generalized eigenspace
for λk . Then {D1 , · · · , Dr } is a basis for Cn .
More can be said. Recall Theorem 13.2.11 on Page 257. From this theorem, there exist unitary
matrices, Uk such that Uk∗ Pk Uk = Tk where Tk is an upper triangular matrix of the form


λk · · · ∗
 .
.. 
..
 .

.
.  ≡ Tk
 .
0 · · · λk
Now let U be the block diagonal matrix defined by

U1 · · ·
 .
..

U ≡  ..
.
0
By Theorem 18.4.2 there exists S such that

···
P1
 .
−1

S AS =  ..
0
Therefore,

U ∗ SASU
U1∗ · · ·
 .
..
.
= 
.
 .
0 ···

U1∗ P1 U1

..
= 
.

0
···
..
.
···

0
P1

.. 
  ..
.  .
Ur∗
0
···
..
.
···

0
.. 

. .
Ur

0
.. 

. .
Pr

0
U1

.. 
  ..
.  .
· · · Pr
0
 
0
T1 · · ·
  .
..
..
= .
.
.
  .
∗
Ur Pr Ur
0 ···
···
..
.
This proves most of the following corollary of Theorem 18.4.2.
···
..
.
···

0
.. 

. .
Tr

0
.. 

. 
Ur
372
CHAPTER 18. LINEAR TRANSFORMATIONS
Corollary 18.4.4 Let A be an n × n matrix. Then A is similar to an upper triangular, block
diagonal matrix of the form


T1 · · · 0
 . .

.
. . ... 
T ≡
 .

0 · · · Tr
where Tk is an upper triangular matrix having only λk on the main diagonal. The diagonal blocks
can be arranged in any order desired. If Tk is an mk × mk matrix, then
r
mk = dim (ker (A − λk I) k )
where the minimal polynomial of A is
p
∏
(λ − λk )
rk
k=1
Furthermore, mk is the multiplicity of λk as a zero of the characteristic polynomial of A.
Proof: The only thing which remains is the assertion that mk equals the multiplicity of λk as
a zero of the characteristic polynomial. However, this is clear from the observation that since T is
similar to A they have the same characteristic polynomial because
(
)
det (A − λI) = det S (T − λI) S −1
)
(
= det (S) det S −1 det (T − λI)
(
)
= det SS −1 det (T − λI)
= det (T − λI)
and the observation that since T is upper triangular, the characteristic polynomial of T is of the
form
r
∏
m
(λk − λ) k . k=1
The above corollary has tremendous significance especially if it is pushed even further resulting
in the Jordan Canonical form. This form involves still more similarity transformations resulting in
an especially revealing and simple form for each of the Tk , but the result of the above corollary is
sufficient for most applications.
It is significant because it enables one to obtain great understanding of powers of A by using the
matrix T. From Corollary 18.4.4 there exists an n × n matrix S 2 such that
A = S −1 T S.
Therefore, A2 = S −1 T SS −1 T S = S −1 T 2 S and continuing this way, it follows
Ak = S −1 T k S.
where T is given in the above corollary. Consider T k . By block multiplication,


0
T1k


..
.
Tk = 
.


0
Trk
The matrix Ts is an ms × ms matrix which is of the form

α ··· ∗
 . .

. . ...
Ts =  ..
0 ··· α
2 The
S here is written as S −1 in the corollary.




(18.4)
18.5. THE MATRIX OF A LINEAR TRANSFORMATION
373
which can be written in the form
Ts = D + N
for D a multiple of the identity and N an upper triangular matrix with zeros down the main diagonal.
Therefore, by the Cayley Hamilton theorem, N ms = 0 because the characteristic equation for N is
just λms = 0. Such a transformation is called nilpotent. You can see N ms = 0 directly also, without
having to use the Cayley Hamilton theorem. Now since D is just a multiple of the identity, it follows
that DN = N D. Therefore, the usual binomial theorem may be applied and this yields the following
equations for k ≥ ms .
Tsk
k ( )
∑
k
= (D + N ) =
Dk−j N j
j
j=0
ms ( )
∑
k
=
Dk−j N j ,
j
j=0
k
(18.5)
the third equation holding because N ms = 0. Thus Tsk is of the form


αk · · · ∗
 .
.. 
..

.
Tsk = 
.
. .
 .
0 · · · αk
Lemma 18.4.5 Suppose T is of the form Ts described above in 18.4 where the constant α, on the
main diagonal, is less than one in absolute value. Then
( )
lim T k ij = 0.
k→∞
Proof: From 18.5, it follows that for large k, and j ≤ ms ,
( )
k
k (k − 1) · · · (k − ms + 1)
≤
.
j
ms !
Therefore, letting C be the largest value of
( k)
T pq ≤ ms C
(
(
Nj
)
pq
for 0 ≤ j ≤ ms ,
k (k − 1) · · · (k − ms + 1)
ms !
)
|α|
k−ms
which converges to zero as k → ∞. This is most easily seen by applying the ratio test to the series
)
∞ (
∑
k (k − 1) · · · (k − ms + 1)
k−ms
|α|
ms !
k=ms
and then noting that if a series converges, then the k th term converges to zero. 18.5
The Matrix Of A Linear Transformation
To begin with, here is an easy lemma which relates the vectors in two vector spaces having the same
dimension.
Lemma 18.5.1 Let q : V → W be one to one, onto and linear. Then q maps any basis of V to a
basis for W . Conversely, if q is linear and maps a basis to a basis, then it is one to one onto.
374
CHAPTER 18. LINEAR TRANSFORMATIONS
Proof: Let {v1 , · · · , vn } be a basis for∑V , why is {qv1 , · · · , qvn } ∑
a basis for W ? First consider
n
n
why it is linearly independent.
Suppose
c
qv
=
0.
Then
q
(
k
k=1 k
k=1 ck vk ) = 0 and since q
∑n
is one to one, it follows that k=1 ck vk = 0 which requires each ck = 0 because {v1 , · · · , vn } is
independent. Next take w ∈ W. Since q is onto, there exists
∑n v ∈ V such that qv = w. Since
{v
,
·
·
·
,
v
}
is
a
basis,
there
are
scalars
c
such
that
q
(
n
k
k=1 ck vk ) = q (v) = w and so w =
∑n1
c
qv
which
is
in
the
span
(qv
,
·
·
·
,
qv
)
.
Therefore,
{qv
k
1
n
1 , · · · , qvn } is a basis as claimed.
k=1 k
Suppose
now
that
qv
=
w
where
{v
,
·
·
·
,
v
}
and
{w
,
·
·
· , wn } are bases for V and W . If
i∑
i
n
1
∑n
∑1n
n
q ( k=1 ck vk ) = 0, then k=1 ck qvk = k=1
c
w
=
0
and
so
each
ck is 0 because it is given that
∑n k k
{w
,
·
·
·
,
w
}
is
linearly
independent.
For
c
w
an
arbitrary
vector
of W, this vector equals
n
k=1 k k
∑n1
∑n
c
qv
=
q
(
c
v
)
.
Therefore,
q
is
also
onto.
k
k=1 k
k=1 k k
Such a mapping is called an isomorphism.
Definition 18.5.2 Let V be a vector space of dimension n and W a vector space of dimension m
with respect to the same field of scalars F. Let
qα
qβ
: Fn → V
: Fm → W
be two isomorphisms as in the above lemma. Here α denotes the basis {qα e1 , · · · , qα en } and β
denotes the basis {qβ e1 , · · · , qβ em } for V and W respectively. Then for L ∈ L (V, W ) , the matrix
of L with respect to these two bases,
{qα e1 , · · · , qα en } , {qβ e1 , · · · , qβ em }
denoted by [L]βα or [L] for short satisfies
Lqα = qβ [L]βα
(18.6)
L
→
◦
→
[L]βα
(18.7)
In terms of a diagram,
V
qα ↑
Fn
W
↑ qβ
Fm
Starting at Fn , you go up and then to the right using L and the result is the same if you go to the
right by matrix multiplication by [L]βα and then up using qβ .
So how can we find this matrix? Let
α
≡
{qα e1 , · · · , qα en } ≡ {v1 , · · · , vn }
β
≡
{qβ e1 , · · · , qβ em } ≡ {w1 , · · · , wm }
and suppose the ij th entry of the desired matrix is aij . Letting b ∈ Fn , the requirement 18.6 is
equivalent to
∑
i
z
=([L]b)i
}| {
∑
∑
∑

aij bj wi = L
bj vj =
bj Lvj .
j
j
j
Thus interchanging the order in the sum on the left,
(
)
∑ ∑
∑
aij wi bj =
(Lvj ) bj ,
j
i
j
18.5. THE MATRIX OF A LINEAR TRANSFORMATION
375
and this must hold for all b which happens if and only if
∑
Lvj =
aij wi
(18.8)
i
It may help to write 18.8 in the form
(
Lv1 · · ·
)
Lvn
where [L] = A = (aij ) .
A little less precisely, you need for b ∈ Fn ,
(
)
(
w1 · · · wm Ab = L v1 · · ·
(
=
w1
···
)
(
b=
vn
)
A
wm
Lv1
···
(18.9)
)
Lvn
b
(18.10)
where we use the usual conventions for notation like the above. Then since 18.10 is to hold for all
b, 18.9 follows.
Example 18.5.3 Let
V ≡ { polynomials of degree 3 or less},
W ≡ { polynomials of degree 2 or less},
and L ≡ D where D is the differentiation operator. A basis for V is {1,x, x2 , x3 } and a basis for
W is {1, x, x2 }.
What is the matrix of this linear transformation with respect to this basis? Using 18.9,
(
) (
)
0 1 2x 3x2 = 1 x x2 [D] .
It follows from this that

0

[D] =  0
0
1
0
0

0

0 .
3
0
2
0
Now consider the important case where V = Fn , W = Fm , and the basis chosen is the standard
(
)T
basis of vectors ei ≡ 0 · · · 1 · · · 0
the 1 in the ith position. Let L be a linear transforn
m
mation from F to F and let [L] be the matrix of the transformation with respect to these bases.
Thus
α = {e1 , · · · , en } , β = {e1 , · · · , em }
and so, in this case the coordinate maps qα and qβ are simply the identity map and the requirement
that A is the matrix of the transformation amounts to
(Lb)i = ([L] b)i
where this above indicates the ith entry of the indicated vectors taken with respect to the standard
basis vectors. Thus, if the components of the vector in Fn with respect to the standard basis
are (b1 , · · · , bn ) ,
(
)T ∑
b = b1 · · · bn
=
bi ei ,
i
then if the entries of [L] are aij , you would need to have
∑
(Lb)i =
aij bj
j
In terms of matrix notation, you would need
(
Lei · · ·
)
Len
= I [L]
The following example illustrates what happens when you consider the matrix of a linear transformation with respect to two different bases.
376
CHAPTER 18. LINEAR TRANSFORMATIONS
Example 18.5.4 You have a linear transformation L : R3 → R3 and it is given
  
   
 
 
1
1
1
1
−1
0
  
   
 
 
L  0  =  −1  , L  1  =  2  , L  1  =  1
1
1
1
−1
0
1
Find the matrix of this linear transformation relative to
    
1
1
    
,
0
   1 ,
1
1
on a basis by



the basis

−1

1 
0
Then find the matrix of this transformation with respect to the usual basis on R3 .
The matrix with columns equal to Lvi for the vi the basis vectors is on the left in what is below.
Thus if A is the matrix of the transformation with respect to this basis,


 
1
1 0
1 1 −1

 

 −1 2 1  =  0 1 1  A
1 1 0
1 −1 1
Then multiplying by the inverse

1 1

A= 0 1
1 1
of the first matrix on
−1 
−1
1
1
 
1   −1 2
0
1 −1
the right, you need
 

0
2 −5 1
 

1  =  −1 4 0 
1
0 −2 1
T
Note how it works. Start with a column vector (x, y, z) . Then do qα to it to get
(
)T
(
)T
(
)T
x 1 0 1
+y 1 1 1
+ z −1 1 0
.
Then you do L to this and get





  
1
1
0
x+y




  

x  −1  + y  2  + z  1  =  2y − x + z 
1
−1
1
x−y+z
Now take the matrix of the transformation times


2 −5 1
x


 −1 4 0   y
0 −2 1
z
the given column vector.
 

2x − 5y + z
 

4y − x
=

z − 2y
Is this the coordinate vector of the above relative to the given basis?
 
 

 

1
1
−1
x+y
 
 

 

(2x − 5y + z)  0  + (4y − x)  1  + (z − 2y)  1  =  2y − x + z 
1
1
0
x−y+z
You see it is the same thing.
Now lets find the matrix of L with respect to the usual basis.
multiplication by B is the same as doing L. Thus

 
1 1 −1
1
1 0

 
B  0 1 1  =  −1 2 1
1 1 0
1 −1 1
Let B be this matrix. That is,



18.5. THE MATRIX OF A LINEAR TRANSFORMATION
Hence

1

B =  −1
1

1 0
1 1

2 1  0 1
−1 1
1 1
−1 
−1
0


1  = 2
0
−3
377
0
3
−2

1

−3 
4
Of course this is a very different matrix than the matrix of the linear transformation with respect
to the funny basis.
What about the situation where different pairs of bases are chosen for V and W ? How are the
two matrices with respect to these choices related? Consider the following diagram which illustrates
the situation.
Fn A2 Fm
−
→
q2 ↓
◦ p2 ↓
V →
L
W
−
q1 ↑
Fn
◦ p1 ↑
A1 Fm
−
→
In this diagram qi and pi are coordinate maps as described above. From the diagram,
−1
p−1
1 p2 A2 q2 q1 = A1 ,
where q2−1 q1 and p−1
1 p2 are one to one, onto, and linear maps. Thus the effect of these maps is
identical to multiplication by a suitable matrix.
Definition 18.5.5 In the special case where V = W and only one basis is used for V = W, this
becomes
q1−1 q2 A2 q2−1 q1 = A1 .
Letting S be the matrix of the linear transformation q2−1 q1 with respect to the standard basis vectors
in Fn ,
S −1 A2 S = A1 .
(18.11)
When this occurs, A1 is said to be similar to A2 and A 7→ S −1 AS is called a similarity transformation.
Here is some terminology.
Definition 18.5.6 Let S be a set. The symbol, ∼ is called an equivalence relation on S if it satisfies
the following axioms.
1. x ∼ x
for all x ∈ S. (Reflexive)
2. If x ∼ y then y ∼ x. (Symmetric)
3. If x ∼ y and y ∼ z, then x ∼ z. (Transitive)
Definition 18.5.7 [x] denotes the set of all elements of S which are equivalent to x and [x] is called
the equivalence class determined by x or just the equivalence class of x.
With the above definition one can prove the following simple theorem which you should do if
you have not seen it.
Theorem 18.5.8 Let ∼ be an equivalence class defined on a set, S and let H denote the set of
equivalence classes. Then if [x] and [y] are two of these equivalence classes, either x ∼ y and
[x] = [y] or it is not true that x ∼ y and [x] ∩ [y] = ∅.
378
CHAPTER 18. LINEAR TRANSFORMATIONS
Theorem 18.5.9 In the vector space of n × n matrices, define
A∼B
if there exists an invertible matrix S such that
A = S −1 BS.
Then ∼ is an equivalence relation and A ∼ B if and only if whenever V is an n dimensional vector
space, there exists L ∈ L (V, V ) and bases {v1 , · · · , vn } and {w1 , · · · , wn } such that A is the matrix
of L with respect to {v1 , · · · , vn } and B is the matrix of L with respect to {w1 , · · · , wn }.
Proof: A ∼ A because S = I works in the definition. If A ∼ B , then B ∼ A, because
A = S −1 BS
implies
B = SAS −1 .
If A ∼ B and B ∼ C, then
A = S −1 BS, B = T −1 CT
and so
−1
A = S −1 T −1 CT S = (T S)
CT S
which implies A ∼ C. This verifies the first part of the conclusion.
It was pointed out in the above definition that if A, B are matrices which come from a single
linear transformation, then they are similar. Suppose now that A, B are similar. A = S −1 BS. Pick
a basis α for V and let qα be as described above. Then in the following diagram, define L ≡ qα Aqα−1 .
L
V → V
qα ↑ ◦ ↑ qα
Fn → Fn
A
Then since A, B are similar,
Let qβ ≡ qα S −1 . Then
L = qα S −1 BSqα−1
Lqβ = qα S −1 BSqα−1 qα S −1 = qα S −1 B = qβ B
and so B is the matrix of L with respect to the basis β. What if the linear transformation consists of multiplication by a matrix A and you want to find
the matrix of this linear transformation with respect to another basis? Is there an easy way to do
it? The answer is yes. It was illustrated in one of the above examples.
Proposition 18.5.10 Let A be an m × n matrix and let L be the linear transformation which is
defined by multiplication on the left by A. Then the matrix M of this linear transformation with
respect to the bases {u1 , · · · , un } for Fn and {w1 , · · · , wm } for Fm is given by
(
)−1 (
)
M = w1 · · · wm
A u1 · · · un
(
where
w1
···
)
wm
is the m × m matrix which has wj as its j th column.
Proof: Consider the following diagram. The situation is that
(
) (
)
A u1 · · · un = w1 · · · wm M
Hence the matrix M is given by the claimed formula. 18.5. THE MATRIX OF A LINEAR TRANSFORMATION
379
Definition 18.5.11 An n × n matrix A, is diagonalizable if there exists an invertible n × n matrix
S such that S −1 AS = D, where D is a diagonal matrix. Thus D has zero entries everywhere except
on the main diagonal. Write diag (λ1 · · · , λn ) to denote the diagonal matrix having the λi down the
main diagonal.
Which matrices are diagonalizable?
Theorem 18.5.12 Let A be an n × n matrix. Then A is diagonalizable if and only if Fn has a
basis of eigenvectors of A. In this case, S of Definition 18.5.11 consists of the n × n matrix whose
columns are the eigenvectors of A and D = diag (λ1 , · · · , λn ) .
Proof: Suppose first that Fn has a basis of eigenvectors,

 {v1 , · · · , vn } where Avi = λi vi . Then
{
uT1
 . 
1 if i = j
−1
T
. 
let S denote the matrix (v1 · · · vn ) and let S ≡ 
. S −1
 .  where ui vj = δ ij ≡
0 if i ̸= j
uTn
exists because S has rank n. Then from block multiplication or the way we multiply matrices,




uT1
uT1
 . 


..  (Av1 · · · Avn ) =  ...  (λ1 v1 · · · λn vn )
S −1 AS = 




uTn
uTn



=


λ1
0
..
.
0
0
λ2
..
.
···
0
..
.
0
···
..
.
···
0
λn



 = D.


Next suppose A is diagonalizable so S −1 AS = D ≡ diag (λ1 , · · · , λn ) . Then the columns of S
form a basis because S −1 is given to exist. It only remains to verify that these columns of A are
eigenvectors. But letting S = (v1 · · · vn ) , AS = SD and so (Av1 · · · Avn ) = (λ1 v1 · · · λn vn ) which
shows that Avi = λi vi . It makes sense to speak of the determinant of a linear transformation as described in the following
corollary.
Corollary 18.5.13 Let L ∈ L (V, V ) where V is an n dimensional vector space and let A be the
matrix of this linear transformation with respect to a basis on V. Then it is possible to define
det (L) ≡ det (A) .
Proof: Each choice of basis for V determines a matrix for L with respect to the basis. If A and
B are two such matrices, it follows from Theorem 18.5.9 that
A = S −1 BS
and so
But
(
)
det (A) = det S −1 det (B) det (S) .
(
)
(
)
1 = det (I) = det S −1 S = det (S) det S −1
and so
det (A) = det (B) Definition 18.5.14 Let A ∈ L (X, Y ) where X and Y are finite dimensional vector spaces. Define
rank (A) to equal the dimension of A (X) .
380
CHAPTER 18. LINEAR TRANSFORMATIONS
The following theorem explains how the rank of A is related to the rank of the matrix of A.
Theorem 18.5.15 Let A ∈ L (X, Y ). Then rank (A) = rank (M ) where M is the matrix of A taken
with respect to a pair of bases for the vector spaces X, and Y.
Proof: Recall the diagram which describes what is meant by the matrix of A. Here the two bases
are as indicated.
X
A
Y
−
→
qα ↑
Fn
◦ ↑ qβ
m
M
−
→ F
−1
The maps qα , a−1
α , qβ , qβ are one to one and onto. They take independent sets to independent sets.
Let {Ax1 , · · · , Axr } be a basis for A (X). Thus {x1 , · · · , xr } is independent. Let xi = qα ui . Then
the {ui } are independent and from the definition of the matrix of the linear transformation A,
A (X) = span (Aqα u1 , · · · , Aqα ur ) = span (qβ M u1 , · · · , qβ M ur ) = qβ span (M u1 , · · · , M ur )
n
n
However, it also follows from the definition of M that A (X) = q∑
β M (F ) . Hence M (F ) =
span (M u1 , · · · , M ur ) and so the span of the M uj equals M (Fn ). If j cj M uj = 0, then
∑
cj qβ M uj = 0 =
j
∑
j
cj Aqα ui =
∑
cj Axj
j
and so each cj = 0. Therefore, dim (M Fn ) = dim (qβ M (Fn )) = dim (A (X)) which shows that the
rank of the matrix equals the rank of the transformation. The following result is a summary of many concepts.
Theorem 18.5.16 Let L ∈ L (V, V ) where V is a finite dimensional vector space. Then the following are equivalent.
1. L is one to one.
2. L maps a basis to a basis.
3. L is onto.
4. det (L) ̸= 0
5. If Lv = 0 then v = 0.
∑n
n
Proof:∑Suppose first L is one to one and let {vi }i=1 be a basis. Then if i=1 ci Lvi = 0 it
n
follows
( i=1 ci vi ) = 0 which means that since L (0) = 0, and L is one to one, it must be the case
∑L
n
that i=1 ci vi = 0. Since {vi } is a basis, each ci = 0 which shows {Lvi } is a linearly independent
set. Since there are n of these, it must be that this is a basis.
Now suppose 2.). Then letting∑{vi } be a basis,∑and y ∈ V, it follows from part 2.) that there
n
n
are constants, {ci } such that y = i=1 ci Lvi = L ( i=1 ci vi ) . Thus L is onto. This shows that 2.)
implies 3.).
Now suppose 3.). Then the operation consisting of multiplication by the matrix of L, [L], must
be onto. However, the vectors in Fn so obtained, consist of linear combinations of the columns of
[L] . Therefore, the column rank of [L] is n. By Theorem 8.6.7 this equals the determinant rank and
so det ([L]) ≡ det (L) ̸= 0.
Now assume 4.) If Lv = 0 for some v ̸= 0, it follows that [L] x = 0 for some x ̸= 0. Therefore,
the columns of [L] are linearly dependent and so by Theorem 8.6.7, det ([L]) = det (L) = 0 contrary
to 4.). Therefore, 4.) implies 5.).
Now suppose 5.) and suppose Lv = Lw. Then L (v − w) = 0 and so by 5.), v − w = 0 showing
that L is one to one. 18.5. THE MATRIX OF A LINEAR TRANSFORMATION
381
Also it is important to note that composition of linear transformation corresponds to multiplication of the matrices. Consider the following diagram.
X
A
Y
−
→
◦
↑ qβ
[A]βα Fm
−−−→
qα ↑
Fn
B
Z
−
→
◦
↑ qγ
[B]γβ Fp
−−−→
where A and B are two linear transformations, A ∈ L (X, Y ) and B ∈ L (Y, Z) . Then B ◦ A ∈
L (X, Z) and so it has a matrix with respect to bases given on X and Z, the coordinate maps for
these bases being qα and qβ respectively. Then
B ◦ A = qγ [B]γβ qβ−1 qβ [A]βα qα−1 = qγ [B]γβ [A]βα qα−1 .
But this shows that [B]γβ [A]βα plays the role of [B ◦ A]γα , the matrix of B ◦ A. Hence the matrix
of B ◦ A equals the product [B]γβ [A]βα . Of course it is interesting to note that although [B ◦ A]γα
must be unique, the matrices, [B]γβ and [A]βα are not unique because they depend on the basis
chosen for Y .
Theorem 18.5.17 The matrix of the composition of linear transformations equals the product of
the matrices of these linear transformations in the same order as the composition.
18.5.1
Some Geometrically Defined Linear Transformations
This is a review of earlier material. If T is any linear transformation which maps Fn to Fm , there is
always an m × n matrix A with the property that
Ax = T x
(18.12)
for all x ∈ Fn . How does this relate to what is discussed above? In terms of the above diagram,
{e1 , · · · , en }
Fn
T
Fm
−
→
qFn ↑ ◦ ↑ qFm
m
Fn
M
−
→ F
where
qFn (x) ≡
n
∑
{e1 , · · · , en }
xi ei = x.
i=1
Thus those two maps are really just the identity map. Thus, to find the matrix of the linear
transformation T with respect to the standard basis vectors,
T ek = M ek
In other words, the k th column of M equals T ek as noted earlier. All the earlier considerations
apply. These considerations were just a specialization to the case of the standard basis vectors of
this more general notion which was just presented.
18.5.2
Rotations About A Given Vector
As an application, I will consider the problem of rotating counter clockwise about a given unit
vector which is possibly not one of the unit vectors in coordinate directions. First consider a pair of
perpendicular unit vectors, u1 and u2 and the problem of rotating in the counterclockwise direction
about u3 where u3 = u1 × u2 so that u1 , u2 , u3 forms a right handed orthogonal coordinate system.
Thus the vector u3 is coming out of the page.
382
CHAPTER 18. LINEAR TRANSFORMATIONS
-
θ
θ
u1
u2R
?
Let T denote the desired rotation. Then
T (au1 + bu2 + cu3 ) = aT u1 + bT u2 + cT u3
= (a cos θ − b sin θ) u1 + (a sin θ + b cos θ) u2 + cu3 .
Thus in terms of the basis {u1 , u2 , u3 } , the matrix of this transformation is


cos θ − sin θ 0


 sin θ cos θ 0  .
0
0
1
I want to write this transformation in terms of the usual basis vectors, {e1 , e2 , e3 }. From Proposition
18.5.10, if A is this matrix,


cos θ − sin θ 0


 sin θ cos θ 0 
0
0
1
(
)−1 (
)
=
A u1 u2 u3
u1 u2 u3
and so you can solve for A if you know the ui .
Suppose the unit vector about which the counterclockwise rotation takes place is (a, b, c). Then
I obtain vectors, u1 and u2 such that {u1 , u2 , u3 } is a right handed orthogonal system with u3 =
(a, b, c) and then use the above result. It is of course somewhat arbitrary how this is accomplished.
I will assume, however that |c| =
̸ 1 since otherwise you are looking at either clockwise or counter
clockwise rotation about the positive z axis and this is a problem which has been dealt with earlier. (If
c = −1, it amounts to clockwise rotation about the positive z axis while if c = 1, it is counterclockwise
rotation about the positive z axis.) Then let u3 = (a, b, c) and u2 ≡ √a21+b2 (b, −a, 0) . This one is
perpendicular to u3 . If {u1 , u2 , u3 } is to be a right hand system it is necessary to have
u1 = u2 × u3 = √
(
1
(a2 + b2 ) (a2 + b2 + c2 )
−ac, −bc, a2 + b2
)
Now recall that u3 is a unit vector and so the above equals
√
1
(a2
+
b2 )
(
)
−ac, −bc, a2 + b2
Then from the above, A is given by

√ −ac
2
+b2 )
 (a−bc
 √
 (a2 +b2 )
√
a2 + b2
√ b
a2 +b2
√ −a
a2 +b2
0
a

cos θ


b   sin θ
0
c
− sin θ
cos θ
0
  √ −ac
0
(a2 +b2 )


0   √ −bc
2
2
√ (a +b )
1
2
a + b2
√ b
a2 +b2
√ −a
a2 +b2
a
0
c
−1

b 

Of course the matrix is an orthogonal matrix so it is easy to take the inverse by simply taking the
transpose. Then doing the computation and then some simplification yields
18.6. THE MATRIX EXPONENTIAL, DIFFERENTIAL EQUATIONS

(
)
a2 + 1 − a2 cos θ

=  ab (1 − cos θ) + c sin θ
ac (1 − cos θ) − b sin θ
ab (1 − cos θ) − c sin θ
(
)
b2 + 1 − b2 cos θ
bc (1 − cos θ) + a sin θ
∗
383

ac (1 − cos θ) + b sin θ

bc (1 − cos θ) − a sin θ  .
(
)
c2 + 1 − c2 cos θ
(18.13)
With this, it is clear how to rotate clockwise about the unit vector (a, b, c) . Just rotate counter
clockwise through an angle of −θ. Thus the matrix for this clockwise rotation is just


)
(
a2 + 1 − a2 cos θ
ab (1 − cos θ) + c sin θ ac (1 − cos θ) − b sin θ
(
)


=  ab (1 − cos θ) − c sin θ
b2 + 1 − b2 cos θ
bc (1 − cos θ) + a sin θ  .
(
)
ac (1 − cos θ) + b sin θ bc (1 − cos θ) − a sin θ
c2 + 1 − c2 cos θ
In deriving 18.13 it was assumed that c ̸= ±1 but even in this case, it gives the correct answer.
Suppose for example that c = 1 so you are rotating in the counter clockwise direction about the
positive z axis. Then a, b are both equal to zero and 18.13 reduces to the correct matrix for rotation
about the positive z axis.
18.6
The Matrix Exponential, Differential Equations
∗
You want to find a matrix valued function Φ (t) such that
Φ′ (t) = AΦ (t) , Φ (0) = I, A is p × p
(18.14)
Such a matrix is called a fundamental matrix.
What is meant by the above symbols? The idea is that Φ (t) is a matrix whose entries are
differentiable functions of t. The meaning of Φ′ (t) is the matrix whose entries are the derivatives of
the entries of Φ (t). For example, abusing notation slightly,
(
)′ (
)
t
t2
1
2t
=
.
sin (t) tan (t)
cos (t) sec2 (t)
What are some properties of this derivative? Does the product rule hold for example?
Lemma 18.6.1 Suppose Φ (t) is m × n and Ψ (t) is n × p and these are differentiable matrices.
Then
′
(Φ (t) Ψ (t)) = Φ′ (t) Ψ (t) + Φ (t) Ψ′ (t)
Proof: By definition,
(
′)
(Φ (t) Ψ (t)) ij
(
=
=
(Φ (t) Ψ (t))ij
∑
)′
=
∑
)′
Φ (t)ik Ψ (t)kj
k
Φ′ (t)ik Ψ (t)kj +
k
=
(
∑
Φ (t)ik Ψ′ (t)kj
k
(Φ′ (t) Ψ (t))ij + (Φ (t) Ψ′ (t))ij
and so the conclusion follows. What do we mean when we say that for {Bn } a sequence of matrices
lim Bn = B?
n→∞
We mean the obvious thing. The ij th entry of Bn converges to the ij th entry of B. One convenient
way to ensure that this happens is to give a measure of distance between matrices which will ensure
that it happens.
384
CHAPTER 18. LINEAR TRANSFORMATIONS
Definition 18.6.2 For A, B matrices of the same size, define ∥A − B∥∞ to be
max {|Aij − Bij | , all ij}
Thus
∥A∥∞ = max {|Aij | , all ij}
Then to say that limn→∞ Bn = B is the same as saying that limn→∞ ∥Bn − B∥∞ = 0. Here is
a useful lemma.
Lemma 18.6.3 If A, Bn , B are p × p matrices and limn→∞ Bn = B, then
lim ABn
= AB,
lim Bn A
= BA,
n→∞
n→∞
Also Ak
k
∞
≤ pk ∥A∥∞ for any positive integer k and
m
∑
≤
Ak
∞
k=1
m
∑
∥Ak ∥∞
k=1
For t a scalar,
∥tA∥∞ = |t| ∥A∥∞
Proof: From the above observation, it suffices to show that limn→∞ ∥ABn − AB∥ = 0.
(
)
∑
(ABn − AB)ij =
Aik (Bn )kj − Bkj
k
≤
∑
∥A∥∞ ∥Bn − B∥∞ = p ∥A∥∞ ∥Bn − B∥∞
k
Hence
∥ABn − AB∥∞ ≤ p ∥A∥∞ ∥Bn − B∥∞
The expression on the right converges to 0 as n → ∞ and so this establishes the first part of the
lemma.
From the way we multiply matrices, the absolute value of the ij th entry of Ak is of the form
∑
∑
Air1 Ar1 r2 · · · Ark j ≤
r1 ,r2 ··· ,rk
k
r1 ,r2 ··· ,rk
k
∥A∥∞ = pk ∥A∥∞
k
This is because there are pk terms in the sum, each term no larger than ∥A∥∞ .
Consider the claim about the sum.
(m
)
m
m
∑
∑
∑
Ak
=
(Ak )ij ≤
∥Ak ∥∞
k=1
k=1
ij
k=1
Since this holds for arbitrary ij, it follows that
m
∑
≤
Ak
∞
k=1
m
∑
∥Ak ∥∞
k=1
as claimed. The last assertion is obvious. By analogy with the situation in calculus, consider the infinite sum
∞
∑
Ak tk
k=0
k!
≡ lim
n→∞
n
∑
A k tk
k=0
k!
18.6. THE MATRIX EXPONENTIAL, DIFFERENTIAL EQUATIONS
∗
385
where here A is a p × p matrix having real or complex entries. Then letting m < n, it follows from
the above lemma that
n
∑
Ak tk
k=0
k!
−
m
∑
Ak tk
=
k!
k=0
n
∑
Ak tk
k!
∞
k=m+1
∞
k
∑
|t|
Ak
k!
≤
n
k
∑
|t|
≤
Ak
k!
∞
≤
k=m
Now the series
that
∑∞
k=0
|t|k pk ∥A∥k
∞
k!
∞
k=m+1
∞
∞
k
k
∑
|t| pk ∥A∥∞
k!
k=m
converges and in fact equals exp (|t| p ∥A∥∞ ). It follows from calculus
lim
m→∞
∞
k
k
∑
|t| pk ∥A∥∞
= 0.
k!
k=m
∑n
k k
It follows that the ij entry of the partial sum k=0 Ak!t is a Cauchy sequence and hence by
completeness of C or R it converges. Therefore, the above limit exists. This is stated as the essential
part of the following theorem.
th
Theorem 18.6.4 Let t ∈ [a, b] ⊆ R where b − a < ∞. Then for each t ∈ [a, b] ,
n
∑
Ak tk
lim
n→∞
k!
k=0
≡
∞
∑
Ak tk
k!
k=0
≡ Φ (t) , A0 ≡ I,
exists. Furthermore, there exists a single constant C such that for tk ∈ [a, b] , the infinite sum
∞
∑
Ak tk
k
k!
k=0
converges and in fact
∞
∑
Ak tk
≤C
k
k=0
∑∞
k!
∞
k k
Proof: The convergence for k=0 Ak!t was just established.
Consider the estimate. From the above lemma,
n
∑
Ak tk
≤
k
k=0
k!
∞
≤
n
k
k
∑
pk (|a| + |b|) ∥A∥
∞
k=0
∞
∑
k=0
k!
k
k
pk (|a| + |b|) ∥A∥∞
k!
= exp (p (|a| + |b|) ∥A∥∞ )
∑n Ak tk
It follows that the ij th entry of k=0 k! k has magnitude no larger than the right side of the above
inequality. Also, a repeat of the above argument after Lemma 18.6.3 shows that the partial sums of
∑∞ Ak tkk
the ij th entry of
k=0 k! form a Cauchy sequence. Hence passing to the limit, it follows from
calculus that
)
(∞
∑ Ak tk
≤ exp (p (|a| + |b|) ∥A∥∞ )
k!
k=0
ij
Since ij is arbitrary, this establishes the inequality. 386
CHAPTER 18. LINEAR TRANSFORMATIONS
Next consider the derivative of Φ (t). Why is Φ (t) a solution to the above 18.14? By the mean
value theorem,
Φ (t + h) − Φ (t)
h
∞
=
∞
1 ∑ (t + h) − tk k
1 ∑ k (t + θk h)
A =
h
k!
h
k!
k
k=0
= A
∞
k−1
∑
(t + θk h)
(k − 1)!
k=1
k−1
k=0
∞
∑
Ak−1 = A
k=0
h
Ak
k
(t + θk h) k
A , θk ∈ (0, 1)
k!
∑∞ k
Does this sum converge to k=0 tk! Ak ≡ Φ (t)? If so, it will have been shown that Φ′ (t) = AΦ (t).
By the mean value theorem again,
∞
k
∑
(t + θk h)
=
≤
k=0
∞
∑
h
h
k=0
∞
∑
k=0
k!
Ak −
∞ k
∑
t
k=0
k (t + η k h)
k!
k−1
k (t + η k h)
k!
k−1
θk
k!
Ak
∞
, η k ∈ (0, 1)
Ak
∞
Ak
∞
Now for |h| ≤ 1, the expression tk ≡ t + η k h ∈ [t − 1, t + 1] and so by Theorem 18.6.4,
∞
k
∑
(t + θk h)
k!
k=0
A −
k
∞ k
∑
t
k=0
k!
k
A
∞
∞
k−1
∑
k (t + η k h)
Ak
= h
k!
k=0
≤ C |h|
(18.15)
∞
for some C. Then
Φ (t + h) − Φ (t)
− AΦ (t)
h
= A
∞
∞
k
∑
(t + θk h)
k!
k=0
Ak − A
∞ k
∑
t
k=0
k!
Ak
∞
This converges to 0 as h → 0 by Lemma 18.6.3 and 18.15.Hence the ij th entry of the difference
quotient
Φ (t + h) − Φ (t)
h
∑∞ tk k
th
converges to the ij entry of the matrix A k=0 k! A = AΦ (t) . In other words,
Φ′ (t) = AΦ (t)
Now also it follows right away from the formula for the infinite sum that Φ (0) = I. This proves the
following theorem.
Theorem 18.6.5 Let A be a real or complex p × p matrix. Then there exists a differentiable p × p
matrix Φ (t) satisfying the following initial value problem.
Φ′ (t) = AΦ (t) , Φ (0) = I.
This matrix is given by the infinite sum
Φ (t) =
∞ k
∑
t
k=0
k!
Ak
with the usual convention that t0 = 1, A0 = I, 0! = 1. In addition to this, AΦ (t) = Φ (t) A.
(18.16)
18.6. THE MATRIX EXPONENTIAL, DIFFERENTIAL EQUATIONS
∗
387
Proof: Why is AΦ (t) = Φ (t) A? This follows from the observation that A obviously commutes
with the partial sums,
n
n
n
∑
∑
tk k ∑ tk k
tk k+1
A
A =
A A=
A
,
k!
k!
k!
k=0
k=0
k=0
and Lemma 18.6.3,
AΦ (t) = lim A
n→∞
(
n
∑
tk
k=0
k
k!
A = lim
Now let
Ψ (t) =
n→∞
n
∑
tk
k=0
k!
)
k
A
A = Φ (t) A. ∞
k
∑
(−A) tk
k!
k=0
In the same way as above Ψ′ (t) = (−A) Ψ (t) , Ψ (0) = I, and AΨ (t) = Ψ (t) A.
Lemma 18.6.6 Φ (t)
−1
= Ψ (t)
Proof:
(Φ (t) Ψ (t))
′
= Φ′ (t) Ψ (t) + Φ (t) Ψ′ (t)
= AΦ (t) Ψ (t) + Φ (t) (−A) Ψ (t)
= AΦ (t) Ψ (t) − AΦ (t) Ψ (t) = 0
Therefore, Φ (t) Ψ (t) is a constant matrix. Just use the usual calculus facts on the entries of the
matrix. This matrix can only be I because Φ (0) = Ψ (0) = I. What follows contains all mathematically significant features of a typical undergraduate differential equations course without the endemic loose ends which result when thoughtful students begin
to wonder whether there exist enough generalized eigenvectors and eigenvectors to represent the solution. This seems to be never explained in these beginning differential equations courses. However,
to see why there are enough of these, read the appendix on the Jordan form.
The formula in the following theorem is called the variation of constants formula. I have to admit
that I don’t see the mathematical significance for a typical undergraduate differential equations class
when a substantial part of such a course is contained in the following theorem.
Theorem 18.6.7 Let f (t) be continuous and let x0 ∈ Rp . Then there exists a unique solution to
the equation
x′ = Ax + f , x (0) = x0
This solution is given by the formula
∫
x (t) = Φ (t) x0 + Φ (t)
t
Ψ (s) f (s) ds
0
Where Φ (t) is the fundamental matrix described in Theorem 18.6.5 and Ψ (t) is the fundamental
matrix defined the same way from −A.
Proof: Suppose x′ = Ax + f . Then x′ − Ax = f . Multiply both sides by Ψ (t). Then
′
(Ψ (t) x)
= Ψ (t) x′ (t) + Ψ′ (t) x (t)
= Ψ (t) x′ (t) − AΨ (t) x (t)
= Ψ (t) x′ (t) − Ψ (t) Ax (t)
= Ψ (t) (x′ (t) − Ax (t))
Therefore,
′
(Ψ (t) x) = Ψ (t) f (t)
388
CHAPTER 18. LINEAR TRANSFORMATIONS
Hence
∫
t
Ψ (t) x (t) − x0 =
Ψ (s) f (s) ds
0
Therefore, multiplying on the left by Φ (t) ,
∫
x (t) = Φ (t) x0 + Φ (t)
t
Ψ (s) f (s) ds
0
Thus, if there is a solution, this is it. Now if x (t) is given by the above formula, then
x (0) = Φ (0) x0 = x0
and also
′
′
∫
′
t
x (t) = Φ (t) x0 + Φ (t)
∫
Ψ (s) f (s) ds + Φ (t) Ψ (t) f (t)
0
t
Ψ (s) f (s) ds + f (t) = Ax (t) + f (t) = AΦ (t) x0 + AΦ (t)
0
Observation 18.6.8 As a special case when A is a real or complex scalar, the above theorem shows
∑∞ k
that Φ (t) x0 ≡ k=0 tk! Ak x0 is the solution of the differential equation
x′ (t) = Ax, x (0) = x0 .
Another solution to this is eAt where this is defined in Section 1.7. It follows that in this special
case, the uniqueness provision of the above theorem shows that
∞ k
∑
t
k=0
k!
Ak = eAt
in the special case that A is a complex number.
18.6.1
Computing A Fundamental Matrix
In case that A is p × p and nondefective, you can easily compute Φ (t). Recall that in this case, there
exists a matrix S such that


λ1


..

S −1 AS = D = 
.


λp
where D is a diagonal matrix. Then A = SDS −1 . Now Φ (t) =

eλ1 t
∞
∞
∑ SDk S −1
∑ Dk

..
tk = S
tk S −1 = S 
.

k!
k!
k=0
k=0

 −1
S

eλ p t
Thus you can easily compute Φ (t) in this case. For the case where a λk is complex, see the above
observation for the meaning of eλt . Thus one can explicitly find the fundamental matrix in this case.
In fact you can find it whenever you can compute the eigenvalues exactly. Suppose that the
matrix A is defective. A chain based on the eigenvector v1 is an ordered list of nonzero vectors
(v1 , v2 , · · · , vm )
where (A − λI) vk+1 = vk and (A − λI) v1 = 0. Given such a chain, m > 1, consider
x (t) ≡
m
∑
k=1
tm−k
vk eλt
(m − k)!
18.6. THE MATRIX EXPONENTIAL, DIFFERENTIAL EQUATIONS
Then
x′ (t) = λ
m
∑
=λ
k=1
=λ
389
m−1
∑ (m − k) tm−(k+1)
tm−k
vk eλt +
vk eλt
(m − k)!
(m − k)!
k=1
m
∑
∗
k=1
tm−k
vk eλt +
(m − k)!
m
∑
k=1
m−1
∑
k=1
tm−(k+1)
vk eλt
(m − (k + 1))!
∑ tm−k
t
vk eλt +
vk−1 eλt
(m − k)!
(m − k)!
m
m−k
k=2
Recalling that Avk − vk−1 = λvk for k > 1, and Av1 − λv1 = 0,
=
m
∑
k=1
∑ tm−k
∑ tm−k
tm−k
Avk eλt −
vk−1 eλt +
vk−1 eλt
(m − k)!
(m − k)!
(m − k)!
=A
m
m
k=2
k=2
m
∑
k=1
tm−k
vk eλt = Ax (t)
(m − k)!
Thus each such chain results in a solution to the system of equations. Also, each such chain of
length m results in m solutions to the differential equation. Just consider the chains
v1 , (v1 , v2 ) , · · · , (v1 , v2 , · · · , vm ) ,
each determining a solution as described above. Letting xk (t) denote the solution which comes from
the chain (v1 , v2 , · · · , vk ) and the above formula involving a sum, it follows that xk (0) = vk . Thus
if you consider the solutions coming from the chains
v1 , (v1 , v2 ) , · · · , (v1 , v2 , · · · , vm )
and consider the vectors obtained by letting t = 0, this results in the ordered list of vectors
v1 , v2 , · · · , vm
This is a linearly independent set of vectors. Suppose for some l ≤ m
l
∑
ck vk = 0
k=1
where not all the ck = 0 and l is as small as possible for this to occur. Then since v1 ̸= 0, it must
be that l ≥ 2. Also, cl ̸= 0. Do A − λI to both sides. This gives
0=
l
∑
k=2
ck vk−1 =
l−1
∑
ck+1 vk
k=1
and so l was not as small as possible after all. Thus the set must be linearly independent after all.
Note that for C (λk ) , a chain based on an eigenvector corresponding to λk ,
A : span (C (λk )) → span (C (λk ))
Letting Ak be the linear transformation which comes from the restriction of A to span (C (λk )) , what
is the matrix of Ak with respect to the ordered basis (v1 , v2 , · · · , vm ) = C (λk )?
(A − λk I) vj = vj−1 , j > 1
while (A − λk I) v1 = 0. Then formally, the matrix of Ak is given by M where
(
) (
)
λk v1 v1 + λk v2 · · · vm−1 + λk vm = v1 v2 · · · vm M
390
CHAPTER 18. LINEAR TRANSFORMATIONS
It follows that M is of the form







λk

1
λk
..
.
..
.



,

1 
λk
a matrix which has all zeros except for λk down the main diagonal and 1 down the super diagonal.
This is called a Jordan block corresponding to the eigenvalue λk .
r
It can be proved that there are chains {C (λk )}k=1 of such vectors associated with each eigenvalue
λk such that the totality of these vectors form a basis for Cn . Then you simply use these solutions as
just described to obtain a matrix Φ (t) whose columns are each solutions to the differential equation
x′ = Ax
and since the vectors just mentioned form a basis, this yields a fundamental matrix.
With respect to this basis, the matrix A becomes similar to one in Jordan Canonical form and
the existence of this basis is equivalent to obtaining the existence of the Jordan form. This very
interesting theorem is discussed in an appendix if you are interested.
Example 18.6.9 Find a fundamental matrix for the system


2
1 −1


x′ =  −1 0
1 x
−1 −2 3
In this case the eigenvectors and eigenvalues are
 


0
−1
 


 1  ↔ 1,  1  ↔ 2
1
1
2
The characteristic polynomial is (λ − 1) (λ − 2) and so 2 is an eigenvalue of algebraic multiplicity
2. Corresponding to this eigenvalue, there is the above eigenvector and a generalized eigenvector v2
which comes from solving the system


−1


(A − 2I) v2 =  1 
1
This leads to the augmented matrix

0
1 −1

 −1 −2 1
−1 −2 1

−1

1 
1


1


a solution to this is  −1  . Therefore, there are three solutions to this differential equation
0

 





0
−1
−1
1

 t 
 2t 
 2t 
 2t
 1  e ,  1  e , t  1  e +  −1  e
1
1
1
0
18.7. EXERCISES
Then a fundamental matrix is
391

0
 t
Φ (t) =  e
et
−e2t
e2t
e2t

e2t − te2t

te2t − e2t 
te2t
You can check and see that this works. Other situations are similar. If you can believe that there
will be enough of these vectors to obtain a basis formed by chains, then this gives a way to obtain a
formula for the fundamental matrix. However, to verify that this is the case, you will need to deal
with the Jordan canonical form. From the above discussion involving a power series, the existence
of a fundamental matrix is not an issue. This is about finding it in closed form using well known
functions and it is only this which involves the necessity of considering the Jordan canonical form.
18.7
Exercises
1. Find the matrix with respect to the standard basis vectors for the linear transformation which
rotates every vector in R2 through an angle of π/3.
2. Find the matrix with respect to the standard basis vectors for the linear transformation which
rotates every vector in R2 through an angle of π/4.
3. Find the matrix with respect to the standard basis vectors for the linear transformation which
rotates every vector in R2 through an angle of π/12. Hint: Note that π/12 = π/3 − π/4.
4. Find the matrix with respect to the standard basis vectors for the linear transformation which
rotates every vector in R2 through an angle of 2π/3 and then reflects across the x axis.
5. Find the matrix with respect to the standard basis vectors for the linear transformation which
rotates every vector in R2 through an angle of π/3 and then reflects across the y axis.
6. Find the matrix with respect to the standard basis vectors for the linear transformation which
rotates every vector in R2 through an angle of 5π/12. Hint: Note that 5π/12 = 2π/3 − π/4.
7. Let V be an inner product space and u ̸= 0. Show that the function Tu defined by Tu (v) ≡
v − proju (v) is also a linear transformation. Here
proju (v) ≡
⟨v, u⟩
|u|
2
u
Now show directly from the axioms of the inner product that
⟨Tu v, u⟩ = 0
8. Let V be a finite dimensional inner product space, the field of scalars equal to either R or C.
Verify that f given by f v ≡ ⟨v, z⟩ is in L (V, F). Next suppose f is an arbitrary element of
L (V, F). Show the following.
(a) If f = 0, the zero mapping, then f v = ⟨v, 0⟩ for all v ∈ V .
(b) If f ̸= 0 then there exists z ̸= 0 satisfying ⟨u, z⟩ = 0 for all u ∈ ker (f ) .
(c) Explain why f (y) z − f (z) y ∈ ker (f ).
(d) Use part b. to show that there exists w such that f (y) = ⟨y, w⟩ for all y ∈ V .
(e) Show there is at most one such w.
You have now proved the Riesz representation theorem which states that every f ∈ L (V, F) is
of the form
f (y) = ⟨y, w⟩
for a unique w ∈ V.
392
CHAPTER 18. LINEAR TRANSFORMATIONS
9. ↑Let A ∈ L (V, W ) where V, W are two finite dimensional inner product spaces, both having
field of scalars equal to F which is either R or C. Let f ∈ L (V, F) be given by
f (y) ≡ ⟨Ay, z⟩
where ⟨⟩ now refers to the inner product in W. Use the above problem to verify that there
exists a unique w ∈ V such that f (y) = ⟨y, w⟩ , the inner product here being the one on V .
Let A∗ z ≡ w. Show that A∗ ∈ L (W, V ) and by construction,
⟨Ay, z⟩ = ⟨y,A∗ z⟩ .
In the case that V = Fn and W = Fm and A consists of multiplication on the left by an m × n
matrix, give a description of A∗ .
10. Let A be the linear transformation defined on the vector space of smooth functions (Those
which have all derivatives) given by Af = D2 + 2D + 1. Find ker (A). Hint: First solve
(D + 1) z = 0. Then solve (D + 1) y = z.
11. Let A be the linear transformation defined on the vector space of smooth functions (Those
which have all derivatives) given by Af = D2 + 5D + 4. Find ker (A). Note that you could first
find ker (D + 4) where D is the differentiation operator and then consider ker (D + 1) (D + 4) =
ker (A) and consider Sylvester’s theorem.
12. Suppose Ax = b has a solution where A is a linear transformation. Explain why the solution
is unique precisely when Ax = 0 has only the trivial (zero) solution.
13. Verify the linear transformation determined by the matrix
(
)
1 0 2
0 1 4
maps R3 onto R2 but the linear transformation determined by this matrix is not one to one.
14. Let L be the linear transformation taking polynomials of degree at most three to polynomials
of degree at most three given by
D2 + 2D + 1
where D is the
operator. Find the matrix of this linear transformation relative
{ differentiation
}
to the basis 1, x, x2 , x3 . Find the matrix directly and then find the matrix with respect to
the differential operator D + 1 and multiply this matrix by itself. You should get the same
thing. Why?
15. Let L be the linear transformation taking polynomials of degree at most three to polynomials
of degree at most three given by D2 + 5D + 4 where D is the{differentiation
} operator. Find
the matrix of this linear transformation relative to the bases 1, x, x2 , x3 . Find the matrix
directly and then find the matrices with respect to the differential operators D + 1, D + 4 and
multiply these two matrices. You should get the same thing. Why?
16. Show that if L ∈ L (V, W ) (linear transformation) where V and W are vector spaces, then if
Lyp = f for some yp ∈ V, then the general solution of Ly = f is of the form ker (L) + yp .
17. Let L ∈ L (V, W ) where V, W are vector spaces, finite or infinite dimensional, and define x ∼ y
if x − y ∈ ker (L) . Show that ∼ is an equivalence relation. (x ∼ x, if x ∼ y, then y ∼ x, and
x ∼ y and y ∼ z implies x ∼ z.) Next define addition and scalar multiplication on the space
of equivalence classes as follows. [x] ≡ {y : y ∼ x} .
[x] + [y] ≡ [x + y]
α [x] =
[αx]
18.7. EXERCISES
393
Show that these are well defined definitions and that the set of equivalence classes is a vector
space with respect to these operations. The zero is [ker L] . Denote the resulting vector space
by V / ker (L) . Now suppose L is onto W. Define a mapping A : V / ker (K) 7→ W as follows.
A [x] ≡ Lx
Show that A is well defined, one to one and onto.
18. If V is a finite dimensional vector space and L ∈ L (V, V ) , show that the minimal polynomial
for L equals the minimal polynomial of A where A is the n × n matrix of L with respect to
some basis.
19. Let A be an n × n matrix. Describe a fairly simple method based on row operations for
computing the minimal polynomial of A. Recall, that this is a monic polynomial p (λ) such
that p (A) = 0 and it has smallest degree of all such monic polynomials. Hint: Consider
2
I, A2 , · · · . Regard each as a vector in Fn and consider taking the row reduced echelon form
or something like this. You might also use the Cayley Hamilton theorem to note that you can
stop the above sequence at An .
20. Let A be an n × n matrix which is non defective. That is, there exists a basis of eigenvectors.
Show that if p (λ) is the minimal polynomial, then p (λ) has no repeated roots. Hint: First
show that the minimal polynomial of A is the same as the minimal polynomial of the diagonal
matrix


D (λ1 )


..

D=
.


D (λr )
Where D (λ) is a diagonal matrix having λ down the
∏r main diagonal and in the above, the λi
are distinct. Show that the minimal polynomial is i=1 (λ − λi ) .
21. Show that if A is an n × n matrix and the minimal polynomial has no repeated roots, then A is
non defective and there exists a basis of eigenvectors. Thus, from the above problem, a matrix
may be diagonalized if and only if its minimal polynomial has no repeated roots. (It turns out
this condition is something which is relatively easy to determine. You look at the polynomial
and its derivative and ask whether these are relatively prime. The answer to this question can
be determined using routine algorithms as discussed above in the section on polynomials and
fields. Thus it is possible to determine whether an n × n matrix is defective.) Hint: You might
want to use Theorem 18.3.1.
22. Recall the linearization of the Lotka Volterra equations used to model the interaction between
predators and prey. It was shown earlier that if x, y are the deviations from the equilibrium
point, then
c
x′ = −bxy − b y
d
a
y ′ = dxy + dx
b
If one is interested only in small values of x, y, that is, in behavior near the equilibrium point,
these are approximately equal to the system
c
x′ = −b y
d
a
y′ =
dx
b
Written more simply, for α, β > 0,
x′
y
′
=
−αy
=
βx
394
CHAPTER 18. LINEAR TRANSFORMATIONS
Find the solution to the initial value problem
)(
) (
) (
(
) (
)
x
x (0)
x0
x′
0 −α
,
=
=
y′
β
0
y
y (0)
y0
Also have a computer graph the vector fields (−xy − y, xy + x) and (−y, x) which result from
setting all the parameters equal to 1.
23. This and the next problems will demonstrate how to solve any second order linear differential
equation with constant coefficients. Suppose you want to find all smooth solutions y to
y ′′ + (a + b) y ′ + aby = 0
where a, b are complex numbers, a ̸= b. Then in terms of the differentiation operator D, this
can be written as
(D + a) (D + b) y = 0
Thus you want to have ker ((D + a) (D + b)) . Thus the two operators (D + a) and (D + b)
commute. Show this carefully. It just comes down to standard calculus manipulations. Also
show that ker (D + a) is of the form Ce−at where C is a constant. Use this to verify that
(D + a) is one to one on ker (D + b) and (D + b) is one to one on ker (D + a). Then use
Lemma 18.3.4 to verify that the solution to the above equation is of the form
ker ((D + a) (D + b)) = C1 e−at + C2 e−bt
Here we write e−at to signify the function t → e−at . This shows how to solve any equation
y ′′ +αy ′ +βy = 0 where α, β are real or complex numbers. This is because you can always write
such an equation in the form discussed above by simply factoring the quadratic polynomial
r2 + αr + β using the quadratic formula. Of course you might need to use DeMoivre’s theorem
to take square roots. This is called the general solution to the homogeneous equation. Once
you have found the general solution, you can then determine the constants C1 , C2 in such a
way that the solution to a given initial condition where y (0) , y ′ (0) are given may be obtained.
For example, if I wanted y (0) = 1, y ′ (0) = 0, this would be easy to find. I would just need to
solve the equations
y (0)
′
=
y (0) =
C1 + C2 = 1
(−a) C1 + (−b) C2 = 0
then solve these equations as in the first few chapters of the book.
2
24. What if the second order equation is of the form (D + a) y = 0? Show that
(
)
2
ker (D + a) = C1 e−at + C2 te−at
(
)
2
To show this, verify directly that the right side is in ker (D + a) . Next suppose y ∈
(
)
2
ker (D + a) . Thus
(D + a) ((D + a) y) = 0
and so
(D + a) y = Ce−at
because you know ker (D + a) . Now simply solve the above equation as follows. You have
y ′ + ay = Ce−at
Multiply both sides by eat and verify that this yields
d ( at )
e y =C
dt
2
Now finish the details to verify that everything in ker (D + a) is of the desired form.
18.7. EXERCISES
395
25. The case of real coefficients is very important and in fact is the case of most interest. Show
that if α, β are real, then if y is a solution to y ′′ + αy ′ + βy = 0, then so is ȳ, the complex
conjugate.
((
))
26. If you have the solution to ker D2 + (a + b) D + ab = C1 eat + C2 ebt and a, b are complex
but a + b, ab are real, show that in fact, they are complex conjugates
so a = α + iβ,
((
))b = α − iβ.
Then explain why you get exactly the same solution for ker D2 + (a + b) D + ab by simply
writing in the form
e−αt (B1 (cos (βt)) + B2 sin (βt))
The advantage in doing this is that everything is expressed in terms of real functions and
typically you are looking for real solutions.
27. Find the solutions to the following initial value problems. Write in terms of real valued
functions.
(a) y ′′ + 4y ′ + 3y = 0, y (0) = 1, y ′ (0) = 1
(d) y ′′ + 5y ′ + 4y = 0, y (0) = 0, y ′ (0) = 1
(b) y ′′ + 2y ′ + 2y = 0, y (0) = 0, y ′ (0) = 1
(e) y ′′ + 4y = 0, y (0) = 1, y ′ (0) = 0
(c) y ′′ + 4y ′ + 4y = 0, y (0) = 1, y ′ (0) = 1
(f) y ′′ − 2y ′ + 5y = 0, y (0) = 2, y ′ (0) = 1
28. If you have an initial value problem of the form
y ′′ + b (t) y ′ + c (t) y = 0, y (t0 ) = y0 , y ′ (t0 ) = y1
Explain why it can be written in the form
(
)′ (
)(
) (
) (
)
x1
0
1
x1
x1 (t0 )
y0
=
,
=
x2
−c (t) −b (t)
x2
x2 (t0 )
y1
Easy if you take x1 = y and x′1 = x2 . Then you just plug in to the equation and get what is
desired.
29. Suppose you have two solutions y, ŷ to the “homogeneous” equation
y ′′ + b (t) y ′ + c (t) y = 0
The Wronskian of these is defined as
(
W (y, ŷ) (t) ≡ W (t) ≡ det
Show that
y (t)
y ′ (t)
(18.17)
ŷ (t)
ŷ ′ (t)
)
W ′ (t) + b (t) W (t) = 0
Now let B ′ (t) = b (t) . Multiply by eB(t) and verify that
(
)′
eB(t) W (t) = 0
Hence
W (t) = Ce−B(t)
This is Abel’s formula. Note that this shows that the Wronskian either vanishes for all t or
for no t.
30. Suppose you have found two solutions y, ŷ to 18.17. Consider their Wronskian. Show that the
Wronskian is nonzero if and only if the ratio of these solutions is not a constant. Hint: This
is real easy. Just use the quotient rule and observe that in the quotient rule the numberator
is a multiple of the Wronskian.
396
CHAPTER 18. LINEAR TRANSFORMATIONS
31. ↑Show that if you have two solutions y, ŷ of 18.17, whose Wronskian is nonzero, then for any
choice of y0 , y1 , there exist constants C1 , C2 such that there is exactly one solution to the initial
value problem
y ′′ + b (t) y ′ + c (t) y = 0, y (t0 ) = y0 , y ′ (t0 ) = y1
which is of the form C1 y + C2 ŷ. When this happens, we say that we have the general solution
in the form C1 y + C2 ŷ. Explain why all solutions to the equation must be of this form.
32. ↑Suppose you have found a single solution to the equation
y ′′ + b (t) y ′ + c (t) y = 0
How can you go about finding the general solution to
y ′′ + b (t) y ′ + c (t) y = 0?
Hint: You know that all you have to do is to find another solution ŷ such that it is not a
scalar multiple of y. Also, you know Abel’s formula. Letting B ′ (t) = b (t) ,
ŷ (t)
ŷ ′ (t)
y (t)
y ′ (t)
= Ce−B(t)
and so you have a differential equation for ŷ
−y (t) ŷ ′ (t) + ŷ (t) y ′ (t) = e−B(t)
You don’t care to find all examples. All you need is one which is not a multiple of y. That is
why it suffices to let C = 0. Then
ŷ ′ (t) −
y ′ (t)
e−B(t)
ŷ (t) = −
y (t)
y (t)
Now solve this equation for ŷ. First multiply by 1/y. Verify that
d
e−B(t)
(ŷ/y) = − 2
dt
y (t)
Now from this describe how to find ŷ which is not a constant multiple of y.
33. If you have the general solution for y ′′ + b (t) y ′ + c (t) y = 0, two solutions y, ŷ having nonzero
Wronskian, show that it is always possible to obtain all solutions to
y ′′ + b (t) y ′ + c (t) y = f (t)
and that such a solution will be of the form A (t) y (t) + B (t) ŷ (t). This is called variation of
parameters. Hint: You want it to work. Plug in and require it to do so.


≡0
z }| {


c (t) (y = Ay + B ŷ) , b (t) y ′ = A′ y + B ′ ŷ + Ay ′ + B ŷ ′ 
1 (y ′′ = A′ y ′ + B ′ ŷ ′ + Ay ′′ + B ŷ ′′ )
Then show we get a solution if A′ y + B ′ ŷ = 0, A′ y ′ + B ′ ŷ ′ = f . Now get a solution to this
using that the Wronskian is not zero. Then if z is any solution and ỹ is the one you just found,
z − ỹ solves y ′′ + b (t) y ′ + c (t) y = 0 and so is of the form C1 y + C2 ŷ.
34. Explain how to write the higher order differential equation
y (k) + ak−1 y (k−1) + · · · + a1 y ′ + a0 y = f (t)
as a first order system of the form
x′ = Ax + F
Thus the theory of this chapter includes all ordinary differential equations with constant
coefficients.
Appendix A
The Jordan Canonical Form*
Recall Corollary 18.4.4. For convenience, this corollary is stated below.
Corollary A.0.1 Let A be an n×n matrix. Then A is similar to an upper triangular, block diagonal
matrix of the form


T1 · · · 0
 . .

.
. . ... 
T ≡
 .

0 · · · Tr
where Tk is an upper triangular matrix having only λk on the main diagonal. The diagonal blocks
can be arranged in any order desired. If Tk is an mk × mk matrix, then
rk
mk = dim ker (A − λk I)
where the minimal polynomial of A is
p
∏
(λ − λk )
.
rk
k=1
The Jordan Canonical form involves a further reduction in which the upper triangular matrices,
Tk assume a particularly revealing and simple form.
Definition A.0.2 Jk (α) is a Jordan block if it is a k × k matrix of the form


α 1
0


 0 ... ...



Jk (α) =  .

..
 . ..

.
. 1 
 .
0 ···
0 α
In words, there is an unbroken string of ones down the super diagonal and the number, α filling
every space on the main diagonal with zeros everywhere else. A matrix is strictly upper triangular
if it is of the form


0 ∗ ∗
 . .

 .
.. ∗  ,
 .

0 ··· 0
where there are zeroes on the main diagonal and below the main diagonal.
397
398
APPENDIX A. THE JORDAN CANONICAL FORM*
The Jordan canonical form involves each of the upper triangular matrices in the conclusion of
Corollary 18.4.4 being a block diagonal matrix with the blocks being Jordan blocks in which the size
of the blocks decreases from the upper left to the lower right. The idea is to show that every square
matrix is similar to a unique such matrix which is in Jordan canonical form. It is assumed here
that the field of scalars is C but everything which will be done below works just fine if the minimal
polynomial can be completely factored into linear factors in the field of scalars.
Note that in the conclusion of Corollary 18.4.4 each of the triangular matrices is of the form
αI + N where N is a strictly upper triangular matrix. The existence of the Jordan canonical form
follows quickly from the following lemma.
Lemma A.0.3 Let N be an n × n matrix which is strictly upper triangular. Then there exists an
invertible matrix S such that


Jr1 (0)
0


Jr2 (0)


−1


S NS = 
..

.


0
Jrs (0)
∑s
where r1 ≥ r2 ≥ · · · ≥ rs ≥ 1 and i=1 ri = n.
Proof: First note the only eigenvalue of N is 0. Let v1 be an eigenvector. Then {v1 , v2 , · · · , vr }
is called a chain if N vk+1 = vk for all k = 1, 2, · · · , r and v1 is an eigenvector so N v1 = 0. It will
be called a maximal chain if there is no solution v, to the equation, N v = vr .
Claim 1: The vectors in any chain are linearly independent and for
{v1 , v2 , · · · , vr }
a chain based on v1 ,
N : span (v1 , v2 , · · · , vr ) 7→ span (v1 , v2 , · · · , vr ) .
(1.1)
Also if {v1 , v2 , · · · , vr } is a chain, then r ≤ n.
Proof: First note that 1.1 is obvious because
N
r
∑
ci vi =
i=1
r
∑
ci vi−1 .
i=2
It only remains to verify the vectors of a chain are independent. Suppose then
r
∑
ck vk = 0.
k=1
Do N r−1 to it to conclude cr = 0. Next do N r−2 to it to conclude cr−1 = 0 and continue this way.
Now it is obvious r ≤ n because the chain is independent. This proves the claim.
Consider the set of all chains based on eigenvectors.
{ Since all have
} total length no larger than n
it follows there exists one which has maximal length, v11 , · · · , vr11 ≡ B1 . If span (B1 ) contains all
eigenvectors of N, then
consider all chains based on eigenvectors not in span (B1 )
{ stop. Otherwise,
}
and pick one, B2 ≡ v12 , · · · , vr22 which is as long as possible. Thus r2 ≤ r1 . If span (B1 , B2 )
contains all eigenvectors of N, stop.
{ Otherwise,
} consider all chains based on eigenvectors not in
span (B1 , B2 ) and pick one, B3 ≡ v13 , · · · , vr33 such that r3 is as large as possible. Continue this
way. Thus rk ≥ rk+1 .
Claim 2: The above process terminates with a finite list of chains
{B1 , · · · , Bs }
because for any k, {B1 , · · · , Bk } is linearly independent.
399
Proof of Claim 2: The claim is true if k = 1. This follows from Claim 1. Suppose it is true for
k − 1, k ≥ 2. Then {B1 , · · · , Bk−1 } is linearly independent. Suppose
p
∑
cq wq = 0, cq ̸= 0
q=1
where the wq come from {B1 , · · · , Bk−1 , Bk } . By induction, some of these wq must come from Bk .
Let vik be the one for which i is as large as possible. Then do N i−1 to both sides to obtain v1k , the
eigenvector upon which the chain Bk is based, is a linear combination of {B1 , · · · , Bk−1 } contrary to
the construction. Since {B1 , · · · , Bk } is linearly independent, the process terminates. This proves
the claim.
Claim 3: Suppose N w = 0. (w is an eigenvector) Then there exist scalars, ci such that
w=
s
∑
ci v1i .
i=1
Recall that v1i is the eigenvector in the ith chain on which this chain is based.
Proof of Claim 3: From the construction, w ∈ span (B1 , · · · , Bs ) since otherwise, it could serve
as a base for another chain. Therefore,
w=
ri
s ∑
∑
cki vki .
i=1 k=1
Now apply N to both sides.
0=
ri
s ∑
∑
i
cki vk−1
i=1 k=2
and so by Claim 2,
cki
= 0 if k ≥ 2. Therefore,
w=
s
∑
c1i v1i
i=1
and this proves the claim.
If N w = 0, then w ∈ span (B1 , · · · , Bs ) . In fact, it was a particular linear combination involving
the bases of the chains. What if N k w = 0? Does it still follow that w ∈ span (B1 , · · · , Bs )?
Claim 4: If N k w = 0, then w ∈ span (B1 , · · · , Bs ) .
Proof of Claim 4: Suppose this true that if N k−1 w, k ≥ 2, it follows that w ∈ span (B1 , · · · , Bs ) .
Then suppose N k w = 0. If N k−1 w = 0, then by induction, w ∈ span (B1 , · · · , Bs ). The other case
is that N k−1 w ̸= 0. Then you have the chain
N k−1 w, · · · , w
where N k−1 w is an eigenvector. The chain has length no longer than the lengths of any of the Bi
by construction. Then by Claim 3,
N k−1 w =
s
∑
ci v1i =
i=1
And so,
(
N k−1
w−
s
∑
ci N k−1 vki
i=1
s
∑
)
ci vki
=0
i=1
which implies by induction that
w−
s
∑
i=1
ci vki ∈ span (B1 , · · · , Bs ) .
400
APPENDIX A. THE JORDAN CANONICAL FORM*
This proves Claim 4.
Since every w satisfies N n w = 0, this shows that span (B1 , · · · , Bs ) = Fn .
{B1 , · · · , Bs } is independent. Therefore, this is a basis for Fn .
Now consider the block matrix
(
)
S = B1 · · · Bs
(
where
Bk =
···
v1k
vrkk
)
.

Thus
S −1

C1
 . 
. 
=
 . 
Cs
From the construction, N vjk ∈ span (Bk ) , and so
Ci N vjk = 0.
It follows that Ci Bi = Iri ×ri and Ci N Bj = 0 if i ̸= j.

uT1
 .
.
Ck = 
 .
uTrk
Let


.

Then noting that Bk is n × rk and Ck is rk × n,


uT1
)
 . (
k
k
. 
Ck N Bk = 
 .  N v1 · · · N vrk
uTrk


uT1
)
 . (
k
k
. 
= 
 .  0 v1 · · · vrk −1
uTrk
which equals an rk × rk matrix of the form




Jrk (0) = 


0 1
.. . .
.
.
..
.
0 ···
···
..
.
..
.
···

0
.. 
. 



1 
0
That is, it has ones down the super diagonal and zeros everywhere else. It follows


C1
)
 .  (
. 
S −1 N S = 
 .  N B1 · · · Bs
Cs


Jr1 (0)
0


Jr2 (0)




= 
..

.


0
Jrs (0)
By Claim 2,
401
as claimed. Now let the upper triangular matrices, Tk be given in the conclusion of Corollary 18.4.4. Thus,
as noted earlier,
Tk = λk Irk ×rk + Nk
where Nk is a strictly upper triangular matrix of the sort just discussed in Lemma A.0.3. Therefore,
there exists Sk such that Sk−1 Nk Sk is of the form given in Lemma A.0.3. Now Sk−1 λk Irk ×rk Sk =
λk Irk ×rk and so Sk−1 Tk Sk is of the form






Ji1 (λk )
0
Ji2 (λk )
..
.
0
where i1 ≥ i2 ≥ · · · ≥ is and
∑s
j=1 ij






Jis (λk )
= rk . This proves the following corollary.
Corollary A.0.4 Suppose A is an upper triangular n × n matrix having α in every position on the
main diagonal. Then there exists an invertible matrix S such that


Jk1 (α)
0


Jk2 (α)


−1


S AS = 
..

.


0
Jkr (α)
where k1 ≥ k2 ≥ · · · ≥ kr ≥ 1 and
∑r
i=1
ki = n.
The next theorem is gives the existence of the Jordan canonical form.
Theorem A.0.5 Let A be an n × n matrix having eigenvalues λ1 , · · · , λr where the multiplicity of
λi as a zero of the characteristic polynomial equals mi . Then there exists an invertible matrix S such
that


J (λ1 )
0


..

S −1 AS = 
(1.2)
.


0
J (λr )
where J (λk ) is an mk × mk matrix of the form






Jk1 (λk )
0
Jk2 (λk )
..
0
where k1 ≥ k2 ≥ · · · ≥ kr ≥ 1 and
∑r
i=1
.






Jkr (λk )
ki = mk .
Proof: From Corollary 18.4.4, there exists S such that S −1 AS is of the form


T1 · · · 0
 .
.. 
..

..
T ≡
.
. 

0 · · · Tr
(1.3)
402
APPENDIX A. THE JORDAN CANONICAL FORM*
where Tk is an upper triangular mk × mk matrix having only λk on the main diagonal. By Corollary
A.0.4 There exist matrices, Sk such that Sk−1 Tk Sk = J (λk ) where J (λk ) is described in 1.3. Now
let M be the block diagonal matrix given by


S1
0


..
.
M =
.


0
Sr
It follows that M −1 S −1 ASM = M −1 T M and this is of the desired form. What about the uniqueness of the Jordan canonical form? Obviously if you change the order of
the eigenvalues, you get a different Jordan canonical form but it turns out that if the order of the
eigenvalues is the same, then the Jordan canonical form is unique. In fact, it is the same for any
two similar matrices.
Theorem A.0.6 Let A and B be two similar matrices. Let JA and JB be Jordan forms of A and
B respectively, made up of the blocks JA (λi ) and JB (λi ) respectively. Then JA and JB are identical
except possibly for the order of the J (λi ) where the λi are defined above.
Proof: First note that for λi an eigenvalue, the matrices JA (λi ) and JB (λi ) are both of size mi ×
mi because the two matrices A and B, being similar, have exactly the same characteristic equation
and the size of a block equals the algebraic multiplicity of the eigenvalue as a zero of the characteristic
equation. It is only necessary to worry about the number and size of the Jordan blocks making up
JA (λi ) and JB (λi ) . Let the eigenvalues of A and B be {λ1 , · · · , λr } . Consider the two sequences of
m
m
numbers {rank (A − λI) } and {rank (B − λI) }. Since A and B are similar, these two sequences
m
m
coincide. (Why?) Also, for the same reason, {rank (JA − λI) } coincides with {rank (JB − λI) } .
m
m
Now pick λk an eigenvalue and consider {rank (JA − λk I) } and {rank (JB − λk I) } . Then


JA (λ1 − λk )
0


..


.




JA − λk I = 
JA (0)





..
.


JA (λr − λk )
0
and a similar formula holds for JB − λk I. Here

Jk1 (0)

Jk2 (0)

JA (0) = 


0
and



JB (0) = 


0
..
.






Jkr (0)
Jl1 (0)
0
Jl2 (0)
..
.






0
Jlp (0)
∑
∑
and it suffices to verify that li = ki for all i. As noted above,
ki =
li . Now from the above
formulas,
∑
m
m
rank (JA − λk I)
=
mi + rank (JA (0) )
i̸=k
=
∑
m
mi + rank (JB (0) )
i̸=k
m
= rank (JB − λk I) ,
403
m
m
which shows rank (JA (0) ) = rank (JB (0) ) for all m. However,

m
JB (0)


=


m
Jl1 (0)

0





m
Jl2 (0)
..
.
m
0
Jlp (0)
∑p
m
m
m
with a similar formula holding for JA (0) and rank (JB (0) ) = i=1 rank (Jli (0) ) , similar for
m
rank (JA (0) ) . In going from m to m + 1,
(
)
m
m+1
rank (Jli (0) ) − 1 = rank Jli (0)
until m = li at which time there is no further change. Therefore, p = r since otherwise, there would
exist a discrepancy right away in going from m = 1 to m = 2. Now suppose the sequence {li } is
not equal to the sequence, {ki }. Then lr−b ̸= kr−b for some b a nonnegative integer taken to be a
small as possible. Say lr−b > kr−b . Then, letting m = kr−b ,
r
∑
i=1
m
rank (Jli (0) ) =
r
∑
m
rank (Jki (0) )
i=1
and in going to m + 1 a discrepancy must occur because the sum on the right will contribute less to
the decrease in rank than the sum on the left. 404
APPENDIX A. THE JORDAN CANONICAL FORM*
Appendix B
Directions For Computer Algebra
Systems
B.1
Finding Inverses
II
B.2
Finding Row Reduced Echelon Form
II
B.3
Finding P LU Factorizations
II
B.4
Finding QR Factorizations
II
B.5
Finding The Singular Value Decomposition
First here is how you do it using Scientific Notebook. It does it very easily.
I
Next, here is the rather elaborate procedure to get the singular value decomposition using maple.
I am using maple 12. Maybe there is an easier way with a more up to date version. I am not sure.
I
B.6
Use Of Matrix Calculator On Web
There is a really nice service on the web which will do all of these things very easily. It is
www.bluebit.gr/matrix-calculator/ or click on following link or google matrix calculator.
I
You enter a matrix row by row, placing a space between each number. When you come to the
end of a row, you press enter on the keyboard to enter the next row. When you have entered the
matrix, you select what you want it to do.
I
405
406
APPENDIX B. DIRECTIONS FOR COMPUTER ALGEBRA SYSTEMS
Bibliography
[1] Apostol T. Calculus Volume II Second edition, Wiley 1969.
[2] Baker, Roger, Linear Algebra, Rinton Press 2001.
[3] Davis H. and Snider A., Vector Analysis Wm. C. Brown 1995.
[4] Edwards C.H. Advanced Calculus of several Variables, Dover 1994.
[5] Chahal J.S., Historical Perspective of Mathematics 2000 B.C. - 2000 A.D. Kendrick Press,
Inc. (2007)
[6] Golub, G. and Van Loan, C.,Matrix Computations, Johns Hopkins University Press, 1996.
[7] Greenberg M.D. Advanced Engineering Mathematics Prentice Hall 1998 Second edition.
[8] Gurtin M. An introduction to continuum mechanics, Academic press 1981.
[9] Hardy G. A Course Of Pure Mathematics, Tenth edition, Cambridge University Press 1992.
[10] Horn R. and Johnson C. matrix Analysis, Cambridge University Press, 1985.
[11] Jacobsen N. Basic Algebra Freeman 1974.
[12] Karlin S. and Taylor H. A First Course in Stochastic Processes, Academic Press, 1975.
[13] Kuttler K.Linear Algebra On web page. Linear Algebra
[14] Nobel B. and Daniel J. Applied Linear Algebra, Prentice Hall, 1977.
[15] Rudin W. Principles of Mathematical Analysis, McGraw Hill, 1976.
[16] Salas S. and Hille E., Calculus One and Several Variables, Wiley 1990.
[17] Strang Gilbert, Linear Algebra and its Applications, Harcourt Brace Jovanovich 1980.
[18] Wilkinson, J.H., The Algebraic Eigenvalue Problem, Clarendon Press Oxford 1965.
[19] Yosida K., Functional Analysis, Springer Verlag, 1978.
407
408
BIBLIOGRAPHY
Appendix C
Answers To Selected Exercises
C.1
Exercises 11
De Moivre’s theorem. Now using De Moivre’s
theorem, derive a formula for sin (5x) and
one for cos (5x).
5 Let z = 5 + i9. Find z −1 .
(5 + i9)
−1
=
5
106
−
9
106 i
6 Let z = 2 + i7 and let w = 3 − i8. Find
zw, z + w, z 2 , and w/z.
62 + 5i, 5 − i, −45 + 28i, and − 50
53 −
37
53 i.
8 Graph the complex cube roots of 8 in the
complex plane. Do the same for the four
fourth roots of 16.
3
The cube roots are
√ the solutions
√ to z +8 =
0, Solution is: i 3 + 1, 1 − i 3, −2
The fourth roots are the solutions to z 4 +
16 = 0, Solution is:
√
√
√
(1 − i) 2, − (1 + i) 2, − (1 − i) 2, (1 + i)
√
2. When you graph these, you will have
three equally spaced points on the circle of
radius 2 for the cube roots and you will
have four equally spaced points on the circle of radius 2 for the fourth roots. Here
are pictures which should result.
sin (5x) = 5 cos4 x sin x − 10 cos2 x sin3 x +
sin5 x
cos (5x) = cos5 x−10 cos3 x sin2 x+5 cos x sin4 x
13 Factor x3 + 8 as a product of linear factors.
√
√
x3 + 8 = 0, Solution is: i 3 + 1, 1 − i 3, −2
and so this polynomial equals
(
(√
)) (
(
√ ))
(x + 2) x − i 3 + 1
x− 1−i 3
(
)
14 Write x3 +27 in the form (x + 3) x2 + ax + b
where x2 + ax + b cannot be factored any
more using only real numbers.
(
)
x3 + 27 = (x + 3) x2 − 3x + 9
16 Factor x4 +16 as the product of two quadratic
polynomials each of which cannot be factored further without using complex numbers.
√
√
(
)(
)
x4 +16 = x2 − 2 2x + 4 x2 + 2 2x + 4 .
You can use the information in the preceding problem. Note that (x − z) (x − z) has
real coefficients.
17 If z, w are complex numbers prove zw = zw
and then show by induction that z1 · · · zm =
z1 · · · zm . Also verify that
m
∑
zk =
m
∑
zk .
9 If z is a complex number, show there exists
ω a complex number with |ω| = 1 and ωz =
|z| .
z
If z = 0, let ω = 1. If z ̸= 0, let ω =
|z|
In words this says the conjugate of a product equals the product of the conjugates
and the conjugate of a sum equals the sum
of the conjugates.
11 You already know formulas for cos (x + y)
and sin (x + y) and these were used to prove
(a + ib) (c + id) = ac − bd + i (ad + bc) =
(ac − bd) − i (ad + bc)
409
k=1
k=1
410
APPENDIX C. ANSWERS TO SELECTED EXERCISES
(a − ib) (c − id) = ac−bd−i (ad + bc) which
is the same thing. Thus it holds for a product of two complex numbers. Now suppose
you have that it is true for the product of
n complex numbers. Then z1 · · · zn+1 =
z1 · · · zn zn+1 and now, by induction this
equals
z1 · · · zn zn+1
As to sums, this is even easier.
n
∑
(xj + iyj ) =
j=1
=
=
n
∑
xj + i
j=1
n
∑
j=1
n
∑
xj − i
n
∑
yj =
j=1
n
∑
yj
j=1
n
∑
xj − iyj
j=1
(xj + iyj ).
j=1
21 De Moivre’s theorem is really a grand thing.
I plan to use it now for rational exponents,
not just integers.
1/4
1 =
1(1/4) = (cos 2π + i sin 2π)
=
cos (π/2) + i sin (π/2) = i.
Therefore, squaring both sides it follows 1 =
−1 as in the previous problem. What does
this tell you about De Moivre’s theorem?
Is there a profound difference between raising numbers to integer powers and raising
numbers to non integer powers?
It doesn’t work. This is because there are
four fourth roots of 1.
22 Here is another question: If n is an integer,
n
is it always true that (cos θ − i sin θ) =
cos (nθ) − i sin (nθ)? Explain.
Yes, this is true.
18 Suppose p (x) = an xn + an−1 xn−1 + · · · +
a1 x + a0 where all the ak are real numbers.
Suppose also that p (z) = 0 for some z ∈ C.
Show it follows that p (z) = 0 also.
You just use the above problem. If p (z) =
0, then you have p (z) = 0 =
an z n + an−1 z n−1 + · · · + a1 z + a0
= an z n + an−1 z n−1 + · · · + a1 z + a0
= an z n + an−1 z n−1 + · · · + a1 z + a0
= an z n + an−1 z n−1 + · · · + a1 z + a0
= p (z)
19 Show that 1 + i, 2 + i are the only two zeros
to
n
(cos θ − i sin θ)
n
= (cos (−θ) + i sin (−θ))
=
=
cos (−nθ) + i sin (−nθ)
cos (nθ) − i sin (nθ)
23 Suppose you have any polynomial in cos θ
and sin θ. By this I mean an expression of
the
∑mform
∑n
β
α
α=0
β=0 aαβ cos θ sin θ where aαβ ∈
C.
Can this always be ∑
written in the form
∑m+n
n+m
τ =−(n+m) cτ sin τ θ?
γ=−(n+m) bγ cos γθ+
Explain.
Yes it can. It follows from the identities for
the sine and cosine of the sum and difference of angles that
p (x) = x2 − (3 + 2i) x + (1 + 3i)
sin a sin b
=
so the zeros do not necessarily come in conjugate pairs if the coefficients are not real.
cos a cos b
=
(x − (1 + i)) (x − (2 + i)) = x2 −(3 + 2i) x+
1 + 3i
sin a cos b
=
20 I claim that 1 = −1. Here is why.
√
√
√ √
2
−1 = i2 = −1 −1 = (−1) = 1 = 1.
This is clearly a remarkable result but is
there something wrong with it? If so, what
is wrong?
Something
is wrong.
√
−1.
There is no single
1
(cos (a − b) − cos (a + b))
2
1
(cos (a + b) + cos (a − b))
2
1
(sin (a + b) + sin (a − b))
2
Now cos θ = 1 cos θ + 0 sin θ and sin θ =
0 cos θ + 1 sin θ. Suppose that whenever k ≤
n,
cosk (θ) =
k
∑
aj cos (jθ) + bj sin (jθ)
j=−k
for some numbers aj , bj . Then
cosn+1 (θ) =
C.2. EXERCISES 23
n
∑
411
(a) x2 + 2x√+ 1 + i √
= 0, Solution is :√x =
−1 + 21 2 − 12 i 2, x = −1 − 12 2 +
√
1
2i 2
aj cos (θ) cos (jθ) + bj cos (θ) sin (jθ)
j=−n
Now use the above identities to write all
products as sums of sines and cosines of
(j − 1) θ, jθ, (j + 1) θ. Then adjusting the
constants, it follows
(b) 4x2 + 4ix − 5 = 0, Solution is : x =
1 − 12 i, x = −1 − 12 i
(c) 4x2 + (4 + 4i) x + 1 + 2i = 0, Solution
is : x = − 12 , x = − 12 − i
cosn+1 (θ) =
n+1
∑
(d) x2 − 4ix − 5 = 0, Solution is : x =
−1 + 2i, x = 1 + 2i
a′j cos (θ) cos (jθ) + b′j cos (θ) sin (jθ)
(e) 3x2 + (1 − i)
Solution
is :
√x + 3i( = 0, √
)
x = − 16 + 61 19 + 16 − 61 19 i, x =
√
√ )
(
− 61 − 61 19 + 16 + 16 19 i
j=−n+1
You can do something similar with sinn (θ)
and with products of the form
cosα θ sinβ θ.
24 Suppose p (x) = an xn + an−1 xn−1 + · · · +
a1 x+a0 is a polynomial and it has n zeros,
27 Prove the fundamental theorem of algebra
for quadratic polynomials having coefficients
in C.
This is pretty easy because you can simply
write the quadratic formula. Finding the
square roots of complex numbers is easy
from the above presentation. Hence, every quadratic polynomial has two roots in
C. Note that the two square roots in the
quadratic formula are on opposite sides of
the unit circle so one is −1 times the other.
z1 , z2 , · · · , zn
listed according to multiplicity. (z is a root
of multiplicity m if the polynomial f (x) =
m
(x − z) divides p (x) but (x − z) f (x) does
not.) Show that
p (x) = an (x − z1 ) (x − z2 ) · · · (x − zn ) .
p (x) = (x − z1 ) q (x) + r (x) where r (x) is
a nonzero constant or equal to 0. However,
r (z1 ) = 0 and so r (x) = 0. Now do to q (x)
what was done to p (x) and continue until
the degree of the resulting q (x) equals 0.
Then you have the above factorization.
25 Give the solutions to the following quadratic
equations having real coefficients.
(a) x2 − 2x + 2 = 0, Solution is: 1 + i, 1 − i
√
(b) 3x2 + x + 3 = 0, Solution is: 16 i 35 −
√
1
1
1
6 , − 6 i 35 − 6
(c) x2 −6x+13 = 0, Solution is: 3+2i, 3−
2i
√
(d) x2 +√
4x + 9 = 0, Solution is: i 5 −
2, −i 5 − 2
(e) 4x2 + 4x + 5 = 0, Solution is: − 12 +
i, − 21 − i
26 Give the solutions to the following quadratic
equations having complex coefficients. Note
how the solutions do not come in conjugate
pairs as they do when the equation has real
coefficients.
C.2
1
(
Exercises 23
50 +
300
√
2
)
√ j
i+ 300
2
2 θ = 9.56 degrees.
3 Will need 68. 966 gallons of gas. Therefore,
it will not make it.
(
)
√
4 At 155 75 3 + 150
(
)
= 155.0 279. 9
5 It takes 2 hours
√
6 16 13 miles. He ends up 1/3 mile down
stream.
( √ )
7 23 , 35 In the second case, he could not
do it.
8 (−3, 2, −5) .
9 (3, 0, 0).
( √
√
10
5 2 + 25 3
√
11 T = 50 26.
11
2
√ )
−5 2
412
APPENDIX C. ANSWERS TO SELECTED EXERCISES
C.3
Exercises 33
7 113
1 This formula says that
u · v = |u| |v| cos θ where θ is the included
angle between the two vectors. Thus
|u · v| = |u| |v| |cos θ| ≤ |u| |v|
and equality holds if and only if θ = 0 or
π. This means that the two vectors either
point in the same direction or opposite directions. Hence one is a multiple of the
other.
3 −0.197 39 = cos θ, Thus θ = 1. 769 5 radians.
4 −0.444 44 = cos θ, θ = 2. 031 3 radians.
5
u·v
u·u u
(
=
7
= −5
14 (1, 2, 3)
5
− 14
− 57
− 15
14
)
12 a × b × c is meaningless.
13 (u × v) ×w = (u · w) v− (v · w) u,
u× (v × w) = (w · u) v− (v · u) w
14 u· (z × w) v − v· (z × w) u
= [u, z, w] v− [v, z, w] u
15 [v, w, z] [u, v, w]
16 0
18 u · u = c, Therefore, u′ · u + u · u′ = 0 so
u′ · u = 0.
C.5
= (1,2,−2,1)·(1,2,3,0)
(1, 2, 3, 0)
1+4+9
(
)
1
3
= − 14
− 17 − 14
0
u·v
u·u u
8 It makes no sense.
9 projD (F) ≡
9 It means that if you place them so that they
all have their tails at the same point, the
three will lie in the same plane.
F·D D
|D| |D|
Exercises 60
[
1 x=
10
13 , y
=
1
13
]
3 [x = 1, y = 0]
[
]
5 x = 35 , y = 15
6 No solution exists.
D
= (|F| cos θ) |D|
= (|F| cos θ) u
( 20 )
π 100 = 3758. 8
11 40 cos 180
(
)
13 20 cos π4 300 = 4242. 6
8 It appears that the solution exists but is
not unique.
9 It appears that there is a unique solution.
15 (−4, 3, −4) · (0, 1, 0) × 10 = 30
(
)
√
17 (2, 3, −4) · 0, √12 , √12 20 = −10 2
11 There might be a solution. If so, there are
infinitely many.
19 (1, 2, 3, 4) · (2, 0, 1, 3) = 17
13 These can have a solution. For example,
2
2
21 |a + b| + |a − b| =
2
2
2
2
|a| + |b| + 2a · b + |a| + |b| − 2a · b
2
= 2 |a| + 2 |b|
2
12 No. Consider x+y+z = 2 and x+y+z = 1.
x + y = 1, 2x + 2y = 2, 3x + 3y = 3 even
has an infinite set of solutions.
14 h = 4
15 Any h will work.
C.4
Exercises 41
1 If a ̸= 0, then the condition says that
|a × u| = |a| sin θ = 0
for all angles θ. Hence a = 0 after all.
√
3 12 374
√
5 8 3
16 Any h will work.
17 If h ̸= 2 there will be a unique solution for
any k. If h = 2 and k ̸= 4, there are no
solutions. If h = 2 and k = 4, then there
are infinitely many solutions.
19 There is no solution.
[
]
20 w = 32 y − 1, x = 23 − 21 y, z = 13
C.6. EXERCISES 80
22 z = t, y =
x=
3
4
+t
413
(1)
4
− 12 t where t ∈ R.
1
2
23 z = t, y = 4t, x = 2 − 4t.
24 x5 = t, x4 = 1 − 6t, x3 = −1 + 7t,
x2 = 4t, x1 = 3 − 9t
25 x5 = t, x3 = s. Then the other variables are
given by x4 = − 12 − 32 t,
x2 =
3
2
− t 12 , x1 =
5
2
+ 12 t − 2s.
26 [x = 1 − 2t, z = 1, y = t]
27 [x = 2 − 4t, y = −8t, z = t]
25 [x = −1, y = 2, z = −1]
29 [x = 2, y = 4, z = 5]
30 [x = 1, y = 2, z = −5]
31 [x = −1, y = −5, z = 4]
32 [x = 2t + 1, y = 4t, z = t]
33 [x = 1, y = 5, z = 3]
34 [x = 4, y = −4, z = −2]
35 These are not legitimate row operations.
They do not preserve the solution set of the
system.
36 {g = 60, I = 90, b = 200, s = 50}
37 [w = 15, x = 15, y = 20, z = 10] .
C.6
(
1.
(
Exercises 80
−3
−6
−6 −9
−3 −21
)
,
)
−5 3
,
5 −4
(
)
−3 3 4
Not possible,
, Not possi6 −1 7
ble, Not possible.
(
)
−3 −9 −3
3
,
−6 −6 3
(
)
5
−18 5
,
−11
4
4


11 2


 13 6  ,
−4 2
8
−11

7


Not possible,  9  ,
−2
(
)
−7 1 5 ,
(
)
2
Not possible,
,
−5
(
)
1 3
, 10
3 9


7
(


4  5  , Not possible, −14 13
−5
Not possible,
(
)
−1 −3
11,
4
12
(
)
x
y
7
.
−x −y


0 −1 −2


8  0 −1 −2  , 1
0 1
2

,
)
2
,
9 [k = 4]
10 There is no possible choice of k which will
make these matrices commute.
11 To get −A, just replace every entry of A
with its additive inverse. The 0 matrix is
the one which has all zeros in it.
12 A =
A+AT
2
+
A−AT
2
.
13 If A = −AT , then aii = −aii and so each
aii = 0.
i
14 Ω × u : ω 1
u1
j
ω2
u2
k
ω3
u3
= iω 2 u3 − iω 3 u2 − jω 1 u3 + jω 3 u1 + kω 1 u2 −
kω 2 u1 .
In terms of matrices, this is


ω 2 u3 − ω 3 u2


 −ω 1 u3 + ω 3 u1  . The matrix is of the
ω 1 u2 − ω 2 u1
form
414
APPENDIX C. ANSWERS TO SELECTED EXERCISES

0
−ω 3 ω 2


0
−ω 1  for suitable ω i since
 ω3
−ω 2 ω 1
0
it is skew symmetric. Now you note that
multiplication on the left by this matrix
gives the same thing as the cross product.

15 −A = −A + (A + B)
= (−A + A) + B = 0 + B = B
16 0′ = 0′ + 0 = 0.
17 0A = (0 + 0) A = 0A+0A. Now add − (0A)
to both sides. Then 0 = 0A.
18 A + (−1) A = (1 + (−1)) A = 0A = 0.
Therefore, from the uniqueness of the additive inverse, it follows that −A = (−1) A.
(
)
T
19 (αA + βB)
ij
= (αA + βB)ji = αAji + βBji
(
)
( )
T
= α AT ij + (βB)ij = αAT + βB T ij
∑
20 (Im A)ij ≡ j δ ik Akj = Aij .
24 Explain why if AB = AC and A−1 exists,
then B = C.
Because you can multiply on the left by
A−1 .
(
)
)−1 (
3
2 1
− 17
7
28
=
2
1
−1 3
7
7
(
29
(
30
(
0 1
5 3
2 1
3 0
2 1
4 2
)−1
(
=
)−1
(
=
)−1
− 35
1
1
5



36 

1
−1
2
1
3
2
−1 0
−2 − 34



37 A = 



38 Write 

)

0
1
4
9
4

2 0
2 0 


3 0 
0 3
x1 + 3x2 + 2x3
2x3 + x1
6x3
x4 + 3x2 + x1



A

where A is

1
 1

A=
 0
1
x1
x2
x3
x4



 in the form






an appropriate matrix.

3 2 0
0 2 0 


0 6 0 
3
does not exist. The row re
1 1
(
)
 1 1
1

1 2
duced echelon form of this matrix is
. 39 A = 
 −1 0
0 0
1 0
(
)
d
b
− ad−bc
ad−bc

32
, assuming ad−bc ̸=
c
a
1 2 0
− ad−bc
ad−bc
 1 1 2
0 of course.

42 


 2 1 −3
−2 4 −5


1 2 1
33  0
1 −2 
1 −2 3
31
1
2
− 52



1 
−1
0
0
3
1
1
0
1

1
2
− 12
)
0
1
0
3
1 − 23

−2 0
3


1
34  0
− 32 
3
1 0 −1


1 2 3


35  2 1 4 , row echelon form:
4 5 10


1 0 53


 0 1 23  . There is no inverse.
0 0 0

0
1

1 0
2 0 


1 0 
0 3
2
0
2
2



 . Thus the solution is

C.7. EXERCISES 99






x
y
z
w
1
 1


 2
1

415

5 −4


=

2
1
1
2
0 2
2 0
−3 2
1 2
6 −6
7 −32





a + 2b + 2d

a + b + 2c

=
 2a + b − 3c + 2d
a + 2b + c + 2d

a
b 


c 
d





43 Multiply on the left of both sides by A−1 .
44 Multiply on both sides on the left by A−1 .
Thus 0 = A−1 0 =
(
)
A−1 (Ax) = A−1 A x = Ix = x
45 A−1 = A−1 I = A−1 (AB)
(
)
= A−1 A B = IB = B.
(
)T
46 You just need to show that A−1
acts
like the inverse of AT because from uniqueness in the above problem, this will imply
it is the inverse. But from properties of the
transpose,
(
)T
AT A−1
( −1 )T T
A
A
=
=
8 63
9 211
11 It does not change the determinant. This
was just taking the transpose.
13 The determinant is unchanged. It was just
the first row added to the second.
15 In this case the two columns were switched
so the determinant of the second is −1 times
the determinant of the first.
17 det (aA) = det (aIA) = det (aI) det (A) =
an det (A) . The matrix which has a down
the main diagonal has determinant equal to
an .
19 This is not true at all. Consider
(
)
1 0
A =
,
0 1
(
)
−1 0
B =
.
0 −1
20 It must be 0 because
( −1 )T
A A = IT = I
( −1 )T
AA
= IT = I
(
)T ( )−1
Hence A−1 = AT
and this last matrix exists.
(
)
47 (AB) B −1 A−1 = A BB −1 A−1 = AA−1 =
I
(
)
B −1 A−1 (AB) = B −1 A−1 A B
= B −1 IB = B −1 B = I
∑
51 Ax · y = k (Ax)k yk
∑ ∑
= k i Aki xi yk =
(
)
∑ ∑ T
T
i
k Aik xi yk = x,A y
0 =
=
2 6
Exercises 99
23 If A = S −1 BS, then SAS −1 = B and so
if A ∼ B, then B ∼ A. It is obvious that
A ∼ A because you can let S = I. Say
A ∼ B and B ∼ C. Then A = P −1 BP and
B = Q−1 CQ. Therefore,
A = P −1 Q−1 CQP = (QP )
4 6
−1
C (QP )
and so A ∼ C.
25 Say M = S −1 N S. Then
=
=
=
3 2
k
(det (A)) .
21 det (A) = 1, or −1.
=
C.7
( )
det (0) = det Ak
=
=
det (λI − M )
(
)
det λI − S −1 N S
(
)
det λS −1 S − S −1 N S
(
)
det S −1 (λI − N ) S
(
)
det S −1 det (λI − N ) det (S)
(
)
det (λI − N ) det S −1 det (S)
det (λI − N )
416
APPENDIX C. ANSWERS TO SELECTED EXERCISES

1

31  − 23
2
3


33 
− 12
3
2
1
2
0
−3
1
3
− 13
5
3
− 23
3
2
− 92
− 12

C.8



0

1 
0
38 Show that if det (A) ̸= 0 for A an n × n matrix, it follows that if Ax = 0, then x = 0.
If det (A) ̸= 0, then A−1 exists and so you
could multiply on both sides on the left by
A−1 and obtain that x = 0.
39 Suppose A, B are n × n matrices and that
AB = I. Show that then BA = I. Hint:
You might do something like this: First explain why det (A) , det (B) are both nonzero.
Then (AB) A = A and then show
BA (BA − I) = 0.
From this use what is given to conclude
A (BA − I) = 0. Then use Problem 38.
You have 1 = det (A) det (B). Hence both
A and B have inverses. Letting x be given,
A (BA − I) x
=
(AB) Ax − Ax
= Ax − Ax = 0
and so it follows from the above problem
that
(BA − I) x = 0.
Since x is arbitrary, it follows that BA = I.


e−t
0
0


40  0
e−t (cos t + sin t) − (sin t) e−t 
0 −e−t (cos t − sin t) (cos t) e−t
Exercises 147
4 a. is not, b. is, and so is c.



1 0
1 0 0 3
 0 1


5  0 1 0 0 
 0 0
0 0 1 −2
0 0


1 0 0 0


 0 1 12 0 
0 0 0 1
0
0
1
0





7 It is because you cannot have more than
min (m, n) nonzero rows in the row reduced
echelon form. Recall that the number of
pivot columns is the same as the number of
nonzero rows from the description of this
row reduced echelon form.
{(
) ( )}
1
1
9
,
is a basis.
2
3

  

1 
 1


  
10  2  ,  3  is a basis.




0
1
12 Yes. This is a subspace. It is closed with
respect to vector addition and scalar multiplication.
14 Yes, this is a subspace.
16 This is a subspace.
18 Not a subspace.
∑k
20
i=1 0xk = 0.
22 If AB = I, then B must be one to one.
Otherwise there exists x ̸= 0 such that
Bx = 0.
41 The columns of the matrix are

 

1 −t
0
2e
 

 1
 2 cos t + 21 sin t  ,  − sin t 
1
1
cos t
2 sin t − 2 cos t



1 −t
2e
1
1
2 sin t − 2 cos t
1
− 2 cos t − 12 sin t



But then you would have
x = Ix = ABx = A0 = 0
In particular, the columns of B are linearly
independent. Therefore, B is also onto.
Also,
(BA − I) Bx = B (AB) x−Bx = 0
Since B is onto, it follows that BA−I maps
every vector to 0 and so this matrix is 0.
Thus BA = I.
C.8. EXERCISES 147
24 These vectors are not linearly independent.
They are linearly dependent. In fact −1
times the first added to 2 times the second
is the third.
26 These cannot be linearly independent because there are 4 of them. You can have at
most three. However, it might be that they
span R3 .


1 4 3 2


 2 3 1 4 , row echelon form:
3 3 0 6


1 0 −1 2


 0 1 1 0  . The dimension of the
0 0 0 0
span of these vectors is 2 so they do not
span R3 .
28 These vectors are not linearly independent
and so they are not a basis. The remaining
question is whether they span.


1 4 1 2


 0 3 2 4 , row echelon form:
3 3 0 0


1 0 0 0


 0 1 0 0  . The dimension of the span
0 0 1 2
of these vectors is 3 and so they do span R3 .
32 Yes it is. It is the span of the vectors

  
2
3

  
 −1  ,  1  .
1
1
Since these two vectors are a linearly independent set, the given subspace has dimension 2.
34 It is a subspace and it equals the span of
the vectors which form the columns of the
following matrix.


0 2 1 0
 0 1 3 0 



, row echelon form:
 1 1 0 1 

0 0
1 0
 0 1


 0 0
0 0
mension
1
0

0 1
0 0 

 . It follows that the di1 0 
0 0
of this subspace equals 3. A basis
417
  
0
2



 0   1
  
is   , 

 1   1



0
0
 
 
 
,
 
1
3
0
1











38 This is obvious. If x, y ∈ V ∩ W, then for
scalars α, β, the linear combination αx+βy
must be in both V and W since they are
both subspaces.
40 Let {x1 , · · · , xk } be a basis for V ∩W. Then
there is a basis for V and W which are respectively
{x1 , · · · , xk , yk+1 , · · · , yp } ,
{x1 , · · · , xk , zk+1 , · · · , zq }
It follows that you must have
k+p−k+q−k ≤n
and so you must have
p+q−n≤k
42 No. There must then be infinitely many solutions. If the system is Ax = b, then there
are infinitely many solutions to Ax = 0 and
so the solutions to Ax = b are a particular solution to Ax = b added to the solutions to Ax = 0 of which there are infinitely
many.
44 Yes. It has a unique solution.
46 a. Infinite solution set. b. This surely can’t
happen. If you add in another column, the
rank does not get smaller. c. You can’t
have the rank equal 4 if you only have two
columns. d. In this case, there is no solution to the system of equations represented
by the augmented matrix. e. In this case,
there is a unique solution since the columns
of A are independent.The columns are independent. Therefore, A is one to one.
50 Suppose ABx = 0. Then Bx ∈ ker (A) ∩
∑k
B (Fp ) and so Bx = i=1 Bzi showing that
x−
k
∑
zi ∈ ker (B) .
i=1
Consider B (Fp ) ∩ ker (A) and let a basis be
{w1 , · · · , wk } .
418
APPENDIX C. ANSWERS TO SELECTED EXERCISES
(
)
Suppose AT Ax = 0. Then AT Ax, x =
(Ax,Ax) and so Ax = 0. Therefore,
( T
)
A b, x = (b,Ax) = (b, 0) = 0
Then each wi is of the form Bzi = wi .
Therefore, {z1 , · · · , zk } is linearly independent and ABzi = 0. Now let {u1 , · · · , ur }
be a basis for ker (B) . If ABx = 0, then
Bx ∈ ker (A) ∩ B (Fp ) and so
Bx =
k
∑
(
((
)T ))⊥
It follows that AT b ∈ ker AT A
and so there exists a solution x to the equation
AT Ax = AT b
ci Bzi
i=1
which implies
by the Fredholm alternative.
x−
k
∑
56
ci zi ∈ ker (B)
i=1
|b−Ay|
and so it is of the form
x−
k
∑
i=1
ci zi =
2
2
= |b−Ax+Ax−Ay|
r
∑
dj uj
j=1
2
2
2
2
2
= |b−Ax| + |Ax − Ay|
(
)
+2 AT b−AT Ax, (x − y)
It follows that if ABx = 0 so that
= |b−Ax| + |Ax − Ay|
x ∈ ker (AB) ,
then
x ∈ span (z1 , · · · , zk , u1 , · · · , ur ) .
Therefore,
dim (ker (AB))
≤ k + r = dim (B (Fp ) ∩ ker (A))
+ dim (ker (B))
≤ dim (ker (A)) + dim (ker (B))
−1
52 If det (A − λI) = 0 then (A − λI) does
not exist and so the columns are not independent which means that for some x ̸= 0,
(A − λI) x = 0.
2
= |b−Ax| + |Ax − Ay|
+2 (b−Ax,A (x − y))
and so, Ax is closest to b out of all vectors
Ay.
2
57 The dimension of Fn is n2 . Therefore, there
exist scalars ck such that
n
∑
2
ck Ak = 0
k=0
Let p (λ) be the monic polynomial having
smallest degree such that p (A) = 0. If q (A) =
0 then from the Euclidean algorithm,
q (λ) = p (λ) l (λ) + r (λ)
where the degree of r (λ) is less than the
degree of p (λ) or else r (λ) equals 0. However, if it is not zero, you could plug in A
and obtain
54 Since A1 is not one to one, it follows there
exists x ̸= 0 such that A1 x = 0. Hence Ax = 0
0 = q (A) = 0 + r (A)
although x ̸= 0 so it follows that A is not
one to one. From another point of view, if A
⊥
and this would contradict the definition of
were one to one, then ker (A) = Rn and so
T
p (λ) as being the polynomial having smallby the Fredholm alternative, A would be
n
T
est degree which sends A to 0. Hence q (λ) =
onto R . However, A has only m columns
p
(λ) l (λ) .
so this cannot take place.
(
)T
55 That AT A = AT A follows from the propC.9 Exercises 164
erties of the transpose. Therefore,
(
√ )
1
1
(
((
))⊥ (
−
)
(
))
T
⊥
2
2 3
√
3
ker AT A
= ker AT A
1
1
2 3
2
C.9. EXERCISES 164
(
4
(
5
(
6
(
7
(
8
(
9
419
√ )
− 12 2
√
1
2 2
√ )
1
2 3
√
1
2 √2
1
2 2
1
2√
− 12 3
Now, why does this matrix preserve distance? For short, call it Q and note that
QT Q = I. Then
(
)
2
2
|x| = QT Qx, x = (Qx,Qx) = |Qx|
1
2
√ )
− 12 − 12 3
√
1
− 12
2 3
√ √
1
4 √2√3
1
4 2 3
√
+ 14 2
√
− 14 2
√
1
− 12
−2 3
√
1
− 12 3
2
and so Q preserves distances.
√ √ )
√
1
− 14 2 3
4 √2√
√
1
1
4 2 3+ 4 2
Q (x − y)
− 12
√
1
2 3
)
√ √
√
√ √
√ )
1
1
1
1
2
3
−
2
−
2
3
−
4√ √
4√
4√ √
4√ 2
16
1
1
1
1
4 2 3+ 4 2
4 2 3− 4 2
 √

1
− 12
0
2 3
√
 1

1
17  2
0 
2 3
0
0
−1


1 5
3

1 
19 35
 5 25 15 
3 15 9
21
2
|x − y|
(x − y, x − y)
= y−x
Q (x + y) =
(
x+y−2
·
x−y
|x − y|
2
·
T
(x − y) (x + y)
= x+y−2
x−y (
2
|x − y|
2
|x| − |y|
2
)
= x+y
and so
Qx − Qy= y − x
Qx+Qy= x + y
Hence, adding these yields Qx = y and then
subtracting them gives Qy = x.
Tu (av+bw)
(av+bw · u)
= av+bw−
u
2
|u|
(w · u)
(v · u)
= av − a
2 u + bw−b
2 u
|u|
|u|
= aTu (v) + bTu (w)
(
25
x−y
= x−y−2
)
√ ) ( 1√
− 12 3
−2 3
1
−2
− 12
1
2√
− 12 3
27 From the above problem this preserves distances and QT = Q. Now do it to x.
a2 − b2
2ab
2ab
2
b − a2
)
I − 2uuT


32 

34 
)T (
I − 2uuT
(
)
(
)
= I − 2uuT I − 2uuT
)
=1
z}|{
= I − 2uu − 2uu + 4uuT uuT
= I
T
Ta (u + v) ̸= Ta u + Ta v.

26
(
28 Linear transformations take 0 to 0. Also
T


36 


38 
 

−3t̂3
0
 

−t̂3  +  −1  , t̂3 ∈ R
t̂3
0
 

3t̂
−3
 

2t̂  +  −1  , t̂ ∈ R
t̂
0
 

−4t̂
0
 

−2t̂  +  −1  , t̂ ∈ R.
t̂
0
 

−t̂
−1
 

2t̂  +  −1 
t̂
0
420
APPENDIX C. ANSWERS TO SELECTED EXERCISES



39 




40 




42 

0
−t̂
−t̂
t̂
0
−t̂
−t̂
t̂
−t̂
t̂
t̂
0

1 −2 −5 0


3  −2 5
11 3  =
3 −6 −15 1



1 0 0
1 −2 −5 0



1 3 
 −2 1 0   0 1
3 0 1
0 0
0 1


1 −3 −4 −3


5  −3 10 10 10  =
1 −6 2 −5



1
0 0
1 −3 −4 −3



 −3 1 0   0 1 −2 1 
1 −3 1
0 0
0
1




 , t̂ ∈ R



 
 
+
 


 
 
+
 
2
−1
−1
0
−8
5
0
5











 
−1

 −1

 1   1
 

44 A basis is 
,
 2   0



1
0
49














−2
−1



 −1/2 
 −1/2



 + t 0
s
1






 0 
 1
0
0

 

1
4

 

 −1   7/2 

 

 

+w 
 0 + 0 

 

 0   0 
1
0



7 









52 If x, y ∈ ker (A) then
A (ax+by) = aAx + bAy = a0 + b0 = 0
and so ker (A) is closed under linear combinations. Hence it is a subspace.
C.10

Exercises 181
1 2

1  2 1
1 2

1 0

 2 1
1 0

0

3 =
3

0
1

0  0
1
0

2 0

−3 3 
0 3








9 






3 −2 1
9 −8 6
−6 2
2
3
2 −7



=


1
0
0 0
3
 0
3
1
0 0 


−2 1
1 0  0
1 −2 −2 1
0
−2
−2
0
0

−1 −3 −1
1
3
0 

=
3
9
0 
4
12 16

1 0 0 0
−1


−1 1 0 0   0

−3 0 1 0   0
−4 0 −4 1
0

−3 −1
0 −1 


0 −3 
0
0
1
3
1
0



.

11 An LU factorization of the coefficient matrix is


1 2 1


 0 1 3 
2 3 0



1 0 0
1 2 1



=  0 1 0  0 1 3 
2 −1 1
0 0 1
C.11. EXERCISES 205
First solve
421



1 0 0
u



 0 1 0  v 
2 −1 1
w
 
1
 
=  2 
6

which yields u =
solve

1

 0
0

1

=  2
6
This
(
0
14
0
(
0
0

1

15  1
2

1

 0
0

1

 0
0

1
 1

17 
 2
3

1
 0


 0
0

1
 0


 0
0
1, v = 2, w = 6. Next


x
2 1


1 3  y 
z
0 1

2
1
0
0
0
1
0
0
1
0
0
1
0
0






2
1
−4 −2 


0 −1 
0
0
√ √
√ 
− 16 2
√ 
− 16 2  ·
√
2
3 2

√
6
11
11
√ √ 
2
2 11 
11 √
2
13
2 11
66 √
√
5
− 66
2 11
√ √
1
33 2 11
√
4
− 11
11
√ √
6
11 2 11
0
22 You would have QRx = b and so then you
would have Rx = QT b. Now R is upper triangular and so the solution of this problem
is fairly simple.
Exercises 205
1 The minimum is −11/2 and it occurs when
x1 = x3 = x6 = 0 and x2 = 7/2, x4 =
13/2, x6 = −11/2.
The maximum is 7 and it occurs when x1 =
7, x2 = 0, x3 = 0, x4 = 3, x5 = 5, x6 = 0.
= −16, x = 27.
)(
)
0
0 1
1
0 0
)(
)
0
0 1
1
0 1
0
1
0
 √
11

 0
0
C.11


yields z = 6, y
) (
1
1
=
1
1
) (
1
1
=
1
0

2 1

2 2 =
1 1

0 0
1

0 1  2
1 0
1

2
1

−3 −1 
0
1

2 1
2 2 

=
4 1 

20 
√
1
11 √11
3
11 √11
1
11 11
2 Maximize and minimize the following if possible. All variables are nonnegative.
(a) The maximum is 7 when x1 = 7 and
x1 , x3 = 0.
The minimum is −7 and it happens
when x1 = 0, x2 = 7/2, x3 = 0.
(b) The minimum is −21 and it occurs
when x1 = x2 = 0, x3 = 7.
The maximum is 7 and it occurs when
x1 = 7, x2 = 0, x3 = 0.

0

0 ·
1
(c) The minimum is 0 and it occurs when
x1 = x2 = 0, x3 = 1.
The maximum is 14 and it happens
when x1 = 7, x2 = x3 = 0.
(d) The maximum is 7 and it happens when
x2 = 7/2, x3 = x1 = 0.
The minimum is 0 when x1 = x2 =
0, x3 = 1.
4 Find solutions if possible.
1
3
2
1
0
1
0
0
0
0
1
−1
0
0
0
1



·

(a) There is no solution to these inequalities with x1 , x2 ≥ 0.
(b) A solution is x1 = 8/5, x2 = x3 = 0.
(c) No solution to these inequalities for
which all the variables are nonnegative.
(d) There is a solution when x2 = 2, x3 =
0, x1 = 0.
(e) There is no solution.
422
APPENDIX C. ANSWERS TO SELECTED EXERCISES
C.12
Exercises 234
4 If it did have λ ∈ R as an eigenvalue, then
there would exist a vector x such that Ax =
λx for λ a real number. Therefore, Ax and
x would need to be parallel. However, this
doesn’t happen because A rotates the vectors.
6 Am x = λm x for any integer. In the case of
−1,
A−1 λx = AA−1 x = x
so A−1 x = λ−1 x. Thus the eigenvalues of
A−1 are just λ−1 where λ is an eigenvalue
of A.
8 Let x be the eigenvector. Then Am x =
λm x,Am x = Ax = λx and so
λm = λ
Hence if λ ̸= 0, then
λm−1 = 1
and so |λ| = 1.
 



 3 


10 eigenvectors:
↔ 1,
1


 


1
This is a defective matrix.

 

5
 −2

 
12 eigenvectors:  1  ,  0


0
1
 

 2 

 
 1  ↔2




1
2
1
1





↔ 2.




 ↔ −1,


This matrix is not defective because, even
though λ = 1 is a repeated eigenvalue, it
has a 2 dimensional eigenspace.
  



0
 1

   1 
14 eigenvectors:  0  ,  − 2  ↔ 3,




0
1



 −1 



 1  ↔6




1
This matrix is not defective.

 

1
5


 −3

6

 

16 eigenvectors:  1  ,  0  ↔ −1




0
1
This matrix is defective. In this case, there
is only one eigenvalue, −1 of multiplicity 3
but the dimension of the eigenspace is only
2.


3


 4 


19 eigenvectors:  14  ↔ 0 This one is




1
defective.


3

 4 



20 eigenvectors:  14  ↔ 1




1
This is defective.
22 eigenvectors:







1



 −i 




 −1  ↔ 4,  −i 








1
1



 i 



↔ 2 − 2i,  i  ↔ 2 + 2i




1
24 eigenvectors:




1




 −1  ↔ 4,




1



 −i 



 −i  ↔ 2 − 2i,




1



 i 



 i  ↔ 2 + 2i This matrix is not




1
defective.
26 eigenvectors:




1




 −1  ↔ −6,




1



 −i 



 −i  ↔ 2 − 6i,




1



 i 



 i  ↔ 2 + 6i




1
This is not defective.
C.12. EXERCISES 234
423
28 The characteristic polynomial is of degree
three and it has real coefficients. Therefore,
there is a real root and two distinct complex
roots. It follows that A cannot be defective
because it has three distinct eigenvalues.
{(
)}
−i
29 eigenvectors:
↔ −i,
1
{( )}
i
↔i
1
 


 0 
 
32 eigenvectors:  0  ↔ −1,




1
   

0 

 1
   
 0 , 1  ↔ 1




0
0
 

 0 

 
34 eigenvectors:  0  ↔ 1,




1
 

 1 

 
 1  ↔ 2,




0



 −1 



 1  ↔3




0
36 In terms of percentages in the various locations,


21. 429


 21. 429 
57. 143
38 Obviously A cannot be onto because the
range of A has dimension 1 and the dimension of this space should be 3 if the matrix
is onto. Therefore, A cannot be invertible.
Its row reduced echelon form cannot be I
since if it were, A would be onto. Aw = w
so it has an eigenvalue equal to 1. Now
suppose Ax = λx. Thus, from the Cauchy
Schwarz inequality,
|x| =
≥
and so |λ| ≤ 1.
|x| |w|
2
|w|
|w|
|(x, w)|
|w|
2
40 Since the vectors are linearly independent,
the matrix S has an inverse. Denoting this
inverse by


w1T
 . 
. 
S −1 = 
 . 
wnT
it follows by definition that
wiT xj = δ ij .
Therefore,
S −1 M S = S −1 (M x1 , · · · , M xn )


w1T
 . 
. 
=
 .  (λ1 x1 , · · · , λn xn )
wnT


λ1
0


..

=
.


0
λn
42 The diagonally dominant condition implies
that none of the Gerschgorin disks contain
0. Therefore, 0 is not an eigenvalue. Hence
A is one to one, hence invertible.
∗
44 First note that (AB) = B ∗ A∗ . Say M x =
λx, x ̸= 0. Then
λ |x|
∗
= λx∗ x = (λx) x
2
=
∗
(M x) x = x∗ M ∗ x
= x∗ M x = x∗ λx = λ |x|
2
Hence λ = λ.
47 Suppose A is skew symmetric. Then what
about iA?
∗
(iA) = −iA∗ = −iAT = iA
and so iA is self adjoint. Hence it has all
real eigenvalues. Therefore, the eigenvalues of A are all of the form iλ where λ
is real. Now what about the eigenvectors?
You need
Ax = iλx
where λ ̸= 0 is real and A is real. Then
A Re (x) = iλ Re (x)
|w| = |λ| |x|
The left has all real entries and the right has
all pure imaginary entries. Hence Re (x) =
0 and so x has all imaginary entries.
424
APPENDIX C. ANSWERS TO SELECTED EXERCISES
C.13
Exercises 271
9 Yes.
1 a. orthogonal and transformation, b. symmetric, c. skew symmetric.
2
4 ∥U x∥ = (U x, U x)
(
)
2
= U T U x, x = (Ix, x) = ∥x∥
Next suppose distance is preserved by U.
Then
(U (x + y) , U (x + y))
2
2
= ∥U x∥ + ∥U y∥ + 2 (U x, U y)
(
)
2
2
= ∥x∥ + ∥y∥ + 2 U T U x, y
But since U preserves distances, it is also
the case that
(U (x + y) , U (x + y))
2
Hence
2
∥x∥ + ∥y∥ + 2 (x, y)
=
(
)
(x, y) = U T U x, y
and so
((
)
)
U T U − I x, y = 0
Since y is arbitrary, it follows that U T U −
I = 0. Thus U is orthogonal.
(
5
)
x
y
z

·


a1
a4 /2 a5 /2
x



a2
a6 /2   y 
 a4 /2
a5 /2 a6 /2 a3
z
7 If A is symmetric, then A = U T DU for
some D a diagonal matrix in which all the
diagonal entries are non zero. Hence A−1 =
U −1 D−1 U −T . Now
(
)−1
U −1 U −T = U T U
= I
−1
  

 1 

 
11 eigenvectors:  0  ↔ c,




0




0




 −i  ↔ −ib,




1




 0 


 i  ↔ ib




1




0




12 eigenvectors:  −i  ↔ a − ib,




1



 0 



 i  ↔ a + ib,




1



 1 



 0  ↔c




0
13 eigenvectors:
  

1 


 
√1  1 
↔ 6,


 3

1
 


−1 




√1  1 
↔ 12,


 2

0
 


−1 




√1  −1 
↔ 18


 6

2
15
T
λxT x = (Ax) x
(CD)T =D T C T
=
=I
A is real
and so A−1 = QD−1 QT , where Q is orthogonal. Is this thing on the right symmetric? Take its transpose. This is QD−1 QT
which is the same thing, so it appears that
a symmetric matrix must have symmetric
inverse. Now consider raising it to a power.
k
T
k
A =U D U
and the right side is clearly symmetric.
=
λ is eigenvalue
=
xT Ax
xT Ax
xT λx = λxT x
and so λ = λ. This shows that all eigenvalues are real. It follows all the eigenvectors
are real. Why?
Because A is real. Ax = λx, Ax = λx,
so x+x is an eigenvector. Hence it can be
assumed all eigenvectors are real.
C.13. EXERCISES 271
425
Now let x, y, µ and λ be given as above.
λ (x · y)
= λx · y
= Ax · y = x · Ay
= x·µy = µ (x · y)
= µ (x · y)
and so
(λ − µ) x · y = 0.
Since λ ̸= µ, it follows x · y = 0.
17
λx · x
= Ax · x
= x·λx
A Hermitian
=
37 If A is given by the formula, then
rule for complex inner product
=
x·Ax
34 eigenvectors:

√ 


− 13 3




0

 ↔ 0,


 1√ √

3 2 3



√
1


3


3 √
 1

 − 2 2  ↔ 1,


√


1
6 6
 √ 
1


 3 √3 

 1
 2 2  ↔ 2. The columns are these



 1√
6 6
vectors.
AT = U T DT U = U T DU = A
λx · x
and so λ = λ. This shows that all eigenvalues are real. Now let x, y, µ and λ be given
as above.
Next suppose A = AT . Then by the theorems on symmetric matrices, there exists
an orthogonal matrix U such that
λ (x · y) = λx · y = Ax · y =
U AU T = D
x · Ay= x·µy
rule for complex inner product
=
for D diagonal. Hence
µ (x · y)
= µ (x · y)
and so
(λ − µ) x · y = 0.
Since λ ̸= µ, it follows x · y = 0.
(
)
1 3
19 Certainly not.
0 2
29 eigenvectors:

√ 


− 16 6

√ 

1
 − 6  ↔ 6,
 1 √6 √



3 2 3

√ 
1

 − 2√ 2 

 1

 2 2  ↔ 12,




0
 √ 
1

 3 √3 

 1

 3 3  ↔ 18.


 1√

3 3
The matrix U has these as its columns.
A = U T DU
39 There exists U unitary such that A = U ∗ T U
such that T is uppser triangular. Thus A
and T are similar. Hence they have the
same determinant. Therefore, det (A) =
det (T ) , but det (T ) equals the product of
the entries on the main diagonal which are
the eigenvalues of A.
(
)
43
2
3
1
3
44 y = −0.125x2 + 1. 425x + 0.925
46 Find an orthonormal basis for the spans of
the following sets of vectors.
(a) (3, −4, 0) , (7, −1, 0) , (1, 7, 1).

 
 

3/5
4/5
0

 
 

 −4/5  ,  3/5  ,  0 
0
0
1
(b) (3, 0, −4) , (11, 0, 2) , (1, 1, 7)

 
  
3
4
0
5
5

 
  
 0 , 0 , 1 
3
− 45
0
5
426
APPENDIX C. ANSWERS TO SELECTED EXERCISES
(c) (3, 0, −4) , (5, 0, 10) , (−7, 1, 1)

 
  
3
4
0
5
5

 
  
 0 , 0 , 1 
3
− 45
0
5
Then
trace (AA∗ )
( (
= trace U
(
47

√  



1
6 √6
1
3 √6
1
6 6
 
,
√ 
7
3
15 √


1
,  − 15
3 
√
− 13 3
√
3
10 √2
− 25 2
√
1
2 2





1
5
√
5
0
√
2


,
5
51 It satisfies the properties of an inner product. Note that
∑∑
Aik Bik
trace (AB ∗ ) =
=
i
k
k
i
∑∑
Aik Bik
= trace (BA∗ )
so
(A, B)F = (B, A)F
The product is obviously linear in the first
argument. If (A, A)F = 0, then
∑∑
i
k
Aik Aik =
∑
= trace U
= trace
5
√ √ 
3
5 14
− 35

 1√ √
 14 5 14 
√
√
3
70 5 14

σ 0
0 0
( (
(
49

V
2
|Aik | = 0
i,k
52 From the singular value decomposition,
(
)
σ
0
U ∗ AV =
,
0 0
(
)
σ 0
A = U
V∗
0 0
)
0
V ∗·
0
)
σ
0
)
σ2
0
U∗
σ2
0
0
0
0
0
)
)
=
)
U
∗
∑
σ 2j
j
∑ ∑
53 trace (AB) = ∑
i ∑
k Aik Bki ,
trace (BA) =
i
k Bik Aki . These give
the same thing. Now
(
)
trace (A) = trace S −1 BS
(
)
= trace BSS −1 = trace (B) .
C.14
Exercises 290

0.39


1  −0.09 
0.53


5. 319 1 × 10−2


2  7. 446 8 × 10−2 
0.712 77


0.143 94


5  0.939 39 
0.280 3


0.205 21


6 
0.117 26

−2
−2. 605 9 × 10

7 It indicates that they are no good for doing
it.
C.15
Exercises 315
1 The actual largest eigenvalue
is8 with cor
−1


responding eigenvector  1 
1
C.16. EXERCISES 321
4 The largest
is −16 and an eigen eigenvalue

1


vector is  −2  .
1
5 Eigenvalue near −1.35 : λ = −1. 341,


1.0


 −0.456 06 
−0.476 32
Eigenvalue near 1.5: λ = 1. 679 0,


0.867 41


 5. 586 9 
−3. 528 2
Eigenvalue

4. 405 2

 3. 213 6
6. 171 7
near 6.5: λ = 6. 662,



8 Eigenvalue near −1 : λ = −0.703 69,


3. 374 9


 −1. 265 3 
0.155 75
Eigenvalue near .25 : λ = 0.189 11,


−0.242 20


 −0.522 91 
1.0
Eigenvalue near 7.5 : λ = 7. 514 6,


0.346 92


1.0


0.606 92
10
λ−
1√
22
2
≤
3
3
12 From the bottom line, a lower bound is −10
From the second line, an upper bound is 12.
C.16
Exercises 321
427
2 First note that either m or −m is in S so S
is a nonempty set of positive integers. By
well ordering, there is a smallest element of
S, called p = x0 m + y0 n. Either p divides
m or it does not. If p does not divide m,
then by the above problem,
m = pq + r
where 0 < r < p. Thus
m = (x0 m + y0 n) q + r
and so, solving for r,
r = m (1 − x0 ) + (−y0 q) n ∈ S.
However, this is a contradiction because p
was the smallest element of S. Thus p|m.
Similarly p|n.Now suppose q divides both
m and n. Then m = qx and n = qy for
integers, x and y. Therefore,
p =
=
mx0 + ny0 = x0 qx + y0 qy
q (x0 x + y0 y)
showing q|p. Therefore, p = (m, n) .
3 Suppose r is the greatest common divisor
of p and m. Then if r ̸= 1, it must equal p
because it must divide p. Hence there exist
integers x, y such that
p = xp + ym
which requires that p must divide m which
is assumed not to happen. Hence r = 1 and
so the two numbers are relatively prime.
4 The only substantive issue is why Zp is a
field. Let [x] ∈ Zp where [x] ̸= [0]. Thus x
is not a multiple of p. Then from the above
problem, x and p are relatively prime. Hence
from another of the above problems, there
exist integers a, b such that
1 = ap + bx
1 The hint is a good suggestion. Pick the first
thing in S. By the Archimedean property,
S ̸= ∅. That is km > n for all k sufficiently
large. Call this first thing q + 1. Thus n −
(q + 1) m < 0 but n − qm ≥ 0. Then
Then
[1 − bx] = [ap] = 0
and it follows that
[b] [x] = [1]
n − qm < m
−1
and so
so [b] = [x]
0 ≤ r ≡ n − qm < m.
.
428
APPENDIX C. ANSWERS TO SELECTED EXERCISES
C.17
Exercises 340
1 No. (1, 0, 0, 0) ∈ M but 10 (1, 0, 0, 0) ∈
/ M.
3 If not, you could add in a vector not in their
span and obtain 6 vectors which are linearly
independent. This cannot occur thanks to
the exchange theorem.
10 For each x ∈ [a, b] , let fx (x) = 1 and
fx (y) = 0 if y ̸= x. Then these vectors
are obviously linearly independent.
12 A field also has multiplication. However,
you can consider the elements of the field
as vectors and then it satisfies all the vector
space axioms. When you multiply a number (vector) in R by a scalar in Q you get
something in R. All the axioms for a vector space are now obvious. For example, if
α ∈ Q and x, y ∈ R,
α (x + y) = αx + αy
from the distributive law on R.
13 Simply let f (i) be the ith component of a
vector x ∈ Fn . Thus a typical thing in Fn
is (f (1) , · · · , f (n)).
∑n
14 Say for some n, k=1 ck ek = 0, the zero
function. Then pick i,
0 =
n
∑
ck ek (i)
(a) These are linearly independent.
(b) These are also linearly independent.
21 This is obvious because when you add two
of these you get one and when you multiply
one of these by{a scalar,
√ } you get another
one. A basis is 1, 2 . By definition, the
span of these gives the collection of √
vectors.
Are they independent? Say a + b 2 = 0
where √
a, b are rational numbers. If a ̸= 0,
then b 2 = −a which can’t happen
√ since a
is rational. If b ̸= 0, then −a = b 2 which
again can’t happen because on the left is
a rational number and on the right is an
irrational. Hence both a, b = 0 and so this
is a basis.
29 Consider the claim about ln σ.
1eln(σ) + (−1) σe0 = 0
The equation shown does hold from the definition of ln σ. However, if ln σ were algebraic, then eln σ , e0 would be linearly dependent with field of scalars equal to the
algebraic numbers, contrary to the Lindemann Weierstrass theorem. The other instances are similar. In the case of cos σ, you
could use the identity
1 iσ 1 −iσ
e + e
− e0 cos σ = 0
2
2
contradicting independence of
eiσ , e−iσ , e0 .
k=1
= ci ei (i) = ci
Since i was arbitrary, this shows these vectors are linearly independent.
15 Say
n
∑
ck yk = 0
k=1
Then taking derivatives you have
n
∑
(j)
ck yk = 0, j = 0, 1, 2 · · · , n − 1
k=1
This must hold when each equation is evaluated at x where you can pick the x at
which the above determinant is nonzero.
Therefore, this is a system of n equations in
n variables, the ci and the coefficient matrix is invertible. Therefore, each ci = 0.
19 Which are linearly independent?
C.18
Exercises 357
1 I will show one of these. Verify that Examples 17.1.1 - 17.1.4 are each inner product
spaces.
First consider Example 17.1.1. All of the
axioms of the inner product are obvious except one, the one which says that if ⟨f, f ⟩ =
0 then f = 0. This one depends on continuity of the functions. Suppose then that it
is not true. In other words, ⟨f, f ⟩ = 0 and
yet f ̸= 0. Then for some x ∈ I, f (x) ̸= 0.
By continuity, there exists δ > 0 such that
if y ∈ I ∩ (x − δ, x + δ) ≡ Iδ , then
|f (y) − f (x)| < |f (x)| /2
It follows that for y ∈ Iδ ,
|f (y)| > |f (x)| − |f (x) /2| = |f (x)| /2.
C.18. EXERCISES 357
Hence
429
be positive. However, eventually a repeat
n
m
will take
( place.
) Thus a = a m < n, and
so am ak − 1 = 0 where k = n − m. Since
am ̸= 0, it follows that ak = 1 for a suitable
k. It follows that the sequence of powers
of a must include each of {1, 2, · · · , p − 1}
and all these would therefore, be positive.
However, 1 + (p − 1) = 0 contradicting the
assertion that Zp can be ordered. So what
would you mean by saying ⟨z, z⟩ ≥ 0? The
Cauchy Schwarz inequality would not even
apply.
∫
2
⟨f, f ⟩ ≥
|f (y)| p (x) dy ≥
Iδ
(
)
2
|f (x)| /2 (length of Iδ ) (min (p)) > 0,
a contradiction. Note that min p > 0 because p is a continuous function defined on
a closed and bounded interval and so it
achieves its minimum by the extreme value
theorem of calculus.
2
7 Let δ = r − |z − x| . Then if y ∈ B (z, δ) ,
∫
|y − x| ≤
f (x) g (x)p (x) dx
I
(∫
)1/2
2
|f (x)| p (x) dx
≤
(∫
I
·
and so B (z, δ) ⊆ B (x,r).
)1/2
8
2
|g (x)| p (x) dx
I
n
∑
≤
n
∑
)1/2 (
|f (xk )|
2
uk wk ≤
where u =
∞
∑
k=1
∑
k
(
ak bk ≤
n
∑
)1/2
|g (xk )|
√
2
1
2
√ √
2 3x
3
4
√ √ 2 1√ √ )
2 5x − 4 2 5
n
∑
)1/2 (
|uk |
2
k=1
′
k=1
∑
)1/2 (
|ak |
)1/2
|wk |
2
k=1
uk vk and w =
∞
∑
n
∑
2
k
∞
∑
wk vk .
)1/2
|bk |
z (p (x) y ′ ) + (λq (x) + r (x)) yz
′
y (p (x) z ′ ) + (µq (x) + r (x)) zy
= 0
= 0
Subtract.
2
k=0
(
k=1
1
2
′
f (xk ) g (xk )
k=0
n
∑
(
9 Let y go with λ and z go with µ.
k=0
(
|y − z| + |z − x| < δ + |z − x|
= r − |z − x| + |z − x| = r
2
k=1
5 It might be the case that ⟨z, z⟩ = 0 and yet
z ̸= 0. Just let z = (z1 , · · · , zn ) where exactly p of the zi equal 1 but the remaining
are equal to 0. Then ⟨z, z⟩ would reduce
to 0 in the integers mod p. Another problem is the failure to have an order on Zp .
Consider first Z2 . Is 1 positive or negative?
If it is positive, then 1 + 1 would need to
be positive. But 1 + 1 = 0 in this case.
If 1 is negative, then −1 is positive, but
−1 is equal to 1. Thus 1 would be both
positive and negative. You can consider
the general case where p > 2 also. Simply take a ̸= 1. If a is positive, then consider a, a2 , a3 · · · . These would all have to
′
z (p (x) y ′ ) −y (p (x) z ′ ) +(λ − µ) q (x) yz = 0
Now integrate from a to b. First note that
′
′
z (p (x) y ′ ) − y (p (x) z ′ )
d
(p (x) y ′ z − p (x) z ′ y)
=
dx
and so what you get is
p (b) y ′ (b) z (b) − p (b) z ′ (b) y (b)
− (p (a) y ′ (a) z (a) − p (a) z ′ (a) y (a))
∫ b
+ (λ − µ)
q (x) y (x) z (x) dx = 0
a
Look at the stuff on the top line. From the
assumptions on the boundary conditions,
C1 y (a) + C2 y ′ (a) = 0
C1 z (a) + C2 z ′ (a) = 0
and so
y (a) z ′ (a) − y ′ (a) z (a) = 0
Similarly,
y (b) z ′ (b) − y ′ (b) z (b) = 0
Hence, that stuff on the top line equals zero
and so the orthogonality condition holds.
430
11
12
APPENDIX C. ANSWERS TO SELECTED EXERCISES
∑5
k=1
1
2π
−
2(−1)k+1
k
sin (kx)
15 |
∑2
4
k=0 (2k+1)2 π
cos ((2k + 1) x)
∑n
i=1
xi yi i| ≤
(∑n
i=1
)1/2 (∑n
)
2 1/2
x2i i
i=1 yi i
16
ei(t/2) Dn (t)
=
n
1 ∑ i(k+(1/2))t
e
2π
=
n
1 ∑ i(k−(1/2))t
e
2π
k=−n
ei(−t/2) Dn (t)
k=−n
=
13
∑5
4
k=1 k2
k
(−1) cos (kx) +
π2
3
1
2π
n−1
∑
ei(k+(1/2))t
k=−(n+1)
(
)
Dn (t) ei(t/2) − e−i(t/2)
)
1 ( i(n+(1/2))t
=
e
− e−i(n+(1/2))t
2π
((
) )
1
1
Dn (t) 2i sin (t/2) =
2i sin
n+
t
2π
2
( (
))
1 sin t n + 12
( )
Dn (t) =
2π
sin 12 t
You know that t → Dn (t) is periodic of
period 2π. Therefore, if f (y) = 1,
∫ π
∫ π
Sn f (x) =
Dn (x − y) dy =
Dn (t) dt
−π
−π
However, it follows directly from computation that Sn f (x) = 1. Just take the integral
of the sum which defines Dn .
17 From Lemma 17.3.1 and Theorem 17.3.2
⟨
⟩
n
∑
y−
⟨y, uk ⟩ uk , w = 0
k=1
C.18. EXERCISES 357
431
n
for all w ∈ span ({ui }i=1 ) . Therefore,
2
|y| = y−
n
∑
⟨y, uk ⟩ uk +
n
∑
without loss of generality, assume that f
has real values. Then the above limit reduces to having both the real and imaginary parts converge to 0. This implies the
thing which was desired. Note also that if
α ∈ [−1, 1] , then
∫ π
lim
f (t) sin ((n + α) t) dt = lim
2
⟨y, uk ⟩ uk
k=1
k=1
Now if ⟨u, v⟩ = 0, then you can see right
away from the definition that
2
2
|u + v| = |u| + |v|
2
n→∞
∫
Applying this to
u = y−
n
∑
n
∑
π
f (t) [sin (nt) cos α + cos (nt) sin α] dt = 0
−π
⟨y, uk ⟩ uk ,
19 From the definition of Dn ,
∫ π
Sn f (x) =
f (x − y) Dn (y) dy
k=1
v=
n→∞
−π
⟨y, uk ⟩ uk ,
−π
k=1
the above equals
=
y−
n
∑
2
⟨y, uk ⟩ uk
+
=
y−
2
⟨y, uk ⟩ uk
k=1
k=1
n
∑
n
∑
2
⟨y, uk ⟩ uk
k=1
+
n
∑
0
2
|⟨y, uk ⟩| ,
∫
k=1
the last step following because of similar
reasoning to the above and the assumption
that the ∑
uk are orthonormal. It follows
∞
2
the sum k=1 |⟨y, uk ⟩| converges and so
limk→∞ ⟨y, uk ⟩ = 0 because if a series converges, then the k th term must converge to
0.
18 Let f be any piecewise continuous function
which is bounded on [−π, π] . Show, using
the above problem, that
∫ π
lim
f (t) sin (nt) dt
n→∞ −π
∫ π
= lim
f (t) cos (nt) dt = 0
n→∞
Now observe that Dn is an even function.
Therefore, the formula equals
∫ π
Sn f (x) =
f (x − y) Dn (y) dy
−π
−π
Then, from the above
} and the fact
{ problem
1
ikx
√
form an
shown earlier that
e
2π
k∈Z
orthonormal set of vectors in this inner product space, it follows that
⟨
⟩
lim f, einx = 0
n→∞
−π
π
f (x − y) Dn (y) dy
∫
f (x − y) Dn (y) dy
=
∫ 0π
+
f (x + y) Dn (y) dy
0
∫
f (x + y) + f (x − y)
2Dn (y) dy
2
0
∫π
Now note that 0 2Dn (y) = 1 because
π
=
∫
π
−π
Dn (y) dy = 1
and Dn is even. Therefore,
Sn f (x) −
Let the inner product space consist of piecewise continuous bounded functions with the
inner product defined by
∫ π
⟨f, g⟩ ≡
f (x) g (x)dx
0
+
f (x+) + f (x−)
=
2
∫
π
2Dn (y) ·
0
f (x + y) − f (x+) + f (x − y) − f (x−)
dy
2
From the formula for Dn (y) given earlier,
this is dominated by an expression of the
form
∫ π
f (x + y) − f (x+) + f (x − y) − f (x−)
C
·
sin (y/2)
0
sin ((n + 1/2) y)dy
432
APPENDIX C. ANSWERS TO SELECTED EXERCISES
for a suitable constant C. The above is equal
to
∫ π
y
(y)·
C
0 sin 2
21 Consider for t ∈ [0, 1] the following.
|y − (x + t (w − x))|
where w ∈ K and x ∈ K. It equals f (t) =
f (x + y) − f (x+) + f (x − y) − f (x−)
y
2
Suppose x is the point of K which is closest
to y. Then f ′ (0) ≥ 0. However,
f ′ (0) = −2 Re ⟨y − x, w − x⟩ .
y
and the expression sin(y/2)
equals a bounded
continuous function on [0, π] except at 0
where it is undefined. This follows from
elementary calculus. Therefore, changing
the function at this single point does not
change the integral and so we can consider
this as a continuous bounded function defined on [0, π] . Also, from the assumptions
on f, y →
Therefore, if x is closest to y,
Re ⟨y − x, w − x⟩ ≤ 0.
Next suppose this condition holds. Then
you have
is equal to a piecewise continuous function
on [0, π] except at the point 0. Therefore,
the above integral converges to 0 by the previous problem. This shows that the Fourier
series generally tries to converge to the midpoint of the jump.
∑ (−1)
π
= lim
n→∞
4
2k − 1
2
k=1
2
You could also find the Fourier series for x
instead of x and get
n
∑
4
π2
k
(−1) cos (kx) +
= x2
2
n→∞
k
3
lim
k=1
because the periodic extension of this function is continuous. Let x = 0
n
∑
4
π2
k
lim
(−1)
+
=0
n→∞
k2
3
k=1
and so
π2
3
=
n
∑
4
k+1
(−1)
n→∞
k2
≡
∞
∑
4
k+1
(−1)
k2
lim
k=1
k=1
This is one of those calculus problems where
you show it converges absolutely by the comparison test with a p series. However, here
is what it converges to.
≥
2
≥ |y − x|
|y − x| + t |w − x|
2
2
By convexity of K, a generic point of K is
of the form x + t (w − x) for w ∈ K. Hence
x is the closest point.
22
2
|x + y| + |x − y|
2
2
2
= |x| + |y| + 2 Re ⟨x, y⟩
2
2
+ |x| + |y| − 2 Re ⟨x, y⟩
=
k+1
2
|y − (x + t (w − x))|
f (x + y) − f (x+) + f (x − y) − f (x−)
y
n
2
|y − x| +t2 |w − x| −2t Re ⟨y − x, w − x⟩
sin ((n + 1/2) y)dy
20
2
2
2 |x| + 2 |y|
2
Of course the same reasoning yields
)
1(
2
2
|x + y| − |x − y|
4
1( 2
2
=
|x| + |y| + 2 ⟨x, y⟩
4 (
))
2
2
− |x| + |y| − 2 ⟨x, y⟩
= ⟨x, y⟩
23 Let xk be a minimizing sequence. The connection between xk and ck ∈ Fk is obvious
because the {uk } are orthonormal. That
is,
|xn − xm | = |cn − cm |Fp .
∑
where x ≡ j cj uj . Use the parallelogram
identity.
y − xk − (y − xm )
2
+
= 2
2
y − xk + (y − xm )
2
y − xk
2
2
+2
2
y − xm
2
C.18. EXERCISES 357
433
Hence
1
2
|xm − xk |
4
1
2
=
|y − xk |
2
2
1
xk + xm
2
+ |y − xm | − y−
2
2
1
1
2
2
|y − xk | + |y − xm | − λ2
≤
2
2
Now the right hand side converges to 0 since
{xk } is a minimizing sequence. Therefore,
{xk } is a Cauchy sequence in U. Hence
{ } the
sequence of component vectors ck is a
Cauchy sequence in Fn and so it converges
thanks to completeness of F. It follows that
{xk } also must converge to some x. Then
since K is closed, it follows that x ∈ K.
Hence
λ = |x − y| .
24
⟨P x − P y, y − P y⟩ ≤
0
⟨P y − P x, x − P x⟩ ≤
0
Thus
⟨P x − P y, x − P x⟩ ≥ 0
Hence
⟨P x − P y, x − P x⟩−⟨P x − P y, y − P y⟩ ≥ 0
saying that ∥x∥ ≤ ∆ |x| . If this is not so,
then there exists a sequence of vectors {xk }
such that
∥xk ∥ > k |xk |
dividing both sides by ∥xk ∥ it can be assumed that 1 > k |xk | = xk . Hence xk →
0 in Fk . But from the triangle inequality,
∥xk ∥ ≤
Therefore, since limk→∞ xki = 0, this is a
contradiction to each ∥xk ∥ = 1. It follows
that there exists ∆ such that for all x,
∥x∥ ≤ ∆ |x|
Now consider the other direction. If it is
not true, then there exists a sequence {xk }
such that
1
|xk | > ∥xk ∥
k
Dividing both sides by |xk | , it can be assumed that |xk | = xk = 1. Hence, by
compactness of the closed unit ball in Fn ,
there exists a further subsequence, still denoted by k such that xk → a ∈ Fn and it
also follows that |a|Fn = 1. Also the above
inequality implies limk→∞ ∥xk ∥ = 0. Therefore,
n
∑
j=1
25
aj uj = lim
k→∞
n
∑
xkj uj = lim xk = 0
k→∞
j=1
which is a contradiction to the uj being linearly independent. Therefore, there exists
δ > 0 such that for all x,
|x − y| |P x − P y| ≥
⟨P x − P y,P x − P y⟩
xki ∥ui ∥
i=1
and so
⟨P x − P y, x − y − (P x − P y)⟩ ≥ 0
n
∑
2
= |P x − P y|
n
Let {uk }k=1 be a basis for
let xi be the components
V and if x ∈ V,
of x relative to
this basis. Thus the xi are defined according to
∑
xi ui = x
i
Then decree that {ui } is an orthonormal
basis. It follows
∑
2
2
|x| =
|xi | .
i
Now letting
{ }{xk } be a sequence of vectors
of V let xk denote the sequence of component vectors in Fn . One direction is easy,
δ |x| ≤ ∥x∥ .
Now if you have any other norm on this
finite dimensional vector space, say |||·||| ,
then from what was just shown, there exist
scalars δ i and ∆i all positive, such that
δ 1 |x|
δ 2 |x|
≤
≤
∥x∥ ≤ ∆1 |x|
|||x||| ≤ ∆2 |x|
It follows that
|||x||| ≤
∆2 ∆1
|x|
δ1
≤
∆2
∥x∥ ≤
δ1
∆2 ∆1
|||x||| .
δ1 δ2
434
APPENDIX C. ANSWERS TO SELECTED EXERCISES
Hence
(e) If w1 , w2 both work, then for every
y,0 =
δ1
∆1
|||x||| ≤ ∥x∥ ≤
|||x||| .
∆2
δ2
In other words, any two norms on a finite
dimensional vector space are equivalent norms.
What this means is that every consideration which depends on analysis or topology is exactly the same for any two norms.
What might change are geometric properties of the norms.
C.19
(
3
1
2
1
2
1
2
√
9 It is required to show that A∗ is linear.
⟨y,A∗ (αz + βw)⟩
≡ ⟨Ay,αz + βw⟩
= α ⟨Ay, z⟩ + β ⟨Ay, w⟩
3
1
2
√
≡ α ⟨y,A∗ z⟩ + β ⟨y,A∗ w⟩
= ⟨y,αA∗ z⟩ + ⟨y,βA∗ w⟩
√ )
3
− 12
1
2
√ )(
− 12 3
1
3
2
( √ √
√
1
2 3 + 14 2
4
√ √
√
=
1
1
4 2 3− 4 2
(
√ )
1
− 21
3
2
√
5
1
1
3
2
2
√ )
− 12 2
√
1
2 2
√ √
√ )
− 14 2 3 − 14 2
√
√ √
1
1
4 2− 4 2 3
= ⟨y,αA∗ z + βA∗ w⟩
√
Since y is arbitrary, this shows that A∗ is
linear. In case A is an m × n matrix as
described,
A∗ = (AT )
1
2
2 √
− 12 2
8 Let f ∈ L (V, F).
11 The two operators D + 1 and D + 4 commute and are each one to one on the kernel of the other. Also, it is obvious that
ker (D + a) consists of functions of the form
Ce−at . Therefore, ker (D + 1) (D + 4) consists of functions of the form
(a) If f = 0, the zero mapping, then f v =
⟨v, 0⟩ for all v ∈ V .
y = C1 e−t + C2 e−4t
⟨v, 0⟩ = ⟨v, 0 + 0⟩ = ⟨v, 0⟩ + ⟨v, 0⟩
so ⟨v, 0⟩ = 0.
(b) If f ̸= 0 then there exists z ̸= 0 satisfying ⟨u, z⟩ = 0 for all u ∈ ker (f ) .
ker (f ) is a subspace and so there exists z1 ∈
/ ker (f ) . Then there exists a
closest point of ker (f ) to z1 called x.
Then let z = z1 − x. Thus ⟨u, z⟩ = 0
for all u ∈ ker (f ).
(c)
f (f (y) z − f (z) y)
= f (y) f (z) − f (z) f (y) = 0.
(d)
0
=
⟨f (y) z − f (z) y, z⟩
2
= f (y) |z| − f (z) ⟨y, z⟩
and so
⟨
f (y) =
y,
f (z)
|z|
= ⟨y, w1 ⟩ − ⟨y, w2 ⟩
= ⟨y, w1 − w2 ⟩
Now let y = w1 −w2 and so w1 = w2 .
Exercises 391
(
1
f (y) − f (y)
2
⟩
z
(z)
so w = f|z|
2 z appears to work.
where C1 , C2 are arbitrary constants. In
other
{ words,}a basis for ker (D + 1) (D + 4)
is e−t , e−4t .
14





1
0
0
0
2
1
0
0
2
4
1
0
0
6
6
1





17 It is obvious that x ∼ x. If x ∼ y, then
y ∼ x is also clear. If x ∼ y and y ∼ z,
then
z−x=z−y+y−x
and by assumption, both z − y and y − x ∈
ker (L) which is a subspace. Therefore, z − x ∈
ker (L) also and so ∼ is an equivalence relation. Are the operations well defined?
If [x] = [x′ ] , [y] = [y′ ] , is it true that
[x + y] = [y′ + x′ ]? Of course. x′ + y′ −
(x + y) = (x′ − x) + (y′ − y) ∈ ker (L) because ker (L) is a subspace. Similar reasoning applies to the case of scalar multiplication. Now why is A well defined? If
C.19. EXERCISES 391
435
[x] = [x′ ] , is Lx = Lx′ ? Of course this is
so. x − x′ ∈ ker (L) by assumption. Therefore, Lx = Lx′ . It is clear also that A is
linear. If A [x] = 0, then Lx = 0 and so
x ∈ ker (L) and so [x] = 0. Therefore, A is
one to one. It is obviously onto L (V ) = W.
19 An easy way to do this is to “unravel” the
2
powers of the matrix making vectors in Fn
and then making these the columns of a
n2 × n matrix. Look for linear relationships between the columns by obtaining the
row reduced echelon form and using Lemma
8.2.5. As an example, consider the following matrix.


1 1 0


 −1 0 −1 
2 1 3
Lets find its minimal polynomial. We have
the following powers

 

1 0 0
1 1 0

 

 0 1 0  ,  −1 0 −1  ,
0 0 1
2 1 3

 

0
1 −1
−3 −1 −4

 

 −3 −2 −3  ,  −7 −6 −7 
7
5
8
18 15 19
By the Cayley Hamilton theorem, I won’t
need to consider any higher powers than
this. Now I will unravel each and make
them the columns of a matrix.


1 1
0 −3
 0 1
1 −1 




 0 0 −1 −4 




 0 −1 −3 −7 


 1 0 −2 −6 


 0 −1 −3 −7 


 0 2
7
18 




 0 1
5
15 
1
3
8
19
trix and then look for linear relationships.

















1
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
2
−5
4
0
0
0
0
0
0

















From this and Lemma 8.2.5, you see that
for A denoting the matrix,
A3 = 4A2 − 5A + 2I
and so the minimal polynomial is
λ3 − 4λ2 + 5λ − 2
No smaller degree polynomial can work either. Since it is of degree 3, this is also the
characteristic polynomial. Note how we got
this without expanding any determinants
or solving any polynomial equations. If you
factor this polynomial, you get
2
λ3 − 4λ2 + 5λ − 2 = (λ − 2) (λ − 1) so this
is an easy problem, but you see that this
procedure for finding the minimal polynomial will work even when you can’t factor
the characteristic polynomial.
20 If two matrices are similar, then they must
have the same minimal polynomial. This
is obvious from the fact that for p (λ) any
polynomial and A = S −1 BS,
p (A) = S −1 p (B) S
So what is the minimal polynomial of the
diagonal matrix shown? It is obviously
r
∏
(λ − λi ) .
i=1
Thus there are no repeated roots.
Next you can do row operations and obtain
the row reduced echelon form for this ma-
21 Show that if A is an n × n matrix and the
minimal polynomial has no repeated roots,
then A is non defective and there exists a
basis of eigenvectors. Thus, from the above
problem, a matrix may be diagonalized if
and only if its minimal polynomial has no
436
APPENDIX C. ANSWERS TO SELECTED EXERCISES
repeated roots. It turns out this condition is something which is relatively easy
to determine. Hint: You might want to
use Theorem 18.3.1.
If A has a minimal polynomial which has
no repeated roots, say
p (λ) =
m
∏
(λ − λi ) ,
j=1
then from the material on decomposing into
direct sums of generalized eigenspaces, you
have
Fn
= ker (A − λ1 I) ⊕ ker (A − λ2 I) ⊕
· · · ⊕ ker (A − λm I)
and by definition, the basis vectors for
ker (A − λ2 I)
are all eigenvectors. Thus Fn has a basis of
eigenvectors and is therefore diagonalizable
or non defective.
Index
∩, 1
∪, 1
σ(A), 369
Abel’s formula, 104
Abelian group, 319
absolute value
complex number, 5
adjoint, 256, 260
adjugate, 94, 113
algebraic multiplicity, 214
algebraic number
minimal polynomial, 337
algebraic numbers, 337
field, 338
angle between vectors, 27
area
parallelogram, 36
area of a parallelogram, 35
augmented matrix, 47
axioms for a norm, 32
back substitution, 46
bases, 137
basic feasible solution, 186
finding one, 199
basic variables, 53, 186
basis, 137, 322
any two same size, 325
column space, 139
row space, 139
basis of eigenvectors
diagonalizable, 218
bijective, 363
binomial theorem, 11
block matrix, 251
block multiplication, 251
box product, 38
Cartesian coordinates, 14
Cauchy Schwarz inequality, 25, 31
Cayley Hamilton theorem, 114, 239, 277
characteristic equation, 208
characteristic polynomial, 114
characteristic value, 208
chemical reactions
balancing, 56
classical adjoint, 94
cofactor, 89, 90, 111
cofactor matrix, 90
column rank, 128
column space, 127
companion matrix, 303, 317
complex conjugate, 4
complex eigenvalues, 222
shifted inverse power method, 302
complex numbers, 4
complex numbers
arithmetic, 4
roots, 6
triangle inequality, 5
component, 33
component of a force, 29
components of a matrix, 66
components of a vector, 16
composition of linear transformations, 381
condition number, 290
conformable, 69
conjugate
of a product, 11
consistent, 55
Coordinates, 13
Cramer’s rule, 98, 113
cross product, 34
area of parallelogram, 35
coordinate description, 35
distributive law, 37, 39
geometric description, 34
cross product
coordinate description, 35
distributive law, 37
geometric description, 34
parallelepiped, 38
De Moivre’s theorem, 6
defective, 214
defective eigenvalue, 214
derivative, 292
determinant, 107
alternating property, 109
cofactor, 88
437
438
cofactor expansion, 111
expanding along row or column, 88
expansion along row (column), 111
linear transformation, 379
matrix inverse formula, 94, 112
minor, 87
nonzero, 146
product, 92, 110
product of eigenvalues, 276
row operations, 91
transpose, 108
zero, 146
determinant rank
row rank, 145
diagonal matrix, 218, 238
diagonalizable, 218, 238, 249, 379
differential equations
first order systems, 343
dimension, 138
definition, 138
dimension of vector space, 326
direct sum, 366
distance formula, 17
Dolittle’s method, 173
dot product, 25
properties, 25
duality, 201
dynamical system, 238
echelon form, 48
eigenspace, 210
eigenvalue, 208, 369
existence, 367
eigenvalues, 114
eigenvector, 208
Einstein summation convention, 40
elementary matrices, 117
elementary matrix
inverse, 121
properties, 121
elementary operations, 45
empty set, 2
entries of a matrix, 66
equivalence class, 332, 377
equivalence relation, 332, 377
exchange theorem, 136, 322
field axioms, 4
field extension, 332
dimension, 334
finite, 334
field extensions, 335
Field of scalars, 319
force, 22
INDEX
Fourier coefficients, 353
Fredholm alternative, 144, 262
free variables, 53, 186
Frobinius norm, 277
fundamental matrix, 383
fundamental theorem of algebra, 8, 9
fundamental theorem of algebra
plausibility argument, 10
rigorous proof, 10
Gauss Elimination, 55
Gauss elimination, 47, 48
Gauss Jordan method for inverses, 77
Gauss Seidel, 282
Gauss Seidel method, 282
general solution, 163
solution space, 162
generalized eigenspace, 369
direct sum, 367
geometric multiplicity, 214
Gerschgorin’s theorem, 232
Gram Schmidt process, 256, 348
Grammian matrix, 349
greatest common divisor, 329
Hermitian, 258
homogeneous coordinates, 166
homogeneous syster, 162
homomorphism, 153
Householder matrix, 178
householder matrix, 165
inconsistent, 53, 55
independent set
extending to a basis, 140
independent set of vectors
extending to form a basis, 140
injective, 363
inner produc
strange examplet, 33
inner product, 25, 30
axioms, 345
Cauchy Schwarz inequality, 346
inner product
properties, 31
integers mod a prime, 321
intersection, 1
intervals
notation, 1
inverse
left inverse, 113
right inverse, 113
inverses and determinants, 96, 112
invertible, 75
irreducible, 329
INDEX
relatively prime, 330
isomorphism, 153
Jacobi, 280
Jacobi method, 280
Jordan block, 397
joule, 30
ker, 162
kernel, 141, 162
Kirchoff’s law, 63, 64
Kroneker delta, 39
Kroneker symbol, 74
Laplace expansion, 89, 111
leading entry, 48
least square approximation, 259
Leontief model, 86
linear combination, 68, 110, 123
linear independence
definition, 133
enlarging to form a basis, 326
equivalent conditions, 133
linear relationships, 124
finding them, 135
linear transformation, 153, 363
defined on a basis, 364
matrix, 155, 161
rotation, 154
linear transformations
commuting, 366
dimension, 363
linearly dependent, 322
linearly independent, 132, 322
linearly independent sets, 136
LU deomposition
non existence, 171
LU factorization, 171
by inspection, 171
justification, 174
multipliers, 172
solving systems, 173
main diagonal, 90
Markov matrices, 225
math induction, 2
mathematical induction, 2
matrices
more columns than rows, 125
multiplication, 69
one to one, onto, 146
similar, 217
matrix, 65
composition of linear transformations, 381
identity, 74
439
inverse, 75
invertible, product of elementary matrices,
146
left inverse, 113
left inverse, right inverse, 160
lower triangular, 90, 114
main diagonal, 218
one to one, onto, 160
polynomial, 115
raising to a power, 220
right inverse, 113
right inverse left inverse and inverse, 127
rotation, 156
rotation about given vector, 157
self adjoint, 238, 273
symmetric, 238
transpose, 73
upper triangular, 90, 114
matrix exponential, 221, 222
matrix inverse
finding it, 76, 77
matrix multiplication
ij entry, 71
properties, 72
vectors, 68
mean square approximation, 354
migration matrix, 225
minimal polynomial, 152, 368
computation, 393
uniqueness, 368
minimal polynomial
algebraic number, 337
minimization and orthogonality, 351
minor, 89, 90, 111
monic, 329
monic polynomial, 368
multipliers, 176
Neuman series, 86
Newton, 22
nilpotent, 101, 373
non defective
minimal polynomial, 393
nondefective, 249
nondefective eigenvalue, 214
normed linear space, 348
normed vector space, 348
null space, 141, 162
nullity, 142
one to one, 155
rank, 151
onto, 155
open ball, 17
440
operator norm, 287
orthogonal matrix, 101, 165, 178, 241
switching two unit vectors, 179
orthogonal projection, 352
orthogonality and minimization, 259
orthonormal, 242, 255
independent, 348
orthonormal set, 348
p norms, 291
parallelepiped, 38
volume, 38
parallelogram identity, 361
particular solution, 161
partitioned matrix, 251
permutation, 106
permutation matrices, 117
permutation symbol, 40
reduction identity, 40
perp, 143
perpendicular, 28
pivot, 53
pivot column, 49, 124
pivot columns, 49
pivot position, 49
pivot positions, 49
PLU factorization, 176
points and vectors, 13
polar form complex number, 6
polarization identity, 361
polynomial, 328
degree, 328
divides, 329
division, 328
equal, 328
greatest common divisor, 329
greatest common divisor description, 329
greatest common divisor, uniqueness, 329
irreducible, 329
irreducible factorization, 330
relatively prime, 329
root, 328
polynomial
matrix coefficients, 115
polynomials
canceling, 330
factoring, 7
factorization, 331
position vector, 15
power method, 293
preserving distance, 264
principal axes, 251
principal directions, 224
product of matrices
INDEX
composition of linear transformations, 381
projection, 29, 148
projection of a vector, 29
projections
matrix, 159
QR decomposition, 256
QR factorization, 179, 256
thin, 256
quadratic form, 251
quadratic formula, 8
rank
column determinant and row, 128
existence of solutions, 143
finding the rank, 130
linear transformation, 379
rank and singular values, 267
rank of a matrix, 128, 145
Rayleigh quotient, 303
reflection
across a given vector, 165
regression line, 261
regular Sturm Liouville problem, 358
resultant, 22
right handed system, 34
right inverse, 77
right polar factorization, 263
rotations
about given vector, 381
row and column space, 129
row equivalent, 125
row operations, 48, 91, 117
row rank, 128
row reduced echelon form, 48, 123
existence, 123
uniqueness, 125
row space, 127
scalar product, 25
scalars, 14, 65
scaling factor, 293
Schur’s theorem, 257
set notation, 1
sgn, 105
uniqueness, 106
shifted inverse power method, 295
complex eigenvalues, 302
sign of a permutation, 106
similar matrices, 377
similarity
block diagonal matrix, 369
upper triangular block diagonal, 371
similarity and equivalence, 378
similarity relation, 218
INDEX
similarity transformation, 377
Wronskian, 104, 350
simple field extension, 339
Wronskian alternative, 343
simplex tableau, 186, 187
zero matrix, 66
simultaneous corrections, 280
singular value decomposition, 266
singular values, 266
skew lines, 44
skew symmetric, 74, 82
slack variables, 186, 188
solution space, 162
span, 110, 123, 322
spanning sets, 136
spectrum, 208
speed, 23
splitting field, 335
strictly upper triangular, 397
Sturm Liouville problem, 358
subspace, 135
has a basis, 327
substituting matrix into polynomial identity, 115
surjective, 363
Sylvester’s theorem, 365
symmetric, 74, 82
symmetric matrix, 243
trace, 277
sum of eigenvalues, 277
transpose
dot product, 143
triangle inequality, 19, 26, 32, 347
complex numbers, 5
trigonometry
sum of two angles, 157
union, 1
unitary, 256
upper Hessenberg form, 311
variation of constants formula, 344
variational inequality, 361
vector
components, 16
vector addition
geometric meaning, 16
vector space, 319
dimension, 326
vector space axioms, 14, 66, 319
vectors, 13, 22, 67
column, 67
row vector, 67
velocity, 23
well ordered, 2
well ordering, 2
work, 29
441