Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What Is Linear Algebra and Why Learn It?
About This Book
Prerequisites
Math
Attitude
Coding
Mathematical Proofs Versus Intuition from Coding
Code, Printed in the Book and Downloadable Online
Code Exercises
How to Use This Book (for Teachers and Self Learners)
1
2
2
3
3
3
4
5
5
6
2. Vectors, Part 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Creating and Visualizing Vectors in NumPy
Geometry of Vectors
Operations on Vectors
Adding Two Vectors
Geometry of Vector Addition and Subtraction
Vector-Scalar Multiplication
Scalar-Vector Addition
Transpose
Vector Broadcasting in Python
Vector Magnitude and Unit Vectors
The Vector Dot Product
The Dot Product Is Distributive
Geometry of the Dot Product
7
10
11
11
12
13
14
15
16
17
18
20
21
iii
Other Vector Multiplications
Hadamard Multiplication
Outer Product
Cross and Triple Products
Orthogonal Vector Decomposition
Summary
Code Exercises
22
22
23
24
24
28
29
3. Vectors, Part 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Vector Sets
Linear Weighted Combination
Linear Independence
The Math of Linear Independence
Independence and the Zeros Vector
Subspace and Span
Basis
Definition of Basis
Summary
Code Exercises
33
34
35
37
38
38
41
44
46
46
4. Vector Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Correlation and Cosine Similarity
Time Series Filtering and Feature Detection
k-Means Clustering
Code Exercises
Correlation Exercises
Filtering and Feature Detection Exercises
k-Means Exercises
49
52
53
57
57
58
60
5. Matrices, Part 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Creating and Visualizing Matrices in NumPy
Visualizing, Indexing, and Slicing Matrices
Special Matrices
Matrix Math: Addition, Scalar Multiplication, Hadamard Multiplication
Addition and Subtraction
“Shifting” a Matrix
Scalar and Hadamard Multiplications
Standard Matrix Multiplication
Rules for Matrix Multiplication Validity
Matrix Multiplication
Matrix-Vector Multiplication
Matrix Operations: Transpose
iv
|
Table of Contents
61
61
63
65
65
66
67
67
68
69
70
72
Dot and Outer Product Notation
Matrix Operations: LIVE EVIL (Order of Operations)
Symmetric Matrices
Creating Symmetric Matrices from Nonsymmetric Matrices
Summary
Code Exercises
73
73
74
74
75
76
6. Matrices, Part 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Matrix Norms
Matrix Trace and Frobenius Norm
Matrix Spaces (Column, Row, Nulls)
Column Space
Row Space
Null Spaces
Rank
Ranks of Special Matrices
Rank of Added and Multiplied Matrices
Rank of Shifted Matrices
Theory and Practice
Rank Applications
In the Column Space?
Linear Independence of a Vector Set
Determinant
Computing the Determinant
Determinant with Linear Dependencies
The Characteristic Polynomial
Summary
Code Exercises
82
83
84
84
88
88
91
94
96
97
98
99
99
100
101
102
103
104
106
107
7. Matrix Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Multivariate Data Covariance Matrices
Geometric Transformations via Matrix-Vector Multiplication
Image Feature Detection
Summary
Code Exercises
Covariance and Correlation Matrices Exercises
Geometric Transformations Exercises
Image Feature Detection Exercises
113
116
120
124
124
124
126
127
8. Matrix Inverse. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
The Matrix Inverse
Types of Inverses and Conditions for Invertibility
129
130
Table of Contents
|
v
Computing the Inverse
Inverse of a 2 × 2 Matrix
Inverse of a Diagonal Matrix
Inverting Any Square Full-Rank Matrix
One-Sided Inverses
The Inverse Is Unique
Moore-Penrose Pseudoinverse
Numerical Stability of the Inverse
Geometric Interpretation of the Inverse
Summary
Code Exercises
131
131
133
134
136
138
138
139
141
142
143
9. Orthogonal Matrices and QR Decomposition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Orthogonal Matrices
Gram-Schmidt
QR Decomposition
Sizes of Q and R
QR and Inverses
Summary
Code Exercises
147
149
150
151
154
154
155
10. Row Reduction and LU Decomposition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Systems of Equations
Converting Equations into Matrices
Working with Matrix Equations
Row Reduction
Gaussian Elimination
Gauss-Jordan Elimination
Matrix Inverse via Gauss-Jordan Elimination
LU Decomposition
Row Swaps via Permutation Matrices
Summary
Code Exercises
159
160
161
163
165
166
167
169
170
171
172
11. General Linear Models and Least Squares. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
General Linear Models
Terminology
Setting Up a General Linear Model
Solving GLMs
Is the Solution Exact?
A Geometric Perspective on Least Squares
Why Does Least Squares Work?
vi
|
Table of Contents
176
176
176
178
179
180
181
GLM in a Simple Example
Least Squares via QR
Summary
Code Exercises
183
187
188
188
12. Least Squares Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Predicting Bike Rentals Based on Weather
Regression Table Using statsmodels
Multicollinearity
Regularization
Polynomial Regression
Grid Search to Find Model Parameters
Summary
Code Exercises
Bike Rental Exercises
Multicollinearity Exercise
Regularization Exercise
Polynomial Regression Exercise
Grid Search Exercises
193
198
199
199
200
204
206
206
206
207
208
210
210
13. Eigendecomposition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Interpretations of Eigenvalues and Eigenvectors
Geometry
Statistics (Principal Components Analysis)
Noise Reduction
Dimension Reduction (Data Compression)
Finding Eigenvalues
Finding Eigenvectors
Sign and Scale Indeterminacy of Eigenvectors
Diagonalizing a Square Matrix
The Special Awesomeness of Symmetric Matrices
Orthogonal Eigenvectors
Real-Valued Eigenvalues
Eigendecomposition of Singular Matrices
Quadratic Form, Definiteness, and Eigenvalues
The Quadratic Form of a Matrix
Definiteness
ATA Is Positive (Semi)definite
Generalized Eigendecomposition
Summary
Code Exercises
214
214
215
216
217
217
220
221
222
224
224
226
227
228
228
230
231
232
233
234
Table of Contents
|
vii
14. Singular Value Decomposition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
The Big Picture of the SVD
Singular Values and Matrix Rank
SVD in Python
SVD and Rank-1 “Layers” of a Matrix
SVD from EIG
SVD of ATA
Converting Singular Values to Variance, Explained
Condition Number
SVD and the MP Pseudoinverse
Summary
Code Exercises
241
243
243
244
246
247
247
248
249
250
251
15. Eigendecomposition and SVD Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
PCA Using Eigendecomposition and SVD
The Math of PCA
The Steps to Perform a PCA
PCA via SVD
Linear Discriminant Analysis
Low-Rank Approximations via SVD
SVD for Denoising
Summary
Exercises
PCA
Linear Discriminant Analyses
SVD for Low-Rank Approximations
SVD for Image Denoising
255
256
259
259
260
262
263
263
264
264
269
272
275
16. Python Tutorial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Why Python, and What Are the Alternatives?
IDEs (Interactive Development Environments)
Using Python Locally and Online
Working with Code Files in Google Colab
Variables
Data Types
Indexing
Functions
Methods as Functions
Writing Your Own Functions
Libraries
NumPy
Indexing and Slicing in NumPy
viii
|
Table of Contents
279
280
280
281
282
283
284
285
286
287
288
289
289
Visualization
Translating Formulas to Code
Print Formatting and F-Strings
Control Flow
Comparators
If Statements
For Loops
Nested Control Statements
Measuring Computation Time
Getting Help and Learning More
What to Do When Things Go Awry
Summary
290
293
296
297
297
297
299
300
301
301
301
302
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Table of Contents
|
ix