No bullshit guide to linear algebra
Ivan Savov
April 30, 2016
N o bullshit guide to linear algebra
by Ivan Savov
Copyright © Ivan Savov, 2015, 2016.
All rights reserved.
Published by M inir ef er ence C o.
Montréal, Québec, Canada
minireference.com | @minireference | fb.me/noBSguide
For inquiries, contact the author at ivan.savov@gmail.com
Near-final release
v0.91 hg changeset: 314:578e9ac33e27
ISBN 978-0-9920010-1-8
10 9 8 7 6 5 4 3 2 1
Contents
P reface
vii
I ntroduction
1
1 M ath fundamentals
1.1 Solving equations . . . . . . . . . . . . . . . . . . . . .
1.2 Numbers . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Variables . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 Functions and their inverses . . . . . . . . . . . . . . .
1.5 Basic rules of algebra . . . . . . . . . . . . . . . . . . .
1.6 Solving quadratic equations . . . . . . . . . . . . . . .
1.7 T he Cartesian plane . . . . . . . . . . . . . . . . . . .
1.8 Functions . . . . . . . . . . . . . . . . . . . . . . . . .
1.9 Function reference . . . . . . . . . . . . . . . . . . . .
1.10 Polynomials . . . . . . . . . . . . . . . . . . . . . . . .
1.11 Trigonometry . . . . . . . . . . . . . . . . . . . . . . .
1.12 Trigonometric identities . . . . . . . . . . . . . . . . .
1.13 Geometry . . . . . . . . . . . . . . . . . . . . . . . . .
1.14 Circle . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.15 Solving systems of linear equations . . . . . . . . . . .
1.16 Set notation . . . . . . . . . . . . . . . . . . . . . . . .
1.17 Math problems . . . . . . . . . . . . . . . . . . . . . .
9
10
12
16
18
21
25
29
32
38
54
58
63
65
67
69
73
79
2 Vectors
89
2.1 Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . 90
2.2 Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
2.3 Vector products . . . . . . . . . . . . . . . . . . . . . . 99
2.4 Complex numbers . . . . . . . . . . . . . . . . . . . . 102
2.5 Vectors problems . . . . . . . . . . . . . . . . . . . . . 107
3 I ntro to linear algebra
111
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 111
3.3
3.4
3.5
3.6
Matrix operations . . . . . . . . . . . . . . . . . . . .
Linearity . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview of linear algebra . . . . . . . . . . . . . . . .
Introductory problems . . . . . . . . . . . . . . . . . .
121
126
131
135
4 C omputational linear algebra
137
4.1 Reduced row echelon form . . . . . . . . . . . . . . . . 138
4.2 Matrix equations . . . . . . . . . . . . . . . . . . . . . 150
4.3 Matrix multiplication . . . . . . . . . . . . . . . . . . . 154
4.4 Determinants . . . . . . . . . . . . . . . . . . . . . . . 158
4.5 Matrix inverse . . . . . . . . . . . . . . . . . . . . . . 169
4.6 Computational problems . . . . . . . . . . . . . . . . . 176
5 G eometrical aspects of linear algebra
181
5.1 Lines and planes . . . . . . . . . . . . . . . . . . . . . 181
5.2 Projections . . . . . . . . . . . . . . . . . . . . . . . . 189
5.3 Coordinate projections . . . . . . . . . . . . . . . . . . 194
5.4 Vector spaces . . . . . . . . . . . . . . . . . . . . . . . 199
5.5 Vector space techniques . . . . . . . . . . . . . . . . . 210
5.6 Geometrical problems . . . . . . . . . . . . . . . . . . 220
6 L inear transformations
223
6.1 Linear transformations . . . . . . . . . . . . . . . . . . 223
6.2 Finding matrix representations . . . . . . . . . . . . . 234
6.3 Change of basis for matrices . . . . . . . . . . . . . . . 245
6.4 Invertible matrix theorem . . . . . . . . . . . . . . . . 249
6.5 Linear transformations problems . . . . . . . . . . . . 256
7 T heoretical linear algebra
257
7.1 Eigenvalues and eigenvectors . . . . . . . . . . . . . . 258
7.2 Special types of matrices . . . . . . . . . . . . . . . . . 271
7.3 Abstract vector spaces . . . . . . . . . . . . . . . . . . 277
7.4 Abstract inner product spaces . . . . . . . . . . . . . . 281
7.5 Gram–Schmidt orthogonalization . . . . . . . . . . . . 288
7.6 Matrix decompositions . . . . . . . . . . . . . . . . . . 292
7.7 Linear algebra with complex numbers . . . . . . . . . 298
7.8 T heory problems . . . . . . . . . . . . . . . . . . . . . 313
8 A pplications
317
8.1 Balancing chemical equations . . . . . . . . . . . . . . 318
8.2 Input–output models in economics . . . . . . . . . . . 320
8.3 Electric circuits . . . . . . . . . . . . . . . . . . . . . . 321
8.4 Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
8.5 Fibonacci sequence . . . . . . . . . . . . . . . . . . . . 330
iii
8.7 Least squares approximate solutions . . . . . . . . . .
8.8 Computer graphics . . . . . . . . . . . . . . . . . . . .
8.9 Cryptography . . . . . . . . . . . . . . . . . . . . . . .
8.10 Error correcting codes . . . . . . . . . . . . . . . . . .
8.11 Fourier analysis . . . . . . . . . . . . . . . . . . . . . .
8.12 Applications problems . . . . . . . . . . . . . . . . . .
333
342
354
366
375
389
9 P robability theory
391
9.1 Probability distributions . . . . . . . . . . . . . . . . . 391
9.2 Markov chains . . . . . . . . . . . . . . . . . . . . . . 398
9.3 Google’s PageRank algorithm . . . . . . . . . . . . . . 404
9.4 Probability problems . . . . . . . . . . . . . . . . . . . 410
10 Quantum mechanics
411
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 412
10.2 Polarizing lenses experiment . . . . . . . . . . . . . . . 418
10.3 Dirac notation for vectors . . . . . . . . . . . . . . . . 425
10.4 Quantum information processing . . . . . . . . . . . . 431
10.5 Postulates of quantum mechanics . . . . . . . . . . . . 434
10.6 Polarizing lenses experiment revisited . . . . . . . . . 448
10.7 Quantum physics is not that weird . . . . . . . . . . . 452
10.8 Quantum mechanics applications . . . . . . . . . . . . 457
10.9 Quantum mechanics problems . . . . . . . . . . . . . . 473
E nd matter
475
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
Social stuff . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 477
General linear algebra links . . . . . . . . . . . . . . . . . . 477
A A nswers and solutions
479
B N otation
499
Math notation . . . . . . . . . . . . . . . . . . . . . . . . . 499
Set notation . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
Vectors notation . . . . . . . . . . . . . . . . . . . . . . . . 500
Complex numbers notation . . . . . . . . . . . . . . . . . . 501
Vector space notation . . . . . . . . . . . . . . . . . . . . . 501
Notation for matrices and matrix operations . . . . . . . . . 502
Notation for linear transformations . . . . . . . . . . . . . . 503
Matrix decompositions . . . . . . . . . . . . . . . . . . . . . 503
Abstract vector space notation . . . . . . . . . . . . . . . . 504
Concept maps
F igure 1: T his concept map illustrates the prerequisite topics of high
school math covered in Chapter 1 and vectors covered in Chapter 2. Also
shown are the topics of computational and geometrical linear algebra covered in Chapters 4 and 5.
F igure 2: Chapter 6 covers linear transformations and their properties.
F igure 4: Matrix operations and matrix computations play an important
role throughout this book. Matrices are used to implement linear transformations, systems of linear equations, and various geometrical computations.
F igure 5: T he book concludes with three chapters on linear algebra applications. In Chapter 8 we’ll discuss applications to science, economics,
business, computing, and signal processing. Chapter 9 on probability theory and Chapter 10 on quantum mechanics serve as examples of advanced
subjects that you can access once you learn linear algebra.
P reface
T his book is about linear algebra and its applications. T he material
is covered at the level of a first-year university course with more advanced concepts also being presented. T he book is written in a clean,
approachable style that gets to the point. Both practical and theoretical aspects of linear algebra are discussed, with extra emphasis on
explaining theconnections between concepts and building on material
students are already familiar with.
Since it includes all necessary prerequisites, this book is suitable
for readers who don’t feel “comfortable” with fundamental math concepts, having never learned them well, or having forgotten them over
the years. T he goal of this book is to give access to advanced
mathematical modelling tools to everyone interested in learning,
regardless of their academic background.
W hy learn linear algebra?
Linear algebra is one of the most useful undergraduate math subjects. T he practical skills like manipulating vectors and matrices
that students learn will come in handy for physics, computer science,
statistics, machine learning, and many other areas of science. Linear
algebra is essential for anyone pursuing studies in science.
In addition to being useful, learning linear algebra can also bea lot
of fun. Readers will experience knowledge buzz from understanding
the connections between concepts and seeing how they fit together.
Linear algebra is oneof themost fundamental subjectsin mathematics
and it’s not uncommon to experience mind-expanding moments while
studying this subject.
T he powerful concepts and tools of linear algebra form a bridge
toward more advanced areas of mathematics. For example, learning
about abstract vector spaces will help students recognize the common
“vector structure” in seemingly unrelated mathematical objects like
matrices, polynomials, and functions. Linear algebra techniques can
viii
P R E FA C E
objects that are vector-like!
W hat’s in this book?
Each section in this book is a self-contained tutorial that covers the
definitions, formulas, and explanations associated with a single topic.
Consult the concept maps on the preceding pages to see the topics
covered in the book and the connections between them.
T hebook begins with a review chapter on numbers, algebra, equations, functions, and trigonometry (Chapter 1) and a review chapter
on vectors (Chapter 2). Anyone who hasn’t seen these concepts before, or who feels their math and vector skills area little“rusty” should
read thesechapters and work through theexercises and problems provided. Readers who feel confident in their high school math abilities
can jump straight to Chapter 3 where the linear algebra begins.
Chapters 4 through 7 cover the core topics of linear algebra: vectors, bases, analytical geometry, matrices, linear transformations, matrix representations, vector spaces, inner product spaces, eigenvectors,
and matrix decompositions.
Chapters 8, 9, and 10 discuss various applications of linear algebra.
T hough not likely to appear on any linear algebra final exam, these
chapters serve to demonstrate the power of linear algebra techniques
and their relevance to many areas of science. T he mini-course on
quantum mechanics (Chapter 10) is unique to this book.
I s this book for you?
T hequick paceand lively explanations in this book provideinteresting
reading for students and non-students alike. Whether you’re learning
linear algebra for a course, reviewing material as a prerequisite for
more advanced topics, or generally curious about the subject, this
guide will help you find your way in the land of linear algebra. T he
short-tutorial format cuts to the chase: we’re all busy adults with no
time to waste!
T his book can be used as the main textbook for any universitylevel linear algebra course. It contains everything students need to
know to prepare for a linear algebra final exam. Don’t be fooled by
the book’s small format: it’s all in here. T he text is compact because
it distills the essentials and removes the unnecessary cruft.
P ublisher
T he genesis of the no bul l shit guide textbook series dates back to
ix
that “something must be done,” and started a textbook company to
produce textbooks that explain math and physics concepts clearly,
concisely, and affordably.
T he goal of Minireference Publishing is to fix the first-year
science textbooks problem. Mainstream textbooks suck, so we’re doing something about it. We want to set the bar higher and redefine
readers’ expectations for what a textbook should be! Using print-ondemand and digital distribution strategies allows us to providereaders
with high quality textbooks at reasonable prices.
A bout the author
I have been teaching math and physics for more than 15 years as a
private tutor. T hrough this experience, I learned to explain difficult
concepts by breaking complicated ideas into smaller chunks. An interesting feedback loop occurs when students learn concepts in small
chunks: the knowledge buzz they experience when concepts “click”
into placemotivates them to continuelearning more. I know this from
first-hand experience, both as a teacher and as a student. I completed
my undergraduate studies in electrical engineering, then stayed on to
earn a M.Sc. in physics, and a Ph.D. in computer science.
Linear algebra played a central role throughout my studies. With
this book, I want to share with you some of what I’ve learned about
this expansive subject.
Ivan Savov
Montreal, 2016
Introduction
T here have been countless advances in science and technology in recent years. Modern science and engineering fields have developed
advanced models for understanding thereal world, predicting theoutcomes of experiments, and building useful technology. We’re still far
from obtaining a theory of everything that can predict the future,
but we understand a lot about the natural world at many levels of description: physical, chemical, biological, ecological, psychological, and
social. Anyone interested in being part of scientific and technological
advances has no choice but to learn mathematics, since mathematical
models are used throughout all fields of study. T he linear algebra
techniques you’ll learn in this book are some of the most powerful
mathematical modelling tools that exist.
At the core of linear algebra lies a very simple idea: linearity. A
function f is linear if it obeys the equation
f (ax1 + bx2) = af (x1) + bf (x2);
where x1 and x2 are any two inputs suitable for the function. We use
the term linear combination to describe any expression constructed
from a set of variables by multiplying each variable by a constant
and adding the results. In the above equation, the linear combination
ax1 + bx2 of the inputs x1 and x2 is transformed into the linear
combination af (x1) + bf (x2) of the outputs of the function f (x 1)
and f (x2). L inear functions transform linear combinations
of their inputs into the same linear combination of their
outputs. T hat’s it, that’s all! Now you know everything there is to
know about linear algebra. T he rest of the book is just details.
A significant proportion of the models used by scientists and engineers describe linear relationships between quantities. Scientists,
engineers, statisticians, business folk, and politicians develop and use
linear models to make sense of the systems they study. In fact, linear
models are often used to model even nonlinear (more complicated)
2
INT RODUCT ION
thereal world. Linear models for nonlinear phenomena arereferred to
as linear approximations. If you’ve previously studied calculus, you’ll
remember learning about tangent lines. T he tangent line to a curve
f (x) at xo is given by the equation
T (x) = f 0(xo) x
xo + f (xo):
T his line has slope f 0(xo) and passes through the point (xo; f (xo)).
T he equation of the tangent line T (x) serves to approximate the function f (x) near xo. Using linear algebra techniques to model nonlinear
phenomena can be understood as a multivariable generalization of
this idea.
Linear models can also be combined with nonlinear transformations of the model’s inputs or outputs to describe nonlinear phenomena. T hese techniques are often employed in machine learning: kernel
methods are arbitrary non-linear transformations of the inputs of a
linear model, and the sigmoid activation curve is used to transform
a smoothly-varying output of a linear model into a hard yes or no
decision.
Perhaps the main reason linear models are widely used is because
they are easy to describe mathematically, and easy to “fit” to realworld systems. We can obtain the parameters of a linear model for a
real-world system by analyzing its behaviour for relatively few inputs.
We’ll illustrate this important point with an example.
E xample At an art event, you enter a room with a multimedia
setup. A drawing canvas on a tablet computer is projected on a
giant screen. Anything you draw on the tablet will instantly appear
projected on the giant screen. T he user interface on the tablet screen
doesn’t give any indication about how to hold the tablet “right side
up.” What is the fastest way to find the correct orientation of the
tablet so your drawing will not appear rotated or upside-down?
T his situation is directly analogous to the tasks scientists face every day when trying to model real-world systems. T he canvas on the
tablet describes a two-dimensional input space, and the wall projection is a two-dimensional output space. We’relooking for theunknown
transformation T that maps the pixels of the tablet screen (the input
space) to coloured dots on the wall (the output space). If the unknown transformation T is a linear transformation, we can learn its
parameters very quickly.
Let’s describe each pixel in the input space with a pair of coordinates (x; y) and each point on the wall with another pair of coordinates (x0; y0). T he unknown transformation T describes the mapping
of pixel coordinates to wall coordinates:
3
F igure 6: An unknown linear transformation T maps “tablet coordinates”
to “screen coordinates.” How can we characterize T ?
To uncover how T transforms(x; y)-coordinatesto (x0; y0)-coordinates,
you can use the following three-step procedure. First put a dot in the
lower left corner of the tablet to represent the origin (0; 0) of the
xy-coordinate system. Observe the location where the dot appears
on the wall—we’ll call this location the origin of the x0y0-coordinate
system. Next, make a short horizontal swipe on the screen to represent the x-direction (1; 0) and observe the transformed T (1; 0) that
appears on the wall. As the final step, make a vertical swipe in the
y-direction (0; 1) and see the transformed T (0; 1) that appears on the
wall. By noting how the xy-coordinate system is mapped to the x0y0coordinate system, you can determine which orientation you must
hold the tablet for your drawing to appear upright when projected on
the wall. K nowing the outputs of a linear transformation T
for all “directions” in its inputs space allows us to completely
characterize T .
In the case of the multimedia setup at the art event, we’re looking
for an unknown transformation T from a two-dimensional input space
to a two-dimensional output space. SinceT is a linear transformation,
it’s possible to completely describe T with only two swipes. Let’s
look at the math to see why this is true. Can you predict what
will appear on the wall if you make an angled swipe in the (2; 3)direction? Observe that the point (2; 3) in the input space can be
obtained by moving 2 units in the x-direction and 3 units in the ydirection: (2; 3) = (2; 0) + (0; 3) = 2(1; 0) + 3(0; 1). Using the fact
that T is a linear transformation, we can predict the output of the
transformation when the input is (2; 3):
T (2; 3) = T (2(1; 0) + 3(0; 1)) = 2T (1; 0) + 3T (0; 1):
T he projection of the diagonal swipe in the (2; 3)-direction will have
4
INT RODUCT ION
of the two swipes T (1; 0) and T (0; 1) is sufficient to determine the
linear transformation’s output for any input (a; b). Any input (a; b)
can be expressed as a linear combination: (a; b) = a(1; 0) + b(0; 1).
T he corresponding output will be T (a; b) = aT (1; 0) + bT (0; 1). Since
we know T (1; 0) and T (0; 1), we can calculate T (a; b).
T L ;D R Linearity allows us to analyze multidimensional processes
and transformations by studying their effects on a small set of inputs. T his is the essential reason linear models are so prominent in
science. Probing a linear system with each “input direction” is enough
to completely characterize the system. Without this linear structure,
characterizing unknown input-output systems is a much harder task.
Linear algebra is the study of linear structure, in all its details. T he
theoretical results and computational procedures of you’ll learn apply
to all things linear and vector-like.
L inear transformations
You can think of linear transformations as “vector functions” and understand their properties in analogy with the properties of the regular
functions you’re familiar with. T he action of a function on a number
is similar to the action of a linear transformation on a vector:
function f : R ! R ,
input x 2 R ,
output f (x) 2 R ,
inverse function f
1
,
zeros of f ,
linear transformation T : R n ! R m
input ~
x 2 Rn
output T (~
x) 2 R m
inverse transformation T
1
kernel of T
Studying linear algebra will expose you to many topics associated
with linear transformations. You’ll learn about concepts like vector
spaces, projections, and orthogonalization procedures. Indeed, a first
linear algebra course introduces many advanced, abstract ideas; yet
all the new ideas you’ll encounter can be seen as extensions of ideas
you’re already familiar with. Linear algebra is the vector-upgrade to
your high-school knowledge of functions.
P rerequisites
To understand linear algebra, you must havesomepreliminary knowledgeof fundamental math concepts likenumbers, equations, and func-
5
feel confident about your basic math skills, don’t worry. Chapter 1
is specially designed to help bring you quickly up to speed on the
material of high school math.
It’snot a requirement, but it helpsif you’vepreviously used vectors
in physics. If you haven’t taken a mechanics course where you saw velocities and forces represented as vectors, you should read Chapter 2,
as it provides a short summary of vectors concepts usually taught in
the first week of Physics 101. T he last section in the vectors chapter
(Section 2.4) is about complex numbers. You should read that section
at some point because we’ll use complex numbers in Section 7.7 later
in the book.
E xecutive summary
T he book is organized into ten chapters. Chapters 3 through 7 are
the core of linear algebra. Chapters 8 through 10 contain “optional
reading” about linear algebra applications. T he concept maps on
pages iv, v, and vi illustrate the connections between the topics we’ll
cover. I know the maps are teeming with concepts, but don’t worry—
the book is split into tiny chunks, and we’ll navigate the material step
by step. It will be like Mario World, but in n dimensions and with a
lot of bonus levels.
Chapter 3 is an introduction to the subject of linear algebra. Linear algebra is the math of vectors and matrices, so we’ll start by
defining the mathematical operations we can perform on vectors and
matrices.
In Chapter 4, we’ll tackle the computational aspects of linear algebra. By the end of this course, you will know how to solve systems
of equations, transform a matrix into its reduced row echelon form,
compute the product of two matrices, and find the determinant and
the inverse of a square matrix. Each of these computational tasks can
be tedious to carry out by hand and can require lots of steps. T here
is no way around this; we must do the grunt work before we get to
the cool stuff.
In Chapter 5, we’ll review the properties and equations of basic
geometrical objects like points, lines, and planes. We’ll learn how
to compute projections onto vectors, projections onto planes, and
distances between objects. We’ll also review the meaning of vector
coordinates, which arelengths measured with respect to a basis. We’ll
learn about linear combinations of vectors, thespan of a set of vectors,
and formally define what a vector space is. In Section 5.5 we’ll learn
how to use the reduced row echelon form of a matrix, in order to
describe the fundamental spaces associated with the matrix.
6
INT RODUCT ION
putational tools from Chapter 4 and the geometrical intuition from
Chapter 5, we can tackle the core subject of linear algebra: linear
transformations. We’ll explore in detail the correspondence between
linear transformations (vectors functions T : R n ! R m ) and their
representation as m n matrices. We’ll also learn how the coefficients in a matrix representation depend on the choice of basis for
the input and output spaces of the transformation. Section 6.4 on
the invertible matrix theorem serves as a midway checkpoint for
your understanding of linear algebra. T his theorem connects several
seemingly disparate concepts: reduced row echelon forms, matrix inverses, row spaces, column spaces, and determinants. T he invertible
matrix theorem links all these concepts and highlights the properties of invertible linear transformations that distinguish them from
non-linear transformations. Invertible transformations are one-to-one
correspondences (bijections) between the vectors in the input space
and the vectors in the output space.
Chapter 7 covers more advanced theoretical topics of linear algebra. We’ll define the eigenvalues and the eigenvectors of a square
matrix. We’ll seehow theeigenvalues of a matrix tell us important information about the properties of the matrix. We’ll learn about some
special names given to different types of matrices, based on the properties of their eigenvalues. In Section 7.3 we’ll learn about abstract
vector spaces. Abstract vectors are mathematical object that—like
vectors—have components and can be scaled, added, and subtracted
component-wise. Section 7.7 will discuss linear algebra with complex
numbers. Instead of working with vectors with real coefficients, we
can do linear algebra with vectors that havecomplex coefficients. T his
section serves as a review of all the material in the book. We’ll revisit
all the key concepts discussed in order to check how they are affected
by the change to complex numbers.
In Chapter 8 we’ll discuss the applications of linear algebra. If
you’ve done your job learning the material in the first seven chapters,
you’ll get to learn all the cool things you can do with linear algebra.
Chapter 9 will introduce the basic concepts of probability theory.
Chapter 10 contains an introduction to quantum mechanics.
T hesections in thebook areself-contained so you could read them
in any order. Feel free to skip ahead to the parts that you want to
learn first. T hat being said, thematerial is ordered to providean optimal knowing-what-you-need-to-know-before-learning-what-you-wantto-know experience. If you’re new to linear algebra, it would be best
to read them in order. If you find yourself stuck on a concept at some
point, refer to the concept maps to see if you’re missing some prerequisites and flip to the section of the book that will help you fill in the
7
Difficulty level
In terms of difficulty of content, I must prepare you to get ready for
some serious uphill pushes. As your personal “trail guide” up the
“mountain” of linear algebra, it’s my obligation to warn you about
the difficulties that lie ahead, so you can mentally prepare for a good
challenge.
Linear algebra is a difficult subject because it requires developing
your computational skills, your geometrical intuition, and your abstract thinking. T he computational aspects of linear algebra are not
particularly difficult, but they can be boring and repetitive. You’ll
have to carry out hundreds of steps of basic arithmetic. T he geometrical problems you’ll be exposed to in Chapter 5 can be difficult at
first, but will get easier once you learn to draw diagrams and develop
your geometric reasoning. T hetheoretical aspects of linear algebra are
difficult because they require a new way of thinking, which resembles
what doing “real math” is like. You must not only understand and use
the material, but also know how to prove mathematical statements
using the definitions and properties of math objects.
In summary, much toil awaits you as you learn the concepts of
linear algebra, but the effort is totally worth it. All the brain sweat
you put into understanding vectors and matrices will lead to mindexpanding insights. You will reap the benefits of your efforts for the
rest of your life as your knowledge of linear algebra will open many
doors for you.
Chapter 1
M ath fundamentals
In this chapter we’ll review the fundamental ideas of mathematics
which are the prerequisites for learning linear algebra. We’ll define
the different types of numbers and the concept of a function, which is
a transformation that takes numbers as inputs and produces numbers
as outputs. Linear algebra is the extension of these ideas to many
dimensions: instead of “doing math” with numbers and functions, in
linear algebra we’ll be “doing math” with vectors and linear transformations.
F igure 1.1: A concept map showing the mathematical topics covered in
this chapter. We’ll learn about how to solve equations using algebra, how to
model the world using functions, and some important facts about geometry.
T he material in this chapter is required for your understanding of the more
advanced topics in this book.
10
1.1
M AT H F U N DA M E N T A L S
Solving equations
Most math skills boil down to being able to manipulate and solve
equations. Solving an equation means finding the value of the unknown in the equation.
Check this shit out:
x2 4 = 45:
To solve the above equation is to answer the question “What is x?”
More precisely, we want to find the number that can take the place
of x in the equation so that the equality holds. In other words, we’re
asking,
“Which number times itself minus four gives 45?”
T hat is quite a mouthful, don’t you think? To remedy this verbosity, mathematicians often use specialized mathematical symbols.
T he problem is that these specialized symbols can be very confusing.
Sometimes even the simplest math concepts are inaccessible if you
don’t know what the symbols mean.
What are your feelings about math, dear reader? Are you afraid
of it? Do you have anxiety attacks because you think it will be too
difficult for you? Chill! Relax, my brothers and sisters. T here’s
nothing to it. Nobody can magically guess what the solution to an
equation is immediately. To find the solution, you must break the
problem down into simpler steps.
To find x, we can manipulate the original equation, transforming
it into a different equation (as true as the first) that looks like this:
x = only numbers.
T hat’s what it means to solve. T he equation is solved because you
can type the numbers on the right-hand side of the equation into a
calculator and obtain the numerical value of x that you’re seeking.
By the way, before we continue our discussion, let it be noted: the
equality symbol (=) means that all that is to the left of = is equal to
all that is to the right of =. To keep this equality statement true, for
every change you apply to the left side of the equation, you
must apply the same change to the right side of the equation.
To find x, we need to correctly manipulate the original equation
into its final form, simplifying it in each step. T he only requirement
is that the manipulations we make transform one true equation into
another true equation. Looking at our earlier example, the first simplifying step is to add the number four to both sides of the equation:
1.1 SO LV I N G E Q U AT I O N S
11
which simplifies to
x2 = 49:
T he expression looks simpler, yes? How did I know to perform this
operation? I was trying to “undo” the effects of the operation 4.
We undo an operation by applying its inverse. In the case where
the operation is subtraction of some amount, the inverse operation is
the addition of the same amount. We’ll learn more about function
inverses in Section 1.4 (page 18).
We’re getting closer to our goal, namely to isolate x on one side of
the equation, leaving only numbers on the other side. T he next step
is to undo the square x2 operation. T he inverse
operation of squaring
p
2
so this is what we’ll do
a number x is to take the square root
next. We obtain
p
p
2
x = 49:
Notice how we applied the square root to both sides of the equation?
If we don’t apply the same operation to both sides, we’ll break the
equality!
p
p
T he equation x2 = 49 simplifies to
jxj = 7:
What’s up with the vertical bars around x? T he notation jxj stands
for the absolute value of x, which is the same as x except we ignore
the sign. For example j5j = 5 and j 5j = 5, too. T he equation
jxj = 7 indicates that both x = 7 and x = 7 satisfy the equation
x2 = 49. Seven squared is 49, and so is ( 7)2 = 49 because two
negatives cancel each other out.
We’re done since we isolated x. T he final solutions are
x=7
or
x=
7:
Yes, there are two possible answers. You can check that both of the
above values of x satisfy the initial equation x2 4 = 45.
If you are comfortable with all the notions of high school math
and you feel you could have solved the equation x2 4 = 45 on your
own, then you should consider skipping ahead to Chapter 2. If on the
other hand you are wondering how the squiggle killed the power two,
then this chapter is for you! In the following sections we will review
all the essential concepts from high school math that you will need to
power through the rest of this book. First, let me tell you about the
different kinds of numbers.
12
M AT H F U N DA M E N T A L S
1.2
N umbers
In the beginning, we must define the main players in the world of
math: numbers.
Definitions
Numbers are the basic objects we use to calculate things. Mathematicians like to classify the different kinds of number-like objects
into sets:
• T he natural numbers: N = f 0; 1; 2; 3; 4; 5; 6; 7; : : : g
• T he integers: Z = f : : : ; 3; 2; 1; 0; 1; 2; 3; : : : g
• T he rational numbers: Q = f 53 ; 22
7 ; 1:5; 0:125; 7; : : : g
p
• T he real numbers: R = f 1; 0; 1; 2; e; ; 4:94: : : ; : : : g
• T he complex numbers: C = f 1; 0; 1; i; 1+ i; 2+ 3i; : : : g
T hese categories of numbers should be somewhat familiar to you.
T hink of them as neat classification labels for everything that you
would normally call a number. Each item in the above list is a set.
A set is a collection of items of the same kind. Each collection has a
name and a precise definition. Note also that each of the sets in the
list contains all the sets above it. For now, we don’t need to go into
the details of sets and set notation (page 73), but we do need to be
aware of the different sets of numbers.
Why do we need so many different sets of numbers? T he answer
is partly historical and partly mathematical. Each set of numbers is
associated with more and more advanced mathematical problems.
T he simplest numbers are the natural numbers N, which are sufficient for all your math needs if all you are going to do is count things.
How many goats? Five goats here and six goats there so the total
is 11 goats. T he sum of any two natural numbers is also a natural
number.
As soon as you start using subtraction (theinverseoperation of addition), you start running into negative numbers, which are numbers
outside the set of natural numbers. If the only mathematical operations you will ever use are addition and subtraction, then the set of
integers Z = f : : : ; 2; 1; 0; 1; 2; : : :g will be sufficient. T hink about
it. Any integer plus or minus any other integer is still an integer.
You can do a lot of interesting math with integers. T here is an
entire field in math called number theory that deals with integers.
However, to restrict yourself solely to integers is somewhat limiting.
You can’t use the notion of 2.5 goats for example. T he menu at
1
1.2 N U M B E R S
13
If you want to use division in your mathematical calculations,
you’ll need the rationals Q. T he rationals are the set of fractions of
integers:
Q = all z such that z =
x
where x and y are in Z; and y 6
=0 :
y
You can add, subtract, multiply, and dividerational numbers, and the
result will always be a rational number. However, even the rationals
are not enough for all of math!
p
In geometry, we can obtain irrational quantities like 2 (the diagonal of a square with side 1) and (the ratio between a circle’s circumferenceand
its diameter).
T hereareno integers x and y such that
p
p
p
x
2 = y . T herefore, 2 is not part of the set Q, and we say that 2
is irrational. An irrational number has an infinitely long decimal expansion that doesn’t repeat. For example, = 3:141592653589793: : :
where the dots indicate that the decimal expansion of continues all
the way to infinity.
Adding the irrational numbers to the rationals gives us all the
useful numbers, which we call the set of real numbers R. T he set R
containstheintegers,
thefractionsQ, aswell asirrational numberslike
p
2 = 1:4142135: : :. By using the reals you can compute pretty much
anything you want. From here on in the text, when I say number, I
mean an element of the set of real numbers R.
T he only thing you can’t do with the reals is take the square root
of a negative number—you need the complex numbers C for that. We
defer the discussion on C until the end of Chapter 3.
Operations on numbers
A ddition
You can add and subtract numbers. I will assume you are familiar
with this kind of stuff:
2+ 5 = 7; 45+ 56 = 101; 65
66 =
1; 9999+ 1 = 10000:
It can help visual learners to picturenumbers as lengths measured out
on the number line. Adding numbers is like adding sticks together:
the resulting stick has a length equal to the sum of the lengths of the
constituent sticks.
Addition is commutative, which means that a + b = b+ a. It is
also associative, which means that if you have a long summation like
a + b+ c you can compute it in any order (a + b) + c or a + (b+ c)
14
M AT H F U N DA M E N T A L S
M ultiplication
You can also multiply numbers together.
+ a = b+ b+
} |
{z
ab= a + a +
|
{z
+ b:
}
a times
b times
Notethat multiplication can be defined in terms of repeated addition.
T he visual way to think about multiplication is as an area calculation. T he area of a rectangle of base a and height b is equal to ab.
A rectangle with a height equal to its base is a square, and this is why
we call aa = a2 “a squared.”
Multiplication of numbers is also commutative, ab= ba; and associative, abc = (ab)c = a(bc). In modern notation, no special symbol
is used to denote multiplication; we simply put the two factors next
to each other and say the multiplication is implicit. Some other ways
to denote multiplication are a b, a b, and, on computer systems,
a b.
D ivision
Division is the inverse operation of multiplication.
a=b=
a
= one bth of a:
b
Whatever a is, you need to divide it into b equal parts and take one
such part. Some texts denote division as a b.
Note that you cannot divide by 0. Try it on your calculator or
computer. It will say “error divide by zero” because this action
simply doesn’t make sense. After all, what would it mean to divide
something into zero equal parts?
E xponentiation
Often an equation calls for us to multiply things together many times.
T he act of multiplying a number by itself many times is called exponentiation, and we denote this operation as a superscript:
ab = aaa
}:
| {z a
b ti mes
We can also encounter negative exponents. T he negative in the exponent does not mean “subtract,” but rather “divide by”:
a b=
1
=
1
:
1.2 N U M B E R S
15
Fractional exponents describe square-root-like operations:
a
1
2
p
p2
a
a;
a
1
3
p3
a;
a
1
4
p4
a=a
1 1
2 2
= a
1
2
1
2
q
=
p
a:
p
inverse operation of x2. Similarly, for any n we
Square root x is the
p
define the function n x (the nth root of x) to be the inverse function
of xn .
It’s worth clarifying what “taking the nth root” means and understanding when to use this operation. T he nth root of a is a number
which, when multiplied together n times, will give a. For example, a
cube root satisfies
p3
p3 p3 p3
p3 3
a a a=
a = a = a3:
p
Do you see why 3 x and x3 are inverse operations?
T hefractional exponent notation makesthemeaningof rootsmuch
more explicit. T he nth root of a can be denoted in two equivalent
ways:
p
1
n
a an :
T he symbol “ ” stands for “is equivalent to” and is used when two
mathematical objects are
identical. Equivalence is a stronger relation
p
1
n
than equality. Writing a = a n indicates we’ve found two mathematical expressions (the left-hand side and the right-hand side of the
equality) that happen to be equal to each other. It is more mathep
p
1
1
matically precise to write n a a n , which tells us n a and a n are
two different ways of denoting the same mathematical object.
T he nth root of a is equal to one nth of a with respect to multi1
plication. To find the whole number, multiply the number a n times
itself n times:
1
1
1
1
n
n
n
n
a
| a a a{z
1
1
1
a n a n} = a n
n
n
= a n = a1 = a:
n times
T he n-fold product of n1 -fractional exponents of any number produces
p
that number with exponent one, thereforetheinverseoperation of n x
is xn .
T he commutative law of multiplication ab = ba implies that we
can see any fraction ab in two different ways: ab = a 1b = 1b a. We
multiply by a then divide the result by b, or first we divide by b and
then multiply the result by a. Similarly, when we have a fraction in
the exponent, we can write the answer in two equivalent ways:
p3
p
p
p m
2
1
m
1
1
a 3 = a2 = ( 3 a)2; a 2 = 1 = p ; a n = n a = n am :
a
a2
Make sure the above notation makes sense to you.
4
As an exer-
16
M AT H F U N DA M E N T A L S
Operator precedence
T here is a standard convention for the order in which mathematical
operations must be performed. T he basic algebra operations have the
following precedence:
1. Exponents and roots
2. Products and divisions
3. Additions and subtractions
For instance, the expression 5 32 +13 is interpreted as “first find the
square of 3, then multiply it by 5, and then add 13.” Parenthesis are
needed to carry out the operations in a different order: to multiply
5 times 3 first and then take the square, the equation should read
(5 3)2 + 13, where parenthesis indicate that the square acts on
(5 3) as a whole and not on 3 alone.
Other operations
Wecan defineall kinds of operations on numbers. T heabovethreeare
special operations sincethey feel simpleand intuitiveto apply, but we
can also define arbitrary transformations on numbers. We call these
transformations functions. Before we learn about functions, let’s first
cover variables.
1.3
Variables
In math we use a lot of variables, which are placeholder names for
any number or unknown.
E xample Your friend invites you to a party and offers you to drink
from a weirdly shaped shooter glass. You can’t quite tell if it holds
25ml of vodka or 50ml or some amount in between. Since it’s a
mystery how much boozeeach shot contains, you shrug your shoulders
and say there’s x ml in there. T he night happens. So how much did
you drink? If you had three shots, then you drank 3x ml of vodka.
If you want to take it a step further, you can say you drank n shots,
making the total amount of alcohol you consumed nx ml.
Variables allow us to talk about quantities without knowing the
details. T his is abstraction and it is very powerful stuff: it allows you
to get drunk without knowing how drunk exactly!
Variable names
1.3 VA R I A B L E S
17
• x: general name for the unknown in equations (also used to denotea function’s input, as well as an object’s position in physics
problems)
• v: velocity in physics problems
• ; ' : the Greek letters theta and phi are used to denote angles
• xi ; xf : denote an object’s initial and final positions in physics
problems
• X : a random variable in probability theory
• C: costs in business along with P for profit, and R for revenue
Variable substitution
Wecan often change variables and replaceoneunknown variablewith
another to simplify an equation. For example, say you don’t feel
comfortable around square roots. Every time you see a square root,
you freak out until one day you find yourself taking an exam trying
to solve for x in the following equation:
6
p
5
x
=
p
x:
Don’t freak out! In crucial moments like this,
p substitution can help
with your root phobia. J ust write, “Let u = x” on your exam, and
voila, you can rewrite the equation in terms of the variable u:
6
5
u
= u;
which contains no square roots.
T he next step to solve for u is to undo the division operation.
Multiply both sides of the equation by (5 u) to obtain
6
5
u
(5
u) = u(5
u);
6 = 5u
u2:
which simplifies to
T his can be rewritten as a quadratic equation, u2 5u+6 = 0. Next,
we can factor the quadratic to obtain the equation (u 2)(u 3) = 0,
for which u1 = 2 and u2 = 3 are the solutions. T he last
p step is
to convert our u-answers into x answers by using u = x, which
is equivalent to x = u2. T he final answers are x1 = 22 = 4 and
x2 = 32 = 9. Try plugging these x values into the original square root
equation to verify that they satisfy it.
18
M AT H F U N DA M E N T A L S
C ompact notation
Symbolic manipulation is a powerful tool because it allows us to manage complexity. Say you’re solving a physics problem in which you’re
told the mass of an object is m = 140 kg. If there are many steps in
the calculation, would you rather use the number 140 kg in each step,
or the shorter variable m? It’s much easier in the long run to use the
variable m throughout your calculation, and wait until the last step
to substitute the value 140 kg when computing the final answer.
1.4
Functions and their inverses
As we saw in the section on solving equations, the ability to “undo”
functions is a key skill for solving equations.
E xample Suppose we’re solving for x in the equation
f (x) = c;
wheref is somefunction and c is someconstant. Our goal is to isolate
x on one side of the equation, but the function f stands in our way.
By using the inverse function (denoted f 1) we “undo” the effects
of f . T hen we apply the inverse function f 1 to both sides of the
equation to obtain
f
1
(f (x)) = x = f
1
(c) :
By definition, the inverse function f 1 performs the opposite action
of the function f so together the two functions cancel each other out.
We have f 1(f (x)) = x for any number x.
Provided everything is kosher (the function f 1 must be defined
for theinput c), themanipulation wemadeaboveis valid and wehave
obtained the answer x = f 1(c).
T he above example introduces the notation f 1 for denoting the
function’s inverse. T his notation is borrowed from the notion of inverse numbers: multiplication by the number a 1 is the inverse operation of multiplication by the number a: a 1ax = 1x = x. In the
case of functions, however, the negative-one exponent does not re1
fer to “one over-f (x)” as in f (x)
= (f (x)) 1; rather, it refers to the
function’s inverse. In other words, the number f 1(y) is equal to the
number x such that f (x) = y.
Be careful: sometimes applying the inverse leads to multiple solutions. For example, the function f (x) = x2 maps two input values
1.4 F U N C T I O N S A N D T H E I R I N V E R SE S
19
p
and x =
c are solutions to the equation x2 = c. In this case,
this equation’s
solutions can be indicated in shorthand notation as
p
x=
c.
Formulas
Here is a list of common functions and their inverses:
function f (x) ,
x+2 ,
1
2x
x ,
x
p
2x
,
3x + 5 ,
1
(x)
2
x
2x ,
x2 ,
exp(x)
inverse f
x
log2(x)
1
3 (x
5)
ax
,
loga (x)
ex
,
ln(x)
loge(x)
sin(x) ,
sin 1(x)
arcsin(x)
cos(x) ,
cos 1(x)
arccos(x)
T he function-inverse relationship is reflexive—if you see a function
on one side of the above table (pick a side, any side), you’ll find its
inverse on the opposite side.
E xample
Let’s say your teacher doesn’t like you and right away, on the first
day of class, he gives you a serious equation and tells you to find x:
q
p
log5 3+ 6 x 7 = 34+ sin(5:5)
(1):
See what I mean when I say the teacher doesn’t like you?
First, note that it doesn’t matter what (the capital Greek letter
psi ) is, since x is on the other side of the equation. You can keep
copying (1) from line to line, until the end, when you throw the
ball back to the teacher. “My answer is in terms of your variables,
dude. You go figure out what the hell is since you brought it up in
the first place!” By the way, it’s not actually recommended to quote
me verbatim should a situation like this arise. T he same goes with