Math 317: Theory of Linear Algebra
Professor: Dr. Tim Chumley
Class Notes
May 3, 2014
Contents
Terms from Textbook, Missed Sections
1
Monday, 10 February 2014
11
Tuesday, 11 February 2014
15
Thursday, 13 February 2014
19
Friday, 14 February 2014
23
Monday, 17 February 2014
26
Tuesday, 18 February 2014
29
Thursday, 20 February 2014
33
Friday, 21 February 2014
36
Professor Notes
39
Tuesday, 25 February 2014
48
Thursday, 27 February 2014
52
Friday, 28 February 2014
57
Monday, 3 March 2014
61
Tuesday, 4 March 2014
65
Thursday, 6 March 2014
68
Friday, 7 March 2014
73
Monday, 10 March 2014
74
ii
Tuesday, 11 March 2014
77
Thursday, 13 March 2014
81
Friday, 14 March 2014
86
Monday, 24 March 2014
89
Tuesday, 25 March 2014
94
Sub Professor Alex Roitershtein) Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
94
Professor Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
97
Thursday, 27 March 2014
102
Friday, 28 March 2014
106
Monday, 31 March 2014 & Tuesday, 1 April 2014
110
Thursday, 3 April 2014
117
Friday, 4 April 2014
120
Monday, 7 April 2014
124
Tuesday, 8 April 2014
128
Thursday, 10 April 2014
134
Friday, 11 April 2014
138
Monday, 14 April 2014
143
Tuesday, 15 April 2014
144
Thursday, 17 April 2014
148
Friday, 18 April 2014
152
iii
Monday, 21 April 2014
156
Tuesday, 22 April 2014
159
Thursday, 24 April 2014
163
Friday, 25 April 2014 and Monday, 28 April 2014
166
Tuesday, 29 April 2014
171
Thursday, 1 May 2014
175
Friday, 2 May 2014
178
iv
Terms from Textbook, Missed Sections
Chapter 1 Vectors and Matrices
1.1 Vectors
Vectors in R2
The vector x has length and direction. The length of x is denoted ∣∣x∣∣ and is given by
√
∣∣x∣∣ = x21 + x22 ,
whereas its direction can be specified, say, by the angle the arrow makes with the positive x1 axis.
Definition 1
A vector x is called a unit vector if it has length 1, i.e., ∣∣x∣∣ = 1.
Definition 2
We say two nonzero vectors x and y are parallel if one vector is a scalar multiple of the other,
i.e., if there is a scalar c such that y = cx. We say two nonzero vectors are nonparallel if they
are not parallel. (Notice that when one of the vectors is 0, they are not considered to be either
parallel or non-parallel.)
Remark. We emphasize here that the notions of vector addition and scalar multiplication make
sense geometrically for vectors that do not necessarily have their tails at the origin. If we wish to
Ð→
Ð→
Ð→
add CD to AB, we simply recall that CD is equal to any vector with the same length and direction,
Ð→
so we just translate CD so that C and B coincide; then the arrow from A to the point D in its new
Ð→
Ð→
position is the sum AB + CD.
1.3 On to Rn
Length, scalar multiplication, and vector addition are defined algebraically in an analogous fashion:
If x, y ∈ Rn and c ∈ R, we define
1. ∣∣x∣∣ =
√
x21 + x22 + ⋯ + x2n ;
1
2. cx = (cx1 , cx2 , . . . , cxn );
3. x + y = (x1 + y1 , x2 + y2 , . . . , xn + yn ).
Definition 3
Let v1 , . . . , vk ∈ Rn . If c1 , . . . , ck ∈ R, the vector
v = c1 v1 + c2 v2 + ⋯ + ck vk
is called a linear combination of v1 , . . . , vk . (See Figure 1.15.)
Definition 4
Let v1 , . . . , vk ∈ Rn . The set of all linear combinations of v1 , . . . , vk is called their span , denoted
Span(v1 , . . . , vk ). That is,
Span(v1 , . . . , vk ) = {v ∈ Rn ∶ v = c1 v1 + c2 v2 + ⋯ + ck vk for some scalars c1 , . . . , ck }.
§1.2 Dot Product
Definition 5
Given vectors x, y ∈ R2 , define their dot product
x ⋅ y = x1 y1 + x2 y2 .
More generally, given vectors x, y ∈ Rn , define their dot product
x ⋅ y = x1 y1 + x2 y2 + ⋯ + xn yn .
Remark. The dot product of two vectors is a scalar. For this reason, the dot product is also called
the scalar product, but it should not be confused with the multiplication of a vector by a scalar,
the result of which is a vector. The dot product is also an example of an inner product, which we
will study in Section 6 of Chapter 3.
2
Proposition 1. The dot product has the following properties:
1. x ⋅ y = y ⋅ x for all x, y ∈ Rn (the commutative property);
2. x ⋅ x = ∣∣x∣∣2 ≥ 0 and x ⋅ x = 0 if and only if x = 0;
3. (cx ⋅ y = c(x ⋅ y) for all x, y ∈ Rn and c ∈ R;
4. x ⋅ (y + z) = x ⋅ y + x ⋅ y for all x, y, z ∈ Rn (the distributive property).
Corollary 6
∣∣x + y∣∣2 = ∣∣x∣∣2 + 2x ⋅ y + ∣∣y∣∣2 .
Definition 7
We say vectors x and y ∈ Rn are orthogonal if x ⋅ y = 0.
In general, when you are asked to prove a statement of the form P if and only if Q,this means that
you must prove two statements: If P is true, then Q is also true (“only if); and if Q is true, then P
is also true (“f). In this example, we gave the two arguments simultaneously, because they relied
essentially only on algebraic identities. A useful shorthand for writing proofs is the implication
symbol, Ô⇒. The sentence
P Ô⇒ Q
can be read in numerous ways:
• “if P , then Q”
• “P implies Q”
• “P only if Q”
• “Q whenever P ”
• “P is sufficient for Q” (because when P is true, then Q is true as well)
• “Q is necessary for P ” (because P can”t be true unless Q is true)
The “reverse implication” symbol, ⇐Ô, occurs less frequently, because we ordinarily write “P ⇐Ô
Q” as “Q Ô⇒ P .” This is called the converse of the original implication. To convince yourself
3
that a proposition and its converse are logically distinct, consider the sentence “If students major
in mathematics, then they take a linear algebra course. The converse is “If students take a linear algebra course, then they major in mathematics. How many of the students in this class are
mathematics majors??
We often use the symbol ⇐⇒ to denote “if and only if”: P ⇐⇒ Q means “P Ô⇒ Q and Q Ô⇒ P .”
This is often read “P is necessary and sufficient for Q” here necessity corresponds to “Q Ô⇒ P ”
and sufficiency corresponds to “P Ô⇒ Q.”
Definition 8
Let x and y be nonzero vectors in Rn . We define the angle between them to be the unique θ
satisfying 0 ≤ θ ≤ π so that
cos θ =
x⋅y
.
∣∣x∣∣∣∣y∣∣
Proposition 2. (Cauchy-Schwarz Inequality) If x, y ∈ Rn , then
∣x ⋅ y∣ ≤ ∣∣x∣∣∣∣y∣∣.
Moreover, equality holds if and only if one of the vectors is a scalar multiple of the other.
Remark. The dot product also arises in situations removed from geometry.
§1.3 Hyperplanes in Rn
Recall that
proja x =
x⋅a
a,
∣∣a∣∣2
the line consists of all vectors whose projection onto the normal vector a is the constant vector
c
a.
∣∣a∣∣2
§1.4 Systems of Linear Equations and Gaussian Elimination
When we are solving a system of equation, there are three basic algebraic operations we can perform
that will not affect the solution set. They are the following elementary operations:
4
(i) Interchange any pair of equations.
(ii) Multiply any equation by a nonzero real number.
(iii) Replace any equation by its sum with a multiple of any other equation.
Theorem 9. If a system of equations Ax = b is changed into a new system Cx = d by elementary
operations, then the systems have the same set of solutions.
Remark. It is important to distinguish between the symbols = and ↝; when we convert one matrix
to another by performing one or more row operations, we do not have equal matrices.
Definition 10
We call the first nonzero entry of a row (reading left to right) its leading entry . A matrix is in
echelon form if
1. The leading entries move to the right in successive rows.
2. The entries of the column below each leading entry are all 0.
3. All rows of 0’s are at the bottom of the matrix.
A matrix is in reduced echelon form if it is in echelon form and, in addition,
4. Every leading entry is 1.
5. All entries of the column above each leading entry are 0 as well.
If a matrix is in echelon form, we call the leading entry of any (nonzero) row a pivot . We refer
to the columns in which a pivot appears as pivot columns and to the corresponding variables
(in the original system of equations) as pivot variables . The remaining variables are called
free variables .
Definition 11
We say that we’ve written the general solution in standard form when it is expressed as the
sum of a particular solution – obtained by setting all the free variables equal to 0 – and a linear
combination of vectors, one for each free variable – obtained by setting that free variable equal to
5
1 and the remaining free variables equal to 0 and ignoring the particular solution.
Proposition 3. All echelon forms of an m × n matrix A have the same number of nonzero rows.
Theorem 12. Each matrix has a unique reduce echelon form.
§1.5 The Theory of Linear Systems
5.1 Existence, Constraint Equations, and Rank
Definition 13
If the system of equations Ax = b has no solutions, the system is said to be inconsistent ; if it
has at least one solution, then it is said to be consistent .
Remark. It is worth noting that since A has different echelon forms, one can arrive at different
constraint equations. We will investigate this more deeply in Chapter 3.
Definition 14
The rank of a matrix A is the number of nonzero rows (the number of pivots) in any echelon
form of A. It is usually denoted by r.
Proposition 4. The linear system Ax = b is consistent if and only if the rank of the augmented
matrix [A ∣ b] equals the rank of A. In particular, when the rank of A equals m, the system Ax = b
will be consistent for all vectors b ∈ Rm .
5.2 Uniqueness and Non-uniqueness of Solutions
Definition 15
A system Ax = b of linear equations is called inhomogeneous when b ≠ 0; the corresponding
equation Ax = 0 is called the associated homogeneous system .
Proposition 5. Let A be an m × n matrix and let x, y ∈ Rn . Then
A(x + y) = Ax + Ay.
(This is called the distributive property of matrix multiplication.)
6
Theorem 16. Assume the system Ax = b is consistent, and let u1 be a particular solution. Then
all the solutions are of the form
u = u1 + v
for some solution v of the associated homogeneous system Ax = 0.
Definition 17
If the system of equations Ax = b has precisely one solution, then we say that the system has a
unique solution.
Proposition 6. Suppose the system Ax = b is consistent. Then it has a unique solution if and only
if the associated homogeneous system Ax = 0 has only the trivial solution. This happens exactly
when r = n.
Definition 18
An n × n matrix of rankr = n is called nonsingular . An n × n matrix of rankr < n is called
singular .
Proposition 7. Let A be an n × n matrix. The following are equivalent:
1. A is nonsingular.
2. Ax = 0 has only the trivial solution.
3. For every b ∈ Rn , the equation Ax = b has a solution (indeed, a unique solution).
7
Chapter 2. Matrix Algebra
§2.1 Matrix Operations
We take this opportunity to warn our readers that the word if is ordinarily used in mathematical
definitions, even though it should be the phrase if and only if. That is, even though we don’t say
so, we intend it to be understood that, for example, in this case, if A = B, then aij = bij for all i
and j. Be warned: This custom applies only to definitions, not to proposition and theorems! See
the earlier discussion of if and only if in p. 21.
Definition 19
Let A be an n × n (square) matrix with entries aij for 1, . . . , n and j = 1, . . . , n.
1. We call A diagonal if every non-diagonal entry is zero, i.e., if aij = 0 whenever
i ≠ j.
2. We call A upper triangular if all the entries below the diagonal are zero, i.e.,
if aij = 0 whenever i > j.
3. We call A lower triangular if all the entries above the diagonal are zero, i.e., if
aij = 0 whenever i < j.
It is important to understand that when we refer to the set of all m × n matrices, Mm×n , we have
not specified the positive integers m and n. They can be chosen arbitrarily. However, when we say
that A, B ∈ Mm×n , we mean that A and B must have the same “shape,” i.e., the same number of
rows (m) and the same number of columns (n).
Proposition 8. Let A, B, and C ∈ Mm×n and let c, d ∈ R.
1. A + B = B + A.
5. c(dA) = (cd)A.
2. (A + B) + C = A + (B + C).
6. c(A + B) = cA + cB.
3. 0 + A = A.
7. (c + d)A = cA + dA.
4. There is a matrix −A so that A +
8. 1A = A.
(−A) = 0.
8
To understand these properties, one might simply examine corresponding entries of the appropriate
matrices and use the relevant properties of real numbers to see why they are equal. A more elegant
approach is the following: We can encode an m × n matrix as a vector in Rmn , for example,
⎡
⎢ 1
⎢
⎢
⎢
⎢ 2
⎢
⎢
⎢ −5
⎢
⎣
⎤
−1 ⎥⎥
⎥
⎥
6
3 ⎥⎥ ∈ M3×2 ↭ (1, −1, 2, 3, −5, 4) ∈ R ,
⎥
4 ⎥⎥
⎦
and you can check that scalar multiplication and addition of matrices correspond exactly to scalar
multiplication and addition of vectors. We will make this concept more precise in Section 6 of
Chapter 3.
Definition 20
Let A be an m × n matrix and B an n × p matrix. Their product AB is an m × p matrix whose
ij-entry is
n
(AB)ij = ai1 b1j + ai2 b2j + ⋯ + ain bnj = ∑ aik bkj ;
k=1
that is, the dot product of the ith row vector of A and the jth column vector of B, both of which
are vectors in Rn . Graphically, we have
⎡
⎢ a11
⎢
⎢
⎢
⎢
⎢
⎢
⎢ a
⎢ i1
⎢
⎢
⎢
⎢
⎢
⎢
⎢ am1
⎣
a12
⋯
⋮
ai2
⋯
⋮
am2
⋯
⎤
a1n ⎥⎥ ⎡
⎥ ⎢ b11
⎥⎢
⎥⎢
⎥⎢ b
⎥ ⎢ 21
ain ⎥⎥ ⎢⎢
⎥⎢ ⋮
⎥⎢
⎥⎢
⎥⎢
⎥ ⎢ bn1
⎥⎣
amn ⎥⎦
⋯
b1j
b1p
b2j
b2p
⋮
⋯
bnj
⋮
bnp
⎡
⎢
⎤ ⎢ ⋯ ⋯
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥=⎢ ⋯ ⋯
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎦ ⎢
⎢ ⋯ ⋯
⎣
⋯
⋮
(AB)ij
⋮
⋯
⎤
⋯ ⎥⎥
⎥
⎥
⎥
⎥
⎥
⋯ ⋯ ⎥⎥
⎥
⎥
⎥
⎥
⎥
⎥
⋯ ⋯ ⎥⎦
⋯
We reiterate that in order for the product AB to be defined, the number of columns of A must
equal the number of rows of B.
Proposition 9. Let A and A′ be m × n matrices, let B and B ′ be n × p matrices, let C be a p × q
matrix, and let c be a scalar. Then
1. AIn = A = Im A. For this reason, In is called the n × n identity matrix.
9
2. (A+A′ )B = AB +A′ B and A(B +B ′ ) = AB +AB ′ . This is the distributive property
of matrix multiplication over matrix addition.
3. (cA)B = c(AB) = A(cB).
4. (AB)C = A(BC). This is the associative property of matrix multiplication.
§2.2 Linear Transformations: An Introduction
Definition 21
A function T ∶ Rn → Rm is called a linear transformation (or linear map ) if it satisfies
(i) T (x + y) = T (x) + T (y) for all x, y ∈ Rn .
(ii) T (cx) = cT (x) for all x ∈ Rn and all scalars c.
These are often called the linearity properties .
If is important to remember that we have to check that the equation T (x + y) = T (x) + T (y)
holds for all vectors x and y, so the argument must be an algebraic one using variables. Similarly,
we must show T (cx) = cT (x) for all vectors x and all scalars c. It is not enough to check a few
cases.
Just a reminder: To check that a multi-part definition holds, we must check each condition. However,
to show that a multi-part definition fails, we only need to show that one of the criteria does not
hold.
10
Monday, 10 February 2014
§2.2 - Wrap Up
Reflections across a line
Let l be the line through the origin in the direction of u ∈ R2 . To find the standard
Figure 1:
⎛ 1 ⎞
⎛ 0 ⎞
matrix of Rl we must find Rl ⎜ ⎟ and Rl ⎜ ⎟. We can find vectors x⊥ and proju x
⎝ 0 ⎠
⎝ 1 ⎠
Figure 2:
so that x = proju x + x⊥ . Also,
Rl (x) = proju x − x⊥
= proju x − (x − proju x)
= 2proju x − x.
This formula holds for any x ∈ R2 . In particular,
⎛ 1 ⎞
⎛ 1 ⎞ ⎛ 1 ⎞
⎟ = 2proju ⎜ ⎟ − ⎜ ⎟ = 2
Rl ⎜
⎝ 0 ⎠
⎝ 0 ⎠ ⎝ 0 ⎠
11
⎛ 1 ⎞
⎟
u⋅⎜
⎝ 0 ⎠
∣∣u∣∣2
⎛ 1 ⎞
u − ⎜ ⎟.
⎝ 0 ⎠
If u = (u1 , u2 ), then
2u1 ⎛ u1 ⎞ ⎛ 1 ⎞ ⎛
⎜
⎟−⎜ ⎟=⎜
= 2
u1 + u22 ⎝ u ⎠ ⎝ 0 ⎠ ⎝
2
⎛
=⎜
⎝
2u21
u21 +u22
−
u21 +u22
u21 +u22
2u1 u2
u21 +u22
⎞ ⎛
⎟=⎜
⎠ ⎝
u21 −u22
u21 +u22
2u1 u2
u21 +u22
2u21
u21 +u22
−1 ⎞
⎟
2u1 u2
⎠
u2 +u2
1
2
⎞
⎟
⎠
So the first column of the standard matrix of Rl is
⎛
⎜
⎝
u21 −u22
u21 +u22
u21 −u22
u21 +u22
2u1 u2
u21 +u22
2u1 u2
u21 +u22
⎞
⎟
⎠
⎛ 0 ⎞
You find the formula for Rl ⎜ ⎟. This will be the second column.
⎝ 1 ⎠
§2.3 Inverse Matrices
Definition 22
Given an m × n matrix A, an n × m matrix B is called a right inverse if AB = Im . Similarly, an
m × n matrix C is called a left inverse if CA = In . So if it is clear by context what the dimension
of In or Im is we just write I.
Remark. .
1. If A, an m × n matrix has a right inverse B, then Ax = b with x ∈ Rn has a solution for all
B ∈ RM .
Proof. Fix b ∈ RM .
Claim Then u = Bb is a solution to Ax = b because Au = A(Bu) = (AB)b = Im b = b.
Conclusion Existence of right inverse of A tells us rank(A) = m.
2. If Am×n has a left inverse C, then for any b Ax = b has a solution and that solution will be
unique.
12
Proof. Fix b ∈ Rm for which Ax = b has a solution, say u. Let v be some other solution to
Ax = b if it exists. Observe
u − v = In (u − v = CA(u − v = C(A(u − v)) = C(Au − Av) = C(b − b) = C(0) = 0.
Conclusion The existence of a left inverse of A implies rank A = n.
Corollary 23
If A is an m × n matrix with left and right inverses, then the rank A = m and rank A = n, so that
implies that A is square n × n matrix and its nonsingular.
Remark (Remark 3). If A is n × n with left and right inverses C and B respectively. Then B = C.
Proof.
C = CIn = C(AB) = (CA)B = In B = B.
Definition 24
An n × n matrix A is called invertible if there exists an n × n matrix B so that AB = In and
BA = In . We usually denote B as A−1 (read “A inverse”).
How to find the inverse of a nonsingular matrix or is a nonsingular matrix invertible?
Let’s try to find a right inverse of a nonsingular n × n matrix A
A nonsingular ⇒ rank A = n ⇒ Ax = c has a unique solution for every c ∈ Rn .
Let
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
ei = ⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
13
⎤
0 ⎥⎥
⎥
⋮ ⎥⎥
⎥
⎥
0 ⎥⎥
⎥
⎥
1 ⎥⎥
⎥
0 ⎥⎥
⎥
⎥
⋮ ⎥⎥
⎥
⎥
0 ⎥⎦
where 1 is in the ith position.
So, in particular Ax = ei has a unique solution for each 1 ≤ i ≤ n.
14
Tuesday, 11 February 2014
If A is an m × n matrix, then the existence of a right inverse implies rank A = m; and the existence
of a left inverse implies rank A = n.
If A is an n × n matrix and has a left and right inverse (i.e., A is invertible) then A is nonsingular.
Next, we’ll try to show that if A is nonsingular, then A is invertible.
In particular, A being nonsingular implies
Ax = e1 , Ax = e2 , . . . , Ax = en
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
all have unique solutions where ei = ⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎤
0 ⎥⎥
⎥
⋮ ⎥⎥
⎥
⎥
0 ⎥⎥
⎥
⎥
1 ⎥⎥, where 1 is the ith entry.
⎥
0 ⎥⎥
⎥
⎥
⋮ ⎥⎥
⎥
⎥
0 ⎥⎦
Let b1 , . . . , bn ∈ Rn be the unique solutions of these equations respectively. So Ab1 = e1 , . . . , Abn =
en . Let
⎡
⎤
⎢ ∣
∣ ⎥⎥
⎢
⎥
⎢
⎥
⎢
B = ⎢ b1 ⋯ bn ⎥
⎥
⎢
⎥
⎢
⎢ ∣
∣ ⎥⎥
⎢
⎦
⎣
⎤
⎡
⎢
∣
∣
∣ ⎥⎥
⎢
⎥
⎢
⎢
⎥
Note that AB = ⎢ A e1 e2 ⋯ en ⎥ = [A ∣ I].
⎢
⎥
⎢
⎥
⎢
∣
∣
∣ ⎥⎥
⎢
⎦
⎣
Procedure for finding right inverse of a nonsingular matrix A.
15
1. Make an augmented matrix.
⎡
⎢
⎢
⎢
⎢
⎢ A
⎢
⎢
⎢
⎢
⎣
∣
∣
e1
e2
∣
∣
⋯
⎤
∣ ⎥⎥
⎥
⎥
en ⎥⎥ = [A ∣ I]
⎥
∣ ⎥⎥
⎦
2. Use RREF algorithm on the augmented matrix to get
[I ∣ B] .
Example
⎡
⎢ 1
⎢
⎢
⎢
A=⎢ 2
⎢
⎢
⎢ 1
⎢
⎣
⎡
⎢ 1
⎢
⎢
⎢
⎢ 2
⎢
⎢
⎢ 1
⎢
⎣
⎤
1 ⎥⎥
⎥
⎥
−1 0 ⎥⎥
⎥
−2 2 ⎥⎥
⎦
−1
−1
0
1
0
−1
0
0
1
02
2 0
0
⎡
⎤
⎢ 1
0 ⎥⎥
⎢
⎢
⎥
⎢
⎥
0 ⎥⎥ Ð→ ⎢⎢ 0
(∗)
⎢
⎥
⎢ 0
1 ⎥⎥
⎢
⎣
⎦
⎡
⎢ 1
⎢
⎢
⎢
Ð→ ⎢ 0
(∗∗) ⎢
⎢
⎢ 0
⎢
⎣
−1
1
1
1
−2
−2
−1
1
−1
⎤
0 ⎥⎥
⎥
⎥
1 0 ⎥⎥
⎥
0 1 ⎥⎥
⎦
0
0
−1
−1
1
1
−2
−2
1
0
1
3
−1
⎤
0 ⎥⎥
⎥
⎥
0 ⎥⎥
⎥
−1 ⎥⎥
⎦
⎡
⎢ 1 0 0
⎢
⎢
⎢ 0 1 0
⎢
Ð→ ⎢⎢
⎢ 0 0 1
⎢
⎢
⎢
⎢
⎣
2
4
3
⎤
−1 ⎥⎥
⎥
1 −2 ⎥⎥
⎥
⎥
−1 −1 ⎥⎥
⎥
⎥
⎥
B
⎦
0
where at (∗) we did −R1 +R2 → R2 and −R1 +R3 → R3 and at (∗∗) we did R2 +R1 → R1
and R2 + R3 → R3 . Need to check by hand that AB = I.
Elementary Matrices
Recall elementary row operations
1) swap
2) scaling by nonzero factor
3) replace
16
An elementary matrix is a matrix formed by performing a single elementary row operation on
the identity matrix I.
Multiply a matrix A (n×n) on the left by an n×n elementary matrix E, then EA is a row equivalent
matrix obtained by performing that same row operation on A.
Example
⎡
⎢ 1
⎢
⎢
⎢
A=⎢ 4
⎢
⎢
⎢ 7
⎢
⎣
2
5
8
⎤
3 ⎥⎥
⎥
⎥
6 ⎥⎥.
⎥
9 ⎥⎥
⎦
⎤
⎡
⎢ 0 0 1 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
To swap the first and third rows, we multiply on the left by E1 = ⎢ 0 1 0 ⎥.
⎥
⎢
⎥
⎢
⎢ 1 0 0 ⎥
⎥
⎢
⎦
⎣
⎡
⎤
⎤ ⎡
⎤⎡
⎢ 0 0 1 ⎥⎢ 1 2 3 ⎥ ⎢ 7 8 9 ⎥
⎥
⎥ ⎢
⎥⎢
⎢
⎥
⎥ ⎢
⎥⎢
⎢
⎥
⎥ ⎢
⎥⎢
⎢
Take E1 A = ⎢ 0 1 0 ⎥ ⎢ 4 5 6 ⎥ = ⎢ 4 5 6 ⎥.
⎥
⎥ ⎢
⎥⎢
⎢
⎥ ⎢
⎥⎢
⎢
⎥
⎢ 1 0 0 ⎥⎢ 7 8 9 ⎥ ⎢ 1 2 3 ⎥
⎢
⎥
⎥ ⎢
⎥⎢
⎦
⎦ ⎣
⎦⎣
⎣
⎤
⎡
⎢ 1 0 0 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
Scale the second row of A by c ≠ 0 we multiply on the left by E2 = ⎢ 0 2 0 ⎥, then
⎢
⎥
⎥
⎢
⎢ 0 0 1 ⎥
⎥
⎢
⎦
⎣
⎤
⎡
⎢ 1 2 3 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
E2 A = ⎢ 4c 5c 6c ⎥.
⎥
⎢
⎥
⎢
⎢ 7 8 9 ⎥
⎥
⎢
⎦
⎣
To replace the third row of A by its sum with d times the second row, d ≠ 0, we multiply
⎡
⎤
⎡
⎤
⎢ 1 0 0 ⎥
⎢
⎥
1
2
3
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥.
on the left by E3 = 0 1 0 , then E3 A =
4
5
6
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢ 0 d 1 ⎥
⎢ 7 + 4d 8 + 5d 9 + 6d ⎥
⎢
⎥
⎢
⎥
⎣
⎦
⎣
⎦
Go back to A nonsingular n × n matrix.
A nonsingular ⇒ row equivalent to I ⇒ there exists a sequence of row operations that we may
perform on A to get to I ⇒ there exists a sequence of elementary matrices E1 , . . . , Ek for some k
17
so that Ek ⋅ E1 A = I.
´¹¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¶
C
So C = Ek ⋯E1 gives us a left inverse of A.
Theorem 25. A n × n matrix is nonsingular if and only if it is invertible.
Corollary 26
Let A and B be n × n matrices satisfying BA = I. Then B = A−1 and A = B −1 .
Proof. Claim: A is nonsingular.
If Ax = 0 ⇒ BAx = B0 = 0 ⇒ A nonsingular ⇒ A is invertible.
=Ix
= x
So A has a true (right and left) inverse A−1 .
Claim: B = A−1
BA = I ⇒ BAA−1 = IA−1 = BI = A−1 ⇒ B = A−1 .
So B is a right and left inverse for A, so AB = I and BA = I. So A is a left and right
inverse for B. So A = B −1 .
18
Thursday, 13 February 2014
A, an n × n matrix is invertible if it has a left and right matrices.
A is nonsingular if rank A = n.
Theorem 27. The following are equivalent for an n × n matrix A:
(5) Ax = b has a solution for all b ∈ Rn .
(1) A is invertible
(2) A is nonsingular
(6) there exists a matrix C so that CA = I.
(3) RREF of A is I
(4) Ax = 0 has only the trivial solution.
(7) there exists a matrix B so that AB = I.
Proposition 10. Suppose A, B are invertible n × n matrices. Then AB is invertible and (AB)−1 =
B −1 A−1 .
Proof. We should show that
(B −1 A−1 )(AB) = I
(AB)(B −1 A−1 ) = I
(B −1 A−1 )(AB) = B −1 (A−1 A)B = B −1 IB = B −1 B = I
(AB)(B −1 A−1 ) = A(BB −1 )A−1 = AIA−1 = AA−1 = I
19
⎡
⎢ a
⎢
Inverse of a 2 × 2 matrix ⎢
⎢
⎢ c
⎣
⎡
⎢ a
⎢
⎢
⎢
⎢ c
⎣
b1
0
d
0
⎤ ⎡
⎥ ⎢
⎥ ⎢
⎥→⎢
⎥ ⎢
1 ⎥⎦ ⎢⎣
⎡
⎢
⎢
→⎢
⎢
⎢
⎣
⎡
⎢
⎢
→⎢
⎢
⎢
⎣
⎡
⎢
⎢
→⎢
⎢
⎢
⎣
Therefore we have
⎡
⎢ a
⎢
⎢
⎢
⎢ c
⎣
⎤
b ⎥⎥
⎥, suppose a ≠ 0.
⎥
d ⎥⎦
⎤ ⎡
⎤
1
b
0 ⎥⎥ ⎢⎢ 1
0 ⎥⎥
a
a
⎥→⎢
⎥
⎥ ⎢
⎥
c
⎥
c d 0 1 ⎥⎦ ⎢⎣ 0 −bc
+
d
−
1
d
d
⎦
⎤ ⎡
⎤
1
1
b
b
0 ⎥⎥ ⎢⎢ 1 a
0 ⎥⎥
1
a
a
a
⎥→⎢
⎥
⎥ ⎢
⎥
a
a
⎥
) ad−bc
0 ad−bc
− ac 1 ⎥⎦ ⎢⎣ 0 1 − ac ( ad−bc
a
⎦
⎤
⎡
bc
1
a
0 ⎥⎥ ⎢⎢ 1 0 a1 + a(ad−bc)
− ab ad−bc
1 ab
a
⎥→⎢
⎥ ⎢
c
a
c
a
⎥ ⎢ 0 1
0 1 − ad−bc
− ad−bc
ad−bc ⎦
ad−bc
⎣
⎤
b
d
⎥
− ad−bc
1 0 ad−bc
⎥
⎥
⎥
c
a
⎥
0 1 − ad−bc
ad−bc ⎦
1
b
a
1
a
⎤−1
⎤
⎡
b ⎥⎥
1 ⎢⎢ d −b ⎥⎥
⎥ =
⎥,
⎢
⎥
ad − bc ⎢⎢ −c a ⎥⎥
d ⎥⎦
⎦
⎣
⎤
⎥
⎥
⎥
⎥
⎥
⎦
ad − bc ≠ 0
LU Decomposition
Definition 28
⎡
⎢ a11
⎢
⎢
⎢
If A = ⎢ ⋮
⎢
⎢
⎢ a
⎢ n1
⎣
⋯
⋱
⋯
⎤
a1n ⎥⎥
⎥
⎥
⋮ ⎥⎥ = [aij ] then the entries a11 , a22 , . . . , ann are the diagonal of A. If
⎥
ann ⎥⎥
⎦
(1) aij = 0 for all i ≠ j, A is called a diagonal matrix .
(2) aij = 0 for all i > j, A is called upper triangular .
(3) aij = 0 for all i < j, A is called lower triangular .
Claim
If A is a matrix that can be put into echelon form via only replace operations (and not
swap operations) then we can write A = LU where L is a lower triangular matrix and
U is an upper triangular matrix.
20
Example
⎡
⎢ 1 2
⎢
⎢
⎢
A = ⎢ −1 1
⎢
⎢
⎢ 2 7
⎢
⎣
⎤
2 ⎥⎥
⎥
⎥
−1 ⎥⎥
⎥
4 ⎥⎥
⎦
U
⎡
⎢ 1 2
⎢
⎢
⎢
⎢ −1 1
⎢
⎢
⎢ 2 7
⎢
⎣
⎤
⎡
⎢ 1
2 ⎥⎥
⎢
⎥
⎢
E
1
⎥
⎢
⎢ 0
Ð→
−1 ⎥⎥
⎢
⎥ R1 +R2 →R2 ⎢
⎥
⎢ 2
4 ⎥
⎢
⎦
⎣
2
3
7
⎤
2 ⎥⎥
⎥
⎥
1 ⎥⎥
⎥
4 ⎥⎥
⎦
⎡
⎢ 1
⎢
⎢
E2
⎢
⎢ 0
Ð→
−2R1 +R3 →R3 ⎢
⎢
⎢ 0
⎢
⎣
2
3
3
³¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ·¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ µ
⎤
⎡
⎤
⎢ 1 2 2 ⎥
2 ⎥⎥
⎢
⎥
⎥ E ⎢
⎥
⎥ 3 ⎢
⎥
1 ⎥⎥ Ð→ ⎢⎢ 0 3 1 ⎥⎥
⎥
⎢
⎥
⎢ 0 0 −1 ⎥
0 ⎥⎥
⎢
⎥
⎦
⎣
⎦
⎡
⎤
⎡
⎤
⎡
⎢ 1
⎢ 1 0 0 ⎥
⎢ 1 0 0 ⎥
⎥
⎢
⎥
⎢
⎢
⎢
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
where E1 = ⎢ 1 1 0 ⎥, E2 = ⎢ 0 1 0 ⎥, and E3 = ⎢ 0
⎢
⎥
⎢
⎥
⎢
⎢
⎥
⎢
⎥
⎢
⎢ 0
⎢ −2 0 1 ⎥
⎢ 0 0 1 ⎥
⎢
⎥
⎢
⎥
⎢
⎣
⎦
⎣
⎦
⎣
E3 E2 E1 = U , because we only used replace operations, E3 , E2 ,
⎤
0 ⎥⎥
⎥
⎥
1 0 ⎥⎥. So we have
⎥
−1 1 ⎥⎥
⎦
E1 , are all lower trian0
gular.
In homework we see that elementary matrices are invertible. In fact, the inverse of an
elementary matrix is an elementary matrix. For example
⎡
⎤
⎡
⎢ 1
⎢ 1 0 0 ⎥
⎢
⎥
⎢
⎢
⎥
⎢
⎥ −1 ⎢
⎢
−1
E1 = ⎢ −1 1 0 ⎥ E2 = ⎢ 0
⎢
⎥
⎢
⎢
⎥
⎢
⎢ 2
⎢ 0 0 1 ⎥
⎢
⎥
⎢
⎣
⎦
⎣
0
1
0
⎡
⎤
⎢ 1
0 ⎥⎥
⎢
⎢
⎥
⎥ −1 ⎢
0 ⎥⎥ E2 = ⎢⎢ 0
⎢
⎥
⎢ 0
1 ⎥⎥
⎢
⎣
⎦
We have:
E3 E2 E1 A = U
⇒ E2 E1 A = E3−1 U
⇒ E1 A = E2−1 E3−1 U
⇒ A = E1−1 E2−1 E3−1 U
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
L
Note the product of lower triangular matrices is lower triangular.
21
0
1
1
⎤
0 ⎥⎥
⎥
⎥
0 ⎥⎥
⎥
1 ⎥⎥
⎦
Example; Continued
⎡ ⎤
⎡ ⎤
⎡
⎤
⎢ 2 ⎥
⎢ 1 ⎥
⎢ 5 ⎥
⎢ ⎥
⎢ ⎥
⎢
⎥
⎢ ⎥
⎢ ⎥
⎢
⎥
⎢ ⎥
⎢ ⎥
⎢
⎥
Let b1 = ⎢ 1 ⎥, b2 = ⎢ 0 ⎥, b3 = ⎢ −1 ⎥. Try to solve Ax = b1 , Ax = b2 , and Ax = b3 .
⎢ ⎥
⎢ ⎥
⎢
⎥
⎢ ⎥
⎢ ⎥
⎢
⎥
⎢ 1 ⎥
⎢ 2 ⎥
⎢ 4 ⎥
⎢ ⎥
⎢ ⎥
⎢
⎥
⎣ ⎦
⎣ ⎦
⎣
⎦
⎧
⎪
⎪
⎪
⎪ y = Ux
. Consider Ly = b
If we want to solve Ax = b for any b ⇔ LU x = b ⇔ ⎨
⎪
⎪
⎪
Ly
=
b
⎪
⎩
⎡
⎢ 1
⎢
⎢
⎢
y = L−1 b = (E3 E2 E1 )b = ⎢ 1
⎢
⎢
⎢ −3
⎢
⎣
Consider
⎡
⎢ 1
⎢
⎢
⎢
Ux = y ⎢ 0
⎢
⎢
⎢ 0
⎢
⎣
2
3
0
⎤⎡
⎤
0 ⎥⎥ ⎢⎢ b1 ⎥⎥
⎥⎢
⎥
⎥⎢
⎥
1 0 ⎥⎥ ⎢⎢ b2 ⎥⎥ .
⎥⎢
⎥
−1 1 ⎥⎥ ⎢⎢ b3 ⎥⎥
⎦⎣
⎦
0
⎤
⎤ ⎡
⎤⎡
2 ⎥⎥ ⎢⎢ x1 ⎥⎥ ⎢⎢ y1 ⎥⎥
⎥
⎥ ⎢
⎥⎢
⎥
⎥ ⎢
⎥⎢
1 ⎥⎥ ⎢⎢ x2 ⎥⎥ = ⎢⎢ y2 ⎥⎥
⎥
⎥ ⎢
⎥⎢
−1 ⎥⎥ ⎢⎢ x3 ⎥⎥ ⎢⎢ y3 ⎥⎥
⎦
⎦ ⎣
⎦⎣
x1 + 2x2 + 2x3 = y1
3x2 + x3 = y2
−x1 = y3
To solve Ax = b, let
⎡
⎢ 1
⎢
⎢
⎢
y=⎢ 1
⎢
⎢
⎢ −3
⎢
⎣
⎤
⎤⎡ ⎤ ⎡
0 ⎥⎥ ⎢⎢ 2 ⎥⎥ ⎢⎢ 2 ⎥⎥
⎥
⎥⎢ ⎥ ⎢
⎥
⎥⎢ ⎥ ⎢
1 0 ⎥⎥ ⎢⎢ 1 ⎥⎥ = ⎢⎢ 3 ⎥⎥
⎥⎢ ⎥ ⎢
⎥
−1 1 ⎥⎥ ⎢⎢ 1 ⎥⎥ ⎢⎢ −6 ⎥⎥
⎦
⎦⎣ ⎦ ⎣
0
and now solve
⎫
⎪
x1 + 2x2 + 2x3 = 2 ⎪
x1 + 2(−1) + 2(6) = 2
⎪
⎪
⎪
⎪
⎪
⎪
3x3 + x3 = 3 ⎬
3x2 + 6 = 3
⎪
⎪
⎪
⎪
⎪
⎪
−x3 = −6⎪
x3 = 6
⎪
⎭
This simplifies our work.
22
⇒ x1 = −8
⇒ x2 = −1
Friday, 14 February 2014
§2.5 Transpose of a matrix
Definition 29
Let A be an m × n matrix. Then the transpose of A is an n × m matrix whose columns are the
rows of A (in the right order). We denote the transpose of A by AT (read “A transpose”). If aij is
the element of A in the ithe row and the jth column of A, then aji is the element of AT in the ith
row and jth column.
Examples
⎡
⎤
⎡
⎢
⎢ 1 2 3 ⎥
⎢
⎥
⎢
⎢
⎥
⎢
⎥ T ⎢
⎢
1. A = ⎢ 4 5 6 ⎥, A = ⎢
⎥
⎢
⎢
⎢
⎥
⎢
⎢
⎢ 7 8 9 ⎥
⎢
⎥
⎢
⎣
⎦
⎣
⎤
⎡
⎢ 1 2 ⎥
⎡
⎥
⎢
⎢
⎥
⎢
⎥ T ⎢ 1
⎢
2. B = ⎢ 3 4 ⎥, B = ⎢
⎢
⎥
⎢
⎢ 2
⎥
⎢
⎣
⎢ 5 6 ⎥
⎥
⎢
⎦
⎣
⎡ ⎤
⎢ 1 ⎥
⎢ ⎥
⎢ ⎥
⎢ ⎥
3. v = ⎢ 2 ⎥, vT = [1, 2, 3].
⎢ ⎥
⎢ ⎥
⎢ 3 ⎥
⎢ ⎥
⎣ ⎦
From now on, all of our vectors are
1
4
2
5
3
6
3
4
⎤
7 ⎥⎥
⎥
⎥
8 ⎥⎥.
⎥
9 ⎥⎥
⎦
⎤
5 ⎥⎥
⎥.
⎥
6 ⎥⎦
columns. Row vectors will be written as the transpose of the
columns.
Properties
Let A, A′ be m × n matrices, B be an n × p matrix, and c ∈ R.
1. (AT )T = A
2. (cA)T = c(AT )
3. (A + A′ )T = AT + A′T
23
4. (AB)‘ = B T AT
Proof. The (i, j) entry of (AB)T = (j, i) entry of AB = (jth row of A) ⋅ (ith column
of B). The (ij) entry of B T AT = (ith row of B T ) ⋅ (jth column of AT ) = (ith
column of B) ⋅ (jth row of A).
5. If A is invertible. Then AT is invertible and (AT )−1 = (A−1 )T .
Proof. Homework Problem
What is a reason for brining up the transpose? Plays nicely with the dot product.
Remark. Let u, v ∈ Rn (column vectors)
bT v = u ⋅ v
Proposition 11. Let A be an m × n matrix, x ∈ Rn , y ∈ Rm . Then Ax ⋅ y = x ⋅ AT y.
Proof.
Ax ⋅ y = (Ax)T y = xT (AT y) = x ⋅ AT y.
An application of transpose – Orthogonal Matrices
Let A be an n × n matrix, A is called an orthogonal matrix if AT A = I.
Notice:
1. A−1 = AT
⎧
⎪
⎪
⎪
⎪0
2. (A A)ij = (ith row of A )⋅(jth column of A) = (ith column of A)⋅(jth column of A) = ⎨
⎪
⎪
⎪
1
⎪
⎩
So we see the columns of an orthogonal matrix are naturally orthogonal.
T
T
3. Let x, y ∈ Rn
Ax ⋅ Ay = x ⋅ AT A y = x ⋅ y
²
±
´¹¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¶
A orthogonal
(∗∗)
(∗)
24
i≠j
i=j
with
(∗) =
cos(Ax ⋅ Ay)
∣∣Ax∣∣∣∣Ay∣∣
and (∗∗) =
cos(x ⋅ y)
∣∣x∣∣ ⋅ ∣∣y∣∣
Think of A as a linear transformation.
∣∣Ax∣∣2 = Ax ⋅ Ax = x ⋅ AT Ax = x ⋅ x = ∣∣x∣∣2
cos ∠(Ax ⋅ Ay) cos ∠(x, y)
=
∣∣Ax∣∣∣∣Ay∣∣
∣∣x∣∣∣∣y∣∣
Orthogonal matrices preserve orthogonality. So orthogonal matrices preserve angles between vectors.
Claim
The standard matrix of a rotation is an orthogonal matrix
⎡
⎢ cos θ
⎢
Aθ = ⎢
⎢
⎢ sin θ
⎣
⎤
− sin θ ⎥⎥
⎥
⎥
cos θ ⎥⎦
ATθ
⎡
⎢ cos θ
⎢
=⎢
⎢
⎢ − sin θ
⎣
⎤
sin θ ⎥⎥
⎥
⎥
cos θ ⎥⎦
ATθ Aθ
⎡
⎢ 1
⎢
=⎢
⎢
⎢ 0
⎣
Reflection matrices are orthogonal.
Claim
The standard matrix of a reflection is an orthogonal matrix.
⎡
⎢ cos 2θ
⎢
R=⎢
⎢
⎢ sin 2θ
⎣
⎡
⎢ cos θ
⎢
R =⎢
⎢
⎢ sin θ
⎣
⎤
− sin 2θ ⎥⎥
⎥
⎥
− cos 2θ ⎥⎦
T
Check RT R = I.
25
⎤
sin θ ⎥⎥
⎥.
⎥
− cos θ ⎥⎦
⎤
0 ⎥⎥
⎥.
⎥
1 ⎥⎦
Monday, 17 February 2014
Let P be a projection onto a line l in R2 . If x ∈ R2 , then x = c1 u + c2 v where u ∈ l, v ∈ l⊥ , then
Figure 3:
P (x) = c1 P (u) + c2 P (v) = c1 u.
Note that we have already done these calculations, we are looking at them in a different way.
Theorem 30 (Spectral Theorem). If A is an n × n matrix with AT = A, where (∗) – A is a
´¹¹ ¹ ¹ ¹ ¹¸¹ ¹ ¹ ¹ ¹ ¶
(∗)
symmetric matrix. Then A = P DP −1 where P is an orthogonal matrix and D is a diagonal matrix.
(Saw this in homework with Fibonacci Sequence)
Our ultimate goal is to prove this theorem.
Chapter 3. Vector Spaces
§3.1 Subspaces of Rn
Definition 31
A subset V ⊆ Rn is called a (linear) subspace of Rn if
(1) 0 ∈ V .
(2) if u, v ∈ V , then u + v ∈ V (closed under addition)
(3) if c ∈ R, u ∈ V , then cu ∈ V (closed under scalar multiplication)
26
Examples
.
Non-Examples
1. l = {(x1 , x2 ) ∈ R2 ∶ x1 + x2 = 5} This does
1. A line in Rm through the
not contain the zero vector; 0 ∈/ V .
origin, is a subspace.
2. {(x1 , x2 , x3 ) ∈ R3 ∶ x1 + x2 + x3 = 5}. This
l = {tu ∶ t ∈ R}.
does not contain the zero vector; 0 ∈/ V .
3. Z ⊆ R, not a subspace This is not closed
2. A plane through the origin
under scalar multiplication.
(picture of plane through
4. Unit sphere in R3 ; s2 = {x ∈ R ∶ ∣∣x∣∣ = 1} =
origin)
{(x1 , x2 , x3 ) ∶ x21 + x22 + x23 = 1}. 0 ∈/ S 2 , not
3. Rn ⊆ Rn = V – “Rn is a sub-
closed under scalar multiples.
space of itself”
5. {(x1 , x2 ) ∈ R2 ∶ x1 , x2 ≥ 0} = first quadrant.
4. V = {0}
If we multiply by a negative scalar it gives
a vector not in the first quadrant.
6. {(x1 , x2 ) ∈ R2 ∶ x2 = 0} ∪ {(x1 , x2 ), x1 = 0}
implies x-axis ∪ y-axis.
Proposition 12. Let {v1 , . . . , vk } ∈ Rn . Then V = span(v1 , . . . , vk ) is a subspace of Rn .
Proof. Recall span(v1 , . . . , vk ) = {c1 v1 + ⋯ + ck vk ∶ c1 , . . . , ck ∈ R}.
1. 0 = 0v1 + ⋅ + 0vk ∈ V .
2. Let u, w ∈ V . So u = c1 v1 + ⋯ + ck vk for some c1 , . . . , ck ∈ R; w = d1 v1 + ⋯ + dk vk
for some d1 , . . . , dk ∈ R. Then u + w = (c1 + d1 )v1 + ⋯(ck + dk )vk ∈ V .
3. Let c ∈ R. Let u ∈ V . Then u = c1 v1 + ⋯ + ck vk for some c1 , . . . , ck ∈ R. So
cu = (cc1 )v1 + ⋯ + (cck )vk ∈ V .
Proposition 13. Let A be an m × n matrix. Then the solution set Ax = 0, V = {x ∈ Rn ∶ Ax = 0}
is a subspace.
Proof. (1) 0 ∈ V because A0 = 0.
27
(2) If u, w ∈ V . Then A(u + w) = Au + Aw = 0 + 0 = 0. So u + w ∈ V .
(3) If c ∈ R, u ∈ V , then A(cu) = cAu = c0 = 0, so cu ∈ V .
Definition 32
Let V be a subset of Rn . Then the orthogonal complement of V , denoted V ⊥ , is {x ∈ Rn ∶
x ⋅ v = 0 for all v ∈ V }.
Example
⎡ ⎤
⎛⎢⎢ 1 ⎥⎥⎞
⎜⎢ ⎥⎟
⎜⎢ ⎥⎟
Let V = span ⎜⎢ 2 ⎥⎟ ⊆ R3 . Then V ⊥ = {x ∈ R3 ∶ x ⋅ v = 0 for all v ∈ V } = {(x1 , x2 , x3 )T ∶
⎜⎢ ⎥⎟
⎜⎢ ⎥⎟
⎝⎢⎢ 1 ⎥⎥⎠
⎣ ⎦
⎡ ⎤
⎛ ⎢⎢ 1 ⎥⎥⎞
⎜ ⎢ ⎥⎟
⎜ ⎢ ⎥⎟
x ⋅ ⎜t ⎢ 2 ⎥⎟ = 0 for all t ∈ R}.
⎜ ⎢ ⎥⎟
⎜ ⎢ ⎥⎟
⎝ ⎢⎢ 1 ⎥⎥⎠
⎣ ⎦
28
Tuesday, 18 February 2014
Definition 33
Orthogonal Complement of V ⊂ Rn is the set V ⊥ = {x ∈ Rn ∶ x ⋅ v = 0 for all v ∈ V }.
Example
⎡ ⎤
⎛⎢⎢ 1 ⎥⎥⎞
⎜⎢ ⎥⎟
⎜⎢ ⎥⎟
If V = span ⎜⎢ 2 ⎥⎟. Then V ⊥ is the set of vectors x which are perpendicular (orthog⎜⎢ ⎥⎟
⎜⎢ ⎥⎟
⎝⎢⎢ 1 ⎥⎥⎠
⎣ ⎦
⎡
⎤
⎢ 1 ⎥
⎢
⎥
⎢
⎥
⎢
⎥
onal) to all v ∈ V . Any vector v ∈ V is of the form v = tu, t ∈ R, u = ⎢ 2 ⎥. If x ∈ V ⊥ ,
⎢
⎥
⎢
⎥
⎢ 1 ⎥
⎢
⎥
⎣
⎦
then x ⋅ tu = 0 for all t ∈ R. That means x ⋅ u = 0. This means x1 + 2x2 + x3 = 0. Then,
if we want to write V ⊥ as a span, x1 = −2x2 − x3 . Then any x ∈ V ⊥ must be of the
⎛ −2x2 − x3 ⎞
⎛ −2 ⎞
⎛ −1 ⎞
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎟ for x2 , x3 ∈ R. Thus x = x2 ⎜ 1 ⎟ + x3 ⎜ 0 ⎟. Now we have
form x = ⎜
x
2
⎜
⎜
⎟
⎜
⎟
⎟
⎜
⎟
⎜
⎟
⎜
⎟
x3
⎝
⎠
⎝ 0 ⎠
⎝ 1 ⎠
⎤
⎤ ⎡
⎡
⎛⎢⎢ −2 ⎥⎥ ⎢⎢ −1 ⎥⎥⎞
⎥⎟
⎥ ⎢
⎜⎢
⎥⎟
⎥ ⎢
⎜⎢
V ⊥ = span ⎜⎢ 1 ⎥ , ⎢ 0 ⎥⎟.
⎥⎟
⎥ ⎢
⎜⎢
⎥ ⎢
⎜⎢
⎥⎟
⎢
⎥
⎢
⎝⎢ 0 ⎥ ⎢ 1 ⎥⎥⎠
⎦
⎦ ⎣
⎣
Example
⎡ ⎤ ⎡ ⎤
⎛⎢⎢ 1 ⎥⎥ ⎢⎢ 0 ⎥⎥⎞
⎜⎢ ⎥ ⎢ ⎥⎟
⎜⎢ ⎥ ⎢ ⎥⎟
Find the orthogonal complement of the plane V = span ⎜⎢ 0 ⎥ , ⎢ 1 ⎥⎟ = span(e1 , e2 ).
⎜⎢ ⎥ ⎢ ⎥⎟
⎜⎢ ⎥ ⎢ ⎥⎟
⎝⎢⎢ 0 ⎥⎥ ⎢⎢ 0 ⎥⎥⎠
⎣ ⎦ ⎣ ⎦
Any vector x ∈ V ⊥ must be perpendicular to every vector in V . Every vector in V is of
the form v = c1 e1 + c2 e2 , c1 , c2 ∈ R. In order for x ∈ V ⊥ it must satisfy x ⋅ (c1 e1 + c2 e2 ) = 0
for all c1 , c2 ∈ R. In order for this equation to be satisfied it suffices for
x ⋅ e1 = 0
and
29
x ⋅ e2 = 0.
⎧
⎧
⎪
⎪
⎪
⎪
⎪
⎪
⎪ x ⋅ e1
⎪ x1 = 0
⎨
Ô⇒ ⎨
⎪
⎪
⎪
⎪
⎪
⎪
x ⋅ e2
x = 0.
⎪
⎪
⎩
⎩ 2
To solve this system it is equivalent to solving Ax = 0 where
⎛ 1
⎜
⎜
A=⎜ 0
⎜
⎜
⎝ 0
0
1
0
⎛ x1 ⎞
0 ⎞
⎟
⎜
⎟
⎟
⎜
⎟
x = ⎜ x2 ⎟
0 ⎟
⎟
⎜
⎟
⎟
⎜
⎟
0 ⎠
⎝ x3 ⎠
The solution set of this system is {x ∈ R3 ∶ x3 ∈ R3 , x1 = x2 = v}.
Proposition 14. Let V ∈ Rn . Then V ⊥ is a subspace of Rn .
Proof. .
(1) 0 ⊂ V ⊥ be any vector in V for any v ∈ V . That is 0 ⋅ v = 0.
(2) If u, w ∈ V ⊥ , then for any v ∈ V we have (u + w)v = u ⋅ v + w ⋅ v; so (u + w) ∈ V ⊥ .
(3) If c ∈ R, u ∈ V ⊥ , then for any v ∈ V we see (cu) ⋅ v = c(u ⋅ v) = c(0) = 0. SO cu ∈ V ⊥ .
Remark. If U, V ⊆ Rn are subspaces, it is not necessarily the case that U ∪ V is a subspace.
U = {x1 − axis} ⊆ R2
V = {x2 − axis} ⊆ R2 .
Definition 34
If U, V ∈ Rn are subspaces, define their sum
as the set U + V = {x ∈ Rn ∶ for every u ∈ U and
v ∈ V }.
Example
If u = {x1 -axis} ⊆ R2 V = {x2 -axis} ⊆ R2
U + V = R2 .
Proposition 15. U + B is a subspace of Rn .
30
Proof. (1) 0 ∈ U + V because 0 = 0 + 0.
0= 0 +0
∈u
∈v
because U, V are subspaces.
(2) x, y ∈ U + V . Then x = u1 + v1 where u1 ∈ U and v1 ∈ V . y = u2 + v2 where u2 ∈ V
and v2 ∈ V . It follows x + y = (u1 + u2 ) + (v1 + v2 ) ∈ U + V .
∈U
∈V
Proposition 16. U + V is the smallest subspace that contains U and V .
Proof. Let W be any subspace that contains U, V and show that U + V ⊆ W . Let
x∈U +V.
x = u + v where u ∈ U ⊆ W, v ∈ V ⊆ W.
U, V ∈ W so u + v ∈ W . So that x ∈ W .
Proposition 17. If U, V ⊆ Rn are subspaces then U ∩ V is a subspace.
Proof. Exercise
Definition 35
If U and V are subspaces then they are said to be orthogonal if u − v = 0 for all u ∈ v.
§3.2 Four Fundamental Subspaces
Definition 36
Let A be a m × n matrix. Then the null space of A is the set of all Ax = 0.
N (A) = {x ∈ Rn ∶ Ax = 0}.
The column space of A is the subspace of Rm spanned by the columns of A.
C(A) = span(a1 , . . . , an ) ⊆ Rm
where
Proposition 18. C(A) = {b ∈ Rn ∶ Ax = b is constant}.
31
a1 , . . . , an ∈ Rn
Proof. x ∈ C(A) ⇐⇒ x = c1 a1 + ⋯ + cn an for some
⎛ c1 ⎞
⎛ ∣ ⋯
⎜
⎟
⎜
⎜
⎟
⎜
x = ⎜ ⋮ ⎟. So that Ax = b where A = ⎜ a1 ⋯
⎜
⎟
⎜
⎜
⎟
⎜
⎝ cn ⎠
⎝ ∣ ⋯
32
scalars c1 , . . . , cn ∈ R ⇐⇒ there exists
∣ ⎞
⎟
⎟
⇐⇒ Ax = b is consistent.
an ⎟
⎟
⎟
∣ ⎠
Thursday, 20 February 2014
Given AB nonsingular we want to prove that A and B are nonsingular.
To prove that B is nonsingular, consider Bx = 0 (show that the only solution is x = 0).
From this we have that ABx = 0 (and again we have that x = 0).
Let u be a solution to this equation, then Bu = 0 and it follows that ABu = 0 and thus
we have u = 0, this tells us that B is nonsingular. Now we know AB and B are both
nonsingular, now we want to show that A is nonsingular. We will show instead that A is
invertible by showing it is the product of two invertible matrices. We have A = ABB −1 .
Therefore A is invertible.
§3.3 Linear Independent and Bases
Definition 37 (Informal Definition)
Let V ⊆ Rn be a subspace. A finite set of vectors {v1 , v2 , . . . , vk } ⊆ V is called a basis for V is
every vector v ∈ V can be written uniquely as a linear combination of v1 , v2 , . . . , vk .
Example
⎛ 1 ⎞
⎛ 0 ⎞
⎟.
1. {e1 , e2 } ⊆ R2 ; e1 = ⎜ ⎟ and e2 = ⎜
⎝ 0 ⎠
⎝ 1 ⎠
⎛ x1 ⎞
⎟ ∈ R2 .
Claim is a basis for R2 because any vector x = ⎜
⎝ x2 ⎠
1 Can be written as a linear combination of e1 , e2 ;
⎛ 1 ⎞
⎛ 0 ⎞
x = x1 e1 + x2 e2 = x1 ⎜ ⎟ + x2 ⎜ ⎟ .
⎝ 0 ⎠
⎝ 1 ⎠
2 This decomposition is unique in the sense that no other coefficients for e1 , e2
will give back x.
33
⎧
⎫
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎛
⎞
⎛
⎞
⎪
⎪
−1 ⎪
⎪ 1
⎪
⎟, ⎜
⎟⎬ is a basis for R2 . How can we decide whether a vector
2. Claim that ⎨⎜
⎪
⎪
⎪
⎝ 1 ⎠ ⎝ 1 ⎠⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
´¹
¹
¹
¹
¸
¹
¹
¹
¶
´¹
¹
¹
¹
¹
¹
¸
¹
¹
¹
¹
¹
¹
¶
⎪
⎪
⎪
⎪
u
u
2
⎩ 1
⎭
b can be written as a linear combination of u1 , u2 ? To decide this we figure out if
⎛ ∣
∣ ⎞
⎜
⎟
⎜
⎟
Ax = b has a solution where A = ⎜ u1 u2 ⎟. To decide whether a given solution
⎜
⎟
⎜
⎟
∣ ⎠
⎝ ∣
is unique we consider the solution set of Ax = b.
Definition 38
A set of vectors {v1 , . . . , vk } is said to be linearly independent if the equation x1 v1 +⋯+xk vk = 0
has only the trivial solution. If the equation has a non-trivial solution, then {v1 , . . . , vk } is said to
be linearly dependent .
Example
⎧
⎪
⎪
⎛
⎪
⎪
⎪
⎪
⎜
⎪
⎪
⎜
⎪
⎪
⎪⎜
1. Decide whether the set ⎨⎜
⎜
⎪
⎜
⎪
⎪
⎜
⎪
⎪
⎜
⎪
⎪
⎪
⎪
⎝
⎪
⎪
⎩
1 ⎞ ⎛
⎟ ⎜
⎜
0 ⎟
⎟ ⎜
⎟,⎜
⎟ ⎜
⎜
1 ⎟
⎟ ⎜
⎟ ⎜
2 ⎠ ⎝
2 ⎞ ⎛
⎟ ⎜
⎜
1 ⎟
⎟ ⎜
⎟,⎜
⎟ ⎜
⎜
1 ⎟
⎟ ⎜
⎟ ⎜
1 ⎠ ⎝
⎫
⎪
1 ⎞⎪
⎪
⎪
⎪
⎟⎪
⎪
⎪
⎟
⎪
1 ⎟⎪
⎟⎪
is linearly independent.
⎟⎬
⎪
0 ⎟
⎪
⎟⎪
⎪
⎟⎪
⎪
⎪
⎪
⎠
⎪
−1 ⎪
⎪
⎭
To decide this we look at the equation
⎛
⎜
⎜
⎜
x1 ⎜
⎜
⎜
⎜
⎜
⎝
⎛
1 ⎞
⎜
⎟
⎜
0 ⎟
⎜
⎟
⎟ + x2 ⎜
⎟
⎜
⎜
1 ⎟
⎜
⎟
⎟
⎜
⎝
2 ⎠
⎛
2 ⎞
⎜
⎟
⎜
1 ⎟
⎜
⎟
⎟ + x3 ⎜
⎟
⎜
⎜
1 ⎟
⎜
⎟
⎜
⎟
⎝
1 ⎠
⎛
⎜
⎜
⎜
We consider instead the matrix equation ⎜
⎜
⎜
⎜
⎜
⎝
1 ⎞ ⎛
⎟ ⎜
⎜
1 ⎟
⎟ ⎜
⎟=⎜
⎟ ⎜
⎜
0 ⎟
⎟ ⎜
⎟ ⎜
−1 ⎠ ⎝
1
2
0
1
1
1
0 ⎞
⎟
0 ⎟
⎟
⎟.
⎟
0 ⎟
⎟
⎟
0 ⎠
⎛ 0 ⎞
1 ⎞
⎟ ⎛ x1 ⎞ ⎜ ⎟
⎟ ⎜
1 ⎟
0 ⎟
⎟⎜
⎟
⎟ ⎜
⎟⎜
⎜ ⎟ and find
⎟
=
x
⎟⎜
⎜
⎟
2 ⎟
⎜
⎟
⎟ ⎜
0 ⎟
⎟⎜
⎜ 0 ⎟
⎟ ⎝ x3 ⎠ ⎜ ⎟
⎝ 0 ⎠
−1 ⎠
2 1
its solution set. If it is {D} then the set is linearly independent.
34
2. Suppose u, v ∈ Rn and {u, v} is linearly dependent.
Then there exists scalars c1 , c2 ∈ R so that c1 u + c2 v = 0 at least one of c1 , c2 being
nonzero. Suppose c1 ≠ 0. Then we have u = − cc21 v which implies u, v are parallel.
3. Suppose {u, v, w} ⊆ Rn are linearly dependent.
There there exists scalars c1 , c2 , c3 ∈ R so that c1 u + c2 v + c3 w = 0; and without loss
of generality let c1 ≠ 0. Then we have u = − cc21 v −
35
c3
w
c1
and thus u ∈ span(v, w).
Friday, 21 February 2014
Definition 39
A set of vectors {v1 , v2 , . . . , vk } ⊆ Rn is linearly independent if x1 v1 + x2 v2 + ⋯ + xk vk = 0 has
only the trivial solution x1 = 0, . . . , xk = 0.
Example
1. If v, w are parallel vectors then {v, w} is a linearly dependent set. That is, w = kv
for some k ≠ 0. Thus we have −kv + w = 0.
2. If v, w, u all lie on the same plane through the origin, i.e., if u ∈ span(v, w),
then u = c1 v + c2 w for some c1 , c2 ∈ R, then u − c1 v − c2 w = 0. So the equation
x1 u+x2 v+x3 w = 0 has a non-trivial solution x1 = 1, x2 = −c1 , x3 = −c2 . So {u, v, w}
is linearly dependent.
Figure 4:
3. Suppose the set {v1 , . . . , vk } contains 0. This et is then linearly dependent because
the equation x1 v1 + ⋯ + xk vk = 0 has a non-trivial solution. Without loss of
generality suppose v1 = 0, then letting x1 = 5, x2 = 0, . . . , xk = 0 makes the equation
above true and is a non-trivial solution.
Proposition 19. Let {v1 , . . . , vk } ∈ Rn be a linearly independent set. Then any v ∈ span(v1 , . . . , vk )
can be written uniquely as a linear combination of v1 , . . . , vk .
Proof. Let v ∈ span(v1 , . . . , v2 ). Then v = c1 v1 + ⋯ + ck vk for some c1 , . . . , ck ∈ R.
Suppose there is a (possibly) different linear combination that gives v such as d1 v1 +
36
⋯ + dk vk for some di ∈ R for 1 ≤ i ≤ k.
0=v−v
= c1 v1 + ⋯ + ck vk − d1 v1 − ⋯ − dk vk
= (c1 − d1 )v1 + ⋯ + (ck − dk )vk
=0
=0
⇒ c1 = d1 , . . . , ck = dk
because of {v1 , . . . , vk } being linearly independent.
Definition 40
Let V be a subspace of Rn . Then the set of vectors {v1 , . . . , vk } ⊆ V is a basis if
1) span(v1 , . . . , vk ) = V
2) {v1 , . . . , vk } is linearly independent.
Example
Consider the set
⎧
⎫
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎛
⎞
⎛
⎞
⎪
⎪
−1
5
⎪
⎪
⎟, ⎜
⎟⎬ ⊆ Rn .
B = ⎨⎜
⎪
⎪
⎪
⎝ 1 ⎠ ⎝ 2 ⎠⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎩ u1
u2 ⎭
We claim that B is a basis for R2 .
To prove this we must show span(u1 , u2 ) = R2 .
Let b ∈ R2 and show that b ∈ span(u1 , u2 ). Show there exists scalars c1 and c2 so
that b = c1 u1
⎛ ∣
∣
⎜
⎜
A = ⎜ u1 u2
⎜
⎜
∣
⎝ ∣
+ c2 u2 , (i.e., must show that there exists a solution to Ax = b, where
⎞
⎟
⎟
⎟.
⎟
⎟
⎠
37
Consider
⎛ −1 5 ⎞ ⎛ x1 ⎞ ⎛ b1 ⎞
⎜
⎟⎜
⎟=⎜
⎟
⎝ 1 2 ⎠ ⎝ x2 ⎠ ⎝ b2 ⎠
⎛ −1 5
⇒⎜
⎝ 1 2
b1 ⎞
⎟
b2 ⎠
⎛ 1
→⎜
⎝ 0
⎞ ⎛ 1
⎟→⎜
c 2 − b1 ⎠ ⎝ 0
−1
7
−b1
0
1
−b1 + 57 (b2 + b1 ) ⎞
⎟.
1
⎠
(b
+
b
)
1
2
7
The equation Ax = b has a solution because rank(A) = 2. The solution is unique because
⎛ 1 ⎞
=
there are no free variables. Suppose b = ⎜ ⎟. Then x1 = −(−1) + 57 (−1 − 1) = −1 − 10
7
⎝ 1 ⎠
− 37 ; x2 = 17 (−1 − 1) = − 27 . Thus we have found a unique solution to b = x1 u1 + x2 u2 , i.e.,
⎛ −1 ⎞
⎛ 5 ⎞
⎛ −1 ⎞
⎜
⎟ = − 73 ⎜
⎟ − 27 ⎜ ⎟.
⎝ 1 ⎠
⎝ 1 ⎠
⎝ 2 ⎠
Definition 41
If B = {v1 , . . . , vk } is a basis for a subspace V , then any vector v ∈ V has a unique representation
as a linear combination of v1 , . . . , vk , v = c1 v1 + ⋯ + ck vk . The scalars c1 , . . . , ck ∈ R are called
the coordinates of v with respect to B. The B-coordinate representation of v is the vector
⎤
⎡
⎢ c1 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
[v]B = ⎢ ⋮ ⎥.
⎥
⎢
⎥
⎢
⎢ c ⎥
⎢ k ⎥
⎦
⎣
⎛ −1 ⎞
⎛ 5 ⎞
⎛ −1 ⎞
⎟, u2 = ⎜ ⎟,b = ⎜
⎟.
In our example b = − 73 u1 − 72 u2 ; u1 = ⎜
⎝ 1 ⎠
⎝ 2 ⎠
⎝ −1 ⎠
⎛ 1 ⎞
⎛ 0 ⎞
e1 = ⎜ ⎟ , e2 = ⎜ ⎟ → b = −e1 − e2
⎝ 0 ⎠
⎝ 1 ⎠
38
Figure 5:
Professor Notes
§3.3 Linear Independence and Bases
Definition 42 (Informal Definition)
Let V ⊆ Rn be s subspace.
A finite set of vectors {v1 , . . . , vk } ⊆ V is called a basis for V if every vector v ∈ V can be
decomposed uniquely as a linear combination of v1 , . . . , vk .
Definition 43
A set of vectors {v1 , . . . , vk } ⊆ Rn is said to be linearly independent if the equation
x1 v1 + ⋯ + xk vk = 0
has only the trivial solution, x1 = ⋅ ⋅ ⋅ = xk = 0. If x1 v1 + ⋯ + xk vk = 0 has a non-trivial solution, the
set {v1 , . . . , vk } is said to be linearly dependent .
Example
1. Decide whether
⎧
⎡
⎪
⎢
⎪
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎨v1 = ⎢⎢
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎪
⎢
⎪
⎣
⎩
is linearly independent.
⎤
⎡
⎢
1 ⎥⎥
⎢
⎥
⎢
⎥
⎢
0 ⎥
⎢
⎥ , v2 = ⎢
⎥
⎢
⎢
1 ⎥⎥
⎢
⎥
⎢
⎥
⎢
⎢
2 ⎥⎦
⎣
⎤
⎡
⎢
2 ⎥⎥
⎢
⎥
⎢
⎥
⎢
1 ⎥
⎢
⎥ , v3 = ⎢
⎥
⎢
⎢
1 ⎥⎥
⎢
⎥
⎢
⎥
⎢
⎢
1 ⎥⎦
⎣
⎤⎫
⎪
1 ⎥⎥⎪
⎪
⎪
⎪
⎥⎪
⎪
⎪
⎥
⎪
1 ⎥⎪
⎥⎪
⎥⎬
⎪
0 ⎥⎥⎪
⎪
⎪
⎥⎪
⎪
⎥⎪
⎪
⎪
−1 ⎥⎦⎪
⎪
⎭
2. Suppose {u, v} is linearly dependent. What can we conclude?
39
3. Suppose {u, v, w is linearly dependent. What can we conclude?
4. Suppose {v1 , . . . , vk } contains the zero vector. What can we conclude?
Proposition 20. If {v1 , . . . , vk } is linearly independent then any vector in span(v1 , . . . , vk ) has a
unique expression as a linear combination of v1 , . . . , vk .
Proof. Let v ∈ span(v1 , . . . , vk ). Then
v = c1 v 1 + ⋯ + ck v k
for some c1 , . . . , ck ∈ R. Suppose there is some (possibly) different expression
v = d1 v1 + ⋯dk vk .
Then
0 = v − v = (c1 − d1 )v1 + ⋯ + (ck − dk )vk ⇒ c1 − d1 = ⋯ = ck − dk = 0
because {v1 , . . . , vk } is linearly independent. So we have c1 = d1 , ⋯, ck = dk implies the
expression for v as a linear combination of v1 , . . . , vk was unique.
Definition 44
Let V ⊆ Rn be a subspace. Then a finite set {v1 , . . . , vk } ⊆ V is called a basis for V if
1) span(v1 , . . . , vk ) = V
2) {v1 , . . . , vk } is linearly independent.
Proposition 21. Suppose {v1 , . . . , vk } ⊆ Rn is a linearly independent set, and suppose v ∈ Rn .
Then {v, v1 , . . . , vk } is linearly independent if and only if v ∈ span(v1 , . . . , vk ).
Proof. We will prove the contrapositive statement:
“{v, v1 , . . . , vk } is linearly dependent if and only if v ∈ span(v1 , . . . , vk ).”
1. First prove {v, v1 , . . . , vk } is linearly dependent implies v ∈ span(v1 , . . . , vk ).
Suppose {v, v1 , . . . , vk } is linearly dependent. Then there exists a non-trivial so-
40
lution to the equation:
0 = x1 v1 + ⋯ + xk vk + xk+1 v,
i.e., there exists solutions c1 , . . . , ck , c not all zero such that 0 = c1 v1 + ⋯ + ck vk + cv.
Claim c ≠ 0.
Proof. If c − 0 then that means at least one of the other scalars, say ci ,
must be nonzero. But then
= c1 v1 + ⋯ + ck vk
cv
0 = c1 v1 + ⋯ck vk + which implies c1 , . . . , ck is a non-trivial solution to the equation 0 = x1 v1 +
⋯ + xk vk , which contradicts the linear independence of {v1 , . . . , vk }. So
our claim is proved.
Therefore
v=−
c1
ck
v1 + ⋯ + − vk ⇒ v ∈ span(v1 , . . . , vk ).
c
c
2. Next we prove v ∈ span(v1 , . . . , vk ) implies {v, v1 , . . . , vk } is linearly dependent.
Suppose v ∈ span(v1 , . . . , vk ). Then v = c1 v1 + ⋯ + ck vk for some scalars c1 , . . . , ck ∈
R.
⇒ 0 = −1v + c1 v1 + ⋯ + ck vk
⇒ 0 = c1 v1 + ⋯ + xk vk + xk+1 v
has a solution x1 = c1 , . . . , xk = ck , xk+1 = −1 which is non-trivial because xk+1 = −1.
This implies {v1 , . . . , vk , v} is linearly dependent.
Homework: 14, 15, 10, 8, 19
Intuition to Last Result:
1. A basis is the smallest linearly independent set of vectors in V that spans V .
41
2. A basis is the biggest spanning set that is linearly independent.
That is, if B = {v1 , . . . , vk } is a basis for V , adding another vectors to B will break
linear independent, while subtracting a vector from B will break the fact that it
spans V .
Theorem 45. Every non-trivial subspace V ⊆ Rn has a basis. (By non-trivial subspace we mean
V ≠ {0}).
Proof. V ≠ {0} means there exists v1 ≠ 0 in V . If v1 spans V then {v1 } is a linearly
independent spanning set and it follows that {v1 } is a basis for V . Else there exists
v2 ∈ V such that v1 ∈/ span(v1 ). Since v2 ∈/ span(v1 ), we have {v1 , v2 } is linearly
independent. If span(v1 , v2 ) = V then {v1 , v2 } is a basis. Else there exists v3 ∈ V
such that v3 ∈/ span(v1 , v2 , v3 ). Since span(v1 , v2 , v3 ) = V then {v1 , v2 , v3 } is a basis.
Continue this process until we have a basis for V . The only question left is why does
the process have to terminate?
Claim The process must terminate in at most n steps.
Proof. Otherwise, at the (n+1)-step we choose vector vn+1 and span(v1 , . . . , vn )
where {v1 , . . . , vn } are linearly independent. However this implies {v, . . . , vn , vn+1 }
is linearly independent.
Homework: Prove any set of n + 1 vectors in Rn must be linearly dependent.
Thus we have a contradiction if the process ran more than n steps.
Lemma 46
If {v1 , . . . , vk } ⊆ V ⊆ Rn is linearly independent and {w1 , . . . , wm } spans V then k ≤ m.
42
Proof. span(w1 , . . . , wm ) = V implies
v1 = c11 w1 + ⋯ + c1m wm
⋮
vk = ck1 w1 + ⋯ + ckm wm
If
⎡
⎢ ∣
⎢
⎢
⎢
B = ⎢ w1
⎢
⎢
⎢ ∣
⎢
⎣
then
∣
⋯
wm
∣
⎡
⎢ c11
⎢
⎢
⎢
v1 = B ⎢ ⋮
⎢
⎢
⎢ c
⎢ 1m
⎣
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
⋮
⎡
⎢ ck1
⎢
⎢
⎢
vk = B ⎢ ⋮
⎢
⎢
⎢ c
⎢ km
⎣
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
So if
A = [ v1 ⋯vk ]
then
⎡
⎛ ⎢⎢ c11
⎜ ⎢
⎜ ⎢
A = ⎜B ⎢ ⋮
⎜ ⎢
⎜ ⎢
⎝ ⎢⎢ c1m
⎣
⎡
⎤
⎥
⎢ ck1
⎢
⎥
⎢
⎥
⎢
⎥
⎥ ⋯B ⎢ ⋮
⎢
⎥
⎢
⎥
⎢ c
⎥
⎢ km
⎥
⎦
⎣
⎤
⎤
⎡
⎢ c11 ⋯ ck1 ⎥
⎥⎞
⎥
⎢
⎥
⎥
⎢
⎥⎟
⎥
⎢
⎥⎟
⎥
⎥⎟ = B ⎢ ⋮
⋱
⋮
⎥
⎢
⎥⎟
⎥
⎢
⎥⎟
⎢ c
⎥⎠
⎥
⎥
⎢ 1m ⋯ ckm ⎥
⎣
⎦
⎦
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
∶=C←m×k
We claim Cx = 0 has only the trivial solution. Observe that if u ∈ Rk is a solution then
Cu = 0
⇒ BCu
=0
⇒ Au
=0
⇒u
=0
43
since {v1 , . . . , vk } linearly independent. Since Cx = 0 has only the trivial solution to
RREF of C has only pivot columns implies the number of columns is less than or equal
to the number of rows and it follows that k ≤ m.
Theorem 47. Every basis of a non-trivial subspace has the same size.
Proof. Let B1 = {v1 , . . . , vk }, B2 = {w1 , . . . , wl } be two bases of V .
Claim 1
k ≤ l. Since B1 , is a basis, it is linearly independent. Since B2 is a basis, it
spans V . By the above Lemma we have k ≤ l.
Claim 2
k ≥ l. Since B1 is a basis, it spans V . Since B2 is a basis it is linearly
independent. By the above Lemma l ≤ k.
We conclude that k = l.
Proposition 22. If V, W ⊆ Rm are subspaces and V ⊆ W with dim V = dim W , then V = W .
Proof. Let {v1 , . . . , vk } be a basis for V (so that dim V = dim W = k). If V ⊊ W then
there exists w ∈ W /span(v1 , . . . , vk ). By the above Lemma, {v1 , . . . , vk , w} ⊆ W is
linearly independent, which implies dim W ≥ k + 1, a contradiction.
We know bases always exist for non-trivial subspaces. How do we find them?
Proposition 23. Let A be an n × n matrix. Then A is nonsingular if and only if its columns form
a basis for Rn .
Proof. (⇒) A nonsingular implies Ax = 0 has a unique solution and it follows that if
A = [v1 ⋯vk ], vi ∈ Rn then 0 = Ax = x1 v1 + ⋯ + xn vn has only the trivial solution.
Thus {v1 , . . . , vn } is linearly independent.
Further if A is nonsingular we have Ax = b has a solution for all b ∈ Rn and thus
for every b ∈ Rn there exists scalars c1 , . . . , cn ∈ Rn such that c1 v1 + ⋯ + cn vn = b.
44
Thus we have span(v1 , . . . , vk ) = Rn . Therefore {v1 , . . . , vn } is a basis for Rn .
(⇐) Left as an exercise.
Remark. So this proposition gives us
1. Given a potential basis a way to check it is a basis. (Put the vectors into columns
of a matrix and check that the matrix is nonsingular).
2. A huge collection of bases for Rn (Any matrix know to be nonsingular has columns
that form a basis).
Next How to find a bases for subspaces?
Example
⎧
⎫
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎛
⎞
⎛
⎞
⎪
⎪
−1
5
⎪
⎪
⎟, ⎜ ⎟⎬.
Consider B = ⎨⎜
⎪
⎪
⎪
⎝ 1 ⎠ ⎝ 2 ⎠⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
´¹
¹
¹
¹
¹
¹
¸
¹
¹
¹
¹
¹
¹
¶
´¹
¹
¹
¹
¸
¹
¹
¹
¶
⎪
⎪
⎪
=u2 ⎪
⎩ =u1
⎭
1. Check that B is a basis for R2 .
⎛ −1 ⎞
⎟ Express v as a linear combination of u1 , u2 .
2. Let v = ⎜
⎝ −1 ⎠
Definition 48
Let B = {u1 , . . . , uk } be a basis for a subspace V ⊆ Rn . For each v ∈ V there is a unique set of
scalars c1 , . . . , ck ∈ R determined by v called the coordinates of v with respect to B so that
v = c1 u1 + ⋯ + ck uk .
⎡
⎤
⎡
⎤
⎢ c1 ⎥
⎢ c1 ⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
The B-coordinate representation of v is the vector ⎢ ⋮ ⎥, sometimes denoted [v]B = ⎢ ⋮ ⎥.
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢ c ⎥
⎢ c ⎥
⎢ k ⎥
⎢ k ⎥
⎣
⎦
⎣
⎦
Example
45
Suppose V = span(v1 , v2 , v3 , v4 ) with
⎛ 1 ⎞
⎜ ⎟
⎜ ⎟
v1 = ⎜ 1 ⎟ ,
⎜ ⎟
⎜ ⎟
⎝ 2 ⎠
⎛ 2 ⎞
⎜
⎟
⎜
⎟
v2 = ⎜ 2 ⎟ ,
⎜
⎟
⎜
⎟
⎝ 4 ⎠
⎛ 0 ⎞
⎜ ⎟
⎜ ⎟
v3 = ⎜ 1 ⎟ ,
⎜ ⎟
⎜ ⎟
⎝ 1 ⎠
⎛ 3 ⎞
⎜ ⎟
⎜ ⎟
v4 = ⎜ 4 ⎟ .
⎜ ⎟
⎜ ⎟
⎝ 7 ⎠
Find a basis for V .
Lemma 49
Let {v1 , . . . , vk } ⊆ Rn be a subset of vectors in Rn . If v1 is a linear combination of v2 , . . . , vk then
span(v1 , . . . , vk ) = span(v2 , . . . , vk ).
Proof. Clearly
span(v2 , . . . , vk ) ⊆ span(v1 , . . . , vk ).
On the other hand, let v ∈ span(v1 , . . . , vk ). Then v = c1 v1 +⋯+ck vk for some c1 , . . . , ck ∈
R. By assumption, v1 = d2 v2 + ⋯ + dk vk for some d2 , . . . , dk ∈ R. Then
v = c1 (d2 v2 +⋯+dk vk )+c2 v2 +⋯+ck vk = (c1 d2 +c2 )v2 +(c1 d3 +c3 )v3 +⋯+(c1 dk +ck )vk .
Thus v ∈ span(v2 , . . . , vk ).
Back to the Exercise
By the Lemma, the goal will be to remove vectors from the set {v1 , v2 , v3 , v4 } which are
linear combinations of the others until we get down to a set that spans and is linearly
independent. Let
A = ( v1
v2
v3
⎛ 1
⎜
⎜
v4 ) = ⎜
⎜ 1
⎜
⎝ 2
2
0
2
1
4
1
3 ⎞
⎟ RREF
⎟
↝
4 ⎟
⎟
⎟
7 ⎠
⎛ 1
⎜
⎜
⎜ 0
⎜
⎜
⎝ 0
2
0
0
1
0
0
3 ⎞
⎟
⎟
=∶ R
1 ⎟
⎟
⎟
0 ⎠
if u is a solution to Rx = 0 then u is a solution to Ax = 0. (Notice these solutions tell
us of linear dependence among the columns of R or U .) The free variable columns of A
(or R) are linear combinations of the pivot columns: solution to Rx = 0:
x1 + 2x2 + 3x4 = 0
x3 + x4 = 0
Ô⇒
46
x1 = −2x2 − 3x4
x3 = −x4
solution set
= {(−2x2 , −3x4 , x2 , −x4 , x4 ) ∶ x2 , x4 ∈ R} = {x2 (−2, 1, 0, 0) + x4 (−3, 0, −1, 1) ∶ x2 , x4 ∈ R}
if x2 = 1 and x4 = 0 then this tells us −2v1 + v2 = 0 if x2 = 0 and x4 = 0 then this tells
us −3v1 − v2 + v4 = 0. So span(v1 , v2 , v3 , v4 ) = span(v1 , v3 ). Moreover, {v1 , v3 } are
linearly independent. (Why?) So {v1 , v2 } is a basis.
47
Tuesday, 25 February 2014
Definition 50
If V ⊆ Rn is a non-trivial subspace (V ≠ {0}) then its dimension , dim V , is the size of one of its
bases.
Lemma 51
If {v1 , . . . , vk } ⊆ V ⊆ Rn is a linearly independent set and {w1 , . . . , wm } ⊆ V spans V then k ≤ m.
Proof. Since span(w1 , . . . , wm ) = V we have
v1 = c11 w1 + ⋯ + c1m wm
⋮
vk = ck1 w1 + ⋯ + ckm wm
⎛ ∣
⎜
⎜
If B = ⎜ w1
⎜
⎜
⎝ ∣
⋯
∣
⋯
wm
⋯
∣
⎞
⎟
⎟
⎟ an n × m matrix, then
⎟
⎟
⎠
⎛ c11
⎜
⎜
v1 = B ⎜ ⋮
⎜
⎜
⎝ c1m
⎛ ck1
⎞
⎜
⎟
⎜
⎟
⎟ , . . . , vk = B ⎜ ⋮
⎜
⎟
⎜
⎟
⎝ ckm
⎠
⎛ c11 ⋯ c1m
⎜
⎜
Let A = ( v1 ⋯ vk ), and C = ⎜ ⋮
⋱
⋮
⎜
⎜
⎝ ck1 ⋯ ckm
Cx = 0 has only the trivial solution. Note that C is
⎞
⎟
⎟
⎟.
⎟
⎟
⎠
⎞
⎟
⎟
⎟, then A = BC. We claim that
⎟
⎟
⎠
an m × k matrix. Given the claim,
this says
rankC = k.
But we know that rankC ≤ m for any matrix C. Let u ∈ Rk be a solution to Cx = 0.
Then Cu = 0. It follows that BCu = B0 = 0 thus Au = 0. Since {v1 , . . . , vk } is linearly
independent, Ax = 0 has only the trivial solution. So u = 0.
48
Theorem 52. Let {v1 , . . . , vk } and {w1 , . . . , wl } be two bases for a subspaces V ⊆ Rn . Then k = l.
Proof. We will show that k ≤ l and that l ≤ k, thus showing that k = l.
• Claim 1: k ≤ l
Since {v1 , . . . , vk } is a basis it is linearly independent. Since {w1 , . . . , wl } is a
basis, it spans V . Thus by the above lemma we have k ≤ l.
• Claim 2: l ≤ k
Same argument, switching roles of {v1 , . . . , vk } and {w1 , . . . , wl }.
Proposition 24. If V, W ⊆ Rn are subspaces and suppose V ⊆ W and dim V = dim W . Then
V = W.
Proof. By way of contradiction suppose there exists w ∈ W /V . Let {v1 , . . . , vk } ⊆
V be a basis for V (so dim V = k = dim W ). Consider {v1 , . . . , vk , w}. This set is
linearly independent since w ∈/ span(v1 , . . . , vk ) = V . This implies that dim W ≥ k + 1, a
contradiction.
Remark (Side Remark). We know that bases exist for all non-trivial subspaces. How do we find
them? We start with the case when V = Rn .
Proposition 25. Let A be an n × n matrix. Then A is nonsingular if and only ifs columns form a
basis for Rn .
Remark. 1. Given {v1 , . . . , vn } ⊂ Rn to check if it is a basis we check if A = (v1 , . . . , vk ) is nonsingular.
2. Any nonsingular matrix’s columns gives us a basis for Rn .
Proof of above Proposition. (⇒) Assume A is nonsingular. Prove its columns form a
basis. Let A = ( v1
⋯
vk ).
49
Claim 1
{v1 , . . . , vk } are linearly independent. Since A is nonsingular
Ax = 0
has only the trivial solution. If we consider the equation
(∗)
x1 v1 + ⋯ + xk vk = 0
the equivalent matrix equation is Ax = 0. So (*) has only the trivial
solution.
Claim 2:
span(v1 , . . . , vk ) = Rn . A nonsingular tells us for every b ∈ Rn there exists
a solution to Ax = b, i.e., there exists scalars c1 , . . . , cn ∈ R such that
⎛ c1 ⎞
⎜
⎟
⎜
⎟
A ⎜ ⋮ ⎟ = b.
⎜
⎟
⎜
⎟
⎝ cn ⎠
⎛ c1 ⎞
⎜
⎟
⎜
⎟
A⎜ ⋮ ⎟ = b
⎜
⎟
⎜
⎟
⎝ cn ⎠
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
=c1 v1 +⋯+cn vn
which says span(v1 , . . . , vk ) = R .
n
(⇐) Exercise (same argument backwards)
50
Definition 53
⎛
⎜
⎜
⎜
⎜
⎛ 1 ⎞
⎜
⎜
⎜
⎟
⎜
⎜ 0 ⎟
⎜
⎜
⎟
n
⎟ ∈ R , ei = ⎜
⎜
Let e1 = ⎜
⎜
⎟
⎜
⎜ ⋮ ⎟
⎜
⎜
⎟
⎜
⎜
⎟
⎜
⎜
⎝ 0 ⎠
⎜
⎜
⎜
⎜
⎝
0 ⎞
⎟
⋮ ⎟
⎟
⎟
⎛ 0 ⎞
⎟
⎜
⎟
0 ⎟
⎟
⎜ ⋮ ⎟
⎟
⎜
⎟
⎟
n
⎟ ∈ Rn . Then {e1 , . . . , en } ⊆ Rn is called the
∈ R , en = ⎜
1 ⎟
⎜
⎟
⎟
⎜
⎟
⎟
⎜ 0 ⎟
⎟
⎜
⎟
0 ⎟
⎟
⎝ 1 ⎠
⎟
⋮ ⎟
⎟
⎟
0 ⎠
standard basis of Rn . (It is a basis because I = ( e1
⋯
en ) is nonsingular).
Example
Suppose V = span(v1 , v2 , v3 , v4 ) ⊆ R3 where
⎛ 1 ⎞
⎜ ⎟
⎜ ⎟
v1 = ⎜ 1 ⎟ ,
⎜ ⎟
⎜ ⎟
⎝ 2 ⎠
⎛ 0 ⎞
⎜ ⎟
⎜ ⎟
v3 = ⎜ 1 ⎟ ,
⎜ ⎟
⎜ ⎟
⎝ 1 ⎠
⎛ 2 ⎞
⎟
⎜
⎟
⎜
v2 = ⎜ 2 ⎟ ,
⎟
⎜
⎟
⎜
⎝ 4 ⎠
⎛ 3 ⎞
⎜ ⎟
⎜ ⎟
v4 = ⎜ 4 ⎟ .
⎜ ⎟
⎜ ⎟
⎝ 7 ⎠
Find a basis for V . We know {v1 , . . . , v4 } is linearly dependent.
Lemma 54
Let {v1 , . . . , vk } ⊆ Rn . If v1 is a linear combination of v2 , . . . , vk then
span(v1 , . . . , vk ) = span(v2 , . . . , vk ).
Proof. Exercise span(v2 , . . . , vk ) ⊆ span(v1 , . . . , vk ).
We now prove span(v1 , . . . , vk ) ⊆ span(v2 , . . . , vk ). Let w ∈ span(v1 , . . . , vk ). Then
there exists c1 , . . . , ck ∈ R such that w = c1 v1 + ⋯ + ck vk . There exists d2 , . . . , dk ∈ R such
that v1 = d2 v2 + ⋯ + dk vk . Then we have
w = c1 (d2 v2 + ⋯ + dk vk ) + c2 v2 + ⋯ + ck vk = (c1 d2 + c2 )v2 + ⋯ + (c1 dk + ck )vk ,
thus, w ∈ span(v2 , . . . , vk ).
51
Thursday, 27 February 2014
Example
V = span(v1 , v2 , v3 , v4 ) ⊆ R3 where
⎛ 1 ⎞
⎜ ⎟
⎜ ⎟
v1 = ⎜ 1 ⎟ ,
⎜ ⎟
⎜ ⎟
⎝ 2 ⎠
⎛ 2 ⎞
⎜
⎟
⎜
⎟
v2 = ⎜ 2 ⎟ ,
⎜
⎟
⎜
⎟
⎝ 4 ⎠
⎛ 0 ⎞
⎜ ⎟
⎜ ⎟
v3 = ⎜ 1 ⎟ ,
⎜ ⎟
⎜ ⎟
⎝ 1 ⎠
We will need the following Lemma to do this example:
Lemma 55
Let
{v1 , . . . , vk } ⊆ Rn .
If v1 is a linear combination of v2 , . . . , vk then
span(v1 , . . . , vk ) = span(v2 , . . . , vk ).
Back to the Example
Goal to Example: Find a basis for V .
It is obvious that we can pull out v2 .
⎛ ∣
⎜
⎜
Let A = ⎜ v1
⎜
⎜
⎝ ∣
∣
∣
v2
v3
∣
∣
∣ ⎞
⎟
⎟
a 3 × 4 matrix.
v4 ⎟
⎟
⎟
∣ ⎠
⎛ 1 2 0 3 ⎞
⎜
⎟
⎜
⎟
A Ð→ ⎜ 0 0 1 1 ⎟ .
⎜
⎟
⎜
⎟
⎝ 0 0 0 0 ⎠
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
RREF
B
Solution set of Ax = 0, this is the same as Bx = 0.
52
⎛ 3 ⎞
⎜ ⎟
⎜ ⎟
v4 = ⎜ 4 ⎟
⎜ ⎟
⎜ ⎟
⎝ 7 ⎠
The solution set of Bx = 0 is the set of all vectors such that
⎧
⎪
⎪
⎛
⎪
⎪
⎪
⎪
⎜
⎪
⎪
⎜
⎪
⎪
⎪⎜
⎨⎜
⎜
⎪
⎜
⎪
⎪
⎜
⎪
⎪
⎜
⎪
⎪
⎪
⎪
⎝
⎪
⎪
⎩
⎫
⎧
⎫
⎫
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎛ −2x2 − 3x4 ⎪
x1 ⎞
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎟
⎜
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎟
⎜
⎪
⎪
⎪
⎪
x
=
−2x
−
3x
⎪
⎪
⎪
2
4⎪
x2 ⎟ 1
x2
⎜
⎪
⎪
⎪
⎪
⎟∶
⎜
⎬ = ⎨⎜
⎬ ∶ x2 , x 4 ∈ R ⎬
⎟
⎪
⎪
⎪
⎪
⎜
⎪
⎪
⎪
⎪
x3 ⎟
−x4
⎪
⎪
⎪
⎪
⎟ x3 = −x4
⎜
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎟
⎜
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎝
⎪
⎪
⎪
⎪
x4 ⎠
x
⎪
⎪
⎪
⎪
4
⎭ ⎩
⎭
⎭
⎧
⎫
⎪
⎪
⎪
⎪
⎛ −1 ⎞
⎛ −3 ⎞
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎜
⎟
⎜
⎟
⎪
⎪
⎪
⎪
⎜
⎟
⎜
⎟
⎪
⎪
⎪
⎪
1 ⎟
⎜ 0 ⎟
⎪ ⎜
⎪
⎜
⎟
⎜
⎟
= ⎨x2 ⎜
+ x4 ⎜
∶ x 2 , x 4 ∈ R⎬ .
⎟
⎟
⎪
⎪
⎜ 0 ⎟
⎜ −1 ⎟
⎪
⎪
⎪
⎪
⎜
⎟
⎜
⎟
⎪
⎪
⎪
⎪
⎜
⎟
⎜
⎟
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎝
⎠
⎝
⎠
⎪
⎪
0
1
⎪
⎪
⎩
⎭
For any vector in this solution set, we get scalars that give a solution to Ax = 0 or
equivalently we get scalars that give a solution to
x1 v1 + x2 v2 + x3 v3 + x4 v4 = 0.
⎛
⎜
⎜
⎜
For instance, if x2 = 1, and x4 = 0 then this gives the solution ⎜
⎜
⎜
⎜
⎜
⎝
−2 ⎞
⎟
1 ⎟
⎟
⎟ which tells us that
⎟
0 ⎟
⎟
⎟
0 ⎠
−2v1 + v2 + 0v3 + 0v4 = 0 ⇒ v2 = 2v1 . If x2 = 0 and x4 = 1 then this gives the solution
⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
−3 ⎞
⎟
0 ⎟
⎟
⎟ which implies that −3v1 + 0v2 − v3 + v4 = 0 ⇒ v4 = 3v1 + v3 .
⎟
−1 ⎟
⎟
⎟
1 ⎠
Check:
⎛ 1 ⎞
⎛ 2 ⎞
⎛ 0 ⎞
⎛ 3 ⎞
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
v1 = ⎜ 1 ⎟ , v2 = ⎜ 2 ⎟ , v3 = ⎜ 1 ⎟ , v4 = ⎜ 4 ⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎝ 2 ⎠
⎝ 4 ⎠
⎝ 1 ⎠
⎝ 7 ⎠
⎛ 3 ⎞ ⎛ 0 ⎞ ⎛ 0 ⎞ ⎛ 3 ⎞ ⎛ 3 ⎞
⎜
⎟ ⎜
⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟
⎜
⎟ ⎜
⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟
⇒ 3 ⎜ 3 ⎟ + ⎜ 1 ⎟ = ⎜ 1 ⎟ + ⎜ 4 ⎟ = ⎜ 4 ⎟ = v4
⎜
⎟ ⎜
⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟
⎜
⎟ ⎜
⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟
⎝ 6 ⎠ ⎝ 1 ⎠ ⎝ 1 ⎠ ⎝ 7 ⎠ ⎝ 7 ⎠
53
✓
V = span(v1 , v2 , v3 , v4 ) = span(v1 , v3 ).
Since {v1 , v3 } is a linearly independent spanning set, it is a basis for V .
⎛ ∣ ⋯
⎜
⎜
Recall if A = ⎜ v1 ⋯
⎜
⎜
⎝ ∣ ⋯
just did was find a basis
∣ ⎞
⎟
⎟
then the column space of A is C(A) = span(v1 , . . . , vk ). What we
vk ⎟
⎟
⎟
∣ ⎠
for C(A).
Theorem 56. If A is an m × n matrix of rank k, then its column space C(A) ⊆ Rm is a kdimensional subspace with basis consisting of the columns of A which correspond to pivot columns
in RREF of A.
Proof. Suppose that the RREF of A looks like
B=
Remark.
⎡
⎢ 1
⎢
⎢
⎢
⎢ ⋮
k ⎢⎢
⎢ 0
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
k
⋯ 0
b11
⋱
⋮
⋮
⋯
1
bk1
0
n−k
⎤
⋯ b1,n−1 ⎥⎥
⎥
⎥
⎥
⋮
⋮
⎥
⎥
⋯ bk,n−k ⎥⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
0
1. The pivot columns of B from a linearly independent set.
2. The free variable columns are linear combinations of the pivot columns, i.e., for
the first free variable column
⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
b11 ⎞
⎟
⎟
⎛
⋮ ⎟
⎟
⎟
⎜
⎜
bk1 ⎟
⎟
⎜
⎟ = b11 ⎜
⎟
⎜
⎜
0 ⎟
⎟
⎜
⎟
⎜
⎟
⎝
⋮ ⎟
⎟
⎟
0 ⎠
⎛
⎜
1 ⎞
⎜
⎟
⎜
⎟
⎜
0 ⎟
⎜
⎟ + b21 ⎜
⎟
⎜
⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎜
⎜
0 ⎠
⎝
54
0
1
0
⋮
0
⎛
⎜
⎜
⎞
⎜
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎟ + ⋯ + bk1 ⎜
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎜
⎠
⎜
⎜
⎝
0 ⎞
⎟
⋮ ⎟
⎟
⎟
⎟
0 ⎟
⎟
⎟
⎟
1 ⎟
⎟
⎟
0 ⎟
⎟
⎟
⎟
⋮ ⎟
⎟
⎟
0 ⎠
Claims
1. The first k columns of A must be linearly independent.
2. The last n − k columns of A must be linearly combinations of the first k
columns of A.
To see why these claims are true, we will make the following general observations. Let
α, β, γ, δ ∈ R and suppose
⎛ a1 ⎞
⎛ b1
⎜
⎟
⎜
⎜
⎟
⎜
α⎜ ⋮ ⎟ +β ⎜ ⋮
⎜
⎟
⎜
⎜
⎟
⎜
⎝ an ⎠
⎝ bn
⎞
⎛ c1 ⎞
⎛ d1
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟+γ⎜ ⋮ ⎟+δ⎜ ⋮
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎠
⎝ cn ⎠
⎝ dn
⎞
⎟
⎟
⎟ = 0.
⎟
⎟
⎠
Observations:
1. Switch the ith and jth entries in each vector does not affect the equation
⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
α⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⎛
a1 ⎞
⎜
⎟
⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
aj ⎟
⎜
⎜
⎟
⎜
⎟
+β⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎜
⎟
ai ⎟
⎜
⎜
⎟
⎜
⎟
⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎝
⎠
an
⎛
b1 ⎞
⎜
⎟
⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
bj ⎟
⎜
⎜
⎟
⎜
⎟
+γ⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎜
⎟
bi ⎟
⎜
⎜
⎟
⎜
⎟
⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎝
⎠
bn
⎛
c1 ⎞
⎜
⎟
⎜
⋮ ⎟
⎟
⎜
⎜
⎟
⎜
⎟
⎜
⎟
cj ⎟
⎜
⎜
⎟
⎜
⎟
+δ⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎜
⎟
di ⎟
⎜
⎜
⎟
⎜
⎟
⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎝
⎠
cn
d1 ⎞
⎟
⋮ ⎟
⎟
⎟
⎟
dj ⎟
⎟
⎟
⎟
= 0.
⋮ ⎟
⎟
⎟
di ⎟
⎟
⎟
⎟
⋮ ⎟
⎟
⎟
dn ⎠
2. Scaling the jth entries by k ≠ 0 does not affect the equation.
3. Replacing the jth entries by the sum of the jth entry with k × ith entry
does not affect the equation
⎛
a1
⎜
⎜
⎜
⋮
⎜
⎜
α⎜
⎜ kai + aj
⎜
⎜
⎜
⋮
⎜
⎜
⎝
an
⎞
⎛
b1
⎟
⎜
⎟
⎜
⎟
⎜
⋮
⎟
⎜
⎟
⎜
⎟ + β ⎜ kb + b
⎟
⎜ i j
⎟
⎜
⎟
⎜
⎟
⎜
⋮
⎟
⎜
⎟
⎜
⎠
⎝
bn
⎞
⎛
c1
⎟
⎜
⎟
⎜
⎟
⎜
⋮
⎟
⎜
⎟
⎜
⎟ + γ ⎜ kc + c
⎟
⎜ i j
⎟
⎜
⎟
⎜
⎟
⎜
⋮
⎟
⎜
⎟
⎜
⎠
⎝
cn
55
⎞
⎛
d1
⎟
⎜
⎟
⎜
⎟
⎜
⋮
⎟
⎜
⎟
⎜
⎟ + δ ⎜ kd + d
⎟
⎜
i
j
⎟
⎜
⎟
⎜
⎟
⎜
⋮
⎟
⎜
⎟
⎜
⎠
⎝
dn
⎞
⎟
⎟
⎟
⎟
⎟
⎟ = 0.
⎟
⎟
⎟
⎟
⎟
⎟
⎠
We conclude that the first k columns of A are linearly independent because the first k
columns of B. Moreover, the last n − k columns of A are linear combinations of the first
k columns of A.
C(A) = span(columns of A) = span(first k columns of A) ⇒ first k columns of A form a
basis for C(A) and we can conclude also dim C(A) = k = rank(A).
56
Friday, 28 February 2014
Given A = ( v1
⋯
n
v ) an m × n matrix with columns v1 , . . . , vn ∈ R .
C(A) = span(v1 , . . . , vn ).
We found that a basis for C(A) consists of the columns of A which correspond to pivot columns in
the RREF of A.
dim C(A) = number of pivot columns in RREF of A.
Example
⎛
⎜
⎜
⎜
Let v1 = ⎜
⎜
⎜
⎜
⎜
⎝
⎛
1 ⎞
⎜
⎟
⎜
1 ⎟
⎜
⎟
⎟, v2 = ⎜
⎜
⎟
⎜
1 ⎟
⎜
⎟
⎜
⎟
⎝
1 ⎠
0 ⎞
⎟
1 ⎟
⎟
⎟ and let V = span(v1 , v2 ) ⊆ R4 . Find a basis for V ⊥ .
⎟
3 ⎟
⎟
⎟
−1 ⎠
V ⊥ = {w ∈ R4 ∶ w ⋅ v = 0 for all v ∈ V }.
⎛
⎜
⎜
⎜
Any vector w = ⎜
⎜
⎜
⎜
⎜
⎝
w1 ⎞
⎟
⎧
⎪
⎪
⎪ w ⋅ v1 = 0
w2 ⎟
⎟
⎟ such that ⎪
⎨
is a vector in V ⊥ , because if w satisfies
⎟
⎪
⎟
⎪
w3 ⎟
⎪
w ⋅ v2 = 0
⎪
⎩
⎟
w4 ⎠
these equations w is perpendicular to any linear combination of v1 and v2
⎧
⎪
⎪
⎪
⎪ w ⋅ v1 = 0
⎨
⎪
⎪
⎪
w ⋅ v2 = 0
⎪
⎩
Ô⇒
⎧
⎪
⎪
⎪
⎪ w1 + w2 + w3 + w4 = 0
⎨
⎪
⎪
⎪
w2 + w3 − w4 = 0.
⎪
⎩
⎛ 1
In fact, we see that V ⊥ is the solution set to the equation Ax = 0 where A = ⎜
⎝ 0
So our goal is to find a basis for N (A) = {x ∶ Ax = 0}. Observe that
⎛ 1 − −2 2 ⎞
RREF
⎟.
A Ð→ ⎜
⎝ 0 1 3 −1 ⎠
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
=B
57
1
1
1
3
1 ⎞
⎟.
−1 ⎠
Note that the solution set of Ax = 0 and Bx = 0 are the same, so N (A) = N (B).
Consider Bx = 0 and write this back in equation form
⎧
⎪
⎪
⎪
⎪ x1 − 2x3 + 2x4 = 0
⎨
⎪
⎪
⎪
x + 3x3 − x4 = 0
⎪
⎩ 2
⇒
⎧
⎪
⎪
⎪
⎪ x1 = 2x3 − 2x4
⎨
⎪
⎪
⎪
x = −3x3 + x4
⎪
⎩ 2
So a general solution to Bx = 0 is
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎤ ⎡
x1 ⎥⎥ ⎢⎢ 2x3 − 2x4
⎥ ⎢
x2 ⎥⎥ ⎢⎢ −3x3 + x4
⎥=⎢
⎥ ⎢
x3 ⎥⎥ ⎢⎢
x3
⎥ ⎢
⎥ ⎢
x4 ⎥⎦ ⎢⎣
x4
⎤
⎡
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥ , x 3 , x4 ∈ R = x3 ⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎦
⎣
⎤
⎡
⎢
2 ⎥⎥
⎢
⎥
⎢
⎢
−3 ⎥⎥
⎢
⎥ + x4 ⎢
⎥
⎢
⎢
1 ⎥⎥
⎢
⎥
⎢
⎥
⎢
⎢
0 ⎥⎦
⎣
⎤
−2 ⎥⎥
⎥
1 ⎥⎥
⎥.
⎥
0 ⎥⎥
⎥
⎥
1 ⎥⎦
⎧
⎡
⎪
⎢
⎪
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎪ ⎢⎢
N (A) = N (B) = ⎨x3 ⎢
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎪
⎢
⎪
⎩ ⎣
⎫
⎤
⎡
⎤
⎤
⎤ ⎡
⎡
⎪
⎢ −2 ⎥
⎪
⎛⎢⎢ −2 ⎥⎥ ⎢⎢ −2 ⎥⎥⎞
2 ⎥⎥
⎪
⎪
⎥
⎢
⎪
⎪
⎥⎟
⎥ ⎢
⎥
⎢
⎜⎢
⎥
⎪
⎪
⎢ 1 ⎥
⎜⎢ 3 ⎥ ⎢ 1 ⎥⎟
⎪
⎪
−3 ⎥⎥
⎥⎟
⎥ ⎢
⎥
⎢
⎜⎢
⎪
⎥⎟ .
⎥,⎢
⎥ ∶ x3 , x4 ∈ R⎬ = span ⎜⎢
⎥ + x4 ⎢
⎢
⎥
⎥
⎢
⎢
⎜
⎥
⎥⎟
⎪
⎢ 0 ⎥
⎜⎢ 1 ⎥ ⎢ 0 ⎥⎟
⎪
1 ⎥⎥
⎪
⎥
⎢
⎜⎢
⎥⎟
⎥ ⎢
⎪
⎪
⎢
⎥
⎥⎟
⎥ ⎢
⎥
⎜⎢
⎪
⎪
⎥⎠
⎥ ⎢
⎥
⎢
⎢
⎥
⎪
⎪
⎝
⎪
⎥
⎢
⎥
⎥
⎢
⎢
⎥
0
1
0 ⎦
1
⎪
⎦
⎦ ⎣
⎦
⎣
⎣
⎭
⎫
⎧
⎤
⎤ ⎡
⎡
⎪
⎢ −2 ⎥ ⎢ −2 ⎥⎪
⎪
⎪
⎪
⎪
⎪
⎥⎪
⎢
⎥
⎢
⎪
⎪
⎪
⎪
⎥
⎢
⎥
⎢
⎪
⎪
⎪
⎪
⎢
⎥
⎢
⎥
⎪
⎪
⎪
⎪
1
3
⎥
⎢
⎥
⎢
⎪⎢
⎥⎪
⎥,⎢
⎬ form a basis for N (A) = V ⊥ .
Since these are linearly independent, ⎨⎢
⎥
⎢
⎥
⎪
⎪
⎥
⎢
⎥
⎢
⎪
⎪
⎪
⎢ 1 ⎥ ⎢ 0 ⎥⎪
⎪
⎪
⎪
⎥⎪
⎥ ⎢
⎢
⎪
⎪
⎪
⎪
⎥
⎢
⎥
⎢
⎪
⎪
⎪
⎪
⎪
⎪
⎥
⎢
⎥
⎢
1
0
⎪
⎦⎪
⎦ ⎣
⎩⎣
⎭
Theorem 57. If A is an m×n matrix of rank k, then N (A) ⊆ Rn is a (n−k)-dimensional subspace
of Rn with consisting of vectors from the solution set of Ax = 0 found by setting each free variable
equal to 1 and the others to 0 one at a time.
Proof. Recall that if B is row equivalent to A, then N (B) = N (A). In particular,
58
N (A) = null space of RREF of A.
B=
⎡
⎢ 1
⎢
⎢
⎢
⎢ ⋮
k ⎢⎢
⎢ 0
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
⋯
k
0
b11
⋱
⋮
⋮
⋯
1
bk1
0
n−k
⎤
⋯ b1,n−1 ⎥⎥
⎥
⎥
⎥
⋮
⋮
⎥
⎥
⋯ bk,n−k ⎥⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
0
Then the general ?? to Bx = 0 is a vector of x ∈ Rn given by
⎡
⎢ −b11 xk+1 − ⋯ − b1n−k xn
⎢
⎢
⎛ x1 ⎞ ⎢⎢ −b21 xk+1 − ⋯ − b2n−k xn
⎜
⎟ ⎢⎢
⎜
⎟
⎜ ⋮ ⎟ ⎢⎢
⋮
⎜
⎟ ⎢
⎜
⎟ ⎢
⎜ x ⎟ = ⎢ −b x
⎜ k ⎟ ⎢
k1 k+1 − ⋯ − bkn−k xn
⎜
⎟ ⎢
⎜
⎟ ⎢
⎜ ⋮ ⎟ ⎢
xk+1
⎜
⎟ ⎢
⎜
⎟ ⎢
⎝ xn ⎠ ⎢⎢
⋮
⎢
⎢
⎢
xn
⎣
⎤
⎡
⎢ −b12 ⎥
⎤
⎡
⎥
⎢
⎤
⎡
⎤
⎢ −b1n−k ⎥
⎥
⎢
⎢ −b11 ⎥
⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎢ ⋮ ⎥
⎥
⎢
⎥
⎥
⎥
⎢
⎢ ⋮ ⎥
⎥
⎥
⎢
⋮
⎢
⎥
⎢
⎥
⎥
⎢
⎥
⎥
⎢
⎥
⎥
⎢
⎢ −b ⎥
⎢
⎥
⎥
⎢
⎢
⎥
k2 ⎥
⎥
⎢ −bkn−k ⎥
⎥
⎢
⎢ −bk1 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎥
⎢
⎢ 0 ⎥
⎥
⎢
⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎥ +⋯+xn ⎢
⎥ = xk+1 ⎢ 1 ⎥ +xk+2 ⎢
⎥
0
⎥
⎢
⎥
⎢
⎥
⎥
⎢
⎢ 1 ⎥
⎥
⎢
⎥
⎥
⎢
⎢ 0 ⎥
⎥
⎥
⎢
⎥
⎢
⋮
⎢
⎥
⎥
⎢
⎥
⎢
⎥
⎥
⎥
⎢
⎢ 0 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎥
⎢
⎥
⎢
⎢ ⋮ ⎥
⎥
0
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎢ ⋮ ⎥
⎥
⎢
⎥
⎥
⎥
⎢
⎥
⎢
⎥
⎥
⎢
⎢
⎢ 0 ⎥
⎥
⎥
⎢
⎥
1
⎦
⎣
⎦
⎦
⎣
⎥
⎢
⎢ 0 ⎥
´¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¸
¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¹
¶
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
⎦
⎣
vn−k
v1
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
v2
for xn+1 , . . . , xn ∈ R. N (A) = N (RREF (A)) = N (B) = span(v1 , v2 , . . . , vk ).
Definition 58
Let A be an m×n matrix whose rows are the vectors vT1 , . . . , vTm ∈ Rn (write transpose to emphasize
rows). Then the row space of A is the span(v1 , . . . , vm ) ⊆ Rn .
Example
⎡
⎢
⎡ ⎤
⎡ ⎤
⎡ ⎤
⎡ ⎤
⎢
⎢ 1 ⎥
⎢ 2 ⎥
⎢ 0 ⎥
⎢ 3 ⎥
⎢
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢ ⎥
Let v1 = ⎢ 1 ⎥, v2 = ⎢ 2 ⎥, v3 = ⎢ 1 ⎥, v4 = ⎢ 4 ⎥. Let A = ⎢⎢
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢
⎢ 2 ⎥
⎢ 4 ⎥
⎢ 1 ⎥
⎢ 7 ⎥
⎢
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢ ⎥
⎢
⎣ ⎦
⎣ ⎦
⎣ ⎦
⎣ ⎦
⎢
⎣
matrix. Then R(A) = span(v1 , v2 , v3 , v4 ).
59
− vT1
− vT2
− vT3
−
v4
⎤
− ⎥⎥
⎥
− ⎥⎥
⎥ be a 4 × 3
⎥
− ⎥⎥
⎥
⎥
− ⎥⎦
Fine a basis for R(A).
Lemma 59
If B is row equivalent to A then R(A) = R(B).
Proof. Given v1 , . . . , vk ∈ Rn . We will look at the following three claims:
C 1: span(v1 , . . . , vi , . . . , vj , . . . , vk ) = span(v1 , . . . , vj , . . . , vi , . . . , vk ).
C 2: span(v1 , . . . , cvj , . . . , vk ) = span(v1 , . . . , vk ).
C 3: span(v1 , . . . , vj + cvi , . . . , vk ) = span(v1 , . . . , vk ), where we are replacing vj with
vj + cvi .
Leave the proof of the claims to us.
The lemma follows from these claims because the rows of B are obtained from elementary
operations on the rows of A.
Back to our Example
⎤
1 ⎥⎥
⎥
0 1 1 ⎥⎥
2 2
⎥=B
⎥
0 0 0 ⎥⎥
0 1
⎥
⎥
0 0 0 ⎥⎦
3 4
⎡ ⎤ ⎡ ⎤⎫
⎡ ⎤ ⎡ ⎤
⎧
⎪
⎢ 1 ⎥ ⎢ 0 ⎥⎪
⎛⎢⎢ 1 ⎥⎥ ⎢⎢ 0 ⎥⎥⎞
⎪
⎪
⎪
⎢ ⎥ ⎢ ⎥⎪
⎪
⎪
⎪
⎢ ⎥ ⎢ ⎥⎪
⎜⎢ ⎥ ⎢ ⎥⎟
⎪
⎪
⎪
⎢ ⎥ ⎢ ⎥⎪
⎜⎢ ⎥ ⎢ ⎥⎟
By the above Lemma we have R(A) = R(B) = span ⎜⎢ 0 ⎥ , ⎢ 1 ⎥⎟. Moreover ⎨⎢ 0 ⎥ , ⎢ 1 ⎥⎬
⎢
⎥
⎢
⎥
⎜⎢ ⎥ ⎢ ⎥⎟
⎪
⎪
⎪
⎢ ⎥ ⎢ ⎥⎪
⎜⎢ ⎥ ⎢ ⎥⎟
⎪
⎪
⎪
⎢ 1 ⎥ ⎢ 1 ⎥⎪
⎪
⎪
⎪
⎪
⎝⎢⎢ 1 ⎥⎥ ⎢⎢ 1 ⎥⎥⎠
⎥
⎢
⎥
⎢
⎪
⎭
⎩
⎣ ⎦ ⎣ ⎦⎪
⎣ ⎦ ⎣ ⎦
is a basis for R(B) = R(A).
⎡
⎢
⎢
⎢
⎢
⎢
A = ⎢⎢
⎢
⎢
⎢
⎢
⎢
⎣
1
1
⎡
⎤
⎢
2 ⎥⎥
⎢
⎢
⎥
⎢
4 ⎥⎥
⎢
⎥ Ô⇒ ⎢
⎢
⎥
⎢
⎥
1 ⎥
⎢
⎢
⎥
⎢
⎥
⎢
7 ⎥⎦
⎣
60
1
0
Monday, 3 March 2014
What we know so far:
1. dim C(A) = the number of pivot columns = rank(A).
2. N (A) = {x ∶ Ax = 0}
dim N (A) = the number of free variables.
3. R(A) = span of rows of A = R(RREF(A))
Example
⎛
⎜
⎜
⎜
A=⎜
⎜
⎜
⎜
⎜
⎝
1
1
2
2
0
1
3
4
⎛
2 ⎞
⎜
⎟
⎜
⎟
4 ⎟
⎜
⎟ Ô⇒ ⎜
⎜
⎟
⎜
1 ⎟
⎜
⎟
⎜
⎟
⎝
7 ⎠
1
0
0
1
0
0
0
0
1 ⎞
⎟
1 ⎟
⎟
⎟ = RREF (A).
⎟
0 ⎟
⎟
⎟
0 ⎠
⎡ ⎤ ⎡ ⎤⎫
⎡ ⎤ ⎡ ⎤
⎧
⎪
⎢ 1 ⎥ ⎢ 0 ⎥⎪
⎛⎢⎢ 1 ⎥⎥ ⎢⎢ 0 ⎥⎥⎞
⎪
⎪
⎪
⎢ ⎥ ⎢ ⎥⎪
⎪
⎪
⎪
⎜⎢ ⎥ ⎢ ⎥⎟
⎢ ⎥ ⎢ ⎥⎪
⎪
⎪
⎪
⎢ ⎥ ⎢ ⎥⎪
⎜⎢ ⎥ ⎢ ⎥⎟
R(A) = R(RREF(A)) = span ⎜⎢ 0 ⎥ , ⎢ 1 ⎥⎟ and ⎨⎢ 0 ⎥ , ⎢ 1 ⎥⎬ are linearly indepen⎥
⎢
⎥
⎢
⎜⎢ ⎥ ⎢ ⎥⎟
⎪
⎪
⎪
⎢ ⎥ ⎢ ⎥⎪
⎜⎢ ⎥ ⎢ ⎥⎟
⎪
⎪
⎪
⎢ 1 ⎥ ⎢ 1 ⎥⎪
⎥
⎢
⎥
⎢
⎪
⎪
⎪
⎪
⎝⎢ 1 ⎥ ⎢ 1 ⎥⎠
⎥
⎢
⎥
⎢
⎪
⎭
⎩⎣ ⎦ ⎣ ⎦⎪
⎣ ⎦ ⎣ ⎦
dent. So they form a basis.
Theorem 60. If A is an m × n matrix of rank k then its row space R(A) ⊆ Rn is a k-dimensional
subspace of Rn with basis consisting of the non-zero row of RREF of A.
Key Idea of Proof. R(A) = row space of RREF of A. If RREF of A looks like
B=
⎡
⎢ 1
⎢
⎢
⎢
⎢ ⋮
k ⎢⎢
⎢ 0
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
⋯
k
0
b11
⋱
⋮
⋮
⋯
1
bk1
0
n−k
⎤
⋯ b1,n−1 ⎥⎥
⎥
⎥
⎥
⋮
⋮
⎥
⎥
⋯ bk,n−k ⎥⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
0
61
The first k rows of RREF form a basis for the row space of RREF of A and hence for
R(A).
Definition 61
Let A be an m × n matrix. Then the left null space of A is N (AT ).
Remark.
N (AT ) = {x ∈ Rm ∶ AT x = 0} ⊆ Rm = {x ∈ Rm ∶ (AT x)T = 0} = {x ∈ Rm ∶ xT (AT )T = 0}.
=A
Remark. Given A an m × n matrix:
1. C(A) ⊆ Rm
dim C(A) = rank(A) = number of pivot columns.
2. N (A) ⊆ Rn
dim N (A) = number of free variable columns.
3. R(A) ⊆ Rn
dim R(A) = number ob pivot columns.
4. N (AT ) ⊆ Rm
Theorem 62. If A is an m × n matrix, then
n = dim R(A) + dim N (A) = rank(A) + dim N (A).
Sometimes the dimension of N (A) is called the nullity of A, and is denoted null(A). Thus we have
n = dim R(A) + dim N (A) = rank(A) + dim N (A) = rank(A) + null(A).
Claim
For any matrix A, we have N (A) = R(A)⊥ .
Proof. First we show
N (A) ⊆ R(A)⊥ .
Let v ∈ N (A). Show that for any w ∈ R(A), v ⋅ w = 0. Let w ∈ R(A). Since v ∈ N (A),
Av = 0, this tells us v is perpendicular to any row of A which tells us v is perpendicular to
62
every linear combination of the rows of A. Thus v ⋅ w = 0 since w is a linear combination
of the rows of A.
Next, we claim R(A)⊥ ⊆ N (A). Let v ∈ R(A)⊥ . Show v ∈ N (A). v ∈ R(A)⊥ implies v is
perpendicular to every row of A. Thus Av = 0 and it follows v ∈ N (A).
Proposition 26. Let V ⊆ Rn be a k-dimensional subspace. Then dim V ⊥ = n − k.
⎛ vT
⎜ 1
⎜
Proof. Let {v1 , . . . , vk } be a basis for V . Let A = ⎜ ⋮
⎜
⎜
⎝ vTk
V ⊥ = R(A)⊥ = N (A).
⎞
⎟
⎟
⎟. Then V = R(A). And so
⎟
⎟
⎠
It follows that dim V ⊥ = dim N (A) = n−dim C(A) = n−dim R(A) = n−dim V = n−k.
Corollary 63
For any V ⊆ Rn subspace, V = (V ⊥ )⊥ .
Proof. Recall from the homework: V ⊆ (V ⊥ )⊥ . Also we proved if V ⊆ W are subspaces
and dim V = dim W then V = W .
Claim that dim(V ⊥ )⊥ = n − dim V ⊥ = n − (n − k) = k = dim V . Therefore V = (V ⊥ )⊥ .
Corollary 64
N (A)⊥ = R(A).
⊥
Proof. We proved that N (A) = R(A)⊥ , this implies N (A)⊥ = (R(A)⊥ ) = R(A).
Figure 6:
63
Corollary 65
For any matrix A,
1. C(A) = N (A⊥ )⊥
2. C(A)⊥ = N (A⊥ ).
Proof.
1.
R(A) = N (A)⊥ ⇒ R(A⊥ ) = N (A⊥ ).
´¹¹ ¹ ¹ ¸¹¹ ¹ ¹ ¶
C(A)
2.
N (A)⊥ = R(A) ⇒ N (A⊥ )⊥ = R(A⊥ ) = C(A).
64
Tuesday, 4 March 2014
Sample Exam
7. Let V ⊆ Rn be a subspace. Prove that V ∩ V ⊥ = {0}.
Proof. (1) {0} ⊆ V ∩ V ⊥ is clear because V and V ⊥ are subspaces.
(2) V ∩ V ⊥ ⊆ {0}. That is, if v ∈ V and v ∈ V ⊥ , then v ⋅ v = ∣∣v∣∣2 = 0 which implies
v = 0.
9. Suppose V ⊆ Rn is a k-dimensional subspace of Rn .
Given Lemma
Given V ⊆ Rn a subspace and {v1 , . . . , vm } ⊆ V that are linearly independent and
{w1 , . . . , wl } ⊆ V that span V . Then m ≤ l.
We want to prove:
1) Any set of k vectors in V that are linearly independent must span V .
2) Any set of k vectors in V that span V must be linearly independent.
Proof of 1. Let B = {w1 , . . . , wk } ⊆ V be a basis of V . And let S = {v1 , . . . , vk } ⊆ V be
any arbitrary linearly independent set. Suppose S does not span V . Then there exists
v ∈ V /span(S). Consider {v1 , . . . , vk , v}. This set is linearly independent. This is a
contradiction because I have a k + 1 element linearly independent set and a k-element
spanning set (B). Thus S spans V .
Theorem 66. Let V ⊆ Rn be a subspace of Rn . Then any vector x ∈ Rn can be written uniquely as
the sum of a vector in V and a vector in V ⊥ . In particular, Rn = V + V ⊥ .
Proof. Suppose dim V = k and let {v1 , . . . , vk } be a basis for V . Recall dim V ⊥ = n − k.
Let {vk+1 , . . . , vn } be a basis for V ⊥ .
65
Figure 7:
Claim is that {v1 , . . . , vk , vk+1 , . . . , vn } is a basis. To prove this claim it suffices to show
this set is linearly independent. Suppose c1 v1 +⋯cn vn = 0 for some scalars c1 , . . . , cn ∈ R.
Hint: V ∩ V ⊥ = {0}, look for another element in V ∩ V ⊥ .
c1 v1 + ⋯ + ck vk = − (ck+1 vk+1 + ⋯ + cn vn ) .
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
V
V⊥
if
w = c1 v1 + ⋯ck vk ∈ V
and w = −(ck+1 vk+1 + ⋯ + cn vn ) ∈ V ⊥
then w ∈ V ∩ V ⊥ = {0}. So we have w = 0 which implies that c1 v1 + ⋯ + ck vk = 0
and ck+1 vk+1 + ⋯ + cn vn = 0. From this we see that c1 = 0, . . . , ck = 0 and ck+1 =
0, . . . , cn = 0. Let x ∈ Rn . Then there exists a unique set of scalars d1 , . . . , dn ∈ R so that
x = x1 v1 + ⋯dn vn .
x = d1 v1 + ⋯ + dk vk + dk+1 vk+1 + ⋯ + dn vn .
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶ ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸⊥¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
∈V
∈V
So x is the sum of a vector in V and a vector in V ⊥ .
Summary of Chapter 3
§3.1 Subspaces
V ⊆ Rn .
• lines and planes through the origin
• solution set of Ax = 0
66
• {b ∶ Ax = b has a solution} is a subspace
• spans of vectors – span(v1 , . . . , vk ) is a subspace
• V ⊥ = {w ∈ Rn ∶ w ⋅ v = 0 for all v ∈ V }.
• V, W ⊆ Rn subspaces implies V + W = {v + w ∶ v ∈ V, w ∈ W }
§3.3 Linear Independent and Bases
A basis is the smallest set of vectors that gives a unique decomposition of any vector in V
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
n
• If V = R , then B = {e1 , . . . , en } where ei = ⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎤
0 ⎥⎥
⎥
⋮ ⎥⎥
⎥
⎥
0 ⎥⎥
⎥
⎥
1 ⎥⎥, with 1 in the ith row.
⎥
0 ⎥⎥
⎥
⎥
⋮ ⎥⎥
⎥
⎥
0 ⎥⎦
Definition 67 (Formal Definition)
A basis for a subspace V ⊆ Rn is a set of vectors that is linearly independent and spans V .
Definition 68
{v1 , . . . , vk } is linearly independent if x1 v1 + ⋯ + xk vk has only the trivial solution.
Lemma 69
If {v1 , . . . , vk } ⊆ V are linearly independent and v ∈ V then {v1 , . . . , vk , v} are linearly independent
if and only if v ∈/ span(v1 , . . . , vk ).
67
Thursday, 6 March 2014
Definition 70
A n × n matrix is invertible if there exists matrices B and C both n × n so that CA = I = AB.
Example
Suppose A2 = 0. Prove that I − A is invertible.
Let B = I + A. Then
(I − A)B = (I − A)(I + A) = I 2 − AI + IA − A2 = I − 0 = I,
thus B is a right inverse. Also, we have B(I − A) = I and thus B is a left inverse and
therefore I − A is invertible.
§2.5
Definition 71
Let A be an m × n matrix. Then AT is an n × m matrix whose rows are the columns of A.
Prove that if A is invertible, then AT is invertible.
Let B = (A−1 )T . Observe that AT B = AT (A−1 )T .
Theorem 72 (Big Theorem 1). If A is an m × n and B be an n × p. (AB)T = B T AT
AT (A−1 )T = (A−1 A)T = I T = I
(I is symmetric)
BAT = (A−1 )T AT = (AA−1 )T = I T = I.
Theorem 73 (Big Theorem 2). If A is an n × n and x, y ∈ Rn then Ax ⋅ y = x ⋅ AT y.
Ax ⋅ y = (Ax)T y = xT AT y = x(AT y) = x ⋅ AT y.
68
Example
Suppose A is an m × n matrix and AT Ax = 0 for some x ∈ Rn . Prove that Ax = 0.
Hint: Consider ∣∣Ax∣∣ (length of Ax or ∣∣Ax∣∣2 ).
∣∣Ax∣∣2 = Ax ⋅ Ax = xAAT x = x ⋅ 0 = 0.
∣∣Ax∣∣ = 0 ⇒ Ax = 0.
§3.1
Definition 74
Let V ⊆ Rn be a subset. It is a subspace if
1) 0 ∈ V ;
2) Closed under Addition: if u, v ∈ V , then u + v ∈ V ;
3) Closed under Scalar Multiplication: if u ∈ V and c ∈ R, then cu ∈ V .
Definition 75
Let V ⊆ Rn be a subspace. Then V T = {w ∈ Rn ∶ w ⋅ v = 0 for all v ∈ V } is called the orthogonal
complement.
Example
Suppose V, W ⊆ Rn are subspaces and suppose v ⋅ w = 0 for all v ∈ V and all w ∈ W .
Prove that V ⊂ W ⊥ .
Let v ∈ V . Then v ⋅ w = 0 for all w ∈ W . So v ∈ W ⊥ . So V ⊂ W ⊥ .
Is it true that V = W ⊥ ? or can you find V, W so that V ⊊ W ⊥ ?
Note that the first question is false!!.
Let W = {x2 -axis}, then W ⊥ = {x1 x2 -plane}. Let V = {x1 -axis}. So V ⊊ W ⊥ .
69
Figure 8:
§3.2
Definition 76
Let A be an m × n matrix. Then
C(A) = span(columns of A) ⊆ Rm
Columns Space
N (A) = {x ∈ Rn ∶ Ax = 0} ⊆ Rn
Null Space.
Theorem 77. N (A), C(A) are subspaces. ← We Prove.
Proposition 27. C(A) = {b ∈ Rm ∶ Ax = b is consistent}. A = (v1 , . . . , vn ) then if b ∈ C(A).
b = c1 v1 + ⋯ + cn vn for some c1 , . . . , cn ∈ R.
⎛ c1 ⎞
⎜
⎟
⎜
⎟
b = A⎜ ⋮ ⎟.
⎜
⎟
⎜
⎟
⎝ cn ⎠
We proved
⊇
C(A) = {b ∈ Rm ∶ Ax = b is consistent}.
⊆
Example
⎛ 1
⎜
⎜
A=⎜ 1
⎜
⎜
⎝ 0
2
2
0
⎛ 2 ⎞
0 ⎞
⎟
⎜
⎟
⎟
⎜
⎟
. Does Ax = b have a solution if b = ⎜ 3 ⎟? any vector in C(A) must
0 ⎟
⎟
⎜
⎟
⎟
⎜
⎟
1 ⎠
⎝ 1 ⎠
⎛ a ⎞
⎜
⎟
⎜
⎟
be of the form ⎜ a ⎟ for a, b ∈ R. It is not possible that b is a solution.
⎜
⎟
⎜
⎟
⎝ b ⎠
70
Example
⎛ 0 ⎞
⎛ 0
⎜
⎟
⎜
⎜
⎟
⎜
Is ⎜ 0 ⎟ in N (A)? No since A ⎜ 0
⎜
⎟
⎜
⎜
⎟
⎜
⎝ 2 ⎠
⎝ 2
0.
⎞
⎛ 0
⎟
⎜
⎟
⎜
⎟ ≠ 0 or because N (A) = R(A)⊥ and ⎜ 0
⎟
⎜
⎟
⎜
⎠
⎝ 1
⎞⎛ 0 ⎞
⎟⎜
⎟
⎟⎜
⎟
⎟⎜ 0 ⎟ ≠
⎟⎜
⎟
⎟⎜
⎟
⎠⎝ 2 ⎠
§3.3 Linearly Independent and Bases
Definition 78
Let {v1 , . . . , vk } ⊆ Rn be a set of vectors. It is linearly independent if x1 v1 +⋯+xk vk = 0 where this
equation only has the trivial solution x1 = 0, . . . , xk = 0. Let A = ( v1
⋯
vk ). Find solution set
of Ax = 0.
Definition 79
Let {v1 , . . . , vk } ⊆ V ⊆ Rn for a subset of a subspaces of V . H is a basis if
1. {v1 , . . . , vk } is a linearly independent,
2. span(v1 , . . . , vk ) = V .
⎛ 0
⎛ 1 ⎞
⎜
⎟
⎜
⎜
⎟
⎜
Give a basis of R3 . The standard basis {e1 , e2 , e3 } where e1 = ⎜ 0 ⎟, e2 = ⎜ 1
⎜
⎟
⎜
⎜
⎟
⎜
⎝ 0
⎝ 0 ⎠
⎞
⎟
⎟
⎟, and e3 =
⎟
⎟
⎠
⎛ 0 ⎞
⎜ ⎟
⎜ ⎟
⎜ 0 ⎟.
⎜ ⎟
⎜ ⎟
⎝ 1 ⎠
Theorem 80 (Big Theorem). Let A be an n × n invertible matrix if and only if its columns form
a basis of Rn .
Why can’t 3 vectors form a basis for R4 .
A4×3 = ( v1
v2
v3 )
has rank ≤ 3, but we would need rank = 4 to get columns to span R4 .
71
Example
Suppose {u, v} ⊆ Rn are linearly independent. Prove {u, u + v} is linearly independent.
Consider x1 u + x2 (u + v) = 0.
Figure 9:
72
Friday, 7 March 2014
Exam 2
73
Monday, 10 March 2014
Abstract Vector Spaces
Definition 81
An abstract real vector space is a set V with operations of addition and scalar multiplication
that satisfy:
1) u + v = v + u for all u, v ∈ V
2) (u + v) + w = u + (v + w) for all u, v, w ∈ V
3) there exist 0 ∈ V such that 0 + v = v for all v ∈ V (additive identity)
4) for all v ∈ V there exist −v ∈ V such that v + (−v) = 0
5) d(cv) = (dc)v for all c, d ∈ R, v ∈ V
6) c(u + v) = cu + cv for all c ∈ R, u, v ∈ V
7) (c + d)u = cu + du for all c, d ∈ R, u ∈ V
8) 1u = u for all u ∈ V .
Examples
1) V = Rn and define addition and scalar multiplication component-wise as usual.
2) The set Mm×n = the set of all m × n matrices with the usual component-wise
addition and scalar multiplication.
3) The set F (I) of all real-valued functions defined on an interval I ⊆ R. Let f, g ∈
F (I):
(f + g)(t) = f (t) + g(t) ∀ t ∈ I
(cf )(t) = cf (t)
74
∀ t ∈ I.
4) Rω = set of all infinite sequences with real number values. If x, y ∈ Rω then
x + y = (x1 + y1 , x2 + y2 , . . . )
def.
cx = (cx1 , cx2 , . . . )
5) The set of all polynomials with real coefficients
Definition 82
A subspace W of an abstract vector space V properties.
1) 0 ∈ W ;
2) if u, v ∈ W then u + v ∈ W ;
3) if u ∈ W and c ∈ R, then cu ∈ W .
Examples of Subspaces
1. a) U = {upper triangular n × n matrices}
b) L = {lower triangular n × n matrices}
c) D = {diagonal n × n matrices} are subspaces of Mn×n .
2. The et C 0 (I) of all continuous functions on I is a subspace of F(I).
3. The set W of all sequences whose terms are equivalently zero. W = {x ∈ Rω ∶ there
exists n ∈ N such that xk = 0 for all k ≥ n}
Proof. If x, y ∈ W then there exist n1 , n2 ∈ N such that
xk = 0 for all k ≥ n1
yk = 0 for all k ≥ n2
x + y ∈ W because xk + yk = 0 for all k ≥ max{n1 , n2 }.
4. The space of all polynomials with real coefficients is also a subspaces of F(I).
75
Theorem 83. If V is an abstract vector space and W is a subspace of V . Then W is an abstract
vector space.
Proof. Left as Exercise.
Example
Let V = F(I) and W = (C 0 (I)). Then W is a subspace of V and W itself is an abstract
space and has subspace P (I) the set of all polynomials on I.
Definition 84
If V is an abstract vector space then we define the span of v1 , . . . , vk ∈ V as the set
span(v1 , . . . , vk ) = {c1 v1 + ⋯ + ck vk ∶ c1 , . . . , ck ∈ R}.
We say a set {v1 , . . . , vk } is linearly independent if the equation x1 v1 + ⋯ + xk vk = 0 has only the
trivial solution. A basis for V is a set of vectors that span V and is linearly independent.
Example
Let p1 (t) = t + 1, p2 (t) = t2 + 2, p3 (t) = t2 − t. Claim {p1 , p2 , p3 } is linearly independent.
Suppose c1 p1 + c2 p2 + c3 p3 = 0 the zero polynomial for some scalars c1 , c2 , c3 ∈ R. This
means that c1 p1 (t) + c2 p2 (t) + c3 p3 (t) = 0 for all t ∈ R, c1 (t + 1) + c2 (t2 + 2) + c3 (t2 − t)
So if t = −1, then c2 (3) + 2c3 = 0. If t = 0, then c1 + 2c2 = 0. If t = 1 , then 2c1 + 2c2 = 0.
Solve the system
⎧
⎪
⎪
3c2 + 2c3 = 0
⎪
⎪
⎪
⎪
⎪
⎪
⎨ c1 + 2c2 = 0
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎩ 2c1 + 3c2 = 0
We see that c1 = c2 = c3 = 0.
76
Tuesday, 11 March 2014
Definition 85
V is finite dimensional if there exists a k ∈ N and a set {v1 , . . . , vk } ⊆ V that is a basis for V .
Otherwise, V is called infinite dimensional .
Theorem 86. Every finite dimensional non-trivial vector space has a basis.
Proof. Just like the proof in Rn .
Lemma 87
If V is finite dimensional and dim V = k then any set of vectors in V containing more than k vectors
must be linearly dependent.
Proof. There exists a basis {v1 , . . . , vk } ⊆ V . Suppose {w1 , . . . , wl } ⊆ V with l > k. We
want to show that this set is linearly dependent. There exists scalars so that
w1 = a11 v1 + ⋯ + a1l vk
⋮
wk = ak1 v1 + ⋯ + akl vk
Let A = [akj ]?? and show that l > k. Then null(A) > 0 because l > k.
⎛ c1 ⎞
⎜
⎟
⎜
⎟
There exist a nonzero vector c = ⎜ ⋮ ⎟ such that Ac = 0.
⎜
⎟
⎜
⎟
⎝ cl ⎠
l
l
j=1
j=1
k
∑ cj wj = ∑ cj (∑ aij vi )
i=1
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
l
k
k
l
= ∑ ∑ cj aij vi = ∑ ∑ ci aij vi
j=1 i=1
i=1 j=1
=wi
k
l
i=1
j=1
= ∑ vi ∑ aij cj = 0
´¹¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¶
= 0 = (∗)
(*) – ith component of Ac. I have written the wi ’s as a non-trivial linear combination
that sums to zero and so {w1 , . . . , wl } is linearly dependent.
77
Example
Rω = {set of all real valued sequences} is infinite dimensional, i.e., there does not exist
a finite basis for Rω .
Proof. (By way of contradiction) suppose Rω is finite dimensional. Then there exist
k ∈ N and a basis for Rω of size k. Our lemma from above tells us that the set containing
the following sequences must be linearly dependent:
e1 = (1, 0, 0, . . . )
e2 = (0, 1, 0, . . . )
⋮
ek+1 = (0, . . . , 0, 1 , 0, . . . )
(∗)
(∗) – k + 1 -entry
By lemma {e1 , . . . , ek } is linearly dependent. A contradiction.
Example
The set of P of all polynomials if infinite dimensional.
Proof. Exercise
Example
Fix k ∈ N. The set Pk of polynomials of degree at most k is finite dimensional with
dimension k + 1.
Proof. Claim: The set {1, t, t2 , . . . , tk } is a basis for Pk , where f0 = 1, . . . , fk = tk .
Proof of Claim.
1. {1, t, . . . , tk } spans Pk .
Any polynomial p ∈ Pk is of the form p(t) = a0 + a1 t + ⋯ + ak tk so this is
in span(1, t, . . . , tk ).
2. {1, t, . . . , tk } is linearly independent.
78
Proof of Linearly Independent. Suppose
c0 f0 + ⋯ck fk = 0
for some scalars c0 , . . . , ck ∈ R. This means
c0 + c1 t + ⋯ + ck tk = 0
for every t ∈ R. In other words every t ∈ R is a root of the polynomial
c0 f0 + ⋯ + ck fk .
Theorem 88 (Algebra Theorem). Any polynomial of degree at most k
has at most k roots, unless it is the zero polynomial.
The polynomial
c0 f0 + ⋯ + ck fk
has more than k roots, therefore it must be the zero polynomial. So all
of its coefficients must be zero.
Definition 89
Let V be a real vector space. V is an inner product space if for all u, v ∈ V there exists a real
number ⟨u, v⟩ call the inner product of u and v such that for all u, v, w ∈ V and c ∈ R:
1) ⟨u, v⟩ = ⟨v, u⟩
2) ⟨cu, v⟩ = c⟨u, v⟩
3) ⟨u + v, w⟩ = ⟨u, w⟩ + ⟨v, w⟩
4) ⟨v, v⟩ ≥ 0 and ⟨v, v⟩ = 0 if and only if v = 0.
79
Example
1. Rn with the usual dot product.
2. The space of n × n matrices Mn×n with the inner product ⟨A, B⟩ = tr(AT B) where
n
tr(A) = ∑[A]ii .
i=1
k+1
3. Fix t1 , . . . , tk+1 ∈ R. Define an inner product on Pk by ⟨p, q⟩ = ∑ p(ti )q(ti ) for
i=1
p, q ∈ Pk .
80
Thursday, 13 March 2014
Last problem on Homework 7 parts (d) and (e).
Part (e):
⎛ 1 ⎞
⎜ ⎟
⎜ ⎟
Column space and null space both have basis ⎜ 0 ⎟.
⎜ ⎟
⎜ ⎟
⎝ 0 ⎠
Observations:
a) The matrix must have 3 rows.
b) The matrix must have 3 columns.
c) Rank-Nullity Theorem: 3 = dim C(A) + dim N (A).
d) But dim C(A) = 1 and dim N (A) = 1.
This is why such a matrix cannot exist.
Part (d):
⎛ 1 ⎞
Column space and null space both have basis ⎜ ⎟.
⎝ 0 ⎠
Observations:
a) The matrix should be 2 × 2.
b) Rank-Nullity Theorem: 2 = dim C(A) + dim N (A) holds.
⎛ 1 ⎞
Every columns should be a multiple of ⎜ ⎟.
⎝ 0 ⎠
Candidate:
⎛ a b ⎞
⎜
⎟
⎝ 0 0 ⎠
81
a, b ∈ R
where a and b are not both zero, otherwise the null space would be R2 .
⎛ a b ⎞⎛ c ⎞ ⎛ 0 ⎞
⎜
⎟⎜
⎟=⎜ ⎟
⎝ 0 0 ⎠⎝ 0 ⎠ ⎝ 0 ⎠
for any c ∈ R. This implies that ac = 0 for any c ∈ R and it follows that a = 0. So any
⎛ 0 b ⎞
⎟, b ≠ 0 works.
matrix of the form ⎜
⎝ 0 0 ⎠
Definition 90
An abstract vector space is called an inner product space if it has an inner product for all
v, w ∈ V . There exist a real number called the inner product of v and w, denoted ⟨v, w⟩ such that
1) ⟨v, w⟩ = ⟨w, v⟩ for all v, w ∈ V
2) ⟨u + v, w⟩ = ⟨u, w⟩ + ⟨v, w⟩
3) ⟨cu, v⟩ = c⟨u, v⟩
4) ⟨v, v⟩ ≥ 0 and ⟨v, v⟩ = 0 if and only if v = 0
Example
n
1) Mn×n with the inner product ⟨A, B⟩ = tr(AT B) where tr(A) = ∑ (A)ii = the sum
i=1
of diagonal entries of A.
2) Consider the following subspace of Rω = space of sequences.
∞
l2 = {x ∈ Rω ∶ ∑ x2k < ∞} .
k=1
∞
l2 is an inner product space with inner product ⟨x, y⟩ = ∑ xk yk .
k=1
Claim: For all x, y ∈ l ⟨x, y⟩ < ∞.
2
Proof.
∞
N
⟨x, y⟩ = ∑ xk yk = lim ∑ xk yk .
N →∞ k=1
k=1
Fix N . Observe that
N
N
N
k=1
k=1
k=1
∑ ∣xk yk ∣ = ∑ ∣xk ∣∣yk ∣ ≤ ( ∑
82
1
2
x2k )
N
(∑
k=1
1
2
yk2 )
.
Cauchy-Swartz Inequality:
∣x ⋅ y∣ = ∣∣x∣∣∣∣y∣∣
Take the limit of both sides as N → ∞.
∞
∞
k=1
k=1
∑ ∣xk yk ∣ ≤ ( ∑
x2k )
1
2
∞
(∑
yk2 )
1
2
< ∞.
k=1
∞
Since ∑ ∣xk yk ∣ < ∞ implies ∑∞
k=1 xk yk < ∞.
k=1
Question
Can you think of a l2 sequence that does not eventually vanish?
Answer
Examples:
xn = r n
∣r∣ < 1
because
∞
∞
n=1
n=1
n 2
2 n
∑ (r ) = ∑ (r ) < ∞
because it is geometric.
1
n
zn =
∞
1
< ∞.
2
n
n=1
∑
Non-Example:
yn = (−1)n
where
∞
∞
∑ yn = ∑ 1
2
n=1
n=1
zn =
1
1
n2
83
where
∞
∞
1
</ ∞
n
n=1
∑ zn = ∑
2
k=1
and diverges.
Example
Consider Pk the space of polynomials of degree at most k. Fix t1 , . . . , tk+1 ∈ R, distinct.
k+1
⟨p, q⟩ = ∑ p(tj )q(tj ).
j=1
Claim: ⟨⋅, ⋅⟩ is an inner product.
Proof.
• ⟨p, q⟩ = ⟨q, p⟩
k+1
⟨p, q⟩ = ∑ p(tj )q(tj )
j=1
k+1
⟨q, p⟩ = ∑ q(tj )p(tj )
✓
j=1
•
k+1
⟨p + q, r⟩ = ∑ (p + q)(tj )r(tj )
j=1
k+1
= ∑ (p(tj )q(tj ))r(tj )
j=1
k+1
k+1
j=1
j=1
= ∑ p(tj )r(tj ) + ∑ q(tj )r(tj )
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶ ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
⟨p,r⟩
⟨q,r⟩
• ⟨cp, q⟩ = c⟨p, q⟩ – Exercise
•
k+1
2
⟨p, p⟩ = ∑ (p(tj )) ≥ 0
j=1
2
⟨p, p⟩ = 0 if and only if (p(tj )) = ⋯ =??? = 0 must be the zero inner
product.
84
Theorem 91 (Lagrange Interpolation Theorem). Let (t1 , b1 ), . . . , (tk+1 , bk+1 ) be k + 1 given points
in R2 , with t1 , . . . , tk+1 are distinct. Then there exists a unique polynomial p ∈ Pk that passes through
these points, i.e., p(ti ) = bi for all 1 ≤ i ≤ k + 1.
Naive Approach
Want p(t) = a0 + a1 t + ⋯ + ak tk with p(ti ) = bi for all i. This gives the following
constraint equations
p(1) = a0 + a1 t1 + ⋯ + ak tk1 = b1
⋮
p(k + 1) = a0 + a1 tk+1 + ⋯ + ak tkk+1 = bk+1
giving
⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
1
t1
⋯
1
t2
⋯
⋮
⋮
⋱
1
tk+1
⋯
tk1 ⎞ ⎛
⎟⎜
⎜
tk2 ⎟
⎟⎜
⎟⎜
⎟⎜
⎜
⋮ ⎟
⎟⎜
⎟⎜
tkk+1 ⎠ ⎝
85
a0 ⎞ ⎛ b1
⎟ ⎜
⎜
a1 ⎟
⎟ ⎜ b2
⎟=⎜
⎟ ⎜
⎜
⋮ ⎟
⎟ ⎜ ⋮
⎟ ⎜
ak ⎠ ⎝ bk+1
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎠
Friday, 14 March 2014
Theorem 92 (Lagrange Interpolation). Let (t1 , b1 ), . . . , (tk+1 , bk+1 ) are points in R2 and that
t1 , . . . , tk+1 are all distinct. Then there exists a unique polynomial of degree at most k that interpolates these points.
Outline of Proof.
1. Construct polynomials p1 , . . . , pk+1 with the property that
⎧
⎪
⎪
⎪ 1
pj (tl ) = ⎨
⎪
⎪
⎪
⎩ 0
if j = l
if j ≠ l
.
2. Our polynomial will be defined to be
p = b1 p1 + ⋯bk+1 pk+1 .
3. Show that {p1 , . . . , pk+1 } forms an orthonormal basis for Pk .
4. Use the fact that {p1 , . . . , pk+1 } form an orthonormal basis to prove uniqueness of
p.
Proof.
1. We want a polynomial p1 so that p1 (t1 ) = 1, p1 (t2 ) = ⋯ = p1 (tk+1 ) = 0.
Let q1 (t) = (t − t2 )(t − t3 )⋯(t − tk+1 ). Let
p1 (t) =
q1 (t)
(t − t2 )⋯(t − tk+1 )
=
.
q1 (t1 ) (t1 − t2 )⋯(t1 − tk+1 )
Now let, q2 (t) = (t − t1 )(t − t3 )⋯(t − tk+1 ). Let
p2 (t) =
q2 (t)
.
q2 (t2 )
Notice p2 (t2 ) = 1 and p2 (tj ) = 0 for all j ≠ 2.
2. Let p = b1 p1 + ⋯bk+1 pk+1 . Notice that
p(tj ) = b1 p1 (tj ) + ⋯ + bk+1 pk+1 (tj ) = bj pj (tj ) = bj
´¹¹ ¹ ¸ ¹ ¹ ¶
=1
for any j = 1, . . . , k + 1.
86
3. To show that this set {p1 , . . . , pk+1 } is a basis for Pk , it actually just suffices to
show that it is linearly independent because any linearly independent set with
k + 1 elements in a (k + 1)-dimensional vector space must span the space. To show
linearly independence, it suffices to show that
⎧
⎪
⎪
⎪ 1
⟨pi , pj ⟩ = ⎨
⎪
⎪ 0
⎪
⎩
i=j
i≠j
.
Why? Because if
c1 p1 + ⋯ + ck+1 pk+1 = 0
⇒ ⟨c1 p1 + ⋯ + ck+1 pk+1 , pj ⟩
= ⟨0, pj ⟩ = 0
⇒ c1 ⟨p1 , pj ⟩ + ⋯ + ck+1 ⟨pk+1 , pj ⟩ = 0
⇒ cj = 0.
Why does the above hold?
⎧
⎪
⎪
⎪ 1, i = j
⟨pi , pj ⟩ = ∑ pi (tl )pj (tl ) = ⎨
.
⎪
⎪
l=1
0,
i
≠
j
⎪
⎩
k+1
4. Let q ∈ Pk that passes through the points (t1 , b1 ), . . . , (tk+1 , bk+1 ). We will show
that q = b. We can write q = d1 p1 + ⋯ + dk+1 pk+1 for some scalars d1 , . . . , dk+1 ∈ R.
To show that q = p, we must show that di = bi for each i = 1, . . . , k + 1. To find the
coefficients on q, we compute
⟨q, pi ⟩ = ⟨d1 p1 + ⋯ + dk+1 pk+1 , pi ⟩
= d⟨p1 , pi ⟩ + ⋯ + dk+1 ⟨pk+1 , pi ⟩
= di ⟨pi , pi ⟩ = di
On the other hand
k+1
⟨q, pi ⟩ = ∑ q(tl )pi (tl )
l=1
= q(ti ) pi (ti ) = q(ti ) = bi .
²
=1
87
so we have di = bi
§4.1 Inconsistent Systems and Projections
⎛
⎜
⎜
⎜
Suppose we want to solve the system Ax = b where A = ⎜
⎜
⎜
⎜
⎜
⎝
2
1
0
1
⎛
1 ⎞
⎟
⎜
⎜
1 ⎟
⎟
⎜
⎟ and b = ⎜
⎟
⎜
⎜
1 ⎟
⎟
⎜
⎟
⎜
⎝
−1 ⎠
2 ⎞
⎟
1 ⎟
⎟
⎟.
⎟
1 ⎟
⎟
⎟
−1 ⎠
Example
How to fudge b so that the resulting system has a solution?
The key idea is to replace b with p ∈ C(A) that is closest to b.
Lemma 93
Suppose V ⊆ Rn is a subspace ad b ∈ Rm . Then if p ∈ V with the property that b − p ∈ V ⊥ then
∣∣b − p∣∣ < ∣∣b − q∣∣ for any other q ∈ V .
Figure 10:
∣∣p − q∣∣2 + ∣∣b − p∣∣2 = ∣∣b − q∣∣2 ⇒ ∣∣b − p∣∣ < ∣∣b − q∣∣.
>0
Remark. Given V ⊆ Rm is a subspace. Every b ∈ Rm can be written uniquely as the sum of a vector
in V and a vector in V ⊥ . So b = p + b − p and this expression is unique.
® ²
∈V
∈V ⊥
88
Monday, 24 March 2014
§4.1 Projections and Inconsistent Systems
Recall - Theorem
Let V ⊆ Rn be a subspace. Then every vector in Rn can be written uniquely as the sum
x = v + w where v ∈ V and w ∈ V ⊥ .
Definition 94
The vector v in the theorem is called the projection of x onto V , denoted projV x. The vector w
is called the projection of x onto V ⊥ , denoted projV ⊥ x.
Proposition 28 (Proposition from Last Time). Let V ⊆ Rm be a subspace and b ∈ Rm . If p ∈ V is
a vector with the property that b − p ∈ V ⊥ then p is the vector in V closest to b. That is,
∣∣b − p∣∣ < ∣∣b − q∣∣ for all other q ∈ V.
The projection of b onto V is the vector in V closest to b.
Computational Goals
1. Given V ⊆ Rm and b ∈ Rm , find p = projV (b).
2. Given a rank n m × n matrix A and b ∈ Rm so that Ax = b is an inconsistent system, solve the
system Ax̄ = p where p = projC(A) (b).
C(A) = span of the columns of A = {b ∈ Rm ∶ Ax = b is consistent}.
We will address 2 first:
We solve the system
⎧
⎪
⎪
⎪
⎪ Ax̄ = p
⎨
⎪
⎪
⎪
p = projC(A) (b)
⎪
⎩
89
given A and b, where b = projC(A) (b) + projC(A)⊥ (b). This system is equivalent to
=
b−p
⎧
⎧
⎪
⎪
⎪
⎪
⎪ Ax̄ = p
⎪ Ax̄ = p
⎨
⇐⇒ ⎨
⎪
⎪
⊥
T
T
⎪
⎪
⎪
⎪
⎩ b − p ∈ C(A) = N (A )
⎩ A (b − p) = 0
⇐⇒ AT (b − Ax̄) = 0
(by substitution)
⇐⇒ AT b = AT Ax̄
Figure 11:
Claim
Since rank(A) = n, AT A is nonsingular where A is an n × n matrix.
Proof. We show that if AT Ax = 0 then x = 0. Suppose AT Ax = 0. Dot
product of both sides with x
(AT Ax) ⋅ x = 0 ⇒ Ax ⋅ Ax = 0 ⇒ ∣∣Ax∣∣2 = 0 ⇒ Ax = 0.
So x = 0 because rank(A) = n.
By our claim
x̄ = (AT A)−1 AT b.
90
Definition 95
Given a rank n, m × n matrix A and b ∈ Rm , the equations
⎧
⎪
⎪
⎪ Ax̄ = p
⎨
⎪
T
⎪
⎪
⎩ A (b − p) = 0
are called the normal equations of the system Ax = b. The solution x̄ = (AT A)−1 AT b of the
normal equations is called the least squares solution to the (possibly) inconsistent system Ax = b.
We address 1 now:
Given V ⊆ Rm subspace and b ∈ Rm . To find p = projV (b).
1. Find a basis {v1 , . . . , vn } ⊆ V .
⎛ ∣
∣ ⎞
⎜
⎟
⎜
⎟
2. Let A = ⎜ v1 ⋯ vn ⎟. Note A is m × n and rank(A) = n, because the columns
⎜
⎟
⎜
⎟
∣ ⎠
⎝ ∣
of A are linearly independent.
3. Since V = span(v1 , . . . , vn ), V = C(A) and so p = projV (b) = projC(A) (b). So p is
“the” p from the normal equations
⎧
⎪
⎪
Ax̄ = p
⎪
⎨
⎪
T
⎪
⎪
⎩ A (b − p) = 0
whose solution is x̄ = (AT A)−1 AT b. Therefore
p = A(AT A)−1 AT b.
Remark. The matrix A(AT A)−1 AT is the standard matrix of projection onto V and is denoted in
the book by PV = A(AT A)−1 AT .
The matrix of projection onto V ⊥ is given by
PV ⊥ = I − A(AT A)−1 AT
91
since
b = projV (b) + projV ⊥ (b)
= PV b + projV ⊥ (b)
⇒ projV ⊥ (b) = b − PV b = (I − PV )b
⇒ P V ⊥ = I − PV .
Example
⎛⎛
⎜⎜
⎜⎜
⎜⎜
⎜
Suppose V = span ⎜
⎜⎜
⎜⎜
⎜⎜
⎜⎜
⎝⎝
2 ⎞ ⎛
⎟ ⎜
⎜
1 ⎟
⎟ ⎜
⎟,⎜
⎟ ⎜
⎜
0 ⎟
⎟ ⎜
⎟ ⎜
1 ⎠ ⎝
⎛
1 ⎞⎞
⎟⎟
⎜
⎟
⎟
⎜
1 ⎟⎟
⎜
⎟⎟ ⊆ R4 and b = ⎜
⎟⎟
⎜
⎟
⎜
1 ⎟
⎟⎟
⎜
⎟⎟
⎜
⎝
−1 ⎠⎠
2 ⎞
⎟
1 ⎟
⎟
⎟. Find proj (b), proj ⊥ (b).
V
V
⎟
1 ⎟
⎟
⎟
−1 ⎠
Solution:
⎛
⎜
⎜
⎜
We first find the least squares solution of Ax = b where A = ⎜
⎜
⎜
⎜
⎜
⎝
−1
2
1
0
1
1 ⎞
⎟
1 ⎟
⎟
⎟ (which
⎟
1 ⎟
⎟
⎟
−1 ⎠
has rank 2). We fine x̄ = (A A) A b.
T
⎛ 2
AT = ⎜
⎝ 1
1
0
1
1
(AT A)−1 =
⎛
⎜
⎜
⎜
T
A A=⎜
⎜
⎜
⎜
⎜
⎝
1 ⎞
⎟
−1 ⎠
1 ⎛ 4
⎜
20 ⎝ −2
T
−2 ⎞
⎟
6 ⎠
2
1
0
1
1 ⎞
⎟
1 ⎟
⎟⎛ 2
⎟⎜
⎟
1 ⎟
⎟⎝ 1
⎟
−1 ⎠
⎛ 2
AT b = ⎜
⎝ 1
1
0
1
1
So
⎛
x̄ = (AT A)−1 AT b = ⎜
⎝
92
3
10
11
10
⎞
⎟
⎠
1
0
1
1
⎛
⎜
1 ⎞⎜
⎜
⎟⎜
⎜
−1 ⎠ ⎜
⎜
⎜
⎝
1 ⎞ ⎛ 6
⎟=⎜
−1 ⎠ ⎝ 2
2 ⎞
⎟
4 ⎠
2 ⎞
⎟
1 ⎟
⎟ ⎛ 4 ⎞
⎟=⎜
⎟.
⎟
⎝
⎠
1 ⎟
5
⎟
⎟
−1 ⎠
⎛
⎜
⎜
⎜
projV (b) = Ax̄ = ⎜
⎜
⎜
⎜
⎜
⎝
93
17/10 ⎞
⎟
7/5 ⎟
⎟
⎟.
⎟
11/10 ⎟
⎟
⎟
−4/5 ⎠
Tuesday, 25 March 2014
Sub (Professor Alex Roitershtein) Notes
Given V ∈ Rn a subspace we have:
x, y ∈ V
α, β ∈ R
⇒ αx + βy ∈ V
Theorem 96. Any vector x ∈ Rn can be represented (uniquely) as x = V + W with V ⊥ W (orthogonal).
Definition 97 (Orthogonal of V )
All x ∈ R such that V in the above represented is 0. Of course, V is all x ∈ Rn such that W in the
above represented is 0.
Theorem 98. (V ⊥ )⊥ = V .
Notation: often people write Rn = V ⊕ V ⊥ where ⊕ is direct sum.
Definition 99 (Notation)
Rn = V ⊕ W means any x ∈ Rn can be represented as x = V + W , v ∈ V , w ∈ W , V ⊥ W .
Theorem 100. a) Rn = V ⊕ W if and only if W = V ⊥
b) (uniqueness) dim(V ) + dim(W ) = n.
c) V = W ⊥ and W = V ⊥
Example n = 3, R3
Figure 12:
94
V = span{e1 , e2 , e3 } = they hyperplane xy x = (x, y, z), then x = xe1 + ye2 + ze3
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶ °
V
W
Figure 13:
Figure 14:
Definition 101
Let v1 , v2 , . . . , vr ∈ Rn . Then the set {v1 , . . . , vr } is called orthogonal basis for a subspace
V ⊂ Rn if
1) v1 , . . . , vr form a basis of V
2) vi ⊥ vj for any pair 1 ≤ i < j ≤ r.
In general, a set {x1 , . . . , xr } vectors in Rn is called an orthogonal set if xi ⊥ xj for any 1 ≤ i <
j ∈ R (only property 2) holds). So orthogonal basis = (1) basis and (2) orthogonal set.
Question
The reverse direction – is it true that any orthogonal set is a basis of some subspace.
Answer
Yes, if none of the vectors in the set is 0 (unless r = 1, V = 0).
95
Proposition 29. Any orthogonal set including all nonzero vectors is independent.
Proof. If v1 , . . . , vr form an orthogonal set. Assume a1 v1 + ⋯ + ar vr = 0 and ak ≠ 0 for
some i ≤ k ≤ r. Multiply both sides by vk .
(a1 v1 + ⋯ak vk ) ⋅ vk = 0 ⋅ vk = 0
we get that the left hand side (LHS) ≠ 0 and the right hand side (RHS) = 0, which is
nonsense.
Corollary 102
If v1 , . . . , vr is an orthogonal set and vi ≠ 0 for all 1 ≤ i ≤ r, then {v1 , . . . , vr } is a basis of
span{v1 , . . . , vr }.
Lemma 103
If V ∈ Rn has orthogonal basis {v1 , . . . , vr }, then for all x ∈ V ,
vk ⋅ x
vk
(∗∗) k=1 ∣∣vk ∣∣2
r
r
x = ∑ projvk (x) = ∑
(∗) k=1
with (*) – being shown today and (**) having been shown in another previous class.
Proof. If x ∈ V . Since {v1 , . . . , vr } is a basis we have
x = c1 v1 + c2 v2 + ⋯ck vk
for some ci ∈ R. Multiply both sides by some vk .
x ⋅ vk = ck (vk ⋅ vk ).
Hence ck =
x ⋅ vk
. Hence
(vk ⋅ vk )
r
r
x ⋅ vk
v
=
projvk (x).
∑
k
2
k=1
k=1 ∣∣vk ∣∣
r
x = ∑ ck vk = ∑
k=1
96
Professor Notes
§4.2 Orthogonal Basis
Definition 104
Let v1 , . . . , vk ∈ Rn . The set {v1 , . . . , vk } is called an orthogonal set of vi ⋅vj = 0 for all i ≠ j. The
set {v1 , . . . , vk } is an orthogonal basis for a subspace V ⊆ Rn if it is a basis and an orthogonal
set. The set is an orthogonal basis if it is an orthogonal basis and all of its vectors are unit
length.
Proposition 30. If {v1 , . . . , vk } ⊆ Rn is an orthogonal set then it is linearly independent.
Lemma 105
If V ⊆ Rn has orthogonal basis {v1 , . . . , vk } then for all x ∈ V ,
vi ⋅ x
v.
2 i
i=1 ∣∣vi ∣∣
k
k
x = ∑ projvi (x) = ∑
i=1
Proof. Let x ∈ V . Then x = c1 v1 + ⋯ + ck vk for some c1 , . . . , ck ∈ R.
⇒ x ⋅ vi
= (c1 v1 + ⋯ + ck vk ) ⋅ vi
= ci ∣∣vi ∣∣2
x ⋅ vi
∣∣vi ∣∣2
⇒ ci
=
⇒x
=∑
for each i = 1, . . . , k
k
x ⋅ vi
= ∑ projvi (x).
2
i=1 ∣∣vi ∣∣
i=1
k
Recall
That the ci are called the coordinates with respect to the basis {v1 , . . . , vk }. Where the
basis is orthogonal, they are sometimes called the Fourier coefficients of x.
Proposition 31. If V ⊂ Rm has orthogonal basis {v1 , . . . , vk }, and b ∈ Rm then
vi ⋅ b
v.
2 i
∣∣v
i ∣∣
i=1
k
k
projV (b) = ∑ projvi (b) = ∑
i=1
97
Proof. Recall that for all b ∈ Rm , b = projV (b) + projV ⊥ (b). Then p ∈ V and
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶ ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
−p
b−p
k
k
p ⋅ vi
b ⋅ vi
vi = ∑
vi = ∑ projvi (b).
2
2
i=1 ∣∣vi ∣∣
i=1 ∣∣vi ∣∣
i=1
k
p=∑
where (b − p) ⋅ vi = 0 for all i implies b ⋅ vi = p ⋅ vi .
Example
Recall V = span(v1 , v2 )
⎛ −1 ⎞
⎛ 1
⎜
⎟
⎜
⎜
⎟
⎜
w1 = ⎜ 0 ⎟, w2 = ⎜ 1
⎜
⎟
⎜
⎜
⎟
⎜
⎝ 1 ⎠
⎝ 1
from previous section. {v1 , v2 } not an orthogonal basis. But,
⎞
⎟
⎟
⎟ is ??. Then
⎟
⎟
⎠
projV (b) = projw1 (b) + projw2 (b)
=
b ⋅ w2
b ⋅ w1
w1 +
w2
∣∣w1 ∣∣2
∣∣w2 ∣∣2
=⋯
Problem
How to find orthonormal basis in general?
Solution
We present an algorithm called the Gram-Schmidt Process for finding an orthogonal
basis given a regular old basis.
Let {v1 , . . . , vk } be a basis for a subspace V . We find an orthogonal basis {w1 , . . . , wk }
for V .
Step 1: Let w1 = v1 Remove span(w1 ) = span(v1 ).
98
Step 2: Let
w2 = projspan(w1 )⊥ (v2 )
= v2 − projspan(w1 ) (v2 )
= v2 − projw1 (v2 )
= v2 −
w1 ⋅ v1
w1 .
∣∣w1 ∣∣2
Figure 15:
Remark. (w1 , w2 ) ⊂ span(v1 , v2 ), {w1 , w2 } orthogonal implies independence.
Step 3: Let
w3 = projspan(w1 ,w2 )⊥ (v3 )
= v3 − projspan(w1 ,w2 ) (v3 )
= v3 − projw1 (v3 ) − projw2 (v3 )
= v3 −
w1 ⋅ v3
w2 ⋅ v3
w1 −
w2 .
∣∣w1 ∣∣2
∣∣w2 ∣∣2
Figure 16:
Remark. w3 span(v1 , v2 , v3 ) ⇒ span(w1 , w2 , w3 ) ⊂ span(v1 , v2 , v3 ) and
{w1 , w2 , w3 } orthogonal.
Step k: wk = vk −
vk ⋅ wk−1
vk ⋅ w1
w1 − ⋯ −
wk−1 .
2
∣∣w1 ∣∣
∣∣wk−1 ∣∣2
99
What is special?
At each step, we add a vector orthogonal to all others, so the set is always linearly
independent, and spans the same subspace that the vi ’s span.
Example
⎛
⎜
⎜
⎜
Let v1 = ⎜
⎜
⎜
⎜
⎜
⎝
⎛
1 ⎞
⎟
⎜
⎜
1 ⎟
⎟
⎜
⎟, v 2 = ⎜
⎟
⎜
⎜
1 ⎟
⎟
⎜
⎟
⎜
⎝
1 ⎠
⎛
3 ⎞
⎟
⎜
⎜
1 ⎟
⎟
⎜
⎟, v3 = ⎜
⎟
⎜
⎜
−1 ⎟
⎟
⎜
⎟
⎜
⎝
1 ⎠
1 ⎞
⎟
1 ⎟
⎟
⎟. Fine orthogonal basis for span(v1 , v2 , v3 ).
⎟
3 ⎟
⎟
⎟
3 ⎠
Solution:
⎛
⎜
⎜
⎜
1. w1 = v1 = ⎜
⎜
⎜
⎜
⎜
⎝
1 ⎞
⎟
1 ⎟
⎟
⎟.
⎟
1 ⎟
⎟
⎟
1 ⎠
⎛
⎜
⎜
⎜
w1 ⋅ v2
⎜
2. w2 = v2 −
w
=
1
⎜
2
∣∣w1 ∣∣
⎜
⎜
⎜
⎝
⎛ 1 ⎞ ⎛
3 ⎞
⎟
⎜ ⎟ ⎜
⎜ ⎟ ⎜
1 ⎟
⎟ 4⎜ 1 ⎟ ⎜
⎟− ⎜ ⎟=⎜
⎟ 4⎜ ⎟ ⎜
⎜ 1 ⎟ ⎜
−1 ⎟
⎟
⎜ ⎟ ⎜
⎟
⎜ ⎟ ⎜
⎝ 1 ⎠ ⎝
1 ⎠
100
2 ⎞
⎟
0 ⎟
⎟
⎟.
⎟
−2 ⎟
⎟
⎟
0 ⎠
3.
w3 = v3 −
v3 ⋅ w1
v 3 ⋅ w2
w1 −
w2
∣∣w1 ∣∣2
∣∣w2 ∣∣2
⎛
⎜
⎜
⎜
=⎜
⎜
⎜
⎜
⎜
⎝
⎛ 1 ⎞
⎛ 2 ⎞
1 ⎞
⎜ ⎟
⎜
⎟
⎟
⎜ ⎟
⎜
⎟
1 ⎟
⎟ 8⎜ 1 ⎟ 4⎜ 0 ⎟
⎟− ⎜ ⎟+ ⎜
⎟
⎟ 4⎜ ⎟ 8⎜
⎟
⎜
⎜
⎟
⎟
⎟
3 ⎟
⎜ 1 ⎟
⎜ −2 ⎟
⎜ ⎟
⎜
⎟
⎟
⎝ 1 ⎠
⎝ 0 ⎠
3 ⎠
⎛
⎜
⎜
⎜
=⎜
⎜
⎜
⎜
⎜
⎝
1 ⎞ ⎛
⎟ ⎜
⎜
1 ⎟
⎟ ⎜
⎟−⎜
⎟ ⎜
⎜
3 ⎟
⎟ ⎜
⎟ ⎜
3 ⎠ ⎝
⎛
⎜
⎜
⎜
=⎜
⎜
⎜
⎜
⎜
⎝
−1 ⎞ ⎛
⎟ ⎜
⎜
−1 ⎟
⎟ ⎜
⎟+⎜
⎟ ⎜
⎜
1 ⎟
⎟ ⎜
⎟ ⎜
1 ⎠ ⎝
2 ⎞ ⎛
⎟ ⎜
⎜
2 ⎟
⎟ ⎜
⎟+⎜
⎟ ⎜
⎜
2 ⎟
⎟ ⎜
⎟ ⎜
2 ⎠ ⎝
1 ⎞
⎟
0 ⎟
⎟
⎟
⎟
−1 ⎟
⎟
⎟
0 ⎠
1 ⎞ ⎛
⎟ ⎜
⎜
0 ⎟
⎟ ⎜
⎟=⎜
⎟ ⎜
⎜
−1 ⎟
⎟ ⎜
⎟ ⎜
0 ⎠ ⎝
0 ⎞
⎟
−1 ⎟
⎟
⎟.
⎟
0 ⎟
⎟
⎟
1 ⎠
Find orthonormal basis
q1 =
w1
,
∣∣w1 ∣∣2
q2 =
w2
,
∣∣w2 ∣∣2
101
q3 =
w3
∣∣w3 ∣∣2
Thursday, 27 March 2014
Sub: Dr. Pendleton
§4.2 Continued
Gram-Schmidt Procedure
Goal: Starting with any basis for some space V , construct an orthogonal (orthonormal) basis for
V.
If {w1 , . . . , wk } is any basis for V , then if x ∈ V then we can write x = c1 w1 + c2 w2 + ⋯ + ck wk
where ci ∈ R for all i.
How does this change?
x ⋅ vi
v
2 i
∣∣v
i ∣∣
i=1
²
′
k
If {w1 , . . . , wk } is an orthogonal basis for V , then if x ∈ V implies x = ∑
ci s
We also recall that
b ⋅ vi
v
2 i
i=1 ∣∣vi ∣∣
k
k
projV b = ∑ projvi b = ∑
i=1
Question: How do we actually do this?(We want: orthogonal basis for W , {v1 , . . . , vk })
Start with a basis {w1 , . . . , wk } for V
Step1: Let v1 = w1 . Then span{v1 } = span{w1 }.
Step2: (Need a vector v2 such that v1 ⊥ v2 and span{v1 , v2 } = span{w1 , w2 }) Take v2 to be the
Figure 17:
102
part of w2 which is orthogonal to w1 , i.e.,
v2 = projspan(w1 )⊥ w2 = projspan(v1 )⊥ w2
(∗)
= w2 − projspan(v1 ) w2
´ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
=v
1
= w2 − projv1 w2
= w2 −
w2 ⋅ v 1
v1
∣∣v1 ∣∣2
(*) – this is an important set. We need to work with an orthogonal basis.
Note:
v2 ≠ 0 since if v2 = 0 then
v2 = w2 −
w2 ⋅ v 1
w2 ⋅ v 1
w2 ⋅ v 1
v1 ⇒ w2 −
w1 ⇒ w2 =
w1
∣∣v1 ∣∣2
∣∣v1 ∣∣2
∣∣v1 ∣∣2
´¹¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¶
constant
which is a contradiction since {w1 , w2 } is a linearly independent set.
Note:
span{v1 , v2 } ⊂ span{w1 , w2 } Since {v1 , v2 } is an orthogonal set it implies {v1 , v2 }
is linearly independent and has the same number of elements, that is span{v1 , v2 } =
span{w1 , w2 }.
Step3: Take v3 to be the part of w3 that is perpendicular (equivalently orthogonal) to the span{v1 , v2 },
i.e.,
v3 = projspan{v1 ,v2 }⊥ w3
= w3 − projspan{v1 ,v2 } w3
= w3 − projv1 w3 − projv2 w3
= w3 −
v1 ⋅ w3
v2 ⋅ w3
v1 −
v2
∣∣v1 ∣∣2
∣∣v2 ∣∣2
Note:
span{v1 , v2 , v3 } = span{w1 , w2 , w3 }
Continue this process (inductively)
103
Stepk: vk = wk −
wk ⋅ v 1
wk ⋅ v2
wk ⋅ vk−1
v1 −
v2 − ⋯ −
vk−1 assuming that {v1 , . . . , vk−1 } is already
2
2
∣∣v1 ∣∣
∣∣v2 ∣∣
∣∣vk−1 ∣∣2
know.
Example
⎡
⎢
⎢
⎢
⎢
⎢
Let w1 = ⎢⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎤
⎡
⎢
1 ⎥⎥
⎢
⎥
⎢
⎥
⎢
1 ⎥
⎢
⎥, w2 = ⎢
⎥
⎢
⎢
1 ⎥⎥
⎢
⎥
⎢
⎥
⎢
⎢
1 ⎥⎦
⎣
⎤
⎡
⎢
3 ⎥⎥
⎢
⎥
⎢
⎥
⎢
1 ⎥
⎢
⎥, w3 = ⎢
⎥
⎢
⎢
−1 ⎥⎥
⎢
⎥
⎢
⎥
⎢
⎢
1 ⎥⎦
⎣
⎤
1 ⎥⎥
⎥
1 ⎥⎥
⎥.
⎥
3 ⎥⎥
⎥
⎥
3 ⎥⎦
(a) Find an orthogonal basis for span{v1 , v2 , v3 }.
⎡
⎢
⎢
⎢
⎢
⎢
Let v1 = w1 = ⎢⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎡
⎤
⎢
1 ⎥⎥
⎢
⎢
⎥
⎢
⎥
1 ⎥
⎢
w
⋅
v
⎥. Let v2 = w2 − 2 1 v1 = ⎢
⎢
⎥
2
∣∣v1 ∣∣
⎢
1 ⎥⎥
⎢
⎢
⎥
⎢
⎥
⎢
1 ⎥⎦
⎣
⎤
⎡ ⎤ ⎡
⎢ 1 ⎥ ⎢
3 ⎥⎥
⎢ ⎥ ⎢
⎥
⎢ ⎥ ⎢
⎥
1 ⎥ 4 ⎢⎢ 1 ⎥⎥ ⎢⎢
⎥− ⎢ ⎥=⎢
⎥
⎢ ⎥ ⎢
−1 ⎥⎥ 4 ⎢⎢ 1 ⎥⎥ ⎢⎢
⎥
⎢ ⎥ ⎢
⎥
⎢ ⎥ ⎢
⎢ 1 ⎥ ⎢
1 ⎥⎦
⎣ ⎦ ⎣
w3 ⋅ v1
w3 ⋅ v2
v1 −
v2
2
∣∣v1 ∣∣
∣∣v2 ∣∣2
⎤
⎤
⎡ ⎤
⎡
⎥
⎢ 1 ⎥
⎢ 2 ⎥
⎥
⎥
⎢ ⎥
⎢
⎥
⎥
⎢ ⎥
⎢
⎥
⎢ 1 ⎥
⎢ 0 ⎥
⎥ 8 ⎢ ⎥ −4 ⎢
⎥
⎥− ⎢ ⎥−
⎥
⎢
⎥ 4⎢ ⎥ 8 ⎢
⎥
⎥
⎢ 1 ⎥
⎢ −2 ⎥
⎥
⎥
⎢ ⎥
⎢
⎥
⎥
⎢ ⎥
⎢
⎥
⎥
⎢ ⎥
⎢
⎥
⎢ 1 ⎥
⎢ 0 ⎥
⎦
⎦
⎣ ⎦
⎣
⎤ ⎡
⎤ ⎡ ⎤ ⎡
⎥ ⎢ 2 ⎥ ⎢ 1 ⎥ ⎢ 0
⎥ ⎢
⎥ ⎢ ⎥ ⎢
⎥ ⎢
⎥ ⎢ ⎥ ⎢
⎥ ⎢ 2 ⎥ ⎢ 0 ⎥ ⎢ −1
⎥ ⎢
⎥ ⎢ ⎥ ⎢
⎥−⎢ ⎥+⎢
⎥=⎢
⎥ ⎢ ⎥ ⎢
⎥ ⎢
⎥ ⎢ 2 ⎥ ⎢ −1 ⎥ ⎢ 0
⎥ ⎢
⎥ ⎢ ⎥ ⎢
⎥ ⎢
⎥ ⎢ ⎥ ⎢
⎥ ⎢
⎥ ⎢ ⎥ ⎢
⎥ ⎢ 2 ⎥ ⎢ 0 ⎥ ⎢ 1
⎦ ⎣
⎦ ⎣ ⎦ ⎣
⎤
2 ⎥⎥
⎥
0 ⎥⎥
⎥. Let
⎥
−2 ⎥⎥
⎥
⎥
0 ⎥⎦
v 3 = w3 −
⎡
⎢
⎢
⎢
⎢
⎢
= ⎢⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎡
⎢
⎢
⎢
⎢
⎢
= ⎢⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎧
⎡
⎪
⎢
⎪
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎪⎢⎢
⎨⎢
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎢
⎪
⎪
⎪
⎢
⎪
⎩⎣
⎤ ⎡
1 ⎥⎥ ⎢⎢
⎥ ⎢
1 ⎥⎥ ⎢⎢
⎥,⎢
⎥ ⎢
1 ⎥⎥ ⎢⎢
⎥ ⎢
⎥ ⎢
1 ⎥⎦ ⎢⎣
⎤ ⎡
2 ⎥⎥ ⎢⎢
⎥ ⎢
0 ⎥⎥ ⎢⎢
⎥,⎢
⎥ ⎢
−2 ⎥⎥ ⎢⎢
⎥ ⎢
⎥ ⎢
0 ⎥⎦ ⎢⎣
1
1
3
3
1
1
1
1
⎤
⎥
⎥
⎥
⎥
⎥
⎥.
⎥
⎥
⎥
⎥
⎥
⎥
⎦
⎤⎫
⎪
0 ⎥⎥⎪
⎪
⎪
⎪
⎥⎪
⎪
⎪
⎥
⎪
−1 ⎥⎪
⎥⎪
is an orthogonal basis for span{w1 , w2 , w3 }.
⎥⎬
⎪
0 ⎥⎥⎪
⎪
⎪
⎥⎪
⎪
⎥⎪
⎪
⎪
1 ⎥⎦⎪
⎪
⎭
(b) Find an orthonormal basis {q1 , q2 , q3 } for span{w1 , w2 , w3 }. Recall that orthonor-
104
mal basis is when qi ⊥ qj for all i ≠ j and ∣∣qi ∣∣ = 1 for all i. Thus we have:
q1 =
v1
,
∣∣v1 ∣∣
q2 =
v2
,
∣∣v3 ∣∣
q3 =
v3
,
∣∣v3 ∣∣
§4.3 Linear Transformations / Change of Basis Formula
Definition 106
T ∶ Rn → Rm (T a mapping) is a linear transformation if
(1) T (x + y) = T (x) + T (y) for all x, y ∈ Rn
(2) T (cx) = cT (x) for all c ∈ R and x ∈ Rn
105
Friday, 28 March 2014
§4.3 The Matrix of a Linear Transformation/Change of Basis Formula
Definition 107 (Linear Transformation)
T ∶ Rn → R is a linear transformation if
1) T (x + y) = T (x) + T (y) for all x, y ∈ Rn
2) T (cx) = cT (x) for all c ∈ R, x ∈ Rn .
Remark. The behavior of T on V = span{v1 , . . . , vk } is completely determined by T (v1 ), . . . , T (vk ),
x ∈ V (we want to compute T (x), where T is a linear transformation). This implies x = c1 v1 +
c2 v2 + ⋯ + ck vk and it follows
T (x) = T (c1 v1 + ⋯ + ck vk ) = T (c1 v1 ) + T (c2 v2 ) + ⋯ + T (ck vk ) = c1 T (v1 ) + c2 T (v2 ) + ⋯ + ck T (vk ).
Examples
(1) If A is an m × n matrix, then T (x) = Ax is a linear transformation.
(2) If V is a subspace of Rn , then projV (⋅) (where ⋅ means it can be anything) is a linear
transformation/linear map (linear operator – when spaces are the same) from Rn
to Rn .
(3) Given V ⊆ Rn , V is a subspace, we can define the map R(x) = projV (x)−projV ⊥ (x),
which is linear (by (2)). R is called the refection across V .
Figure 18:
106
The Standard Matrix for a Linear Transformation T
Let T ∶ Rn → Rn be a linear map (linear transformation) and let E = {e1 , . . . , en } be the stan⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
n
dard basis for R , i.e., ei = ⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
0 ⎞
⎟
⋮ ⎟
⎟
⎟
⎟
0 ⎟
⎟
⎟
⎟
with 1 in the ith position. The m × n matrix whose columns are
1 ⎟
⎟
⎟
0 ⎟
⎟
⎟
⎟
⋮ ⎟
⎟
⎟
0 ⎠
T (e1 ), T (e2 ), . . . , T (en ) is called the standard matrix of T and is denoted by [T ]E or [T ]stand .
Proposition 32. Given a linear transformation T ∶ Rn → Rm and x ∈ Rn , then T (x) = [T ]E x.
⎡
⎤
⎢ x1 ⎥
⎢
⎥
⎢
⎥
⎢
⎥
Proof. Let x = ⎢ ⋮ ⎥ ∈ Rn . Then x = x1 e1 + x2 e2 + ⋯ + xn en . Then
⎢
⎥
⎢
⎥
⎢ x ⎥
⎢ n ⎥
⎣
⎦
⎡
⎢
∣
⎢
⎢
⎢
T (x) = x1 T (e1 )+x2 T (e2 )+⋯+xn T (en ) = ⎢ T (e1 )
⎢
⎢
⎢
∣
⎢
⎣
∣
T (e2 )
∣
⋯
⎡
⎤ ⎢⎢
⎥
∣
⎥ ⎢⎢
⎥⎢
⎥
T (en ) ⎥⎥ ⎢⎢
⎥ ⎢⎢
⎥⎢
∣
⎥⎢
⎦⎢
⎣
⎤
x1 ⎥⎥
⎥
x2 ⎥⎥
⎥ = [T ]E x.
⎥
⋮ ⎥⎥
⎥
⎥
xn ⎥⎦
Example
Let V = {(x1 , x2 , 0) ∶ x1 , x2 ∈ R} be the x1 x2 -plane in R. Find [projV ]E or [projV ]stand .
x ∈ R3 ⇒ x
x∈V ⇒x
⎡
⎤
⎢ x1 ⎥
⎢
⎥
⎢
⎥
⎢
⎥
= x1 e1 + x2 e2 + x3 e3 where x = ⎢ x2 ⎥
⎢
⎥
⎢
⎥
⎢ x ⎥
⎢ 3 ⎥
⎣
⎦
⎡
⎤
⎢ x1 ⎥
⎢
⎥
⎢
⎥
⎢
⎥
= x1 e1 + x2 e2 + 0e3 where x = ⎢ x2 ⎥
⎢
⎥
⎢
⎥
⎢ 0 ⎥
⎢
⎥
⎣
⎦
107
This implies V = span{e1 , e2 }.
⎡
⎢
∣
⎢
⎢
⎢
projV tE = ⎢ projV e1
⎢
⎢
⎢
∣
⎢
⎣
= [ e1
⎡
⎢ 1
⎢
⎢
⎢
=⎢ 0
⎢
⎢
⎢ 0
⎢
⎣
e2
0
1
0
∣
projV e2
∣
⎤
⎥
⎥
⎥
⎥
projV e3 ⎥⎥
⎥
⎥
∣
⎥
⎦
∣
0 ]
⎤
0 ⎥⎥
⎥
⎥
0 ⎥⎥
⎥
0 ⎥⎥
⎦
= [projV ]E
⎤
⎡
⎡ ⎤
⎢ 1 ⎥
⎢ 1 ⎥
⎥
⎢
⎢ ⎥
⎥
⎢
⎢ ⎥
⎥
⎢
⎢ ⎥
3
⎥. Find [projV ]E .
⎢
⎥
⎢
Now let V ⊆ R be the plane spanned by v1 = 0 , v2 =
1
⎢
⎢ ⎥
⎥
⎢ ⎥
⎥
⎢
⎢ −1 ⎥
⎢ 1 ⎥
⎥
⎢
⎢ ⎥
⎦
⎣
⎣ ⎦
Note:
{v1 , v2 } is an orthogonal set (in fact, it is an orthogonal basis for V ). So, projV (x) =
projv1 (x) + projv2 (x) for all x ∈ R3 .
projV (e1 ) = projv1 (e1 ) + projv2 (e2 )
projV (e1 ) = projv1 (e1 ) + projv2 (e2 )
projV (e1 ) = projv1 (e1 ) + projv2 (e2 )
⎤ ⎡
⎡ ⎤
⎡
⎢ 1 ⎥
⎢ 1 ⎥ ⎢
⎥ ⎢
⎢ ⎥
⎢
⎥ ⎢
v1 ⋅ e1 v2 ⋅ e2 1 ⎢⎢ ⎥⎥ 1 ⎢⎢
⎥ ⎢
⎥
⎥=⎢
⎢
⎢
+
+
=
=
0
1
⎥ ⎢
∣∣v1 ∣∣2
∣∣v2 ∣∣2 2 ⎢⎢ ⎥⎥ 3 ⎢⎢
⎥ ⎢
⎢ 1 ⎥
⎢ −1 ⎥ ⎢
⎥ ⎢
⎢ ⎥
⎢
⎦ ⎣
⎣ ⎦
⎣
⎡ 1 ⎤
⎢
⎥
⎢ 3 ⎥
⎢
⎥
⎢
⎥
= ⎢ 13 ⎥
⎢
⎥
⎢
⎥
⎢ −1 ⎥
⎢ 3 ⎥
⎣
⎦
⎡ 1 ⎤
⎢
⎥
⎢ 6 ⎥
⎢
⎥
⎢ 1 ⎥
= ⎢ −3 ⎥
⎢
⎥
⎢
⎥
⎢ 5 ⎥
⎢ 6 ⎥
⎣
⎦
108
5
6
1
3
1
6
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
⎡
⎢
⎢
⎢
⎢
[projV ]E = ⎢
⎢
⎢
⎢
⎢
⎣
5
6
1
3
1
6
1
3
1
3
− 31
1
6
− 13
5
6
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
Example
Find the [projV ⊥ ]E where V is a given in the last example. For any x ∈ Rn , x =
projV (x) + projV ⊥ (x) which implies
projV ⊥ (x) = x − projV (x) = x − [projV ]E x
= Ix − [projV ]E x
⎤⎤
⎤ ⎡
⎡⎡
1
1 ⎥⎥
⎢⎢ 1 0 0 ⎥ ⎢ 5
⎥ ⎢ 6
⎢⎢
3
6 ⎥⎥
⎥⎥
⎥ ⎢
⎢⎢
⎥⎥
⎥ ⎢
⎢⎢
1
= ⎢⎢ 0 1 0 ⎥ − ⎢ 13
− 31 ⎥⎥⎥⎥
3
⎥ ⎢
⎢⎢
⎥⎥
⎥ ⎢
⎢⎢
⎢⎢ 0 0 1 ⎥ ⎢ 1 − 1
5 ⎥⎥
⎥⎥
⎥ ⎢ 6
⎢⎢
3
6
⎦⎦
⎦ ⎣
⎣⎣
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
⎡ 1
⎢
⎢ 6
⎢
⎢
= ⎢ − 31
⎢
⎢
⎢ −1
⎢ 6
⎣
[projV ⊥ ]E
− 13
2
3
1
3
109
⎤
− 16 ⎥⎥
⎥
1 ⎥
⎥x
3 ⎥
⎥
1 ⎥
6 ⎥
⎦
Monday, 31 March 2014 & Tuesday, 1 April 2014
Example
⎛ 1 ⎞
⎛ 1 ⎞
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
Let V = Span(v1 , v2 ) where v1 = ⎜ 0 ⎟ and v2 = ⎜ 1 ⎟.
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎝ 1 ⎠
⎝ −1 ⎠
Note: {v1 , v2 } is an orthogonal basis for V . To find the standard basis of projection P
onto v1 , you compute P (e1 ), P (e2 ), and P (e3 ).
[P ]stand = P (e1 )
(
P (e2 )
⎡
⎢
⎢
⎢
⎢
P (e3 ) = ⎢⎢
⎢
⎢
⎢
⎣
)
5
6
1
3
1
6
1
3
1
3
− 13
1
6
− 13
5
6
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
where P (ei ) = projv1 (ei ) + projv2 (ei ).
Definition 108
Let T ∶ Rn → Rn be a linear transformation, let B = {v1 , v2 , . . . , vn } be a basis for Rn . Then
the n × n matrix whose columns are the B-coordinate vectors of T (v1 ), . . . , T (vn ) is called the
matrix of T with respect to B , denoted [T ]B . That is, if T (vj ) = a1j v1 + a2j v2 + ⋯ + anj vn
⎡
⎤
⎢ a1j ⎥
⎢
⎥
⎢
⎥
⎢ a ⎥
⎢ 2j ⎥
⎥.
then the jth column of [T ]B is ⎢⎢
⎥
⎢ ⋮ ⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢ anj ⎥
⎣
⎦
Example
⎧
⎫
⎪
⎪
⎪
1 ⎪
⎧
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
x1 + x3 = 0
⎪
⎪
⎪
⎪
⎪
⎪
If we solve ⎨
we find that ⎨ −2 ⎬ is a basis for V ⊥ . So B = {v1 , v2 , v3 }
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
x + x2 − x3 = 0
⎪
⎪
⎪
⎪
⎪
⎩ 1
⎪
⎪
−1
⎪
⎪
⎩
⎭
´¹¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¶
is a basis for R3 . Find [P ]B . Note
P (v1 ) = v1 ,
v3
P (v2 ) = v2 ,
110
P (v3 ) = 0.
Figure 19:
P (v1 ) = v1
= 1v1 + 0v2 + 0v3
P (v2 ) = v2
= 0v1 + 1v2 + 0v3
P (v3 ) = v3
= 0v1 + 0v2 + 0v3 .
⎤
⎡
⎢ 1 0 0 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
[P ]P = ( [P (v1 )]B [P (v2 )]B [P (v3 )]B ) = ⎢ 0 1 0 ⎥ .
⎥
⎢
⎥
⎢
⎢ 0 0 0 ⎥
⎥
⎢
⎦
⎣
n
n
Theorem 109 (Change of Basis). Let T ∶ R → R be a linear transformation, let B = {v1 , . . . , vn }
be a basis for Rn . Let P = ( v1
⋯
−1
vn ). Then [T ]stand = P [T ]B P .
Proof. We prove that [T ]stand P = P [T ]B .
´¹¹ ¹ ¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¹ ¹ ¶
²
B
A
AP = A ( v1
⋯
vn
P B = P ( [T (v1 )]B
Note: That if [T (vi )]B
⎛ ∣
⎜
⎜
) = ⎜ Av1
⎜
⎜
⎝ ∣
⋯
∣
⋯
Avn
∣
⎞
⎟
⎟
⎟ = (T (v1 ) + ⋯ + T (vn ))
⎟
⎟
⎠
[T (vn )]B ) = ( P [T (v1 )]B
⋯
P [T (vn )]B ) .
⎛ α1 ⎞
⎜
⎟
⎜
⎟
= ⎜ ⋮ ⎟ that this implies T (vj ) = α1 v1 + ⋯ + αn vn and it
⎜
⎟
⎜
⎟
⎝ αn ⎠
111
follows
P [T (vj )]B
⎛ α1 ⎞
⎜
⎟
⎜
⎟
= P ⎜ ⋮ ⎟ = ( v1
⎜
⎟
⎜
⎟
⎝ αn ⎠
⋯
vn
⎛ α1 ⎞
⎜
⎟
⎜
⎟
) ⎜ ⋮ ⎟ = α1 + v1 + ⋯ + αn vn = T [vj ].
⎜
⎟
⎜
⎟
⎝ αn ⎠
Remark. P is called a change of basis matrix .
Example
Let T ∶ R3 → R3 be the linear transformation that rotates vectors by an angle
2π
about
3
⎛ 1 ⎞
⎟
⎜
⎟
⎜
the axis by the span of ⎜ −1 ⎟ in the counterclockwise direction. When viewed from
⎟
⎜
⎟
⎜
⎝ 1 ⎠
⎛ 1 ⎞
⎟
⎜
⎟
⎜
the tip of ⎜ −1 ⎟ toward the origin. Find [T ]stand .
⎟
⎜
⎟
⎜
⎝ 1 ⎠
Solutions Outline
1. Find a “natural” basis B for R3 that is adapted to this problem.
2. Find [T ]B
3. [T ]stand = P [T ]B P −1
Solution
⎛ 1 ⎞
⎜
⎟
⎜
⎟
1. Let v3 = ⎜ −1 ⎟. Then x1 − x2 + x3 = 0 gives the Cartesian op. of the
⎜
⎟
⎜
⎟
⎝ 1 ⎠
plane orthogonal to v1 . Let x2 = 1 and x3 = 0 then we get x1 = 1 so
⎛ 1 ⎞
⎜ ⎟
⎜ ⎟
w1 = ⎜ 1 ⎟ is in the plane. Let x1 = 0 and x2 = 1 then we get x3 = 0 so
⎜ ⎟
⎜ ⎟
⎝ 0 ⎠
112
⎛ 0 ⎞
⎜ ⎟
⎜ ⎟
w2 = ⎜ 1 ⎟ is in the plane. Now let us use Gram-Schmidt:
⎜ ⎟
⎜ ⎟
⎝ 1 ⎠
Let v1 = w1
⎛ 0 ⎞
⎛ 1 ⎞
⎜ ⎟ 1⎜ ⎟
v 1 ⋅ w2
⎜ ⎟
⎜ ⎟
v1 = ⎜ 1 ⎟ − ⎜ 1 ⎟ =
Let v2 = w2 −
⎜ ⎟ 2⎜ ⎟
∣∣v1 ∣∣2
⎜ ⎟
⎜ ⎟
⎝ 1 ⎠
⎝ 0 ⎠
v2
v3
v1
,q =
, and q3 =
which yields:
∣∣v1 ∣∣ 2 ∣∣v2 ∣∣
∣∣v3 ∣∣
⎛ 1 ⎞
⎟
1 ⎜
⎜ ⎟
q1 = √ ⎜ 1 ⎟ ,
⎟
2⎜
⎜ ⎟
⎝ 0 ⎠
⎛ −1 ⎞
⎟
1 ⎜
⎟
⎜
q2 = √ ⎜ 1 ⎟ ,
⎟
⎜
6⎜
⎟
⎝ 2 ⎠
⎛ −1
⎜ 2
⎜ 1
⎜ 2
⎜
⎜
⎝ 1
⎞
⎟
⎟
⎟. Let q1 =
⎟
⎟
⎠
⎛ 1 ⎞
⎟
1 ⎜
⎟
⎜
q3 = √ ⎜ −1 ⎟ .
⎟
⎜
3⎜
⎟
⎝ 1 ⎠
Figure 20:
T (q3 ) = q3
2π
2π
) q1 + sin ( ) q2 + 0q3
3
3
2π
2π
T (q2 ) = − sin ( ) q1 + cos ( ) q2 + 0q3
3
3
⎤
⎡
⎢ cos ( 2π ) − sin ( 2π ) 0 ⎥
⎢
⎥
3
3
⎥
⎢
⎢
⎥
2π
2π
B = {q1 , q2 , q3 }[T ]B = ⎢ sin ( 3 ) cos ( 3 ) 0 ⎥
⎢
⎥
⎢
⎥
⎢
0
0
1 ⎥⎥
⎢
⎣
⎦
T (q1 ) = cos (
2. [T ]B = ( [T (q1 )]
B
[T (q2 )]B
[T (q3 )]B ).
113
Figure 21:
2π
2π
) q1 + sin ( ) q2
3
3
2π
2π
T (q2 ) = − sin ( ) q1 + cos ( ) q2 + 0q3
3
3
T (q1 ) = cos (
T (q3 = q3
⎤
⎡
⎢ cos ( 2π ) − sin ( 2π ) 0 ⎥
⎥
⎢
3
3
⎥
⎢
⎥
⎢
2π
2π
[T ]B = ⎢ sin ( 3 ) cos ( 3 ) 0 ⎥ .
⎥
⎢
⎥
⎢
⎢
0
0
1 ⎥⎥
⎢
⎦
⎣
3. Then [T ]stand = P [T ]B P −1
⎡ 1
⎢ √
⎢ 2
⎢
⎢
where P = ⎢ √1
⎢ 2
⎢
⎢ 0
⎢
⎣
we row reduce the augmented matrix ( P
⎛ 0
⎜
⎜
and P [T ]B P −1 = ⎜ 0
⎜
⎜
⎝ 1
−1
0
0
− √16
√1
3
√1
6
⎛
⎜
⎜
I ) Ð→ ⎜
⎜
⎜
⎝
⎤
⎥
⎥
⎥
⎥
⎥. To find P −1
⎥
⎥
⎥
⎥
⎦
√1
√1
0 ⎞
2
2
⎟
⎟
√1
√6 ⎟
− √16
6
5 ⎟
⎟
1
0
− √13
⎠
3
√1
3
1
− √3
√1
3
I
0 ⎞
⎟
⎟
.
−1 ⎟
⎟
⎟
0 ⎠
Example
⎛ 3
Let T ∶ R2 → R2 given by T (x) = Ax where A = ⎜
⎝ 2
T is not obvious from the structure of A.
114
1 ⎞
⎟. The geometric meaning to
2 ⎠
⎛ 1 ⎞
⎛ −1 ⎞
⎟ and v2 = ⎜
⎟, then
Note if v1 = ⎜
⎝ 1 ⎠
⎝ 2 ⎠
⎛ 3
T (v1 ) = ⎜
⎝ 2
⎛ 1 ⎞
1 ⎞⎛ 1 ⎞ ⎛ 4 ⎞
⎟⎜ ⎟ = ⎜ ⎟ = 4⎜
⎟ = 4v1
⎝ 1 ⎠
2 ⎠⎝ 1 ⎠ ⎝ 4 ⎠
⎛ 3
T (v2 ) = ⎜
⎝ 2
1 ⎞ ⎛ −1 ⎞ ⎛ −1 ⎞
⎟⎜
⎟=⎜
⎟ = v2 .
2 ⎠⎝ 2 ⎠ ⎝ 2 ⎠
Let B = {v1 , v2 }. That is a basis for R2 .
Figure 22:
[T ]B = ( [T (v1 )]B
[T (v2 )]B
⎡
⎢ 4
⎢
)=⎢
⎢
⎢ 0
⎣
⎤
0 ⎥⎥
⎥.
⎥
1 ⎥⎦
The matrix T with respect to B makes the geometric action of T on R classes, but how
to find B?
Definition 110
Let A and B be n × m matrices. Then A and B are similar if there exists an invertible matrix P
such that A = P BP −1 .
Remark. A and B are similar if they are matrices of the same linear transformation with respect
to difference bases.
Definition 111
A matrix is called diagonalizable if it is similar to a diagonal matrix.
115
§4.4 Linear Transformations on Abstract Vector Spaces.
Definition 112
Let V and W be vectors spaces. A map T ∶ V → W is called a linear transformation (or
linear map ) if
1) T (u + v) = T (u) + T (v) for all u, v ∈ V
2) T (cu) = cT (u) for all c ∈ R and u ∈ V .
Examples
1) D ∶ C ′ [0, 1] → C ′ ([0, 1]) where C ′ ([0, 1]) is a different function defined on [0, 1]
with continuous ??? D(f ) = f ′ .
Proof. This is linear!
1) D(f + g) = (f + g) = f ′ + g ′ = D(f ) + D(g).
2) D(cf ) = (cf )′ = cf ′ = cD(f )
2) C ○ [0, 1] → R T (f ) = ∫
1) T (f + g) = ∫
2) T (cf ) = ∫
0
1
1
0
1
0
f (x)dx is linear because
(f + g)(x)dx = ∫
cf (x)dx = c ∫
0
1
0
1
f (x)dx + ∫
f (x)dx = cT (f )
3) S ∶ C [0, 1] → R S(f ) = f ′ (0).
116
1
0
g(x)dx = T (f ) + T (g)
Thursday, 3 April 2014
Example
1. D ∶ C ′ ([0, 1]) → C 0 [0, 1]
2. D(f ) = f ′
3. T ∶ C ′ ([0, 1]) → R, T (f ) = f ′ (o)
Definition 113
Let T ∶ V → W be a linear map. Then the set ker(T ) = {v ∈ V ∶ T (v) = 0 ∈ W } is called the
kernel of T . The set image of T = {w ∈ W ∶ w = T (v) for some v ∈ V } is called the image (or
range ) of T .
Definition 114
T is called onto or surjective if for every w ∈ W there exists v ∈ V such that T (v) = w.
Proposition 33. Let T ∶ V → W be a linear map. Then
1) T is onto if image(T ) = W .
2) T is one-to-one of ker T = {0}.
Proof.
1. Follows from definitions
2. Suppose T (v1 ) − T (v2 ) implies T (v1 ) − T (v2 ) = 0 and because of linearity T (v1 −
v2 ) = 0 which implies v1 − v2 ∈ ker T and it follows v1 − v2 = 0 and thus v1 = v2 .
117
Example
Find the kernel and image of
1. D ∶ P3 → P2 , D(P ) = P ′ .
ker(D) = {constant polynomial}
image(D) = P2 because if p(t) = a1 t + a2 t2 if we let q(t) = a0 t +
a1 2
t
2
+
a2 3
t
3
then
D(q) = p
⎛ f (0) ⎞
⎟
2. T ∶ P2 → R2 , T (f ) = ⎜
⎝ f (1) ⎠
ker T = {p ∈ P2 , with roots at 0 and 1} = { polynomial of degree 2 with roots at 0
and 1}
Claim: image (T ) = R2 .
⎛ a ⎞
⎟ ∈ R and let f (t) = a + (b − a)t then f (0) = a, f (1) = b so
Proof. Let ⎜
⎝ b ⎠
⎛ a ⎞
⎟.
T (f ) = ⎜
⎝ b ⎠
⎛ f (0) ⎞
⎜
⎟
⎜
⎟
3. S ∶ P2 → R3 , S(f ) = ⎜ f (1) ⎟.
⎜
⎟
⎜
⎟
⎝ f (2) ⎠
Let (S) = {polynomial of degree at most 2 with zeros at x = 0, 1, 2} = {zero
polynomial}.
image S = R3 by Lagrange Interpolation Theorem
Definition 115
Any linear map that is one-to-one and onto is called a (linear) isomorphism.
118
Proposition 34. T ∶ V → W is a linear map if and only if an inverse, i.e., there exists a map
S ∶ W → V such that S ○ T = v for all v ∈ W and T ○ S(W ) = w for all w ∈ W .
Proof. (⇒) Suppose T ∶ V → W is an isomorphism. We construct an inverse map
S ∶ W → V . Let w ∈ W . Since T is onto there exists v ∈ V such that T (v) = W .
S(W ) = vector in V that gets mapped by T to W = V . Because T is one-to-one
this v is the unique vector in V that gets mapped to w. By construction, this is
an inverse.
(⇐) Exercise
119
Friday, 4 April 2014
Definition 116
A linear map T ∶ V → W that is one-to-one and onto is called an isomorphism.
Proposition 35. T ∶ V → W is an isomorphism if and only if there exists S ∶ W → W such that
T ○ S = IdW and S ○ T = IdV .
Example
Given a finite dimensional space V with basis B = {v1 , . . . , vn }. Then there exists an
isomorphism CB ∶ V → Rn . Any finite vector space of dim V = n is isomorphic to Rn .
Let v ∈ V then there
⎛ c1
⎜
⎜
define CB (v) = ⎜ ⋮
⎜
⎜
⎝ cn
exists unique scalars c1 , . . . , cn so that v = c1 v1 + ⋯ + cn vn . Then
⎞
⎟
⎟
⎟.
⎟
⎟
⎠
Let us check that it is an isomorphism.
1. One-to-One
Suffices to show that ker CB = {0}. Let v ∈ ker CB and show v = 0. We
⎛ c1 ⎞
⎜
⎟
⎜
⎟
can express v = c1 v1 + ⋯ + cn vn . 0 = CB (v) = ⎜ ⋮ ⎟, ⋆ – because 0 is in
⎜
⎟
⋆
⎜
⎟
⎝ cn ⎠
the kernel. So v = 0.
2. Onto
⎛ a1 ⎞
⎛ a1 ⎞
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
Let ⎜ ⋮ ⎟ ∈ Rn . Let v = a1 v1 + ⋯ + an vn . Then CB (v) = ⎜ ⋮ ⎟.
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎝ an ⎠
⎝ an ⎠
Corollary 117
Let V and W be two vector spaces of the same dimension. Then they are isomorphic.
Proof. Let n = dim V = dim W . Let B1 be a basis for V and B2 be a basis for W . Then
120
by the example, there exists isomorphism.
CB1 ∶ V → Rn ,
CB2 ∶ W → Rn .
Let
CB1
CB2
−1
T = CB
○ CB1 ∶ V Ð→ Rn Ð→ W.
2
−1
Why is T ∶ CB
○ CB1 an isomorphism?
2
−1
To prove this, we should show that has an inverse. Let S = CB
○ CB (recall - CBi are
1
−1
isomorphism and therefore CB
exist).
i
Exercise: Check that T ○ S = IdW and S ○ T = IdV .
Definition 118
Let V, W be finite dimensional vector space. Let BV = {v1 , . . . , vn } be a basis for V , let BW =
{w1 , . . . , wm } be a basis for W . Let T ∶ V → W be a linear transformation. For each 1 ≤ j ≤ n
T (vj ) = a1j w1 + ⋯ + amj wm for some scalars a1j , . . . , amj . Then matrix A = (aij ), an m × n matrix,
is called the matrix of T with respect to the basis V and W denoted [T ]BV BW .
V
T
W
CBV
Rn
CBW
A
Rm
Example
Write the matrix of the derivative map D ∶ P3 → P2 . Df = f ′ with respect to the
standard basis for P3 and P2 .
P3 = {1, t, t2 , t3 },
D(1) = 0;
D(t) = 1;
P2 = {1, t, t2 }.
D(t2 ) = 2t;
D(1) = 0 ⋅ 1 + 0t + 0t2
D(t) = 1 ⋅ 1 + 0t + 0t2 .
121
D(t3 ) = 3t2 .
⎛ 0
⎜
⎜
The matrix of D with respect to these bases ⎜ 0
⎜
⎜
⎝ 0
1
0
0
2
0
0
0 ⎞
⎟
⎟
.
0 ⎟
⎟
⎟
3 ⎠
Suppose f (t) = 4 + 3t + t2 + 14t3
⎛ 0
⎜
⎜
⎜ 0
⎜
⎜
⎝ 0
1
0
0
2
0
0
⎛
0 ⎞⎜
⎟⎜
⎟⎜
⎜
0 ⎟
⎟⎜
⎟⎜
⎜
3 ⎠⎜
⎝
4 ⎞
⎟ ⎛ 3 ⎞
⎟
3 ⎟
⎟ ⎜
⎟
⎟=⎜
⎟
2
⎟ ⎜
⎜
⎟
⎟
1 ⎟
⎟ ⎜
⎟ ⎝ 42 ⎠
14 ⎠
and note that f ′ (t) = 3 + 2t + 42t2 .
Example Two Questions
1) Find all polynomials of degree at most 3 that satisfy the differential equation
f ′′ + 4f − 5f = 0. That is can you find the kernel of T ?
2) Given g ∈ P3 . Can you find f ∈ P3 so that f ′′ + 4f ′ − 5f = g.
Let T be a linear transformation that acts on P3 and is given by T (f )(t) = f ′′ (t) +
4f ′ (t) − 5f (t).
Question 1 Find kernel of T ?
Question 2 Is image(T ) = P3 i.e., is T onto?
Lemma 119
Let V and W be vector spaces with bases BV = {v1 , . . . , vm } and BW = {w1 , . . . , wn }. Let
T ∶ V → W be a linear transformation and let A = [T ]BV ,BW . Then
⎛ x1
⎜
⎜
1) x = ⎜ ⋮
⎜
⎜
⎝ xm
⎞
⎟
⎟
⎟ ∈ N (A) if and only if x1 v1 + ⋯ + xm vm ∈ ker T .
⎟
⎟
⎠
⎛ y1 ⎞
⎜
⎟
⎜
⎟
2) y = ⎜ ⋮ ⎟ ∈ C(A) if and only if y1 w1 + ⋯ + yn wn ∈ image(T ).
⎜
⎟
⎜
⎟
⎝ yn ⎠
122
Example
⎛
⎜
⎜
⎜
The matrix of T with respect to these basis is A = ⎜
⎜
⎜
⎜
⎜
⎝
Answer 1 Yes, only x = 0, trivial solution.
C(A) = R4 .
123
−5
4
2
0
−5
8
0
0
−5
0
0
0
0 ⎞
⎟
6 ⎟
⎟
⎟.
⎟
12 ⎟
⎟
⎟
−5 ⎠
Monday, 7 April 2014
Example
Let T ∶ V → W be a linear transformation. Let B1 = {v1 , . . . , vn } and B2 = {w1 , . . . , wm }
be bases for V and W .
Definition 120
The matrix for T with respect to B1 and B2 is the m×n matrix whose columns are the B2 -coordinates
of T (v1 ), . . . , T (vn ), denoted [T ]B1 ,B2 .
Theorem 121. Let CB1 ∶ V → Rn and CB2 ∶ W → Rm be the isomorphism associated with B1 and
B2 . Let µA ∶ Rn → Rm given by µA (x) = Ax, where A = [T ]B1 ,B2 . Then T = CB−12 ○ µA ○ CB1 .
V
T
W
CB1
CB2
Rn
Rm
µA
C stands for coordinate.
Given V with basis B = {v1 , . . . , vn } then CB ∶ V → R is defined by sending v ∈ V to its B-coordinate
vector.
This is equivalent to defining CB in the following way
v1 → e1
CB ∶
v2 → e2
⋮
vn → en
124
Then
CB (v) = CB (c1 v1 + ⋯ + cn vn )
= c1 CB (v1 ) + ⋯ + cn CB (vn )
= c1 e1 + ⋯ + cn en
[T ]B
Rn
Rn
P
P
T ∶ Rn
[T ]stand
Rn
[T ]stand = P [T ]
Example
T ∶ P3 → P3 given by T (f ) = f ′′′ + 4′ − 5f . Let B1 = {1, t, t2 , t3 } and B2 = {1, t, t2 , t3 }.
T (1) = −5
T (t) = 4 − 4t
T (t2 ) = 2 + 8t − 5t2
T (t3 ) = 6t + 12t2 − 5t3
⇒ [T (1)]B2
⎡
⎢
⎢
⎢
⎢
⎢
= ⎢⎢
⎢
⎢
⎢
⎢
⎢
⎣
Then
A = [T ]B1 ,B2
⎤
⎡
⎤
⎡
⎤
⎡
⎤
⎢ 0 ⎥
⎢ 2 ⎥
⎢ 4 ⎥
−5 ⎥⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
6
8
0 ⎥
⎢
⎥
⎢
⎢ −5 ⎥
⎥
⎥ , [T (t)]B = ⎢
⎥.
⎥ , [T (t3 )]B = ⎢
⎥ , [T (t2 )]B = ⎢
2
2
2
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
0 ⎥
⎢ 12 ⎥
⎢ −5 ⎥
⎢ 0 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢ −5 ⎥
⎢ 0 ⎥
⎢ 0 ⎥
0 ⎥⎦
⎦
⎣
⎦
⎣
⎦
⎣
⎡
⎢
⎢
⎢
⎢
⎢
= ⎢⎢
⎢
⎢
⎢
⎢
⎢
⎣
−5
4
2
0
−5
8
0
0
−5
0
0
0
125
⎤
0 ⎥⎥
⎥
6 ⎥⎥
⎥
⎥
12 ⎥⎥
⎥
⎥
−5 ⎥⎦
Use the theorem to compute, for example T (f ) where f (t) = 5 + 3t + 12t2 + 2t3 .
⎛ ⎛
⎜ ⎜
⎜ ⎜
⎜ ⎜
−1 ⎜ ⎜
T (f ) = CB2 ⎜A ⎜
⎜ ⎜
⎜ ⎜
⎜ ⎜
⎝ ⎝
⎛
5 ⎞⎞
⎟⎟
⎜
⎟
⎜
3 ⎟
⎟⎟
⎜
−1
⎟⎟ = C ⎜
B2 ⎜
⎟⎟
⎟
⎜
12 ⎟
⎟⎟
⎜
⎟⎟
⎜
⎠
⎠
⎝
2
11 ⎞
⎟
93 ⎟
⎟
⎟ = 11 + 93t − 36t2 − 10t3 .
⎟
−36 ⎟
⎟
⎟
−10 ⎠
Lemma 122
Let A = [T ]B1 ,B2 . Then
1) N (A) ⊆ Rn is isomorphic to ker T ⊆ V .
2) C(A) ⊆ Rm is isomorphic to image(T ) ⊆ W .
RREF
For our example A ↝ I, which implies C(A) = R4 and N (A) = {0}. By Lemma 122 and Lemma
123, ker T = {0} and image T = W .
Lemma 123
If T ∶ V → W is an isomorphism and dim V = n then dim W = n.
Exercise: Likely to be on the exam.
We have shown image (T ) = W = P3 . This says for any g ∈ P3 there exists f ∈ P3 such that T (f ) = g
implies f ′′ + 4f ′ − 5f = g.
ker(T ) = {0} says the only f for which T (f ) = 0 is f = 0. If your interested in the differential
equation f ′′ + 4f ′ − 5f = 0 only has the zero polynomial solution among P3 .
126
Chapter 5 Determinants
§5.1 Properties of Determinants
Goal: Make a function that takes in n × n matrices and spits out a number.
Definition 124
A determinant map is a map det ∶ Mn×n → R that satisfies the following properties.
⎡
⎤
⎢ A1 ⎥
⎢
⎥
⎢
⎥
⎢
⎥
1) Linear in each row, i.e., if A = ⎢ ⋮ ⎥, then
⎢
⎥
⎢
⎥
⎢ A ⎥
⎢ n ⎥
⎣
⎦
⎛
⎜
⎜
⎜
⎜
⎜
det ⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
A1
⋮
cAj
⋮
An
⎛ A1 ⎞
⎞
⎟
⎟
⎜
⎟
⎜
⎟
⎜ ⋮ ⎟
⎟
⎟
⎜
⎟
⎟
⎜
⎟
⎟ = c det ⎜ A ⎟
⎜ j ⎟
⎟
⎟
⎜
⎟
⎜
⎟
⎟
⎜ ⋮ ⎟
⎟
⎟
⎜
⎟
⎟
⎜
⎟
⎝ An ⎠
⎠
and
⎛
A1
⎜
⎜
⎜
⋮
⎜
⎜
′′
det ⎜
⎜ Aj + Aj
⎜
⎜
⎜
⋮
⎜
⎜
⎝
An
2) Skew-Symmetric
⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
det ⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⎛
A1 ⎞
⎜
⎟
⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
Aj ⎟
⎜
⎟
⎜
⎜
⎟
= − det ⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎜
⎟
Ak ⎟
⎜
⎜
⎟
⎜
⎟
⎜
⋮ ⎟
⎜
⎟
⎜
⎟
⎝
⎠
An
3) det I = 1
127
A1 ⎞
⎟
⋮ ⎟
⎟
⎟
⎟
Ak ⎟
⎟
⎟
⎟
⋮ ⎟
⎟
⎟
Aj ⎟
⎟
⎟
⎟
⋮ ⎟
⎟
⎟
An ⎠
⎛ A1 ⎞
⎛ A1 ⎞
⎞
⎟
⎜
⎟
⎜
⎟
⎟
⎜
⎟
⎜
⎟
⎜ ⋮ ⎟
⎜ ⋮ ⎟
⎟
⎟
⎜
⎟
⎜
⎟
⎟
⎜
⎟
⎜
⎟
⎟ = det ⎜ A ⎟+det ⎜ A′′ ⎟ .
⎟
⎜ j ⎟
⎜ j ⎟
⎟
⎜
⎟
⎜
⎟
⎟
⎜
⎟
⎜
⎟
⎜ ⋮ ⎟
⎜ ⋮ ⎟
⎟
⎟
⎜
⎟
⎜
⎟
⎟
⎜
⎟
⎜
⎟
⎝ An ⎠
⎝ An ⎠
⎠
Tuesday, 8 April 2014
A determinant map is a function det ∶ Mn×n → R with the properties:
1) linear in each row
2) skew-symmetric (anti-symmetric)
3) det I = 1
Proposition 36. If two rows of A are equal then det A = 0.
Proof. det A = − det A from swapping the two equal rows of A and this implies det A =
0.
Proposition 37. If A has a row of zeros then det A = 0.
Proof. Homework.
Use linearity and we can pull out scalars.
(Proposition 38 is key for doing computations)
Proposition 38. Doing a replace elementary row operation does not change the determinant.
Proof. Consequence of linearity in each row.
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
Suppose A = ⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
A1
⋮
Aj
⋮
Ak
⋮
An
⎤
⎥
⎛
A1
⎥
⎜
⎥
⎜
⎥
⋮
⎜
⎥
⎜
⎥
⎜
⎥
⎥
⎜
Aj
⎜
⎥
⎜
⎥
⎜
⎥
⎥. Let B = ⎜
⋮
⎜
⎥
⎜
⎥
⎜ A + cA
⎥
⎥
⎜ k
j
⎥
⎜
⎜
⎥
⎜
⎥
⋮
⎜
⎥
⎜
⎥
⎥
⎝
⎥
An
⎦
128
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟.
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎠
⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
det B = det ⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⎛
A1 ⎞
⎟
⎜
⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎟
⎜
⎜
Aj ⎟
⎟
⎜
⎟
⎜
⎟
⎜
+ det ⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎜
Ak ⎟
⎟
⎜
⎟
⎜
⎟
⎜
⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎝
An ⎠
⎛
A1 ⎞
⎟
⎜
⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎟
⎜
⎜
Aj ⎟
⎟
⎜
⎟
⎜
⎟
⎜
= det ⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎜
cAj ⎟
⎟
⎜
⎟
⎜
⎟
⎜
⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎝
An ⎠
⎛
A1 ⎞
⎟
⎜
⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎟
⎜
⎜
Aj ⎟
⎟
⎜
⎟
⎜
⎟
⎜
+ c det ⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎜
Ak ⎟
⎟
⎜
⎟
⎜
⎟
⎜
⎜
⋮ ⎟
⎟
⎜
⎟
⎜
⎝
An ⎠
A1 ⎞
⎟
⋮ ⎟
⎟
⎟
⎟
Aj ⎟
⎟
⎟
⎟
⋮ ⎟
⎟
⎟
cAj ⎟
⎟
⎟
⎟
⋮ ⎟
⎟
⎟
An ⎠
= det A + c ⋅ matrix with equal rows = det A + c ⋅ 0 = 0.
Remark. We can use elementary row operations to compute determinants.
Examples
⎛ 2
⎜
⎜
1) det ⎜ 2
⎜
⎜
⎝ 1
⎛ 2
⎜
⎜
det ⎜ 2
⎜
⎜
⎝ 1
4
1
0
4
1
0
6 ⎞
⎟
⎟
0 ⎟
⎟
⎟
1 ⎠
⎛ 1
6 ⎞
⎜
⎟
⎜
⎟
= det ⎜ 2
0 ⎟
⎜
⎟
⎜
⎟
1 ⎠
⎝ 2
⎛ 1
⎜
⎜
= det ⎜ 0
⎜
⎜
⎝ 0
0
4
1
0
4
1
1 ⎞
⎟
⎟
6 ⎟
⎟
⎟
0 ⎠
⎛ 1
⎜
⎜
= − det ⎜ 0
⎜
⎜
⎝ 0
1 ⎞
⎟
⎟
4 ⎟
⎟
⎟
−2 ⎠
⎛ 1
⎜
⎜
= − det ⎜ 0
⎜
⎜
⎝ 0
129
0
1
4
0
1
0
1 ⎞
⎟
⎟
−2 ⎟
⎟
⎟
4 ⎠
1 ⎞
⎟
⎟
−2 ⎟
⎟
⎟
12 ⎠
⎛ 1
⎜
⎜
= −12 det ⎜ 0
⎜
⎜
⎝ 0
⎛ 1
⎜
⎜
= −12 det ⎜ 0
⎜
⎜
⎝ 0
= −12 ⋅ 1 = −12.
0
1
0
0
1
0
1 ⎞
⎟
⎟
−2 ⎟
⎟
⎟
1 ⎠
0 ⎞
⎟
⎟
0 ⎟
⎟
⎟
1 ⎠
⎛ 2
⎜
⎜
2) det ⎜ 4
⎜
⎜
⎝ 1
4
8
2
6 ⎞
⎟
⎟
12 ⎟
⎟
⎟
3 ⎠
⎛ 2
⎜
⎜
det ⎜ 4
⎜
⎜
⎝ 1
4
8
2
⎛ 1
6 ⎞
⎟
⎜
⎟
⎜
= − det ⎜ 4
12 ⎟
⎟
⎜
⎟
⎜
3 ⎠
⎝ 2
2
8
4
⎛ 1
3 ⎞
⎟
⎜
⎟
⎜
= − det ⎜ 0
12 ⎟
⎟
⎜
⎟
⎜
6 ⎠
⎝ 0
2
0
0
3 ⎞
⎟
⎟
=0
0 ⎟
⎟
⎟
0 ⎠
since we have a row of zeros.
Theorem 125. Let A be an n × n matrix. Then A is nonsingular if and only if det A ≠ 0.
Talk Through Proof. If A is nonsingular then the RREF of A is the identity matrix,
recalling that we never scale rows by zero, implying that det A ≠ 0.
If A is singular then the RREF of A is a matrix with at least one row of zeros implying that det A = 0.
Proposition 39. If A is upper (or lower) triangular with diagonal elements a11 , . . . , ann then
det A = a11 ⋯ann .
Proof. Two cases:
Case I:
aii ≠ 0 for all i.
⎛ a11
⎜
⎜
A=⎜
⎜
⎜
⎝
0
⋱
*
ann
130
⎞
⎟
⎟
⎟.
⎟
⎟
⎠
⎛ 1
⎜
⎜
⎜
det A = a11 det ⎜
⎜
⎜
⎜
⎜
⎝
a22
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎠
*
⋱
0
ann
⎛ 1
⎜
⎜
⎜
⎜
⎜
= a11 a22 det ⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
0
1
∗
a22
∗
a22
a33
*
⋱
0
⎞
⎟
⎟
⋯ ⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
ann ⎠
= a11 ⋯aann det I = a11 ⋯ann
Case II:
aii = 0 for some i
⎛ a11
⎜
⎜
⎜
⎜
⎜
A=⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⋱
*
0
⋱
0
ann
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎠
In this case our row reduction algorithm will reduce this matrix to a REF
with a row of zeros.
Proposition 40. (Intermediate) Let A be an n × n matrix and let E be an elementary matrix.
Then det(EA) = det E det A.
Proof. Case I:
Suppose E corresponds to a swap operation. Then det E = −1.
det(EA) = − det A = det E det A.
131
Case II:
D corresponds to a scale operation for c ≠ 0. Then det E = c and det EA =
c det A = det E det A.
Case III:
E corresponds to a replace. Exercise.
Proposition 41. Let A and B be n × n matrices then det AB = det A det B.
Proof. We will look at two cases; when 1) A is nonsingular and when 2) A is singular.
Recall If AB is nonsingular then A and B are nonsingular also (the contrapositive),
if A or B is singular, then AB is singular.
Case I:
A is nonsingular implies there exists some elementary matrices E1 , . . . , Ek so
that Ek ⋯E1 A = I. This tells us A = E1−1 ⋯Ek−1 , ⋆ – these are elementary
⋆
matrices as well. So AB = E1−1 ⋯Ek−1 B. So then
det AB = det(E1−1 E2−1 ⋯Ek−1 B)
= det(E1−1 ) det(E2−1 ⋯Ek−1 B)
= det(E1−1 ) det(E2−1 ) det(E3−1 ⋯Ek−1 B)
= det(E1−1 ) det(E2−1 )⋯ det(Ek−1 ) det B
= det(E1−1 ⋯Ek−1 ) det B
= det A det B.
Case II:
A is singular. Then AB is singular ( det A det B = det AB ). So det A = 0 =
²
´¹¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¶
det AB. Thus det AB = det A det B.
=0
132
=0
Corollary 126
Suppose A is nonsingular det A−1 =
1
.
det A
Proof. Given that AA−1 = I implies det(AA−1 ) = 1 and it follows
det A det A−1 = 1
Ô⇒
133
det A−1 =
1
.
det A
Thursday, 10 April 2014
Homework Discussion
Let V and W be vector spaces of dimensions n and m respectively. Let V and W be bases for V
and W .
Give these bases there exists isomorphism
CV ∶ V → Rn
CW ∶ W → Rm
Let T ∶ V → W be a linear transformation and let A be its matrix with respect to V and W.
V
T
W
CV
Rn
CW
µA
Rm
−1
Theorem T = CW
○ µA ○ CV
4.4.12 Claim:
If v ∈ ker T then x = CV (v) ∈ N (A).
−1
−1
−1
0 = T (v) = (CW
○ µA ○ CV )(v) = (CW
○ µA )(x) = CW
(Ax).
−1
Because CW
is one-to-one we get Ax = 0.
4.4.5 T ∶ P3 → P3 where T (f )(t) = 2f (t) + (1 − t)f ′ (t).
a. Show T is linear. f, g ∈ P3 .
T (f + g)(t) = 2(f + g)(t) + (1 − t)(f + g)′ (t) = 2(f (t) + g(t)) + (1 − t)(f ′ (t) + g ′ (t)) = ...
⎛
⎜
⎜
⎜
2 3
2
3
Let B = {1, t, t , t } where CB (a0 + a1 t + a2 t + a3 t ) = ⎜
⎜
⎜
⎜
⎜
⎝
a0 ⎞
⎟
a1 ⎟
⎟
⎟.
⎟
a2 ⎟
⎟
⎟
a3 ⎠
T (1) = 2(1) − (1 − t)(0) = 2
134
⎛
⎜
⎜
⎜
thus CB (T (1)) = ⎜
⎜
⎜
⎜
⎜
⎝
2 ⎞
⎟
0 ⎟
⎟
⎟.
⎟
0 ⎟
⎟
⎟
0 ⎠
b. Basis of the Null Space = kernel of T and the basis of the Column Space = image of T .
⎛
⎜
⎜
⎜
d. g(t) = 1 + 2t, we want to find all f such that T (f ) = g. Let b = ⎜
⎜
⎜
⎜
⎜
⎝
1 ⎞
⎟
2 ⎟
⎟
⎟ and solve Ax = b.
⎟
0 ⎟
⎟
⎟
0 ⎠
§5.1
Proposition 42. Let A be an n × n matrix. Then det AT = det A.
Corollary 127
.
1) The determinant map is linear in each column.
2) Swapping two columns of a matrix changes the sign of the determinant.
Proof of Proposition 42. If A is singular then so is AT and so det A = 0 = det AT . If A
is nonsingular then Ek ⋯E1 A = I and we have A = E1−1 ⋯Ek−1 = E1′ ⋯Ek′ where E1′ , . . . , Ek′
are elementary matrices. Then we have AT = EkT ⋯E1T , This gives:
′
′
det A = det E1′ ⋯ det Ek′
det AT = det E1T ⋯ det EkT
′
135
′
Claim: det E = det E ′ for any elementary matrix.
Scale:
⎛ 1
⎜
⎜
⎜
⎜
⎜
E=⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⋱
0
c
⋱
0
⎞
⎟
⎟
⎟
⎟
⎟
⎟ = ET
⎟
⎟
⎟
⎟
⎟
⎟
1 ⎠
implies det E = det E T .
Swap:
⎛ 1
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
E=⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟ = ET
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⋱
⎟
⎟
1 ⎠
0
⋱
0
1
⋱
1
0
0
which implies det E = det E T .
Replace: Exercise.
By the claim, det A = det AT .
§5.2 Cofactors and Cramer’s Rule
Formula for a 2 × 2 determinant using row reduction.
Let us assume a ≠ 0 then we have
⎛ a
det ⎜
⎝ c
⎛ 1
b ⎞
⎟ = a det ⎜
⎝ c
d ⎠
= a (d ⋅
⎞
⎛ 1
⎟ = a det ⎜
⎝ 0
d ⎠
b
a
⎛ 1
bc
) det ⎜
a
⎝ 0
b
a
d−
bc
a
⎞
⎟
⎠
⎞
⎛ 1
bc
⎟ = a (d ⋅ ) det ⎜
a
⎝ 0
1 ⎠
b
a
136
0 ⎞
⎟ = ad − bc.
1 ⎠
Formula for 3 × 3 determinant using Cofactors
⎛ a11 a12
⎜
⎜
We want to calculate det ⎜ a21 a22
⎜
⎜
⎝ a31 a32
a11 (1, 0, 0) + a12 (0, 1, 0) + a13 (0, 0, 1).
⎛ a11
⎜
⎜
det ⎜ a21
⎜
⎜
⎝ a31
a12
a22
a32
a13 ⎞
⎟
⎟
. First we will not that (a11 + a12 + a13 ) =
a23 ⎟
⎟
⎟
a33 ⎠
Thus we have:
⎛ 1
a13 ⎞
⎟
⎜
⎟
⎜
= a11 det ⎜ a21
a23 ⎟
⎟
⎜
⎟
⎜
a33 ⎠
⎝ a31
0
a22
a32
⎛ 1
⎜
⎜
= a11 det ⎜ 0
⎜
⎜
⎝ a31
0
a22
0
⎛ 0
0 ⎞
⎟
⎜
⎟
⎜
+ a12 det ⎜ a21
a23 ⎟
⎟
⎜
⎟
⎜
a33 ⎠
⎝ a31
⎛ 0
0 ⎞
⎟
⎜
⎟
⎜
+ a12 det ⎜ a21
a23 ⎟
⎟
⎜
⎟
⎜
a33 ⎠
⎝ a31
⎛ 0
0 ⎞
⎟
⎜
⎟
⎜
+ a13 det ⎜ a21
a23 ⎟
⎟
⎜
⎟
⎜
a33 ⎠
⎝ a31
1
a22
a32
1
0
0
⎛ 0
0 ⎞
⎟
⎜
⎟
⎜
+ a13 det ⎜ a21
a23 ⎟
⎟
⎜
⎟
⎜
a33 ⎠
⎝ a31
= a11 (a22 a33 − a23 a32 ) − a12 (a21 a33 − a23 a31 ) + a13 (a21 a32 − a22 a31 )
Example
⎛ 2
⎜
⎜
Compute det ⎜ 1
⎜
⎜
⎝ 0
⎛ 2
⎜
⎜
det ⎜ 1
⎜
⎜
⎝ 0
1
−2
2
1
−2
2
3 ⎞
⎟
⎟
.
3 ⎟
⎟
⎟
1 ⎠
⎛ 0
3 ⎞
⎜
⎟
⎜
⎟
= det ⎜ 2
3 ⎟
⎜
⎟
⎜
⎟
1 ⎠
⎝ 1
2
1
−2
1 ⎞
⎟
⎟
3 ⎟
⎟
⎟
3 ⎠
⎛ 1 3 ⎞
⎛ 2
⎟ − 2 det ⎜
= 0 det ⎜
⎝ −2 3 ⎠
⎝ 1
⎛ 2
3 ⎞
⎟ + 1 det ⎜
⎝ 1
3 ⎠
= − − 2(6 − 3) + (−4 − 1) = −6 − 5 = −11
137
1 ⎞
⎟
−2 ⎠
0
a22
a32
0
a22
a32
1 ⎞
⎟
⎟
a23 ⎟
⎟
⎟
a33 ⎠
1 ⎞
⎟
⎟
0 ⎟
⎟
⎟
0 ⎠
Friday, 11 April 2014
Homework Question 7
S ∶ V → V and T ∶ V → V , where V is finite dimensional. S ○ T − T ○ S = Id implies AB − BA = I
where A is the matrix for S with respect to B and B is the matrix for T with respect to B.
Then we have
AB − BA = I
0 = tr(AB − BA) ≠ tr(I) = n.
Example
From last class:
⎛ 2
⎜
⎜
Compute det ⎜ 1
⎜
⎜
⎝ 0
⎛ 2
⎜
⎜
det ⎜ 1
⎜
⎜
⎝ 0
1
−2
2
1
−2
2
3 ⎞
⎟
⎟
.
3 ⎟
⎟
⎟
1 ⎠
⎛ 0
3 ⎞
⎜
⎟
⎜
⎟
= det ⎜ 2
3 ⎟
⎜
⎟
⎜
⎟
1 ⎠
⎝ 1
2
1
−2
1 ⎞
⎟
⎟
3 ⎟
⎟
⎟
3 ⎠
⎛ 1 3 ⎞
⎛ 2
⎟ − 2 det ⎜
= 0 det ⎜
⎝ −2 3 ⎠
⎝ 1
⎛ 2
3 ⎞
⎟ + 1 det ⎜
⎝ 1
3 ⎠
= − − 2(6 − 3) + (−4 − 1) = −6 − 5 = −11
⎛ −2 ⎞
⎛ 1 ⎞
⎛ 0
⎜
⎟
⎜ ⎟
⎜
⎜
⎟
⎜ ⎟
⎜
⎜ 1 ⎟ = −2 ⎜ 0 ⎟ + 1 ⎜ 1
⎜
⎟
⎜ ⎟
⎜
⎟
⎜ ⎟
⎜
⎜
⎝ 2 ⎠
⎝ 0 ⎠
⎝ 0
138
⎞
⎛ 0 ⎞
⎟
⎜ ⎟
⎟
⎜ ⎟
⎟ + 2⎜ 0 ⎟.
⎟
⎜ ⎟
⎟
⎜ ⎟
⎠
⎝ 1 ⎠
1 ⎞
⎟
−2 ⎠
⎛ 1
⎜
⎜
det ⎜ 2
⎜
⎜
⎝ 0
−1
1
2
⎛ 1
3 ⎞
⎟
⎜
⎟
⎜
= −2 det ⎜ 2
3 ⎟
⎟
⎜
⎟
⎜
1 ⎠
⎝ 0
⎛ 1
⎜
⎜
= 2 det ⎜ 0
⎜
⎜
⎝ 0
1
0
0
⎛ 2
= 2 det ⎜
⎝ 0
⎛ 1
3 ⎞
⎟
⎜
⎟
⎜
+ 1 det ⎜ 0
3 ⎟
⎟
⎜
⎟
⎜
1 ⎠
⎝ 0
1
2
0
⎛ 1
⎜
⎜
= 2 det ⎜ 0
⎜
(∗)
⎜
⎝ 0
⎛ 1
3 ⎞
⎟
⎜
⎟
⎜
+ 1 det ⎜ 2
3 ⎟
⎟
⎜
⎟
⎜
1 ⎠
⎝ 0
0
2
0
⎛ 1
0 ⎞
⎟
⎜
⎟
⎜
+ det ⎜ 0
3 ⎟
⎟
⎜
⎟
⎜
1 ⎠
⎝ 0
⎛ 1
3 ⎞
⎟ + det ⎜
⎝ 0
1 ⎠
0
1
0
2
1
0
0
1
0
⎛ 1
3 ⎞
⎟
⎜
⎟
⎜
+ 2 det ⎜ 1
3 ⎟
⎟
⎜
⎟
⎜
1 ⎠
⎝ 0
⎛ 1
3 ⎞
⎟
⎜
⎟
⎜
+ 2 det ⎜ 0
3 ⎟
⎟
⎜
⎟
⎜
1 ⎠
⎝ 0
⎛ 1
0 ⎞
⎟
⎜
⎟
⎜
− 2 det ⎜ 0
3 ⎟
⎟
⎜
⎟
⎜
1 ⎠
⎝ 0
⎛ 1
3 ⎞
⎟ − 2 det ⎜
⎝ 2
1 ⎠
0
0
1
0
1
2
0
1
2
3 ⎞
⎟
⎟
3 ⎟
⎟
⎟
1 ⎠
1 ⎞
⎟
⎟
3 ⎟
⎟
⎟
3 ⎠
0 ⎞
⎟
⎟
3 ⎟
⎟
⎟
3 ⎠
3 ⎞
⎟ = 2(2) + 1 − 2(3 − 6) = 11.
3 ⎠
(*) – take the transpose
Definition 128
The (i, j) cofactor of an n × n matrix A is
Cij = (−1)i+j det Aij
where Aij is the (n − 1) × (n − 1) submatrix of A which comes from deleting the ith row and the
jth column of A.
Proposition 43. A = (aij )
n
for any 1 ≤ i ≤ n
det A = ∑ aij Cij
j=1
n
= ∑ aij Cij
for any 1 ≤ j ≤ n.
i=1
139
Example
⎛
⎜
⎜
⎜
Compute det ⎜
⎜
⎜
⎜
⎜
⎝
⎛
⎜
⎜
⎜
det ⎜
⎜
⎜
⎜
⎜
⎝
1
4
1
2
10
0
0
0
2
0
0
−2
1
4
1
2
10
0
0
0
2
0
0
−2
−3 ⎞
⎟
1 ⎟
⎟
⎟
⎟
2 ⎟
⎟
⎟
1 ⎠
−3 ⎞
⎛ 10 0
⎟
⎜
1 ⎟
⎟
⎟ = 1 ⋅ (−1)1+1 det ⎜
⎜ 0
2
⎟
⎜
⎜
2 ⎟
⎟
⎟
⎝ 0 −2
1 ⎠
⎛ 4
1 ⎞
⎟
⎜
⎟
⎜
+ 2 ⋅ (−1)2+1 det ⎜ 0
2 ⎟
⎟
⎜
⎟
⎜
1 ⎠
⎝ 0
1
2
−2
−3 ⎞
⎟
⎟
2 ⎟
⎟
⎟
1 ⎠
⎛ 2 2 ⎞
⎛
⎛ 2 2 ⎞⎞
⎟ − 2 ⎜4 det ⎜
⎟⎟
= 10 det ⎜
⎝ −2 1 ⎠
⎝
⎝ −2 1 ⎠⎠
= 10(2 + 4) − 8(2 + 4) = 2(2 + 4) = 12.
Theorem 129 (Cramer’s Rule). Let A be
⎛
⎜
⎜
solution of Ax = b is given by a vector x = ⎜
⎜
⎜
⎝
a nonsingular n × n matrix. Let b ∈ Rn . Then the
x1 ⎞
⎟
⎟
.
⋮ ⎟
⎟
⎟
xn ⎠
xi =
det Bi
det A
where Bi is the matrix obtained by replacing the ith column of A with b.
Proof. Suppose A = ( a1
⋯
an ), then Bi = ( a1
140
⋯
ai−1
b
ai+1
⋯
an ). If
b = Ax = ( a1
⋯
an
⎛ x1 ⎞
⎜
⎟
⎜
⎟
) ⎜ ⋮ ⎟ = x1 a1 + ⋯xn an .
⎜
⎟
⎜
⎟
⎝ xn ⎠
Bi = ( a 1
⋯
ai−1
= x1 det ( a1
⋯
x1 a1 + ⋯ + xn an
ai−1
a1
ai+1
⋯
ai+1
+ ⋯ + xi det ( a1
⋯
ai−1
ai
+ ⋯ + xn det ( a1
⋯
ai−1
an
= xi det A ⇒ xi =
⋯
an )
an )
ai+1
ai+1
⋯
⋯
an )
an )
det Bi
.
det A
Example
⎛ 2
Solve ⎜
⎝ 4
3 ⎞ ⎛ x1 ⎞ ⎛ 3 ⎞
⎟⎜
⎟=⎜
⎟.
7 ⎠ ⎝ x2 ⎠ ⎝ −1 ⎠
By Cramer’s Rule we have
x1 =
⎛ 3 3 ⎞
⎟
det ⎜
⎝ −1 7 ⎠
2
= 12;
x2 =
⎛ 2
det ⎜
⎝ 4
3 ⎞
⎟
−1 ⎠
2
= −7.
A corollary that follows from Cramer’s Rule is
Corollary 130
Let A be nonsingular. Let C = (Cij ) be the matrix of cofactors of A. Then
A−1 =
1
CT
det A
where C T is called “classical adjoint of A” or “adjugate of A” or adj(A).
141
Example
⎛ 1
⎜
⎜
Find inverse of A = ⎜ 2
⎜
⎜
⎝ 1
⎛ 2
1) det A = 1 det ⎜
⎝ 1
1 ⎞
⎟
⎟
.
−1 0 ⎟
⎟
⎟
−2 2 ⎠
−1
⎛ 1
−1 ⎞
⎟ + 2 det ⎜
⎝ 2
−2 ⎠
−1 ⎞
⎟ = −3 + 2(1) = −1.
−1 ⎠
2) Remember Cij = (−1)i+j det Aij .
C11 = (−1)1+1 (−2) = −2
C21 = 0
C31 = 1
C12 = (−1)1+2 (4) = −4
C22 = 1
C32 = 2
C13 = (−1)1+3 (−3) = −3
C23 = 1
C33 = 1
thus we have
⎛ −2
⎜
⎜
C=⎜ 0
⎜
⎜
⎝ 1
−4
1
2
⎛ −2
⎜
⎜
C T ⎜ −4
⎜
⎜
⎝ −3
−3 ⎞
⎟
⎟
;
1 ⎟
⎟
⎟
1 ⎠
giving
⎛ 2
⎜
⎜
A−1 = ⎜ 4
⎜
⎜
⎝ 3
142
0
−1
−1
−1 ⎞
⎟
⎟
.
−2 ⎟
⎟
⎟
−1 ⎠
0
1
1
1 ⎞
⎟
⎟
2 ⎟
⎟
⎟
1 ⎠
Monday, 14 April 2014
Exam III
Class Average: 48/60
143
Tuesday, 15 April 2014
Definition 131
A n × n matrix A is called diagonalizable if it is similar to a diagonal matrix,
A = P DP −1
⎛ d1
⎜
⎜
where P is invertible and D = ⎜
⎜
⎜
⎝
0
⋱
0 ⎞⎟⎟
dn
⎟.
⎟
⎟
⎠
Recall:
If A is diagonalizable
Ak = (P DP −1 )k = P DP 1− ⋅ P DP −1 ⋯P DP −1 = P Dk P −1 .
⎛ dk
⎜ 1
⎜
k
Also D = ⎜
⎜
⎜
⎝
0
⋱
0 ⎞⎟⎟
dkn
⎟.
⎟
⎟
⎠
Raising matrices to a power comes up a lot in applications
144
Observation
Suppose A = P DP −1 where P = ( v1
A ( v1
⋯
⋯
vn ) = ( v1
⋯
vn ) and note AP = P D implies
vn
⎛ d1
⎜
⎜
)⎜
⎜
⎜
⎝
⋱
0
0 ⎞⎟⎟
dn
⎟
⎟
⎟
⎠
⎛ ∣
∣ ⎞ ⎛ ∣
∣
⎜
⎟ ⎜
⎜
⎟ ⎜
⇒ ⎜ Av1 ⋯ Avn ⎟ = ⎜ d1 v1 ⋯ dn vn
⎜
⎟ ⎜
⎜
⎟ ⎜
∣ ⎠ ⎝ ∣
∣
⎝ ∣
⎧
⎪
⎪
Av1 = d1 v1
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪ Av2 = d2 v2
⇒⎨
⎪
⎪
⎪
⋮
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
Avn = dn vn
⎪
⎩
So the geometric action of A on v1 , . . . , vn is quite simple. Further B =
⎞
⎟
⎟
⎟
⎟
⎟
⎠
{v1 , . . . , vn }
forms a basis for Rn . Note that the converse is also true.
Definition 132
A nonzero vector v is called an eigenvector for an n × n matrix A is there exists a scalar λ such
that Av = λv, λ is called the eigenvalue associated to v.
Proposition 44. An n × n matrix A is diagonalizable if and only if there exists a basis for Rn
consisting of eigenvalues.
Proof. (⇒) Prove by our observation.
(⇐) Suppose {v1 , . . . , vn } is a basis for Rn consisting of eigenvectors with associated
eigenvalues λ1 , . . . , λn . We don’t know that the λi ’s for 1 ≤ i ≤ n are distinct. Let
⎛ λ1
⎞
⎜
⎟
⎜
⎟
⎟.
P = ( v1 ⋯ vn ) and D = ⎜
⋱
⎜
⎟
⎜
⎟
λn ⎠
⎝
0
0
145
Claim A = P DP −1
It is equivalent to show AP = P D.
AP = A ( v1
= ( v1
⋯
⋯
vn ) = ( Av1
vn
⎛ λ1
⎜
⎜
)⎜
⎜
⎜
⎝
⋱
0
⋯
Avn
⎛ ∣
⎜
⎜
) = ⎜ λ1 v1
⎜
⎜
⎝ ∣
∣
⋯
λn vn
∣
⎞
⎟
⎟
⎟
⎟
⎟
⎠
0 ⎞⎟⎟
λn
⎟ = P D.
⎟
⎟
⎠
How to find eigenvalues and eigenvectors?
Lemma 133
Let A be an n × n matrix and let λ ∈ R. Then E(λ) = {x ∈ Rn ∶ Ax = λx} = {x ∈ R ∶ (A − λI)x = 0} =
{x ∈ Rn ∶ Ax − λIx = 0} is a subspace for Rn . Moreover E(λ) ≠ {0} if and only if λ is an eigenvalue.
In this case, E(λ) is called the λ-eigenspace.
Proof. E(λ) = N (A − λI) and null spaces are always subspaces.
Part II holds simply by the definition of eigenvector.
Proposition 45. Let A be an n×n matrix. Then λ ∈ Rn is an eigenvalue if and only if det(A−λI) =
0.
Proof. We have that λ is an eigenvalue if and only if E(λ) ≠ {0} if and only if N (A−λI) ≠
{0} if and only if A − λI is singular if and only if det(A − λI) = 0.
146
Example
⎛ 3 1 ⎞
⎟
A=⎜
⎝ −3 7 ⎠
⎛ 3 1 ⎞ ⎛ λ
⎟−⎜
A − λI = ⎜
⎝ −3 7 ⎠ ⎝ 0
0 ⎞ ⎛ 3−λ
1 ⎞
⎟=⎜
⎟.
λ ⎠ ⎝ −3 7 − λ ⎠
Then we have
det(A − λI) = (3 − λ)(7 − λ) + 3 = 24 − 10λ + λ2 = (λ − 6)(λ − 4).
So the eigenvalues of A are λ = 4, 6.
Question What are the associated eigenvectors?
We have
⎛ −1
E(4) = N (A − 4I) = N ⎜
⎝ −3
⎛ 1
1 ⎞
⎟=N⎜
⎝ −3
1 ⎠
⎛ 1
−1 ⎞
⎟=N⎜
⎝ 0
3 ⎠
−1 ⎞
⎟
0 ⎠
⎧
⎫
⎪
⎪
⎛⎛ 1 ⎞⎞
⎪
⎪
⎪⎛ x2 ⎞
⎪
⎟ ∶ x2 ∈ R⎬ = Span ⎜⎜
⎟⎟
⇒ x1 − x2 = 0 ⇒ x1 = x2 , so N (A − 4I) = ⎨⎜
⎪
⎪
⎪
⎪
⎝
⎠
⎝
⎝
⎠⎠
x
1
⎪
⎪
2
⎩
⎭
⎛ −3
E(6) = N (A − 6I) = N ⎜
⎝ −3
⎛⎛ 1 ⎞⎞
1 ⎞
⎟ = Span ⎜⎜ ⎟⎟ .
⎝⎝ 3 ⎠⎠
1 ⎠
Example
⎛ 0
Let A = ⎜
⎝ 1
−1 ⎞
⎟.
0 ⎠
⎛ λ
A − λI = ⎜
⎝ 1
−1 ⎞
⎟,
λ ⎠
det(A − λI) = λ2 + 1 > 0.
147
Thursday, 17 April 2014
Recall
• Definition 131
A matrix is diagonalizable if it is similar to a diagonal matrix, i.e., A = P DP −1
where P is invertible and D is diagonal.
• Proposition 44
A is diagonalizable if there exists a basis for Rn consisting of eigenvectors for A.
• Definition 132
A nonzero vector v ∈ Rn is an eigenvector for A is there exits λ ∈ R such that
Av = λv. The number λ is called the corresponding eigenvalue.
Notation: Given x ∈ Rn , E(λ) = {x ∈ Rn ∶ Ax = λx}.
If E(λ) ≠ {0}, then λ is an eigenvalue and E(λ) contains all of the eigenvectors corresponding to λ.
E(λ) ≠ {0} (non-trivial) precisely when A − λI s singular ⇔ det(A − λI) = 0 where (*) –
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
(∗)
as a function of λ this is a polynomial with variable λ called the characteristic polynomial of A.
Example
⎛ 1
⎜
⎜
Find the eigenvalues of A = ⎜ 0
⎜
⎜
⎝ 1
2
1
3
1 ⎞
⎟
⎟
.
0 ⎟
⎟
⎟
1 ⎠
148
⎛ 1−λ
⎜
⎜
A − λI = ⎜ 0
⎜
⎜
⎝ 1
2
1−λ
3
⎞
⎟
⎟
⇒ det(A − λI)
0 ⎟
⎟
⎟
1−λ ⎠
1
⎛ 1−λ
⎜
⎜
= det ⎜ 0
⎜
⎜
⎝ 1
⎞
⎟
⎟
0 ⎟
⎟
⎟
1−λ ⎠
2
1
1−λ
3
⎛ 1−λ
= (1 − λ) det ⎜
⎝ 1
⎞
⎟
1−λ ⎠
1
= (1 − λ)[(1 − λ)2 − 1]
= (1 − λ)(1 − 2λ + λ2 − λ
= (1 − λ)(λ)(λ − 2).
So A has three eigenvalues 0, 1, and 2.
What are the corresponding eigenvectors to λ = 0, 1, 2?
⎛ −1 ⎞
⎟
⎜
(∗)
⎟
⎜
E(0) = N (A − 0I) = N (A) = Span ⎜ 0 ⎟
⎟
⎜
⎟
⎜
⎝ 1 ⎠
⎛ 3 ⎞
⎟
⎜
⎟
⎜
E(1) = N (A − I) = Span ⎜ −1 ⎟
⎟
⎜
⎟
⎜
⎝ 2 ⎠
⎛ 1 ⎞
⎜
⎟
⎜
⎟
E(2) = N (A − 2I) = Span ⎜ 0 ⎟
⎟
⎜
⎜
⎟
⎝ 1 ⎠
(*) by row reduction.
Recall the rational roots test:
If p(t) = a0 + a1 t + ⋯ + an tn with integer coefficients, then if t =
r
s
in lowest terms is a
root then r is a factor of a0 and s is a factor of an , i.e., r ∣ a0 and s ∣ an .
149
Example
⎛ 4
⎜
⎜
A=⎜ 0
⎜
⎜
⎝ 2
−3
1
−2
3 ⎞
⎟
⎟
4 ⎟
⎟
⎟
1 ⎠
⎛ 4−t
⎜
⎜
From this we have: A − tI = ⎜ 0
⎜
⎜
⎝ 2
⎞
⎟
⎟
.
1−t
4 ⎟
⎟
⎟
−2 1 − t ⎠
−3
3
⎛ 1−t
⎛ −3 3 ⎞
4 ⎞
⎟ + 2 det ⎜
⎟
det(A − tI) = (4 − t) det ⎜
⎝ −2 1 − t ⎠
⎝ 1−t 4 ⎠
= (4 − t)((1 − t)2 + 8) + 2( − 12 − 3(1 − t)) = 6 − 11t + 6t2 − t3 .
The only possible rational roots are ±6, ±3, ±2, and ±1. Note p(1) = 0. So p(t) =
(t − 1)q(t), where q(t) is second degree. To find q(t) we do long division.
− x2 + 5x − 6
x − 1) − x3 + 6x2 − 11x + 6
x3 − x2
5x2 − 11x
− 5x2 + 5x
− 6x + 6
6x − 6
0
So q(t) = −t2 + 5t − 6 = −(t − 3)(t − 2). Therefore p(t) = (t − 1)(t − 3)(t − 2). So A has
eigenvalues t = 1, 2, 3.
§6.2 Diagonalizabllity
Theorem 134. Let A be an n × n matrix and suppose v1 , . . . , vk are eigenvectors of A with distinct
eigenvalues λ1 , . . . , λk ∈ R respectively. Then {v1 , . . . , vk } are linearly independent.
150
Proof. Let m be the largest number between 1 and k such that {v1 , . . . , vk } is linearly
independent. Then our goal is to prove that m = k.
Suppose m < k by way of contradiction. Then {v1 , . . . , vm } is linearly independent and
vm+1 ∈ Span(v1 , . . . , vm ). So vm+1 = c1 v1 + ⋯ + cm vm for some c1 , . . . , cm ∈ R. Multiply
both sides by (A − λm+1 I).
(A − λm+1 I)vm+1 = Avm+1 − λm+1 vm+1 = λm+1 vm+1 − λm+1 vm+1 = 0
and
(A − λm+1 I)(c1 v1 + ⋯cm vm ) = c1 ( Av1 −λm+1 v1 ) + ⋯ + cm ( Avm −λm+1 vm )
±
²
λ1 v1
λm vm
= c1 (λ1 − λm+1 )v1 + ⋯ + cm (λm − λm+1 )vm .
Therefore c1 = ⋯ = cm = 0 because λ1 ≠ λm+1 , . . . , λm ≠ λm+1 . This implies vm+1 = 0, a
contradiction of the fact that it is an eigenvector.
Corollary 135
Suppose A is an n × n matrix with n distinct real eigenvalues. Then A is diagonalizable.
Proof. Let v1 . . . , vn be eigenvectors associated to the n distinct eigenvalues. By Theorem 134, {v1 , . . . , vn } is linearly independent, in particular it is a basis for Rn . Since A
is diagonalizable if and only if it has an eigenbasis, A is diagonalizable in this case.
Remark. n distinct eigenvalues is sufficient for diagonalizablitiy but not necessary. The identity
matrix has one distinct eigenvalue 1 and it is diagonalizable (trivially, using D = I, and P any
invertible matrix).
151
Friday, 18 April 2014
Last time we showed that if A is an n × n matrix with n distinct real eigenvalues it is diagonalizable.
Lemma 136
If A and B are similar matrices, then they have the same characteristic polynomial.
(Recall: PA (t) = det(A − tI) and PB (t) = det(B − tI) are the characteristic polynomials.)
Proof. Given that A and B are similar then we have A = P BP −1 for some invertible
matrix P . This implies
PA (t) = det(A − tI) = det(A − tI) det P det P −1 = det P −1 det(A − tI) det P
= det(P −1 (A − tI)P ) = det(P −1 AP −P −1 (tI)P ) = det(B − tI) = PB (t).
´¹¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¶
B
Definition 137
Let λ be an eigenvalue of A. (Recall that this means that λ is a root of PA (t)).
1) The algebraic multiplicity of λ is its multiplicity as a root of PA (t), i.e., the
highest power of (t − λ) that divides PA (t).
e.g.: p(t) = (t − 3)5 (t − 2)3 , then t = 3 has multiplicity 5 and t = 2 has multiplicity
3.
2) The geometric multiplicity of λ is the dimension of the eigenspace associated
to the λ (dim E(λ) = dim{x ∈ Rn ∶ Ax = λx}.
Proposition 46. Let A be an n × n matrix with eigenvalue λ. Then the geometric multiplicity of
λ is less then or equal to the algebraic multiplicity of λ.
Let d = dim E(λ) = geometric multiplicity of λ. Let {v1 , . . . , vd } be a basis for E(λ)
152
and extend to a basis B = {v1 , . . . , vd , vd+1 , . . . , vn } to make a basis for Rn . Note that
Av1 = λ1 v1
⋮
Avd = λd vd
The matrix of A with respect to B
⎛ λ
⎜
⎜
⎜
⎜
⎜
⎜
⎜
E=⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
Note that if P = ( v1
⋯
⋱
0
B
λ
⋱
0
λ
0
C
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎠
−1
vn ) then A = P EP which implies
⎛ λ
⎜
⎜
⎜
⎜
⎜
⎜
⎜
PA (t) = det(E−tI) = det ⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⋱
0
B
λ
⋱
0
λ
C − tIn−d
0
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟ = (λ − t)d det(C−tIn−d )
⎟
⎟ ´¹¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹¶
⎟
±(t−1)
⎟
⎟
⎟
⎟
⎟
⎠
⎛ A B ⎞
⎟ = det A det C.
Noting that det ⎜
⎝ 0 C ⎠
Recall the algebraic multiplicity of λ is the highest power of (t − λ) that divides PA (t). We see here
that (1 − λ)d divides PA (t), so d ≤ algebraic multiplicity.
Remark. .
1) The sum of the algebraic multiplicities of the real eigenvalues is at most n. PA (t) = ±(t −
λ1 )m1 (t − λ2 )m2 ⋯(t − λk )mk (t2 + at + a2 )α1 .
2) If PA (t) has complex roots, then the algebraic multiplicities sum to something less than n. Then
the dimensions of the eigenspaces sum to something less than n. So there can’t be n eigenbasis.
153
3) If PA (t) has only real roots and the geometric multiplicities are equal to the algebraic multiplicities, is this enough to say we have an eigenbasis? Yes (still linear by independent definition.)
Theorem 138. Let A be an n×n matrix with distinct real eigenvalues λ1 , . . . , λk . Then the following
are equivalent
1) A is diagonalizable.
2) PA (t) = (t − λ1 )m1 ⋯(t − λk )mk where mi = dim E(λi ).
3) dim E(λ1 ) + ⋯ + dim E(λk ) = n.
Proof. We will show
(i) (1) implies (2),
(ii) (2) implies (3), and
(iii) (3) implies (1)
⎛ α
⎞
⎜
⎟
⎜
⎟
⎟ . PA (t) =
(i) If A is diagonalizable it implies A = P DP −1 where D = ⎜
⋱
⎜
⎟
⎜
⎟
αn ⎠
⎝
PD (t) = det(D − tI) = (λ1 − t)⋯(αn − t). Since λ1 , . . . , λk are the real roots of
0
0
PA (t), namely α1 , . . . , αn must be λ1 , . . . , λk with some multiplicities. So D =
⎛ λ1
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
⋱ }m1
λ1
λ2
⋱
}m2
λ2
⋱
λk
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟ where m1 , . . . , mk are the eigen⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⋱ }mk ⎟
⎟
⎟
λk ⎠
value multiplicities of λ1 , . . . , λk .
Claim: dim(D − λI) = dim N A − λI).
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
´¹¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹¶
mi
E(λi )
Sub-claim: Similar matrices have isomorphic null spaces.
154
155
Monday, 21 April 2014
Recall:
• Corollary 135
A matrix is diagonalizable if and only if there exists a basis for Rn that consists of
eigenvectors.
• Definitions 137
– The characteristic of A is P (t) = det(A − tI) (with degree n = the number of
rows/columns of A).
– The zeros of the characteristic polynomials are the eigenvalues. The algebraic
multiplicity of an eigenvalue λ is the highest power of (t − λ) that divides
P (t). The geometric multiplicity of λ s the dimension the eigenspace E(λ) =
N (A − λI) = {v ∈ Rn ∶ Av = λv} = {the set of eigenvectors when λ is an
eigenvalue}.
• Theorem 138
Let A be an n × n matrix with distinct real eigenvalues λ1 , . . . , λk . Then the
following statements are equivalent.
1) A is diagonalizable
2) p(t) = ±(t − λ1 )d1 ⋯(t − λk )dk where di = dim E(λi ).
Proof of Theorem 138. .
(1)⇒(2) Suppose A = P DP −1
⎛ α1
⎜
⎜
where D = ⎜
⎜
⎜
⎝
⋅
0
0 ⎞⎟⎟
αn
⎟.
⎟
⎟
⎠
Note
PA (t) = det(A − tI) = det P −1 det(A − tI) det P
= det (P −1 (A − tI)P ) = det(P −1 AP − tI) = det(D − tI) = PD (t).
156
So the α1 , . . . , αn are λ1 , . . . , λk with possible repetitions of k < n. So PA (t) = ±(t −
λ1 )m1 ⋯(t − λk )mk where mi , . . . , mk are the algebraic multiplicities of λ1 , . . . , λk .
Now we consider the dimensions of the eigenspaces E(λi )
claim
Claim: dim E(λi ) = dim N (A − λi I) = dim N (D − λi I)
Notice D − λi I = diagonal matrix with exactly mi rows of zeros which
implies dim N (D − λi I)
observation
=
mi .
If there exists an isomorphism between two subspaces of Rn . Then those
subspaces have equal dimension.
Sub-Claim: Two similar matrices have isomorphic null spaces.
Suppose A, B (where A is different from the matrix above) are
similar matrices, then A = P BP −1 and P −1 ∶ N (A) → N (B).
Rn
A
P −1
Rn
Rn
P −1
B
Rn
Proof that this is an isomorphism:
1) Well-Defindness of P −1 .
Let x ∈ N (A). Prove that P −1 x ∈ N (B). Observe that
BP −1 x = P −1 Ax = P −1 (0) = 0.
2) One-to-One
As a map from Rn to Rn P −1 is one-to-one and o restricted
to N (A) it is still one-to-one.
3) Onto
Let y ∈ N (B). Prove that there exists an x ∈ N (A) so that
P −1 x = y. Let x = P y and check that x ∈ N (A). Indeed,
157
Ax = AP y = P By = P (0) = 0.
By the sub-claim N (A − λi I) and N (D − λi I) are isomorphic so
there dimensions are equal.
(2)⇒(3) Suppose P (t) = ±(t−λ1 )d1 ⋯(t−λk )dk where di = dim E(λi ). Prove that dim E(λ1 ) +⋯+
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
d1
dim E(λk ) = n. Because P has degree n.
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
dk
(3)⇒(1) Suppose dim E(λ1 )+⋯+dim E(λk ) = n. Let B1 , . . . , Bk be bases for E(λ1 ), . . . , E(λk ).
Let B = B1 ∩ ⋯ ∩ Bk . Note that ∣B∣ = n (the cardinality), by assumption.
Claim: B is a linear independent set.
Proof of the claim is a homework problem. The key is that λ1 , . . . , λk are
all distinct.
Given the claim, B is a bases which consists of eigenvectors of A, so A is diagonalizable.
§6.3 Applications of Diagonalizability
Recall Fibonacci sequence 1, 1, 2, 3, 5, 8, 13, 21, ...
⎛ ak ⎞
⎛ ak+1
⎟ then xk+1 = ⎜
a0 = 1 = a1 . Let xk = ⎜
⎝ ak+1 ⎠
⎝ ak+2
given recurrsively by ak+2 = ak+1 + ak where
⎞
⎛ 1 ⎞
⎟. Let x0 = ⎜
⎟ then xk = Ak x0 . If A is
⎠
⎝ 1 ⎠
diagonalizable, i.e., A = P DP −1 , then Ak has a nice formula, Ak = P Dk P −1 .
158
Tuesday, 22 April 2014
Recall the Fibonacci Sequence is given by the recursive relation
an+2 = an+1 + an
and a0 = a1 = 1
⎛ ak ⎞
⎛ 0
⎟, A = ⎜
such that {1, 1, 2, 3, 5, 8, 13, . . . }. Let xk = ⎜
⎝ ak+1 ⎠
⎝ 1
⎛ 0
Axk = ⎜
⎝ 1
⎛ 1 ⎞
1 ⎞
⎟, and x0 = ⎜ ⎟.
⎝ 1 ⎠
1 ⎠
⎞ ⎛ ak+1 ⎞
1 ⎞ ⎛ ak ⎞ ⎛
ak+1
⎟⎜
⎟=⎜
⎟=⎜
⎟.
1 ⎠ ⎝ ak+1 ⎠ ⎝ ak + ak+1 ⎠ ⎝ ak+1 ⎠
Note that xk = Ak x0 . Show that A is diagonalizable.
(1) Find the eigenvalues of A,
⎛ −t
p(t) = det(A − tI) = det ⎜
⎝ 1
The roots of p(t) are
t=
1±
√
⎞
⎟ = t(t − 1) − 1 = t2 − t − 1.
1−t ⎠
1
√
1 − 4(1)(−1) 1 ± 5
=
.
2
2
√
√
1+ 5
1− 5
and λ2 =
. So A is diagonalizable because it is 2 × 2 and
2
2
has two distinct eigenvalues.
Let λ1 =
Write the digitalization of A
⎛ λ1
D=⎜
⎝ 0
0 ⎞
⎟. To find P we need corresponding eigenvectors.
λ2 ⎠
⎛ −λ1
E(λ1 ) = N (A−λ1 I) = N ⎜
⎝ 1
And observe that 1 − λ1 +
⎛ 1
⎞
⎛ 1 −1 ⎞
⎜
λ1
⎟=N⎜
⎟=N⎜
⎜ 0
⎜
⎝ 1 1 − λ1 ⎠
1 − λ1 ⎠
⎝
− λ11
1
1
λ1
1 − λ1 +
√
√
1
1− 5
2
1+ 5
2
√ =
√
1 − λ1 +
=1−
+
+
λ1
2
2
1− 5
1− 5
√
√
(1 + 5)(1 − 5)
4
1−5
4
√
√ =
√ +
√ = 0.
=
+
2(1 − 5)
2(1 − 5) 2(1 − 5) 2(1 − 5)
159
=0
1
λ1
⎞
⎟
⎟.
⎟
⎟
⎠
⎛ 1
So E(λ1 ) = N ⎜
⎝ 0
by λ1 .
− λ11 ⎞
⎟ = Span (
0 ⎠
1
λ1
⎛ 1 ⎞
⎟, where we scaled
) = Span ⎜
⎝ λ1 ⎠
⎛ 1 ⎞
⎛ 1
⎟. Let P = ⎜
Similarly, E(λ2 ) = Span ⎜
⎝ λ2 ⎠
⎝ λ1
1 ⎞
⎟. By previous work
λ2 ⎠
⎛ λk1
A = P DP −1 . So Ak = (P DP −1 )k = P Dk P −1 = P ⎜
⎝ 0
0 ⎞
⎟.
λk2 ⎠
(2)
If you are at an even integer you can either stay with probability P or you can jump to an odd
with probability 1 − p.
If you are at an odd number you can either jump to an even with probability q or you can stay
with probability 1 − q.
Figure 23:
Let ek = probability of being at an even at a time k = pek−1 qok−1 and ok = probability of being at
⎛ ek ⎞
⎟, then
an odd at time k = (1 − p)ek−1 + (1 − q)ok−1 . Notice that if xk = ⎜
⎝ ok ⎠
⎛ p
q ⎞ ⎛ ek−1 ⎞
⎟ = Axk + 1.
⎟⎜
x=⎜
⎝ 1 − p q ⎠ ⎝ ok−1 ⎠
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
A
⎛ e0 ⎞
⎟.
And xk = Ak x0 where x0 = ⎜
⎝ o0 ⎠
What we saw from numerical experiments is that for large xk seems to be independent of x0 .
160
Is A diagonalizable?
(1) What are the eigenvalues of A?
⎛ p−t
det(A−tI) = det ⎜
⎝ 1−p
⎞
⎟ = (p−t)(1−q−t)−q(1−p) = t2 +(q−p−q)t+p−q.
1−q−t ⎠
q
Find roots:
t=
p−q+1±
√
(q − p − 1)2 − 4(p − q) p − 1 + 1 ±
=
2
√
(p − q)2 + 2(p − q) + 1 − 4(p − q) p − q + 1 ±
=
2
So long as p − 1 ≠ 1 we have two eigenvalues which implies A is diagonalizable.
⎛ p−1
E(1) = N ⎜
⎝ p
⎛ q
E(p − q) = N ⎜
⎝ p
⎛ p
Let P = ⎜ 1−q
⎝ 1
⎛ p−1 q ⎞
⎛
q ⎞
⎟=N⎜
⎟ = Span ⎜
⎝ 0
⎝
−q ⎠
0 ⎠
⎛ 1
⎞
⎟=N⎜
⎝ 0
1−p ⎠
q
⎛ 1
−1 ⎞
⎟ and D = ⎜
⎝ 0
1 ⎠
⎛ 1
P Dk P −1 where Dk = ⎜
⎝ 0
q
1−p
1
⎞
⎟
⎠
⎛ −1 ⎞
1 ⎞
⎟ = Span ⎜
⎟.
⎝ 1 ⎠
0 ⎠
⎞
⎟ and note A = P DP −1 . So Ak =
p−q ⎠
0
⎞
⎛ e0 ⎞
⎟ for all k < 1. Let x0 = ⎜
⎟ probability of
⎝ o0 ⎠
(p − q)k ⎠
0
initial states. Then x = Ax0 . Note P −1 =
1
(q+p+1)
⎛ 1
⎜
⎝ −1
1
q
1−p
⎛ e0 ⎞
⎟
P −1 x0 = P −1 ⎜
⎝ o0 ⎠
=
⎛ 1
1
⎜
+ 1 ⎝ −1
q
1−p
1
q
1−p
⎞ ⎛ e0 ⎞
⎟⎜
⎟
⎠ ⎝ o0 ⎠
=
⎞
e0 + o0
1−p ⎛
⎜
⎟
q + 1 − p ⎝ −e + q o ⎠
0
0
1−p
=
⎞
1
1−p ⎛
⎜
⎟
q + 1 − p ⎝ −e + q o ⎠
0
0
1−p
161
⎞
⎟ and
⎠
√
((p − q) − 1
2
⎛ c0 ⎞
⎛ 1
⎟=P⎜
xk = P D k ⎜
⎝ c1 ⎠
⎝ 0
⎛ q−1
=⎜
⎝ 1−p
⎞ ⎛ c0 ⎞
⎛
⎞
c0
⎟⎜
⎟=P⎜
⎟
⎝ c1 (p − q)t ⎠
(p − q)k ⎠ ⎝ c1 ⎠
0
⎞ ⎛
⎞ k→∞ ⎛
⎞ ⎛
−1 ⎞ ⎛
c0
c0 q − c1 (p − q)k
c0 q
⎟⎜
⎟=⎜
⎟ Ð→ ⎜
⎟=⎜
⎝ c0 (1 − p) ⎠ ⎝
1 ⎠ ⎝ c1 (p − q)k ⎠ ⎝ c0 (1 − p) + c1 (p − q)k ⎠
162
q
q−p+1
1−p
q−p+1
⎞
⎟.
⎠
Thursday, 24 April 2014
Suppose A is n × n and diagonalizable. This implies A = P DP −1 with P = ( v1
⎛ λ1
⎜
⎜
D=⎜
⎜
⎜
⎝
⋱
0
( λ1 v1
0 ⎞⎟⎟
λn
⋯
⎟ (with λi eigenvalues and vi eigenvectors) AP = P D
⎟
⎟
⎠
( Av1
⋯
⋯
vn ) and
Avn ) =
λn vn ).
⎛ λk
⎜ 1
⎜
k
k −1
k
From this we have A = P D P , with D = ⎜
⎜
⎜
⎝
λkn
0
⎛ c1 ⎞
⎜
⎟
⎜
⎟
Let c = ⎜ ⋮ ⎟ = P −1 x. Then
⎜
⎟
⎜
⎟
⎝ cn ⎠
⎛ λk
⎜ 1
⎜
k
k
A x = PD C = P ⎜
⎜
⎜
⎝
⋱
⋱
λkn
⎛ c1 λk
⎞ ⎛ c1 ⎞
1
⎜
⎟⎜
⎟
⎜
⎟⎜
⎟
⎟⎜ ⋮ ⎟ = P ⎜ ⋮
⎜
⎟⎜
⎟
⎜
⎟⎜
⎟
⎝ cn λkn
⎠ ⎝ cn ⎠
0 ⎞⎟⎟
⎟. If x ∈ Rn , then Ak x = P Dk P −1 x.
⎟
⎟
⎠
⎞
⎟
⎟
⎟ = ( v1
⎟
⎟
⎠
⋯
vn
⎛ c1 λ k
1
⎜
⎜
)⎜ ⋮
⎜
⎜
⎝ cn λkn
⎞
⎟
⎟
⎟ = c1 λk1 v1 + ⋯ + cn λkn vn .
⎟
⎟
⎠
These eigenvalues are telling us growth and decay rate.
§6.4 Spectral Theorem
Remember:
We have seen that not every matrix is diagonalizable.
Example
⎛ 0 1 ⎞
⎛ −t 1 ⎞
⎟ with p(t) = det(A−tI) = det ⎜
⎟ = t2 +1. Not diagonalizable
1 A=⎜
⎝ −1 0 ⎠
⎝ −1 −t ⎠
(over the reals) because it is characteristic polynomial cannot be factored into linear
factors.
163
⎛
⎜
⎜
⎜
2 A=⎜
⎜
⎜
⎜
⎜
⎝
2
0
0
0
2
0
0
0
3
0
0
0
0 ⎞
⎟
0 ⎟
⎟
⎟, p(t) = det(A − tI) = (2 − t)2 (3 − t)2 .
⎟
1 ⎟
⎟
⎟
3 ⎠
We need for dim E(2) + dim E(3) = 4.
⎛
⎜
⎜
⎜
E(2) = N (A − 2I) = N ⎜
⎜
⎜
⎜
⎜
⎝
0
0
0
0
0
0
0
0
1
0
0
0
⎛ −1
⎜
⎜
⎜ 0
⎜
E(3) = N (A − 3I) = N ⎜
⎜
⎜
⎜
⎜
⎝
0
0
−1
0 ⎞
⎟
0 ⎟
⎟
⎟ ⇒ dim E(2) = 2
⎟
1 ⎟
⎟
⎟
1 ⎠
0
0
0
⎞
⎟
⎟
⎟
⎟
⎟ ⇒ dim E(3) = 1
⎟
1 ⎟
⎟
⎟
0 ⎠
Not diagonalizable because dim E(2) + dim E(3) < 4.
Theorem 139 (Spectral Theorem). Let A be a symmetric n × n matrix. Then
1) A has n real eigenvalues (possibly with repetition)
2) There is an orthonormal basis of eigenvectors of A, i.e., A = QDQ−1 where Q has
columns which are the vectors in this orthonormal basis (o.n.b) (so Q is orthogonal,
i.e., QT Q = I).
Proof to come.
Remark. If A is symmetric. Then A = QDQ−1 where Q = ( q1
⋯
qn
⎛ λ1
⎜
⎜
), D = ⎜
⎜
⎜
⎝
0
164
⋱
0 ⎞⎟⎟
λn
⎟.
⎟
⎟
⎠
Let x ∈ Rn . Then
⎛ qT
⎜ 1
⎜
−1
Ax = QDQ x = QD ⎜ ⋮
⎜
⎜
⎝ qTn
⎛ λ1
⎜
⎜
= Q⎜
⎜
⎜
⎝
⋱
0
= ( q1
0 ⎞⎟⎟ ⎛⎜⎜ q
λn
⋯
⎞
⎛ q ⋅x
⎟
⎜ 1
⎟
⎜
⎟ x = QD ⎜
⋮
⎟
⎜
⎟
⎜
⎠
⎝ qn x
qn
⎟⎜
⎟⎜
⎟⎜
⎠⎝
⎞
⎟
⎟
⎟
⎟
⎟
⎠
⎛ λ1 (q ⋅ x)
⋅x ⎞
1
⎟
⎜
⎟
⎜
⎟ = Q⎜
⋮
⋮
⎟
⎜
⎟
⎜
qn ⋅ x ⎠
⎝ λn (qn ⋅ x)
1
⎛ λ1 (q ⋅ x)
1
⎜
⎜
)⎜
⋮
⎜
⎜
⎝ λn (qn ⋅ x)
⎞
⎟
⎟
⎟
⎟
⎟
⎠
⎞
⎟
⎟
⎟
⎟
⎟
⎠
= λ1 (q⋅ x) q1 +⋯ + λn (qn ⋅ x) qn
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶ ¯n
´¹¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹¶ ¯n
∈R
∈R
= λ1
∈R
∈R
q ⋅x
q1 ⋅ x
q + ⋯ + λn n 2 qn
∣∣q1 ∣∣2 1
∣∣qn ∣∣
= λ projq1 (x) + ⋯ + λn projqn (x).
The action of A is to project onto A’s eigenspaces and then scale by the corresponding eigenvalues.
With this in mind for some symmetric matrices we can define the notion of a square root
√
A. That
is, we can find a matrix B such that B 2 = A. For such a B to exist it is enough for the eigenvalues
of A to be nonnegative and we let
⎛
⎜
√
√
⎜
Bx = λ1 projq1 (x) + ⋯ + λn projqn (x) = Q ⎜
⎜
⎜
⎝
√
λ1
⋱
0
0
√
λn
⎞
⎟
⎟ T
⎟ Q x.
⎟
⎟
⎠
If B is symmetric with real eigenvalues σ = {λ1 , . . . , λn } and f ∶ R → R whose domain D contains σ
⎛ f (λ1 )
⎞
⎜
⎟
⎜
⎟ T
⎟Q .
then we can define f (A) = Q ⎜
⋱
⎜
⎟
⎜
⎟
f (λn ) ⎠
⎝
0
0
165
Friday, 25 April 2014 and Monday, 28 April 2014
Theorem 139 (Spectral Theorem)
Let A be an n × n symmetric matrix (so AT = A). Then
1) A has only real eigenvalues (with possible repetitions).
2) A is diagonalizable with A = QDQT where Q is an orthogonal matrix
(therefore QT Q = I).
Remark. Fundamental theorem of algebra says p(t) = det(A − λI) = ±(t − λ)⋯(t − λn ) if we allow
λi to be complex for any A.
In the case of symmetric matrices, these λi are actually real numbers.
Example
⎛ 5
⎜
⎜
Orthogonally diagonalize the matrix A = ⎜ −4
⎜
⎜
⎝ −2
2
B so that B = A.
−4
5
−2
−2 ⎞
⎟
⎟
and if possible find a matrix
−2 ⎟
⎟
⎟
8 ⎠
⎛ 5 − t −4
−2 ⎞
⎜
⎟
⎜
⎟
Step 1: p(t) = det(A−tI) = ⎜ −4 5 − t −2 ⎟ = −t(t−9)2 . So the eigenvalues
⎜
⎟
⎜
⎟
−2 8 − t ⎠
⎝ −2
of A repeated according to multiplicity are t = 0, 9, 9.
Step 2:
⎛ 2 ⎞
⎜ ⎟
⎜ ⎟
E(0) = N (A) = Span ⎜ 2 ⎟,
⎜ ⎟
⎜ ⎟
⎝ 1 ⎠
´¹¹ ¹ ¹¸ ¹ ¹ ¹¶
E(1)
v1
⎛
⎞
⎜
⎟
⎜⎛ −1 ⎞ ⎛ −1 ⎞⎟
⎜
⎟
⎜⎜
⎟ ⎜
⎟⎟
⎜⎜
⎟ ⎜
⎟⎟
= N (A − 9I) = Span ⎜⎜ 1 ⎟, ⎜ 0 ⎟⎟
⎜⎜
⎟ ⎜
⎟⎟
⎜⎜
⎟ ⎜
⎟⎟
⎜⎝ 0 ⎠ ⎝ 2 ⎠⎟
⎜
⎟
⎜
⎟
⎝´¹¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹¶ ´¹¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹¶⎠
v2
where these form a basis for the null space. Then {v1 , v2 , v3 } is a basis
of eigenvectors of R3 . Immediately know diagonalizable.
166
v3
Step 3: Use the Gram Schmit Process (GSP) on {v1 , v2 , v3 } to find an orthonormal basis for R3 , {q1 , q2 , q3 }. (These are still eigenvectors with eigenvalues 0, 9, 9).
⎛ 0
⎜
⎜
Step 4: Let Q = (q1 , q2 , q3 ) and D = ⎜ 0
⎜
⎜
⎝ 0
⎛ 2 ⎞
⎜ ⎟
⎜ ⎟
q1 = 13 ⎜ 2 ⎟, q2 =
⎜ ⎟
⎜ ⎟
⎝ 1 ⎠
⎛ −1 ⎞
⎜
⎟
⎜
⎟
√1 ⎜ 1 ⎟, q3 =
2⎜
⎟
⎜
⎟
⎝ 0 ⎠
0
9
0
0 ⎞
⎟
⎟
. Then, A = QDQT where
0 ⎟
⎟
⎟
9 ⎠
⎛ −1 ⎞
⎜
⎟
⎟
1 ⎜
√
⎜
⎟.
−1
3 2⎜
⎟
⎜
⎟
⎝ 4 ⎠
Now for the second part of the question.
⎛ 0 0
0 ⎞ ⎛ 0 0 0 ⎞
⎟
⎟ ⎜
⎜
√
√
⎟
⎟ ⎜
⎜
⎟ = ⎜ 0 3 0 ⎟ and let B = Q DQT . Observe that B 2 =
Let D = ⎜ 0
9
0
⎟
⎟ ⎜
⎜
⎟
⎜
√ ⎟ ⎜
9 ⎠ ⎝ 0 0 3 ⎠
⎝ 0 0
√
√
√ √
Q DQT Q DQT = Q D DQT = QDQT = A. We call B the “square root of A”.
√
Proof of Spectral Theorem 139.
1) Prove the eigenvalues of A are real.
Let λ be an eigenvalue for A (i.e., a root of p(t) = det(A − tI). A prior, λ could
√
be complex i.e., λ = a + bi for some a, b ∈ R (where i = −1). We show b = 0. Let
S = (A − λI)(A − λI) where λ = a − bi, then
S = A2 − λA − λA + λλI
= A2 − (λ − λ)A + (a2 + b2 )I = A2 − 2aA + (a2 + b2 )I
= (A − aI)2 + b2 I.
Note det S = det(A − λI) det(A − λI) implies S is singular and it follows that there
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
0
exists x ≠ 0 ∈ Rn such that Sx = 0 and it follows that Sx ⋅ x = 0 since Sx = 0 that
means
0 = Sx ⋅ x = ((A − aI)2 x + b2 x) ⋅ x = (A − aI)2 x ⋅ x + b2 x ⋅ x = (A − aI)(A − aI)x ⋅ x + b2 ∣∣x∣∣2
= (A − aI)x ⋅ (A − aI)T x + b2 ∣∣x∣∣2 = ∣∣(A − aI)x∣∣2 + b2 ∣∣x∣∣2 ⇒ b2 ∣∣x∣∣2 = 0 ⇒ b2 = 0 ⇒ b = 0.
2) Prove that there exists an orthonormal basis for Rn of eigenvectors for A.
167
Let λ1 be an eigenvalue for A and let q1 be a unit eigenvector corresponding to
λ1 . We can find some orthonormal basis (onb) of Rn which includes q1 ∶ B =
{q1 , v2 , v3 , . . . , vn }. Note
Aq1 = λ1 q1 , Av2 =?, . . . , Avn =?
⎛
⎜
⎜
⎜
Let B = ⎜
⎜
⎜
⎜
⎜
⎝
λ1
0
* ⋯ *⎞⎟⎟
⋮
C
0
⎟
⎟. Then if P = (q , v2 , . . . , vn ) then A = P BP −1 = P BP T
1
⎟
⎟
⎟
⎟
⎠
implies B = P T AP and it follows B T = P T AT P T
⎛
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎜
⎝
λ1
0
⋮
T
= P T AP = B thus B =
* ⋯ *⎞⎟⎟
C
⎟
⎟ where C is an (n − 1) × (n − 1) symmetric matrix. Note that
⎟
⎟
⎟
⎟
⎠
0
the eigenvalues of C are λ2 , . . . , λn because ±(t − λ1 )⋯(t − λn ) = det(A − tI) =
det(B − tI) = −(t − λ1 ) det(C − tI) which implies det(C − tI) = ±(t − λ2 )⋯(t − λn ).
Let q′2 ∈ Rn−1 be a unit eigenvector for C corresponding to λ2 . Now consider the
⎛ 0 ⎞
⎛ 0 ⎞
⎟. We have the following three claims:
⎟ ∈ Rn . Let q2 = P ⎜
vector ⎜
⎝ q′2 ⎠
⎝ q′2 ⎠
Claim 1 q2 is an eigenvector for A with eigenvalue λ2 .
Indeed,
⎛ 0 ⎞
⎛ λ2 ⋅ 0 ⎞
⎛ 0 ⎞
⎛ 0 ⎞
⎟=P⎜
⎟ = λ2 P ⎜
⎟ = λ2 q2 .
⎟ = PB ⎜
Aq2 = P BP T P ⎜
⎝ q′2 ⎠
⎝ λ2 q′2 ⎠
⎝ q′2 ⎠
⎝ q′2 ⎠
Claim 2 q2 is orthogonal to q1 .
⎛ 0 ⎞
⎛ 0 ⎞
⎟ ⋅ q1 = ⎜
⎟ ⋅ P T q1 .
q1 ⋅ q1 = P ⎜
′
′
⎝ q2 ⎠
⎝ q2 ⎠
168
Note
⎛ qT1
⎜
⎜ vT
⎜ 2
T
P q1 = ⎜
⎜
⎜ ⋮
⎜
⎜
⎝ vTn
⎞
⎛
⎟
⎜
⎟
⎜
⎟
⎜
⎟q = ⎜
⎟ 1 ⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎠
⎝
⎛ 1
1 ⎞
⎟
⎜
⎟
⎛ 0 ⎞ ⎜
0 ⎟ ⎛ 0 ⎞ T
⎜ 0
⎟⇒⎜
⎟ ⋅ P q1 = ⎜
⎟⋅⎜
⎟
⎜
′
′
⎝ q2 ⎠ ⎜
⋮ ⎟
⎟ ⎝ q2 ⎠
⎜ ⋮
⎟
⎜
⎝ 0
0 ⎠
⎞
⎟
⎟
⎟
⎟ = 0.
⎟
⎟
⎟
⎟
⎠
Claim 3 q is a unit vector
Observe
⎛ 0 ⎞
⎛ 0 ⎞
⎛ 0 ⎞ ⎛ 0 ⎞
⎟⋅P ⎜
⎟ = PTP ⎜
⎟⋅⎜
⎟ = ∣∣q′2 ∣∣2 = 1.
q2 ⋅ q2 = P ⎜
′
′
′
′
⎝ q2 ⎠
⎝ q2 ⎠
⎝ q2 ⎠ ⎝ q2 ⎠
an orthonormal basis for Rn , B2 = {q1 , q2 , v′3 , v′4 , . . . , v′n }. Let
⎤
0 ⋆ ⋯ ⋆ ⎥⎥
⎥
⎥
λ2 ⋆ ⋯ ⋆ ⎥⎥
⎥
⎥. B2 = P T AP2 where P2 = ( q q v′ ⋯ v′ ).
0
2
⎥
1
2
n
3
⎥
⎥
⎥
⋮
⎥
⎥
⎥
⎥
0
⎦
⎤
⎡
⎢ λ1 0 0 ⋯ 0 ⎥
⎥
⎢
⎥
⎢
⎥
⎢
⎢ 0 λ2 0 ⋯ 0 ⎥
⎥
⎢
⎥
⎢
⎥.
Then we have B2 = ⎢⎢ ⋮
0
⎥
⎥
⎢
⎥
⎢
⎥
⎢ ⋮
⋮
⎥
⎢
⎥
⎢
⎥
⎢
⎥
⎢ 0
0
⎣
⎦
Extend {q1 , q2 } to
⎡
⎢ λ1
⎢
⎢
⎢
⎢ 0
⎢
⎢
B2 = [A]B2 = ⎢⎢ ⋮
⎢
⎢
⎢ ⋮
⎢
⎢
⎢
⎢ 0
⎣
C
C
Fact: Let A be symmetric, then det A is equal to the product of the eigenvalues of A.
Proof. Given that A is symmetric then we have
p(t) = det(A−tI) = (−1)n (t−λ1 )⋯(t−λn ) ⇒ p(0) = (−1)n (0−λ1 )⋯(0−λn ) = (−1)n (−1)n λ1 ⋯λn = λ1 ⋯λn .
169
Polar Decomposition
Suppose A is an n × n matrix with λ1 , . . . , λn with real eigenvalues repeated according
to multiplicity. Then A can be written as the product of an orthogonal matrix and the
square root of AT A. Note (AT A)T = AT AT
T
= AT A implies AT A is symmetric. It
Figure 24:
follows by the Spectral Theorem that AT A = QDQT for some diagonal matrix D.
√
√
Claim: The eigenvalues of A are λ21 , . . . , λ2n which implies AT A = Q DQT where
√
⎞
⎛ λ2
⎞ ⎛ ∣λ1 ∣
1
⎟
⎜
⎟ ⎜
⎟
⎜
⎟ ⎜
⎟.
⎟=⎜
D=⎜
⋱
⋱
⎟
⎜
⎟ ⎜
⎟
⎜
⎟ ⎜
√
2
∣λn ∣ ⎠
λn ⎠ ⎝
⎝
√
Now the big claim is that A = P AT A for some orthogonal P .
√
−1
Prove in the case that A is invertible. Let P = A ( AT A) . (Note the eigenvalues of
√
AT A are ∣λ1 ∣, . . . , ∣λn ∣ ≠ 0 implies nonsingular.
170
Tuesday, 29 April 2014
Problem:
⎡
⎢ 8 0
⎢
⎢
⎢
Let A be a 3 × 2 matrix, B be a 2 × 3 matrix, and suppose AB = ⎢ − 32 9
⎢
⎢
⎢ −2 0
⎢
⎣
is BA?
⎤
−4 ⎥⎥
⎥
⎥
−6 ⎥⎥. What
⎥
1 ⎥⎥
⎦
Polar Decomposition
Every matrix A with real eigenvalues can be written as A = P B where P is an orthogonal
⎞
⎛ ∣λ1 ∣
⎟
⎜
√
√
√
⎟
⎜
⎟, ∣λi ∣ = absolute
matrix and B = AT A. B = Q DQT where D = ⎜
⋱
⎟
⎜
⎟
⎜
∣λn ∣ ⎠
⎝
values of the eigenvalues of A.
⎛
⎜
√
√
√
−1
−1
⎜
B −1 = (Q DQT )−1 = Q ( D) QT ⇒ ( D) = ⎜
⎜
⎜
⎝
1
∣λ1 ∣
⋱
1
∣λn ∣
⎞
⎟
⎟
⎟.
⎟
⎟
⎠
If A has n (possibly complex) eigenvalues, then we can write A = P B where P is an
√
orthogonal matrix and B = A∗ A, where A∗ = “adjoint” or “conjugate transpose of
A.”
Theorem 140. Let A be an n × n matrix. Let Ω ⊂ Rn . Then volume(A(Ω)) = ∣ det A∣volume(Ω).
Example
⎛ 2
⎜
⎜
Suppose Ω is the unit cube and A = ⎜ 0
⎜
⎜
⎝ 0
0
3
0
0 ⎞
⎟
⎟
, then A(Ω) is a rectangular region,
0 ⎟
⎟
⎟
4 ⎠
with volume(A(Ω)) = 2 × 3 × 4 = 24 and det A = 2 × 3 × 4.
Let A be any 3 × 3 matrix. Then A = P B where P is orthogonal and B =
171
√
AT A by
Figure 25:
Figure 26:
⎛ ∣λ1 ∣
⎜
√
√
⎜
Polar decomposition = Q DQT where D = ⎜
⋱
⎜
⎜
⎝
√
T
A(Ω) = P B(Ω) = P (Q DQ (Ω)).
0
⎞
⎟
⎟
⎟, Q is orthogonal so
⎟
⎟
∣λn ∣ ⎠
0
If Q is an orthogonal matrix, then Q = ( v1 , v2 , v3 ) where {v1 , v2 , v3 } is an orthonormal basis for R3
Qe1 = ( v1
⎛ 1 ⎞
⎜ ⎟
⎜ ⎟
⎟ = v1 ,
v3 ) ⎜
⎜ 0 ⎟
⎜ ⎟
⎝ 0 ⎠
v2
Qe2 = v2 ,
Qe3 = v2 .
√
√
Thus we have P (Q DQT (Ω)) = a new box that is stretched by D. Its volume is
∣λ1 ⋯λn ∣ = det A.
Suppose we are given a system of differential equations
y1′ (t) = a11 y1 (t) + a12 y2 (t) + ⋯ + a1n yn (t)
y2′ (t) = a21 y1 (t) + a22 y2 (t) + ⋯ + a2n yn (t)
⋮
yn′ (t) = an1 y1 (t) + an2 y2 (t) + ⋯ + ann yn (t).
172
⎛ y1 (t) ⎞
⎛ y ′ (t)
⎜
⎟
⎜ 1
⎜
⎟
⎜
⎟ and y(t) = ⎜
Let y(t) = ⎜
⋮
⋮
⎜
⎟
⎜
⎜
⎟
⎜
⎝ yn (t) ⎠
⎝ yn′ (t)
(Think back to calculus if
y ′ = ay ⇔
⎛ a11
⎞
⎜
⎟
⎜
⎟
⎟. The system is given by y ′ = Ay where A = ⎜ ⋮
⎜
⎟
⎜
⎟
⎝ an1
⎠
⋯
⋱
⋯
a1n ⎞
⎟
⎟
.
⋮ ⎟
⎟
⎟
ann ⎠
dy
dy
dy
= ay ⇔
= adt ⇔ ∫
= ∫ a dt ⇔ ln y = at + c
dt
y
y
⎛ a11
⎜
⎜
thus y = eat+c = eat ec = ceat .) A = ⎜ ⋮
⎜
⎜
⎝ an1
⋯
⋱
⋯
a1n ⎞
⎟
⎟
, y(t) = etA .
⋮ ⎟
⎟
⎟
ann ⎠
In fact, we can define the matrix exponential for any matrix etA = I + tA +
(tA)2 (tA)3
+
+ ⋯ (which
2!
3!
converges in some sense).
∞
xk
x2 x3
=1+x+
+
+ ⋯.
2! 3!
k=0 k!
Recall ex = ∑
Our guess is that something like y = etA solve the differential equation y ′ = Ay. We should take a
derivative of y = etA with respect to t. Because the power series converges uniformly we can take a
term by term derivative and get
d tA
A
A
A
tA2 tA3 tA4
(e ) = A + tA2 + 3tA2 ( ) + 4tA3 ( ) + 5tA4 ( ) + ⋅ ⋅ ⋅ = A [I + tA +
+
+
+ ⋯] = AetA = Ay.
dt
3!
4!
5!
2!
3!
4!
If A is diagonalizable, then P DP −1 and
t2
t3
t2 A2
2
3
+ ⋯ = +tP DP 1 + (P DP −1 ) + (P DP −1 )
2!
2!
3!
2
3
t
t
t2 D2 t3 D3
= I + tP DP −1 + P D2 P −1 + P D3 P −1 + ⋯ = P (I + tD +
+
+ ⋯) P −1
2!
3!
2!
3!
etA = I + tA +
= P etD P −1 , and etD
⎛ etλ1
⎜
⎜
=⎜
⎜
⎜
⎝
⋱
etλn
173
⎞
⎟
⎟
⎟.
⎟
⎟
⎠
If A is not diagonalizable Jordan Canonical Form
⎛ C1 I
⎞
⎜
⎟
⎜
⎟
⋱ ⋱
⎜
⎟ −1
⎜
⎟P ,
A=P⎜
⎟
⎜
⋱ I ⎟
⎜
⎟
⎜
⎟
⎝
Cn ⎠
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
0
J
and λ = α1 + iβ1 . Ak = P J k P −1.
174
⎛ α12
Ci = ⎜
⎝ β12
β12 ⎞
⎟
α12 ⎠
Thursday, 1 May 2014
Review
• §6.1 How to find eigenvalues and eigenvectors.
– for eigenvalues, find roots of the characteristic polynomial.
– for eigenvectors corresponding to eigenvalues λ, find a basis for N (A − λI).
• §6.2 When is a matrix diagonalizable?
Answer: When the matrix has a set of eigenvectors that form a basis for Rn .
An easy sufficient condition for this to happen is when A has n distinct real eigenvalues.
The most general necessary and sufficient conditions for diagonalability of A are
1) A has only real eigenvalues, i.e., p(t) = ±(t − λ1 )d1 ⋯(t − λk )dk .
2) The dimension of the eigenspaces corresponding to λ1 , . . . , λn are d1 , . . . , dk .
Key Proposition (# 20 of last homework assignment)
The union of linearly independent sets of eigenvectors corresponding to
distinct eigenvalues is linearly independent.
• §6.3 Power of a diagonalizable matrices if A = P DP −1 then Ak = P Dk P −1 .
• §6.4 Symmetric matrices have only real eigenvalues and are orthogonally diagonalizable, i.e.,
A = QDQT , where Q is orthogonal.
175
Practice Exam
⎛ 2
1. Let A = ⎜
⎝ 1
5 ⎞
⎟. Calculate Ak for all k ≥ 1.
−2 ⎠
(Check if diagonalizable).
(a) Find eigenvalues, i.e., find roots of
⎛ 2−t
5 ⎞
⎟ = −(2 − t)(2 + t) − 5 = −4 + t2 − 5 = t2 − 9 = (t − 3)(t + 3).
det(A − tI) = det ⎜
⎝ 1
−2 − t ⎠
⎛ 3
(λ = ±3 implies diagonalizable. Thus D = ⎜
⎝ 0
0 ⎞
⎟.
−3 ⎠
(b) Find
E(3) = (v1 )
E(−3) =(v2 ).
(c) Let P = ( v1
−1
v2 ) and find P .
⎛ 3k
(d) Write Ak = P Dk P −1 where Dk = ⎜
⎝ 0
⎞
⎟.
(−3)k ⎠
0
2. Let A and B be similar.
(a) AP = P B for some invertible P . A = P BP −1 . Show that det(A − tI) = det(B − tI) for
all t.
First step, write down
det(A − tI) = det(P BP −1 − tI)
= det(P BP −1 − tP IP −1 ) = det(P BP −1 − P tIP −1 )
Now use multiplicity of determinants.
(b) Automatic from (a).
3. Let A be symmetric n × n matrices. Prove det A is the product of its eigenvalues.
176
= det (P (B − tI)P −1 ).
A symmetric implies the eigenvalues of A are λ1 , . . . , λn ∈ R. That is det(A − λI) = (−1)n (t −
λ1 )⋯(t−λn ) and now plugging in t = 0 we have det(A) = (−1)n (0−λ1 )⋯(0−λn ) = (−1)n (−1)n λ1 ⋯λn =
λ1 ⋯λn .
⎛ λ1
⎜
⎜
Or we have A = P BP −1 , where B = ⎜
⎜
⎜
⎝
det B = λ1 ⋯λn .
⋱
λn
⎞
⎟
⎟
⎟ implies det A = det P det B det P −1 =
⎟
⎟
⎠
⎛ 1 ⎞ ⎛ 2 ⎞
⎟. and det A = 6.
4. Suppose A is symmetric, A ⎜ ⎟ = ⎜
⎝ 1 ⎠ ⎝ 2 ⎠
Observations:
⎛ 1 ⎞
⎛ 1 ⎞ ⎛ 2 ⎞
⎛ 1 ⎞ ⎛ 2 ⎞
⎟ implies 2 ⎜
⎟ = ⎜ ⎟.
1) 2 is an eigenvalue, ⎜ ⎟ is eigenvector A ⎜ ⎟ = ⎜
⎝ 1 ⎠
⎝ 1 ⎠ ⎝ 2 ⎠
⎝ 1 ⎠ ⎝ 2 ⎠
2) A is 2 × 2.
3) A has only real eigenvalues and their product is det A = 6, so 3 is an eigenvalue also.
⎛ 2
4) A = P DP −1 with D = ⎜
⎝ 0
0 ⎞
⎟ and some invertible P .
3 ⎠
Goal is to find eigenvectors that correspond to 2 and 3 respectively and then we will let
P = ( v1
v2 ).
Fact:
The eigenspaces E(2) and E(3) are orthogonal subspaces.
(More generally, if A is symmetric and λ, µ are distinct eigenvalues, the E(λ) and
⎛ 1 ⎞
⎛ 1
⎟. So A = ⎜
E(µ) are orthonormal subspaces. Let v2 = ⎜
⎝ −1 ⎠
⎝ 1
5. Save for Later
177
1 ⎞⎛ 2
⎟⎜
−1 ⎠ ⎝ 0
0 ⎞⎛ 1
⎟⎜
3 ⎠⎝ 1
1 ⎞
⎟.
−1 ⎠
Friday, 2 May 2014
Final Exam – May 7 from 0730-0930 in our classroom.
Roughly:
• 1/2 to 2/3 questions on Chapter 6.
• The rest is old material similar to old exam questions.
6. If λ1 , . . . , λk are distinct eigenvalues with corresponding eigenvectors v1 , . . . , vk then {v1 , . . . , vk }
is linearly independent.
λ
Proof. Suppose c1 v1 +⋯+ck vk = 0. Multiply by (A−λ1 I)⋯(A−λk−1 I) implies ck
(λ
(λ
k − λ1 )vk =
k−
k−1 )⋯
0 ⇒ ck = 0.
7. Let V = {x ∈ R3 ∶ −x1 + x2 + x3 = 0}. Let T ∶ R3 → R3 be the projection onto V . Let
B = {q1 , q2 , q3 } be an orthonormal basis with one vector q3 in the direction of v3 . Then
Figure 27:
[T ]B = ( [T (q1 )]B
[T (q2 )]B
[T (q3 )]B
⎛ 1
⎜
⎜
)=⎜ 0
⎜
⎜
⎝ 0
0
1
0
0 ⎞
⎟
⎟
.
0 ⎟
⎟
⎟
0 ⎠
Eigenvalues of [T ]B are 1 and 0. Then the eigenvalues of [T ]stand = P [T ]B P −1 where P =
( q1
q2
q3 ). Since [T ]stand and [T ]B are similar, they have the same characteristic poly-
nomial and hence the same eigenvalues Let S ∶ R3 → R3 be a rotation about the axis spanned by
178
Figure 28:
v3 .
Note S(v3 ) = v3 so v3 is an eigenvector (any vector along the axis of rotation is an eigenvector)
with eigenvalue 1. There are no other vectors in R3 such that S(v) = λv for some λ. So λ = 1 is
the only eigenvalue and E(1) = Span(v3 ).
8. Skipped
9. True or False:
a) If A is invertible then A is diagonalizable.
⎛ 1
A=⎜
⎝ 0
False
1 ⎞
⎟.
1 ⎠
This is invertible because det A ≠ 0 and it only has real eigenvalues (λ = 1), but E(1) =
⎛ 0
N (A − I) = N ⎜
⎝ 0
1 ⎞
⎟ so dim E(1) = 1. Since dim E(1) < 2, A is not diagonalizable.
0 ⎠
b) If A is diagonalizable then A is invertible.
⎛ 1
A=⎜
⎝ 0
False:
0 ⎞
⎟
0 ⎠
diagonalizable, but not invertible.
c) If A is invertible and diagonalizable, then AT is diagonalizable.
True:
179
Proof. Let λ be some eigenvalue for A with eigenvector v. Av = λv. Multiply both sides
by A−1 implies v = λA−1 v = λ−1 v = A−1 v and it follows (λ−1 , v) is an eigenpair, thus the
bases of eigenvectors for A is also a basis for eigenvectors for A−1 which implies that A−1 is
diagonalizable.
10.
11. Suppose A is diagonalizable. Let p(t) be its characteristic polynomial. Show p(A) = O.
Proof. p(t) = ±(t − λ1 )d1 ⋯(t − λk )dk where λ1 , . . . , λk are the distinct eigenvalues of A which
implies p(A) = ±(A − λ1 I)d1 ⋯(A − λk I)dk . Let {v1 , . . . , vn } be an eigenbasis.
Claim: p(A)vi = 0.
Observe that p(A)vi = ±(A−λ1 I)di ⋯(A−λi I)dk vi = ±(λi −λk )dk (A−λ1 I)d1 ⋯(A−λi−1 )dk−1 vi = 0.
So for any v ∈ Rn , there exists scalars c1 , . . . , cn so that v = c1 v1 + ⋯ + vn implies p(A)v =
p(A)(c1 v1 + ⋯ + cn vn ) = c1 p(A)v1 + ⋯ + cn p(A)vn = 0.
⎛ 1
⎛ 8 ⎞
⎜
⎟
⎜
⎜
⎟
⎜
12. x = ⎜ −4 ⎟ with u1 = ⎜ 0
⎜
⎟
⎜
⎜
⎟
⎜
⎝ 3 ⎠
⎝ 1
⎛ 2
⎛ −1 ⎞
⎞
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎟, u2 = ⎜ 4 ⎟, and u3 = ⎜ 1
⎜
⎟
⎜
⎟
⎜
⎟
⎜
⎟
⎝ −2
⎝ 1 ⎠
⎠
⎞
⎟
⎟
⎟. Write x as a linear combination
⎟
⎟
⎠
⎛ c1 ⎞
⎜
⎟
⎜
⎟
of u1 , u2 , u3 . Let A = ( u1 u2 u3 ). Let c = ⎜ c2 ⎟. Then we have to find c so that Ac = x.
⎜
⎟
⎜
⎟
⎝ c3 ⎠
Since {u1 , u2 , u3 } ⊆ R3 is orthogonal so it is an orthogonal basis. So x = c1 u1 +c2 u2 +⋯+(u3 +u4 )
u1 ⋅ x
u1 ⋅x
1 ⋯x
for some c1 , c2 < 3. Dot both sides with (u1 ) implies u1 ○ u1 = u
= ∣∣u
u1 +
2 . So x =
u1 ⋅u1
1 ∣∣
∣∣u1 ∣∣2
u2 ⋅ x
u3 ⋅ x
u2 +
u3 = ....
∣∣u2 ∣∣2
∣∣u3 ∣∣2
13. (a) Prove that if {u1 , . . . , uk } is an orthogonal set of nonzero vectors then its linearly independent.
Suppose both sides with u1
c1
u1 = 0 ⇒ c1 = 0
180
because ∣∣u1 ∣∣2 ≠ 0 because u1 ≠ 0.
Show that BV ∪ BW is a basis for V + W .
181