Linear Algebra
Lecture 6.1 – 6.5
Woongsun Jeon
School of Electrical and Electronics Engineering
Chung Ang University
1
6.1 INNER PRODUCT,
LENGTH, AND
ORTHOGONALITY
2
1
INNER PRODUCT
▪
If u and v are vectors in ℝ𝑛 , then we regard u and v
as n 1matrices.
▪
The transpose uT is a 1 × 𝑛 matrix, and the matrix
product uTv is a 1 × 1 matrix, which we write as a
single real number (a scalar) without brackets.
▪
The number uTv is called the inner product of u
and v, and it is written as 𝑢 ∙ 𝑣.
▪
The inner product is also referred to as a dot
product.
INNER PRODUCT
 u1 
 v1 
u 
v 
2
2
▪ If u =   and v =   ,
 
 
u 
v 
 n
 n
then the inner product of u and v is
u
1
u2
 v1 
v 
un   2  = u1v1 + u2v2 +
 
v 
 n
+ unvn
2
INNER PRODUCT
▪
Theorem 1: Let u, v, and w be vectors in ℝ𝑛 , and
let c be a scalar. Then
a. 𝑢 ∙ 𝑣 = 𝑣 ∙ 𝑢
b. 𝑢 + 𝑣 ∙ 𝑤 = 𝑢 ∙ 𝑤 + 𝑣 ∙ 𝑤
c. 𝑐𝑢 ∙ 𝑣 = 𝑐 𝑢 ∙ 𝑣 = 𝑢 ∙ (𝑐𝑣)
d. 𝑢 ∙ 𝑢 ≥ 0, and 𝑢 ∙ 𝑢 = 0 if and only if u = 0
▪
Properties (b) and (c) can be combined several
times to produce the following useful rule
(𝑐1 𝑢1 +…+𝑐𝑝 𝑢𝑝 ) ∙ 𝑤 = 𝑐1 𝑢1 ∙ 𝑤 + ⋯ + 𝑐𝑝 𝑢𝑝 ∙ 𝑤
THE LENGTH OF A VECTOR
▪ If v is in ℝ𝑛 , with entries v1, …, vn, then the square
root of 𝑣 ∙ 𝑣 is defined because 𝑣 ∙ 𝑣 is nonnegative.
▪ Definition: The length (or norm) of v is the
nonnegative scalar v defined by
𝑣 = 𝑣∙𝑣 =
𝑣12 + 𝑣22 + ⋯ + 𝑣𝑛2 , and 𝑣 2 = 𝑣 ∙ 𝑣
a 
b 
▪ Suppose v is in ℝ2 , say, v =  
3
THE LENGTH OF A VECTOR
▪ A vector whose length is 1 is called a unit vector.
▪ If we divide a nonzero vector v by its length—that is,
multiply by 1/ v .
▪ The process of creating u from v is sometimes called
normalizing v, and we say that u is in the same
direction as v.
DISTANCE IN ℝ𝑛
▪ Definition: For u and v in ℝ𝑛 , the distance between u
and v, written as dist (u, v), is the length of the vector
u − v. That is,
dist (u,v) = u − v
4
DISTANCE IN ℝ𝑛
▪ Example 4: Compute the distance between the
vectors u = (7,1) and v = (3, 2) .
▪ Solution: Calculate
 7   3  4 
u−v= −  = 
 1  2   −1
u − v = 42 + (−1) 2 = 17
DISTANCE IN ℝ𝑛
▪ Notice that the parallelogram in the above figure
shows that the distance from u to v is the same as the
distance from u − v to 0.
5
ORTHOGONAL VECTORS
▪ Consider ℝ2 or ℝ3 and two lines through the origin
determined by vectors u and v.
▪ The two lines shown in the figure are geometrically
perpendicular if and only if the distance from u to v is
the same as the distance from u to − v.
▪ This is the same as requiring the squares of the
distances to be the same.
ORTHOGONAL VECTORS
▪ Now
[dist u, −v ]2 = u − (−v) 2 = u + v 2
= u+v ∙ u+v
= u ∙ u + v + v ∙ u + v Theorem 1(b)
= u ∙ u + u ∙ v + v ∙ u + v ∙ v Theorem 1(a), (b)
= u 2 + v 2 + 2u ∙ v Theorem 1(a)
▪ The same calculations with v and − v interchanged show
that
[dist u, v ]2 = u 2 + −v 2 + 2u ∙ (−v)
= u 2 + v 2 − 2u ∙ v
▪ The two squared distances are equal if and only if
2𝑢 ∙ 𝑣 = −2𝑢 ∙ 𝑣, which happens if and only if 𝑢 ∙ 𝑣 = 0.
6
ORTHOGONAL VECTORS
▪ Definition: Two vectors u and v in ℝ𝑛 are orthogonal (to
each other) if 𝑢 ∙ 𝑣 = 0.
▪ The zero vector is orthogonal to every vector in ℝ𝑛
because 0T v = 0 for all v.
▪ Theorem 2: Two vectors u and v are orthogonal if and
only if
u+v = u + v
2
2
2
THE PYTHOGOREAN THEOREM
Orthogonal Complements
▪ If a vector z is orthogonal to every vector in a subspace
W of ℝ𝑛 , then z is said to be orthogonal to W.
▪ The set of all vectors z that are orthogonal to W is
called the orthogonal complement of W and is
denoted by W ⊥ (and read as “W perpendicular” or
simply “W perp”).
⊥
1. A vector x is in W if and only if x is orthogonal
to every vector in a set that spans W.
⊥
2. W is a subspace of ℝ𝑛 .
7
ORTHOGONAL COMPLEMENTS
▪
Theorem 3: Let A be an m  n matrix. The orthogonal
complement of the row space of A is the null space
of A, and the orthogonal complement of the column
space of A is the null space of AT:
(Row A) ⊥ = Nul A and (Col A) ⊥ = Nul AT
ORTHOGONAL COMPLEMENTS
▪ Proof: The row-column rule for computing Ax shows that
if x is in Nul A, then x is orthogonal to each row of A (with
the rows treated as vectors in ℝ𝑛 ).
▪ Since the rows of A span the row space, x is orthogonal to
Row A.
▪ Conversely, if x is orthogonal to Row A, then x is certainly
orthogonal to each row of A, and hence Ax = 0 .
▪ This proves the first statement of the theorem.
8
ORTHOGONAL COMPLEMENTS
▪ Since this statement is true for any matrix, it is true
for AT.
▪ That is, the orthogonal complement of the row space
of AT is the null space of AT.
▪ This proves the second statement, because
Row AT = Col A
ANGLES IN ℝ2 AND ℝ3 (OPTIONAL)
▪ If u and v are nonzero vectors in either ℝ2 or ℝ3 ,
then there is a nice connection between their inner
product and the angle  between the two line
segments from the origin to the points identified with
u and v.
▪ The formula is
𝑢u
∙ 𝑣v = u v cos 
9
ANGLES IN ℝ2 AND ℝ3 (OPTIONAL)
▪ By the law of cosines,
u − v = u + v − 2 u v cos 
2
2
2
6.2 ORTHOGONAL
SETS
20
10
ORTHOGONAL SETS
▪ A set of vectors {u1,…,up} in ℝ𝑛 is said to be an
orthogonal set if each pair of distinct vectors from
the set is orthogonal, that is,
if 𝑢𝑖 ∙ 𝑢𝑗 = 0 whenever i  j
▪ Theorem 4: If S = {u1 , ,u p } is an orthogonal set of
nonzero vectors in ℝ𝑛 , then S is linearly independent
and hence is a basis for the subspace spanned by S.
ORTHOGONAL SETS
▪ Proof: If 0 = c1u1 + + c p u p for some scalars c1,…,cp,
then
0 = 0 ∙uu11= (c1u1 + c2 u 2 + + c p u p ) ∙uu11
= (c1u1 ) ∙uu11+ (c2 u 2 ) ∙uu11+
+ (c p u p ) ∙uu11
= c1 (u1 ∙uu11) + c2 (u 2 ∙uu11))+
+ c p (u p ∙uu11))
= c1 (u1 ∙uu11))
because u1 is orthogonal to u2,…,up.
▪ Since u1 is nonzero, u1 ∙ u1 is not zero and so c1 = 0
▪ Similarly, c2,…,cp must be zero.
▪ Thus S is linearly independent.
11
ORTHOGONAL SETS
▪ Definition: An orthogonal basis for a subspace W of ℝ𝑛
is a basis for W that is also an orthogonal set.
▪ Theorem 5: Let {u1,…,up} be an orthogonal basis for a
subspace W of ℝ𝑛 . For each y in W, the weights in the
linear combination
y = c1u1 + + c p u p
are given by
𝑐𝑗 =
𝑦 ∙ u𝑗
u𝑗 ∙ u𝑗
( j = 1,
, p)
AN ORTHOGONAL PROJECTION
▪ Given a nonzero vector u in ℝ𝑛 , consider the problem
of decomposing a vector y in ℝ𝑛 into the sum of two
vectors, one a multiple of u and the other orthogonal
to u. We wish to write
y = yˆ + z
where ŷ =  u for some scalar α and z is some vector
orthogonal to u.
▪ The vector ŷ is called the orthogonal projection of y
onto u, and the vector z is called the component of y
orthogonal to u.
12
AN ORTHOGONAL PROJECTION
▪ If c is any nonzero scalar and if u is replaced by cu in
the definition of ŷ, then the orthogonal projection of
y onto cu is exactly the same as the orthogonal
projection of y onto u.
▪ Hence this projection is determined by the subspace L
spanned by u (the line through u and 0).
▪ Sometimes ŷ is denoted by projLy and is called the
orthogonal projection of y onto L.
▪ That is,
yy∙ u
ŷ = projL y =
uu∙ u
u
AN ORTHOGONAL PROJECTION
7 
6
4
2
▪ Example 3: Let y =   and u =   .
▪ Find the orthogonal projection of y onto u.
Then write y as the sum of two orthogonal vectors,
one in Span{u} and one orthogonal to u.
▪ Solution: Compute
7 4
∙
= 40
6 2
4 4
u∙u=
∙
= 20
2 2
y∙u=
13
AN ORTHOGONAL PROJECTION
▪ The orthogonal projection of y onto u is
ŷ =
 4   8
40
𝑦y∙ u
𝑢
u = u = 2  =  
𝑢u∙ u𝑢
20
2  4
and the component of y orthogonal to u is
7   8  −1
y − yˆ =   −   =  
 6  4  2
AN ORTHOGONAL PROJECTION
▪ The sum of these two vectors is y.
7   8  −1
 6 =  4 +  2
     
y
ŷ
ˆ
(y − y)
▪ The decomposition of y is illustrated in the following
figure:
14
ORTHONORMAL SETS
▪ A set {u1,…,up} is an orthonormal set if it is an
orthogonal set of unit vectors.
▪ If W is the subspace spanned by such a set, then
{u1,…,up} is an orthonormal basis for W, since the set is
automatically linearly independent, by Theorem 4.
▪ The simplest example of an orthonormal set is the
standard basis {e1,…,en} for ℝ𝑛 .
▪ Any nonempty subset of {e1,…,en} is orthonormal, too.
ORTHONORMAL SETS
▪ Example 2: Show that {v1, v2, v3} is an orthonormal
basis of ℝ3 , where
 −1/
3 / 11 



v1 = 1/ 11, v 2 =  2 /



1/
11


 1/
 −1/ 66 
6



,6  v =  −4 / 66 
3



7
/
66
6 


15
ORTHONORMAL SETS
▪ Solution: Compute
vv11 ∙ vv22 = −3 / 66 + 2 / 66 + 1/ 66 = 0
vv11 ∙ vv33 = −3 / 726 − 4 / 726 + 7 / 726 = 0
vv2 2∙ vv33 = 1/ 396 − 8 / 396 + 7 / 396 = 0
▪ Thus {v1, v2, v3} is an orthogonal set.
ORTHONORMAL SETS
▪ Also,
vv
v11 = 9 /11 + 1/11 + 1/11 = 01
1 1∙ v
vv22 ∙ v22 = 1/ 6 + 4 / 6 + 1/ 6 = 1
vv33∙ v33 = 1/ 66 + 16 / 66 + 49 / 66 = 1
which shows that v1, v2, and v3 are unit vectors.
▪ Thus {v1, v2, v3} is an orthonormal set.
▪ Since the set is linearly independent, its three vectors
form a basis for ℝ3 .
16
ORTHONORMAL SETS
▪ When the vectors in an orthogonal set of nonzero
vectors are normalized to have unit length, the new
vectors will still be orthogonal, and hence the new set
will be an orthonormal set.
ORTHONORMAL SETS
▪ Theorem 6: An m  n matrix U has orthonormal columns
if and only if
T
U U =I
▪ Proof: To simplify notation, we suppose that U has only
three columns, each a vector in ℝ𝑚 .
▪ Let U =  u1 u 2 u 3  and compute
 u1T 
 
U TU =  u T2   u1
 u T3 
u2
 u1T u1 u1T u 2

u 3  =  u T2 u1 u T2 u 2
 u T3 u1 u T3 u 2
u1T u 3 

u T2 u 3 
u T3 u 3 
17
ORTHONORMAL SETS
▪ The entries in the matrix at the right are inner
products, using transpose notation.
▪ The columns of U are orthogonal if and only if
u1T u 2 = u T2 u1 = 0 u1T u 3 = u T3 u1 = 0 u T2 u 3 = u T3 u 2 = 0
▪ The columns of U all have unit length if and only if
u1T u1 = 1
u T2 u 2 = 1
u T3 u 3 = 1
ORTHONORMAL SETS
▪
Theorem 7: Let U be an m  n matrix with
orthonormal columns, and let x and y be in ℝ𝑛 .
Then
Ux = x
𝑈𝑥 ∙ 𝑈𝑦 = 𝑥 ∙ 𝑦
𝑈𝑥 ∙ 𝑈𝑦 =0 if and only if 𝑥 ∙ 𝑦=0
▪
Properties say that the linear x Ux mapping
preserves lengths and orthogonality.
18
6.3 ORTHOGONAL
PROJECTIONS
37
ORTHOGONAL PROJECTIONS
▪ The orthogonal projection of a point in ℝ2 onto a line
through the origin has an important analogue in ℝ𝑛 .
▪ Given a vector y and a subspace W in ℝ𝑛 , there is a
vector ŷ in W such that (1) ŷ is the unique vector in W
ˆ is orthogonal to W, and (2) ŷ is the
for which y − y
unique vector in W closest to y.
19
THE ORTHOGONAL DECOMPOSITION
THEOREM
▪ These two properties of ŷ provide the key to finding
the least-squares solutions of linear systems.
▪ Theorem 8: Let W be a subspace of ℝ𝑛 . Then each y
in ℝ𝑛 can be written uniquely in the form
y = yˆ + z
where ŷ is in W and z is in W ⊥.
(1)
▪ In fact, if {u1,…,up} is any orthogonal basis of W, then
ŷ =
𝑦y∙ 𝑢
u11
𝑢u11 ∙ u𝑢11
u1 +
+
𝑦y ∙ u𝑢p𝑝
𝑢
u𝑝p ∙u𝑢p𝑝
up
(2)
and z = y − yˆ .
THE ORTHOGONAL DECOMPOSITION
THEOREM
▪ The vector ŷ is called the orthogonal projection of y
onto W and often is written as projWy.
▪ Proof: Let {u1,…,up} be any orthogonal basis for W,
and define ŷ by (2).
▪ Then ŷ is in W because ŷ is a linear combination of
the basis u1,…,up.
20
THE ORTHOGONAL DECOMPOSITION
THEOREM
▪ Let z = y − yˆ .
▪ Since u1 is orthogonal to u2,…,up, it follows from (2)
that
y∙u
1
z ∙ u1 = y − yො ∙ u1 = y ∙ u1 −
u1 ∙ u1 − 0…0-0
u1 ∙u1
= y ∙ u1 − y ∙ u1 = 0
▪ Thus z is orthogonal to u1.
▪ Similarly, z is orthogonal to each uj in the basis for W.
▪ Hence z is orthogonal to every vector in W.
▪ That is, z is in W ⊥.
THE ORTHOGONAL DECOMPOSITION
THEOREM
▪ Example 1: Let
 2
 −2 
 1
u1 =  5 ,u 2 =  1 ,and y =  2 
 
 
 
 −1
 1
 3
Observe that {u1, u2} is an orthogonal basis for
W = Span{u1 ,u 2 } . Write y as the sum of a vector in W
and a vector orthogonal to W.
21
THE ORTHOGONAL DECOMPOSITION
THEOREM
▪ Solution: The orthogonal projection of y onto W is
ŷ =
𝑦y ∙ u
𝑢11
𝑢𝑢u111 ∙∙ u
𝑢𝑢111
u1 +
𝑦y∙ u
𝑢22
𝑢u22 ∙ u
𝑢22
u2
 2
 −2 
 2
 −2   −2 / 5
9   3   9   15   
=
5 +
1 =
5 +
1 =
2

30   6   30   30   
 −1
 1
 −1
 1  1/ 5
▪ Also
 1  −2 / 5  7 / 5
y − yˆ =  2  − 
2 = 
0
  
 

 3  1/ 5 14 / 5
THE ORTHOGONAL DECOMPOSITION
THEOREM
▪ Theorem 8 ensures that y − yˆ is in W .⊥
▪ To check the calculations, verify that y − yˆ is
orthogonal to both u1 and u2 and hence to all of W.
▪ The desired decomposition of y is
 1  −2 / 5  7 / 5
y = 2 = 
2 + 
0
  
 

 3  1/ 5 14 / 5
22
PROPERTIES OF ORTHOGONAL
PROJECTIONS
▪ If {u1,…,up} is an orthogonal basis for W and if y
happens to be in W, then the formula for projWy is
exactly the same as the representation of y given in
Theorem 5 in Section 6.2.
▪ In this case, projW y = y
▪ If y is in W = Span{u1 , ,u p } , then projW y = y .
THE BEST APPROXIMATION THEOREM
▪ Theorem 9: Let W be a subspace of ℝ𝑛 , let y be any
vector in ℝ𝑛 , and let ŷ be the orthogonal projection of y
onto W. Then ŷ is the closest point in W to y, in the
sense that
y − yˆ  y − v
(3)
for all v in W distinct from ŷ.
▪ The vector ŷ in Theorem 9 is called the best
approximation to y by elements of W.
▪ The distance from y to v, given by y − v , can be
regarded as the “error” of using v in place of y.
▪ Theorem 9 says that this error is minimized when v = yˆ .
23
PROPERTIES OF ORTHOGONAL
PROJECTIONS
▪ Example 4:The distance from a point y in ℝ𝑛 to a
subspace W is defined as the distance from y to the
nearest point in W. Find the distance from y to
W = Span{u1 , u 2 } , where
 −1
 5
 1
y =  −5 , u1 =  −2  , u 2 =  2 
 
 
 
 10 
 1
 −1
▪ Solution: By the Best Approximation Theorem, the
distance from y to W is y − yˆ , where ŷ = projW y
PROPERTIES OF ORTHOGONAL
PROJECTIONS
▪ Since {u1, u2} is an orthogonal basis for W,
 5
 1  −1
15
−21
1
7
ŷ = u1 +
u 2 =  −2  −  2  =  −8
30
6
2  2   
 1
 −1  4 
 −1  −1 0 
y − yˆ =  −5 −  −8 =  3
     
 10   4  6 
y − yˆ = 32 + 62 = 45
2
▪ The distance from y to W is 45 = 3 5 .
24
PROPERTIES OF ORTHOGONAL
PROJECTIONS
▪ Theorem 10: If {u1,…,up} is an orthonormal basis for a
subspace W of ℝ𝑛 , then
proj𝑊 𝑦 = y ∙ u1 u1 + y ∙ u2 u2 + ⋯ + y ∙ up up (4)
If
U =  u1
for all y in ℝ𝑛
u2
u p  , then
projW y = UU T y
(5)
6.4 THE GRAMSCHMIDT PROCESS
50
25
THE GRAM-SCHMIDT PROCESS
51
THE GRAM-SCHMIDT PROCESS
52
26
THE GRAM-SCHMIDT PROCESS
▪ Theorem 11: The Gram-Schmidt Process
▪ Given a basis {x1, . . . , xp} for a nonzero subspace W of ℝ𝑛 ,
define
v1 = x1
𝐱 ∙𝐯
v2 = x2 − 2 1 v1
v3 = x3 −
.
.
.
𝐯1 ∙𝐯1
𝐱3∙𝐯1
𝐯1 ∙𝐯1
𝐱 ∙𝐯
v1 − 3 2 v2
𝐯2 ∙𝐯2
𝐱 ∙𝐯
𝐱 ∙𝐯
𝐯1 ∙𝐯1
𝐯2 ∙𝐯2
v𝑝 = x𝑝 − 𝑝 1 v1 − 𝑝 2 v2− . . . −
𝐱𝑝∙𝐯𝑝−1
v
𝐯𝑝−1 ∙𝐯𝑝−1 p–1
▪ Then {v1 , . . . , vp} is an orthogonal basis for W. In addition
(1)
Span{v1 , . . . , vk} = Span{x1 , . . . , xk} for 1 ≤ 𝑘 ≤ 𝑝.
THE GRAM-SCHMIDT PROCESS
▪ Proof For 1 ≤ 𝑘 ≤ 𝑝, let Wk = Span{x1 , . . . , xk}. Set v1 = x1, so
that Span{v1} = Span {x1}. Suppose for some k < p, we have
constructed v1, . . . , vk so that {v1 , . . . , vk} is an orthogonal basis
for Wk. Define
vk+1 = xk+1 – projWkxk+1
(2)
▪ By the Orthogonal Decomposition Theorem, vk+1 is orthogonal to
Wk. Furthermore, vk+1 ≠ 0 because xk+1 is not in
Wk = Span{x1 , . . . , xk}
▪ Hence {v1 , . . . , vk} is an orthogonal set of nonzero vectors in the
(k + 1)-dimensional space Wk+1. By the Basis Theorem in Section
4.5, this set is an orthogonal basis for Wk+1. Hence Wk+1 = Span{v1
, . . . , vk+1}. When k + 1 = p, the process stops.
27
ORTHONORMAL BASES
▪ Example 3 Example 1 constructed the orthogonal basis
3
0
v1 = 6 , v2 = 0
0
2
▪ An orthonormal basis is
3 1/ 5
1
1
u1 =
v1 =
6 = 2/ 5
v1
45
0
0
0
1
u2 =
v = 0
v2 2
1
QR FACTORIZATION OF MATRICES
▪ Theorem 12: The QR Factorization
▪ If A is an m × n matrix with linearly independent columns,
then A can be factored as A = QR, where Q is an m × n
matrix whose columns form an orthonormal basis for Col
A and R is an n × n upper triangular invertible matrix with
positive entries on its diagonal.
28
QR FACTORIZATION OF MATRICES
1 0 0
▪ Example 4 Find a QR factorization of A = 1 1 0 .
1 1 1
1 1 1
▪ Solution The columns of A are the vectors x1, x2, and x3 in
Example 2. An orthogonal basis for
Col A
= Span{x1, x2, x3} was found in that example:
0
1
−3
−2/3
1
1
𝐯1 =
, 𝐯2 =
, 𝐯3 =
1/3
1
1
1/3
1
1
QR FACTORIZATION OF MATRICES
▪ To simplify the arithmetic that follows, scale v3 by replacing
v3 by 3v3. Then normalize the three vectors to obtain u1, u2,
and u3, and use these vectors as the columns of Q:
1/2 −3/ 12
0
1/2 1/ 12 −2/ 6
Q=
.
1/2 1/ 12
1/ 6
1/2 1/ 12
1/ 6
▪ By construction, the first k columns of Q are an
orthonormal basis of Span{x1 , . . . , xk}.
29
QR FACTORIZATION OF MATRICES
▪ From the proof of Theorem 12, A = QR for some R. To find
R, observe that QTQ = I, because the columns of Q are
orthonormal. Hence
𝑄𝑇𝐴 = 𝑄𝑇 𝑄𝑅 = 𝐼𝑅 = 𝑅
▪ and
1/2
1/2
1/2
R = −3/ 12 1/ 12 1/ 12
0
−2/ 6 1/ 12
1/2
1/ 12
1/ 6
1
1
1
1
0
1
1
1
0
0
1
1
2
3/2
1
= 0 3/ 12 2/ 12
0
0
3/ 12
6.5 LEASTSQUARES
PROBLEMS
60
30
LEAST-SQUARES PROBLEMS
▪ Definition: If A is m  n and b is in ℝ𝑚 , a least-squares solution
of Ax = b is an x̂ in ℝ𝑛 such that
𝑛
b − Axˆ  b − Ax
for all x in ℝ .
▪ The most important aspect of the least-squares problem is that
no matter what x we select, the vector Ax will necessarily be in
the column space, Col A.
▪ So we seek an x that makes Ax the closest point in Col A to b.
LEAST-SQUARES PROBLEMS
▪ Solution of the General Least-Squares Problem
▪ Given A and b, apply the Best Approximation Theorem to the
subspace Col A.
▪ Let
b̂ = proj b
Col A
▪ Because b̂ is in the column space A, the equation Ax = bˆ is
consistent, and there is an x̂ in ℝ𝑛 such that
Ax̂ = bˆ
(1)
▪ Since b̂ is the closest point in Col A to b, a vector x̂ is a leastsquares solution of Ax = b if and only if x̂ satisfies (1).
31
SOLUTION OF THE GENREAL LEASTSQUARES PROBLEM
▪ Suppose x̂ satisfies Ax̂ = bˆ
▪ By the Orthogonal Decomposition Theorem, the projection b̂
has the property that b − Axˆ is orthogonal to Col A, so b − bˆ
is orthogonal to each column of A.
▪ If aj is any column of A, then 𝑎𝑗 ∙ 𝑏 − 𝐴𝑥ො = 0, and
𝑎𝑗𝑇 𝑏 − 𝐴𝑥ො = 0.
SOLUTION OF THE GENREAL LEASTSQUARES PROBLEM
▪ Since each a Tj is a row of AT,
ˆ =0
AT (b − Ax)
▪ Thus
(2)
AT b − AT Axˆ = 0
AT Ax̂ = AT b
▪ These calculations show that each least-squares solution of Ax = b
satisfies the equation
AT Ax = AT b (3)
▪ The matrix equation (3) represents a system of equations
called the normal equations for Ax = b .
▪ A solution of (3) is often denoted by x̂ .
32
SOLUTION OF THE GENREAL LEASTSQUARES PROBLEM
▪ Theorem 13: The set of least-squares solutions of Ax = b coincides
with the nonempty set of solutions of the normal equation
AT Ax = AT b
SOLUTION OF THE GENREAL LEASTSQUARES PROBLEM
▪ Example 1: Find a least-squares solution of the system Ax = b
for
4 0
 2
A = 0 2 , b =  0


 
 1 1
11
33
SOLUTION OF THE GENREAL LEASTSQUARES PROBLEM
▪ Solution: To use normal equations (3), compute:
4 0
 4 0 1 
 = 17 1
A A=
0
2



 0 2 1  1 1  1 5


 2
 4 0 1   19 
T
A b=
 0 = 
 0 2 1 11 11
 
T
▪ Then the equation AT Ax = AT b becomes
17 1  x1  19 
 1 5  x  = 11

 2  
SOLUTION OF THE GENREAL LEASTSQUARES PROBLEM
▪ Row operations can be used to solve the system on the previous
slide, but since ATA is invertible and 2  2, it is probably faster to
compute
( AT A) −1 =
1  5 −1
84  −1 17 
and then solve AT Ax = AT b as
x̂ = ( AT A) −1 AT b
=
1  5 −1 19  1  84   1
=
=
84  −1 17  11 84 168  2 
34
SOLUTION OF THE GENREAL LEASTSQUARES PROBLEM
▪
Theorem 14: Let A be an m  n matrix. The following
statements are logically equivalent:
a. The equation Ax = b has a unique least-squares solution
for each b in ℝ𝑚 .
b. The columns of A are linearly independent.
c. The matrix ATA is invertible.
When these statements are true, the least-squares solution x̂ is given by
(4)
T
−1 T
x̂ = ( A A) A b
▪
When a least-squares solution x̂ is used to produce Ax̂ as an
approximation to b, the distance from b to Ax̂ is called the
least-squares error of this approximation.
ALTERNATIVE CALCULATIONS OF
LEAST-SQUARES SOLUTIONS
▪ Theorem 15: Given an m  n matrix A with linearly independent
columns, let A = QR be a QR factorization of A. Then, for each b in
ℝ𝑚 , the equation Ax = b has a unique least-squares solution,
given by
x̂ = R −1QT b
35
Question?
71
36