MAC 2103
Module 11
lnner Product Spaces II
1
Learning Objectives
Upon completing this module, you should be able to:
1.
2.
3.
4.
5.
6.
Rev.F09
Construct an orthonormal set of vectors from an
orthogonal set of vectors.
Find the coordinate vector with respect to a given
orthonormal basis.
Construct an orthogonal basis from a nonstandard basis
in ℜⁿ using the Gram-Schmidt process.
Find the least squares solution to a linear system Ax = b.
Find the orthogonal projection on col(A).
Obtain the best approximation.
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
2
General Vector Spaces II
The major topics in this module:
Orthogonal Bases, Gram-Schmidt Process,
Least Squares and Best Approximation
Rev.09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
3
How to Construct an Orthonormal Set of Vectors
from an Orthogonal Set of Vectors?
We have learned from the previous module that two
vectors u and v in an inner product space V are
orthogonal to each other iff <u,v> = 0.
To obtain an orthonormal set, we will normalize each of
the vectors in the orthogonal set.
How to normalize the vectors? This can be done by
dividing each of them by their respective norm and
making each of them a unit vector.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
4
How to Construct an Orthonormal Set of Vectors
from an Orthogonal Set of Vectors? (Cont.)
Example 1: Find the orthonormal set of vectors from the
following set of vectors: Let S ={v1, v2} where v1 = (5,0) and
v2 = (0,-3).
Step 1: Verify that the set of vectors are mutually orthogonal
with respect to the Euclidean inner product on ℜ².
r r
r r
v1, v2   v1  v2  (5)(0)  (0)(3)  0 S is an orthogonal set.
Step 2: Find the norm for both vectors.
v1  5 2  0 2  5, and
r
v2  0 2  (3)2  3.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
5
How to Construct an Orthonormal Set of Vectors
from an Orthogonal Set of Vectors? (Cont.)
Step 3: Normalize the vectors in the orthogonal set.
r
r
v1  5 0 
v2  0 3
r
q1  r   ,   (1,0), q2  r   ,   (0,1)
v1  5 5 
v2  3 3 
Step 4: Verify that the set S is orthonormal by showing that
r
r r
q1, q2   0 and q1  q2  1.
r r
r r
q1 , q2   q1  q2  (1)(0)  (0)(1)  0,
r
r r 12
r r 12
q1  q1 , q1   (q1  q1 )  12  0 2  1, and
1
1
r
r r
r r
q2  q2 , q2  2  (q2  q2 ) 2  0 2  (1)2  1.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
6
Orthonormal Set, Orthonormal Basis, and
Orthogonal Basis
Orthonormal Set: An orthogonal set in which each vector is a
unit vector.
Orthonormal basis: A basis consisting of orthonormal vectors
in an inner product space. Example: The standard basis for
ℜⁿ.
Orthogonal basis: A basis consisting of orthogonal vectors in
an inner product space.
Note that if S is an orthogonal set, then S is a linearly
independent set.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
7
Orthonormal Set, Orthonormal Basis, and
Orthogonal Basis (Cont.)
If S ={v1, v2 , … , vn} is an orthogonal basis of W, then for any w
∈ W,
r r
r r
r r
r r
w, vi  r w, v1  r w, v2  r
w, vn  r
w   r r vi  r r v1  r r v2  ... r r vn ,
v1, v1 
v2 , v2 
vn , vn 
i 1 vi , vi 
r r
r r
r r
w, v1  w, v2 
w, vn 
where r r , r r , ... , r r
v1, v1  v2 , v2 
vn , vn 
n
are called the Fourier coefficients.
So the coordinate vector of w,
r r
r r
r r

w, v1  w, v2 
w, vn  
r
wS  (w)S   r r , r r , ... , r r  .
vn , vn  
 v1, v1  v2 , v2 
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
8
Orthonormal Set, Orthonormal Basis, and
Orthogonal Basis (Cont.)
If S ={q1, q2 , … , qn} is an orthonormal basis of W, then for any
w ∈ W,
r r
n
w, qi  r
r r r
w   r r qi   w, qi qi
i 1 qi , qi 
i 1
r r r
r r r
r r r
 w, q1 q1  w, q2 q2  ... w, qn qn ,
r r
r 2
as qi , qi   qi  1 for i  1, 2, ... , n.
n
r r
r r
r r
where w, q1 ,w, q2 , ... ,w, qn  are called the Fourier
coefficients.
So the coordinate vector of w,
r
r r
r r
r r
wS  (w)S  (w, q1 ,w, q2 , ... ,w, qn ).
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
9
How to Find the Coordinate Vector with Respect to
a Given Orthogonal Basis?
Example 2: Compute the coefficients and determine the
coordinate vectors in Example 1 for u = (10,3).
From Example 1, we have v1 = (5,0), v2 = (0,-3) and
r
v1  5, and v2  3.
In this case, the coefficients are:
r r
r r
u, v1  u  v1 (10)(5)  (3)(0) 50

2
r r  r 2 
2
v1 , v1 
5
25
v1
r r
r r
u, v2  u  v2 (10)(0)  (3)(3) 9

 1
r r  r 2 
2
v2 , v2 
3
9
v2
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
10
How to Find the Coordinate Vector with Respect to
a Given Orthogonal Basis? (Cont.)
So the coordinate vector of u,
r r
r r
 u, v1  u, v2  
r
uS  (u)S   r r , r r   (2,1).
 v1 , v1  v2 , v2  
We can see that a nice advantage of working with an
orthogonal basis is that the coefficients in any basis
representation for a vector are immediately known; they
are called Fourier coefficients.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
11
How to Find the Coordinate Vector with Respect to
a Given Orthonormal Basis?
Example 3: Find the coordinates of w = (2,3) relative to
the orthonormal basis for ℜ², B = {v1, v2}, where
1
1
1
1
r
v1  (
,
), and v2  (
,
).
2
2
2
2
Since B is orthonormal, we have
1
1
1
1
5
r r
r r
w, v1   w  v1  (2, 3)  (
,
)  (2)(
)  3(
)
,
2
2
2
2
2
1
1
1
1
1
r r
r r
w, v2   w  v2  (2, 3)  (
,
)  (2)( )  3(
)
,
2
2
2
2
2
5
1
r
r
r r
r r
and wB  (w)B  (w, v1 , w, v2 )  (
,
).
2
2
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
12
The Gram-Schmidt Process
Let S = {u1, u2, …, um} with nonzero ui ∈ ℜⁿ for i = 1, 2, … ,
m. S does not have to be a linearly independent set. It
might be that A = [u1 u2 … um] is an n x m matrix and the
source of S.
The Gram-Schmidt Algorithm: S = {u1, u2, …, um}
r
1. Let v1  u1 .
2. For k = 2, 3, … , m, let
r r
uk , vi  r
r
vk  uk   r r vi .
r
i 1 vi , vi 
If vk  0, we discard it since it is linearly dependent.
k 1
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
13
The Gram-Schmidt Process (Cont.)
3. We then have r ≤ m orthogonal and linearly
independent vectors in B = {v1, v2, …, vr}.
span(S) = span(B) = W, a r dimensional subspace of ℜⁿ
and B is an orthogonal basis for W. W = col(A) if A = [u1
u2 … um].
An orthogonal basis for W is {q1, q2, …, qr} where
r
vi
qi  r
vi
for i =1, 2, … , r.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
14
How to Construct an Orthonormal Basis from a Nonstandard
Basis in ℜⁿ using the Gram-Schmidt Process?
Let ℜⁿ be the usual Euclidean inner product space of
dimension n. Let {u1, u2, …, un} be a nonstandard basis
in ℜⁿ.
Step 1: Use the Gram-Schmidt method to construct an
orthogonal basis {v1, v2, …, vn} from the basis vectors
{u1, u2, …, un}.
Step 2: Normalize the orthogonal basis vectors to obtain
the orthonormal basis {q1, q2,…, qn}.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
15
How to Construct an Orthonormal Basis from a Nonstandard
Basis in ℜⁿ using the Gram-Schmidt Process? (Cont.)
Example 4:
r r r
The set {u1, u2 , u3 }  {(1,1,0),(1,2,1),(3,1,1)}is a nonstandard
basis in ℜ³.
Step 1:
r
r 2
r
v1  u1  (1,1, 0), v1  2, W1  span({v1 }), dim(W1 )  1.
Projections onto W1 have one component.
r r
u2 , v1  r
r
r
r
v2  u2  projW1 (u2 )  (1, 2,1)  r 2 v1
v1
1
3 3
r 2 11
r r
 (1, 2,1)  (1,1, 0)  ( , ,1), v2  , W2  span({v1 , v2 }) dim(W2 )  2.
2
2 2
2
Projections onto W2 have two components.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
16
How to Construct an Orthonormal Basis from a Nonstandard
Basis in ℜⁿ using the Gram-Schmidt Process? (Cont.)
r r
r r
u3 , v1  r u3 , v2  r
r
r
v3  u3  projW2 (u3 )  (2,1,1)  r 2 v1  r 2 v2
v1
v2
(2)
7 3 3
(1,1, 0) 
( , ,1)
2
11 / 2 2 2
14 3 3
 (3,1,1)  (1,1, 0)  ( , ,1)
11 2 2
21 21 14
1 1
3
 (3,1,1)  (1,1, 0)  ( , , )  ( , ,  ).
11 11 11
11 11 11
3 3
1 1
3
r
r
r
Thus, v1  (1,1, 0), v2  ( , ,1), v3  ( , ,  ).
2 2
11 11 11
form an orthogonal basis for ℜ³.
 (3,1,1) 
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
17
How to Construct an Orthonormal Basis from a Nonstandard
Basis in ℜⁿ using the Gram-Schmidt Process? (Cont.)
Step 2: Normalize the orthogonal basis vectors to obtain
the orthonormal basis B = {q1, q2, q3}.
The norms of these vectors are:
11
1
r
r
v1  2, v2 
, and v3 
.
2
11
So an orthonormal basis is B where
r
v1
1
1 1
q1  r 
(1,1, 0)  (
,
, 0),
v1
2
2 2
r
v2
2 3 3
3 2 3 2
2
r
q2  r 
( , ,1)  (
,
,
), and
v2
11 2 2
2 11 2 11 11
r
v
1 1
3
11 11 3 11
r
3
q3  r  11( , ,  )  (
,
,
).
v3
11 11 11
11 11
11
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
18
How to Find the Least Squares Solution?
The linear system Ax = b always has the associated normal
system ATAx = ATb which is consistent and has one or more
solutions. Any solution of this system is a least squares
solution of Ax = b. Moreover, rthe orthogonal projection of b
r
on W = col(A) is Ax  projW (b), where x is a least squares
solution.
Example 5: Find the least squares solution of the linear
system Ax = b given by




A


1
2
1
0
1
4
1
r
 and b  
.
2




 7 
Observe that A has two linearly independent column vectors,
so ATA is invertible, and there is a unique solution to ATAx =
ATb, which will be our least squares solution to Ax = b.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
19
How to Find the Least Squares Solution? (Cont.)
We have
 1 2 1   1
  2
AT A  
 0 1 4   1

0
1
4

 6 6 
  6 17 

 
 1 2 1   1   10 
r
  2   
AT b  



30
 0 1 4  7


 
So the normal system ATAx = ATb in this case is
 6 6   x1   10 




6
17
x
30

  2  

Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
20
How to Find the Least Squares Solution? (Cont.)
By Gauss Elimination, the row-echelon form of
r

 A A | A b    6 6
 6 17
T
T
10 

30 
is
 1 1

 0 1
5 / 33 
.
20 / 11 
The solution to the normal system ATAx = ATb in this case is
x1  5 / 33, and x2  20 /11.
 x1  r

  x  (x1, x2 )  (5 / 33,20 / 11)
 x2 
is our unique least square solution to Ax = b.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
21
How to Find the Orthogonal Projection and Obtain the
Best Approximation?
r
r
Ax  projW (b) is therorthogonal projection of b on W =
col(A). The projW (b) is the Best Approximation
to b in
r
W since the distance between b and projW (b) is the
minimum for all vectors in W,
r
r r
r
b  projW (b)  b  w for all w W .
Example 6: From the previous example, the orthogonal
projection of b on W = col(A) is
 1
r
r
projW (b)  Ax   2

 1
0
1
4

 5 / 33
  5 / 33   
  20 / 11   50 / 33
  235 / 33
 


.


Thus, we have obtained the best approximation to b in W.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
22
What have we learned?
We have learned to :
1. Construct an orthonormal set of vectors from an
orthogonal set of vectors.
2. Find the coordinate vector with respect to a given
orthonormal basis.
3. Construct an orthogonal basis from a nonstandard basis in
ℜⁿ using the Gram-Schmidt process.
4. Find the least squares solution to a linear system Ax = b.
5. Find the orthogonal projection on col(A).
6. Obtain the best approximation.
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
23
Credit
Some of these slides have been adapted/modified in part/whole from the
following textbook:
• Anton, Howard: Elementary Linear Algebra with Applications, 9th Edition
Rev.F09
http://faculty.valenciacc.edu/ashaw/
Click link to download other modules.
24