3.8 Inner Product Spaces

Euclidean n-space:
Rn was defined to be the set of all ordered n-tuples of real
numbers. When Rn is combined with the standard operations of
vector addition, scalar multiplication, vector length, and the dot
product, the resulting vector space is called Euclidean n-space.
The dot product of two vectors is defined to be
u  v  u1v1  u2v2  L  unvn
The definitions of the vector length and the dot product are
needed to provide the metric concept for the vector space.
5-1

Axioms of inner product:
Let u, v, and w be vectors in a vector space V, and let c be
any scalar. An inner product on V is a function that
associates a real number <u, v> with each pair of vectors
u and v and satisfies the following axioms.
(1) 〈u , v〉〈v , u〉
(2) 〈u , v  w〉〈u , v〉〈u , w〉
(3)
c〈u , v〉〈cu , v〉
(4) 〈v , v〉 0 and 〈v , v〉 0 if and only if v  0
5-2

Note:
u  v  dot product (Euclidean inner product for R n )
 u , v  general inner product for vector space V

Note:
A vector space V with an inner product is called an inner
product space.
Vector space:
Inner product space:
V ,
V ,
, 
, , , 
5-3

Ex: (A different inner product for Rn)
Show that the function defines an inner product on R2,
where u  (u1 , u2 ) and v  (v1 , v2 ) .
〈u , v〉 u1v1  2u 2 v 2
Sol: (a) 〈u , v〉 u1v1  2u2v2  v1u1  2v2u2 〈v , u〉
(b) w  (w1 , w2 ) 〈u , v  w〉 u1 (v1  w1 )  2u2 (v2  w2 )
 u1v1  u1w1  2u2 v2  2u2 w2
 (u1v1  2u2v2 )  (u1w1  2u2 w2 )
〈u , v〉〈u , w〉
(c) c〈u , v〉 c(u1v1  2u2v2 )  (cu1 )v1  2(cu2 )v2 〈cu , v〉
(d ) 〈v , v〉 v1  2v2  0
2
2
〈v , v〉 0  v1  2v2  0
2
2
5-4
 v1  v2  0 ( v  0)

Ex: (A function that is not an inner product)
Show that the following function is not an inner product on R3.
〈u  v〉 u1v1  2u 2 v 2  u 3 v3
Sol:
Let v  (1 , 2 , 1)
Then  v , v  (1)(1)  2(2)(2)  (1)(1)  6  0
Axiom 4 is not satisfied.
Thus this function is not an inner product on R3.
5-5

Ex: (A function that is not an inner product)
Show that the following function is not an inner product on R3.
〈u  v〉 u1v1  2u 2 v 2  u 3 v3
Sol:
Let v  (1 , 2 , 1)
Then  v , v  (1)(1)  2(2)(2)  (1)(1)  6  0
Axiom 4 is not satisfied.
Thus this function is not an inner product on R3.
5-6

Norm (length) of u:
|| u ||  〈u , u〉
|| u ||2 〈u , u〉
For a norm, there are many possibilities.
(5) If A and B are two matrices, an inner product can be <A,B>=Tr(A†B),
where † is the transpose complex conjugate of the matrix and Tr means the
trace. Therefore
5-7
For a norm, there are many possibilities.


There is an example in criminal law in which the distinctions
between some of these norms has very practical consequences. If
you’re caught selling drugs in New York there is a longer sentence
if your sale is within 1000 feet of a school. If you are an attorney
defending someone accused of this crime, which of the norms
would you argue for? The legislators didn’t know linear algebra,
so they didn’t specify which norm they intended. The prosecuting
attorney argued for norm #1, “as the crow flies.” The defense
argued that “crows don’t sell drugs” and humans move along city
streets, so norm #2 is more appropriate.
The New York Court of Appeals decided that the Pythagorean
norm (#1) is the appropriate one and they rejected the use of the
pedestrian norm that the defendant advocated (#2).
5-8

Distance between u and v:
d (u , v) || u  v || u  v, u  v

Angle between two nonzero vectors u and v:
〈u , v〉
cos  
, 0  
|| u || || v ||

Orthogonal: (u  v )
u and v are orthogonal if 〈u , v〉 0 .
Note: If || v ||  1 , then v is called a unit vector.
v 1
v 0
 
Normalizin g
v
v
5-9
(the unit vector in the
direction of v)

Properties of norm:
(1) || u ||  0
(2) || u ||  0 if and only if u  0
(3) || cu ||  | c | || u ||

Properties of distance:
(1) d (u , v)  0
(2) d (u , v)  0 if and only if u  v
(3) d (u , v)  d ( v , u)
5 - 10

Ex: (Finding inner product)
 p , q  a0b0  a1b1  L  anbn is an inner product
Let p( x)  1  2 x 2 , q( x)  4  2 x  x 2 be polynomial s in P2 ( x)
(a) 〈p , q〉 ?
(b) || q || ?
(c ) d ( p , q )  ?
Note : p  (1, 0, 2) and q  (4, 2,1) with basis  {1, x, x 2 }
Sol:
(a) 〈p , q〉 (1)(4)  (0)(2)  (2)(1)  2
(b) || q ||  〈q , q〉 42  ( 2)2  12  21
( c ) Q p  q  3  2 x  3 x 2
 d ( p , q )  || p  q ||   p  q, p  q
 ( 3)2  22  ( 3)2  22
5 - 11

Thm 3.21：
Let u and v be vectors in an inner product space V.
(1) Cauchy-Schwarz inequality:
〈
| u , v〉|  ||u|| || v||
(2) Triangle inequality:
||u  v||  ||u||  || v||
(3) Pythagorean theorem :
u and v are orthogonal if and only if
|| u  v || 2  || u || 2  || v || 2
5 - 12

Orthogonal projections in inner product spaces:
Let u and v be two vectors in an inner product space V,
s
such that v  0. Then the orthogonal projection
of u onto
v is given by
u
u , v
projv u  u , v 
v
v
 v , v
Note:
r
r
rv
We can solve for this coefficient by noting
that
because
v

Proj
s
r
is orthogonal to
r a scalar multiple of s , it must be
orthogonal to s itself. Therefore, the consequent fact that
r
r r
r
the dot product (v  Projs v )gs is zero,
giving that projr vr  vr , sr  sr
r
s
s
5 - 13

Ex: (Finding an orthogonal projection in R3)
Use the Euclidean inner product in R3 to find the
orthogonal projection of u=(6, 2, 4) onto v=(1, 2, 0).
Sol:
Q  u , v  (6)(1)  (2)(2)  (4)(0)  10
 v , v  12  22  02  5
u v
 projv u 
v
vv
10
5
(1 , 2 , 0)  (2 , 4 , 0)
5 - 14

Thm 3.22: (Orthogonal projection and distance)
Let v and s be two vectors in an inner product space V,
such that s  0. Then
r
r
r r
d (v , projsr v )  d (v , cs ) ,
5 - 15
r r
v , s 
c r r
s , s
3.9 Orthonormal Bases: Gram-Schmidt Process

Orthogonal:
A set S of vectors in an inner product space V is called an
orthogonal set if every pair of vectors in the set is orthogonal.
S  v1 , v2 ,L , v n   V

 vi , v j   0 i  j
Orthonormal:
An orthogonal set in which each vector is a unit vector is
called orthonormal.
S   v1 , v 2 ,   , v n   V
5 - 16
1
 vi , v j   
0
i j
i j

Ex: (An orthonormal basis for P3 ( x) )
In P3 ( x ) , with the inner product
 p, q  a0b0  a1b1  a2b2
The standard basis B  {1, x , x 2 }  {v1 , v2 , v3 }
is orthonormal.
2
2
v

0

0
x

x
,
v

0

x

0
x
,
Sol: v1  1  0 x  0 x , 2
3
2
Then
 v1 , v 2   (1)(0)  (0)(1)  (0)(0)  0,
v1   v1 , v1  
 v1 , v 3   (1)(0)  (0)(0)  (0)(1)  0,
v2  v2 , v2  
 v 2 , v3   (0)(0)  (1)(0)  (0)(1)  0
v3   v3 , v3  
11  00  00  1,
00  11  00  1,
00  00  11  1
Thus, B is an orthonormal basis for P3 ( x)
5 - 17
.

Thm 3.23: (Orthogonal sets are linearly independent)
If S  v1 , v 2 ,L , v n  is an orthogonal set of nonzero vectors
in an inner product space V, then S is linearly independent.
Pf:
S is an orthogonal set of nonzero vectors
i.e.
Let


 v i , v j   0 i  j and  v i , v i   0
c1 v1  c2 v 2  L  cn v n  0
 c1 v1  c2 v 2  L  cn v n , v i    0, v i   0
i
c1  v1 , v i   c2  v 2 , v i   L  ci  v i , v i   L  cn  v n , v i 
0
Q  vi , vi   0
 ci  0 i
5 - 18
 S is linearly independent.

Ex: (Using orthogonality to test for a basis)
Show that the following set is a basis for R 4 .
v1
v2
v3
v4
S {(2 , 3 , 2 ,  2) , (1 , 0 , 0 , 1) , (1 , 0 , 2 , 1) , (1 , 2 ,  1 , 1)}
Sol:
v1 , v 2 , v 3 , v 4 : nonzero vectors
v1  v 2  2  0  0  2  0
v 2  v 3  1  0  0  1  0
v1  v 3  2  0  4  2  0
v 2  v 4  1  0  0  1  0
v1  v 4  2  6  2  2  0
v3  v 4  1  0  2  1  0
 S is orthogonal .
 S is a basis for R 4
5 - 19

Thm 3.24: (Coordinates relative to an orthonormal basis)
If B  {v1 , v 2 , L , v n } is an orthonormal basis for an inner
product space V, then the coordinate representation of a vector
w with respect to B is
[w]B   w , v1  v1   w , v 2  v 2  L   w , v n  v n
Pf:
B  {v1 , v 2 , L , v n } is a basis for V
w V
w  k1 v1  k2 v 2  L  kn v n (unique representation)
Q B  {v1 , v 2 , L , v n } is orthonormal
1
  vi , v j   
0
i j
i j
5 - 20
 w , v i  ( k1 v1  k2 v 2  L  kn v n ) , v i 
 k1  v1 , v i   L  ki  v i , v i   L  kn  v n , v i 
 ki
i
 w   w, v1  v1   w, v 2  v 2  L   w, v n  v n
 w B
  w, v1  
  w, v  
2 

 M 


  w, v n  
5 - 21

Ex: (Representing vectors relative to an orthonormal basis)
Find the coordinates of w = (5, -5, 2) relative to the following
orthonormal basis for R3 .
B  {v1 , v2 , v3 }  {( 53 , 54 , 0) ,(  54 , 53 , 0) , (0 , 0 , 1)}
Sol:
 w, v1   w  v1  (5 ,  5 , 2)  ( 53 , 54 , 0)  1
 w, v 2   w  v 2  (5,  5 , 2)  ( 54 , 53 , 0)  7
 w, v 3   w  v 3  (5 ,  5 , 2)  (0 , 0 , 1)  2
  1
 [w ]B   7
 2 
5 - 22

Gram-Schmidt orthonormalization process:
B  {u1 , u 2 , L , u n } is a basis for an inner product space V
Let v1  u1
w1  span({v1})
〈 u 2 , v1〉
v 2  u 2  projW1 u 2  u 2 
v1
w 2  span({v1 , v 2 })
〈 v1 , v1〉
 u 3 , v1 
 u3 , v 2 
v 3  u3  projW2 u3  u3 
v1 
v2
 v1 , v 1 
 v2 , v2 
M
 vn , vi 
v n  u n  projWn1 u n  u n  
vi
i 1  v i , v i 
n 1
 B '  {v1 , v 2 , L , v n } is an orthogonal basis.
vn
v1
v2
 B ''  {
,
,L ,
} is an orthonormal basis.
v1
v2
vn
5 - 23

Ex: (Applying the Gram-Schmidt orthonormalization process)
Apply the Gram-Schmidt process to the following basis.
u1
B  {(1 , 1 , 0) ,
u2
(1 , 2 , 0) ,
u3
(0 , 1 , 2)}
Sol: v1  u1  (1 , 1 , 0)
u 2  v1
3
1 1
v2  u2 
v1  (1 , 2 , 0)  (1 , 1 , 0)  ( , , 0)
v1  v1
2
2 2
u 3  v1
u3  v 2
v3  u3 
v1 
v2
v1  v1
v2  v2
1
1/ 2 1 1
 (0 , 1 , 2)  (1 , 1 , 0) 
( , , 0)  (0 , 0 , 2)
2
1/ 2 2 2
5 - 24
Orthogonal basis
 B'  {v1 , v 2 , v 3}  {(1, 1, 0), (
1 1
, , 0), (0, 0, 2)}
2 2
Orthonormal basis
v1 v 2 v 3
1 1
1 1
 B' '  {
,
,
}  {(
,
, 0), (
,
, 0), (0, 0, 1)}
v1 v 2 v 3
2 2
2 2
5 - 25

Ex: Find an orthonormal basis for the solution space of the
homogeneous system of linear equations.
x1  x 2
 7 x4  0
2 x1  x 2  2 x3  6 x 4  0
Sol: 1 1 0 7 0
 2 1 2 6 0



G. J .E
1 0 2  1 0
0 1  2 8 0


 x1   2s  t 
 2  1 
 x   2s  8t 
 2   8
r
r
2









s
t
 sv1  tv2
 x3   s 
1 0
  

   
x
t

0 1
 4 
5 - 26
Thus one basis for the solution space is
B  {v 1 , v 2 }  {(2 , 2 , 1 , 0) , (1 ,  8 , 0 , 1)}
v1  u1   2, 2, 1, 0 
 u 2 , v1 
 18
 2, 2, 1, 0
v2  u2 
v1  1,  8, 0, 1 
 v1 , v1 
9
  3,  4, 2, 1
 B'   2,2,1,0 3,4,2,1
(orthogonal basis)
  2 2 1    3  4
2
1 
 B' '  
, , ,0  , 
,
,
,

 3 3 3   30 30 30 30 
(orthonormal basis)
5 - 27
3.10 Mathematical Models and Least-Squares Analysis

Orthogonal complement of W:
Let W be a subspace of an inner product space V.
(a) A vector u in V is said to orthogonal to W, if u is
orthogonal to every vector in W.
(b) The set of all vectors in V that are orthogonal to W is
called the orthogonal complement of W.
W   {v V |  v , w  0 ,  w W }
W  (read “ W perp”)
5 - 28

Direct sum:
Let W1 and W2 be two subspaces of R n . If each vector x  R n
can be uniquely written as a sum of a vector w1 from W
1
and a vector w 2 from W2 , x  w1  w 2 , then R n is the
direct sum of W1 and W2 , and you can write R n  W1  W2

Thm 3.25: (Properties of orthogonal subspaces)
Let W be a subspace of Rn. Then the following properties
are true.
(1) dim( W )  dim( W  )  n
(2) R n  W  W 
(3) (W  )   W
5 - 29

Thm 3.26: (Projection onto a subspace)
If {u1 , u 2 , L , u t } is an orthonormal basis for the
subspace W of V, and for v V , then
projW v   v , u1  u1   v , u 2  u 2  L   v , u t  u t
||
v  projW  v
Q  projW  v, ui   0
i
5 - 30

Ex: (Projection onto a subspace)
w1  0, 3, 1, w 2  2, 0, 0, v  1, 1, 3
Find the projection of the vector v onto the subspace W.
Sol:
W  span({w1 , w 2})
w1, w2 
: an orthogonal basis for W
 w1 w 2  
3
1

 u1 , u 2   
,
,
), 1,0,0 :
  (0,
 w1 w 2  
10 10

an orthonormal basis for W
projW v   v  u1  u1   v  u 2  u 2
6
3
1
9 3

(0,
,
)  1, 0, 0   (1, , )
5 5
10
10 10
5 - 31
Fitting by Least-Squares
Scientists are often presented with a system that has no solution
and they must find an answer that is as close as possible to being
an answer.
Suppose that we have a coin to use in flipping and this coin has
some proportion m of heads to total flips.
Because of randomness, we do not find the exact proportion with
this sample
The vector of experimental
data {16, 34, 51} is not in
the subspace of solutions.
5 - 32
However, we want to find the m that most nearly works. An orthogonal
projection of the data vector into the line subspace gives our best guess.
The estimate (m = 7110/12600 ~ 0.56) is a bit high than 1/2 but not much, so
probably the penny is fair enough. The line with the slope m= 0.56 is called
the line of best fit for this data.
Minimizing the distance
between the given vector and
the vector used as the lefthand side minimizes the total
of these vertical lengths. We
say that the line has been
obtained through fitting by
least-squares.
5 - 33
The different denominations of U.S. money have different average
times in circulation
The linear system with equations
has no solution, but we can use orthogonal projection to find a
best approximation.
1 1 
1.5 
1 5 
 2


 
tr r
1 10   b   3 
Ax  v  
    
1 20   m   5 
1 50 
9


 
1 100 
 20 
5 - 34
The method on the Projection into a Subspace says that coefficients
b and m so that the linear combination
of the columns of A is as
r
close as possible to the vector v are the entries of
r b
T
1 T r
x     ( A A) A v 
m 
Some calculation gives an intercept of b = 1.05 and a slope of
m = 0.18.
5 - 35

Thm 3.27: (Orthogonal projection and distance)
Let S be a subspace of an inner product space V, and v .V
Then for all s  S , s  projS v
||v  projS v||  ||v  s ||
or ||v  projS v||  min ||v  s ||
( projS v is the best approximation to v from S)
5 - 36

Pf:
v  s  (v  projS v)  ( projS v  s)
(v  projS v)  ( projS v  s)
By the Pythagorean theorem
 ||v  s ||2  ||v  projS v||2  || projS v  s ||2
s  projS v  projS v  s  0
 ||v  s ||2  ||v  projS v||2
 ||v  projS v||||v  s ||
5 - 37
Fundamental subspaces of a matrix
For a given mn matrix [Amn]: The spaces NS(A) and RS(A) are orthogonal
complements of each other within Rn. This means that any vector from NS(A) is
orthogonal to any vector from CS(AT), and the vectors in these two spaces span
Rn, i.e.,
n
T
R  NS ( A)  CS ( A )
.
5 - 38

Thm 3.28:
If A is an m×n matrix, then
(1) (CS ( A))  NS ( A )
( NS ( A ))  CS ( A)
(2) (CS ( A ))  NS ( A)
( NS ( A))  CS ( A )
(3) CS ( A)  NS ( AT )  Rm
CS ( A)  ( NS ( A))  Rm
(4) CS ( AT )  NS ( A)  Rn
CS ( AT )  (CS ( A ))  Rn
5 - 39

Ex: (Fundamental subspaces)
Find the four fundamental subspaces of the matrix.
1
0
A
0

0
Sol:
2 0
0 1
0 0

0 0
CS ( A)  span
(reduced row-echelon form)
1,0,0,0  0,1,0,0
is a subspace of R4
1, 2, 0  0, 0,1 is a subspace of R
NS ( A)  span   2,1, 0  is a subspace of R
CS ( A )  RS  A  span
3
5 - 40
3
1 0 0 0
1 0 0 0
A   2 0 0 0   R   0 1 0 0 
 0 1 0 0 
 0 0 0 0 
s t
NS ( A )  span

 0, 0,1, 0
 0, 0, 0,1
Check:
(CS ( A))   NS ( A )
(CS ( A ))   NS ( A)
CS ( A)  NS ( AT )  R 4
CS ( AT )  NS ( A)  R 3
5 - 41
is a subspace of R4

Ex:
W  span({w1 , w 2})
Let W is a subspace of R4 and w1  (1, 2, 1, 0), w 2  (0, 0, 0, 1) .
(a) Find a basis for W
(b) Find a basis for the orthogonal complement of W.
Sol:
1
2
A 
1

0
0
1
0
0 
R
0
0


1
0
w1 w 2
0
1 
(reduced row-echelon form)
0

0
5 - 42
(a ) W  CS  A 

1, 2,1, 0  ,  0, 0, 0,1
is a basis for W
(b ) W   CS  A    NS  A 


1 2 1 0
QA  

0
0
0
1



  2,1, 0, 0 

Notes:
(1)
 x1   2 s  t 
  2    1
x  

1 0
s
  s t 
 2  
 x3   t 
0 1
  

   
0 0
 x4   0 
 1, 0,1, 0 
is a basis for W 
dim(W )  dim(W  )  dim( R 4 )
(2) W  W   R 4
5 - 43

Least-squares problem:
Axb
m  n n 1 m 1
(A system of linear equations)
(1) When the system is consistent, we can use the Gaussian
elimination with back-substitution to solve for x
(2) When the system is consistent, how to find the “best possible”
solution of the system. That is, the value of x for which the
difference between Ax and b is small.
5 - 44

Least-squares solution:
Given a system Ax = b of m linear equations in n unknowns,
the least squares problem is to find a vector x in Rn that
minimizes
Ax  b with respect to the Euclidean inner
product on Rn.
Such a vector is called a least-squares solution of Ax = b.
5 - 45
A  M m n
x  Rn
Ax  CS ( A) (Q CS  A  is a subspace of R m )
Let W  CS ( A)
Ax̂  ProjW b
ˆ  (b  Pr ojW b)  W
 (b  Ax)
ˆ  CS ( A)
 (b  Ax)
 b  Axˆ  (CS ( A))  NS ( A )
ˆ 0
 A (b  Ax)
i.e.,
A Axˆ  A b or
xˆ  ( AT A)1 AT b
(This is the solution of the normal system associated
with Ax = b)
5 - 46

Note: The problem of finding the least-squares solution of
is equal to the problem of finding an exact solution of the Ax  b
associated normal system A Axˆ  A b .

Thm:
If A is an m×n matrix with linearly independent column vectors,
then for every m×1 matrix b, the linear system Ax = b has a unique
least-squares solution. This solution is given by

1

x LS  ( A A) A b
Moreover, if W is the column space of A, then the orthogonal
projection of b on W is projW b  Ax  [ A( A A)1 A ] b
5 - 47

Ex: (Solving the normal equations)
Find the least squares solution of the following system
Ax  b
1 1 
0
1 2   x1    1 

 x   
1 3   2   3 
and find the orthogonal projection of b on the column space of A.
5 - 48

Sol:
1 1 
1 1 1 
3 6 

T
A A
1 2  



1
2
3
6
14





1
3


0 
1 1 1    4 
T
A b
1   


1 2 3   11
3
the associated normal system
3 6   x1   4 
A Ax  A b  
 



6 14  x2  11
T
T
5 - 49
the least squares solution of Ax = b
x LS
  53 
 3 
 2 
the orthogonal projection of b on the column space of A
projCS ( A)b  Ax LS
 61 
1 1  5
3 8


 1 2   3    6 
1 3  2   176 
5 - 50