3.V. Projection
3.V.1. Orthogonal Projection Into a Line
3.V.2. Gram-Schmidt Orthogonalization
3.V.3. Projection Into a Subspace
3.V.1. & 2. deal only with inner product spaces.
3.V.3 is applicable to any direct sum space.
3.V.1. Orthogonal Projection Into a Line
Definition 1.1:
Orthogonal Projection
The orthogonal projection of v into the line spanned by a nonzero s is the vector.
projs  v    v  sˆ  sˆ 
sˆ 
s

s
Example 1.3:
 x
s 
 2x 
vs
s
ss
s
ss
Orthogonal projection of the vector ( 2 3 )T into the line y = 2x.
s  x  4x  5 x
2
2
 2 1
1 1 8  1
projs   
 2  6     
5
5  2 5  2
 3
1
ˆs 
5
1
 2
 
Example 1.4: Orthogonal projection of a general vector in R3 into the y-axis
 0
e 2   1 
 0
 
 x   x   0 

proje2  y    y    1 
 z   z   0 
     
  0
 0
    y 1
 
  1
 0
  0 
 
0
  y 
0
 
Example 1.5:
Project = Discard orthogonal components
A railroad car left on an east-west track without its brake is pushed by a wind
blowing toward the northeast at fifteen miles per hour; what speed will the car reach?
15
w
2
 1
 1
 
15  1 
v  proje1 w 
 
2  0
speed  v 
15
2
Example 1.6:
Nearest Point
A submarine is tracking a ship moving along the line y = 3x + 2.
Torpedo range is one-half mile.
Can the sub stay where it is, at the origin on the chart, or must it
move to reach a place where the ship will pass within range?
1
Ship’s path is parallel to vector sˆ 
10
1
 3
 
Point p of closest approach is
 0
 0
p     projs  
 2
2
 0
6
1
 

10
10
 2
p 
1
10
5
0.632
1
 0  3  1
1  3 
 3   2   5  3  5  1 
 
 
 
 
(Out of range)
Exercises 3.V.1.
1. Consider the function mapping a plane to itself that takes a vector to its
projection into the line y = x.
(a) Produce a matrix that describes the function’s action.
(b) Show also that this map can be obtained by first rotating everything in
the plane π/4 radians clockwise, then projecting into the x-axis, and then
rotating π/4 radians counterclockwise.
3.V.2. Gram-Schmidt Orthogonalization
Given a vector s, any vector v in an inner product space can be decomposed as
v  projs v   v  projs v   v //  v 
where
v //  v   0
Definition 2.1:
Mutually Orthogonal Vectors
Vectors v1, …, vk  Rn are mutually orthogonal if
vi · vj = 0
ij
Theorem 2.2:
A set of mutually orthogonal non-zero vectors is linearly independent.
Proof:
c v
i
i
0
→
v j   ci vi  c j v j  v j  0
i
i
→
cj = 0  j
Corollary 2.3:
A set of k mutually orthogonal nonzero vectors in V k is a basis for the space.
Definition 2.5:
Orthogonal Basis
An orthogonal basis for a vector space is a basis of mutually orthogonal vectors.
Example 2.6:
 1  0   1 
 1 ,  2  ,  0 
     
 1  0   3 
     
Turn
 1
κ1   1
 1
 
1
 1  1 
1
projκ1  0    1   0 
 3  3  1  3 
 
   
 0
 1  0 
1
projκ1  2    1   2 
 0  3  1  0 
 
   
κ1  κ1  3
 0
 0
κ 2   2   projκ1  2 
 0
 0
 
 
 1
 1
 1  2  2 
 
3  
 1
 
 1
 0
2
  2  
 0 3
 
 1
 1
4
 1   1
  3  
 1
 1
 
 
1
1
1
κ 3   0   projκ1  0   projκ2  0 
 3
 3
 3
 
 
 
into an orthogonal basis for R3.
1
4
  0  
 3 3
 
 1
 1
 
 1
 
κ2  κ2 
 1
2
  1
3  
 1
24 8

9 3
1
 1  1   1
3
2
2
2
projκ2  0     2    0   2   
3
 3  8 3  1  3  3  1
 
     
 1
 1  2
  3
 1
 
 1
 1
2  0
 
 
 1
1
 
 
K
 1 
2
 
 1 
 
 1  1  1
 1 , 2  2  ,  0 
  3   
 1   1   1 
     
Theorem 2.7: Gram-Schmidt Orthogonalization
If  β1 , …, βk  is a basis for a subspace of Rn then the κj s calculated from the
following scheme is an orthogonal basis for the same subspace.
κ1  β1

κ1 κ1 
β 2  κ1
 β2 
κ1   I 
 β2
κ
κ
κ1  κ1
1
1 

κ 2  β2  projκ1 β2
κ 3  β3  projκ1 β3  projκ2 β3  β3 
k 1
κ k  βk   projκ j βk
j 1
Proof:
k 1
βk  κ j
j 1
κj κj
 βk  
κj

κ1 κ1
κ2 κ2 
 I 

 β2
κ
κ
κ
κ
1
1
2
2 

k 1 κ

κj
j
 I  

j 1 κ j κ j


 βk


For m  2 , Let  β1 , …, βm  be mutually orthogonal, and
m
βm1  κ j
j 1
κj κj
κ m1  βm1  
Then
β 3  κ1
β κ
κ1  3 2 κ 2
κ1  κ1
κ2  κ2
κj
m
βm1  κ j
j 1
κ j κj
κ i  κ m1  κ i  βm 1  
κ i  κ j  κi  βm1  βm1  κi  0
 i  1,
,m
QED
If each κj is furthermore normalized, the basis is called orthonormal.
The Gram-Schmidt scheme simplifies to:
κ1  β1
e1 
κ1
κ1
κ 2  β2  projκ1 β2  β2  β2  e1  e1
  I  e1 e1

κ 3  β3  projκ1 β3  projκ2 β3  β3  β3  e1  e1  β3  e2  e2
k 1
κ k  βk   projκ j βk
j 1
k 1
 βk    βk  e j  e j
j 1
e2 
β2
  I  e1 e1  e 2 e 2
k 1


  I   e j e j  βk
j 1


κ2
κ2
β
e3 
2
ek 
κk
κk
κ3
κ3
Exercises 3.V.2.
1. Perform the Gram-Schmidt process on this basis for R3,
 2  1   0
 2  ,  0  ,  3
     
 2   1   1 
     
2. Show that the columns of an nn matrix form an orthonormal set if and
only if the inverse of the matrix is its transpose. Produce such a matrix.
3.V.3. Projection Into a Subspace
Definition 3.1:
For any direct sum V = M  N and any v  V such that
v=m+n
with m  M and n  N
The projection of v into M along N is defined as
projM, N (v) = m
Reminder:
• M & N need not be orthogonal.
• There need not even be an inner product defined.
Example 3.2:

M 

The space M22 of 22 matrices is the direct sum of

N 


a, b  R 

a b
 0 0


Task: Find projM , N (A), where
0 0 
c d 



c, d  R 

3 1
A

0 4
Solution:
Let the bases for M & N be
→
∴
B  BM º BM

BM 
 1 0  0 1
 0 0 ,  0 0

 

 1 0  0 1  0 0  0 0
 0 0 ,  0 0 ,  1 0 ,  0 1

 
 
 

 1 0
 0 1
 0 0
 0 0
A  3

1

0

4

 0 0
 1 0
 0 1
 0 0






1 0
0 1
projM , N  A   3 

1

 0 0
 0 0


 3 1


 0 0
BN 
 0 0  0 0
 1 0 ,  0 1

 

is a basis for M22 .
Example 3.3:
Consider
and subspaces
Both subscripts on projM , N (v) are significant.


M 


 x
 y
 
z
 


y  2z  0 


  0

N   k  0 
 1
  
It’s straightforward to verify


k R 


with basis
&
1  0
 0 ,  2
   
 0  1
   
BM 
  0

L   k  1 
  2 
  


k R 


R3  M  N  M  L
Task: Find projM , N (v) and projM , L (v)
where
 2
v   2 
 5
 
Solution:
For
BM N  BM º BN 
 1  0  0
 0 ,  2 ,  0
     
 0  1  1
     
→
2
v   1 
4
 B M N
 2
2


proj M , N  v    1 
  2 
 0
 B M N  1 
For
BM L  BM º BL 
1  0  0 
 0 ,  2 ,  1 
     
 0   1   2 
     
→
 2 
 2
v   2    9 / 5 
 8 / 5 
 5

B M L
 
 2 
 2 
proj M , L  v    9 / 5 
  18 / 5 
 0 



B M N  9 / 5 
Note: BML is orthogonal but BMN is not.
2a  b  2
a 2b  5
Definition 3.4:
Orthogonal Complement
The orthogonal complement of a subspace M of Rn is
M = { v  Rn | v is perpendicular to all vectors in M }
( read “M perp” ).
The orthogonal projection projM (v ) of a vector is its projection into M along M .
Example 3.5:
In R3, find the orthogonal complement of the plane
Solution:
→
P   v
Natural basis for P is
v  β1  0, v  β2  0






 v1 
v 
 2
v 
 3
B


P


 x
 y
 
z
 
1  0
 0 ,  1
   
 3  2 
   
 v1 
1
0
3

   0
 0 1 2   v2    0 

 v   
 3


3x  2 y  z  0 


( parameter = z)





  3 

 k  2 
  1
  


k R 


Lemma 3.7:
Let M be a subspace of Rn.
Then M is also a subspace and Rn = M  M .
Hence,  v Rn, v  projM (v) is perpendicular to all vectors in M.
Proof: Construct bases using G-S orthogonalization.
Theorem 3.8:
Let v be a vector in Rn and let M be a subspace of Rn with basis  β1 , …, βk .
If A is the matrix whose columns are the β’s then
projM (v ) = c1β1 + …+ ck βk
where the coefficients ci are the entries of the vector (AT A) AT v.
That is,
projM (v ) = A (AT A)1 AT v.
Proof:
projM  v   M
By lemma 3.7,
→
0  AT  v  Ac 
projM  v   A c
→
AT A c  AT v
where c is a column vector
→
c   AT A

1
AT v
Interpretation of Theorem 3.8:
If B =  β1 , …, βk  is an orthonormal basis, then ATA = I.
In which case, projM (v ) = A (AT A)1 AT v = A AT v.
 β1T 


βk  
  v   β1
 βk T 


projM  v    β1
 β1T  v 


βk  

 βk T  v 


 v1 
βk   
v 
 k B
  β1
k
 v j β j
vj  βj  v
with
j 1
In particular, if B = Ek , then A = AT = I.
In case B is not orthonormal, the task is to find C s.t. B = AC and BTB = I.
I   AC 
Hence
T
 AC
 C A AC
T
T
→
AT A   CT  C1   CCT 
1
projM  v   BBT v  AC  AC  v
T
1
→
 ACCT A T v  A  AT A  AT v
1
CCT   AT A 
1
Example 3.9:
To orthogonally project


P


From
 0
y  1 
 0
 
into subspace


x, y  R 


 1 0
 2 0
1
0

1




T

A A
 0 1  0 1
0
1
0


  1 0  


→
A  AT A 
1
x  0  
 1
 
1
v   1
1
 
1
we get


P


 x
 y
 
z
 


xz0 


 1 0
A   0 1 
 1 0 


1
 1/ 2 0 
T
A
A

   0 1


 1 0
 1 0
 1/ 2 0 1/ 2 
1/
2
0
1
0

1
1/
2
0

1/
2


 


AT   0 1  
  0 1  
  0
1
0 




0 
 1 0   0 1   0 1 0   1 0   0 1
 1/ 2 0 1/ 2 






 1/ 2 0 1/ 2   1 
0
projP  v    0
1
0   1   1
 1/ 2 0 1/ 2   1 
0

 
 
Exercises 3.V.3.
1. Project v into M along N, where
 3
v   0 
1
 
2. Find M for


M 




M 


 x
 y x  y  0
 
z
 





 x
 y   x  3y  z  0
 
z
 


N 







1
c  0  c  R
1
 





3. Define a projection to be a linear transformation t : V → V with the property
that repeating the projection does nothing more than does the projection alone:
( t  t )(v) = t (v) for all v  V.
(a) Show that for any such t there is a basis B =  β1 , …, βn  for V such that
β
t  βi    i
0
for
i  1, 2, , r
i  r  1, r  2,
,n
where r is the rank of t.
(b) Conclude that every projection has a block partial-identity representation:
 I 0
RepB  B  t   

0
0

