Inner Product Space
Inner Product Space
• A
An inner product
i
d t on a real vector space V is a l
t
Vi
function that associates a real number ⟨u, v⟩ with each pair of vectors u and v
each pair of vectors u
and v in V
in V in such a way in such a way
that the following axioms are satisfied for all vectors u, v, and w
, ,
in V and all scalars k.
–
–
–
–
⟨u, v⟩ = ⟨v, u⟩
⟨u + v, w⟩ = ⟨u, w⟩ + ⟨v, w⟩
⟨ku, v⟩ = k ⟨u, v⟩
⟨u, u⟩ ≥ 0 and ⟨u, u⟩ = 0 if and only if u = 0
A real vector space with an inner product is called a real inner product space.
1
n
Euclidean Inner Product on R
Euclidean Inner Product on R
• If u = (u1, u2, …, un) and v = (v1, v2, …, vn) are vectors in Rn, then the formula ⟨v, u⟩ = u ∙ v = u1v1 + u2v2 + … + unvn
d fi
defines ⟨v, u⟩
⟨ ⟩ to be the Euclidean inner b h
lid
i
product on Rn. • Inner products can be used to define norm and distance in a general inner product space
and distance in a general inner product space. 2
Theorem 6 1 1
Theorem 6.1.1
• If u and v are vectors in a real inner product space V, and if k
is a scalar, then
–
–
–
–
||v|| ≧
||v||
≧ 0 with equality if and only if v
0 with equality if and only if v = 0
0
||kv|| = |k| ||v||
d(u,v) = d(v,u)
( , ) ( , )
d(u,v) ≧ 0 with equality if and only if u = v
3
Euclidean Inner Product vs. Inner Product
• Th
The Euclidean inner product is the most E lid
i
d t i th
t
important inner product on Rn. However, there are various applications in which it is desirable to
are various applications in which it is desirable to modify the Euclidean inner product by weighting its terms differently. y
• More precisely, if w1,w2,…,wn are positive real numbers, and if u=(u1,u2,…,un) and v=(v1,v2,…,vn) are vectors in Rn, then it can be shown which is called weighted Euclidean inner product which
is called weighted Euclidean inner product
with weights w1,w2,…,wn
4
Weighted Euclidean Inner Product
Weighted Euclidean Inner Product
• Let u = (u1, u2) and v = (v1, v2) be vectors in R2. Verify that the weighted Euclidean inner product ⟨u, v⟩ = 3u1v1 + 2u2v2
satisfies the four product axioms
satisfies the four product axioms.
• Solution:
– Note first that if u and v are interchanged in this equation, the right side remains the same. Therefore, ⟨u, v⟩ = ⟨v, u⟩.
– If w = (w1, w2), then ⟨u + v, w⟩ = 3(u1+v1)w1 + 2(u2+v2)w2 = (3u1w1 + 2u2w2) + (3v1w1 + 2v2w2)= ⟨u, w⟩ + ⟨v, w⟩
which establishes the second axiom.
5
Weighted Euclidean Inner Product
Weighted Euclidean Inner Product
– ⟨ku, v⟩ =3(ku1)v1 + 2(ku2)v2 = k(3u1v1 + 2u2v2) = k ⟨u, v⟩
which establishes the third axiom.
– ⟨v, v⟩
⟨ ⟩ = 3v
3 1v1+2v
2 2v2 = 3v
3 12 + 2v
2 22 2 .Obviously , ⟨v, v⟩
Ob i l ⟨ ⟩ = 3v
3 12 + 2v22 ≥0 . Furthermore, ⟨v, v⟩ = 3v12 + 2v22 = 0 if and only if v1 = vv2 = 0, That is , if and only if v
0 That is if and only if v = (v
(v1,vv2))=0
0. Thus, the Thus the
fourth axiom is satisfied.
6
Applications of Weighted Euclidean Inner Products
• Suppose that some physical experiment can produce any of n possible numerical values x1,x2,…,xn. • We perform m repetitions of the experiment and yield these values with various frequencies; that is, x1
occurs f1 times, x2 occurs f2 times, and so forth. f1+f2+…+fn=m.
• Thus, the arithmetic average, or mean, is 7
Applications of Weighted Euclidean Inner Products
• If we let f=(f1,f2,…,fn), x=(x1,x2,…,xn), w1=w2=…=wn=1/m
• Then this equation can be expressed as the Then this equation can be expressed as the
weighted inner product
8
Weighted Euclidean Inner Product
Weighted Euclidean Inner Product
• The norm and distance depend on the inner product used. – If the inner product is changed, then the norms and distances between vectors also change. – For example, for the vectors u = (1,0) and v = (0,1) in R2
with the Euclidean inner product, we have
u = 1 and d (u, v ) = u − v = (1,−1) = 12 − ( −1) 2 = 2
– However, if we change to the weighted Euclidean inner product ⟨u v⟩ = 3u
product ⟨u, v⟩
3u1v1 + 2u
+ 2u2v2 , then we obtain
then we obtain
u = u, u
1/ 2
= (3 ⋅1⋅1 + 2 ⋅ 0 ⋅ 0)1/ 2 = 3
d (u, v ) = u − v = (1,−1),
) (1,−1)
1/ 2
= [3 ⋅1 ⋅1 + 2 ⋅ (−1) ⋅ (−1)]1/ 2 = 5
9
Norm and Distance
Norm and Distance • If
If V
V is an inner product space, then the norm
i
i
d t
th th
(
(or length) of a vector u in V is denoted by ||u|| and is defined by
is defined by ||u|| = ⟨u, u⟩½
• The distance
The distance between two points (vectors) u
between two points (vectors) u and and
v is denoted by d(u,v) and is defined by d(u v) = ||u – v||
d(u, v) = ||u
• If
If a vector has norm 1, then we say that it is a a vector has norm 1 then we say that it is a
unit vector. 10
Unit Circles and Spheres in Inner Product Space
• If V is an inner product space, then the set of points in V that satisfy p
y
||u|| = 1 i
is called the unite sphere
ll d h
i
h
or sometimes the i
h
unit circle in V. In R2 and R3 these are the points that lie 1 unit away form the origin.
11
Unit Circles in R
Unit
Circles in R2
• Sketch the unit circle in an xy‐coordinate system in R2 using the Euclidean inner product ⟨u, v⟩ = u1v1 + u2v2
• Sketch the unit circle in an xy‐coordinate system in R
Sketch the unit circle in an xy‐coordinate system in R2 using the using the
Euclidean inner product ⟨u, v⟩ = 1/9u1v1 + 1/4u2v2
• Solution
– If u = (x,y), then ||u|| = ⟨u, u⟩½ = (x2 + y2)½, so the equation of the unit circle is x2 + y2 = 1.
– If u
If = (x,y), then ||u|| = ⟨u, u⟩
( ) th || || ⟨
⟩½ = (1/9x
(1/9 2 + 1/4y
1/4 2)½, so the equation of th
ti
f
the unit circle is x2 /9 + y2/4 = 1.
12
Inner Products Generated by Matrices
Inner Products Generated by Matrices
•
⎡ u1 ⎤
⎡ v1 ⎤
⎢u ⎥
⎢v ⎥
L u = ⎢⎢ :2 ⎥⎥ andd v = ⎢⎢ :2 ⎥⎥ b
Let be vectors in R
i Rn (expressed as n×1 (
d
1
⎢ ⎥
⎢ ⎥
⎢⎣un ⎥⎦
⎢⎣vn ⎥⎦
matrices), and let A be an invertible n×n matrix. • If u ∙ v is the Euclidean inner product on Rn, then the formula ⟨u, v⟩ = Au ∙ Av
defines an inner product; it is called the inner product on Rn
generated by A
generated by A.
• Recalling that the Euclidean inner product u ∙ v can be written as the matrix product vTu, the above formula can be written in as the matrix product v
u, the above formula can be written in
the alternative form ⟨u, v⟩ = (Av) TAu, or equivalently, ⟨u, v⟩ = vTATAu
13
Inner Product Generated by the Identity Matrix
• The inner product on Rn generated by the n×n identity matrix is the Euclidean inner product: Let A = I, we have ⟨u, v⟩ = Iu ∙ Iv
=u∙v
= u ∙
• The weighted Euclidean inner product ⟨u, v⟩ = 3u1v1 + 2u2v2 is the inner product on R2 generated by the inner product on R
generated by
⎡ 3
A=⎢
⎣⎢ 0
0 ⎤
⎥
2 ⎥⎦
since u , v = [v1
⎡ 3
v2 ]⎢
⎣0
0 ⎤⎡ 3
⎥⎢
2 ⎦⎣ 0
0 ⎤ ⎡u1 ⎤
⎥ ⎢ ⎥ = [v1
2 ⎦ ⎣u 2 ⎦
⎡3 0⎤ ⎡u1 ⎤
v2 ]⎢
⎥ ⎢u ⎥ = 3u1v1 + 2u 2 v2
0
2
⎣
⎦⎣ 2 ⎦
14
Inner Product Generated by the Diagonal Matrix
• In general, the weighted Euclidean inner product ⟨u, v⟩ = w1u1v1 + w2u2v2 + … + wnunvn is the inner product on Rn generated by
15
An Inner Product on M22
An Inner Product on M
• If If U = ⎢⎡
u1
⎣ u3
u2 ⎤
⎡ v1
and
d
V
=
⎢v
u4 ⎥⎦
⎣ 3
v2 ⎤
v4 ⎥⎦
are any two 2×2 matrices, then
⟨U V⟩ = tr(U
⟨U, V⟩
t (UTV) = tr(V
V) t (VTU) = u1v1 + u2v2 + u3v3 + u4v4
defines an inner product on M22
• For example, if p
⎡1 2 ⎤
⎡ −1 0 ⎤
U =⎢
and V = ⎢
⎥
⎣3 4 ⎦
⎣3
2 ⎥⎦
then ⟨U, V⟩ = 1(‐1)+2(0) + 3(3) + 4(2) = 16
• The norm of a matrix U relative to this inner product is U =< U , U >1/ 2 = u12 + u22 + u32 + u42
and the unit sphere in this space consists of all 2×2 matrices U
f
whose entries satisfy the equation ||U|| = 1, which on squaring yields u12 + u22 + u32 + u42 = 1
16
An Inner Product on P2
An Inner Product on P
• If
If p = a0 + a1x + a2x2 and q
d = b
b0 + b
b1x + b
b2x2 2 are any two vectors in P2, then the following formula defines an inner product on P2: :
defines an inner product on P
⟨p, q⟩ = a0b0 + a1b1 + a2b2
• The norm of the polynomial p
The norm of the polynomial p relative to this relative to this
inner product is p =< p, p >1/ 2 = a02 + a12 + a22
and the unit sphere in this space consists of all polynomials p in P2 whose coefficients satisfy the equation || p || = 1, which on squaring yields || ||
h h
ld
a02+ a12 + a22 =1
17
Theorem 6 1 2
Theorem 6.1.2
• Properties of Inner Products
• If u, v, and w
If u, v, and w are vectors in a real inner are vectors in a real inner
product space, and k is any scalar, then:
– ⟨0, v⟩
⟨0 ⟩ = ⟨v, 0⟩
⟨ 0⟩ = 0
0
– ⟨u, v + w⟩ = ⟨u, v⟩ + ⟨u, w⟩
– ⟨u, kv⟩ =k ⟨u, v⟩
– ⟨u ⟨u – v, w⟩
v, w⟩ = ⟨u, w⟩
⟨u, w⟩ – ⟨v, w⟩
⟨v, w⟩
– ⟨u, v – w⟩ = ⟨u, v⟩ – ⟨u, w⟩
18
Example Example
• ⟨u – 2v, 3u + 4v⟩
⟨ ,
⟩ – ⟨ 2v, 3u+4v⟩
,
⟩
= ⟨u, 3u + 4v⟩
= ⟨u, 3u⟩ + ⟨u, 4v⟩ – ⟨2v, 3u⟩ – ⟨2v, 4v⟩
= 3 ⟨u u⟩ + 4 ⟨u, v⟩
= 3 ⟨u, u⟩
+ 4 ⟨u v⟩ – 6 ⟨v, u⟩
6 ⟨v u⟩ – 8 ⟨v, v⟩
8 ⟨v v⟩
= 3 || u ||2 + 4 ⟨u, v⟩ – 6 ⟨u, v⟩ – 8 || v ||2 = 3 || u ||
3 || u ||2 2 – 2 ⟨u, v⟩
2 ⟨u v⟩ – 8 || v ||
8 || v ||2
19
Example
• We are guaranteed without any further proof p p
g
that the five properties given in Theorem 6.1.2 are true for the inner product on Rn generated by any matrix A
by any matrix A.
• ⟨u, v + w⟩ = (v+w)TATAu
=(vT+wT)ATAu
[Property of transpose]
=(vTATAu) + (w
Au) + (wTATAu) Au) [Property of matrix multiplication]
[Property of matrix multiplication]
= ⟨u, v⟩+ ⟨u, w⟩
20
Theorem 6 2 1
Theorem 6.2.1 • Cauchy‐Schwarz Inequality
C h S h
I
li
– If u and v are vectors in a real inner product space, then |⟨u, v⟩| | ≤ ||u|| ||v||
|| || || ||
– The inequality can be written in the following two forms
The inequality can be written in the following two forms
– The Cauchy‐Schwarz inequality for Rn follows as a special case of this theorem by taking y
g ⟨u, v⟩ to be the Euclidean inner product u‧v.
21
Angle Between Vectors
Angle Between Vectors
• Cauchy‐Schwarz inequality can be used to h
h
i
li
b
d
define angles in general inner product cases. • We define to be the angle between u and v. 22
Example
• Let R4 have the Euclidean inner product. Find the cosine of the angle θ between the vectors u = (4, 3, 1, ‐2) and v = (‐2, 1, 2, 3).
23
Theorem 6 2 2
Theorem 6.2.2
Properties of Length
i
f
h
• If u, v and w are vectors in an inner product space V, and if k is any scalar, then :
– || u + v || ≤ || u || + || v || (Triangle inequality for vectors)
– d(u,v) ≤ d(u,w) + d(w,v) (Triangle inequality for distances)
• Proof of (a)
P f f( )
(Property of absolute value)
(Theorem 6.2.1)
24
Orthogonality
• Two vectors u and v
d in an inner product space
i
i
d
are called orthogonal if ⟨u, v⟩ = 0.
• Example (⟨U, V⟩ = tr(UTV) = tr(VTU) = u1v1 + u2v2 + u3v3 + u4v4)
– If M22 has the inner project defined previously, then the matrices ⎡1 0⎤
⎡0 2 ⎤
U =⎢
and V = ⎢
⎥
⎥
1
1
0
0
⎣
⎣
⎦
⎦
are orthogonal, since ⟨U, V⟩ = 1(0) + 0(2) + 1(0) + 1(0) = 0.
25
Orthogonal Vectors in P2
Orthogonal Vectors in P
1
• LLet P
t P2 have the inner product and let p = x
h
th i
d t < p, q >= ∫ p( x)q( x))ddx
dl t
and d
−1
q = x2. • Then
⎡1
⎤
1/ 2
p =< p, p > = ⎢ ∫ xxdx ⎥
⎣ −1
⎦
1/ 2
⎡1 2 2 ⎤
1/ 2
q =< q, q > = ⎢ ∫ x x dx ⎥
⎣ −1
⎦
1
< p, q >=
1/ 2
⎡1 2 ⎤
= ⎢ ∫ x dx ⎥
⎣ −1
⎦
1/ 2
2
3
=
1/ 2
⎡1 4 ⎤
= ⎢ ∫ x dx ⎥
⎣ −1
⎦
=
2
5
1
3
xx
dx
=
x
∫
∫ dx =0
2
−1
−1
because ⟨p, q⟩ = 0, the vectors p = x and q = x 2 are orthogonal relative to the given inner product.
26
Theorem 6 2 3
Theorem 6.2.3
Generalized Theorem of Pythagoras
• If u
If u and v
and v are orthogonal vectors in an inner are orthogonal vectors in an inner
product space, then
|| + v ||2 = || u
|| u
|| ||2 + || v
|| ||2
• Proof:
27
Orthogonality
• Definition
D fi i i
– Let W be a subspace of an inner product space V. A vector u in V
in V is said to be orthogonal to W
is said to be orthogonal to W if it is orthogonal to if it is orthogonal to
every vector in W, and the set of all vectors in V that are orthogonal to W is called the orthogonal complement of W. – If V is a plane through the origin of R3 with Euclidean inner product, then the set of all vectors that are orthogonal to every vector in V forms the line L
every vector in V
forms the line L through the origin that is through the origin that is
perpendicular to V. L
V
28
Theorem 6 2 4
Theorem 6.2.4
Properties of Orthogonal Complements
i
f
h
l
l
• If W is a subspace of a finite‐dimensional inner p
product space V, then:
– W⊥ is a subspace of V. (read is a subspace of V. (read “W
W perp
perp”))
– The only vector common to W and W⊥ is 0; that is ,W ∩ W⊥ = 0. is ,W
0.
29
Proof of 6 2 4(a)
Proof of 6.2.4(a)
• Note first that ⟨0, w⟩=0 for every vector w in W, so W ⊥ contains at least the zero vector. • We want to show that the sum of two vectors in W ⊥ is orthogonal to every vector in W
in W
is orthogonal to every vector in W
(closed under addition) and that any scalar ⊥ is orthogonal to multiple of a vector in W
l l f
h
l
every vector in W (closed under scalar multiplication). 30
Proof of 6 2 4(a)
Proof of 6.2.4(a)
⊥, let k
• Let u and v
d be any vector in W
b
i
l k be any b
scalar, and let w be any vector in W. Then from the definition of W ⊥, we have ⟨u, w⟩=0 and ⟨v, w⟩=0. • Using the basic properties of the inner p
product, we have ,
⟨u+v, w⟩ = ⟨u, w⟩ + ⟨v, w⟩ = 0 + 0 = 0
⟨k
⟨ku, w⟩
⟩ = k⟨u, w⟩
k⟨
⟩ = k(0) = 0
k(0) 0
• Which proves that u+v and ku are in W ⊥. 31
Proof of 6 2 4(b)
Proof of 6.2.4(b)
The only vector common to W and W⊥ is 0; that is ,W ∩ W⊥ = 0. • If v is common to W and W ⊥, then ⟨v, v⟩=0, p
by Axiom 4 for inner y
which implies that v=0
products. 32
Theorem 6 2 5
Theorem 6.2.5
• If W
f is a subspace of a finite‐dimensional inner i
b
f fi i di
i
li
product space V, then the orthogonal complement of W ⊥ is W; that is
(W⊥)⊥ = W
33
Example (Basis for an Orthogonal Complement)
•
•
LLet W
t W be the subspace of R
b th
b
f R6 spanned d
by the vectors w1=(1, 3, ‐2, 0, 2, 0), w2=(2, 6, ‐5, ‐2, 4, ‐3), w3=(0, 0, 5, 10, 0, 15), w4=(2,
0, 15), w
(2, 6, 0, 8, 4, 18). Find a 6, 0, 8, 4, 18). Find a
basis for the orthogonal complement of W.
Solution
– Si
Since the row space and null space of th
d ll
f
A are orthogonal complements, this problem reduces to finding a basis for the null space of A.
the null space of A. – The space W spanned by w1, w2, w3, and w4 is the same as the row space of the matrix
form a basis for this null space.
34
Orthonormal Basis
• A
A set of vectors in an inner product space is called an t f
t i
i
d t
i
ll d
orthogonal set if all pairs of distinct vectors in the set are orthogonal.
are orthogonal. • An orthogonal set in which each vector has norm 1 is called orthonormal.
• Example
– Let u1 = (0, 1, 0), u2 = (1, 0, 1), u3 = (1, 0, ‐1) and assume that R3 has the Euclidean inner product. – It follows that the set of vectors S
It follows that the set of vectors S = {u
= {u1, u
u2, u
u3} is } is
orthogonal since ⟨u1, u2⟩ = ⟨u1, u3⟩ = ⟨u2, u3⟩ = 0.
35
Orthonormal
• If v is a nonzero vector in an inner product space, then the vector has norm 1
• Since
• The process of multiplying a nonzero vector v by the reciprocal of its length to obtain a unit vector is called
reciprocal of its length to obtain a unit vector is called normalizing v. • An orthogonal set of nonzero vectors can always be converted An orthogonal set of nonzero vectors can always be converted
to an orthonormal set by normalizing each of its vectors. 36
Example
• LLett u1 = (0, 1, 0), u2 = (1, 0, 1), u3 = (1, 0, ‐1). The Th
Euclidean norms of the vectors are
u1 = 1, u 2 = 2, u 3 = 2
• Normalizing u1, u2, and u3 yields
v1 =
u
u1
u
1
1
1
1
= (0,1,0), v 2 = 2 = (
,0,
), v 3 = 3 = (
,0,−
)
u1
u2
u3
2
2
2
2
• The set S = {v1, v2, v3} is orthonormal since ⟨v1, v2⟩ = ⟨v1, v3⟩ = ⟨v2, v3⟩ = 0 and ||v1|| = ||v2|| = ||v3|| = 1
37
Theorem 6 3 1
Theorem 6.3.1
• If
If SS = {v
{ 1, v2, …, vn} is an orthogonal set of }i
th
l t f
nonzero vectors in an inner product space, then S is linearly independent.
then S
is linearly independent
38
Proof of 6 3 1
Proof of 6.3.1
• A
Assume that k
th t k1v1+kk2v2+…+kknvn = 0. To 0 T
demonstrate that S is linearly independent, we must prove that k1=k2=…=0. must prove that k
= =0
• For each vi in S, ⟨k1v1+k2v2+…+knvn, vi⟩ = ⟨0,vi⟩=0
or equivalently k1⟨v1,vvi⟩+ k2⟨v2,vvi⟩+…+
or, equivalently k
⟩+ + kn⟨vn,vvi⟩⟩=0
0
• From the orthogonality of S it follows that ⟨vj,vi⟩⟩=0
0 when j
when j is not equal to i, so the equation is not equal to i, so the equation
reduces to ki⟨vi,vi⟩ =0
• Since the vectors in S are assumed to be nonzero, ,
⟨vi,vi⟩ ≠0. Therefore, ki=0. Since the subscript i is arbitrary, we have k1=k2=…=kn=0. 39
Orthonormal Basis
• In an inner product space, a basis consisting of i
d
b i
i i
f
orthonormal vectors is called an orthonormal basis, and a basis consisting of orthogonal vectors is called an orthogonal basis. • A familiar example of an orthornormal basis is the standard basis for R3
i=(1,0,0), j=(0,1,0), k=(0,0,1)
• The standard basis for R
Th t d d b i f Rn
e1=(1,0,0,…,0), e2=(0,1,0,…,0), …, en=(0,0,0,…,1)
40
Example
• The vectors
v1 =
u1
u
1
1
u
1
1
= (0,1,0), v 2 = 2 = (
)
,0,
), v 3 = 3 = (
,0,−
u1
u2
u3
2
2
2
2
form an orthonormal set with respect to the Euclidean inner product on R
d
R3
• From Theorem 6.3.1, we know that they are linearly independent
• Therefore, S={v1,v2,v3} is an orthonormal basis for R3
41
Coordinates Relative to Orthogonal Bases
• O
One way to express a vector u as a linear combination t
t
li
bi ti
of basis vectors S={v1,v2,…,vn} is to convert the vector equation u=c
q
1v1+c2v2+…+cnvn
• Theorem 6.3.2
– (a) If S = {v1, v2, …, vn} is an orthogonal basis for a vector space V, u is any vector in V, then
h
u=
u , v1
v1
2
v1 +
u, v 2
v2
2
v2 + " +
u, v n
vn
2
vn
– (b) If S = {v1, v2, …, vn} is an orthonormal basis for an inner product space V, and u
d
V d is any vector in V, then i
i V h
u = ⟨u, v1⟩ v1 + ⟨u, v2⟩ v2 + ∙ ∙ ∙ + ⟨u, vn⟩ vn
42
Proof of 6 3 2(a)
Proof of 6.3.2(a)
• Si
Since S={v
S { 1,v2,…,vn} is a basis for V, every vector u
}i b i f V
t
in V can be expressed in the form u=c1v1+c2v2+…+c
+ +cnvn
• We complete the proof by showing that • We observe first that We observe first that
• Since S is an orthogonal set, all of the inner products in the last equality are zero except for
products in the last equality are zero except for the ith, so
43
Orthonormal Basis
Orthonormal Basis
• The coordinates of the vector u relative to the g
{ 1, v2, …, vn}} is
orthogonal basis S = {v
and relative to an orthonormal basis S = {v
{ 1,, v2, …, vn}
(u)S = (⟨u, v
= (⟨u v1⟩,
⟩ ⟨u, v
⟨u v2⟩, ⟩ … , ⟨u, v
⟨u vn⟩)
44
Example Example
• Let
Let vv1 = (0, 1, 0), v
= (0 1 0) v2 = (‐4/5, 0, 3/5), v
= ( 4/5 0 3/5) v3 = (3/5, 0, 4/5). = (3/5 0 4/5)
It is easy to check that S = {v1, v2, v3} is an orthonormal basis for R3
with the Euclidean inner product. Express the vector u = (1, 1, 1) as a linear combination of the p
(
)
vectors in S, and find the coordinate vector (u)
df d h
d
( )s.
• Solution:
– ⟨u, v1⟩ = 1, ⟨u, v2⟩ = ‐1/5, ⟨u, v3⟩ = 7/5
– Therefore, by Theorem 6.3.1 we have u = v1 – 1/5 v2 + 7/5 v3
7/5 v
– That is, (1, 1, 1) = (0, 1, 0) – 1/5 (‐4/5, 0, 3/5) + 7/5 (3/5, 0, 4/5)
– The coordinate vector of u
Th
di t
t
f relative to S
l ti t S is i
(u)s=(⟨u, v1⟩, ⟨u, v2⟩, ⟨u, v3⟩) = (1, ‐1/5, 7/5)
45
Example
• Show that the vectors w1=(0,2,0), w2=(3,0,3), w3=(‐4,0,4) from an orthogonal basis for R3 with the Euclidean inner product, and use that basis to find an orthonormal basis d t d
th t b i t fi d
th
lb i
by normalizing each vector. • Solution: S l ti
– ⟨w1, w2⟩ = 0, ⟨w1, w3⟩ = 0, ⟨w2, w3⟩ = 0
– From Theorem 6.3.1, these vectors are linearly independent and hence from a basis for R3. 46
Example
• Express the vector u=(1,2,4) as a linear combination of the orthonormal basis vectors obtained previously. • Solution: Solution:
– u = ⟨u, v1⟩v1 + ⟨u, v2⟩v2 + ⟨u, v3⟩v3
47
Theorem 6 3 3
Theorem 6.3.3 • Projection Theorem
j i
h
– If W is a finite‐dimensional subspace of an inner product space V, then every vector u in V can be expressed in exactly one way as
u = w1 + w2
where w1 is in W and w2 is in W⊥.
u
w1
w2
u
W
w1
w2
W
48
u
Projection
w1
w2
W
• The vector w1 is called the orthogonal projection of u
on W and is denoted projWu. • The vector w2 is called the component of u
orthogonal to W and is denote by projW⊥ u. • u = projWu + projW⊥ u
• Since w
Since w2 = u‐w
= u‐w1, it follows that proj
it follows that projW⊥ u = u
= u – projWu
• So we can write u = projWu + (u – projWu)
u
W
(u – projWu)
projWu
49
Theorem 6 3 4
Theorem 6.3.4
• Let W be a finite‐dimensional subspace
b fi i di
i
l b
of an f
inner product space V.
– If {v1, …, vr} is an orthonormal basis for W, and u is any vector in V, then projwu = ⟨u,v1⟩ v1 + ⟨u,v2⟩ v2 + … + ⟨u,vr⟩ vr
– If {v1, …, vr} is an orthogonal basis for W, and u is any vector in V, then
projW u =
u, v 1
v1
2
v1 +
u, v 2
v2
2
v2 + " +
u, v r
vr
2
vr
N dN
Need Normalization
li ti
50
Proof
• u = w1+w2, where w1=projWu is in W and w2 is in W⊥
• From Theorem 6.3.2
• Since w2 is orthogonal to W, it follows that • Rewrite
51
Example Example
• LLet R
R3 have the Euclidean inner product, and let W
h
h E lid
i
d
d l W be the b h
subspace spanned by the orthonormal vectors v1 = (0, 1, 0) and v2 2 = (‐4/5, 0, 3/5). ( / , , / )
• From the above theorem, the orthogonal projection of u = (1, 1, 1) on W is projw u =< u, v1 > v1 + < u, v 2 > v 2
1
4
3
4
3
=(1)(0, 1, 0) + (− )(− , 0, )=( , 1, − )
5
5
5
25
25
• The component of u
h
f orthogonal to W
h
l
is 4
3
21
28
projw ⊥ u = u − projw u = (1, 1, 1) − ( , 1, − ) = ( , 0,
)
25
25
25
25
• Observe that projW⊥u is orthogonal to both v1 and v2.
52
Geometric Interpretation of Orthogonal Projections
• If W is a one‐dimensional subspace of an inner product space V, say span{a}, then • In the special case where V is R3 with the Euclidean inner product this exactly Formula (10) of Section 3 3 for the
product, this exactly Formula (10) of Section 3.3 for the orthogonal projection of u along a. • This suggests that we can think the equation as the sum of This suggests that we can think the equation as the sum of
orthogonal projections on axes determined by the basis vectors for the subspace W. projW u =
u , v1
v1
2
v1 +
u, v 2
v2
2
v2 +" +
u, v r
vr
2
vr
53
Finding Orthogonal/Orthonormal Bases
Finding Orthogonal/Orthonormal Bases
• Theorem 6.3.5
– Every nonzero finite‐dimensional inner product y
p
space has an orthonormal basis.
• Remark – The step‐by‐step construction for converting an arbitrary basis into an orthogonal basis
y
g
is called the Gram‐Schmidt process.
54
Proof of 6 3 5
Proof of 6.3.5
• Let V be an nonzero finite‐dimensional inner b
fi i di
i
li
product space, and suppose that {u1, u2, …, un} is any basis for V. It suffices to show that V has an orthogonal basis, since the vectors in the orthogonal basis can be normalized to produce an orthonormal basis for V. • The following sequence of steps will produce an orthogonal basis {v1, v
an orthogonal basis {v
, v2, …, v
, …, vn} for V. } for V.
• Step 1: Let v1 = u1. 55
Proof of 6 3 5
Proof of 6.3.5
u2
W1
v1
v2 = u2 – projW1u2
projW1u2
• Step 2: We can obtain a vector v2 that is orthogonal to v1 by computing the component of u2 that is orthogonal to the computing the component of u
that is orthogonal to the
space W1 spanned by v1. • Of course if v
O cou se 2=0, then v
0, t e 2 is not a basis vector. But this cannot s ot a bas s ecto ut t s ca ot
happen. Assume v2=0, • Which says that u2 is multiple of u1, contradicting the linear independence of the basis S={u1,u2,…,un}
56
Proof of 6 3 5
Proof of 6.3.5
• Step 3: To construct a vector v3 that is h i
orthogonal to both v1 and v2, we compute the component of u3 orthogonal to space W2
spanned by v1 and v2. From Theorem 6.3.5(b):
• As in the Step 2, the linear i d
independence of {u
d
f { 1,u2,…,un}
ensures that v3 ≠ 0
u3
v1
v2
W
57
Proof of 6 3 5
Proof of 6.3.5
• St
Step 4: To demonstrate a vector v
4 T d
t t
t
i th
l
4 is orthogonal to v1, v2, and v3, we compute the component of u4 orthogonal to the space W
orthogonal to the space W3 spanned by v
spanned by v1, vv2, and v3. • Continuing
Continuing in this way, we will obtain, after n
in this way we will obtain after n
steps, an orthogonal set of vectors {v1,v2,…,vn}. Since V is n‐dimensional and every orthogonal set y
g
is linearly independence, the set {v1,v2,…,vn} is an orthogonal basis for V. 58
Example (Gram Schmidt Process)
Example (Gram‐Schmidt Process)
• Consider the vector space R3 with the Euclidean inner product. Apply the Gram‐Schmidt process to transform the basis vectors u1 = (1, 1, 1), u
= (1 1 1) u2 = (0, 1, 1), u
= (0 1 1) u3 = (0, 0, 1) = (0 0 1)
into an orthogonal basis {v1, v2, v3}; then normalize the orthogonal basis vectors to obtain an orthonormal basis {q
q1, q
q2, q
q3}.
• Solution: – Step 1: Let v1 = u1.That is, v1 = u1 = (1, 1, 1)
– Step 2: Let v2 = u2 – projW1u2. That is, v 2 = u 2 − projw1 u 2 = u 2 −
< u 2 , v1 >
v1
2
v1
2
2 1 1
= (0,
(0 1,
1 1) − (1
(1, 11, 1) = (− , , )
3
3 3 3
59
Example (Gram Schmidt Process)
Example (Gram‐Schmidt Process) We have two vectors in W2 now!
– Step 3: Let v3 = u3 – projW2u3. That is, h
v 3 = u3 − projw 2 u3 = u3 −
< u3 , v1 >
v1
2
v1 −
< u3 , v 2 >
v2
2
v2
1
1/ 3 2 1 1
1 1
= (0, 1, 1) − (1, 1, 1) −
(− , , ) = (0, − , )
3
2/3 3 3 3
2 2
– Thus, v1 = (1, 1, 1), v2 = (‐2/3, 1/3, 1/3), v3 = (0, ‐1/2, 1/2) form an orthogonal basis for R3. The norms of these vectors are v1 = 3, v 2 =
6
1
, v3 =
3
2
so an orthonormal basis for R3 is q1 =
v1
v
1
1
1
2 1
=( ,
,
), q 2 = 2 = (−
, ,
v1
v2
3
3
3
6 6
q3 =
v3
1 1
= (0,
(0 , )
v3
2 2
1
),
6
60
Theorem 6 3 6
Theorem 6.3.6
• If W
If W is a finite‐dimensional inner product space, then
i fi it di
i
li
d t
th
– Every orthogonal set of nonzero vectors in W can be enlarged to an orthogonal basis for W
g
g
– Every orthonormal set in W cam be enlarged to an orthonormal basis for W. • Proof (b)
P f (b)
– Suppose that S={v1, v2, …, vs} is an orthonormal set of vectors in W. Theorem 4.5.5 tells us that we can enlarge S
vectors in W. Theorem 4.5.5 tells us that we can enlarge S
to some basis S’={v1,v2,…,vs, vs+1, …, vk} for W. If we apply the Gram‐Schmidt process to set S’, then the vectors v1,v2, …,v
, …,vs will not be affected since they are already will not be affected since they are already
orthonormal, and the resulting set will be an orthonormal basis for W. 61
Theorem 6 3 7
Theorem 6.3.7
• QR‐Decomposition
QR D
iti
– If A is an m×n matrix with linearly independent column vectors, then A
,
can be factored as A = QR
where Q is an m×n matrix with orthonormal column vectors, and R
t
d R is an n
i
×n invertible upper triangular matrix.
i
tibl
ti
l
ti
• Remark Remark
– In recent years the QR‐decomposition has assumed growing importance as the mathematical foundation for a wide variety of practical algorithms, including a widely used algorithm for computing eigenvalues of large matrices.
62
QR Decomposition
QR‐Decomposition
• Suppose that the column vectors of A are u1, u2, …, un and the orthonormal column vectors of Q are q1, q2, …, qn; thus A = [u1 | u2 | … | un] and Q = [q1 | q2
| … | qn]
• From Theorem 6.3.2b, the vectors u1, u2, …, un are expressible in terms of the vectors q1, q2, …, qn
…
…
…
…
63
QR Decomposition
QR‐Decomposition
• Recalling from Section 1.3 that the jth column vector of a matrix product is a linear combination of the column vectors of the first factor with coefficients coming from the jth column of the second factor. A
=
Q
R
64
QR Decomposition
QR‐Decomposition
• IIt is a property of the Gram‐Schmidt process that for i
f h G
S h id
h f
j≧ 2, the vector qj is orthogonal to u1, u2, …, uj‐1; thus all entries below the main diagonal of R are zero
all entries below the main diagonal of R
are zero
• The diagonal entries of R are nonzero, so R is invertible. 65
QR Decomposition of a 3×3 Matrix
QR‐Decomposition of a 3×3 Matrix
⎡1 0 0 ⎤
• Find the QR‐decomposition of A = ⎢⎢1 1 0 ⎥⎥
⎢⎣1 1 1 ⎥⎦
• Solution:
S l ti
– The column vectors A are – Applying the Gram‐Schmidt process with subsequent normalization to these column vectors yields the orthonormal vectors
⎡1/ 3 ⎤
⎡ −2 / 6 ⎤
⎡ 0 ⎤
⎢
⎥
⎢
⎥
⎢
⎥
q1 = ⎢1/ 3 ⎥ , q 2 = ⎢ 1/ 6 ⎥ , q3 = ⎢ −1/ 2 ⎥
⎢
⎥
⎢
⎥
⎢ 1/ 2 ⎥
1/
3
1/
6
⎢⎣
⎥⎦
⎢⎣
⎥⎦
⎣
⎦
Q
66
QR Decomposition of a 3×3 Matrix
QR‐Decomposition of a 3×3 Matrix
– The matrix R is
⎡ u1 , q1
⎢
R=⎢ 0
⎢⎣ 0
u 3 , q1 ⎤ ⎡3 / 3 2 / 3 1 / 3 ⎤
⎥
⎥ =⎢ 0
2 / 6 1/ 6 ⎥
u3 , q 2 ⎥ ⎢
0
1 / 2 ⎥⎦
u 3 , q 3 ⎥⎦ ⎢⎣ 0
u 2 , q1
u2 , q2
0
– Thus, the QR‐decomposition of A is ⎡1 0 0 ⎤ ⎡1/ 3 −2 / 6
⎢1 1 0 ⎥ = ⎢1/ 3 1/ 6
⎢
⎥ ⎢
⎢⎣1 1 1 ⎥⎦ ⎢1/ 3 1/ 6
⎢⎣
A
Q
⎤ ⎡3 / 3 2 / 3 1/ 3 ⎤
⎥⎢
⎥
−1/ 2 ⎥ ⎢ 0
2 / 6 1/ 6 ⎥
⎥⎢
⎥
1/ 3 ⎥⎦ ⎢⎣ 0
0
1/ 2 ⎥⎦
0
R
67