Chapter 5 Inner Product Spaces 5-1 Inner Products and Norms Inner product <x,y>: V is a vector space over F, and <x,y> has the following characteristics: (a) <ax+bz,y>=a<x,y>+b<z,y>, (b) <x,y>=<y,x>*, (c) <x,x> is positive if x≠0. Eg. Let u=(u1,u2,u3) and v=(v1,v2,v3). Determine which of the following are inner products on R3. (i) <u,v>=u12v12+u22v22+u32v32, (ii) <u,v>=2u1v1+u2v2+3u3v3, (iii) <u,v>=u1v1-u2v2+u3v3. [2003交大電 信所] (Sol.) Only (ii) is the inner product. (i) does not fulfill (a), (iii) does not fulfill (c) if u1v1 +u3v3<u2v2. Eg. The following calculations fulfill the definition of the inner product: n (a) x=(a1,a2,…,an), y=(b1,b2,…,bn), <x,y>= ai bi , (b) A,B Mnxn(F), <A,B>=tr(B*A), i 1 (c) f, g V, <f,g>= f x g ( x)dx . a 0 In Matlab language, we can use the following instructions to obtain the inner product and the outer product of two vectors: >>A=[4,-1,3]; B=[-2,5,2]; C=dot(A,B); D=cross(A,B) A= 4 -1 3 B= -2 5 2 -14 18 C= -7 D= -17 Norm ||.||: A real-valued function on V and satisfies the following conditions for all x, y V and c F. (a) ||cx||=|c|.||x||, (b) ||x||≧0, and ||x||=0 if x=0, (c) ||x+y||≦||x||+||y||. Eg. Show that ||-x||=||x||. [1993 台大電研] (Proof) ||-x||=|-1|.||x||=||x|| Eg. The following calculations fulfill the definition of the norm: (a) V=Mmxn(F), ||A||= max |Aij|, (b) V=C[0,a >0] , ||f||= max f (t ) , (c) t[ 0 ,a ] i, j Euclidean norm of a vector: x V, ||x||= V=Fn, n 2 x a i i 1 1 2 x, x Theorem (a)| <x,y>|≦||x||.||y|| (Cauchy-Schwartz Inequality), (b) ||x+y||2 + ||x-y||2 =2||x||2 +2||y||2. (Remark) By definition of the Euclidean norm. In Matlab language, we can use the following instructions to obtain the Euclidean norm of a vector: >> x=[3 4 5]; c=norm(x) c= 7.0711 Orthogonal vectors x and y: x⊥y <x,y>=0. Orthonormal vectors x and y: ||x||=||y||=1 and <x,y>=0 if x≠y. Eg. Show that if {v1, v2, v3, …, vn} is an orthogonal set of non-zero vectors in an inner product space, then v1, v2, v3, …, and vn are linearly independent. [2015 中央電研固態組、生醫電子組] (Proof) Let y= be a linear combination of v1, v2, v3, …, and vn. Since {v1, v2, v3, …, vn} is an orthogonal set of non-zero vectors in an inner product space, we have < vi, vj>=0 if j≠i but < vi, vi>≠0 in case of j=i. Consider an inner product <y,vj>=< , vj >= < vi, vj >=ai< vi, vi> Set y=0 and then we have ai=0 because < vi, vi>≠0 for each i. It implies that v1, v2, v3, …, and vn are linearly independent. m Theorem V is an inner product space and S={x1,…, xm}. If y= a i xi , and then (a) aj=<y,xj>/<xj,xj> i 1 for all j if S is orthogonal, and (b) aj=<y,xj> for all j if S is orthonormal. m m y, x j i 1 i 1 xj, xj (Proof) (a) y , x j ai xi , x j ai xi , x j a j x j , x j a j Eg. (8,-7)=8(1,0)-7(0,1) where 8=<(8,-7),(1,0)> and -7=<(8,-7),(0,1)> Theorem A linear operator T: V→W, let β={x1,…,xm} and orthogonal set β’={y1,…,ym} are the bases T ( x j ), y i of V and W, respectively. Then the ij-entry Tij= . If β’ is an orthonormal basis, then yi , yi Tij= T ( x j ), y i . m m m k 1 k 1 k 1 (Proof) ∵ T ( x j ) Tkj y k , ∴ T ( x j ), y i Tkj y k , y i Tkj y k , y i Tij y i , y i Tij T ( x j ), y i yi , yi Eg. For a linear operator T(a1,a2)=(a1-a2,a1,2a1+a2), let β={(1,2),(2,3)} be a basis for R2, and β’={(1,0,0),(0,1,0),(0,0,1)} be an orthonormal basis for R3, then we have T(1,2)=(-1,1,4)=-1(1,0,0)+1(0,1,0)+4(0,0,1) and T(2,3)=(-1,2,7)=-1(1,0,0)+2(0,1,0)+7(0,0,1) T ' 1 1 4 1 2 7 , where T11=-1=<T(1,2),(1,0,0)>, T21=1=<T(1,2),(0,1,0)>, T31=4=<T(1,2),(0,0,1)>, T12=-1<T(2,3),(1,0,0>, T22=2=<T(2,3),(0,1,0)>, and T32=7=<T(2,3),(0,0,1)> 5-2 Gram-Schmidt Orthogonalization Process Gram-Schmidt Orthogonalization Process: Let S={y1,y2,y3,…,yn} be a linearly independent subset y 2 , x1 of V and S’={x1,x2,x3,…,xn} be an orthogonal subset of V. Set x1=y1, x2= y2 x1 , x3= x1 , x1 y3- k 1 y , x y3 , x1 y3 , x2 k j x j . Then Span(S)=Span(S’). x1 x2 , …, and xk=yk- x1 , x1 x2 , x2 j 1 x j , x j Eg. For V=R3, β={(1,1,0),(2,0,1),(2,2,1)}, find an orthogonal basis for V by the Gram-Schmidt orthogonalization process. [2007 台科大電研] (Sol.) Let x1 (1,1,0), x2 (2,0,1) x3 (2,2,1) (2,0,1), (1,1,0) (1,1,0) (2,0,1) 22 (1,1,0) (1,1,1) (1,1,0), (1,1,0) (2,2,1), (1,1,0) (2,2,1), (1,1,1) (1,1,0) (1,1,1) ( 13 , 13 , 23 ) . (1,1,0), (1,1,0) (1,1,1), (1,1,1) Then β’={x1,x2,x3} is an orthogonal basis. Eg. Let the vector space P2 have the inner product <p,q>= 1 1 p( x)q( x)dx . Apply the Gram-Schmidt process to transform the standard basis S={1,x,x2} into an orthonormal basis. [2005 北科大電腦通訊所] 1 y 2 , x1 x 1dx 1 =x-0=x, (Sol.) S={1,x,x2}={ y1, y2, y3}. Let x1= y1=1, x2= y2 x1 =x- 11 x1 , x1 1 1dx 1 1 1 x 2 1dx x 2 xdx y3 , x1 y3 , x2 1 1 1 1 x3= y3 x1 x2 =x2- 1 1 - 1 x =x2- -0= x2- . 3 3 x1 , x1 x2 , x2 1 1dx x xdx 1 1 1 45 2 1 1 3 ∴ Orthogonal basis: {1,x,x2- } Orthonormal basis: { , x, (x - )} 3 3 8 2 2 Sperp: S⊥={x V: <x,y>=0 for all y S} is an orthogonal set of S. Eg. For V=C3, S=Span{(1,0,i),(1,2,1)}, compute S⊥. (Sol.) Suppose S⊥=Span{(a,b,c)}, a, b, c , 1,0, i 0 a ci 1 i ⊥ a, b, c , 1,2,1 0 b 1 i c , S = Span{( i, 2 ,1)} 2 Eg. Show that (a) {x1,x2,…,xk} is an ordered basis for W (subspace of V) and {x1,x2,…,xk,xk+1,…,xn} is an ordered basis V, then {xk+1,…,xn} is an order basis of for W⊥. (b) W is a subspace of V, then dim(V)=dim(W)+dim(W⊥). [台大電研] n (Proof) (a) x V x x, xi xi i 1 If x W , then x, xi 0 for 1 i k . Therefore, x n x, x i k 1 (b) According to (a), dim( V )=n=k+(n-k)=dim( W )+dim( W ) i xi Spanx k 1 , , x n Eg. Show that (a) W1 W2 W1 W2 and (b) W1 W2 W1 W2 . [1990, 1999 台大電 研] (Proof) W1 W2 W1 W2 W1 W1 W2 W2 V W1 and W2 W1 W2 , W1 W2 W1 and W2 W1 W2 W1 W2 x W1 W2 , then y W1 and z W2 y , x z, x 0 cy dz W1 W2 , cy dz, x c y, x d z, x 0 x W1 W2 W1 W2 (W1 W2 ) , ∴ W1 W2 W1 W2 On the other hand, U 1 U 2 U 1 U 2 U 1 U 2 U 1 U 2 Set U 1 W1 , U 2 W2 W1 W2 W1 W2 5-3 Various Matrices Adjoint matrix: A* is the complex conjugate transpose of A. Note: A*≠adj(A) and (AB)*=B*A*. 1 i 2 1 i 3i Eg. For A= , then A*= . 4 3i 4 2 Eg. For T: C2→C2 by T(a1,a2)=(2ia1+3a2,a1-a2), if β is the standard ordered basis, find T*(a1,a2). [1998 台大電研] (Sol.) T 1,0 2i,1 2i (1,0) 1 (0,1) and T 0,1 3,1 3 (1,0) (1) (0,1) 2i 3 2i 1 T T* , ∴ T*(a1,a2)=( -2ia1+a2,3a1-a2). 1 1 3 1 Theorem For A Mm×n(F), x Fn, y Fm, then <Ax,y>=<x,A*y>. (Note: T x , y x, T * y for x, y V , T : V V ) Eg. 1 1 x u x y u x 1 2i u x u 2iv 2i 3 y , v 2ix 3 y , v xu yu 2ixv 3 yv y , 1 3 v y , u 3v Theorem (a) A Mm×n(F), Rank(A*A)=Rank(A). (b) A Mm×n(F), if Rank(A)=n, then A*A is invertible. (Proof) (a) A*Ax=0 Ax=0 1. " ": Ax 0 implies A* Ax 0 2. "": 0 A* Ax, x n Ax, A** x m Ax, Ax m , ∴ Ax=0 (b) A M mn F A* A M nn F Rank(A*A)=Rank(A)=n, ∴ A*A is invertible Orthogonal matrix A: AAt=AtA=I. Unitary matrix A:. AA*=A*A=I. 1 / 2 Eg. A= 1 / 2 1/ 2 t 1/ 2 1/ 2 t t , A = , AA =A A=I, ∴ A is orthogonal. 1/ 2 1 / 2 1 / 2 i / 2 Eg. A= i / 2 i / 2 i / 2 , A*= i/ 2 i/ 2 i / 2 , AA*=A*A=I, ∴ A is unitary. i / 2 Normal Matrix A: AA*=A*A. i i i i Eg. A= , A*= , AA*=A*A, ∴ A is normal. i i i i Eg. T is a normal operator on V. Show that (a) ||T(x)||= ||T*(x)||, (b) T(x)=λx T * x x , (c) λ1 and λ2 are distinct eigenvalues of T with corresponding eigenvector x1 and x2. Then x1 and x2 are orthogonal. [2012 台大電研] (Proof) (a) T x T x , T x T *T x , x TT * x , x T * x , T * x T * x 2 2 (b) Suppose T x x for some x V . Let U T I , 0 U x U * x (T * I )x , ∴ T * x x . (c) Let T x1 1 x1 , T x2 2 x2 , 1 x1 , x2 1 x1 , x2 T x1 , x2 x1 , T * x2 x1 , 2 x2 2 x1 , x2 1 2 , x1 , x2 0 . Self-adjoint (Hermitian) matrix A: A=A*. 2 5i 1 Eg. is a self-adjoint matrix. 2 5i 3 (Note: A self-adjoint matrix is also a normal matrix, but a normal matrix may not be self-adjoint) Theorem If A is a self-adjoint matrix, all eigenvalues of A are real. [2000 台大電研] (Proof) x Ax A* x x for some x≠0, λ is real. Gramian matrix A: B Mm×n(F) such that A=BBt then A is called the Gramian matrix. A is symmetric. Theorem A is a Gramian matrix all eigenvalues of A are nonnegative. (Proof) (1) At=(BBt)t=BBt=A, ∴ A is symmetric. (2) For some x 0, and Ax=λx, then Ax, x x, x x, x BB t x, x B t x, B t x 0, x, x 0, 0 Eg. Which of the following matrices is Hermitian? Which is normal? i 2 i 1 B 2 A 2 i 1 1 2 1 i 1 3 1 i i 0 2 C i 0 2 i D 1 i 1 3 [交大電子所] i i 1 2 i 3 1 0 2 (Sol.) A=A* and D=D*, ∴ A and D are both Hermitian matrices, 1 i 1 i CC*=C*C, BB*= , B*B= , ∴ C is normal but B is not normal. i 1 i 1 5-4 Special Characteristics of Matrices Ax, x >0 for all x≠0, then A is positive definite. Positive definite: If Eg. If A is a positive-definite matrix, then all the eigenvalues of A are positive. (Proof) For some x≠0 and Ax=λx, positive definite: Ax, x x, x x, x 0 0 . Eg. Let A be a complex normal (or real symmetric) n×n matrix with eigenvalues: λ1, λ2, …, λn, n n i 1 i 1 2 n and then show that (a) tr(A)= i and tr(A*A)= i , (b) det(A)= i . [1998 台大電研] i 1 n (Proof) Let A=QDQ-1 and D be diagonal, tr(A)=tr(QDQ-1)=tr(DQQ-1)=tr(D)= i i 1 n det A det QDQ 1 det Q det D det Q 1 det Q det D det Q 1 det D i . i 1 2 1 Eg. Let λi be the eigenvalues of A= - 1 0 1 -1 0 3 4 2 , 1≦i≦4, (a) 4 1 2 2 21 4 i i 1 ? (b) 4 i ? i 1 ( c ) I s A p o s i t i v e definite? [台大機研] 4 (So1.) ∵ A is a real symmetric matrix, ∴ (a) i tr(A) = 2+3+1+1=7, (b) i 1 4 =det(A) = 10, i i 1 (c) Method 1: λ<0 A is not positive definite. 2 1 -1 2 1 Method 2: det([2])=2>0 and det( )=5>0, but det( 1 3 4 )=-38<0 Not positive definite. 1 3 - 1 4 1 A is unitarily equivalent to B: P fulfills B=P*AP and P-1=P*. Theorem (a) A is complex normal A is unitary equivalent to a diagonal matrix. (b) A is real symmetric A is orthogonally equivalent to a real diagonal matrix. (Proof) AA* PDP * PDP * * PDP * PD * P * PDD * P * A PDP * , A* A PDP * PDP * PD * DP * . DD * D * D, A* A AA* * Projection: V=W1⊕W2, x1 W1, and x2 W2, x=x1+x2. If T(x)=x1, then T is projection on W1. That is, R(T)=W1={x: T(x)=x} and N(T)= W2. (Note: T2=T if T is a projection.) Orthogonal Projection: If R(T)⊥=N(T) and N(T)⊥= R(T) for T: V→V be a projection. Theorem T is a linear operator on V. Then T is an orthogonal projection. T2=T=T*. Eg. Let T: R2→R2 be a linear operator. For any z R2, T(z)=p, where p is the projection of z on the line x=y. Find [T]. [交大電信所] 1 2 } and the standard basis of R2 is { 1 , 0 }. (Sol.) The orthonormal basis of line x=y is { 0 1 1 2 1 1 2 >= 1 , so the orthogonal projection of 1 onto the line x=y is Inner product: < , 0 2 0 1 2 1 1 1 2 = 2 = 1 1 1 2 2 2 2 Inner 1 2 1 1 0 + 2 0 1 1 0 2 >= 1 , so the orthogonal projection of 0 onto the line x=y is product: < , 1 2 1 1 2 1 1 1 1 2 = 2 = 1 1 + 1 0 , ∴ [T]= 2 2 , and we have [T]2=[T]=[T]* 1 1 1 1 2 0 2 1 2 2 2 2 1 0 0 Eg. Find the orthogonal projection of the vector v= 0 onto the subspace S=Span( 1 , 1 ). 2 1 1 [交大電子所] 0 0 0 0 (Sol.) Transform the orthogonal basis { 1 , 1 } into the orthonormal one: { 1 , 1 }. 2 2 1 1 1 1 2 2 1 < 0 2 0 1 0 1 , >=-√2, so the orthogonal projection of the vector v= 0 onto 1 is 2 1 2 1 2 0 0 -√2 1 = 1 . 2 1 1 2 1 0 1 0 1 < 0 , >=-√2, so the orthogonal projection of the vector v= 0 onto 1 is 2 2 1 2 1 2 0 0 -√2 1 = 1 . 2 1 1 2 0 0 1 0 0 1 1 1 1 0 0 1 1 ∵ 0 -(-√2 )-(-√2 )= 0 - 1 - 1 = 0 , ∴ v= 0 = 0 + 1 + 1 2 2 1 1 2 1 1 0 2 2 0 1 1 2 2 1 0 0 and { 0 }⊥S=Span( 1 , 1 ) 0 1 1 Rotation: orthogonal basis β={x1,x2} for W, and θ is real and fulfills T(x1)=x1cosθ+x2sinθ, T(x2)=-x1sinθ+x2cosθ, and T(y)=y, y W⊥, where W is a 2-dimensional subspace. And then T is a ⊥ ⊥ rotation of W about W , where W is the axis of rotation. Reflection: T(x)=-x, x W and T(y)=y, y W⊥. And then T is a reflection of W about W⊥, where W is a 1-dimensional subspace. Eg. T: R2→R2 by T(a,b) = (-a,b). Let W=Span({e1}) and W⊥=Span({e2}). It is reflection of R2 about W⊥ (the y-axis). Theorem T is an orthogonal operator on a 2-dimensional real inner product space. If det(T)=1, then T is a rotation. If det(T)=-1, then T is a reflection. cos sin (Proof) By definition, T is a rotation, ∴ det(T) = 1. sin cos Choose z1 , z 2 ,... be an orthogonal basis for V and z1 W, T 1 0 0 0 1 det(T) = -1. 0 0 0 1 Theorem The composition of a reflection and a rotation is a reflection. (Proof) det(T1T2)=det(T1)det(T2)= -1 normal on V over F C Spectral Theorem If T is , and λ1, …, λk are distinct eigenvalues of self - adjoint on V over F R T. Let Wi be the eigenspace of T corresponding to λi, and Ti be an orthogonal projection on Wi. (Ti(x)=xi Wi). Then (a) V=W1⊕W2⊕W3…⊕Wk, (b) Wi⊥= W j (direct sum), (c) TiTj=δijTi, (d) ji I=T1+T2+…+Tk, (e) T=λ1T1+λ2T2+…+λkTk. 0, i j (Proof) (c) Let x 0 and T j ( x) x j , then Ti Tj ( x) Ti ( x j ) i j Ti ( x) TiT j i j Ti xi , i j (d) x Ix x1 ... xk T1 ( x) T2 ( x) ... Tk ( x) (T1 T2 ... Tk )( x) , ∴ I T1 T2 ... Tk . (e) For x V , then x x1 ... xk , where xi Wi , T ( x) T ( x1 ) ... T ( xk ) 1 x1 ... k xk 1T1 ( x) ... k Tk ( x) (1T1 ... k Tk )( x) , ∴ T 1T1 ... k Tk Theorem (a) F=C, T is a unitary operator T* is normal and |λ|=1 for all eigenvalues. (b) T=-T* each λ is pure imaginary. (c) T is a projection each λ is either 0 or 1. 5-5 Bilinear Forms Bilinear form, H: If (a) H(ax1+bx2,y)=aH(x1,y)+bH(x2,y) and (b) H(x,ay1+by2)=aH(x,y1)+bH(x,y2) for x1, y1, x2, y2, x, y V, a, b F. Theorem For each H B(Fn), ! A Mn×n(F) fulfills H(x,y)=xtAy for all x, y Fn. a b a b Eg. Define H : R 2 R 2 R by H ( 1 , 1 ) 2a1b1 3a1b2 4a 2 b1 a 2 b2 for all 1 , 1 R 2 , a2 b2 a2 b2 a1 b1 2 3 t then let x , y , A H ( x, y ) x Ay [a1 a 2 ] a b 4 1 2 2 2 3 4 1 Symmetric bilinear form: H(x,y)=H(y,x) for all x, y V. Quadratic form: K: V→F if H B(V) fulfills K(x)=H(x,x)=xtAx. Eg. Let f(t1,t2,t3)= 2t1 t 2 2 f (t1 , t 2 , t 3 ) t1 t 2 2 t3 0 2 3 t1 6t1t 2 4t 2 t 3 K ( t 2 ) , and then A= 3 1 2 t 3 0 2 0 0 t 1 2 3 3 1 2 t is a quadratic form. 2 0 2 0 t 3 b1 b . 2 Eg. There are two bases (x1,y1) and (x2,y2) in R2. If the equations of the same ellipse represented by two distinct bases are described as follows, respectively: 2x12-4x1y1+5y12-36=0 and x22+6y22-36=0. Please find the transformation matrix between these two bases. [2004 台大電研] x 2 (Sol.) Let K 1 2 x1 2 5 y1 2 4 x1 y1 , A 2 y1 2 a1 a 2 0 1 a , a 0, 4 a2 1 a2 a 1 2 b 4 2 b 1 1 A 1I 1 a 2 2 2 2 , det 2 5 1 6, 1, 6 5 b 1 2 6 A 2 I 1 1 0 1 b , b 0 2 b2 2 1 b2 b2 Q 2 5 1 5 x ∴ 1 y 1 1 5 fulfills Q 1 AQ Q t AQ 1 0 for 1 2, 1 1 . 0 6 2 5 1 5 2 5 2 5 1 5 1 5 x 2 K x2 x 2 6y 2 2 x1 2 4 x1 y1 5 y1 2 36 0 x 2 2 6 y 2 2 36 0 2 2 y 2 y 2 2 5 Eg. [t1,t2,t3]t R3, 3t12+3t22+3t32-2t1t3+2√2 (t1+t3)+1=0. Find a basis β for R3 such that the above equation is simplified. t1 3 0 1 2 2 2 (Sol.) Let K t 2 3t1 3t 2 3t 3 2t1t 3 , then A 0 3 0 . It is found that an orthogonal t 3 1 0 3 0 matrix Q 1 0 1 0 3 0 0 1 1 2 1 1 t 0 fulfills Q AQ 0 4 0 for 1 , 0 , 0 . 2 2 1 0 0 0 2 1 1 2 1 2 0 1 2 t1 0 ∴ t 2 1 t 3 0 1 2 0 1 2 1 2 s1 0 s 2 K ( x) 3s12 4s2 2 2s3 2 1 s3 2 3t1 3t 2 3t 3 2t1t3 2 2 (t1 t 3 ) 1 0 3s1 4s2 2s3 4s3 1 0 . 2 2 2 2 2 2