CS 485 / 685
Computer Vision
Instructor: Mircea Nicolescu
Lecture 8
Orthogonal/Orthonormal Vectors
• A set of vectors x1, x2, . . . , xn is orthogonal if
k
• A set of vectors x1, x2, . . . , xn is orthonormal if
2
Linear Combinations of Vectors
• A vector v is a linear combination of the vectors v1,
..., vk:
where c1, ..., ck are scalars
• Example: any vector in R3 can be expressed as a
linear combinations of the unit vectors i = (1, 0, 0),
j = (0, 1, 0), and k = (0, 0, 1)
3
Space Spanning
• A set of vectors S = (v1, v2, . . . , vk ) span some
space W if every vector in W can be written as a
linear combination of the vectors in S
w
• Example: the vectors i, j, and k span R3
4
Linear Dependence
• A set of vectors v1, ..., vk are linearly dependent if
at least one of them is a linear combination of the
others.
(where vj does not appear on the right side)
5
Linear Independence
• A set of vectors v1, ..., vk is linearly independent if
implies
Example:
6
Vector Basis
• A set of vectors (v1, ..., vk) is said to be a
basis for a vector space W if
(1) (v1, ..., vk) are linearly independent
(2) (v1, ..., vk) span W
• Standard bases:
R2
R3
Rn
7
Orthogonal Basis
• A basis with orthogonal basis vectors:
k
• Any set of basis vectors (x1, x2, . . . , xn) can be
transformed to an orthogonal basis (o1, o2, . . . , on) using
the Gram-Schmidt orthogonalization.
8
Orthonormal Basis
• A basis with orthonormal basis vectors:
9
Uniqueness of Vector Expansion
• Suppose v1, v2, . . . , vn represents a basis in W,
then any v є W has a unique vector expansion in
this basis:
• The vector expansion provides a meaning for
writing a vector as a “column of numbers”.
Note: to interpret v, we need to know
what basis was used for the expansion!
10
Computing Vector Expansion
(1) Assuming the basis vectors are orthogonal, to
compute xi, take the inner product of vi and v:
vi .v  vi .( x1v1  x2v2  ...  xn vn ) 
x1 (vi .v1 )  ...  xi (vi .vi )  ...  xn (vi .vn )  xi (vi .vi )
(2) The coefficients of the expansion can be computed as
follows:
11
Matrix Operations
• Matrix addition/subtraction
− Matrices must be of same size.
• Matrix multiplication
mxn
qxp
mxp
n
Condition: n = q
12
Identity Matrix
13
Matrix Transpose
14
Symmetric Matrices
Example:
15
Determinants
2x2
3x3
mxm
16
Determinants
diagonal matrix:
17
Matrix Inverse
• The inverse A-1 of a (square) matrix A has the
property:
AA-1=A-1A=I
• A-1 exists only if
• Terminology
− Singular matrix: A-1 does not exist
− Ill-conditioned matrix: A is close to being singular
18
Matrix Inverse
• Properties of the inverse:
19
Pseudo-Inverse
• The pseudo-inverse A+ of a matrix A (could be
non-square, e.g., m x n) is given by:
• It can be shown that:
20
Matrix Trace
Properties:
21
Rank of Matrix
• Equal to the dimension of the largest square submatrix of A that has a non-zero determinant.
Example:
has rank 3
22
Rank of Matrix
• Alternative definition: the maximum number of
linearly independent columns (or rows) of A.
Example:
Therefore,
rank is not 4 !
23
Rank and Singular Matrices
24
Orthogonal Matrices
Notation:
A is orthogonal if:
Example:
25
Orthonormal Matrices
A is orthonormal if:
Note that if A is orthonormal, it easy to find its inverse:
Property:
26
Eigenvalues and Eigenvectors
• The vector v is an eigenvector of (square) matrix
A and λ is an eigenvalue of A if:
(assume non-zero v)
• Interpretation: the linear transformation implied
by A cannot change the direction of the
eigenvectors v, only their magnitude.
27
Computing λ and v
• To find the eigenvalues λ of a matrix A, find the
roots of the characteristic polynomial:
Example:
28
Properties
• Eigenvalues and eigenvectors are only defined for
square matrices (i.e., m = n)
• Eigenvectors are not unique (e.g., if v is an
eigenvector, so is kv)
• Suppose λ1, λ2, ..., λn are the eigenvalues of A,
then:
29
Properties
xTAx > 0 for every
30
Matrix Diagonalization
• Given A, find P such that P-1AP is diagonal (i.e.,
P diagonalizes A)
• Take P = [v1 v2 . . . vn], where v1,v2 ,. . . vn are the
eigenvectors of A:
31
Matrix Diagonalization
Example:
32
Are All n × n Matrices Diagonalizable?
• Only if P-1 exists (i.e., A must have n linearly
independent eigenvectors, that is, rank(A)=n)
• If A has n distinct eigenvalues λ1, λ2, ..., λn , then
the corresponding eigenvectors v1,v2 ,. . . vn form
a basis:
(1) linearly independent
(2) span Rn
33
Diagonalization  Decomposition
• Let us assume that A is diagonalizable, then:
34
Decomposition: Symmetric Matrices
• The eigenvalues of symmetric matrices are all
real.
• The eigenvectors corresponding to distinct
eigenvalues are orthogonal.
P-1=PT
A=PDPT=
35
Singular Value Decomposition (SVD)
• Any real m x n matrix A can be decomposed
uniquely:
• U is m x n and column orthonormal (UTU=I)
• D is n x n and diagonal
− σi are called singular values of A
− It is assumed that σ1 ≥ σ2 ≥ … ≥ σn ≥ 0
• V is n x n and orthonormal (VVT=VTV=I)
36
SVD
• If m=n, then:
• U is n x n and orthonormal (UTU=UUT=I)
• D is n x n and diagonal
• V is n x n and orthonormal (VVT=VTV=I)
37
SVD
• The columns of U are eigenvectors of AAT
• The columns of V are eigenvectors of ATA
• If λi is an eigenvalue of ATA (or AAT), then λi =σi2
38
SVD – Example
U = (u1 u2 . . . un)
V = (v1 v2 . . . vn)
D
39
SVD – Another Example
The eigenvalues of AAT , ATA are:
λ1
λ2
λ3
The eigenvectors of AAT , ATA are:
40
SVD Properties
• A square (n × n) matrix A is singular iff at least
one of its singular values σ1, …, σn is zero.
• The rank of matrix A is equal to the number of
nonzero singular values σi
41
Matrix “Condition”
• SVD gives a way of determining how singular A is.
• The condition of A measures the degree of
singularity of A:
cond (A)=
(ratio of largest singular value to its smallest singular value)
• Matrices with a large condition number are called
ill conditioned.
42
Computing A-1 Using SVD
• If A is a n x n nonsingular matrix, then its inverse
can be computed as follows:
easy to compute!
(UTU=UUT=I so UT=U-1, and VTV=VVT=I so VT=V-1)
43
Computing A-1 Using SVD
• If A is singular (or ill-conditioned), we can use SVD
to approximate its inverse as follows:
?
where
(t is a small threshold)
44