Linear Algebra Review
By Tim K. Marks
UCSD
Borrows heavily from:
Jana Kosecka
http://cs.gmu.edu/~kosecka/cs682.html
Virginia de Sa (UCSD)
Cogsci 108F Linear Algebra review
Vectors
Quic kTime™ and a
TIFF (LZW) decompress or
are needed to see this pic ture.
Qu ic kTi me™ a nd a
TIFF (LZW)d ec omp res so r
are n ee de d to s ee th is pi ctu re .
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
The length of x, a.k.a. the norm or 2-norm of x, is
x  x12  x 22 
 x n2
e.g.,

x  32  22  52  38
Good Review Materials
http://www.imageprocessingbook.com/DIP2E/dip2e_downloads/review_material_downloads.htm
(Gonzales & Woods review materials)
Chapt. 1: Linear Algebra Review
Chapt. 2: Probability, Random Variables, Random Vectors
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompress or
are needed to s ee this pic ture.
Online vector addition demo:
http://www.pa.uky.edu/~phy211/VecArith/index.html
Vector Addition
v
u+v
u
Vector Subtraction
u
u-v
v
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Example (on board)
Inner product (dot product) of
two vectors
6 
 
a  2 

3




4
 
b  1

5

a  b  aT b
4
 
 6 2 3 1

5

 6 4  21 (3)  5

 11
Inner (dot) Product
v
The inner product is a SCALAR.

u
uT v  0  u  v

QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.

Transpose of a Matrix
Transpose:
Cmn  AT nm
(A  B)T  AT  BT
c ij  a ji
(AB)T  BT AT
Examples:
6 2 6 1

   

1 5 2 5
T


6 2

 6 1 3

1 5  
2
5
8




3
8


T
If AT  A , we say A is symmetric.

Example of symmetricmatrix
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.


Matrix Product
Product:
Cn p  Anm Bm p
A and B must have
compatible dimensions
m
c ij   aikbkj
In Matlab:
>> A*B
k1
Examples:
2 5 6 2 17 29

 
 

3
1
1
5
19
11

 
 

6 2 2 5 18 32

 
 

1
5
3
1
17
10

 
 

Matrix Multiplication is not commutative:
Ann Bnn Bnn Ann
Matrix Sum
Sum:
Cnm  Anm  Bnm
c ij  aij  bij

Example:
A and B must have the same
dimensions

2 5 6 2 8 7

 
 

3 1 1 5 4 6
Determinant of a Matrix
Determinant:
A must be square
a11 a12  a11 a12
det
 a11a22  a21a12

a21 a22 a21 a22
Example:
a11 a12

deta
a22
21

a31 a32
2 5
det
 2 15  13
3 1
a13 
a22

a23 a11
a32
a33

a23
a33
 a12
a21 a23
a31 a33
 a13
a21 a22
a31 a32
Determinant in Matlab
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Inverse of a Matrix
If A is a square matrix, the inverse of A, called A-1,
satisfies
AA-1 = I
and A-1A = I,
Where I, the identity matrix, is a diagonal matrix
with all 1’s on the diagonal.
1 0
I2  

0
1


1 0 0


I3  0 1 0

0 0 1

Inverse of a 2D Matrix
Quick Time™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Example:
1
6 2


1
5
  
1
6 2
1 5 2

  

1
5
1
6

 28 

6 2 1 5 2 6 2 1 28 0  1 0
 
 
 
 
 

1
5
1
6
1
5
0
28
0
1
28
28



 


 

Inverses in Matlab
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Trace of a matrix
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Example (on board)
Matrix Transformation: Scale
A square, diagonal matrix scales each dimension by
the corresponding diagonal element.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Example:
2 0 0 6  12

    
0
.5
0

 8   4 

0 0 3
 
10
 
30


Linear Independence
• A set of vectors is linearly dependent if one of
the vectors can be expressed as a linear
combination of the other vectors.
1 0 2
Example:
     
0 , 1 , 1

0
 
0
 
0

• A set of vectors is linearly independent if none of
the vectors can be expressed as a linear

combination
of the other vectors.
Example:
1 0 2
     
0 , 1 , 1

0
 
0
 
3

Rank of a matrix
• The rank of a matrix is the number of linearly
independent columns of the matrix.
Examples:
1 0 2


has rank 2
0 1 1



0 0 0


1 0 2


0
1
0



0 0 1

has rank 3
• Note: the rank of a matrix is also the number of
linearlyindependent rows of the matrix.
Singular Matrix
All of the following conditions are equivalent. We
say a square (n  n) matrix is singular if any one
of these conditions (and hence all of them) is
satisfied.
–
–
–
–
–
The columns are linearly dependent
The rows are linearly dependent
The determinant = 0
The matrix is not invertible
The matrix is not full rank (i.e., rank < n)
Linear Spaces
A linear space is the set of all vectors that can be
expressed as a linear combination of a set of basis
vectors. We say this space is the span of the basis
vectors.
– Example: R3, 3-dimensional Euclidean space, is
spanned by each of the following two bases:
1 0 0
     
0 , 1 , 0

0
 
0
 
1


1 0 0
     
0 , 1 , 0

0
 
2
 
1


Linear Subspaces
A linear subspace is the space spanned by a subset
of the vectors in a linear space.
– The space spanned by the following vectors is a
two-dimensional subspace of R3.
1 0
   
What does it look like?
0 , 1

0
 
0

– The space spanned by the following vectors is a
two-dimensional subspace of R3.
1 0

   
What does it look like?
1
,
0
   

0
 
1

Orthogonal and Orthonormal
Bases
n linearly independent real vectors
span Rn, n-dimensional Euclidean space
• They form a basis for the space.
– An orthogonal basis, a1, …, an satisfies
ai  aj = 0 if i ≠ j
– An orthonormal basis, a1, …, an satisfies
ai  aj = 0 if i ≠ j
ai  aj = 1 if i = j
– Examples.
Orthonormal Matrices
A square matrix is orthonormal (also called
unitary) if its columns are orthonormal vectors.
– A matrix A is orthonormal iff AAT = I.
• If A is orthonormal, A-1 = AT
AAT = ATA = I.
– A rotation matrix is an orthonormal matrix with
determinant = 1.
• It is also possible for an orthonormal matrix to have
determinant = –1. This is a rotation plus a flip (reflection).
Matrix Transformation:
Rotation by an Angle 
2D rotation matrix:
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
http://www.math.ubc.ca/~cass/courses/m309-8a/java/m309gfx/eigen.html
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Some Properties of
Eigenvalues and Eigenvectors
– If 1, …, n are distinct eigenvalues of a matrix, then
the corresponding eigenvectors e1, …, en are linearly
independent.
– If e1 is an eigenvector of a matrix with corresponding
eigenvalue 1, then any nonzero scalar multiple of e1 is
also an eigenvector with eigenvalue 1.
– A real, symmetric square matrix has real eigenvalues,
with orthogonal eigenvectors (can be chosen to be
orthonormal).
SVD: Singular Value Decomposition
Any matrix A (m  n) can be written as the product of three
matrices:
A = UDV T
where
– U is an m  m orthonormal matrix
– D is an m  n diagonal matrix. Its diagonal elements, 1, 2, …, are
called the singular values of A, and satisfy 1 ≥ 2 ≥ … ≥ 0.
– V is an n  n orthonormal matrix
Example: if m > n
A








  
 
   |
   u1
 
   |
 
 


U

|
u2
|
u3
|
|


 

| 
um 

| 
 

D
VT
1 0 0 


T
0

0

v
2
1


0 0  n 


T
0
0
0

v
n



0 0 0 






SVD in Matlab
>> a = [1 2 3; 2 7 4; -3 0 6; 2 4 9; 5 -8 0]
a=
1 2 3
2 7 4
-3 0 6
2 4 9
5 -8 0
>> [u,d,v] = svd(a)
u=
-0.24538 0.11781 -0.11291 -0.47421
-0.53253 -0.11684 -0.52806 -0.45036
-0.30668 0.24939 0.79767 -0.38766
-0.64223 0.44212 -0.057905 0.61667
0.38691 0.84546 -0.26226 -0.20428
d=
-0.82963
0.4702
0.23915
-0.091874
0.15809
14.412
0
0
0
8.8258
0
0
0
5.6928
0
0
0
0
0
0
v=
0.01802
-0.68573
-0.72763
0.48126
-0.63195
0.60748
-0.87639
-0.36112
0.31863
Some Properties of SVD
– The rank of matrix A is equal to the number of nonzero
singular values i
– A square (n  n) matrix A is singular
if and only if
at least one of its singular values 1, …, n is zero.
Geometric Interpretation of SVD
If A is a square (n  n) matrix,
A
U

D
VT
   
  1 0 0  v1T

 
 

un  0  2 0 
    u1
T

 
  
 

 
0 0  n 

 v n
– U is a unitary matrix: rotation (possibly plus flip)
– D is a scale matrix
– V (and thus V T) is a unitary matrix





Punchline: An arbitrary n-D linear transformation is
equivalent to a rotation (plus perhaps a flip), followed by a
scale transformation, followed by a rotation
T
Advanced: y = Ax = UDV x
– V T expresses x in terms of the basis V.
– D rescales each coordinate (each dimension)
– The new coordinates are the coordinates of y in terms of the basis U