Matrix Factorizations:
Singular Value Decomposition
Presented by Nik Clark
MTH 421
Introduction
• In the exciting world of numerical analysis,
one may wonder “Why? Why do I study
matrices and their factorizations?”
• Such a simple answer, “Because matrices
and their decompositions can take you
anywhere!”
Matrix Decompositions
Some interesting ways to decompose a matrix
•
•
•
•
•
Choleski
Jordan
LU
Polar
Proper Orthogonal
•
•
•
•
QR
Schur
Singular Value
Spectral
(Eigendecomposition)
Singular Value Decomposition
We’d like to more formally introduce you to
Singular Value Decomposition (SVD) and
some of its applications.
What is SVD?
• SVD is a type of factorization for a
rectangular, real or complex, matrix.
• THEOREM:
All matrices Amxn have a singular value
decomposition.
Why should we care?
• All math is awesome.
• Also because SVD can be applied to several
situations:
–
–
–
–
Data Compression
Analyzing DNA Gene Expression Data
Solving Least Squares Problems
Information Retrieval
More Caring
– Image Processing
• De-blurring
– Seismology
– Digital Signal Processing
• Noise Reduction
– Data Hiding
• Cryptography
• Watermarks
– Researching Databases
Back to the SVD
• Each mxn matrix A decomposes into the product of
three matrices A=UΣVT
• U is orthogonal and mxm. Its columns span
col(A).
• V is also orthogonal, but is nxn. Its columns span
row(A)
•  is a diagonal matrix where the singular values
of A are along the main diagonal, and all other
values are zero.
More on SVD
• The m columns of matrix U are the
eigenvectors of AAT
• The n columns of matrix V are the
eigenvectors of ATA
• The entries of the main diagonal of matrix 
are the singular values of A, denoted by i
Another Way to Write
• An additional way to write A, apart from
A=UΣVT :
A = 1u1vT1 + 2u2vT2 + …+rurvTr ,
Where r = rank(A), and is defined to be the
number of linearly independent columns of A.
Importance
• Each term of the expansion is already in
order of importance.
• 1 ≥  2 ≥ … ≥  k ≥ 0
Rank k Approximation
• Ak=UkΣkVTk
• Where Ak is the first k terms of the SVD
factorization of A
• i.e. A = 1u1vT1 + 2u2vT2 + …+kukvTk
More on Rank
• A = 1u1vT1 + 2u2vT2 + …+kukvTk
• Where each term of the expansion is a
rank 1 matrix.
An Example: Code breaking
• Cipher- a method for encrypting a message.
• Cryptography – The art of creating a coded
message using a cipher.
• Cryptanalysis – The art of breaking a code
by finding its weakness.
• Cryptology – the study of the
aforementioned definitions.
How do we decode the code?
• Most commonly, a cryptogram is created by
substituting one letter for another.
• When decoding a code (in english) it is
easiest to first decode the vowels.
• Vowel pairs are less frequent than
consonant-vowel pairs.
• More frequently, vowels follow consonants,
vfc.
vcf
• When vowels follow consonants, there is a
mathematical proportion:
number of vowel pairs
number of vowels
<
number consonant-vowel pairs
number of consonants
The matrix
• We define matrix A to be a digram frequency of
the letter combinations of our text.
• aij is the number of times the ith letter is followed
by the jth letter.
• We define vector wi to equal 1 when the letter is a
vowel and 0 otherwise.
• Vector ci is defined to equal 1 when the letter is a
consonant and 0 otherwise.
Digram Matrix
Thus
• Our vectors w and c are orthogonal.
• wTAw is the number of vowel pairs.
• cTAw is the number of consonant vowel
pairs.
• We can now rewrite our equation:
wTAw
wTA(w+c)
<
cTAw
cTA(w+c)
Singular Vectors
• A ~ A1 = 1u1vT1
• f = k1 * u1
• f = k2 * v1
• Where f represents the frequency of the
words in the text.
Digram Matrix
Rank Two
• A ~ A2 = 1u1vT1 + 2u2vT2
• Rank two is the frequency of vowel pairs.
• Each vowel (except for u) corresponds to a
(+, -) vector pair.
• Each consonant corresponds to a (-, +)
vector pair.
• There are some exceptions
Neuter Letters
• Those letters that correspond to a (+, +)
vector pair or a (-, -) vector pair are called
neuter.
• These letter patterns correspond to any text
we consider.
An Encrypted Example
• The following matrix is the digram matrix
of an encrypted text (cryptogram) similar to
the sentence below.
• What does this mean??
• Gsviv rh ml zkkorw nzgsvnzgrxk ru gsviv rh
ml nzgsvnzgrxh gl zkkob.