Mathematics for Machine
Learning
Amit Chattopadhyay
IIIT-Bangalore
Module 2: Matrix Diagonalization
1/33
3. Eigen Decomposition (SVD)
References
1. Introduction to Linear Algebra, Gilbert Strang, 4th Edition.
2. Linear Algebra and Its Applications, David C. Lay, 4th Edition.
Eigenvalues and Eigenvectors
Goal: To diagonalize a matrix. First, start with a square matrix.
Eigenvectors: Certain special vectors x remain in the same direction (or
reversed) as Ax i.e. Ax = l x.
Eigenvalues: l tells if the special vector is stretched or shrunk or
reversed or unchanged.
Definition:
Let A be an n ⇥ n matrix over a field R. A non-zero vector x 2 Rn is an
eigenvector of A if there exists a scalar l such that:
Ax = l x =) (A
l I)x = 0.
Note: For a non-zero solution of the above homogeneous system:
det(A l I) = 0.
3/33
Eigenvalues and Eigenvectors
Computation:
1. First, compute eigenvalues by solving characteristic equation:
fA (l ) := det(A l I) = 0.
(The eigenvalues lie in a suitable algebraic extension C.)
2. Then corresponding to each eigenvalue l , the eigenvectors are the
special solutions of (A l I)x = 0. These special solutions (including null
vector) form the nullspace N(A l I): characteristic subspace.
Example:

1 3
Compute the eigenvalues and eigenvectors of A =
.
4 5
4/33
Characteristic Equation
For an n ⇥ n matrix A = (aij ), fA (l ) =
=c0 l n + c1 l n
1 +...+c
n
a11 l
a21
...
an1
a12
a22 l
...
an2
...
a1n
...
a2n
...
...
... ann l
where
c0 = ( 1)n ,
c1 = ( 1)n 1 (a11 + a22 + . . . + ann ),
cr = ( 1)n r [sum of all the principal minors of order r ],
cn = det(A).
Again if l1 , . . . , ln be the roots of polynomial of fA (l ) then
 li =
C1
C0 ,
Âi<j li lj =
Thus, Â li = trace(A),
C2
C0 ,
··· ,
l1 l2 . . . ln = ( 1)n CCn0 .
l1 l2 . . . ln = det(A) etc.
5/33
Eigenvalues and Eigenvectors
Properties:
1. To each eigenvalue of A, 9 at least one eigenvector.
2. To each eigenvector of A, 9 a unique eigenvalue.
3. If A be a singular matrix, then 0 is an eigenvalue of A and conversely.
4. If A be non-singular and l be one eigenvalue of A, then l
eigenvalue of A 1 .
1
is one
5. If A and B be two matrices of same order, then AB and BA have the
same eigenvalues.
6. If l is eigenvalue of A, then (i) l + k is an eigenvalue of A + kI (k: a
scalar), (ii) l k is an eigenvalue of kA, (iii) l m is an eigenvalue of Am
(m: a positive integer).
6/33
Eigenvalues and Eigenvectors
Properties:
7. The eigenvalues of a diagonal matrix are its diagonal elements.
8. (i) Product of n eigenvalues of A is equals to the determinant.
(ii) Sum of n eigenvalues equals the sum of the n diagonal entries.
9. The eigenvectors of an n ⇥ n matrix A (over R) corresponding to an
eigenvalue l together with the zero vector, form a subspace of Rn
(characteristic subspace).
10. If x1 , x2 , . . . , xr (r  n) be r eigenvectors of an n ⇥ n matrix A
corresponding to r distinct eigenvalues l1 , l2 , . . . , lr respectively,
then x1 , x2 , . . . , xr are linearly independent.
11. (Cayley Hamilton Theorem) Every square matrix satisfies its own
characteristic polynomial, i.e. fA (A) = 0.
7/33
Similar Matrices
Definition:
An n ⇥ n matrix A is said to be similar to an n ⇥ n matrix B if there exists
a non-singular n ⇥ n matrix P such that B = P 1 AP.
Properties:
1. If A is similar to B, then B is similar to A.
2. Similar matrices have same: (i) eigenvalues, (ii) determinant and trace,
(iii) rank, (iv) number of independent eigenvectors.
Example
(Converse is not true)

1 0
1 2
A=
and B =
have same eigenvalues, but they are not
0 1
0 1
similar. (Hints. Since A is an identity matrix, it is similar only to itself.)
8/33
Diagonalization of a Matrix
Definition
An n ⇥ n matrix A is said to be diagonalizable if A is similar to an n ⇥ n
diagonal matrix.
Properties:
1. If A is similar to a diagonal matrix D = diag(l1 , l2 , . . . , ln ) then
l1 , l2 , . . . , ln are the eigenvalues of A.
9/33
Eigen Decomposition (Spectral Decomposition)
2. If an n ⇥ n matrix A has n independent eigenvectors then A is
diagonalizable.
Proof: Let v1 , v2 , . . . , vn be n linearly independent eigenvectors of A
corresponding to eigenvalues l1 , l2 , . . . , ln , respectively. Then
[Av1 , Av2 , . . . , Avn ] = [l1 v1 , l2 v2 , . . . , ln vn ]
=) AP = PD
=) D = P
1
AP or A = PDP
1
where
P = [v1 , v2 , . . . , vn ] and D = diag(l1 , l2 , . . . , ln )
.
10/33
Diagonalization of a Matrix
Note:
(i) For diagonalizability, matrix A should possess a full set of linearly
independent eigenvectors.
(ii) Using Prop-10, if A has all its eigenvalues distinct, then it has n
linearly independent eigenvectors and so A is diagonalizable.
Question: What if A has one eigenvalue multiple times?
11/33
Geometric and Algebraic Multiplicities
Method:
For each l consider two types of multiplicities:
I. Geometric Multiplicity (GM)=dimension of N(A
subspace of l )
l I) (Characteristic
II. Algebraic Multiplicity (AM)=count of repetitions of l
Property:
3. In general, AM(l ) GM(l )
Regularity condition: l is regular if AM(l )=GM(l )
Property:
4. A is diagonalizable if all its eigenvalues are regular.
12/33
Diagonalization of a Matrix
Examples:

1 0
1. Show that the matrix A =
is not diagonalizable.
3 1
(Hints. Show AM(1)= 2, GM(1)= 1.)
2. Find
 a matrix P such that P
1 3
A=
.
4 5
1 AP
2
1
3. Diagonalize the matrix: A = 43
6
is a diagonal matrix where
3
3 3
5 35.
6 4
13/33