ECE 530 – Analysis Techniques for
Large-Scale Electrical Systems
Lecture 22: Singular Value Decomposition
Prof. Hao Zhu
Dept. of Electrical and Computer Engineering
University of Illinois at Urbana-Champaign
haozhu@illinois.edu
11/18/2015
1
Announcements
•
•
HW 6 posted, due Tuesday, December 1
Final exam on Monday Dec 14 from 1:30 to 4:30pm in
this room (ECEB-3081)
– Closed book, closed notes; you can bring in two note sheets
(one new note sheet and midterm exam note sheet), along
with simple calculators
2
Singular Value Decomposition
•
•
An extremely useful matrix analysis technique is the
singular value decomposition (SVD).
For any m by n real matrix A, it can be represented by
A  U Σ VT
where U (V) is an m by m (n by n) orthogonal matrix
(UTU = I), 𝚺 is an m by n diagonal matrix with entries
𝜎𝑖 ≥ 0 (singular values of A)
𝜎1 ≥ 𝜎2 ≥ ⋯ ≥ 𝜎𝑝 ≥ 0 with 𝑝 = min(𝑚, 𝑛)
•
• Computational order is O(n2m); ok if n is small
Source: Y. Saad, Lecture notes for CSCI 5304 at UMN
http://www-users.cs.umn.edu/~saad/csci5304/
3
SVD Existence and Uniqueness
•
Let 𝜎1 = 𝐀
•
There exists a pair of unit vectors 𝐮1 and 𝐯1 such that
𝐀𝐯𝟏 = σ1 𝐮𝟏
Complete 𝐯1 into an orthonormal basis as 𝐕 = 𝐯1 ; 𝐕2
Similarly, 𝐔 = 𝐮1 ; 𝐔2
•
•
•
•
•
2
= max 𝐀𝐱
𝐱 =1
𝜎1 𝐰′
𝜎1 𝒘′
𝑇
Thus, 𝐀𝐕 = 𝐔 ×
→ 𝐔 𝐀𝐕 =
≜ 𝐀𝟏
0 𝐁
0 𝐁
𝜎1
𝜎1
𝟐
𝟐
2
2
𝐀1
≥ 𝜎1 + 𝐰 = 𝜎1 + 𝐰
𝐰
𝐰
Only holds if 𝐰 = 𝟎
4
Induction proof: proceed with B
Two SVD Cases
5
The “Thin” SVD
•
•
•
For the Case-1, it can be rewritten as
𝚺1 𝑇
𝐀 = 𝐔1 ; 𝐔2
𝐕
0
which gives rise to
𝐀 = 𝐔1 𝚺1 𝐕 𝑇
with U1 is m by n (same shape as 𝐀) and 𝚺1 is n by n
Referred to as the “thin” SVD; useful for tall matrices
Q: how can one obtain the thin SVD from the QR
factorization of A and the SVD of an n by n matrix?
6
SVD Properties
•
•
Assume that
𝜎1 ≥ 𝜎2 ≥ ⋯ ≥ 𝜎𝑟 > 0 and 𝜎𝑟+1 = ⋯ = 𝜎𝑝 = 0
Then we have the following properties
– rank 𝐀 = 𝑟 = number of positive singular values
– Range 𝐀 = span 𝐮1, 𝐮2, … , 𝐮r
– Null 𝐀 = span 𝐯𝑟+1 , 𝐯𝑟+2 , … , 𝐯𝑛
•
The matrix A admits the SVD expansion
𝑟
𝜎𝑖 𝐮𝑖 𝐯𝑖𝑇
𝐀=
𝑖=1
7
SVD Properties
•
•
•
•
•
Largest singular value 𝜎1 = 𝐀
Frobenius norm 𝐀
2
=
2
2 1/2
Σ𝑖 𝜎𝑖
−1
If A is an n by n nonsingular matrix then 𝜎𝑖
are
singular values of 𝐀−1 and
1/𝜎𝑛 = 𝐀−1 2
Consider the r by r diagonal 𝚺𝑟 = diag (𝜎1 , … , 𝜎𝑟 ), we
can show that
2 𝟎
𝚺
𝑇
𝑇
𝑇
𝐀 𝐀 = 𝐕𝚺 𝚺𝐕 = 𝐕 𝑟
𝐕𝑇
𝟎 𝟎
This gives the eigen-decomposition of 𝐀𝑇 𝐀 (or 𝐀𝐀𝑇 )
8
SVD Applications
•
Based on the SVD expansion
A   u v +  2u 2 v 
T
1 1 1
T
2
  nu n v
T
n
the best rank-k approximation of A is
𝑘
𝜎𝑖 𝐮𝑖 𝐯𝑖𝑇
𝐀𝑘 =
•
•
𝑖=1
Sometimes each rank-1 component is termed as a mode
An immediate application is data compression for e.g.,
an image; only a small number of modes suffice
9
SVD Image Compression Example
Image source http://fourier.eng.hmc.edu/e161/lectures/svdcompression.html
10
SVD Applications
•
•
•
•
A related application is denoising.
𝐀 + 𝐍 = 𝐔 𝚺 + 𝜂𝐈 𝐕 𝑇
Noise strength is uniform across all modes, so it affects
more the smaller singular values
To remove noise effects, one can take the rank-k SVD
approximation by ignoring small singular-value modes
This is also related to principal component analysis
(PCA), where the singular vectors can serve as the
transformation to make any input data uncorrelated
11
PCA Example
•
Inexplicitly assuming input to be zero-mean Gaussian
12
Pseudo-inverse of a Matrix
•
•
•
•
The pseudo-inverse of a matrix generalizes concept of a
matrix inverse to a non-square matrix
– Often called as Moore-Penrose Matrix Inverse
Notation for the pseudo-inverse of A is A+ (or )
𝑟
𝑇
−1
𝐯
𝐮
𝑖 𝑖
𝚺
𝟎
+
𝑇
+ 𝑇
𝑟
𝐀 =𝐕
𝐔 = 𝐕𝚺 𝐔 =
σ𝑖
𝟎
𝟎
𝑖=1
Has to satisfy (i) AA+A = A,
(ii) A+AA+ = A,
(iii) (AA+)H = AA+, (iv) (A+A)H = A+A
If A is nonsingular, then A+ = A-1
13
Pseudo-inverse and LS
•
•
Pseudo-inverse can be directly obtained from SVD
Recall the LS solution is given by the normal equations
 A T A x
= A Tb
•
Thus we have 𝐱 𝐿𝑆 = 𝐀𝑇 𝐀
•
Even if 𝐀 is not of full column rank, the pseudo-inverse
based LS solution is the minimizer to ‖𝐛 − 𝐀𝐱‖ with
the smallest 2-norm among all possible minimizers
−1 𝐀𝑇 𝐛
= 𝐀+ 𝐛
14
Simple Least Squares Example
•
•
•
Assume we which to fix a line (mx + b = y) to three data
points: (1,1), (2,4), (6,4)
Two unknowns, m and b; hence x = [m b]T
Setup in form of Ax = b
1 1
1 
1 1
 2 1  m    4  so A =  2 1

 b  


 6 1    4 
 6 1
15
Simple Least Squares Example
•
•
Doing the SVD
 0.182 0.765
0   0.976 0.219 
6.559


T
A  UΣV   0.331 0.543 
  0.219 0.976 
0
0.988


 0.926 0.345  
Computing the pseudo-inverse
0   0.182 0.331 0.926 
 0.976 0.219  0.152
A   V Σ  UT  
 0
  0.765 0.543 0.345 

0.219

0.976
1.012




 0.143 0.071 0.214 

 T
A VΣ U  

0.762
0.548

0.310


16
Simple Least Squares Example
•
Computing x = [m b]T gives
1 
 0.143 0.071 0.214    0.429 

A b
4  



0.762
0.548

0.310
1.71

 4 

 
•
With the pseudo-inverse approach we immediately see
the sensitivity of the elements of x to the elements of b
17
Modal Identification
•
An important application of SVD in power systems is
related to its linearized dynamic modeling
𝐱 𝑡 = 𝐀𝐱(𝑡)
with each state
𝑎𝑘 𝑒 𝑏𝑘𝑡 cos 𝜔𝑘 𝑡 + 𝜃𝑘
𝑥𝑖 (𝑡) =
•
•
𝑘
Values 𝑎𝑘 and 𝜃𝑘 are determined by the initial
conditions at 𝑡 = 0
Parameters 𝑏𝑘 and 𝜔𝑘 are based on eigenvalues of 𝐀
18
Matrix Pencil Method
•
•
•
The goal of modal identification is to find the poles 𝑧𝑖
of the dynamic systems
The matrix pencil (MP) method measures the output
𝑦 𝑡 to produce a matrix with roots as 𝑧𝑖
A generalized eigenvalue problem
19
MP Procedure
•
•
•
•
•
•
Choose pencil parameter L s.t. 𝑛 ≤ 𝐿 ≤ 𝑁 − 𝑛 (~N/2)
Construct matrix [𝑌]
Perform an SVD of 𝑌 = 𝑈Σ𝑉 𝑇
Construct two matrices 𝑉1 and 𝑉2 from 𝑉 as
𝑉1 = 𝑣1 ; 𝑣2 ; … ; 𝑣𝑛−1
𝑉2 = [𝑣2 ; 𝑣3 ; … ; 𝑣𝑛 ]
Construct two matrices 𝑌1 = 𝑉1 𝑇 𝑉1
𝑌2 = 𝑉2 𝑇 𝑉2
The desired 𝑧𝑖 obtained as the generalized eigenvalues
of { 𝑌1 , 𝑌2 }; i.e., 𝑌1 𝑥𝑖 = 𝑧𝑖 𝑌2 𝑥𝑖
20
Toy Example
•
Three-mode waveform
•
MP solution
21
Power System Example
•
•
•
Dynamic PSS/E simulation of
a large Midwest utility system
Number of modes unknown
FFT results
22
Power System Example
•
Use MP to extract five modes
•
Reconstruction results (MP + two other methods)
23