Carnegie Mellon
18-202
Mathematical Foundations of
Electrical Engineering
Lecture 22: SVD, Systems of ODEs
José M. F. Moura and Soummya Kar
Instructors
1
Carnegie Mellon
Today’s Lecture
⬛
Last Lecture
▪
▪
⬛
Eigenvalues and EVD
Singular Value Decomposition (SVD): Introduction
Today’s Lecture
▪
▪
SVD: Analysis and Properties
ODE Systems
o Matrix Exponential Function
o Autonomous Systems and Solutions
2
Carnegie Mellon
Eigenvalue Decomposition (EVD)
3
Carnegie Mellon
Eigenvalue Decomposition (EVD)
4
Carnegie Mellon
Matrix Decompositions
5
Singular Values
Let A ∈ Cm×n and note the following:
the matrix AH A ∈ Cn×n is positive semi-definite;
let λ1 ≥ λ2 ≥ · · · ≥ λn ≥ 0 be the n non-negative eigenvalues of
AH A arranged in decreasing order;
. √
define σi = λi for all i; the numbers σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0 are
called the n singular values of A.
Proposition
The number of non-zero eigenvalues of AH A or equivalently the number
of non-zero singular values of A is equal to rank A.
Proof.
Follows by noting that rank(A) = rank(AH A).
Recall: Rank of a diagonalizable matrix is equal to the number of its
non-zero eigenvalues.
3 / 23
Singular Value Decomposition (SVD)
Definition
Let A ∈ Cm×n with rank(A) = r . Denote by σ1 ≥ σ2 ≥ · · · ≥ σr > 0 the
r non-zero singular values of A.
A singular value decomposition of A is a factorization of the form:
A = UΣV H ,
where
U ∈ Cm×m and V ∈ Cn×n are unitary matrices;
Σ ∈ Cm×n is of the form
Σ1 0
Σ=
0 0
r ×r
and Σ1 ∈ C
is a diagonal matrix whose i-th diagonal entry is the
singular value σi , for i = 1, · · · , r .
4 / 23
Singular Value Decomposition (SVD)
First, we need to show that a SVD exists!
The SVD is not unique.
Consider the SVD A = UΣV H and let
U = u1 u2 · · · um and V = v1 v2 · · · vn .
Then U may be written as the sum of r rank-one matrices, i.e.,
U = σ1 u1 v1H + σ2 u2 v2H + · · · + σr ur vrH .
Alternatively, partitioning U and V as
U = U1 U2 , V = V1
where U1 ∈ Cm×r and V1 ∈ Cn×r , we obtain
V2
A = U1 Σ1 V1H .
5 / 23
Singular Value Decomposition (SVD)
Proposition
Let A ∈ Cm×n with rank(A) = r and let A = UΣV H be a singular value
decomposition of A. Partition U and V as:
U = U1 U2 , V = V1 V2
where U1 ∈ Cm×r and V1 ∈ Cn×r . Then
the columns of U1 and U2 form orthonormal bases for R(A) and
N (AH ) respectively;
the columns of V1 and V2 form orthonormal bases for R(AH ) and
N (A) respectively.
6 / 23
Singular Value Decomposition: Construction
Proposition
Let A ∈ Cm×n with rank(A) = r . Then A admits a singular value
decomposition A = UΣV H , where U and V are given as follows:
the columns of V are othonormal eigenvectors v1 , v2 , · · · , vn of AH A
such that
AH Avi = σi2 vi , ∀i = 1, · · · , r ,
for i ≤ r , the i-th column ui of U is given by ui = σi−1 Avi ; the
remaining (m − r ) columns of U are obtained by arbitrarily
extending to an orthonormal basis for Cm .
Proof.
First, verify that U thus obtained is a unitary matrix.
Then just show that UΣV H evaluates to A (using properties of
eigenvalues and orthonormality)!
7 / 23
Carnegie Mellon
SVD: Example
6
Carnegie Mellon
SVD: Example
7
Carnegie Mellon
SVD: A Note on Ranks
8
Singular Value Decomposition: Best Approximation
For A ∈ Cm×n , recall the Frobenius norm of A:

1/2
m X
n
X
∥A∥F = 
|aij |2  .
i=1 j=1
Proposition
Let A ∈ Cm×n with rank(A) = r and suppose that A = UΣV H is a
singular value decomposition of A. Then
r
X
∥A∥2F =
σi2 .
i=1
8 / 23
Singular Value Decomposition: Best Approximation
For A ∈ Cm×n , recall the Frobenius norm of A:

1/2
m X
n
X
∥A∥F = 
|aij |2  .
i=1 j=1
Proposition
Let A ∈ Cm×n with rank(A) = r and suppose that A = UΣV H is a
singular value decomposition of A. Then, for q ≤ r , the matrix
q
X
Aq =
σi ui viH
i=1
is the best approximation of A with matrices of rank at most q in the
Frobenius sense, i.e.,
b F
∥A − Aq ∥F ≤ ∥A − A∥
b ≤ q.
for all Ab ∈ Cm×n with rank(A)
9 / 23
Systems of Linear Differential Equations
Often we model coupled multivariable dynamical systems of the
form:
ẋ1 (t) = a11 x1 (t) + a12 x2 (t) + · · · + a1n xn (t) + f1 (t)
ẋ2 (t) = a21 x1 (t) + a22 x2 (t) + · · · + a2n xn (t) + f2 (t)
······
······
ẋn (t) = an1 x1 (t) + an2 x2 (t) + · · · + ann xn (t) + fn (t)
Here, each xi (t) is a scalar valued process (function of time t)
ẋi (t) = dxdti (t) denotes derivative
fi (t) is a forcing function applied to the i-th variable
Note: if n = 1, we reduce to a standard (single variable) ODE.
10 / 23
Systems of Linear Differential Equations
Matrix Representation:

 
ẋ1 (t)
a11
ẋ2 (t) a12

 
 ..  =  . .
 .   .
ẋn (t)
Or equivalently:
a1n
a12
a22
···
···

 

a1n
x1 (t)
f1 (t)

 

a2n 
 x2 (t) f2 (t)
  ..  +  .. 
 .   . 
a2n
···
ann
xn (t)
fn (t)
ẋ(t) = Ax(t) + f(t)
Here:
x(t) ∈ Rn : often called state of the system
f(t) ∈ Rn : often called input to the system
A ∈ Rn×n : called the system matrix.
11 / 23
Systems of Linear Differential Equations
The system
ẋ(t) = Ax(t) + f(t)
together with a vector of initial conditions x(0) = x0 ∈ Rn is called
a multivariable initial value problem (IVP).
Multivariable IVPs are used to model multivariable control systems,
also called state-space systems:
ẋ(t) = Ax(t) + Bu(t),
x(0) = x0 ∈ Rn .
Here, Bu(t) plays the role of the forcing function f(t), u(t) being a
control vector in Rm and B is a given ×m control matrix.
In this class, we focus on solving the above state space model for a
given f(t) (or equivalently Bu(t)).
In control systems theory, the goal is to design u(t) (hence f(t))
such that the system x(t) evolves in a specified manner!
12 / 23
Carnegie Mellon
Social Systems Modeling: Example
Opinion Dynamics
𝑢!(𝑡)
2
1
Variant of DeGroot model of opinion
diffusion
𝑥! 𝑡 : opinion state at agent 𝑛, a real number
𝑢! 𝑡 : input (exogenous) at agent 𝑛, a real number
State (opinion) evolution:
̇
𝑥" (𝑡) = 𝐴"# 𝑥# 𝑡 + 𝐴"" 𝑥" 𝑡 + 𝐴"$ 𝑥$ 𝑡 + 𝑢" (𝑡)
3
𝑥̇ = 𝐴𝑥 + 𝐵𝑢
State-space model
• How to design the input to drive the opinion states to a desired
value?
2
Higher-Order ODEs
A general n-th order ODE in the scalar variable z(t) ∈ R evolves as:
an z (n) (t) + an−1 z (n−1) (t) + · · · + a0 z(t) = f (t),
where
f (t) is the forcing function
. k
z (k) (t) = d dtz(t)
denotes the k-th order derivative
k
an , an−1 , · · · , a0 ∈ R and an ̸= 0
initial conditions: z (n−1) (0), z (n−2) (0), · · · , z(0), if we are interested
in solving an initial value problem
Can we model this n-th order ODE as a system of (first-order) ODE of
the form
ẋ(t) = Ax(t) + f(t)
by introducing appropriate variables and transformations?
13 / 23
Higher-Order ODEs
Define variables:
.
x1 (t) = z(t)
.
x2 (t) = z (1) (t) = ẋ1 (t)
...
. (n−1)
xn (t) = z
(t) = ẋn−1 (t)

x1 (t)
x2 (t)


Let x(t) =  . .
 .. 

xn (t)

 
ẋ1 (t)
ẋ2 (t) 

 
=⇒ ẋ(t) =  .  = 
 ..  
ẋn (t)
x2 (t)
x3 (t)
..
.
(n−1)
(n−2)
− an−1
(t) − an−2
(t) − · · · + f (t)
an z
an z





14 / 23
Higher-Order ODEs
Thus,

x2 (t)
x3 (t)
..
.





ẋ(t) = 



an−2
an−1
1
− an xn (t) − an xn−1 (t) − · · · + an f (t)

 

0
0
1
0 ...
0

 .. 

0
0
1
.
.
.
0

 

=
 x(t) +  .  f (t)
..

0

.
an−1
a1
a2
a0
1
− an − an − an . . . . . . − an
an
An n-th order ODE in a scalar variable can be represented by a
n-dimensional system of (first order) ODE in the standard form, i.e.,
the A matrix is n × n
The A matrix obtained above has a special structure called the
companion form
The n initial conditions on z and its derivates becomes a vector
initial condition on x
15 / 23
Systems of ODEs
We will first focus on the autonomous system (setting f(t) ≡ 0) and
the corresponding initial value problem, i.e., the system
ẋ(t) = Ax(t), subject to x(0) = x0 ∈ Rn .
In the scalar case (n = 1) this reduces to a standard first-order
scalar IVP:
ẋ(t) = ax(t), subject to x(0) = x0 ∈ R.
What is the solution in this case?
Can we extend the scalar solution to the general vector case?
16 / 23
Matrix Exponentials
Recall:
Definition
For a n × n matrix A, the matrix exponential e A is defined as:
∞
1
1
. X1
eA =
(A)j = I + A + A2 + · · · + Aj + · · · +
j!
2!
j!
j=0
Note, the above expression is well-defined, i.e., the defining series
converges (absolutely).
17 / 23
Matrix Exponential Function
Definition
For a n × n matrix A, the matrix exponential function e At is defined as:
∞
1
1
. X1
(At)j = I + At + A2 t 2 + · · · + Aj t j + · · · +, ∀t ∈ R.
e At =
j!
2!
j!
j=0
Note, for a fixed A, the matrix exponential e At is a function that
maps t ∈ R to Rn×n .
In what follows, we will fix A and study properties of e At as a
function of t.
18 / 23
Matrix Exponential Function
The matrix exponential function satisfies most properties of standard
(scalar) exponentials with certain modifications:
Proposition
For any A ∈ Rn×n , the following hold:
1
e A0 = I
2
e A(t+s) = e At e As
19 / 23
Matrix Exponential Function
The matrix exponential function satisfies most properties of standard
(scalar) exponentials with certain modifications:
Proposition
For any A, B ∈ Rn×n , if AB = BA, the following hold:
e (A+B)t = e At e Bt = e Bt e At ,
∀t.
Note, the above property generally requires the commutativity of A
and B, i.e., may fail to hold if AB ̸= BA.
20 / 23
Matrix Exponential Function
The matrix exponential function satisfies most properties of standard
(scalar) exponentials with certain modifications:
Proposition
For any A ∈ Rn×n and t ∈ R, the following hold:
At
= e Tr(At) > 0
1 det e
2
e At is nonsingular for all t ∈ R and we have that
−1
e At
= e −At .
21 / 23
Matrix Exponential Function
The matrix exponential function satisfies most properties of standard
(scalar) exponentials with certain modifications:
Proposition
For any A ∈ Rn×n , the matrix exponential function e At is differentiable
for all t and we have that:
de At
= Ae At , ∀t.
dt
Note, derivative of the matrix e At is interpreted component-wise;
At
hence, both e At and dedt are n × n.
22 / 23
Autonomous (Homogeneous) Systems
We are now ready to obtain solutions for homogeneous systems of ODEs:
Theorem
For A ∈ Rn×n and x0 ∈ Rn , the unique solution of the initial value
problem for the system of linear ordinary differential equations,
ẋ(t) = Ax(t), subject to x(0) = x0 ,
is given by x(t) = e At x0 for all t ∈ R.
23 / 23