+
Compressed Sensing
+
Compressed
Sensing

Mobashir Mohammad

Aditya Kulkarni

Tobias Bertelsen

Malay Singh

Hirak Sarkar

Nirandika Wanigasekara

Yamilet Serrano Llerena

Parvathy Sudhir
3
+
Introduction
Mobashir Mohammad
+
4
The Data Deluge

Sensors: Better… Stronger… Faster…

Challenge:

Exponentially increasing amounts of data

Audio, Image, Video, Weather, …

Global scale acquisition
+
5
+
6
Sensing by Sampling
Sample
N
+
7
Sensing by Sampling (2)
N >> L
Sample
N
Compress
L
JPEG
…
N >> L
L
Decompress
N
+
8
Compression: Toy Example
+
9
Discrete Cosine Transformation
Transformation
+
10
Motivation

Why go to so much effort to acquire all the data when most of
the what we get will be thrown away?

Cant we just directly measure the part that wont end up
being thrown away?
Donoho
2004
11
Outline
•
•
•
•
•
•
•
+
Compressed Sensing
Constructing Φ
Sparse Signal Recovery
Convex Optimization Algorithm
Applications
Summary
Future Work
12
+
Compressed Sensing
Aditya Kulkarni
+
13
What is compressed sensing?

A paradigm shift that allows for the saving of time and space
during the process of signal acquisition, while still allowing
near perfect signal recovery when the signal is needed
Analog
Audio
Signal
High-rate
Nyquist rate
Sampling
Compressed Compression
(e.g. MP3)
Sensing
Low-rate
+
14
Sparsity

The concept that most signals in our natural world are sparse
a. Original image
c. Image reconstructed by discarding the zero coefficients
+
15
How It Works
+ Dimensionality Reduction
Problem
𝒚=𝚽𝒙
I.
Measure 𝑦
II. Construct sensing matrix Φ
III. Reconstruct 𝑥
16
+
17
Sampling
Φ=I
𝑦
𝑥
𝑁× 1
𝑁×1
measurements
sparse signal
=
𝐾
nonzero
entries
𝑁×𝑁
+
18
Φ
𝑦
𝑥
𝑁× 1
𝑀×1
measurements
𝐾<𝑀≪𝑁
sparse signal
=
𝐾
nonzero
entries
𝑀×𝑁
+
19
Ψ
𝑥
𝑁×1
𝑁
nonzero
entries
𝛼
𝑁×1
=
𝐾
nonzero
entries
𝑁×𝑁
+
20
Sparsity

The concept that most signals in our natural world are sparse
a. Original image
c. Image reconstructed by discarding the zero coefficients
+
21
Φ
𝑦
Ψ
𝛼
=
𝑀×1
𝑀×𝑁
𝑁×𝑁
𝑁×1
22
+
Constructing Φ
Tobias Bertelsen
+
23
RIP - Restricted Isometry Property
A matrix 𝛷 satisfies the RIP of order K if there exists a 𝛿 ∈ (0,1) such that:
Φ𝑥1 − Φ𝑥2 22
1−𝛿 ≤
≤1+𝛿
2
𝑥1 − 𝑥2 2
holds for all 𝐾-sparse vectors 𝑥1 and 𝑥2
Or equally
Φ𝑥 22
1−𝛿 ≤
≤1+𝛿
𝑥 22
holds for all 2K-sparse vectors 𝑥

The distance between two points are approximately the same
in the signal-space and measure-space
+
24
RIP - Restricted Isometry Property

RIP ensures that measurement error does not blow up
1−𝛿 ≤
Φ𝑥1 −Φ𝑥2 22
𝑥1 −𝑥2 22
≤1+𝛿
Image: http://www.brainshark.com/brainshark/brainshark.net/portal/title.aspx?pid=zCgzXgcEKz0z0
+
25
Randomized algorithm
1.
Pick a sufficiently high 𝑀
2.
Fill Φ randomly according to some random
distribution
 Which
 How
distribution?
to pick 𝑀?
 What
is the probability of satisfying RIP?
+
26
Sub-Gaussian distribution


Defined by 𝐸 𝑒 𝑋𝑡 ≤ 𝑒
𝑐2 𝑡2
2

Tails decay at least as fast as the Gaussian

E.g.: The Gaussian distribution, any bounded distribution
Satisfies the concentration of measure property (not RIP):
For any vector 𝑥 and a matrix Φ with sub-Gaussian entries, there exists a
𝛿 ∈ [0,1] such that
Φ𝑥 22
1−𝛿 ≤
≤ 1+𝛿
𝑥 22
holds with exponentially high probability 1 − 2−𝑑 𝛿 𝑀
where 𝑑(𝛿) is a constant only dependent on 𝛿
+
27
Johnson-Lidenstrauss Lemma

Generalization to a discrete set of 𝑃 vectors

For any vector the magnitude are preserved with:
𝑝𝑟𝑜𝑏 = 1 − 2−𝑑 𝛿 𝑀

For all P vectors the magnitudes are preserved with:
𝑝𝑟𝑜𝑏 = 1 − 𝑃2−𝑑 𝛿 𝑀 = 1 − 2log 𝑃−𝑑 𝛿 𝑀

To account for this 𝑀 must grow with 𝑂 log 𝑃
+
28
Generalizing to RIP
Φ𝑥 22
𝑥 22

RIP: 1 − 𝛿 ≤

We want to approximate all 2𝐾-sparse vectors with 𝑃 unit vectors


≤ 1+𝛿 ,
𝑥
0
≤ 2𝐾
𝑁
The space of all 2𝐾-sparse vectors is made up of
2𝐾
2𝐾-dimensional subspaces
– one for each position of non-zero entries in 𝑥
We sample points on the unit-sphere of each subspace

𝑁
≤
2𝐾
𝑁𝑒 2𝐾
2𝐾
𝑁 𝐾
𝐾

𝑃=𝑂

𝑀 = 𝑂 log 𝑃 = 𝑂 𝐾 log
𝑁
𝐾
+
29
Randomized algorithm
 Use
sub-Gaussian distribution
 Pick
𝑀 = 𝑂 𝐾 log
 Exponentially
 Formal
𝑁
𝐾
high probability of RIP
proofs and specific formulas for
constants exists
+
30
Sparse in another base

We assumed the signal itself was sparse

What if the signal is sparse in another base, i.e. 𝑥 = Ψ𝛼 is
sparse.

ΦΨ must have the RIP

As long as Ψ is an orthogonal basis, the random construction
works.
+
31
Characteristics of Random Φ

Stable


Universal


Works with any orthogonal basis
Democratic



Robust to noise, since it satisfies RIP
Any element in has equal importance
Robust to data loss
Other Methods


Random Fourier submatrix
Fast JL transform
32
+
Sparse Signal Recovery
Malay Singh
+
33
The Hyperplane of 𝑦 = Φ𝑥
+
34
𝐿𝑝 Norms for
N dimensional vector x
1
𝑝
𝑁
𝑥𝑗
𝑥
𝑝
=
𝑝
𝑗=1
𝑠𝑢𝑝𝑝(𝑥)
max 𝑥𝑗
𝑗=1,2,…,𝑁
Unit Sphere of
𝐿1 norm
if 𝑝 > 0
Unit Sphere of
𝐿2 norm
if 𝑝 = 0
if 𝑝 = ∞
Unit Sphere of
𝐿∞ norm
Unit Sphere of
𝐿0.5 quasinorm
+
35
𝐿𝑝 Balls in higher dimensions
+
36
How about 𝐿0 minimization
𝑥 = arg min 𝑥
𝑥∈𝐵(𝑦)
0
𝐵 𝑦 = 𝑥 ∶ Φ𝑥 = 𝑦
But the problem is non-convex
and very hard to solve
+
37
We do the 𝐿2 minimization
𝑥 = arg min 𝑥
𝑥∈𝐵(𝑦)
2
𝐵 𝑦 = 𝑥 ∶ Φ𝑥 = 𝑦
We are minimizing the Euclidean
distance. But the arbitrary angle of
hyperplane matters
+
38
What if we convexify the 𝐿0 to 𝐿1
𝑥 = arg min 𝑥
𝑥∈𝐵(𝑦)
1
𝐵 𝑦 = 𝑥 ∶ Φ𝑥 = 𝑦
+
39
Issues with 𝐿0 minimization

⋅ 0 is non-convex and 𝐿0 minimization is
potentially very difficult to solve.
 We
convexify the problem by replacing ⋅
⋅ 1 . This leads us to 𝐿1 Minimization.
0
by
 Minimizing
𝐿2 results in small values in some
dimensions but not necessarily zero.
 𝐿1 provides
a better result because in its solution
most of the dimensions are zero.
40
+
Convex Optimization
Hirak Sarkar
+
41
What it is all about …
 Find
a sparse representation 𝑥
𝑦 = Φ𝑥
 Here
Φ ∈ ℝ𝑀×𝑁 and 𝑥 ∈ ℝ𝑁 , Moreover 𝑀 ≪ 𝑁
 Two
ways to solve
(P1)
min 𝐽 𝑥 : 𝑦 = Φ𝑥
𝑥
where 𝐽 𝑥 is a measure of sparseness
(P2)
min{𝐽 𝑥 : 𝐻 Φ𝑥, 𝑦 ≤ 𝜖}
𝑥
+
42
How to chose 𝐻 𝑥 and 𝐽 𝑥



Take the simplest convex function
𝐽 𝑥 = 𝑥
1
𝐻 Φ𝑥, 𝑦 =
1
2
A simple 𝐻 𝑥
Φ𝑥 − 𝑦
Final unconstrained version
min{𝜇 𝑥
𝑥
1
+ 𝐻(𝑥)}
2
2
+
43
Versions of the same problem
1
min{𝜇 𝑥 1 + Φ𝑥 − 𝑦
𝑥
2
2
2}
1
min 𝑥 1 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜
Φ𝑥 − 𝑦
𝑥
2
2
2
≤𝜖
1
min Φ𝑥 − 𝑦
𝑥 2
1
≤𝑡
2
2
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥
+
44
Formalize
 Nature



Convex
Differentiable
𝛻𝐻 𝑥 ′ = Φ𝑇 (Φ𝑥 ′ − 𝑦)
 Basic




of 𝐻
Intuition
Take an arbitrary 𝑥 ′
Calculate 𝑥 ′ − 𝜏𝛻𝐻(𝑥′)
Use the shrinkage operator
Make corrections and iterate
+
45
Shrinkage operator
 We
define the shrinkage operator as follows
𝑥−𝛼
𝑠ℎ𝑟𝑖𝑛𝑘 𝑥, 𝛼 =
0
𝑥+𝛼
if 𝛼 < 𝑥
if − 𝛼 ≤ 𝑥 ≤ 𝛼
if 𝑥 < −𝛼
+
46
Algorithm
Input: Matrix Φ, Signal measurement 𝑦, parameter sequence 𝜇𝑛
Output: Signal estimate 𝑥
Initialization: 𝑥0 = 0, 𝑟 = 𝑦, 𝑘 = 0
+
47
Performance
 For
closed and convex function 𝐽 𝑥 any 𝜇 > 0 the
algorithm converges within finite steps
 For
𝐽 𝑥 = 𝑥 1 and a moderate 𝜇 number of
iterations needed is less than 5
48
+
Single Pixel Camera
Nirandika Wanigasekara
+
49
Single Pixel Camera

What is a single pixel camera

An optical computer

sequentially measures the 𝑦[𝑖]

Directly acquires 𝑀 random linear measurements without first
collecting the 𝑁 pixel values
+
50
Single Pixel Camera- Architecture
+
51
Single Pixel Camera- DMD Array

Digital Micro mirror Device

A type of a reflective spatial light modulator

Selectively redirects parts of the light beam

Consisting of an array of N tiny mirrors

Each mirror can be positioned in one of two states(+/-10
degrees)

Orients the light towards or away from the second lens
+
52
Single Pixel Camera- Architecture
+
53
Single Pixel Camera- Photodiode

Find the focal point of the second lens

Place a photodiode at this point

Measure the output voltage of the photodiode

The voltage equals 𝑦𝑗 , which is the inner product between 𝜙𝑗
and the desired image 𝑥.
+
54
Single Pixel Camera- Architecture
+
55
Single Pixel Camera- measurements

A random number generator (RNG) sets the mirror
orientations in a pseudorandom 1/0 pattern

Repeats the above process for 𝑀 times

Obtains the measurement vector 𝑦 and ϕ

Now we can construct the system in the
=
𝑦𝑗
Φj
𝑥
+
56
Single Pixel Camera- Architecture
+
57
Sample image reconstructions

256*256 conventional image of
black and white ‘R’

Image reconstructed from
𝑀 = 1300
How can we improve the reconstruction further? M
256 ⇥ 256
256 ⇥ 256
= 1300
256 ⇥ 256
M = 1300
+
58
Utility

This device is useful when measurements are expensive

Low Light Imager

Conventional Photomultiplier tube/ avalanche photodiode

Single Pixel Camera  Single photomultiplier
Original
800
1600
65536 pixels
from 6600
+
59
Utility

CS Infrared Imager


IR photodiode
CS Hyperspectral Imager
60
+
Compressed Sensing MRI
Yamilet Serrano Llerena
+
61
Compressed Sensing MRI
Magnetic Resonance Imaging
(MRI)
Essential medical imaging tool with
slow data acquisition process.
Applying Compressed Sensing (CS)
to MRI offers that:
• We can send much less
information
reducing
the
scanned time
• We are still able to reconstruct
the image in based on they are
compressible
+
62
Compressed Sensing MRI
Scan Process
+
63
Scan Process
Signal Received
K-space
Space where MRI data is
stored
K-space trajectories:
K-space is 2D Fourier
transform of the MR image
+
64
In the context of CS
Φ:
•
•
•
y=Φx
Is depends on the acquisition device
Is the Fourier Basis
Is an M x N matrix
y:
•
Is the measured k-space data from
the scanner
x:
+
65
Recall ...

The heart of CS is the assumption that x has a sparse
representation.
Medical Images are naturally compressible by sparse
coding in an appropriate transform domain (e.g. Wavelet
Transform)
Not significant
+
66
Compressed Sensing MRI
Scan Process
+
67
Image Reconstruction
CS uses only a fraction of the MRI data to reconstruct image.
+
68
Image Reconstruction
+
69
Benefits of CS w.r.t Resonance

Allow for faster image
cardiac/pediatric imaging)
acquisition
(essential
for

Using same amount of k-space data, CS can obtain higher
Resolution Images.
70
+
Summary
Parvathy Sudhir Pillai
+
71
Summary

Motivation

Data deluge

Directly acquiring useful part of the signal

Key idea: Reduce the number of samples

Implications

Dimensionality reduction

Low redundancy and wastage
+
72
Open Problems

‘Good’ sensing matrices

Adaptive? Deterministic?

Nonlinear compressed sensing

Numerical algorithms

Hardware design
Coefficients (𝛼)
Intensity (x)
Phase (𝜃)
+
73
Impact

Data generation and storage

Conceptual achievements

Exploit minimal complexity efficiently

Information theory framework

Numerous application areas

Legacy - Trans disciplinary research
Information
Hardware
CS
Software
Complexity
+
74
Ongoing Research

New mathematical framework for evaluating CS schemes

Spectrum sensing


Not so optimal
Data transmission - wireless sensors (EKG) to wired base stations.

90% power savings
+
75
In the news
+
76
References

Emmanuel Candes, Compressive Sensing - A 25 Minute Tour, First EU-US Frontiers of
Engineering Symposium, Cambridge, September 2010

David Schneider, Camera Chip Makes Already-Compressed Images, IEEE Spectrum,
Feb 2013

T.Strohmer. Measure what should be measured: Progress and Challenges in
Compressive Sensing. IEEE Signal Processing Letters, vol.19(12): pp.887-893, 2012.

Larry Hardesty, Toward practical compressed sensing, MIT news, Feb 2013

Tao Hu and Mitya Chklovvskii, Reconstruction of Sparse Circuits Using Multi-neuronal
Excitation (RESCUME), Advances in Neural Information Processing Systems, 2009

http://inviewcorp.com/technology/compressive-sensing/

http://ge.geglobalresearch.com/blog/the-beauty-of-compressive-sensing/

http://www.worldindustrialreporter.com/bell-labs-create-lensless-camera-throughcompressive-sensing-technique/

http://www.lablanche-and-co.com/
+
77
THANK YOU