Introduction to Compressed
Sensing and its applications
mehul.raval@ahduni.edu.in
Age of Digital World
• Our life revolve around the digital world
– Entertainment, communication, business, life !!
• Digital bits streams running at the background
is expected to deliver “natural” performance.
– Surround sound, 3D TV, sixth sense !!
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
2
Human centric conversion process
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
3
Nyquist Theorem
• Band limited signal with highest frequency of
B Hz can be reconstructed perfectly from its
samples with rate > 2B. (Nyquist Rate).
• Relation in X(t) and X(nT).
– Digital operation replacing analog counter parts.
– Relationship in power spectral densities of analog
and discrete random process.
• Estimation and detection by DSP is possible.
21-May-14
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
4
Spectrums of time domain signal and its samples
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
5
Wide band signal acquisition
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
6
ADC
• Analog to digital converters forms heart.
• Physical (analog) information streams of
numbers  digital processing by software.
• Intriguing task Snap shot of fast varying
signal + acquiring measurements.
• Unprecedented strain on ADC’s and DSP.
– Demand is ever increasing.
21-May-14
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
7
Ever increasing demand
• After sampling, we retain large number of bits.
• Conventional solution to storage space
– Sampling  Compression (exploiting redundancy)
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
8
Ultimate question[1]
• Why so much effort is spent on acquiring
(sampling) the data(redundancy) when most
of it will be thrown away (compression)?
• Can’t we directly measure the part
(information) which will not be thrown away?
• Why can’nt we ask such a question??
[1] D.L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52, no.4, pp.
1289-1306, Sep. 2006.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
9
Idea of simultaneous compression and
sampling
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
10
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
11
Common link
• Basis Function
• Coefficients
• Signal
• k - sparse signal 𝑥 ≅ Φ𝜃
• Support of non zero indices for 𝜃 denoted as
𝑆(𝜃) .
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
12
Transform domain image
representation
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
13
Basics of Compressed Sensing
• Demo
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
14
Whittaker–Shannon–Kotelnikov (WSK)
v/s
Compressive sensing
•
•
•
•
WSK
Greater than Nyquist rate.
Uniform / Non uniform
sampling
Non uniform sampling is
based
on
Lagrange
interpolation.
Theory developed for
continuous time signals
CS
• Sub Nyquist sampling.
• Randomized
measurement matrix.
• Samples are inner
product.
• Initial focus on finite
dimensional signal.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
15
Whittaker–Shannon–Kotelnikov (WSK)
v/s
Compressive sensing
WSK
• No underlying
structure.
CS
signal • Initial sparse. Now
structure
beyond
sparsity is explored.
• Transform coding based • Signal structure over
and above sparsity gives
compression.
higher
compression.
• Does
not consider
• Structured non random
Hardware
measurement matrix.
implementation
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
16
.
Basic CS frame work
• 𝑥 is a N x 1 vector,
• 𝑦 is a M x 1 vector with M << N
•
is a random measurement matrix.
• Sparsifying dictionary of basis
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
17
Recovery
• Recover
• Given:
and
•
: Class with all k sparse signal.
• CS makes exhaustive search in such that
• Solution using
optimization problem
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
18
Issues with
•
•
optimization is computationally very
expensive: it is an non-deterministic
polynomial-time (NP) hard problem.
• E.g. M = 1000, N = 5000, k = 100
•
search space.
• Non convex
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
19
Questions to be tackled
1. Can this problem be solved by any other
mechanisms?
2. How can we get an estimated solution and
what level of estimation accuracy is
acceptable? (after all Engineers always looks
for the workable approximate solutions !!!);
3. What kind of approximation will yield the
solution closer to the desired one?
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
20
Convex and Non convex set
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
21
.
Convex relaxation -
norm
•
•
• Set
is convex, so above
problem is also convex.
• A strictly posed convex problems leads to a
close form solution and guaranteed to
converge at the local minima.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
22
norm
• Find Xi’s for which
norm is minimal.
• It is not strictly convex and it may have multiple
solution.
• However, these solutions are
1. clustered around in a convex set as all optimal
solutions will have an penalty and their
combination would also be convex;
2. the set is bounded;
3. among them at least one has at most k non zero
elements.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
23
Best possible approximation
•
•
approximation
ℓ𝑝 norm with 𝑝<1 .
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
24
Heart: The measurement matrix
•
;
:
with M << N
• Basis Fixed independent of signal.
• Most fundamental design questions:
1. how much of information about signal x is
retained in its linear measurements y ?;
2. how can the linear measurements y uniquely
represent x?;
3. how can the original signal be recovered from its
measurement?
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
25
The measurement matrix
• It is rank deficient with non empty Null space
• Consequence:
• Unable to recover the signal
measurements.
• Design: For distinct k sparse signals
Should have a unique measurement
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
from
26
Characterization of Uniqueness
• Spark = Sparse + Rank
• If spark( ) > 2k 
uniquely
represents a k- sparse signal belonging to class
• The spark of the measurement matrix is used
to ensure stability and consistency.
• Computing is search over all
sub-matrices.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
27
Coherence
• Coherence can be used to identify the sparse
signal.
• It is describing the dependency between two
columns.
• Supremum on sparsity k is
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
28
Uniqueness property in presence of
noise
• Modified
• In CS, it is assumed that is available during
sparse signal recovery.
• Modified
• Measurement process should be robust to
such noise.
• Sparse recovery is possible if
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
29
Restricted Isometric property (RIP)
• A matrix satisfies (k, ) restricted iso-metry
property of order k if for
,
• Isometry is a function between the two spaces
which has a property to preserve distance
between each pair of points.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
30
.
.
Building Sensing Matrices
• Vandermonde matrix
has spark M + 1
•
: Geometric progression.
• Poor conditioning.
• Gabor Frame
= n x n time shift matrix
• Bernoulli, Gaussian, or sub Gaussian
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
31
Research avenue
• Structured measurement matrix: Application
dependent.
• Subjected to the physical constraint of the
application.
M.F. Duarte & Y.C. Eldar, "Structured compressed sensing: from theory to applications,"
IEEE Trans. Sig. Proc., vol.59, no. 9, pp. 4053-4085, Sept. 2011.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
32
CS recovery
• CS recovery = L1 minimization
• The choice of algorithm is based on various factors,
namely:
1. signal reconstruction timing from measurement
vector;
2. the number of measurement required for
recovery to determine the storage requirements;
3. the simplicity of the implementation;
4. possible portability to the hardware for
execution;
5. fidelity of the signal recovery.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
33
BP: Basis Pursuit, BPIC: Basis Pursuit with Inequality
Constraint, BPDN: Basis Pursuit De-noising, MP: Matching
Pursuit, OMP: Orthogonal Matching Pursuit, IHT: Iterative
Hard Thresholding
21-May-14
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
34
Basic
•
• Its power stems from the fact that it converts
the search into convex problem and provides
the accurate recovery.
• Basis pursuit (BP):
norm has a tendency to
locate the sparse solutions if ever they exist.
21-May-14
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
35
2D reconstruction
(a)
(b)
• a) original Image N = 6400 (80 x 80); b) reconstructed
Image with M = 2400, MSE: 0.0 reconstruction time:
18.64 sec.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
36
BPIC
Bound based on noise.
Could be user defined
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
37
N = 1024; k
= 50; M =
220;
MSE=2.02 x
10-4
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
38
(a)
(b)
• (a) reconstructed Image M = 1600, MSE: 0.25,
reconstruction time: 16.27 sec;
• (b) reconstructed Image M = 800, MSE: 0.31,
reconstruction time: 31.19 sec
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
39
Watermarking application
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
40
Detector
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
41
Over determined system
•
• Construct matrix
with M > N
s.t
= 0.
• Estimate using CS formulation.
•
s.t
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
42
Fidelity
Non malicious
Malicious manipulations
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
43
Single Pixel Camera
• webee.technion.ac.il
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
44
Summary
• One of the most exciting domain.
• Interdisciplinary: signal processing, statistics,
probability theory, computer science,
optimization, linear programming.
• Look beyond the random measurement
matrix.
• Developing a better signal models: finite rate
innovation (FRI), Xampling framework
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
45
References
•
•
•
•
•
•
•
•
•
•
•
•
D.L. Donoho, "Compressed sensing", IEEE Trans. Inf. Theory, vol. 52, no.4, pp. 1289 - 1306, Sep. 2006.
E.J. Candès & T.Tao, "Near optimal signal recovery from random projections: Universal encoding
strategies," IEEE Trans. Inf. Theory, vol.52, no. 12, pp.5406 - 5425, Dec. 2006.
E.J. Candès, J. Romberg, & T. Tao, "Robust uncertainty principles: exact signal reconstruction from highly
incomplete frequency information," IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489 - 509, Dec. 2006.
Richard Baraniuk, "Compressive sensing," IEEE Sig. Proc. Mag., vol. 24, no. 4, pp. 118 -124, 2007.
M.F. Duarte & Y.C. Eldar, "Structured compressed sensing: from theory to applications," IEEE Trans. Sig.
Proc., vol.59, no. 9, pp. 4053-4085, Sept. 2011.
Compressed sensing, Theory and applications, (eds. Y.C. Eldar & Gitta Kutyniok), Cambridge university
press, Cambridge, UK, 2012.
E. J. Candès, J. Romberg and T. Tao, “Stable signal recovery from incomplete and inaccurate
measurements”, Comm. Pure Appl. Math., vol.59,pp. 1207–1223, 2006.
B. K. Natarajan, “Sparse approximate solutions to linear systems”, SIAM Journal on computing, vol. 24,
pp.227–234, 1995.
Nonlinear Optimization: Complexity Issues, S. A. Vavasis, Oxford University Press, New York, 1991.
E. J. Candès and T. Tao, "The power of convex relaxation: Near-optimal matrix completion," IEEE Trans.
Inform. Theory, vol. 56, no. 5, pp. 2053-2080, 2009.
Convex Optimization, Stephen Boyd & Lieven Vandenberghe, Cambridge University Press, 2004.
S. S. Chen, D. L. Donoho, & M. A. Saunders, " Atomic decomposition by basis pursuit," SIAM J. Scientific
Computing, vol. 20, no. 1, pp.33–61, 1998.
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
46
Compressive sensing solvers
•
•
•
•
•
http://users.ece.gatech.edu/~justin/l1magic/
http://sparselab.stanford.edu/
http://www.lx.it.pt/~mtf/GPSR/
http://www.stanford.edu/~boyd/l1_ls/
http://www.personal.soton.ac.uk/tb1m08/sparsify/spa
rsify.html
• http://www.lx.it.pt/~mtf/SpaRSA/
• https://sites.google.com/site/igorcarron2/cs#reconstru
ction (Comprehensive Listing of solvers)
• http://videolectures.net/bmvc09_chellappa_cscv/
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
47
mehul.raval@ahduni.edu.in
Workshop on Compressive sensing, MNIT
Jaipur, 12th - 13th Sept. 2015
48