Sparse Solutions of Linear Systems of
Equations and Sparse Modeling of
Signals and Images
Alfredo Nava-Tudela
John J. Benedetto, advisor
AMSC 663
1
Outline
- Problem: Find “sparse solutions” of Ax = b
- Why do we care?
- An application in signal processing: compression
- Working definition of “sparse solution”
- Why things eventually work, for some cases
- How do we find sparse solutions? OMP algorithm
- Case study: What happens when we combine the
DCT and DWT?
- Conclusions/Recap
- Project timeline
AMSC 663
2
Problem
Let A be an n by m matrix with n < m, and rank(A)=n.
We want to solve
Ax = b,
where b is a data or signal vector,
and x is the solution with the fewest number of nonzero entries possible, that is, the “sparsest” one.
Observations:
- A is underdetermined and, since rank(A)=n,
there is an infinite number of solutions. Good!
- How do we find the “sparsest” solution? What does
this mean in practice? Is there a unique sparsest
solution?
AMSC 663
3
But, why do we care?
QuickTime™ and a
decompressor
are needed to see this picture.
231 kb, uncompressed,
320x240x3x8 bit
QuickTime™ and a
decompressor
are needed to see this picture.
74 kb, compressed 3.24:1
JPEG
AMSC 663
4
But, why do we care?
QuickTime™ and a
decompressor
are needed to see this picture.
QuickTime™ and a
decompressor
are needed to see this picture.
512 x 512 Pixels,
24-Bit RGB,
Size 786 Kbyte
75:1, 10.6 Kbyte
JPEG2000
AMSC 663
5
“Sparsity” equals compression
Both JPEG and JPEG2000 achieve their compression
mainly because at their core one finds a linear transform
(DCT and DWT, respectively) that reduces the number
of non-zero entries required to represent the data, within
an acceptable error.
We can then think of signal compression in terms of our
problem
Ax = b,
x is sparse, b is dense, store x!
AMSC 663
6
Working definitions of “sparse”
- Convenient to introduce the l0 “norm”:
||x||0 = # {k : xk ≠ 0}
- (P0): minx ||x||0 subject to ||Ax - b||2 = 0
- (P0e): minx ||x||0 subject to ||Ax - b||2 < e
Observation: Solving (P0) is NP-hard, bummer.
AMSC 663
7
Some theoretical results
Definition: The spark of a matrix A is the minimum
number of linearly dependent columns of A. We write
spark(A) to represent this number.
Theorem: If there is a solution x to Ax = b, and
||x||0 < spark(A) / 2, then x is the sparsest solution.
That is, if y ≠ x also solves the equation, then
||x||0 < ||y||0.
Observation: Computing spark(A) is combinatorial,
therefore hard. Alternative?
AMSC 663
8
More theoretical results
Definition: The mutual coherence of a matrix A is the
number
Lemma: spark(A) ≥ 1+1/mu(A).
Theorem: If x solves Ax = b, and ||x||0 < (1+mu(A)-1)/2,
then x is the sparsest solution, as before.
Observation: mu(A) is a lot easier and faster to compute,
but 1+1/mu(A) far worse bound than spark(A), in general.
AMSC 663
9
Finding sparse solutions:OMP
Orthogonal Matching Pursuit algorithm:
AMSC 663
10
Revisiting compression
Propose to study the compression properties of the
matrix
A = [DCT,DWT]
and compare it with the compression properties of
DCT or DWT alone.
Study the behavior of OMP for this problem.
Many wavelet options available to try, e.g., reversible
5/3 or floating-point 9/7 Daubechies, as in JPEG2000.
Interested in compression vs error graph properties.
AMSC 663
11
Conclusions/Recapitulation
- Finding sparse solutions to the linear system of
equations Ax = b, when A is an n by m full rank
matrix and n < m, is of interest to the signal
processing community.
- There are simple criteria to assert the uniqueness of
a given sparse solution.
- There are algorithms to find sparse solutions, e.g.,
OMP; and their convergence can be guaranteed
when there are “sufficiently sparse” solutions.
- Studies on the performance of OMP missing when
A is the concatenation of unitary matrices.
AMSC 663
12
Project timeline
- Oct. 15, 2010: Complete project proposal.
- Oct. 18 - Nov. 5: Implement OMP.
- Nov. 8 - Nov. 26: Validate OMP.
- Nov. 29 - Dec. 3: Write mid-year report.
- Dec. 6 - Dec. 10: Prepare mid-year oral presentation.
- Some time after that, give mid-year oral presentation.
- Jan. 24 - Feb. 11: Testing of OMP. Reproduce paper
results.
- Jan. 14 - Apr. 8: Testing of OMP on A = [DCT,DWT].
- Apr. 11 - Apr. 22: Write final report.
- Apr. 25 - Apr. 29: Prepare final oral presentation.
- Some time after that, give final oral presentation.
AMSC 663
13
References
A. M. Bruckstein, D. L. Donoho, and M. Elad, From sparse solutions of
systems of equations to sparse modeling of signals and images, SIAM
Review, 51 (2009), pp. 34–81.
S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, 1998.
B. K. Natarajan, Sparse approximate solutions to linear systems, SIAM
Journal on Computing, 24 (1995), pp. 227-234.
G. W. Stewart, Introduction to Matrix Computations, Academic Press,
1973.
D. S. Taubman and M. W. Mercellin, JPEG 2000: Image Compression
Funda- mentals, Standards and Practice, Kluwer Academic Publishers,
2001.
G. K. Wallace, The JPEG still picture compression standard,
Communications of the ACM, 34 (1991), pp. 30-44.
http://www.stanford.edu/class/ee398a/handouts/lectures/08-JPEG.pdf
AMSC 663
14