Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Geometry in the sparse methods
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
Zhiyue Huang and Paul Marriott
Department of Statistics and Actuarial Science
University of Waterloo
What is the role of
randomness?
Application
April 6, 2011
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Outline
Geometry in
the sparse
methods
1
Introduction
From basis pursuit to compressed sensing
2
Geometry in sparse methods
Why is the solution sparse?
How sparse the solution is ?
What is the role of randomness?
3
Application
Mixture models
Statistical model space
4
Summary
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Signal representation
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
Represent f (t ) as linear combination of basis elements
f (t ) =
X
αi ψi (t ) or f = Ψα
{ψi (t )} could be sinusoids, wavelets, curvelets...
The coefficient sequence α
What is the role of
randomness?
Application
(1)
i
αi = hf , ψi i
or α = ΨT f .
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
(2)
Sparsity
Geometry in
the sparse
methods
Introduction
Original signal
40
30
signal f(t)
Zhiyue Huang
and
Paul Marriott
20
10
0
−10
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
50
Mixture models
Model space
200
250
200
250
200
250
100
50
0
−50
50
Application
150
time t
Transformed coefficients
How sparse the
solution is ?
What is the role of
randomness?
100
100
150
Sorted transformed coefficients
100
50
0
−50
50
100
150
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Basis pursuit
[Chen et al., 1999]
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Consider underdetermined linear equation
Ψn×d βd ×1 = yn×1 ,
Introduction
(3)
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
where Ψj , the j th column of Ψ, is basis spanning the space
of y and d ≫ n.
Basis pursuit
min kβkℓ1
subject to Ψβ = y
Application
Mixture models
Model space
can provide sparse solution β.
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
(4)
Compressed sensing
[Donoho, 2006]
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
x0 is k -sparse in Rn , a random matrix A obeys RIP of
order 2k .
A vector is said to be k -sparse if it has at most k nonzero
entries.
x0 can be compressed to Ax0 in Rd , d = O (k log n).
x0 can be recovered exactly by
What is the role of
randomness?
Application
min kx kℓ1
subject to Ax = y .
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Outline
Geometry in
the sparse
methods
1
Introduction
From basis pursuit to compressed sensing
2
Geometry in sparse methods
Why is the solution sparse?
How sparse the solution is ?
What is the role of randomness?
3
Application
Mixture models
Statistical model space
4
Summary
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
ℓ1 minimization in basis pursuit
Geometry in
the sparse
methods
min kx kℓ1 subject to y = Ax
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
ℓ1 ball
Rk
x0
How sparse the
solution is ?
x̂
ℓ2 ball
x̂
x0
Rk
What is the role of
randomness?
Application
Mixture models
Model space
solution space
{x; Ax = y}
solution space
{x; Ax = y}
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Outline
Geometry in
the sparse
methods
1
Introduction
From basis pursuit to compressed sensing
2
Geometry in sparse methods
Why is the solution sparse?
How sparse the solution is ?
What is the role of randomness?
3
Application
Mixture models
Statistical model space
4
Summary
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Solid simplex
Geometry in
the sparse
methods
Rn
Zhiyue Huang
and
Paul Marriott
Solid simplex
Introduction
(k − 1)-faces
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
k-sparse vector x
Origin
Standard simplex
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Standard simplex T n−1 : the convex hull of the unit basis
vectors.
Summary
Solid simplex T0n : the convex hull of T n−1 and the origin.
References
T n−1 is the outward part of T0n .
Model space
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Polytopes
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Rd
Polytope
P = AT
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
origin
aj
zerofree face
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
P = AT0n is a convex polytope in Rd , where A : Rn → Rd .
P = conv ({0} ∪ {aj }nj=1 ), where aj is the j th column of A.
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
k -neighborly polytope
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
k -neighborly ploytope:
every set of k vertices spans a (k − 1)-face of P .
outwardly k -neighborly polytope:
every set of k vertices not including the origin spans a
(k − 1)-face of P .
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Neighborliness
[Donoho and Tanner, 2005]
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Suppose the polytope P = AT has N vertices and outwardly
k -neighborly, if and only if, for all ℓ = 0, · · · , k − 1, and for all
ℓ-dimensional faces F of T n−1 , AF is a ℓ-dimensional face of P .
Introduction
n
A: R → R
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Simplex T
x ∈ (k−1)−face F
d
y ∈ (k−1)−face AF
Polytope P
Why is the solution
sparse?
a
How sparse the
solution is ?
origin
What is the role of
randomness?
Application
Mixture models
Model space
origin
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
j
Unique representation
[Donoho and Tanner, 2005]
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
For k -neighborly polytopes, every low-dimensional face is
a simplex.
Each point y on the face has a unique convex combination
by the vertices of the simplex.
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
(k − 1)-face of
k-neighborly polytope P = Ax
aj
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
k-simplex
Application
y = Ax
Mixture models
Model space
Summary
ak
ai
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Equivalent properties
[Donoho and Tanner, 2005]
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
Let A ∈ Rd ×n , d < n. The two properties of A are equivalent
1
The polytope P is outwardly k -neighborly.
2
Whenever y = Ax has a nonnegative solution x0 having at
most k nonzeros, x0 is unique nonnegative solution to ℓ1
minimization problem.
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Outline
Geometry in
the sparse
methods
1
Introduction
From basis pursuit to compressed sensing
2
Geometry in sparse methods
Why is the solution sparse?
How sparse the solution is ?
What is the role of randomness?
3
Application
Mixture models
Statistical model space
4
Summary
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Restricted isometry property
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
A ∈ Rd ×n obeys a RIP for sets of size k if
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
d
d
(1 − δk ) kβk2ℓ2 ≤ kAβk2ℓ2 ≤ (1 + δk ) kβk2ℓ2
n
n
for every k -sparse vector β.
RIP preserve all k and lower dimensional planes from Rn
to Rd .
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Randomness and RIP
[Baraniuk et al., 2008]
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
A obeys RIP for
k ≤ Cd / log(n/d + 1),
with high probability, when
A is random Gaussian matrix.
A is random binary matrix.
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
(5)
Outline
Geometry in
the sparse
methods
1
Introduction
From basis pursuit to compressed sensing
2
Geometry in sparse methods
Why is the solution sparse?
How sparse the solution is ?
What is the role of randomness?
3
Application
Mixture models
Statistical model space
4
Summary
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Local mixture model
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Given observed data X : x1 , · · · , xN with t1 , · · · , tn distinct
values.
Introduction
According to [Lindsay, 1995], embedding the likelihood of
mixture model in Rn with −1-affine geometry.
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
The likelihood of mixture model is
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
f (t ; θ, Q ) = f (t ; θ) +
n
X
βj sj (t ),
j =1
Application
Mixture models
Model space
where sj (t ) are the basis generated from data.
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
(6)
Robust local mixture model
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
sj (t ) can be chosen to be sparse for robustness of the
model.
For each ti in observed data,
f (ti ; θ, Q ) = f (ti ; θ) +
What is the role of
randomness?
Application
Mixture models
Model space
βj sj (ti )
(7)
j ∈Ωi
Why is the solution
sparse?
How sparse the
solution is ?
X
For example, one x = ti is added to X, it will not affect the
inference on βj , j < Ωi .
Sparse basis sj (t ) can be obtained by sparse methods.
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Outline
Geometry in
the sparse
methods
1
Introduction
From basis pursuit to compressed sensing
2
Geometry in sparse methods
Why is the solution sparse?
How sparse the solution is ?
What is the role of randomness?
3
Application
Mixture models
Statistical model space
4
Summary
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Statistical model space
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Consider a family of statistical models, Mn , whose
likelihood can be embedded in Rn space.
The true model locates in Rk , k ≪ n.
By random orthogonal matrix A with RIP, Mn can be
projected into Rm , where model sufficiency is kept.
What is the statistical inference property on such Rm ?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Outline
Geometry in
the sparse
methods
1
Introduction
From basis pursuit to compressed sensing
2
Geometry in sparse methods
Why is the solution sparse?
How sparse the solution is ?
What is the role of randomness?
3
Application
Mixture models
Statistical model space
4
Summary
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Summary
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
ℓ1 minimization.
Sparse solution in ℓ1 minimization.
Outwardly k -neighborly polytope.
RIP and randomness.
Application to robust mixture models.
Future work on statistical model space.
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Reference I
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
R. Baraniuk, M. Davenport, R. DeVore, and M. Wakin. A simple
proof of the restricted isometry property for random
matrices. Constructive Approximation, 28(3):253–263,
2008. ISSN 0176-4276.
S.S. Chen, D.L. Donoho, and M.A. Saunders. Atomic
decomposition by basis pursuit. SIAM journal on scientific
computing, 20(1):33–61, 1999. ISSN 1064-8275.
D.L. Donoho. Compressed sensing. Information Theory, IEEE
Transactions on, 52(4):1289–1306, 2006. ISSN 0018-9448.
D.L. Donoho and J. Tanner. Sparse nonnegative solution of
underdetermined linear equations by linear programming.
Proceedings of the National Academy of Sciences of the
United States of America, 102(27):9446, 2005.
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods
Reference II
Geometry in
the sparse
methods
Zhiyue Huang
and
Paul Marriott
Introduction
Basis Pursuit and
compressed sensing
Geometry in
sparse
methods
B.G. Lindsay. Mixture models: theory, geometry and
applications. In NSF-CBMS Regional Conference Series in
Probability and Statistics, pages 67–163. JSTOR, 1995.
Why is the solution
sparse?
How sparse the
solution is ?
What is the role of
randomness?
Application
Mixture models
Model space
Summary
References
Zhiyue Huang and Paul Marriott
Geometry in the sparse methods