Sparse & Redundant Representation
Modeling of Images
Problem Solving Session 2: Relaxation
Algorithms
Task 1:
Dantzig Selector
Matan Protter
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
2
Dantzig Selector Practicalities
Dantzig Selector : Reminder
• There are several ways to write the Dantzig selector
as linear programming
• We choose to write
• The Basis Pursuit says that we should solve:
2
min α 1 s.t. Dα − x 2 ≤ ε2
α
α = u − v s.t. u ≥ 0, v ≥ 0
(
min l T u + l T v
u,v
)
min α 1 s.t. DT (Dα − x) ≤ T
α
⎫
⎧
u≥0 v≥0
⎪
⎪
s.t. ⎨
and
⎬
⎪ − σλl ≤ D T x − Du + D v ≤ σλl⎪
⎩
⎭
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
(
.
• The Dantzig Selector suggests instead:
• And formulate the LP problem
1
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
∞
.
• Intuition: Shaping of the noise is done better this way.
)
• Note: this is a linear programming problem for the noisy case
as well.
4
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
3
Dantzig Selector Practicalities – Cont.
(
T
T
min f u + f v
u,v
)
Dantzig Selector Practicalities – Cont.
⎧
⎫
u≥0 v≥0
⎪
⎪
s.t. ⎨
and
⎬
⎪ − σλ f ≤ D T x − Du + D v ≤ σλ f ⎪
⎭
⎩
(
u,v
• We set f = 1 (a vector of ones)
• We re-write the second constraint
6
Dantzig Selector notes
(
)
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
5
Task 1: Implement the Dantzig Selector algorithm
• Use Matlab’s linprog function to solve this:
function [resX , resXproj] = Dantzig_Selector(D , X , param)
% Run Dantzig Selector
%
Plan the function to
% Inputs :
run on a set of signals
% D : dictionary
(each column of X)
% X : set of vectors to run on
% param : A struct containing the fields:
%
'lambda' - the relative weight of the L1 term
%
'noiseSig ' - the noise power
%
% Outputs :
% resX : Result signals
% resXproj : Result signals from sparsifying the coefficients
LINPROG Linear programming.
X = LINPROG(f,A,b) attempts to solve the linear programming problem:
T
min f x s.t. A x ≤ b
x
X = LINPROG(f,A,b,Aeq,beq) solves the problem above while additionally satisfying the
equality constraints Aeq*x = beq.
X = LINPROG(f,A,b,Aeq,beq,LB,UB) defines a set of lower and upper bounds on the
design variables, X, so that the solution is in the range LB <= X <= UB. Use empty
matrices for LB and UB if no bounds exist. Set LB(i) = -Inf if X(i) is unbounded below; set
UB(i) = Inf if X(i) is unbounded above.
X = LINPROG(f,A,b,Aeq,beq,LB,UB,X0) sets the starting point to X0. This option is only
available with the active-set algorithm. The default interior point algorithm will ignore any
non-empty starting point.
2
)
⎧
⎫
u≥0 v≥0
⎪
⎪
s.t. ⎨
and
⎬
⎪ − σλ f ≤ D T x − Du + D v ≤ σλ f ⎪
⎭
⎩
• Since α = u − v and u ≥ 0, v ≥ 0
we can compute α 1
as the sum of elements of u and v
• Therefore, we set f = 1 (a vector of ones)
⎛ DTD − DTD⎞ ⎛ u⎞ ⎛ DT x ⎞
⎟ + σλ
⎟⋅⎜ ⎟ ≤ ⎜
⎜
⎜ − D T D D T D ⎟ ⎜⎝ v ⎟⎠ ⎜ − D T x ⎟
⎠
⎝
⎠
⎝
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
T
min f u + f v
)
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
(
T
8
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
7
Dantzig Selector notes – Cont.
Dantzig Selector notes
• Implement for running on a set of signals at
once
• The unknown vector is
•
•
•
•
•
•
⎛ D T D − D T D ⎞ can be computed only
⎟
⎜
⎜ − D T D D T D ⎟ once
⎠
⎝
• Construct f = 1 and the upper and lower
bounds only once
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
10
⎛ u⎞
⎜⎜ ⎟⎟
⎝v⎠
The linear multiplier is f = 1 (of length 2n)
The lower bound on this vector is 0
The upper bound is 20 ⋅ 1
There are no equality constrains
The inequality constraints
⎛ DTD − DTD⎞ ⎛ u⎞ ⎛ DT x ⎞
⎟ + σλ
⎟⋅⎜ ⎟ ≤ ⎜
⎜
⎜ − D T D D T D ⎟ ⎜⎝ v ⎟⎠ ⎜ − D T x ⎟
⎠
⎝
⎠
⎝
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
9
Dantzig Selector Variant
Task 1: Implement the Dantzig Selector algorithm
• The algorithm may create coefficients with
small values.
• These atoms might not belong in the support
• So:
– Compute the Dantzig Selector estimate
– Eliminate coefficients with small (e.g., ≤ 10 −3 )
magnitudes
– Compute LS over the remaining support to obtain
a new estimate
– This is the resXproj in the function header
Raise your hand if you’ve got a question,
or if you are done
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
3
12
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
11
IRLS: Reminder
2
1
Dα − x 2 .
2
p
λ αp +
• Suppose we aim to solve min
α
p
• Notice that α p =
K
∑α
p
i
i=1
K
= ∑ αi
2−p
i=1
Task 2:
⋅ αi = α A ( α) α .
2
T
IRLS
• Thus, at the j-th iteration, freeze A(αj) and minimize the
quadratic problem (and then update A)
Iterative Re-weighted Least Squares
2
1
α
ˆj+1 = Argmin λα A ( α
ˆj ) α + Dα − x .2
2
α
T
• This algorithm is called IRLS (also FOCUSS).
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
14
IRLS algorithm
T
T
0
x =0
W =I
n
Iteration
(
n−1
x = 2λ ⋅ W
n := n + 1
)
T
+D D \p
(
)
−1
n
W = diag⎛⎜ Xn + ε ⎞⎟
⎝
⎠
T
p=D x
Add small ε
for stability
Initialization
n
No
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
4
13
Task 2: Implement the IRLS algorithm
• Pre-compute D D and D x
• For each signal (in Matlab’s notation)
0
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
x −x
n −1
2
2
function resX = IRLS(D , X , param)
% Run Iterative Reweighted Least-Squares
%
% Inputs :
% D : dictionary (normalized columns)
% X : set of vectors to run on
% param :
%
'lambda' the value of the parameter in the minimization
%
Plan the function to
% Outputs :
run
on a set of signals
% resX : The result vectors
(each column of X)
≤T
2
Yes
Done!
16
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
15
Task 2: Implement the IRLS algorithm
Task 3:
Synthetic
Experiments
Raise your hand if you’ve got a question,
or if you are done
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
18
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
Task 3: Compare the performance of the algorithms
Comparing the algorithms
• Download the package Sparse_ProblemSession02.zip
• Extract the files
• Understand and run the function
Synthetic_Tests
• You can use your implementations for the algorithms
or the ones provided
• We’ve implemented 4 algorithms:
17
– Greedy
• Thresholding
• OMP
– Relaxation
• Dantzig Selector
• IRLS
• Let’s compare their performance
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
5
20
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
19
Task 3: Compare the performance of the algorithms
Task 3: Compare the performance of the algorithms
• Feel free to change the parameters of the tests
– However, the optimal parameters hard-coded, and the
parameter ranges might need changing.
• Run on a small (~20) number of signals
– Run on more signals during the night
Raise your hand if you’ve got a question,
or if you are done
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
22
Task 3: Expected Results
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
6
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
21
Task 3: Expected Results
24
Sparse & Redundant Representation Modeling of Images
Problem Solving Session 1: Greedy Pursuit Algorithms
By: Matan Protter
23