Pixel Club, November, 2013
A Transform-based Variational
Framework
Guy Gilboa
In a Nutshell
Fourier inspiration:
𝐹
𝐿𝑃𝐹
𝐹 −1
Fourier Scale
Φ
𝐼
Fourier Scale
𝐻
𝑆
Spatial
Input
3000
2500
2500
2000
2000
1500
1500
1000
1000
500
0
𝑆𝐻
Transform
Analysis
3000
Φ−1
Transform
Filtering
Spectral
500
0
20
40
60
80
100
120
140
0
0
20
40
60
80
100
TV Scale
TV Scale
TV Flow
120
140
𝐼
Spatial
Output
Relations to eigenvalue problems
General linear:
(L linear operator)
 Functional based
Lu  u
J H 1   | u | dx
 div(u )  u  u
J TV   | u |dx
 u 
  u
 div
 | u | 

2


What can a transform-based
approach give us?
Scale analysis based on the spectrum.
 New types of filtering – otherwise hard
to design: nonlinear LPF, BPF, HPF.
 Nonlinear spectral theory – relation
to eigenfunctions and eigenvalues.
 Deeper understanding of the
regularization, optimal design with respect
to data, noise and artifacts.

Examples of spectral applications today:
Eigenfunctions for 3D processing
Taken from L Cai, F Da, “Nonrigid deformation recovery..”, 2012.
Taken from Zhang et al, “Spectral mesh processing”, 2010.
Image Segmentatoin
Eigenvectors of the graph Laplacian
[Taken from I. Tziakos et al, “Color image segmentation using Laplacian
eigenmaps”, 2009 ]
Some Related Studies








Andreu, Caselles, Belletini, Novaga et al 20012012– TV flow theory.
Steidl et al 2004 – Wavelet – TV relation
Brox-Weickert 2006 – scale through TV-flow
Luo-Aujol-Gousseau 2009 – local scale measures
Benning-Burger 2012 – ground states (nonlinear
spectral theory)
Szlam-Bresson – Cheeger cuts.
Meyer, Vese, Osher, Aujol, Chambolle, G. and
many more – structure-texture decomposition.
Chambolle-Pock 2011, Goldstein-Osher 2009 –
numerics.
Scale Space – a Natural Way to
Define Scale
Scale space as a gradient descent:
ut   p, u |t 0  f , p  u J (u)
We’ll talk specifically about total-variation
(TV-flow, Andreu et al - 2001):
 Du 
u
, in (0, )  
 div
t
 | Du | 
u
 0,
n
u (0; x)  f ( x),
on (0, )  
in x  
TV-Flow:
A behavior of a disk in time
[Andreu-Caselles et al–2001,2002, Bellettini-Caselles-Novaga-2002, Meyer-2001]
t
…
…
Center of disk, first and second time derivatives:
𝑢
𝑢𝑡𝑡
𝑢𝑡
2
0
0
−0.5
0
Spectral TV basic framework
Phi(t) definition
 (t; x)  utt (t; x)t
Reconstruction
Reconstruction formula

fˆ    (t ) dt  f
0
f 
1
f ( x)dx

||
Th. 1: The reconstruction formula
recovers 𝑓𝜖𝐵𝑉
Spectral response
Spectrum S(t) as a function of time t:
S (t )   (t; x)
1
L
  |  (t; x) | dx

f
S(t)
3000
2500
2000
1500
1000
500
0
0
20
40
60
t
80
100
120
140
Spectrum example
f
S(t)
4
2
x 10
1.8
1.6
1.4
S(t)
1.2
1
0.8
0.6
0.4
0.2
0
0
5
10
15
20
25
t
30
35
40
45
50
Dominant scales
4
2
x 10
1.8
1.6
1.4
S(t)
1.2
1
0.8
0.6
0.4
0.2
0
0
5
10
15
20
25
30
t
 (t  2; x)
 (t  10; x)
 (t  37; x)
35
40
45
50
Eigenvalue problem
The nonlinear eigenvalue problem with
respect to a functional J(u) is defined by:
p  u ,
p  J (u ),   
We’ll show a connection to the spectral
components  (t ) .
Solution of eigenfunctions
Th. 2: For 𝐽 𝑢 = |𝑢|𝐵𝑉 , if 𝑓𝜖𝐵𝑉 is an
eigenfunction with eigenvalue α then:

 f ( x)(1  t ),
u (t ; x)  
0,

1
 (t ; x)   (t  ) f ( x)

S (t )   (t 
1

) f ( x)
L1
0t 
t
1

1

What are the TV eigenfunctions?
In 2D, 𝐶 is a characteristic function of a convex set.
If sup𝑝∈𝜕𝐶 𝜅 𝑝 ≤
𝑃 𝐶
𝐶
then 𝐶 is an eigenfunction.
Perimeter
max(curvature on boundary) 
Area
1 𝑃 𝐶
2𝜋𝑟 2
𝜅 𝑝 = ;
= 2=
𝑟
𝐶
𝜋𝑟
𝑟
[Giusti-1978], [Finn-1979],[Alter-CasellesChambolle-2003].
Filtering
H(t)
𝜙(𝑡)
𝜙𝐻 (𝑡)
Let H(t) be a real-valued function of t.
The filtered spectral response is
H (t; x) :  (t; x) H (t )
The filtered spatial response is

f H ( x)   H (t; x)dt  f
0
Filtering, example 1:
TV Band-Pass and Band-Stop filters
f
S(t)
4
2.5
x 10
2
1.5
1
0.5
0
Band-pass
0
2
4
6
8
Band-stop
10
12
14
Disk band-pass example
S(t)
3000
2500
2000
1500
1000
500
0
0
20
40
60
80
100
120
140
We have the basic framework
Φ
𝐼
Spatial
Input
𝐻
𝑆
Φ−1
𝑆𝐻
Transform
Analysis
Transform
Filtering
 (t; x)  utt (t; x)t
Spatial
Output

f
H ( x)    H (t ; x) dt  f
0
S (t )   (t ; x)
1
L
3000
2500
2000
1500
1000
500
0
0
20
40
60
80
100
120
140
𝐼
Numerics


ut   p (u )
Many ways to solve.
Variational approach was chosen:
u (n  1)  u (n)  tp (u (n  1))  0
J (u ) 
2
1
||u  u (n)||L 2
2t
Currently use Chambolle’s projection algorithm
(some spikes using Split-Bregman, under
investigation).
 In time:

◦ 2nd derivative - central difference
◦ 1st derivative - forward differnce
◦ Discrete reconstruction algorithm proved for any
regularizing scale-space (Th. 4).
TV-Flow as a LPF
Th. 3: The solution of the TV-flow 𝑢(𝑡1 ) is
equivalent to spectral filtering with:
1
0.9
0.8
0.7
HTFV,t1
H TVF ,t1
0  t  t1
0,

  t  t1
 t , t1  t  
0.6
0.5
0.4
t1 = 1
t1 = 5
t1 = 10
t1 = 20
0.3
0.2
0.1
0
0
10
20
30
40
50
t
60
70
80
90
100
Nonlocal TV

Reminder: NL-TV (G.-Osher 2008):
Gradient
wu( x)  u( y)  u( x) w( x, y)
x, y  
Functional
J NL TV (u )   |  wu ( x) |dx

Spectral NL-TV?

The framework can fit in principle many
scale-spaces, like NL-TV flow. We can
obtain a one-homogeneous regularizer.
What is a generalized nonlocal disk?
 What are possible eigenfunctions?
 It is expected to be able to process
better repetitive textures and structures.

Sparseness in the TV sense
3000
Sparse spectrum – the signal has
only a few dominant scales.
 Can be a large objects
2500


Or many small ones
(here TV energy is large)
2000
1500
1000
500
0
0
20
40
60
80
100
120
140
S(t)
2500
2000
1500
1000
500
Natural images – are not
very sparse in general
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
30
35
40
45
50
t
4
2
x 10
1.8
1.6
1.4
1.2
S(t)

0
1
0.8
0.6
0.4
0.2
0
0
5
10
15
20
25
t
Noise Spectrum
S(t)
1400
1200
𝑆(𝑡)
1000
800
600
400
200
0
0
0.5
1
1.5
2
2.5
t
Various standard deviations:
S(t)
3
3.5
4
4.5
5
Noise + signal
12000
Signal
Noise
Signal + Noise
10000
8000
6000
4000

Not additive.
Spreads original image spectrum.

Needs to be investigated.

2000
0
0
5
10
15
20
25
t
Band-pass filtered
2500
f
u
Clean image
Noise only
Image with noise
2000
1500
1000
500
f-u
0
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Spectral Beltrami Flow?
Initial trials on Beltrami flow with parameterization such that it is closer to TV
Original
Beltrami Flow
Spectral Beltrami
Difference
images:
1.1
1
• Keeps sharp contrast
• Breaks extremum
principle
0.9
0.8
Original
0.7
Spectral Beltrami
0.6
Beltrami Flow
0.5
0.4
0
20
40
60
80
100
120
Values along one line (Green channel)
Segmentation priors
Swoboda-Schnorr 2013 –
convex segmentation with
histogram priors.
 We can have 2D spectrum with
histograms


Use it to improve segmentation
S(t,h)
Texture processing

Many texture bands
t i1
Band(i)    (t )dt
ti


i  1,2,..
ti  ti 1
We can filter and manipulate certain bands
and reconstruct a new image.
Generalization of structure-texture
decomposition.
Processing approach

Deconstruct the image into bands

Identify salient textures

Amplify / attenuate / spatial process the bands.

Reconstruct image with processed bands
Color formulation
Vectorial TV – all definitions can be
generalized in a straightforward
manner to vector-valued images.
 Bresson-Chan (2008) definition and
projection algorithm is used for the
numerics.

Orange example
Orange – close up
Original
Modes 2,3=0
Modes 2-5=x1.5
Selected phi(t) modes (1, 5, 15, 40)
f
residual
Old man
Old man – close up
Original
2 modes attenuated
7 modes attenuated
Old Man - First 3 Modes
Modes: 1
2
3
Take Home Messages

Introduction of a new TV transform
and TV spectrum.

Alternative way to understand and
visualize scales in the image.

Highly selective scale separation,
good for processing textures.

Can be generalized to other
functionals.
Thanks!
Refs. Google “Guy Gilboa publications”
• Preliminary ideas are in SSVM 2013 paper.
• Most material is in CCIT Tech report 803.
• Up-to-date and organized - submitted journal version –
contact me.