What Energy Functions Can
be Minimized Using
Graph Cuts?
Shai Bagon
Advanced Topics in Computer Vision
June 2010
What is an Energy Function?
Image
For a given
Segmentation:
problem:
Useful Energy function:
1. Good solution  Low energy
2.suggested
Tractablesolution
 Can be minimized
E
a number
-20
237
Families of Functions or Outline
• F2 submodular
• Non submodular
• F3
• Beyond F3
Foreground Selection
Let
yi – color of ith pixel
xi ϵ {0,1} BG/FG labels (variables)
Given BG/FG scribbles:
Pr(xi|yi)=How likely each pixel to be FG/BG
Pr(xm|xn)=Adjacent pixels should have same label
F2 energy:
E(x)=∑iEi(xi)+∑ijEij(xi,xj)
yi
xi
xm
xn
Submodular
Known concept from set-functions:
f x   f  y   f x  y   f x  y 
f 1, 0   f 0 ,1   f 1,1   f 0 , 0 
x, y  S
S 2
E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj), xi ϵ {0,1}
xj
0
1
0
A
B
1
C
D
xi
Eij(xi,xj):
What does it
mean?
B+C-A-D ≥ 0
F2 submodular
How to
Minimize?
E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj), xi ϵ {0,1}
Local “beliefs”:
Prior knowledge:
Data term
Smoothness term
Graph Partitioning
A weighted graph G=( V E w )
Special Nodes: s t
s-t cut: S  V , T  V , s S, t T
S T  f , S  T  V
s
Cost of a cut: Cut ( S , T )  wij
t
iS , jT
Nice property: 1:1 mapping
s-t cut ↔ {0,1}|V|-2
wij
Graph Partitioning - Energy
D-C
C-A
E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj)
s
Ej(1)
B+C-A-D
j
i
Eij(xi,xj)
xj
xi
0
0
1
A
B
Ei(0)
0
0
=A+
1
C
D
0 D-C
+
C-A C-A
t
0 B+C-A-D
+
0 D-C
0
0
Graph Partitioning - Energy
E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj)
st

C-A
s
Ej(1)
B+C-A-D
cut  binary assignment
j
i
0 v  S
cut
x v  cost 
1 v  T
cut  S , T
D-C
energy of assignment
Ei(0)
t
min cut  Energy min.

A
w
ij
i  S , j T

A  E j 1   E i 0   D  C  B  C  A  D

E x 
B=Eij(0,1)
Recap
F2 submodular:
E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj)
Eij(1,0)+Eij(0,1)≥Eij(0,0)+Eij(1,1)
Mapping from energy to graph partition
Min Energy = computing min-cut
Global optimum in poly time
for submodular functions!
Next…
Multi-label F2
E(x)=∑i Ei(xi) + ∑ij Eij(xi,xj) s.t. xi ϵ {1,…,L}
– Fusion moves: solving binary sub-problems
– Applications to stereo, stitching, segmentation…
Solve Binary problem:
xi=0
xi=1
●
Current
labeling
=
suggested
labeling
Fusion
“Alpha expansion”
Stereo matching
see http://vision.middlebury.edu/stereo/
Input:
Pairwise MRF
[Boykov et al. ‘01]
Ground truth
slide by Carsten Rother, ICCV’09
Panoramic stitching
slide by Carsten Rother, ICCV’09
Panoramic stitching
slide by Pushmeet Kohli, ICCV’09
AutoCollage
http://research.microsoft.com/en-us/um/cambridge/projects/autocollage/
[Rother et. al. Siggraph ‘05 ]
Next…
Multi-label F2
E(x)=∑i Ei(xi) + ∑ij Eij(xi,xj) s.t. xi ϵ {1,…,L}
– Fusion moves: solving binary sub-problems
– Applications to stereo, stitching, segmentation…
Non-submodular
Beyond pair-wise interactions: F3
Merging Regions
“edge” prob.
regions (Ncuts)
input image
j
i
pi
“weak” edge
pi – prob. of boundary being edge
GOAL: Find labeling
x
ϵ{0,1}
that
max:
min:
i
“strong” edge
Pr  x  

 p log p1  p  log 1  p 
i
i: x i 1
i
i: x i  0
i
i: x i  0
i
Taking -log
Merging Regions
 x  
  log
pi 
i: x i 1

  log
pi 

i
i: x i  0
pi 
i: x i  0
  log 1  p   log
i
  log
pi
i: x i  0
pi 
i: x i  0
i
C 
  log 1  p     log
log
i
pi
1Adding
 x i  and
C   wi xi
1  p isubtracting

the i
same number
def
w i  log
wi  0
pi

: wlog 0 p 
  x1p i

i
i
i: x i 1
pi  1
2

likely to
p log1 21 p likey to
i
i: x i  0
i
be an edge
be merged
Merging Regions
Solving for edges:
arg min x C   w i x i
i
Consistency constraints:
No “dangling” edge
J
x1
x2
x3
EJ
0
0
0
0
1
1
1
0
0
1
1
0
0
0
1
λ
xi
wi
No
E J  1  x1 
1  longer
x 2 x3
 x 2 x 3  1  x1  x 3
  1  x1 pair-wise:
F3
Minimization trick
 1  x1  x 2 x 3  min  z 1  x1  x 2  x 3  2 
z   0 ,1 
K
  x i  min  z
i 1
z 0 ,1

K
i 1
xi  K  1

Freedman D., Turek MW, Graph cuts with many pixel interactions:
theory and applications to shape modeling. Image Vision Computing 2010
Merging Regions
The resulting energy:
E x 


 
min
z
n
wn xn
l , m , n J
 z lmn   x l  x m  x n  1   1  x l  x n
+ Pair-wise
- Non submodular!
Quadratic Pseudo-Boolean Optimization
s
j
i
i
t
j
Kolmogorov V., Carsten R., Minimizing non-submodular functions with graph cuts – a review.
PAMI’07
Quadratic Pseudo-Boolean Optimization
+ All edges with positive capacities
- No constraint i  1  i
Labeling rule: 0

y i  1
f

partial labeling
if i  S , i  T
s
i
j
i
j
if i  T , i  S
otherwise
t
Quadratic Pseudo-Boolean Optimization
Properties of partial labeling y:
1. Let z=FUSE(y,x) 
E(z)≤E(x)
2. y is subset of optimal y*
y is complete:
1. E submodular
2. Exists flipping
(inference in trees)
s
i
j
i
j
t
QBPO - Probing
QPBO:
0
p
?
q
?
r
?
s
?
t
?
Probe Node p:
0
0
0
0
?
p
q
r
s
t
?
1
0
p
q
0
r
1
s
0
t
?
What can we say about variables?
•r -> is always 0
•s -> is always equal to q
•t -> is 0 when q = 1
slide by Pushmeet Kohli, ICCV’09
QBPO - Probing
• Probe nodes in an order until energy
unchanged
• Simplified energy preserves global
optimality and (sometimes) gives the global
minimum
slide by Pushmeet Kohli, ICCV’09
Merging Regions
Result using QPBO-P:
input image
regions (Ncuts)
Result
Recap
• F3 and more
– Minimization trick
• Non submodular
– QPBO approx. – partial labeling
Beyond F3…
[Kohli et. al. CVPR ‘07, ‘08, PAMI ’08, IJCV ‘09]
Image Segmentation
n = number of pixels
E(X) =
∑ ci xi + ∑ dij |xi-xj|
i
Image
E: {0,1}n → R
0 →fg, 1→bg
i,j
Unary Cost
Segmentation
[Boykov and Jolly ‘ 01] [Blake et al. ‘04] [Rother et al.`04]
Pn Potts Potentials
Patch Dictionary
(Tree)
h(Xp) =
{
0
if xi = 0, i ϵ p
Cmax otherwise
Cmax  0
p
[slide credits: Kohli]
Pn Potts Potentials
n = number of pixels
E: {0,1}n → R
0 →fg, 1→bg
E(X) =
h(Xp) =
∑ ci xi + ∑ dij |xi-xj| + ∑
i
i,j
{
0
if xi = 0, i ϵ p
Cmax otherwise
p
hp (Xp)
p
[slide credits: Kohli]
Image Segmentation
n = number of pixels
E: {0,1}n → R
0 →fg, 1→bg
E(X) =
∑ ci xi + ∑ dij |xi-xj| + ∑
i
Image
i,j
Pairwise Segmentation
p
hp (Xp)
Final Segmentation
[slide credits: Kohli]
Application:
Recognition and Segmentation
Image
Unaries only
TextonBoost
[Shotton et al. ‘06]
One superpixelization
Pairwise CRF only
[Shotton et al. ‘06]
another superpixelization
Pn Potts
from [Kohli et al. ‘08]
Robust(soft) Pn Potts model
h(xp) =
{
0 if xi = 0, i ϵ p
f(∑xp) otherwise
p
Pn Potts
p
Robust Pn Potts
from [Kohli et al. ‘08]
Application:
Recognition and Segmentation
Image
Unaries only
TextonBoost
[Shotton et al. ‘06]
Pairwise CRF only
[Shotton et al. ‘06]
One superpixelization
Pn Potts
another superpixelization
robust Pn Potts
robust Pn Potts
(different f)
From [Kohli et al. ‘08]
Same idea for surface-based stereo
[Bleyer ‘10]
One input
image
Stereo with
Ground truth
hard-segmentation
depth
This approach gets best result on
Middlebury Teddy image-pair:
Stereo with
robust Pn Potts
How is it done…
Most general binary function:
H (X) = F ( ∑ xi ) concave
H (X)
0
∑ xi
The transformation is to a submodular pair-wise MRF, hence
optimization globally optimal
[slide credits: Kohli]
Higher order to Quadratic
• Start with Pn Potts model:
f(x) =
min f(x)
x
Higher Order
Function
∑xi = 0
∑xi > 0
{
=
0 if all xi = 0
C1 otherwise
x ϵ {0,1}n
min C1a + C1 (1-a) ∑xi
x,a ϵ {0,1}
Quadratic Submodular
Function
f(x) = 0
a=0
f(x) = C1
a=1
[slide credits: Kohli]
Higher order to Quadratic
min C1a + C1 (1-a) ∑xi
=
min f(x)
x
x,a ϵ {0,1}
Higher Order Function
Quadratic Submodular
Function
C1∑xi
C1
1
2
3
∑xi
[slide credits: Kohli]
Higher order to Quadratic
min C1a + C1 (1-a) ∑xi
=
min f(x)
x
x,a ϵ {0,1}
Higher Order
Submodular Function
Quadratic Submodular
Function
C1∑xi
a=1
a=0
Lower envelope
of concave
functions is
concave
C1
1
2
3
∑xi
[slide credits: Kohli]
Summary
• Submodular F2
• F3 and beyond: minimization trick
• Non submodular
– QPBO(P)
• Beyond F3 – Robust HOP
f2(x)
a=1
a=0
f1(x)
∑xi
s
i
j
i
j
t