Extensions of submodularity and
their application in computer vision
Vladimir Kolmogorov
IST Austria
Heidelberg, 14 November 2014
Discrete Optimization in Computer Vision
• Minimize

• Known as
 MAP inference in a graphical model
- maximim a posteriori estimation
 Energy minimization
Ising model
• Submodular function
• Minimizing f : maxflow algorithm (“graph cuts”)
• In Microsoft Powerpoint 2010
Interactive
image segmentation
Potts model
• NP-hard problem (for k>2)
• Efficient approximation algorithms
 “alpha-expansion” [Boykov et al.’01]: 2-approximation
Object recognition
Partial optimality
• Partial labeling:
1
5
6
2
3
4
• Optimal if can be extended to a minimizer of
7
f
Partial optimality
• Strategy:
 construct function g over partial labelings
- k-submodular relaxation of f
 minimize g
1
5
6
2
3
4
7
Classes of problems
• submodular functions
• k-submodular functions
• ...
tractable
• Potts functions
• ...
NP-hard
Part I: Complete classification of tractable classes
• For finite-valued VCSP languages
Part II: Application of k-submodular functions
• Obtaining partial optimality
• Efficient algorithm for the Potts model
Valued Constraint Satisfaction Problem (VCSP)
• D: fixed set of labels, e.g. D = {a,b,c}
• Language G: a set of cost functions
• VCSP(G): class of functions that can be expressed
as a sum of functions from G with overlapping sets of vars
-
Goal: minimize this sum
• Complexity of G?
G  g , h 
example:
g ( x)
h( x, y, z )
G-instance:
f ( x )  g ( x 3 )  h ( x1 , x1 , x 2 )  h ( x 4 , x 3 , x 2 )
Classifications for finite-valued CSPs
Theorem
If G admits a binary symmetric fractional polymorphism
then it can be solved in polynomial time by
Basic LP relaxation (BLP) [Thapper,Živný FOCS’12], [K ICALP’13]
Otherwise G is NP-hard
[Thapper,Živný STOC’13]
• For CSPs classification still open [Feder-Vardi conjecture]
 Every G is either tractable or NP-hard
 CSP: contains functions
Submodular functions
4
3
2
1
0
New classes of functions
Useful classes
• Submodular functions
-
Pairwise functions: can be solved via maxflow (“graph cuts”)
Lots of applications in computer vision
• k-submodular functions
-
Partial optimality for functions of k-valued variables
This talk: efficient algorithm for Potts energy [Gridchyn, K ICCV’13]
Partial optimality
• Input: function
• Partial labeling is optimal if it can be extended
to a full optimal labeling
Partial optimality
• Input: function
• Partial labeling is optimal if it can be extended
to a full optimal labeling
• Can be viewed as a labeling
k-submodular relaxations
• Input: function
k-submodular relaxations
• Input: function
• Construct extension
which is k-submodular
• Minimize
Theorem:
Minimum of
partially optimal
k-submodularity
• Function
is k-submodular if
k-submodular relaxations
• Case k = 2 ([K’10,12])
 Bisubmodular relaxation
 Characterizes extensions of QPBO
• Case k > 2
 [Gridchyn,K ICCV’13] :
- efficient method for Potts energies
 [Wahlström SODA’14] :
- used for FPT algorithms
k-submodular relaxations for Potts energy
• k-submodular relaxation:
k-submodular relaxations for Potts energy
• k-submodular relaxation:
d(a,b) : tree metric
k-submodular relaxations for Potts energy
• k-submodular relaxation:
3
4
gi (·) : k-submodular relaxation of fi (·)
1.5
0
10
k-submodular relaxations for Potts energy
• Minimizing g : O(log k) maxflows
• Alternative approach: [Kovtun ’03,’04]
- Stronger than k-submodular relaxations (labels more)
- Can be solved by the same approach!
 complexity: k => O(log k) maxflows
- Part of “Reduce, Reuse, Recycle’’ [Alahari et al.’08,’10]
- Our tests for stereo: 50-93% labeled
 with 9x9 windows
- Speeds up alpha-expansion for unlabeled part
Tree Metrics
• [Felzenszwalb et al.’10]: O(log k) maxflows for
Tree Metrics
• [Felzenszwalb et al.’10]: O(log k) maxflows for
• [This work]: extension to more general unary terms
- new proof of correctness
Special case: Total Variation
• Convex unary terms
• Reduction to parametric maxflow [Hochbaum’01],
[Chambolle’05], [Darbon,Sigelle’05]
New condition: T-convexity
• Convexity for any pair of adjacent edges:
Algorithm: divide-and-conquer
• Pick edge (a,b)
• Compute
Algorithm: divide-and-conquer
• Pick edge (a,b)
• Compute
• Claim: g has a minimizer as shown below
• Solve two subproblems recursively
Achieving balanced splits
• For star graphs, all splits are unbalanced
• Solution [Felzenszwalb et al.’10]: insert a new short edge
- modify unary terms gi (·) accordingly
Algorithm illustration
• k = 7 labels:
2
3
1
4
7
5
6
1,2,3,4,5,6,7
2
3
1
4
7
5
6
Algorithm illustration
• k = 7 labels:
2
3
1
4
7
5
6
5,6,7
1,2,3,4
2
3
1
4
7
5
6
Algorithm illustration
• k = 7 labels:
1,2
5,6,7
3,4
Algorithm illustration
• k = 7 labels:
5,6
1,2
3,4
7
Algorithm illustration
• k = 7 labels:
1
5
6
2
3
“Kovtun labeling”
•
4
7
unlabeled part,
run alpha-expansion
maxflows
Stereo results
Proof of correctness (sketch)
Proof of correctness (sketch)
• For labeling x and edge (a,b) define
Proof of correctness (sketch)
• For labeling x and edge (a,b) define
Proof of correctness (sketch)
• Coarea formula:
Proof of correctness (sketch)
• Coarea formula:
Proof of correctness (sketch)
• Coarea formula:
Proof of correctness (sketch)
• Coarea formula:
Proof of correctness (sketch)
• Coarea formula:
Proof of correctness (sketch)
• Coarea formula:
• Equivalent problem: minimize
with
subject to consistency constraints
• Equivalent to independent minimizations of
- consistency holds automatically due to convexity of
Extension to trees
• Coarea formula:
Summary
Part I:
• Complete characterization of tractable finite-valued CSPs
Part II:
• k-submodular relaxations for partial optimality
• For Potts model:
-
cast Kovtun’s approach as k-submodular function minimization
O(log k) algorithm
generalized alg. of [Felzenswalb et al’10] for tree metrics
• Future work: k-submodular relaxations for other functions?
postdoc & PhD student
positions are available