Outline
• Image Segmentation by Data-Driven Markov
Chain Monte Carlo
– Z. Tu and S. C. Zhu, “Image segmentation by
data-driven Markov chain Monte Carlo,” IEEE
Transactions on Pattern Analysis and Machine
Intelligence, vol. 24, no. 5, 657-673, 2003
Image Segmentation
The process to decompose an image into its constituent regions
May 29, 2016
Computer Vision
2
Image Segmentation Motivation – cont.
May 29, 2016
Computer Vision
3
Image Segmentation
• Image Lattice:
• Image:
• For any point
either
or
• Lattice partition into K disjoint regions:
• Region is discrete label map:
• Region Boundary is Continuous:
May 29, 2016
Computer Vision
4
Bayesian Formulation
• Each Image Region
is a realization from a
probabilistic model
•
are parameters of model indexed by
• A segmentation is denoted by a vector of hidden
variables W; K is number of regions
• Bayesian Framework:
Posterior
May 29, 2016
Likelihood
Computer Vision
Prior
5
Prior Over Segmentations
Want fewer regions
Want regions with smooth boundaries
~ uniform
Want less complex models
Want large regions
•
May 29, 2016
Computer Vision
6
Likelihood for Images
• Visual Patterns are independent stochastic processes
Grayscale
•
is model-type index
•
•
is model parameter vector
is image appearance in of the ith region
May 29, 2016
Computer Vision
Color
7
Four Gray-level Models
Uniform
Gaussian
Clutter
Texture
Intensity
Histogram
FB Response
Histogram
Shading
B-Spline
• Uniform:
May 29, 2016
Computer Vision
8
Four Gray-level Models
Uniform
Gaussian
Clutter
Texture
Intensity
Histogram
FB Response
Histogram
Shading
B-Spline
• Clutter:
May 29, 2016
Computer Vision
9
Four Gray-level Models
Uniform
Gaussian
Clutter
Texture
Intensity
Histogram
FB Response
Histogram
Shading
B-Spline
• Texture:
May 29, 2016
Computer Vision
10
Four Gray-level Models
Uniform
Gaussian
Clutter
Texture
Intensity
Histogram
FB Response
Histogram
Shading
B-Spline
• Shading:
May 29, 2016
Computer Vision
11
Four Gray-level Models
Uniform
Gaussian
Clutter
Texture
Intensity
Histogram
FB Response
Histogram
Shading
B-Spline
• Gray-level model space:
May 29, 2016
Computer Vision
12
Calibration
• Likelihoods are calibrated using empirical study
• Calibration required to make likelihoods for different
models comparable (necessary for model competition)
May 29, 2016
Computer Vision
13
Calibration – cont.
May 29, 2016
Computer Vision
14
Three Color Models (L*,u*,v*)
• Gaussian
• Mixture of 2 Gaussians
• Bezier Spline
• Color model space:
May 29, 2016
Computer Vision
15
What do we do with scores?
Search
May 29, 2016
Computer Vision
16
Anatomy of Solution Space
• Space of all k-partitions
• General partition space
Scene
Space
Partition
space
• Space of all segmentations
May 29, 2016
or
Computer Vision
K Model
spaces
17
Anatomy of the Solution Space
May 29, 2016
Computer Vision
18
Searching Through Segmentations
Exhaustive Enumeration of all segmentations
Takes too long!
Greedy Search (Gradient Ascent)
Local minima!
Stochastic Search
Takes too long
MCMC based exploration
May 29, 2016
Computer Vision
19
Why MCMC
• What is it?
- A clever way of searching through a high-dimensional space
- A general purpose technique of generating samples from a probability
• What does it do?
- Iteratively searches through space of all segmentations by constructing
a Markov Chain which converges to stationary distribution
May 29, 2016
Computer Vision
20
Designing Markov Chains
• Three Markov Chain requirements
• Ergodic: from an initial segmentation W0,
any other state W can be visited in finite time
(no greedy algorithms); ensured by jumpdiffusion dynamics
• Aperiodic: ensured by random dynamics
• Detailed Balance: every move is reversible
 ( x ) P ( x, y )   ( y ) P ( y , x )
May 29, 2016
Computer Vision
23
5 Dynamics
1.) Boundary Diffusion
2.) Model Adaptation
3.) Split Region
4.) Merge Region
5.) Switch Region Model
At each iteration, we choose a dynamic with probability q(1),q(2),q(3),q(4),q(5)
May 29, 2016
Computer Vision
24
Dynamics 1: Boundary Diffusion
• Diffusion* within
Boundary
Between
Regions i and j
Temperature
Decreases over
Time
Brownian Motion
Along
Curve Normal
*Movement within partition space
May 29, 2016
Computer Vision
25
Dynamics 2: Model Adaptation
• Fit the parameters* of a region by steepest ascent
*Movement within cue space
May 29, 2016
Computer Vision
26
Dynamics 3-4: Split and Merge
• Split one region into two
Remaining
Variables
Are
unchanged
Probability of
Proposed Split
Conditional Probability
of how likely chain
proposes to move
to W’ from W
May 29, 2016
Data-Driven Speedup
Computer Vision
27
Dynamics 3-4: Split and Merge
Remaining
Variables
Are
unchanged
• Merge two Regions
Data-Driven Speedup
Probability of
Proposed Merge
May 29, 2016
Computer Vision
28
Dynamics 5: Model Switching
• Change models
• Proposal Probabilities
Data-Driven Speedup
May 29, 2016
Computer Vision
29
Motivation of DD
• Region Splitting: How to decide where to
split a region?
Vs.
• Model Switching: Once we switch to a new
model, what parameters do we jump to?
Model Adaptation Required some initial parameter vectors
May 29, 2016
Computer Vision
30
Data Driven Methods
• Focus on boundaries and model parameters
derived from data: compute these before
MCMC starts
• Cue Particles: Clustering in Model Space
• K-partition Particles: Edge Detection
• Particles Encode Probabilities Parzen
Window Style
May 29, 2016
Computer Vision
31
Cue Particles In Action
Clustering in Color Space
May 29, 2016
Computer Vision
33
Cue Particles In Action
May 29, 2016
Computer Vision
34
Cue Particles In Action
May 29, 2016
Computer Vision
35
Cue Particles In Action
May 29, 2016
Computer Vision
36
Cue Particles In Action
May 29, 2016
Computer Vision
37
Cue Particles In Action
May 29, 2016
Computer Vision
38
Cue Particles In Action
May 29, 2016
Computer Vision
39
Cue Particles In Action
May 29, 2016
Computer Vision
40
Edge Detection
May 29, 2016
Computer Vision
41
Edge Detection
May 29, 2016
Computer Vision
42
K-partition Particles in Action
• Edge detection gives us a good idea of where we
expect a boundary to be located
May 29, 2016
Computer Vision
44
Particles or Parzen Window* Locations?
• What is this particle
about?
• A particle is just the
position of a Parzenwindow which is used
for density estimation
1D particles
May 29, 2016
Computer Vision
46
Multiple Solutions
• MAP gives us one solution
• Output of MCMC sampling
How do we get multiple solutions?
Parzen Windows: Again
May 29, 2016
Scene Particles
Computer Vision
49
Why multiple solutions?
• Segmentation is often not the final stage of
computation
• A higher level task such as recognition can
utilize a segmentation
• We don’t want to make any hard decision
before recognition
• multiple segmentations = good idea
May 29, 2016
Computer Vision
50
K-adventurers
• We want to keep a fixed number K of segmentations
but we don’t want to keep trivially different
segmentations
• Goal: Keep the K segmentations that best preserve the
posterior probability in KL-sense
• Greedy Algorithm:
- Add new particle, remove worst particle
May 29, 2016
Computer Vision
51
Results (Multiple Solutions)
May 29, 2016
Computer Vision
52
Results
May 29, 2016
Computer Vision
53
Results (Color Images)
http://www.stat.ucla.edu/~ztu/DDMCMC/benchmark_color/benchmark_color.htm
May 29, 2016
Computer Vision
54
Conclusions
• DDMCMC: Combines generative (top-down)
and discriminative (bottom-up) approaches
• Traverse the space of segmentations via
Markov Chains
May 29, 2016
Computer Vision
55
References
• DDMCMC Paper:
http://www.cs.cmu.edu/~efros/courses/AP06/Papers/tu-pami-02.pdf
• DDMCMC Website:
http://www.stat.ucla.edu/%7Eztu/DDMCMC/DDMCMC_segmentation.htm
• MCMC Tutorial by Authors:
http://civs.stat.ucla.edu/MCMC/MCMC_tutorial.htm
May 29, 2016
Computer Vision
56