Embedding Gestalt Laws
in Markov Random Fields
by Song-Chun Zhu
Purpose of the Paper
Proposes functions to measure Gestalt
features of shapes
Adapts [Zhu, Wu Mumford] FRAME method to
shapes
Exhibits effect of MRF model obtained by
putting these together.
Recall Gestalt Features
(à la [Lowe], and others)
Colinearity
Cocircularity
Proximity
Parallelism
Symmetry
Continuity
Closure
Familiarity
FRAME
[Zhu, Wu, Mumford]
F ilters
R
andom fields
A
nd
M aximum
E
ntropy
A general procedure for constructing
MRF models
Three Main Parts
Data
Learn MRF models from data
Test generative power of learned model
Elements of Data
A set of images representative of the chosen
application domain
An adequate collection of feature measures or
filters
The (marginal) statistics of applying the
feature measures or filters to the set of images
Data: Images
Zhu considers 22 animal shapes and their
horizontal flips
The resulting histograms are symmetric
More data can be obtained
But are there other effects?
Sample Animate Images
Contour-based Feature Measures
Goal is to be generic
But generic shape features are hard to find
φ1 = κ(s), the curvature
κ(s) = 0 implies the linelets on either side of Γ(s)
are colinear
φ2 = κ'(s), its derivative
κ'(s) = 0 implies three sequential linelets are cocircular
“Other contour-based shape filters can be defined in
the same way”
Zhu's Symmetry Function
Ψ(s) pairs linelets across medial axes
Defined and computed by minimizing an energy
functional constructed so that
Paired linelets are as close, parallel and symmetric as
possible, and
There are as few discontinuities as possible
Region-based Feature Measures
φ3(s) = dist(s, ψ(s))
Measures proximity of paired linelets across a region
φ4(s) = φ3'(s), the derivative
φ4(s) = 0 implies paired linelets are parallel
φ5(s) = φ'4(s) = φ3''(s)
φ5(s) = 0 implies paired linelets are symmetric
Another Possible Shape Feature
φ6(s) = 1 where ψ(s) is discontinuous
0 otherwise
Counts the number of “parts” a shape has
Can Gestalt “familiarity” be (statistically?)
measured?
The Statistic
The histogram of feature φ over curve Γ is
H(z; φk, Γ) = ∫δ(z-φk(s)) ds
δ is the Dirac function: mass 1 at 0, and 0
otherwise
μ(z; φk) denotes the average over all images
Zhu claims μ is a close estimation of the marginal
distribution of the “true distribution” over shape
space, assuming the total number of linelets is
small.
Statistical Observations
On 22 images and their flips
φ1 at scales 0, 1, 2
φ3
φ4
φ5
Construct a Model
Ω is the space of shapes
Φ is a finite subset of feature filters
We seek a probability distribution p on Ω
∫Ω p(Γ) dΓ = 1
(1)
That reproduces the statistics for all φ in
Ω
∫Ω p(Γ) δ(z-φ(s)) dΓ = μ(z; φ)
(2)
Construct a Model, 2
Idea: Choose the p with maximal entropy
Seems reasonable and fair, but is it really
the best target/energy function?
Lagrange multipliers and calculus of variations
lead to
p(Γ; Φ, Λ) = exp(–∑φЄΦ
∫ λ (z) H(φ, Γ, z) dz) / Z
φ
where Z is the usual normalizing factor
Λ = { λφ | φЄΦ }
It's a Gibbs Distribution
In other words, it has the form of a Gibbs
distribution, and therefore determines a
Markov Random Field (MRF) model.
Markov Chain Monte Carlo
Too hard to compute λ's and p analytically
Idea: Sample Ω according to the distribution
p, stochastically update Λ to update p, and
repeat until p reproduces all μ(z; φ) for φ Є
Φ
Monte Carlo because of random walk
Markov Chain in the nature of the loop
Markov Chain Monte Carlo, 2
From the sampling produce μ'(z; φ)
Same as μ(z; φ) except based on a random sample
of shape space
For the purposes of today's discussion, the details
are not important
For φ
ЄΦ
μ'(z; φ) = μ(z; φ)
Zhu et al. assume there exists a “true underlying
distribution”
The Nonaccidental Statistic
For φ' not in the set Φ we expect
μ'(z; φ') ≠ μ(z; φ')
μ'(z; φ') is the accidental statistic for φ'
It is a measure of correlation between φ' and Φ
The “distance” (L1, L2, or other) between
μ'(z; φ') and μ(z; φ') is the nonaccidental
statistic for φ'
It is a measure of how much “additional
information” φ' carries above what is already in Φ
The Algorithm (simplified)
Enter your set Γ = { γ } of shapes
Enter a (large) set { φ } of candidate feature
measures
Compute μ(φ, Γ) for all φ in Φ
Compute μ'(φ) relative to a uniform
distribution on Ω
Until the nonaccidental statistic of all unused
features is small enough, repeat:
Algorithm, 2
Of the remaining φ , add to Φ one with maximal
nonaccidental statistic
Update:
Set of Lagrange multipliers Λ = { λ }
Probability distribution model p(Φ, Λ)
The μ'(φ) for remaining candidate features φ
Experiments and Discussion
Let my description of these experiments
stimulate your thoughts on such issues as
Are there better Gestalt feature measures?
What is the best possible outcome of a generative
model of shape?
What feature measures should be added to the
Gestalt ones?
How useful were these experiments and what
other might be worth doing?
Experiment 1
When the only feature used is the curvature
the model generated
Experiment 1, continued
A Gaussian model (with the same κ-variance)
produced
Experiment 2
Experiment 2 uses both κ and κ'
The nonaccidental statistic of κ' with respect to the
model based on κ can be seen here
Experiment 2, continued
This time the model generated these shapes,
purported to be smoother and more scale invariant
Experiment 3
The nonaccidental statistics of the three regionbased shape features relative to the model
produced in Experiment 2
Experiment 3, continued
So r'' was omitted, this model has
Φ = { κ, κ', r, r' }
Experiment 3, continued
This model produced such shapes as
Concluding Discussion
Zhu acknowledges that the selection of training
shapes might introduce a bias; but
Discussion, continued
Zhu acknowledges that the paucity of Gestalt
features limits the possible neighborhood
structures used to define a MRF.
Zhu acknowledges that these models do not
account for high-level shape properties, and
suggests that a composition system might
address this problem.
Questions and Comments
Although it is in the nature of an MRF-model to
propagate local properties, I think there needs
to be a higher-level basis (than linelets) for
measuring the Gestalt features of a shape!
Are there better Gestalt feature measures?
What feature measures should be added to the
Gestalt ones?
More Questions for Discussion
What is the best possible outcome of a generative
model of shape? Is such a thing worth pursuing?
How useful were Zhu' experiments and what
others might be worth doing?