Learning to Detect Natural Image
Boundaries Using Local Brightness,
Color and Texture Cues
by David R. Martin, Charless C. Fowlkes, Jitendra Malik
Heather Dunlop
16-721: Advanced Perception
January 25, 2006
What is a Boundary?
Canny
Human
Martin, 2002
Dataset
“You will be presented a photographic image. Divide the
image into some number of segments, where the segments
represent ‘things’ or ‘parts of things’ in the scene. The
number of segments is up to you, as it depends on the
image. Something between 2 and 30 is likely to be
appropriate. It is important that all of the segments have
approximately equal importance.”
Dataset
Database of over 1000 images and 5-10
segmentations for each
Martin, 2002
Boundaries
Non-boundaries
Boundaries
Intensity
Brightness
Color
Texture
Martin, 2002
Method
Goal: learn the probability of a boundary, Pb(x,y,θ)
Image
Optimized Cues
Brightness
Boundary Strength
Cue Combination
Color
Model
Texture
Benchmark
Human Segmentations
Martin, 2002
Image Features
CIE L*a*b* color space (luminance, red-green,
yellow-blue)
2
2
Oriented Energy: OE ,  I  fe,  I  fo,

fe: Gaussian second derivative
fo: Its Hilbert transform
 
Brightness
L* distribution
Color
a* and b* distributions (joint or marginal)
Texture

Texture
Convolve with a filter bank:
Gaussian second derivative
Its Hilbert transform
Difference of Gaussians
Filter responses give a measure of
texture
Other Filter Banks
Leung-Malik filter set:
Schmid filter set:
Maximum Response 8 filter set:
Textons
Convolve image with filter bank
Cluster filter responses to form textons
Adapted from Martin, 2002 and Varma, Zisserman, 2005
Texton Distribution
Assign each pixel to nearest texton
Form distribution of textons
Adapted from Martin, 2002 and Varma, Zisserman, 2005
Gradient-based Features
Brightness (BG), color (CG), texture
(TG) gradients
Half-disc regions described by
histograms
Compare distributions with
r
(x,y) 
2
χ statistic
2
(
g

h
)
1
 2 ( g , h)   i i
2 i g i  hi
Texture Gradient
Texton distribution in two half circles
Martin, 2002
Localization
Tightly localize
boundaries
Reduce noise
Coalesce double
detections
Improve OE and TG
features
OE
OE localized
TG
TG localized
Martin, Fowlkes, Malik, 2004
Optimization
Texture parameters:
type of filter bank
scale of filters
number of textons
universal or image-specific textons
Other possible distance/histogram
comparison metrics
Number of bins for histograms
Scale parameter for all cues
Evaluation Methodology
Posterior probability
of boundary:
Pb(x,y,θ)
Evaluation measure:
precision recall curve
F-measure:
F  PR R  (1   ) P 
  0.5
Martin, 2002
Cue Combination
Which cues should
be used?
OE is redundant
when other cues are
present
BG+CG+TG produces
best results
Martin, 2002
Classifiers
Until now, only logistic
regression was used
Other possible
classifiers:
Density estimation
Classification trees
Hierarchical mixtures of
experts
Support vector machines
Martin, 2002
Result Comparison
Alternative methods:
Matlab’s Canny edge
detector with and
without hysteresis
Spatially-averaged
second moment
matrix (2MM)
Martin, 2002
Image
Canny
Results
2MM
BG+CG+TG Human
Martin, 2002
Image
Canny
Results
2MM
BG+CG+TG Human
Martin, 2002
Image
Canny
Results
2MM
BG+CG+TG Human
Martin, 2002
Conclusions
Large data set used for testing
Texture gradients are a powerful cue
Simple linear model sufficient for cue
combination
Outperforms existing methods
An approach that is useful for higher-level
algorithms
Code is available online:
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/