Is this a
Building or a
Horse?
Do these edges
and contours
represent
anything?
Top-Down & Bottom-Up
Segmentation
Presented By:
Joseph Djugash
What’s wrong with Segmentation
from Image Statistics?

Is this an object boundary?
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Not all Image Statistics are Helpful!
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
How can Class Information help?

Where is the object boundary?
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
The Class can help resolve ambiguities!
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Motivation

Bottom-Up segmentation
Capture image properties
 Segmentation based on similarities between
image regions


How can we capture prior knowledge of a
specific object (class)?

Answer: Top-Down Segmentation
Class-Specific, Top-Down
Segmentation
Eran Borenstein and Shimon Ullman
Method
Input
Fragments
Matching
Cover
Method Outline

Fragment Extraction
Figure Ground Label
 Reliability Value


Fragment Matching
Individual Correspondences
 Consistency
 Reliability


Segmentation

Optimal Cover
Fragment Extraction

Want to find fragments that:




Generalize well
Are specific to the class
Add information that other fragments haven’t already
given us
Fragment Size varies from 1/50 to 1/7 of object size
Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.
Fragment Extraction

C (u, v) 
  T ( x, y)I ( x  u, y  v)
  I ( x  u , y  v)
x
y
2
x
y
Figure-ground label
Manual labeling
 Learned from relative motion or grey level
Strength of Response –
Maximal normalized correlation
variability


of a fragment i with each image I
in C and NC
Reliability Value – Class Specific
Hit rate:
 A fixed level of false alarms  is achieved by
the criterion:
 Select the k best fragments according to the
Hit rate

Method Outline

Fragment Extraction
Figure Ground Label
 Reliability Value


Fragment Matching
Individual Correspondences
 Consistency
 Reliability


Segmentation

Optimal Cover
Fragment Matching –
Individual Correspondences

Measuring Similarity

Region Correlation
 Normalized
Correlation
 Restrict to pixels with the “figure” label

Edge Information
 Derived
from the boundary of the
figure-ground label

The Similarity Measure:
Fragment Matching –
Special Requirements
The Difference
Input Template
Entire Template
Figure Part Only
Figure & Edge Similarity
Fragment Matching –
Consistency


To acquire a good global cover of the shape,
each local match needs to satisfy the
consistency measure.
Consistency Measure:
Fragment Matching –
Reliability
Using more “reliable” (anchor) fragments is
likely to increase the chance of finding the
optimal cover
 A fragment’s reliability is evaluated by the
likelihood ratio between the hit/detection
rate and the false alarm rate


Reliable fragments used first to guide the
covering process
Fragment Matching –
Reliability
Problem Areas –
Do not exactly follow
image discontinuities
Easy to identify and
more commonly seen
fragments used first
Method Outline

Fragment Extraction
Figure Ground Label
 Reliability Value


Fragment Matching
Individual Correspondences
 Consistency
 Reliability


Segmentation

Optimal Cover
Segmentation –
The Cover Algorithm
The best cover should maximize individual
match quality, consistency and reliability
 Thus the cover score is written:

Rewards for
match quality
and reliability
Zero for non-overlapping pairs
Penalizes for
inconsistent
overlapping
fragments
Constant that determines the
magnitude of the penalty for
insufficient consistency
Segmentation –
The Cover Algorithm

Initialize with a sub-window that has the
maximal concentration of reliable
fragments


Similarity of all the reliable fragments is
examined at 5 scales at all possible locations
Iterative Algorithm:
Select a small number (M=15) of good
candidate fragments
 Add to cover a subset of the M fragments that
maximally improve the score
 Remove existing fragments inconsistent with
new cover (fragments with cumulative
negative score)

Results I
Results I (cont.)
Results I (cont.)
Learning to Segment
Eran Borenstein and Shimon Ullman
Method – Old
Input
Fragments
Matching
Cover
Method – Updated
Learning Figure-Ground
Segmentation – Degree of Cover
Start with over-segmented fragments –
each fragment now contains many regions
 Degree of Cover (ri)



Calculated by counting the average number of
fragments (from C) overlapping the region Ri
The fragment selection method extracts
most fragments from the figure region
Higher ri  higher likelihood to be “figure”
 Lower ri  lower likelihood to be “background”

Learning Figure-Ground
Segmentation – Degree of Cover
Most likely
figure region

By thresholding the degree of cover, ri, we can
choose the figure part to be:
Learning Figure-Ground
Segmentation – Border Consistency
A fragment often contains multiple edges
 Determine the boundary that optimally
separates figure from the background
 Fragment hit (Hj={1,n}) – image patches
where fragment Fi is detected
 Border Consistency:

Learning Figure-Ground
Segmentation – Border Consistency

This approach emphasizes consistent edges (border and
interior edges) while diffusing noise edges (background
features).
Learning Figure-Ground
Segmentation

Combining degree of cover and boarder
consistency we get the figure part (P)


Maximized when P contains the
most of the consistent edges 

Fragments detected in an image
applies its figure-ground “vote” for all
the pixels it covers
Li(x,y) = +1 – vote for figure label
Maximized when the boundary
Li(x,y) = –1 – vote for background label
between the figure and ground are
i supported
w(i) Li(x,y) by
– total
votes for pixel
the consistent
edges
(x,y)
Reliability of fragment i
Improving Figure-Ground Labeling
Fragments that are not consistent with the
cover (S) is removed and a new cover (S')
is generated
 Further Refinements:

Modify the degree of cover to be the average
number of times its pixels cover figure parts
 With a more accurate degree of cover,
individual pixels can be substituted for the
sub-regions
 This new degree of cover can them produce
an improved cover

Results II
Results II (cont.)
Results II (cont.)
Results II (cont.)
Bottom-Up Segmentation
“Segmentation and Boundary Detection
Using Multiscale Intensity Measurements”
Eitan Sharon, Achi Brandt, and Ronen Basri
Segmentation by Weighted
Aggregation

Normalized-cuts measure in graphs


Detect segments that optimize a NCut
measure
Hierarchical Structure
Recursively coarsen a graph reflecting
similarities between intensities of neighboring
points
 Aggregates of pixels of increasing size are
gradually collected to form segments


Modify the graph to reflect the coarse
scale measurements based on computed
properties of the aggregates
Normalized-Cut Measure
E(S )   wij (ui  u j )2
i j
1 i  S
ui  
0 i  S
N (S )   wij uiu j
Minimize:
E (S )
( S ) 
N (S )
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Segmentation by Weighted
Aggregation

Normalized-cuts measure in graphs


Detect segments that optimize a NCut
measure
Hierarchical Structure
Recursively coarsen a graph reflecting
similarities between intensities of neighboring
points
 Aggregates of pixels of increasing size are
gradually collected to form segments


Modify the graph to reflect the coarse
scale measurements based on computed
properties of the aggregates
Bottom-Up
Segmentation
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Segmentation by Weighted
Aggregation

Normalized-cuts measure in graphs


Detect segments that optimize a NCut
measure
Hierarchical Structure
Recursively coarsen a graph reflecting
similarities between intensities of neighboring
points
 Aggregates of pixels of increasing size are
gradually collected to form segments


Modify the graph to reflect the coarse
scale measurements based on computed
properties of the aggregates
Full Texture – Lion Cub
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Full Texture – Polar Bear
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Full Texture - Zebra
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.
Benefits of the Hierarchical Structure
Able to detect regions that differ by fine as
well as coarse properties
 Accurate detection of individual object
boundaries
 Able to detect regions separated by weak,
yet consistent edges


By combining intensity difference with
measures of boundary integrity across
neighboring aggregates
Combining Top-Down and
Bottom-Up Segmentation
Eran Borenstein, Eitan Sharon and Shimon Ullman
Another step towards the middle
Top-Down
Bottom-Up
Some Definitions & Constraints
Measure of saliency h(i), hi є [0,1)
 A configuration vector s contains labels si
(1/-1) of all the segments (Si) in the tree
 The label si can be different from its
parent’s label s i –
 Cost function for a given s

Defines the weighted
edge between Si & Si–
Top-down term
Bottom-up term
Classification Costs
The terminal segments of the tree
determine the final classification
 The top-down term is defined as:

The saliency of a segment should restrict
its label (based on its parent’s label)
 The bottom-up term is defined as:

Minimizing the Costs –
Information Exchange in a Tree

Bottom-up message:
 Top-down
Cost of si = –1message:
Cost
Message
of si = +1
and s = x

from
and
si =
s –1
=x
Message
from si = +1
Min-cost Label:
Computed at each node – minimal of the
values is the selected label of node s in s
Minimal Cost if the region was
classified as background
Minimal Cost if the region was
classified as figure
Confidence Map

Evaluating the confidence of a region:

Causes of Uncertainty of Classification
Bottom-up uncertainty – regions where there
is no salient bottom-up segment matching the
top-down classification
 Top-down uncertainty – regions where the topdown classification is ambiguous (highly
variable shape regions)


The type of uncertainty and the confidence
values can be used to select appropriate
additional processing to improve
Results III
Results III (cont.)
Results III (cont.)
Results III (cont.)
Results III (cont.)
Results III (cont.)
Questions?
– Appendix –
Why Fragments?
Vs.

Image fragments make good features


Image fragments contain more information than
wavelets


especially when training data is limited
allows for simpler classifiers
Slides from theory
David Bradley, ”framework
Object Recognition with Informative
Features and Linear Classification”.
Information
for feature
– Appendix –
Intermediate complexity
Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.