Design and Perceptual Validation
of Performance Measures
for Salient Object Segmentation
Vida Movahedi, James H. Elder
Centre for Vision Research
York University, Canada
Evaluation of Salient Object Segmentation
Source: Berkeley Segmentation Dataset
2
Centre for Vision Research, York University
Evaluation of Salient Object Segmentation
How do we measure success?
3
Centre for Vision Research, York University
Existing literature
Salient object segmentation


[Liu07, Zhang07, Park07, Zhuang09, Achanta09, Pirnog09, …]
Evaluation of salient object segmentation algorithms


[Ge06,?]
Evaluation of segmentation algorithms


4
[Huang95, Zhang96, Martin01, Monteiro06, Goldmann08, Estrada09]
Centre for Vision Research, York University
Contributions

Analysis of previously suggested measures

Contour Mapping Measure (CM)

Order-preserving matching

A new dataset of salient objects (SOD)

Psychophysics experiments

Evaluation of above measures
Matching paradigm in Precision and Recall measures

5
Centre for Vision Research, York University
Evaluation measures in literature
Region-based error measures


Based on false positive/ false negative pixels

[Young05], [Ge06], [Goldmann08], ...
Boundary-based error measures


Based on distance between boundaries

[Huttenlocher93], [Monteiro06], ...
Mixed measures

6

Based on distance of misclassified pixels to the boundaries

[Young05], [Monteiro06], ...
Centre for Vision Research, York University
Region-based error measures
[Young05], [Ge06], [Goldmann08], ...
A and B two boundaries
RA the region corresponding to a boundary A and |RA| the
area of this region,


False Positives
RI ( A, B) 1 
RA  RB
RA  RB

| RA |  RA  RB
RA  RB
False Negatives

| RB |  RA  RB
RA  RB
Not sensitive to shape differences
7
Centre for Vision Research, York University
Boundary-based error measures
[Huang95],[Huttenlocher93], [Monteiro06], ...

A and B two boundaries
Distance of one point a on A from B is d B (a )  min d (a, b) 

Hausdorff distance:
HD ( A, B)  max max d B (a ), max d A (b)

Mean distance:
MD ( A, B)  mean mean d B (a ), mean d A (b)

a
8


bB
aA
aA
bB
bB


Not sensitive to shape differences
Centre for Vision Research, York University
Mixture error measures
[Young05], [Monteiro06], ...
Penalizing the over-detected and under-detected regions by
their distances to intersection

False Negatives
1
MM ( A, B) 
2 Ddiag
False Positives
 1 N fn
1

dA( pj ) 
N 
N fp
 fn j 1

d B ( qk ) 


k 1

N fp
Not sensitive to shape difference
9
Centre for Vision Research, York University
Another example
Different shapes with low errors
10
Centre for Vision Research, York University
Comparing two boundaries
B A
Small False
Negative Region
Small False
Positive Region



The two boundaries need to follow each other
Thus it is not sufficient to map points to the closest
point on the other boundary
The ordering of mapped points must be preserved
11
Centre for Vision Research, York University
Order-preserving Mapping

The order of mapped points on the two boundaries must be
monotonically non-decreasing.
If ai  bm , a j  bn and i  j then m  n

Allowing for different levels of detail:



12
One-to-one
Many-to-one
One-to-many
Centre for Vision Research, York University
Contour Mapping Measure

Given two contours A=a1a2..an and B=b1b2..bm,

Find the correct order-preserving mapping
Contour mapping error measure:
Average distance between matched pairs of points

Bimorphism [Tagare02]

Elastic Matching [Geiger95, Basri98, Sebastian03, ..]
13
Centre for Vision Research, York University
Contour Mapping Measure

A dynamic programming implementation to
find the optimum mapping

Closed contours  point indices are assigned cyclically

Based on string correction techniques [Maes90]

Complexity: O(nm log m)
if m<n and m, n points on two boundaries
14
Centre for Vision Research, York University
Contour Mapping Example
Ground Truth Boundary
Algorithm Boundary
Matched pairs shown as line segments
CM= average length of
line segments connecting
matched pairs
15
Centre for Vision Research, York University
Contour Mapping Measure

Order- preserving mapping avoids problems experienced
by other measures
16
Centre for Vision Research, York University
SOD: Salient Object Dataset




A dataset of salient objects
Based on Berkeley Segmentation Dataset (BSD) [Martin01]
300 images
7 subjects
1
1
1
1
1
Source: Berkeley Segmentation Dataset
17
Centre for Vision Research, York University
Available in SOD
Psychophysical experiments

Which error measure is closer to human judgements of
shape similarity?

9 subjects
5 error measures






18
Regional Intersection (RI)
Mean distance (MD)
Hausdorff distance (HD)
Mixed distance (MM)
Contour Mapping (CM)
Centre for Vision Research, York University
Psychophysical Experiments
Experiment 1 - SOD
Reference & test shapes all
from SOD
Experiment 2 - ALG
Reference from SOD,
test shapes algorithm-generated
Reference:
Human
segmentation
Reference:
Human
segmentation
Test cases:
Human
segmentations
19
Test cases:
Algorithmgenerated
Centre for Vision Research, York University
Agreement with Human Subjects

Human subject chooses Left or
Right

An error measure M also
chooses Left or Right, based on
their error w.r.t. the reference
shape

If M chooses the same as the
human, it is a case of agreement

Human-Human consistency:
defined based on agreement
between human subjects
20
Reference
Left
Centre for Vision Research, York University
Right
Psychophysical Experiments
Experiment 1- SOD
Reference & tests shapes all
from SOD
Experiment 2 - ALG
Reference from SOD,
test shapes algorithm-generated
RI: region intersection, MD: mean distance, HD: Hausdorff distance, MM: mixed measure, CM: contour mapping
21
Centre for Vision Research, York University
Precision and Recall measures

For algorithm boundary A and ground truth boundary B

Precision: proportion of true positives on A

Recall:

Martin’s PR (M-PR)[Martin04]


Minimum cost bipartite matching, cost proportional to distance
Estrada’s PR (E-PR)[Estrada09]


proportion of detected points on B
matched(A, B)
| A|
matched(B, A)

|B|

‘No intervening contours’ and ‘Same side’ constraints
Contour Mapping PR (CM-PR)

22
Order-preserving mapping
Centre for Vision Research, York University
Matching paradigm in Precision/Recall
Experiment 1- SOD
Reference & test shapes all
from SOD
23
Experiment 2 - ALG
Reference from SOD,
test shapes algorithm-generated
Centre for Vision Research, York University
Summary

Analysis of available measures for evaluation of salient object
segmentation algorithms

A new measure- contour mapping measure (CM)


A new dataset of salient objects


Dataset available online: http://elderlab.yorku.ca/SOD
Psychophysical Experiment


Code available online: http://elderlab.yorku.ca/ContourMapping
CM has a higher agreement with human subjects
Order-preserving matching paradigm in Precision/Recall
analysis

24
Code available online: http://elderlab.yorku.ca/ContourMapping
Centre for Vision Research, York University
Thank You!
25
Centre for Vision Research, York University