Shape Representation and Similarity
Longin Jan Latecki
Computer and Information Sciences Dept., Temple Univ.,
latecki@temple.edu
With contributions from: Rolf Lakaemper, Zygmunt Pizlo, Eamonn Keogh, Remco Veltkamp
Based on IS&T/SPIE EI2006 Short Course #SC754; Jan 2006
Overview
1. The role of shape in object recognition
2. Example applications
3. What is shape?
4. Shape similarity measures
5. Comparison based on MPEG-7 data set
6. Contour-based shape representation
7. Partial shape similarity
Why Shape ?
Why Shape ?
•
Shape is probably the most important property
that is perceived about objects. It allows to
predict more facts about an object than other
features, e.g. color (Palmer 1999)
•
Thus, recognizing shape is crucial for object
recognition. In some applications it may be the
only feature present, e.g. logo recognition
Why Shape ?
These objects are recognized by…
Why Shape ?
These objects are recognized by…
Texture
Color
X
X
Context
X
Shape
X
X
X
X
X
X
Object Recognition Process:
Source:
2D image of a 3D object
Object Segmentation
Contour Extraction
Contour Cleaning, e.g.,
Evolution
Contour Segmentation
Matching: Correspondence
of Visual Parts
Why Shape ?
Several applications use shape processing:
• Object recognition
• Object comparison
• Image retrieval
• Processing of pictorial information
• Video compression (e.g., MPEG-7)
…
0
200
400
600
800
Courtesy of Eamonn Keogh, UCR
1000
1200
1400
All these are in the genus Cercopithecus,
except for the skull identified as being
either a Vervet or Green monkey, both of
which belong in the Genus of Chlorocebus
which is in the same Tribe (Cercopithecini)
as Cercopithecus.
Tribe Cercopithecini
Cercopithecus
De Brazza's Monkey, Cercopithecus neglectus
Moustached Guenon, Cercopithecus cephus
Red-tailed Monkey, Cercopithecus ascanius
Chlorocebus
Green Monkey, Chlorocebus sabaceus
Vervet Monkey, Chlorocebus pygerythrus
These are in the same species
Bunopithecus hooloc (Hoolock
Gibbon)
All these are in the tribe Papionini
Tribe Papionini
Genus Papio – baboons
Genus Mandrillus- Mandrill
These are in the Genus Pongo
These are in the family Lemuridae
All these are in the family Cebidae
Family Cebidae (New World monkeys)
Subfamily Aotinae
Aotus trivirgatus
Subfamily Pitheciinae sakis
Black Bearded Saki, Chiropotes satanas
White-nosed Saki, Chiropotes albinasus
These are in the genus Alouatta
These are in the same species
Homo sapiens (Humans)
Note: This is a phenogram, not a phylogenetic tree. For the current best phylogenetic tree, see
the Tree of Life project http://tolweb.org/tree/phylogeny.html
Lowland Gorilla
(Gorilla gorilla graueri)
Mountain Gorilla
(Gorilla gorilla beringei)
Application: Human Activity Recognition
Trajectory
Rao and M. Shah,
University of Central Florida
Activity Recognition: Typical Set of Trajectories
Another Shape Application
A, B: one individual tall male
(bounce in B is truncated)
C: relatively short female with a
radically different style
• Notice that DTW is forced to
map A’s bounce section to
the end of sequence B
• MVM is free to ignore
sections that do not have
natural correspondence
• Hence, MVM is able to
produce the more natural
clustering
Shape-Based Retrieval of Vertebra
Sameer Antani, Daniel M. Krainak, et al.
User text
Query
Shape
User sketch
Shape database
search, compare,
& retrieve similar
segment
Results
User example image
Query


Provide an image
example
Extract shape with
the same process as
in the original
segmentation
Retrieval results ranked by shape
similarity
Retrieval in multimedia databases
Query by Shape / Texture / …(Color / Keyword)
Blobworld: http://elib.cs.berkeley.edu/photos/blobworld/start.html
Query by Shape / Texture / Location / Color
Selected Blob
Query:
by Color and
Texture of
Blob
Result:
Blobs with
similar Color
and Texture
Satisfying ?
Blobworld
Selected Blob
Query:
by Shape of
Blob
Result:
…are these
shapes
similar ?
Satisfying ?
What is Shape ?
What is Shape ?
Plato, "Meno", 380 BC:
• "figure is the only existing thing that
is found always following color“
• "figure is limit of solid"
What is Shape ?
Shape is not only perceived by visual means:


tactical sensors can also provide shape
information that are processed in a similar
way.
robots’ range sensor provide shape
information, too.
What is Shape ?
Typical problems:
• How to describe shape ?
• What is the matching
transformation?
• No one-to-one correspondence
• Occlusion
• Noise
What is Shape ?
… let’s start with some properties easier
to agree on:
• Shape describes a spatial region
Shape is a (the ?) specific part of spatial cognition
• Typically addresses 2D space
why ?
What is Shape ?
• the original 3D (?) object
What is Shape ?
• 3D => 2D projection
What is Shape ?
Moving on from the naive
understanding, some questions arise:
• Is there a maximum size for a shape to be a
shape?
• Can a shape have holes?
• Does shape always describe a connected
region?
• How to deal with/represent partial shapes
(occlusion / partial match) ?
What is Shape ?
Shape or Not ?
Continuous transformation from shape to no
shape: Is there a point when it stops being
a shape?
What is Shape ?
Shape or Not ?
Continuous transformation from shape to two
shapes: Is there a point when it stops being
a single shape?
What is Shape ?
But there’s no doubt that
a single, connected region
is a shape.
Right ?
What is Shape ?
A single, connected region.
But a shape ?
A question of scale !
What is Shape ?
• There’s no easy, single definition of shape
•
In difference to geometry, arbitrary shape is not
covered by an axiomatic system
• Different applications in object recognition
focus on different shape related features
•
Special shapes can be handled
• Typically, applications in object recognition
employ a similarity measure to determine a
plausibility that two shapes correspond to
each other
Shape Similarity
So the new question is:
What is Shape Similarity ?
or
How to Define a Similarity Measure?
Shape Similarity
which similarity measure,
depends on
which required properties,
depends on
which particular matching problem,
depends on
which application.
However, there still may exist a single
universal measure that we have build
in our heads.
Shape Similarity
…which application
Simple Recognition (yes / no)
... robustness
Common Rating (best of ...)
Analytical Rating (best of, but...)
... invariance to basic transformations
Shape Similarity
…which problem
• computation problem: d(A,B)
• decision problem: d(A,B) <e ?
• decision problem: is there g: d(g(A),B) <e ?
• optimization problem: find g: min d(g(A),B)
Similarity Measure
Requirements to a similarity measure
• Should not incorporate context
knowledge, thus computes generic shape
similarity
Similarity Measure
Requirements on a similarity measure
• Must be in accord with human perception
• Must be able to deal with noise
• Must be invariant with respect to basic
transformations
Scaling (or resolution)
Next:
Strategy
Rotation
Rigid / non-rigid deformation
Similarity Measure
Some other aspects worth consideration:
• Similarity of structure
• Similarity of area
Can all these aspects be expressed by a
single number?
Similarity Measure
Desired Properties of a Similarity Function d
(Basri et al. 1998)
• d should be a metric
• d should be continuous
• d should be invariant (to…)
Properties
Metric Properties
S set of patterns
Metric: d: S  S R satisfying
1. Nonnegativity: "x ,yS, d(x,y)≥0
2. Self-identity: "xS, d(x,x)=0
3. Uniqueness: d(x,y)=0 implies x=y
4. Symmetry: "x,yS, d(x,y)= d(y,x)
5. Triangle inequality: "x,y,zS,
d(x,z)d(x,y)+d(y,z)
• S with fixed metric d is called metric space
Properties
Properties
Properties
In general:
• a similarity measure in accordance with
human perception is NOT a metric. This
leads to deep problems in further
processing, e.g. clustering, since most of
these algorithms need metric spaces !
Properties
Properties
Similarity Measures
Classes of Similarity Measures:
Similarity Measure depends on
• Shape Representation
• Boundary
• Area (discrete: = point set)
• Structural (e.g. Skeleton)
• Comparison Model
• feature vector, also called shape signature
• direct
Similarity Measures
Boundary
Area (point set)
direct
feature based
Spring model, Cum.
Angular Function,
Chain Code, Arc
Decomposition
Central Dist., Fourier
Hausdorff
Moments
…
Zernike Moments
Distance histogram
…
…
Structure
Skeleton
…
---
MPEG-7
• We focus on measures evaluated on
MPEG-7 CE-Shape-1 data set.
K. B. Sun and B. J. Super. Classification of Contour Shapes ... CVPR 2005.
MPEG-7
Retrieval Rate in Part B of the MPEG-7 CE-Shape-1 data set:
Each shape is as a query, and the retrieval rate is expressed by
Bull’s Eye Percentage:
the fraction of images that belong to the same class in the top 40
matches.
Strong shape variations within the same classes imply that no
shape similarity measure achieves a 100% retrieval rate.
Retrieval results obtained on the MPEG-7 CE-Shape-1 Part B data set
Improving shape retrieval
by semi-supervised
learning with graph
transduction
Courtesy Remco C. Veltkamp
Contour-Based Similarity Measures
•We focus on similarity measures based
on contour representation.
Contour is a 1D manifold. It is mapped to a
1D function from arc length to R:
• Distance to the centroid
• Tangent direction
• Curvature
Comparison of object shape based on object contours
Object contours are naturally obtained
in Computer Vision, Robot Navigation, and other
applications as polylines (polygonal curves).

Shape similarity reduces to similarity of polylines.
Shape similarity of polylines is not so simple:
•simple 1-1 vertex correspondence does not work
•a scale problem
Cognitive Similarity Requirements
Since polylines are obtained as boundary parts of objects
in usually noisy sensor data (e.g., digital images):
1. two similar polylines do not need to have the same
number of vertices,
i.e., do not have to be of comparable level of detail,
2. do not have to be of comparable size,
3. may have only a subpart that is similar and that has a
significant contribution to their shape = visual part
Vector
Contour
Comparison
Representation
It requires that the object is segmented or edges are extracted.
Contour is given as list of
Euclidean coordinates:
0,0; 1,0; 2,0; 2,1; 2,2; 3,3; …
Vector Chain
Comparison
Code
A binary image can be converted into a ‘chain code’
representing the boundary. The boundary is traversed
and a string representing the turns is constructed.
3
2
1
4
C
0
5
6
7
5,6,6,3,3,4,3,2,3,4,5,3,…
Vector Chain
Comparison
Code
Digital curves suffer from effects caused by
digitalization, e.g. rotation:
Distance to the centroid as function of angle or arc length
0
200
400
600
800
1000
1200
Skull classification by Eamonn Keogh
is based on distance to the centroid representation.
1400
Distance Histogram, example for a 3D surface
Tangent or Turn Angle Space
Transformation from image space to tangent or turn angle space
The angle with x-axis of the tangent to the contour at each contour
point s is denoted Ψ(s), where s is arc length normalized.
angle with x-axis of the tangent
arc length of the contour
Shape Similarity Measure: Arkin at al. PAMI 1991.
Shape Comparison: Measure
Drawback: not adaptive to unequally distributed noise
Shape Comparison: Contour Segmentation
Solution: use this measure only locally,
i.e., apply only to corresponding parts:
Shape Similarity
Correspondence of visual parts: Results
Visual parts = max. convex arcs obtained by
Discrete Curve Evolution (DCE)
Visual Parts Play a Key Role in Human Shape Perception
Divide a plane curve into parts at
negative minima of curvature
D. D. Hoffman and W. A. Richards (1984) Parts of Recognition, Cognition 18,
65–96, 1984.
D. D. Hoffman and M. Singh. (1997) Salience of Visual Parts. Cognition 63, p.
29-78, 1997.
K. Siddiqi, K. J. Tresness, and B. B. Kimia. Parts of visual form: Ecological
and psychophysical aspects. Perception 25, p. 399-424, 1996.
Correspondence of visual parts: non-rigid deformation
L. J. Latecki and R. Lakämper: Shape Similarity Measure Based on
Correspondence of Visual Parts.
IEEE Trans. Pattern Analysis and Machine Intelligence 22, 2000.
ISS Database
ISS-Database
http://knight.cis.temple.edu/~shape
Rolf Lakaemper and Longin Jan Latecki
The Interface (JAVA – Applet)
The Sketchpad: Query by Shape
The First Guess: Different Shape - Classes
Selected shape defines query by shape – class
Result
ISS Database
ISS: Query by Shape / Texture
Sketch of
Shape
Query:
by Shape
only
Result:
Satisfying ?
Vector Comparison
Curvature
• Curvature is the rate of change of slope.
• It is also the reciprocal of the radius of the local-fit circle.
Experimental Results (cont.)
Whole Sequence Matching (faces datasets)

Converting shapes into time series in a Face dataset:

trace the head outline and the local turn angle is used to create
a 1-dimensional signal
Curvature Scale Space (CSS), included in MPEG7 standard
Mokhtarian, Abbasi, Kittler
CSS
Inflection points (curvature zero crossings)
CSS
•
•
•
Smooth (continuous) boundary curve using convolution with
an increasing Gaussian kernel.
Use the run length position of curvature zero-crossings on the
boundary as index set for each kernel size; creating the
‘Curvature Scale Space’.
The maxima of the CSS are used for shape representation
Similarity of shape is defined by difference between the
maxima of the CSS representation
CSS
•
Shape Similarity is reduced to correspondence
of points on the plane.
CSS
•
Similarity of shape is defined by difference
between the maxima of the CSS representation
CSS
Problems of CSS:
• Convex shapes don’t have inflection points
• Different shapes can have identical CSS !
Symmetry Axis Representation by Liu, Kohn and Geiger
Shape
Shape Axis
(SA)
SA-Tree
The idea goes back to Blum’s Medial Axis Transformation
Vector
PartialComparison
Shape Similarity
All similarity measures shown cannot deal with
occlusions, noise, segmentation errors, change
of point of view, and non-rigid deformations.
They are useful (and used) for specific
applications, but are not sufficient to deal with
these problems.
Solution: Part – based similarity !
Partial Shape Similarity
Partial shape similarity of
contours is needed for object
recognition!
Partial Shape Similarity
Partial shape similarity
Partial Shape Similarity
• Partial match:
only part of
query appears
in part of
database shape
Courtesy of R. Veltkamp
Partial Shape Similarity
Motivation



Once a significant visual part is recognized the whole
recognition process is strongly constrained in possible topdown object models.
(H1) object recognition is preceded by, and based on
recognition of visual parts.
(H2): contour extraction is driven by shape similarity
to a known shape.
What do you see?
With grouping constraints we can see (i.e., recognize the object).
Salient visual parts can influence the object recognition
(Singh and Hoffman 2001)
Salient visual parts can influence the object recognition
(Singh and Hoffman 2001)
Salient visual parts can influence the object recognition
(Singh and Hoffman 2001)
Partial Shape Similarity
Object contours


Psychophysical and neurophysiological studies
provide an abundance of evidence that contours of
objects are extracted in early processing stages of
human visual perception.
Contours play a central role in the Gestalt-theory.
Partial Shape Similarity
Contour detection is a difficult inverse problem

A given image could be produced by infinitely many possible 3D
scenes. In order to produce a unique, stable and accurate
interpretation, the visual system must use a priori constraints (see
Pizlo, 2001 for a review).


The solution is obtained by optimizing a cost function which consists
of two general terms:
1. how close the solution is to the visual data
2. how well the constraints are satisfied
Partial Shape Similarity
Challenges of Partial Shape Similarity
Given only a part (of a shape ), find similar shapes
(1) length problem,
(2) scale problem,
(3) distortion problem
Query Shape
Target Shape
Target Shape
Experimental results: 90% retrieval accuracy with single query part
bird:05.17
bird:05.17
bird:05.15
bird:05.18
lizzard:52.02
bird:05.14
bone:06.01
bone:06.01
bone:06.05
bone:06.04
bone:06.02
bone:06.03
cellphone:14.15
cellphone:14.15
cellphone:14.16
cellphone:14.17
cellphone:14.14
device0:23.02
crown:20.16
crown:20.16
crown:20.17
teddy:66.01
crown:20.15
crown:20.18
glas:42.13
glas:42.13
glas:42.16
glas:42.14
glas:42.17
glas:42.15