A Survey on
Shape Correspondence
O. van Kaick1, H. Zhang1, G. Hamarneh1, and D. Cohen-Or2
Simon Fraser University, Canada
2 Tel Aviv University, Israel
1
Correspondence
Webster dictionary definition
• the agreement of things with one another
• a particular similarity
• a relation between sets in which each member
of one set is associated with one or more
members of the other
Correspondence
Webster dictionary definition
• the agreement of things with one another
• a particular similarity
• a relation between sets in which each member
of one set is associated with one or more
members of the other
Focus on shapes
• Correspondence between shape representations
Problem statement
• Given n input shapes, search for a meaningful
relation R between their elements
Registration  correspondence
• Registration: to bring several (partial) shapes to
(almost) exact alignment, rigidly or non-rigidly
• Correspondence: no repeated geometry expected
Why is correspondence hard?
• Given n input shapes, search for a meaningful
relation R between their elements
Meaningful correspondence
• Appropriate notion of “similarity” or “agreement”
• Similarity: corresponding points/regions look
similar  local geometric similarity
• Agreement: close-by points should match close-by
points  proximity or distortion
Local similarity + distortion
Local similarity
Local similarity
Proximity or distortion
Local similarity + distortion
Not always so easy
• Local similarity + distortion often does not work
Correspondence by human
• Recognition is involved and that is hard!
Human recognition
• Can tolerate many shape variations
▫ Rigid transforms: e.g., rotation
▫ Isometric transform and pose
▫ Local geometric details
Human recognition
• More drastic shape variations
▫ Non-homogenous part scaling
▫ Even topological differences
• Human recognition is often
[Lipman et al. 2009]
▫ beyond individual components  in a context
▫ beyond what something looks like
 what it what it does  functionality
Computer algorithms
• Relatively strict geometry constraints
▫ Rigid transforms: e.g., rotation
▫ Isometric transform and pose
▫ Local geometric details
[Ovsjanikov et al. 2008]
Computer algorithms
• Challenging: more drastic variations (semantics?)
▫ Non-homogenous part scaling (what is a part?)
▫ Topological differences
[Lipman et al. 2009]
Challenge: objective
• Correspondence is a search problem
• Search objective can be hard to define
▫ Easy: relatively strict geometry constraints/criteria
▫ Hard: shape semantics, modeling of functionality
What is a
handle?
What is an arm-rest?
Challenge: search space
• Search space can be large to intractable
▫ Perhaps only rigid transforms are low-dim (6D)
▫ Difficult to parameterize search space for nonrigid or topology-modifying transforms
Partial correspondence
• Matching partial shapes  adds more challenge
▫ Larger search space: also
need to find the subsets
 many (2n) of them!
Partial correspondence
• Matching partial shapes  adds more challenge
▫ Harder to define objective:
small subsets = small cost,
but is that desired result ?
Not to be discouraged
• I am skipping ahead: this is a 90-min talk! 
• A LOT of interesting stuff even in rigid case!
▫ Well-formulated
▫ Challenges: efficiency, partial matching, …
Not to be discouraged
• More open: non-homogenous part scaling, topology
▫ Strong tie to part analysis
▫ Semantic analysis
▫ Use of prior knowledge
ShapeAnnotator [Attene et al. 09]
Overview
• Motivations: why shape correspondence?
• Problem and specialization/classification
• Representative approaches
• Validation
• Conclusion and future perspectives
Overview
• Motivations: why shape correspondence?
• Problem and specialization/classification
• Representative approaches
• Validation
• Conclusion and future perspectives
Motivating applications
• Registration
• Transfer
• Understanding
• Synthesis
Registration
Registration for reconstruction
Upcoming EG talk
Q. Zheng, A. Sharf, A. Tagliasacchi, B. Chen, H. Zhang, A. Sheffer,
and D. Cohen-Or, “Consensus skeleton for non-rigid space-time
registration”.
Thursday, 13:30 – 15:30, Concert Hall
Attribute transfer
Texture mapping [KSG03]
Understanding
• Recognition by corresponding query to models
tag: leg
tag: head
Part query
Matched parts from model database
Part analogy [Shapira et al. *09]
Synthesis
• Correspondence allows statistical modeling of
shape/part variations
Morphable faces [BV99]
Human modeling [ASK*05]
Overview
• Motivations: why shape correspondence?
• Problem and specialization/classification
• Representative approaches
• Validation
• Conclusion and future perspectives
Recall the problem
• Given n input shapes, search for a meaningful
relation R between their elements
Input representation
• Given n input shapes, search for a meaningful
relation R between their elements
Full mesh
Segments or
parts
Features
Skeleton
Input: 2D, 3D, 2D+time, 3D+time?
Input size
• Given n input shapes, search for a meaningful
relation R between their elements
Group correspondence: n > 2
Consensus skeleton
[Zheng et al. 2010]
Output: full vs. partial
• Given n input shapes, search for a meaningful
relation R between their elements
Full
Partial
Output: dense vs. sparse
• Given n input shapes, search for a meaningful
relation R between their elements
The hard problem
can be initialization
Output: transform or not?
• Given n input shapes, search for a meaningful
relation R between their elements
Correspondence only
rigid
isometric
non-rigid
topology
1-many
many-many
fuzzy
Correspondence +
aligning transformation
Objective
• Given n input shapes, search for a meaningful
relation R between their elements
Meaningful = local shape similarity and distortion
Objective
• Given n input shapes, search for a meaningful
relation R between their elements
Meaningful = good rigid alignment
Objective
• Given n input shapes, search for a meaningful
relation R between their elements
Meaningful = good non-rigid alignment
Objective
• Given n input shapes, search for a meaningful
relation R between their elements
Meaningful = agreement with human understanding
Overview
• Motivations: why shape correspondence?
• Problem and specialization/classification
• Representative approaches
• Validation
• Conclusion and future perspectives
Representative approaches
• Rigid transform
▫ ICP: iterative closest points
▫ Transformation search by voting
• Piece-wise rigidity and isometry
▫ Rigidity decomposition
▫ Spectral embedding
• Large non-rigid deformation
▫ Deformation-driven search
Representative approaches
• Rigid transform
▫ ICP: iterative closest points
▫ Transformation search by voting
• Piece-wise rigidity and isometry
▫ Rigidity decomposition
▫ Spectral embedding
• Large non-rigid deformation
▫ Deformation-driven search
Iterative closest point (ICP)
• One of the most classic correspondence schemes
• Input: data and model shapes
• Objective:
▫ Rigid transform = rotation + translation
▫ Minimize mean squared error from data points to
closest points in model
• Correspondence obtained by Euclidean proximity
[Besl and Mckay 92]
ICP algorithm
model
data
Model and data shapes (point samples)
[Besl and Mckay 92]
ICP algorithm
model
data
Find closest points from data to model
[Besl and Mckay 92]
ICP algorithm
model
data
Find best rigid transform to align the
corresponding points
[Besl and Mckay 92]
ICP algorithm
model
data
Iterate …
[Besl and Mckay 92]
Issues with ICP
• Many variants
[Rusinkiewicz and Levoy 01]
• Greedy: bad starts lead to bad local minima
• Usually, initial data and model almost aligned
• ICP works best as a refinement scheme
• Harder: initial alignment even w/ arbitrary pose
▫ Feature-guided initial alignment
▫ Transform-space voting
Robust global registration
Feature selection on data
Registration by features
ICP refinement
• Feature points selected by robust and scaledependent local curvature measure
• Register feature points from data to model
[Gelfand et al. 05]
Correspond features to model
• Use an exhaustive search (branch-and-bound)
• Given feature point on data, matching candidates
on model are filtered by local shape similarity
• Rigidity constraints (distortion measure) further
prune the search
pi  p j
2
 qi  q j
2

[Gelfand et al. 05]
Optimization criteria
• The implicit goal is to minimize a combination
of similarity distance and distortion

 SimlarityD ist( p , q ) (1   )  Distortion ( p , p , q , q )
i
i : feature
i
i
j
i
i , j:feature
• More difficult here due to partial matching 
exhaustive search over subsets of varying size
j
A general formulation

 SimlarityD ist( p , q ) (1   )  Distortion ( p , p , q , q )
i
i : feature
i
i
j
i
i , j:feature
• For geometry-driven shape correspondence
▫ SimiarityDist() and Distortion() can vary
• Essentially the (NP-hard) Quadratic Assignment
Problem (QAP), if feature set is given
j
Issues
• Issues with feature-aided ICP
▫ Needs sufficient number of features
▫ Feature extraction can be unreliable with much
outlier, noise, and missing data
• Transform-space voting
▫ Let data “vote” for the rigid transform that
aligns the largest common subsets
▫ Voting in space of transforms via peak detection
Transform-space voting
Transform-space voting
Simple triples of points (triples define a rigid transform)
Transform-space voting
T
Estimate transform and test quality of fit
Transform-space voting
Good T
Voting
Transform space
Good transform cats vote in low-D transform space
Transform-space voting
Good T
Voting
Transform space
• No features, any initial poses, but costly  O(n3)
An insight: choose 4 points
• Choosing special sets of 4 points, planar congruent
sets, makes the problem easier  O(n2)
Choose a base on one shape
S1: co-planar 4-point set
Find all 4-point sets on other shape
S2 approx. congruent to {a, b, c, d}
[Aiger et al. 08]
Basic idea
• Two edge ratios r1 and r2 are invariant to affine
transform (then certainly rigid transform)
[Aiger et al. 08]
Basic idea
• Find 4-point sets in S2 sharing
the same ratios
• Looking at all quadruples too
costly  O(n4)
• Check for “collision” between
intermediate points on all
edges in S2  O(n2) edges
▫ 4 intermediate points per edge
[Aiger et al. 08]
Example
• Only two intermediates points shown for clarity
[Aiger et al. 08]
Representative approaches
• Rigid transform
▫ ICP: iterative closest points
▫ Transformation search by voting
• Piece-wise rigidity and isometry
▫ Rigidity decomposition
▫ Spectral embedding
• Large non-rigid deformation
▫ Deformation-driven search
Relaxing rigidity …
• Piece-wise rigidity  mainly motivated by
character modeling/capture with pose
• Poses are approximately isometric transforms
Extension and reduction to rigid

 SimlarityD ist( p , q ) (1   )  Distortion ( p , p , q , q )
i
i
i : feature
Need pose-oblivious shape
descriptor and distance
i
j
i
i , j:feature
Replace Euclidean for rigid by
geodesic for approx. isometry
• Relatively straightforward extension from rigid
• Reduction to the rigid case: rigid decomposition
or isometric embedding
j
Pose-oblivious (scalar) descriptor
Top: Shape diameter function
Bottom: Centricity function
[Gal et al. 07]
More on descriptors
• Descriptions of local geometry
• Key to correspondence: measure local similarity
• Important issues:
▫ Robustness to noise and tessellation,
e.g., integral invariants [Potmann et al. 06]
▫ Use of context, e.g., shape context, spin
image, part analogy [Shapira et al. 09]
▫ Use the right scale (neighbourhood)
Rigid decomposition
• For deformable scan registration
• Instead of one global rigid transform, find rigid
transforms for different pieces to register
Decomposition of shape into rigidly moving pieces
[Chang and Zwicker 08]
A labelling problem
• Input: two (possibly incomplete) scans
• Output:
▫ Vertices labelled with candidate rigid transforms
▫ Labelling implying a rigid decomposition
▫ Piece-wise rigid registration
[Chang and Zwicker 08]
Algorithm overview
1. Candidate correspondence by feature matching
[Chang and Zwicker 08]
Algorithm overview
1. Candidate correspondence by feature matching
2. Map rigid transform between corresponded features
(their local frames) into transform space
[Chang and Zwicker 08]
Algorithm overview
1. Candidate correspondence by feature matching
2. Map rigid transform between corresponded features
(their local frames) into transform space
3. Cluster in transform space to get candidate transforms
[Chang and Zwicker 08]
Many other works
• Reliance of large scan overlaps between frames
• Use of “reduced models”, e.g., skeletons
▫ Pair-wise correspondence
[Chang and Zwicker 09]
▫ Skeleton consensusization [Zheng et al. 2010]
• Many recent papers on deformable shape
capture  an EG tutorial this year
T2: Geometric Registration for Deformable Shapes (Monday) by
Will Chang, QiXing Huang, Hao Li, Niloy Mitra, Mark Pauly, Michael Wand
Use of embedding
• Transforming (embedding) task into new domain
• Solution facilitated in that domain
▫ E.g., Spectral embedding 
pose normalization
• General topic: spectral geometry processing
▫ Eurographics 2007 STAR and survey
[Zhang et al. 10]
▫ SIGGRAPH 2010 course with Bruno Levy
Spectral embedding
• Use of eigenvectors of appropriate matrices to
embed original shape into the spectral domain
[Jain et al. 07]
[Au et al. 2010]
Rigidity reduction
Rigid alignment
Rigid alignment
• Reduction to rigid alignment in spectral domain
(more or less)
[Jain et al. 07]
[Au et al. 2010]
Overall structure
Eigendecomposition: A = UUT
Input mesh y
A
=
U
e.g.
y’ = U(3)U(3)Ty
Some mesh
operator A
=

UT
Spectral correspondence
1. Define appropriate distance
2. Distances  affinities
3. Spectrally embed two shapes
based on affinity matrix
4. Correspondence analysis in the
Rigid alignment
between mesh elements
spectral domain
▫ Perhaps rigid transform suffices
▫ At least good initialization
[Jain et al. 07]
A specific algorithm
Pose normalization
Geodesic-based
spectral embedding
Best
matching
Size of data sets
Eigenvector
scaling
Non-rigid ICP via
thin-plate spline
Dealing with a bit of nonuniform deformation
[Jain et al. 07]
Embedding
• Affinities defined by geodesics
Eigenvector scaling
1 u1
2 u2
k uk
…
Spectral embeddings using 2nd,
3rd, and 4th scaled eigenvectors
Eigenvectors
u1, u2, …, uk
[Jain et al. 07]
Why does spectral work?
• The right distance/affinity
of correspondence into the
distance measure
• E.g., pose invariance via
use of geodesic distance
Rigid alignment
• Build invariance properties
• Affinities via Gaussian 
scale normalization
[Jain et al. 07]
Eigenvector switching
• Eigenvector plots reveal switching due to part stretching
• Switching causes a reflection, as does a sign flip
y
x
y
y
x
x
[Jain et al. 07]
Fixing eigenvector switching
• “Un-switching” and “un-flipping” by exhaustive search
• Search through all orthogonal transformations (rotation
+ reflection) in spectral domain [Mateus et al., 07]
[Jain et al. 07]
switching
Some results
symmetry
Models with articulation and moderate stretching
[Jain et al. 07]
Cause of eigenvector switching
• Stretching of parts
Large variations in embedding
Stretch-invariant correspondence?
• Probably not arbitrary stretching
• But stretching that preserves part structure
▫ Need distance measure to capture part structure
Distance measure (metric)
• Euclidean
• Geodesic
• Isophotic or
angular
[Pottmann 04]
• Diffusion
[Coifman 06]
Surface based!
Most used: geodesic & angular
• Geodesic distance: insensitive to parts
• Angular distance: subject to “leakage” problem
No angular difference
Diffusion distance
• Models connected-ness between points
▫ Can sense a “narrowing”
over the surface
• More global than geodesic
▫ Consider more paths (those with length t or less)
between two points
An example
Geodesic
Angular
Diffusion
Still cannot fully sense part information
Part-ware!
Geodesic
Diffusion
Angular
Part-aware
[Liu et al. 09]
Missing: connect to volume
Parts are volumetric!
[Liu et al. 09]
Volumetric view
• Look at the object from inside
[Liu et al. 09]
Volumetric view
• Look at the object from inside to define a metric
[Liu et al. 09]
Use of visibility information
• Significant visibility changes across part boundary
• Design part-aware metric to capture change
• Combine with geodesic and angular distances
[Liu et al. 09]
Visibility change
[Liu et al. 09]
Visibility change
[Liu et al. 09]
Visibility change
[Liu et al. 09]
Visibility change
[Liu et al. 09]
Visibility change
[Liu et al. 09]
Visibility change along path
d
VSI Diff.
c
a
b
e
[Liu et al. 09]
Results
source
Part-aware distance field
[Liu et al. 09]
Use of part-aware metric
Homer
Stretched
[Liu et al. 09]
Use of part-aware metric
Homer
Stretched
Geodesic only
Geodesic only
Part-aware metric
Part-aware metric
Spectral embeddings
[Liu et al. 09]
Robustness against part stretching
Geodesic only
Geodesic only
Part-aware metric
Part-aware metric
Spectral embeddings
Rigid ICP registration
Homer Stretched
[Liu et al. 09]
Limitations to spectral embedding
• Intrinsic affinities
▫ Cannot distinguish symmetric switches
▫ Topology sensitivity
• Exhaustive search for eigenvector un-switching
▫ Use of heat kernels (a sum)
[Sun et al., Gebel et al. 09]
• Global and inappropriate for partial matching
Representative approaches
• Rigid transform
▫ ICP: iterative closest points
▫ Transformation search by voting
• Piece-wise rigidity and isometry
▫ Rigidity decomposition
▫ Spectral embedding
• Large non-rigid deformation
▫ Deformation-driven search
Difficulties

 SimlarityD ist( p , q ) (1   )  Distortion ( p , p , q , q )
i
i
i : feature
Descriptor insensitive to pose
and local geometry change
i
j
i
j
i , j:feature
No longer appropriate in the
case of large deformations
• More drastic shape variations + partial matching
• Embedding or rigidity reduction no longer applies
• Only few works so far
Large non-rigid deformation
Pose + non-homogenous
part scaling
Local shape variability
[Zhang et al. 08]
A more global approach
• Local vs. global criteria
▫ Local: feature similarity
▫ Local criterion less reliable with large shape variations
▫ Focus more on global consistency of correspondence
• Use of non-rigid mesh deformation
Correspondence cost =
effort to deform one
<
mesh into other
[Zhang et al. 08]
A result
[Zhang et al. 08]
Advantages
• Automatic
• Handles variations in pose, local scale, part
composition, geometric details
• Self distortion cost
▫ Energy measured on surface of deformed mesh
▫ Not error of fit  partial matching and variations
▫ Feature similarity and geodesic distances do not
enter cost; they only prune the search
Advantages
• Symmetry breaking  extrinsic
▫ Symmetry switch captured by surface distortion
• Priority-driven combinatorial search
▫ Priority = (self) deformation cost
▫ Exploration of large solution space
▫ Avoid initial align and local minima
Symmetry switch
Algorithm
Step 1: feature (extremeties) extraction
Algorithm
Step 1: feature (extremeties) extraction
Step 2: combinatorial search
- Priority = deformation cost
- Pruning by feature similarity and
geodesic distance
Search tree
……
Search tree
Search tree
……
Search tree
……
Search tree
……
Search tree
……
All partial
…… matchings
listed in tree
Some results
Observe partial matching
Limitations
• Still limited by use of geodesic distances
• Pure geometry w/o modeling of semantics
Limitations
• Quite expensive due to exhaustive search and
costly mesh deformation steps
• Upcoming talk:
Oscar K-C. Au, C-L. Tai, D. Cohen-Or, Y. Zhang, H. Fu, “Electors
Voting for Fast Automatic Shape Correspondence”.
Thursday, 13:30 – 15:30, Concert Hall
Overview of talk
• Motivations: why shape correspondence?
• Problem and specialization/classification
• Representative approaches
• Validation
• Conclusion and future perspectives
Qualitative validation
• Visual examination
Quantitative evaluation
• Indirect:
▫ Retrieval, e.g., Princeton Shape Benchmark
• Direct: shapes with known correspondence
▫ Isometric deformations (TOSCA dataset)
▫ Simulated deformations [HJT08]
▫ No general benchmark for shape correspondence
▫ ShapeAnnotator [Attene et al. 09]
▫ Princeton segmentation benchmark [Chen et al. 09]
Overview of talk
• Motivations: why shape correspondence?
• Problem and specialization/classification
• Representative approaches
• Validation
• Conclusion and future perspectives
Summary
• Fundamental building blocks
▫ Local descriptors (robustness, context, scale)
▫ Surface metric to build the right neighbourhood
or invariance properties (geodesic, angular,
diffusion, commute-time, part-aware)
• The right problem specialization
▫ Correspondence only: solve assignment (QAP),
lots of work in vision, survey: [Pardalos et al. 2000]
▫ Alignment sought in most graphics applications
Summary
• Solution paradigms
▫ Solve assignment: relaxation labelling, ACO, HMM
▫ Searching for alignment: ICP, voting
▫ Priority-driven search: deformation, graph editing
▫ Embedding: spectral, mobius [Lipman & Funkhouser 09]
• Partial matching is hard
▫ Exhaustive search, voting (find largest common
subsets), part-in-whole signatures [Shapira et al. 09]
Progressing from low- to high-level
Rigid
Progressing from low- to high-level
Rigid
Isometry/pose
Progressing from low- to high-level
Rigid
Isometry/pose
General deformation
Progressing from low- to high-level
Rigid
Isometry/pose
General deformation
Non-rigid but topology-preserving
Progressing from low- to high-level
Rigid
Isometry/pose
General deformation
Non-rigid but topology-preserving
Topology
Progressing from low- to high-level
Rigid
General deformation
Isometry/pose
Topology
Non-rigid but topology-preserving
Efficiency, partial matching
Purely geometric
Formulation, part analysis
Semantic
Knowledge-driven
Upcoming paper
V. Kalogerakis, A. Hertzmann, and K. Singh, “Learning 3D mesh
segmentation and labelling”, SIGGRAPH 2010, to appear.
Aside: visualizing correspondence
• Matching color tags
We are not visualizing but recognition!
Visualizing correspondence
• Edge links: becoming cluttered quickly
Question: effective visualization of correspondence?
Acknowledgement
• The anonymous reviewers
• My co-authors and many other people with
their excellent works and discussions
• Images and PPTs taken w/ or w/o permission 
• Funding agencies
Questions
Partial correspondence?