Yunhai Wang1 Minglun Gong1,2 Tianhua Wang1,3 Hao (Richard) Zhang 4
Daniel Cohen-Or 5 Baoquan Chen1,6
1Shenzhen
2 Memorial
4 Simon
5
Institutes of
Advanced Technology
Fraser University
University
of Newfoundland
3Jilin
Tel-Aviv University
6 Shandong
University
University
 One of the most fundamental tasks in shape
analysis
 Low-level cues (minimal rule; convexity) alone
insufficient
2/40
Learning segmentation
Unsupervised co-analysis
[Kalograkis et al. 10]
[Sidi et al. 2011]
Keys to success: amount & quality of
labelled or unlabelled 3D data
3
Joint segmentation
Active co-analysis
[Huang et al. 2011]
[Wang et al. 2012]
3/40
 How many 3D models of strollers, golf carts, gazebos, …?
 Not enough 3D models = insufficient knowledge
 Labeling 3D shapes is also a non-trivial task
380 labeled meshes over 19 object
categories
4/40
About 14 million images across almost 22,000 object
categories
Labeling images is quite a bit easier than labeling 3D
shapes
5/40
Self-intersecting; non-manifold
Incomplete
Real-world 3D models (e.g., those from Tremble
Warehouse) are often imperfect
6
6/40
 Treat a 3D shape as a set of projected binary
images
 Label these images by
learning from vast amount of
image data
 Then propagate the image
labels to the 3D shape
 Alleviate various data
artifacts in 3D, e.g., selfintersections
7/40
 Joint image-shape analysis via projective
analysis for semantic 3D segmentation
 Utilize vast amount of available image data
 Allowing us to analyze imperfect 3D shapes
8/40
 Bi-class Symmetric Hausdorff distance = BiSH
 Designed for matching 1D binary images
 More sensitive to topology changes (holes)
 Caters to our needs: part-aware label transfer
9/40
Many works on 2D-3D fusion, e.g., for reconstruction
[Li et al.11]
Image-guided 3D modeling
[Xu et al.11]
10
10/40
Image-space simplification error
[Lindstrom and Turk 10]
Light field descriptor for
3D shape retrieval
[Chen et al.03]
We deal with the higher-level and more
delicate task of semantic 3D segmentation
11/40
11
 PSA for 3D shape segmentation
 Region-based binary shape matching
 Results and conclusion
12/40
Labeling involves GrabCut and some user assistance
13/40
 Assume all objects are upright oriented; they mostly are!
 Project an input 3D shape from multiple pre-set
viewpoints
14/40
 For each projection of the input 3D shape, retrieve top
matches from the set of labelled images
15/40
 Select top (non-adjacent) projections with the smallest
average matching costs for label transfer
16/40
 Label transfer is done per
corresponding horizontal slabs
Later …
 Pixel correspondence
straightforward
17/40
 Label transfer is weighted by a confidence value per pixel
 Three terms based on image-level, slab-level,
and pixel-level similarity: more similar = higher
confidence
18/40
 Probabilistic map over input 3D shape: computed by
integrating per-pixel confidence values over each shape
primitive

One primitive projects to multiple pixels in multiple
images
 Per-pixel confidence gathered over multiple retrieved
images
19/40
 Final labeling of 3D shape: multi-label alpha expansion
graph cuts based on the probabilistic map
20/40
 PSA for 3D shape segmentation
 Region-based binary shape matching
 Results and conclusion
21/40
…
Projections of input 3D shape
…
Database of (labeled) images
 Goal: find shapes most suitable for label transfer and
FAST!
 Not a global visual similarity based retrieval
 Want part-aware label transfer but cannot reliably
segment
 Characteristics of the data to be matched
Classical descriptors, e.g., shape context, interior
distance
shape
context
(IDSC),
GIST,
Zenikenot
moments,

Possibly
complex
topology
(lots
of holes),
just a
Fourier
descriptors, etc., do not quite fulfill our needs
contour
22/40
 All upright orientated: to be exploited
Takes advantage of upright orientation
23/40
 Cluster scan-lines into smaller number of slabs --efficiency!
 Hierarchical clustering by a distance between adjacent
slabs
Classical choice for distance:
symmetric Hausdorff (SH)
But not sensitive to topology
changes; not part-aware
24/40
C
SH(C,B)=2, SH(Cc, Bc)=2
B
A
SH(A,B)=2, SH(Ac, Bc)=10
B
SH for only one class may not be topologysensitive
A bi-class SH distance is!
25/40
C
B
SH(C,B)=2, SH(Cc, Bc)=2
BiSH(C,B) = 2
A
B
SH(A,B)=2, SH(Ac, Bc)=10
BiSH(A,B) = 10
26/40
BiSH is more part-aware: new slabs near part boundaries
BiSH
SH
27/40
 Slabs are scaled/warped vertically for better alignment
 Another measure to encourage part-aware label transfer
Warp
Slabs of labeled image warped to
better align with slabs in projected
image
Recolor
Slabs recolored: many-to-one slab
matching possible
28/40
 Dissimilarity between slabs: BiSH scaled by slab
height
 Slab matching allows linear warp: optimized by a
dynamic time warping (DTW) algorithm
 Dissimilarity between images: sum over slab
dissimilarity after warped slab matching
29/40
 PSA for 3D shape segmentation
 Region-based binary shape matching
 Results and conclusion
30/40
 Same inputs, training data (we project), and experimental
setting
 Models in [K 2010]: manifold, complete, no selfintersections
 PSA allows us to handle any category and imperfect
31/40
 11 object categories; about 2600 labeled images
 All input 3D shapes tested have self-intersections
as well as other data artifacts
32/40
Pavilion
(465 pieces)
Bicycle
(704 pieces)
33/40
34/40
 Matching two images (512 x 512) takes 0.06
seconds
 Label transfer (2D-to-2D then to 3D): about 1
minute for a 20K-triangle mesh
 Number of selected projections: 5 – 10
 Number of retrieved images per projection: 2
35/40
 Projective shape analysis (PSA): semantic
3D segmentation by learning from labeled
2D images
 Demonstrated potential in labeling 3D
models: imperfect, complex topology, over
any category
36/40
36
 Utilize the rich availability and ease of
processing of photos for 3D shape analysis
 No strong requirements on quality of 3D model
37/40
 Inherent limitation of 2D
projections: they do not fully
capture 3D info
 Inherent to data-driven: knowledge has to be in
data
 Relying on spatial and not feature-space
 analysis
Assuming upright; not designed for articulated
shapes
38/40
 Labeling 2D images is still tedious:
unsupervised projective analysis
 Additional cues from images and projections,
e.g., color, depth, etc.
 Apply PSA for other knowledge-driven
analyses
39/40
More results and data can be found
from http://web.siat.ac.cn/~yunhai/psa.html
40/40
40