Bag-of-Feature-Graphs:
A New Paradigm for Non-rigid Shape
Retrieval
T i n gbo HOU, X i a ohua HOU, M i n g ZHON G a n d Ho n g QI N
De p a rtm ent o f Co m p u te r S c i enc e
S to ny Bro o k U n i ve rsit y ( S U N Y S B)
Department of Computer Science
Center for Visual Computing
ICPR 2012
Nonrigid Shape Retrieval
Shape Query
Shape Database
Retrieved Shapes
…
…
Department of Computer Science
Center for Visual Computing
ICPR 2012
Overview of BoFG
 Inspired by the ideas from Bag-of-Words (BoW) and SpatialSensitive Bag-of-Words (SS-BoW)
 Feature-driven
 Concise and fast to compute
 Spatially informative
Department of Computer Science
Center for Visual Computing
ICPR 2012
Previous Works Relevant to
This Project
 Bag-of-Words
1.
2.
3.
4.
Y. Liu, H. Zha, and H. Qin. CVPR, 2006.
H. Tabia, M. Daoudi, J. P. Vandeborre, and O. Colot. 3DOR, 2010.
R. Toldo, U. Castellani, and A. Fusiello. VC, 2010.
G. Lavoué. 3DOR, 2011.
 Shape Google (Spatially-Sensitive Bag-of-Words)
1. M. Ovsjanikov, A. M. Bronstein, L. J. Guibas and M. M. Bronstein. NORDIA,
2009.
2. (SI-HKS) M. M. Bronstein and I. Kokkinos. CVPR, 2010.
3. A. M. Bronstein, M. M. Bronstein, L. J. Guibas, and M. Ovsjanikov. ACM
TOG, 2011.
Department of Computer Science
Center for Visual Computing
ICPR 2012
Background (1)
 Heat Kernel on surface 𝑀
Amount of heat transferred from a point 𝑥 to 𝑦 in time 𝑡
−𝜆𝑙 𝑡
ℎ𝑡 𝑥, 𝑦 = ∞
𝜙𝑙 𝑥 𝜙𝑙 (𝑦)
𝑙=0 𝑒




𝜆𝑙 , 𝜙𝑙 : 𝑙-th eigenvalue and eigenfunction of the Laplace-Beltrami operator
Heat Kernel Signature (HKS): ℎ𝑡 (𝑥, 𝑥)
 HKS descriptor


A vector of HKS probed at different values of 𝑡
𝐾 𝑥 = (ℎ𝑡 1 𝑥, 𝑥 , ℎ𝑡2 𝑥, 𝑥 , … , ℎ𝑡𝑟 𝑥, 𝑥 )
 Properties of Heat Kernel



Intrinsic (Invariant to rigid and isometric deformation)
Informative (locally and globally shape aware)
Stable
Department of Computer Science
Center for Visual Computing
ICPR 2012
Background (2)
 Geometric words 𝑊𝑖


A representative HKS vector
Clustered in the HKS descriptor space by the k-means algorithm
 Vocabulary
𝑾 = {𝑊1 , … , 𝑊𝑉 }
 Similarity of point 𝑥 and word 𝑊𝑖
𝜃𝑖 𝑥 = 𝑐 𝑥
Department of Computer Science
Center for Visual Computing
𝐾 𝑥 −𝑊𝑖 2
−
2𝜎2
𝑒
ICPR 2012
Shape-Google Revisit (1)
 Bag-of-Words


Word distribution of each point
Θ 𝑥 = 𝜃1 𝑥 , … , 𝜃𝑉 𝑥
BoW descriptor: 𝑉 × 1 vector
𝑓 𝑀 =
Θ 𝑥 𝑑𝜇(𝑥)
𝑀

Measure the frequencies of words appearing on the shape
Department of Computer Science
Center for Visual Computing
ICPR 2012
Shape-Google Revisit (2)
 Spatially-Sensitive Bag-of-Words

SS-BOW descriptor: 𝑉 × 𝑉 matrix
𝐹 𝑀 =
𝑀×𝑀

Θ 𝑥 ΘT y ℎ𝑡 𝑥, 𝑦 𝑑𝜇 𝑥 𝑑𝜇(𝑦)
Measure the frequencies of word pairs
Department of Computer Science
Center for Visual Computing
ICPR 2012
New Paradigm:
Bag-of-Feature Graphs (1)
 Motivation: Reduce computation complexity
 Considering all points on shape -> only considering feature points
 Vector/matrix of word frequencies -> feature graphs associated
with words
Department of Computer Science
Center for Visual Computing
ICPR 2012
Formulation (1)
 Feature set: 𝐹




Feature graph 𝐺𝑖 associated with the 𝑖-th geometric word 𝑊𝑖
𝐺𝑖 represented as 𝐹 × |𝐹| matrix
𝐺𝑖 𝑥, 𝑦 = 𝜃𝑖 𝑥 𝜃𝑖 𝑦 ℎ𝑡 (𝑥, 𝑦)
𝑥, 𝑦 ∈ 𝐹
ℎ𝑡 (𝑥, 𝑦): Heat Kernel
 Bag-of-Feature-Graphs representation of shape 𝑀
𝐺 𝑀 = {𝐺1 , 𝐺2 , … , 𝐺𝑉 }
…
𝐺1
Department of Computer Science
Center for Visual Computing
𝐺2
ICPR 2012
𝐺3
Formulation (2)
 BoFG descriptor


Multi-dimensional scaling (MDS): Choosing the 6 largest eigenvalues of
each graph matrix denoted by 𝑆𝑖 𝑀
6𝑉 × 1 vector 𝑆1 𝑀 , … , 𝑆𝑉 𝑀 𝑇
 Shape distance
𝑉
𝑑 𝑀1 , 𝑀2 =
𝑆𝑖 𝑀1 − 𝑆𝑖 𝑀2
2
𝑖=1
 Retrieval by approximate nearest neighbor (ANN) search
Department of Computer Science
Center for Visual Computing
ICPR 2012
Nonrigid Shapes and Their BoFG
Descriptors
Department of Computer Science
Center for Visual Computing
ICPR 2012
𝑁: Number of vertices
𝑂 𝐷 : Time complexity
for computing HKS
descriptor of a vertex
Time Complexity of BoW, SS-BOW and BoFG
Department of Computer Science
Center for Visual Computing
ICPR 2012
Experiments
 Test dataset: TOSCA1


12 classes of 148 non-rigid shapes
Each shape has 3K ∼ 30K vertices
 Evaluated methods: BoW, FSS-BoW, SI-HKS,
 Vocabulary


48 words for BoW and SS-BoW (clustered from all shape points)
4 words for BoFG (clustered only from feature points)
 Feature numbers in BoFG: 30 ∼ 50 for each shape
1http://toca.cs.technion.ac.il/book/shrec.html
Department of Computer Science
Center for Visual Computing
ICPR 2012
Department of Computer Science
Center for Visual Computing
ICPR 2012
Experiments
Time performance (in seconds) of three descriptors on two shapes with 3K and 30k vertices
Department of Computer Science
Center for Visual Computing
ICPR 2012
Experiments
(1)
(2)
(3)
Precision-recall curves of evaluated methods, with categories of
(1) null, (2) scale changes and (3) holes.
Department of Computer Science
Center for Visual Computing
ICPR 2012
Partial shape retrieval
 Query shape is only a part of a complete model
 Online feature alignment is required to extract corresponding subgraphs
Department of Computer Science
Center for Visual Computing
ICPR 2012
Summary
 Bag-of-Feature-Graphs (BoFG) is a new paradigm for shape
representation
 This representation is feature-driven, concise, and spatially-aware
 The key idea is to construct graphs of features associated with
geometric words
 BoFG has much improved time-performance and competitive
retrieval results in comparison with other state-of-the-art methods
Department of Computer Science
Center for Visual Computing
ICPR 2012
Future Work
 Investigate graph comparison with heavy outliers
 Improve the performance on partial shape retrieval
 Acknowledgements: Research Grants from
National Science Foundation
Department of Computer Science
Center for Visual Computing
ICPR 2012
Thank You!
Department of Computer Science
Center for Visual Computing
ICPR 2012