Shape Distinction for 3D Object Retrieval Philip Shilane Shape Matching Shape matching is important for numerous projects including constructing 3D models, object recognition, and protein analysis. Molecular Biology Protein Databank H.M. Berman et al. 2000. Shape Matching Shape matching is important for numerous projects including constructing 3D models, object recognition, and protein analysis. Molecular Biology Protein Databank H.M. Berman et al. 2000. Mechanical CAD National Design Repository W. Regli et al. 2001. Engineering Shape Benchmark N. Iyer et al. 2005. Shape Matching Shape matching is important for numerous projects including constructing 3D models, object recognition, and protein analysis. Molecular Biology Protein Databank H.M. Berman et al. 2000. Tracking E3D Mechanical CAD National Design Repository W. Regli et al. 2001. Engineering Shape Benchmark N. Iyer et al. 2005. Shape Matching Shape matching is important for numerous projects including constructing 3D models, object recognition, and protein analysis. Molecular Biology Protein Databank H.M. Berman et al. 2000. Tracking Computer Graphics Princeton Shape Benchmark P. Shilane et al. 2004. E3D Mechanical CAD National Design Repository W. Regli et al. 2001. Engineering Shape Benchmark N. Iyer et al. 2005. Min et al. Min et al. Shape Retrieval Find similar shapes in a database using a query shape. 3D Model Model Database Best Matches Shape Descriptors Shape descriptors are feature vector representations for shapes 3D Model Model Database Best Matches Example Shape Descriptors • • • • • • • • • • • • • • Extended Gaussian Image (EGI) Surface Normals Complex Extended Gaussian Image (CEGI) Shape Histograms (Shells) Shape Histograms (Sectors) Shape Histograms (SecShells) Surface D2 Shape Distributions Distribution Spherical Extent Function (EXT) Radialized Spherical Extent Function (REXT) Voxel Gaussian Euclidean Distance Transform (GEDT) Morphing Harmonic Shape Descriptor (HSD) Distance Fourier Shape Descriptor (FSD) Light Field Descriptor (LFD) ImageDepth Buffer Descriptor (DBD) Based Properties of Shape Descriptors Shape Descriptor Size bytes Create seconds Compare μ-seconds EGI 1,032 0.41 14 CEGI 2,056 0.37 27 Shells 136 0.66 2 Sectors 552 0.90 14 32,776 1.38 451 D2 136 1.12 2 EXT 562 1.17 8 REXT 17,416 2.22 229 Voxel 32,776 1.34 450 GEDT 32,776 1.69 450 HSD 2,184 1.69 27 FSD 32,768 1.82 450 LFD 4,700 3.25 1300 DBD 1,752 0.55 18 SecShells Shape Retrieval with Descriptors • • Which shape descriptors are the best? How do we evaluate retrieval success? Princeton Shape Benchmark Philip Shilane, Patrick Min, Michael Kazhdan, and Thomas Funkhouser Princeton Shape Benchmark • Large shape database • 1,814 classified models, 161 classes • Separate training and test sets • Standardized suite of tests • Multiple classifications • Targeted sets of queries • Standardized evaluation tools • Visualization software • Quantitative metrics Princeton Shape Benchmark (PSB) Num Classes Num Models Largest Class Osada 25 133 20% MPEG-7 15 227 15% Hilaga 32 230 15% Technion 17 258 10% Zaharia 23 362 14% CCCC 54 416 13% Utrecht 6 512 45% Taiwan 47 549 12% Viewpoint 85 1,280 12% 161 1,814 6% PSB Princeton Shape Benchmark 51 potted plants 33 faces 15 desk chairs 22 dining chairs 100 humans 28 biplanes 14 flying birds 11 ships Evaluation Tools Visualization tools • Precision/recall plot • Best matches • Distance image • Tier image Quantitative metrics • Nearest neighbor • First and Second tier • E-Measure • Discounted Cumulative Gain (DCG) Multiple Classifications • Granularity • Model Properties Base Classification (92) LFD Evaluation Tools Visualization tools • Precision/recall plot • Best matches • Distance image • Tier image Quantitative metrics • Nearest neighbor • First and Second tier • E-Measure • Discounted Cumulative Gain (DCG) Multiple Classifications • Granularity • Model Properties EGI Nearest Neighbor LFD 100% EGI 100% LFD Evaluation Tools Visualization tools • Precision/recall plot • Best matches • Distance image • Tier image Quantitative metrics • Nearest neighbor • First and Second tier • E-Measure • Discounted Cumulative Gain (DCG) Multiple Classifications 1 if correct • Granularity n 0 if incorrect log 2 (i 1) • Model DCGProperties i Max Score EGI Nearest Neighbor DCG LFD 100% 88% EGI 100% 75% PSB Contributions • Methodology to compare shape descriptors • Vary classifications • Query lists targeted at specific properties • Unexpected results • EGI: good at discriminating man-made vs. natural objects, though poor at fine-grained distinctions • DBD: good overall performance across tests • Freely available Princeton Shape Benchmark • classified polygonal models • Source code for evaluation tools • Downloaded ~8,700 times since 2003 • http://shape.cs.princeton.edu/benchmark Shape Retrieval Matching the whole shape has drawbacks. 3D Model Model Database Best Matches Local Matches for Retrieval Matching local regions of shapes to improve retrieval. 3D Model Model Database D(A,B) = ∑ ΔFeatureShape Correspondences + Best Matches ∑ ΔSpatialConsistency Correspondence Pairs Local Matches for Retrieval Focusing on matching local regions of shapes can improve retrieval results. Using many local descriptors is slow. 3D Model Model Database D(A,B) = ∑ ΔFeatureShape Correspondences + Best Matches ∑ ΔSpatialConsistency Correspondence Pairs Local Matches for Retrieval Focusing on matching local regions of shapes can improve retrieval results. Using many local descriptors is slow. 3D Model Best Matches ModelMany Database descriptors do D(A,B) = ∑ ΔFeatureShape Correspondences represent ∑not ΔSpatialConsistency Correspondence distinguishing parts. Pairs + Local Matches for Retrieval Focusing on matching local regions of shapes can improve retrieval results. 3D Model D(A,B) = Focusing on the distinctive regions Best improves retrieval time Matches Model Database and accuracy. ∑ ΔFeatureShape + ∑ ΔSpatialConsistency Correspondences Correspondence Pairs Outline • • • • • • • • Introduction Shape Descriptors for Retrieval Related Work System Overview Evaluating Distinction Computational Improvements Applications Conclusion Related Work • Random Selection • Perceptual Criteria • Eye Tracking • Saliency • Shape Matching • Likelihood • Stability Related Work • Random Selection • Perceptual Criteria • Eye Tracking • Saliency • Shape Matching • Likelihood • Stability Mori 2001 Frome 2004 Related Work • Random Selection • Perceptual Criteria • Eye Tracking • Saliency • Shape Matching • Likelihood • Stability Howlett et al. Related Work • Random Selection • Perceptual Criteria • Eye Tracking • Saliency • Shape Matching • Likelihood • Stability Lee et al. Other Projects: Li et al. Novotni et al. Frintrop et al. Watanabe et al. Hoffman et al. Gal et al. Related Work • Random Selection • Perceptual Criteria • Eye Tracking • Saliency • Shape Matching • Likelihood • Stability Johnson et al. Chua et al. Related Work • Random Selection • Perceptual Criteria • Eye Tracking • Saliency • Shape Matching • Likelihood • Stability Shan et al. Distinct Regions Distinctive regions distinguish an object from other types. 0 Mesh Mesh Distinction Distinction 1 Key Idea Determining which regions are important is based on the other shapes under consideration. Mesh Shape DB Mesh Distinction Key Idea A classified shape database provides a ground truth to assess which regions of a shape are distinctive. Mesh Classified Shape DB Mesh Distinction Outline • • • • • • • • Introduction Shape Descriptors for Retrieval Related Work System Overview Evaluating Distinction Computational Improvements Applications Conclusion System Overview Mesh Regions Shape Descriptors Distinct Regions Distinctive Regions of 3D Surfaces Philip Shilane and Thomas Funkhouser ACM Transactions on Graphics Vertex Distinction Constructing Regions Consistent segmentation is difficult and unnecessary. Regions could be disjoint, overlap, or exist at multiple scales. Segmentation Katz et al. Constructing Regions Randomly selecting points on the surface is a simple solution. Constructing Regions Regions are created at multiple scales at each position. Size of Regions 0.5 0.25 0.5 1.0 1.0 2.0 Describing Shapes • • • • • Quick to compute Compact to store Fast to compare Robust to mesh problems Discriminating of similar models Describing Shapes • • • • • Quick to compute Compact to store Fast to compare Robust to mesh problems Discriminating of similar models Experimented with several descriptors • Shells Shape Histogram • Harmonic Shape Descriptor • Fourier Shape Descriptor Measuring Distinction We would like to calculate distinction scores for each region. Mesh Distinction Measuring Distinction Our approach is to analyze each region relative to a database to determine which regions match shapes that are similar. Distinction Mesh Classified Shape DB Measuring Distinction We would like to know the distinction of all combinations of regions. Distinction Mesh Classified Shape DB Measuring Distinction We make an independence assumption to calculate distinction for each region. Distinction Mesh Classified Shape DB Distinction = Retrieval Performance The distinction of each local descriptor is based on how well it retrieves shapes of the correct class. Query Descriptors Retrieval Results Distinction = Retrieval Performance The distinct descriptors that distinguish between classes are classification dependent. Query Descriptors Retrieval Results Mapping to Vertices For visualization, mesh simplification, and icon generation, it is useful to have distinction scores for each vertex. Mapping to Vertices We map distinction scores to each vertex using a Gaussian weighted average. 2 xv x D( x) exp[ 2 2 ] D (v ) 2 xv x exp[ 2 2 ] v is a vertex x is a point with distinctio n σ 0.1 of mesh radius Outline • • • • • • • • Introduction Shape Descriptors for Retrieval Related Work System Overview Evaluating Distinction Computational Improvements Applications Conclusion Evaluating Distinction 1. What regions are distinctive? 2. Does distinction improve retrieval? Analysis Distinctive regions of meshes correspond to important regions that define a class of objects. Analysis Compared to a database of flying objects, the helicopter blades are most distinctive. Mesh Saliency Analysis For human models in this pose, the elbow area is most distinctive. Analysis The distinctive regions of a mesh change with the database under consideration. Princeton Shape Benchmark Plane DB Vehicle DB Mesh Saliency Evaluating Distinction 1. What regions are distinctive? 2. Does distinction improve retrieval? Partial Matching of 3D Shapes with Priority-Driven Search Thomas Funkhouser and Philip Shilane Symposium on Geometry Processing, Sardinia, Italy, July 2006 Distinction within a Database Classification Preprocess Shape DB Local Descriptors Retrieval Evaluation Distinction Function Descriptor DB Query Shape Local Descriptors Match Retrieval List Matching Local Shapes 1. Generate local shape features 2. Find correspondences minimizing distance function Surface A B Matching Local Shapes 1. Generate local shape features 2. Find correspondences minimizing distance function Features Points A D(A,B) = ∑ ΔFeatureShape Correspondences B + ∑ ΔSpatialConsistency Correspondence Pairs Matching Local Shapes 1. Generate local shape features 2. Find correspondences minimizing distance function Feature Correspondences Features A D(A,B) = ∑ ΔFeatureShape Correspondences B + ∑ ΔSpatialConsistency Correspondence Pairs Matching Local Shapes 1. Generate local shape features 2. Find correspondences minimizing distance function Feature Correspondences Spatial Consistency (Deformation) Features A D(A,B) = ∑ ΔFeatureShape Correspondences B + ∑ ΔSpatialConsistency Correspondence Pairs Priority-Driven Search Use a priority queue to perform best-first search for the optimal set of K correspondences in the database. B2 A1 A6 A5 C1 B3 C2 C5 B5 A2 A3 A4 B1 Query Object Feature Correspondences _______________ D(match) (A1,B2) (A3,B1) (A5,B3) 0.9 Low Cost B4 C4 C3 Target Objects (A2,B1) 1.1 (A1,B2) (A2,B1) 1.5 (A3,B1) (A2,B1) 1.8 Priority Queue (A4,C1) 9.1 (A2,C3) 9.6 High Cost Matching Local Shapes Filter the database to the most distinctive regions during preprocessing Mesh Descriptors Distinction Scores 4 Selected Descriptors Alternative Selection Techniques The database can be filtered according to various selection techniques. Distinction Likelihood Saliency Random Shape Matching Results Using the most distinctive descriptors improves matching versus other techniques. Shape Matching Results Using multiple descriptors improves over using a single descriptor. Descriptor Selection Distinction Random Distinction Random K NN% DCG% 3 3 1 1 74.3% 66.5% 62.4% 55.6% 70.6% 63.4% 65.3% 57.8% Centroid Oracle 1 1 53.3% 89.5% 54.4% 79.7% Shape Matching Results Using a single descriptor on the surface is better than using a global descriptor. Descriptor Selection Distinction Random Distinction Random K NN% DCG% 3 3 1 1 74.3% 66.5% 62.4% 55.6% 70.6% 63.4% 65.3% 57.8% Centroid Oracle 1 1 53.3% 89.5% 54.4% 79.7% Shape Matching Results The oracle case shows an upper bound on matching performance, suggesting research areas for improvement. Descriptor Selection Distinction Random Distinction Random K NN% DCG% 3 3 1 1 74.3% 66.5% 62.4% 55.6% 70.6% 63.4% 65.3% 57.8% Centroid Oracle 1 1 53.3% 89.5% 54.4% 79.7% Shape Matching Results Priority-driven search with distinctive regions has better retrieval performance than previous global techniques. Outline • • • • • • • • Introduction Shape Descriptors for Retrieval Related Work System Overview Evaluating Distinction Computational Improvements Applications Conclusion Computational Bottlenecks • Computing distinction requires a full retrieval list [2.4 minutes] • Computing 128 descriptors at 4 scales [3 minutes] Times are per model Computational Bottlenecks • Computing distinction requires a full retrieval list [2.4 minutes] • Computing 128 descriptors at 4 scales [3 minutes] Times are per model Calculating Distinction Discounted cumulative gain requires a full retrieval list of n results. 1 if correct n 0 if incorrect log 2 (i 1) i DCG Max Score Calculating Distinction Approximate distinction with a short retrieval list of length k < n. 1 if correct k 0 if incorrect log 2 (i 1) i DCG Max Score Calculating Distinction Cover tree index is a spatial structure for efficiently finding neighbors in high dimensional space. Query descriptor Image adapted from Qin Lv Calculating Distinction 64 neighbors can be found in a fraction of the time of searching the whole database, with similar retrieval accuracy. Updating Distinction A cover tree index can also be used for updating distinction 1. Build cover tree for DB descriptors 2. Record R distances in red-black tree 3. Find K neighbors for new descriptor, K > R Rth Neighbor Kth Neighbor K>R 4. Update distinction for K neighbors and DB descriptors with R distance > K distance Updating Distinction A fraction of the DB must be updated depending on K neighbor search and R-distinction Computational Bottlenecks • Computing distinction requires a full retrieval list [2.4 minutes] • Computing 128 descriptors at 4 scales [3 minutes] Times are per model Predicting Distinction We want a predicted distinction score for each position on the model. Descriptors Distinction Predict Selecting Distinctive 3D Shape Descriptors for Similarity Retrieval Philip Shilane and Thomas Funkhouser Shape Modeling International, Matsushima, Japan, June 2006 Predicting Distinction We map descriptors into a compact space where we learn distinction from a training set. Descriptors Distinction Prediction Overview Training Shape DB Local Descriptors Classification Likelihood Distinction Function Descriptor DB Retrieval Evaluation Query Shape Local Descriptors Likelihood Evaluate Distinction Select Descriptors Match Retrieval List Likelihood of Descriptors Multi-dimensional normal density [Johnson 2000] density ( x ) 1 2 2 d 2 1 t 1 exp ( x ) ( x ) 2 x d dimensiona l feature vector mean feature vector d x d covariance matrix Build Predicted Distinction Function Measure likelihood and retrieval performance of each descriptor. Build Predicted Distinction Function Measure likelihood and retrieval performance of each descriptor. Build Predicted Distinction Function Measure likelihood and retrieval performance of each descriptor. Build Predicted Distinction Function Retrieval performance is averaged within each likelihood bin. Predicted Distinction Function A likelihood mapping separates descriptors with different retrieval performance. Less Likely More Likely Predicted Distinction Function The most common features are the worst for retrieval. Less Likely More Likely Mapping Descriptors to Distinction During the query phase, we predict distinction as we generate descriptors Descriptors Distinction Likelihood Mapping Local Matches for Retrieval Y X 3D Model Model Database Cost Function C( X ,Y ) i i Best Matches Matching with Distinctive Descriptors Y X 3D Model Model Database Cost Function k C( X ,Y ) i i Best Matches Alternative Prediction Methods Distinction improves retrieval more than other techniques. Selection Techniques % Improvement Precision 20% Distinction 15% Likelihood 10% Random 5% 0% 10% 30% 50% -5% Recall 70% 90% Outline • • • • • • • • Introduction Shape Descriptors for Retrieval Related Work System Overview Evaluating Distinction Computational Improvements Applications Conclusion Applications Importance scores on the surface of a mesh are useful for numerous graphics and geometric processing applications: • • • • • • • Shape Matching Mesh Simplification Icon Generation Alignment Rendering Morphing Segmentation Mesh Simplification Garland 1,700 tri 300 tri Mesh Saliency Distinct Regions Mesh Simplification 97K tri Mesh Saliency Distinct Regions 2K tri 2K tri zoom Icon Generation Icon Generation Focusing on the most distinctive regions often (but not always) highlights important features. Outline • • • • • • • • Introduction Shape Descriptors for Retrieval Related Work System Overview Evaluating Distinction Computational Improvements Applications Conclusion Conclusion • • • • • • • • PSB: data set and tools for evaluating retrieval methods Defined distinctive regions based on retrieval performance Analyzed properties of distinction Focused shape-matching on distinctive regions Calculate/update distinction efficiently Predict distinctive regions from training set Apps of distinction: icons and mesh simplification Focus on generic methods, independent of shape descriptor Future Work • • • • Efficiency of shape retrieval Distinction for an unclassified database Scalability of updating distinction Predict distinction with improved likelihood model or other mapping • Apply distinction analysis to image retrieval and other applications Acknowledgements • Committee: Thomas Funkhouser, Szymon Rusinkiewicz, Adam Finkelstein, David Dobkin, Andrea LaPaugh, and Kai Li • Graphics Group and entire CS Department • Family and friends • Funding Sources: Princeton University Departmental Award National Science Foundation Grants IIS-0612231, CCR-0093343, CNS 0406415, and 11S-0121446 Air Force Research Laboratory Grant FA8650-04-1-1718 Google Research Grant Extra Slides Constructing Regions Selecting the vertices biases the sampling to the underlying polygon representation. Vertices Applications: Shape Matching •General strategy: • Sample set of points on surface of object • Build shape descriptor centered at every point at multiple scales • Find matches with high descriptor similarity & low geometric deformation Mesh Simplification Simplifying a complex mesh is important for improving rendering time. Many polygons Fewer polygons (quadric error shown) Garland et al. Icon Generation Recent work selects the viewpoint that maximizes the amount of salient surfaces or that minimizes the symmetric surfaces. Lee et al. Podolak et al. Alternative Prediction Methods Distinction improves retrieval more than other techniques. Selection Techniques 20% % Improvement Precision Distinctive 15% Johnson 2000 (Model) Johnson 2000 (DB) 10% Random 5% 0% 10% 30% 50% -5% Recall 70% 90% Typical Shape Databases Osada MPEG-7 Hilaga Technion Zaharia CCCC Utrecht Taiwan Viewpoint Num Models 133 1,300 230 1,068 1,300 1,841 684 1,833 1,890 Num Num Classes Classified 25 133 15 227 32 230 17 258 Letter ‘C’ 23 362 54 416 6 512 47 549 85 1,280 Largest Class 20% 15% 15% 10% 14% 13% 45% 12% 12% Typical Shape Databases Osada MPEG-7 Hilaga Technion Zaharia CCCC Utrecht Taiwan Viewpoint Num Models 133 1,300 230 1,068 1,300 1,841 684 1,833 1,890 Num Num Classes Classified 25 133 15 227 32 230 17 258 23 362 54 416 6 512 47 549 85 1,280 Largest Class 20% 15% 15% 10% 14% 13% 45% 12% 12% Typical Shape Databases Osada MPEG-7 Hilaga Technion Zaharia CCCC Utrecht Taiwan Viewpoint Num Num Num Models Classes Classified 133 25 133 1,300 15 227 230 32 230 1,068 17 258 Aerodynamic 23 1,300 362 1,841 54 416 684 6 512 1,833 47 549 1,890 85 1,280 Largest Class 20% 15% 15% 10% 14% 13% 45% 12% 12% Evaluation Tools Visualization tools • Precision/recall plot • Best matches • Distance image • Tier image Quantitative metrics • Nearest neighbor • First and Second tier • E-Measure • Discounted Cumulative Gain (DCG) Multiple Classifications • Granularity • Model Properties Evaluation Tools Visualization tools • Precision/recall plot • Best matches • Distance image • Tier image Quantitative metrics • Nearest neighbor • First and Second tier • E-Measure • Discounted Cumulative Gain (DCG) Dining Chair Multiple Classifications • Granularity • Model Properties Desk Chair Related Work • Random Selection • Perceptual Criteria • Eye Tracking • Saliency • Shape Matching • Likelihood • Stability Lee et al. Other Projects: Li et al. Novotni et al. Frintrop et al. Watanabe et al. Hoffman et al. Gal et al. Analysis We investigated whether alternative, faster techniques for calculating importance are correlated with distinction. • Distance: from center of mass • Surface Area: percentage of mesh enclosed within a region • Likelihood: consider each shape descriptor as a feature vector • Saliency: change in curvature as calculated by saliency.exe Analysis We investigated whether alternative, faster techniques for calculating importance are correlated with distinction. • Distance: from center of mass • Surface Area: percentage of Correlation Coefficient mesh enclosed within a region n 1 • Likelihood: consider each r as a feature ( xi x)( yi shape descriptor ( n 1 ) x y i 1 vector • Saliency: change in curvature as calculated by saliency.exe y) Analysis We investigated whether alternative, faster techniques for calculating importance are correlated with distinction. • Distance: from center of mass • Surface Area: percentage of mesh enclosed within a region • Likelihood: consider each shape descriptor as a feature vector • Saliency: change in curvature as calculated by saliency.exe r = -0.04 r = 0.07 r = 0.04 r = 0.03 Shape Matching Results Priority-driven search with distinctive regions has better retrieval performance than previous global techniques. Mesh Simplification When simplifying a mesh, important regions should be preserved while less important regions are simplified. 69K tri Garland et al. 1K tri Mesh Simplification Quadric Error is related to the distance each vertex has moved during simplification. Ee Qv1 Qv 2 Garland et al. Mesh Simplification We modified the standard error metric to include distinction scores for each vertex. We augment quadric error with distinction scores. Dv1 Ee Dv1Qv1 Dv 2Qv 2 Garland et al. After an edge collapse, the remaining vertex is given the maximum distinction score of the two vertices involved. Icon Generation Selecting the best view point is important for creating icons that are quickly recognizable. Blanz et al. Icon Generation Important surfaces should be visible when creating icons for a catalog of shapes. Good View Lee et al. Poor View Good View Poor View Podolak et al. Evaluation Tools Man-made vs. Natural (2) LFD REXT SHD GEDT Precision Visualization tools • Precision/recall plot • Best matches 1 • Distance image 0.8 • Tier image Quantitative metrics 0.6 • Nearest neighbor • First and Second tier 0.4 • E-Measure 0.2 • Discounted Cumulative Gain (DCG) 0 Multiple Classifications 0 • Granularity • Model Properties EXT SecShells Voxel Sectors CEGI EGI 0.2 0.6 0.4 Recall 0.8 1 D2 Shells