3D-Based Reasoning with Blocks,
Support, and Stability
Zhaoyin Jia
School of Electrical and Computer Engineering
Cornell University
Computer Vision with RGB-D
Pose Recognition
Activity Detection
J. Shotton et al. 2011; G. Girshick et al. 2013.
J. Sung et al. 2012; H. Koppula et al. 2013.
2
Object Recognition
3D Scene Labeling
K. Lai et al. 2011; A. Janoch et al. 2011
H. Koppula, et al. 2011; N. Silberman et al 2011, 2012.
Jia, Gallagher, Saxena and Chen
RGB-D Images
3
Jia, Gallagher, Saxena and Chen
3D Reasoning on RGB-D Images

Free Space:


Physical Stability:


one book is supported
by the table and wall.
Foresee
Consequences:

4
objects can be placed
in empty spaces.
the camera and the
book will fall if the box
moves.
Jia, Gallagher, Saxena and Chen
Reasoning with Blocks, Support, & Stability
Input: RGB-D
5
Segmentation
Jia, Gallagher, Saxena and Chen
Reasoning with Blocks, Support, & Stability
Input: RGB-D
6
Blocks, Support, and Stability
Jia, Gallagher, Saxena and Chen
Reasoning with Blocks, Support, & Stability
Input: RGB-D
7
Final 3D representation
Jia, Gallagher, Saxena and Chen
Algorithms
8
Jia, Gallagher, Saxena and Chen
Overview
Input Segmentation*
3D Block Fitting
Support and Stability
Evaluate Energy
Function
* "Indoor Segmentation and Support Inference from RGBD Images," N. Silberman et al. ECCV, 2012.
9
Jia, Gallagher, Saxena and Chen
Overview
Input Segmentation
3D Block Fitting
Support and Stability
Evaluate Energy
Function
10
Jia, Gallagher, Saxena and Chen
Single Block Fitting



3D orientated bounding box on depth data
Partially observed. Minimum volume may fail *
Minimum surface distance (Min-surf)
* "Fast oriented bounding box optimization on the rotation group SO(3, R)," C. Chang et al, ACM Transactions
on Graphics, 2011.
11
Jia, Gallagher, Saxena and Chen
Overview
Input Segmentation
3D Block Fitting
Support and Stability
Evaluate Energy
Function
12
Jia, Gallagher, Saxena and Chen
Support and Stability
Support
Relations
13
Supporting
Area
Stability
Support Relation
Surface On-top
Support
14
Partial On-top
Support
Side
Support
Jia, Gallagher, Saxena and Chen
Separate axis is
parallel to y
Surface On-top
Support
15
Partial On-top
Support
Separate axis is
perpendicular to y
Side
Support
Jia, Gallagher, Saxena and Chen
Surface On-top
Support
16
Partial On-top
Support
Side
Support
Jia, Gallagher, Saxena and Chen
From Support To Stability

Supporting Area
17
Jia, Gallagher, Saxena and Chen
From Support To Stability

Supporting Area

Stability
Stable
18
Jia, Gallagher, Saxena and Chen
From Support To Stability

Supporting Area

Stability
Stable
19
Unstable
Jia, Gallagher, Saxena and Chen
Overview
Input Segmentation
3D Block Fitting
Support and Stability
Evaluate Energy
Function
20
Jia, Gallagher, Saxena and Chen
Reasoning Through an Energy Function
Segmentation
Energy Function
1
1
F(S)   (si )   (si ,s j )
N i
M i, j
Use Support Relations, Stability, Other Box-based/RGB-D info as features.
Better Segmentation
Smaller F(S)
21
RGB-D
Worse Segmentation
Larger F(S)
Jia, Gallagher, Saxena and Chen
Energy Function: Single Box Potential
F(S) 
1
1

(s
)

 (si ,s j )


i
N i
M i, j
Features: minimum surface distance, visibility, single box stability, etc.
Worse
Box
Better
Box
22
Jia, Gallagher, Saxena and Chen
Energy Function: Pairwise Box Potential
F(S) 
1
1

(s
)

 (si ,s j )


i
N i
M i, j
Features: box intersection, support, supporting area distance etc.
Worse
Boundary
Better
Boundary
23
Jia, Gallagher, Saxena and Chen
Segmentation Energy
Function:
1
1
F(S)  (s )   (s ,s )
N
M
i
i
i
j
i, j
1.4
……
……
2.3
Segmentation
at one step
1.2
……
24
……
……
Jia, Gallagher, Saxena and Chen
Summary
Input Segmentation
3D Block Fitting
Support and Stability
Evaluate Energy
Function
25
Jia, Gallagher, Saxena and Chen
Experiments
26
Jia, Gallagher, Saxena and Chen
Experiments:

Block dataset

Cornell Support Object dataset (SOD)


300 RGB-D images with ground-truth segments and support
relations
NYU-2 RGB-D dataset
27
Jia, Gallagher, Saxena and Chen
Experiment: Segmentation Results

Pixel-wise object segmentation
accuracy:
28
Cornell Dataset
NYU Dataset
ECCV-12’
60.2%
60.1%
Ours
70.0%
61.7%
Jia, Gallagher, Saxena and Chen
Experiment: Segmentation Results
Input RGB-D images
29
Jia, Gallagher, Saxena and Chen
Experiments: Support Inference
Block
Dataset
Cornell
Dataset
Neighbor
80.6%
52.9%
Stability
91.7%
72.9%
30


Neighbor: object is supported by
its neighbors
Stability: trim unnecessary
support after reasoning
Jia, Gallagher, Saxena and Chen
Color Segmentation
D. Hoiem et al. ICCV, 2007;
P. Arbelaez et al. CVPR, 2012.
……
31
Blocks world revisited
A. Gupta et all, ECCV, 2010.
Semantic 3D Labeling
H. Koppula et. al. NIPS 2011.
Object Placement
Indoor Segmentation & Support
Y. Jiang et al. IJRR, 2012.
N. Silberman et al. ECCV 2012.
Jia, Gallagher, Saxena and Chen
Conclusion
 3D support and stability


Object segmentation in 3D scene


Based on box representations
Learning algorithm.
Future work



32
Non-uniform density
Semantic classification on blocks
Occluded supports
Jia, Gallagher, Saxena and Chen
3D-Based Reasoning with Blocks,
Support, and Stability
Zhaoyin Jia, Andrew Gallagher, Ashutosh Saxena,
Tsuhan Chen
Cornell University
Thanks. Questions?
33