Registration for Robotics
Patrick Mihelich
Kurt Konolige
Willow Garage
Stanford University
JD Chen
James Bowman
Helen Oleynikova
Freiburg TORO group:
Giorgio Grisetti
Cyrill Stachness
Rainer Kummerle
Outline
•
•
•
•
•
•
•
Applications
Overview of Registration
Front end and image matching
Visual Odometry
Place recognition
Global SBA
Extensions
Applications of Registration
• Photo tourism [Snavely, Seitz, Szeliski 2006]
Applications of Registration
• Object Modeling [Lovi et al. 2010]
• [Newcombe and Davison 2010]
Applications of Registration
• Face Tracking [WATSON: Morency 2003]
Applications of Registration
• Visual
Odometry
[Willow Garage]
Applications of Registration
• Visual
SLAM
[Willow Garage]
Outline
•
•
•
•
•
•
•
Applications
Overview of Registration
Front end, image matching
Visual Odometry
Place recognition
Global SBA
Extensions
Registration Elements
Incremental
Global
Extract keypoints
Compute features
Add new view to map
Match against local views
Match against global views
[Place recognition]
Estimate pose
Estimate pose
Optimize
Optimize
Mapping Systems
Map type
MonoSLAM
[Davison 2003]
World points
Parallel Tracking and Mapping Views and world
[Klein and Murray 2007]
View-based Maps
[Konolige et al. 2009]
points
Views [and
implicit world
points]
Map Representations
Map type
World points
Views and world
points
Views [and
implicit world
points]
Covariance Matrix over points
p0 p1 p2 …
p0
p1
p2
…
Extended Kalman Filter update
MonoSLAM
Map Representations
Map type
World points
Views and world
points
Views [and
implicit world
points]
Information matrix over views
and world points
c0 c1 c2 …
p0 p1 p2 …
c0
c1
c2
…
p0
p1
p2
…
Nonlinear least squares update
HDx = -g
Parallel Tracking and Mapping
Map Representations
Map type
World points
Views and world
points
Views [and
implicit world
points]
Information matrix over views
c0 c1 c2 …
c0
c1
c2
…
Nonlinear least squares update
HDx = -g
View-based Maps
Comparison of Mapping Methods
Map type
World points
Views and world
points
Views [and
implicit world
points]
Useful in small spaces
- World points for global matching
- World point Covar matrix update
Not as accurate [Strasdat et al. 2010]
Useful in small spaces
- World points for global matching
- View and point Info matrix update
Highly accurate
Useful in larger spaces
- View matching
- View Info matrix update
Not as accurate
Outline
•
•
•
•
•
•
•
Applications
Overview of Registration
Front end and image matching
Visual Odometry
Place recognition
Global SBA
Extensions
Interest points
• FAST [Rosten and Drummond 2006]
• Adaptive threshold
• Harris
• Gridding for spatial
diversity
• SIFT [Lowe 1999]
• SUSAN, CenSure, MSER, …
Feature matching
• SIFT / SURF features
• Tracking with motion model
• Planar patches
• Windowed brute-force
• Random-tree signatures [Calonder et al. 2009]
• KD tree nearest-neighbor
• …
930 features 354 matched
183 inliers
Outlier detection and pose estimation
• RANSAC to estimate pose
– 3 points for stereo (3D-3D)
– 3-point PnP for monocular (2D-3D)
• Outlier rejection
• Refinement
– Least-squares
– Reprojection
error
• Scavenging
930 features 354 matched
183 inliers
Outline
•
•
•
•
•
•
•
Applications
Overview of Registration
Front end and image matching
Visual Odometry
Place recognition
Global SBA
Extensions
Visual Odometry – Sliding Window
• 2-view pose estimates are unreliable
– Triangulation is narrow-baseline
• Multi-view estimates are more accurate
– Triangulation is wide-baseline
– More points, wider baseline => more accuracy
• Keyframes
– Many close-together views don’t add much
– Too far apart leads to low inlier count
• Sliding window of views
– Track points as long as possible
– ~20 – 40 keyframes in window
Visual Odometry – Bundle Adjustment
Measurement model:
Dz  z  z (x)
For Gaussian distributions (W 1 is covariance)
  log p( z | x)    logexp(
i
i
1
2
DziTWi Dzi )   DziTWi Dzi
i
i
[Gauss-Newton /
T
T
(
J
WJ
)

x


J
WDz
Normal]
J
z
x
H
c0 c1 c2 …
Sparseness of H
20 camera views
5000 features
H is (20x6 + 5000x3)2
c0
c1
c2
…
p0
p1
p2
…
p0 p1 p2 …
Visual Odometry examples
[courtesy Andrew Comport, INRIA]
•
Outdoor sequence
in Versailles
•
1 m stereo baseline,
narrow FOV
•
~400 m sequence
•
Average frame
distance: 0.6 m
•
Max frame distance:
1.1 m
Visual Odometry examples
Indoor Willow
Garage sequence
[courtesy Andrew Comport, INRIA]
10 cm stereo, wide
FOV
~100 m sequence
Average frame
distance: 0.3 m
26
Outline
•
•
•
•
•
•
•
Applications
Overview of Registration
Front end and image matching
Visual Odometry
Place recognition
Global SBA
Extensions
Place Recognition
• K-nearest neighbor feature
matching across database
of images
test
DB
– KD tree
[Lowe 1999,
Eade and Drummond 2008,
Williams et al. 2007]
• Bag-of-words
test
– visual vocabulary
[Stewenius and Nister 2006,
Cummins and Newman 2008]
DB
Place Recognition: Vocabulary Trees
[Nister and Stewenius CVPR06]
- “Bag of words” retrieval
- Vocab tree created offline
- For recognition:
- Image keypoints extracted
- Tree encodes approximate NN
search
- Inverted index of images at leaves
[Cummins and Newman ICRA07
Cullmer et al. ACRA08
Fraundorfer et al. IROS07]
[Image from Nister and Stewenius CVPR06]
Place Recognition: Vocabulary Trees
Performance on Indoor dataset
View-based Maps [Konolige et al. 2009]
Performance on Indoor dataset
View-based Maps [Konolige et al. 2009]
Performance on Indoor dataset
Trajectory synthesis
Outline
•
•
•
•
•
•
•
Applications
Overview of Registration
Front end and image matching
Visual Odometry
Place recognition
Global SBA
Extensions
Comparison of Mapping Methods
Map type
World points
Views and world
points
Views [and
implicit world
points]
Useful in small spaces
- World points for global matching
- World point Covar matrix update
Not as accurate [xxx and Davison 2010]
Useful in small spaces
- World points for global matching
- View and point Info matrix update
Highly accurate
Useful in larger spaces
- View matching
- View Info matrix update
Not as accurate
Fast SBA
[Konolige BMVC 2010, Jeong et al. CVPR 2010]
• Take advantage of sparse secondary structure of Hessian
• Use fast linear solvers
– Davis’ CHOLMOD
– Block preconditioned conjugate gradient
Fast SBA in VSLAM
Fast SBA in Reconstruction
Open-Source in ROS
• sba package for fast SBA
• frame_common and posest for front-end
and pose estimation
• vocabulary_tree for place recognition
• vslam_system for Visual Odometry and
VSLAM