Image Stitching
Shangliang Jiang
Kate Harrison
What is image stitching?
What is image stitching?
Introduction
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
Introduction
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
– Human FOV
= 200 x 135°
Introduction
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
– Human FOV
= 200 x 135°
– Panoramic Mosaic
= 360 x 180°
Recognizing Panoramas
• 1D Rotations (q)
– Ordering  matching images
Recognizing Panoramas
• 1D Rotations (q)
– Ordering  matching images
Recognizing Panoramas
• 1D Rotations (q)
– Ordering  matching images
Recognizing Panoramas
• 1D Rotations (q)
– Ordering  matching images
• 2D Rotations (q, f)
– Ordering  matching images
Recognizing Panoramas
• 1D Rotations (q)
– Ordering  matching images
• 2D Rotations (q, f)
– Ordering  matching images
Recognizing Panoramas
• 1D Rotations (q)
– Ordering  matching images
• 2D Rotations (q, f)
– Ordering  matching images
Recognizing Panoramas
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
• Feature Matching
– SIFT Features
– Nearest Neighbor Matching
•
•
•
•
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
SIFT Features
• SIFT features are…
– Geometrically invariant to similarity transforms,
• some robustness to affine change
– Photometrically invariant to affine changes in
intensity
Overview
• Feature Matching
– SIFT Features
– Nearest Neighbor Matching
•
•
•
•
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Nearest Neighbor Matching
• Find k nearest neighbors for each feature
– k  number of overlapping images (we use k = 4)
• Use k-d tree
– k-d tree recursively bi-partitions data at mean in the
dimension of maximum variance
– Approximate nearest neighbors found in O(nlogn)
Overview
• Feature Matching
– SIFT Features
– Nearest Neighbor Matching
•
•
•
•
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
• Feature Matching
• Image Matching
– Random Sample Consensus (RANSAC) for
Homography
– Probabilistic model for verification
• Bundle Adjustment
• Image Compositing
• Conclusions
Overview
• Feature Matching
• Image Matching
– Random Sample Consensus (RANSAC) for
Homography
– Probabilistic model for verification
• Bundle Adjustment
• Image Compositing
• Conclusions
RANSAC for Homography
RANSAC for Homography
• SIFT features of two similar images
RANSAC for Homography
• SIFT features common to both images
RANSAC for Homography
• Select random subset of features (e.g. 6)
• Compute motion estimate
• Apply motion estimate to all SIFT features
• Compute error: feature pairs not
described
by motion estimate
• Repeat many times (e.g. 500)
• Keep estimate with best error
RANSAC for Homography
Overview
• Feature Matching
• Image Matching
– Random Sample Consensus (RANSAC) for
Homography
– Probabilistic model for verification
• Bundle Adjustment
• Image Compositing
• Conclusions
Probabilistic model for verification
• Potential problem:
• Two images don’t match…
• … but RANSAC found a motion estimate
• Do a quick check to make sure the images do
match
• MAP for inliers vs. outliers
Finding the panoramas
Finding the panoramas
Finding the panoramas
Overview
• Feature Matching
• Image Matching
– RANSAC for Homography
– Probabilistic model for verification
• Bundle Adjustment
• Image Compositing
• Conclusions
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
– Error function
• Image Compositing
• Conclusions
Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
– Error function
• Image Compositing
• Conclusions
Bundle Adjustment
• New images initialised with rotation, focal
length of best matching image
Bundle Adjustment
• New images initialised with rotation, focal
length of best matching image
Error function
• Sum of squared projection errors
–
–
–
–
n = #images
I(i) = set of image matches to image i
F(i, j) = set of feature matches between images i,j
rijk = residual of kth feature match between images i,j
• Robust error function
Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
– Error function
• Image Compositing
• Conclusions
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Blending
Gain compensation
How do we blend?
Linear blending
Multi-band blending
Multi-band Blending
• Burt & Adelson 1983
– Blend frequency bands over range  l
2-band Blending
Low frequency (l > 2 pixels)
High frequency (l < 2 pixels)
3-band blending
Band 1: high frequencies
3-band blending
Band 2: mid-range frequencies
3-band blending
Band 3: low frequencies
Panorama straightening
Heuristic: people tend to shoot pictures in a certain way
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Overview
•
•
•
•
•
Feature Matching
Image Matching
Bundle Adjustment
Image Compositing
Conclusions
Conclusion
Algorithm
AutoStitch program
AutoStitch.net
Open questions
• Advanced camera modeling
– radial distortion, camera motion, scene motion,
vignetting, exposure, high dynamic range, flash …
• Full 3D case – recognizing 3D objects/scenes
in unordered datasets
Credits
• Automatic Panoramic Image Stitching Using
Invariant Features, 2007
– Matthew Brown and David G. Lowe (Uni. of British
Columbia)
• Recognising Panoramas, 2003
– Matthew Brown and David G. Lowe (Uni. of British
Columbia)
– 2003
– Thanks for the slides!
• Image Alignment and Stitching: A Tutorial,
2006
– Richard Szeliski (Microsoft)
Questions?