Matching of Satellite, Aerial and Close-range Images
Luigi Barazzetti
Politecnico di Milano, ABC Departement
Lab. G@icarus, via Ponzio 31, Milan
Lab. IC&T, Corso Premessi Sposi, Lecco
luigi.barazzetti@polimi.it - http://www.icet-rilevamento.lecco.polimi.it
Introduction to image matching
Satellite, aerial and close-range data
The image matching problem
What stuff in the left image matches with stuff on the right?
‘Stuff’ … what does it mean? points, lines, areas, … ?
Image matching
2
Introduction to image matching
3
Satellite, aerial and close-range data
Image acquisition platforms (Photogrammetry and Remote Sensing)
> 700 km
Satellite images
Aerial images
UAVs
Close-range images
Waterproof equipment
<1m
Image matching
Introduction to image matching
4
Satellite, aerial and close-range data
Matching is easy! Why?
Point of view
Illumination
Good texture
Absence of occlusions
…
Figure by Noah Snavely
Image matching
Introduction to image matching
5
Satellite, aerial and close-range data
Harder case! Why?
An important consideration: human operators can easily extract
corresponding points, objects, …
Figure by Noah Snavely
Image matching
Introduzione al controllo statico
delle strutture
6
Harder? (Nasa Mars Rover images)
Figure by Noah Snavely
Image matching
Introduction to image matching
7
Satellite, aerial and close-range data
Harder? SIFT key-points (automatic)
Figure by Noah Snavely
Image matching
Introduction to image matching
8
Satellite, aerial and close-range data
Applications
Photogrammetry and Remote Sensing
•
fiducial mark measurement (target detection and matching)
•
tie point extraction
•
DTM/DSM generation
•
image registration
• …
Computer Vision
•
object recognition
•
multiple view analysis (3D reconstruction, panoramic images, HDR images, medical
data alignment, …)
•
motion capture, object tracking
•
…
… too many to be exhaustively listed
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Motion capture: tracking a limited number of points
Image matching
9
Introduction to image matching
Satellite, aerial and close-range data
2.5D vs 3D models from aerial images
Image matching
10
Introduction to image matching
Satellite, aerial and close-range data
Dense image matching and oblique imagery
Image matching
11
Introduction to image matching
Satellite, aerial and close-range data
Example: Apple iOS Maps (the Cathedral of Milan)
Image matching
12
Introduction to image matching
13
Satellite, aerial and close-range data
Example: Apple iOS Maps
The Statue of Liberty that
was
missing
before
now
shows a 3D imagery of the
landmark and the details of
the structures surrounding it
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Shaded and textured model
Image matching
14
Introduction to image matching
15
Satellite, aerial and close-range data
Point matching
• Goal: extraction of good points
• How can we define “good” and “bad” candidates?
• Moving the window in any direction gives a big change
“flat” region:
“edge”:
“corner”:
no change in all
no change along the
significant change in all
directions
edge direction
directions
Slide adapted from Darya Frolova, Denis Simakov, and Noah Snavely
Image matching
Introduction to image matching
16
Satellite, aerial and close-range data
Classification of point matching techniques
Intensity-based matching: image data is used in form of grey values. Most prominent
methods are cross-correlation and least squares matching (LS-matching). Also called
"area-based" matching. Give sub-pixel accuracy, in extreme cases 1/100 pixel and better
Feature-based matching: requires the extraction of basic image features, like blobs,
corners, junctions, edges, etc. first. Matching is performed between these features.
Features are sometimes more stable with regard to reflectance characteristics
Relational matching: uses geometrical or other relations between features and structures
(combination of features). Correspondence is established by tree-search techniques. These
methods are not very accurate but usually robust
Image matching
Introduction to image matching
17
Satellite, aerial and close-range data
Relational matching: uses geometrical or other relations between features and
structures (combination of features). These methods are not very accurate but
usually robust. They do not require good approximation
Structural description: set of primitives and their inter-relationships
Image matching
Introduction to image matching
18
Satellite, aerial and close-range data
Intensity-based matching: Normalized Cross-Correlation
The solution (best match) is found at max(r(x, y)). r is limited to the region [-1, 1]. False or weak matches
are indicated by small (r≤0,5). However, large r do not always indicate good, stable matches (e.g. in case
of multiple solutions or in cases of weak signal in template and patch)
r=
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: Foerstner operator
Image matching
19
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: modern operators
New trend based on local features with detectors and descriptors
Computer Vision methods: their use in Photogrammetry and RS is attractive
Example with the SIFT operator: panoramic photography
Brown, M. and Lowe, D.G., 2003. Recognizing Panoramas. International
Conference on Computer Vision, 2: 1218-1225.
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: modern operators
Picture with a Compact Camera (FOV)
50 x 35°
Do you want a 360° reconstruction?
Brown, M. and Lowe, D.G., 2003. Recognizing Panoramas. International
Conference on Computer Vision, 2: 1218-1225.
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: modern operators
Several pictures with a Compact Camera (FOV)
Do you want a 360° reconstruction?
Panoramic images (panoramas)
Brown, M. and Lowe, D.G., 2003. Recognizing Panoramas. International
Conference on Computer Vision, 2: 1218-1225.
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Image matching plus constraints (geometry)
Take pictures with a rotating camera
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: modern operators
Extract SIFT features: detector / descriptor
Geometrically invariant to similarity transforms
Some robustness to affine change
Automated methods  outliers !!!
Brown, M. and Lowe, D.G., 2003. Recognizing Panoramas.
International Conference on Computer Vision, 2: 1218-1225.
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: modern operators
Outlier rejection is fundamental  RANSAC
LS line
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: modern operators
Outlier rejection is fundamental  RANSAC
RANSAC
line
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: modern operators
Outlier rejection is fundamental  RANSAC (homography)
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching: modern operators
Final radiometric correction
Extension to N images and final blending
Brown, M. and Lowe, D.G., 2003. Recognizing Panoramas. International Conference on Computer Vision, 2: 1218-1225.
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching and satellite images
Similar approach but different operator (SURF)
Input data over Las Vegas:
•
1 ASTER image
5627x5001 pixels
•
1 LANSAT TM image 7811x7011 pixels
resolution 15 m
resolution 30 m
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching and satellite images
Image 1  4141 interest points
Image 2  14787 interest points
Descriptor comparison  502 matches (outliers??)
Image matching
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching and satellite images
Hypothesis: similarity transformation
 Xn 
 Yn 
 
 a b   Xo   c 
 cos α
=
 −b a  *  Yo
 +  d  S *  − sin α

    

sin α   Xo   DXo 
*  + 

cos α   Yo   DYo 
Yn
Y0
Yn
Yn
Y0
Y0
X0
Xn
Xn
X0
Translation
Scaling
Image matching
Xn
X0
Rotation
Introduction to image matching
Satellite, aerial and close-range data
Feature-based matching and satellite images
RANSAC: 395 inliers (from 502 matches)
Final Least Squares adjustment
Image matching
Introduction to image matching
Satellite, aerial and close-range data
References
Brown, M. and Lowe, D.G., 2003. Recognizing Panoramas. International Conference on Computer
Vision, 2: 1218-1225.
Brown, M. and Lowe, D.G., 2007. Automatic panoramic image stitching using invariant features.
International Journal of Computer Vision, 74(1): 59-73.
Gruen, A., 1985. Adaptative least squares correlation: a powerful image matching technique. South
African Journal of Photogrammetry, Remote Sensing and Cartography, 14(3): 175-187.
Gruen A., Geomatics in the 21th Century State of the art and future perspectives, Course at Politecnico
di Torino
Gruen A., Image Matching for DSM Generation, Compact Course at Politecnico di Milano
Some slides and pictures from A. Gruen, M. Scaioni, S. Seitz, R. Szeliski, N. Snavely, M Brown, D.
Lowe, D. Frolova, D. Simakov
Image matching