‫פברואר ‪ ,1968‬בסיום מסע למצדה‬
‫יוני ‪ ,1969‬אודיטוריום פיסיקה‪ ,‬בית‪-‬בירם‬
‫פברואר ‪ ,1968‬בסיום מעלה האיסיים‬
‫פברואר ‪ ,1969‬עובדת (מסע גדנ"ע)‬
Image Registration of Remotely Sensed Data
Nathan S. Netanyahu
Dept. of Computer Science, Bar-Ilan University
and Center for Automation Research, University of Maryland
Collaborators:
Jacqueline Le Moigne
David M. Mount
Arlene A. Cole-Rhodes, Kisha L. Johnson
Roger D. Eastman
Ardeshir Goshtasby
Jeffrey G. Masek, Jeffrey Morisette
Antonio Plaza
Harold Stone
Ilya Zavorin
Shirley Barda, Boris Sherman
Yair Kapach
NASA / Goddard Space Flight Center
University of Maryland
Morgan State University, Maryland
Loyola College of Maryland
Wright State University, Ohio
NASA / Goddard Space Flight Center
University of Extremadura, Spain
NEC Research Institute (Ret.)
CACI International, Maryland
Applied Materials, Inc., Israel
Bar-Ilan University
What is Image Registration / Alignment / Matching?
The above image is rotated and shifted
with respect to the left image.
2
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Motivation
• A crucial, fundamental step in image
analysis tasks, where final information is
obtained by the combination / integration
of multiple data sources.
3
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Motivation / Applications
• Computer Vision (target localization, quality
control, stereo matching)
• Medical Imaging (combining CT and MRI data,
tumor growth monitoring, treatment verification)
• Remote Sensing (classification, environmental
monitoring, change detection, image mosaicing,
weather forecasting, integration into GIS)
4
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Application Examples
• Global Wafer Alignment
5
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Wafer Alignment (cont’d)
Courtesy: Shirley Barda and Boris Sherman, Applied Materials, Inc.
6
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Application Examples (cont’d)
• Integration of medical images
Registration of MR and PET images of the same person (courtesy: A. Goshtasby)
7
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Application Examples (cont’d)
• Change Detection
1975
2000
Satellite images of Dead Sea, United Nations Environment Programme (UNEP) website
8
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Change Detection (cont’d)
1990
2005
Satellite images of Amona hilltop, Peace Now website
9
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Change Detection (cont’d)
IKONOS images of Iran’s Bushehr nuclear plant, GlobalSecurity.org
10
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
What is the “Big Deal”?
Bydo
matching
control
points,
e.g., corners, high-curvature points.
How
humans
solve
this?
Zitova and Flusser, IVC 2003
11
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Automatic Image Registration
•
Books:
–
–
–
–
•
Surveys:
–
–
–
•
A Survey of Image Registration Techniques, ACM Comp. Surveys, L.G. Brown, 1992
A Survey of Medical Image Registration, Medical Image Analysis, J.B.A. Maintz and M.A. Viergever, 1998
Image Registration Methods: A Survey, Image and Vision Computing, B. Zitova and J. Flusser, 2003
Sample Papers:
–
–
–
–
–
–
12
Medical Image Registration, J. Hajnal, D.J. Hawkes, and D. Hill (Eds.), CRC 2001
Numerical Methods for Image Registration, J. Modersitzki, Oxford University Press 2004
2-D and 3-D Image Registration, A. Goshtasby, Wiley 2005
Image Registration for Remote Sensing, J. LeMoigne, N.S. Netanyahu, and R.D. Eastman (Eds.), Cambridge
University Press, in preparation.
Image Sequence Enhancement Using Sub-pixel Displacements, CVPR, D. Keren, S. Peleg, and R. Brada, 1988
Improving Resolution by Image Registration, CVGIP, M. Irani and S. Peleg, 1991
Computing Correspondence Based on Regions and Invariants without Feature Extraction and Segmentation,
CVPR, C. Lee, D. Cooper, and D. Keren, 1993
Robust Multi-Image Sensor Image Alignment, ICCV, M. Irani and P. Anandan, 1998
Fast Block Motion Estimation Using Gray Code Kernels, Israel CV Workshop, Y. Moshe and H. Hel-Or, 2006
Image Matching Using Photometric Information, Israel CV Workshop, M. Kolomenkin and I. Shimshoni, 2006
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Automatic Image Registration Components
0. Preprocessing
– Image enhancement, cloud detection, region of interest masking
1. Feature extraction (control points)
– Corners, edges, wavelet coefficients, segments, regions, contours
2. Feature matching
– Spatial transformation (a priori knowledge)
– Similarity metric (correlation, mutual information, Hausdorff distance)
– Search strategy (global vs. local, multiresolution, optimization)
3. Resampling
Tp
I2
I1
13
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Example of Image Registration Steps
Feature extraction
Resampling
Registered images after transformation
Zitova and Flusser, IVC 2003
14
Feature matching
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Automatic Image Registration
for Remote Sensing
• Sensor webs, constellation, and exploration
• Selected NASA Earth science missions
• Domain-dependent characteristics
15
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Sensor Webs, Constellation, and Exploration
Satellite/Orbiter,
and In-Situ Data
Automatic
Multiple Source
Integration
Planning and
Scheduling
Intelligent Navigation
and Decision Making
16
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Selected NASA Earth Science Missions
0.1
0.4
0.5
0.6
0.7
Instrument
(Spat. Resol.)
Number of
Channels
AVHRR (D)
(1.1 km)
5 Channels
1
TRMM/VIRS
(2 km)
5 Channels
1
Lands at4-MSS
(80 m)
4 Channels
Ultra
Violet
Visible
Lands at5&7-TM&ETM+
(30 m)
7 Channels
2.0
3.0
2
3
2
3
4.0
5.0
6.0
7.0
8.0
Mid-IR
2
2
9.0
10.0
11.0
12.0
13.0
14.0
Thermal-IR
3
4
5
3
4
5
4
4
5
7
6
1
1
JERS-1
8 Channels
(Ch1-4:18m; Ch5-8:24m)
EO/1
ALI-MultiSpectr.
ALI-Panchrom.
Hyperion
(30m)
LAC
1.3
Near-IR
1
1
Lands at7-Panchromatic
(15m)
IRS-1
4 Channels
LISS-I (73m) - LISS-2 (36.5m)
SPOT-HRV Panchromatic
(10m)
1 Channel
Spot-HRV Multis pectral
(20 m)
3 Channels
MODIS
36 Channels
(Ch1-2:250 m;3-7:500m;8-36:1km)
1.0
4
2
3
1
2
3
&4
3
5
678
6
7
1
3,
8-10
1'
9 Channels (30m)
1 Channel (10m)
220 Channels
1
2
1 1,
4,
12
1,
13,
14
2
3
1
2,
1619
15
5
26
5'
4
5
7
4
5-9
20-25
27
28
29
30
31
32
33-36
1
1 to 220
256 Channels
1 to 256
(250m)
IKONOS-Panchromatic
(1m)
1 Channel
IKONOS-MS
4 Channels (4m)
ASTER
14 Channels
1
1
2
3
4
1
2
3
2 3
4
5
6
7
(Ch1-3:15m;4-9:30m;10-14:90m)
CZCS
(1 km)
6 Channels
SeaWiFS (D)
(1.1 km)
8 Channels
TOVS-HIRS2 (D)
20 Channels
1
1
2 3
4
5
20
8
19
(15 km)
GOES
5 Channels
(1 km:1, 4km:2,4&5, 8km:3)
METEOSAT
3 Channels
(V:2.5km,WV&IR:5km)
17
.
1
Visible
13,14
1 0,1 1 1 2
2
1
8
17
to
13
12
3
11
10
9
8
7 to 1
4
Water
Vapor
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
5
IR
15.0
MODIS Satellite System
From the NASA MODIS website
18
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
MODIS Satellite Specifications
19
Orbit:
705 km, 10:30 a.m. descending node (Terra) or 1:30 p.m. ascending
node (Aqua), sun-synchronous, near-polar, circular
Scan Rate:
20.3 rpm, cross track
Swath Dimensions:
2330 km (cross track) by 10 km (along track at nadir)
Telescope:
17.78 cm diam. off-axis, afocal (collimated), with intermediate field stop
Size:
1.0 x 1.6 x 1.0 m
Weight:
228.7 kg
Power:
162.5 W (single orbit average)
Data Rate:
10.6 Mbps (peak daytime); 6.1 Mbps (orbital average)
Quantization:
12 bits
Spatial Resolution:
250 m (bands 1-2)
500 m (bands 3-7)
1000 m (bands 8-36)
Design Life:
6 years
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Landsat 7 Satellite System
New Orleans, before and after Katrina 2005 (from the USGS Landsat website)
20
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Landsat 7 Satellite Specifications
Launch
Date
April 15, 1999
Vehicle
Delta II
Site
Vandenberg AFB
Orbit Characteristics
21
Reference system
WRS-2
Type
Sun-synchronous, near-polar
Altitude
705 km (438 mi)
Inclination
98.2°
Repeat cycle
16 days
Swath width
185 km (115 mi)
Equatorial crossing time
10:00 AM +/- 15 minutes
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
IKONOS Satellite System
22
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
IKONOS Satellite Specifications
23
Launch Date
24 September 1999
Vandenberg Air Force Base, California, USA
Operational Life
Over 7 years
Orbit
98.1 degree, sun synchronous
Speed on Orbit
7.5 kilometers per second
Speed Over the Ground
6.8 kilometers per second
Number of Revolutions Around the Earth
14.7 every 24 hours
Orbit Time Around the Earth
98 minutes
Altitude
681 kilometers
Resolution
Nadir:
0.82 meters panchromatic
3.2 meters multispectral
26° Off-Nadir
1.0 meter panchromatic
4.0 meters multispectral
Image Swath
11.3 kilometers at nadir
13.8 kilometers at 26° off-nadir
Equator Crossing Time
Nominally 10:30 a.m. solar time
Revisit Time
Approximately 3 days at 40° latitude
Dynamic Range
11-bits per pixel
Image Bands
Panchromatic, blue, green, red, near IR
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Domain-Dependent Characteristics
• Very large images (~ 6200 x 5700 of typical
Landsat 7 scene)
• Practically “flat”, 2D images
• Rigid/similar transformations
• A priori knowledge (e.g., small rotation and scale)
24
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Challenges in Processing of
Remotely Sensed Data
• Multisource data
• Multi-temporal data
• Various spatial resolutions
• Various spectral resolutions
• Subpixel accuracy
• 1 pixel misregistration ≥ 50% error in NDVI classification
• Computational efficiency
• Fast procedures for very large data sets
• Accuracy assessment
• Synthetic data
• “Ground Truth" (manual registration?)
• Consistency ("circular" registrations) studies
25
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Fusion of Multi-temporal Images
Improvement of NDVI classification accuracy due to fusion of multi-temporal
SAR and Landsat TM over farmland in The Netherlands (source: The Remote
Sensing Tutorial by N.M. Short, Sr.)
26
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Integration of Multiresolution Sensors
Registration of Landsat ETM+ and IKONOS images over coastal VA and agricultural Konsa site
(source: LeMoigne et al., IGARSS 2003)
27
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Feature Extraction
Gray levels
BPF wavelet coefficients
Binary feature map
Top 10% of wavelet coefficients (due to Simoncelli) of Landsat image over Washington, D.C.
(source: N.S. Netanyahu, J. LeMoigne, and J.G. Masek, IEEE-TGRS, 2004)
28
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Feature Extraction (cont’d)
Image features (extracted from two overlapping scenes over D.C.) to be matched
29
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Feature Matching / Transformations
• Given a reference image, I1(x, y), and a sensed image I2(x, y), find the
mapping (Tp, g) which “best” transforms I1 into I2, i.e.,
I 2 ( x, y )  g ( I1 (Tp ( x, y ), Tp ( x, y ))),
where Tp denotes spatial mapping and g denotes radiometric mapping.
•
Spatial transformations:
Translation, rigid, affine, projective, perspective, polynomial
•
Radiometric transformations (resampling):
Nearest neighbor, bilinear, cubic convolution, spline
30
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Transformations (cont’d)
Objective: Find parameters of a transformation Tp (consisting of a translation,
a rotation, and an isometric scale) that maximize similarity measure.
x '  s cos   x  s sin   y  t x
y '  s sin   x  s cos   y  t y
 s cos 

Tp   s sin 
 0

31
 s sin 
s cos 
0
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
tx 

ty 
1 
Similarity Measures (cont’d)
• L2 norm:
Minimize sum of squared errors over overlapping subimage
  I
2
 Tp  I1  
2
• Normalized cross correlation (NCC):
Maximize normalized correlation between the images
  I ( x, y)  I   I
1
NCC ( I1 , I 2 ) 
x
1
2
y
( x, y )  I 2 
  I1 ( x, y)  I1     I 2 ( x, y)  I 2 
2
x
32
y
x
y
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
2
Similarity Measures (cont’d)
• Mutual information (MI):
Maximize the degree of dependence between the images
 pI1 , I2  g1 , g 2  
MI  I1 , I 2    pI1 , I2  g1 , g 2   log 
,

g1 g 2
 pI1  g1   pI2  g 2  
or using histograms, maximize
1
MI  I1 , I 2  
M
33
 MhI1 , I2  g1 , g 2  
hI1 , I2  g1 , g 2   log 



g1 g 2
 hI1  g1   hI2  g 2  
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Similarity Measures (cont’d), An Example
MI vs. L2-norm and NCC applied to Landsat 5 images
(source: Chen, Varshney, and Arora, IEEE-TGRS, 2003)
34
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Similarity Measures (cont’d), an MI Example
Source: Cole-Rhodes et al., IEEE-TIP, 2003
35
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Similarity Measures (cont’d)
• (Partial) Hausdorff distance (PHD):
H K  I1 , I 2   K pth1I1 min p2I2 dist  p1 , p2  ,
where 1  K  I1
K 1
K 2
K  I1
36
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Similarity Measures (cont’d), PHD Example
PHD-based matching of Landsat images over D.C. (source: Netanyahu,
LeMoigne, and Masek, IEEE-TGRS, 2004)
37
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Feature Matching / Search Strategy
• Exhaustive search
• Fast Fourier transform (FFT)
• Optimization (e.g., gradient descent; Thévenaz,
Ruttimann, and Unser (TRU), 1998; Spall, 1992)
• Robust feature matching (e.g., efficient subdivision
and pruning of transformation space)
38
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Computational Efficiency
• Extraction of corresponding regions of interest (ROI)
• Hierarchical, pyramid-like approach
• Efficient search strategy
39
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Computational Efficiency (cont’d), ROI Extraction
UTM of 4 scene corners known from systematic correction
Reference Scene
Extract reference chips and
corresponding input
windows using mathematical
morphology
2.
Register each (chip-window)
pair and record pairs of
registered chip corners
(refinement step)
3.
Compute global registration
from multiple local ones
4.
Compute correct UTM of 4
scene corners of input scene
Input Scene
Advantages:
• Eliminates need for chip database
• Cloud detection can easily be included in process
• Process any size images
• Initial registration closer to optimal registration =>
reduces computation time and increases accuracy.
40
1.
Source: Plaza, LeMoigne, and
Netanyahu, MultiTemp, 2005
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Computational Efficiency (cont’d),
An Example of a Pyramid-Like Approach
32 x 32
64 x 64
0
1
2
128 x 128
 
ss
t x  2t x
t y  2t y
256 x 256
41
3
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
IR Example Using Partial Hausdorff Distance
64 x 64
128 x 128
256 x 256
42
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
IR Example Using PHD (cont’d)
Source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004
43
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
IR Components (Revisited)
Features
Similarity
Measure
Strategy
44
Gray Levels
Correlation
Fast Fourier Transform
Wavelets or
Wavelet-Like
Edges
L2-Norm
Gradient
Descent
Mutual Information
Thevenaz,
Ruttimann,
Unser
Optimization
Hausdorff Distance
Spall’s
Optimization
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Robust
Feature
Matching
IR Components (Revisited)
Features
Similarity
Measure
Strategy
FFT
Correlation
L2-Norm
Gradient
Descent
Thevenaz,
Ruttimann,
Unser
Optimization
L2-Norm
MI
MI
Hausdorff Distance
Robust
Feature
Matching
Spall’s
Optimization
Gradient
Descent
45
Simoncelli
BPF
Spline or Simoncelli
LPF
Gray Levels
Thevenaz,
Ruttimann,
Unser
Optimization
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Spall’s
Optimization
Goals of a Modular Image Registration Framework
• Testing framework to:
– Assess various combinations of components
– Assess a new registration component
• Web-based registration tool would allow user to “schedule”
combination of components, as a function of:
– Application
– Available computational resources
– Required registration accuracy
•
Prototype of web-based registration toolbox:
– Several modules based on wavelet decomposition
– Java implementation; JNI-wrapped functions
46
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Web-Based Image Registration Toolbox
TARA (“Toolbox for Automated Registration & Analysis”)
47
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Web-Based Image Registration Toolbox
TARA (“Toolbox for Automated Registration & Analysis”)
48
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Current and Future Work
• Conclude component evaluation
– Sensitivity to noise, radiometric transformations, initial
conditions, and computational requirements
– Integration of digital elevation map (DEM) information
• Build operational registration framework/toolbox
– Web-based
– Applications:
• EOS validation core sites
• Other EOS satellites (e.g., Hyperion vs. ALI registration) and beyond
• Image fusion, change detection
49
Akavia Memorial Lecture, Haifa Univ., 11.01.2007
Back to the Moon
View of the moon, as seen from Apollo 11 (1969(
“Icarus fell because he flew too close to the sun. Columbia - and the
whole American manned space program today - fell because it flies too
close to the Earth” – Charles Krauthammer (Feb. 2003)