CPSC 641:
Image Registration
Jinxiang Chai
Review
Image warping
Image morphing
Image Warping
Warping function
- similarity, affine, projective etc
Image warping
x
- forward warping and two-pass 1D
y
warping
- backward warping
u
Inverse
v
forward
Resampling filter
- point sampling
- bilinear filter
- anisotropic filter
S(x,y)
T(u,v)
Image Morphing
Point based image morphing
Vector based image morphing
Image Registration
Image warping: given h and f, compute g
g(x) = f(h(x))
g?
f
h
Image registration: given h and g, compute f
g
f
h?
Why Image Registration?
Lots of uses
– Correct for camera jitter
(stabilization)
– Align images (mosaics)
– View morphing
– Image based modeling/rendering
– Special effects
– Etc.
Image Registration
How do we align two images automatically?
Two broad approaches:
– Feature-based alignment
• Find a few matching features in both images
• compute alignment
– Direct (pixel-based) alignment
• Search for alignment where most pixels agree
Outline
Image registration
- feature-based approach
- pixel-based approach
Readings
Bergen et al. Hierarchical model-based motion
estimation. ECCV’92, pp. 237–252.
Shi, J. and Tomasi, C. (1994). Good features to
track. In CVPR’94, pp. 593–600.
Baker, S. and Matthews, I. (2004). Lucas-kanade
20 years on: A unifying framework. IJCV, 56(3),
221–255.
Outline
Image registration
- feature-based approach
- pixel-based approach
Feature-based Alignment
1. Find a few important features (aka Interest
Points)
2. Match them across two images
3. Compute image transformation
Feature-based Alignment
1. Find a few important features (aka Interest
Points)
2. Match them across two images
3. Compute image transformation
How to choose features
– Choose only the points (“features”) that are
salient, i.e. likely to be there in the other image
Feature-based Alignment
1. Find a few important features (aka Interest
Points)
2. Match them across two images
3. Compute image transformation
How to choose features
– Choose only the points (“features”) that are
salient, i.e. likely to be there in the other image
– How to find these features?
• windows where
• Harris Corner detector
has two large eigenvalues
Feature Detection
-Two images taken at the same place with different
angles
- Projective transformation H3X3
Feature Matching
?
-Two images taken at the same place with different
angles
- Projective transformation H3X3
Feature Matching
• Intensity/Color similarity
– The intensity of pixels around the
corresponding features should have similar
intensity
– SSD, Cross-correlation
• Distance constraint
– The displacement of features should be
smaller than a user-defined threshold
Feature-space Outlier Rejection
Can we now compute H3X3 from the blue points?
– No! Still too many outliers…
– What can we do?
Robust Estimation
What if set of matches contains gross outliers?
RANSAC
Objective
Robust fit of model to data set S which contains outliers
Algorithm
(i) Randomly select a sample of s data points from S and
instantiate the model from this subset.
(ii) Determine the set of data points Si which are within a
distance threshold t of the model. The set Si is the
consensus set of samples and defines the inliers of S.
(iii) If the subset of Si is greater than some threshold T, reestimate the model using all the points in Si and terminate
(iv) If the size of Si is less than T, select a new subset and
repeat the above.
(v) After N trials the largest consensus set Si is selected, and
the model is re-estimated using all the points in the
subset Si
RANSAC
Repeat M times:
– Sample minimal number of matches to
estimate two view relation (affine, perspective,
etc).
– Calculate number of inliers or posterior
likelihood for relation.
– Choose relation to maximize number of
inliers.
RANSAC Line Fitting Example
Task:
Estimate best line
RANSAC Line Fitting Example
Sample two points
RANSAC Line Fitting Example
Fit Line
RANSAC Line Fitting Example
Total number of
points within a
threshold of line.
RANSAC Line Fitting Example
Repeat, until get a
good result
RANSAC Line Fitting Example
Repeat, until get a
good result
RANSAC Line Fitting Example
Repeat, until get a
good result
How Many Samples?
Choose N so that, with probability p, at least one random
sample is free from outliers. e.g. p=0.99
1  1  e 
s N
 1 p

N  log 1  p  / log 1  1  e 
s

proportion of outliers e
s
2
3
4
5
6
7
8
5%
2
3
3
4
4
4
5
10% 20% 25% 30% 40% 50%
3
5
6
7
11
17
4
7
9
11
19
35
5
9
13
17
34
72
6
12
17
26
57 146
7
16
24
37
97 293
8
20
33
54 163 588
9
26
44
78 272 1177
How Many Samples?
Choose N so that, with probability p, at least one random
sample is free from outliers. e.g. p=0.99
1  1  e 
s N
 1 p

N  log 1  p  / log 1  1  e 
s

proportion of outliers e
Affine
transform
s
2
3
4
5
6
7
8
5%
2
3
3
4
4
4
5
10% 20% 25% 30% 40% 50%
3
5
6
7
11
17
4
7
9
11
19
35
5
9
13
17
34
72
6
12
17
26
57 146
7
16
24
37
97 293
8
20
33
54 163 588
9
26
44
78 272 1177
How Many Samples?
Choose N so that, with probability p, at least one random
sample is free from outliers. e.g. p=0.99
1  1  e 
s N
 1 p

N  log 1  p  / log 1  1  e 
s

proportion of outliers e
Projective
transform
s
2
3
4
5
6
7
8
5%
2
3
3
4
4
4
5
10% 20% 25% 30% 40% 50%
3
5
6
7
11
17
4
7
9
11
19
35
5
9
13
17
34
72
6
12
17
26
57 146
7
16
24
37
97 293
8
20
33
54 163 588
9
26
44
78 272 1177
RANSAC for Estimating
Projective Transformation
RANSAC loop:
1.
2.
3.
4.
5.
Select four feature pairs (at random)
Compute the transformation matrix H (exact)
Compute inliers where SSD(pi’, H pi) < ε
Keep largest set of inliers
Re-compute least-squares H estimate on all of the
inliers
RANSAC
Outline
Image registration
- feature-based approach
- pixel-based approach
Direct (pixel-based) Alignment :
Optical flow
Will start by estimating motion of each pixel separately
Then will consider motion of entire image
Problem Definition: Optical Flow
How to estimate pixel motion from image H to image I?
Problem Definition: Optical Flow
How to estimate pixel motion from image H to image I?
– Solve pixel correspondence problem
• given a pixel in H, look for nearby pixels of the same
color in I
Problem Definition: Optical Flow
How to estimate pixel motion from image H to image I?
– Solve pixel correspondence problem
• given a pixel in H, look for nearby pixels of the same
color in I
Key assumptions
– color constancy: a point in H looks the same in I
• For grayscale images, this is brightness constancy
– small motion: points do not move very far
This is called the optical flow problem
Optical Flow Constraints
Let’s look at these constraints more closely
– brightness constancy: Q: what’s the equation?
Optical Flow Constraints
Let’s look at these constraints more closely
– brightness constancy: Q: what’s the equation?
H(x,y) - I(x+u,v+y) = 0
Optical Flow Constraints
Let’s look at these constraints more closely
– brightness constancy: Q: what’s the equation?
H(x,y) - I(x+u,v+y) = 0
– small motion: (u and v are less than 1 pixel)
• suppose we take the Taylor series expansion of I:
Optical Flow Constraints
Let’s look at these constraints more closely
– brightness constancy: Q: what’s the equation?
H(x,y) - I(x+u,v+y) = 0
– small motion: (u and v are less than 1 pixel)
• suppose we take the Taylor series expansion of I:
Optical Flow Equation
Combining these two equations
Optical Flow Equation
Combining these two equations
Optical Flow Equation
Combining these two equations
Optical Flow Equation
Combining these two equations
Optical Flow Equation
Combining these two equations
In the limit as u and v go to zero, this becomes
exact
Optical Flow Equation
How many unknowns and equations per pixel?
Optical Flow Equation
How many unknowns and equations per pixel?
Intuitively, what does this constraint mean?
Optical Flow Equation
How many unknowns and equations per pixel?
Intuitively, what does this constraint mean?
– The component of the flow in the gradient direction is
determined
– The component of the flow parallel to an edge is
unknown
Optical Flow Equation
How many unknowns and equations per pixel?
Intuitively, what does this constraint mean?
– The component of the flow in the gradient direction is
determined
– The component of the flow parallel to an edge is
unknown
This explains the Barber Pole illusion
http://www.sandlotscience.com/Ambiguous/barberpole.htm
Optical Flow Equation
How many unknowns and equations per pixel?
Intuitively, what does this constraint mean?
– The component of the flow in the gradient direction is
determined
– The component of the flow parallel to an edge is
unknown
This explains the Barber Pole illusion
http://www.sandlotscience.com/Ambiguous/barberpole.htm
Aperture Problem
Aperture Problem
Stripes moved
upwards 6 pixels
Stripes moved
left 5 pixels
Solving the Aperture Problem
How to get more equations for a pixel?
– Basic idea: impose additional constraints
• most common is to assume that the flow field is smooth locally
• one method: pretend the pixel’s neighbors have the same (u,v)
– If we use a 5x5 window, that gives us 25 equations per pixel!
RGB Version
How to get more equations for a pixel?
– Basic idea: impose additional constraints
• most common is to assume that the flow field is smooth locally
• one method: pretend the pixel’s neighbors have the same (u,v)
– If we use a 5x5 window, that gives us 25 equations per pixel!
Lukas-Kanade Flow
Prob: we have more equations than unknowns
Lukas-Kanade Flow
Prob: we have more equations than unknowns
Solution: solve least squares problem
Lukas-Kanade Flow
Prob: we have more equations than unknowns
Solution: solve least squares problem
– minimum least squares solution given by solution
(in d) of:
Lukas-Kanade Flow
– The summations are over all pixels in the K
x K window
– This technique was first proposed by Lukas
& Kanade (1981)
Lukas-Kanade Flow
When is this Solvable?
• ATA should be invertible
• ATA should not be too small due to noise
– eigenvalues l1 and l2 of ATA should not be too small
• ATA should be well-conditioned
– l1/ l2 should not be too large (l1 = larger eigenvalue)
Edge
Bad for motion estimation
- large l1, small l2
Low Texture Region
Bad for motion estimation:
- gradients have small magnitude
- small l1, small l2
High Textured Region
Good for motion estimation:
- gradients are different, large magnitudes
- large l1, large l2
Observation
This is a two image problem BUT
– Can measure sensitivity by just looking at one of the
images!
– This tells us which pixels are easy to track, which are
hard
• very useful later on when we do feature tracking...
Errors in Lucas-Kanade
What are the potential causes of errors in this
procedure?
– Suppose ATA is easily invertible
– Suppose there is not much noise in the image
Errors in Lucas-Kanade
What are the potential causes of errors in this
procedure?
– Suppose ATA is easily invertible
– Suppose there is not much noise in the image
When our assumptions are violated
– Brightness constancy is not satisfied
– The motion is not small
– A point does not move like its neighbors
• window size is too large
• what is the ideal window size?
Iterative Refinement
Iterative Lukas-Kanade Algorithm
1. Estimate velocity at each pixel by solving LucasKanade equations
2. Warp H towards I using the estimated flow field
- use image warping techniques
3. Repeat until convergence
Revisiting the Small Motion
Assumption
Is this motion small enough?
– Probably not—it’s much larger than one pixel (2nd order terms
dominate)
– How might we solve this problem?
Reduce the Resolution!
Coarse-to-fine Optical Flow
Estimation
u=1.25 pixels
u=2.5 pixels
u=5 pixels
image H
Gaussian pyramid of image H
u=10 pixels
image II
image
Gaussian pyramid of image I
Coarse-to-fine Optical Flow
Estimation
run iterative L-K
warp & upsample
run iterative L-K
.
.
.
image JH
Gaussian pyramid of image H
image II
image
Gaussian pyramid of image I
Beyond Translation
So far, our patch can only translate in (u,v)
What about other motion models?
– rotation, affine, perspective
Warping Function
w(x;p) describes the geometric relationship
between two images:
x
Input Image I(x)
 wx ( x; p) 

w( x; p)  

w
(
x
;
p
)
y


Template Image H(x)
Warping Functions
Translation:
Affine:
Perspective:
 x  p1 

w( x; p)  
 y  p2 
 p1 x  p2 y  p3 

w( x; p)  
 p4 x  p5 y  p6 


w( x; p)  



p1 x  p2 y  p3 

p7 x  p8 y  1 
p4 x  p5 y  p6 
p7 x  p8 y  1 
Image Registration
Find the warping parameter p that minimizes the intensity
difference between template image and the warped
input image
Image Registration
Find the warping parameter p that minimizes the intensity
difference between template image and the warped
input image
Mathematically, we can formulate this as an optimization
problem:
2
arg min  I ( w( x; p))  H ( x)
p
x
Image Registration
Find the warping parameter p that minimizes the intensity
difference between template image and the warped
input image
Mathematically, we can formulate this as an optimization
problem:
2
arg min  I ( w( x; p))  H ( x)
p
x
The above problem can be solved by many optimization
algorithm:
- Steepest descent
- Gauss-newton
- Levenberg-marquardt, etc.
Image Registration
Find the warping parameter p that minimizes the intensity
difference between template image and the warped
input image
Mathematically, we can formulate this as an optimization
problem:
2
arg min  I ( w( x; p))  H ( x)
p
x
The above problem can be solved by many optimization
algorithm:
- Steepest descent
- Gauss-newton
- Levenberg-marquardt, etc
Image Registration
Mathematically, we can formulate this as an
optimization problem:
2
arg min  I ( w( x; p))  H ( x)
p
x
Image Registration
Mathematically, we can formulate this as an
optimization problem:
2
arg min  I ( w( x; p))  H ( x)
p
x
Similar to optical flow:
arg min
p
2


I
(
w
(
x
;
p


p
))

H
(
x
)

x
Taylor series expansion
Image Registration
Mathematically, we can formulate this as an
optimization problem:
2
arg min  I ( w( x; p))  H ( x)
p
x
Similar to optical flow:
arg min
p
2


I
(
w
(
x
;
p


p
))

H
(
x
)

Taylor series expansion


W
 arg min   I ( w( x; p))  I
p  H ( x)
p
p
x 

x
x
2
Image Registration
Mathematically, we can formulate this as an
optimization problem:
2
arg min  I ( w( x; p))  H ( x)
p
x
Similar to optical flow:
arg min
p
2


I
(
w
(
x
;
p


p
))

H
(
x
)

Taylor series expansion


W
 arg min   I ( w( x; p))  I
p  H ( x)
p
p
x 

x
x
 I
I  
 x
I 
y 
Image gradient
2
Image Registration
Mathematically, we can formulate this as an
optimization problem:
2
arg min  I ( w( x; p))  H ( x)
p
x
Similar to optical flow:
arg min
p
2


I
(
w
(
x
;
p


p
))

H
(
x
)

Taylor series expansion
2


W
 arg min   I ( w( x; p))  I
p  H ( x)
p
p
x 
 w
x
1 0

p 0 1
x
 I
I  
 x
I 
y 
Image gradient
 wx
w  p1

p  wy
 p1
wx 
pn 

wy 
...
p2 
translation
...
Jacobian matrix
w  x y 1 0 0 0

p 0 0 0 x y 1
……
affine
Gauss-newton Optimization


W
arg min   I ( w( x; p))  I
p  H ( x)
p
p
x 

2
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero
Gauss-newton Optimization


W
arg min   I ( w( x; p))  I
p  H ( x)
p
p
x 

2
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero
T
 W  

W

I
I
(
w
(
x
;
p
))


I

p

H
(
x
)
x  p  
  0

p

 

Gauss-newton Optimization


W
arg min   I ( w( x; p))  I
p  H ( x)
p
p
x 

2
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero
T
 W  

W

I
I
(
w
(
x
;
p
))


I

p

H
(
x
)
 0 Rearrange
x  p  

p

 

 W 
x I p 


T
T
 W 
 W 

I

p

x I p  ( H ( x)  I (w( x; p))



p




Gauss-newton Optimization


W
arg min   I ( w( x; p))  I
p  H ( x)
p
p
x 

2
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero
T
 W  

W

I
I
(
w
(
x
;
p
))


I

p

H
(
x
)
 0 Rearrange
x  p  

p

 

 W 
x I p 


T
A Δp
T
 W 
 W 

I

p

x I p  ( H ( x)  I (w( x; p))



p




=
b
Linear equation
Gauss-newton Optimization


W
arg min   I ( w( x; p))  I
p  H ( x)
p
p
x 

2
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero
T
 W  

W

I
I
(
w
(
x
;
p
))


I

p

H
(
x
)
 0 Rearrange
x  p  

p

 

 W 
x I p 


T
T
 W 
 W 

I

p

x I p  ( H ( x)  I (w( x; p))



p




 W 
p  ( I
p 
x 
T
T
 W  1  W 
I p  )  I p  ( H ( x)  I ( w( x; p))

 x 

A-1
b
Lucas-Kanade Registration
 W 
p  ( I
p 
x 
T
T
 W  1  W 
I p  )  I p  ( H ( x)  I ( w( x; p))

 x 

Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
Lucas-Kanade Registration
 W 
p  ( I
p 
x 
T
T
 W  1  W 
I p  )  I p  ( H ( x)  I ( w( x; p))

 x 

Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
Lucas-Kanade Registration
 W 
p  ( I
p 
x 
T
T
 W  1  W 
I p  )  I p  ( H ( x)  I ( w( x; p))

 x 

Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient I with w(x;p)
Lucas-Kanade Registration
 W 
p  ( I
p 
x 
T
T
 W  1  W 
I p  )  I p  ( H ( x)  I ( w( x; p))

 x 

Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient I with w(x;p)
w
4. Evaluate the Jacobian p at (x;p)
Lucas-Kanade Registration
 W 
p  ( I
p 
x 
T
T
 W  1  W 
I p  )  I p  ( H ( x)  I ( w( x; p))

 x 

Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient I with w(x;p)
w
4. Evaluate the Jacobian p at (x;p)
5. Compute the p
Lucas-Kanade Registration
 W 
p  ( I
p 
x 
T
T
 W  1  W 
I p  )  I p  ( H ( x)  I ( w( x; p))

 x 

Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient I with w(x;p)
w
4. Evaluate the Jacobian p at (x;p)
5. Compute the p
6. Update the parameters p  p  p
Lucas-Kanade Algorithm
Iteration 1:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Iteration 2:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Iteration 3:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Iteration 4:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Iteration 5:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Iteration 6:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Iteration 7:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Iteration 8:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Iteration 9:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Lucas-Kanade Algorithm
Final result:
H(x)
I(w(x;p))
H(x)-I(w(x;p))
Image Alignment
Lucas-Kanade for Image
Alignment
Pros:
– All pixels get used in matching
– Can get sub-pixel accuracy (important for good
mosaicing!)
– Relatively fast and simple
Cons:
– Prone to local minima
– Relative small movement
Beyond 2D Registration
So far, we focus on registration between 2D
images
The same idea can be used in registration
between 3D and 2D
3D-to-2D Registration
The transformation between 3D object and
2D images
3D-to-2D Registration
From world coordinate to image coordinate
Perspective
projection
Viewport
projection
u
sx
0
u0
v
1
0
0
-sy
0
v0
1
View
transformation
3D-to-2D Registration
Similarly, we can formulate this as an optimization
problem: