Computer Vision
Optical Flow
Some slides from K. H. Shafique [http://www.cs.ucf.edu/courses/cap6411/cap5415/] and T. Darrell
Correspondence

Which pixel went where?
Time: t
Bahadir K. Gunturk
Time: t + dt
EE 7730 - Image Analysis II
2
Motion Field vs. Optical Flow



Scene flow: 3D velocities of scene points.
Motion field: 2D projection of scene flow.
Optical flow: Approximation of motion field derived
from apparent motion of brightness patterns in image.
Bahadir K. Gunturk
EE 7730 - Image Analysis II
3
Motion Field vs. Optical Flow

Consider perfectly uniform sphere rotating in front of
camera.



Motion field follows surface points.
Optical flow is zero since motion is not visible.
Now consider stationary sphere with moving light
source.


Motion field is zero.
But optical flow results from changing shading.
But, in general, optical flow
is a reliable indicator of
motion field.
Bahadir K. Gunturk
EE 7730 - Image Analysis II
4
Applications







Object tracking
Video compression
Structure from motion
Segmentation
Correct for camera jitter (stabilization)
Combining overlapping images (panoramic image
construction)
…
Bahadir K. Gunturk
EE 7730 - Image Analysis II
5
Optical Flow Problem

How to estimate pixel motion from one image to another?
H
Bahadir K. Gunturk
I
EE 7730 - Image Analysis II
6
Computing Optical Flow

Assumption 1: Brightness is constant.
H ( x, y )  I ( x  u , y  v)

Assumption 2: Motion is small.
(from Taylor series expansion)
Bahadir K. Gunturk
EE 7730 - Image Analysis II
7
Computing Optical Flow

Combine
It
In the limit as u and v goes to zero, the equation becomes exact
0  I t  I xu  I y v
Bahadir K. Gunturk
EE 7730 - Image Analysis II
(optical flow equation)
8
Computing Optical Flow

At each pixel, we have one equation, two unknowns.
0  I t  I xu  I y v

(optical flow equation)
This means that only the flow component in the
gradient direction can be determined.
The motion is parallel to
the edge, and it cannot be
determined.
This is called the aperture problem.
Bahadir K. Gunturk
EE 7730 - Image Analysis II
9
Computing Optical Flow


We need more constraints.
The most commonly used assumption is that optical
flow changes smoothly locally.
Bahadir K. Gunturk
EE 7730 - Image Analysis II
10
Computing Optical Flow

One method: The (u,v) is constant within a small
neighborhood of a pixel.
Optical flow equation:
0  I t  I xu  I y v
Use a 5x5 window:
Two unknowns, 25 equation !
Bahadir K. Gunturk
EE 7730 - Image Analysis II
11
Computing Optical Flow

Find minimum least squares solution
Lucas – Kanade method
Bahadir K. Gunturk
EE 7730 - Image Analysis II
12
Computing Optical Flow

Lucas – Kanade
When is This Solvable?
• ATA should be invertible
• ATA should not be too small. Other wise noise will be
amplified when inverted.)
Bahadir K. Gunturk
EE 7730 - Image Analysis II
13
Computing Optical Flow

What are the potential causes of errors in this
procedure?



Brightness constancy is not satisfied
The motion is not small
A point does not move like its neighbors

Bahadir K. Gunturk
window size is too large
EE 7730 - Image Analysis II
14
Improving accuracy


Recall our small motion assumption
This is not exact


To do better, we need to add higher order terms back in:
This is a polynomial root finding problem

Can solve using Newton’s method


Also known as Newton-Raphson method
Lucas-Kanade method does one iteration of Newton’s method

Bahadir K. Gunturk
Better results are obtained via more iterations
EE 7730 - Image Analysis II
15
Iterative Refinement

Iterative Lucas-Kanade Algorithm
1.
2.
Estimate velocity at each pixel by solving Lucas-Kanade
equations
Warp H towards I using the estimated flow field
- use image warping techniques
3.
Repeat until convergence
Bahadir K. Gunturk
EE 7730 - Image Analysis II
16
Revisiting the small motion
assumption

What can we do when the motion is not small?
Bahadir K. Gunturk
EE 7730 - Image Analysis II
17
Reduce the resolution!
Bahadir K. Gunturk
EE 7730 - Image Analysis II
18
Coarse-to-fine optical flow estimation
u=1.25 pixels
u=2.5 pixels
u=5 pixels
u=10 pixels
image H
Gaussian pyramid of image H
Bahadir K. Gunturk
EE 7730 - Image Analysis II
image I
Gaussian pyramid of image I
19
Coarse-to-fine optical flow estimation
run iterative L-K
warp & upsample
run iterative L-K
.
.
.
image H
image I
Gaussian pyramid of image H
Bahadir K. Gunturk
EE 7730 - Image Analysis II
Gaussian pyramid of image I
20
Multi-resolution Lucas-Kanade
Algorithm
Bahadir K. Gunturk
EE 7730 - Image Analysis II
21
Block-Based Motion Estimation
Bahadir K. Gunturk
EE 7730 - Image Analysis II
22
Block-Based Motion Estimation
Bahadir K. Gunturk
EE 7730 - Image Analysis II
23
Block-Based Motion Estimation
Hierarchical search
image H
image I
Gaussian pyramid of image H
Bahadir K. Gunturk
EE 7730 - Image Analysis II
Gaussian pyramid of image I
24
Parametric (Global) Motion

Sometimes few parameters are enough to represent the
motion of whole image
translation
rotation
scale
Bahadir K. Gunturk
EE 7730 - Image Analysis II
25
Parametric (Global) Motion

Affine Flow
u ( x , y )  a1 x  a 2 y  b1
v ( x , y )  a 3 x  a 4 y  b2
u 
x
   A  B
v 
 y
Bahadir K. Gunturk
 u   x   x   a1
   
 
v   y  y  a3
EE 7730 - Image Analysis II
a 2   x   b1 
    
a 4   y  b2 
26
Parametric (Global) Motion

Affine Flow
u ( x , y )  a1 x  a 2 y  b1
v ( x , y )  a 3 x  a 4 y  b2
 u   x   x   a1
   
 
v   y  y  a3
u   x
  
v  0
Bahadir K. Gunturk
EE 7730 - Image Analysis II
a 2   x   b1 
    
a 4   y  b2 
y
1
0
0
0
0
x
y
 a1 
 
a
 2
0   b1 
 
1   a3 
a 
4
 
 b 2 
27
Parametric (Global) Motion

Affine Flow – Approach
 u 1   x1
  
v
0
 1 
u 2   x2
 
 v2   0
u   x
3
3
  
 v 3   0
Bahadir K. Gunturk
y1
1
0
0
0
0
x1
y1
y2
1
0
0
0
0
x2
y2
y3
1
0
0
0
0
x3
y3
EE 7730 - Image Analysis II
0   a1 
 
1
a
 2
0   b1 
 
1   a3 
0   a4 
 
1   b 2 
28
Iterative Refinement

Iterative Estimation
1.
2.
Estimate parameters by solving the linear system.
Warp H towards I using the estimated flow field
- use image warping techniques
3.
Repeat until convergence or a fixed number of iterations
Bahadir K. Gunturk
EE 7730 - Image Analysis II
29
Coarse-to-fine global flow estimation
Compute Flow Iteratively
warp & upsample
Compute Flow Iteratively
.
.
.
image H
J
image I
Bahadir K. Gunturk
30
EE 7730 - Image Analysis II
Gaussian
pyramid of image H
Gaussian pyramid of image
I
Global Flow
Find features
Match features
Fit parametric model
Application: Mosaic construction
Bahadir K. Gunturk
EE 7730 - Image Analysis II
31
Image Warping
x, y
x, y 
 u   x   x   a1
   
 
v   y  y  a3
 x   1  a 1
   
 y   a 3
 x  1  a 1
   
 y   a3
Bahadir K. Gunturk
a 2   x   b1 
    
a 4   y  b2 
a 2   x   b1 
    
1  a 4   y  b2 
a2 

1  a4 
1
  x    b1  
    
 y

   b2  
EE 7730 - Image Analysis II
32
Image Warping
Interpolate to find the intensity at (x’,y’)
x, y
x, y 
Pixel values are known in
this image
Bahadir K. Gunturk
Pixel values are to found in
this image
EE 7730 - Image Analysis II
33
Other Parametric Motion Models
x'
Perspective
y'
a1 x  a 2 y  a 3
a 7 x  a8 y  1
a 4 x  a5 y  a6
a 7 x  a8 y  1
and
u  x ' x
v  y ' y
Pseudo-Perspective:
Approximation to perspective.
(Planar scene + Perspective
projection)
Bahadir K. Gunturk
EE 7730 - Image Analysis II
34