Where have all the flowers gone?
(motion and tracking)
Computer Vision
COS429 Fall 2014
10/02
Guest lecture by
Andras Ferencz
Many slides adapted from James Hays, Derek Hoeim, Lana Lazebnik, Silvio Saverse, who in turn
adapted slides from Steve Seitz, Rick Szeliski, Martial Hebert, Mark Pollefeys, and others
Motivation: Mobileye
Camera-based Driver Assistance System
Safety Application based on single
forward looking camera:
• Lane Detection
- Lane Departure Warning (LDW)
- Lane Keeping and Support
• Vehicle Detection
- Forward Collision Warning (FCW)
- Headway Monitoring and Warning
- Adaptive Cruise Control (ACC)
- Traffic Jam Assistant
- Emergency Braking (AEB)
• Pedestrian Detection
- Pedestrian Collision Warning (PCW)
- Pedestrian Emergency Braking
Under the Hood: what we detect
For Videos, visit
www.mobileye.com
Detect... Detect … Detect...
Or Track?
Once target has been located, and we “learn” what it looks like, should be easier to
find in later frames... this is object tracking.
Future Image Frame
Template
Approaches to Object Tracking





Motion model (translation, translation+scale, affine, non-rigid, …)
Image representation (gray/color pixel, edge image, histogram, HOG,
wavelet...)
Distance metric (L1, L2, normalized correlation, Chi-Squared, …)
Method of optimization (gradient descent, naive search, combinatoric
search...)
What is tracked: whole object or selected features
Template
Distance Metric
• Goal: find
in image,
assume translation only:
no scale change or
rotation,
using search (scanning the
image)
• What is a good similarity
or distance measure
between two patches?
Matching with filters
Goal: find
in image
• Method 0: filter the image with eye patch
h[m, n]   g[k , l ] f [m  k , n  l ]
k ,l
f = image
g = filter
What went wrong?
response is stronger
for higher intensity
Input
Filtered Image
0-mean filter
• Goal: find
in image
• Method 1: filter the image with zero-mean eye
h[m, n]  ( f [k , l ]  f ) ( g[m  k , n  l ] )
k ,l
mean of f
True detections
False
detections
Input
Filtered Image (scaled)
Thresholded Image
L2
• Goal: find in image
• Method 2: SSD
h[m, n]  ( g[k , l ]  f [m  k , n  l ] )2
k ,l
True detections
Input
1- sqrt(SSD)
Thresholded Image
L2
• Goal: find
in image
• Method 2: SSD
h[m, n]  ( g[k , l ]  f [m  k , n  l ] )2
k ,l
One potential downside of
SSD:
Brightness Constancy
Assumption
Input
1- sqrt(SSD)
Normalized Cross-Correlation
• Goal: find
in image
• Method 3: Normalized cross-correlation
(= angle between zero-mean vectors)
mean template
h[ m, n] 
mean image patch
 ( g[k , l ]  g )( f [m  k , n  l ]  f
m ,n
)
k ,l

2
2
  ( g[ k , l ]  g )  ( f [ m  k , n  l ]  f m,n ) 
k ,l
 k ,l

Matlab: normxcorr2(template, im)
0.5
Normalized Cross-Correlation
• Goal: find
in image
• Method 3: Normalized cross-correlation
True detections
Input
Normalized X-Correlation
Thresholded Image
Normalized Cross-Correlation
• Goal: find
in image
• Method 3: Normalized cross-correlation
True detections
Input
Normalized X-Correlation
Thresholded Image
Search vs. Gradient Descent
• Search:
– Pros: Free choice of representation, distance metric; no
need for good initial guess
– Cons: expensive when searching over complex motion
models (scale, rotation, affine)
• If we have a good guess, can we do something
cheaper?
– Gradient Descent
Lucas-Kanade Object Tracker
• Key assumptions:
• Brightness constancy: projection of the same point looks the
same in every frame (uses SSD as metric)
• Small motion: points do not move very far (from guessed
location)
• Spatial coherence: points move in some coherent way
(according to some parametric motion model)
• For this example, assume whole object just translates in (u,v)
Template
The brightness constancy constraint
I(x,y,t)
I(x,y,t+1)
• Brightness Constancy Equation:
I ( x , y , t )= I ( x+ u , y + v ,t+ 1 )
Take Taylor expansion of I(x+u, y+v, t+1) at (x,y,t) to linearize the right side:
Image derivative along x
Difference over frames
I ( x+ u , y + v ,t + 1)≈ I ( x , y , t )+ I x⋅ u+ I y⋅ v + I t
I ( x+ u , y + v ,t + 1)− I ( x , y ,t )=+ I x⋅ u+ I y⋅ v+ I t
Hence,
I x⋅ u+ I y⋅v+ I t ≈ 0
T
→∇ I⋅ [u v ] + I t = 0
How does this make sense?
T
∇ I⋅ [u v ] + I t = 0
• What do the static image gradients have to do
with motion estimation?
Intuition in 1-D
Frame t+1
Intensity
Error: It
Frame t
X position
Ix
Solve for u in:
I x⋅ u+ I t ≈ 0
The brightness constancy constraint
Can we use this equation to recover image motion (u,v) at
each pixel?
T
∇ I⋅ [u v ] + I t = 0
• How many equations and unknowns per pixel?
•One equation (this is a scalar equation!), two unknowns (u,v)
The component of the motion perpendicular to the
gradient (i.e., parallel to the edge) cannot be measured
If (u, v) satisfies the equation,
so does (u+u’, v+v’ ) if
gradient
(u,v)
∇ I⋅ [u' v ' ]T = 0
(u’,v’)
(u+u’,v+v’)
edge
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusion
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusion
The aperture problem
Perceived motion
The aperture problem
Actual motion
Solving the ambiguity…
B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In
Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674–679, 1981.
• Spatial coherence constraint: solve for many pixels and
assume they all have the same motion
•
In our case, if the object fits in a 5x5 pixel patch, this gives us 25 equations:
Solving the ambiguity…
• Least squares problem:
Matching patches across images
• Over-constrained linear system
Least squares solution for d given by
The summations are over all pixels in the K x K window
Dealing with larger movements: Iterative
refinement
Original (x,y) position
1. Initialize (x’,y’) = (x,y)
2. Compute (u,v) by
2nd moment matrix for feature
patch in first image
It = I(x’, y’, t+1) - I(x, y, t)
displacement
1. Shift window by (u, v): x’=x’+u; y’=y’+v;
2. Recalculate It
3. Repeat steps 2-4 until small change
•
Use interpolation to warp by subpixel values
Schematic of Lucas-Kanade
[Baker & Matthews, 2003]
Dealing with larger movements
• How to deal with cases where the initial guess is not
within a few pixels of the solution?
Dealing with larger movements: coarse-tofine registration
run iterative L-K
upsample
run iterative L-K
.
.
.
image J1
Gaussian pyramid of image 1 (t)
image I2
image
Gaussian pyramid of image 2 (t+1)
Coarse-to-fine optical flow estimation
u=1.25 pixels
u=2.5 pixels
u=5 pixels
image Himage 1
Gaussian pyramid of image 1
u=10 pixels
image I2
image
Gaussian pyramid of image 2
Summary
• L-K works well when:
– Have a good initial guess
– L2 (SSD) is a good metric
– Can handle more degrees of freedom in motion model
(scale, rotation, affine, etc.), which are too expensive for
search
Two Problems
• Outliers: bright strong features that are wrong
• Complex, high dimensional, or non-rigid motion
One Solution: feature tracking
• Idea: track small, good features using translation only
(u,v) instead of whole object
– use outlier rejection to get consensus of only the points
that agree to common solution
Conditions for solvability
Optimal (u, v) satisfies Lucas-Kanade equation
When is this solvable? I.e., what are good points to
track?
• ATA should be invertible
• ATA should not be too small due to noise
– eigenvalues 1 and  2 of ATA should not be too small
• ATA should be well-conditioned
–  1/  2 should not be too large ( 1 = larger eigenvalue)
M = ATA is the second moment matrix !
(Harris corner detector…)
• Eigenvectors and eigenvalues of ATA relate to
edge direction and magnitude
• The eigenvector associated with the larger eigenvalue points
in the direction of fastest intensity change
• The other eigenvector is orthogonal to it
Low-texture region
– gradients have small magnitude
– small1, small 2
Edge
– gradients very large or very small
– large1, small 2
High-texture region
– gradients are different, large magnitudes
– large1, large 2
Feature Point tracking
• Find a good point to track (harris corner)
• Track small patches (5x5 to 31x31) (e.g. using
Lucas-Kanade)
• For rigid objects with affine motion: solve motion
model parameters by robust estimation (RANSAC
– to be covered later)
• For motion segmentation, apply Markoff Random
Field (MRF) algorithms (later?)
[Kanade, Lucas,Tamasi]
Implementation issues
• Window size
– Small window more sensitive to noise and may miss larger
motions (without pyramid)
– Large window more likely to cross an occlusion boundary
(and it’s slower)
– 15x15 to 31x31 seems typical
• Weighting the window
– Common to apply weights so that center matters more
(e.g., with Gaussian)
Dense Motion field
• The motion field is the projection of the 3D
scene motion into the image
What would the motion field of a non-rotating ball moving towards the camera look like?
Optical flow
• Definition: optical flow is the apparent motion
of brightness patterns in the image
• Ideally, optical flow would be the same as the
motion field
• Have to be careful: apparent motion can be
caused by lighting changes without any actual
motion
– Think of a uniform rotating sphere under fixed
lighting vs. a stationary sphere under moving
illumination
Lucas-Kanade Optical Flow
• Same as Lucas-Kanade feature tracking, but
densely for each pixel
– As we saw, works better for textured pixels
• Operations can be done one frame at a time,
rather than pixel by pixel
– Efficient
Example
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Multi-resolution registration
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Errors in Lucas-Kanade
• The motion is large
– Possible Fix: Keypoint matching
• A point does not move like its neighbors
– Possible Fix: Region-based matching
• Brightness constancy does not hold
– Possible Fix: Gradient constancy