LUCAS KANADE FEATURE
TRACKER
a pyramidal implementation
di Stefano Pacifico – A.A. 2005/2006
Introduction



Feature tracking goal: Given I,J
grayscaled 2D images, and a
point u on I, find v = u+d on J
such as u and v are similar.
Optical flow evaluation to
estimate motion field (Hyp:
Lambertian surfaces, infinity
ranged light sources, no
photometric distortion)
Feature based methods vs. Direct
methods
Stefano Pacifico - 2005/2006
2
A first approach…



B.K.P. Horn and B.G. Schunck.
Determining optical flow. AI Memo 572.
Massachusetts Institue of Technology,
1980.
This article is the first proposal of an
iterative method to compute optical flow.
Dense motion vector field, weak to
noise.
Stefano Pacifico - 2005/2006
3
Optical flow constraint

Assumptions:

Pixel luminosity is the same in both images for each
pair of pixels.

Motion perpendicular to local gradient direction is not
discernible.
Uniform motion for near pixlel clusters (to avoid
aperture problem).
Little displacemente of pixels between image I and J
Using a window of 5x5 size -> 25 equations per pixel;
only u and v as unknowns.



Stefano Pacifico - 2005/2006
4
Lucas Kanade

Overdetermined equation system



Equivalent to solving with least squares method

Iterate with Newton – Raphson method
Stefano Pacifico - 2005/2006

A’A must be
invertible.
L1 and L2
eigenvalues of
A’A, L1/L2
must be not
too big.
L1 and L2 not
too little.
5
Pyramidal Lucas Kanade
build pyramid representations of I and J ILm, JLm
Initialization of pyramidal guess gLm
for L = Lm to 0
uL = location of u on ILm = u/2^L
Ix = derivative of IL wrt x
Iy = derivative of IL wrt y
G = spatial gradient matrix
init iterative L-K (guess V0)
for k = 1 to K or until optical flow < treshold
Vk = iterative L-K loop estimation
end for
final optical flow at level L dL = Vk
calculate guess for next level gLm-1
end for
final optical flow vector V = g0 + d0
return v = u +d
Stefano Pacifico - 2005/2006

Relaxes iterative L-K
assumption of little
displacement

Algorithm complexity
O(n^2*N+n^3)

n= #warp parmeters

N = integration window
size

See Baker, Matthews
papers.
6
Pros, cons and alternatives




Problem and its actual solution are well studied; KLT
widely used with almost optimal performances (see
Barron et al. paper).
Bare KLT less reliable than affine tracker, but quite faster.
Possible integration with compositional algorithm (see
Baker,Matthews papers)
Feature trackers alone don’t satisfy the needs of today AI
applications. KLT is integrated into more complex
architectures (e.g. RANSAC, ML)
Alternatives: Mean shift tracking and Histogram tracking
(see Bradski paper), Kalman Filter, Block Matching,
Warping Theory (see Brax paper).
Stefano Pacifico - 2005/2006
7
Software and demo

Stanford KLT implementation





OpenCV implementation





Pyramidal implementation
Affine model feature checking
Can track features in non-consecutive frames
http://www.ces.clemson.edu/~stb/klt/
Iterative implementation
Pyramidal implementation
Affine transformation estimation
http://Opencv.sourceforge.net
Both have means to detect features, and to
declare them lost (see also Shi, Tomasi paper).
Stefano Pacifico - 2005/2006
8
Bibliography

B.K.P. Horn and B.G.: Schunck. Determining optical flow.
AI Memo 572. Massachusetts Institue of Technology, 1980.


Lucas, B. D.; Kanade, T.: An Iterative Image
Registration Technique with an Application to Stereo
Vision : Proc. of 7th International Joint Conference on Artificial Intelligence, (1981), pp. 674679.
Shi, J.; Tomasi, C.: Good Features to Track. In: Proc. IEEE
Conference on Computer Vision and Pattern Recognition, (1994), pp. 593-600.

Barron, J. L.; Fleet, D. J.: Beauchemin, S. S.; Burkitt,
T. A.: Performance of optical flow techniques. In: International
Journal of Computer Vision 12, Vol. 1 (1992-02), pp. 43-77.

Bouguet, J.-Y.: Pyramidal Implementation of the Lucas
Kanade Feature Tracker. Part of OpenCV Documentation.
Stefano Pacifico - 2005/2006
9
Bibliography(2)


Bradski , G. R.: Computer vision face tracking for use
in a perceptual user interface. Intel Technology Journal, 2nd Quarter, 1998.
Baker, S. ; Matthews, I.: Lucas-Kanade 20 years on: A
unifying framework: Part 1. Technical Report CMU-RI-TR-02-16, CMU Robotics
Institute, (2002).

U. Knauer, T. Dammeier, and B. Meffert.:The Structure Of
Road Traffic Scenes As Revealed By Unsupervised Analysis
Of The Time Averaged Optical Flow. 17th International Conference on the
Applications of Computer Science and Mathematics in Architecture and Civil Engineering


Brox T., Bruhn, A. Papenberg, N. Weickert J.: High
Accuracy Optical Flow Estimation Based on a Theory
for Warping In Proc. 8th European Conference on Computer Vision, 2004
F. Pirri – Lecture notes A.A. 2005-2006
Stefano Pacifico - 2005/2006
10