Object Tracking
Evangelos Loutas and Ioannis Pitas*
Dept. of Informatics
Aristotle University of Thessaloniki
Thessaloniki, GREECE
*email: pitas@zeus.csd.auth.gr
Department of Informatics
Aristotle University of Thessaloniki
Definition
Object Tracking: Trace the
progress of objects or (object
features) as they move about
in a visual scene.
Department of Informatics
Aristotle University of Thessaloniki
Basic concepts needed
•
•
•
•
•
Correlation.
Edge Information.
Color Information.
Spatio-temporal information.
Kalman Prediction.
Department of Informatics
Aristotle University of Thessaloniki
Correlation
• Given a template T in the form of a small array of image
intensities find the likely locations of that template in some
larger test image I.
This means that the mathematical correlation has to be
maximized.
Maximize :  I ( x  x ' , y  y ' )T ( x, y ) dxdy
Department of Informatics
Aristotle University of Thessaloniki
Edge Detection
Edge detection using:
• Edge templates.
• Laplacian.
• Hough transform.
Department of Informatics
Aristotle University of Thessaloniki
Extract Color Information
• Use the Hue function
• Use the Fisher Linear Discriminant function.
Fischer(I)=f.I,
I=(r,g,b)
Department of Informatics
Aristotle University of Thessaloniki
SPATIO-TEMPORAL INFORMATION
• Information about objects from single
(spatial) as well as multiple (temporal)
frames/images.
• Previous frames can be used to predict
object motion. (Motion Prediction)
Department of Informatics
Aristotle University of Thessaloniki
Kalman Filtering(I)
• A Kalman filter estimates the state of a
dynamic system recursively at each time,in
the linear minimum mean square error sense,
given a time series of vector or scalar
observations that are linearly related to these
state variables.
Department of Informatics
Aristotle University of Thessaloniki
Kalman Filtering (II)
• If the state variables and the noise are
modeled as uncorrelated, Gausian random
processes then the Kalman filter is the
minimum mean square error estimator.
Department of Informatics
Aristotle University of Thessaloniki
Kalman Filtering Basics(I)
• Linear state transition equation :
Φ(k,k-1): State transition matrix.
W(k): Zero mean white random sequence.
z(k )  (k , k  1)z(k  1)  w(k ), k  1,..., N
Department of Informatics
Aristotle University of Thessaloniki
Kalman Filtering Basics(II)
• The measurements are related to the state
variables as:
H(k) : Observation matrix.
V(k) : Zero mean white observation noise
sequence
y (k )  H(k )z (k )  v(k ), k  1,..., N
Department of Informatics
Aristotle University of Thessaloniki
Kalman Filtering Basics (III)
• The Kalman filter minimizes, at each time k,
the trace of the error covariance matrix
conditioned on all observations up to time k,
defined us:
 E{e1(k ) 2 | y (k ),  y (1)}
 E{e1(k )eM (k ) | y (k ),  y (1)}







2
 E{eM (k )e1(k ) | y (k ),  y (1)} 
E{eM (k ) | y (k ),  y (1)} 

^
ei (k )  zi (k )  zi (k ),
zi(k) is an estimate of k.
Department of Informatics
Aristotle University of Thessaloniki
Algorithm Categorization
• Knowledge-base.
• Camera motion.
• Rigid body vs. non-rigid body.
Department of Informatics
Aristotle University of Thessaloniki
Some Major Problem Areas of Object
Tracking
• Feature Selection
• Occlusion
Department of Informatics
Aristotle University of Thessaloniki
Feature Selection-Various
Approaches
• The need of feature selection.
• In general, temporal as well as spatial
variations are used to select features.
• The Kanade Lucas Tomasi Algorithm
approach.
• Active Contour approach.
Department of Informatics
Aristotle University of Thessaloniki
Background Subtraction
• Background subtraction is used for
separating moving objects from their
backgrounds. It is used as a pre-process in
advance of feature detection to suppress the
background features. The foreground areas
are those that satisfy:
I ( x, y )  IB ( x, y )  
Department of Informatics
Aristotle University of Thessaloniki
Occlusion
Definition
• Occlusion is a set of points that appear in one
image whose corresponding world points are
not visible in another image because an
opaque object is blocking the view of those
points in the other image.
Department of Informatics
Aristotle University of Thessaloniki
Examples
Department of Informatics
Aristotle University of Thessaloniki
Algorithm Examples
• Active Contours
• Object tracking using a set of point features
Department of Informatics
Aristotle University of Thessaloniki
Snakes
• Snake : Deformable curve r(s) 0s1.
• Maximize F(r(s)) over 0s1.
• The tendency to maximize F is formalized as the
“external” potential energy of the dynamical
system.
• The “external” potential energy is counterbalanced
by “internal” potential energy
Department of Informatics
Aristotle University of Thessaloniki
Deformable Templates
• A parametric shape model r(s,X) called
deformable template.
Department of Informatics
Aristotle University of Thessaloniki
Active Contours
• Newton’s law of motion for a snake with mass
driven by internal and external forces :
• w1, w2 : Elastic coefficients
• ρ : Mass density
• γ : viscous resistance from a medium surrounding the snake.
( w1r)  ( w2r)
rtt  (rt 

)  F
2
s
s
2
Department of Informatics
Aristotle University of Thessaloniki
Kalman Filtering Prediction to Active
Contours
• The dynamical model is used for prediction.
• The predicted position is refined using
measured image features.
• Form of Dynamical equation :
..
.
X  f ( X, X, w )
Department of Informatics
Aristotle University of Thessaloniki
Resistance to occlusion
• The active contours algorithm shows
resistance to partial occlusions. A partial
occlusion causes loss of measurements . The
remaining successful observations, together
with the dynamical model compensate for
lost measurements. Observations resume
after disocclusion.
Department of Informatics
Aristotle University of Thessaloniki
Limitations of the traditional model
• Geometrically and topologically simple
objects can be handled.
• The model is inadequate for objects with
deep cavities or multi-part objects.
• The topology of structure of interest must be
known in advance in order to define a
parametric model.
Department of Informatics
Aristotle University of Thessaloniki
Possible Features
• Edges
• Valleys
• Ridges
Department of Informatics
Aristotle University of Thessaloniki
Another Approach
• Select the region to be tracked.
• Define a set of N point features inside the
region.
• Track the point features using Kanade Lucas -Tomasi algorithm.
• Estimate the tracked region.
Department of Informatics
Aristotle University of Thessaloniki
Tracking of point features
Kanade - Lucas -Tomasi approach (I)
Find the displacement vector d=[dx,dY] by
minimizing over a window W the dissimilarity
between the current and the previous frame:
d
d 2
   [ J (x  )  I (x  )] w(x)dx
W
2
2
Department of Informatics
Aristotle University of Thessaloniki
Tracking of point features
Kanade - Lucas -Tomasi approach (II)
• After setting the derivative equal to zero:

d
0
it is found that in order to perform one
iteration of the minimization the equation :
Zd  e
must be solved
Department of Informatics
Aristotle University of Thessaloniki
Tracking of point features
Kanade - Lucas -Tomasi approach (III)
• Z is a 2x2 matrix depending on the image
gradient, e is a 2x1 matrix (error vector)
depending on the frame difference and the
image gradient.
• The final solution
is
achieved
by
solving
Zd  e
repeatedly
and shifting I and J
image by the computed amount.
Department of Informatics
Aristotle University of Thessaloniki
Feature Selection
• The feature selection on Kanade - Lucas Tomasi algorithm is based on theZd  e
requirement that the equation
is
well conditioned.
• A good feature is defined as one for which the
matrix Z has two large eigenvalues.
Department of Informatics
Aristotle University of Thessaloniki
Reliability of Optical Flow
• The reliability of optical flow can be defined
as the angle between two lines corresponding
Zd  e.
to equation
Department of Informatics
Aristotle University of Thessaloniki
Occlusion Handling in Kanade Lucas - Tomasi Algorithm
• Rejection of features whose cumulative
residue is above a certain threshold.
• A large residue implies that different motions
exist within the search range of the tracker.
Department of Informatics
Aristotle University of Thessaloniki
Confronting the Occlusion Problem
(I)
PARTIAL OCCLUSION
Predict the motion of occluded features using
motion of the unoccluded features.
Department of Informatics
Aristotle University of Thessaloniki
Confronting the occlusion
problem (II)
•
•
•
•
•
FULL OCCLUSION
Find the occluding region.
Predict the overall occluded region motion using
Kalman Filtering.
Determine region disocclusion.
Perform a verfication procedure.
Continue track the disoccluded region.
Department of Informatics
Aristotle University of Thessaloniki
Results
Results On Artificial Images
The Region Being Tracked is Found after total occlusion
Department of Informatics
Aristotle University of Thessaloniki
Results
“Walker Video Sequence”
Department of Informatics
Aristotle University of Thessaloniki
Results
Results On Football Image Sequence (I)
The Region Being Tracked is found after total occlusion and
disocclusion.
Department of Informatics
Aristotle University of Thessaloniki
Results
Results On Football Image Sequence (II)
The Region Being Tracked is found after total occlusion and
disocclusion.
Department of Informatics
Aristotle University of Thessaloniki
Results
Results On Football Image Sequence (III)
The Region Being Tracked is found after total occlusion and
disocclusion.
Department of Informatics
Aristotle University of Thessaloniki
Resistance to partial occlusion(I)
The tracker is resistant to partial occlusion
Department of Informatics
Aristotle University of Thessaloniki
Resistance to partial occlusion(II)
Department of Informatics
Aristotle University of Thessaloniki
Resistance to partial occlusion(III)
Department of Informatics
Aristotle University of Thessaloniki
Bibliography
• Andrew Blake and Michael Isard “Active
Contours”,Springer-Verlag 1998.
• A. Murat Tekalp “Digital Video Processing”,
Prentice Hall 1995.
• C. Tomasi and T. Kanade, “Shape and Motion
from Image Streams: a Factorization Method Part 3 Detection and Tracking of Point
Features”,
Department of Informatics
Aristotle University of Thessaloniki
Bibliography (II)
• Tech. Report CMU-CS-91-132, Computer
Science Department Carnegie Mellon
University, April 1991.
• J. Shi and C. Tomasi, “Good Features to
Track”, IEEE International Conference on
Computer Vision and Pattern Recognition
(CVPR94), Seattle, June 1994.
Department of Informatics
Aristotle University of Thessaloniki
Bibliography (III)
• Tim McInerney and Demetri Terzopoulos,
“Topologically Adaptable Snakes”, Int. Conf.
On Computer Vision (ICCV ‘95),
Cambridge, MA, USA, June 1995.
• Ryuzo Okada, Yoshiaki Shirai and Jun
Miura "Object Tracking based on Optical
Flow and Depth", Proc. of IEEE/SICE/RSJ
Int. Conf. on MFI, pp.565-571, 1996
Department of Informatics
Aristotle University of Thessaloniki