Introduction To
Tracking
Mario Haddad
What is Tracking?
• Estimating pose (state)
• Possible from a variety of measured
sensors
–
–
–
–
–
–
Electrical
Mechanical
Inertial
Optical
Acoustic
Magnetic
2
DYNAMIC SCENE ANALYSIS

The input to the dynamic scene analysis is a sequence
of image frames 𝐹 𝑥, 𝑦, 𝑡 taken from the changing
world.

x, y are spatial coordinates.

Frames are usually captured at fixed time intervals.

𝑡 represents 𝑡 𝑡ℎ frame in the sequence.
Typical Applications

Motion detection. Often from a static camera.

Object localization.

Three-dimensional shape from motion.

Object tracking.
Example Application
Object Tracking Definition

Object tracking is the problem of determining
(estimating) the positions and other relevant
information of moving objects in image sequences.
Difficulties In Reliable Object
Tracking


Rapid appearance changes caused by

image noise,

illumination changes,

non-rigid motion,

...
Non-stable background

Interaction between multiple objects

...
Difficulties In Reliable Object
Tracking
Robust Density Comparison for Visual Tracking (BMVC 2009)
Difficulties In Reliable Object
Tracking
Motion Estimation
Block Matching Method

For a given region in one frame, find the corresponding
region in the next frame by finding the maximum
correlation score (or other block matching criteria) in a
search region
Block Matching Method
Block Matching Method
Optical Flow  Motion Field
(a)
(b)
Visible Motion and True
Motion

OPTIC FLOW - apparent motion of the same (similar)
intensity patterns

Generally, optical flow corresponds to the motion field,
but not always:
Local Features for Tracking

If strong derivatives are observed in two
orthogonal directions then we can hope
that this point is more likely to be unique.

Many trackable features are called
corners.

Harris Corner Detection !
Aperture Problem
The Aperture Problem

Different motions – classified as similar
source: Ran Eshel
The Aperture Problem

Similar motions – classified as different
source: Ran Eshel
Tracking Methods
Mean-Shift
The mean-shift algorithm is an efficient
approach to tracking objects whose
appearance is defined by histograms.
(not limited to only color)
Motivation

Motivation – to track non-rigid objects, (like
a walking person), it is hard to specify
an explicit 2D parametric motion model.

Appearances of non-rigid objects can
sometimes be modeled with color
distributions
Mean Shift Theory
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Intuitive Description
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Intuitive Description
Region of
interest
Center of
mass
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Mean Shift Vector

Given:
Data points and approximate location of the mean of this
data:

Task:
Estimate the exact location of the mean of the data by
determining the shift vector from the initial mean.
Mean Shift Vector
A Quick PDF Definition
A probability density function (pdf),
is a function that describes the
relative likelihood for this random
variable to take on a given value.
Mean-Shift Object Tracking
Target Representation
Choose a
reference
target model
Represent the
model by its
PDF in the
feature space
Choose a
feature space
0.35
Quantized
Color Space
Probability
0.3
0.25
0.2
0.15
0.1
0.05
0
1
2
3
.
color
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt
.
.
m
Mean-Shift Object Tracking
Target Model
Target Candidate
(centered at 0)
(centered at y)
0.35
0.3
0.3
0.25
0.25
Probability
Probability
PDF Representation
0.2
0.15
0.1
0.2
0.15
0.1
0.05
0.05
0
0
1
2
3
.
.
.
m
1
2
color
q  qu u 1..m
3
.
.
.
m
color
m
q
u 1
u
1
Similarity f  y   f  q , p  y  


Function:
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt
p  y    pu  y u 1..m
m
p
u 1
Q is the target histogram,
P is the object histogram
(depends on location y)
u
1
Mean-Shift Object Tracking
Target Localization Algorithm
Start from
the position
of the model
in the current
frame
q
Search in the
model’s
neighborhood
in next frame
p  y
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt
Find best
candidate by
maximizing a
similarity func.
f  p  y  , q 
Mean Shift

Mean-Shift in tracking task:

track the motion of a cluster of interesting
features.

1. choose the feature distribution to represent
an object (e.g., color + texture),

2. start the mean-shift window over the feature
distribution generated by the object

3. finally compute the chosen feature
distribution over the next video frame.
Mean Shift

Starting from the current window location, the
mean-shift algorithm will find the new peak or
mode of the feature distribution, which
(presumably) is centered over the object that
produced the color and texture in the first
place.

In this way, the mean-shift window tracks
the movement of the object frame by frame.
Examples
Examples
Other Mean Shift
Applications
Edge Preserving Smoothing
Segmentation
Contour Detection
Kalman Filter
Rudolf Emil Kalman
•
•
•
•
•
Born in 1930 in Hungary
BS and MS from MIT
PhD 1957 from Columbia
Filter developed in 1960-61
Now retired
Kalman Filter
• Noisy data in
hopefully less noisy data out
• The Kalman filter operates recursively on streams
of noisy input data to produce a statistically
optimal estimate of the underlying system state.
Motivation
Kalman Filter Applications

Tracking objects (e.g., missiles, faces, heads, hands)

Navigation

Many computer vision applications
– Stabilizing depth measurements
– Feature tracking
– Cluster tracking
– Fusing data from radar, laser scanner and
stereo-cameras for depth and velocity measurements
– Many more
Intuition


Robot

Odometer

GPS
Sand
Previous
state
We may encounter:

Wheel spin

GPS inaccuracy
Odometer
GPS
Kalman Filter
Not perfectly sure. Why ?
•
A𝐬𝐬𝐮𝐦𝐞 𝑲𝒌 = 𝟎. 𝟓 , what would we get?
Kalman Filter

Kalman filter finds the most optimum averaging factor for
each consequent state.

“somehow” remembers a little bit about the past
states.
Kalman Filter
State Prediction:
Measurement Prediction:
𝑥𝑘 - state prediction
𝑢𝑘 - control signal (Most of the time there is no control signal)
𝑤𝑘 - process noise
A,B,H - define the physics of interest ( acceleration, position, speed… )
𝑧𝑘 - measurement prediction
𝑣𝑘 - measurement noise
Kalman Filter
• Two groups of the equations for the Kalman filter:
o Time update equations (Prediction)
o Measurement update equations. (Correction)
• The time update equations are responsible for projecting forward (in time)
the current state and error covariance estimates to obtain the a priori
estimates for the next time step.
• The measurement update equations are responsible for the feedback—i.e. for
incorporating a new measurement into the a priori estimate to obtain an
improved a posteriori estimate.
Brace Yourselves..
Kalman Filter
Predict
1.
Predict the state ahead:
Update
1.
xˆt  xˆt  Kt zt  Hxˆt 
xˆt  Axˆt 1
2.
2.
Predict the error covariance
ahead:
t  At 1 AT  Q
Update the state estimate:
Update the error
covariance:
t  I  Kt H t
where Kalman gain Kt is:

K t   t H T H t H T  R
55

1