Motion Tracking
Introduction

Finding how objects have moved in an
image sequence



Movement in space
Movement in image plane
Camera options


Static camera, moving objects
Moving camera, moving objects
Image Processing and Computer Vision: 8
2
Contents

Acquiring targets



Image differencing
Moving edge detector
Following targets






Matching
Minimum path curvature
Model based methods
Kalman filtering
Condensation
Hidden Markov Model
Image Processing and Computer Vision: 8
3
Camera calibration revisited

Image to camera co-ordinate
transformation


Intrinsic parameters
Camera to world co-ordinate
transformation

Extrinsic parameters
Image Processing and Computer Vision: 8
4
Camera to Image Co-ordinates
Distortionless Camera

If



no distortions
uniform sampling
Co-ordinates linearly related

offset and scale
x  xim  o x  s x


y   y im  o y s y
Image Processing and Computer Vision: 8
5
Camera to Image Co-ordinates
Distorting Camera

Periphery is distorted


y  y d 1  k1 r 2  k 2 r 4 
x  x d 1  k1 r 2  k 2 r 4
2
2
2


y
x
r

k2 = 0 is good enough
Image Processing and Computer Vision: 8
6
Pinhole Camera
f
Z
Optical
centre
Image
Image and centre, object and
centre are similar triangles.
Image Processing and Computer Vision: 8
Object
X
x f
Z
Y
y f
Z
7
Camera and World Coordinates
translate and
rotate
World frame
Image Processing and Computer Vision: 8
8
System architecture
Start
Acquire
target
End
Follow
target
Image Processing and Computer Vision: 8
9
Target acquisition



Finding a target to follow
Differencing
Moving edge detector
Image Processing and Computer Vision: 8
10
Change and Moving Object
Detection


Simplest method of detecting change
Compute differences between


Live and background images
Adjacent images in a sequence
Image Processing and Computer Vision: 8
11
Image Differencing

Differences due to




Moving object overlying static background
Moving object overlying another moving
object
Moving object overlying same moving
object
Random fluctuation of image data
Image Processing and Computer Vision: 8
12
Image Processing and Computer Vision: 8
13
Demo


Inter frame differencing
Difference from a background

pFinder
Image Processing and Computer Vision: 8
14
Background image



Detecting true differences required an
accurate background
Lighting changes?
Camera movement?
Image Processing and Computer Vision: 8
15
Background image updates

Periodically modify whole background


Will include changes in new background
Systematically incorporate non-changed
portions of image into background
if Li ,t  backgroundthen Bi ,t  Bi ,t 1  Li ,t
else Bi ,t  Bi ,t 1
Image Processing and Computer Vision: 8
16
Critique

Can identify changes in the image data


But what do the changes mean?
Need a second layer of processing


To recognize changes
Optical flow sidesteps this problem...
Image Processing and Computer Vision: 8
17
Target following




Observing the positions of an object or
objects in a time sequence of images.
Object matching
Minimum path curvature
Model based methods
Image Processing and Computer Vision: 8
18
Matching



Locate objects in each image
Match objects between images
Use methods of previous lectures
Image Processing and Computer Vision: 8
19
Minimum path curvature

Observations of two objects in three images
Image Processing and Computer Vision: 8
20
Image Processing and Computer Vision: 8
21
Image Processing and Computer Vision: 8
22
Image Processing and Computer Vision: 8
23
Image Processing and Computer Vision: 8
24
Which is “best” solution?

One with overall straightest paths
For each solution
For each path
Compute total curvature
Sum
Image Processing and Computer Vision: 8
25
Model based tracking

Mathematical model of objects’
motions:


position, velocity (speed, direction),
acceleration
Can predict objects’ positions
Image Processing and Computer Vision: 8
26
System overview
Motion
Model
Predict
Location
Update?
Verify
Location
Image Processing and Computer Vision: 8
27
Simple Motion Model

Newton’s laws
1
st   s  ut  a t
2
2
0



s = position
u = velocity
a = acceleration


all vector quantities
measured in image co-ordinates
Image Processing and Computer Vision: 8
28
Prediction

Can predict position at time t knowing




Position
Velocity
Acceleration
At t=0
Image Processing and Computer Vision: 8
29
Uncertainty


If some error in a - Da or u - Du
Then error in predicted position - Ds
1
Dst   s0  Dut  Da t 2
2
Image Processing and Computer Vision: 8
30
Verification

Is the object at the predicted location?

Matching


How to decide if object is found
Search area

Where to look for object
Image Processing and Computer Vision: 8
31
Object Matching

Compare



A small bitmap derived from the object vs.
Small regions of the image
Matching?

Measure differences
Image Processing and Computer Vision: 8
32
Search Area:
Why? and Where?

Uncertainty in knowledge of model
parameters



Limited accuracy of measurement
Values might change between
measurements
Define an area in which object could be

Centred on predicted location, s  Ds
Image Processing and Computer Vision: 8
33
Update the Model?


Is the object at the predicted location?
Yes


No change to model
No


Model needs updating
Kalman filter is a solution
Image Processing and Computer Vision: 8
34
Kalman filter
Mathematically rigorous methods of using
uncertain measurements to update a
model
Image Processing and Computer Vision: 8
35
Kalman filter notation

Relates

Measurements y[k]


System state x[k]



Relates system state to measurements
Evolution matrix A[k]


Position, velocity of object, etc
Observation matrix H[k]


e.g. positions
Relates state of system between epochs
Measurement noise n[k]
Process noise v[k]
Image Processing and Computer Vision: 8
36
Mathematically
How do observations relate to model?
yk   Hk xk   nk 
 xk 
 xk 


x
k
1
0
0
0

 

  nk 

 yk  0 0 1 0  yk 

 



 y k 
Image Processing and Computer Vision: 8
37
Prediction of System State

Relates system states at epochs k and
k+1
xˆ k  1 | k   Ak xk | k   vk 
 xˆk  1 | k  1
 xˆk  1 | k   0


 yˆ k  1 | k  0
ˆ
 
 y k  1 | k   0
T 0 0  xk | k 
1 0 0   xk | k 
 vk 
0 1 T   yk | k 


0 0 1  y k | k 
Image Processing and Computer Vision: 8
38
Prediction of Observation

From predicted system state, estimate
what observation should occur:
yˆ k  1 | k   Hk xˆ k  1 | k   nk 
 xˆk  1 | k 


ˆ
 xˆk  1 | k  1 0 0 0  xk  1 | k 
 yˆ k  1 | k   0 0 1 0  yˆ k  1 | k   nk 

 

ˆ

 y k  1 | k 
Image Processing and Computer Vision: 8
39
Updating the Model




Predict/estimate a measurement yˆ k  1 | k 
Make a measurement
yk  1
Predict state of model
xˆ k  1 | k 
How does the new measurement
contribute to updating the
model?
Image Processing and Computer Vision: 8
40
Updating the Model
xˆ k  1 | k  1  xˆ k  1 | k   Gk  1Dyk  1 | k 
Dyk  1 | k   yk  1  yˆ k  1 | k 

G is Kalman Gain

Derived from A, H, v, n.


1
T
T
G  Ck | k H HC k | k  H  n
Ck  1 | k   Ck | k   GHCk | k 
C  system covariance
Image Processing and Computer Vision: 8
41
Example



Tracking two corners of a minimum
bounding box
Matching using colour
Image differencing to locate target
Image Processing and Computer Vision: 8
42
Condensation Tracking


So far considered single motions
What if movements change?



Bouncing ball
Human movements
Use multiple models

plus a model selection mechanism
Image Processing and Computer Vision: 8
43
Selection and Tracking


Occur simultaneously
Maintain


Predict


A distribution of likely object positions plus
weights
Select N samples, predict locations
Verify


Match predicted locations against image
Update distributions
Image Processing and Computer Vision: 8
44
Tracking Using Hidden Markov
Models

Hidden Markov model describes




States occupied by a system
Possible transitions between states
Probabilities of transitions
How to recognise a transition
Image Processing and Computer Vision: 8
45
Optic Flow



Assume each pixel moves but does not
change intensity
Pixel at location (x,y) in image 1 is pixel
at (x+Dx,y+Dy) in image 2.
Optic flow associates displacement
vector with each pixel
Image Processing and Computer Vision: 8
46
I  x, y, t   I  x  Dx, y  Dy, t  Dt 
0
0
0
 
I
I
I
2
 I  x, y, t   Dx  Dy  Dt  O Dx
x
y
t
I
I
I
 Dx  Dy  Dt
x
y
t
I Dx I Dy I



x Dt y Dt t
I
I
I
 v u
x
y
t
Image Processing and Computer Vision: 8
47
Aperture Problem
I/  x  horizontal difference
 I/  y  vertical difference
 I/  t  difference between images
One equation, two unknowns
 cannot solve equation
Could solve for movement perpendicular
to gradient
Image Processing and Computer Vision: 8
48
Solution
Impose smoothness constraint:
Minimise total of sum of squares of
velocity component gradients
   
 u 2 u 2 v 2 v
  x  y  x  y
image 
Image Processing and Computer Vision: 8
2


49
P
u  u average  E x
D
P
v  v average  E y
D
P  E x u average  E y v average  E t
D   2  E 2x  E 2y
 is a constant
Iterate over a pair of frames, or
over a sequence
Image Processing and Computer Vision: 8
50
Critique

Assumptions



Linear variation of intensities
Velocity changes smoothly
These are invalid

Especially at object boundaries
Image Processing and Computer Vision: 8
51
Area Based Methods


Match small regions in image 1 with
small regions in image 2
Assume objects move but do not
deform

Same formulation as for optical flow,
different areas involved….
I x  Dx, y  Dy, t  Dt   I x, y, t 
Image Processing and Computer Vision: 8
52
Matching
Compute (Dx, Dy) by finding (u,v) that
minimises
iw jw
2
Du, v   
I  x  u  i, y  v  j , t  Dt   I  x  i, y  j, t 

i  w j  w


Then (u,v) = (Dx, Dy)
Image Processing and Computer Vision: 8
53
Summary

Target acquisition



Image differencing
Background model
Target following




Matching
Minimum path curvature
Model based methods
Optic flow
Image Processing and Computer Vision: 8
54
This “telephone” has too many
shortcomings to be seriously considered
as a means of communication. The
device is of no value to us.
Western Union internal memo, 1876
Image Processing and Computer Vision: 8
55