DSP Based Motion Tracking System
Dani Cherkassky
Ronen Globinski
Advisor: Mr Slapak Alon
Presentation contents
Introduction
•Objective
•The tracking task
•Utilization potential
•Design stages
Implementation
•System diagram
•Data flow diagram
•Tracking flow chart
•Motion detection algorithm
•Tracking algorithm
Summery
•Conclusions
•Future work
Results
Introduction
Implementation
Results
Summery
Objective
Design and implement a DSP based vision system that
can detect and track a single, rigid, moving object
Introduction
Implementation
Results
Summery
The tracking task
• The problem:
Detecting and Tracking moving objects in a camera vision field often
requires the constant attention of human operators, thus making the
tracking task exposed to human errors and working hours problems
•The solution:
We offer a DSP based machine vision application that can perform the
motion detection and tracking tasks automatically.
Our solution is based on an image processing algorithms that can be
implemented on a digital signal processor platform and deliver a real-time
performance.
Introduction
Implementation
Results
Utilization potential
Motion Detection and tracking system
Security
Surveillance
Traffic
control
Military
systems
Summery
Introduction
Implementation
Results
Summery
Design stages
Stage A
Stage B
Possible
Implementing
Algorithms
The selected
Examination
Algorithms
(literature survey) (Matlab real time)
Stage C
Implementing
The selected
Algorithms
(DSP platform)
Stage D
Building the
Camera motor
system
Stage E
Tests and
Benchmarks
Introduction
Implementation
Results
Summery
System Diagram
Pan & tilt system
Motors
B&W CMOS
Camera
Analog
Video
Stream
B&W TV
Monitor
PWM Motor
Angle Control
Analog
Video
Stream
Digital Image
Stream
Via SPORT
Master DSP Board
Video Encoding & Image Processing
Slave DSP Board
Video Decoding
Introduction
Implementation
Results
Summery
Data Flow Diagram
Video Data Flow on the slave DSP board
Slave SDRAM
Input Image
244 x 360
Gray scale
SPORT, DMA
Core
Update NTSC frame
with new image data
Bank 1
Input
Bank 2
NTSC frame 1
Init empty NTSC
frame before
transfer begins
Bank 3
NTSC frame 2
Software access (core)
Hardware access (DMA)
TV
DMA, PPI
Slave
DSP
Introduction
Implementation
Tracking Flow Chart
Results
Digital
Video Stram
Motion tracking algorithms
running on the
master DSP board
Master
DSP
Separate Active
Video
Current State ß Detecting
Current State ß Detecting
Idle
Current State ß Tracking
Current State ß Tracking
Tracking
Current State
Detecting
∑-∆
Algorithm
EKLT Tracking
Algorithm
NO
YES
Lost
Summery
NO
YES
Motion Detected
Introduction
Implementation
Results
Summery
Motion
Detection Algorithm
The principle of the motion detection method is to build a model of the static
scene (i.e. without moving objects) called background, and then to compare every
In this section aframe
motion
detection
algorithm(∑-∆
filter) proposed
Manzanera
of the
sequence
to this background
in orderby
to A.
discriminate
theJ.regions of
C. Richefeu is introduced. unusual motion, called foreground (the moving objects).
Motion detection , how ???
Methods for motion detection :
• Background subtraction algorithms (the naive method)
•
•
The background must stay constant
Irrelevant motion is not discarded
Not compatible with noisy real world situations!
• Last K frames analysis algorithms (histogram , entropy , linear prediction)
•
Consumes memory (K have to be large)
Not suitable for DSP applications !
• Recursive methods that use fixed number of estimates (Kalman filter , ∑-∆ filter)
OK! low complexity, low memory consumption, statistical measure on
the temporal activity
Introduction
Implementation
Results
Summery
Motion Detection Algorithm
The ∑-∆ algorithm – a recursive approximation of the temporal statistics,
allowing a simple and efficient pixel-level change detection
Initializ ation
for each pixel x :
1
M 0 ( x)  I0 ( x)
for each pixel x :
if M t-1 ( x )  I t ( x ) then M t ( x )  M t-1 ( x )  X
if M t-1 ( x )  I t ( x ) then M t ( x )  M t-1 ( x )  X
for each pixel x :
2
for each pixel x :
 t ( x)  M t ( x)  It ( x)
for each fram e t
Initialization
for each fr am e t
3
V0 ( x )   0 ( x )
Background estimation:
Computation
of the ∑-∆
mean (an
Motion likelihood
measure:
approximation
Computationofofthe
themedian)
difference between
the image and the median
fo r each fram e t
4
fo r each p ix el x :
if  ( x )  V ( x )
t
t
Computation
of the
variance:
for each fram e t
D t ( x )  0;
Defined
the ∑-∆ mean of N times
Motion as
computation:
for each pixel x such that  t ( x )  0 :
else
the
non-zero differences
Comparison
between the difference and
if V t-1 ( x )  N   t ( x ) then V t ( x )  V t-1 ( x )  Y
D t ( x )  1;
the variance
if V t-1 ( x )  N   t ( x ) then V t ( x )  V t-1 ( x )  Y
It
- Input Fram e
X, Y, N 
M t - E stim ated B ackground value (approxim ation of the m edian of I t )
param eters used for
t
- First observation field / differe nce field (the m ot ion likelihood m easure )
sensitivity adjustm ents
Vt
- S econd observation fi eld (the variance)
D t - M otion field
Implementation
Introduction
Results
Summery
Motion Detection Algorithm
Example: two frames from a sequence acquired by a stationary camera
Still area
Algorithm results for 3
particular pixels from 3
different areas
Moving object
(foreground)
Motion area
Clutter area
The algorithm can discard
irrelevant (clutter) motion - tree
moving with the wind, and also
has low sensitivity to noise
The motion field corresponds to the Boolean indicator of the condition: “the green line is over
the black line” – every time the difference is greater than the variance the pixel is considered
to be moving.
Introduction
Implementation
Results
Summery
Motion Detection Algorithm
Example - cont’d
The motion field of the moving car
The effect of the parameters X, Y, N on the motion field:
Salt noise
X=1, Y=1, N=4
The sensitivity in this case
is too high, thus the
motion field is very noisy
and unusable
X=1, Y=3, N=4
Changing the parameter Y
affects the variance
calculation and in this case
lowering the sensitivity
which results in a cleaner
motion field
X=1, Y=3, N=4 Filtered
The motion field can be
further cleaned by applying
a 3x3 median filter to
remove the “salt” noise.
Introduction
Implementation
Results
Summery
Motion Tracking Algorithm
In this section a method proposed by J. Shi and C. Tomasi is introduced.
It is an extension of a method proposed by B. D. Lucas and T. Kanade and
will therefore, from now on, be called Extended Lucas-Kanade Tracking (ELKT).
Lets look at two sequential frames and
express theModel
transformation
of the
movingmodel)
Image Motion
(affine
motion
object in mathematical terms
The ELKT is based on the assumption that most changes between two frames are caused
by image motion and affine transformations.
Affine
Model
Moving template in
Frame 1, I m ( x , t )
Moving template in
Frame 2, I m ( x , t   )
I m  im age sequence
x  im age coordinates w ith respect to center of tem plate
  transform ation function
D  deform ation m atrix
d  translation (displacem ent) of the center of the tem plate
  the six param eters of the A ffine m otion m odel
I m ( x , t   )  I m ( ξ ( x , t ,  ), t )
ξ = Ax + d
A = I+D
 d xx
D 
 d yx
d xy 
d x 
 ,d   
d yy 
d y 
μ   d xx d xy d yx d yy d x d y 
T
Introduction
Implementation
Results
Motion Tracking Algorithm
Affine transform
We cannot assume
between J(x) and
J  I m ( x , t perfect
), K  I mcorrespondence
( x, t0   )
K ( A xK(Dx
+ d)  +
J (d).
x ) This due
0
to image noise and imperfections in the image motion model. There for, some
dissimilarity measure should be minimized. In ELKT the sum of squared
two differences
samples of the
image is
sequence
(SSD),
chosen as the dissimilarity measure.
Image Motion Estimation
T he dissim ilarity:
 
  J ( x )  K ( A x + d ) 
2
dx
W
W  T he area of integration (the object tem p lat e spatial coordinates )
This equation would however be non-linear, and
Motion estimation = minimizing
thedifficult
dissimilarity
. a good solution for
therefore
to solve. Not
The SSD method ensures that theour
dissimilarity
function
has only one minimum.
DSP Real-time
system.
Methods for minimizing the dissimilarity:
• The natural approach to minimize the dissimilarity would be
to differentiate the dissimilarity function and set the results to zero.
• The better solution is to linearize the function with a truncated Taylor expansion with
respect to some good estimate and solve the new system by iterating the procedure.
Summery
Implementation
Introduction
Results
Motion Tracking Algorithm
The tracking algorithm
Tracking = determining the six parameters motion vector µ.
μ   arg μ m in  ( μ )
The final equation for estimating the six parameters of the Affine motion vector:
μ'
( n  1)
 μ'
(n)
T
  


1


(n)
 φ  φ
T

1
T
 φ   E 
(n)
E ( x )  J ( x )  K ( ξ 0 ( x ))
For each frameφ (n:
x)  X ( x)   x J (x)
1.Calculate µ’(n+1) by iterating this
equation.  1
T
D 

0
0
0the motion vector µ.
2.Update


1
3.Use
the
 andof X
   µ to
0 update
0
(x) 
 D 0  the position


1
target object.

0
0
D  

0

x

0

y

0
1

0
0

x

0

y
0

1
The tracking is performed by updating µ using the iterative calculation of µ’ for every input
frame. The number of iterations can be determined in advance according to the tracking
performance.
Summery
Introduction
Implementation
Results
Summery
Motion Tracking Algorithm
The EKLT algorithm is recursive, we need to determine the
number of iterations
( n  1)
• It was suggested in [4] to continue the iterations till :
E
T
We have found that method problematic for our system because :
The Advantages of our method are :
E
(1)
•T
be larger
then some
picture
level
must
be measured).
1.have
Noto
need
to measure
picture
noisenoise
level level
because
iscalculated
an
The (noise
average
error
using Alfa filter
approximation
that
level.
• It may
take too longoffor
the
algorithm to converge (not capable
with RT) systems).
( n )  (1  the
E Tracking ( n  1) for n  0
  E Tracking
2. Tracking quality is estimated for eachEframe,
it
is
easy
to
change
( n  1)  
Tracking
for n  0
template when that estimate goes low.

 E Tracking ( n  1)
• We have discovered that if the algorithm doesn't converge
after 4-5 iterations there will be
practically no convergence at all. Instead of making the exhaustive iterative process we
offer a different approach:
Tracking
• Run the EKLT algorithm for a fix number of iterations.
• Calculate the tracking error using the next equation :
• Check the tracking quality:
E T racking 
1
N
N

Ei
i 1
E T racking ( n  1)    E T racking (n)
• If the previous inequality is satisfied continue tracking, else, go back to motion
detection.
Introduction
Implementation
Results
• Algorithm Complexity:
a.
Motion detection: O{mxn} , mxn = Frame Size
b.
Feature extraction: O {N} , N = Template Size
c.
Tracking initialization – compute  φ  φ T 
d.
Solving EKLT equation: O {N}
1
φ
•CPU Utilization (for 10 x 20 Template size ) :
a.
Motion detection: 82,070,156 (Cycles)
b.
Feature extraction: 20,857 (Cycles)
c.
Tracking initialization: 6,550,668 (Cycles)
d.
Solving EKLT equation: 5,205,380 (Cycles)
: O {N}
Results
Summery
Introduction
• System limitation :
Implementation
Results
Results - cont’d
The system’s tracking speed is limited
• Tracking results :
Calculation of the angular velocity of the target object:
x
Camera position
y
Object plain
α
Summery
Introduction
Implementation
Results
Summery
Conclusions
• The motion detection algorithm:
The ∑-∆ filter, combined with 3x3 median filter, proved to be a robust and accurate method of detection of
moving objects for a small cost in memory consumption and computational complexity.
• The tracking algorithm:
The EKLT tracking algorithm performed well on our system. The algorithm is efficient – the on-line
calculations are relatively small and the convergence time is fairly good. The performance can be greatly
improved by better selection of the initial features for the tracking process.
The good performance of the algorithms proved our
concept that a real-time tracking system can be
implemented successfully on DSP platforms
Introduction
Implementation
Results
Summery
Future work
Consider using a DSP with a dual PPI core to improve the performance
The DSP implementation must be optimized
Improve the pan and tilt mechanism
Improve the system’s transition from motion detection to tracking
Introduction
Implementation
Results
Summery
References
[1] A.Manzanera, J.C.Richefeu "A robust and computationally efficient motion
detection algorithm based on Background estimation"
[2] G.Jing, C.Eng and D.Rajan. "Foreground motion detection by difference-based
spatial temporal entropy"
[3] J.Shi and C.Tomashi, "Good Features to Track" Proc. IEEE Conf. Computer
Vision and Pattern Recognition, pp.593-600 IEEE CS Press 1994.
[4] D.Hager and N.Belhumeur, "Efficient Region Tracking Whit Parametric Models of
Geometry and Illumination" IEEE Transaction on Pattern Analysis and Machine
Intelligence, Vol. 20, No.10, October 1198.