Tracking Using A Highly Deformable
Object Model
Nilanjan Ray
Department of Computing Science
University of Alberta
Overview of Presentation
•
Tracking deformable objects
– Motivations: desirable properties of a deformable object model
– An example application (mouse heart tracking)
•
Some technical background
– Level set function and its application in image processing
– Non-parametric probability density function (pdf) estimation
– Similarity/dissimilarity measures for pdfs
•
•
•
Proposed tracking technique
Results, comparisons and demos
Ongoing investigations
–
–
–
–
•
•
Incorporating color cues, and other features
Adding constraints on object shape
Application in morphing (?)
Incorporating object motion information (??)
Summary
Acknowledgements
Tracking Deformable Objects
• Desirable properties of deformable models:
– Adapt with deformations (sometimes drastic
deformations, depending on applications)
– Ability to learn object and background:
• Ability to separate foreground and background
• Ability to recognize object from one image frame to the next,
in an image sequence
Show cine MRI video
Some Existing Deformable Models
• Deformable models:
– Highly deformable
• Examples: snake or active contour, B-spline snakes, …
• Good deformation, but poor recognition (learning) ability
– Not-so-deformable
• Examples
– Active shape and appearance models
– G-snake
– …
• Good recognition (learning) capability, but of course poor
deformation ability
So, how about good deformation and good recognition capabilities?
Technical Background: Level Set
Function
• A level set function represents a contour or a
front geometrically
• Consider a single-valued function (x, y) over
the image domain; intersection of the x-y plane
and  represents a contour:
2
2

 ( x  X ( x, y ))  ( y  Y ( x, y )) , if ( x, y ) is inside the object
 ( x, y )  
2
2

 ( x  X ( x, y ))  ( y  Y ( x, y )) , otherwise,
(X(x, y), Y(x, y)) is the point on the curve that is closest to the (x, y) point
• Matlab demo (lev_demo.m)
Applications of Level Set: Image
Segmentation
•
•
•
•
•
•
Matlab segmentation demo (yezzi.m)
Vessel segmentation
Brain reconstruction
Virtual endoscopy
Trachea fly through
…tons out there
Show videos
Level Set Applications: Image
Denoising
• Two example videos
Show video
Level Set Applications: Robotics
• Finding shortest path
Show video
Level Set Applications: Computer
Graphics
•
•
•
•
Morphing
Simulation
Animation
….
http://www.sci.utah.edu/stories/2004/fall_levelset.html
Go to http://graphics.stanford.edu/~fedkiw/
for amazing videos
More Applications of Level Set
Methods
• Go to
http://math.berkeley.edu/~sethian/2006/Ap
plications/Menu_Expanded_Applications.h
tml
Technical Background: Non-Parametric
Density Estimation
Normalized image intensity histogram:
1
( I ( x, y )  i ) 2
H (i )   exp( 
)dxdy
2
C
2 i
I(x, y) is the image intensity at (x, y)
i is the standard deviation of the Gaussian kernel
C is a normalization factor that forces H(i) to integrate to unity
Technical Background: Similarity and
Dissimilarity Measures for PDFs
Kullback-Leibler (KL) divergence (a dissimilarity measure):
KL( P, Q)   Q( z ) log(
Q( z )
)dz
P( z )
Bhattacharya coefficient (a similarity measure):
BC ( P, Q)   Q( z ) P( z ) dz
P(z) and Q(z) are two PDFs being compared
Proposed Method: Tracking Deformable
Object
• Deformable Object model (due to Leventon [1]):
– From the first frame learn the joint pdf of level set function and
image intensity (image feature)
• Tracking:
– From second frame onward search for similar joint pdf
[1] M. Leventon, Statistical Models for Medical Image Analysis, Ph.D. Thesis, MIT, 2000.
Deformable Object Model
• Joint probability density estimation with Gaussian kernels:
1
( ( x, y )  l ) 2
( J ( x, y )  i ) 2
Q(l , i)   exp( 
) exp( 
)dxdy
2
2
C
2 l
2 i
Level set function value: l
Image intensity: i
J(x, y) is the image intensity at (x, y) point on the first image frame
(x, y) is the value of level set function at (x, y) on the first image frame
C is a normalization factor
We learn Q on the first video frame given the object contour (represented
by the level set function)
Proposed Object Tracking
• On the second (or
subsequent) frame
compute the density:
1
( ( x, y )  l ) 2
( I ( x, y )  i ) 2
P(l , i)   exp( 
) exp( 
)dxdy
2
2
C
2 l
2 i
• Match the densities P and
Q by KL-divergence:
KL   Q(l , i ) log
• Minimize KL-divergence
by varying the level set
function (x, y)
Q(l , i)
dldi
P(l , i )
Note that here only P is
a function of (x, y)
I(x, y) is the image intensity at (x, y) on the second/subsequent frame
(x, y) is the level set function at on the second/subsequent frame
Minimizing KL-divergence
• In order to minimize KL-divergence we use
Calculus of variations
• After applying Calculus of variations the rule of
update (gradient descent rule) for the level set
function becomes:
Q(l , i) ( t ( x, y )  l )
( t ( x, y )  l ) 2
( I ( x, y )  i ) 2
 ( x, y)   ( x, y)  (t ) 
exp( 
) exp( 
)dldi
2
2
2
P(l , i)
l
2 l
2 i
t 1
t
t : iteration number
t : timestep size
Minimizing KL-divergence:
Implementation
• There is a compact way of expressing the update
rule:
Q(l , i) ( t ( x, y )  l )
( t ( x, y )  l ) 2
( I ( x, y )  i ) 2
 ( x, y)   ( x, y)  (t ) 
exp( 
) exp( 
)dldi
P(l , i)
 l2
2 l2
2 i2
t 1
t
convolution
Q
P
 t 1 ( x, y )   t ( x, y )  (t )(  g1 )( t ( x, y ), I ( x, y ))
Where g1 is a convolution kernel:
l2
i2
g1 (l )  2 exp(  2 ) exp(  2 )
l
2 l
2 i
l
Q
P
is a function defined simply as:
Q Q(l , i )

P
(l , i )
Minimizing KL-divergence: A Stable
Implementation
• The previous implementation is called explicit scheme and is
unstable for large time steps; if small time step is used then the
convergence will be extremely slow
• One remedy is a semi-implicit scheme of numerical implementation:
 t 1 ( x, y ) 
 t ( x, y )  (t )(
1  (t )(
Where g is a convolution kernel:
l2
i2
g (l )  exp(  2 ) exp(  2 )
2 l
2 i
lQ
 g )( ( x, y ), I ( x, y ))
P
Q
 g )( ( x, y ), I ( x, y ))
P
lQ
is a function defined simply as:
P
lQ
Q(l , i )
l
P
(l , i )
In this numerical scheme t can be large and still the solution will
be convergent; So very quick convergence is achieved in this scheme
Results: Tracking Cardiac Motion
A few cine MRI frames and delineated boundaries on them
Show videos
Numerical Results and Comparison
Segmentation Score
Pratt FOM
1
0.5
0
10
20
30
40
50
Frame number
60
GVF snake method
Proposed method
70
80
90
1
0.5
0
10
20
30
40
50
Frame number
60
GVF snake method
Proposed method
70
80
90
Sequence with slow heart motion
Segmentation Score
Pratt FOM
1
0.5
0
20
40
60
80
100
Frame number
120
GVF snake method
140
160
Proposed method
180
1
0.5
0
20
40
60
80
100
Frame number
Sequence with rapid heart motion
Comparison of mean performance measures
Pratt’s FOM
Segmentation Score
Slow Seq.
Rapid Seq.
Slow Seq.
Rapid Seq.
GVF Snake Method
0.51
0.62
0.51
0.44
Proposed Method
0.88
0.77
0.74
0.79
120
GVF snake method
140
160
180
Proposed method
Extensions: Tracking Objects in
Color Video
• If we want to learn joint distribution of level set
function and color channels (say, r, g, b), then
non-parametric density estimation suffers from:
– Slowness
– Curse of dimensionality
• Another important theme is combine edge
information and region information of objects
• One remedy sometimes is to take a linear
combination of r, g, and b channels
– Fisher’s linear discriminant can be used to learn the
coefficients of linear combination
• A demo
Extensions: Adding Object Shape
Constraint
• Can we constrain the
object shape in this
computational
framework?
Minimize:
E ( )  KL(Q, P)  KL(q, p)
where
Frame number: 1
q(l ) 
1
K
 exp( 
( ( x, y )  l ) 2
)dxdy
2 l2
p(l ) 
1
K
 exp( 
( ( x, y )  l ) 2
)dxdy
2 l2
Frame number: 2
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
180
180
50
100
150
200
50
100
150
200
Application in Computer Graphics:
Morphing
Initial object Shape and
intensity/texture
(J1, 1)
Final object Shape and
intensity/texture
(I2, 2)
(I1, 1), (I2, 2), …..
Morphing: generate realistic intermediate tuples (It, t)
Morphing: Formulation
• Generate intermediate shapes, i.e., level
set function t (say, via interpolation):
t 1 
t  2
1  t
,
1   1
• Next, generate intermediate intensity It by
maximizing:
E ( I t )  BC ( PDF ( 1 , J1 ), PDF (t , I t )) 
BC ( PDF ( 2 , J 2 ), PDF (t , I t ))
• Once again we get a similar PDE for It
Morphing: Preliminary Results
Target
Source
20
20
40
40
60
60
80
80
100
100
120
120
20
40
60
80
100
20
120
Show videos
40
60
80
100
120
Summary
• Highly deformable object tracking:
Variational minimization of KL-divergence
leading to fast and stable partial
differential equations
• Several exciting extensions
• Application in morphing
Acknowledgements
•
•
•
•
Baidyanath Saha
CIMS lab and Prof. Hong Zhang
Prof. Dipti P. Mukherjee, Indian Statistical Institute
Department of Computing Science, UofA