Snakes with Some Math
Acknowledgements: University of Western Ontario, University of
Manchester, demos from Visual Dynamics Group (University of
Oxford),
Snakes
•Snakes, active contours [Kass, Witkin, Terzopoulos 1987]
•In general, deformable models are widely used
Given: initial contour (model) near desirable object
Snakes
•Snakes, active contours [Kass, Witkin, Terzopoulos 1987]
•In general, deformable models are widely used
Given: initial contour (model) near desirable object
Goal: evolve the contour to fit exact object boundary
Tracking via deformable models
1. Use final contour/model extracted at frame
t as an initial solution for frame t+1
2. Evolve initial contour to fit exact object
boundary at frame t+1
3. Repeat steps 1 and 2 for t ‘= t+1
Tracking via deformable models
Tracking Heart Ventricles
“Snakes”
• A smooth 2D curve which matches to image data
• Initialized near target, iteratively refined
• Can restore missing data
initial
intermediate
final
Parametric Curve
Representation
(continuous case)
• A curve can be represented parametrically
 ( s)  ( x( s), y ( s))
open curve
0  s 1
closed curve
Note: in computer vision and medical image analysis communities the term “snake”
is normally associated with such parametric representation of contours.
Internal Energy
• The bending energy of a continuous curve
is
Ein
d
( ( s ))   ( s )
ds
Elasticity
2
d
  (s)
2
d s
2
2
Stiffness
The more the curve bends, the larger this value
1
Ein   Ein ( ( s)) ds
0
External energy
• The external energy describes how well
the curve matches the image data locally
• Numerous forms can be used, attracting
the curve toward different image features
Simple Edge Strength
• Suppose we have an image I(x,y)
• Generate gradient images Gx ( x, y) & G y ( x, y )
• External energy based on edge strength at
a point is then
Eex ( (s))  ( | Gx ( (s)) |2  | Gy ( (s)) |2 )
(Negative in order that minimizing it forces the curve toward large edges)
• An external energy term for a snake is
1
thus
Eex   Eex ( ( s)) ds
0
Snake Energy (continuous form)
• The total energy of the snake is defined as
Etotal  Ein  Eex
1
Ein   Ein ( ( s)) ds
e.g. bending energy
0
1
Eex   Eex ( ( s)) ds
0
e.g. total edge strength
under curve
Discrete approach
discrete image
discrete snake
representation
discrete optimization
(dynamic programming)
Discrete and Continuous
approaches
Continuous
Models in
Image Analysis
Discrete
Models in
Image Analysis
The relationship is somewhat analogous to relation between
continuous and discrete models in physics, e.g.
Continuous Model of Light
Electro-Magnetic Waves
Discrete Model of Light
Photons
Parametric Curve
Representation
(discrete case)
• Represent the curve with a set of n points
 i  ( xi , yi )
i  0  n 1
Discrete Representation
• If the curve is represented by n points
 i  ( xi , yi )
d vi 1  i 1

ds
2
Ein 
n 1

i 0
i  0  n 1
d 2
 ( i 1  i )  ( i  i 1 )   i 1  2 i  i 1
2
ds
 | i 1  i |   | i 1  2 i  i 1 |
2
Elasticity
Stiffness
2
Simple Elastic Curve (example)
• For a curve represented as a set of points
a simple elastic energy term is
n 1
Ein
    Li
2
i 0
n 1
    ( xi 1  xi ) 2  ( yi 1  yi ) 2
i 0
This encourages the closed curve to shrink to a
point (like a very small elastic band)
Encouraging point spacing
• To stop the curve from shrinking to a point,
add a term such as [v-v’]^2 where v’ is
average distance between snaxels.
Simple Edge Strength
• An external energy term for a (discrete)
snake based on image edge
n 1
Eex
   | Gx ( xi , yi ) |2  | G y ( xi , yi ) |2
i 0
(Negative in order that minimizing it forces the curve toward large edges)
Simple Elastic Snake
• A simple elastic snake is thus defined by
– A set of n points,
– An internal elastic energy term
– An external edge based energy term
• To use this to locate the outline of an
object
– Initialize in the vicinity of the object
– Modify the points to minimize the total energy
Simple Elastic Snake
• In this case we have
Etotal ( x)  Ein ( x)  Eex ( x)
• where
x  ( x0 ,, xn 1 , y0 ,, yn 1 )T
• Optimization problem (2n variables)
– note that we can easily compute the derivatives, which
allows efficient optimization (via gradient descent)
converging to a local minima
• more robust option: Dynamic Programming
Synthetic example
(1)
(2)
(3)
(4)
Dealing with missing data
• The smoothness constraint can deal with
missing data (sometimes maybe wrong!):
Relative weighting
• Notice that the strength of the internal
elastic component can be controlled by a
n 1
parameter,
2
Ein     Li

i 0
• Increasing this increases stiffness of curve
large

medium

small

Note: values of energy are normalized from 0 to 1, so if 0.1 times 10 becomes .01, etc.,
Then re-scale it. Basically this weight makes it more sensitive to internal energy.
Simple shape prior
• If object is some smooth variation on a
known shape, use
n 1
Ein
2
ˆ
    ( i  i )
i 0
• where {ˆi } give points of the basic shape
Alternative External (image)
Energies
• Directed gradient measures
n 1
Eex
   u x ,i Gx ( i )  u y ,i G y ( i )
i 0
– Where ui  (u x ,i , u y ,i )
is the unit normal to
the boundary at contour point  i
– This gives a good response when the
boundary has the same direction as the edge,
but weaker responses when it does not
Open and Closed Curves
 0  n
0
 n 1
closed curve
open curve
• When using an open curve we can impose
constraints on the end points (e.g. end points may
have fixed position)
Additional Constraints
• Snakes originally developed for interactive image
segmentation
• Initial snake result can be nudged where it goes wrong
(correct the incorrect ones)
• Simply modify the external energy term to
– Pull nearby points toward cursor, or
– Push nearby points away from cursor
Interactive (external) forces
• Pull points towards cursor:
n 1
E pull
r2
 
2
i 0 |  i  p |
Nearby points get
pulled hardest
Negative sign gives
better energy for
positions near p
Interactive (external) forces
• Push points from cursor:
n 1
E push
r2
 
2
i 0 |  i  p |
Nearby points get
pushed hardest
Positive sign gives
better energy for
positions far from p
Dynamic snakes
• Adding motion parameters into hidden
variables (for each snake node)
• Introduce energy terms for motion
consistency
• Example: use optic flow to get candidate
points for the snaxels in the next frame.
Discrete Snakes Optimization
• At each iteration we compute a new snake position
within proximity to the previous snake
• New snake energy should be smaller than the
previous one
• Stop when the energy can not be decreased within
local neighborhood of the snake (local energy
minima)
Optimization Methods
1. Gradient Descent
2. Dynamic Programming
Gradient Descent
• Example: minimization of functions of 2 variables
E ( x, y )
( x0 , y0 )
 E
  xEi 
  E 
  yi 
negative gradient at point (x,y) gives direction of the
steepest descent towards lower values of function E
Gradient Descent
• Example: minimization of functions of 2 variables
E ( x, y )
( x0 , y0 )
 Ex 
 x   x 
      t   E 
 y   y 
 y 


p  p  t  E
Stop at a local minima where

E  0
Gradient Descent
• Example: minimization of functions of 2 variables
E ( x, y )
High sensitivity wrt. the initialisation !!
Gradient Descent for Snakes
simple elastic snake
n 1 energy
Etotal ( x0 , , xn 1 , y0 , , yn 1 )    | Gx ( xi , yi ) |2  | G y ( xi , yi ) |2
i 0
n 1
energy as a function 2n variables
    ( xi 1  xi ) 2  ( yi 1  yi ) 2
i 0

Fi

F  E
Force is a negative gradient
of energy function
(2n-dimentional vector)
We can break force/gradient into
components corresponding to
individual snake nodes

 F0 
  
F
  1 
 ... 
 
 Fn 1 

 xEi 
Fi    E 
 yi 
Discrete Snakes:
“Gradient Flow” evolution
dC  dt  E
Update equation for each node

 i   i  Fi  t



  i
i
Contour evolution via
“Gradient flow”
i  0,, n  1

 i
Stopping criteria:

Fi  E |i  0
for all i
at a local minima of energy
Difficulties with Gradient
Descent
• Very difficult to obtain accurate estimates of highorder derivatives on images (discretization errors)
– E.g., estimating
E
x
requires computation of G x
x

2I
x 2
• Gradient descent is not trivial even for onedimensional functions. Robust numerical
performance for 2n-dimensional function is
problematic.
– Choice of parameter
t is non-trivial
• Small t , the algorithm may be too slow
• Large t , the algorithm may never converge
– Even if “converged” to a good local minima, the snake is
likely to oscillate near it
Alternative solution for 2D
snakes:
Dynamic Programming
• In most cases, snake energy can be
rewritten as a sum of pair-wise interaction
potential
Etotal ( 0 , , n 1 ) 
n 1
 E ( ,
i
i 0
i
i 1
)
More generally, it can be written as a sum of
triple-interaction potentials.

Etotal ( 0 , , n 1 ) 
n 1
 E (
i 0
i
i 1
, i , i 1 )
Snake energy:
pair-wise interactions
Example: simple elastic
snake energy
n 1
Etotal ( x0 , , xn 1 , y0 , , yn 1 )    | Gx ( xi , yi ) |2  | G y ( xi , yi ) |2
i 0
n 1
    ( xi 1  xi ) 2  ( yi 1  yi ) 2
i 0
n 1
n 1
Etotal ( 0 , , n 1 )    || G ( i ) ||     ||  i 1  i ||2
2
i 0
Etotal ( 0 ,  , n 1 ) 
where
i 0
n 1
 E ( ,
i 0
i
i
i 1
)
Ei ( i , i 1 )   || G ( i ) ||2  ||  i  i 1 ||2
Q: give an example of snake with triple-interaction potentials?
DP Snakes
[Amini, Weymouth, Jain, 1990]
v2
v3
v1
v4
v5
v6
control points
First-order interactions
E (v1 , v2 ,..., vn )  E1 (v1 , v2 )  E2 (v2 , v3 )  ...  En1 (vn1 , vn )
Energy E is minimized via Dynamic Programming
DP Snakes
[Amini, Weymouth, Jain, 1990]
v2
v3
v1
v4
v5
v6
control points
First-order interactions
E (v1 , v2 ,..., vn )  E1 (v1 , v2 )  E2 (v2 , v3 )  ...  En1 (vn1 , vn )
Energy E is minimized via Dynamic Programming
Iterate until optimal position for each point is the center of the box,
i.e. the snake is optimal in the local search space constrained by boxes
Dynamic Programming (DP)
Viterbi Algorithm
Here we will concentrate on first-order interactions
sites
E1 (v1 , v2 )  E2 (v2 , v3 )  ...  En1 (vn1 , vn )
states
1
2
…
m
v1
E1 (v1 , v2 )
v2
E2 (v2 , v3 )
v3
E3 (v3 , v4 )
v4
E4 (v4 , vn )
vn
E1 (1)  0
E2 (1)
E3 (1)
E4 (1)
E n (1)
E1 (2)  0
E2 (2)
E3 ( 2 )
E4 (2)
En (2)
E1 (3)  0
E2 (3)
E3 (3)
E4 (3)
En (3)
E1 (4)  0
E2 (4)
E3 ( 4 )
E4 (4)
En (4)
Complexity:
2
O(nm )
This is iterative, as search nbd changes
each time.
Dynamic Programming
for a closed snake?
Clearly, DP can be applied to optimize an open ended snake
E1 (v1 , v2 )  E2 (v2 , v3 )  ...  En1 (vn1 , vn )
1
n
Can we use DP for a “looped” energy in case of a closed snake?
E1 (v1 , v2 )  E2 (v2 , v3 )  ...  En1 (vn1 , vn )  En (vn , v1 )
 n 1
n
1
2 3 4
Dynamic Programming
for a closed snake
E1 (v1 , v2 )  E2 (v2 , v3 )  ...  En1 (vn1 , vn )  En (vn , v1 )
Unfortunately, DP can not be directly applied in case of a “loop”.
However, some (approximation) tricks are often used in practice…
1. Use DP to optimize snake energy
with fixed  1 (according to a
given initial snake position).
2. Use DP to optimize snake energy
again. This time fix position of an
intermediate node  n / 2  ˆn / 2
where ˆ is an optimal position
obtained in step 1.
Problems with snakes
• Depends on number and spacing of control
points
• Snake may oversmooth the boundary
• Not trivial to prevent curve self intersecting
(having a model helps!)
• Can not follow topological changes of objects
Problems with snakes
• May be sensitive to initialization
• may get stuck in a local energy minimum near initial contour
• Numerical stability can be an issue for gradient descent
and variational methods (continuous formulation)
– E.g. requires computing second order derivatives
• The general concept of snakes (deformable models) does
generalize to 3D (deformable mesh), but many robust
optimization methods suitable for 2D snakes do not apply
in 3D
– E.g.: dynamic programming only works for 2D snakes
Problems with snakes
• External energy: may need to diffuse image gradients,
otherwise the snake does not really “see” object
boundaries in the image unless it gets very close to it.
image gradients I
are large only directly on the boundary
I
Diffusing Image Gradients
image gradients diffused via
Gradient Vector Flow (GVF)
Chenyang Xu and Jerry Prince, 98
http://iacl.ece.jhu.edu/projects/gv
f/
Alternative Way to
Improve External Energy
n 1
n 1
• Use Eex   D(vi ) instead of Eex   | I | (vi ) where
i 0
i 0
D() is
– Distance Transform (for detected binary image features, e.g.
edges)
Distance Transform can
be visualized as a grayscale image
binary image features
(edges)
Distance Transform
D ( x, y )
• Generalized Distance Transform (directly for image gradients)
Distance Transform
Image features (2D)
1
1
1
1
2
3
4
5
Distance Transform
0 1 2 3 4 3
0 1 2 3 3 2
0 1 2 3 2 1
0 0 1 2 1 0
1 1 2 1 0 1
2 2 2 1 0 1
3 3 2 1 0 1
4 4 3 2 1 0
2
1
0
1
2
2
2
1
Distance Transform is a function D() that for each image
pixel p assigns a non-negative number D ( p ) corresponding to
distance from p to the nearest feature in the image I
Some more Variations..
Internal Energy
• Controls the contour smoothness and
continuity
Eint (vi )  1 (1 
vi 1vi  vi vi 1
vi 1vi  vi vi 1
)  1 vi  vi 1  d
External Energy
• Limitations of the constraint of intensity
homogeneity
– Cannot be applied to open contour
– Does not work for band-shaped object
Ultrasound image
Key-chain ring image
Band Energy
• For the contour element vi , we can define
its tangent as:
• Two regions R and
contour element vi
i
R'i
can be defined for
Band Energy
• Normalized mean intensity difference
• Band energy (given ring example is opposite)
• Combination of intensity and gradient:
Penalty
Performance
Without band energy
Image
Initialization
With band energy
Without band energy
With band energy
3D reconstruction Experiments
Input
Results