Active Contours / Planes
Sebastian Thrun, Gary Bradski, Daniel Russakoff
Stanford CS223B Computer Vision
http://robots.stanford.edu/cs223b
Some slides taken from: Julien Jomier
Outline
Snakes
 Planes

Sebastian Thrun
Stanford University
CS223B Computer Vision
Introduction (1)
 The
active contour model, or snake, is
defined as an energy-minimizing spline.
 Active contours results from work of Kass
et.al. in 1987.
 Active contour models may be used in
image segmentation and understanding.
 The snake’s energy depends on its shape
and location within the image.
 Snakes can be closed or open
Sebastian Thrun
Stanford University
CS223B Computer Vision
Example
Sebastian Thrun
Stanford University
CS223B Computer Vision
Introduction (2)
Aorta segmentation using active contours
Sebastian Thrun
Stanford University
CS223B Computer Vision
Introduction (3)
First an initial spline (snake) is placed on the
image, and then its energy is minimized.
 Local minima of this energy correspond to
desired image properties.
 Unlike most other image models, the snake is
active, always minimizing its energy functional,
therefore exhibiting dynamic behavior.
 Also suitable for analysis of dynamic data or 3D
image data.

Sebastian Thrun
Stanford University
CS223B Computer Vision
Kass Algorithm

The snake is defined parametrically as v(s)=[x(s),y(s)],
where s[0,1] is the normalized arc length along the
contour.
The energy functional to be minimized may be written as
*
Esnake
 0 Esnake (v( s )) ds
1
  0 Econt (v( s )) ds   0 Ecurv (v( s )) ds
1
1
  0 Eimage (v( s )) ds
1
Econt = snake continuity
Ecurv = snake curvature
Eimage = image forces (e.g., edge attraction)
Sebastian Thrun
Stanford University
CS223B Computer Vision
Internal Energy

Continuity:
dv
Econt (v ( s )) 
ds

2
Curvature:
2
d v
Ecurv (v ( s )) 
ds 2
Sebastian Thrun
2
Stanford University
CS223B Computer Vision
Image Forces

Dark/Bright Lines
Eimage (v( s ))  I (v( s ))

Edges
Eimage(v(s))   I (v(s))
Sebastian Thrun
Stanford University
CS223B Computer Vision
Trade-offs
*
Esnake
 0 Esnake (v( s )) ds
1
  0 Econt (v( s )) ds   0 Ecurv (v( s )) ds
1
1
  0 Eimage (v( s )) ds
1

, ,  determine trade-off
Sebastian Thrun
Stanford University
CS223B Computer Vision
Numerical Algorithm
*
Esnake
   | pk  pk 1 |2   | pk 1  2 pk  pk 1 |   | I ( pk ) |
2
k


Select N initial locations p1,…, pN
Update until convergence

pi  pi  0.1
pi
Sebastian Thrun

k


 | pk  pk 1 |2

2
  | pk 1  2 pk  pk 1 | 




|

I
(
p
)
|
k


Stanford University
CS223B Computer Vision
Snakes, Done Right

Define Spline over p1,…, pN
Optimize criterion for all points on spline
Allow for corners

(optimization becomes tricky, fills entire literature)


Sebastian Thrun
Stanford University
CS223B Computer Vision
Applications
http://www.markschulze.net/snakes/
Main Applications are:
 Segmentation
 Tracking
 Registration
Sebastian Thrun
Stanford University
CS223B Computer Vision
Examples (1)
Julien Jomier
Sebastian Thrun
Stanford University
CS223B Computer Vision
Examples (2)
Julien Jomier
Sebastian Thrun
Stanford University
CS223B Computer Vision
Examples (3)
Heart
Sebastian Thrun
Stanford University
Julien Jomier
CS223B Computer Vision
Examples (4)
3D Segmentation of the Hippocampus
Julien Jomier
Sebastian Thrun
Stanford University
CS223B Computer Vision
Examples (5)
Julien Jomier
Sebastian Thrun
Stanford University
CS223B Computer Vision
Example (6)
Julien Jomier
Sebastian Thrun
Stanford University
CS223B Computer Vision
Example (7)
Julien Jomier
Sebastian Thrun
Stanford University
CS223B Computer Vision
Problems with snakes
 Snakes
sometimes degenerate in shape
by shrinking and flattening.
 Stability and convergence of the contour
deformation process unpredictable.
Solution: Add some constraints
 Initialization
is not straightforward.
Solution: Manual, Learned, Exhaustive
Sebastian Thrun
Stanford University
CS223B Computer Vision
References
M. Kass, A. Witkin, and D. Terzopoulos.
Snakes: Active contour models. In Proc. 1st
ICCV, pages 259-268, June 1987. London, UK.
 Yongjik Kim. A summary of Implicit Snake
Formulation.
 Jorgen Ahlberg. Active Contours in Three
Dimensions.
 M. Bertalmio, G. Sapiro and G. Randall.
Morphing Active Contours. IEEE PAMI, Vol
22, No 7, July 2000

Sebastian Thrun
Stanford University
CS223B Computer Vision
Outline
Snakes
 Planes

Sebastian Thrun
Stanford University
CS223B Computer Vision
Finding Planes



Lines in Image (e.g., Hough transform)
Planes in 3D space (e.g., stereo reconstruction,
SFM)
Problems:
–
–
–
–
Number of planes
Parameters of planes
Data association: which point belongs to which plane
Outlier removal (noise, non-flat surfaces)
Sebastian Thrun
Stanford University
CS223B Computer Vision
Range Data in Multi-Planar Environment
Taken with Laser Range Finder, but similar for
Sebastian Thrun
Stanford University
CS223B Computer Vision
Basic Idea
*
Sebastian Thrun
Stanford University
*
CS223B Computer Vision
Mathematical Model: Expectation Maximization

3D Model:
  {1 , 2 ,, J }
Planar surface in 3D
 j   j ,  j  3  
y
surface
normal 
surface
Distance point-surface
z
displacement 
dist(  j , zi )   j  zi  
x
Sebastian Thrun
Stanford University
CS223B Computer Vision
Mixture Measurement Model

Case 1: Measurement zi caused by plane j
1
p ( zi |  j ) 
2 2
e
1 ( j  zi   j )

2
2
2
 Case 2: Measurement zi caused by something else
p ( zi |  * ) 
Sebastian Thrun
1
zmax

1
2 2
Stanford University
e
2
1 z max
 ln
2 2 2
CS223B Computer Vision
Measurement Model with Correspondences

1
p( zi |  , c1 ,, cJ , c* ) 
( j  zi   j )
z max 2 J
1 
 c* ln

c
j
2
2 2 j 1
2

}
2 2
e
2




correspondence variables C:
c* , c j {0,1}
J
c*   c j  1
j 1
 p( Z |  , C )  
i 1
Sebastian Thrun

1
2
( j  zi   j )
z max 2 J
1 
 ci* ln

c
ij
2
2 2 j 1
2

2
e
Stanford University
2




CS223B Computer Vision
Expected Log-Likelihood Function
p( Z |  , C )  
i 1
…after some simple math
Ec ln p( Z , C |  )

1
2

( j  zi   j )
z max 2 J
1 
 ci* ln

c
ij
2
2
2

2
j

1

2
e




1


ln


2
 ( J  1) 2



2
  1 E[c ] ln zmax

i  2 i* 2 2


2 
J
(


z


)
j
  1 E[c ] j i

ij
2
 2


j 1


probabilistic
data association
Sebastian Thrun
2
Stanford University
mapping with
known data association
CS223B Computer Vision
E-Step

Calculate expectations with fixed model  :
E[ci* ]  e
E[cij ]  e
2
1 z max
 ln
2 2 2
1 ( j  zi   j )

2
2
2
J
(normalize so that
E[ci* ]   E[cij ]  1 )
j 1
Sebastian Thrun
Stanford University
CS223B Computer Vision
M-Step

Maximize
Ec ln p ( Z , C |  )

subject to  j  j  1

1


 ln

2
 ( J  1) 2



2
  1 E[c ] ln z max

i  2 i* 2 2


2 
  1 E[c ] ( j  zi   j ) 
ij
2
 2




Is equivalent to minimizing
J
2
E
[
c
](


z


)
 ij j i j
i
subject to  j  j  1
j 1
Sebastian Thrun
Stanford University
CS223B Computer Vision
M-Step: Solve via Lagrange Multipliers
J
2
E
[
c
](


z


)
 ij j i j  min
i

j :  j   j  1
j 1
Define
J
J
L   E[cij ]( j  zi   j )    j j  j
2
i

j 1
j 1
And observe that
!
L
 E[cij ]( j  zi   j ) zi   j j  0
 j
!
L
 E[cij ]( j  zi   j )  0
 j
 j  j  1
Sebastian Thrun
Stanford University
CS223B Computer Vision
M-Step: Solve via Lagrange Multipliers

Solve for  :
L
 E[cij ]( j  zi   j )  0
 j
!

 j 
 E[c ]  z
 E[c ]
kj
j
k
k
kj
j
k
Substitute back:
!
L
 E[cij ]( j  zi   j ) zi   j j  0
 j
E[ckj ] j  z k

!
L
k

 E[cij ]( j  zi 
) zi   j j  0
 j
 E[ckj ] j
k

Is of the linear form:
A j   j j

Solution: two Eigenvectors of A with smallest Eigenvalues.
Sebastian Thrun
Stanford University
CS223B Computer Vision
Determining Number of Surfaces
Add
Firstmodel
model
Prune
E/M
M-Step
M-step
E-Step
Steps
components
model
component
*
*
Sebastian Thrun
J =2
=1
=3
*
Stanford University
CS223B Computer Vision
Choosing the “Right” Number of Planes: AIC
J=0
J=1
J=2
J=3
J=4
J=5
increased data likelihood
increased prior probability
log p( J | d )  const  log p(d | J )  log p( J )
Sebastian Thrun
Stanford University
CS223B Computer Vision
Model Selection
Approximately every 20 iterations of EM:
 Start new surfaces
– Near any set of collinear measurements

Terminate unsupported surfaces
– If not supported by enough measurements
– If density of measurements too low
– If two planes are too close to each other
Sebastian Thrun
Stanford University
CS223B Computer Vision
Results
Sebastian Thrun
Stanford University
CS223B Computer Vision
Online Robotic Mapping @ CMU
Sebastian Thrun
Stanford University
CS223B Computer Vision
Online Robotic Mapping @ Stanford
Sebastian Thrun
Stanford University
CS223B Computer Vision