Image Segmentation of Multiple
Objects and Their Compartments
Jerry L. Prince
Image Analysis and Communications
Laboratory (IACL)
http://iacl.ece.jhu.edu
Johns Hopkins University
© 2010
Acknowledgments
•
•
•
•
•
•
•
•
•
•
Chenyang Xu
Dzung Pham
Xiao Han
Duygu Tosun
Bai Ying
Daphne Yu
Kirsten Behnke
Xiaodong Tao
Sarah Ying
Xian Fan
•
•
•
•
•
•
•
•
•
Susan Resnick
Mike Kraut
Maryam Rettmann
Christos Davatzikos
Nick Bryan
Aaron Carass
Ulisses Braga-Neto
Lotta Ellingsen
Pierre-Louis Bazin
Funding sources: NSF, NIH/NINDS, NIH/NIA
Outline
• Introduction
• Deformable models
• TGDM: topology-preserving geometric
deformable model
• MGDM: multi-object geometric deformable
model
• Conclusion
Outline
• Introduction
• Deformable models
• TGDM: topology-preserving geometric
deformable model
• MGDM: multi-object geometric deformable
model
• Conclusion
Conventional Structural Image
www.medical.philips.com
Segmentation of Brain Structures
TOADS
CRUISE
Bazin and Pham, TMI, 2007
Xu et al., TMI, 1999
Bazin and Pham, MedIA, 2008
Han et al., NeuroImage, 2004
Tosun et al., NeuroImage, 2006
Cortex
Subcortical Structures
Volumetric MR Data
Multi-Compartment Anatomy
Cerebellar lobules
Thalamic nuclei
Other Multi-Object Scenarios
Satellite imagery
Aerial photographs
Retinal examination
Cell counting
Circuit board inspection
Traffic camera
Outline
• Introduction
• Deformable models
• TGDM: topology-preserving geometric
deformable model
• MGDM: multi-object geometric deformable
model
• Conclusion
Cortical Surface Segmentation
Partial Inflation
Ventricle Segmentation
Deformable Models
• Parametric
deformable models
(PDMs)
─explicit
parameterization
• Geometric
deformable models
(GDMs)
– implicit
representation
Parametric to Geometric
[Osher & Sethian 1988]
Contour Deformation:
Level Set PDE:
Visual Concept of GDM
Properties of GDMs
• Advantages:
– Produce closed, non-self-intersecting contours
– Independent of contour parameterization
– Easy to implement: numerical solution of PDEs on
regular computational grid
– Stable computations
– Automatically changes topology
• Potential disadvantage:
– Does not maintain topology
Topology Behavior
• GDM cannot control topology
• TGDM (ours)  preserves topology
GDM: Standard Geometric
Deformable Model
TGDM: Topology-preserving
Geometric Deformable Model
Why Maintain Topology?
GDM: Standard Geometric
Deformable Model
TGDM: Topology-preserving
Geometric Deformable Model
Outline
• Introduction
• Deformable models
• TGDM: topology-preserving geometric
deformable model
• MGDM: multi-object geometric deformable
model
• Conclusion
Marching Cubes Isosurface
• Where is the boundary
defined by a level set
function?
• Consider voxel values on
corners of a cube
• Label as
– above isovalue
– below isovalue
• Determine position of
triangular mesh surface
passing through the cubes
by linear interpolation
Voxel values
> 0.5
< 0.5
Digital Connectivity
6-connectivity
18-connectivity
26-connectivity
• Consistent pairs:
(foreground,background) → (6,18), (6,26), (18,6), (26,6)
Digital Embedding of Contour Topology
• Contour topology is
determined by signs of the
level set function at pixel
locations
• Topology of the implicit
contour is the same as the
topology of the digital
object
White Points:
Black Points:
Connectivity Rule of Contour
• Topology of digital contour determined by connectivity rule
Same digital object, different topologies
Topology Preservation Principle
[Han et al., PAMI, 2003]
• Preserving surface topology is equivalent to
maintaining the topology of the digital object
• The digital object can only change topology when
the level set function changes sign at a grid point
• Which sign changes can be allowed, and which
cannot?
• To prevent the digital object from changing
topology, the level set function should only be
allowed to change sign at simple points
Simple Point
• Definition: a point is simple if adding or removing
the point from a binary object will not change the
object topology
• Determination: can be characterized locally by
the configuration of its neighborhood (8- in 2D,
26- in 3D) [Bertrand & Malandain 1994]
Simple
NonSimple
x is a Simple Point
 ( x)  0
 ( x)  0
x
x
x is Not a Simple Point
( x)  0
X
X
( x)  0
Topology Preserving Geometric
Deformable Model (TGDM)
• Evolve level set function according to GDM
• If level set function is going to change sign, check
whether the point is a simple point
– If simple, permit the sign-change
– If not simple, prohibit the sign-change
(replace the grid value by epsilon with same sign)
– (Roughly, this step adds 7% computation time.)
• Extract the final contour using a connectivity
consistent isocontour algorithm
Ambiguous Faces
Two possible tilings:
Ambiguous Cubes
Two possible tilings:
Connectivity Consistent MC Algorithm
Ambiguous
Face
(a )
(b)
(c)
Ambiguous
Cube
(d)
(e)
• (black,white)
• (18,6)  choose b, f
• (26,6)  choose b, e
(f)
• (6,18)  choose c, f
• (6,26)  choose c, f
Nested Deformable Surfaces
TGDM-1
Initial WM
Isosurface
TGDM-2
Inner
Surface
TGDM-3
Central
Surface
Pial
Surface
TGDM for Inner Surface
[Han et al., NeuroImage, 2004]
Initial WM Isosurface
Evolving GM/WM Interface
TGDM for Central Surface
Initialize with GM/WM surface
IACL
Evolving toward Central Surface
TGDM for Outer Surface
Start from Central Surface
Evolving toward Outer Surface
Results—Visual Inspection
• surfaces overlaid on cross-sections of
the original image
Axial
Sagittal
Outline
• Introduction
• Deformable models
• TGDM: topology-preserving geometric
deformable model
• MGDM: multi-object geometric deformable
model
• Conclusion
Multiple Object Challenges
1. Maintenance of
multiple level sets
2. Maintenance of object’s
individual topologies
and relationships
between objects
Anatomical parcellation is not arbitrary
Prior Strategies
• N level set functions for N objects
[Paragios00, Brox06, …]
– Pros: Flexibility between objects
– Cons: Objects might overlap or form
gaps; large memory and computational
demands
• Multi-phase [Vese02] (4-color
theorem)
– Pros: Log(N) level set functions for N
objects; low computational complexity;
no overlaps or gaps;
– Cons: Forces limited to region and
length terms; little control over
individual object forces; no 3D
equivalent
Principle of MGDM
• Simple point criterion can
be replaced by digital
homeomorphism criterion
• Movement of collection
of objects occurs primarily
at:
– edges between two objects
or
– junctions between three
objects
• Higher-order relationships
can be ignored
Fan et al., CVPR’08, MMBIA’08
Objects are not digitally homeomorphic.
Level Set Function Decomposition
• N objects Oi, i=1,…N
• Distance to objects:
L0 = Object
• Label functions:
L1 = Nearest neighbor
L2 = 2nd nearest neighbor
Distance and Level Set Functions
• Distance-based functions:
0(x)
• Reconstruction of level set functions:
1(x)
^
2D
3D
Approximation: valid assuming 3 objects max per junction
2(x)
Evolution
• Recall GDM:
• Required evolutions:
• Distance-based functions:
MGDM Algorithm (2D case)
• Assume
1. Compute forces
2. Find “third” neighbor:
3. Compute:
4. Compute:
If
then set
and
•
If
then set
Set
Digital topology and
homeomorphism are readily
added
Simulation Experiments
• Compare algorithms:
– multiphase (MP)
– coupled level sets (CLS)
– ours (MGDM)
• Objective function (classic Mumford
Shah energy; also Chan-Vese for GDM)
• Evaluate:
– convergence, memory usage, computation
time, and misclassification percentage
Visual Comparison
Convergence Comparison
E
Iteration
Quantitative Results
Experiment I: Whole Brain Segmentation
Structure memberships from TOADS [Bazin 07]
(Topology-preserving, Anatomy-driven segmentation)
Sulcal
CSF
Brainstem
Membership
function:
Cerebral
GM
Ventricles
ui
Cerebral WM
Cerebellar
GM
Thalamus
Caudate
Force:
Cerebellar
WM
Putamen
fi  0.5  ui     




Balloon force
Smooth force
Whole Brain Segmentation: 2D Visualization
(a) Original Image
(c) No Topology or Smooth
(b) Result from Toads
(d) No Topology but Smooth
Topology is
preserved with DHC
(e) Single Topology
(f) Group Topology
Whole Brain Segmentation: 3D Visualization
(a) No Topology or Smooth
(c) Single Object Topology
(b) No Topology but Smooth
(d) Group Objects Topology
Object topology and relationships
between objects can be preserved.
Experiment II: Thalamic Nuclei Parcellation
Force for the
thalamus
boundary.
TOADS
Apply to voxels
whose label or the
first neighbor is the
background.
Thalamus
Membership
MP-RAGE
image co-registered
Different forces applied to
different parts of an object
F
Homogeneous
Orientation
FA with PEV color
map
Force for the
thalamus
nuclei.
Apply to voxels whose
label or the first neighbor
belongs to thalamic nuclei.
Thalamic Nuclei Parcellation
 The force for thalamus nuclei parcellation is designed
based on the assumption that the orientations for each nuclei
is homogeneous.
fi ( x)  v( x)  Vi ( x) i
Mean principal orientation
from region i
 The force for thalamus boundary is a combination of
balloon and smooth terms.
f ( x)  0.5  u( x)    ( x) 
Membership function of
thalamus at voxel x
Thalamic Nuclei Parcellation: Result
(a) Membership Function
for Thalamus
Front View
(b) Initialization
Back View
(c) Principal Orientation
of Thalamus
Off Left View
(d) Result
Off Diagonal View
Outline
• Introduction
• Deformable models
• TGDM: topology-preserving geometric
deformable model
• MGDM: multi-object geometric deformable
model
• Conclusion
General Principles
• Object topology can be strictly preserved in
geometry deformable models: TGDM
• Multiple objects can be simultaneously
segmented and
– topology preserved
– object relationships preserved
– memory efficient
– all conventional forces can be applied
– guaranteed to have no overlaps or gaps
Remaining Concerns and the Future
• How to establish initial object or collection?
– digital topology is not always preserved under simple
transformations such as rotation
– recent work on manual skeleton is promising, but
tedious
– automatic topology correction is known only for
spherical topology, and it is not globally optimum
• Problem of objects getting “stuck”
– not so bad in TGDM
– much worse in MGDM
Questions?