1
Real-time segmentation
of 3D echocardiograms,
using a state estimation approach
with deformable models
Fredrik Orderud
Norwegian University of Science & Technology
2
Outline
• Background and motivation
• Framework description
• Publications
1.
2.
3.
4.
5.
6.
7.
8.
Ellipsoid model (computers in cardiology, 2006)
Spline model (MICCAI, 2007)
Active-shape model (CAIP, 2007)
Subdivision models (CVPR, 2008)
Speckle-tracking (f-MICCAI, 2008)
Edge + speckle-tracking (IUS, 2008)
Coupled segmentation (IUS, 2008)
Automatic alignment (SPIE, 2009)
• Conclusions
3
Ultrasound background
Cardiac ultrasound is moving
from 2D to 3D
• Latest generation of scanners
are capable of acquiring
dense image volumes in realtime
• Important competitive
advantage to MR & CT
Image analysis is lagging
behind
• Only post-processing tools
are currently available
• We want analysis tools
robust and fast enough to
operate during imaging
4
Left ventricle
• One of four heart
chambers.
• Pumps oxygenated
blood from lungs to
the rest of the body.
• The most clinically
interesting heart
chamber.
(c) 2006, Wikipedia
5
Ultrasound segmentation
Image segmentation:
•
•
•
Problem of labeling image data
with information about
boundaries, structure locations
etc.
Difficult problem, especially in
ultrasound.
Clutter, reverberation, angle
dependency and drop-out
disrupts images, and makes
segmentation difficult.
Example:
• How to handle that part of the
cardiac wall is missing?
Approach:
• Use a geometric model to
assist the segmentation process
6
Common types of 3D models
•
Level sets
–
–
•
Discrete model
–
–
•
Flat polygonal surface, e.g. simplex mesh (Delingette,
IJCV 199)
Iterative force-based update scheme
Statistical shape model
–
–
–
•
Implicit surface: Zero-crossing of finite diff
quantification (Malladi, PAMI 1995)
PDE-based update scheme to evolve surface
Discrete surface, with predefined deformation modes
(Cootes, IPMI 1993)
Iterative displacement-based update scheme
Can be extended to also capture texture and motion
pattern - AAMM (Bosch, TMI 2002)
Parametric surface
–
–
Continuous surface description (FEM, spline,
subdivision etc.)
Typ. iterative gradient descent update scheme
Iterative
update
schemes
7
Kalman segmentation background
• Prof. Andrew Blake & al., • Kalman filter to update a
univ. Oxford, 1990’ies
spline contour
– Noniterative least-squares
fitting
[1] Blake A, Curwen R, Zisserman A. A framework for spatiotemporal
control in the tracking of visual contours. International
Journal of Computer Vision 1993;11(2):127–145.
[2] Blake A, Isard M, Reynard D. Learning to track the visual
motion of contours. Artificial Intelligence 1995;78(12):179–212.
[3] Blake A, Isard M. Active Contours. Secaucus, NJ, USA:
Springer-Verlag New York, Inc., 1998. ISBN 3540762175.
8
Prior real-time cardiac work
2D: G. Jacob, A. Noble (univ. Oxford):
• ASM model, updated by a Kalman filter
• Investigated relationship between shape
variation and wall thickening to pathology.
2D: Siemens corporate research:
• Similar Kalman-filter algorithms
• Subspace-constrained deformations
• Anisotropic measurement error
3D: Q Duan, E. Angelini (Columbia univ.):
• Cubic spline surface, updated by “level-set”
Mumford-Shah gradient descent.
• 33 ms processing time per frame
[4] Jacob G, Noble JA, Mulet-Parada M, Blake A. Evaluating
a robust contour tracker on echocardiographic sequences.
Medical Image Analysis 1999.
[5] Jacob G, Noble JA, Kelion AD, Banning AP. Quantitative
regional analysis of myocardial wall motion. Ultrasound in
Medicine Biology 2001.
[6] Jacob G, Noble JA, Behrenbruch CP, Kelion AD, Banning
AP. A shape-space based approach to tracking myocardial
borders and quantifying regional left ventricular function
applied in echocardiography. IEEE Medical Imaging 2002
[7] D. Comaniciu, X.S. Zhou & al: Robust real-time myocardial border
tracking for echocardiography: An information fusion approach, IEEE
Medical imaging, 2004
[8] X.S. Zhou, A. Gupta, D. Comaniciu: An information fusion
framework for robust shape tracking, IEEE Pattern analysis and
machine intelligence, 2004
[9] Qi Duan, E. Angelini & al: Real-time segmentation of 4D ultrasound
by Active Geometric Functions, IEEE Intl. Symposium on Biomedical
Imaging (ISBI) 2008.
9
Kalman filter:
• Invented in 1960 (R.
Kalman).
• Widely used for
navigation and RADAR
tracking.
Kalman filtering has some
limitations related to
linearity and Gaussian
assumptions.
Is it out of date?
Apollo guidance computers
10
Research goals
• Extend Kalman tracking
framework to 3D, and
support:
– Different types of
deformable models.
– Different image
measurement.
– Multiple simultaneous
models, arranged in a
hierarchy.
• Use the framework to
measure aspects of left
ventricular function:
– Chamber volumes.
– Myocardial muscle mass.
– Regional myocardial
strain.
11
Real-time 3D Kalman tracking
framework
12
Approach
• Use deformable surface
models
– Described by a limited set of
parameters
– Combine with global transform for
position, size and orientation
• Treat tracking as a sequential
state estimation problem
– Multivariate normal distribution
– Use a Kalman filter to predict and
update state, based on image
measurements and a kinematic
model
Tracking driven by propagation
of uncertainty through time
13
Tracking framework
Segmentations
Edge detection
Kinematic prediction
Deformable model(s)
Volume curves
Speckle tracking
Apical [%]
Lateral
Posterior
Inferior
Septum
100
100
100
100
50
50
50
50
50
0
0
0
0
0
0
-50
-50
10
Midwall [%]
20
-50
10
20
-50
10
20
-50
10
20
-50
10
20
100
100
100
100
100
100
50
50
50
50
50
50
0
0
-50
0
-50
10
Basal [%]
Anterior
100
50
Strain curves
Kalman filter
Ant.Sept.
100
20
0
-50
10
20
0
-50
10
20
20
20
100
100
100
100
100
50
50
50
50
50
50
0
0
0
-50
10
20
0
-50
10
20
0
-50
10
20
20
Next frame
Kinematic prediction
10
20
-50
10
20
Confidence maps
Attractors
20
0
-50
10
10
-50
10
100
-50
20
0
-50
10
10
Anatomic landmarks
14
Processing chain
1. Predict
–
Predict state, based on a kinematic model
and previous estimates
2. Model
–
–
Evaluate surface points, based on state
vector.
Compute Jacobian matrices
3. Measure
–
–
Search for edges in surface normal
direction, and/or
Track speckle pattern.
•
State-vector parameters:
–
–
Local shape.
Global pose (trans, rot, scale).
4. Assimilate
–
–
Perform outlier rejection.
Assimilate measurement in information
space.
5. Update
–
–
Compute updated state estimate, based
on prediction and measurements.
Update surface model on screen.
Bayesian least squares
solution instead of iterative
refinement
15
Edge detection
• Extract intensity “profiles” in
surface normal direction
• 1D edge-detection:
– “Transition criterion”, where the
edge forms a transition from
one intensity level to another
– Determine edge position that
minimizes the sum of squared
errors.
Green
Red
Yellow
Black
- edge discovered outside the surface
- edge detected inside the surface
- discarded outlier edge
- discarded too weak edge
16
Speckle tracking
Block matching:
• Extract “kernel volumes” in
myocardium
• Match to bigger “search
volumes” in next the frame
• Search for best integer
displacement (SAD).
• Sub-sample refinement (optical
flow).
Green
Red
Yellow
- displacement vectors
- discarded displacement vectors
(weak or outside sector)
- discarded outlier displacements
17
Tracking Confidence
•
The Kalman filter does not only
estimate state (shape), but also
covariance (uncertainty)
–
–
–
•
Region of low
confidence
Spatial uncertainty of each node is
preserved at all stages through the
processing chain, since the Kalman
filter is based on 2nd order statistics
(state + cov. Matrix)
Unique property of the Kalman filter!
Model mechanics and measurement
weights will not only affect shape,
but also unscertainty across the
model.
Ccovariance matrices can be
projected onto the surface to
color-code regions of low
confidence
Node uncertainty ellipsoids
18
Advantages
• Flexible
– Supports a range of different
parametric models.
– Models can be arranged in a
hierarchy to support multimodel tracking
• Robust
– Wide range of convergence.
– Enables fully automatic
initialization.
• Intuitive
– Parameterized by physical
quantities
– Spatial uncertainty for image
measurements
– No unobservable “forces” to
tune.
• Efficient
– 4 ms/frame for segmentation
using edge-detection.
– 40 ms/frame when using
speckle-tracking.
10-100x faster than
existing methods
19
Papers
20
A Framework for Real-Time Left Ventricular
Tracking in 3D+T Echocardiography,
Using Nonlinear Deformable Contours and
Kalman Filter Based Tracking
F. Orderud
Proceedings of Computers in Cardiology
2006
21
Ellipsoid Model
• Truncated ellipsoid
• Parameters:
– Translation (tx, ty, tz)
– Rotation/orientation (rx, ry)
– Scaling (sx, sy, sz)
– “Bending” (cx, cy)
10 degrees of freedom.
22
Results
Data:
•
•
Collection of 21 unselected 3D
echocardiography recordings
Initial contour placed at 8 cm depth in first
frame
Objective:
•
•
Not accurate segmentation
Crude approximation to LV size, position
and orientation
initialization
Quality
Good
After one cycle
Count
16
Description
Tracking performed well
Adequate
3
Tracking with reduced
accuracy
Fair
1
Tracking with low accuracy
Poor
1
Unable to track
Subjectively scored by author
azimuth view
elevation view
23
Real-time Tracking of the Left Ventricle in 3D
Echocardiography Using a State Estimation
Approach
F. Orderud, J. Hansegård, S.I. Rabben
Proceedings of the 10th International
Conference on Medical Image Computing
and Computer Assisted Intervention (MICCAI)
2007
24
1.5
Spline Model
1
0.5
0
-0.5
• Using a quadratic splinesurface to segment the
endocardial wall
– 4 x 6 control points that
are allowed to move in/out
to adjust shape
• Global transform for
positioning, scaling and
orientation
-1
-1.5
-2
-2.5
-1.5
-1
-0.5
0
0.5
1
25
Example
26
Results
Protocol:
• Tested in 21 unselected 3D
echocardiography recordings (GE
Vivid 7).
• Compared to GE AutoLVQ.
Results:
• Bland-Altman: 4.1±24.6 ml
agreement (95%), and a strong
correlation (r = 0.92).
• 2.7 mm average point to surface
distance against reference.
27
Real-Time Active Shape Models for
Segmentation of 3D Cardiac Ultrasound
J. Hansegård, F. Orderud, S.I. Rabben
Proceedings of the 12th International
Conference on Computer Analysis of Images
and Patterns - CAIP 2007
28
Deformable model
3D active shape model (ASM)
Linear model consisting of:
•Average shape
•Deformation modes Ai
• Shape controlled by state xl
• Project deformations to normal
direction ni to reduce
computational cost.
Built by PCA on training set (N=31)
segmented with AutoLVQ (J.
Hansegård).
20 deformation modes explains
98% of variation in training set
+
+
+
29
Initialization example
30
Results
Good agreement in volumes
and EF.
2.2 ±1.1 mm point-to-surface
error.
Discussion:
• Few parameters to estimate
• Regularizes segmentation to
physiological realistic shapes
31
GE GRC model (unpublished)
32
Real-time 3D Segmentation of the Left Ventricle
Using Deformable Subdivision Surfaces
F. Orderud, S.I. Rabben
Proceedings in the IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition (CVPR) 2008
33
Subdivision surfaces
•
•
Model the left ventricle with a
subdivision surface
Extension of spline surfaces to
arbitrary topology
–
–
–
–
•
Parameterized by a mesh of control
points
Surface defined as the limit of recursive
refinement process (implicit)
Smooth surface with compact
description
Intuitive, easy to model
Utilized method of J. Stam to
evaluate arbitrary surface points
without recursive subdivision.
J. Stam: Exact Evaluation of Catmull-Clark Subdivision
Surfaces at Arbitrary Parameter Values, SIGGRAPH'98
34
Example
5ms processing time/frame
35
Results
• Better agreement
compared to spline/ASM.
Discussion:
• Very popular in computer
graphics
36
Real-time Left Ventricular SpeckleTracking in 3D Echocardiography With
Deformable Subdivision Surfaces
F. Orderud, G. Kiss, S. Langeland, E.
Remme, H. Torp, S.I. Rabben
Proceedings of the MICCAI workshop on
Analysis of Medical Images 2008
37
Speckle tracking
• Idea
– Edge-detection to initialize
– 3D speckle-tracking as
measurement
• Block matching:
– Integer displacement estimation
with sum of absolute differences
(SAD)
– Sub-sample correction with LucasKanade optical flow
38
Simulations
•
•
Finite-element (FEM)
simulation of a left
ventricle deformation (E.
Remme)
“K-space” ultrasound
simulator (S. Langeland)
infarction
infarction
Dog-heart simulation
Ellipsoid simulation
E. Remme, O. Smiseth: Characteristic Strain Pattern of Moderately Ischemic
Myocardium Investigated in a Finite Element Simulation Model, Functional
Imaging and Modeling of the Heart, 2007, 330-339
T. Hergum, J. Crosby, M. Langhammer, H. Torp: The Effect of Including Fiber
Orientation in Simulated 3D Ultrasound Images of the Heart, IEEE Ultrasonics
Symposium, 2006, 1991-1994
39
Simulation results
• Able to identify infarcted
region in both simulations
• Absolute strain values are
underestimated
End systolic area strain:
Estimated
Sim. A:
Sim. B:
Ground truth
40
In-vivo results
• Tested in 21 unselected invivo recordings
– 50% with cardiac disease
– No ground truth available
• In-vivo challenges:
– Lower image quality, especially
in the near-field
– Lower spatial & temporal
resolution than 2D recordings
Drift after tracking in one cycle:
37ms processing
time per frame
(decimated data)
Absolute drift
Relative drift
Simulation drift
0.58 +/- 0.70mm
8.58 +/- 10.59%
In-vivo drift
2.7 +/- 1.0mm
12.08 +/- 2.09%
95% conf. intervals (mean +/- 1.96std)
41
Combining Edge Detection With SpeckleTracking for Cardiac Strain Assessment in
3D echocardiography
F. Orderud, G. Kiss, S. Langeland, E.
Remme, H. Torp, S.I. Rabben
Proceedings of the IEEE Ultrasonics
symposium (IUS) 2008
42
3D strain
• Approach:
– Combine edge-detection with
speckle-tracking.
– Edge-detection to align to
myocardium
– Speckle-tracking for material
deformations
• Advantages:
– Only track residual deformation,
after shape changes are
corrected for
– Smaller search-windows
– Less drift
43
Results
Combined edge + tracking
yields:
• Improved tracking
accuracy
• Improved processing time
Processing time:
Edge:
130ms
Edge+speckle: 68ms
(non-decimated data)
ES displacement errors
ES strain
Speckletracking
Speckle
+ edge
ES displacement vectors
44
Automatic coupled segmentation of endo- and
epicardial borders in 3D echocardiography
F. Orderud, G. Kiss, H. Torp
Proceedings of the IEEE Ultrasonics
symposium (IUS) 2008
45
LV mass
• Simultaneous
segmentation:
– Endocard LV model
– Epicard LV models
– Models share position, size
and orientation.
• Applications:
– Myocardial mass/muscle
volume
– Wall thickening analysis
10ms processing time/frame
46
LV mass
•
•
Dataset: 5 recordings of
high image quality.
Reference: Endo+epi
segmentation with
AutoLVQ at ED and ES
Results:
• Good correspondence
(<20ml difference) in 4 out
of 5 recordings.
47
Automatic Alignment of Standard Views in 3D
Echocardiograms Using Real-time Tracking
F. Orderud, H. Torp, S.I. Rabben
Proceedings of the SPIE Medical Imaging
conference 2009
48
Apical View Alignment
•
Combine LV model, with
“RV sail” and “LVOT
cylinder”
Extract:
• Long-axis and rotation
from landmarks on
models
• Generate 4CH, 2CH &
LAX views (assume fixed
angle between slices)
SAX
4CH
2CH
LAX
49
Short-axis Alignment
•
•
Dynamic long-axis vector
from LV
Equidistant short-axis slices
that compensate for out-ofplane motion
Basal SAX slice example:
in atrium
MV
50
Results
Experiment:
• Dataset of 35 recordings (>70%
of myocardium visible).
• Compared to manually
annotated landmarks from GE
AutoLVQ.
Mean +/- std. absolute position error for
apex and mitral valve points
Mean +/- std. absolute angle error around
long-axis for A4C, A2C and LAX views
Discussion:
• Use image analysis to improve
visualization.
• Does not depend on 100%
accurate segmentation.
51
Conclusions
52
Conclusions
• Extend Kalman-tracking
framework to 3D
• Extend the framework to
support:
– Different types of
deformable models
(ellipsoid, spline, activeshape, subdivision)
– Different image
measurement (edge,
speckle-tracking)
– Multiple simultaneous
models arranged in a
hierarchy (thick-wall,
alignment)
• Demonstrated feasibility
of:
–
–
–
–
LV volume (in-vivo).
LV mass (in-vivo)
LV strain (simulations)
Automatic view alignment
(in-vivo)
Conducted in/close to realtime in 3D echocardiograms.
53
Other applications
Technology is general,
and not limited to
either ultrasound or
cardiac imaging.
Bladder segmentation
Simultaneous LV + LA segmentation
Cardiac CT LV+RV segmentation
54
Future extensions
• Developments for 2D echo
– Auto segmentation for handheld scanner
• Scanning feedback
– Image quality scoring.
– Probe positioning feedback.
– Auto-positioning of color-flow
areas
• Algorithmic developments
– Simultaneous edge-detection
in all profiles
– Multi-resolution segmentation
(coarse to fine)
– Bidirectional tracking
– Distributed filtering
55
Thank you
56
Acknowledgements
• Dep. computer science
(NTNU) for financing my
PhD program
• PhD Stein Inge Rabben
(GE), for supervising my
work related to model-based
segmentation.
• Prof. Hans Torp (NTNU), for
supervising my work related
to cardiac ultrasound and
speckle-tracking
+ many more...