th 2008
Tuesday, July 29th
th 2008
Tuesday, July 29th
8:30 AM – 9:25 AM
8:30 AM – 9:25 AM
By 9:25 AM, you will be able to …
Deformable Image Registration:
Methods and Endpoints
Marc L Kessler, PhD
The University of Michigan
9 Describe the basic mechanics (recipes ) of image
registration techniques used in commercial &
research planning and delivery systems
9 Describe the different techniques used to combine,
display & interact with multimodality and 4D image
data
9 Understand the clinical use & application of these
techniques for treatment planning, treatment
delivery & plan adaptation
th 2008
Tuesday, July 29th
The Recipe
8:30 AM – 9:25 AM
Recipe for Image Registration
2 Image Volumes
What’s the recipe for registering
1 Transformation Model
and fusing 3D & 4D image data?
1 Registration Metric
1 Optimization Engine
What’s the cake ?
n Evaluation Tools
AAPM
~ 50 ~
Recipe for
Recipe
for Image
ImageRegistration
Registration
2 Image Volumes
Volume 1
1 Transformation Model
1 Registration Metric
1 Optimization Engine
n Evaluation Tools
Recipe for
Recipe
for Image
ImageRegistration
Registration
stationary
Registration
Metric
Volume 2’
transformed
Volume 2
moving
Geometric
Transformation
#
Optimization
Engine
{ß}
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
1
Recipe for
Optimization
Recipe
for Plan
Image
Registration
Criteria
Plan Quality
Metric
The Cake !
CT
#
Dose
Distribution
MR
Optimization
Engine
NM
Physics
Dose
Calculation
Model
{ß}
US
The Cake !
Tx Plan
3D Dose
Portal
Images
Generalized
Adaptable
Patient
Model
CBCT
3D Dose
Day n
4D
CBCT
1…n
1…n
US
The Cake !
PET
PET
• anatomic
• physiologic
MR
• geometric
• density
CT
MR
CT
CT+ MR + NM + Dose (τ)
(τ)
Scalar Data
4-D Vectors
Scalar Data
4-D Vectors
Pouliot
Pouliot // UCSF
UCSF
The Cake !
The Cake !
Map structure outlines from
one image volume to another
Map and combine doses from
multiply treatment fractions
Fraction 1
=
Fraction 2
Δ Dose
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
2
Recipe for
Recipe
for Image
ImageRegistration
Registration
The Cake !
2 Image Volumes
Dose – Function Analysis
Chose 2 from the plethora of
imaging volumes we now have
available in radiation therapy!
Simulation
Simulation
Planning CT
SPECT Imaging
Gregoire
Gregoire // St-Luc
St-Luc
Available Image Volumes
Balter
Balter // UM
UM
Available Image Volumes
?
?
Physics
X-ray
CT
Anatomy
MRI
Physiology
Nuc Med
Motion Information
…multiple modalities
…4-D imaging
Available Image Volumes
Normal
Normal
Dawson
Dawson // PMH
PMH
Available Image Volumes
MR !
Target
Target
?
Varian
OBI™
?
…repeat imaging during therapy
Elekta
Synergy™
Siemens
PRIMATOM™
TomoTherapy
Hi-Art™
ViewRay
Renaissance™
Resonant
Restitu™
…at the treatment unit too!
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
3
Recipe
Recipefor
forImage
ImageRegistration
Registration
2 Image Volumes
Prostate Example
Can you tell what is different in the 2 volumes?
Depending on the application,
choose either entire volumes
or the relevant sub-volumes
Volume 1
Prostate Example
Bones aligned, prostate region not aligned
Entire Volume
Volume 2
Prostate Example
Bones ignored, prostate region aligned
Sub volumes
Prostate Example
Liver Example
MR data
discarded
?
curved
curved
flat
flat
MR
CT
radioactive
seeds
anatomic-based
cropping
ignore anatomy
outside liver
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
4
Liver Example
The Past / Present
Registration
Segmentation
ignore anatomy
outside liver
anatomic-based
cropping
The Present / Future
Treatment Delivery Example
Registration
Segmentation
Rigid assumption used
for regional registration
Pick Your Battles Wisely!
Treatment Delivery Example
Sonke
Sonke // NKI
NKI
Registration at Delivery
Regional
registration
of serial
CBCT scans
and TP CT
Oops!
6 DOF
Choose your battles wisely!
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
5
Recipe
Recipefor
forImage
ImageRegistration
Registration
Transformation Model
1 Transformation Model
X2
X1
Depending on the input data
T
and clinical situation, chose
a transformation model
Volume 2
Recipe
Recipefor
forImage
ImageRegistration
Registration
1 Transformation Model
X1 =
Volume 1
Degrees of Freedom {
PET/CT
MR - CT
ß}
4D CT
T ( X2 , { ß })
Choose the flavor of
T
and
the set of parameters {
ß}
Available Flavors of T
0?
3 to 6
3xN
None ?
Few
Many
Available Flavors of T
Affine Assumption
¾ Rigid / Affine
… or regional rigid / affine
¾ Full 3D / 4D Deformation
Parametric transformations
Free-form transformations
y = mx+b … in three dimensions
x11 = A x22 + b
Otherwise
rotation (3)
… up to 12 DOF
translation (3)
scale (3)
Spatially
variant transformations
shear parametric
(3)
Various
& free form models
… up to 3 x N !
… lots of degrees of freedom!
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
6
www.gnome.org
www.gnome.org
Affine Transformations
Volume 2
www.gnome.org
www.gnome.org
Affine Transformations
Volume 2
A
A Square
Square
Volume 1
6 DOF
A
A Square
Square
Volume 1
Rotation
Rotation
Translation
Translation
Scaling
Scaling
Shearing
Shearing
Rotation
Rotation
Translation
Translation
Scaling
Scaling
Shearing
Shearing
3
3
3
3
3
3
3
3
Parallel lines stay parallel!
Parallel lines stay parallel!
Munro
Munro // Varian
Varian
Affine Transformations
www.gnome.org
www.gnome.org
Affine Transformations
3 or 4 DOF
Rotation
Rotation
CT
Translation
Translation
Scaling
Scaling
Shearing
Shearing
CT
CBCT
CBCT
Parallel lines stay parallel!
non- Affine
Transformations
Volume 1
Volume 2
Parallel lines don’t stay parallel!
non- Affine
Transformations
Volume 1
Volume 2
X1 = T ( X2 , { ß })
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
7
non- Affine
Transformations
Study A
Study B
Available Flavors of T
Parametric Models
Freeform Models
Transformation parameters
to apply to a particular voxel
depends on the location of
the voxel (spatially variant) !
X1 = T ( X2 , { ß(X2) })
Available Flavors of T
Spatially variant transformations
¾ Parametric
Brock / PMH
Warp Space /
… Drag Objects
Warp Objects /
… Drag Space
Available Flavors of T
Spatially variant transformations
¾ Free Form
Thin-plate Splines
Intensity Flow Methods
Global Deformations
Fluid Mechanics-like
B-Splines
Finite Element Models
Local Deformation
Account for Physical Properties
Thin-Plate Splines
Liver
Liver
Bladder
Bladder
B-Spline Deformations
Each have some distinct properties
¾ B-Splines
X2 = X1 + ΔX
= X1 + Σ wi·Β(X-ki)
weights
nn
T ( P ) = a00 + a xx x + a yy y + a zz z + ∑ wiiU ( P − Pii )
basis function
( … parameters! )
ii==11
affine
warping
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
8
B-Spline Deformations
B-Spline Deformations
Construct overall function T from
a weighted sum of local deformations
Each have some distinct properties
¾ B-Splines
1-D Example
ΔX
w
w77
ΔX
w
w77
knots
knots
knots
kk11 kk22 kk33 kk44 kk55 kk66 kk77 kk88 kk99 kk10
k 11
10 k11
k
k 11
k11 k
k33 k
k55 k
k77 k
k99 k
k10
k22 k
k44 k
k66 k
k88 k
10 k11
X22 = X11 + ΔX = X11 + Σ wii·Β(X-kii)
X
X22 = X11 + ΔX = X11 + Σ wii·Β(X-kii)
parameters!
B-Spline Deformations
B-Splines
Construct overall function T from
a weighted sum of local deformations
Construct overall function from
a weighted sum of local deformations
ΔX
Deformlets?
ΔX
w
w77
knots
w
w77
k
k 11
k11 k
k33 k
k55 k
k77 k
k99 k
k10
k22 k
k44 k
k66 k
k88 k
10 k11
X
X22 = X11 + ΔX = X11 + Σ wii·Β(X-kii)
parameters!
B-Spline Transformation Model
3 weights / knots !
knots
k
k 11
k11 k
k33 k
k55 k
k77 k
k99 k
k10
k22 k
k44 k
k66 k
k88 k
10 k11
X
X22 = X11 + ΔX = X11 + Σ wii·Β(X-kii)
This seems a lot like intensity modulation!
Multiresolution Deformations
Divide and Conquer
60
60 xx 60
60 xx 48
48 mm
mm
3-D Grid of Knots
X
4
4 xx 4
4 xx 3
3 mm
mm
Coarse
Fine
Knot Spacing
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
9
Recipe
Recipefor
forImage
ImageRegistration
Registration
1 Registration Metric
Registration Metrics
?
Choose a registration metric
?
depending on the type of
imaging data involved
Geometry-based
Geometry-Based Metrics
¾ Point Matching
Least Squares
Σ( X
22
Point Matching
- X’11 ) 2
Σ( X
22 – X’11
¾ Surface Matching
Chamfer Matching
Intensity-based
Σ min distance 2
)2
Volume 2
Volume 1
Define corresponding anatomic points
Surface Matching
a
?
b
Catallo
Catallo // UM
UM
Sonke
Sonke // NKI
NKI
Registration at Delivery
c
Regional
registration
of serial
CBCT scans
and TP CT
dii
Σ di 2
objects
misaligned
compute
mismatch
mismatch
minimized
Define corresponding anatomic surfaces
6 DOF
Choose your battles wisely!
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
10
Intensity-Based Metrics
Sum of Squared Differences
… subtract one image from the other
¾ Mono-modality
Sum of Squared Diff.
¾ Multimodality Data
Mutual Information
Σ(I
22
- I11 ) 2
CT 22
p(A,B)
p(A,B)
p(A,B) log
log
Σ p(A,B)
p(A)
p(A) p(B)
p(B)
-
I CT 22
CT 11
I CT 11
=
Difference
Σ (I CT
1
1
-I CT 22 )22
Sum
Sum of
of the
the Squares
Squares of
of
the
the Differences
Differences
Individual
Individual Intensity
Intensity
Distributions
Distributions
Sum of Squared Differences
Sum of Squared Differences
… subtract one image from the other
… subtract one image from the other
=
I CT 22
I CT 11
Individual
Individual Intensity
Intensity
Distributions
Distributions
Difference
-
CT
I CT
Zero
MR
I MR
Individual
Individual Intensity
Intensity
Distributions
Distributions
Registered
Mutual Information
=
Difference
Not Zero
This
This doesn’t
doesn’t usually
usually
make
make much
much sense
sense
Mutual Information
… using an information theory-based approach
H(I1,I2) = H(I1) + H(I2) - MI(I1,I2)
CT
MR
H(ICT) + H(IMR) = H(ICT,IMR)
Individual Information
Contents
The mutual information is the
CT, MR
Joint Information
Content
information that is common to
?
both image volumes!
‘48 Shannon - Bell Labs / ‘95 Viola - MIT
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
11
Mutual Information
Mutual Information
MI(I1,I2) = H(I1) + H(I2) - H(I1,I2)
MI(I1,I2) =
Σ p(I , I ) log
1
1
2
2
2
p(I11, I22)
p(I11) p(I22)
The mutual information is the
The mutual information of two image
information that is common to
volumes is a maximum when they are
both image volumes!
geometrically registered …
‘48 Shannon - Bell Labs / ‘95 Viola - MIT
‘48 Shannon - Bell Labs / ‘95 Viola - MIT
Mutual Information
Mutual Information
original MR
How About An Example?
Rotate - Translate
CT
MI = .62
R
I
I
CT
2D joint intensity
histogram
Transformation
p(ICT, IMR)
reformatted
CT
I
CT
R
I
reformatted
Not so
Aligned!
MI = .99
M
p(ICT, IMR)
M
Aligned!
original MR
2D joint intensity
histogram
How About An Example?
PET
PET
Registration Metric
CT
Mutual Information
CT
Optimizer
Simplex Algorithm
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
12
How About An Example?
How About An Example?
PET
PET
CT
CT
Recipe
Recipefor
forImage
ImageRegistration
Registration
2 Image Volumes
Volume 1
stationary
1 Transformation Model
transformed
1 Optimization Engine
Volume 2
n Evaluation Tools
moving
Recipe for
Optimization
Recipe
for Plan
Image
Registration
Plan Quality
Metric
Dose
Distribution
Physics
Model
Dose
Calculation
Registration
Metric
Volume 2’
1 Registration Metric
Criteria
Recipe
Recipefor
forImage
ImageRegistration
Registration
Geometric
Transformation
#
Optimization
Engine
{ß}
Interactive Registration
#
9 Translate
9 Rotate
Optimization
Engine
? Deform
{ß}
Provide tools to transform and visualize!
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
13
Automated Registration
Multiresolution Deformations
Divide and Conquer
60
60 xx 60
60 xx 48
48 mm
mm
Rigid first , ... then B-Spline deformation
Multiresolution Deformations
Coarse
Fine
Knot Spacing
Multiresolution B-Splines
Registration Metric vs. Iteration
ABC CT Example
2.5
2.5
Registration Metric
4
4 xx 4
4 xx 3
3 mm
mm
Change
Change in
in
knot
knot spacing
spacing
2.4
2.4
Low Res
2.3
2.3
2.2
2.2
2.1
2.1
High Res
2.0
2.0
1.9
1.9
Exhale State
1.8
1.8
0
0
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
Inhale State
180
180
Iteration
Iteration Number
Number
Multiresolution B-Splines
Multiresolution B-Splines
ABC CT Example
Multiphasic CT Data
Exhale State
Inhale State
deformed
Exhale State
Inhale State
deformed
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
14
Ruan
Ruan // UM
UM
Multiresolution B-Splines
We Are Not Really Splines !
Multiphasic CT Data
9
9
No “stiffness”
information
Exhale State
Inhale State
deformed
Add Some Physics?
Extracted
Ribcage
Exhale
Deform Inhale
Spatially Variant Stiffness
Etotal = Esimilarity + α Estiffness
intensity similarity metric
tissue-dependent regularization
Evol =
∫
wc(x) |det JT(x) – 1|2 dx
Ruan
Ruan // UM
UM
“Stiffness” Weighting
Using “Prior” Information
wc(x)
using “stiffness”
information
Extracted
Ribcage
Exhale
Deform Inhale
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
15
Balter
Balter // UM
UM
Tissue Sliding
Tissue Sliding
Balter
Balter // UM
UM
Deal with different organs individually?
Deal with different organs individually?
Segmentation + Registration
Registration / Segmentation
No masking
Masking
Registration
Segmentation
Ribs driven by large
lung deformations
Ribs not affected
by lung registration
Brock
Brock // UM
UM
Finite Element Modeling
Brock
Brock // PMH
PMH
Finite Element Modeling
Exhale
Exhale
Inhale
Inhale
Take into account physical
tissue properties (directly)
… thorough segmentation is necessary
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
16
The Future ?
Are We Already There?
Several independent
products are there
or almost there!
Family of
Generalized,
Customizable,
Patient Models
II have
have no
no commercial
commercial interest
interest in
in any
any of
of these
these company
company
olivier.commowick.org
olivier.commowick.org
Meyer/UM
Meyer/UM
…from Atlas to Individual
thesis_work/img1.png
without
without
with
with
9
Segment /register / average
www.mimvista.com
www.mimvista.com
…from Atlas to Individual
II have
have no
no commercial
commercial interest
interest in
in this
this company!
company!
Segment using atlas
Thompson
Thompson // UCLA
UCLA
…from Individuals to Atlas
Brain
Brain Mapping:
Mapping: The
The Disorders
Disorders,, Academic
Academic Press,
Press, 1999
1999
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
17
Recipe for
Recipe
for Image
ImageRegistration
Registration
Let’s Recap
9 The geometric transformation T
Affine / not - Affine
Volume 1
stationary
Registration
Metric
Volume 2’
9 The Registration Metric
transformed
Geometry / Intensity
9 Optimization Engine
Volume 2
Interactive / Automated
moving
Recipe for
Recipe
for Image
ImageRegistration
Registration
2 Image Volumes
#
Optimization
Engine
Geometric
Transformation
{ß}
Evaluation Tools
How do we know how well these
registration methods perform?
1 Transformation Model
¾ build phantoms and test them
1 Registration Metric
we can know the truth!
1 Optimization Engine
¾ provide tools to examine results
we don’t know the truth!
n Evaluation Tools
Circa
Circa 1986
1986
Multimodality Phantoms
4D Phantoms
Kashani
Kashani // UM
UM
CT
MR
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
18
Evaluation Tools
Visualization Tools
Evaluation Tools
Numerical Tools
¾ Color gel or wash
overlay
¾ Split /dual screen
displays
Point
Description
1
2
3
4
5
6
2nd branch of bronchial tree
3rd branch of bronchial tree
4th branch of bronchial tree
Vessel bifurcation 1
Vessel bifurcation 2
Vessel bifurcation 3
X
-5.37
-5.73
-6.50
-8.12
-8.06
-10.69
Study
Study A
A
Exhale
Y
0.98
2.12
2.77
3.37
-1.95
2.47
*
Inhale
X
Y
-4.62
-0.22
A
A
A
A 0.74A
A
-5.40
-6.24
0.80
-8.12
1.40
-7.67
-3.20
Study
Study
B 1.16
-10.78B
Z
-3.42
-5.42
-8.42
-9.92
-4.42
0.58
(x , y , z )
All numbers in centimeters
all values in cm.
¾ Anatomic boundary
overlay!
Exhale' ( w/ TPS alignment )
-4.71
-0.47
-3.36
-5.35
0.58
-5.83
-6.27
0.69
-9.51
-8.19
0.91
-11.60
-7.27
-2.83
-3.63
-10.85
0.87
1.24
σ
AAPM Task Group 132
Use of Image Registration and Data
Fusion Algorithms and Techniques in
Radiotherapy
ΔX
-0.09
0.05
B
B
-0.03
-0.07
0.40
-0.07
0.26
2 Image Volumes
1 Transformation Model
1 Optimization Engine
Caution: The Full Recipe?
0.29
Recipe for
Recipe
for Image
ImageRegistration
Registration
¾ Issues related to acceptance testing
and quality assurance for image
registration and fusion
Exhale' - Inhale
ΔY
ΔZ
-0.25
-0.44
0.09
B
B -0.16B
B
-0.11
-0.09
-0.49
-0.18
0.37
0.29
-0.29
0.16
0.19
1 Registration Metric
C o o n!
So
*
(x , y , z )
¾ Methods to assess the accuracy
g of
in
image registration andmfusion
o
Z
-2.92
-5.92
-9.42
-11.42
-3.92
1.08
n Evaluation Tools
Dose Mapping?
Dealing with volume elements that may:
9 deformation
9 weight loss
9 resection
… not just
deformations!
9 shrinkage
9 Δ vascular
change shape / appear / disappear
… need proper spatial re-sampling
don’t necessarily add in a linear fashion
… need some sort of radiobiology
exist in homogenous intensity regions
… hard to evaluate registration
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
19
Opportunities & Challenges
What Now ?
Flair
Flair
T
T22
T
T11
More than just mechanics!
Gd
Gd
Diff
Diff
MR volumes mapped to CT study
Summary
Product Comparison
¾ Tools are now available to register and
integrate image, anatomy & dose for
both Tx planning and Tx delivery
www.ITNonline.net
¾ These tools can be used to help build
better models of the patient and to
help customize and adapt therapy
¾ Work towards more standard and
robust tools and validations methods
(for non-rigid) situations continues
th
Wednesday , July 30th
Don’t try this at home!
4:00 PM – 5:30 PM
Joint Imaging -Therapy Symposium
Image Processing and Analysis
for Radiotherapy Guidance
www.itk.org
Deformable Image Registration
Methods and Clinical Endpoints
Marc L Kessler, PhD
20