Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
AAPM 2003
Multimodality Imaging in
Radiation Oncology:
Marc L Kessler, PhD
The Big Picture
Patient
Imaging
Studies
Process
Images
Registration and Fusion
(Actual)
(Actual)
(Digitized)
(Digitized)
(Register,
(Register,
Segment)
Segment)
Marc L Kessler, PhD
The University of Michigan
Department of Radiation Oncology
Tx
Plan
DRRs
Radio
graphs
(Virtual)
(Virtual)
(Virtual)
(Virtual)
(Actual)
(Actual)
Marc L Kessler, PhD
Objectives
AAPM 2003
Ø What is image registration ?
Ø Why do we want to do it ?
AAPM 2003
Patient
(Virtual)
(Virtual)
…
Patient
+
Dose
(Actual)
(Actual)
Marc L Kessler, PhD
AAPM 2003
What is Image Fusion?
Multi-Sensor Data Fusion
imaging devices
Ø How do we do it ?
outputs of the devices
the mechanics !
integration and display
Marc L Kessler, PhD
AAPM 2003
Multi-Sensor Data Fusion
Marc L Kessler, PhD
AAPM 2003
Multi-Sensor Data Fusion
Golf
Courses
Galore!
This is
NOT New!
LANDSAT
VII
~~ 30
30 years
years old,
old, LANDSAT
LANDSAT II launched
launched 7/23/72
7/23/72
Marc L Kessler, PhD
1
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
AAPM 2003
Multi-Sensor Data Fusion
• Landsat MSS / TM
Marc L Kessler, PhD
AAPM 2003
Multi-Sensor Data Fusion
... over both space and time!
• Radar / Sonar
• Aerial Photographs
• Elevation Models
• Geophysical Data
Why ?
Marc L Kessler, PhD
MR Spin Echo
3D Scalar Data
AAPM 2003
FDG-PET
MR + PET
1984
CT + Dose
1993
Change!
Why ?
… patient models must
include functional and
physiologic information
as well as anatomic and
physical data.
Marc L Kessler, PhD
AAPM 2003
MR Spin Echo
FDG-PET
MR + PET
CT + Dose
… not just 3D scalars,
but 4D vectors which
include dose-to-date.
4D Vectors
Why ?
Marc L Kessler, PhD
Target
Target Volume
Volume Definition
Definition using
using FDG-PET
FDG-PET
AAPM 2003
Why ?
Marc L Kessler, PhD
AAPM 2003
Functional
Functional Imaging
Imaging using
using Magnetic
Magnetic Resonance
Resonance
choline
choline
?
creatine
creatine
citrate
citrate
C
P
?
ppm
ppm
S
X-ray CT
A
3.5
3.5
3.0
3.0
2.5
2.5
2.0
2.0
1.5
1.5
Localized
Localized spectroscopy
spectroscopy and
and MR
MR image
image data
data
FDG-PET
18
18
Marc L Kessler, PhD
Pickett
Pickett // UCSF
UCSF
2
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Why ?
Marc L Kessler, PhD
AAPM 2003
Why ?
Functional
Functional Imaging
Imaging using
using Magnetic
Magnetic Resonance
Resonance
Marc L Kessler, PhD
AAPM 2003
Repeat
Repeat imaging
imaging during
during treatment
treatment course
course
- silent word task -
CT
CT Images
Images before
before and
and after
after chemotherapy
chemotherapy
BOLD
BOLD -- Blood
Blood Oxygen
Oxygen Level
Level Dependent
Dependent Technique
Technique
Hamilton
Hamilton // U
U Chicago
Chicago
Why ?
Marc L Kessler, PhD
Wolfe
Wolfe // Michigan
Michigan
AAPM 2003
How ?
1
“… construct as complete
and accurate a model of
Marc
Marc LL Kessler,
Kessler, PhD
PhD
August
August 1
12
2,,2003
2003
Marc L Kessler, PhD
a2
?
Compute the geometric transformation
that relates the coordinate systems of
two datasets
2
Apply the computed transformation
to map information from one dataset
to another
data fusion
AAPM 2003
Marc L Kessler, PhD
Fusion
X
XBB =
=
b2
a3
Study A
rotate,
rotate, scale
scale
translate,
translate, ...
...
AA
tumor
tumor volumes
volumes
3D
3D doses,
doses, ...
...
b3
Study B
XB = F ( XA , { ß })
Study A
AAPM 2003
F ( XX ,, {{ ßß }})
b1
a1
AAPM 2003
registration
the patient as possible.”
Registration
Marc L Kessler, PhD
?
Study B
Computer
Computer graphics
graphics and
and image
image processing
processing
Marc L Kessler, PhD
3
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
What is F ?
AAPM 2003
Marc L Kessler, PhD
What is F ?
AAPM 2003
Affine Assumption
Ø Rigid / Affine
y= mx+b … in three dimensions
Ø Piecewise Rigid ( & Limited FOV)
Ø Full 3D / 4D Warping
xbb = A xAA + b
… up to 12 DOF
Otherwise
Ø Finite Element Models
Spatially variant function
Thin-plate spline
Marc L Kessler, PhD
Degrees of Freedom
AAPM 2003
Patient dependent
… lots of DOF
How ?
Marc L Kessler, PhD
AAPM 2003
Prospective
–– rotation
rotation
(( θθxx,, θθyy,, θθzz))
Ø reproduce imaging geometry exactly
–– translation
translation
(( ttxx,, ttyy,, ttzz ))
–– distortion
distortion
many
many DOF
DOF
Ø attach coordinate system to patient
Machine dependent
–– pixel
pixel size
size
–– slice
slice thickness
thickness
–– distortion
distortion
•• frames
frames // fiducials
fiducials
{ß}
Retrospective
(( ssxx,, ssyy ,, sszz ))
Ø patient intrinsic
many
many DOF
DOF
•• anatomy
anatomy // shape
shape // image
image intensities
intensities
Marc L Kessler, PhD
Prospective
AAPM 2003
Marc L Kessler, PhD
Prospective
AAPM 2003
… attach a
coordinate
system to
the patient!
ouch
F = Identity
X B = XA
… stereotactic radiosurgery
Marc L Kessler, PhD
4
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
Retrospective
CT
–
–
–
–
–
?
AAPM 2003
rotate
translate
scale
shear
warping
Marc L Kessler, PhD
Retrospective
AAPM 2003
1 Construct a metric that measures the
mismatch between two datasets
2 Apply a minimization algorithm to
determine the parameters (DOF)
that minimize this metric
MR
Surface-based
Intensity-based
Interactive or Automated
Marc L Kessler, PhD
Retrospective
Ø Point Matching
Least
Least Squares
Squares
Ø
Ø
AAPM 2003
Marc L Kessler, PhD
Surface-based
aa
Σ (( XX
BB
-- X
XAA )) 22
b
b
?
AAPM 2003
cc
Surface Matching
Chamfer
Chamfer Matching
Matching
min distance
distance 22
Σ min
Σ dii22
Ø Voxel Intensities
Mutual
Mutual Information
Information
--
p(A,B)
p(A,B)
p(A,B) log
log
Σ p(A,B)
p(A)
p(A) p(B)
p(B)
objects
misaligned
compute
mismatch
mismatch
minimized
Catallo
Catallo //U
U Mich
Mich
Marc L Kessler, PhD
Intensity-based
CT
AAPM 2003
AAPM 2003
H(IAA ,IBB) = H(IAA ) + H(IBB) - MI(I AA ,IBB)
MR
?
H(IAA )
Marc L Kessler, PhD
Mutual Information
H(IBB)
H(IAA ,IBB)
Mutual Information
Joint
Individual
Mutual
The
mutual information
image
Entropy
Entropiesof two
Information
datasets is a maximum when they are
geometrically registered … p(IAA, IBB)
MI(IAA,IBB) =
p(IAA, IBB) log22
p(IAA) p(IBB)
… MI can be used as a metric
Σ
‘48 Shannon - Bell Labs / ‘95 Viola - MIT
Marc L Kessler, PhD
5
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
AAPM 2003
Mutual Information
Yes, … go to next slide
H(X) = - EX [ log22 p(x)]
No, … skip past this section
Marc L Kessler, PhD
.75
.75
2
2
.50
.50
1
1
0
0
1
Σx p(x) log p(x)
2
2
Marc L Kessler, PhD
Mutual Information
1.0
1.0
Σ pp log 1p1p
3
3
.75
.75
2
2
.50
.50
.25
.25
1
1
.25
.25
0
0
0
0
0
0
ii
0
0
1
1
2
2
22
ii
3
3
11
11
11
11
+
+
+
+ 00 ** log
log22
+ 00 ** log
log22
+ 00 ** log
log22
11
00
00
00
=
=
Marc L Kessler, PhD
air
grayscale
fat
probability
frequency
AAPM 2003
p(I
p(X)
histogram
CT )
tissue
bone
intensity
intensity
3D CT
Grayscale Data
1
1
2
2
22
ii
3
3
11
11
1
1
.25
+
+
.25 ** log
log22
+ .25
.25 ** log
log22
+ .25
.25 ** log
log22 1 +
+ .25
.25 ** log
log22 1 =
=
.25
.25
.25
.25
.25
.25
.25
.25
binary
binary units
units =
= “bits”
“bits”
Mutual Information
ii
Histogram
Histogram
2D
2D image
image
0
0
AAPM 2003
Σ pp log 1p1p
0
0
Histogram
Histogram
2D
2D image
image
11 ** log
log22
=
AAPM 2003
Mutual Information
1.0
1.0
AAPM 2003
The entropy or information content of
an image with a frequency distribution
p(X), is defined by:
Should we really describe this?
3
3
Marc L Kessler, PhD
Mutual Information
2.0
2.0
222 = 4
Marc L Kessler, PhD
Mutual Information
AAPM 2003
The joint entropy or combined
information content of a pair of
images with image with a joint
frequency distribution p(X,Y), is:
1
Σy Σx p(x,y) log p(x,y)
H(X,Y) = -
2
2
Frequency
Distribution
Marc L Kessler, PhD
6
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
Mutual Information
AAPM 2003
Marc L Kessler, PhD
Mutual Information
AAPM 2003
H(X,Y) = H(X) + H(Y)
counts
ICT
reformatted
reformatted
CT
CT
H(X)
0
original
original MR
MR
n+1
n
256
IMR
256
2D joint intensity
histogram
Marc L Kessler, PhD
Mutual Information
AAPM 2003
H(I CT
,I MR) = H(I CT
) + H(I MR
)
CT MR
CT
MR
H(I CT
)
CT
Marc L Kessler, PhD
Mutual Information
Marc L Kessler, PhD
AAPM 2003
H(ICT
,IMR) = H(ICT
) + H(IMR
) - MI
CT MR
CT
MR
MI
H(IMR
MR)
AAPM 2003
H(ICT
,IMR) = H(ICT
) + H(IMR
)-?
CT MR
CT
MR
?
H(ICT
CT)
… if CT and MR are independent
H(ICT
CT)
… if X and Y are independent
H(I MR
)
MR
Mutual Information
H(Y)
H(IMR
MR)
CT and MR dependent
Marc L Kessler, PhD
Mutual Information
AAPM 2003
H(IAA ,IBB) = H(IAA ) + H(IBB) – MI
MI(IAA,IBB) =
Σ p(
IAA, IBB) log22
p(IAA, IBB)
p(IAA) p(IBB)
This is our metric!
CT and MR dependent
Marc L Kessler, PhD
7
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
p( I CT, I MR)
Not so
Aligned!
CT
IM
I
reformatted
reformatted
CT
CT
2D joint intensity
histogram
original
original MR
MR
Marc L Kessler, PhD
Mutual Information
MI = .62
CT
2D joint intensity
histogram
original
original MR
MR
AAPM 2003
H(IAA ,IBB) = H(IAA ) + H(IBB) - MI(I AA ,IBB)
p( I CT, I MR)
AAPM 2003
IM
I
reformatted
reformatted
CT
CT
Marc L Kessler, PhD
Mutual Information
MI = .99
R
Aligned!
AAPM 2003
R
Marc L Kessler, PhD
Mutual Information
Marc L Kessler, PhD
Mutual Information
AAPM 2003
- MI vs . transformation parameters
fixed
fixed
z
H(
H(IICT
)=
= 6.44
6.44
CT)
(bits)
(bits)
The mutual information of two image
datasets is a maximum when they are
geometrically registered …
x
… MI can be used as a metric
‘48 Shannon - Bell Labs / ‘95 Viola - MIT
1.0
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
-0.8
-1.0
AAPM 2003
0.8
Marc L Kessler, PhD
Mutual Information
Reference
Homologous
Marc L Kessler, PhD
Mutual Information
AAPM 2003
FDG PET
0.0
CT
CT
-1.0
-MI
-2.0
H(I
)
H(IPET
PET)
1.83
1.83
-3.0
-5.0
-6.0
PET
PET
X-ray CT
-4.0
zz rotation
rotation degrees
degrees
xx translation
translation cm
cm
-7.0
H(ICT)
3.98
Manipulate CT to
match FDG-PET
-- rotate
rotate // translate
translate --
parameter
parameter offset
offset
Marc L Kessler, PhD
8
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
Mutual Information
AAPM 2003
Marc L Kessler, PhD
AAPM 2003
Marc L Kessler, PhD
AAPM 2003
Mutual Information
#
# Iterations
Iterations
Starting
Starting MI
MI
150
150
-0.197
-0.197
Wall
Wall Clock
Clock
CT
CT
~1
~1 minutes
minutes
-MI
-MI
-0.197
-0.197
PET
PET
?? ?? ??
-0.479
-0.479
Marc L Kessler, PhD
In the Brain
AAPM 2003
Multimodality image registration in
the cranium is a solved problem!
Outside the Brain
Ø Rigid / Affine ( if we’re careful ! )
Ø Full 3D / 4D Warping
Ø Finite Element Models
Ø Piecewise Rigid ( & Limited FOV)
CT / MR
PET / CT
MR / PET
Marc L Kessler, PhD
Rigid Assumption ?
AAPM 2003
Reproduce patient orientation as closely as
possible using an immobilization device
Custom-molded
Thorax Board
Styrofoam cradle
Sinmed BV
Marc L Kessler, PhD
Rigid Assumption ?
AAPM 2003
Register CT data with PET emission data
FDG
FDG PET
PET
crop
crop spurious
spurious PET
PET data
data
Marc L Kessler, PhD
X-ray
X-ray CT
CT
window/level
window/level datasets
datasets
9
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
Rigid Assumption ?
AAPM 2003
Visual validation of registration
Marc L Kessler, PhD
PET in cradle, CT not
Auto-segmented
Auto-segmented boundaries
boundaries from
from CT
CT mapped
mapped to
to PET
PET
AAPM 2003
What about Warping ?
PET
CT
Allow Spatial Variant Transformations
Thin-Plate Splines, Finite Element Models, etc
Meyer
Meyer // UM
UM
Marc L Kessler, PhD
AAPM 2003
What about Warping ?
Marc L Kessler, PhD
AAPM 2003
Finite Element Modeling
Exhale
Exhale
Inhale
Inhale
Take into account physical
tissue properties
Thin-plate Spline Transformation and
Mutual Information Metric
Brock
Brock // UM
UM
Meyer
Meyer // UM
UM
Marc L Kessler, PhD
The Future ?
AAPM 2003
Marc L Kessler, PhD
Piecewise Rigid
AAPM 2003
Premise
A simple rotate- translate model can
work over a limited spatial extent.
Finite Element Family
Global
Global Rigid
Rigid
Too
Too Little
Little ??
Marc L Kessler, PhD
Piecewise
Piecewise Rigid
Rigid
Full
Full Warping
Warping
Too
Too Much
Much ??
10
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
Piecewise Rigid
AAPM 2003
Marc L Kessler, PhD
AAPM 2003
Piecewise Rigid Assumption
Partition data into 1 or more sub-regions
Region
Region
1
1
Region
Region
2
2
CT
Geometric
Geometric primitives
primitives
Anatomic
Anatomic structures
structures
Marc L Kessler, PhD
AAPM 2003
Piecewise Rigid Assumption
(( Therapy
Therapy ))
MR
(( Diagnostic
Diagnostic ))
Regions
Regions
Of
Of Interest
Interest
Marc L Kessler, PhD
AAPM 2003
Marc L Kessler, PhD
AAPM 2003
Limited Field-of-View
MR data
discarded
curved
curved
flat
flat
MR
CT
Region 1
Registration
Marc L Kessler, PhD
Limited Field-of-View
particularly
particularly
poor
poor
MR
(( Diagnostic
Diagnostic ))
radioactive
seeds
Region 2
Registration
AAPM 2003
Limited Field-of-View
CT
(( Therapy
Therapy ))
Marc L Kessler, PhD
11
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
AAPM 2003
The Big Picture
Patient
Imaging
Studies
Process
Images
(Actual)
(Actual)
(Digitized)
(Digitized)
(Register,
(Register,
Segment)
Segment)
Tx
Plan
DRRs
Radio
graphs
(Virtual)
(Virtual)
(Virtual)
(Virtual)
(Actual)
(Actual)
Patient
(Virtual)
(Virtual)
…
Marc L Kessler, PhD
Does what we just saw mean that
we could use soft tissue imaging to
localize and position targets ?
Patient
+
Dose
Tx
Plan
DRRs
Radio
graphs
(Actual)
(Actual)
(Virtual)
(Virtual)
(Virtual)
(Virtual)
(Actual)
(Actual)
Marc L Kessler, PhD
Piecewise Rigid
AAPM 2003
Anatomy outside sub-regions ?
Patient
+
Dose
…
(Actual)
(Actual)
Marc L Kessler, PhD
Piecewise Rigid
AAPM 2003
Distance-weighted interpolation
Ø Ignore
A
A
OK in many or most cases!
Ø Interpolate ?
A
B
B
B
distance weighting, …
Rigid objects
Marc L Kessler, PhD
Piecewise Rigid
AAPM 2003
Distance-weighted interpolation
3D Distance fields
Marc L Kessler, PhD
How …
AAPM 2003
do we know these algorithms work?
A
A
Ø build phantoms and test them
A
we can know the truth!
Ø provide tools to examine results
B
B
B
Rotate objects
AAPM 2003
The Big Picture
we don’t know the truth!
Distance fields
Marc L Kessler, PhD
12
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
AAPM 2003
Multimodality Phantoms
Marc L Kessler, PhD
AAPM 2003
Multimodality Phantoms
CT
MR
1986
1986
Marc L Kessler, PhD
AAPM 2003
van
van Herk
Herk // NKI
NKI
Marc L Kessler, PhD
AAPM 2003
Registration Validation
Registration Validation
- Visual Analysis -
- Visual Analysis -
Original
MR
“Magic Window”
Reformatted
CT
Image Compare
Marc L Kessler, PhD
AAPM 2003
Registration Validation
- Visual Analysis -
Display datasets as “color gels”
Marc L Kessler, PhD
Clinical Examples
AAPM 2003
Target Volume Definition
• Magnetic Resonance Imaging
- soft tissue contrast
• Positron Emission Tomography
Native Outlines
Mapped Outlines
- tumor metabolism
Marc L Kessler, PhD
13
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
AAPM 2003
Marc L Kessler, PhD
AAPM 2003
Target Volume Definition
w/ Magnetic Resonance Imaging
w/ Magnetic Resonance Imaging
MR axial
CT axial
% Volume
Target Volume Definition
MR coronal
Dose-Volume
Histogram
Composite
% Dose
( Logical OR )
Marc L Kessler, PhD
Dose Mapping
AAPM 2003
Marc L Kessler, PhD
AAPM 2003
Target Volume Definition
w/ Positron Emission Tomography
?
P
C
S
A
?
Compute
Compute using
using CT
CT
X-ray CT data
Display
Display over
over MR
MR
Marc L Kessler, PhD
AAPM 2003
FDG-PET Data
Marc L Kessler, PhD
AAPM 2003
Target Volume Definition
Target Volume Definition
w/ Positron Emission Tomography
w/ Positron Emission Tomography
B
PET > CT
CT
CT >
> PET
PET
A
A
PET > CT
CT > PET
front
left
PET - Pink surface
back
right
CT - Blue contours
Green
Green surface
surface -- CT
CT cord
cord
front
left
B
PET - Pink surface
CT - Blue contours
Marc L Kessler, PhD
14
Multimodality Imaging in Radiation Oncology:
Registration and Fusion
Marc L Kessler, PhD
Summary
AAPM 2003
Marc L Kessler, PhD
Summary
AAPM 2003
Taxonomy of Registration Process
“… construct as complete
and accurate a model of
the patient as possible.”
Marc
Marc LL Kessler,
Kessler, PhD
PhD
August
August 12,2003
12,2003
Marc L Kessler, PhD
Summary
Surface-based
Voxel-based
Interactive
Automated
Affine
Warping
AAPM 2003
Marc L Kessler, PhD
Challenges
AAPM 2003
Ø Techniques are now available to register
3D image data from different modalities
Ø Measure improvement in therapy realized by
incorporation of additional image information
Ø Registered data can be fused to create more
Ø Correlation of post-therapy radiologic findings
complete models of the patients
Ø Accuracy on the order of image resolutions
reported ...
with pre-therapy computed dose distributions
Ø Extend techniques to better handle non-rigid
organ motion … morphometrics
Marc L Kessler, PhD
15