Circa
Circa 1986
1986
In The Beginning!
Imaging in Therapy:
Image Registration &
Data Fusion Algorithms
*
*
Marc L Kessler, PhD
Department of Radiation Oncology
The University of Michigan, Ann Arbor
*
* Disclosure
Disclosure online
online
And here!
Look here!
Acknowledgements!
Many wonderful people have contributed
material for this presentation !
Outline
¾ What is image registration ?
¾ Why do we want to do it ?
¾ How do we do it ?
the mechanics !
Data Handling for IGRT
Imaging
Studies
Process
Images
(Actual)
(Digitized)
(Segment,
Register)
Tx
Plan
DRRs
Radio
graphs
(Virtual)
(Virtual)
(Actual)
Patient
Patient
(Virtual)
Patient
+
…
Dose
(Actual)
Data Handling for IGRT
Imaging
Studies
Process
Images
(Actual)
(Digitized)
(Segment,
Register)
Tx
Plan
DRRs
Radio
graphs
(Virtual)
(Virtual)
(Actual)
Patient
Patient
(Virtual)
Patient
+
…
Dose
(Actual)
Data Handling for IGRT
Volumetric
data
Imaging
Patient
acquired Studies
at the
treatment unit!
(Actual)
(Digitized)
Tx
Plan
DRRs
(Virtual)
Process
Images
(Segment,
Register)
On-line and
(Virtual)
Patient
(Virtual)
Patient
+
…
off-line
Dose
Radio
graphs
(Actual)
(Actual)
Data Handling for IGRT
Once you look,
… you will see!
Once you see,
… you will have to act !
maybe
Data Handling for IGRT
!
?
… what do we do now?
Data Handling for IGRT
adaptive
Imaging
Planning
Delivery
Imaging
…
bi-directional
… flow
betterbecomes
bulk up on
DICOM too!
Data Handling for IGRT
Txx Planning
CT
MRI
cor,sag,axial
cor,sag,axial
NucMed
TBD
Tx Plan
3D Dose
“Adapting”
Patient
Model
3D Dose
Day n
Cone Beam
1…n
Portal
Images
4D
Cone Beam
Marker
Locations
How ?
To get the plethora of data to all
groove together we need to know
the geometric transformations
F
that relate the coordinates of the
different imaging studies
How ?
1
Compute the geometric transformation
that relates the coordinate systems of
two datasets
registration
2
Apply the computed transformation
to map information from one dataset
to another
data fusion
How ?
Reference
Image
Moving
Image
Data Fusion
Image Registration
Image Registration
a2
?
b2
b1
a1
a3
Study A
rotate, scale
translate,
deform ...
b3
Study B
XB = F ( XA , { ß })
Data Fusion
XBB =
F ( XAA , { ß })
tumor volumes
3D doses, ...
Study A
?
Study B
Computer graphics and image processing
Degrees of Freedom
Patient dependent
– rotation
( θxx , θyy , θzz )
– translation
( txx , tyy , tzz )
– distortion
many DOF
Machine dependent
– pixel size
– slice thickness
– distortion
( sxx , syy , szz )
many DOF
{ß}
What is F ?
¾ Rigid / Affine
¾ Piecewise Rigid / Affine
… limited field-of-view
¾ Full 3D / 4D Deformation
Parametric models
Free-form models
What is F ?
Affine Assumption
y = mx+b … in three dimensions
xB = A xA + b
Otherwise
rotation (3)
… up to 12 DOF
translation (3)
scale (3)
Spatially
variant function
shear
(3)
Various splines and free form models
… lots of degrees of freedom!
www.gnome.org
www.gnome.org
Affine Transformations
Parallel lines stay parallel!
non- Affine
Transformations
Parallel lines don’t stay parallel!
non- Affine
Transformations
XB = F ( XA , { ß })
non- Affine
Transformations
XB = F ( XA , { ß(XA) })
non- Affine
Transformations
Transformation parameters
to apply to a voxel depends
on the location of the voxel
… up to 3N parameters
XB = F ( XA , { ß(XA) })
DICOM 3 Parts 3 & 17
DICOM handles only up to affine
... (and most Tx planning systems)
New DICOM Objects ?
OK Marc
New DICOM Objects ?
No Marc
How ?
Prospective
¾ reproduce imaging geometry exactly
¾ attach coordinate system to patient
• frames / fiducials
Retrospective
¾ patient intrinsic
• anatomy / shape / image intensities
Prospective
PET / CT Hybrid
GE Discovery LS
F = Identity
XB = XA
Prospective
… attach a
coordinate
system to
the patient!
ouch
… Stereotactic Radiosurgery
Retrospective
CT
?
– rotate
– translate
– scale
– warp?
MR
Geometry-Based
Intensity-based
Interactive or Automated
Automated Registration
1 Construct a metric that measures the
mismatch (or similarity) between a
pair of datasets
2 Apply a minimization algorithm to
determine the parameters (DOF) that
minimize (maximize) this metric
Automated Optimization
Twiddle
Parameters
Evaluate
Metric
(DOF)
(Cost)
Registration Metrics
?
?
Geometry-based
Intensity-based
Retrospective
¾ Point Matching
Least Squares
¾
Surface Matching
Chamfer Matching
¾ Voxel Intensities
Sum of squares diff
Σ(X
B
B
- XAA
) 22
Σ min distance 22
Σ
( IBB - IAA ) 22
Mono-modality
Retrospective
¾ Point Matching
Least Squares
¾
Surface Matching
Chamfer Matching
Σ(X
B
B
) 22
Σ min distance 22
¾ Voxel Intensities
Mutual Information
- XAA
-
Σ
p(A,B) log
p(A,B)
p(A) p(B)
Multi-modality
Catallo
Catallo // UM
UM
Geometry-based
… using an extracted anatomic surfaces
a
?
b
c
Σ di 2
objects
misaligned
compute
mismatch
mismatch
minimized
Intensity-based
… using an information theory-based approach
CT
MR
?
H(ICT)
H(IMR)
Individual Information
Content
H(ICT,IMR)
Joint Information
Content
Information Theory
H(IA,IB) = H(IA) + H(IB) - MI(IA,IB)
Joint
Entropy
MI(IA,IB) =
Individual
Entropies
Mutual
Information
Σ p(I , I ) log
A
A
B
B
2
p(IAA, IBB)
p(IAA) p(IBB)
These are just intensity histograms!
Information Theory
H(IA,IB) = H(IA) + H(IB) - MI(IA,IB)
The mutual information of two image
datasets is a maximum when they are
geometrically registered …
… MI can be used as a metric
‘48 Shannon - Bell Labs / ‘95 Viola - MIT
Mutual Information
I
reformatted
I
CT
CT
MI = .99
M
R
Aligned!
p(ICT, IMR)
original MR
2D joint intensity
histogram
Mutual Information
p(ICT, IMR)
I
reformatted
I
CT
CT
MI = .62
M
R
Not so
Aligned!
original MR
2D joint intensity
histogram
Automated Optimization
Rotate,
Translate,
Evaluate
MI
Deform?
Metric
Automated Registration
MR
CT
Maximize Mutual Information
In the Brain
Multimodality image registration in
the cranium is a solved problem!
CT / MR
PET / CT
MR / PET
Outside the Brain ?
There is a need to better handle data
from other regions of the body!
Prostate
Liver
Spine
Outside the Brain ?
Rotate-translate and MI can still be used
effectively, … over a limited field-of-view
geometric
anatomic
piecewise
Region
Region 11
Region
Region 22
Prostate Example
MR data
discarded
curved
curved
flat
flat
MR
CT
radioactive
seeds
Prostate Example
Can you tell what is different in the 2 images?
Prostate Example
Bones aligned, prostate region not aligned
No Cropping
The “answer” depends on the region defined!
Prostate Example
Bones ignored, prostate region aligned
Cropping
The “answer” depends on the region defined!
Liver Example
Dawson
Dawson // PMH
PMH
Most of the motion of
the liver seems to be
rigid or affine!
… some deformation
does occur though.
Liver Example
anatomic-based
cropping
ignore anatomy
outside liver
Liver Example
anatomic-based
cropping
ignore anatomy
outside liver
Liver Example
anatomic-based
cropping
ignore anatomy
outside liver
Limited Field-of-View
Rigid assumption used
for regional registration
Pick Your Battles Wisely!
Limited Field-of-View
Oops!
What about Deformations ?
Image Registration 201
4 credit course
3 credits - science
1 credit - art
non- Affine
Transformations
Displacement
Function
Displacement
Vectors
non- Affine
Transformations
XB = F ( XA , { ß })
non- Affine
Transformations
XB = F ( XA , { ß(XA) })
non- Affine
Transformations
Transformation parameters
to apply to a voxel depends
on the location of the voxel
… up to 3N parameters
XB = F ( XA , { ß(XA) })
What about Deformations ?
Spatially variant transformations
¾ B-Splines
¾ Thin-Plate splines
parametric
¾ Finite element models
¾ Intensity flow models
free-form
What about Deformations ?
Each have some distinct properties
¾ B-Splines
… local
¾ Thin-Plate splines … global
¾ Finite element
… bio-mechanical
¾ Intensity flow
… image forces
( mono-modality )
What about Deformations ?
Each have some distinct properties
¾ B-Splines
X’ = X + ΔX = X + Σ wi·Β(X-ki)
weights
( … parameters! )
basis function
What about Deformations ?
Each have some distinct properties
¾ B-Splines
Before
1-D Example
voxel
voxel 11 voxel
voxel 22 voxel
voxel 33 voxel
voxel 44 voxel
voxel 55 voxel
voxel 66 voxel
voxel 77
voxel
voxel 33
After
voxel
voxel 11
voxel
voxel 22
voxel
voxel 44
voxel
voxel 55
voxel
voxel 77
voxel
voxel 66
What about Deformations ?
Each have some distinct properties
¾ B-Splines
Before
After
1-D Example
voxel
voxel 11 voxel
voxel 22 voxel
voxel 33 voxel
voxel 44 voxel
voxel 55 voxel
voxel 66 voxel
voxel 77
What about Deformations ?
Each have some distinct properties
¾ B-Splines
1-D Example
ΔX
w
w77
knots
kk11 kk22 kk33 kk44 kk55 kk66 kk77 kk88 kk99 kk10
k 11
10 k11
X
X’ = Xa +lot
ΔXlike
= Xintensity
+ Σ wii·Βmodulation!
(X-kii)
This seems
B-Spline Transformation Model
“Knots”
3-D Grid of Control Points
Multiresolution Deformations
Divide and Conquer
60
60 xx 60
60 xx 48
48 mm
mm
4
4 xx 4
4 xx 3
3 mm
mm
Coarse
Fine
Knot Spacing
ABC Breath hold
Segmental 4D CT
Balter
Balter // UM
UM
Multiresolution Deformations
ABC CT Example
Exhale State
Inhale State
Multiresolution Deformations
ABC CT Example
Exhale State
Inhale State
deformed
B-Spline Deformations
ABC CT Example
Inhale
Inhale deformed
to match exhale
B-Spline Deformations
MR - CT Example
B-Spline Deformations
MR - CT Example
CT
CT
MR
MR
CT
CT
MR
MR
CT
CT
MR
MR
Coming to a DICOM server near you soon?
X
X deformation
deformation
Y
Y deformation
deformation
Z
Z deformation
deformation
magnitude
magnitude
Multiresolution Deformations
MR-CT Example
Reference Dataset
Homologous Dataset
Multiresolution Deformations
MR-CT Example
Reference Dataset
Homologous Dataset
Multiresolution Deformations
MR-CT Example
Split screen
Image Switch
We are not really Splines!
Tissue Sliding
Rigid + Deformation
Brock
Brock // UM
UM
Finite Element Modeling
Exhale
Exhale
Inhale
Inhale
Take into account physical
tissue properties
Brock
Brock // PMH
PMH
Finite Element Modeling
… thorough segmentation is necessary
Data Fusion
Image Mapping / Fusion
resample one image series to match
the scale and orientation of another
Structure Mapping / Transfer
transfer contours defined in one
image series (MR) to another (CT)
Image Mapping / Fusion
MR
Study A
Study A’
CT
... resample one study to match scale and
orientation of the other
Image Mapping / Fusion
CT
CT’
MR
CT’
... resample one study to match scale and
orientation of the other
Structure Mapping
?
... map structure outlines defined on one study
on the other
Structure Mapping
Stack the
Outlines
A
Create a
3D model
B
Structure Mapping
Transform
and Slice
C
Apply Outlines
to CT
D
How …
do we know these algorithms work?
¾ build phantoms and test them
we can know the truth!
¾ provide tools to examine results
we don’t know the truth!
Multimodality Phantoms
CT
MR
1986
1986
van
van Herk
Herk // NKI
NKI
Multimodality Phantoms
Validation
Visualization Tools
¾ Color gel or wash
overlay
¾ Split /dual screen
displays
¾ Anatomic boundary
overlay!
Validation
Split Screen Display
CT
CT
MR
MR
Validation
Image Switch
Validation
Structure Overlay
Auto-segmented boundaries from CT mapped to PET
Validation
Threshold + Compositing
MR
CT
bone window
Validation
Threshold + Compositing
MR
CT
bone window
Validation
Colorwash
MR + PET overlay
CT + PET overlay
Validation
+
CT
GE
GE Web
Web Site
Site
=
PET
CT + PET
Validation
Numerical Tools
Point
1
2
3
4
5
6
Description
2nd branch of bronchial tree
3rd branch of bronchial tree
4th branch of bronchial tree
Vessel bifurcation 1
Vessel bifurcation 2
Vessel bifurcation 3
X
-5.37
-5.73
-6.50
-8.12
-8.06
-10.69
Study A
Exhale
Y
0.98
2.12
2.77
3.37
-1.95
2.47
*
Z
-3.42
-5.42
-8.42
-9.92
-4.42
0.58
X
-4.62
A
A
-5.40
-6.24
-8.12
-7.67
Study
-10.78B
(x , y , z )
All numbers in centimeters
all values in cm.
Inhale
Y
-0.22
A
A 0.74A
A
0.80
1.40
-3.20
1.16
Exhale' ( w/ TPS alignment )
-4.71
-0.47
-3.36
-5.35
0.58
-5.83
-6.27
0.69
-9.51
-8.19
0.91
-11.60
-7.27
-2.83
-3.63
-10.85
0.87
1.24
σ
ΔX
-0.09
0.05
B
B
-0.03
-0.07
0.40
-0.07
Z
-2.92
-5.92
-9.42
-11.42
-3.92
1.08
*
Exhale' - Inhale
ΔY
ΔZ
-0.25
-0.44
0.09
B
B -0.16B
B
-0.11
-0.09
-0.49
-0.18
0.37
0.29
-0.29
0.16
(x , y , z )
0.19
0.29
0.26
Examples
Treatment Planning
using Magnetic Resonance Imaging
Treatment Delivery
using volumetric information
Target Volume Definition
… draw on MR
Axial
“Target”
Coronal
“Target”
+
Optic
Structures
Target Volume Definition
… map to CT
Boolean OR
OR ?
Target Volume Definition
% Volume
… optimize as usual !
Dose-Volume
Histogram
% Dose
Dose Visualization
CT-based dose
Displayed on MR
Dose Mapping
Inhale CT
Exhale CT
Rosu
Rosu // UM
UM
Munro
Munro // Varian
Varian
Registration @ Delivery
CT
CBCT
CT
CBCT
Munro
Munro // Varian
Varian
Registration @ Delivery
Multi-res
Gustavo
Gustavo // Tomo
Tomo
Registration @ Delivery
Original planning CT
Reference CT
Daily CT
Daily CT mapped
to Reference CT
Gustavo
Gustavo // Tomo
Tomo
Registration @ Delivery
Original planning CT
Reference CT
Daily CT
Daily CT mapped
to Reference CT
Gustavo
Gustavo // Tomo
Tomo
Registration @ Delivery
Map contours too!
Summary
Taxonomy of Registration Process
Geometry
Intensity
Interactive
Automated
Affine
Warping
Summary
¾ Techniques are now available to register
3D/4D image data from different modalities
¾ Registered data can be fused to create more
complete models of the patients
¾ Accuracy on the order of image resolutions
reported for “well behaved” situations
Take Home Quiz!
www.ITNonline.net
Thank you
for your
time !
There’s more
than one way
to scan a cat