Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Disclosure Statement
Act
Act II
I
Acquisition
Planning
I receive research funding from …
ImageImage
Processing
Acquisition
for
Planning
Tx Basics
National Center Institute
U.S. National Institutes of Health | www.cancer.gov
Jeff
Siewerdsen
Marc
Kessler
Johns
Hopkinsof
University
The
University
Michigan
Act I
Question 1
Turning Patients into Numbers
The best metric for image resolution is
#
# images
images
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
16
bits
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
#
# rows
rows
#
# cols
cols
16 bits
14%
1. Pixel Size
4%
2. Slice thickness
7%
3. Number of bits
39%
4. Minimum resolvable line pair
36%
5. Slice thickness x pixel size x pixel size
∆∆x, ∆∆y, ∆∆z
Marc L Kessler, PhD
1
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Question 1
Act I
The best metric for image resolution is
Turning Patients into Numbers
4. Minimum resolvable line pair
Reference ACT I
Act II
Computed
Tomography
Magnetic
Resonance
Nuclear
Medicine
Physics
Anatomy
Physiology
Image Processing
2008
2008
Turning numbers into other numbers
Patient
Patient Modeling
Modeling
Definition
Definition of
of Plan
Plan Geometry
Geometry
Enhancement
Visualization
Segmentation
Registration
Quantification
Plan
Plan Evaluation
Evaluation
Implementation
Implementation of
of Therapy
Therapy
Marc L Kessler, PhD
2
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Question 2
Image Processing for Tx Planning
Enhancement
enhance / suppress features or noise for delineation
I would like you to concentrate on …
1. Enhancement
Visualization
n-D rendering for delineation, planning & evaluation
2. Segmentation
Segmentation
3. Registration
delineation for planning, evaluation & registration
4. Visualization
Registration
fusion to support improved delineation
5. Quantification
Quantification
improved delineation and dose calculations
Answer Now
Question 2
10
0
1
2
3
4
5
6
7
8
9
Question 2
I would like you to concentrate on …
I would like you to concentrate on …
20%
1. Enhancement
20%
1. Enhancement
21%
2. Segmentation
21%
2. Segmentation
20%
3. Registration
20%
3. Registration
19%
4. Visualization
19%
4. Visualization
Automated
Segmentation
for
Radiotherapy
Volumetric
Definition
20%
5. Quantification
20%
5. Quantification
Ballroom D 10-12
Marc L Kessler, PhD
3
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Object Representation
Region
Boundary
Contours / Surfaces
Pixel / Voxel Masks
Object Representation
Object
Boundary
Region
Each representation has pros and cons!
Image Segmentation
Image Segmentation
numbers turned into other numbers
numbers turned into other numbers
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
40
40 -- 100+
100+ images
images // series
series
40
40 -- 100+
100+ images
images // series
series
5
5 -- 10+
10+ structures
structures // image
image
5
5 -- 10+
10+ structures
structures // image
image
Marc L Kessler, PhD
4
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Image Segmentation
Image Segmentation
numbers turned into other numbers
numbers turned into other numbers
40
40 -- 100+
100+ images
images // series
series
5
5 -- 10+
10+ structures
structures // image
image
margins &
field shaping
40
40 -- 100+
100+ images
images // series
series
5
5 -- 10+
10+ structures
structures // image
image
Image Segmentation
Image Segmentation
numbers turned into other numbers
numbers turned into other numbers
point samples
for IMRT calcs
40
40 -- 100+
100+ images
images // series
series
circa 1988 ( … 2009?)
5
5 -- 10+
10+ structures
structures // image
image
Marc L Kessler, PhD
40
40 -- 100+
100+ images
images // series
series
5
5 -- 10+
10+ structures
structures // image
image
5
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Image Segmentation
Lee
Lee /UCL
/UCL
numbers turned into other numbers
Image Segmentation
numbers turned into other numbers
We need to automate!
Image Processing to the Rescue
user beware!
adjust
intensity
mapping
Question 3
Question 3
Image processing is important for
0%
1. Enhancement
6%
2. Segmentation
0%
3. Registration
3%
4. Visualization
91%
5. All of the above
5. All of the above
Marc L Kessler, PhD
6
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Image Segmentation
Edge detection
Level set methods
Clustering methods
Graph partitioning
Histogram-based
Watershed transform
Region growing
Model based
Image Segmentation
Boundary Methods
simple edge detection - high contrast objects
deformable models - active contours / surfaces
Region Methods
feature space - image intensity
region growing / voxel recruitment
morphologic techniques
Image Processing Basics
f(x)
System
H
g(x)
Image Processing Basics
f(x)
System
H
g(x) = H [ f(x) ]
g(x) = H [ f(x) ]
image processing is any form of signal
processing for which the input is an image
Correlation
Convolution
Marc L Kessler, PhD
g(x)
7
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Image Processing Basics
Image Processing Basics
g(x) = kernel(τ) .f(x-τ)dτ
System
H
f(x)
a ⊗ b = F(a) • F(b)
g(x)
Convolution theorem
g(x) = kernel ⊗ f(x)
Convolution
Convolution
Image Processing Basics
Image Processing Basics
1
kernel
1
1
1
1
1
1
1
1
1
1
1
1
1
0
1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
0
0
1
1
1
0
0
0
0
1
1
0
1
0
0
1
1
1
1
1
0
0
0
0
0
1
1
0
1
0
0
1
1
1
1
1
0
0
0
0
0
1
1
0
1
0
0
1
1
1
1
1
0
0
0
0
0
1
1
0
1
0
0
0
1
1
1
0
0
0
0
0
1
1
0
1
0
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
0
0
0
0
0
0
0
0
0
1
1
1
kernel
1
1
1
1
1
1
1
1
1
Convolution
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Did we just
add a margin
to the object?
0
0
0
1
1
1
1
0
0
0
0
0
0
1
1
1
1
1
1
1
0
0
1
1
1
1
1
1
1
0
0
1
1
1
1
1
1
1
0
0
1
1
1
1
1
1
1
0
0
0
1
1
1
1
1
1
1
0
0
0
0
0
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Convolution
Marc L Kessler, PhD
8
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Image Processing Basics
Edge Detection
Good detection
algorithm should mark as many real edges in
the image as possible
Good localization
edges marked should be as close as possible
to the edge in the real image
The gradient of an image is one of the
basic building blocks in image processing
Edge Detection
Minimal response
a given edge in the image should only be
marked once, and where possible, image noise
should not create false edges
Image Processing Basics
-1
0
+1
+1 +2 +1
-2
0
+2
0
0
0
-1
0
+1
-1
-2
-1
x- gradient
… just a bunch
of numbers
y-gradient
2
|G|
G == √|GG
Gyy2|
x|x++|G
2
2
Original
Sobel kernel
Marc L Kessler, PhD
9
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Edge Detection
-1
-1 0
0 +1
+1
-2
0 +2
+2
-2 0
*
-1
0 +1
+1
-1 0
+1
+1 +2
+2 +1
+1
0
0
0
0
Smoothing
1
115
=
0
0
-1
-1 -2
-1
-2 -1
Original
2
4
5
4
2
4
9
2
9
4
5 12 15 12
5
4
9
12
9
4
2
4
5
4
2
σσ=1.4
Filtered
Sobel kernel
Gaussian kernel
Smoothing
Edge Detection
*
=
Original
*
Filtered
Gaussian kernel
Original
=
our objects
are “closed”
Filtered
Gaussian then Sobel kernel
Marc L Kessler, PhD
10
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Lots o’ Kernels
Image Segmentation
Boundary Methods
Sobel
Prewitt
simple edge detection - high contrast objects
Canny
Canny-Deriche
deformable models - active contours / surfaces
Differential
∇2 of Gaussian
Roberts
…
Region Methods
voxel recruitment - region growing
feature space - image intensity
morphologic techniques
Model-based Segmentation
Simple analytic shapes
Population averaged models
super-quadrics,
super-quadrics, spherical
spherical harmonics
harmonics
mean
mean position
position and
and variance
variance
Model-based Segmentation
3D extension of Snakes / Active Contours
Marc L Kessler, PhD
11
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Model-based Segmentation
Model-based Segmentation
model
Place model into
image volume
right kidney
Model-based Segmentation
Optimize model to
match image data
Model-based Segmentation
Etotal = ωEint + Eext
Eint
= model forces
curvature, elastic,
population variance
Eext
model
= image forces
edges/ surfaces
raw edges
Marc L Kessler, PhD
filtered edges
12
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Model-based Segmentation
Model-based Segmentation
1988
1988
Object
Model
optimizing
Etotal minimized
Model-based Segmentation
Object
1998
1998
Model-based Segmentation
EEext
ext
Model
Initialized
Marc L Kessler, PhD
Optimized
13
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Image Segmentation
Meyer/UM
Meyer/UM
Object Representation
Object
Boundary
Region
Each representation has pros and cons!
Image Segmentation
Image Segmentation
Boundary Methods
Boundary Methods
simple edge detection - high contrast objects
simple edge detection - high contrast objects
deformable models - active contours / surfaces
deformable models - active contours / surfaces
Region Methods
Region Methods
voxel recruitment - region growing
feature space - image intensity
morphologic techniques
use some feature of the data to determine the
intrinsic grouping in a set of unlabeled data
… think “membership”
Marc L Kessler, PhD
14
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Intensity Feature
Intensity Feature
a
b
fat
fat
air,
air, lung
lung
frequency
frequency
frequency
frequency
histogram
soft
soft tissue
tissue
fat
fat
bone
bone
CT
CT number
number
CT
CT number
number
soft
soft tissue
tissue
air,
lung
air, lung
simple thresholds
c
“fuzzy” thresholds
d
Simple intensity thresholds
frequency
frequency
frequency
frequency
bone
bone
CT
CT number
number
Intensity Feature
CT
CT number
number
Intensity Feature
fat
fat
muscle
muscle
air,
air, lung
lung
bone
bone
frequency
“fuzzy” thresholds
Simple intensity thresholds
Marc L Kessler, PhD
a
voxel
can be a technique
member
use
a clustering
of
onegrouping
group
to more
decidethan
“best”
…
or “cluster”
fuzzy
c-means
CT number
15
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Vector Intensity Feature
Original images
T11, T22
2-D feature plot
w/ clusters
Vector Intensity Feature
Labeled image
color-coding
… use intensity vectors
PET
CT
PET / CT
… from different modalities!
Image Segmentation
Lee
Lee /UCL
/UCL
Image Segmentation
Lee
Lee /UCL
/UCL
numbers turned into other numbers
user beware!
adjust
intensity
mapping
Marc L Kessler, PhD
16
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Image Segmentation
Image Segmentation
Lee
Lee /UCL
/UCL
Watershed Transform
… consider gradient magnitude of
an image as a topographic surface
Validation Studies
Patient
Surgical
specimen
Simple
Threshold
Gradient-based
method
1
4.1
8.7
4.7
2
5.2
7.4
5.5
3
5.6
16.3
8.2
6
24.3
34.1
25
Image Processing to the Rescue
Lee
Lee /UCL
/UCL
Total laryngectomy – surgical specimen is
4
15.4
37.2
19.7
sliced, digitized,
and 25.3
registered**
5
17.3delineated,
35.4
7
30.9
33.4
27.8
mean
14.7
24.7
16.6
RMSE
0
12.22
3.78
Segmentation / Registration
Segmentation
Registration
Registration
*
* Daisne,
Daisne, Gregoire
Gregoire
Marc L Kessler, PhD
17
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Image Registration
Transformation Models
Image Registration
How many DOF?
Geometric / Physical Methods
Series B
B
Series
Point matching
Surface matching
Finite element models
0?
3 or 6
3xN
Intensity Methods
Series A
A
Series
Cross correlation / SSD
Diffusion / Demons
Mutual information
Identity?
Rigid
Deformable
Registration / Segmentation
Registration / Segmentation
Structure Mapping / Propagation
Several independent
products are there
or almost there!
resample
…what about doing
this for doses too?
Original Segmentation
Mapped Structure
II have
have no
no commercial
commercial interest
interest in
in any
any of
of these
these company
company
Marc L Kessler, PhD
18
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Ling
Ling // MSKCC
MSKCC
Multimodality Targeting
Morphology
GTV
PTV
Tumor Growth
• PET
• IUDR
Hypoxia
• PET
• F-miso
Tumor Burden
• MRI/MRS
• choline / citrate
Biology versus
Morphology
Biological
Target Volume
Multimodality Targeting
Multimodality Targeting
Flair
T22
T11
RTH
RTH // UM
UM
Gd
Diff
Multimodality Targeting
Apply Boolean Operator
AND, NOT, OR, XOR
Other Numbers!
MR volumes mapped to CT study
Marc L Kessler, PhD
19
Multimodality Imaging in Radiation Oncology:
Imaging vs. Imagining
Act II
Turning numbers into other numbers
Act
Act III
II
Planning
Delivery
Patient
Patient Modeling
Modeling
Definition
Definition of
of Plan
Plan Geometry
Geometry
Image Processing
for
T
Tx
Delivery
x Planning
Jan-Jakob
Marc Kessler
Sonke
Netherlands
The University
Cancer
of Michigan
Institute
Plan
Plan Evaluation
Evaluation
Implementation
Implementation of
of Therapy
Therapy
Marc L Kessler, PhD
20