8/15/2011
Disclosure Statement
Act II
I
Act
Acquisition
Planning
ImageImage
Processing
Acquisition
for
Tx Basics
Planning
I receive research funding from …
National Center Institute
U.S. National Institutes of Health | www.cancer.gov
Jeff
Siewerdsen
Marc
Kessler
Johns
Hopkinsof
University
The
University
Michigan
Act I
Turning Patients into Numbers
# images
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
16
bits
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
René Magritte
# rows
# cols
16 bits
Dx, Dy, Dz
1
8/15/2011
Imaging Basics
Background
Imaging Basics
Background
Background
Background
Object
Object
Object
Scene
Image
Scene
Imaging Basics
Background
Background
Scene
Object
Finite Resolution
Finite
Sampling
Question 1
The best metric for image resolution is
19%
Object
Object
1. Pixel Size
20%
2. Slice thickness
21%
3. Number of bits
22%
4. Minimum resolvable line pair
18%
5. Slice thickness x pixel size x pixel size
2
8/15/2011
Question 1
Act I
The best metric for image resolution is
Turning Patients into Numbers
4. Minimum resolvable line pair
Reference ACT I
Act II
Computed
Tomography
Magnetic
Resonance
Nuclear
Medicine
Physics
Anatomy
Physiology
Image Processing
2008
Turning numbers into other numbers
Patient Modeling
Definition of Plan Geometry
Enhancement
Visualization
Segmentation
Registration
Quantification
Plan Evaluation
Implementation of Therapy
3
8/15/2011
Question 2
Image Processing for Tx Planning
Enhancement
enhance / suppress features or noise for delineation
I would like you to concentrate on …
1. Enhancement
Visualization
2. Segmentation
n-D rendering for delineation, planning & evaluation
Segmentation
3. Registration
delineation for planning, evaluation & registration
4. Visualization
Registration
fusion to support improved delineation
5. Quantification
Quantification
improved delineation and dose calculations
Answer Now
Question 2
I would like you to concentrate on …
10
0
1
2
3
4
5
6
7
8
9
Question 2
I would like you to concentrate on …
20%
1. Enhancement
20%
1. Enhancement
21%
2. Segmentation
21%
2. Segmentation
20%
3. Registration
20%
3. Registration
19%
4. Visualization
19%
4. Visualization
20%
5. Quantification
20%
5. Quantification
4
8/15/2011
Segmentation / Registration
Object Representation
Segmentation
Registration
Registration
Region
Boundary
Contours / Surfaces
Object Representation
Pixel / Voxel Masks
Image Segmentation
numbers turned into other numbers
Object
Boundary
Region
Each representation has pros and cons!
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
40 - 100+ images / series
5 - 10+ structures / image
5
8/15/2011
Image Segmentation
Image Segmentation
numbers turned into other numbers
numbers turned into other numbers
40 - 100+ images / series
40 - 100+ images / series
5 - 10+ structures / image
5 - 10+ structures / image
Image Segmentation
Image Segmentation
numbers turned into other numbers
numbers turned into other numbers
margins &
field shaping
40 - 100+ images / series
5 - 10+ structures / image
point samples
for IMRT calcs
40 - 100+ images / series
5 - 10+ structures / image
6
8/15/2011
Brock/PMH
Image Segmentation
Image Segmentation
numbers turned into other numbers
numbers turned into other numbers
… now we also have 4D CT!
Multiple breathing states!
circa 1988 ( … 2009?)
40 - 100+ images / series
40 - 100+ images / phase
5 - 10+ structures / image
5 - 10+ structures / image
Chao/MDACC
Lee /UCL
Image Segmentation
Image Segmentation
numbers turned into other numbers
numbers turned into other numbers
Numerous MD outlines
Couple of Experts +
Image Processing
user beware!
adjust
intensity
mapping
7
8/15/2011
J Dawson / UCSD
Different Thresholds
Image Segmentation
Background
PET / CT
% max
SUV
% of Max SUV
Intensity
Object
Scene
Image Segmentation
Background
Object
Scene
Background
RTH / UM
Does a mm make a difference?
Volume of Sphere = 4/3 r3 = d3 / 6
and V = (d2 / 2) d
then V/V = {(d2 / 2) d} / {d3 / 6}
or
V/V = 3 d/d
Object
… adding a 1 mm margin to a 6 cm
target increases the volume by 10%!
V/V = 3 d/d = 3 • 2x1mm / 60 mm
= 0.1 = 10%
8
8/15/2011
Image Segmentation
numbers turned into other numbers
 We need to automate!
 We need to standardize!
Image Processing to the Rescue
Question 3
Question 3
Image processing is important for
19%
1. Enhancement
21%
2. Segmentation
25%
3. Registration
15%
4. Visualization
20%
5. All of the above
Image Segmentation
Edge detection
Level set methods
Clustering methods
Graph partitioning
Histogram-based
Watershed transform
Region growing
Model based
5. All of the above
9
8/15/2011
Image Segmentation
Edge Detection
Good detection
Boundary Methods
 simple edge detection - high contrast objects
 deformable models - active contours / surfaces
algorithm should mark as many real edges in
the image as possible
Good localization
edges marked should be as close as possible
to the edge in the real image
Region Methods
 feature space - image intensity
Minimal response
 region growing / voxel recruitment
a given edge in the image should only be
marked once, and where possible, image noise
should not create false edges
 morphologic techniques
Edge Detection
+1 +2
Image Processing Basics
+1
-1
0
+1
-2
0
+2
0
0
0
-1
0
+1
-1
-2
-1
x- gradient
… just a bunch
of numbers
y-gradient
|G|
G ==
|GG
Gyy2|
x|x++|G
2
Original
Sobel kernel
10
8/15/2011
Edge Detection
*
-1
0 +1
-2
0 +2
-1
0 +1
+1 +2 +1
0
0
Smoothing
1
=
115
0
-1 -2 -1
Original
2
4
5
4
2
4
9
2
9
4
5 12 15 12
5
4
9
12
9
4
2
4
5
4
2
s=1.4
Filtered
Sobel kernel
Gaussian kernel
Smoothing
Edge Detection
*
=
Original
*
Filtered
Gaussian kernel
Original
=
our objects
are “closed”
Filtered
Gaussian then Sobel kernel
11
8/15/2011
Image Segmentation
Model-based Segmentation
Boundary Methods
 simple edge detection - high contrast objects
 deformable models - active contours / surfaces
Region Methods
 voxel recruitment - region growing
 feature space - image intensity
 morphologic techniques
Simple analytic shapes
Population averaged models
super-quadrics, spherical harmonics
mean position and variance
Wiesmeyer/UM
Model-based Segmentation
Model-based Segmentation
Circa 1992
3D extension of Snakes / Active Contours
right kidney
Place model into
image volume
12
8/15/2011
Wiesmeyer/UM
Model-based Segmentation
Wiesmeyer/UM
Model-based Segmentation
Etotal = wEint + Eext
Circa 1992
Eint
= model forces
curvature, elastic,
population variance
Eext
model
Optimize model to
match image data
model
Wiesmeyer/UM
Model-based Segmentation
filtered edges
edges/ surfaces
Wiesmeyer/UM
Model-based Segmentation
Circa 1992
raw edges
= image forces
Circa 1992
optimizing
Etotal minimized
13
8/15/2011
1988
Model-based Segmentation
1998
Model-based Segmentation
Object
Object
Eext
Model
Model
Segmentation Example
Segmentation Example
Tracking prostate during treatment
using segmentation from ultrasound
14
8/15/2011
Qazi / PMH
Model-based Segmentation
Initialized
Model-based Segmentation
Optimized
Atlas-based Segmentation
Varian
Meyer/UM
Image Segmentation
15
8/15/2011
Object Representation
Image Segmentation
Boundary Methods
 simple edge detection - high contrast objects
 deformable models - active contours / surfaces
Region Methods
 voxel recruitment - region growing
Object
Boundary
Region
 feature space - image intensity
 morphologic techniques
Each representation has pros and cons!
Image Segmentation
Intensity Feature
Boundary Methods
 simple edge detection - high contrast objects
 deformable models - active contours / surfaces
Region Methods
fat
air, lung
use some feature of the data to determine the
intrinsic grouping in a set of unlabeled data
… think “membership”
soft tissue
bone
Simple intensity thresholds
16
8/15/2011
Intensity Feature
a
b
air, lung
frequency
frequency
histogram
CT number
bone
fa
t
“fuzzy” thresholds
d
CT number
muscle
air, lung
bone
frequency
frequency
soft tissue
fat
CT number
simple thresholds
c
Intensity Feature
CT number
Intensity Feature
Simple intensity thresholds
Vector Intensity Feature
frequency
“fuzzy” thresholds
a
voxel
can be a technique
member
use
a clustering
of
onegrouping
group
to more
decidethan
“best”
…fuzzy
or “cluster”
c-means
Original images
T1, T2
CT number
2-D feature plot
w/ clusters
Labeled image
color-coding
… use intensity vectors
17
8/15/2011
Lee /UCL
Vector Intensity Feature
Image Segmentation
numbers turned into other numbers
PET
CT
PET / CT
user beware!
… from different modalities!
adjust
intensity
mapping
Lee /UCL
Image Segmentation
Image Segmentation
Watershed Transform
… consider gradient magnitude of
an image as a topographic surface
18
8/15/2011
Lee /UCL
Lee /UCL
Image Segmentation
Validation Studies
Patient
Surgical
specimen
Simple
Threshold
Gradient-based
method
1
4.1
8.7
4.7
2
5.2
7.4
5.5
3
5.6
16.3
4
15.4
37.2
6
24.3
34.1
25
7
30.9
33.4
27.8
mean
14.7
24.7
16.6
RMSE
0
12.22
3.78
8.2
Total laryngectomy – surgical specimen
is
19.7
sliced, digitized,
delineated,
and 25.3
registered*
5
17.3
35.4
Image Processing to the Rescue
* Daisne, Gregoire
Segmentation / Registration
Image Registration
Segmentation
How many DOF?
Series B
Transformation Models
Registration
Registration
3 or 6
3xN
Series A
0?
Identity?
Rigid
Deformable
19
8/15/2011
Image Registration
Geometric / Physical Methods
Registration / Segmentation
Structure Mapping / Propagation
 Point matching
resample
 Surface matching
…what about doing
this for doses too?
 Finite element models
Intensity Methods
 Cross correlation / SSD
 Diffusion / Demons
 Mutual information
Original Segmentation
Mapped Structure
Ling / MSKCC
Registration / Segmentation
Several independent
products are there
or almost there!
Multimodality Targeting
Morphology
GTV
PTV
Tumor Growth
• PET
• IUDR
Hypoxia
• PET
• F-miso
Tumor Burden
• MRI/MRS
• choline / citrate
Biology versus
Morphology
Biological
Target Volume
I have no commercial interest in any of these company
20
8/15/2011
RTH / UM
Multimodality Targeting
T2
T1
Multimodality Targeting
Flair
Gd
Diff
MR volumes mapped to CT study
Act II
Multimodality Targeting
Turning numbers into other numbers
Apply Boolean Operator
AND, NOT, OR, XOR
Patient Modeling
Definition of Plan Geometry
Plan Evaluation
Implementation of Therapy
Other Numbers!
21
8/15/2011
Act
Act III
II
Planning
Delivery
Image Processing
for
T
Tx
Delivery
x Planning
Jan-Jakob
Marc Kessler
Sonke
Netherlands
The University
Cancer
of Michigan
Institute
Image Processing Basics
f(x)
System
H
g(x)
g(x) = H [ f(x) ]
image processing is any form of signal
processing for which the input is an image
Image Processing Basics
f(x)
System
H
g(x)
g(x) = H [ f(x) ]
Correlation
Convolution
22
8/15/2011
Image Processing Basics
Image Processing Basics

g(x) = kernel(t) .f(x-t)dt
System
H
f(x)
a  b = F(a)  F(b)
g(x)
Convolution theorem
g(x) = kernel  f(x)
Convolution
Convolution
Image Processing Basics
Image Processing Basics
kernel
1
1
1
1
1
1
1
1
1
1
1
1
1
1
0
1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
0
0
1
1
1
0
0
0
0
1
1
0
1
0
0
1
1
1
1
1
0
0
1
1
0
1
0
0
1
1
1
1
1
0
1
1
0
1
0
0
1
1
1
1
1
1
1
0
1
0
0
0
1
1
1
1
1
0
1
0
0
0
0
0
1
1
0
1
0
0
0
0
1
1
0
1
0
0
0
0
1
1
1
kernel
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Did we just
add a margin
to the object?
0
0
0
1
1
1
1
0
0
0
0
0
0
1
1
1
1
1
1
1
0
0
0
0
0
1
1
1
1
1
1
1
0
0
0
0
0
0
1
1
1
1
1
1
1
0
0
0
0
0
0
0
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Convolution
Convolution
23
8/15/2011
Image Processing Basics
The gradient of an image is one of the
basic building blocks in image processing
24