BVM-Award 2015
– PhD Thesis –
Sketch-Based Interactive Segmentation
and Segmentation Editing for Oncological
Therapy Monitoring
Frank Heckel
March 17, 2015
Medical Background
Oncological Therapy Response Monitoring
 Change in tumor size is an important criterion for assessing the success
of a chemotherapy
 RECIST1 1.1: Sum of maximum diameters of target lesions  Relative change
Complete
Response
Partial
Response
Stable
Disease
Progressive
Disease
Disappearance
< -30%
-30% … 20%
> 20%
 Volume is a more accurate measure
 Many tumors grow/shrink irregularly in 3D
 Requires appropriate segmentation
1
RECIST: Response Evaluation Criteria In Solid Tumors
2 / 22
The Segmentation Problem
 Ultimate Goal: Automatic segmentation




Reproducible results with no effort for the user
Solutions for specific purposes
Might fail (low contrast, noise, biological variability)
Unsolved or insufficient for many real-world problems
 Solutions:
 Manual segmentation
 Interactive tools
 Automatic segmentation + manual correction
 Drawbacks:
 Higher effort
 Lower reproducibility
3 / 22
Interactive Segmentation
Variational Interpolation
 Based on common 2D user interaction: drawing contours
 Segmentation as an object reconstruction problem
 Energy-minimizing surface reconstruction from a point cloud based on RBFs
𝑘
𝑓 𝑐𝑖 = 𝑃 𝑐𝑖 +
𝑤𝑗 𝜙 𝑐𝑖 − 𝑐𝑗 = ℎ𝑖
𝑗=1
 3D surface based on contours from a few slices in arbitrary orientations
4 / 22
Interactive Segmentation
Main Challenges
 Computation time optimization
 Shape preserving constraint reduction
 Parallelization
 Robustness improvement
 Approximation instead of interpolation for resolving contradictions
 Detection and consideration of self-intersection points
5 / 22
Interactive Segmentation
Results
 Computation time: Speedup ≈80
 Evaluation:
Before1
After2
Metastasis
57,53 s
0.7 s
Liver
629,1 s
8.3 s
 Data: 15 liver metastases, 1 liver
 Participants: 2 experienced radiology technicians
Manual
Metastasis 111 s
Liver
1 CLAPACK,
RBF-based Interpolation
21 contours
64 s
1272 s 106 contours 665 s
7 contours
Overlap: 75%
22 contours
Overlap: 94%
1 thread, no reduction 2 MKL, 4/8 threads, reduction by ≈80%
6 / 22
Segmentation Editing
Stop
Segmentation Algorithm
Automatic
yes
Segmentation
Result
yes
Satisfying?
no
Initial
Algorithm allows
modification?
no
Segmentation
Editing Algorithm
Semi-automatic
Start
no
Segmentation Algorithm
Interactive
Segmentation
Result
Satisfying?
Stop
yes
 Most existing methods are low-level and unintuitive in 3D
 High-level correction has not received much attention in research
7 / 22
Segmentation Editing
Sketch-Based Editing in 2D
add
remove
add +
remove
replace
8 / 22
Segmentation Editing
The Correction Depth
 Estimate 3D size of the error by the „diameter“ of the edited region in 𝑠
𝑪𝒔𝒖
𝒔
𝑪𝒔𝒆
9 / 22
Segmentation Editing
Image-Based 3D Extrapolation
 Sample user contour into reference points
 Move reference points to next slice using a block matching
 Connect seed points using a shortest-path algorithm
10 / 22
Segmentation Editing
Image-Independent 3D Extrapolation
 Utilizes the RBF-based interpolation approach
 Reconstruct the new segmentation with contours in the edited slice
and a start / end slice given by the correction depth
 Restrict the new segmentation to the edited region
11 / 22
Evaluation of Editing Tools
Qualitative Evaluation
 131 representative tumor segmentations in CT (lung nodules, liver
metastases, lymph nodes)
 5 radiologists with different level of experience
 Editing rating score: 𝑟edit =
1
0.0𝑟−− + 0.25𝑟− + 0.5𝑟0 + 0.75𝑟+ + 1.0𝑟++
𝑁
12 / 22
Evaluation of Editing Tools
Quantitative Evaluation
 Analyze quality over time
 Editing quality score: 𝑚edit,𝑠max =
1
𝑆max
min(𝑆,𝑆max )
𝑚𝑖
𝑖=1
+ 𝑆 ∙ 𝑚𝑆
13 / 22
Evaluation of Editing Tools
Simulation-Based Evaluation
 Problem: High effort and bad reproducibility of user studies
 Idea: Replace user by a simulation
 Benefits:




Objective and reproducible validation
Objective comparison
Improved regression testing
Better parameter tuning
Start
Validation
Intermediate
Segmentation
Stop
yes
Satisfying?
no
Reference
Target
Segmentation
Simulation
User
User
Input
Previous
Inputs
Control flow
Data flow
Segmentation
Editing
14 / 22
Evaluation of Editing Tools
Simulation-Based Evaluation




Step 1: Find most probably corrected 3D error
Step 2: Select slice and view where the error is most probably corrected
Step 3: Generate user-input for sketching
Step 4: Apply editing algorithm
15 / 22
Evaluation of Editing Tools
Simulation-Based Evaluation
16 / 22
Partial Volume Correction
The Partial Volume Effect
 Smoothing effect caused by limited spatial resolution (of CT)
 Ill-defined border between tumor and healthy tissue, making
segmentation an ill-defined problem
 Could cause significant differences in size measurements
28.4 ml
(-27.5%)
39.2 ml
56.8 ml
(+44.9%)
17 / 22
Partial Volume Correction
Method
 Spatial subdivision into spherical sectors
to cover different tissues
 Define reference tissue values inside and
outside of the object (𝑡𝑖 and to) per sector
 For each sector 𝑠: compute the weight w
of each partial volume voxel
𝑡𝑜𝑠 − 𝑣
𝑤 𝑉 =
, 𝑉 ∈ 𝑃𝑖𝑠 ∪ 𝑃𝑜𝑠
𝑡𝑜𝑠 − 𝑡𝑖𝑠
𝑉𝑜𝑙𝐿 =
1.0
0.75
0.5
𝑤 𝑉 𝑉𝑜𝑙𝑉
0.25
𝑉∈𝐿
70.8 ml
71.1 ml
0.0
18 / 22
Partial Volume Correction
Software Phantom Results
19 / 22
Partial Volume Correction
Hardware Phantom Results
20 / 22
Partial Volume Correction
Multi-Reader Data Results
21 / 22
Summary
 Contributions:
 General image-independent interactive segmentation method
 Efficient and intuitive segmentation editing tools + methodologies for their
evaluation
 Fast algorithm for compensation of partial volume effects
 Future Work:






Improve algorithms for irregular and large objects
Combine image-based and image-independent editing
Make editing simulation more realistic
HCI aspects in editing
4D and multi-label segmentations
Establish volumetric measurements in clinical routine
22 / 22
Thanks to all colleagues at (Fraunhofer) MEVIS, particularly
Dr. Jan Moltz, Lars Bornemann, Dr. Hans Meine, Dr. Stefan
Braunewell, Dr. Markus Lang, Michael Schwier, Dr. Volker
Dicken, Dr. Benjamin Geisler, Olaf Konrad, Wolf Spindler and
Prof. Horst Hahn.
Special thanks to Dr. Christian Tietjen, Dr. Grzegorz Soza,
Andreas Wimmer, Dr. Ola Friman, Prof. Bernhard Preim,
Prof. Andreas Nüchter, all clinical partners and the Visual
Computing in Biology and Medicine community.
An finally, my wife and my children!
Acknowledgement
Bei Herausforderungen geht es nicht ums
Gewinnen, sondern darum, herauszufinden,
was für ein Mensch man ist.
Thank you!
frank.heckel@mevis.fraunhofer.de