Appearance Changes in
Image Registration
Marc Niethammer
Department of Computer Science
Biomedical Research Imaging Center (BRIC)
UNC Chapel Hill
Appearance Changes in Image Registration Warning
Warning
This talk is about approaches I have personally been involved
with. Other people have been working in this area also and
I will not do proper justice in terms of referencing.
STIA 2014
2
Appearance Changes in Image Registration Motivation
Image Registration
Goal: Spatial alignment of two images
?
Source Image
Target Image
minimize Irregularity of transformation + image mismatch
Sounds simple enough …
STIA 2014
3
Appearance Changes in Image Registration Practical Image Registration
The MD’s perspective
Example Goal: Combine endoscopy and CT images
for improved cancer treatment planning.
CT surface
Endoscopogram
Julian Rosenman, MD
Approach: Registration (=spatial alignment)
STIA 2014
4
Appearance Changes in Image Registration Practical Image Registration
The MD’s perspective
A millions hits. ”This is probably a solved problem.”
STIA 2014
5
Appearance Changes in Image Registration Practical Image Registration
The MD’s perspective
now we are down to 52 results
“Certainly 52 solutions to my problem are enough.”
STIA 2014
6
Appearance Changes in Image Registration Practical Image Registration
The MD’s perspective
now we are down to 52 results
A
Image: www.shapingyouth.org
STIA 2014
A closer look reveals:
None of these programs
solve this problem.
7
Appearance Changes in Image Registration State of Affairs
The sad reality: with a bit of hyperbole
Registration works really well when there is not much to register!
Similar
images

Simple
transforms
However, in reality deformations are often more complex.
 Need for general deformable registration methods.
STIA 2014
8
Appearance Changes in Image Registration Motivation
Deformable Image Registration
Goal: Spatial alignment of two images (beyond affine)
… for example to capture changes over time as in STIA …
?
Source Image
Target Image
minimize Irregularity of transformation + image mismatch
Why is this hard?
STIA 2014
9
Appearance Changes in Image Registration Motivation
Why is Deformable Image Registration Hard?
A: affine, NP: non-parametric (elastic, fluid, …)
Within subject (WS); between subject (BS); with pathology (P)
model realism
A-P
A-P A-BS A-WS
A-BS
NP-WS
NP-BS
model complexity
model robustness
A-WS
NP-P
NP-BS NP-WS
NP-P
model realism
STIA 2014
10
Appearance Changes in Image Registration Motivation
Why is Deformable Image Registration Hard?
Problem 1: Unsuitable deformation models
Does your brain, leg, heart, …
behave like a fluid?
Should it between two subjects?
Image: classicwatersolutions.com
Problem 2: Unsuitable similarity measures
What if images have
different appearances?
Problem 3: Unsuitable numerical solutions
STIA 2014
11
Appearance Changes in Image Registration Similarity Measures
Basis of Similarity Measures
Intensity Image1
SSD
Intensity Image2
Intensity Image2
Intensity Image2
What is considered similar?
Intensity Image1
Normalized
cross correlation
Intensity Image1
Mutual
Information
But sometimes correspondences are not quite so easy …
STIA 2014
12
Appearance Changes in Image Registration Appearance Change Example
Motivating issue 1: Intensity changes over time
Normal brain development (macaque)
Locally changing image intensities cause problems for
image similarity measures (SSD, mutual information, …)
STIA 2014
13
Appearance Changes in Image Registration Appearance Change Example
Motivating issue 2: Texture-blending
Texture blending for computer graphics.
Extreme case: images are vastly different.
STIA 2014
14
Appearance Changes in Image Registration Appearance Change Example
Motivating issue 3: Intensity/Structural changes
Traumatic brain injury
(real data)
Brain tumor
(simulated data)
“Similar looking regions” do not correspond.
Structures only exist in one of the two images.
STIA 2014
15
Appearance Changes in Image Registration Appearance Change Example
Motivating issue 4: Multimodal Registration
Electron microscopy
Light microscopy
Image intensities differ, but additional complexities such as
blurring are present and need to be accounted for.
STIA 2014
16
Appearance Changes in Image Registration Possible Solutions
What to do?
Is all hope lost here?
STIA 2014
17
Appearance Changes in Image Registration Possible Solutions
Possible Solutions
(Non)parametric modeling
vs
Data-driven approaches
STIA 2014
18
Appearance Changes in Image Registration Possible Solutions
Possible Solutions
Parametric Modeling
STIA 2014
19
Appearance Changes in Image Registration Intensity changes over time
Motivating issue 1: Intensity changes over time
Normal brain development (macaque)
Possible solution:
Impose a model over time to change your similarity measure
STIA 2014
20
Appearance Changes in Image Registration Intensity changes over time
Modeling of Appearance Changes
Normal temporal change (for a macaque monkey)
Concretely: Impose logistic model
[your favorite model] to account
for expected changes over time
STIA 2014
21
Appearance Changes in Image Registration Intensity changes over time
Appearance Change: Brain Development
Inferior (bottom)
superior (top)
This strategy not only allows for improved registration results,
but also provides interesting information about the general
brain maturation process …
STIA 2014
22
Appearance Changes in Image Registration Non-corresponding regions
What do we do with non-corresponding regions?
Traumatic brain injury
(real data)
Brain tumor
(simulated data) –
TumorSim [Prastawa et al.]
As we cannot match them we somehow need to
model or ignore the changes.
STIA 2014
23
Appearance Changes in Image Registration Non-corresponding regions
Solution 1: Cost-function masking
One solution: Cost function masking [Brett2001]
= ignoring matching cost in region of source image
X
Image: gerdleonhard.typepad.com
STIA 2014
… let’s just not look at it!
24
Appearance Changes in Image Registration Non-corresponding regions
Solution 2: Geometric Metamorphosis
Model lesion/tumor/… and its
temporal change
• Composition of two deformations
- e.g., tissue displacement & infiltration
- jointly estimated
• Image composition model
- = “glorified cost-function masking”
Model is probably more interesting
from a deformation modeling
perspective..
STIA 2014
25
Appearance Changes in Image Registration Non-corresponding regions
Geometric Metamorphosis: Synthetic Results
Only infiltration
Only deformation
Deformation + infiltration
Deformation + recession
STIA 2014
26
Appearance Changes in Image Registration Possible Solutions
Possible Solutions
Non-parametric Modeling
w/o learning
to deal with very complex changes
STIA 2014
27
Appearance Changes in Image Registration Possible Solutions
Possible Solutions
One solution is metamorphosis,
which requires a little LDDMM detour
[Miller, Trouve, Younes, …]
STIA 2014
28
Appearance Changes in Image Registration Metamorphosis
LDDMM detour
?
Fluid flow setup [Miller, Younes, Trouve, …]:
What is the best
velocity field, v,
to deform one
image into
the other?
s.t.
Optimal control problem, which can can be rewritten as …
STIA 2014
29
Appearance Changes in Image Registration Metamorphosis
Constrained Optimization for LDDMM
?
Can be rewritten as
s.t.
This just requires
infinite-dimensional
constrained
optimization.
… and I am telling you this because …
STIA 2014
30
Appearance Changes in Image Registration Metamorphosis
Image Registration + Image Metamorphosis
Standard Image Registration
Image Metamorphosis
?
Assumes 1-1 correspondence
Registration and blending
add
STIA 2014
to energy
31
Appearance Changes in Image Registration Metamorphosis
Image Metamorphosis
Metamorphosis is an elegant model,
but work remains to be done
• to improve the numerical solution methods
• to extend it to longitudinal data
• to possibly couple it with some parametric models
for greater control over the appearance change
STIA 2014
32
Appearance Changes in Image Registration Possible Solutions
Possible Solutions
Let’s switch gears a bit …
Data-Driven (Learning) Approaches
… will tell you about two different approaches:
1. Image analogies (for microscopy)
2. Recap of “Low-rank to the rescue”
STIA 2014
33
Appearance Changes in Image Registration Image Analogies
Registration for Microscopy
Registration is difficult between
different image modalities
(mutual information may be the best hope)
A’
Light microscopy
B
?
Electron microscopy
Other possible solution: Image Analogies (from graphics)
STIA 2014
34
Appearance Changes in Image Registration Image Analogies
Registration for Microscopy: Image Analogies
Solution approach:
Image Analogies:
learned
A
B
A’
B’
A relates to B
A’ relates to B’
unknown
STIA 2014
35
Appearance Changes in Image Registration Image Analogies
Image-Analogies by Lookup
Generate corresponding
Image patches
STIA 2014
A
B
36
Appearance Changes in Image Registration Image Analogies
Image-Analogies by Lookup
Synthesize a new image
by lookup
C
STIA 2014
D
?
37
Appearance Changes in Image Registration Image Analogies
Image-Analogies by Lookup
Synthesize a new image
by lookup
STIA 2014
38
Appearance Changes in Image Registration Image Analogies
Image-Analogies by Lookup
Synthesize a new image
by lookup
STIA 2014
39
Appearance Changes in Image Registration Image Analogies
Image-Analogies by Lookup
Synthesize a new image
by lookup
STIA 2014
40
Appearance Changes in Image Registration Image Analogies
Image-Analogies by Lookup
Synthesize a new image
by lookup
STIA 2014
41
Appearance Changes in Image Registration Image Analogies
Image-Analogies by Dictionary Learning
Use LASSO for dictionary
learning to learn a basis
for patches which can be
used to predict one modality
from the other.
STIA 2014
42
Appearance Changes in Image Registration Possible Solutions
Possible Solutions
Learning from Populations
Super-brief summary of Roland Kwitt’s talk from yesterday:
Low-rank to the rescue:
Atlas-Based Analyses in the Presence of Pathologies
STIA 2014
43
M ot ivat ion
Appearance Changes in Image Registration Low-rank to the rescue
Problem: Atlas-based tissue segmentation
Motivating Problem: Registration w/ Pathologies
Ij
◦ φ−
1
j
At las
+ t issue s
X
7
φ⇤
e
c
◦
n
e
I⇤
d
n
o
p
s
r e blem
r
o
c
pr o
−1
STIA 2014
Previous registration strategies in similar settings:
44
Appearance Changes in Image Registration Low-rank to the rescue
What if?
What if there were a method to transform an image with
pathology into a healthy-looking image?
magic
This is exactly what “Low-rank to the Rescue” does!
STIA 2014
45
Appearance Changes in Image Registration Low-rank to the rescue
Low-rank + sparse decomposition
STIA 2014
46
Appearance Changes in Image Registration Low-rank to the rescue
Low-rank + sparse decomposition for images
Images are represented by column-vectors.
STIA 2014
47
Appearance Changes in Image Registration Low-rank to the rescue
Low-rank + sparse decomposition example
Now we can work with “almost-normal” images!
STIA 2014
48
Appearance Changes in Image Registration Summary
Summary
Multiple possibilities to deal with image differences:
The classics: mutual information, normalized cross correlation
Explicit modeling:
- parametric (e.g., logistic curve)
- non-parametric (metamorphosis)
- cost-function masking
Data-driven modeling:
- image analogies through dictionary learning
- creating “normal images” using low-rank + sparse
Which method to use will of course be application-dependent.
STIA 2014
49
Appearance Changes in Image Registration The End
Questions?
Questions?
STIA 2014
50