JOURNAL CLUB:
Yang and Ni, Xidian University, China
“Multimodality medical image fusion
based on multiscale geometric analysis of
contourlet transform.”
Jul 21, 2014
Jason Su
Motivation
• Visualization of multiple image modalities or contrasts
is difficult
– Side by side comparisons are often not precise
– Flipping back and forth helps to pronounce changes but
some modalities may have no structural landmarks
• Beginning to collect time-resolved “MP-nRAGE” with
view-sharing methods
– What is the best way to visualize such data, esp. for
thalamic nuclei?
– Can we do something else other than fitting a T1 map?
Goal: Image Fusion
• The combination of multiple images into one while
preserving the important information from each
• Common examples:
–
–
–
–
fMRI overlays on structural images
Segmentation overlays
Nuclear medicine overlays
HDR photography
• Compared to quantitative imaging, the goal is to
achieve a pleasing effect to the eye instead of fitting to
a model
– Thus there are many possible algorithms and no
necessarily “correct” way to do things
Background: Types of data fusion
• Signal level, pixel level
– Image fusion, e.g. averaging, SOS, MIP
– Region-based: consider neighborhood around
current pixel
• Feature level
– Label fusion segmentation: combine multiple
candidate labels to identify features
• Decision level
– Image biomarkers
Gaussian and Laplacian Pyramid
•
•
Pyramids are multiresolution
decompositions of images
Each level is subsampled by a
factor of 2, i.e. each level is an
octave
•
GP: Successively blurred and
downsampled versions of the
image
– Gives scale of features in
the image
•
LP: take differences between
Gaussian levels
– Gives information about
edges of varying widths
ROLP/Contrast Pyramid
• Ratio of low-pass pyramid: take ratios between Gaussian levels
• Contrast = (L-Lb)/Lb
• R = L/Lb = C + 1
Background: The -lets
• Discontinuities destroy the sparsity of a Fourier series, the Gibbs
phenomenon
• Wavelets – are localized and multi-scale
– Perform well in 1D, but poor sense of orientation for 2D
– Only horizontal, vertical or diagonal
• How to better represent a 2D image?
•
– Want multiresolution, localization, critical sampling, directionality, anisotropy
Curvelets – Candés et al.
– Developed in continuous domain then adapted to discrete
– Optimally sparse representation
for smooth 2D functions except
for a discontinuity along a curve
– Models wave propagation
• Contourlets – Do and Vetterli
– Developed in discrete domain
Pointillism-like
Background: Curvelet Decomposition
Directional filter bank.
(a) Frequency
partitioning where l=3
and there are 23 = 8 real
wedge-shaped
frequency bands.
Methods: Algorithm
• Take 2 input images, how to combine them?
• Tale contourlet transform of each
– Each level apears to gain a factor of 2 in angular
resolution Yang’s decomposition
– How does this effect the quality?
Algorithm: Lowpass Subband
• Treat the lowest level of pyramid differently
– This is a tiny thumbnail of the original information
– Higher-level detail is added to this to reconstruct the whole image
• 2 modes of operation: selection or averaging
• Choose based on a threshold criterion: salience
– If the correlation between the input windows in a 3x3 patch in
curvelet space is above a threshold -> weighted averaging
– Else choose the one with more energy (sum sq. over window)
• Averaging is only done at a fixed alpha blend amount
– Not variable dependent on data
• A bit ad hoc in that there are many unspecified preset tunable
parameters: thresholds, blend factors
– They can be optimized for nuclei
Highpass Subband Algo
• Contrast = (L-Lb)/Lb = Lh/Lb
– Ratio of a high level curvelet coefficient to lowest level
1. Compute contrast as above, Lb comes from the pixels
in the lowest level that contribute to the highest level
2. Blur this to get weighted neighborhood contrast
3. Select coefficients from the image that has the higher
value on this metric, i.e. the one that has more local
contrast
• “Using contourlet contrast, more dominant features
can be preserved precisely at all the resolution levels”
Reconstruction
• Take the inverse curvelet transform of the
blended pyramid
Methods
• Test cases
– CT-MR
– Gd and T2w
– PD and T1w
• Compared against existing methods: average, PCA, wavelet
maximum
• Metrics
– Standard deviation – image variability
– Entropy – how much information is in the image
– Overall cross entropy – how close are the distributions, is information
preserved in the fused result?
– Spatial frequency – amount of energy energy in high frequencies
• Only looking at horizontal and vertical freqs.
– Correlation – how similar is the fusion to the inputs
Results: Metrics
• Proposed algorithm generally shows to have
more variability and capture more information
from the inputs
Notes
• PCA table values seem off?
• There is a Matlab implementation of curvelets
by the creators
• How to handle multiple image fusion?