Rigid Registration of Echoplanar and
Conventional Magnetic Resonance Images by
Minimizing the Kullback-Leibler Distance
by
Saull Soman
MAS SACHUSETTS INSTITUTE
OF TECHNOLOGY
B.A. Chemistry (1997)
The Johns Hopkins University
JUL 0 7 2003
LIBRARIES
Submitted to the Department of Electrical Engineering and Computer
Science
in partial fulfillment of the requirements for the degree of
Master of Science in Computer Science and Electrical Engineering
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
May 2003
© Massachusetts Institute of Technology 2003. All rights reserved.
.......................
A u th or ............
Department of Electrical Engineering and Coiputer Science
May 22, 2003
Certified by.
William M Wells III
Associate Professor, Harvard Medical School
RIsearch Scientist, EECS Department
Ties is.Stipervisor
Accepted by ............
.
..
'ARnhur C. Smith
Chairman, Department Committee on Graduate Students
2
Rigid Registration of Echoplanar and Conventional Magnetic
Resonance Images by Minimizing the Kullback-Leibler
Distance
by
Salil Soman
Submitted to the Department of Electrical Engineering and Computer Science
on May 22, 2003, in partial fulfillment of the
requirements for the degree of
Master of Science in Computer Science and Electrical Engineering
Abstract
Functional Magnetic Resonance Imaging (fMRI) studies are derived from a time series
of Echoplanar images (EPIs). Compared to conventional Magnetic Resonance Images
(MRIs), EPIs are of relatively poor quality for discerning anatomic features and are
often registered with corresponding MRIs to map brain activity to neuroanatomy.
In this paper we demonstrate the utility of a technique to register an EPI-MRI pair
by minimizing the discrepancy between its joint intensity probability mass function
(PMF) and a previously learned one for a properly registered EPI-MRI pair, using
the Kullback-Leibler Distance (KLD). In probing experiments Joint Entropy (JE)
and Mutual Information showed significant bias relative to KLD along the axial direction and JE along a rotation axis. A comparison of searches using random starting
poses showed KLD to have lower final pose errors than JE. Results of variation on
parameters of the KLD based EPI-MRI registration technique are also presented.
Thesis Supervisor: William M Wells III
Title: Associate Professor, Harvard Medical School, Research Scientist, EECS
3
4
Acknowledgments
"When you set out on your journey to Ithaca,
pray that the road is long,
full of adventure, full of knowledge..."
- excerpt from Ithaca by Constantine P. Cavafy
I would like to thank Sandy Wells for his endless support, guidance and time
commitment. It was truly a pleasure to work with someone so patient, insightful, and
above all else, energized by his field of work.
I would also like to thank Prof. Grimson for giving me the opportunity to explore
medical imaging research in his group. He has assembled a marvelous group of imaging
scientists, and I am grateful for the opportunity to have worked with them.
I would like to thank Prof. Peter Szolovits for his patience, wisdom, guidance,
and support throughout my time at MIT. He has been a great resource, and I owe
him much for bringing me to MIT.
Thanks are due also to Prof. Lozano-Perez, who provided me with much needed
insight and guidance into the fundamentals of computer science as a discipline.
I would like to thank the SPL at the Brigham and Women's Hospital for providing
me data for this research as well as other data I used in developing the code for this
project.
I also owe many thanks to the members of the AI Lab, both past and present, for
their mentorship, friendship, and inspiration. I owe them the pleasure of the most
intellectually rich and stimulating environment I have ever experienced.
In particular, I would like to thank Erik G. Miller, for his countless hours of
mini-lectures, advice, and friendship. I would also like to thank Pollina Golland, who
introduced me to the AI lab and to Sandy, and to what have been some of the most
rewarding experiences I have ever known. And last but not least, I need to thank
Albert C.S. Chung for his advice, insight, and for introducing me to Matlab hacking
and the many wonders of the Kullback-Leibler Distance.
And finally, I would like to take this opportunity to thank my family, whose
5
endless love and support have made all daunting things seem more possible, and
whose guidance and inspiration gave me the courage to dream.
This work was supported by a Medical Informatics training grant from the National Library of Medicine, a grant from the Pfizer Corporation, and the NSF ERC
grant.
6
Abbreviations
AJP Aligned Joint Probability Mass Function
BOLD blood oxygenation level dependent
EEG electronencephalography
EPI Echoplanar Image
fMR functional Magnetic Resonance data
fMRI functional Magnetic Resonance Imaging
G Gauss
JE Joint Entropy
KLD Kullback-Leibler Distance
MEG magnetoencephalography
MI Mutual Information
MR Magnetic Resonance
MRI Magnetic Resonance Image
PET positron emission tomography
Poses positions and orientations
PMF Probability Mass Function
RR Registration Reference
SPM Statistical Parametric Mapping
T Tesla
Voxel Volume Element
7
Contents
1
15
Introduction
1.1
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2
Medical Applications of functional Magnetic Resonance Imaging Data
1.3
15
Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
2 Functional Magnetic Resonance Studies and Echoplanar Imaging
21
2.1
fMRI And Echo Planar Image Formation . . . . . . . . . . . . . . . .
22
2.2
EPI and fMRI Caveats . . . . . . . . . . . . . . . . . . . . . . . . . .
23
2.3
Spatial Map Formation . . . . . . . . . . . . . . . . . . . . . . . . . .
24
27
3 Image Registration
3.1
Overview of Registration Algorithms . . . . . . . . . . . . . . . . . .
27
3.2
Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
3.2.1
Rigid Transformations . . . . . . . . . . . . . . . . . . . . . .
30
3.2.2
Image Dimensionality and Degrees of Freedom . . . . . . . . .
31
Objective Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
3.3.1
Joint Entropy and Mutual Information . . . . . . . . . . . . .
32
3.3.2
Kullback Leibler Distance
. . . . . . . . . . . . . . . . . . . .
33
3.3.3
Histogramming . . . . . . . . . . . . . . . . . . . . . . . . . .
33
Optimization Methods . . . . . . . . . . . . . . . . . . . . . . . . . .
34
3.4.1
Downhill Simplex . . . . . . . . . . . . . . . . . . . . . . . . .
34
3.4.2
Image Smoothing . . . . . . . . . . . . . . . . . . . . . . . . .
36
3.3
3.4
37
4 Methods
8
4.1
4.2
. . . . . . . . . . . . . . . . .
37
4.1.1
Image Preparation . . . . . . . . . . . . . . . . . . . . . . . .
38
4.1.2
Blurring the Images . . . . . . . . . . . . . . . . . . . . . . . .
38
4.1.3
Image Transformations . . . . . . . . .... . . . . . . . . . . .
38
4.1.4
Histogramming to Estimate Joint Intensities . . . . . . . . . .
39
4.1.5
Kullback-Leibler Distance (KLD) . . . . . . . . . . . . . . . .
40
4.1.6
Interpolation vs. Rounding . . . . . . . . . . . . . . . . . . . .
40
4.1.7
Optimization of the Transformation T - Downhill Simplex Searches 40
Overview of Registration Experiments
Objective Function Probing . . . . . . . . . . . . . . . . . . . . . ..
41
4.2.1
Intensity Scaling Image Pairs from Different Acquisitions . . .
41
4.2.2
Pseudo 3D Experiment - Histogramming over Multiple Slices
42
.
5 Registration Experiments
43
5.1
Probe Comparisons: KLD, JE, MI
. . . . . . . . . . . . . . . . . . .
43
5.2
Objective Function Downhill Simplex Searches . . . . . . . . . . . . .
44
5.3
Effect of Varying Epsilon . . . . . . . . . . . . . . . . . . . . . . . . .
44
5.4
Effect of Different Training Pairs on KLD
. . . . . . . . . . . . . . .
49
5.5
KLD Using Training and Test Pairs from Different Volumetric Images
49
5.6
Effects of Bucket Size and Interpolation on KLD . . . . . . . . . . . .
52
5.7
Combining Histograms across Multiple Slices - Pseudo 3D X Probes
52
6 Discussion
57
6.1
Observations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
58
6.2
Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
58
9
List of Figures
1-1
Example of how brain function information can be highly localized to an individual patient's neuroanatomy by registration. Pictured above as black regions are brain areas found to be
activated during a functional MRI study. This information was then
spatially correlated to specific gyri (a high level of detail of anatomy)
for that patient. Registering fMRI data to higher resolution MR images for individual patients holds great promise for improvement of
neurosurgical applications. Provided by Florin Talos M.D . . . . . . .
2-1
An echoplanar image (left) and its corresponding magnetic
resonance image (right). . . . . . . . . . . . . . . . . . . . . . . .
3-1
19
22
Nine-tiered classification scheme of registration methods presented by Maintz et al. [MV98]. The problem statement, registration paradigm, and optimization procedure provide a unique classification of any registration algorithm. Items from categories 7,8, and
9 describe problem statements. Categories 2, 3, 4, and 5 list items
that can describe registration paradigms. Lastly, category 6 lists optimization procedures. These divisions, however, are in no way absolute
[M V98]. . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . .
10
28
3-2
Overview of Rigid, Intensity Based Image Registration algorithm. Of the two image sets to be registered, one is designated the
floating set, and the other the reference set. The two sets are spatially
oriented to one another in an initial pose. A scoring metric is then
used to assess the quality of the spatial alignment of the two image
sets. Rigid transformations (translations and rotations) can then be
applied to the floating image to improve the spatial alignment scoring.
3-3
29
Downhill Simplex - map of a 3D search space with a minimum
being traversed by a simplex. A simplex is a geometrical figure
consisting, in 2 dimensions, of 3 points (a triangle) and all of their
interconnecting line segments. (In N dimensional space, a simplex is
of N+1 points, including all line segments, polygonal faces, etc). In
the downhill simplex method, the simplex is used to search a multiparametered search space that is non-linear in its parameters for a
global minimum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-1
4-2
Corresponding MR (left) and Echo-Planar (right) Image Pairs. 39
(From Left to Right) Increasing strength blur applied to an
Echo-planar image.
5-1
. . . . . . . . . . . . . . . . . . . . . . . . . .
39
X Probes - (from top to bottom) KLD, JE, MI. JE shows
significant bias relative to KLD and MI. . . . . . . . . . . . . .
5-2
35
45
Y Probes - (from top to bottom) KLD, JE, MI. Both MI and
JE show significant bias relative to KLD.
. . . . . . . . . . . .
46
5-3 0 Probes - (from top to bottom) KLD, JE, MI. JE shows
significant bias relative to KLD and MI. . . . . . . . . . . . . .
11
47
5-4
Results of KLD and joint entropy-driven downhill simplex
searches for a test image given 100 random start poses for
the test EPI. Trials were classified as being successful if the final
pose error fell under a threshold. The KLD based successful searches
showed average pose parameters closer to the expert's chosen pose than
the joint entropy based method . . . . . . . . . . . . . . . . . . . . .
5-5
48
Significant EPI intensities overlaid on corresponding MRI.
(Left) Hand registration of the EPI-MRI pair by an expert, (middle)
EPI transformed to the average final pose parameters of successful
KLD method trials for downhill simplex, and (right) EPI transformed
according to the average final pose parameters of successful Joint Entropy based method. . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-6
Effect of Varying
=
49
on X Probes - (Top) ( = 1x10- 6 , (Middle) (
1x101 3 , and (Bottom) 6 = 1x10-2 3 . No significant performance
changes were seen varying the 6 values within this range. . . . . . . .
50
5-7 KLD X probes of EPI-MRI Pair 12 using (top) Pair 9 as
training pair, and (bottom) Pair 14 as training pair. Both
training pairs were sampled from the same EPI-MRI volume set as
pair 12.
5-8
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51
Comparison of X Probes (left) using training and test pairs
from the same volume of Echoplanar and MR data, and (right)
using training and test pairs where each set was sampled from
a different EP-MR volume.
. . . . . . . . . . . . . . . . . . . . .
12
53
5-9
Effects of varying bucket size with (left column) and without
(right column) use of bi-linear interpolation in KLD based
registration. Probes in top row used 32 buckets, in middle row used
128 buckets, and in bottom row used 256 buckets. . . . . . . . . . . .
5-10 Pseudo 3D Experiment Results - modified KLD X probes in
54
which all of the EPI-MRI pairs from one time point were
divided into two sets (anterior and posterior halves), with
information from one set used for training, and the other
used as the test set. Intensity pair information from all the image
pairs in a set was combined to make the histograms for these probes.
Shown above are probes where the anterior half (top) and the posterior
half (bottom) of the image set were used. . . . . . . . . . . . . . . . .
13
55
List of Tables
1.1
Comparison of Functional Neuro-Imaging Modalities [HBHH01] 18
14
Chapter 1
Introduction
1.1
Background
Researchers often use functional Magnetic Resonance Imaging (fMRI) to study brain
activity. These studies derive spatial activation maps from a time series of Echoplanar
Images (EPIs) of the patient's head taken as the patient responds to specific tasks or
stimuli. While EPIs provide brain activity information on a time scale adequate for
making inferences about regional brain function, they are of relatively poor quality
for discerning anatomical features. Because there are a variety of tasks that are facilitated by the ability to attribute features such as control of motor function to specific
neuroanatomy (e.g. tasks such as planning neurosurgical procedures or performing
neuroscience experiments), researchers often desire to register EPIs to their corresponding Magnetic Resonance Images (MRI). This registration allows brain activity
noted in regions of EPIs over successive volumes of the time series to be attributed
to anatomic regions of the brain. The widely used Mutual Information (MI) technique provides a partial solution to this problem, but shows limitations in capture
range for rigid transformations. In this work we demonstrate the utility of a method
that leverages a priori domain knowledge to register an Echoplanar Image (EPI) Magnetic Resonance Image (MRI) pair by utilizing information about a properly
registered EPI-MRI pair. The technique searches the space of transformations of the
EPI of a test image pair for a pose that results in a joint intensity Probability Mass
15
Function (PMF) that most resembles a learned one for a properly registered training
pair. The difference between the Probability Mass Functions (PMFs) is determined
using the Kullback-Leibler Distance (KLD), an information theoretic similarity measure commonly used in machine learning and information theory [Bis95]. The KLD
registration approach has previously been used to solve challenging angiographic registration problems [CWNG02, CCY+03].
In fMRI studies, a time series of EPIs of a patient's head are taken and regions
of the EPIs are analyzed for changes over time [KBC+92].
Regions of successive
EPIs where intensity changes correlate with the presence or absence of the activation
condition are indicative of brain activity. Brain activation indirectly causes intensity
variations in the EPI images by way of the blood oxygenation level dependent (BOLD)
effect [OL90] (see Section 2.1). Time series of EPIs must capture successive images
at intervals of approximately once every two seconds to be able to reflect these rapid
occurring changes in blood flow. Conventional Magnetic Resonance Images (MRIs)
are acquired over a period of some minutes per image capture, and provide more
detailed images of brain anatomy [JC99]. Registering a set of EPIs to a set of corresponding MR images allows researchers to correlate areas of apparent activity in the
EPI images with specific brain anatomy.
It is widely recognized that EPIs can contain significant distortions in addition to
the intensity voids that are due to magnetic susceptibility effects [JC99]. A definitive
solution to the EPI/MRI registration problem will likely address this issue directly, by
simulating the effect, or by field mapping at the time of acquisition. Nevertheless, we
feel that in the interim, a robust method of rigid registration would be well received
by the research community that uses fMRI.
One pragmatic approach to the MRI
/
EPI registration problem is to acquire a
conventional MRI scan just before the EPI images for use as a Registration Reference (RR) [WWY+99] that may be registered to other conventional MRIs using, for
example, MI. This method depends on the EPI and RR being in correspondence by
the design of the scanning protocols. While the method is viable, it does require
an additional scan, and it is not usable retrospectively for data that was acquired
16
without the RR. In addition, there is frequently residual misregistration between the
EPIs and the RR that needs to be corrected manually.
Many medical image registration problems have been solved using the MI approach
[VW97, WPH+96, MCV+97]. The MI approach to registration seeks a transformation
which maximizes the statistical dependence among the two images, without regard to
particulars of the relationship. There are strong similarities between the MI approach
and that of minimizing the Joint Entropy (JE). The objective functions share a JE
term, which is responsible for a pronounced extremum at the correct pose. The
MI approach uses additional individual entropy terms that can enhance long range
capture in some applications. As will be demonstrated in Chapter 5, the widely-used
MI registration approach can, however, perform poorly on the EPI/MRI registration.
1.2
Medical Applications of functional Magnetic
Resonance Imaging Data Registration
fMRI is becoming the diagnostic method of choice for learning how a normal, diseased,
or injured brain is working. It is also a powerful tool for assessing the potential risks
of surgery or other invasive treatment of the brain. The term fMRI can also include
other MRI techniques that are sensitive to physiologic changes (such as changes in
water motion). Together, these techniques provide means for identifying and monitoring brain tumors, strokes, and certain chronic disorders of the nervous system such
as multiple sclerosis. fMRI can also serve as a useful means of documenting brain
abnormalities related to dementia or seizures.
Other functional neuroimaging methods exist, such as positron emission tomography (PET), electronencephalography (EEG), and magnetoencephalography (MEG).
These differ from fMRI in the type of physiological response measured and their
temporal and spatial resolutions (see Table 1.1). For instance, fMRI measures local
blood oxygenation changes, PET measures either blood flow or metabolic activity (a
more direct measure of activity than blood flow), and EEG measures induced elec17
MEG
Measurement
Blood flow
or metabolic activity
Induced cortical
currents
Induced cortical
fMRI
Blood oxygenation
Modality
PET
EEG
Temporal Res
tens of
seconds
milliseconds
Spatial Res
many
millimeters
cm
milliseconds
cm
seconds
mm
Safety
Radiation
Risk
low risk
high safety
low risk
high safety
magnetic fields
low risk
high safety
changes
Table 1.1: Comparison of Functional Neuro-Imaging Modalities [HBHH01]
trical currents within the cortex. EEG and MEG have very high temporal resolution
(milliseconds) but poor spatial resolution (centimeters); PET has poor temporal resolution (tens of seconds) and intermediate spatial resolution (many millimeters), while
fMRI has an intermediate temporal resolution (seconds) and good spatial resolution
(millimeters). PET also requires that a radioactive agent be injected into the subject,
while the other methods are relatively non-invasive [HBHH01] (see Table 1.1. Overall,
fMRI is one of the safer, high resolution neuro-imaging modalities.
Facilitating the integration of fMRI information with modalities that provide
structural information can be useful to a broad spectrum of clinical endeavors, including surgical, neurologic and cognitive psychological applications. Figure 1-1 shows an
example of what such a registration effort produces.
Among their many uses, fMRI studies are useful in planning radiation therapy
to treat brain tumors, since they can help physicians monitor both metabolism of
the tumor as well as normal brain function (both of which can be inferred by noted
changes in blood flow - see Chapter
2). fMRI can also help assess the effects of
stroke, trauma, or degenerative disease like Alzheimer's on brain function [AR03J.
1.3
Thesis Outline
The remainder of this document is divided into five Chapters - in Chapter 2 the basics
of MRI and EPI image formation are presented, and how they relate to fMRI studies.
18
14
Figure 1-1: Example of how brain function information can be highly localized to an individual patient's neuroan atomy by registration. Pictured
above as black regions are brain areas found to be activated during a functional MRI
study. This information was then spatially correlated to specific gyri (a high level of
detail of anatomy) for that patient. Registering fMRI data to higher resolution MR
images for individual patients holds great promise for improvement of neurosurgical
applications. Provided by Florin Talos M.D
19
In Chapter 3 the fundamental concepts of image registration techniques are reviewed.
Chapter 4 introduces the methodologies implemented in this research project along
with the basic outline of the registration algorithm of this thesis. Chapter 5 presents
the results of this work. And finally, observations and potential future work are
described in Chapter 6.
20
Chapter 2
Functional Magnetic Resonance
Studies and Echoplanar Imaging
As described in Chapter 1, registration of functional Magnetic Resonance data to
MRI holds the promise of marrying clinically useful brain function data with equally
useful structural anatomy information. The fundamental differences between functional Magnetic Resonance data and Magnetic Resonance (MR) images have made
the registration of these modalities very challenging. fMRI studies result in spatial
activation maps derived from statistical analysis of a time series of images, while
MRIs represent intensity information about structural neuroanatomy. However, 6he
images from which fMRI studies generate spatial activation maps of the brain, EPIs,
do contain spatial intensity information, and so can serve as a proxy for registering
fMR activation maps to their corresponding MR image set. Consequently, the transformation that registers a set of EPIs with its corresponding set of MRIs should bring
the set of spatial activation maps derived from those EPIs into spatial alignment with
those MRIs as well.
While there are some distortions in EPI formation, there appears to be enough
spatial intensity information in EPI images to make EPI to MRI registration a viable
proxy for registering fMRI data to MRI registration.
In this chapter the basic facets of MR and Echoplanar (EP) image formation, and
their relation to fMRI studies, are presented. Current methods for registering fMR
21
Figure 2-1: An echoplanar image (left) and its corresponding magnetic resonance image (right).
data to MRIs are then described.
2.1
fMRI And Echo Planar Image Formation
Spatial activation maps are the product of fMR imaging studies designed to gather
evidence concerning which areas of the brain become activated as part of a particular
activity or exposure. In a typical study, a patient is placed inside a magnetic resonance
imaging machine, a MRI of the patient is taken (to serve as a reference later), and
then a series of images of the patient are taken rapidly - one every few seconds. These
images are taken before, during, and after the patient is asked to partake in specific
tasks or exposures [JMS01]. In most studies, these images are EPIs [Man77], though
in some cases other imaging types are used [AKC86, MM87].
EPIs are sensitive to changes in blood flow and
/
or blood oxygenation in the
brain caused by brain activity - this is known as the BOLD effect [KBC+92, OL90].
The EPIs taken while the patient is engaged in the study's activities (called periods
of activation) can be compared with those taken during periods of rest in order to see
which parts of the brain are activated by the stimulus or activity.
Despite being captured by the same hardware as MRIs, EPIs have much less
22
spatial resolution (EPIs usually contain 64x64 pixels [JC99], while MRIs have 256x256
pixels).
The lower resolution of EPIs is a consequence of their higher acquisition
speed, relative to MRIs. While EPIs are acquired once every few seconds, a single
MRI scan, in contrast, is acquired over several minutes. The higher acquisition speed
of EPIs, however, is what allows them to depict the rapidly occurring changes that
occur during periods of activation.
As mentioned earlier, fMR data is generated by statistical analysis of EPIs taken
during activity and rest periods. This is usually done using statistical parametric
mapping packages like SPM [FHP+96I. Often the same experiment is repeated several
times in order to allow for the collection of a number of activation
/
rest EPI sets,
thus boosting the signal to noise ratio. This is necessary because the BOLD response
typically varies from between 0.5 to 5 percent of the average image intensity within the
brain [JMS011. These sets are used to generate mappings of probability of activation.
This topic, along with MR and EP image formation, is covered at length by Jezzard
et al. [JMS01]. Ultimately, the analyses applied to the sets of EPIs result in spatial
activation maps, which have less spatial information than the EPIs.
2.2
EPI and LMRI Caveats
There are a number of caveats to using echo-planar images for registration. While
those concerns are not directly involved in the two-dimensional registration experiments performed in this thesis, they will likely have some bearing on three-dimensional
EPI to MRI registration experiments, and so warrant brief mention. All of these topics are covered at length in Jezzard et al's book [JMS01].
EPIs can experience field deformations due to inhomogeneities inside the human
head. This deformation is most notable where there is a tissue/air interface, such as
near the temporal lobes and the sinuses. A method of warping distorted areas of the
image and correcting unwarped areas based on the computed degree of distortion is
currently used to correct this problem [JMS01].
In the formation of spatial activation maps from EPIs, small head movements
23
made by the patient during successive EPI acquisitions can cause significant errors in
the statistical analysis. This problem has been largely addressed by the implementation of registration algorithms that register series of EPIs before performing statistical
parameter mapping. However, this problem is substantially more simple than EPI to
MRI registration because both the floating and reference images considered are of the
same modality. These images will be more similar than an MRI and its corresponding
EPI, and consequently easier to bring into proper spatial orientation.
And finally, a phenomenon called the Nyquist ghost, in which shadows of the image
appear surrounding the central image, can occur. This too is currently correctable
[JC99}.
2.3
Spatial Map Formation
In fMRI analysis a large number of processing stages are required before the final
activation results are obtained. Although no standard analysis protocol is universally accepted, the SPM software package is most widely used [FHP+96]. A typical
sequence of analysis steps for a single-session fMRI experiment is:
1. Acquire and reconstruct the individual EP images
2. Phase-correct all time series for variations in timings of scanning slices within
the volume scan time.
3. Apply motion correction to correct for head motion.
4. Spatially smooth the data to increase SNR and precondition later statistics.
5. Filter each voxel's (analog of a pixel in three dimensions) time series, to remove
slow temporal drifts and high frequency noise.
6. Perform the statistical analysis (creating a statistical parametric map, or SPM)
7. Threshold the SPM to find the "significant activated regions".
The result is a map showing the voxels or clusters of voxels indicated to have been
activated during the fMRI study. These activation maps are often then subjected to
higher level analysis, such as combining the low resolution results across a group of
subjects [HHH01].
24
To combine EPIs effectively with other imaging modalities, an accurate linear or
nonlinear (warping) registration method is required. The transformations involved in
linear registration (rigid body or affine) will be described in Chapter 3. Registration is
the main means for combining low resolution statistical images with a high resolution
structural image, for combining statistical results across several subjects in a group,
or for transforming the results into a standard coordinate system. For best results, the
EPIs should be corrected for geometric distortion that occurred during the functional
MRI experiment, prior to registration. In general, the fact that the spatial activation
maps produced by fMRI studies do not have particularly good spatial resolution or
contrast makes accurately registering these images with standard anatomical images
a challenging problem [HHH01].
The solution to this problem proposed by this thesis is the methodology for performing structural registration of EPIs to MRIs, which will be presented in Chapters
4 and 5.
25
26
Chapter 3
Image Registration
Information from different imaging modalities acquired over the course of clinical
treatment is often complementary, and more useful to physicians if provided in an
integrated form. An important first step in this integration process is to bring the
images into spatial alignment through a process referred to as registration [MV98].
At its most abstract, a registration procedure consists of a data integration problem to be addressed, a registration paradigm, and an optimization procedure [MV98].
In this chapter we will introduce the basic methodologies used in developing image
registration algorithms.
3.1
Overview of Registration Algorithms
Registration algorithms seek to bring two image sets into spatial alignment. This task
intrinsically requires both the ability to describe different spatial orientations of these
sets relative to one another (referred to as poses) and a means by which to assess
one orientation being better or worse than another (often called cost or objective
functions).
Figure 3-2 depicts the basic elements of an intensity-based registration algorithm.
The process of bringing image sets into spatial alignment entails searching the space of
positions and orientations (poses) for an orientation that minimizes the cost function.
This pose represents the best possible spatial alignment of the two image sets given
27
4. Domain Transformation
1. Dimensionality
(a) Local (subsection of image)
(a) Spatial Dimensions
(b) Global (entire image)
i. 2D-2D
ii. 2D-3D
iii. 3D-3D
(b) Temporal Dimensions - spatial
dimensions recorded at different
time points
i. 2D-2D
ii. 2D-3D
iii. 3D-3D
2. Nature of Registration Basis
5. Interaction
(a) Interactive
(b) Semi-automatic
(c) automatic
6. Optimization Procedure
(a) Parameters
data)
(a) Extrinsic
computed
(from
(b) Parameters searched for (seeking
optimum)
i. Invasive
ii. Non-Invasive
(b) Intrinsic
7. Modalities Involved
i. Landgark Based
(a) Monomodal
A. Anatomical
(b) Multimodal
B. Geometrical
ii. Segmentation Based
(c) Modality to model
A. Rigid Models (points,
(d) Patient to modality
curves, surfaces)
models
B. Deformable
8. Subject
(snakes, nets)
(a) Intrasubject
iii. Voxel Property Based (Intensity Based)
(b) Intersubject
A. Reduction to scalars/vectors
(c) Atlas
B. using frll image content
9. Object
(c) Non-image Based
i. Calibrated coordinate systems
(a) Head
(b) Thorax
(c) Abdomen
3. Nature of Transformation
(a) Rigid
(d) Pelvis and Perineum
(b) Affine
(e) Limbs
(c) Projective
(f) Spine and Vertebrae
(d) Curved
Figure 3-1: Nine-tiered classification scheme of registration methods presented by Maintz et al. [MV98}. The problem statement, registration paradigm,
and optimization procedure provide a unique classification of any registration algorithm. Items from categories 7,8, and 9 describe problem statements. Categories 2,
3, 4, and 5 list items that can describe registration paradigms. Lastly, category 6 lists
optimization procedures. These divisions, however, are in no way absolute [MV98}.
28
Starting
Pose
Floating Image
Reference Image
Excellent
3or
New Pose
of Floating
image/
Spatial
Alignment
Score
Rigid Transformation
of Floating Image
Rotation
Translation
Figure 3-2: Overview of Rigid, Intensity Based Image Registration algorithm. Of the two image sets to be registered, one is designated the floating set,
and the other the reference set. The two sets are spatially oriented to one another
in an initial pose. A scoring metric is then used to assess the quality of the spatial
alignment of the two image sets. Rigid transformations (translations and rotations)
can then be applied to the floating image to improve the spatial alignment scoring.
29
the cost function. To find this pose, the cost function is evaluated on the two image
sets in a variety of orientations. To explore the space of poses, one image set is held
stationary (the reference set) while the orientation of the other set (the floating set)
is altered, resulting in a new pose. This act of changing the spatial orientation of a
single image set is called a transformation.
Ultimately, the registration algorithm seeks the transformation T over the space
of allowable transformations ST of the floating image that minimizes cost function C
[JBBS02].
T = arg min C(Y, T(X))
(3.1)
TEST
3.2
Transformations
The space of allowable transformations is one of the defining features of registration
algorithms (See Figure 3-1 for classification details). A common choice lies between
rigid (where the image sets are treated as rigid bodies) and non-rigid (where images
can be deformed) transformations. As the thesis title indicates, only rigid transformations are used here for the EPI to MRI registrations, and so only these techniques
will be discussed. The reader is directed to Hajnal et al. [HHH01] for a detailed
discussion of the full range of possible transformations.
3.2.1
Rigid Transformations
There are only two kinds of rigid transformations: 1) translation and 2) rotation
of images. The dimensionality of the images being registered determines how may
parameters along which rigid transformations can be performed. The parameters for
the allowed rotations and translations for the floating image set define the space of
spatial orientations (i.e., poses) between the image sets.
The formula for rigid 2D translation is:
=1 XI
T
X2
X2
30
+
[1
D2
(3.2)
The formula for rigid 2D rotations is
T(x) =
cos 0 - sin
sin0
1x
(3.3)
cos0
where x is the vector of x,y coordinate pairs, and T(x) is the new x,y position for
the coordinate having just been transformed.
To apply both rotations and translations, the following equation is used:
T(x)=
cos 0 - sin 0
x+D
(3.4)
cos0
sin0
Rigid transformations performed on 2D images have the caveat of requiring tight
control of the image geometry for both the floating and reference image sets. Scaling
of images may also be required [HHH01].
3.2.2
Image Dimensionality and Degrees of Freedom
Once the rigidity of the transformations has been chosen, the dimensionality of the
images being registered dictates the degrees of freedom along which those transformations can be applied. As Figure 3-1 shows, there are three major dimensionality
combinations used for registration experiments: 2D-2D, 2D-3D, and 3D-3D. 2D refers
to images of only 1 plane, which are mathematically represented as a two-dimensional
matrix of intensities. 3D refers to images that present intensities for a volume, and
so are represented by a three-dimensional matrix of intensities.
The highest dimensionality of the image sets being registered determines the degrees of freedom that can be applied to the floating image set. For example, if the
reference set is 3D, and the floating set is 2D, then the floating image will have 6
degrees of freedom for rigid transformations - 3 translational, and 3 rotational degrees. Despite being only two-dimensional (and so inherently having only 3 degrees
of freedom - 2 translational and 1 rotational), the floating set can be transformed
across all 6 degrees of freedom within the 3D coordinate space of the reference image.
This thesis project entailed working with two-dimensional images, and so trans31
formations relevant to 2D-2D registration will be presented. However, some threedimensional data was considered by exploring registration of sequences of 2D slices
which when considered as a unit represent a 3D structure (see Section 5-10).
Clinically useful systems usually require some degree of 3D data, and so future
work will be aimed at exploring the 3D applications of the 2D technique developed
in this thesis. Again, a full discussion of 2D-3D and 3D-3D registration techniques is
presented in Hajnal et al. [HHH01].
3.3
Objective Functions
A variety of cost functions have been proposed for image registration problems. A
large comparative study of different registration methods indicates that intensitybased cost functions are more accurate and reliable than geometrically based ones
[Wa97]. Ultimately, the objective function is tailored to the image modality domain
that the registration algorithm seeks to assess.
For most registration algorithms, the space of transformations searched to minimize the objective function is defined as sets of parameter values for allowed transformations along the degrees of freedom defined for the floating image set.
3.3.1
Joint Entropy and Mutual Information
When using MI or JE as an objective function, the spatial orientation of the floating
set that maximizes MI or minimizes JE will be selected.
Under MI, the correct
pose returns a maximal value because it represents the point at which knowledge
of one image maximally explains the intensity information of the other. Under JE,
the pose providing correct alignment yields a minimum JE score, because under this
orientation, the joint variation in intensity between the images is at its minimum
[CT91}.
32
The definitions of MI and JE are:
MI =
P(ii,
,
i2) 109
il~i2
JE =
-
j
P,(i
Po(ii)PoT(i2)
1, i 2 ) log
Po(i , i 2 )
il,i2
where P7)(ii, i 2 ) is the pose-dependent joint PMF of the image intensities and PFO(ii)
and PoT(i 2 ) are the marginal distributions.
3.3.2
Kullback Leibler Distance
The Kullback-Leibler distance calculates how closely an estimated probability represents a true probability distribution. It can be considered as a kind of a distance
between the two probability densities, although it is not a real distance measure because it does not satisfy the triangle inequality and is not symmetric (i.e., changing
the PMF used as Po' and P will change the resulting KLD [CT91J). KLD is an important concept in quantum information theory, as well as statistical mechanics and
other areas [Qia00].
D(Pj|P|)
=
E PF(ii,i2 ) log
,'.
(3.5)
P(ii,i 2 )
i02
Once joint histograms are obtained for an image pair in a fixed alignment, the
probability mass functions can be estimated from the data by normalizing the histograms. In the algorithm presented in this thesis, KLD is used to determine the
difference between the observed PMF of the current spatial alignment of images and
the PMF for the approved alignment of the training pair.
3.3.3
Histogramming
A histogram of intensity values is helpful for summarizing the pixels that make up
a grayscale image. A histogram is simply a plot of the frequency of different pixel
intensities (from an image) plotted against those intensities.
33
Joint Histogramming
A joint histogram is a counting of the co-occurrence of intensity values for pixels from
two separate images in a current spatial alignment.
Histograms are made into probability mass functions by normalization.
3.4
Optimization Methods
As mentioned earlier in this chapter, the goal of a registration algorithm is to find the
transformation T(x) that minimizes a cost function. A number of methods are available for searching the space of transformations, such as Powell's method or gradient
descent [Pre92J. In this thesis, Downhill Simplex was employed.
3.4.1
Downhill Simplex
A simplex is a geometrical figure consisting, in N dimensions, of N+1 points (or vertices) and all of their interconnecting line segments, polygonal faces, etc. Therefore,
in two dimensions, a simplex is a triangle. In three dimensions it is a tetrahedron,
(but not necessarily a regular tetrahedron) [NM65].
In 1965 Nelder and Mead [NM65} reported a sequential simplex with variable
size by introducing two modifications into the original algorithm of Spendley et al.
[SHH62}. These two modifications lead the simplex to expand in directions that are
favorable and to contract in directions that are unfavorable, which allows the figure to
accelerate its progress toward a region of an optimum and can then effectively reduce
its search space until the extremum is located within desired accuracy.
Using sequential simplex to fit nonlinear models has the advantages of being compact in size, relatively independent from starting point parameters, and having no
need for derivative calculation with respect to parameters. However, as with other
methods for fitting a model with nonlinear parameters, the sequential simplex requires
initial estimates of those parameters [RB002].
34
t2
Simplex
Search Space
Uh.
0.2
0.1
0.2
'03
0.4
1-.5
06
0.7
Figure 3-3: Downhill Simplex - map of a 3D search space with a minimum
being traversed by a simplex. A simplex is a geometrical figure consisting, in
2 dimensions, of 3 points (a triangle) and all of their interconnecting line segments.
(In N dimensional space, a simplex is of N+1 points, including all line segments,
polygonal faces, etc). In the downhill simplex method, the simplex is used to search
a multi-parametered search space that is non-linear in its parameters for a global
minimum.
35
3.4.2
Image Smoothing
It is common in registration problems to employ a coarse-to-fine resolution search
strategy to avoid converging local minima for objective functions (See work by Lucas
et al for an example [LK81]). In this approach, objective functions are assessed of
copies of the original images that have been smoothed (weighted samples of neighboring intensities are used to replace intensity values). To a point, smoothing the images
in this manner should reduce the number of local minima (or maxima, depending
upon the objective function being used) that can cause an optimization method to
converge in the incorrect location. Often, an optimization function is evaluated on
the objective function using the most smoothed copies of the images to be registered. Once this procedure converges, the parameters on which it stops are used as
the starting point on another run of the optimization procedure, this time using a
less smoothed set of copies of the original images. This process is repeated until the
optimization procedure is performed using the original images and a starting pose
provided by the previous run of the optimization procedure.
36
Chapter 4
Methods
In Chapter 1, the motivation for MR to fMR data registration was introduced. The
fundamental relationships between MR, EPI, and fMR data were next presented in
Chapter 2 and the idea of EPI as a proxy for registering fMR activation data to
MRIs was introduced. The general specifics of registration algorithms were then
described in Chapter 3. In this chapter we present methodology for implementing a
registration algorithm for EPI and MRI using KLD as the objective function. Means
for validating this algorithm and for evaluating its efficacy against the widely used
objective functions JE and MI are also presented.
4.1
Overview of Registration Experiments
In a training operation, an EPI and its corresponding MRI are registered by an expert,
blurred to 4 levels, and joint intensity PMFs are computed for all 5 of the EPI-MRI
pairs. These PMFs will be referred to as Aligned Joint Intensity PMFs (AJPs). In
the registration phase, a test EPI-MRI pair is blurred to the same four levels that
were used on the training pair. Then, starting with the most blurred image (blur level
4 in the figures), the algorithm searches for a rigid transformation of the test pair's
EPI that results in a test pair joint intensity PMF that has a minimal KLD from the
AJP for that blur level. Once this pose is found, the algorithm repeats this search on
the test pair's EPI blurred to the highest blur level that has not yet been searched,
37
using the final pose from the previous search as the start pose. The search continues
until a pose is found to minimize the KLD between the test pair's PMF and the AJP
of the non-blurred training pair. This final pose is returned as the result.
4.1.1
Image Preparation
All image data in our experiments was acquired on a GE 1.5 Tesla Signa system
equipped with the HORIZON hardware/software package. The MRI data in our
experiments is a standard gradient echo (SPGR) scan that consists of 124 1.5mm
thick slices, FOV 24cm, image resolution 256 X 256 pixels. The EPI data sets contain
21 contiguous 7mm-slice images. The functional images are acquired in an auditory
experiment using the EPI-BOLD pulse sequence with the following set of parameters:
TE = 50msec, TR = 3sec, FOV 24cm, image resolution = 64 x 64 pixels. EPI and
MRI volumes were rigidly aligned for good visual agreement throughout, the EPI
volume was re-sampled, using tri-linear interpolation, into the lattice of the MRI
data, and corresponding 256 X 256 slice pairs were generated to be used as 2D test
and training pairs.
4.1.2
Blurring the Images
Four rotationally symmetric Gaussian low-pass filters of the following sizes and standard deviations (a) were created to blur the original images: Level 1 20x20 pixels, a:
3, Level 2 40x40 pixels, a: 5, Level 3 80x80 pixels, a :8, Level 4 100x100 pixels, a:10.
4.1.3
Image Transformations
All transformations of the EPI images were performed using a rigid transformation
model, with rotations occurring around the center of the image (see Equation 3.3).
38
Figure 4-1: Corresponding MR (left) and Echo-Planar (right) Image Pairs.
4.1.4
Histogramming to Estimate Joint Intensities
For the KLD objective function, a histogram was computed on an image pair by
rastering over the EPI and MRI simultaneously and counting the co-occurrence of
intensities in regions where the images overlapped. The co-occurring pixel intensity
i from the EPI and
matrix.
j
from the MRI led to incrementing bucket (i,j) in a 256 x 256
The matrix was then normalized to convert the histogram into a PMF.
Because it was possible for some buckets to have counts of zero after normalization,
Figure 4-2: (From Left to Right)
Increasing strength blur applied to an
Echo-planar image.
39
an
value was added to all buckets in the matrix before normalization. Varying
( between 1x10- 6 and 1x10
23
resulted in no significant difference in the results of
X probing experiments (see Objective Function Probing Experiments section). The
experiments presented in this paper used an
of 1x10- 2 3 .
The histogramming techniques used for KLD and the JE or MI probes differed
slightly. The JE and MI objective functions were computed in the space of the fixed
image, with the moving image zero padded as necessary. Histograms for the MI and
JE method were calculated with and without interpolation, respectively. For the
KLD-based method, histograms were calculated only in the area of overlap between
the floating and reference images.
4.1.5
Kullback-Leibler Distance (KLD)
As described in Section 3.3.2 - the the Kullback-Leibler distance uses the learned P
and the pose-dependent observed P7 joint intensity PMFs to measure the difference
between two PMFs using the following equation: [CT91, Kul68]:
D(Pfl|I)
4.1.6
=
E PF(i
1 ,i 2 )log
.(.
P(i, i2 )
(4.1)
Interpolation vs. Rounding
Histograms were generated by applying a standard rounding function to image intensities. For comparison, we also implemented an interpolation function that spread
counts of intensity co-occurrences over the buckets corresponding to the combinations
of floors and ceilings of each intensity value.
4.1.7
Optirniiation of the Transformation T - Downhill Simplex Seirches
The goal of the KLD based legistration is to find the optimal transformation T by
minimizing the difference between the observed P and the expected P intensity
PMFs, 'T = arg min D(P'
T
IIP).
For the registration experiments presented, the value
40
of KLD or JE was minimized using the Nelder-Mead Downhill Simplex (direct search)
method with a multi-resolution strategy [NM65, Pre92}. This approach does not use
derivatives. One hundred start poses were randomly chosen, where the X, Y, and 0
were varied within boundaries implied by Figures 5-1, 5-2, and 5-3 to cause most,
but not all, downhill searches using KLD and JE to succeed. The parameters ranged
from: -45 to 55 for X, -25 to 35 for Y, and -50' to 130* for 9. The KLD searches
were then carried out as described in the overview part of the Methods section. The
JE searches were performed in the same manner, except that there was no training
phase for JE, and so the images used by KLD as a training set were not used by JE.
4.2
Objective Function Probing
Two EPI-MRI slice pairs were selected from a pair of corresponding volumetric images
to be used as training and test pairs on which to perform rigid transformation probes
(see Figure 4-1). Both image pairs were blurred to the four levels described earlier,
and the AJPs for the blurred and non-blurred training pairs were computed. The
probes then consisted of performing the following for the image pairs at all levels of
blurring: transforming the test pair's EPI in fixed steps along a single axis (X, Y, or
9), computing the joint intensity PMF for the transformed EPI and its corresponding
MRI blurred to the same level, and then computing the KLD between this PMF and
the AJP for that blur level.
The X probe shifted the test pair EPI along the X axis from -65 to 65 pixels in
one-pixel increments, the Y probe performed the same shift along the Y axis, and the
9 probe rotated the test EPI from -180* to 180' in two-degree increments.
4.2.1
Intensity Scaling Image Pairs from Different Acquisitions
To account for inter-volumetric intensity variations for training and test pairs (when
the training and test EPI-MRI pairs were sampled from different volumes), experi41
ments were performed in which an image-dependent scaling factor was applied to all
intensities during histogramming. The factor consisted of the number of buckets used
divided by the intensity of the brightest pixel occurring in that image.
4.2.2
Pseudo 3D Experiment - Histogramming over Multiple
Slices
For these experiments, X probes were performed as previously described, with the
exception that every step requiring a single image pair used a set of images. Registered EPI-MRI pairs from a single time point were divided into two consecutive sets:
anterior and posterior halves. Two probes were then performed, each in which one
set was used for training and the other as a test pair. Generating the training AJPs
entailed blurring all of the images in the training set, and then histogramming over all
intensity pairs occurring in areas of image pair overlap. Computing the test pair PMF
required transforming all EPIs of the test set and then histogramming cumulatively
as described for generating the AJPs. (See Figure 5-10).
42
Chapter 5
Registration Experiments
In this chapter we represent the results of the experiments and evaluations methodologies described in Chapter 4. Probing experiments are presented showing how the
KLD objective function varies over a search space, and how MI and JE based methods
compare. Variations on parameters involved in KLD evaluation are also presented,
to examine potential implementation caveats to using the method. Next the results
of an optimization experiment, in which JE and KLD are separately used to find the
proper spatial alignment for an EPI-MRI pair are presented. And lastly, we describe
the results of pseudo 3D experiments, designed to assess the viability of KLD based
registration for 3D registration applications.
5.1
Probe Comparisons: KLD, JE, MI
Figures 5-1, 5-2, and 5-3 show the results of Joint Entropy [MCV+97], Mutual Information [Uki02, WPH+96], and KLD probes of X, Y or 0 on a test pair of images.
The plots show the variations in the objective functions for each transformation of the
EPI of the test pair for each of the blur levels. Negated MI is presented to facilitate
comparison with JE and KLD, which both show minima near proper registration.
The transformation resulting in the pose chosen by an expert is indicated by the
vertical bar plus
(+)
symbols. The square on each line indicates the pose resulting
in the lowest value for the objective function. Joint Entropy showed significant bias
43
on the Y and 6 probes, while MI showed significant bias on the Y probe.
5.2
Objective Function Downhill Simplex Searches
Figure 5-4 summarizes the results of 100 downhill simplex searches using KLD and
JE, with the final pose error shown as a function of the start pose error. The start and
final pose errors were defined as /ErrorX + Error + Errors, where the error terms
are the distance of the pose in X, Y, and 6 respectively from the correct registration
pose. These searches began with the test image pair blurred at the highest blur level
and the EPI of the test pair transformed to the start pose. For a majority of the
starting points, the KLD objective function returned a pose with a lower final pose
error than the JE based method. A final pose error threshold was selected to classify
searches into successful and unsuccessful. This threshold selected the best-scoring
significant groups of results. The average final pose for successful searches was then
calculated. On average, successful KLD trials converged to pose parameters closer
to those chosen by the expert (x=0, y=0,e=O) and showed smaller variation than
did the JE trials. JE was significantly biased in the Y axis, and showed its largest
variation in the Y axis pose parameter. Figure 5-5 shows the overlay of the test EPI
transformed to the pose chosen by the expert with its corresponding MRI (left image),
the overlay for the test EPI transformed to the average pose parameters suggested
by KLD (middle image) and by JE (right image). Visibly, the JE image is most
discrepant from the expert's pose, with its registration aligning some brain activation
to the scalp. The KLD based method showed a pose similar to the one chosen by the
expert.
5.3
Effect of Varying Epsilon
values are added uniformly to all buckets before histograms are normalized. X
probes were performed using
at 1x10- 6 , 1x10
ences were noted.
44
13
and 1x10-
23 .
No significant differ-
Kuliback-Leibler Distance over Blurs for TrainO9 & Test12
25
10
5
**
-
Bur1
(Lmurs)
NoBSw
.
-t-
--
GoncIT.Pas.
Pt dfLoestStat
+
-
o
20
0
40
60
80
120
100
140
Probe in X from -65 to 65 in steps of 1 pixel
Joint Entropy over Blurs for 12
2.1
.T
2.05f
2
-+
-
190
--
+
E
1.85
)
-
+
Our 2oh
+
1.8
+
1.75-
0
20
40
CaneT
+
o
80
s0
-
Ow41(Ht.s
4
+lur
Pt
Poe
of Lowet St it
100
120
Probe in X from -65 to 65 in steps of 1 pixel
- Mutual Information over Blurs for 12
-04
+
-0.5
r--0.6
+ /--Bu +3
+~
+2
-------
-----------
-0.9
u 4(H1ghst)
--
-1.1
.
+
-1.2
-+-+
13
0
20
40
60
80
100
No
I (Lwst)
BH~
.
COcsn Pose
Pt of Lowesd Stat
120
.
140
Probe in X from -65 to 65 in steps of 1 pixel
Figure 5-1: X Probes - (from top to bottom) KLD, JE, MI. JE shows
significant bias relative to KLD and MI.
45
Kulback-Leibler Distance over Blurs for TrainO9 & Testi 2
Is
16
+
4
1,4
+
j512
+
+
-7Ji
+
+
36
Mir 4(Highest
--- BLr2
Blur 2
-+-...
-.
*e+
-
Lr 1 (Lowest)
No Blur
Correct Pose
a Pt d4 Lowest Stat
+
0
20
40
60
80
100
120
140
Probe in Y from -65 to 65 in steps of 1 pixel
Joint Entropy over Blurs for 12
2.1
2.05
2
U. 1.96
V
19
-31.
+
1.85
Bl ur 2
Blur 1(Lowest)
-+
1.8
-+- No Bur
+ Correct Pose
o Pt of Lowest S
+
1.75' 0
20
40
80
60
ir 4 (Hgiest
--
100
120
140
Probe in Y from -65 to 65 in steps of 1 pixel
- Mutual Information over Blurs for 12
-0.6
+
+
+
**.
0
-0.8
+
E
+
+
+
+
0.
.(ows+
-
.
B'r
+
N
u
+
Bluir 42H~et
+
-1.2
Bkjr1 (Lowest)
No Bitt
+++
+
c
0
20
40
60
80
100
Correct Pose
Pt of Lowest Stat
120
140
Probe in Y from -65 to 65 in steps of 1 pixel
Figure 5-2: Y Probes - (from top to bottom) KLD, JE, MI. Both MI and
JE show significant bias relative to KLD.
46
Kullback-Leibler Distance over Blurs for Train09 &Test12
12
+
0
+
+1
+1
+1
Mir 4 (Hqghst
BNur3
4
Blur 2
~
0
+
+
BaluI (Lowest)
NoaOL,
CiedPoe
o3 Pt ofLowoest S
--4+
20
40
so
80
too
120 140 1eo ISO
Probe in Theta from -Pi to Pi in steps of 2pi/180
Joint Entropy over Blurs for 12
200
2
1.95
'E
W
+
1.9
.5
1.95
+-
Bw 4(Highse
+
1.8
-O
ur 2
Bkur I
(Lowest)
-+- No Our
-
135
1 75
20
40
so
Coret Pose
Pt of Lowest St
+
+3
T
8o
100o
120
140
18a
160
200
Probe in Theta from -Pi to Pi in steps of 2pV180
-Mutual Information over Blurs for 12
D.
E
1
-05
1
1
1
+
-0.6
+
Noe
'I
0-0.7
i;
E
-0.9
-1
-
-1.1
--
*
-1.2
131
0
i
20
i
40
i
80
-i
80
i
100
i
120
BluBk34 (H9M
Bar 2
+ Br1 (Lowe
No B
Coned PoS
o Pt of Lowest Ssat
- +
+
i
14O
i
160
1
180
200
Probe in Theta from -Pi to Pi in steps of 2pi/180
Figure 5-3: 6 Probes - (from top to bottom) KLD, JE, MI. JE shows significant bias relative to KLD and MI.
47
DownHill Simplex Results for KLD & JE for 100 Random Start Points
1
1
300
A
*
Kullback-Leibler Distance Method
Joint Entropy Method
Success Cut-Off Line
250-
*
Y
0
X
Error stdev
- K-L Distance 1.238 2.558 0.756
Joint Entropy 1.877 6.493 1.890
200
0
LI 150C.
0
Average Pose for Successful Trials
Y
e
X
JE: -2.414 12.833 -0.009
KLD: -0.751 1.451 -0.033
Standard Deviation
0
X
Y
2.093 6.493 0.077
1.334 3.428 0.027
*
100 -*-
50-
0__
0
10
20
30
40
50
60
70
80
Start Error
Figure 5-4: Results of KLD and joint entropy-driven downhill simplex
searches for a test image given 100 random start poses for the test EPI.
Trials were classified as being successful if the final pose error fell under a threshold.
The KLD based successful searches showed average pose parameters closer to the
expert's chosen pose than the joint entropy based method
48
Figure 5-5: Significant EPI intensities overlaid on corresponding MRI. (Left)
Hand registration of the EPI-MRI pair by an expert, (middle) EPI transformed to
the average final pose parameters of successful KLD method trials for downhill simplex, and (right) EPI transformed according to the average final pose parameters of
successful Joint Entropy based method.
5.4
Effect of Different Training Pairs on KLD
We examined the effect of choice of training pair on KLD X probes. One EPI-MRI
pair was designated as a test pair, and then two KLD X probes were performed, each
using a different EPI-MRI pair (all sampled from the same volumetric data) as the
training pair. As Figure 5-7 depicts, both probes showed a minimum near the pose
selected by the expert.
5.5
KLD Using Training and Test Pairs from Different Volumetric Images
KLD X probes using a test pair from a similar but different volume of EPI and MR
data than the training pair were performed directly and with the use of an imagedependent scaling factor. Figure 5-8 depicts the results. Both probes yielded minima
near the expert's pose for the non-blurred images, and showed minimal variations of
minima for KLD at the four blur levels. The non-scaled probe showed more minima
at the expert's pose, but the probe that scaled the image intensities showed less
variation in the position of its minima. The final pose of either method would likely
49
Kuilback-Lebler Distance over Blurs for TrainEp212 &TestO9 Pairs
I14
-
-
*
IT
Bur4 (HLghest)
+.+
NoBLur
Correct Pose
Pt of Lowet Stst
n**-+
+
o
0
20
60
40
60
100
0
120
Probe in X from -65 to 65 in steps of 1 pixel
Kullback-Leibler Distance over Blurs for TrainEp1 12 & TestO9 Pairs
16,
le
0 14
812
W10
Blur41(Liheet)
-
2
-
'
.
.-.+
o
01
0
1
20
i
40
i
60
80
100
No Blur
Correct Pose
Pt Lweet
tt
120
140
Probe in X from -65 to 65 in steps of 1 pixel
Kulback-Leibler Distance over Blurs for Train12 &TestO9
+
25
I20
-
,/-
-+
~15
to
10
B
.3
-
*-
BLr 4 (HIghest)
-Bur
3
*ur 2
Blur I (Lowes)
+No Bur
+ Coto Pose
o Pt at Wee Sta
0
20
40
80
60
100
120
140
Probe in X from -65 to 65 in steps of 1 pixel
Figure 5-6: Effect of Varying ( on X Probes - (Top) 6 = 1x10- 6 , (Middle) 6
= 1x1013, and (Bottom) 6 = lxlO1 3 . No significant performance changes were
seen varying the 6 values within this range.
50
Kulback-Leibler Distance over Blurs for TrainO9 &Test12
25
+C
0
.915
0
-+--
tu 4 (HIghest)
5
-
.
+**
++
o
20
0
40
60
80
100
Iu I (Lowest)
Z
Pt
Pose
fLcwest Stgt
120
Probe in X from -65 to 65 in steps of 1 pixel
Kullback-Leibler Distance over Blurs for Train14 & Test12
140
22
20 -
18
0+
o+
16
+r
4+
014-+
12+
0 -
+
EM* 1
+
o
0
20
40
80
60
100
g5
CoredT Poe
Pt o Lowest Std
120
.
.
140
Probe in X from -65 to 65 in steps of 1 pixel
Figure 5-7: KLD X probes of EPI-MRJ Pair 12 using (top) Pair 9 as training
pair, and (bottom) Pair 14 as training pair. Both training pairs were sampled
from the same EPI-MRI volume set as pair 12.
51
agree with that of the expert.
5.6
Effects of Bucket Size and Interpolation on
KLD
Non-integer intensity values were encountered during histogramming, and were rounded
to allow bucketing. To see if rounding was leading to loss of important information,
KLD X probes using a bi-linear interpolation of the intensity value were performed.
As can be seen in Figure 5-9, the interpolation method did not show significant improvement of the objective function. Similarly, increases in bucket size from 32 to
256 showed relatively small improvements in the smoothness or minima of the KLD
X probes.
5.7
Combining Histograms across Multiple Slices
- Pseudo 3D X Probes
Figure 5-10 shows the two KLD X pseudo 3D probes performed. The left plot used the
posterior half of the image set for training, and the anterior as the test set. The right
plot used the opposite configuration. In general, both plots show much less variation
in minima at all blur levels than the X probes performed using single training and
test pairs. The right plot showed more uniform minima at the correct pose than the
probe depicted on the left.
52
Kuliback-Leibler Distance over Blurs for Trainl2 &TestO9
30
25-,,+
+
0
0+
0+
~
+
15
BIw 1 (Lowest)
No Blw
+ Correct Pose
o Pt of Lowest Stat
f+
- .-
20
0
40
80
60
w
100
1
120
140
Probe in X from -65 to 65 in steps of 1 pixel
Kuliback-Leibler Distance over Blurs for Train 12 & TestO9-4
30
+
+
+
+
25
+
+
0
1
5
+
C+
(9
+
+
~20>
o
+
+
+
+
0
+
+
++k
+
+3i
5-
Hget
Bj
--
+-4
+
a
0
20
40
60
80
100
1i(Lowest)
No SIr
Comc Pose
Pt of Lowest Stat
120
140
Probe in X from -65 to 65 in steps of I pixel
Figure 5-8: Comparison of X Probes (left) using training and test pairs from
the same volume of Echoplanar and MR data, and (right) using training
and test pairs where each set was sampled from a different EP-MR volume.
53
Kullback-Leibler Distance over Blurs for Trainb3212 &TestO9
K
Kullback-Leibler Distance over Blurs for Trainb3212BL &TestO9
+
12
10
+-
C.)
S10
++2
(D1
C.)
=4
-
2
-
'$
+
B66r166,r(Lowest)
.
'
o
20
60
40
60
100
140
Probe in X from -65 to 65 in steps of 1 pixel
Kullback-Leibler Distance over Blurs for Trainb12812 & TestO9
0
6
40
20
.**+
\
.
No B6r
+
+'....-'
Cowedt Pose
Pt ofLosegtStat
120
Blr 1(Lowesl)
**.....-''
-
+No
+
~lu
ghest)
B
+
Coosot Pose
Pt of Lowest Stat
o
80
120
100
140
Probe in X from -65 to 65 in steps of 1 pixel
Kullback-Leibler Distance over Blurs for Trainb12812BL &TestO9
20,
1
1
-+
+
016
*20
C
C
659
015
512
-+
914
0
0
0 20
60
6
60
60
10
14
0 20
=6
75
*-
.a
--
B6ur4
0
60
60
10
12
(Highest)
.- Bur4
(HIghest)
lurI
(Lowest)
4
-.. +
,
N,-,
'
20
40
W0
W
(Lowest)
'B9ur
100
120
140
Probe in X from -65 to 65 in steps of 1 pixel
Kullbck-Leibler Distance over Blurs for Train12 & TestO9
-
'Q
2
Cowedt Pose
o~ Pt of Loeest Stet
+
.-
&
'0
'
+
+
0
0
20
W
40
W
No
Corred Pose
Pt of Lowest St
12D
100
1 40
Probe in X from -65 to 65 in steps of 1 pixel
Kullback-Leibler Distance over Blurs for Train12BL & TestO9
20
20
+-
C
20
*
0
*+
-+
*-+
0
+lr4(iset
-
-
20
40
60
--'
''. . \
'.,
.5
6, 1 (Loet)
+4 No 6ur
+ CooedPoe.
o Pt o stmt96gt
*
Probe in X from -85
B6r4(4h~ee4)
*.
60
160
to 65 an steps of l
126
+
+
..
.
***W'
14.0
0
pixel
20
40
60
-
+lr
s0
.
~
Blur+
oB
ur1(o
et
9 fLowest)Sa
1(
100
120
140
Probe in X from -65 to 65 in steps of 1 pixel
Figure 5-9: Effects of varying bucket size with (left column) and without
(right column) use of bi-linear interpolation in KLD based registration.
Probes in top row used 32 buckets, in middle row used 128 buckets, and in bottom
row used 256 buckets.
54
Kullback-Leibler Distance over Blurs for Train3D1 & Test3D2
12
+
C0
U
6
Z4
20
2
40
8
0
2i0
40
0
+
+:
-
1
+
+
o
80
W0
Our 12 (
No EMu
con'"c Pose
Pt
1ed)
of Lmme SWa
100
120
14
Probe in X from -65 to 65 in steps of 1 pixel
Kullback-Leibler Distance over Blurs for Train3D2 & Test3D1
+
2D
CS
015
0
+
Bw
-5
-
-..
+
s
,
..-*
-+
0
20
40
s0
80
100
+No
1w 4 (HIghest)
1w 1 (Lwet)
Owr
CcrrecdPe
a PtolLow..t Stat
120
140
Probe in X from -65 to 65 in steps of 1 pixel
Figure 5-10: Pseudo 3D Experiment Results - modified KLD X probes in
which all of the EPI-MRI pairs from one time point were divided into two
sets (anterior and posterior halves), with information from one set used
for training, and the other used as the test set. Intensity pair information from
all the image pairs in a set was combined to make the histograms for these probes.
Shown above are probes where the anterior half (top) and the posterior half (bottom)
of the image set were used.
55
56
Chapter 6
Discussion
This thesis has explored the registration of echoplanar to magnetic resonance images
using the Kullback-Leibler Distance. The algorithm presented leverages knowledge
of the joint pixel intensity distribution for a properly aligned MR-EP image pair to
register a second EP-MR pair. The Kullback-Leibler Distance was used to assess
the degree to which rigid transformatiohs of the EP image made the joint intensity
distribution for the test EPI-MRI pair more or less like the joint intensity distribution for the properly aligned training pair (AJP). The performance of the KLD based
method was then compared with the widely used Joint Entropy and Mutual Information functions. Probing experiments were performed to characterize the behavior of
these methods over the space of rigid transformations. Additionally, an optimization
experiment using downhill simplex was run comparing the poses to which JE and
KLD converged for a test EPI-MRI pair, with 100 randomly chosen starting points.
Next, a series of probing experiments were run to examine the effects of: 1)bucket
size, 2)epsilon values used in PMF formation, 3)use of different EPI-MRI pairs for
training where all images were sampled from the same volumetric EPI-MRI image
pair and 4) where the training and test pairs were sampled from different volumetric
images. And lastly, pseudo 3D probing experiments were carried out in which joint
intensity observations were made over sets of contiguous slices from corresponding
EP and MR volumes in place of single EP-MR slice pairs, for both the training and
test sets.
57
6.1
Observations
While MRI/EPI registration is a 3D-3D problem, in the present work, informative
results have been obtained from a series of 2D experiments on representative slices.
Clear shortcomings of the MI and JE objective functions were shown, especially in the
axial direction, which are unlikely to be ameliorated by 3D registration. Variations
on bucket size and the use of rounding or interpolation did not appear to significantly
degrade performance of the KLD objective function.
Similarly good results were
achieved when using images from two different acquisition sets as training and test
pairs to perform registration. The pseudo-3D experiments provide good indication
that the KLD approach will be useful for the full 3D-3D problem.
6.2
Future Work
The KLD technique needs to be assessed within a three-dimensional framework, where
probing experiments and searches are performed over all six degrees of freedom. The
influences of the various correction methods regularly applied to echoplanar images
(see Section 2.2) on the KLD method's performance need to be investigated as well.
58
Bibliography
[AKC86]
C. B. Ahn, J. H. Kim, and Z. H. Cho. High-speed spiral-scan echo planar
nmr imaging .1. IEEE Transactions on Medical Imaging, 5(1):2-7, 1986.
[AR03]
ACR and RCSNA.
2003.
Functional mr imaging (fmri), October 31, 1999
http://www.radiologyresource.org/content/functional-
mr.htm#Uses.
[Bis95
C. M Bishop. Neural networks for pattern recognition. Clarendon Press;
Oxford University Press, Oxford, New York, 1995.
[CCY+03]
H-M. Chan, A.C.S. Chung, S.C.H. Yu, A. Norbash, and W. M. Wells.
Multi-modal image registration by minimizing kullback-leibler distance
between expected and observed joint class histograms. In Chuck Dyer
Perona and Pietro, editors, Computer Vision and Pattern Recognition
(CVPR'03), Wisconsin, 2003. To Appear in CVPR 2003.
[CT91]
T. M. Cover and Joy A. Thomas. Elements of information theory. Wiley
series in telecommunications. Wiley, New York, 1991.
[CWNG02] A.C.S. Chung, W.M. Wells, A. Norbash, and W.E.L. Grimson. Multimodal image registration by minimizing kullback-leibler distance.
In
Takeyoshi Dohi Kikinis and Ron, editors, Medical Image Computing and
Computer-Assisted Intervention (MICCAI'02), volume 2488 of LNCS,
pages 525-532, Tokyo, 2002. Springer.
59
[FHP+96]
K. J. Friston, A. Holmes, J. B. Poline, C. J. Price, and C. D. Frith.
Detecting activations in pet and fmri: Levels of inference and power.
Neuroimage, 4(3):223-235, 1996.
[HBHH01I
D. L. Hill, P. G. Batchelor, M. Holden, and D. J. Hawkes. Medical image
registration. Phys Med Biol, 46(3):R1-45, 2001.
[HHH01]
Joseph V. Hajnal, D. J. Hawkes, and Derek L. G. Hill. Medical image
registration. CRC Press, Boca Raton, 2001.
[JBBSO2]
M. Jenkinson, P. Bannister, M. Brady, and Stephen. Smith. Improved
optimization for the robust and accurate linear registration and motion
correction of brain images. Neurolmage, 17:825-841, 2002.
[JC99]
P. Jezzard and S. Clare. Sources of distortions in functional mri data.
Human Brain Mapping, 8(2-3):80-85, 1999.
[JMS01]
Peter Jezzard, Paul M. Matthews, and Stephen M. Smith. Functional
MRI : an introduction to methods. Oxford University Press, Oxford
New York, 2001.
[KBC+92]
K. K. Kwong, J. W. Belliveau, D. A. Chesler, I. E. Goldberg, R. M.
Weisskoff, B. P. Poncelet, D. N. Kennedy, B. E. Hoppel, M. S. Cohen, R. Turner, H. M. Cheng, T. J. Brady, and B. R. Rosen. Dynamic
magnetic-resonance-imaging of human brain activity during primary sensory stimulation. Proceedings of the National Academy of Sciences of the
United States of America, 89(12):5675-5679, 1992.
[Kul68]
S. Kullback, editor. Information Theory and Statistics. Telecommunications and Signal Processing Series. Dover Publications, Inc, 1968.
[LK81]
B.D. Lucas and T Kanade. An iterative image registration technique with
an application to stereo vision. In Imaging Understanding Workshop,
pages 121-130, 1981.
60
[Man77]
P. Mansfield. Multi-planar image-formation using nmr spin echoes. Journal of Physics C-Solid State Physics, 10(3):L55-L58, 1977.
[MCV+97]
F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens.
Multimodality image registration by maximization of mutual information. Ieee Transactions on Medical Imaging, 16(2):187-198, 1997.
[MM87]
CH. Meyer and A. Macovski. Square spiral fast imaging:interleaving and
off-resonance effects. In Proc 6th Soc Magn Reson Med, volume 1, page
230, 1987.
[MV98]
J. Maintz and M. Viergever. A survey of medical image registration.
Medical Image Analysis, 2(1):1-36, 1998.
[NM65]
J. A. Nelder and R Mead. A simplex method for function minimization.
Computer Journal, 7:308-313, 1965.
[OL90]
S. Ogawa and T. M. Lee. Magnetic-resonance-imaging of blood-vessels
at high fields - invivo and invitro measurements and image simulation.
Magnetic Resonance in Medicine, 16(1):9-18, 1990.
[Pre92]
W. H. Press. Numerical recipes in C : the art of scientific computing.
Cambridge University Press, Cambridge Cambridgeshire ; New York,
2nd edition, 1992.
[QiaOO]
H Qian. Relative entropy: Free energy associated with equilibrium fluctuations and nonequilibrium deviations., 2000. http://xxx.lanl.gov/
abs/math-ph/0007010/.
[RB002]
E.G. Romero-Blanco and J.F. Ogilvie.
Optimization with sequential
simplex of variable size, July 2002 2002. http://www.mapleapps. com/
categories/mathematics/combinatorics/html/simplexopt
[SHH62]
.html.
W. Spendley, G. R. Hext, and F. R Himsworth. Sequential application
of simplex designs in optimisation and evolutionary operation. Technometrics, 4(4):441-461, 1962.
61
[Uki02]
S. Ukil. 2d rigid image registration by maximization of mutual information, 2002.
http://www.engineering.uiowa.edu/taip/248_s02_
solutions/soumik-ukil/hw3%/index.htm.
[VW97]
P. Viola and W. M. Wells. Alignment by maximization of mutual infor-
mation. InternationalJournal of Computer Vision, 24(2):137-154, 1997.
[Wa97]
J. West and et al.
Comparison and evaluation of retrospective inter-
modality brain images registration techniques. J. Computer Assist. To-
mogr., 17(4):554-566, 1997.
[WPH+96]
W M. Wells, Viola P., Atsumi H., Nakajima S., and Kikinis R. Multimodal volume registration by maximization of mutual information. Med-
ical Image Analysis, 1(1):35-52, 1996.
[WWY+99] C. G. Wible, W. M. Wells, S. S. Yoo, D. Kacher, R. Kikinis, F. Jolesz, and
R. W. McCarley. The registration of fmri and high resolution anatomical mri scans using mutual information. Schizophrenia Research, 36(1-
3):237-237, 1999.
62