A Robust System for Registration of 3D and 4D Image Data
Marc Kessler, *Charles Meyer, James Balter, and Daniel McShan
Department of Radiation Oncology and *Radiology, The University of Michigan Medical School, Ann Arbor, Michigan, 48109, USA
Abstract
We describe a system for handling a wide variety of clinical image registration problems for 3D and 4D image data. The core of
this system is a mutual information-based registration algorithm with support for geometric transformations from simple rotatetranslate to 3D and 4D deformations using thin-plate splines. An important feature of the system is the ability to apply one or more
3D intensity masks to limit the field-of-view of the data considered. These masks can be either simple geometric shapes or arbitrary
volumes derived from patient specific anatomy. These masks permit the removal of confounding or irrelevant data and can help
reduce the degrees of freedom required to obtain accurate registrations. Examples of the use of this system for several clinical sites
and are presented. Supported in part by NIH P01-CA59827and P01-CA87634-01
Keywords
Image Registration, Mutual Information, Thin-plate Splines, Intensity Masks, Piecewise-rigid, Piecewise-deformable
Introduction
Material and methods
Accurate registration of 3D image data of the brain is a solved
problem. Most commerical treatment planning systems provide
some manual or automated technique for rapid registration and
fusion of diagnostic data from MR or PET/SPECT with the
treatment planning CT. The assumption of a global rigid
transformation applies in most cases and the image data is
usually of sufficient resolution and extent to allow most
registration techniques to work well.
A detailed description of the underlying algorithms comprising
this system is described in [1] and [2]. Basically, the system
uses a Nelder-Mead simplex algorithm to drive the positions of
a set of control points to minimize the mutual information
between two image datasets. The location of the control points
are used to determine either an affine or thin-plate spline
transformation between the two image datasets. A scheduler is
used to support hierarchical optimization by successive
refinement of the image data and optimization parameters.
Outside the cranium, however, several factors may confound
accurate and fast image registration. The assumption of a
single global mapping from one image volume to another is
often violated. Even when tissue deformation is considered,
the use of a single mapping to transform the entire image
volume can result in a trade off of overall registration quality at
the expense of more accurate local registration, possibly in a
clinically important region. Increasing the degrees of freedom
of the transformation model to handle disparate local
deformations (or a mixture of rigid and deformable anatomy)
can lead to excessive calculation times and undesirable local
minima.
In order to handle more imaging situations while maintaining a
robust and simple approach to image registration, we have
extended our mutual information-based image registration
system to allow specification of one or more regions of local
anatomy that are rigid, approximately rigid, or locally
deformable and register the regions independently and then
combine the results when appropriate. Because of the
robustness of the mutual information metric to limited or
sparse data, accurate registration of the individual sub-regions
can be achieved. This approach has been applied successfully
in regions such as the prostate, lung, liver and spine. In this
presentation, details of these approach and clinical examples of
their use will be described.
The main enhancement described here is the use of intensity
masks to segment regions of the image data to be considered
separately or to be ignored. These masks can be either simple
geometric shapes or arbitrary volumes derived from patient
specific anatomy (Figure 1). Data from the different regions
are registered independently, potentially using different
transformation models (Figure 2). If desired, image data
outside the masked regions can be recombined using a
distance-weighted interpolation scheme (Figure 3).
Figure 1: a) Simple geometric masks b) anatomic-based mask.
Results and discussion
Several hundred clinical registrations have been performed
using this system. Many of these cases benefited from the use
of either simple or complex data masking. Benefits ranged
from straightforward computation savings because of the
reduced field-of-view to registrations that became feasible only
with the use of masking. Between these extremes are also
cases that could be registered using fewer degrees of freedom
(e.g., by ignoring deformations of unimportant tissues or
handling them in a piecewise-rigid manner) while maintaining
sufficient clinical accuracy. Registrations of 3D and 4D data
from different clinical sites are presented to illustrate the use
and results of this system.
Figure 2: Overall algorithm flow.
MR-CT Prostate Example
The prostate is an example of a fairly rigid anatomic structure
situated in a milieu of intentionally deformable or mobile
structures. While MR provides superior delineation of the
sensitive anatomic structures in and adjacent to the prostate,
imaging is usually performed on a curved rather than flat
tabletop. Not only does this distort the body shape, it can result
in rotation of the hip and femurs (Figure 4). Simple masking
of prostate to include a few millimeters of surrounding
connective tissue allows fast and accurate registration using
only a simple rotate-translate transformation [3].
Figure 3: Distance-weighted interpolation between piecewise-rigid
transformations for two simple objects. a) Two simple objects 1 and
2. b) Mask and distance field for object 1. c) Same for object 2. d)
objects with grid overlay. e) and f) object 1 held fixed and object 2
rotated clockwise by a few degrees. Points inside the objects are
transformed according to the separate transformation while the points
outside both objects are transformed according to the distanceweighted interpolation of the two transformations.
Simple rotate-translate transformations are computed using
three approximately corresponding sets of control points, full
affine transformations require at least 4 and thin-plate splines
transformations at least 5. In complex cases such as 4D or
cone-beam CT data of the lung, 20 to 30 points are used.
These control points are distributed throughout the organ
interior and across the surface. Using a subset of the points
and lower resolution image volumes in the early stages of the
registration can speed up the overall registration process.
Registration accuracy is determined qualitatively using visual
displays such as split screen, linked-cursor and contour overlay
displays. Quantitative registration accuracy is measured using
root-mean-squared deviations between sets of corresponding
anatomic points such as vessel and airway bifurcations.
Figure 4: MR-CT Prostate registration
MR-CT Liver Example
Most of the problems discussed above when imaging the
prostate with MR are also present and can be more severe
when imaging the liver. The liver is also less rigid and more
mobile than the prostate. Because of this and the complex
shape of the liver, the use of anatomic-based masks has been
necessary to achieve proper registrations. This is illustrated in
Figure 5. The MR shows the tumor volume much clearer than
the CT, but is acquired on curved table top and with much
different filling of the stomach. The kidneys are also displaced
relative to one another. Registration of the entire field-of-view
when the clinician was only interested in the boundary of the
tumor visualized using MR would have unnecessarily
comprised the accuracy of the registration in the region of
interest even if a full deformation model was applied,
especially since obvious sliding of adjacent tissues is present.
Figure 5: MR-CT Liver Registration. a) Native MR image. b) Native
CT image. c) Native MR with CT-defined liver outline and vessel
locations. d) CT reformatted using computed transformation.
Figure 6: 4D-CT Lung Registration. a) Exhale CT. b) Inhale CT. c)
and d) Split screen displays of masked inhale and TPS deformed
exhale CT data.
The mask shown in Figure 1b was used to limit the registration
to consider only voxels inside the liver. Figure 5 shows the
results of the registration. There is very good correspondence
between the anatomy inside the liver but dramatic
disagreement outside. Also, a small part of the liver in the MR
is noticeably displaced by the increased filling of the stomach.
Very local but small deformations of the liver, most likely
resulting from different pressure from the ribs, are also not
accounted for.
4-D CT Lung Example
The lung is clearly known to deform due to breathing (Figure
6). The level of complexity involved in modeling this
deformation is still not well understood, however. A separate
study was performed to demonstrate the ability of the system to
model and accurately predict the deformation of the lung
between inhale and exhale states and to determine the loss in
accuracy of using affine transforms solely compared to thin
plate splines with 30 control points [4]. CT scans acquired at
breath-hold inhale and exhale states were registered. For all
registrations, the masked right lung was used as a reference
intensity map. Accuracy was assessed by comparing predicted
and known locations of anatomic reference points (vascular
and bronchial bifurcations). The results showed an accuracy of
2 mm or better can be achieved with TPS (Table 1), and that
registration errors increase when deformations are not included
in the transformation.
Table 1: Spreadsheet demonstrating geometric relationship between
corresponding anatomic landmarks for 4D CT deformation using TPS.
References
[1] Meyer C R, Boes J L, Kim B, Bland B H, et al 1997
Demonstration of accuracy and clinical versatility of
mutual information for automatic multimodality image
fusion using affine and thin-plate spline warped geometric
deformations. Medical Image Analysis 1(3):195-206.
[2] Kessler M L, Li K and Meyer C 2000 Automated image
registration using mutual information for both affine and
thin-plate spline geometric transformations. Proc. 13th Int.
Conf. on the Use of Computers in Radiation Therapy ed W
Schlegel and T Bortfeld (Heidelberg: Springer) pp 96–98.
Conclusion
The addition of regional rigid and deformable alignment as
well as to tools to define 3D masks based on geometric and
anatomic features has improved the robustness and extended
the utility of this registration system. This infrastructure
provides the tools to improve the value of multiple modality
imaging and to assess temporal changes in patients to better aid
in both treatment planning and adaptive radiotherapy.
[3] Kessler M L, Roberson P, Narayana V, et al 2002 Fast
and Accurate Registration of Prostate CT and MR Data for
Post Implant Dosimetry using Mutual Information.
Radiotherapy and Oncology 64(S1):S283
[4] Coselmon M, Balter J, McShan D, Brock K, Kashani R,
Kessler M 2003 Building a deformable model of the lung
using thin-plate splines. Medical Physics 30(6):1427