Multi-Scale, Multi-Modal Fusion of Histological and MRI
Volumes for Characterization of Lung Inflammation
Mirabela Rusua,* , Haibo Wanga,* , Thea Goldenb , Andrew Gowb , and Anant Madabhushia
a Biomedical
b Department
Engineering, Case Western Reserve University, Cleveland, OH;
of Pharmacology and Toxicology, Rutgers University, Piscataway, New Jersey;
ABSTRACT
Mouse lung models facilitate the investigation of conditions such as chronic inflammation which are associate
with common lung diseases. The multi-scale manifestation of lung inflammation prompted us to use multi-scale
imaging - both in vivo , ex vivo MRI along with ex vivo histology, for its study in a new quantitative way.
Some imaging modalities, such as MRI, are non-invasive and capture macroscopic features of the pathology,
while others, e.g. ex vivo histology, depict detailed structures. Registering such multi-modal data to the same
spatial coordinates will allow the construction of a comprehensive 3D model to enable the multi-scale study of
diseases. Moreover, it may facilitate the identification and definition of quantitative of in vivo imaging signatures
for diseases and pathologic processes. We introduce a quantitative, image analytic framework to integrate in
vivo MR images of the entire mouse with ex vivo histology of the lung alone, using lung ex vivo MRI as conduit
to facilitate their co-registration. In our framework, we first align the MR images by registering the in vivo and
ex vivo MRI of the lung using an interactive rigid registration approach. Then we reconstruct the 3D volume of
the ex vivo histological specimen by efficient groupwise registration of the 2D slices. The resulting 3D histologic
volume is subsequently registered to the MRI volumes by interactive rigid registration, directly to the ex vivo
MRI, and implicitly to in vivo MRI. Qualitative evaluation of the registration framework was performed by
comparing airway tree structures in ex vivo MRI and ex vivo histology where airways are visible and may be
annotated. We present a use case for evaluation of our co-registration framework in the context of studying
chronic inflammation in a diseased mouse.
Keywords: Mouse lung model, in vivo MRI, ex vivo MRI, branchi, histopathology, inflammation, chronic
1. INTRODUCTION
Often multi-modal images including MRI and CT allow for characterizing the lung pathology across different
scales. However, multi-modal in vivo imaging needs to be closely and tightly integrated with histopathology
in order to be able to define and characterize imaging signatures of inflammation and other lung diseases.
Additionally it is desireable to be able to merge the in vivo imaging and ex vivo pathology into an integrated
canonical framework which would provide a unified, integrated multi-scale model for disease characterization.
Such an integrated model would be valuable to characterize in vivo imaging signatures of diseases.1–3 Recently
there has been great interest in developing fused multi-modal multi-scale models by combining ex vivo and in
vivo imaging and pathology data to study diseases, such as prostate cancer1, 4–6 and the brain.7, 8
Mouse models facilitate the study of various pulmonary diseases, e.g. chronic inflammation.9 Lung inflammation is a condition that can be visualized at different imaging scales from the thickening of the airways walls
through the accumulations of foamy appearing alveolar macrophages.10 Methods to investigate in vivo lung
inflammation currently used include micro-CT11 as well as transillumination and fluorescence molecular tomography12 (see13 for a review). However, none of these studies involved a precise co-registration and fusion
with ex vivo histopathology. With pre-clinical models (e.g. mouse) there is the opportunity to quantitatively
and rigorously evaluate imaging features of pathology by correlating the in vivo imaging attributes with ex
vivo histopathology on a voxel-by-voxel basis. For example, in vivo MRI shows macroscopic characteristics
in three dimensions (3D), while histology captures higher level of details in two dimensions (2D) allowing the
visualization of bronchi and their divisions within the lung.
MR and HW contributed equally to this work. Corresponding author: AM (anant.madabhushi@case.edu)
Multi-modal registration is however encumbered by a few significant challenges. Also, ex vivo histology
images often suffer from artifacts caused by fixation and sectioning. Additionally in vivo imaging might suffer
from deformation induced by the presence of other organs such as the heart. Consequently the multi-modal
image registration algorithms need to be able to deal with the various sources of deformation induced within the
different modalities.
While registration of in vivo imaging and ex vivo pathology images have been previously presented in the
literature, the different approaches employed are typically a function of the type of pathology data available. For
instance if only a few pathology sections are available, the only option to co-register with the in vivo imaging
might be to first determine slice correspondences between the two modalities. This can then be followed by 2D-2D
co-registration methods.4, 5 Alternately if multiple concurrent histologic sections are available there might be an
opportunity to reconstruct histologic volumes14–17 which could then allow for a true volumetric co-registration
with the in vivo imaging.6 Such 3D-3D registration has the advantage of introducing additional spatial constrains
to facilitate the accurate alignment. Also 3D-3D registration approaches for MRI and histology typically do not
require establishment of slice correspondences from an expert.
In this work we introduce a co-registration framework that allows the integration of in vivo MRI and ex
vivo histology data for the investigation of chronic inflammation in the lung. Our framework involves the coregistration of in vivo MRI, which shows the lung in situ, with ex vivo digitized histopathology of the entire
lung. The 2D histological slices are collected from the excised lung at very fine intervals and are stained with
hematoxylin and eosin (H&E) to capture inflammation at high resolution. Such high level of detail is needed,
particularly in the case of inflammation, to visualize and characterize the pathology to be studied at a very fine,
granular length scale. The inflammation may be manually annotated by a pathologist on each individual slice
and serves as ground truth that is then mapped via our co-registration and image analytic framework on the
corresponding in vivo MRI. Preceding the co-registration is a three dimensional (3D) reconstruction performed
on the H&E slices to create the corresponding volume of the histology specimen.17 In addition to the in vivo MRI
and histology slices, ex vivo MRI was also collected to visualize smaller structure, particularly airways, that may
not be discernible in vivo; hence, the ex vivo MRI is used by the framework as a conduit to facilitate the accurate
co-registration of in vivo MRI and corresponding ex vivo histology specimens. Our fusion framework allows us
to construct a 3D imaging multiscale model of the lung, allowing us to quantitatively characterize inflammation
on the in vivo imaging.
Our work introduces several novel contributions. (1) We present a fusion framework to reconstruct multi-scale
3D model of the mouse lung by integrating ex vivo histology with in vivo MRI. The solution is robust and allows
to potentially include additional imaging modalities, such as micro CT or PET. (2) The multi-scale, multi-length
scale model allows for quantitative identification of in vivo imaging signatures of chronic inflammation in a
pre-clinical setting for the first time.
The remainder of the paper is organized as follows. The experimental design section describes first the
data collected in the current study followed by the detailed description of the framework including the 3D
reconstruction. Next, the results of the fusion are presented for each step in the framework. Finally, we discuss
our results and present future directions.
2. EXPERIMENTAL DESIGN
2.1 Data Acquisition
In this work, two mice were considered: a mouse that lacks the surfactant protein D (SP-D) showing lung
inflammation, and the wild type mouse without inflammation. Utilizing the Aspect MRI, the in vivo MRI data
was collected on the peak inspiratory volume, which allows for the reconstruction of the in situ lung volume.
Upon completion of the MRI, the mouse lung was extracted and inflation fixed with 4% paraformaldehyde. The
fixed lung, while still in suspension within fluid, was examined by MRI using a 6-hour data collection window,
which allows for a much higher resolution image to be gathered. The fixed lung was then embedded in paraffin
and 5 µm sections were cut with a spacing of 55 µm (Figure 3). These sections were stained with Hematoxylin
and Eosin (H&E) and examined using the Olympus VS120-SL scanning microscopy system.
Our mouse model is constructed using various images.
(a) In vivo MRI
(d) In vivo Lung
(g) In vivo - Ex vivo
MRI fusion
Lung
Segmentation
(b) Ex vivo MRI
(i) 3D Multi-Scale
Lung Model
Rigid
Registration
(e) Ex vivo Lung
Lung
Segmentation
Rigid
(c) Histological Slices
(f) 3D Reconstruction Registration
3D
Reconstruction
(h) Ex vivo MRI 3D histology fusion
Figure 1. Fusion framework: (a) in vivo MRI, (b)ex vivo MRI and (c) histological slices are sequentially acquired. (d-e)The
lung is manually segmented in MRI, (f) a 3D volume is reconstructed from the histology slices to create a volume of the
histopathology. (g) The ex vivo MR volume is aligned to the in vivo MR volume. (h) The histology volume is register
to the ex vivo MRI using rigid transformations. (i) Finally, the lung model is obtained that captures multi-scale details
from the different modalities.
1. Whole body in vivo MRI has the lowest resolution and shows deformation of the lung due to the
presence of other organs, e.g. heart or liver.
2. Ex vivo MRI shows the ex vivo lung at medium resolution allowing the visualization of the trachea and
large bronchi, yet might show artifacts due to the sample preparation.
3. Ex vivo histopathology slices of the entire lung are imaged at high resolution; they may suffer from
imaging artifacts due to sample preparation, e.g. folding and shrinking.
2.2 Annotation of pathology for ground truth extent determination
The ground truth extent for inflammation is determined via manual annotation on the ex vivo digitized histologic
image sections. Inflammation is visible not only in the airways that are expected to thicken, but also at the
level of the alveoli.10 Accurate delineation of inflammation is needed, such that a 3D map of inflammation can
be identified. Accurate registration of the histology slices is thus needed to ensure a meaningful 3D map of the
inflammation onto in vivo MRI.
2.3 Annotation of airways for qualitative validation
Along with the extent of inflammation, large airways are also delineated on each slice using a semi-automatic
thresholding based approach. The segmentation of the large airways provides us the opportunity to (a) qualitatively evaluate the reconstruction results as well as to (b) guide the interactive registration of the histologic
volume with the ex vivo MRI. Specifically, we used the airways to assess the tissue shrinkage caused by the slide
preparation and staining.
1
2
3
4
5
4
5
...
n
(a)
1
2
3
...
...
n
(b)
Figure 2. One-to-one vs One-to-many registration. Each triangle symbolizes an H&E slice, while the number represents
its order in the stack. Black arrow represents the registration direction, i.e. arrow heading on the template image.
The difference between the one-to-one and one-to-many registration approaches is in the number of templates employed.
Only one template is required in (a) allowing for fast piecewise image registration but may cause registration errors to
propagate. On the other hand, groupwise registration in (b) uses multiple templates, requiring additional computations
yet providing better global accuracy.
2.4 Registration of ex vivo MRI to in vivo MRI
First, the lung was manually segmented in both ex vivo MRI and the in vivo MRI. Then, the lung segmented in
ex vivo MRI was rigidly aligned to the lung segmented in in vivo MRI using 3D Slicer.18 In our framework, the
in vivo MRI is the reference fix image to which all other imaging protocols are aligned.
2.5 Reconstruction of the 3D histological volume
Several steps are employed in order to reconstruct the histological volume from the 2D histological slices. First,
we apply one-to-one piecewise19–21 registration to generate an initial alignment of the slices which will be further
optimized by the one-to-many groupwise approach.17 Given a set of 2D histological slices, adjacent slices are
iteratively registered until all slices are aligned (Figure 2(a)). The alignment of two slices is achieved by
maximizing the mutual information using rigid transformations. This piecewise step is able to quickly align the
large number of slices corresponding to the lung. However with piecewise approaches, registration errors may
propagate between slices resulting in slice drifting.
The groupwise step eliminates the drifting by considering more than one section as a template. Here we apply
one-to-many groupwise registration.17 Given the set of slices aligned via the piecewise approach, we choose one
slice that is most similar to the mean shape as an initial template and regard all others as moving slices to be
registered (Figure 2(b)). A moving slice is first selected and aligned to the template such that the mean distance
between the template and moving image is reduced (Algorithm 1). Next, the transformed moving slice is added
to the template stack. Then, the next moving slice is selected and aligned to all the template slices. The process
iterates until all moving slices have been aligned. The metric optimized here is the sum of squared differences
between slices. We adopt the efficient Gauss-Newton22 optimization approach to minimize the metric. Since we
align one moving slice to all previously aligned slices, our approach keeps the global consistency between slices
during the registration.
2.6 Registration of the Histology Volume with ex vivo MRI
The 3D histology lung was aligned with the ex vivo MRI via interactive rigid registration using the outlined
airways of the two modalities as landmarks to guide the alignment. Through the registration of the 3D histology
lung and ex vivo MRI, it became apparent that the histogical specimen has suffered considerable shrinkage during
sample preparation. We assessed the amount of shrinkage by iteratively updating the scaling factor that was
applied to the histological volume. For each scaling factor, the alignment of the airway tree structure in the
ex vivo histology volume and ex vivo MRI was uses to qualitatively estimate of the registration. In this work,
we found a scaling factor of 2 to allow the best alignment between the histology volume and ex vivo MRI. In
Data: a set of N unaligned slices;
Result: aligned images;
Init: Pick the first slice as template;
for each unaligned slices do
Pick one as the moving image while not converged do
compute a small transformation displacement via gradient decent;
add the displacement to the computed transformation;
warp the moving image with the computed transformation;
end
foreach unaligned slices do
warp the slice with the computed transformation;
end
add the aligned moving image as a new template;
end
Algorithm 1: Summary of our groupwise registration approach
the future, we consider acquiring additional data such as block face image of the histological sections before the
sample preparation to accurately assess the scaling factor.6
Following these steps, the ex vivo histology specimen is aligned to the in vivo MRI. Such alignment enables
the mapping of lung inflammation (annotated on histology slices) on to the in vivo MRI.
3. RESULTS AND DISCUSSION
Qualitative results are shown for each step of the framework (Figures 3-6).
3.1 Histology Volume Reconstruction
The groupwise registration allowed the building of the 3D volume representing the lung from the histopathology
sections (Figure 3(b)). A careful inspection of the reconstruction reveals that the approach was able to align the
slices without causing significant drifting between slices. Such drifting is visible when misalignment errors are
propagated between consecutive slices and the model progressively rotates. Moreover, a fairly smooth surface
of the volume suggests that the groupwise registration was able to properly identify the rigid transformations
that were required to align the slices. Occasional minimal misalignment may be observed at the edges of the
reconstruction, yet they are caused by the folding of the underlying tissue during the histology preparation. We
also evaluate the alignment of slices by visual inspection of the airways (colored red in Figure 3(c)). When the
proper transformation was identified, the airways were closely aligned by reconstructing the tree like structure.
3.2 Registration
3.2.1 Histology - ex vivo MRI registration
Following 3D reconstruction, the histological volume was registered to the ex vivo MRI, using iterative scaling
followed by interactive rigid registration. Figure 4 shows the results of the histology volume fusion with ex
vivo MRI. Although the airways align fairly well between the two modalities (Figure 4(b)), it may be observed
that the lung surfaces appear misaligned between the ex vivo MRI and the corresponding 3D histologic reconstruction (green and purple surfaces in Figure 4(a)). Such misalignment is in part due to the imaging artifacts
induced during the histology preparation, e.g. tissues shrinkage, and in the ex vivo MRI acquisition, e.g. large
displacement of the lobes due to the fixation protocol.
...
(a)
(b)
(c)
Figure 3. Illustration of the individual of 2D histology slices (a) and corresponding 3D reconstruction(b-c); the entire
lung is sectioned into individual slices and stained with H&E. Three non-adjacent slices resulting from the sectioning are
displayed in (a). Following the groupwise registration17 of the individual histologic sections a 3D reconstruction of the
lung is obtained (purple outline in (b)-(c)). The airways segmented on the histology slices are aligned in 3D following the
registration protocol indicating a smooth contiguous alignment of the underlying histology slices (red in (c)).
O
90
(a)
(b)
Figure 4. Fusion of ex vivo MRI with the reconstructed 3D histological volumen (a) Overlay of the ex vivo MRI (green)
with the 3D reconstructed histologic volume (purple) following rigid registration. The airways of the two modalities
were used to identify the scaling of the histology to correct for the shrinkage caused by sample preparation. (b) Airway
alignment between the ex vivo MRI (blue) and histology (red) reveals a relatively accurate overlay.
3.2.2 In vivo and ex vivo MRI registration
The segmented lungs on the in vivo and ex vivo MRI were aligned using interactive rigid registration in 3D
Slicer.18 Anatomic landmarks such as the trachea were utilized to guide the alignment. Overall, the shape of
the lung was preserved between the two acquisitions (yellow versus green surfaces in Figure 5), allowing for a
good alignment between the different lung surfaces.
Following the registration of ex vivo to in vivo MRI and subsequently co-registering the ex vivo MRI and the
ex vivo histology, all 3 modalities were co-registered to a common reference frame allowing for the construction
of a 3D imaging model of the lung in mice (Figure 6).
(a)
(b)
Figure 5. Fusion of in vivo MRI with ex vivo MRI seen from different viewing angles, (a) front view and (b) tilted. Outline
of the lung and mouse from in vivo MRI are shown in yellow, while the ex vivo MRI lung is shown in green. The main
airways segmented from ex vivo MRI are depicted in blue.
(a)
(b)
(c)
Figure 6. 3D Multi scale model of the mouse lung; The 3D reconstruction of the histology slices are shown in purple
while the airways segmented from the later showed in red. The ex vivo MRI airways are represented in blue, while the in
vivo MRI is shown in yellow. (a-c) different structures are made visible or invisible to show the fusion results.
4. CONCLUDING REMARKS
In this work, we present a unified imaging framework for building a 3D imaging model of the lung with the goal
of identifying imaging signatures of lung inflammation in vivo. The new framework was utilized to co-register
the in vivo MRI with histology using the ex vivo MRI as conduit to facilitate the data fusion. The framework
was applied here to create a 3D imaging model of a SP-D knock out mouse which shows lung inflammation,
thus allowing for the mapping of regions of inflammation onto the corresponding in vivo MRI. Registration
methods such as described in this work are required as lung inflammation shows a discontinuous distribution23
within the lung. In future work we intend to further develop our framework to allow incorporation of non-linear
transformations and allowing for independent transformations for both sides of the lung. Moreover, we hope to
increase the size of our cohort and utilize fully automatic registration approaches to integrate the multi-modal
data. The co-registration framework will allow for direct measurement of changes that occur with inflammation,
thus allowing for identification and evaluation of in vivo imaging signature of inflammation.
5. ACKNOWLEDGEMENTS
This work was made possible by grants from the National Institute of Health (R01CA136535, R01CA140772,
R43EB015199, R21CA167811), National Science Foundation (IIP-1248316),the QED award from the University
City Science Center and Rutgers University, and HL086621.
REFERENCES
[1] Viswanath, S. E., Bloch, N. B., Chappelow, J. C., Toth, R., Rofsky, N. M., Genega, E. M., Lenkinski,
R. E., and Madabhushi, A., “Central gland and peripheral zone prostate tumors have significantly different
quantitative imaging signatures on 3 tesla endorectal, in vivo T2-weighted MR imagery,” J. Magn. Reson.
Imaging 36, 213–224 (2012).
[2] Segal, E., Sirlin, C. B., Ooi, C., Adler, A. S., Gollub, J., Chen, X., Chan, B. K., Matcuk, G. R., Barry, C. T.,
Chang, H. Y., and Kuo, M. D., “Decoding global gene expression programs in liver cancer by noninvasive
imaging,” Nature biotechnology 25(6), 675–680 (2007).
[3] Gevaert, O., Xu, J., Hoang, C. D., Leung, A. N., Xu, Y., Quon, A., Rubin, D. L., Napel, S., and Plevritis,
S. K., “Non–small cell lung cancer: identifying prognostic imaging biomarkers by leveraging public gene
expression microarray datamethods and preliminary results,” Radiology 264(2), 387–396 (2012).
[4] Chappelow, J., Madabhushi, A., Rosen, M., Tomaszeweski, J., and Feldman, M., “Multimodal image registration of ex vivo 4 tesla prostate mri with whole mount histology for cancer detection,” SPIE Medical
Imaging 6512, S1–S12 (2007).
[5] Chappelow, J., Bloch, B. N., Rofsky, N., Genega, E., Lenkinski, R., DeWolf, W., and Madabhushi, A., “Elastic registration of multimodal prostate MRI and histology via multiattribute combined mutual information,”
Med. Phys. 38(4), 2005–18 (2011).
[6] Park, H., Piert, M. R., Khan, A., Shah, R., Hussain, H., Siddiqui, J., and Meyer, C. R., “Registration
methodology for histological sections and in-vivo imaging of human prostate,” AR 15(8), 1027 (2008).
[7] Dauguet, J., Delzescaux, T., Conde, F., Mangin, J.-F., Ayache, N., Hantraye, P., and Frouim, V., “Threedimensional reconstruction of stained histological slices and 3D non-linear registration with in-vivo MRI for
whole baboon brain,” in [Journal of Neuroscience Methods], 164, 191–204 (2007).
[8] Palm, C., Penney, G. P., Crum, W. R., Schnabel, J. A., Pietrzyk, U., and Hawkes, D. J., “Fusion of rat
brain histology and MRI using weighted multi-image mutual information,” 69140M–69140M–9 (2008).
[9] Sheth, V., van Heeckeren, R., Wilson, A., van Heeckeren, A., and Pagel, M., “Monitoring infection and
inflammation in murine models of cystic fibrosis with magnetic resonance imaging,” J. Magn. Reson. Imaging 28(2), 527–532 (2008).
[10] Atochina, E. N., Beers, M. F., Hawgood, S., Poulain, F., Davis, C., Fusaro, T., and Gow, A. J., “Surfactant
protein-D, a mediator of innate lung immunity, alters the products of nitric oxide metabolism.,” American
Journal of Respiratory Cell and Molecular Biology 30(3), 271 (2004).
[11] Jobse, B. N., Johnson, J. R., Farncombe, T. H., Labiris, R., Walker, T. D., Goncharova, S., and Jordana,
M., “Evaluation of allergic lung inflammation by computed tomography in a rat model in vivo,” European
Respiratory Journal 33(6), 1437–1447 (2009).
[12] Ntziachristos, V., “Optical imaging of molecular signatures in pulmonary inflammation,” Proceedings of the
American Thoracic Society 6(5), 416–418 (2009).
[13] Jannasch, K., Missbach-Guentner, J., and Alves, F., “Using in vivo imaging for asthma,” Drug Discovery
Today: Disease Models 6(4), 129 – 135 (2009).
[14] Yushkevich, P., Avants, B., Ng, L., Hawrylycz, M., Burstein, P., Zhang, H., and Gee, J., “3D mouse brain
reconstruction from histology using a coarse-to-fine approach,” Biomedical Image Registration , 230–237
(2006).
[15] Cifor, A., Pridmore, T., and Pitiot, A., “Smooth 3-D reconstruction for 2-D histological images,” in [Information Processing in Medical Imaging], 350–361, Springer (2009).
[16] Feuerstein, M., Heibel, H., Gardiazabal, J., Navab, N., and Groher, M., “Reconstruction of 3-D histology
images by simultaneous deformable registration,” MICCAI , 582–589 (2011).
[17] Wang, H., Rusu, M., Golden, T., Gow, A., and Madabhushi, A., “Mouse lung volume reconstruction from
efficient groupwise registration of individual histological slices with natural gradient,” in [SPIE Medical
Imaging], Accepted (2013).
[18] Pieper, S., Halle, M., and Kikinis, R., “3D Slicer,” IEEE International Symposium on Biomedical Imaging
, 632–635 (04 2004).
[19] Bagci, U. and Bai, L., “Automatic best reference slice selection for smooth volume reconstruction of a mouse
brain from histological images,” IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1688–1696 (2010).
[20] Pitiot, A., Bardinet, E., Thompson, P. M., and Malandain, G., “Piecewise affine registration of biological
images for volume reconstruction,” MIA 10(3), 465–483 (2005).
[21] Schmitt, O., Modersitzki, J., Heldmann, S., Wirtz, S., and Fischer, B., “Image registration of sectioned
brains,” International Journal of Computer Vision 73(1), 5–39 (2007).
[22] Baker, S. and Matthews, I., “Lucas-Kanade 20 Years On: A Unifying Framework: Part 1,” Tech. Rep.
CMU-RI-TR-02-16, Robotics Institute (July 2002).
[23] Korfhagen, T. R., Sheftelyevich, V., Burhans, M. S., Bruno, M. D., Ross, G. F., Wert, S. E., Stahlman, M. T.,
Jobe, A. H., Ikegami, M., Whitsett, J. A., and Fisher, J. H., “Surfactant protein-D regulates surfactant
phospholipid homeostasis in vivo,” Journal of Biological Chemistry 273(43), 28438–28443 (1998).