Title: A Computational Framework for Ultrastructural Mapping of Neural Circuitry
Running Head: A Framework for Mapping Neural Circuitry
Authors: J.R. Anderson [1], B.W. Jones [1], J-H Yang [1], M.V. Shaw [1], C.B. Watt [1], P. Koshevoy
[2,3], J. Spaltenstein [3], E. Jurrus [3], U.V. Kannan [3], R. Whitaker [3], D. Mastronarde [4], T. Tasdizen
[3,5], R.E. Marc [1]
Affiliations:[1] Dept. Ophthalmology, Moran Eye Center, University of Utah; [2] Sorenson Media, Salt
Lake City, UT; [3] Scientific Computing and Imaging Institute, University of Utah; [4] The Boulder Laboratory For 3-D Electron Microscopy of Cells, University of Colorado, Boulder; [5] Dept. Electrical and
Computer Engineering, University of Utah;
Corresponding Author: R. E. Marc
University of Utah, Moran Eye Center, 65 Mario Capecchi Dr., Salt Lake City 84132 UT
PHONE 801-585-6500
FAX 801-587-7724
EMAIL: robert.marc@hsc.utah.edu
This is the markup version indicating significant changes such as text insertion (underlined),deletion
[bracketed], or relocation to supplemental material (colored). The manuscript has been extensively rePage 1 of 70
vised throughout but changes amounting to less than two lines insertion or deletion are not marked. Major
changes include addition of new Table 2 (the performance reappraisal sought by reviewer 1), two new
figures (requested by reviewer 1) and 5 pages of deletion (requested by reviewer 3).
.
Page 2 of 70
Anderson et al: A Framework for Mapping Neural Circuitry ABSTRACT
Abstract
Circuity mapping of metazoan neural systems is difficult because canonical neural regions (regions containing one or more copies of all components) are large, regional borders are uncertain, neuronal diversity
is high, and potential network topologies so numerous that only anatomical ground truth can resolve
them. Complete mapping of a specific network requires synaptic resolution, canonical region coverage
and robust neuronal classification. Though transmission electron microscopy (TEM) remains the optimal
tool for network mapping, the process of building large serial section TEM (ssTEM) image volumes is
rendered difficult by the need to precisely mosaic distorted image tiles and register distorted mosaics.
Moreover, most molecular neuronal class markers are poorly compatible with optimal TEM imaging.
Our objective was to build a complete framework for ultrastructural circuitry mapping. This framework
combines strong TEM-compliant small molecule profiling with automated image tile mosaicking, automated slice-to-slice image registration and gigabyte-scale image browsing for volume annotation. Specifically we show how ultrathin molecular profiling data sets and their resultant classification maps can be
embedded into ssTEM data sets and how scripted acquisition tools (SerialEM), mosaicking and registration (ir-tools), and large slice viewers (MosaicBuilder, Viking) can be used to manage terabyte-scale volumes. These methods enable large-scale connectivity analyses of new and legacy data. In well-posed
tasks (e.g. complete network mapping in retina), terabyte scale image volumes that previously would require decades of assembly can now be completed in months. Perhaps more importantly, the fusion of molecular profiling, image acquisition by SerialEM, ir-tools volume assembly and data viewers/annotators
also allow ssTEM to be used as prospective tool for discovery in non-neural systems and a practical
screening methodology for neurogenetics. Finally, this framework provides a mechanism for parallelization of ssTEM imaging, volume assembly, and data analysis across an international user base, enhancing
the productivity of a large cohort of electron microscopists.
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
INTRODUCTION
Neural network reconstruction is a grand challenge in neuroscience and vision science in particular. Defining complete network (CN) maps or connectomes (Sporns et al. 2005; Lichtman et al. 2008) for canonical regions of any metazoan neural assembly requires robust cataloguing of neuronal classes (MacNeil et
al. 1999; Marc and Jones 2002; MacNeil et al. 2004; Otsuna and Ito 2006; Micheva and Smith 2007),
mapping statistically distinct neuronal patterns (Vaney 1990; Peters et al. 1997; Lund et al. 2003; Reese
2008) and tracing characteristic connections (Kolb and Famiglietti 1974; Calkins and Sterling 1996; Klug
et al. 2003). Moreover, anatomic methods for network analysis have not kept pace with the demands for
phenotyping an immense and expanding library of genetic models of neurologic disorders in general
(Crawley 2008) and retinal disorders in particular (Marc et al. 2003). This is all the more distressing
since, historically, anatomy has shown far more power to define neural network ground truth1 than either
modeling or physiological strategies and, in practice, ssTEM has been the most powerful generator of validated existing network maps. The need to expand these abilities beyond the purview of a limited number
of specialized laboratories has never been more acute.
We have developed a complete suite of software tools and strategies that leverage existing ultrastructural
resources (Table 1). Commercial (Marc and Liu 2000) and academic (Fiala and Harris 2001; Fiala 2005)
software solutions for small-scale, user-guided mosaicking and multimodal registration have long been
available, but have not proven viable for large datasets or high throughput. By providing tools to precisely
and automatically tile many images ( ≈ 1000) into large mosaics, to precisely register serial mosaics (including multimodal frames) and to browse gigabyte image sets and terabyte volumes, we hope to enable
expanded analysis of connectivity patterns in legacy as well as new image databases.
1
The term "ground truth" emerged first in remote sensing and refers to specific ground information used to validate optical data
captured from afar. In neuroscience, ground truth is the physical connectivity of identified neurons. Patterns of connectivity inferred from behavior, modeling or physiology are thus subject to the test of anatomical ground truth.
Page 4 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
Why are such tools important? Simply, the unraveling of connective patterns in complex neural tissue and
the characterization of deranged circuitry in disease states requires sampling scales that have been impractical. Some important neural reconstruction tasks are so large that they transcend investigator lifetimes
using current resources (Fiala 2002). The volume that must be constructed to approach sampling completeness in the inner plexiform layer of the mammalian retina is three orders of magnitude larger than
most typical ssTEM volumes used in CNS (Fiala and Harris 2001; Harris et al. 2006). Other programs are
addressing these challenges by developing novel platforms to acquire pre-aligned serial electron microscope images (Denk and Horstmann 2004; Briggman and Denk 2006; Hayworth et al. 2006; Knott et al.
2008). However these platforms alone are not the sole nor optimal strategies for ssTEM volume assembly, as some new methods destroy samples, are limited in resolution and speed (Table 2), and most of the
platforms are developmental or highly restricted in availability. Conversely, ssTEM has high resolution,
tremendously flexibility in staining and immunocytochemical options, very fast acquisition times and the
potential for parallelization, by analogy with grid computing, via the subdivision of ssTEM samples into
packets for parallel data acquisition. Data assembly could be readily done if tools to harmonize the effort
were widely available. We present the essential software tools here, specifically those for assembling
large-scale mosaics and achieving slice-to-slice image registration.
The challenge of network diversity.
Is ssTEM really necessary? Why can’t we deduce networks from physiology, confocal imaging or behavior? The answer is that potential network motifs derived by these methods are not unique. Diversity in
potential network topologies is so high (Marc and Liu 2000; Chklovskii et al. 2004) that only anatomical
ground truth can produce a valid connectome (Marc and Liu 2000). For example, mammalian retinas are
simpler than those of most other vertebrates (Marc and Cameron 2002) but even so, no fewer than 70
Page 5 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
unique cell classes exist (Masland 2001). And though the flow of signals from cone photoreceptors to
ganglion cells (GCs) involves stereotyped networks that seem simple (Sterling 2004; Marc et al. 2008), a
vast number of synaptic motifs can be produced from even a limited neuron set (Marc and Liu 2000). A
small network of two different bipolar cells (BCs) driving two GC channels, interconnected by one amacrine cell (AC) class can be connected in 90 formal motifs and at least 40 of these are biologically tenable
(Fig. 1, S1, S2). This is further compounded by unknown synaptic weights, molecular diversity of receptors and channels, gap junction display, and electrotonic constraints. With electrotonic constraints, the
geometric locus of a synaptic contact matters (Miller and Bloomfield 1983; Rall 1995; Weiss 1996) and
the number of structural motifs possible in the simple 5 element example extends to at least 640 (Fig. S1).
In stark contrast, we know that the outflow of signals from the mammalian retina is represented by only
15-20 ganglion cell classes, each representing a discrete filter channel (Marc and Jones 2002; Rockhill et
al. 2002; Berson 2008).
[12 lines of text deleted here]
Anatomy uncovers unique network motifs
An anatomical approach that unambiguously determines motifs is required. This is justified by the fact
that the efficacy of anatomy discovering complex motifs is unrivaled. Mammalian night (scotopic) vision
is a prime example. The main scotopic signal flow network is rod → rod BC → rod AC, which then bifurcates into two synaptic arms that re-enter the ON and OFF cone BC pathways. This motif was discovered by Helga Kolb and E.V. Famiglietti Jr. (Kolb and Famiglietti 1974) by ssTEM. Subsequent physiological and genetic analyses (Müller et al. 1988; Deans et al. 2002) provided correlative support for the
anatomical model, but neither study would have uniquely yielded the correct topology. Moreover, both
TEM (Famiglietti and Kolb 1975; Marc and Liu 1985; Sterling and Lampson 1986; Marc 1989; Strettoi et
al. 1992) and LM imaging studies (Vaney 1986; Li et al. 2002) reveal that this network is even more
complex. There are, in fact, no physiological data that either explain or predict these network submotifs.
Page 6 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
And despite five decades of robust physiology of retinal rod signaling, the discovery of a second scotopic
pathway was also based on ssTEM (Soucy et al. 1998; Tsukamoto et al. 2001). Finally, corruption of scotopic motifs in retinal disease was discovered by TEM (Peng et al. 2000), again despite decades of electroretinographic analysis. It is unlikely that the day of ultrastructural discovery is past and we argue that it
is just dawning.
Requirements for building CN maps.
Manually acquiring even small maps by ssTEM requires Herculean effort, high technical skill (Harris et
al. 2006). Three essential factors in building CN maps are (i) proper resolution, (ii) statistical coverage
and (iii) complete classification.
Resolution must be sufficient to unambiguously identify synaptic contacts and gap junctions (Harris et al.
2006) but not so high as so be unmanageable: nominally 2 nm/pixel. This yields synaptic vesicles
spanned by 8-10 pixels which are robust for circuitry tracing.
Coverage scales with neuronal diversity and density: a canonical region must be sampled. We define two
coverage units (Fig. 2). A canonical tile is bounded by the Voronoi domain (see Reese 2008) of the rarest
neuronal element in a cellular array. In retina, this might be the dopaminergic polyaxonal cell (Eglen et al.
2003) or OFF α GCs (Marc and Jones 2002). A slightly larger element is the canonical field, bounded by
three somas (in planar systems such as retina) or four somas (in brain volumes) of the sparsest neuronal
class. This ensures inclusion of multiple somas of all element classes in the region. However, not all canonical fields are known a priori and, arguably, molecular and anatomic data are key prerequisites to defining such domains. As we will show, a canonical field of retina can be acquired and assembled in less
Page 7 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
than 5 months. Cortex is more challenging. Using normal primate ocular dominance column dimensions
(>0.5 mm) and assuming a canonical repeat of 0.25 mm, the acquisition of V1 visual cortex would take
12 years. Reducing resolution to 10 nm/pixel (similar to bloc-face imaging) reduces acquisition to less
than 1 year and new high-throughput TEM configurations (e.g. the Harvard Connectome Project) are allowing much larger capture fields, with as much as a 10x improvement in speed. Thus even canonical
high-resolution cortical volumes are possible with ssTEM.
Classification (i.e. neuronal phenotyping) must be sufficiently robust to identify most major elements in a
canonical field and may be the most effective way of identifying a field in the first instance (Marc and
Jones, 2002) . It is not practical to decipher networks from large reconstructions when the number of classes of neurons is unknown. In general, panels of markers compatible with mapping networks are applied
to or expressed in regions of interest (Deerinck et al. 2007; Lichtman et al. 2008).
In conjunction with serial section light microscopy (ssLM), we use computational molecular phenotyping
(CMP: see Marc, Murry and Basinger, 1995; Marc and Jones 2002). CMP is a high-resolution optical imaging strategy that exploits sets of immunoglobulins (IgGs) targeting small molecules, precise multichannel image registration and cluster analysis to extract defined neuronal classes in any tissue (Marc et al.
1995; Marc 1999b; Marc and Cameron 2002; Marc and Jones 2002). Small molecule optical CMP is
compatible with ssTEM and enables classification of neuronal elements via direct multimodal image registration (Marc and Liu 2000; Jones et al. 2003). A tutorial on CMP is available in the Supplementary
Material. In summary, building a CN map of any neural region such as retina requires canonical field imaging, neuronal phenotyping, image mosaicking, image registration and image annotation. In this paper
we will detail complementary applications that can be used with a variety of datasets to produce large,
high-resolution, aligned mosaics.
Page 8 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
Results
We present here our complete framework and workflow, focussing on the image acquisition and processing pipeline for neural network mapping, with examples of image capture, mosaicking and registration; fused molecular and ultrastructural data; volume construction; and browsing/annotating large data
sets. The mapping workflow is built around a suite of image registration tools (ir-tools) and involves 12
basic steps (diagrammed in Fig. 3):
1. Harvest target tissue
2. Process for optimized CMP & EM
3. Section correlated ssTEM and ssLM libraries
4. Capture image tiles [SerialEM, Syncroscan]
5. Build mosaics [ir-fft/ir-refine-grid, ir-translate/ir-refine-grid]
6. Register ssTEM to bounding ssLM sets [ir-tweak]
7. Classify ssLM sets [CMPView]
8. Build ssTEM volumes [ir-stos-brute, ir-stos-refine]
9. Register ssTEM processes to intercalated ssLM classes [ir-tweak]
10. Visualize and track processes in the volume (MosaicBuilder, Viking)
11. Tag synaptic connections (Viking)
12. Summarize local circuitry data into network maps.
The collection of software tools for steps 1-11 are summarized in Table 1 and all are available for download. This framework is obviously not restricted to analysis of the nervous system or any TEM or SEM
platform, but analyses of synaptic connectivity specifically require a characteristic resolution and canonical field, as will reconstruction of any other tissue volume.
Page 9 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
Acquiring mosaic tiles manually.
Many TEM facilities lack automated montaging, but this does not mean that high quality imagery cannot
be obtained. Standard TEM imaging with sufficient image overlap can be obtained manually and images
scanned at high resolution and bit depth. We use magnifications ranging from 5,000x-10,000x and a typical manual ssTEM project size would be 100 image tiles per section. Similarly, corresponding bounding
or intercalated ssLM sections can be imaged manually and require only a few tiles even at high resolution.
However, as the positions of each ssLM and ssTEM image tile in the original sections are often lost,
software tools to provide precise mosaic alignments are necessary.
Acquiring mosaic tiles with SerialEM.
Larger ssTEM datasets can be captured with automated imaging, exceeding 1000 tiles. Such montaging
requires robust control of stage position, camera behavior, metadata collection and efficient use of resources. All of these are available through use of SerialEM software developed at the University of Colorado. SerialEM allows the use irregular capture patterns. [7 lines of text deleted here] No further user attention is required once all sections on a grid have been queued which allows one to utilize the TEM during commonly idle night and weekend periods. In all regards we have found automated capture very resource efficient compared to manual approaches. Our current configuration captures 3000 tiles in a day.
The expanded capacity of ssTEM imaging requires a corresponding automation of ssLM tile collection.
There are a number of commercial microscope tiling stages and our initial experience showed that the
highest precision stages were essential to building mosaics of sufficient quality for CMP. However, the
development of software tools for building mosaics informed by but not dependent upon stage metadata
makes the X-Y precision of the stage less critical as long as overlap is adequate.
Page 10 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
Building mosaics with ir-fft, ir-translate and ir-refine-grid.
There are several challenges in ssTEM or ssLM image mosaicking with manual tile acquisition. First,
every section exhibits an unpredictable rotation when placed in the TEM or on a slide and the number of
tiles in each scan-line will differ. Thus it is typically not known which tiles are neighbors in a section. We
developed the ir-fft algorithm to deduce the tile ordering automatically (see Supplementary Material for a
detailed description of ir-fft). The next challenge is the correction of non-linear warps introduced into
each tile from variations in electron imaging quality (Fig. 4; also see Methods and Materials). Our solution finds pairs of overlapping tiles, computes their relative displacement; deduces a tile ordering, builds a
layout of the mosaic without non-linear warping, and refines the mosaic by applying non-linear warps to
each tile.
A multipage section of text was deleted here
Ultimately, it is more efficient to build ssLM and ssTEM mosaics when coordinate information is available in the image metadata and this exploits the tool ir-translate. [4 lines of text deleted here] Only overlapping tiles are matched using the Fourier shift method (ir-fft). This reduces complexity from quadratic
to linear in the number of tiles. Next, we define a tension vector proportional to the offset between the
approximate position,and the preferred position as found by matching. These tensions are relaxed by iteratively moving the tiles (see Supplementary Material).
Regardless of tile placement, most mosaics require some non-linear warp refinement and this is accomplished with ir-refine-grid, an approach that resamples each tile into a coarse triangular mesh. Details of
algorithm development are in the Supplementary Material. Each tile is sampled onto a coarse uniform
triangle mesh and small, neighborhoods is sampled from all of the tile neighbors in the mosaic and the
best matches determined as in ir-fft. [extensive text deleted here] A simple example of the ability of irrefine-grid to manage subtle distortions is shown in Fig. 5, where the TEM image tiles previously shown
Page 11 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
intractable under translation are readily aligned without user intervention. These are low resolution tiles: a
worst-case scenario.
Even when thousands of tiles are assembled, the alignment remains excellent. Fig. 6 displays a randomly
selected tile from a 120+ mosaic series of >1000 tiles each, aligned with ir-translate. At the screen resolution used for synaptic markup, the tile edges are rarely visible. The error in aligment in this set of four
overlapping tiles is extremely small, ranging from no detectable misalignment across three tiles to 7.8 nm
x shift in one tile: roughly ⅓ of a vesicle. Such errors are random rather than systematic through the volume, and do not accumulate.
Together, ir-tools assemble superb mosaics. As an example, our Syncroscan system builds mosaics from
arrays of LM tiles but invariably shows subtle misalignments or blurring at boundaries (Fig. 7A). Conversely, ir-tools perform beautifully on exactly the same image tiles, generating seamless mosaics (Fig
7B). In truth, the Syncroscan errors are so small (≈ 200-2000 nm) that they are generally invisible when
the full image is viewed, but a Laplacian transform (Fig. 7C) shows that there are many of them. When
multiple channels are registered for CMP, these errors are additive, resulting in corruption of classification and ssLM-ssTEM registration. The LM images mosaicked by ir-tools are essentially perfect.
Image registration with ir-tweak and ir-brute-stos / ir-refine-stos.
Both user-guided and automated image registration tools are needed for ssLM and ssTEM. User-guided
applications are essential because some images (e.g. certain CMP imagery) lack sufficient information to
drive automation. Ir-tweak is an interactive, multi-threaded, cross-platform application for manual sliceto-slice registration. [Extensive lines of text deleted here] As control points are placed by the user in one
Page 12 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
image, their locations in the other image are estimated by the current thin-plate spline transform parameters. When the user corrects the locations of estimated points in the second image, the transform parameters are updated. Figure 8 shows the ir-tweak interface where the operator places points in the fixed image, adjusts their locations on the moving target image and observes the registration dynamically.
While automated multimodal slice-to-slice registration remains an open challenge (see
http://prometheus.med.utah.edu/~marclab/gallery_CS.html for publicly available test sets), such boundary
or intercalated registrations are manageable with user-guided tools such as ir-tweak. Conversely, automated ssTEM slice-to-slice (stos) image registration is essential to building volumes, even when image
metadata are unavailable. As any section may be distorted by stretching or electron-optical defects, stos
registration is similar to ir-refine-grid, with two differences. Since the orientation of slice pairs is arbitrary, we cannot use image correlation to estimate image to image translation parameters. Instead, we first
perform a brute force search (ir-stos-brute) for tile translation/rotation parameters by downscaling the
section mosaics to 128x128 pixel thumbnails and pre-processing (ir-blob) to enhance large blob-like features, preventing feature washout when downscaling. [4 lines of text deleted in this region] These parameters are then used to initialize the mesh transform at a fine resolution (ir-stos-refine) and applied to a
“moving” slice relative to a chosen fixed slice. Figure 9 is a down-sampled QuickTime® movie of a
mouse retinal microneuroma ssTEM dataset acquired manually on film and automatically built into a volume of 45 auto-registered mosaics. This is a representative legacy dataset and is by far the most challenging type of data for automatic mosaicking and volume assembly due to lack of metadata and the presence
of many section defects (stain artifacts, folds, dirt, beam burns). Even so, the alignment is excellent and
suggests that many legacy ssTEM datasets can be exploited.
Page 13 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
The availability of a large-format digital camera for TEM (e.g. the Gatan Ultrascan 4000) coupled with
the most recent builds of SerialEM now make it possible to acquire large image fields at synaptic resolution from any specimen and begin assembling volumes automatically, such as the retinal circuitry volume
for the rod BC layer in the mouse retina (Fig. 10). In this example, each slice was automatically mosaicked from 16 tiles (5000x) with ir-translate and ir-refine-grid and a volume of 20 slices automatically
registered with ir-stos (Fig. 10A,B). Manually registering these datasets are impossible because of the
many required distortion corrections among tiles and slices. With the ir-tools a year’s manual work can be
done in a day. Upon browsing the volume, characteristic connection motifs can be quickly extracted (Fig.
10C,D; S4) and graphically summarized (Fig. 10E) from a text list of relations. Again, the goal is not to
render 3D shapes, but rather browse and markup synaptic motifs. This volume readily detects characteristic reciprocal feedback, GABAergic local feedforward and glycinergic long-range feedforward synaptic
arrangements in the locale of the BC (see animation in Fig. S3). Importantly, all regions are registered,
not just the ones of local interest. We have similarly built volumes from 1000-tile datasets of mouse retinal microneuromas (see below) and over 100 serial 1000-tile mosaics from a 300+ section series through
the rabbit inner plexiform layer (Fig. 11). This demonstrates the quality of large-scale automated alignment and the absence cumulative distortions. After automated registration through a volume of >100 sections (2 Tb), no error emerges from transforming all sections into the same volume space. While subtle
slice-to-slice distortions exist due to physical deformation of sections, they do not accumulate and section-to-volume distortions are statistically indistinguishable from any those of any slice pair. Should such
unlikely distortions emerge, our fast transform management method (see Visualization and Annotation
below) allows the volume to be partitioned at any point and structures tracked across the parts. We can
define break points and reference slices anywhere in the volume and rapidly create new series of transforms. This method is ideal for automated registration.
Computational Molecular Phenotyping: CMP
Page 14 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
[extensive text revision & deletion here]
CMP is a thin-section optical method that provides molecular signals for classification of cells and large
processes. Ultrathin sections are immunoprobed for different small molecules, imaged optically, registered by ir-tools, and visualized as multichannel molecular signatures of different cell types. A tutorial on
CMP is provided in the Supplementary Material. All cells have small molecule signatures and these are
most evident in the central nervous system (Hill et al. 2000, 2001) and retina (Marc et al. 1995; Kalloniatis et al. 1996; Marc and Cameron 2002; Marc and Jones 2002). A library of 4-8 small molecules can
segment retinal populations into 20 or more natural molecular cell classes (Marc 1999b; Marc and Jones
2002). CMP can also segment many cell processes into different functional classes with high fidelity
(Marc and Liu, 2000; Jones et l. 2003). Figure 11 displays a retinal microneuroma ssTEM (Fig 11A), its
bounding CMP ssLM images as multispectral overlays (Fig. 11B,C), and its corresponding theme map
after K-means classification with CMPView (Fig. 11D). The four 90 nm sections preceding the ssTEM set
were processed for CMP using IgG E, IgG G, IgG τ, and IgG γ and aligned with the initial ssTEM image
with ir-tweak. After classification with these four signatures alone, we show that there are four superclasses of ACs (γ1, γ2, G1, G2), two BC superclasses (Eτ, EτG), two GC superclasses (E, Eγ), the glial
Müller cell class (τQ) and the retinal pigmented epithelium class, similar to results in normal mouse, primate and rabbit retinas. One critical feature of such theme maps is completeness: every cell in the TEM
mosaic is classified into a known biological group and every process traced from it is similarly tagged. No
other method has yet achieved this scale of functional coverage.
On a larger scale, a 0.75 mm wide sample of the mouse inner plexiform layer was mosaicked and augmented with CMP at sufficient resolution to identify many synapses directly (Fig. 12). An example of the
value of CMP signatures in defining circuits is shown in Fig. 12B, where an ON cone BC (Kalloniatis et
al. 1996; Marc 1999a) is presynaptic to a class γ1 AC process, which also makes a reciprocal synapse
Page 15 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
back onto the bipolar cell. This is an archetypal feedback motif (see Fig. 10), one of the most common in
retina (Marc and Liu 2000). In addition a G1 glycinergic AC process is presynaptic to the BC. This illustrates the powerful segmentation possible with ssLM CMP, even at the ultrastructural scale, enabled by
ir-tweak. But why isn’t simply sampling random examples sufficient? As shown by Marc and Liu
(2000), one of the most common motifs in retinal signaling is the nested feedback synapse, yet its full
topology is rarely observed without ssTEM reconstruction.
Visualization and Annotation.
[one page deleted here]
Individual TEM mosaics can be many gigabytes in size, while final ssTEM volumes can be multiple terabytes. This requires tools optimized for viewing large datasets. A single section can easily exceed the 32bit limit (64K x 64K pixels) of most contemporary image file formats. Even if we exported full resolution mosaics to an image file for use with conventional imaging tools, each 8-bit grayscale 1000-tile mosaic would require 16GB of memory. This is not yet common on desktop computers. To enable real-time
viewing of the completed mosaics we used the established technique (e.g. Google Earth) of constructing an image pyramid for each tile and transforming them with the graphical processing unit. Only tiles
visible on the screen are loaded and displayed, and at that the needed resolution. This makes the viewer
memory footprint essentially volume-independent, providing several advantages. (1) Tile versions enhanced for contrast (ir-clahe), features (ir-blob), or any other processing can be substituted in real time
by pointing the viewer at a different pyramid and using the same transforms. (2) Different transformations
can be substituted to view results at each pipeline stage. (3) Reduced memory and bandwidth requirements of the pyramid-GPU approach make it possible for viewers to work over an HTTP connection.
This is an important feature for collaborative annotation since the terabyte scale of the completed volume
makes it difficult to relocate. (4) Transformations can be applied annotation loci in volume space back
Page 16 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
into section space for persistent storage, allowing one to update transforms or even reorder sections in the
volume without losing the locations of established annotations.
MosaicBuilder is our completed Mac OSX viewer for viewing single sections and was our first visualization/annotation tool. MosaicBuilder imports the images files and transformation definitions generated by the ir-tools and then creates a single project file containing the image pyramid for the section and
any annotations. A single logical file allows the final mosaic to be easily moved and shared among colleagues.
Viking is our web based volume viewer which allows the viewing of volumes over a reasonably fast internet connection. It uses the same image pyramid display strategy as MosaicBuilder, but instead of
importing files into a single package, Viking reads an XML file containing HTTP links to all transforms
and image files. Viking uses the slice-to-slice transforms (ir-stos-grid) to register all slices to a single
reference section. The user can display any section in register with the volume and can easily page to
adjacent sections to track structures. Viking also supports switching to view any grid transformation
generated by the pipeline or alternate image pyramids generated by running image filters over the tiles.
Scripting.
As tools for the framework were developed, we were faced with the option to blend the tools into a single
integrated application with a rich user interface or preserve each algorithm as a separate executable in a
library of tools. We chose the latter as it is more flexible for code refinement and enhancement. However,
by scripting each stage of the pipeline as a separate function in a Python package, we can invoke them
with additional short scripts, automating pipeline execution. This allows building the entire volume startPage 17 of 70
Anderson et al: A Framework for Mapping Neural Circuitry INTRODUCTION
ing from raw microscope output using a single command. Data can be driven from any source into any
stage of the volume building process via addition of a new function. The current scripting approach for
building volumes does have a higher barrier to entry for new users compared to a single application.
Though Python is not nearly as technical as the C++ environment used to create the ir-tools, changing the
pipeline (e.g. adding support for a new microscope platform) does require some programming skill. The
ir-tools have eliminated the most difficult technical challenges to volume construction, but the current
state of the technology still mandates support from skilled programming personnel for the computational
side of the reconstruction effort to be successful.
Framework Parameters.
Though sectioning and staining a 400+ section dataset is in itself a tour-de-force, it is well within the abilities of many ultrastructural laboratories and can be done in a few work days. And while even manual EM
capture can take much longer than sectioning, multiplexing the task across several TEMs and operators
also makes the task of acquiring ssTEM data practical. The image processing step has always been the
real “show-stopper” when large scale ssTEM projects were conceived. Table 3 summarizes the canonical
field, capture, and image processing parameters and timelines for a concrete project: a CN map of the
rabbit retinal inner plexiform layer. This project specifies a resolution of 2.18 nm/pixel, which is sufficient to identify conventional/ribbon synapses and moderate scale gap junctions. In broad terms, an optimal canonical volume can be captured in about 3-5 months with a complete volume build on a single machine.
Page 18 of 70
Anderson et al: A Framework for Mapping Neural Circuitry DISCUSSION
Discussion
Our ultrastructural mapping framework removes three major barriers to large scale ssTEM reconstruction: mosaicking, registration and viewing. While mathematically robust tools have long existed for
analyst-guided nonlinear mosaicking and registration (e.g. PCI Geomatica ®); see Marc and Cameron
2002), and many solid efforts have been made to provide small-volume tools (Fiala 2005), the scale of
ssTEM canonical volume reconstruction precludes a user-guided software solution. The ability of ir-fft /
ir-grid-refine to automatically mosaic individual tiles and ir-stos-brute / ir-stos-refine to automatically
register mosaics means that we have enabled any laboratory to build high-performance ssTEM volumes.
Since scanned film imagery can be readily managed, we have also enabled volume construction and exploration of legacy datasets. Many extremely high quality ssTEM datasets have been produced in the past
three decades (e.g., Stevens et al. 1980a; Smith et al. 1986; Linberg and Fisher 1986; Strettoi et al. 1992;
Kolb and Nelson 1993; Harris et al. 1997; Calkins and Sterling 2007) but their analyses have been restricted to one-time manual tabulations, drawings and representative halftone imagery. Arguably, a key
advance for anatomy would be the ability to allow primary data access, similar to gene accessions. Our
tools provide the framework for such global access via a central repository. And despite the development
of early far-sighted reconstruction frameworks (Stevens et al. 1980b) and subsequent enhancements, the
code, platforms and throughput of those schemata reached neither the performance nor availability required for canonical field reconstructions.
CMP and ssTEM.
The importance of molecular classification of neural data cannot be overstated. Without even partial classification, ssTEM reconstructions remain of limited value. This remains true even with the ability to
nominally identify individual cells by stochastic, multivariate protein expression (Lichtman et al. 2008.)
In contrast, small molecule CMP allows the categorization of class partners in networks before the netPage 19 of 70
Anderson et al: A Framework for Mapping Neural Circuitry DISCUSSION
work is built from ssTEM. Classification by post-hoc unraveling of connectivity is undoubtedly the most
unwieldy and statistically challenging way to identify synaptic partners.
[a half page deleted here]
The retinal CN mapping framework.
Our specific objective in developing these tools is retinal CN mapping. We have begun implementation of
this process by developing a rabbit retinal preparation with strong image segmentation. As shown previously, (Marc 1999a; Marc and Jones 2002; Marc et al. 2007) augmenting CMP libraries with the activity
marker 1-amino-4-guanidobutane (AGB) generates a nearly complete neuronal classification. These signals are also fully compatible with ssTEM (e.g. Jones et al. 2003). We have prepared a single retinal
preparation with sixteen patches each defined as a canonical field for CN mapping. These patches are being sectioned, stained and captured with an estimated completion within a year. The strategy uses horizontal serial sections (sections in the plane of the retina) beginning from either the AC or GC side of the
inner plexiform layer (Fig. 14). Those cellular layers are first classified as CMP bounding layers registered to the ssTEM set of >400 sections, with each section captured in mosaics of 950-1100 tiles. Upon
completion of each volume, it will be available for our own and community browsing and annotation,
described as follows.
A proposal for multi-TEM projects.
Most of the example ssTEM volumes our group has produced so far have been collected with a single
high-performance microscope. We can capture 3000 tiles/day. However, the install base of manual TEM
systems or film-based systems with montaging stages far exceeds those with high-resolution digital cameras. Further, the performance of film is still superior to any digital system and the potential for capturing
high bit-depth scanned images manually augmented with positional metadata makes our ultrastructural
Page 20 of 70
Anderson et al: A Framework for Mapping Neural Circuitry DISCUSSION
framework even more practical. By fragmenting large projects into packets of grids that can be captured
in parallel, it is possible to speed tile acquisition multiplicatively and then distribute tiles to a central resource for volume builds.
The next phase of CN mapping is analysis: building a description of connectivity by tagging cells and
processes and marking synapses. Our goal is not to render 3D ultrastructural images, but rather tabulate
connections within the volume. While it is plausible to develop automated synapse and gap junction
recognition tools (perhaps augmented by molecular probes), those tools are in early development stages.
Our experience is that analysts can perform excellent tagging and synapse markup with these tools. Furthermore, large datasets can be analyzed in parallel by large groups. A wonderful example of this is the
www.galaxyzoo.org project to classify millions of galaxies imaged by various platforms such as the Sloan
Digital Sky Survey (www.sdss.org). Given the importance of mammalian CNS circuitry analysis in neurological disorders, the notion of a single lab performing cradle-to-grave processing on a system is increasingly impractical, as is the notion that computational pattern recognition can adequately screen data
without missing important observations. Human eyes remain the best pattern recognition systems for
ssTEM data. The value of our strategy to develop a scalable, web-compliant viewer for community
markup lies in the fact that new, powerful acquisition platforms (Briggman and Denk 2006; Hayworth et
al. 2006; Knott et al. 2008) and their descendants will soon create an additional deluge of high-quality
data.
Future Developments.
Our next phases of development target six areas. (1) Auto-tracking: Computational techniques for segmenting and tracing individual neurons across a large number of ssTEM sections (Macke et al. 2008;
Jurrus et al. 2008a,b) are critical to speed network data collection. (2) Auto-markup: We are exploring a
Page 21 of 70
Anderson et al: A Framework for Mapping Neural Circuitry DISCUSSION
large library of identified synapses to train automate synapse markup and implement logical rules for synapse identification and signal polarity. These efforts will not replace human tracking and markup in the
short term, but needn’t be extremely efficient to accelerate analysis several-fold. (3) Enhanced pipeline
speed: Even though we can speed mosaic and volume builds by using more machines, we still seek to
vastly improve tool speed to accommodate larger canonical volumes. For example, a canonical volume in
primary visual cortex spanning an ocular dominance column is several times larger than the retinal canonical volume. (4) Simplified pipeline integration: Our efforts to develop acquisition, classification, mosaicking, registration, browsing and markup tools have originated with several developers using different
platforms. How much multi-platform development is justified? Certainly we argue for an open, platform
neutral code base for future development. However, data transport across platforms is now so simple that
it is not essential to spend development resources in replicating applications. Rather, improved pipeline
scripts and interfaces will be our short-term focus. (5) Volume viewing with http compliance: We are creating a volume slice-by-slice viewer that enhances the speed of synapse tagging for building CN maps.
While not essential to our framework, it offers the ability to lever current web protocols to facilitate
community markup. (6) Enhanced CMP power: We are screening libraries of macromolecules (e.g. Marc
et al. 2008; Micheva and Smith 2008) for ssTEM-compliance to augment our CMP library.
Finally, the informatics challenges deserve mention. We are hardly alone in this venture. Many groups
have addressed the informatics of neuroscience data collections, especially the need to aggregate resources, e.g the Neuroscience Information Framework (http://nif.nih.gov), soon to be transferred to the
supervision of the the National Center for Microscopy and Imaging Research at the University of California at San Diego (http://www-ncmir.ucsd.edu/), and the International Neuroinformatics Coordinating Facility (http://incf.org). However, a key issue is a lack of multiresolution annotation tools and task administration for large-scale distributed, multiresolution datasets. Though large scale navigational tools have
been built by the Allen Institute for Brain Science (http://www.brain-map.org/) and Brain Maps develPage 22 of 70
Anderson et al: A Framework for Mapping Neural Circuitry DISCUSSION
oped by Ed Jones and colleagues at Univ California Davis (http://brainmaps.org/), robust tools for community markup that can navigate high resolution TEM datasets have yet to be created or validated. Ontologies for neural systems are under rapid development (http://ccdb.ucsd.edu/CCDBWebSite/sao.html),
which will be essential to building these markup tools. These and other groundbreaking efforts validate
the need for next-generation software including rapid navigation of fused TEM-multivariate molecular
data, true volumetric atlases, graph-theory based analyses and dynamic ontology updating.
Summary.
High-performance ssTEM is a powerful technology coupled to traditionally artisanal data presentation
and analysis methods. These are poorly adapted to large-scale collaborations or high-throughput screening. Many laboratories have attempted to develop stronger tools for ssTEM throughput, but most efforts
were hampered by many barriers: code that did not scale, limited processor speed, expensive storage, and
small canonical volumes. Our framework largely overcomes all computational barriers, providing highly
standardized collaborative environments that enable ssTEM to serve as both a statistically practical CN
mapping tool and an effective screening / phenotyping tool for modern neurogenetics.
Page 23 of 70
Anderson et al: A Framework for Mapping Neural Circuitry MATERIALS AND METHODS
Materials and Methods
Tissue harvest, processing and sectioning.
Retinal samples were taken from Dutch Belted rabbits (Oregon Rabbitry, OR), C57Bl/6J mice (The Jackson Laboratories, Bar Harbor, ME) and TG9N transgenic mice that have an aggressive photoreceptor degeneration and neural remodeling defect (Jones et al. 2003). Light-adapted adult male and female pigmented rabbits tranquilized with intramuscular ketamine/xylazine were deeply anesthetized with intraperitoneal urethane in saline, euthanized by thoracotomy in accord with University of Utah Institutional
Animal Care and Use Committee guidelines, and the eyes immediately injected with 0.1 ml fixative and
an additional 18 Ga needle pressure relief. Rabbit eyes were enucleated, hemisected, and fixed in 1%
formaldehyde, 2.5% glutaraldehyde, 3% sucrose, 1 mM MgSO4, in 0.1 M phosphate or cacodylate buffer,
pH 7.4. Light-adapted mice were rapidly euthanized with halothane or an isoflurane vaporizer. Mouse
eyes were slit at the limbus and injected slowly with 0.1 ml fixative before enucleation and immersion
fixation for 24h. All tissues were osmicated 45-60 min in 0.5-1% OsO4 in 0.1M cacodylate buffer, processed in maleate buffer for en bloc staining with uranyl acetate, and processed for resin embedding as
described in Marc and Liu (2000). Specifically for CN maps, flat resin mounts of retina are remounted for
serial sections in the horizontal plane through the inner plexiform layer (Marc et al. 1990; Marc and Cameron 2002). Vertical sections of mouse retina were used to define normal C57Bl6/j and disordered TG9N
mouse retinal circuitries. Serial sections were cut at 60-90 nm with various models of Leica ultramicrotomes onto carbon-coated Formvar® films on gold slot grids and imaged at 80 KeV in either a Hitachi H600 or JEOL JEM 1400 electron microscope at 5,000-10,000x magnification. Images were captured directly on film (Kodak 4489 Electron Microscope Film) and digitized at 16 bits greyscale on Ultramax or
Creoscitex scanners, or captured digitally on a GATAN Ultrascan 4000 16 megapixel 16-bit camera.
Page 24 of 70
Anderson et al: A Framework for Mapping Neural Circuitry MATERIALS AND METHODS
Image Capture.
Creating the CN map of the retina requires digitizing each tissue section and registering it to its neighbors.
Creating a volume of this scale is a significant undertaking: The CN map for the rabbit inner plexiform
layer in the visual streak requires a volume delimited by a canonical field 250 μm in diameter x 30 μm
high: roughly 1.47 x 106 μm3. A cylindrical volume is a more efficient capture object than rhomboidal
prisms which will have extremities clipped out as sections are rotated during registration. While the issue
is irrelevant at small volumes (Fiala and Harris 2001), it tremendously impacts beam time at canonical
scales. In practice, at a magnification of 5000x on the JEOL JEM-1400, we capture 950-1100 images or
tiles/section and ≈ 333 sections at 90 nm thickness. Storage of unprocessed 16-bit images requires 10.4
terabytes. With a time of capture at roughly 30 s/frame, this requires some 70-100 calendar days on a single TEM, which argues for automated capture scripts and efficient capture geometries. To ensure the images can be positioned properly in the total mosaic, each image has 15% area overlap with its neighbors.
With some of the software tools developed below, it is also evident that such tasks can be parallelized
across microscopes and users.
We capture ssTEM data using SerialEM software developed by D. Mastronarde at the
University of Colorado at Boulder (Mastronarde 2005). Though originally developed for TEM tomography, SerialEM is ideal for large-scale mosaicking. The most recent build version of SerialEM allows definition of multiple circular or polygonal regions of interest on a grid and automates stage drive and image
capture within the regions of interest on the JEOL JEM 1400 TEM (and other recent TEMs as well such
as FEI Tecnais) and, critically, stores stage position metadata for each tile. This greatly reduces the computational cost of the initial positioning of mosaic tiles from O(n2) to O(n). The program includes a scripting capability that provides the flexibility needed to optimize the acquisition strategy, for example, by
focusing only on an appropriately spaced subset of the image tiles. While automated capture is ideal for
Page 25 of 70
Anderson et al: A Framework for Mapping Neural Circuitry MATERIALS AND METHODS
the microscope's Gatan Ultrascan 4000 (4K x 4K) camera, it can also serve on a smaller scale with film
capture.
Mosaicking.
We use the same software suite (ir-tools) to mosaic ssLM and ssTEM data. In an ideal setting, we have
stage position metadata for both kinds of datasets (from Syncroscan and SerialEM), which can be used by
ir-translate to produce precise initial image mosaics. This is then followed by ir-refine grid to correct for
image aberrations in each tile. However, some users may lack capture automation. In this more general
case, an operator manually adjusts the position of each tile, aiming for a specified but often imprecise
overlap between tiles. Even approximate coordinate information is often not recorded. Given a large
number of tiles specified in no particular order, a mosaic must be constructed and individual tiles corrected for distortions, especially subtle electron-optical aberrations (spherical aberrations, magnification
astigmatism, local heating motions, charging motions, etc.) that are often undetectable until precise
alignment is attempted (Fig. 4). It is rare that the magnification in the average TEM is exactly the same at
the field center and edge. At a modest magnification yielding a 7.5 μm wide field (about 5000x), a 1%
error in magnification at the edge of the field (4950x) yields a displacement of over 3 vesicles: potentially
a massive misalignment in tracing circuits. A mosaicking scheme that addresses this general problem is
ir-fft followed by ir-refine-grid.
Image registration and volume construction.
Slice-to-slice image registration is further complicated by differences in imaging modalities (ssLM vs
ssTEM), changing shapes of cells and processes, and physical distortions affecting individual sections
Page 26 of 70
Anderson et al: A Framework for Mapping Neural Circuitry MATERIALS AND METHODS
(folds, stretching). At the boundaries of or intercalated within the ssTEM volume, we collect ssLM data
for CMP analysis. Similar to the multimodal alignment strategies used by Marc and colleagues (Marc and
Liu 2000; Jones et al. 2003), ssLM images are operator-registered to adjacent ssTEM slices using irtweak. However, manually registering large ssTEM volumes is impractical and automated slice-to-slice
tools are needed: (ir-stos-brute, ir-stos-refine).
CMP and classification.
CMP is a method to extract quantitative molecular signatures from cells or even cellular processes (Marc
et al. 1995; Marc and Jones 2002) and fuse molecular signature data with ssTEM datasets (Marc and Liu
2000; Jones et al. 2003). ssLM samples (40-90 nm) are arrayed on 12-spot Teflon®-coated slides (CelLine; Erie Scientific Inc), fully sodium ethoxide etched, probed with IgGs targeting various molecules,
and visualized with silver-intensified 1.4 nm gold granules conjugated to goat anti-rabbit or goat antimouse IgG (Nanoprobes, Yaphank, NY). Immunoreactivity in these samples is a pure surface phenomenon independent of section thickness (Marc et al. 1995). The IgG library includes (but is not limited to)
anti-L-aspartate [IgG D], anti-L-glutamate [IgG E], anti-glycine [IgG G], anti-L-glutamine [IgG Q], antitaurine [IgG τ], and anti-GABA [IgG γ] (Signature Immunologics Inc, Salt Lake City, UT). All data were
captured as 8-bit 1388 pixel x 1036 line frames under voltage regulated tungsten halogen flux with a variation of 1.2 ± 0.6%/min (mean ± sd). Image mosaic tiles were captured with a Peltier-cooled QImaging
Fast 1394 QICAM (QImaging, Burnaby, BC, Canada) and a Syncroscan montaging system (Synoptics
Inc, Frederick, MD) on a Scan 100x100 stage (Märzhäuser Wetzlar GMBH & Co, Germany). ssLM mosaics were prepared using various ir-tools and aligned using ir-tweak (see below). CMP classification
(clustering, histogram analysis, PCA, etc.) is performed on the CMP dataset using CMPView (© James
Anderson 2008) to phenotype processes and cells. CMPview is built in MATLAB2008a. Clustering is
based on the robust K-means and isodata algorithms (see Marc et al. 1995; Marc and Jones 2002), aug-
Page 27 of 70
Anderson et al: A Framework for Mapping Neural Circuitry MATERIALS AND METHODS
mented with interactive histogram splitting tools. CMPView operates on either pixel-based or objectbased images. The intermediate product of relevance for this paper is the production of a high-resolution
theme map of cell classes that is then registered to ssTEM data (Marc and Liu 2000; Jones et al. 2003).
Finally, image analysis for characterization of mosaicking and registration performance was performed
using ImageJ (Rasband 2007).
Image Browsing and Annotation.
Once large mosaics are built, it is essential to have tools to browse, annotate and record annotations.
Some mosaics are 15-30 gigabyte datasets, which are unmanageable by most commercial imaging tools.
We have developed two applications, MosaicBuilder (Mac OS X) and Viking (Win) , that allow a single
slices to be viewed, browsed and annotated. By using image pyramids, these tools quickly navigates between low and high magnification views. Volumes are even more challenging, as they expand into terabyte size and Viking allows rapid paging through the build over an HTTP connection, enabling single image repositories to serve multiple users.
Managing the processing pipeline.
This twelve stage framework requires significant user management and exceptional digital hygiene in data archiving, access and revision control. Since each computational stage invokes a different program and
occasional transitions between data formats we elected to use the Python scripting language and Python
Imaging Library to bridge stages. The scripts perform tasks such as conversion from microscope-specific
formats to the plain text “.mosaic” format of ir-tools, image cropping, contrast enhancement, downsampling and launching multiple instances of single-threaded algorithms to ensure each CPU core is fully
Page 28 of 70
Anderson et al: A Framework for Mapping Neural Circuitry MATERIALS AND METHODS
utilized. Pipeline automation improves throughput, eases integration, produces consistent results, and stabilizes performance.
Page 29 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FUNDING, etc.
Funding
NEI R01 EY02576, R01 EY015128, P01 EY014800 (REM); support from the Cal and JeNeal Hatch
Presidential Endowed Chair (REM); an unrestricted grant from Research to Prevent Blindness to the Moran Eye Center; a Research to Prevent Blindness Career Development Award (BWJ); NIBIB EB005832
(TT). Tolga Tasdizen would like to acknowledge the support of the Utah Science Technology and Research Initiative (USTAR). D. Mastronarde’s work was supported by Grant Number P41RR00592 to A.
H. Hoenger from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH). This paper's contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH.
Author Roles:
Anderson: 1° Author, CMP, ssTEM, imaging, code development, pipeline development
Jones: 2° Author, CMP, ssTEM, imaging
Yang: 3° Author, CMP, ssTEM
Shaw: 3° Author, CMP, imaging
Watt: 2° Author, CMP, ssTEM, imaging
Koshevoy: 3° Author, code development, pipeline development
Spaltenstein: 2° Author, code development, pipeline development
Jurrus: 3° Author, code development
Kannan: 3° Author, code development
Page 30 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FUNDING, etc.
Tasdizen: 1° Author, code development, pipeline development
Whitaker: 3° Author, code development, pipeline development
Mastronarde: 2° Author, ssTEM, code development, pipeline development
Marc: 1° Author, CMP, ssTEM, imaging, code development
Role Definitions:
1° Author: Generated primary manuscript
2° Author: Generated manuscript sections and contributed substantial revisions
3° Author: Generated area-specific manuscript revisions
CMP: computational molecular phenotyping - sample array probing, visualization, capture
ssTEM: serial section TEM - ultrathin section arrays, TEM imaging
imaging: ssLM (serial section LM) capture, image mosaicking, registration and markup
code development: primary code for CMP, mosaicking, registration, browsing, markup
pipeline development: code / scripts for interoperability and automation
Page 31 of 70
Anderson et al: A Framework for Mapping Neural Circuitry REFERENCES
References
Berson DM (2008) Retinal ganglion cell types and their central projections. In: Masland RH, Albright T,
editors. The Senses: A comprehensive reference, Vision. Amsterdam: Elsevier. pp. 491-519.
Briggman KL, Denk W (2006) Towards neural circuit reconstruction with volume electron microscopy
techniques. Current Opinion in Neurobiology 16: 562-570.
Calkins DJ, Sterling P (1996) Absence of spectrally specific lateral inputs to midget ganglion cells in
primate retina. Nature 381: 613-615.
Calkins DJ, Sterling P (2007) Microcircuitry for two types of achromatic ganglion cell in primate fovea. J
Neurosci 27: 2646-2653.
Chklovskii DB, Mel BW, Svoboda K (2004) Cortical rewiring and information storage. Nature 431: 782788.
Crawley JN (2008) Behavioral phenotyping strategies for mutant mice. Neuron 57(6): 809-818.
Deans MR, Volgyi B, Goodenough DA, Bloomfield SA, Paul DL (2002) Connexin36 is essential for
transmission of rod-mediated visual signals in the mammalian retina. Neuron 36: 703-712.
Deerinck TJ, Giepmans BN, Smarr BL, Martone ME, Ellisman MH (2007) Light and electron microscopic localization of multiple proteins using quantum dots. Methods Mol Biol 374: 43-53.
Denk W, Horstmann H (2004) Serial block-face scanning electron microscopy to reconstruct threedimensional tissue nanostructure. PLoS Biol. 2:e329.
Eglen SJ, Raven MA, Tamrazian E, Reese BE (2003) Dopaminergic amacrine cells in the inner nuclear
layer and ganglion cell layer comprise a single functional retinal mosaic. J Comp Neurol 466:
343-355.
Page 32 of 70
Anderson et al: A Framework for Mapping Neural Circuitry REFERENCES
Famiglietti EV Jr, Kolb H (1975) A bistratified amacrine cell and synaptic circuitry in the inner plexiform
layer of the retina. Brain Res 84: 293-300.
Fiala JC (2002) Three-dimensional structure of synapses in the brain and on the web. 2002 World Congress on Computational Intelligence. May 12-17, Honolulu, Hawaii.
Fiala JC (2005) Reconstruct: a free editor for serial section microscopy. J Microscopy 218: 52-61.
Fiala JC, Harris KM (2001) Extending unbiased stereology of brain ultrastructure to three-dimensional
volumes. J Am Med Inform Assoc 8: 1-16.
Harris KM, Jensen FE, Tsao B (1997) Three-dimensional structure of dendritic spines and synapses in rat
hippocampus (CA1) at postnatal day 15 and adult ages: implications for the maturation of synaptic physiology and long-term potentiation. J Neurosci 12: 2685-2705.
Harris KM, Perry E, Bourne J, Feinberg M, Ostroff L et al. (2006) Uniform serial sectioning for transmission electron microscopy. J Neurosci 26: 12101-12103.
Hayworth KJ, Kasthuri N, Schalek R, Lichtman JW (2006) Automating the collection of ultrathin serial
sections for large volume TEM reconstructions. Microscopy and Microanalysis 12: 86-87.
Hill E, Kalloniatis M, Tan S-S (2000) Glutamate, GABA and precursor amino acids in adult mouse neocortex: Cellular diversity revealed by quantitative immunocytochemistry. Cereb Cortex 10: 11321142.
Hill E, Kalloniatis M, Tan S-S (2001) Cellular diversity in mouse neocortex revealed by multispectral
analysis of amino acid immunoreactivity. Cereb Cortex 11: 679-690.
Horton JC, Hocking DR (1996) Intrinsic variability of ocular dominance column periodicity in normal
macaque monkeys. J Neurosci 16:7228 –7339.
Page 33 of 70
Anderson et al: A Framework for Mapping Neural Circuitry REFERENCES
Jones BW, Watt CB, Frederick JM, Baehr W, Chen CK et al. (2003) Retinal remodeling triggered by
photoreceptor degenerations. J Comp Neurol 464: 1-16.
Jurrus E, Whitaker RT, Jones BW, Marc RE, Tasdizen T. (2008a) An optimal-path approach for neural
circuit reconstruction. Proceedings of the IEEE International Symposium on Biomedical Imaging
(ISBI): From Nano to Macro: 1609-1612.
Jurrus E, Hardy M, Tasdizen T, Fletcher PT, Koshevoy P, Chien CB, Denk W, Whitaker R (2008b) Axon
tracking in serial block-face scanning electron microscopy. Med Image Anal. Jun 5. [Epub ahead
of print]
Kalloniatis M, Marc RE, Murry RF (1996) Amino acid signatures in the primate retina. J Neurosci 16:
6807-6829.
Kolb H, Nelson R (1993) OFF-alpha and OFF-beta ganglion cells in cat retina: II. Neural circuitry as revealed by electron microscopy of HRP stains. J Comp Neurol 329: 85-110.
Kolb H, Famiglietti EVJ (1974) Rod and cone pathways in the retina of the cat. Invest Ophthal 15: 935946.
Klug K, Herr S, Ngo IT, Sterling P, Schein S (2003) Macaque retina contains an S-cone OFF midget
pathway. J Neurosci 23: 9881-9887.
Knott G, Marchman H, Wall D, Lich B (2008) Serial section scanning electron microscopy of adult brain
tissue using focused ion beam milling. J Neurosci 28: 2959-2964.
Lichtman JW, Livet J, Sanes JR (2008) A technicolour approach to the connectome. Nat Rev Neurosci 9:
417-422.
Linberg KA, Fisher SK (1986) An ultrastructural study of interplexiform cell synapses in the human retina. J Comp Neurol 243:561-576.
Page 34 of 70
Anderson et al: A Framework for Mapping Neural Circuitry REFERENCES
Lund JS, Angelucci A, Bressloff PC (2003) Anatomical substrates for functional columns in macaque
monkey primary visual cortex. Cereb Cortex 13: 15-24.
Macke JH, Maack N, Gupta R, Denk W, Schölkopf B, Borst A (2008) Contour-propagation algorithms
for semi-automated reconstruction of neural processes. J Neurosci Methods 167: 349-357.
MacNeil MA, Heussy JK, Dacheux RF, Raviola E, Masland RH (1999) The shapes and numbers of amacrine cells: matching of photofilled with Golgi-stained cells in the rabbit retina and comparison
with other mammalian species. J Comp Neurol 413: 305-326.
MacNeil MA, Heussy JK, Dacheux RF, Raviola E, Masland RH (2004) The population of bipolar cells in
the rabbit retina. J Comp Neurol 472: 73-86.
Marc RE (1989) The role of glycine in the mammalian retina. In: Osborne N, Chader G, editors. Progress
in Retinal Research. London: Pergamon Press. pp. 67-107.
Marc RE (1999a) Kainate activation of horizontal, bipolar, amacrine, and ganglion cells in the rabbit retina. J Comp Neurol 407: 65-76.
Marc RE (1999b) Mapping glutamatergic drive in the vertebrate retina with a channel-permeant organic
cation. J Comp Neurol 407: 47-64.
Marc RE, Liu WL (1985) (3H) glycine-accumulating neurons of the human retina. Journal of Comparative Neurology 232: 241-260.
Marc RE, Liu W (2000) Fundamental GABAergic amacrine cell circuitries in the retina: nested feedback,
concatenated inhibition, and axosomatic synapses. J Comp Neurol 425(4): 560-582.
Marc RE, Cameron DA (2002) A molecular phenotype atlas of the zebrafish retina. J Neurocytol 30: 593654.
Marc RE, Jones BW (2002) Molecular phenotyping of retinal ganglion cells. J Neurosci 22: 413-427.
Page 35 of 70
Anderson et al: A Framework for Mapping Neural Circuitry REFERENCES
Marc RE, Murry RF, Basinger SF (1995) Pattern recognition of amino acid signatures in retinal neurons.
J Neurosci 15: 5106-5129.
Marc RE, Jones BW, Watt CB, Strettoi E (2003) Neural remodeling in retinal degeneration. Prog Ret Eye
Res 22: 607-655.
Marc RE, Liu WL, Kalloniatis M, Raiguel SF, van Haesendonck E (1990) Patterns of glutamate immunoreactivity in the goldfish retina. J Neurosci 10: 4006-4034.
Marc RE, Jones BW, Watt CB, Vazquez-Chona F, Vaughan DK et al. (2008) Extreme retinal remodeling
triggered by light damage: implications for age related macular degeneration. Molecular Vision
14: 782-806.
Masland RH (2001) Neuronal diversity in the retina. Curr Opinion Neurobiol 11: 431-436.
Mastronarde DN (2005) Automated electron microscope tomography using robust prediction of specimen
movements. J Struct Biol 152: 36-51.
Micheva KD, Smith SJ (2007) Array tomography: A new tool for imaging the molecular architecture and
ultrastructure of neural circuits. Neuron 55: 25-36.
Miller RF, Bloomfield, SA (1983) Electroanatomy of a unique amacrine cell in the rabbit retina. Proc
Natl Acad Sci USA 80: 3069-3073.
Müller F, Wässle H, Voigt T (1988) Pharmacological modulation of the rod pathway in the cat retina. J
Neurophysiol 59: 1657-1672.
Otsuna H, Ito K (2006) Systematic analysis of the visual projection neurons of Drosophila melanogaster I: Lobula-specific pathways. J Comp Neurol 497: 928-958.
Peters A, Cifuentes JM, Sethares C (1997) The organization of pyramidal cells in area 18 of the rhesus
monkey. Cereb Cortex 7: 405-421.
Page 36 of 70
Anderson et al: A Framework for Mapping Neural Circuitry REFERENCES
Rall W (1995) “The Theoretical Foundation of Dendritic Function: Selected Papers of Wilfred Rall with
Commentaries.” Segev I, Rinzel J, Shepherd GM, editors. Cambridge, MA. The MIT Press.
Rasband WS (2007) ImageJ. U. S. National Institutes of Health
Reese B (2008) Mosaics, tiling and coverage by retinal neurons. In: Masland RH, Albright T, editors. The
Senses: A comprehensive reference, Vision. Amsterdam: Elsevier. pp. 439-456.
Rockhill RL, Daly FJ, MacNeil MA, Brown SP, Masland RH (2002) The diversity of ganglion cells in a
mammalian retina. J Neurosci 22: 3831–3843.
Smith RG, Freed MA, Sterling P (1986) Microcircuitry of the dark-adapted cat retina: functional architecture of the rod-cone network. J Neurosci 6: 3505-3517.
Soucy E, Wang Y, Nirenberg S, Nathans J, Meister M (1998) A novel signaling pathway from rod photoreceptors to ganglion cells in mammalian retina. Neuron 21: 481-493.
Sporns O, Tononi G, Kötter R (2005) The human connectome: A structural description of the human
brain. PLoS Computational Biology 1: e42.
Sterling P (2004) How retinal circuits optimize the transfer of visual information. In: Chalupa LM, Werner J, editors. The Visual Neurosciences, Vol. 1 Cambridge: MIT Press. pp. 234–259.
Sterling P, Lampson LA (1986) Molecular specificity of defined types of amacrine synapse in cat retina. J
Neurosci 6: 1314-1324.
Stevens JK, Davis TL, Friedman N, Sterling P (1980a) A systematic approach to reconstructing microcircuitry by electron microscopy of serial sections. Brain Res 2: 265-293.
Stevens JK, McGuire BA, Sterling P (1980b) Toward a functional architecture of the retina: serial reconstruction of adjacent ganglion cells. Science 207: 317-319.
Page 37 of 70
Anderson et al: A Framework for Mapping Neural Circuitry REFERENCES
Strettoi E, Raviola E, Dacheux RF (1992) Synaptic connections of the narrow-field, bistratified rod amacrine cell (AII) in the rabbit retina. J Comp Neurol 325: 152-168.
Tsukamoto Y, Morigiwa K, Ueda M, Sterling P (2001) Microcircuits for night vision in mouse retina.
Journal of Neuroscience 21(21): 8616-8623.
Vaney DI (1986) Morphological identification of serotonin-accumulating neurons in the living retina.
Science 233: 444-446.
Vaney DI (1990) The mosaic of amacrine cells in the mammalian retina. Progress in retinal research. New
York: Pergamon. pp. 49–100.
Weiss TF (1996) “Cellular Biophysics: Electrical Properties” Cambridge, MA. The MIT Press.
Page 38 of 70
Anderson et al: A Framework for Mapping Neural Circuitry COMPETING INTERESTS & ABBREVIATIONS
Competing Interests
Robert E. Marc is a principal of Signature Immunologics. All other authors declare no other competing
interests.
Abbreviations
AC, amacrine cell; AGB 1-amino-4-guanidobutane; BC, bipolar cell; CMP, computational molecular
phenotyping; CN, complete network; D, aspartate; E, glutamate; en endothelial cell; GC, ganglion cell; G,
glycine; γ, GABA; IgGs, immunoglobulins; IgG D, anti-L-aspartate; IgG E, anti-L-glutamate; IgG G, anti-glycine; IgG Q, anti-L-glutamine; IgG τ, anti-taurine; IgG γ, anti-GABA; L, Linux; M, Mac OS X;
mGluR6, type 6 metabotropic glutamate receptor; rgb, red, green, blue image mapping; RPE retinal pigmented epithelium; ssLM, serial section light microscopy; ssTEM, serial section transmission electron
microscopy; τ, taurine; TEM, transmission electron microscopy; W, Wintel.
Page 39 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Figure Legends
Figure 1. Neuronal elements for building networks.
A five element micronetwork involving one amacrine cell (A) mediating cross talk between two vertical
channels (i,j) with bipolar cells (B) synaptically driving ganglion cells (G). There are eight discrete amacrine cell connnections (0-7) and the network can be configured in 90 formal motifs, at least 40 of which
are of biological significance. Solid dots and arrows are excitatory; open dots and arrows are inhibitory.
Figure 2. Voronoi tiling of ganglion cell (GC) classes in the rabbit retina.
Panels 1-15 represent GC or AC groups defined by CMP (Marc and Jones 2002). The bottom panel is the
aggregate GC pattern. Each class forms its own independent tiling. Class 6 (red) is the most orderly. Class
9 (yellow) is the rarest and defines a canonical field for the GC cohort, containing three members of the
class. Class 14 (cyan) is the densest and is the starburst displaced AC population. The canonical field superimposed on the entire GC cohort (green) captures >100 cells and guarantees a robust sample of all
network types. Modified from Marc and Jones (2002).
Figure 3. The workflow for the ssTEM ultrastructural circuitry framework.
Parallel serial section grid (ssTEM) and slide (ssLM) libraries are built. The ssLM libraries define either
the bounds of the canonical field or are intercalated. Each library is acquired as a set of tiles mosaicked by
ir-tools. ssLM and ssTEM mosaicks are registered by ir-tweak and ssTEM volumes built with ir-stos applications. CMP classified ssLM imagery is merged with the volume to tag neurons and processes. The
volumes are browsed with MosiacBuilder/Viking for process tracking and annotation.
Figure 4. Distortions in overlapping tile regions visualized on film capture of ssTEM data.
A typical low magnification (3000x) field for synaptic screening in the rabbit inner plexiform layer was
captured on a Hitachi H-600 with film images at ≈ 25% overlap. Panels A and B represent part of the
Page 40 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
overlapping fields with slightly different densities due to the auto-exposure of different images. Panels C
and D represent the difference of panels A and B after translational/rotational best alignments for two regions (circles 1 and 2). When imagery in circle 2 is aligned best (panel C) the regions in circle 1 are
shifted and have a higher image dispersion. The same is true when imagery in circle 1 is aligned best
(panel D). The quality of alignment is quantified by normalized intensity histograms of corresponding
patches. When spots are well-aligned (blue) the histograms are narrow, when poorly aligned (yellow) the
intensity variance is high. The two spots are 6 μm apart. The histograms are peak normalized pixel number (ordinate) versus pixel value (abscissa, 0-255). Arrows indicate various ribbon and conventional synapses.
Figure 5. Recovery of distortion errors in tile overlaps with ir-fft and ir-grid-refine.
Panels A and B are mirror images of the transparent overlays of two overlapping tiles, both with best
alignment centers on a large mitochondrion (white spot). Panel A was auto-registered by ir-translate and
many membranes appear as double images (arrows) due to non-linear image distortions between the image pairs. Panel B was registered with ir-grid-refine yielding improved membrane definition, even at very
low magnification (resolution is about 5nm/pixel, which accounts for the blurring). This is a worst-case
scenario. With higher resolutions (more pixels) recovery is even more effective. The inset panels are
high-pass 3x3 pixel filtered patches of the same region, showing severe moiré defects in panel A. Scale =
2 μm.
Figure 6. Representative tile overlaps randomly selected from a 1000 tile array.
A: A randomly selected region of rabbit retinal inner plexiform layer displaying parts section #105 containing many tiles. The overlaps are invisible at this magnification. Image width = 46.6 μm, M Müller cell
processes. B. Randomly selected boxed region from A containing tile overlaps, width = 4.76 um. Arrows
indicate a corner region among four tiles. A pair of vesicles (circled) is enlarged in the inset showing a
Page 41 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
misalignment between upper and lower tiles (arrows) corresponding to 7.8 nm or roughly 1/3 vesicle.
Most tiles have no measurable misalignment. AC, amacrine cell terminal.
Figure 7. Auto-registration of ssLM image tiles with ir-translate and ir-grid-refine.
A thin 200 nm section was probed with anti-AGB IgGs after excitation of the rabbit retinal GC layer
(Marc 1999a), visualized by silver-intensified immunogold detection (Marc et al. 1995), captured on a
SyncroscanRT montaging system (182 nm/pixel), and aligned with Syncroscan software (A,C) and irtranslate / ir-grid-refine (B,D). At low magnification, both images appear perfect, but at near pixel level,
many small defects emerge in the Syncroscan-aligned mosaic (arrows in A,C) that include 200-2000 nm
image shifts and image blurring (box). By using the raw image tiles and their metadata, ir-translate and
ir-grid-refine create defect-free mosaics. While the image shifts shown in A are irrelevant (indeed invisible) for image display, they are highly corrupting in mathematically sensitive procedures such as clustering and multimodal alignments with ssTEM datasets. Panels A and B are bright-field images and C and D
are contrast-stretched Laplacian filtered images that enhance discontinuities and clearly show alignment
defects. The circle in image D represents a lysosome of approximately 200 nm diameter. Its contrast is
better preserved in the ir-translate and ir-grid-refine mosaic. Scale = 20 μm.
Figure 8. Registering ssTEM image tiles with ir-tweak.
The entire image represents two windows of the ir-tweak interface. The top window shows two serial sections from a manual film capture with tiles in different orientations (arrows), the left being the fixed and
the right the moving or warped image. Successive control points (dots) entered on the fixed image by the
user are predictively placed on the moving image based on the model calculated from all previous points,
with a thin-plate spline strategy for accommodating local warps. The bottom panel shows the superimposed fixed (blue) and warped (orange) in real-time.
Figure 9. A QuickTime ® movie of a volume slice through a mouse retinal microneuroma.
Page 42 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
The microneuroma is 27 μm long and 16 μm wide at mid-length. The volume slice spans 45 sections, 90
nm each for a thickness of 4 μm. The original data were scanned from manually acquired TEM film images, aligned using ir-stos tools, and converted to a smaller movie using ir-stom (slice-to-movie) which
generates 90 frames. Each slice is cropped such that only pixels with valid mapping onto every slice in
the volume are kept. The raw serial image output from ir-stom was imported into QuickTime Pro v7 and
saved as a movie.
Movie link: http://prometheus.med.utah.edu/~marclab/cn_plos_movies.html.
Figure 10. Automatic neural volume assembly. A,B. Automatically mosaicked slice 1 (A) and 10 (B)
from an automatically registered 20 slice volume. Scale, 10 μm. C,D. Slices 5 (C) and 9 (D) from the
boxed regions in A and B showing that definition of reciprocal synapse identity requires ssTEM data. AC
process A1 receives excitatory synaptic ribbon (r3) input from BC terminal B1 in panel C, but does not
show a feedback synapse until slice 9 (panel D). Similarly, AC process A11 show a feedbacks synapse in
slice 5 (panel C) but does not receive excitatory synaptic ribbon (r4) input from BC terminal B1 until slice
9 (panel D). Arrows denotes synaptic polarity. Scale, 1 μm. E. Partial summary of connections from
markup of the volume. BC B1 drives five GABAergic (γ) ACs with reciprocal feedback and five glycinergic (gly) ACs. Four putative γ ACs provide feedforward inhibition onto some of the gly AC profiles.
The origin of those processes is yet unknown.
Figure 11. Automatic registration of canonical scale mosaics. The left two columns are six 1000+ tile mosaics from a series of over 120 horizontal plane 70 nm sections of the rabbit inner nuclear layer (sections
1, 20, 40, 61, 80, 103) spanning over 9 μm. Each mosaics is 250 μm wide. The middle column shows mosaics 20, 40, 61, 80, 103 with a colored overlay of the tile adjustment mesh (the true subtile mesh is much
finer). The high contrast version of the mosaic 20 mesh shows that the bounding and bisecting lines only
slight deviations from linearity due to slice-to-slice distortions. However, these do not accumulate. The
Page 43 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
arrow indicates a patch of the true mesh density. The right column (53 μm wide) is a magnified region of
each slice showing the excellent cell-to-cell and subcellular alignment achieved by purely automatic image registration with ir-tools.
Figure 12. Fusion of ssLM CMP and ssTEM data.
The mouse retinal microneuroma data shown in previous figures are comprised of the initial section in the
ssTEM set (A), a set of four bounding 90 nm ssLM sections visualized by IgG γ, IgG G, IgG E and IgG τ,
all registered by ir-tweak, mapped as γGE :: rgb (B) and γτE :: rgb (C) triplets, converted by cluster
analysis in CMPView into a classified theme map of 9 discrete superclasses (see text).
Figure 13. Fusion of ssLM CMP and ssTEM data at the synaptic scale.
A. A 20 μm wide strip from a 750 μm wide mouse retinal dataset of the inner plexiform layer extending
from the AC layer (top) to the GC layer (bottom). The color map is a γGτ :: rgb mapping visualized as a
transparency overlay onto the TEM data. Scale 10 μm. B. The synaptic terminal of an ON cone BC (as
identified by its signature and the region of the inner plexiform layer from which it was sampled, outlined
in Panel A with a synaptic ribbon (r) presynaptic to two profiles (arrows) and postsynaptic to profiles γ1
and G1. C. The ssLM CMP overlay, showing the characteristic blue τ+ signature of BCs, two different
red GABAergic profiles (γ1 and γ2), and the green glycinergic profile (G1). D. Enlargement of the classic
BC ↔ AC GABAergic reciprocal feedback synapse. Scale 1 μm for both B and C, and 400 nm for D.
Figure 14. Browsing 30 Gigabyte datasets with MosaicBuilder.
For this image, 1001 TEM images were captured at 5000x (2.18 nm/pixel) with SerialEM, tiled into a
single mosaic dataset by ir-translate and ir-grid-refine, and visualized with MosaicBuilder. The overlap
of each image with its neighbors is shown in panel A and the entire seamless image visualized in B. The
polygonal region in B is visualized in C simply by “zooming” in MosaicBuilder and the classifications
Page 44 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
obtained in Fig. 11 annotated onto the initial section of the dataset. The rectangles in C are further enlargements that extend to the synaptic level. The ease of use of MosaicBuilder in synaptic markup is
shown in the QuickTime ® movie of Supplementary Figure 3.
Figure 15. The Retinal CN mapping framework.
Canonical fields of rabbit retina are being sectioned from the GC to the AC layers at 90 nm and tiled mosaics acquired for volume assembly. Bounding the ssTEM set are classified sets of GCs (top) and ACs
(bottom) whose processes enter the field and can be tagged and tracked. The GC patch is shown as a
theme map and the AC patch as a γ.AGB.E :: rgb mapped image. At 5000x it is possible to unambiguously identify both conventional and ribbon synapses as well as most gap junctions that exceed ≈ 200 nm in
lateral extent. As CMP can be performed on sections as thin as 40 nm, selected molecular signals can be
intercalated into ssTEM sets without significant disruption of volume builds by saving spaced sections for
CMP, using them if needed, or reinserting them as ssTEM elements if not.
Supplementary Material
Supplementary Item 1
Figure 1. How many network motifs can you make with five retinal neurons?
Assume that you have 5 kinds of cells: 2 different BCs (Bi, Bj), 2 different GCs (Gi, Gj), 1 AC (A) connecting them.
The required vertical channels are Bi → Gi; Bj → Gj
The lateral channel options are:
A → none: processes 0 and 1
A → Bi,Bj (feedback): processes 2 and 3
A → Gi, Gj (feedforward): processes 4 and 5
A → A (nested feedback): loop x
Page 45 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
We start with a summary of submotifs. There are 12 allowed BCAC submotifs:
motif
label
0
i mono input, no feedback
1
j mono input, no feedback
01
dual input, no feedback
02
i mono input, in-channel feedback
03
i mono input, cross-channel feedback
12
j mono input, cross-channel feedback
13
j mono input, in-channel feedback
012
dual input, feedback i
013
dual input, feedback j
023
i mono input, dual feedback
123
j mono input, dual feedback
0123
dual input, dual feedback
There are 4 possible AC → GC feedforward submotifs.
A → none
A → Gi
A → Gj
A → Gi & Gj
There are 2 AC → AC nested feedback submotifs.
A → none
A→A
Page 46 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
The total combined motif number is
((12 BC → AC)(4 AC →GC))-3)(2 AC → AC) = (45 basic submotifs)(2 nested forms) = 90
In connective terms this calculation includes -3 because motifs 0,1, 01 cannot also have AC → no GCs.
However, there are AC volume release mechanisms (peptides, NO, monoamines) that could have this motif. In a strict biological sense, there could be least 96 motifs. But in practice we might require constraints
(see Supplementary Fig. 2).
Supplementary Item 2
Figure 2. How many motifs can you make with five or six retinal neurons?
Even if we add the strong biological constraint that a minimal submotif must have a path to both GCs and
one cross channel AC path (i.e. Bi → Ai → Bj), there are still many possible motifs. Here we redraw the
pattern in Fig. 1 to simplify the analysis. A. This is the complete five neuron network. Three connections
(5AA, 4Ai, 4Aj) can be independently removed: 23 = 8 motifs are possible with these connections varied.
B. After removing 5AA, 4Ai, and 4Aj we have a basic submotif (1) that can be decimated by removing
connection pairs. Submotifs 1-5 satisfy our constraint, thus admitting 40 biologically likely networks.
Submotifs 6 and 7 are biologically plausible, but don’t form a single network.
What happens if we add one more AC? C. This is the complete 40 motif network. D. If one more AC is
added (Aj), it represents an independent 40 motif network and the total possible paths becomes 402 =
1600. E. By adding connections between Ai and Aj increases the number 4-fold (none, Ai → Aj, Aj →
Ai, Ai ↔ Aj) or 6400. Thus the diversity of possible connections is so high that the most effective way to
discover neural circuits is to actually map them by ssTEM
Supplementary Item 3.
QuickTime ® movie of auto-registered synaptic volume.
Page 47 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
This movie is a 20 slice series (looped back & forth) through BC terminal B1 (highlighted in orange)
shown in Fig. 10 of the manuscript. Each instance of a BC synaptic ribbon in B1 is denoted by an aqua
dot for the duration of the ribbon across slices.
Movie link: http://prometheus.med.utah.edu/~marclab/cn_plos_movies.html.
Supplementary Item 4.
QuickTime ® movie of the MosaicBuilder interface.
Here we zoom to the synaptic level with structural markups (captured in real-time by iShowU,
http://shinywhitebox.com). Tiles from a vertical section of rabbit retina prepared for assessment of CN
capture parameters were captured at 5000x with SerialEM, pipelined through ir-translate and ir-refinegrid into a large mosaic and visualized in MosaicBuilder for synaptic markup. The movie shows sliderbased zooming of the dataset from a moderate scale up to synaptic level, first to a characteristic AC →
GC synapse in the OFF sublamina. The synapse is marked by a red pin in the synaptic cleft and an arrow
denoting polarity. These markup metadata are recorded in an *.xml export. The field is then zoomed out
and repositioned to show a characteristic AC → ON cone BC synapse. Two blurring events in zooming
represent the transitions in resolution in the image pyramid scheme.
Movie link: http://prometheus.med.utah.edu/~marclab/cn_plos_movies.html.
Supplementary Item 5.
ir-fft
The purpose of ir-fft is to find the ordering of random image tiles. The first sub-problem is to find pairs of
overlapping tiles. The main constraint at this stage of the algorithm is computational complexity because
this procedure will be applied to approximately n2 pairs where n is the total number of tiles in a section. If
we restrict the class of allowed coordinate transformations between pairs of tiles to translation, a fast
closed-form solution exists (Girod and Kuo 1989). Let F[S](u,v) denote the two-dimensional Fourier
transform of image S(x,y). For simplicity we refer to F[S](u,v) as F[S]. The Fourier transform shift properPage 48 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
ty (Gonzalez and Woods 2008) provides a simple rule relating the Fourier transforms of an image and a
shifted version of it:
F[S(x-xo, y-yo)] = e-j(uxo+vyo) F[S(x,y)] (1)
Following Girod and Kuo (1989), we define a displacement probability image between two images Si and
Sj :
Pi,j (x,y) = Real [F-1 [ Φij (Φii Φjj)-1/2] (2)
where Φij = F[Si ] F[Sj ]* is the cross-correlation of the two images, F[Sj]* being the complex conjugate.
Similarly, Φii and Φjj are the auto-correlations. Using the Fourier transform shift property defined in
Equation (1) it can be shown that if Si and Sj differ only by a displacement vector (xo,yo), the displacement
probability image in Equation (2) reduces to a Dirac function
Pij (x,y) = δ (x-xo,y-yo),
assuming periodicity of the images. In other words, Pij (x,y) will have a non-zero entry for only a single
displacement vector. For partially overlapping images, the exact relationship Pij (x,y) = δ(x-xo,y-yo) does
not hold. However, if the amount of overlap is sufficient, the maximum of Pij (x,y) should correspond to
the true displacement between images Si and Sj. In practice, finding this maximum is non-trivial because
for most TEM images P is very noisy. Also, Pij for two non-overlapping images may contain several
weak maxima, or none at all. These problems were not addressed in Girod and Kuo (1989).
We have found four steps necessary to identify the location of the correct maxima in P. The first is to presmooth images to reduce noise. The second is to select and apply a threshold to P to isolate global peaks.
We choose the threshold at the 99th percentile of the histogram of P: i.e. 1% of the total pixels in P are
considered as possible maxima locations. Third, we locate clusters of at least five 8-connected pixels, inPage 49 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
dicating a strong maximum. If the maxima are scattered across P, it is likely there is no strong maximum.
Maxima coordinates are calculated as the centers of mass of the corresponding clusters. The final step is
to verify which, if any, of these maxima is the true displacement between the image pair. Nonoverlapping image pairs typically produce a Pij with several maxima points at roughly the same value,
while the Pij of two matching tiles produces one maximum significantly stronger than the rest. If this
maximum is at least twice as strong as all others, it is marked as a good match; otherwise, we conclude
that the tiles do not overlap. Displacement vectors with less than 5% overlap are discarded since true tiles
are expected to have as much as 15% overlap. The method works best for tiles with at least 10% overlap.
For further details, we refer the reader to Koshevoy et al. (2006).
The second sub-problem is finding the correct layout of the tiles into the mosaic space. For any pair of
tiles, alternative mappings exist. A direct mapping exists if the pair was determined to overlap. But we
also track cascaded mappings via intermediate tiles. For example, there may exist a mapping S0 : S1 between tiles S0 and S1, and another mapping S1 : S4 between tiles S1 : S4. A mapping S0 : S4 between tiles S0
and S4 can be created via the intermediate tile S1 as an alternative to the direct mapping. The mapping
with the least cost is preferred even when it has greater cascade length. The cost function is the sum of
squared intensity differences in the overlap region, normalized by the overlap area. Using these redundant
mappings between any two tiles presents a great opportunity to select the best mapping possible, especially if an error occurred in the computation of the direct mapping. To build a mosaic, we select an arbitrary
tile as an anchor or first tile. Then the tile with the lowest best mapping cost to the anchor image, computed as described above, is placed into the mosaic. Tiles are successively placed always choosing the best
mapping to tiles already in the mosaic.
Page 50 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
It is possible that meso-scale brightness variations might compromise mosaic alignments. Tiles with problematic intensity variations can be preprocessed using ir-clahe (Contrast Limited Adaptive Histogram
Equalization; CLAHE) before auto-tiling with ir-fft, and/or with ir-blob (a coarse scale blob and edge detector) prior to registering assembled slices with ir-stos-brute (see below). Depending on the user’s preferences, the computed transforms can be applied to either the pre- or unprocessed tiles for visualization
with ir-assemble, ir-stom (slice-to-movie) or the other viewers.
Supplementary Item 6.
ir-translate
Using the approximate positions from metadata, only tiles that are known to overlap are matched using
the Fourier shift method previously described. This reduces the complexity of the method from quadratic
to linear in the number of tiles. Next, a tension vector proportional to the offset between the approximate
position, and the preferred position as found by matching, is defined. This tension vector can be thought
of as a spring between the two tiles pulling the two tiles toward their preferred relative position. Most tiles
will have multiple neighbors and will therefore experience multiple tensions. The sum of the tension vectors experienced by each tile will be a net force pushing each tile towards a more desirable position. Continuing the analogy of multiple springs, the energy of each tension vector can be thought of as the potential energy stored in a spring, thus a global system energy can be calculated by computing the sum of the
squares of all the tension vectors. The algorithm iterates over each tile in the slice multiple times, nudging
all the tiles with each pass. This will result in finding a local minimum of the global system energy.
Supplementary Item 7.
ir-refine-grid
Non-linear warp refinement and is accomplished with ir-refine-grid. During the earlier stages of algorithm development, several continuous polynomial transforms were explored, in particular a bivariate cuPage 51 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
bic radial distortion transform and a bivariate cubic Legendre polynomial transform. These transforms
suffer from a trade-off where the stability of the transform is related inversely to the degree of the polynomial. Our final approach uses a discontinuous transform with local phase correlation. Each tile is sampled onto a coarse uniform triangle mesh. Each vertex in the mesh stores two sets of coordinates: the local
tile coordinates and the mosaic space coordinates. The image is warped by changing the mosaic space
coordinates directly. To map a coordinate from the mosaic space into the tile space, the tile mesh is
searched for the triangle containing the given mosaic space point. Then, the barycentric coordinates of the
intersection point are used to calculate the corresponding tile space point by interpolating the tile space
vertex coordinates. The mapping from tile space into mosaic space is trivial due to the uniform structure
of the triangle mesh in the tile space. One has to find the mesh quad containing the tile space point and
perform a bilinear interpolation between the mosaic space coordinates of the quad vertices. These operations are implemented in an efficient manner using OpenGL (Open Graphics Library).
At each vertex in the mesh a small, corresponding neighborhood is sampled from all of the tile neighbors
in the mosaic. The neighborhood need only be large enough to capture a meaningful amount of image
texture for phase correlation to work: 96x96 pixels. The mesh nodes are spaced at approximately one
third of the neighborhood size. The two neighborhoods are matched using the same Fourier shift method
described for ir-fft. The dense field of displacement vectors produced by this matching is used to correct
the mosaic space coordinates of the vertex. It is possible for tile matching to produce mismatches, so the
calculated vectors at each vertex are median filtered to remove the outliers. The vectors are further denoised with a Gaussian smoothing filter. This post-processing requires 2-3 passes to ensure convergence.
For further details, we refer the reader to Koshevoy et al. (2007).
Supplementary Item 8.
Page 52 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
ir-tweak
ir-tweak was implemented using OpenGL, Qt (the “Cute” Widget Toolkit), ITK (The Insight Segmentation and Registration Toolkit), GLEW (OpenGL Extension Wrangler Library), and Cg (C for Graphics
shading library developed by Nvidia and Microsoft) . Ir-tweak implements fragment shading, reducing
the memory footprint of the image textures and uses a thin-plate spline transform.
Supplementary Item 9.
A Tutorial on Computational Molecular Phenotyping (CMP)
Concept
CMP is based on four technologies:
• quantitative immunoprobes
• ultrathin section arrays
• imaging
• computational N-space classification
CMP transforms complex image sets that must otherwise be qualitatively perused by humans (>102 sections, > 106 cells, 8-36 probes) into N-space data matrices viewed by an algorithmic operator – a statistical pattern classifier (Duda, Hart & Stork, 2000). The classifier parses the matrices and allows a reverse
transform into a theme map optimized for human visualization. Theme maps index the underlying Nspace molecular signatures as well as structural signatures (cell size, position, density, texture, patterning,
etc). A molecular+structural signature is the quantitative realization of a phenotype. It is continuously
extensible and managed by well-known statistical rules. A detailed description of CMP calibration, imagPage 53 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
ing, visualization and clustering is provided in the appendix to Marc et al (1995) Pattern recognition of
amino acid signatures in retinal neurons. J Neurosci 15: 5106-5129. This is an abbreviated overview of
some of that content.
History
The notion of a characteristic molecular signature was developed independently by Thomas Hökfelt and
Dominic Lam in the mid-1980’s (e.g. Lam et al. 1986). The practical concept of characterizing cells by
small molecule signatures was developed by Ottersen and his colleagues (Ottersen and Mathisen, 1984;
Ottersen 1987, 1989; Ottersen et al. 1992). The Marc laboratory began developing optical CMP using
silver-intensified immunogold methods in 1986, some four years after code for analysis of multispectral
satellite imagery was declassified and commercialized (see PCI Geomatica,
http://www.pcigeomatics.com/index.html). The data structures and interpretive objectives of immunocytochemical analyses are congruent to remote sensing structures and goals (Marc et al, 1990; 1995) and are
based on classification theory and Bayesian statistics. Studying classes of structures involves two questions.
• Traditional parametric statistics asks: Are x and y drawn from the same distribution?
• Classification asks a more challenging question: Did x come from distribution A or B or C …?
This challenge is addressed in the publications of Fix & Hodges (1951, 1952), Cover & Hart (1967), Tou
& Gonzales (1974), Duda & Hart (1973). These are the “Hodgkin-Huxley” papers of classification theory.
Page 54 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Rationale
CMP approaches the ideal classification method for heterocellular metazoan tissues. There are two antipodal designs for classifying cells: Univariate and multivariate classification. The univariate design is the
standard immunocytochemical approach. It requires one validated probe per class and hundreds of samples to test them all. Validation is tissue and species specific. For example, the retina clearly has more
than >60 cell classes based on several estimates and tracking them all would require >60 validated probes.
An example of the best univariate work is Haverkamp and Wässle (2000), and they specifically note:
“Only a few markers labeled only one cell type: Most antibodies recognized specific groups of neurons”.
The weaknesses of the univariate approach are:
• most essential probes don’t yet exist or haven’t been validated
• species-independence cannot be assured
• data fusion is usually impossible
• no proof of completeness or correctness is possible
Conversely, multivariate classifiers use basis sets (linearly independent measures in a vector space) of
probes targeting overlapping classes to create an N-space matrix for a single sample. The strengths of
multivariate CMP approach are:
• probes for concurrent use exist
• all small molecule probes are species independent and work in all tissues, in all organisms
• data fusion methods exist and are robust
Page 55 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
• mathematically complete coverage is assured
Fig. CMP 1 graphically compares the classification power of univariate immunocytochemistry with CMP
in the mammalian retina. Even a theoretically optimal conventional preparation on one frozen section
with 4 channels of immunocytochemical data of captures but a tiny segment of the known classes with
qualitative signals and poor coverage. Conversely CMP and CMP + excitation mapping (Marc 1999a,
1999b; Marc and Jones 2002) successively expand class resolution to high levels with 100% cell coverage. Every cell gets tagged with an N-space signature, even if the biological meaning is yet unknown.
The multivariate CMP approach is also extensible as selected macromolecular (Marc et al., 2007, 2008)
and ultrastructural signals (Marc and Liu, 2000; Jones et al., 2003) can be included. CMP packs data to
unprecedented densities. CMP analysis is independent of section thickness (Marc et al, 1995), and ultrathin sectioning allows high sampling densities in every cell. A single 10 µm wide cell yields = 250 ultrathin 40 nm sections. We can compress up to 12-36 channels of quantitative data into twelve such serial
sections, using less than 5% of the cell.
Probe Libraries: Theory & Practice
CMP uses multiple basis probes to increase N-space resolution and coverage. CMP probes can track intracellular concentration differences as small as 40 µM (Marc and Jones, 2000; Fig. CMP 2). This gives
robust classifying power. We think of discrete 2D image metrics as pixels, 3D volume metrics as voxels,
and N-space metrics as “nixels”. The resolution R of a classification space is R = ∏∆sn where n is a molecular channel, ∆S is signal resolution, and the power function is evaluated over the interval n=1 to N,
the total number of channels. With N=8 channels, a minimum ∆S=8 resolved steps per channel (3 bits,
much less than actually achieved), R = 16.8 × 106 nixels of molecular resolution for segmenting retina
into 60 hypervolumes of > 2 × 105 nixels/class. Thus CMP has intrinsic classification power.
Page 56 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Karl Landsteiner (1945) demonstrated regio- and stereospecific binding of small molecules by IgGs and
tissue level discrimination of even enantiomers (e.g. D- and L- aspartate) is now routine (Marc et al.,
1995), as is targeting free amines in general (Storm-Mathisen et al., 1983; Geffard, Segula and HeinrichLock, 1984; Ottersen and Storm-Mathisen, 1985; Steinbusch et al. 1988; Storm-Mathisen and Ottersen,
1990; Marc et al, 1990; Pow, 1993; Pow and Crook, 1993; Marc et al, 1995; Kalloniatis, Marc and Murry,
1996; Marc et al., 1998; Marc and Jones, 2002). Anti-hapten IgGs have successfully been made against
targets as complex as C60 fullerenes, as nonpolar as cholesterol, or as large as vitamin B12 (Izhaky and
Pecht, 1998; Chen et al. 1998; Birn et al., 2002. Most small molecules are potential haptens. A large library of probes targeting small molecules is currently in use in the Marc laboratory (see
http://prometheus.med.utah.edu/~marclab/CMP_probes.html).
Quantitation
IgG binding in resin-based CMP is limited to the surfaces of sections (Marc et al., 1995), making CMP a
2-phase assay with better physical properties for quantitation than traditional immunocytochemistry
(Marc et al., 1990; Marc and Jones, 2002). CMP can be calibrated against standards containing known
amounts of target hapten (e.g. Marc and Jones, 2002). Either fluorescence (intensity) or density methods
may be used. CMP based on silver-intensified immunogold (Kalloniatis and Fletcher, 1993), has advantages of high sensitivity, good dynamic range and archival imaging. Fig. CMP 2 is reproduced from
Marc and Jones (2002) and demonstrates both the range and precision of CMP, calibrated directly against
artificial standards. Marc et al. (1990) also showed direct calibration using immunodot assays. The practical sensitivity of CMP ranges from 50 uM to 10 mM, which covers the range of differential expression in
most cells. By using standardized image capture protocols (Marc and Jones, 2002) CMP imagery is quantitative and allows direct readout of cellular concentrations according to the “Silver” equation: P = (C n⋅
Page 57 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Pmax)/(Cn + σn), where P = pixel value, Pmax is the saturated pixel value, C is concentration, n is the cooperativity (analogous to video gamma) and σ is the concentration at P=0.5 (Fig. CMP 3). However,
transforming images to concentration maps is not necessary for quantitative statistical comparisons, segmentation, or classification.
Ultrathin section arrays
The second key to CMP is production of high-quality section arrays. A detailed CMP protocol is located
at http://prometheus.med.utah.edu/~marclab/protocols_CMP.html. There are three elements to the production of high quality arrays: fixation, resin embedding, and sectioning.
Fixation. The detection regime for small amines is a uses glutaraldehyde (GA) cross-linking (e.g. Marc
and Liu, 2000). This forms a dense matrix with an average pore diameter of 2-5 nm. IgGs (≈ 150 kD)
cannot penetrate the matrix. Thus the precise surfaces produced by resin ultramicrotomy are ideal and
independent of section thickness (Fig. CMP. 4). Immunogold detection using section etching is quantitative and reproducible (Ottersen, 1987; Marc et al. 1990, 1995). The epitope surface ‘above’ the matrix is
likely to be only ≈ 1 nm deep and, in this volume, a 10 mM target thus exposes ≈ 6000 randomly oriented
epitopes/µm2. Our routine detection limit with silver-intensification is ≈ 50 µM, corresponding to 30 true
targets/µm2. These and other calculations suggest that the efficiency of small molecule trapping is very
high, perhaps 85% or higher (see http://prometheus.med.utah.edu/~marclab/CMP_substrates.html)
Resin embedding and sectioning. We have tested many resin options and none approaches the flexibility
and stability of epoxide resins with dodecenylsuccinic anhydride cross-linking and 2,4,6 tris(dimethyaminomethyl)-phenol end-linking. This resin is electron-beam stable and allows ultrathin sectioning, but remains fully removable by anhydrous methoxide etching. Routine CMP is done with 200 nm
Page 58 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
diamond-cut serial sections floated onto 25 µl ultrapure water droplets on ultraclean array spot slides
(HTC® hydrophobic mask slides, Thermo Scientific). Sectioning is done manually and any qualified ultrastructural technician can readily prepare section arrays. Resin embedding also allows multiplexing of
samples (Marc et al., 1990; Marc, 1999) and as many as a dozen tissues can be reformatted into a single
bloc (see http://prometheus.med.utah.edu/~marclab/protocols_plastic.html). Figure CMP 5 shows a typical 12-well tissue array.
Imaging
Because silvered CMP is completely archival, images can be captured under optimal conditions of signalto-noise, with stabilized power supplies and any good 8-bit scientific monochrome camera. We prefer
capturing entire sections at a resolution of 183 nm/pixel (slightly oversampled) with a 1.0 NA planapochromatic oil immersion objective and high-resolution scanning stage. Capture and montaging protocols
are detailed in Marc and Cameron (2003) and Marc and Jones (2002) and at
http://prometheus.med.utah.edu/~marclab/protocols_CMP.html. Captured images are mosaicked and registered into large multispectral datasets for classification.
Classification
Pattern recognition theory was developed in large part for multispectral analysis of planetary imaging data. A set of serial sections probed for small molecules is similar to a series of planetary photographs taken
through different spectral filters: common loci across images will index a list of values representing planetary spectral reflectance or small molecule contents. Viewing the data as color triplets aids in finding the
obvious relations, but is not a statistical assessment. The more powerful is pattern recognition in N-space,
and clustering in particular. Algorithms such K-means and isodata clustering extract N-dimensional
Page 59 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
means and variances for classes by calculating hypersurface decision boundaries (Duda and Hart, 1973;
Tou and Gonzales, 1974). How such algorithms work is explained in great detail in Marc et al. (1995).
Fig. CMP 6 is an example of data from cells with overlapping distributions in univariate space though
separable in 2-space. Visually interpreting data sets of higher dimensionality is difficult or impossible, but
computing classes and decision boundaries is not. Thus a map of theme class memberships can be generated (Marc et al., 1995; Marc and Jones, 2002; Marc and Cameron, 2003) and explored to visualize the
underlying signatures in more detail. Derived classes can be tested for their statistical separabilities by a
probability of error (pe) assessment, typically by calculating the N-space distances of the data clouds (e.g.
transformed divergence or Bhattchyara distance) and modeling pe. This is a powerful measure, because
separable classes are always significant, but not vice versa. Though the bivariate distributions have little
overlap and a low pe, the univariate data overlap extensively with high pe since univariate data cannot
encode covariance. If one were required to name the group from which a given sample arose based on the
strength of one type of signal alone, the error would be unacceptably high. For example, the gray cell indicated at the top of Figure CMP 5 would be misclassified 50% of the time and the integrated error for the
entire range of data would be 0.2, over an order of magnitude higher than the bivariate pe. Such univariate
errors are non- trivial and common in immunocytochemistry.
The results of K-means classifications can be visualized many ways:
• As theme maps (i.e. political maps) by color coding every pixel in the original image according to
theme class (Marc and Jones, 2002; CMPView © JA Anderson; Fig. CMP 7)
• As bivariate probability density distributions (Fig. CMP 6)
Page 60 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
• As matrices of univariate probability density distributions viewed as a small multiple plot (Tufte, 1983;
CMPView © JA Anderson)
• As 2N-plots where each cell is mapped as a set of superimposed bivariate plots (Marc and Jones, 2002;
CellKit © RE Marc; Fig. CMP 8)
• As parallel plots (CMPView © JA Anderson) developed by Inselberg (1990) and on which 2N-plots
were based.
Supplementary Material References
Birn H, Willnow TE, Nielsen R, Norden AG, Bönsch C, Moestrup SK, Nexø E, Christensen EI.
(2002) Megalin is essential for renal proximal tubule reabsorption and accumulation of transcobalaminB12. Am J Physiol Renal Physiol 282: F408-F416.
Chen BX, Wilson SR, Das M, Coughlin DJ, Erlanger BF (1998) Antigenicity of fullerenes: antibodies
specific for fullerenes and their characteristics. Proc Natl Acad Sci 95: 10809-10813.
Cover, T.M. and P.E. Hart, Nearest neighbor pattern classification. IEEE Transactions on Information
Theory, 1967. IT-13: p. 21-27.
Duda RO, Hart PE (1973) Pattern classification and scene analysis. New York: Wiley.
Duda RO, Hart PE, Stork DG (2000) Pattern Classification. 2d ed, Wiley-Interscience.
Fix E, Hodges LJL (1951) Discriminatory analysis: Nonparametric discrimination: Consistency properties. Project 21-49-004, Report No. 4, pp. 261-279. USAF school of Aviation Medicine: Randolph
Field, TX.
Page 61 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Fix E, Hodges LJL (1952) Discriminatory analysis: Nonparametric discrimination: Small sample performance. Project 21-49-004, Report No. 11, pp. 280-322. USAF school of Aviation Medicine: Randolph Field, TX.
Girod B, Kuo D (1989) Direct estimation of displacement histograms. in Proceedings of the Optical Society of America Meeting on Understanding and Machine Vision, pp. 73–76.
Gonzalez RC, Woods RE (2008) "Digital Image Processing," Third Edition, Pearson Prentice Hall.
Geffard M, Segula P, Heinrich-Rock AM (1984) Antisera against catecholamines: specificity studies and
physicochemical data for anti-dopamine and anti-p-tyramine antibodies. Molecular Immunology 21:
515-522.
Haverkamp S, Wässle H (2000) Immunocytochemical analysis of the mouse retina. J Comp Neurol 242:
1-23.
Inselberg, A, Dimsdale B (1990) Parallel coordinates: a tool for visualizing multi-dimensional geometry.
In 1st IEEE Conference on Visualization. 1990. San Francisco CA.
Izhaky D, Pecht I (1998) What else can the immune system recognize? Proc Natl Acad Sci 95: 1150911510.
Kalloniatis M, Fletcher EL. (1993) Immunocytochemical localization of the amino acid neurotransmitters
in the chicken retina. J Comp Neurol 336:174-193.
Kalloniatis M, Marc RE, Murry RF (1996) Amino acid signatures in the primate retina. J Neurosci 16:
6807-6829.
Koshevoy PA, Tasdizen T, Whitaker RT, Jones BW, Marc RE (2006) Assembly of large threedimensional volumes from serial section transmission electron microscopy. Proceedings of 2006
MICCAI Workshop on Microscopic Image Analysis with Applications in Biology, pp. 1-8.
Page 62 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Koshevoy PA, Tasdizen T, Whitaker RT (2007) Automatic assembly of TEM mosaics and mosaic stacks
using phase correlation. SCI Institute Technical Report, No. UUSCI-2007-004, University of
Utah.
K. Landsteiner, The specificity of serological reactions (Harvard University Press, Boston, 1945), pp.
450.
Lam DM, Li HB, Su YY, Watt CB. 1985 The signature hypothesis: co-localizations of neuroactive substances as anatomical probes for circuitry analyses. Vision Res 25: 1353-1364.
Marc RE, Cameron DA (2002) A molecular phenotype atlas of the zebrafish retina. J Neurocytol 30: 593654.
Marc RE, Jones BW (2002) Molecular phenotyping of retinal ganglion cells. J Neurosci 22: 413-427.
Marc RE, Murry RF, Basinger SF (1995) Pattern recognition of amino acid signatures in retinal neurons.
J Neurosci 15: 5106-5129.
Marc RE, Liu WL, Kalloniatis M, Raiguel SF, van Haesendonck E (1990) Patterns of glutamate immunoreactivity in the goldfish retina. J Neurosci 10: 4006-4034.
Marc RE, Murry RF, Fisher SK, Linberg K, Lewis G, Kalloniatis M (1998) Amino acid signatures in the
normal cat retina. Invest Ophth & Vis Science. 39: 1685-1693.
Ottersen OP (1987) Postembedding light- and electron microscopic immunocytochemistry of amino acids: description of a new model system allowing identical conditions for specificity testing and tissue
processing. Exp Brain Res 69: 167-174.
Ottersen OP (1989) Postembedding immunogold labeling of fixed glutamate - an electron-microscopic
analysis of the relationship between gold particle density and antigen concentration. J Chem Neuroanat 2: 57-66.
Page 63 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Ottersen OP, Storm-Mathisen J (1984) Glutamate-containing and GABA-containing neurons in the
mouse and rat-brain, as demonstrated with a new immunocytochemical technique. J Comp Neurol
229: 374-392.
Ottersen OP, Storm-Mathisen J (1985) Different neuronal localization of aspartate-like and glutamate-like
immunoreactivities in the hippocampus of rat, guinea-pig and senegalese baboon (Papio papio), with
a Note on the distribution of gamma-aminobutyrate. Neuroscience, 16: 589-606.
Ottersen OP, Zhang N, Walberg F (1992) Metabolic compartmentation of glutamate and glutamine morphological evidence obtained by quantitative immunocytochemistry in rat cerebellum. Neuroscience 46: 519-534.
Pow DV (1993) Immunocytochemistry of amino-acids in the rodent pituitary using extremely specific,
very high-titer antisera. J Neuroendocrinol 5: 349-356.
Pow DV, Crook DK (1993) Extremely high-titer polyclonal antisera against small neurotransmitter molecules - rapid Production, characterization and use in light-microscopic and electron-microscopic immunocytochemistry. J Neurosci Meth 48: 51-63.
Steinbusch HW, Wouterlood FG, de Vente J, Bol JG, Berkenbosch F (1988) Immunohistochemical localization of monoamines and cyclic nucleotides. Acta Histochem S35: 85-106.
Storm-Mathisen J, Leknes AK, Bore AT, Vaaland JL, Edminson P, Haug FM, Ottersen OP (1983) 1st
visualization of glutamate and GABA in neurons by immunocytochemistry. Nature 301: 517-520.
Storm-Mathisen J, Ottersen OP (1990) Immunocytochemistry of glutamate at the synaptic level. J Histochem Cytochem 38: 1733-1743.
Tou JT, Gonzales RC (1974) Pattern Recognition Principles. New York: Addison-Wesley.
Page 64 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Supplementary Material: CMP captions
Figure CMP 1
Histograms of classification comparing four-channel immunocytochemistry (red column), resin microscopy (green column), CMP (blue column), and CMP + excitation mapping (yellow column) over the
known class range of the mammalian retina (white column). Each horizontal band represents a class or
superclass. The numbers of bands in a column indicate a method’s absolute resolution. CMP expands resin classes into 22 unique superclasses and CMP + excitation mapping nearly doubles that. Only CMP
provides strong classification and 100% coverage. “4 ch icc” uses cone opsin, PKC α, TH (tyrosine hydroxylase) and GC (glutamine synthetase) as markers. NNCs , non-neuronal cells.
Figure CMP 2. Differential GABA content of cells in the ganglion cell layer of the rabbit retina visualized
by quantitative GABA mapping on a 250 nm section. Scaling was determined from artificial standards.
Cellular contents range from 50 μM to 10 mM. Some ganglion cells are literally invisible in the array because they have the same GABA content as the surrounding Müller cells (ellipse). Other cells can be differentiated from the Müller cells by having contents roughly 40 μM higher (box); a strongly immunoreactive starburst amacrine cell is also contained in the box. The image is density scaled. Scale bar, 25 μm.
From Marc and jones, 2002.
Figure CMP 3. Competitive displacement curves for IgG D, IgG E, IgG J, IgG Q, IgG γ run concurrently
and aligned at σ, the half saturation point of the Silver Equation. The curves were pinned by visualizing
IgG E binding to an artificial standard stack.
Page 65 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
Figure CMP 4. Ultrathin resin sections of rabbit retina probed for glutamate and visualized with calibrated
silver metallized immunogold binding. All signal in the images is glutamate detection. There is no ‘background’ and the SNR > 103:1. The left section is 40 nm thick and the right 160 nm. The scatterplot
demonstrates that the signals are identical and correlated over the entire metallization range.
Figure CMP 5. A typical 12-well tissue array with sections positioned in a repeating reverse N pattern,
visualized with silver metallization.
Figure CMP 6. Summary bivariate distributions of idealized glutamate and aspartate signals for two model cell classes 1 and 2. The ellipses are the 2 SD borders of the bivariate data sets and the unimodal profiles are the marginal distributions for each cell class. A datapoint in class 2 possessing an indeterminate
signal in univariate space is indicated by arrows. Revised from Marc et al., 1995.
Figure CMP 7. A formal theme map of the zebrafish retinal marginal zone. Each cell type is coded by
class membership based on isodata clustering and histogram deconvolution. Abbreviations: iGCs, immature ganglion cells; MCPs, Müller cell processes; mGCs, maturing ganglion cells; GABA mACs, maturing GABA amacrine cells; MZ1Cs, marginal zone 1 cells; MZ2aCs, marginal zone 2a cells; MZ2bCs,
marginal zone 2b cells; MZ3Cs, marginal zone 3 cells; MZ4Cs, marginal zone 4 cells, PRs, photoreceptors; ?Cs, unknown cells associated with MZ4; VCs, vascular cells. From Marc and Cameron, 2003.
Figure CMP 8. N-plot for the ADEGJQτγ basis set. This N-plot was derived from the molecular phenotype data of zebrafish class 1 GABA amacrine cells (GABA AC1). Each dot encodes a bivariate mean on
Page 66 of 70
Anderson et al: A Framework for Mapping Neural Circuitry FIGURE LEGENDS
a logarithmic mM scale (i.e. –1 = 0.1 mM, 0 = 1 mM, 1 = 10 mM), and is bounded by a 1 standard deviation ellipse. Four pairs of points represent a projection of 8-space into 2-space, and distinctions among
classes are thus represented by the differing patterns of signals in N-plots (see Marc and Jones, 2001).
The XY pairs are arbitrarily formed but constant throughout this manuscript: AJ (grey), DQ (cyan), Eγ
(gold), Gτ (magenta). In this plot, it can be quickly determined that the mean A, D, G, E and J values are
less than 1 mM in this class, because their coordinates are all below the log 0 mM line. Conversely, the
mean Q, γ and τ values are all >> 1 mM.
Page 67 of 70
Anderson et al: A Framework for Mapping Neural Circuitry TABLES
Tables
Table 1: Software Tools and Sources
Application
Function
Platform(s) Link
SerialEM [1]
TEM Acquisition
W
Univ Colorado, Boulder
SyncroscanRT
LM Acquisition
W
Objective Imaging
ir-fft [2]
autotiling
L, M
Univ Utah SCI Institute
ir-refine-grid [2]
autotiling
L, M
Univ Utah SCI Institute
ir-translate [3]
metadata tiling
L, M
Univ Utah SCI Institute
ir-tweak [2]
registration
L, M, W
Univ Utah SCI Institute
ir-stos-brute [2]
autoregistration
L, M
Univ Utah SCI Institute
ir-stos-refine [2]
autoregistration
L, M
Univ Utah SCI Institute
ir-blob [2]
feature enhancement
L,M
Univ Utah SCI Institute
ir-clahe [2]
image equalization
L, M
Univ Utah SCI Institute
CMPView [4]
classification
L, M, W
Univ Utah SCI Institute
MosaicBuilder [3]
2D viewer & annotation
M
Univ Utah SCI Institute
Viking [4]
Multi-slice pager & annotation W
Univ Utah SCI Institute
Key: W Wintel, L Linux, M Macintosh OS X
Primary Authors: [1] D. Mastronarde, [2] P. Koshevoy, [3] J. Spaltenstein, [4] J. Anderson
Links: Colorado: http://bio3d.colorado.edu/SerialEM/, Objective Imaging: http://www.objectiveimaging.com, Utah
http://code.sci.utah.edu/Microscopy
Table 2: Platform Comparisons for a 250 μm diameter x 30 μm high volume
Page 68 of 70
Anderson et al: A Framework for Mapping Neural Circuitry TABLES
Operation
ssTEM
Sectioning
Manual
Sectioning speed
< 1 wk
Risk of section loss
High
Thickness range
40-90 nm
Staining
Manual, fast
Staining options**
Numerous
Immunochemistry
Compatible
Single tile size
16 megapixels
Single mosaic size
15-30 Gb
Image capture time
3130 tiles/day
Mosaic capture time
3/day
Resolution
≈ 2 nm/pixel
Use constraints***
None
Scalability****
High
Mosaic build times……………………………. †
MRC to TIF convert
55 m
Build image pyramids 43 m
ir-translate time
16 m
ir-refine time
64 m
ir-assemble time
12 m
mosaic builds/day
8
Mosaic register times…………………………..
ir-stos-brute
43 s
ir-stos-grid
8m
pairs aligned/day
>100
Block-face SEM*
Automatic
NA
Low
20-70 nm
Manual en bloc
Limited
Not yet compatible
16 megapixels*
16 megapixels*
NA
0.5/day*
≈ 10 nm/pixel
Extreme
Low
NA
NA
NA
NA
NA
0.5
NA
NA
0.5
* Data from GATAN Corporation; otherwise from Denk and Horstman (2004)
** All standard ssTEM strategies, including CMP and post-embedding immunocytochemistry are available
*** ssTEM may be interrupted at any time to image other samples. Block-face imaging cannot be interrupted.
**** ssTEM capture and processing speed can be enhanced further by parallelization
† Assumes that no image processing is required for bloc-face imaging
NA not applicable
Table 3: Parameters & Timeline for CN Mapping a Canonical Field in the Rabbit Retina
Parameter / Operation
Value
Page 69 of 70
Anderson et al: A Framework for Mapping Neural Circuitry TABLES
Ideal field diameter
Optimal section thickness
Stack thickess
Stack sections
TEM magnification
TEM tile size
Tiles / mosaic
mosaic acquisition time
volume acquisition time
volume build time
high BC, AC, GC, dAC sample
low BC, AC, GC, dAC sample
estimated high: low synapses
high class 6 / class 9 numbers
low class 6 / class 9 numbers
250 μm
70 nm
0.035 mm (GCL - IPL - ACL)
400
5000x, 2.18 nm/pixel
8.7 x 8.7 μm
900-1000 (15% overlap)
7h
133 d
33 d
≈ 1600 BCs, 800 ACs, 200 GCs, 100 dACs
≈ 400 BCs, 200 ACs, 50 GCs, 25 dACs
72,000 : 18,000
30 : 3
7:1
Key: GCL ganglion cell layer, IPL inner plexiform layer, ACL amacrine cell layer, BC bipolar cell, AC amacrine
cell, GC ganglion cell, dAC displaced amacrine cell. Class 6 GCs are the most abundant class and Class 9 GCs the
rarest.
Page 70 of 70