>> Eric Horvitz: It's an honor to have Paxon Frady here with us today.
Paxon did his undergrad work at Cal Tech working in Christof Koch's lab where
he worked on the kinds of things that Christof Koch is known to be interested
in, visual tension and deep neural networks, and then went off to join UCSD
for his Ph.D. work with William Kristan, one of our long-term colleagues here
we've been talking to for years about the kinds of things that Paxon will
talk about today as being possible. The phrase that Bill and I came up with
years ago was the whole idea of pursuing a computational microscope by
analogy and by transition from light microscopy to electromicroscopy,
electromicroscopy to computational microscopy where you have potentially
noisy data coming in in high-dimensional spaces and you want to sort of
process it and visualize this with a new kind of sets of views that collapse,
clarify, compare, contrast, cluster, and that these kinds of new microscopes
would be really critical one day for looking at neurobiological data
especially multi-neuron data, among other kinds of complex biological data.
And he's been working in Bill's lab for several years now. Just finished his
Ph.D. work there, computational neuroscience. I think Paxon was probably our
first and only since then neurobiology intern. I guess second because we had
one right before that actually who went on to finish his Ph.D. work. And so
it was fun having a summer of neurobiology with him here building tools and
tool kits that might one day be the basis for computational microscopy and I
think that looking at his work and his recent efforts what happened over four
years I'm proud to think that some of his internship work was actually
formative. We'll be hearing about that today. Paxon.
>> Paxon Frady: Thanks, Eric. Hi everybody. I'm Paxon. I'm in Bill
Kristan's lab at UCC San Diego. Like Eric said, I'm going to be telling but
this concept we call computational microscopy. And it's really about using
machine learning and developing tools in an interactive way so that we can
really try to understand complicated systems. The brain is just the key
example of a complicated system. We really need a lot of progress in this
technology to move forward in understanding the brain. So I just want to
start with this simple question, why is the brain so hard to understand. Why
have we spent the last hundred plus years trying to figure this thing out and
yet we still don't have a sound theory about how the brain works at all. And
I'm going to just blame this thing we call reductionism and so we know the
brain is very collection. It exists over a large number of scales. We think
all the way down to the molecular nanometer size scale or synapsis and how
these are kind of the foundational units of the brain up into neurons and
small neural circuits up into these complex hierarchies of different brain
areas into the entire brain itself. So bridging across these scales is a
huge technical challenge. And using reductionism is also a philosophical
problem. So reductionism we're taking this complex system and we're breaking
it apart. And this has taken us a long way to understanding how the brain
works. We now know that there are
Synapses, there are neurons and these are all the pieces. We've learned,
we've used reductionism to learn the pieces. But with the brain and with
other complex systems you have to know more than the pieces. It's really
also about how the pieces are put together and how they interact so these
types of questions are outside of the scope of reductionism. So that's why
we need to develop these new technologies and experimental ideas to get past
this and understand how complexity emerges from organization. So this has
kind of been the foundation of the brain initiative. And so this paper which
is called the brain activity map project came out a few years ago just before
the brain initiative kind of pointing out this huge problem and the very
first quote that it starts off with is just talking about this phenomenon of
emergence and how complexity and new properties can appear when lots of
pieces are put together. And then along these types of problems, we have
this book, the forth paradigm that came out when I first started here. This
is centered around using all this data, this new revolution in big data and
how to harness that. Part of the problem in big data is the ability to
understand these complicated systems. When we were here at Microsoft, Eric
wrote in this book this idea of a computational microscope and how to use
these machine learning tools to make sense of these big data problems. The
brain initiative is also another push in this direction. In neuroscience,
we're kind of going through this big data resolution. We have all these
types of new techniques that enable us to record from nervous systems at
unprecedented scale and resolution. I'm going to tell you about voltage
sensitive dye imaging but there's other imaging modalities like calcium
imaging and multiunit electrode arrays. There's even new technology like
light sheet microscopy that's giving us even more neurons. In the classical
regime, we're doing electrophysiology. We can record from a couple of
neurons. Maybe if you're bold you can record from a half a dozen neurons.
And now, at the turn of the 21st Century, we've kind of entered into the next
scale. So we're now at the dozens to hundreds of neurons scale with imaging.
And then in the future, we're going to have to keep pushing. And then to get
to the scale of the brain, the brain has a hundred billion neurons so we're
still a long way away. But just like broaching this first scale is a huge
issue. And it needs a lot of work. So I think over the last few decades
we've started to realize something fundamentally important about how brains
work and how neurons work and the insight is that in high-dimensional space,
you can do a lot of powerful things in regards to computation. We have these
chaotic neural networks now where if you just take a bunch of neurons and you
randomly wire them together, you can use their chaotic Dynamics to learn
arbitrary output patterns in really the power of these simple things is just
that these neural networks are so high-dimensional that you can find a
projection of them that gives you whatever pattern you want. So this is kind
of illustrated in this little GIF where we have information of these words,
words are these two-dimensional projections of information but if the they
exist in a three dimensional space, we can look at them from different
perspectives and the information is entirely different. So the word no and
the word yes, it's there always in this three dimensional space but the
information just depends on which way you look at it. So as we go up into
higher and higher dimensions to really understand these types of systems, we
have to look at them from all sorts of different angles and that's what the
computational microscope is supposed to do. So how do we explore these
high-dimensional systems? So we need lots of data. I'm going to tell but
the voltage sensitive dye recordings been doing that give us large scale
recording of a nervous system. We want to have multiple behavioral outputs
so we want to see kind of a diverse array of complexity. And then we want to
be able to consolidate it cross multiple experiments. So then we need the
algorithms also to make sense of this data and two of the key things are
going to be visualization so that we can see what we're looking at and then
what I call synthesis where we take new things and we put them together with
old things and finds the canonical relationships. We also want it to be
scaleable because we're just now broaching the beginning of what needs to be
a long increase in scale and as we keep going higher and higher in scale,
we're going to need machine learning and algorithms more and more. And the
other really important part of it is we need realtime and guided experiments.
We need to be able to have these tools, take in some data, give us some
visualizations and then use the tool to tell us what to do next or what's the
target or what questions should we try to figure out with our experiments in
realtime. I work on the leech. It's kind of a weird organism and before I
started grad school, I didn't even know that was a thing people studied. But
it's actually a pretty amazing beautiful organism. It has kind of its unique
advantages over all the other model organisms out there in neuroscience. So
this picture is just from a course I teach called neuro systems and behavior.
And in this course, the students, they get to run through every organism that
kind of the main organisms they use in neuroscience. We have mice and
electric fish, zebra fish, fruit flies, the crab STG and leeches and I think
there's a sea elegance somewhere. Maybe that's a sea elegance. I think that
was too old school. For the leech, I'll get a little bit more into its
nervous system in just a second but essentially it has these kinds of
advantages. It's extremely accessible. We can cut the brain out and get
access to the certification system. The nervous system is relatively simple.
It's only the hundreds of neurons scale. We can get access to it while it's
still performing behavior so the leech can an array of pretty interesting
behavior while we can monitor the nervous system. It has defined cells. So
cross neuron, across brains, across leech brains, it actually has the same
cells individually cell by cell. This is pretty unique for almost compared
to any other organism. Another important thing is we can do
electrophysiology. So we can actually stick electrodes into cells and record
voltage whereas like in sea elegance, there's another simple organism you
actually can't do electrophysiology. They're just now getting
electrophysiology and [indiscernible]. [Indiscernible] neurons are so tiny
that if you try to stick an electrode in them, they essentially explode. But
if you're really talented, you can do it. We also have these new voltage
sensitive dyes which gives us a very unique perspective. In the imaging
world, almost everyone images with calcium and this is kind of the second
messenger of voltage and everyone really wants voltage but in the leech, we
now have this new technology that allows us to image voltage directly. So
this is what a leech looks like. It's just a segmented worm. Anterior,
posterior end. If you stimulate it in different behaviors, you can get -- if
you stimulate it in different places, you can get different behaviors so if
you kind of shock it or poke it and the anterior end it will perform this
behavior called shortening where it kind of withdraws. If you shock it in
the back, it will try to swim away. So this is just like the sign wave
muscle oscillation it uses to propel itself through the water. And then if
you kind of apply a light stimulus somewhere to the middle of the body, it
does this aversive reflex called local bending. So the way we've studied
these behaviors over the years is what you do is first you take a full leech
and then you just watch the behavior and you kind of maybe attach beads to
the leech and look at the beads and get the kinematics of its movements.
Then you can do these cool things where this is what my advisor got kind of
famous on where you can actually cut out part of the animal to expose the
nervous system and then you can do recordings from the nervous system while
front and back half of the animal are still fully in tact. So you can see
this leech like get shocked and then the front and back halves of trying to
swim away while the middle of it is sitting there spinning out actual
potentials. So this is kind of how we've been able to relate the behaviors
to the motor neuron outputs. And for more complicated things like imaging,
you can't have the animal flopping around or everything is going to be ruined
so you can essentially take out the entire nervous system, take away all the
muscles and the animal still performs these behaviors. So we can still shock
the nervous system and get swimming and at the same time image one of the
ganglion with voltage dyes. So the leech's nervous system is made out of a
head brain and a tail brain and then this repeating structure we call a
ganglion. There's 21 of them in the mid body. So all in all, there's about
10,000 neurons in the whole nervous system. But what's really cool is that
each of these ganglia appears to be the same structure of 400 neurons just
repeated. And even the head and tail brain appear to be ganglia just fused
together over the course of evolution. And so we think that the leech's
brain is made up of this one core structure made up of only 400 neurons. And
so we're trying to just map this one core structure. So to do this, I'm
going to image from this particular ganglion, ganglion ten. And then I'm
going to be recording throughout the animal different extra cellular nerves
that go -- these nerves go up to the muscle. So this is behavioral readout
so I can record what the motor neurons are doing as well as input. So I can
stimulate these nerves to kind of stimulate the animal in different places.
Actually the different behaviors. So then this is what just a single
ganglion looks like. So this is the ventral face of one of these ganglion.
There's also a dorsal face. And then so there's cell bodying on the ventral
face cell bodies on the dorsal face, and in between the two layers of cell
body is where the leech nerve system makes all its synaptic connections. So
in this picture, all these colored cells are the cells we've known about that
over the last 50 or 60 years of leech nerve physiology, we've uncovered these
colored cells mostly the sensory and some motor neurons. But then the vast
majority of the rest of the cells are white because we don't really know
anything about them. And actually, the white cells are highly under
represented in this diagram. So we only know about a third of the cells so
far. So to activate the different behaviors, we can just shock the nervous
system so get shortening you can shock either one of these anterior nerves to
get the local bend response actually and target this sensory P cell and
activate the sensory P cell and we know this activates the local bend reflex
and then to get swimming we stimulate these nerves back here posteriorly. So
the whole nervous system will generate all these behaviors without any
muscles attached. So we call these fictive behaviors. There's only about a
third of the neurons that have been identified. So some of them, you can
identify just by looking at them. So these two R cells, they're by far the
biggest cell in the ganglion. They're always sitting right here. So you can
just see by eye, you can know that they're the [indiscernible] cells. But
then I kind of grayed out the sensory neurons over here so there's this line
of sensory neurons but you can't necessarily distinguish them by eye. So you
can guess that like these pretty large-ish shells over here, some of them are
sensory neurons, like this might be an N or might be a P, or a T, but you
can't tell just by I. Unfortunately what you can do is stick an electrode in
and record their action potentials. And so a lot of cells have these unique
action potential shapes so like the T cells are kind of bursty. The N cells
have these really long after hyperpolarizations. They have unique defining
characteristics. And that allows people to study them experimentally because
you can reliably identify them across animals. There's a handful of other
cells that have kind of these unique S potentials like the S cell. But the
vast majority of the other cells, there's nothing really that characteristic
about them. You see these kind of little dinky action potentials but there's
nothing really obvious. And the techniques we used to identify the rest of
the neurons rely on observing what they're doing during these different
behaviors. Or some other kind of complex way of distinguishing them from
other neurons. But this is hard to kind of do at scale and we can sues
voltage sensitive dye imaging to actually kind of solve this problem and so
we can record from almost all of these cells all at once while they're each
doing individual behaviors and use that to distinguish the cells. So we use
a voltage sensitive dye to record from these neurons. I'm not going to like
do the mechanism of it but essentially, the dye gives us these amazing
recordings so here, these are optical traces so we actually can see
individual action potentials in these optical traces. We can see these large
bursts. These are little 5-millivolt spikes in this particular cell and then
we can even see oscillation. So down here in black is an intracellular
voltage recording and then in green is an optical recording and so even the
cell that's oscillating about 3 or 5 millivolts during swimming even this
little oscillation you can pick up on the voltage dye or like little
subthreshold synaptic potentials. So we have this new voltage chart a gives
us an unprecedented resolution.
>>:
[Indiscernible] that's not actually potential [indiscernible].
>> Paxon Frady:
>>:
Yeah.
So a lot of the cells --
[Indiscernible] basically.
>> Paxon Frady: Right. Unlike most neurons, the leech neurons are all
Monday know pore. So the soma is like way out here and they send the little
process down into the neuro pill. And then they make all their synaptic
connections and generate the axons out here. I mean generate axon potentials
out here. So what you record at the soma is a very filtered axon potential.
So a lot of times you can't even see them and you see just like the slow
membrane oscillation.
>>:
[Indiscernible].
>> Paxon Frady: Right. The way I kind of envision this, the kind of idea of
the computational microscope is to go from here where we have just an
experiment we collect some raw imaging data, we transform that into cell by
cell data, and then we're going transform that into like characteristic
features and then we're going consolidate this into some kind of canonical
framework or like the homologs of the same cells. And I kinds of like to
think of this as like a deep neural network kind of where we have like these
are just different layers and so this is like 10,000 dimensional feature
space this is like a hundred dimensional feature space and then this is like
ten dimensional feature space and then we do some clustering here to come up
with some canonical formation. And this kind of process is what I call the
imaging computational microscope. The first step is to taker a raw movie.
So just like pixels and an image and a bunch of frames and turn that into
individual cellular components. Here's an old screen shot of what in mal app
GUI looks like. So essentially like here's just some raw data over here I've
selected this particular ROI. So up here is exactly what the ROI is over
time. So like the average within the ROI over all frames. So you can see
this cell spiking. And down here is actually the automatically extracted
component that corresponds to this ROI. So all these ROIs are actually
generated automatically. So I'm going to tell you how that works now. So
the first thing do you is take your image ask we're going to try to reduce
the dimensionality down and we're just going to use PCA. The other thing
that I've already done is I've taken all of the data. All these datas I have
multiple trials so I have multiple trials of shortening, swimming. Each
trial is a ten second long movie. So what I've done is I've used an image
registration algorithm to actually align all the trials up and pretend like
everything is just like a single time series. So I just had this one big
movie. So then you do PCA on the individual pixels of this big movie and
then you get two things out. So you get a map and you get sources. So here
since I have several trial in this one movie, the map is going to be the same
for all the trials, but then these traces can be just broken back up into the
individual trials. So for a single animal then, you'll do PCA and then each
component will have a map and then it will have multiple traces and then the
number of traces are just the same number of trials. So we're mainly going
to use PCA to reduce conventionality. PCA tell us a lot about what's in the
data. Usually if you just look at these maps, what you see is these mixtures
of cells and that's just because PCA is just looking at volume and mixtures
of cells, louder than individual cells. But there's this algorithm called
ICA which looks for independence. And this will pull out signals that we're
interested in and so most of the time what you'll see something like this
where you have this kind of localized spatial region and then you have these
traces corresponding to the activity at that region so this is like pulling
out an individual cell. So here's just another cell and so here this is
actually two different trials so one trial is swimming and this cell
oscillates and this trial is shortening and it just does this early part.
But ICA, these algorithms are just completely naïve to what you're telling it
and they don't know the difference between a cell and an artifact but what's
really nice about ICA is it actually will extract and separate cellular
components and artifactual components and then in the GUI you can go in,
manipulate the parameters, all these kind of magic numbers like the
dimensionality of your PCA and toy with it until you have something that you
like and then you also can then manually kind of remove these components that
you don't want. So then after you've removed the artifacts, there's a
segmentation stage essentially just applying a threshold to these ICA maps to
kind of get a binary mask and then occasionally the ICA will actually cluster
cells together into a single component. So in this case, there are these two
bilateral cells and they essentially are firing spike for spike because
they're strongly electrically coupled to each other and so ICA just thinks
they're the same cell because their activity is the same. By using spatial
segmentation just to break them apart again and then that generates a bunch
of ROIs. Finally, sometimes it does the opposite thing of this where it
breaks apart individual cells. And again, you can just use the GUI to put
them back together. So that's like how we get that data out. And next we're
going to try to describe the individual cell data with a handful of features.
But first let me just show you kind of what the data looks like. So here is
shortening. So this is kind of false colored. Effectively, you hear the
stimulus come on and then you'll see a bunch of cells turn green and red.
Green just means it's depolarizing and red means that are hyperpolarizing.
If you look really closely, right after the stimulus, you see this kind of
flash of green and this slower flash of red. This is realtime. A lot of
stuff is happening, right?
>>:
[Indiscernible].
>>: [Indiscernible] after the actual signal's input to the system.
you're specifically looking at change.
>> Paxon Frady:
>>:
Right.
Yes.
So
So all of the optical --
[Indiscernible] color all the time, right?
>> Paxon Frady: Yeah. But the optical recordings are always relative.
you can't get the absolute voltage values.
>>:
You're conditioning on this background, stabilize --
>> Paxon Frady:
>>:
So
And then it changes.
It changes.
>> Paxon Frady: Right. So for shortening, you see a bunch of different
types of response. Some cells go up, some cells kind of go down. Some cells
go up and then down. Some cells have this late phase. Some cells very
early. But really, all these responses can be pretty well characterized by
these two factors. You can think about these as the principal components.
So essentially there's factor one which is this slower component which
corresponds to cells kind of getting depolarized or hyperpolarize in
conjunction with the actual motor output and then there's this factor two
which is this fast component which is cells mainly getting depolarized due to
the stimulus. So then you fit these factors on to all of these traces and
you can get coefficients and then we can plot all of the individual cells
into these two dimensions. And then I can give the cells colors just based
on where these coefficients are and then use the ICA maps color the maps and
create this activity map. And so with the activity map we can kinds of get a
visual sense of Wells Fargo what the cells are doing because now we can plot
four dimensions here, two per space and color representing these two
shortening factors. So now we're kind of characterizing the cells with this
slow dimensional representation. So here's swimming. So swimming is this
repetitive motor burst so that leeches doing this sign wave oscillation into
the water. So what you can see is a bunch of cells that oscillate with this
motor burst so this is like the easiest most obvious one. All the circles
oscillate so just going red to green red to green red to green and then if
you look at a different circle, if you look at this yellow circle you'll see
it oscillating too but it's actually at a different phase. So to
characterize swimming we're going to do what we call a coherence analysis so
here's this motor, this repetitive motor neuron output burst and then here a
bunch of different cells and the voltage is oscillating with this motor
neuron burst and so we do the correlation and foray space which tells us a
correlation magnitude and phase so this is just giving us now another two
dimensions to characterize these oscillations and so you can see a bunch of
different cells are oscillating in different phases and then give these
phases a color and again you can make an activity map and get a visualization
of what the cells are doing during swimming. And then for local bending
we're going to do the same analysis so the local bending experiment is
actually just activating this century P cell at a repetitive input so this is
just like a half a hertz stimulus input and then I'm measuring the coherence
between this stimulus and the rest of the voltage so effectively it pulls out
cells in this pink phase which are getting depolarized by this stimulus and
cells in this scion phase which are getting hyperpolarized and again, we can
make an activity map and get visualization. So the important thing to make
sure that's true is to make sure that like the ganglion across trials, so
individual animal performing shortening, recognize that the neurons do
something consistent or at least some of them are consistent, right? So it
wouldn't be very useful if it's neurons were doing different things on every
trial shortening because you couldn't really use that to identify them. So
here I'm just going to show you like some visualizations to get a sense of
the variability across animals and across trials. So each of these four
boxes is from a different animal and then for each animal there's two trials
of shortening being shown. Okay. So these are just two separate trials.
And you can see like the activity maps so you can see across individual
trials they're really similar. And then over here are just the same
coefficients that I described to you just a second ago. So from one trial, I
fitted those factors and to every trace and that gives me one end of each of
these line segments. And then for another trial, I fit the factors again and
other end so each line segment is the activity of a single cell across two
trials. So the start and end indicate what that single cell is doing across
these two different trials. And so then the size and the kind of isolation
of the line segment is an indicator of how consistent that cell was across
trials and how different it was from other cells so small isolated line
segments mean that we have a lot of identifying information. And then
visually you can kind of look across animals and get an appreciation like the
ganglion during shortening is effectively doing something pretty similar.
And so that's good because we need across animals for the neurons to be doing
similar things during these behaviors.
>>: How would it change the picture if you had same animal performing
different actions, swimming versus some other thing? Would you see a very
different picture?
>> Paxon Frady:
So you mean like if you applied this --
>>: This is kind of qualitative, right? It's not quantitative. And I'm
trying to -- so you make the claim that this is reproducible, so if we would
have had one image in which it was swimming and the other image in which it
was performing some other action, we expect to see that we'll have long
lines, right?
>> Paxon Frady:
>>:
Right.
Very long line, stuff like that.
Do you have something like that?
>> Paxon Frady: I've never even really thought of trying do that. So here's
swimming. Comparing it across? I mean, what it would look like is that in
one trial, you would have something -- if you did the swim analysis like a
swim, it looks like this, right? You have a bunch of neurons that are
significantly coherent. If you try to do this analysis on a shortening like
nothing is oscillating, everything is going to just be clumped in the middle,
like this, so all the lines will be lines out here pointing in to the middle
like this. Radiate like that.
>>: Let me ask a question. I'm curious if you showed this to Bill, your
committee and your advisors or other neurobiologists, be interesting to know
what kind of insights just a question that they held for a long time might
come to their mind for like helping the traditional people in this field who
have standing questions. First step sort of visualizing that it's beautiful,
say lots of things about these animals, for example, but I wonder if we
assume the question Bill's about circuits for example and circuitry could be
answered directly by these visualizations.
>> Paxon Frady:
>>:
Yeah.
Obviously this is towards that, right?
>> Paxon Frady:
This is a step towards that.
>>: Visualizations, one reaction is they're beautiful and I see lots of
things going on and I can learn many things including maybe the personality
of a cell and the match of a cell among animals. But I'm just curious if any
other questions came up over time that people say, wow, I just had this one
functionality here where I'm seeing something about these two cells that
makes me think they're interesting way beyond ninety degrees.
>> Paxon Frady: Yeah. So one of the things that stands out long these
lines, maybe a little different than what you're saying, but does the same
cell across behaviors do the same correlated things. So if you're in this
phase of swimming, does that imply you're over here during shortening or are
these things independent? Right? And that goes a long way in talking about
how neurons encode information about what they're doing and what they're
sensing and I think a big thing in the field now is that you have these
neurons that are multifunctional that really are kind of representing these
joint features and for whatever -- I mean, there's computational theoretical
reasons like every neuron trying to be as independent as possible something
to go for. So then observing the same neurons doing a bunch of behaviors,
what you see is like there's only a handful of cases where like they're
really correlated across behaviors and actually I think pulling out those
correlations tells you some of the structure that you're talking about. And
then I think when I get to the preparatory network too, that will also be
like some illumination.
>>: So when I look at these images, there are differences between animals.
So in each cell, the left and right image are more similar [indiscernible]
same animal.
>> Paxon Frady:
Yes.
These two are the same animal.
>>: Do you know or speculate about the differences between animals? Does it
stem from structure differences between animals are just random formation
differences or if these are more like the way they initiate the swimming
patterns are different between animal? Do you know where the variation comes
from?
>> Paxon Frady: Yeah. So there's certainly variation. But really, it's
remarkably consistent and so it's consistency can be expressed in that set of
cells that we've already identified and so if you open up any leech ganglion,
you can immediately find a bunch of cells, P cells, a bunch of motor neurons.
And so for about a third of the cells that we've identified so far, they're
always there and they're always in the same function. So there's definitely
slight variability in exactly how the ganglion puts together and that's
actually why I actually didn't mention this but like the reason you can't
just like solely rely on where the cells are is because there is like this
anatomical variability. Sometimes the cells get pushed around. And even
sometimes by doing this dissection I stab it or something. I have to cut off
this little tiny capsule and so the cells can move around and the capsule is
actually kind of compressing holding cells and so when I cut it off, they
kind of like expand out so there's slight anatomical variability. And so
that's what makes it hard to identify the cells too. But as far as there
being functional variability or there being learning or something,
environment dependent, it's not entirely clear. Everything we studied so far
points no, but we're kinds of biased because we have mainly only studied the
sensory and motor neuron and perhaps there's this huge array of interneurons
and that kind of space is flexible. That's not entirely clear but from this
where it seems like all these cells seem to be very consistent as well, so it
seems like the animal has this pretty well defined functional homology. So
swimming, so again, it's the same as before. There's two trials of swimming.
And then each cell has a line segment indicating each of these two trials.
And so as you can see, swimming is remarkable. There's a bunch of line
segments. They're really well spread out around the outside edges. They're
very short. So swimming is very promising feature for identifying cells.
And then here's the same thing with local bending. And so mainly what you
see across local bending is essentially this one access so there's these pink
cells which are getting depolarized and these kind of scion or green cells
which are getting hyperpolarized. But again, at least at these extremes,
there seem to be a handful of lines or so that are pretty well isolated. So
then the final step is taking all these animals and putting them into a
single canonical representation. And so here's kind of the gist of the
strategy. So if we just zoom in on these five cells, so based on just their
position and their size, you couldn't really distinguish them from each other
so they're too close to the other. There's nothing really too characteristic
about them. But then if you do shortening and you get these shortening
features, then they start so get separated so now you can tell like one from
five and five from two and three. But you can't really distinguish between
two and three. So they're still -- they're doing the same thing and they're
in the same place so you can't really distinguish them. And then if you look
at swimming, get the same gist. So seeing swimming helps you kind of
distinguish some of them but maybe not all of them from each other. So now
we can distinguish between two and four and three and five because they're in
the same place, doing the same thing but by like combining all these
features, we're creating this higher dimensional feature space and then this
allows separation of all the cells individually. So this is our strategy so
we're going to take all -- we're going to make a ten-dimensional feature
space so we're going to have two for the position of the cells, two just
indicating the size of the cells, and then two more for each of these three
behaviors. So that's going to be our canonical feature space. So then with
this, we kind of developed these ideas a long time ago.
>>:
Back in the 60s.
Rebels.
[Laughter]
>> Paxon Frady: Yeah. So again, there's another interface that kind of lets
you explore, compare all these feature maps and then explore this feature
space and then we have a couple of machine learning algorithms that
manipulate the feature space to kind of accent certain things. So there's an
iterative kind of strategy. So essentially, the idea is like we're comparing
these two animals. This is animal H, this is animal C and then here just
like the activity maps so if you want to reference that. So then animal C
I'm going to select three cells manually and just indicate it by these three
ROIs and then the algorithm is going to use like the Hungarian algorithm for
instance in this particular case to guess which cells are homologous in this
other animal. So the computer has told me that it thinks this cell is the
green cell, this cell is a blue cell, corresponding blue cell, and this red
cell is a corresponding red cell and then it also shows me in this color heat
map the kind of relationships so the brighter green indicates that these
other cells are more like this so this is like the distance in this
ten-dimensional feature space that all these cells are away from this
particular green cell. And so we can essentially visualize this
high-dimensional space by picking a cell and then you can see the cells that
are similar to it and that gives you an idea of like what are the possibles.
So then we can then tell the computer that oh, I'm very confident that these
cells are matches and I'm very confident that these cells are matches. And
based on these matches, we develop some learning algorithms that warp the
future space to kind of accent the important features. So just as this is
the very simplest version of the algorithm, but effectively what you can see
is so here's just like the normal regular feature space for these two
animals, so this plot over here is indicating the position of the cells.
This is indicating the shortening factors. This is indicating the local bend
coherence, and this is indicating the swim coherence. So this is just what
it looks like when you don't do anything. You just plug in the features so
then after warping so I've assigned -- so there's really 8 different animals
in this data set and then I've gone through and I've assigned matches across
all these different 8 animals and then in this iterative procedure, once I
assigned matches, this algorithm can show me this warped distance space and
that emphasizes the features that are important for the matches. So it kind
of -- and it ends up showing you kinds of what our visual intuitions that led
us to believe earlier. So position gets kind of stretched out. Swimming
gets a little stretched out. And so those feature dimensions are emphasized.
Whereas shortening kinds of gets squashed because it's not quite as
distinguishing. And then local bending gets squished long this one access.
And as we saw, it was essentially this one dimension of local bending that
was informative. So then this gives us ways to use machine learning to kind
of warp the future space and pull out the things that are more interesting
and relevant. And this is how we kind of explore this high-dimensional
space. Here's some gifts. So this is just to like kind of show you give you
some visual sense of what's happening so this is like the principal
components of the ten dimension feature space and then every single cell that
I've matched across 8 animals or shown, okay, and then so each cell that's in
a single cluster is colored the same and connected by a little convex hall.
The point of this is really just so you can see that after doing this warping
algorithm that the clusters come out much more nicely and that you can start
to tease apart the cluster cells a little bit better. Really, there's a lot
of corruption in feature space, like the position of the ganglion like some
ganglions are oriented like this a little bit relative to others so just
kinds of compensating for those minor changes can take you a long way in
cleaning up this high-dimensional space. And then also from this, you can
really get a sense of like which cells you're going to get confused so the
nut and the AE cells are kind of like in the same cluster but it's hard to
understand how to separate them because I'm not just solely relying on this
to do the clustering. I'm also going in and visually assigning these matches
so this is like this algorithm and my own refinement. So this gives us all
these cells and now I'm going to describe the canonical set of cells. This
is an entry I'm going to show you for a single cell so this particular cell
is called 151, just numbered based on this standard number system that the
leech community has developed. And then again we have the position
shortening local bend and swimming features plotted here.
>>:
[Indiscernible].
>> Paxon Frady: Yeah. That's exactly what I was about to say. So these
eight circles correspond to the eight animals that are in this experimental
data set. And then the fill-in just indicates which experiments this
particular set was seen. So cross experiments, we're not going to
necessarily see all the cells. We're going to see some subset of the cells
just because there's anatomical difference sometimes it's just due to the dye
not giving enough signal. So this just tells us which experiments this cell
has founds and then these circles tell you where the cells were. This tells
are what it does during shortening. So this cell has a big fast component
and a pretty large depolarization with a motor neuron and this cell almost
always gets excited by P cell stimulus. Got inhibited one time. And then it
typically oscillates in this red phase during swimming. So that's kinds of
how you read this table. So just another cell. So this cell down here kind
of posterior lateral. So this cell gets hyperpolarized during shortening.
It has both kind of responses during local bending and then it oscillates in
this kind of green scion face. So here's the whole table of cells that we
were able to identify across animals. So almost all of them are swim
oscillators. The only ones that aren't are indicated by these asterisks.
And then I have kind of outlined in yellow these particular cells which share
a kind of interesting set of features. So like we were kind of saying
earlier how these features, there's computational reasons why the nervous
system would want to make its featured representation independent but then
sometimes the features come out to be correlated and I think this points out
like some intrinsic structure. So this yellow network is what I call the
preparatory network and you can kind of see it has this characteristic two
things. So one is that during local bending it's excited by the P cell
stimulus so directly excited by sensory input and then during shortening it
has this big fast component. So this AP cell is kind of the mother of this
network so you see this big fast component so it has this rapid
depolarization during shortening and it has direct sensory input. So then
this is just the canonical maps of these networks so I went in and looked at
the preparatory network much more closely so in this case, if you remember,
I'm always imaging from ganglion ten. But then I'm going to be stimulating
different ganglion to elicit these different behaviors so just depending on
how far away the ganglia I stimulate, it will take longer for the cells to
depolarize. So here in the bottom row, it makes a little more sense. So the
ganglia are just sorted by how far away they are from ganglion ten. Then I
would go in and you can find these inflection points where the cells
depolarize and you can mark the beginning of these depolarizations. And that
tells you the timing of the response. So remember that like the 14 and 17
and tail are eliciting swimming and then 7 and three are eliciting shortening
and so you can see these neurons, they don't really look different based on
the ganglion. The timing is slightly slower, but it's effectively always
depolarizing. So then, here's just like three examples that are in the
preparatory network and then just here's some other cells that are not in the
preparatory network in the same kind of gist.
>>:
[Indiscernible].
>> Paxon Frady:
>>:
No.
This is not in the literature.
So what do you mean?
>> Paxon Frady: This is why we call it the preparatory network. The first
thing is that it's rapid. It's receiving rapid input from every behavior.
This is showing us this rapid not only during shortening which we got from
the factor two, but also, it's rapid now from swimming as well. And it's
also receiving sensory input directly.
>>:
Solution between [indiscernible]?
>> Paxon Frady:
Yeah.
Because they shared this common feature dimension.
>>: [Indiscernible] jumping up a level, when it comes to single cells, how
likely to richer categorizations of sets of cells that might be same phase,
shared attributes and then once you have [indiscernible], does this set
have -- I guess my question would be calling it network versus set of cells.
So the question is what gives you the sense to call it a network? Everything
is connected eventually.
>> Paxon Frady: Now that we have this hypothesis, the preparatory network
and the ability to use this computational microscope in realtime, I can go
into new animal, I can identify the preparatory network, and then I can
target it and do more experiments so that's why I'm heading with this. I'm
trying to establish why calling a preparatory network. You can just fit
these response curves based on distances and so then you can see why you see
it's like you plot the response times of all the preparatory cells, they're
all faster in this blue clump which are just another handful of cells that
I've chosen out of cell identified. So the preparatory network is
consistently fast and it has nothing to do with behavior and essentially
looks the same across animals too. So the reason ->>:
[Indiscernible].
>> Paxon Frady:
ten.
>>:
That's just like how far away the stimulus from ganglion
[Indiscernible].
>> Paxon Frady: Yeah. Well, I mean, so this is like something that Bill
kind of hypothesized a long time ago that the sequence is more like there's a
stimulus, do something, he called it the do-something network, just like get
ready, like prime the muscles. And then there's like the decision and then
there's the actual execution of the behavior.
>>:
Just the do something.
>> Paxon Frady: That's what I think. There's no real behavioral defined
thing. It's all about the stimulus and it looks the same regardless of the
behavior. Even just activating center cells activates this similar network.
>>:
It's consistent.
What do you mean by consistent?
>> Paxon Frady: I mean it's the same cells regardless of the behavior. So
this is all one animal and these are cross behaviors. So these are just the
on set times during these behaviors of these cells.
>>:
You cannot distinguish but you can see them piling up every time.
>> Paxon Frady: Right. Yeah. You can't tell a behavior is going happen
from just the preparatory network.
>>: But you could just suggest [indiscernible] differences are in the
decision.
>> Paxon Frady:
Yeah.
>>: If you know that some prep going on and you hypothesize that that was
some sort of a queueing up of something. You think something about the
prep's end, show you little bit more about the do something, what it is do.
>> Paxon Frady:
>>:
Yeah.
We don't know.
Queuing frame to look for the differences.
>> Paxon Frady:
Right.
>>: Look over here now as the -- actually compare more deeply the difference
between the prep activities for different behaviors downstream. Maybe show
earlier in the prep level.
>> Paxon Frady:
>>:
It's possible.
Prep is really a decision being made.
>> Paxon Frady: Maybe. So then the really important part of the computation
microscope is to close the loop.
>>:
It's like this is our dream, BOI dream.
>> Paxon Frady: So this is like kind of what we call computational guided
experiments or computational guided electrophysiology. And so there's just
like a verification study. So I like find a new animal and then I generate
this swim activity map and that allows me to identify a handful of swim
oscillators and then I can go in and target them and so I'm just trying to
verify that these cells are really doing what the volt dyes say they're doing
and turns out essentially yes. So I can now come and have a new animal. I
can image a trial of swimming, do this analysis, produce canonical
relationships and then target these cell specifically for further
experiments. So there's just three ->>: So new animal, got your old data set, old analysis, and then it's going
to image it and the analysis [indiscernible]?
>> Paxon Frady: Yeah. I think that this component is closest to in the
future space to this cell that we see already, like 152. Right. And then
the computer can tell me what it thinks the cells are and then I can go in
and target them.
>>:
The interesting thing do would be to [indiscernible].
>> Paxon Frady:
>>:
Yeah.
Hasn't gotten that far yet.
You're close.
>> Paxon Frady:
I'm close.
>>: You could basically say, hey, listen, I don't need to waste my time
sticking the needle in 152. Where there's uncertainty right now is 201.
>> Paxon Frady:
>>:
Right.
And that's where I put my electrodes.
>>: That's where [indiscernible] directly.
[indiscernible].
>>:
Not waste time.
The same that we're using
Correspondence.
>> Paxon Frady: Yeah. So I mean, still building towards that but this is
the first. So then I can tell you that the S cell network and so we know
about this for a long time and so one of the cells that are part of this
table is part of the preparatory network is this cell called the S cell. And
it's really interesting because it's one of the few cell there are not
bilaterally paired so most cells have bilateral pairs. But it also forms
this like electrically coupled network that goes up and down the entire
length of the animal. So every S cell has this huge axon that's electrically
coupled to every other S cell up and down the length of the animal. So it's
been characterized as like a giant fiber system. So the hypothesis is that
the S cell is mediating the preparatory network so that the distal stimuli
are coming in through this giant fiber system activating the preparatory
network to get the animal ready to produce the behaviors. So then we can use
the voltage dyes to kind of like answer this question. And so fortunately
the S cell you can identify it just with electrophysiology. It has these
really big, really skinny action potentials. And in this particular
experiment, I ran a swim, I identified a couple cells and then I found the S
cell and then I targeted the S cell with an electrode and I just excited it
while also recording the rest of the ganglion with the voltage dyes. So then
I'm just passing this square wave of current into the S cell so that's the
optical recording of the S cell. And then you can see like a bunch of cell
getting depolarized and a bunch of cells getting hyperpolarized in response
to this S cell stimulus. And it turns out that these three cells are also
part of the preparatory network. So this is just exploring for the S cell
connections and it seems like the S cell is likely connected to a handful of
other preparatory cells and then we can verify this by again -- so here's a
different way of identifying preparatory cells so if you remember like a lot
of the preparatory cells had this pink phase during this local bin behavior.
So this is just like one of the easiest behaviors to do to identify it. It's
like the simplest experiment so then with this, we can identify AP 511 and
153 which are all part of the preparatory network and then I can then go in
with electrophysiology and actually do the intra cellular recording to verify
that this synapsis are real. And so you can see like the S cell to the AP
cell where the AP cell is just kind of the mother of the preparatory network
you can see like this one for one UPSP for SAP and then the S cell is also
connected to 155. Another preparatory neuron. It's weakly connected to 153.
And then you can see that it has no connections to these other two cells in
the same area with that or not part of the preparatory network. So then
here's kind of the summary so their cell bodying kind of spread out like this
through the ganglion. We have the S cell which is going to be forming this
electrical giant fibro system with other ganglion animal. And then sensory
evoked stimuli are going to come into either anterior or posterior through
this S cell network, activate the S cell which in turn activates the rest of
the network or like these local sensory cells are going to activate the
preparatory network directly and also activate the S cell which likely sends
this information to the rest of the animal.
>>: [Indiscernible] on this last three slides. So can you summarize what
about the analysis for virtualization help you identify preparatory network
and then do links between AP and the preparatory network, what about
visualization helped with that that you couldn't have done without
visualizations?
>> Paxon Frady: Well, so mostly, I rely on visualization more than the
actual algorithms like for the matching. The visualizations are simple
enough especially because I'm not really looking at all ten dimensions
because like the cells that I've targeted, you can just do like one local
bend to I'd 155 instead of having do all the behaviors. So just like getting
a visualization, like for me now, I've seen enough swim activity maps that
like I can just look at it and now exactly what cells are. So just like the
training on the visualizations.
>>:
[Indiscernible].
>> Paxon Frady:
Yeah.
>>: That helped you recognize the -- identify formula network and then link
it to explore further in terms of the S cell connection with the other cells.
>> Paxon Frady:
Yeah.
>>: Again, nice to know what would it have taken to get to these insights
without doing the investment in visualization matching.
>> Paxon Frady: Yeah. Like just doing the mapping was important for
feigning the preparatory network because then I really was starting to see
that a lot of these cells were really similar and that they shared these
features and that prompted me to go look at them much more closely.
>>: Do you think like zoning [indiscernible] kind of zooming into counter
matches helped? For instance, instead of [indiscernible] few candidates. Do
you think that was helpful at all?
>> Paxon Frady: Yes. I think being able to see the distances. Sort of like
I click on a cell and you can see, that helps a lot. A lot of times it
will ->>:
[Indiscernible].
>> Paxon Frady: Exactly. And a lot of times it will point a cell that you
weren't even expecting. There's one cell 232 which it can be almost anywhere
and down this central packet and at first it took me a long time to realize
it was there but then I started seeing it and then it started highlighting it
to me and then I was able to visually confirm it. So here's the overview of
the computational microscope and now I plotted to mapping the leech's nervous
system. So we have these now automated techniques which can extract signals
from raw imaging data without really much user input and then we can make
these activity maps or visualization and put them together into some kind of
canonical space and then we can map out the function of a bunch of the cell
and make these canonical maps for future reference and then we can kind of
use these tools to ask new questions and kind of do these experiments which
are at a whole another scale that we can never really do before. Then I kind
of want to tell you like this other -- where we've been doing this brain
initiative and trying to kind of fully characterize this nervous system and
so everything I've showed you was on one face of the ganglions really only
imaging half of ganglion but our collaborators have now built a new
microscope that has essentially an objective on top and on the bottom and I
went out there and helped set this up and now we can literally image from
every single cell in the ganglion at once and so now record from the entire
nervous system. They're in Cincinnati now. [Indiscernible] just started his
lab in Cincinnati about two years ago.
>>:
[Indiscernible].
>> Paxon Frady:
>>:
I see.
What's done about that other half of the sphere?
>> Paxon Frady:
>>:
And that is Daniel's post doc.
There's quite a bit.
It's kind of like the same thing.
Kind of like another duplication.
>> Paxon Frady: Yeah. It's like the motor neurons in this. There are more
motor neurons on the other side. It's actually kind of like I think like
Bill and Anna Free son agreed in the 1960s that one would do the ventral side
and one would do the dorsal side. I mean, it's like kind of similar. It's
about a third of the cells, mostly the large kind of motor neurons that are
known. There aren't quite as many cells on the other side either. You can
see this like stripe down the middle. This is actually the connective so
like the axons coming in from the other ganglia and going through.
>>:
[Indiscernible].
>> Paxon Frady:
>>:
Yeah.
Daniel is a post doc in Bill's lab.
Speaking of the old days.
>> Paxon Frady: Yeah. Yeah. He was still there when I started. He was
just leaving. So we have this and then we have my graduate student colleague
JSON and Bill's lab. We're working on getting the serial block face EM
reconstruction of the leech so if you guys heard of the connect dome, the
connect dome stuff, this is like how you do the connect dome. So it's our
dream has always been to like image the ganglion, image all these behaviors
and then take the same ganglion and get the EM connect dome so that we can
relate the function of all the neurons with the structure and anatomy and so
the EM is just amazing. You can see these nano meter resolutions. This is
leech EM, yeah. Just like look how complicated it is. It's just so
incredible. It's like here, just this cell makes like 17 synaptic
connections to this other cell. Just one cell. 17 different synapsis.
>>:
[Indiscernible].
>> Paxon Frady: Yeah. Need more than one. Need like 17. And then we are
also working with Larry Abbott on doing in computational modeling so he has
these recurrent neural networks that can produce these patterns and so we're
trying to consolidate all this data into the single framework. They're at
Columbia. So I just wanted to thank everyone. This is the lab and then my
advisor Bill. And thank Kevin Brinkman who gave a lot of input at the
beginning of this project. Ashish and Eric. And then Roger Chen's lab for
the voltage dyes and that's my committee. So do you guys have any questions?
[Applause]
>>: So those last 3 or 4 slides, do those folks, are they intrigued by the
computational microscope and visualization ideas or are they just going off
in the directions, those tools?
>> Paxon Frady: I think they are. I think it's kind of like another step.
Like after we kind of defined the canonical space, the mapping, the next step
would be to take the data and make a model from it and kind of incorporate
the Dynamics and the models too.
>>: [Indiscernible]. It's great. Very exciting to see how far you came
from the internship project to real world impact. I have to say, I've seen a
bunch of BRAIN acronym initiative studies but even though this is on a simple
system, this seems like it's made the most progress in terms of understanding
what's going on. What comment here, even leeches, it's pretty clear that the
fabric upon which we're perceiving and thinking now is based on some
derivative of these earlier systems, earlier evolutionary tree. It's hard to
know whether or not the insights that will come out of these tools directly
apply but certainly tools will apply to understand the systems. Just one
comment that excites me about -- besides [indiscernible] about the existence
of doing, the fact that we're listening, hearing, and seeing based on these
same kinds of networks to the unseasoned eye or microscope or computational
microscope was about the same close up. Any other questions or comments?
Thanks a lot.
[Applause]
>> Paxon Frady:
Thanks guys.