>>: Okay let’s get started. In the lecture announcement Daron Green is the official host, but he is busy
with other meetings so I’m not Daron. I’m Zane [inaudible]. It is my real pleasure to introduce our
today’s speaker Pascal Fua. Pascal Fua and I, we know each other from a long, long time ago, maybe 25
years ago. I don’t know, a long time ago, from France [inaudible]. He is a world leading expert in
[inaudible], especially in [inaudible]. But I don’t know today’s topic, so I am looking forward to it. He is
a professor at EPFL, Swiss Federal Institute of Technology in Lausanne and he is a [inaudible]. Pascal.
>> Pascal Fua: Thank you. Okay, so yes …
>>: I have not seen this one.
>> Pascal Fua: Yeah it is good. But, okay so we knew each other way back when, when we were doing
our thesis, our post docs and at that time I was working on cartography and aerial photos and that sort
of things and it may sound weird that I have moved now to modeling brain structure, but in fact, to me
it’s kind of a return to things I was doing a long time ago. Because what it is, is cartography of the brain.
So instead of being aerial cartography it’s brain cartography. But in many ways it is very similar.
So to give you a notion of what kind of images we are playing with, here is one. So what you are seeing
here on the left is what you see if you open the skull of a rat and stake what’s called a 2-photon
microscope. And you see this network of dendrites that are fluorescent.
Then if you zoom in, still with a 2-photon microscope, you see what’s in the middle. And basically the
dendrites are slightly out of resolution. You have kind of a speckedy soup and what you would want to
do is get a topology out of it. The ultimate goal of this sort of thing is something called connector mix
where you want to get a wiring diagram and not of the small part of the whole brain, but that’s still
some distance away.
And if once you have zoomed that much, you can change microscopes and use an Exton microscope to
zoom even more and now you are looking inside the cells and you see the structures inside the cells
which are referred to as organelles. So this is micrometer resolution. This is 5nm resolution. There is a
factor 200 between here and there.
And this is the range of scales we are interested in. We want to build circuitry maps from this. And in a
sense it’s, as we will see, it’s not that different from building maps of the earth from images of very
different resolutions. You want to extract very different kinds of structures and then put them together
into some form of unified framework.
So this is, I mean the long range goal, we are nowhere near there. So what I’d like to talk about today is
what we’ve actually been doing. And what we’ve been doing is to try to delineate some of the dendritic
trees of the speckedy soup in the low resolution images, meaning micrometer in this case, segmenting
the higher resolution images, so the Exton microscopy images, to find these intracellular structures and
then registering the one and the other so that in the end we can build integrated maps.
So let’s start with the delineation. So here is an image. This one is actually relatively easy. But let me
outline the process. So here is an image. In fact of course we will work with low dimension data. All the
images we deal with are cubes of data. And we first run the filter that is going to respond maximally at
the center of a tube. It’s looking for tubular structures. We are going to look for maxima of this
tubularity measure which are these red dots. And given the red dots, we are going to connect each one
to all the dots in some neighborhood, in some sphere, by completing a path of maximum tubularity of
the finest [inaudible] precisely in the moment. And finally, given this graph, we look for a tree. Because
we know that the dendrites actually physiologically form a tree.
So let’s look at these steps in a little bit more detail. So we start with a cube of data like this one. And I
like this one not because it’s particularly high quality, quite the reverse actually. It’s a very noisy data
set. And if you look at it carefully you will see that these so called tubes are in fact globs of things
because the way you obtain an image like this is you take the tissue, which initially is just gray.
Everything is gray. And you dye some cells to make them stand out. And the dyeing process is not
perfect. It’s actually very noisy. And you get these nasty looking tubes and that’s what you have to
work with.
So the first step is to run the filter and the filter that’s fairly standard. It’s called [inaudible] flux filter.
It’s designed to respond at really, at maximally at pixel, the value center of a tube. And you run this for
several scales because these things have width of course. So you are going to generate a 4D volume
which is the response at every location for a range of possible width. And you are going to look for local
maxima in this 4D volume. That will give you a …
>>: [inaudible] 4D.
>> Pascal Fua: So it’s, I mean if you ran it for a single width you get a 3D volume which is all the
responses, but you run it for several possible widths of the filter and then you get a 4D volume.
>>: So [inaudible] the volume here.
>> Pascal Fua: The volume is 3D initially.
>>: It is not just a slice.
>> Pascal Fua: No, no. Everything we do is on 3D volumes. Except debugging, I mean often times you
will debug on the 3D slice but the real computation is on 3D.
And then once you have these local maxima, you connect them by a path which I am going to call a
tubular path. By essentially you formulate that as finding a geodesic path in this 4D volume, where the 4
dimension of the 3 spatial dimension plus width, which, how you get this network of things where it’s
rooted as being irregular because the width are not constant along the path. And it is a graph because
for every local maximum you connect it to all the other local maximum within some distance.
And the final step is within this graph, you are going to look for a tree. And a tree is going to be a tree
that maximizes the aperture probability of it being the right tree, given the quality of this path that you
measure. And that actually involves some machine learning. So if I compare this to what we were doing
when we do the [inaudible] roads way back when at [inaudible] last millennium, all this was hand tuned.
It was a ton of parameters that were all hand tuned. Now we don’t do this anymore. What we do is we
compute, as in this case, gradient base features, graded [inaudible] of direction of gradients, we run
machine learning algorithm and that will actually give us a way to evaluate the quality of this individual
path in a slightly more principle way.
>>: This graph is 2D [inaudible]
>> Pascal Fua: This is, what I am showing is a projection but the real thing is in 3D.
>>: Okay. But then this particular projection is there and the particular meaning of [inaudible].
>> Pascal Fua: The meaning to what …
>>: [inaudible] the x and the y …
>> Pascal Fua: No, they …
>>: [inaudible] scale.
>> Pascal Fua: No there is actually no meaning. It is just the only thing that has meaning is the width of
the tube really is correspond to real measurement of the image. Okay. And that’s another thing that’s
changed in the last say 20 or 25 years. 25 years ago the algorithms we were using to do this would have
been basically the basic algorithm was to look for a minimum spanning tree. And it turns out that’s not
the right thing, or it’s not the best thing you can do in this case because when you look for this local
maxima, I’ll go back to this slide, the local maxima are these red dots you are trying to connect. And
typically what’s going to happen is you always find a few in the middle of nowhere.
So if you run a spanning tree, a minimum spanning tree style algorithm, it’s going to want to span
everything and give you a weird topology and then simply pruning the bad branches is not going to be
enough. So a better way to do this is to use more recent algorithms that say that you can actually
formulate finding the best tree to graph as a linear program. And not only that, but you now have
algorithms that will, it’s [inaudible]. So for any relatively big tree, you can do it but you have good
approximation algorithm that gives you to within really epsilon where it’s not a small of the optimum.
So now finding the best tree is really finding the maximum of the well defined objective function which
is always nice.
>>: This tree [inaudible]
>> Pascal Fua: Yes, that’s actually the whole point. Some points can be completely ignored. So it will, it
might, accept a few bad edges. Because if you had two very good part of the tree and you need a bad
edge to connect them, it will accept it but it will not, it, it’s just as happy to leave a whole bunch of
points unconnected.
>>: But would you allow like two trees that are disconnected?
>> Pascal Fua: Yes, yes.
>>: Okay.
>> Pascal Fua: So actually you will see this in a moment.
>>: [inaudible] of what this formula means?
>> Pascal Fua: Oh, no. [laughter] Now this is being [inaudible]. So the optimization problem, so what
does QMIP mean? QMIP means quadratic mixed integer program. And the objective function at the top
is what you’re trying to minimize, is, you have these edges or TIJ is a binary variable that says whether
the edge connected node I to node J is active or not. So you have this big graph and you are trying to
turn on and turn off some edges. And the cost here is something that has to do with the fact that every
individual edge has a cost, which is what the machine learning was all about. So the individual edges are
some number between zero and one. Saying do these look like they could be dendrites or not, based on
my training data. Plus, and that’s the reason it’s a binary thing, we have terms that says that successive
edges have to be compatible. So they have to be compatible in the sense that their width ought to be
compatible. But you don’t connect a very wide thing to a very thin thing. You don’t, this being
[inaudible] structures, and you prefer to have edges that are smooth to edges that are doing right
angles. And that’s essentially what this encodes.
And then the rest is essentially is a way of saying that if you turn on and off this TIJ and TJK, but you
want to turn them in such a way so that the result is a tree. And it turns out that this essentially is a flow
formulation and you express the fact that you turn them on and off in such a way that if you start from
the root there will always be a connective path from the root to whatever edge is turned off.
And so of course the reason I am not, I mean this is nothing that we invented right? This is standard.
This has become standard ways of finding trees in graphs.
Okay, so in this example, we have this stack and this is what you could do by hand. So when I say by
hand, the way this is done, is you have a victim. This one is a grad student, who essentially went and
clicked at various points and then we have a semi-automated path. It computes the paths semiautomatically to generate this. So this is the best we as people can do. And this is what this version of
the algorithm does, which is actually not exactly the same. But it’s pretty close. And actually here are
shades of things that were already true in the 1980’s, which is when you were doing automated
mapping software. Full automation was a pipedream. The system that truly worked are the ones that
did a lot of the job automatically but then was a good interface to actually fix the mistakes. And I think
it’s still going to be true in 2015, whenever people start using this.
So this is what we did that was our [inaudible] last year. And one thing we’ve realized this year is
actually, when I told you we are looking for trees, it’s actually not what we should be doing. Because in
this kind of data, the data we are working on is a relatively low resolution. So what often happens is you
have two branches. There are really two branches. But because the thing is blurry, they look like they
touch each other. Because especially in the Z direction these optical microscopes have a lot of blur.
So if in a case like this one, where you have two branches, these are really two different branches. In
reality they do not touch. But in practice what happens is you find one point of intersection and only
one. So if you enforce strictly the tree constraints meaning no cycles, you essentially force the algorithm
to make mistakes because it cannot use this node for two different branches. So it turns out that we get
better results even for structures that truly are trees by not enforcing strictly the tree constraints.
Modifying a little bit [inaudible] function to say okay we are going to allow loops, but we are going to
penalize, we don’t want too many free hanging branches. Not too many terminations. And we actually
get better results this way. So it’s essentially the same algorithm with a slightly different twist. And we
produce this. So this, of course this looks hopefully relatively nice, but I’ll give you some numbers in a
moment because in the end that’s the proof of the pudding.
And also we’ve run this. Here’s another example that I like, which is we’ve been collaborating with
people at Harvard who use a technique known as the Brainbow. That’s actually beautiful. You
genetically engineer a mouse so that different neurons generate, produce some different proteins that
fluoresce in different colors when you shine a laser light on them. And so this is where the rainbow
comes from or the Brainbow.
And the idea is to do you know you have some extra connectors where the different wires have
different colors. That makes it a lot easier to know what goes where. And it’s the same idea here
except of course it’s nowhere near as clear so here is what you do by hand. Here’s what the algorithm
does and it’s reasonably close. So, the question of course is how close?
Now let’s get to some actual numbers. And it’s actually, I mean by the way, and that’s true not only
here, quantifying how close two graphs are is not a trivial task. Because saying this is, I mean, because
you have many, many different mistakes you can make and quantifying which ones are more important
so that the obvious one, it’s worse to do a mistake close to the root than further away. And you have to
quantify that. And that’s actually not, there, I don’t think there is any settled way of doing this. But
there was a challenge in 2011 called the DIADEM challenge in which a bunch of people were invited to
run these kinds of algorithms. So there’s a two stage challenge where first you got some images and
you run them in your lab. Send back the results. Based on this there were five finalists chosen and
invited to [inaudible] the forum in D.C. to do it on the spot.
So we ended up being one of the five and sadly enough we didn’t win. And actually it was interesting
because the people who won, who are these people. There’s a team, the team here is somebody called
Roy Badrey [phonetic] from Texas. And what they did is they had a real pipeline. What they had, the
ultimate algorithm, I don’t think they were that much better than ours. I would even claim they
weren’t. But they had a decent interface. They had the process. They had a whole production line
assembly and it worked. And we were the only ones who were actually stupid enough. We didn’t have
an interface at all. It was truly fully automated. Because the contest rules allowed for interaction, you
would be penalized for the more interactive, the more penalized you would be, but it was allowed.
Okay. They won. Okay so of course we are competitive in nature so we’ve been working. We’ve been
at it since. And now we get numbers that are better at least that, the one they are using, the code they
published on their work. So of course they would argue I’m sure that we are using their code badly or
it’s not the best they can do and essentially we need a rematch. But …
>>: [inaudible]
>> Pascal Fua: They no, these are, okay, so these are, they are not these, these are as I said, they are,
they have the scoring functions that are very complicated and they have no, well okay, so I’ll let you
decide whether they have or not. Okay, the truth is in the real competition the DIADEM score was, is
really an automated thing. But when came time to actually decide who had won the competition, they
threw it away and they had human judge look at it.
>>: [inaudible]
>> Pascal Fua: So, sorry, but there was ground truth. There was ground truth. But they were to decide
whose, they even, the people decided the artist is not believable, which is why actually the NETMET
scores also come from this group. So we are using their metrics to try to do it right. To be better, in the,
so as I said actually coming up with a good metric, is an open research problem in of itself.
>>: So the number here on top is the wrong [inaudible]?
>> Pascal Fua: Well as usual in these slides ours, the bold ones are ours. It’s always true.
>>: [inaudible]
>> Pascal Fua: Yeah, yeah, yeah, yeah, yes. So this is I mean I don’t think you should. I mean, okay so
this is one of my pet peeves with papers in computer vision that I’m sure was opened the most recent
[inaudible] proceedings. Everybody is the best [laughter] because everybody’s numbers are involved.
There is something seriously wrong with this.
>>: [inaudible]
>> Pascal Fua: I mean again, these numbers are not computed voxels. They are computed on rewritten
graphs. So it’s the graphs and these measures, essentially you have measures of topology. How many,
you know wrong branches. It’s not the distance.
>>: It’s not positional.
>> Pascal Fua: It’s not the position. It’s an attempt. And that’s what makes it hard actually. It’s an
attempt at measuring topology.
>>: So I missed the beginning of your [inaudible].
>> Pascal Fua: These are, so the dendrites and the axons are essentially the wires in the brain that
connect, that connect the various cells.
>>: Each wire is actually a nerve?
>> Pascal Fua: No. They are, essentially what, okay you have a brain cell.
>>: Yeah.
>> Pascal Fua: It has a nucleus like all cells. And out of this nucleus comes something they call an axon.
And the axon, it generates the nervous influx or whatever you want to call, I mean whatever the right
name is. And this and it is being connected to dendrites which actually, so the axon goes out. You have
connections between the axons and the dendrites of a different nerve cell, which actually receives this
input and that travels to the nucleus of the second cell which then itself will generate something. And
so you have a very extremely complex circuitry and I had the number at the end. I think you had about a
hundred billion of these in your typical brain. A thousand times more connections so it’s really
complicated circuitry. So, that’s what I was saying at this point [inaudible] level we try to do one or two
or three. Really the target is billions. And so there is still some work to be done.
And so that’s the same algorithm. So different, so this is really back to basics. Or back to my Ph.D.
Which is you can run exactly the same algorithm on aerial images. So what’s changed here is okay we
are not in 3D. We are now in regular 3D images. We use different training data to score the [inaudible]
of this path. But the features we use are the same. It’s just we use different training data. And we run
exactly the same algorithm. And we get fairly decent results. So here I mean the results have been
offset on purpose just so that you can see the [inaudible] image. But you get fairly decent results with
exactly the same algorithm on a very different kind of data.
>>: So you actually did compute image gradients?
>> Pascal Fua: Well yeah, yeah. We literally compute image gradients. We compute tubularity
measures based on these image gradients and we run the algorithm exactly the way I have described it.
And that’s actually the point that was made this morning. I think we talked about it, but Daron is one of
the, in computer vision, there seems to be a division between biomedical imaging and regular computer
vision which does not exist. I mean to say an algorithm, and that’s, fortunately that’s what I’m
interested in. I mean I am doing computer vision algorithms. The brain data is a fascinating application,
but it is an application. It could be applied to something else.
>>: [inaudible]
>> Pascal Fua: I’m sorry? The rules in this case, they are similar rules. So in some sense it’s easier here.
But here, even here you have some I think maybe not in those images. No, these images it is true. They
are mostly all similar but they don’t have to be. Because after all the real, you know your freeways and
small, small streets and …
>>: There is actually no tree [inaudible]?
>> Pascal Fua: Right.
>>: [inaudible]
>> Pascal Fua: You could. We tried that. We had an earlier version of the algorithm where wherever we
suspected the junction we would put two vertices. But it was a little bit you know, as often, it could be a
vision, you are never so sure and we have to tweak the numbers and tweak the parameters and it
wasn’t really satisfactory.
Okay, so that was the delineation part of the talk. So now let’s move to something else, which is the
processing of the high level images. So the, in this case what we are looking for this intracellular
structures which are, there are many others, but the ones we have looked at are the synapses and the
mitochondria. So the synapses, so here is maybe a diagram that explains a little bit what’s happening, is
you have the neuron, it generates an electrical impulse and that electrical impulse is caught. I mean it
goes through a junction called the synapse and then travels to the nucleus of a different neuron. And
added to the junction, the electrical junction, or the chemical junction is called a synapse.
So the synapses are connectors and so they are, why are they important? I mean because if we are
going to get a wiring diagram, the delineation gives you the topology, and who is talking to whom. And
the synapse essentially gives you the strength of the connection.
And another very important structure is something called mitochondria. All of this has to be powered
somehow. And the mitochondria are the structures inside the cells that produce the energy that’s
required to do this.
And one thing that of course where does this come from, as any respectable computer scientist, I got it
off Wikipedia. And this is, if you go read the Wikipedia page, this is what the mitochondria looks like.
We look and we’ll see in a moment what they really look like.
Okay, so the algorithm is in fact relatively standard. So the first one we did was, you start with an
image. So here I am actually showing you a 2D slice because it’s easier to show. But the real
computation is done in 3D blocks. So you take your image. You compute super, what we call super
pixels. For voxel you just group. Why? Because this is huge amounts of data, you want to condense it
to something a bit more manageable by aggregating pixels.
So once you’ve done that based on local statistics, you can assign to every super pixel a probability of
being either inside or outside a mitochondria. So, the blue ones look like they are inside. The red ones
look like they are outside. You can also, because these super voxels sit on a grid, so again you have a
graph. And it turns out that mitochondria have very recognizable membranes. They have thick double
membranes. So you, what you can learn is when you look at two neighboring super pixels, do they look
like one is inside, the other is outside? Does this look like the boundary of mitochondria? So you can
look in this graph. You can assign property to edges of being transitions from inside to outside. In this
case the red ones are the ones that are the most likely to be transition. And so the place where you
would want to put the boundary. So that gives you a unary term for the super voxels, a binary term for
the connection between super voxels. And you feed that to a graph cut out of them and you get your
segmentation.
And so now if you look at it in 3D, now the super, the mitochondria voxels have been colored blue,
which means you can only keep them. You throw away everything else. And that’s what the
mitochondria look like in this particular volume. So they look like these long, snaky things, which is quite
different from the, from the Wikipedia page. So, by the way, it’s not that the Wikipedia page is wrong.
It’s in fact, it’s showing you the mitochondria in some specific muscle cells and they look very different.
>>: How are the 3D images obtained?
>> Pascal Fua: So the 3D, so it’s an actual microscope. It’s called FIPSEM. So FIPS stands for focus IM
beam. So you take your sample. You encase it in resin and then you use an actual microscope to image
the top so it’s called block face, to the face of the block. Then with the focus IM beam you abrase 5nm,
a very thin slice. And you take another image and you do it again and again.
>>: Well you actually cut it then.
>> Pascal Fua: Yes, this, I mean there are other ways, but this particular process is destructive.
>>: I see.
>> Pascal Fua: It is destructive, but it has one very nice feature. For us, it gives you images of isotropic.
A lot of the biomedical data often you have good resolution in x, y and terrible resolution in z. This
particular modality is the same in all three directions.
Okay. And actually something that’s interesting and that was striking, that shows it’s important to
visualize things, I showed this to the president of our university and he is a, he is a neuroscientist by
training. And actually he too was surprised to see this. Because all his career he had seen slices. And
it’s hard, even for somebody as bright as our president, to make the connection between the slices and
these weird shapes.
>>: How big is this tissue?
>> Pascal Fua: So this would be about, so it’s about 1000 x 1000 at 5nm x 5uM. Yes, 5uM. 5 x 5 x 5.
Okay so this works. We’ve worked enough to get at least some results. But actually one thing that, as
I’ve described it, it’s not quite right, which is I told you we were computing a unary term and then a
binary term. And we were computing them independently. So the property of the voxel is inside the
mitochondria and then the property of transition from inside to outside. But in fact if you think about it,
these two properties are highly correlated.
And so that’s not, conceptually that’s not quite right. And the way to get around this problem is to use
what’s called a structured SVM, which essentially you can learn. You have a binary objective function
with a unary term and a binary term. All of these are essentially controlled by ways that you learn.
That’s what you learn when you learn the SVM. And you can learn them jointly, which is much better
way of doing it. The problem is all structured SVM’s that I know are linear. Because the cost of doing
the learning is such that if you try to do it with kernelized SVM’s the computational cost explodes.
So our solution to this is, was to introduce what we call kernelized features. And the idea is very simple.
It is, you, first you can first train a normal SVM under unary terms of about what’s inside and outside.
That gives you a bunch of support vectors. And then you use as your feature vectors, not the original
ones, but essentially it distances all the kernel distances from what you are trying to classify to the
support vectors and you learn a linear structured SVM on this. And in this way you get the benefits of
both the kernelized version of SVM and the structured SVM at an acceptable computational cost. And if
you look at the cost [inaudible] paper, we have actually tried that for example in the Pascal data base
and we get decent results. So again, this is an example of the algorithm was designed for biomedical
stuff, but the technique itself is applicable to whatever you want.
And the other thing that seemed to have helped is to modify a little bit the way you learn the model by
replacing the standard way this is being learned by essentially a [inaudible] and dissent approach, which
appears to give better convergence when you do the learning.
>>: [inaudible]
>> Pascal Fua: Sorry?
>>: [inaudible] the second one, not the first one. I mean we were on the …
>> Pascal Fua: No the first one is a standard. It’s a no [inaudible] standard. It’s, you go back and learn
again, yeah. So basically what we are saying about this technique is the supported vectors for the
structured SVM are going to be pretty much the same. It’s just the weights are going to be different.
And it turns out that in practice, this seems to be a reasonable assumption.
Okay, and final little refinement is, and that’s an idea we got, Yuri Boykov has been doing graph cut for a
long time and always comes up with incredibly cute tricks. Is that, we have a problem where we have
mitochondria that have an inside, a membrane that surrounds them in the [inaudible]. In fact it turns
out that in this particular configuration you can do a three class, a three label segmentation using a
sound graph cut and be guaranteed the global optima. So we’ve actually adapted this trick to our
problem and that also seems to give a boost to the performance. So when you put all this together, in
terms of you know, the Jaccard index or the Pascal VLC score, we progressively get improvements in
terms of where you get better ground truth.
So these numbers are, get better, but of course they are not 100 percent which is where we want them
to be. And as of today, the biggest problem we still have is the fact that these mitochondria, sure we
have realized that a little bit late, are not as isolated as we’d like them to be. What seems often to
happen is in fact that they, they are glued to each other, or they touch each other. So for example, an
image like this one, it will do this. I mean basically these are really two mitochondria. They end up
being modeled as one. And if you look at what that does is this is what they look like. So they are really
even more complicated structures than the one I have shown you before. And the current algorithm,
what it does, it will, these are, they are really four different mitochondria here. It will just return one.
So in practice what we’ve done to be able to give to the neuroscientist something they could actually
use, is actually a tool of not a [inaudible]. I mean that style of thing where you will allow a user to click.
Sorry.
>>: Is there any shape higher than [inaudible]?
>> Pascal Fua: No, there isn’t and actually we, that, I mean maybe we can talk about this later, but it’s
missing. Clearly this whole big blob is not a, once you get the results, this is not a single mitochondria.
>>: So even at the loop, the purple or blue one, is that …
>> Pascal Fua: They all, I mean, this is the final result when we fixed it by, with this semi tubular
approach. So they all are pieces of mitochondria. That’s correct. But what the algorithm generates is
one big connective component for all of them. We’ve had to do some manual processing to actually cut
them apart and be able to color them. There are really four different mitochondria.
>>: But is that, so I guess you are talking about the four colors?
>> Pascal Fua: Yes.
>>: But, even the blue …
>> Pascal Fua: The blue one is one, yes. The fact that there is the branch in here, yes the neuroscientist
tells us yes, that’s correct.
And so in some sense actually that’s something that’s, this delineation stuff, it’s getting to the point to
where maybe we should use some of the delineation techniques. Because if we start getting these tree
shaped things …
>>: So [inaudible]
>> Pascal Fua: Yeah.
>>: It’s almost like a pipe.
>> Pascal Fua: Yeah it is kind of a pipe. And yet, what, in this kind of tissue, because I know that, there
can be, there are also places where this kind of little bean things.
>>: [inaudible]
>> Pascal Fua: And actually that’s why this kind of research is interesting, is, I don’t know that anybody
truly knows the answer to your question because there are so many different stuff in the brain or
anywhere else, that you need these tools to go and actually gather some decent statistics about this.
>>: Well I was going to say, how do they know [inaudible]? You, as a known specialist look at the
images. Can you tell that [inaudible]?
>> Pascal Fua: No, I can’t. I mean basically I [laughter] you know at some point we have some, we
collaborate with the people, the neuroscience guys and basically we believe them. [laughter] There
really isn’t a much better approach to this.
So, let me yeah, the way this is done, I mean, okay so in practice this is, if you want to be a little bit more
scientific about this, is you would have several different guys do it by hand independently and then
compare. I mean that’s about the best you can do. The problem is I mean doing this by hand takes
hours, days. I mean it’s seriously expensive in terms of man hours.
>>: Why is it important to [inaudible]? Why is that important?
>> Pascal Fua: Because they want to know how many mitochondria you have per cubic micrometer of
brain in that particular kind of tissue. And if you lump all four together, then your statistics are
completely off.
>>: So [inaudible].
>> Pascal Fua: Well okay so the … [laughter] For example, so we are involved in a project called a human
brain project. And the goal is to simulate a piece, a substantial piece of brain. And while in the
simulation you are going to have to put statistics about virtual neurons, virtual mitochondria, virtual
everything, and the virtual numbers should be close to reality. And so conceivably it’s our job to come
up with the right numbers.
>>: So, do the mitochondria have comparable qualities other than similar sizes, because if you’re
recognizing that they’re [inaudible] the size, [inaudible] one particular size range. And then you’ve got a
big blob which is showing up as four times the volume.
>> Pascal Fua: Which, yeah, that would be an [inaudible]. So, after the fact we can probably process
this and say this looks weird. So you could imagine in the system at least would tell the operator this
looks weird. You better look at it.
>>: Right, but it still [inaudible] potentially.
>> Pascal Fua: Yeah so would want to limit this as much as possible.
>>: Yeah, so maybe for those areas where you know there’s a problem because the volume’s large, you
go back and you do a different resolution of, you know when you do the segmentation? [inaudible]
You do that slightly different.
>> Pascal Fua: Yeah we would. I mean, sorry to get into this issue. I mean all the people here who have
dealt with segmentation know it the algorithm way using, still use very local information. Our algorithm
does not understand. Somebody mentioned, you mentioned shape priors. There isn’t any. But it’s not
good, but that’s also because it’s really hard to do. And so that’s why that this is basically what I’m
showing you is one Ph.D. I think there will be others on the same topic.
Okay. So for example, to try to get a handle about whether this is useful at all or not, here is essentially
what algorithm produces and the colors here are random. It’s just the various connected component
and just gave them different colors just to visualize. And this is a cleaned up output with a person in the
loop who’s actually down to cutting. Refine these things a bit and classify the mitochondria as belonging
to either the axons or the dendrites. Because these are the things they want to know. Now it is getting
to what the neuroscientists want out of this.
And some timing information, so this is what happens if you do it purely by hand. This is what it does
when you use the computing system. And that’s us. And here I have broken down the time into the
automated algorithm I have described and the reason they still have manual intervention is because we
need to train. So we need to get some data initially to do the training. Then we need to do a little bit of
splitting. So we use your [inaudible] algorithm to split the mitochondria and this actually this, we have
a debate. The guy who does this is a serious perfectionist. So that’s him. This is the time. He
[inaudible] correcting the things and cleaning them up really the way he wants them. And I think this
actually could be shortened quite a bit if we used things again. 1980’s technology, different role models
makes [inaudible]. So we’ll probably have to do that as well.
And in the end, with all this, and that’s what we are after. We get similar precisions to what you would
do by hand or using the computation, but much faster. So I don’t think we are going to get rid of
manual intervention any time soon. What we can hope to do is it used to take a week. It will be done in
two hours. Yes?
>>: [inaudible]
>> Pascal Fua: Yeah. We’ve been, essentially that’s what we’re doing. Essentially they, we showed
them the result. They say, so what’s the biggest problem? What’s the biggest problem and that’s why
I’m showing this thing about the [inaudible] mitochondria. That’s, at this point, that’s what bothers
them most. And once we fix that I’m sure they will come up with something else I don’t like, but …
>>: [inaudible]. Do you put different weight on different mistakes?
>> Pascal Fua: We haven’t done it formally yet, but I mean we’re not, basically we’re not yet at the
point where we do a real or clinical evaluation. It’s still too early for that but …
>>: I take it you, back to the closed loop issue. I take that you are not at a point where you have enough
data that has been human corrected to be able to model those corrections.
>> Pascal Fua: No. Not yet. No. Because our, it’s also because the algorithms change too much and
it’s, I think it’s …
>>: [inaudible]
>> Pascal Fua: Eventually, I mean it’s an important thing. But we’re not there yet.
Okay. So this was mitochondria. Let me say a few words about synapses. So, how would you detect a
synapse if you are a person? What you know is that synapses are very, are characterized by what’s
called a post, a post synaptic region and a pre synaptic region. And if you look at them the pre synaptic
region has these little things called vesicles of very specific texture. The post synaptic texture, the post
synaptic region does not. And between them there is this thing called synaptic cleft, which is this thick
black thing, which means that if you look at an image like this one, there is a thick black thing. But it’s
not surrounded by a pre and a post synaptic region. So I don’t know what it is, but it’s not a synapse.
So what you want is a detector that understands this. And this is what we’ve designed, which is a
detector that essentially is [inaudible] so you compute your own interpretation that’s relatively easy.
And then it has learned to essentially sample the space at various orientations and distances from the
center, computing features, and learning to recognize the synaptic voxels from the non synaptic ones.
>>: So the features you describe there are evidenced in 2D?
>> Pascal Fua: In 3D.
>>: Okay. So there isn’t, I was going to, that was the question I guess. Is there something else to do
with the volumetrics of the …
>> Pascal Fua: No. There’s really three. Again I make my drawings in 2D, but they are really 3D things
because these boxes are essentially referenced with respect to the center in 3, you sample the 3D
orientations here.
>>: Right. But, so are they circular in shape?
>> Pascal Fua: Okay, so …
>>: The synapse that, that region?
>> Pascal Fua: The region that you explore yes, but okay. So we, normally we should, these guys should
be sphere. But we use cubes because they’re close enough and a lot easier to compute.
>>: No, what I was meaning well does the form of the synapse itself …
>> Pascal Fua: Is spherical, yeah.
>>: Is spherical.
>> Pascal Fua: Right
>>: Okay. So there is another property potentially that you could explore, which is, is this. [inaudible]
>> Pascal Fua: Right. Yeah.
Okay and so here’s what it does. And so what happens is, you, now the synaptic pixels have been
colored red and I have the usual thing saying well we do [inaudible] on our friends. And what it looks
like is in 3D again. In that volume, these are all the synapses. The color will mean nothing. They are just
for, to look pretty. And essentially the synapses, or the synaptic cleft to be precise, looked like it is kind
of flattened pancakes with occasional holes in them. And that’s actually, according to the
neuroscientists, this is right. This is what they look like.
>>: Well some of them are really thin then, or long in shape, some of them.
>> Pascal Fua: Yeah. I mean they come in various shapes and that’s actually, there again, they want
statistics. Because this tells them what the strengths of the connection between two neurons is.
>>: So different connections have different [inaudible].
>> Pascal Fua: Different strength of connection. I mean the volume or the thickness or the area of the
connection, all of these have meaning to them.
>>: And the hole means where you have forgotten something. [laughter]
>> Pascal Fua: But the synapses, especially doing more for brain, more for genesis, they appear, they
disappear. Actually one of the guys here I was, from Harvard, tells that learning is actually about losing
synapses, not gaining them. And that, in at least for muscles, when the baby is born everything is
connected to everything. It’s a mess. And progressively as the baby learns how to handle its muscles, a
lot of the connection is severed until only the right ones remain.
>>: [inaudible] pruning process?
>> Pascal Fua: The pruning process, so arguably, by listening to me you’ve been losing synapses because
you have learned something. [laughter]
>>: Nice.
>> Pascal Fua: Anyway, maybe a little detour is, we can detect synapses. One thing we’ve been looking
at in connection with this is, so you were asking about the process. So the process is to take this cube,
abrase a thin layer, and then scan. And in fact, it’s called a scanning electron microscope because it
scans many, many, many times. In fact I think for the image I was showing you it has 44 scans for every
pixel. Because individual scans are very noisy and you need to average them to get something decent.
But then that gives rise to the following idea. If you want to speed things up because okay, 5nm, you
know this cube of 5nm, there’s a lot of them in the brain. So, is, everything that can speed up the
process is good to have. So what you could say is I know I’m looking for synapses. I’m not interested in
anything else. I should only scan the places where there are synapses. And so one of the things we have
been working on is to, they have techniques where you scan once. You train your classifiers. You say
okay, this area might contain synapses, but this one definitely not. And you rescan only the one where it
is likely that you’re going to find something you are interested in.
And in the end, so that’s what it looks like, meaning you end up for your whole image, you only scan the
maximum number of times some specified areas where it looks most likely that there will be something.
And that’s actually, we talked about astronomy this morning. I mean I don’t know anything about
astronomy, but it seems to me that if you are scanning vast portions of the sky, some of the same
problems are bound to arise.
>>: [inaudible]
>> Pascal Fua: Yes. So because the scanning software is designed to scan [inaudible] only, so you can
specify please scan. So of course then it becomes a slightly, if you want to do it right, because there is a
cost of scanning a rectangle, to moving the scanning head, to exactly so. I should say these are
simulations. We have not done this for real yet. But we are now, on the basis of this we are going to
collaborate with a small company in Canada that builds, that makes the scanning software. And to see if
we can actually integrate that in the real thing so that eventually it ends into a real microscope.
And we have some simulation results that show that with this approach we get, we lose a little bit of
performance in terms of [inaudible]. But not much, but we can make things a lot faster.
Okay, and final things. Sorry.
>>: So I was actually curious [inaudible] mitochondria and synapses. I was wondering, like the learning
problem of like you’re seeing a lot of data, little data of mitochondria and you can model something
about the mid existing cell.
>> Pascal Fua: Uh huh.
>>: Is there a way to transfer that to other sort of object synapses or other spots?
>> Pascal Fua: Okay so that’s, again that’s a very good question. Right now what we’ve done is we
know our target object. So I am sure there is a way of doing this without a specific target in mind. At
least, okay, all you know it’s brain tissue and you knew something about the statistics of brain tissue and
then you could probably use things. There’s been a lot of techniques about de-noising and all these
techniques in general that probably could be used. Because in fact, if, for those of you who worked on
de-noising, the way the averaging is done should have you scream because they take individual pixels
and they average them. They never look and that’s, we know that’s not the right thing to do so there is
a lot of unexplored stuff to be looked at.
Okay, so final thing and maybe I’ll go quickly, which is okay, I’ve told you about finding linear structure
on the image on the left. I have told you about finding intracellular structures and this and that are a
measure of the same thing at a very different resolution. In other words, this is somewhere in here.
The question is where. So this is a good old registration problem. The problem is here, is we can’t do it
the way we traditionally do this by finding feature points and matching them. The appearance is just
too different. It won’t work.
So the solution we have been working on is to use, to find linear structures at one scale, linear
structures at the other scale, and now you know that this is a sub-graph of that. And the question is
where. So we have [inaudible] algorithm that essentially does not use appearance but tries to match
junction in the two graphs so that you can register them. So it’s actually a two phase process. If I give
you a blue graph and a red graph, you will first try to find corresponding junctions in a two graph to get
a coarse registration. And then, we’ll do fine alignment to bring the two graphs into correspondence to
each other.
And that’s, and that actually works on lots of things. So you can for example, you can register retinal
scans. So these are two different retinal scans, same person, but because they were taken at two
different times, there is a bit of deformation between one and the other. You can do that on
angiograms. Arguably you could also do that, I have talked to you know, soldiers in the field who you
take a picture of what you see, you will see the new structured streets. You could match them against
your, the mapping software you have on your, in you backpack. And you can do it, which of course
doesn’t have the same implications on our live microscopy and electron microscopy data. With this you
can bring it to registration the high resolution block and the low resolution block. The idea being in the
end you end up with a situation like this one where you have the lower resolution stuff that you fill with
the structures you found at the high resolution.
And now the challenge of course is we can do that for little itsy bitsy parts of cells. The idea would be to
do that for the whole brain, which of course is going to keep us amused for awhile and just to give you
an idea of the challenge that this presents, if I were, I mean nobody is ever going to do this but, for a
moment let’s, a human brain has a hundred billion neurons roughly and a hundred trillion synapses.
And if you were to represent all this at the 5nm resolution I talked about, this would be a hundred
exabytes. I think I don’t know what the number is today, but I think the total storage capacity of all
computers on earth is about a tenth of that or some fraction of that. So if you need applications that
require seriously big processing of really big data, I have it for you, and with that I think I’ll stop. Thank
you. [applause]
>>: [inaudible] ask a lot of questions.
>>: [inaudible]. What is the higher level objective?
>> Pascal Fua: It really is computing the wiring diagram. So it’s a piece of the puzzle. You want to know
who’s [inaudible]. You want a wiring diagram of the brain the same way you want a wiring diagram of a
computer. So if I give you a computer, you have never seen one and you want to understand how it
works, if I just give you the wiring diagram that probably is not going to be enough but it’s one of the
things you have to have to understand what it does. And that’s what this [inaudible] field is all about.
Of course, in addition to that you need all the functional aspects which, so there are lots of other people
who work also on in vivo data, where you also, you only, here I am interested in getting the pathways.
But you want to know what goes along these pathways and that’s, then you get into the issues of
connective, of correlating. The kind of imagery I showed you was functional imagery.
>>: [inaudible]
>> Pascal Fua: I don’t think anybody knows. I mean they are good questions. I mean these are the
questions that eventually will get answered. So for example I have been told that the, I don’t know if
you’ve seen but the images they have, they look, they have this kind of striped appearance. There is this
kind of bands or how thick these things are. I think it’s connected to how much energy they produce.
And so again, these are things nobody, I mean basically I think nobody knows much. What is known for
example and actually what probably drives a lot of these researchers, all the new [inaudible] diseases, I
mean in Alzheimer’s patients, the mitochondria are bizarre. They don’t have the right shape. They are
scrunched up. So is it a symptom of the disease or is it a cause of the disease? I don’t think anybody
knows for sure, but there is certainly great interest in finding out.
>>: [inaudible]
>> Pascal Fua: It would. I mean I know that all the learning to count papers, yes, sure. I mean this is
another way of doing it and both are valid. In some cases you still, even if you have this learning to
count technique, you still want to find the usual people and find out where they want to go. In some
case you don’t. So in some sense, the, what I have shown, what I have talked about, speeding up the
scanning is of the learning to count flavor right? You don’t detect the wrong things. You just want to
know in that area well probably there’s something and that’s good enough for you.
>>: So for your last [inaudible].
>> Pascal Fua: Well …
>>: [inaudible]
>> Pascal Fua: Okay. So we need to go back to these images. Okay so in this, so you, especially need
these images to get the long range connectivity. But you don’t see any of the details. There is nothing.
It’s just white.
>>: No, no. I don’t understand. I’m asking you for the [inaudible] part.
>> Pascal Fua: For the what?
>>: What [inaudible]?
>> Pascal Fua: Right.
>>: Where there’s a higher resolution [inaudible].
>> Pascal Fua: Well if we just get to the nearest, essentially if get to the nearest structures and we do
some work and then we do graph matching because the one thing you will find in both places it’s the
same junctions. So the equivalent in a case of say normal photos is you have a map, you have a map of
your area with all the overlays. You climb on the roof. You take a picture of the streets around you.
And the junctions are going to be the same. They look very different because the resolution is different.
But you will have the same street junctions and you can try to match them. Now these are features that
you find on both scales.
>>: What I don’t really understand about that is I guess you were describing before how you take a resin
or something. You slice off …
>> Pascal Fua: Right.
>>: That’s destructive. Right?
>> Pascal Fua: Right.
>>: So how can you, that’s imaging …
>> Pascal Fua: Right.
>>: So how did you get the other LM [inaudible]?
>> Pascal Fua: Typically what you would do is you would for example do this in vivo for awhile, which
you can. And then also you can observe an animal for weeks at a time so I was saying [inaudible] of one
of one of our colleagues in Geneva has a way of, he takes rats, replaces part of the skull by a transparent
thing. And then every week he takes an image like this one. And the idea is to see the dynamics, to see
the network is changing. Then at the end, so you do what you politely call sacrifice the rat. And then
you can do, you can do it only once of course. You can do destructive imaging of this thing. And so this
is, and then this is a great data set because you have, it will in vivo evolution for awhile and then at the
end you get one snapshot of what’s inside all this.
>>: There was also a slide you showed where the experts segmented the mitochondria into either
dendritic or axon.
>> Pascal Fua: Axon, yeah.
>>: [inaudible] mitochondria. So presumably that would, you know that’s because a bunch of them
were in some dendrites and other of them were in some …
>> Pascal Fua: Yeah because they were looking at the images and say okay that looks like a …
>>: Oh, okay.
>> Pascal Fua: So basically we have done overlay on the actual images and we’re looking at the images.
Okay yes this is a dendrite. This is an axon. And I can’t tell the difference right? I mean if they tell me
it’s an axon, it’s an axon.
>>: Okay [inaudible].
[applause]