1
>> Alessandro Forin: Good morning, everybody. It's my pleasure to introduce
Professor Steve Liu this morning, who has worked with us for quite some time
and has graduated in a number of students who have come to with our company.
Steve has got his Ph.D. from Ann Arbor, Michigan in '89, has been on the
faculty of Texas A&M ever since. He is full professor computer science. He's
very active in realtime and cyber physical systems community, work on security,
specialize computing machines.
And today, he'll talk about his work on biomedical data, specifically on
retinal image analysis. Steve?
>> Steve Liu: Good morning, everybody. Thank you for coming to the seminar
either on site or on the internet, okay? Today I would like to give a
presentation on what we have done in the past ten years about the work related
to retina image analysis.
This work really start a long time ago. When the first time I saw this retinal
image, I was really stunned. But the beauty of this image, for no other
reason, I decide that this is really cool, you know, I want to, you know,
understand, to play with it.
So since then, we have come through three generations of evolution. The way we
understand the image, understand the structure, and develop different ways of
processing and structure the medical structures in the retina. It's probably
one of the most beautiful structures that I can think of within the human body,
but it also give us some real challenge.
So in the three generation, we have three Ph.D. students graduates here, and
they now work in different places. And this talk pretty much is the -- based
on the last student, Harry, or Huajun Ying's work. We evolve from the typical
kernel-based, you know, shape matching to the next level of we realized that
really, they are very difficult sensitivity versus sensitivity of the filters,
so with our own filters.
And then so the last level, the most current one of how do we really
personalize the parameter selections. Now we all individual, different. So it
come to the point that we realize this is a property much bigger than, you
know, just select the right filters.
2
So the work we present here is pretty much based on Harry's dissertation, and I
make a lot of reduction of the process. Through this year, I've been working
with him. So we finish this theoretical work, and right now we're looking into
how do we expand this into, generalize into image-based information processing
and how do we do a larger scale image management.
So what is retina? Retina is deep behind our eye, behind the eyeballs. So it
should not be confused with iris. Iris is in front of the eye. So that you
can use iris for biometric security application, but you don't use retina for
biometric security purpose, because what you need to do is pretty much put your
eye in front of this fundus camera. This is a very old model fundus camera,
which your eye stays here, and there's a digital camera and from pretty much is
a microscope, very high performance optical systems then can take picture of
your eye.
So as I said that when I start work on this area, it's really for no other
purpose but look at these pictures. This is really beautiful. Beautiful
[indiscernible] structures, and I want to know what is this about.
So pretty much it's like a look back into your eye, that your eye look at the
world, but this is the way you look back into the eye, which has a circular
structure in the middle. This part is your optical disk, which is all the
nerves and the blood vessels pull back from the eyeball into your brains. And
then the most critical area, in fact, is this dark area, this macula area,
which is your sensory, vision sensory area.
But our work very much are most surrounding on the mapping of these blood
vessels, because these blood vessels, they widespread around, and they're
dynamic, and they're the network supply the nutritions and to take away the
waste. So they reflect a lot of change in the bow structure. So even though
this macular area is the most sensitive side, but it's very silent and it's
very quiet. It has virtually no blood vessels in the surrounding area. But if
there's any kind of a fluid going to this area, you can be in trouble, okay?
Okay. So therefore, as I said earlier, our first generation is pretty much
look at how do we map, just like everybody else, all these blood vessels based
on the cross-cutting the shape. But then we evolve into the situation, how do
we truly understand if there's any relationship between this more dynamic
structure of the blood vessels which change or blood vessel more with disease,
3
the modification and the healing, versus this kind of very quiet area that only
have a very subtle change.
So here is some idea about what do they look like. This is a still relatively
good image in which that is darker. Our retina reflect our skin color. So
this is someone with darker pigments on their skin, and this is what we detect
of some of micro spots on the image.
Okay. Now in the literature, usually you will find the only very high quality,
very bright pictures, but we found some of the most challenging issue really is
related to this darker skin pigments. And, of course, we solve some of these
problems. But still, technology has its limits. And therefore, that it is
necessary and important to know that you don't get the silver bullets in this
kind of a caring -- this kind of technology. But rather, how do we personalize
individuals in [indiscernible] caring is essential.
This one is another example of retinas that, in fact, when we first look at it,
get a shock. But this look really ugly. But, in fact, that's not the case.
In fact, this is bright bands, they reflect very vibrant growth of young
people. Their nervous system is still growing; and therefore this
[indiscernible] like a very dirty cloud. But that, in fact, is very good.
It's young and healthy and the macula is very big. And this only means bad
pictures, okay?
So in our experiment we found that, in fact, a lot of time that the bad
pictures hinders any kind of analysis. That's a [indiscernible], excuse me,
okay?
But still is analyzable, because the reason why we want to do all this analysis
is to pay attention to whether or not the [indiscernible] network is healthy
with respect to the vision critical area, macula and optical disk, okay?
As we grow older, this is a situation that the image becomes a little quieter.
Now, the color reflect the skin color. So this is an Asian person,
middle-aged, very healthy, and you don't see any kind of artifacts around this
area, and the region still okay. It's very pronounced, okay?
And also, interesting to see that the -- if you have near-sighted, then your
retina stretch. So this is very interesting. In addition to the algorithm
design problems, so on, so on, I learn a lot about this kind of the health,
4
really, the issues, okay?
So if you have a glaucoma, then your cup and disk ratio change. And therefore,
this kind of a work, in fact, not only very challenging, it's also very useful.
So what happen if our eye get sick? In fact, they generally call it
retinopathy. Our work mainly focus on the disease induced by blood vessel
change.
So in this case, showing the example that you have a lot of fragmented blood
vessel. That means that this often happen to diabetes patients when they come
to a later stage. And pretty much is like acid in the, you know, garden hose,
and it just broken up and then just begin to leak everywhere.
So that is really a very bad situation, and our goal is not to diagnose all
these kind of problems, because if person come to this situation, then pretty
much anyone can tell, with some basic training, that vessel structure already
is broken down.
Our interest is more the earlier stage of detections, okay? So that when small
-- before they really begin to become pronounced or some of this bubble, like
micro aneurysm begin to emerge, then that's the time you want to detect them.
This is another example that when the [indiscernible] come to a certain extent,
the plasma, the protein deposit become fat, partly the fat deposit become
gel-ish on your retinas. [indiscernible] thing about this is that there is
asymptomatic. That means there's no symptoms associated with it.
So in doing this study, I learn one thing, that we really need to pay attention
something that occur, last for long time and then eventually come back to us.
Usually, in that situation if you come to the range time of needing clinical
intervention, that's already -- the damage likely already done. And corrective
process usually is nonreversible. They need to just cure the surrounding
vessels, not to make the situation worse.
So we did develop some algorithms to be able to detect all these dots, okay?
And this is represent a very, very high quality pictures that you see a lot of
dots, and you count, and that can report to the physicians.
So our objective is really to as a tool for the healthcare providers to know
5
that how bad the progress is going, because they need to use the different
medicine to help the patient to cope with the disease. At earlier stage is
easy to reverse. But at a later stage, thing a lot harder.
So past that stage, we begin looking to the issue of can we -- how much we can
go advance the resolution. And this is a very, very poor quality image in the
left-hand side that is very shallow. In fact, this is -- we suspect this is
some kind of a new vascularization, meaning that when the blood vessels went
bad, the body tried to repair it, produce some of these small vessels. And
this is very hard to see. And the computer can do a lot better job than human
in visualizing this.
So we pretty much [indiscernible] like digital amplifier, magnifiers to make it
easier to see. And being able to, in fact, use Gabor filters to see the shades
and the structures. I think this will be more useful for interactive process
of seeing this -- the evolution of the network.
So if you want to make a full [indiscernible], then you face real issue of -we face the real issue of the trade-off between sensitivity and the false
detection ratio. Okay?
So with time, we came to ask some even more fundamental questions. What do we
do -- what can we do with respect to so many diverse, you know, retinal images
from such a broad population base? We have different color, we have different
age, and we have different type of health conditions.
So is there any kind of a fundamental structures that we can lean on in order
to do a better job of mapping the vessels? Why we still talk about mapping the
vessels? Why we're maybe we say we're only interested in disease is because
this is similar to the situation that in an environment, in order to tell what
color is this, how many white balls in the mix of ball, you also need to know
how many black balls. To tell the white, you must tell black. And therefore,
that there's no other better mechanism.
And we learn this from experience, that if you try to handle a very complicated
imaging system, then you first have baseline. And the blood vessels are most
pronounced in the base structure, so if we can map that reliably and we can
build on top other analysis functions.
So this is what we did in the past few years.
We decided that this is the
6
first step, personalize it and then the answer is [indiscernible] with it, find
some very interesting [indiscernible] laws to do that. And then you can go
about recognize the placement of different objects and their relationship, then
you can go into higher level statistic analysis.
So what I discuss earlier is really mostly related to low level. And then in
this work, in this most recent work, we begin to see can we -- how can we
define and generate the, you know, reliable distribution, statistical
distribution. And then from that we develop the [indiscernible] of mapping and
then higher level work, which we are in the process of trying to understand,
how can we use, you know, specialized computer imaging, like FPGA, to do larger
scale analysis.
So the basis of the whole study here, that we learn from the nature. From the
nature, we found that the [indiscernible] generation is already observed by
scientists. This happens in the leaf. This happens in the blood vessels. And
therefore, that can we classify them based on the similarity of different
branches of these vessels.
You know, if we can do this for blood vessels, we certainly can also do this
kind of on the leaf, the transport of different stems, okay.
So therefore, that we first use the -- a contrast base to transform, which is
also developed by our team, and then we discover that when the map into certain
feature space, that we can map, then we can feed them into a [indiscernible]
distribution, which is a very significant step so that we can, based on the
model base, the prediction techniques, begin to understand where is the optimal
point to set the parameters. And thus exactly the notion of a personalized
parameter setting.
From that, then we have -- if we have a good way of personalize the mapping
[indiscernible], then the next level is to identify where is the target we
really are interest in, which is the macula area.
Now, it's easy, when everything is normal, okay? It's fairly routine. For
this one, there's no disease. But, in fact, there are a lot of challenges
actually to make this work.
The challenges is that when we have all kinds of retinas lesions, you know,
either they get really bright spots, so if you use intensity for landmark
7
recognition; for example, in this case, the optical disk is noted for brighter
spots. But what if you have something brighter, brighter here, then you could
mistaken this as optical disk and everything mess up.
So therefore, that being able to have the first low level reliable mapping of
the blood vessels, then we can use this flow structure itself as a way to
determine, because the ratio between the optical disk and to macula is fixed,
okay.
I think two or three radius of the optical disk. So therefore, that you can
allow more reliably position that macula area even when the macula area become
almost invisible, okay?
This can happen in many situation. For example, if we get older, our colors
block the area or if the person's pigment is darker, then you [indiscernible]
the situation, and we also see the situation that simply the macula become very
plain and is almost invisible, okay?
And then from that, we begin to look into given very silent structures of the
macula areas, can we classify them and how can we study. So this is still the
exploration stage.
The reason why we do this is because we realize in our experiments, any kind of
experiment involved with the human is really expensive. Really, really
expensive.
So if we want to analyze, understand persons in vision health conditions, say
in this room, where at a different age, in order to say that at a certain age
threshold that is my vision getting worse, progressing really bad, or is my
vision is just declining, follow a normal path, you simply cannot predict that
until you get that point. Okay?
But so therefore, that this is a very difficult question. But it's very
important question. So we try to use the population statistics and we hope and
we think there is some clue that may work that based on the relationship
analysis between the macula, which is a subtle image area, versus the blood
vessel structures, there may be a possibility that we can, based on the change
of the blood vessel structures, to tell maybe you have something going on with
the macula area which cannot be observed yet.
8
And there's a scientific principle in this, is because that first is that we do
find is fairly reliable to partition the three different type of patterns.
That younger age, healthy, which is in the middle, you notice that there's a
big ring. That because the nerve system is very strong. The second is a
quieter, so you [indiscernible] go down quieter. The third is the kind of
diseased eye so they're subject to all kind of lesions. They basically become
sometimes even [indiscernible]. Then they, when you project them into the
right feature space, they have a very good distance.
But then the question, interesting question even to me is if I am in -- in the
feature space, for example, in myself, if I can project myself into the feature
space, one of this, because the green one is the middle age, and I'm in the
middle age, okay, am I diverging -- am I moving my pattern into the diseased
area, or am I just staying around, okay? Even bigger is that when we are
young, are we moving toward a disease, or are we moving toward the normal path
of normal age?
So this kind of correlated analysis, as far as we can tell, that there is
nothing done in the literature yet, and our study did show there are some
strong statistics. Let me bring some clue about this relationship.
Now, what application? Now, I can talk about this equations all along two,
three years, a lot of study. But the bottom line is think of the possibility.
So if I can find a correlation, because blood vessel network change a lot
faster than the macula.
So if we can identify the regions that the blood vessel is changing, then that
will give a strong indicator to the healthcare providers, say you better keep
track of this individual. Then that will save a lot of cost in terms of
selective caring.
This is nothing to do with the policy, per se, but at least we know that this
kind of healthcare cost, most of the money is spent on the healthy people, on
healthy people. And we want to focus the resource on those that most need it.
So therefore, this will provide the [indiscernible] some sort of tools to allow
in the triage of the health providing that you don't have to bombard with
everybody with very, very expensive service.
Okay. So then let's get into the low level discussion of what why we think
that there's a chance that we can do this, we can find some sort of a pattern
9
associated with different structures. Okay? It's because the basic
observation that in the first generation of design, everybody try to fit this
with some sort of a kernel. Okay? Some sort of kernel.
But the difficulty really is how to select the parameters. Based on this
example, you already see the cross-section, they really have different, very,
very different structures. So therefore, if you try to [indiscernible] of
going to a different place, then you really face a major challenge of how do
you set the parameters.
So our approach is to say, well, I go through some sort of filters, and then I
collect them and transform them into a different feature space in the hope such
that the features gathered from vessel location of similar size can be put into
the same cluster. And it did happen.
And from that -- and therefore, through that, I mentioned earlier the normal
lognormal distributions, that you can use [indiscernible] thresholding and
techniques very effectively set the parameter which part you want to do the
thresholding. So to make this happen, we also design a different types of ->>:
So are you saying that size defines age?
>> Steve Liu: Age is different questions, okay? The challenge here is that
every -- most work is taken a lot of populations data. And then try to decide
optimal parameters. So basically, the parameters are for average person of a
population.
So we completely do away from that approach. We take your picture, your
pictures. From this picture, you have a lot of vessels. And then we try -doing some filters that I'm going to show next, okay, and then by mapping all
this filters outcome into a feature space so that your large vessels pixels in
the feature space all concentrate on the same point. So therefore, when you do
thresholding, it's a lot more reliable. And therefore, that process become
personalized.
Okay. So to do this work, and, in fact, we also change it from intensity based
filtering to contrast based filtering, because we observed that, in fact,
because of the very different intensity setting, and because of a very
significant pigment color change difference of different population, so this is
a way to normalize what you can see by the contrast, okay?
10
Now, this contrast filter basically scan at every pixel along different
directions. So we give this name daisy graph, because it like a daisy flower.
So you scale on different angles and see what are contrast values.
So these four figures here show four different location. One, two, three,
four. They are all on a some blood vessel through 36, 32, I think. Different
angles. You can see that they show some very interesting properties.
First is that they appear to have two different lobes, okay? And the lobes,
the size seems to be related to the location, okay, of the blood vessels. The
blue means they are negative value. Red means positive. Most of them are
negative. That's because blood vessels are darker when you're going to the
green channel, not red channel. Darker than the [indiscernible].
So therefore, that this give us some idea, if I move across a blood vessels, in
fact, the shape change. And the shape change, and if there any kind of a
consistency, then we can summarize all this cluster in the feature space. And
thus [indiscernible] to difference, okay?
So I don't need any kind of a population anymore. I just base on your image
that I can determine what's optimal parameters. Optimal parameters I come to
next, okay?
So we do a lot more experiments on this one, then we see, for the 16 points,
they have a very different structure, characteristics of this daisy graph,
okay?
Now, then the next question, this daisy graph is good for humans vision, human
visualization, but it is useless for computing the viewpoints. I don't know
how to compute this.
Okay, so therefore, that we need to transform this into some sort of numerical
structures that can be computed. Okay? And from here, it is clear, of course,
we have a lot more that we study, that we see that some major, major points
that from 11 to 14, they are all positive, okay. But then from the boundary,
you change sharply from negative into the positive territories.
Background, you may ask what about background?
of random non-structured, okay?
Background in general is kind
11
So then the key point, the key solution come to all this is that we eventually
find out you can use two feature indicators. One is energy, simply sum them
together. If they're negative, okay, then most likely they are blood vessels.
If they are positive, most likely is background.
Then the other one is called symmetry difference. Initially, we use the term
symmetry, then we find that this is kind of misleading, because what we really
did is compute the opposite direction, what's the difference on the two
different lobes if they're on the some sort of lobes.
If this is small, then that means they're symmetric. Okay? So therefore, that
if they are -- if they have any structure at all, then they can cancel each
other. This turn out to work very reliably, and you see this divided by CP and
so on, this is just nothing but normalizing the evaluation.
Okay. To verify our theory, we went through very lengthy analysis based on
some ground truths. If you gone to the web, you can Google the word drive
retina image database. You can see all kinds of nasty retina pictures.
But the good thing about this database, one good thing about this database is
that you have two human experts mark these blood vessels so you can have blood
vessel map out. One is more conservative, one is more, you know, detail
oriented.
So there's some interesting difference. And we use this to evaluate, what is
this feature space? How does that work based on the -- this is one example,
based on these two [indiscernible] join. Very, very beautiful.
We also discover quite often people say human drawn image is -- that's a very
common way to evaluate work is ground truths, and now we will say, we will take
a position, you know, the ground truths is subjective, okay? So therefore,
that really, based on that kind of an evaluation, sometimes it's not very
reliable.
Now, this is the feature space that we got here, and why is the energy is
either positive or negative?
The S is a symmetry difference, okay? So based
on this map, because the two have difference, so we live -- either a double
mark or single mark. Mean either both person agree this is blood vessel
pixels, or the other case is only one person says mark blood vessels.
12
So in this case, we knew that double marked blood vessels all have negative
energy. In fact, quite reliable. So this, the two. And for the reason why F
here, single marked blood vessel, have some positive, that's because sometimes
the human interpolate the natural gap between blood vessels and pixels, because
they are nervous fibers cross it, so it become shallow, okay?
But from the computing perspective, you can't just so completely be correct.
So therefore, that you do need to make some trade-off. But through this
mechanism, it give you a very systematic way to make that decision. And and
for this one, E, non-blood vessels, you also see some of them are negative,
okay? And negative energy but the vast majority of them are positive.
So if you -- so we use a zero crossing. That means the energy zero is a
negative or positive as a threshold to decide the process as positive vessel
and non-vessel.
You are subject to some small false detection, but that can be compensated with
very easy in the post-processing techniques. So I'm not going to bug you with
all these detailed statistics. Basically, it's done through a very extensive
analysis.
Then the next is that if we just
what can we learn from this? So
representation, once we take the
difference. It's negative value
do thresholding, it's kind of routine. But
based on the plots, and this is a different
zero crossing, we look at the symmetry
to a larger values. Okay?
Then we begin to observe very, very interesting phenomena. That across
different blood vessels map, the -- in the feature space, they all appear to
have a nice distribution, which turn out to be lognormal, okay? This represent
two different type of distribution.
The first kind is using the hand drawn, and we fit into a lognormal
distribution. That means that we can assign the parameters. Once we assign
the parameters, because lognormal distribution has an expression, right? Once
we estimate the parameters, then you would know that -- and you can reverse
back from the mean, from the media, from the, you know, one deviation, two
deviation, do they represent anything meaningful. And the answer is a
positive, okay? In fact, it's very strong relationship.
Then in addition to the mapping of the ground truths, there are another issue
13
of how do I use it. Okay? Now, because when you try to apply this -- when you
try to apply some sort of thresholding in the automated process, you don't want
to have human intervention, okay? And there's no easy way to determine that
what is the actual boundary.
So therefore, Harry is a very clever kid, so he basically come with the idea,
how is this compared to the [indiscernible] edge detectors? Now, edge
detectors cannot be used for blood vessel mapping, because they tend to have a
lot of noise, but do they have any kind of similarity? We use the actual map,
because this is the human drawn map. So eventually, we say well, why don't we
try this. And eventually, we found out, in fact, these two incredibly
consistent with each other.
So we also apply this lognormal distribution
fitting and turn to be also very consistent.
So that said, that means because of the consistency, and this is the -relationship between boundary and the blood vessels, they highly linear.
this is the different threshold values. Highly linear correlated.
And
So that means we can use regular edge detecters to decide where is the
threshold and then use that actually to RCTs based, because RCT and age
detectors are two different type of features, okay, to do thresholding.
So then again, over and over and over, for many different type of ground truth
image, this is two different persons mappings outcome, because one is more -the two human experts. One more conservative, one is more detail oriented.
So through this process, we can draw conclusion. Based on all these immediate
features, based on this model-based techniques, we can reliably determine where
we want to map, to what extent the detail the blood vessel will map. And this
is the outcome.
So if you set this threshold higher, that means you are most likely -- you're
going to map the RCT, you're going to map a small fraction. And we never do
that anyway.
Next, you go to next level, T-2, you get more details. And then T-3. Okay?
And here you notice that the white is human drawn pictures. Red what is we map
[indiscernible] with human drawn. The blue is what we detect human data did
not draw, okay, because a computer cannot be 100%. The anatomic structures,
this is basically the optical disk ring. Optical disk ring here. It has the
14
same signature.
But then we begin to see when we go to T-3, T-4, T-5, T-6, human stop adding,
but the computer add more, okay? Then we may ask the question, is computer
wrong? The answer really is no. Because one you tell the algorithm, if it's
darker, you pick it up. And therefore, in fact, this is a point I personally
felt that if the algorithm be in partnership with the human experts, you can do
a lot more than just human or just the computer, okay?
So in this case, the computer pick up a lot of very subtle, very small vessels
that human cannot pick up, okay? When you come to this level, even capillaries
level that didn't pick up. Because we see that this is not random. This is
all structured, okay?
So from that, we can say we believe that this is a very good way and very
effective way to actually map blood vessels. Mission accomplished. Okay?
Now, if you go in the literature, you will find that pretty much most of the
algorithms, including ours, already reach very, very high performance in terms
of mapping the vessels. Okay? But the difference is that we are able to come
up with a non-trivial personalization techniques that can do this reliably.
And this has a lot of implication in terms of for individualized caring for
anyone. For mass population.
So with that as base, so I'm not going to any of this detail mathematic details
here now. Basically, we proceeded with mapping of the major flow. Major flow
of the vessels to show that with the goal of localize the macula area.
Now, in this case, this become a pretty important challenge, because you can
see in this case, this person has a very severe lesions in that area that, as
far as I know, that pretty hard to use any kind of an intensity based or
[indiscernible] methods, except that you, if you first map the blood vessel's
topology, then you can determine that is in the central area, and this is two
different view, and this is optical disk area, quite reliably.
Okay. So basically show that in different diseases situation, the algorithm
work. Basically, you fit again the idea of fitting some sort of a topology.
In this case, they're a circle shape, and then take the center as the macula.
Then we can position that.
15
Okay. So after this basic work, we begin to ask this question. In fact, when
you try to push the envelope, you begin to realize there's a good reason why
people don't get into that kind of an area to work on the problem, because it's
a lot harder. Because you have much less clue to work with.
So why is it harder? Because as we discuss earlier, the macula area is very
quiet. It's very subtle change. So how can we possibly even tell something
going on?
So from that, we use the statistic correlation and study techniques. In fact,
with some very good result. And then with that work going on, then we go down
to the path of, well, if I want to really develop a health [indiscernible]
infrastructures, it will be very useful that if we can have some way to develop
some sort of a really [indiscernible] databases of the objects, image objects.
So that's where that we have some initial work, and we hope that in the future,
we can get into this direction based on the foundation we have developed so
far.
So for this macula area analysis, it's very routine. People apply some sort of
[indiscernible] techniques to say sick -- disease or not disease. Because
usually, that for macula area that you have disease, the lot of time, you'll
see this kind of bright lesion or kind of a scar is quite visible, even though
it's dark. For trained eye, they can spot it right away, okay?
But to us, what is interesting is how do we evolve? How do we evolve from a
normal, young-age eyes. In fact, in this case, you see a lot of small blood
vessel, even around the macula area supplied in nutrients and the
[indiscernible].
When we get older, they become much less obvious, okay? And the fovea area is
getting smaller and smaller. How do we know that my health, my macula health
vision area is decaying? Okay. So age related macular degeneration is a big
deal. Can we have some way to model it?
So the standard methods of Gabor filters, you know, pretty much is the
[indiscernible]. You apply this, and we use entropy statistics, basically just
take the absolute values and throw that in, into an LDA analysis, and you can
get some sort of classification with reasonable accuracy.
16
But the bigger question, as I mentioned earlier, if I were in some sort of
borderline area. For example, if I'm a person here or here or here, where is
my future? Am I going to this way, am I going to this way? We may say, I
mean, the vision probably usually associated with old age. Nobody can avoid
that.
But we begin to see more and more of proliferation of, you know, obesity, that
kind of problem causing vision problem. So therefore that if we have some way
to analyze and predict the future regression path or even for middle-aged
person, am I going to this way okay, or going this way, or am I really going
out of whack?
Now, this is a very small population for these small cases. If we add one
million or ten millions, what would happen, okay? So this is not trivial
problem. And I -- our studies only humble beginning of trying to understand
the dynamics.
Now, if we say waiting for some prestigious medical community to say go in and
lifelong study, that will be 30 years later, okay? I have no patience waiting
for that. So therefore, we try to think something crazy. Can we predict,
based on the population, similarities?
Now, within this population, what's the vessel structure look like? Because we
know that the vessel, blood vessels are a lot more dynamic and active. So they
actively go through modification and the healing.
If we can get some sort of statistic and understanding, then hopefully things
can speed a little bit, okay? So basically, the idea is to use the notion of
extractions. Am I more close to this area and that area and that area? So
this is all the difference, the distance between three different centroids, and
with respect to one and divided by the other two. Okay?
So if bigger, that means you're far away from that, okay? So this is where you
can derive some sort of similar data. Moving towards or deviating from, okay?
It's very interesting idea, okay? Even though we don't have direct scientific
evidence yet, because this is just recently developed. But hope that there
might be some opportunity, someone pick this up and get this going. Okay?
So the change of the structure maybe is a clue.
But we got to have some
17
systematic way to handle this. So is this change of the macular structure,
with respect to the global vessels, going to everywhere or some selective area?
Okay?
So, in fact, there's some way to find the medical definition of a different
areas. Macula, surrounded by optical disk. Temporal, superior, inferior,
these nine areas. Can we find any kind of relationship between them?
Okay. So again, we get into this official analysis. That, you know, how do we
characterize the blood vessels? You can characterize it based on fractal
dimension. You can calculate based on fircation point. The branches is an
indication of some new blood vessel may want to come up, okay? The length or
maybe because there's worn out or the curvatures or some curly structures.
We don't fully understand whether or not they have any relationship, but we do
know that doing the injury modification healing process, some of the blood
vessel tend to become very small but grow fast, very fast, okay?
So therefore, there are already some known structures analysis on this problem.
Then we go through the correlation analysis. And the detail of the discussion,
we can go either offline later, but basically, it's to define -- we realize
there's some correlation interpretation about the physical change.
From this nine regions, based on the three classification, is quite clear the
fractal dimension has very clear division between the three population groups.
For others, it's less clear. And this is still relatively, you know, young
people, they're all away from the older generation, older people. So no wonder
they have different behavior, right? Okay.
But it's interesting to note, you see that for this region, length, maybe they
have not much difference. But from here, the region 8 and the 9, this is the
one there's a pronounced difference between the normal one. Okay? And here,
region 8, which we usually don't pay too much attention at all, 8 and the 9,
they're from the peripheral area.
Now, from this camera, you can ask the patient change the angle they view and
you can take different regions and they have a commercial software to
[indiscernible] together.
18
This, in terms of a screening, we never thought of one to even look at. But it
turn out to be pretty important. So this is a very interesting discovery. And
then the statistics come out. It shows that when a subject deviate certain
directions, there are some pretty strong correlations at the different regions.
Okay? It's change from negative 1 to positive 1. And as I show earlier,
positive and negative [indiscernible] and you notice -- we notice that in this
case, superior to the length and the curvature, there's some pronounced
correlation. And so that means if we see something change in these regions in
terms of the curvature and the length, okay, something may be going on.
Similar story can be told about another one, okay? From young to middle age,
if you see that the curvature and the lengths going to the different regions
like this may be no big deal, okay? So this become an interesting and useful
approach before the disease occurs, and we actually take a look at the
progression, hopefully this is -- this can have some impact on the way that we
use the medical service, which we all know that is going on the roof because
they're the [indiscernible] deal with disease. But we are aging population.
The so-called active senior citizens, this kind of issue become very important,
right?
Same story.
Similar story.
Okay?
So as we progress with our work, we begin to look into a bigger problem? We
have some good handling of the low level issues, even those after decades of
the focus on this work, but we know that they still far from a larger scale,
you know, image processing, even though there's a lot of commercial service on
the internet and you can do curvy analysis.
But this is a very specialized area that we are looking at, how can we have
some way to classify a medical information based on certain criteria, like
health conditions, you know, can we help the health science experts, you know,
classify population based on the health conditions and decide, you know, what
is the possible best policies to handle all these services structures. And so,
okay.
So this come down to we need to have a way to process a very large amount of
image very effectively, very fast and being able to index them based on what we
need to do.
So this is early work, still based on our structure or our low level work that
19
we hope that we can advance from here.
So the particular [indiscernible] we're looking into is based on FPGA
implementation, okay. I will discuss this in the next slide. It will be very
short. I know that this kind of [indiscernible] just about anyway.
So then in imaging implementation, we know that still, the bottom line of this
whole processing is saying the computing every pixels, okay? And so look at
these competing structures how do we map into FPGA. We certainly know that
this can be applied to GPU. And there is one option. The FPGA, we have been
working that on the side quite some time. So we also know that this is a very
good performance factor with very, very small amount of power, okay. So the
issue is in the low level processing, how do we fill the pipeline? With such a
large chips, and, in fact, this is our [indiscernible] points. But in the
actual experiments, we basically use fixed, you know, precisions. And that got
get -- already got really good results.
The next issue is how do we fill the pipeline? So we divide a structure based
on the mapping between the computing of the
pixel -- the kernels with respect to the architectures of the FPGA. Then
eventually drew the conclusion that we can take every four pixels during the
processing and based on a sliding window techniques to fill the structure. I
have another slide about this. Basically just keep going, okay? And doing
this computation.
Now, even though we say per pixel processing, but really is involved is
[indiscernible] neighbors, okay? So even though this work is only related to
our own kernel, we believe this computing method will be applicable to a
further range of kernel-based processing.
>>: So this is the part in which you're trying to [indiscernible] the blood
vessel.
>> Steve Liu: This is -- no, yeah. This is the part you compute the filters
coefficients, okay. So this is the most expensive part and because every
pixel, you need to go through 32 orientation, right? Then you get a
coefficient and then you construct higher level features. That part's a lot
cheaper, okay. And I already have high level model. So the key of getting
[indiscernible] still is at the lowest level. And how do you optimize the -as all the experts here knows that how do you optimize the computing pipelines
20
into the communication pipelines.
>>:
So for every pixel [indiscernible].
>> Steve Liu:
>>:
Say again?
For every pixel --
>> Steve Liu:
>>:
Okay?
Oh, 32.
32 numbers per pixel?
>> Steve Liu: Right. And then eventually, this is the pipeline structures,
okay? So you basically fill the pipelines and the parallels on the larger
number blocks. Okay? It ends, basically, with the [indiscernible] we got. We
got 34 times [indiscernible] with respect to a regular PC in lab and this is
nothing but a variation board. So you imagine if we apply this in a
[indiscernible] processing environment, I think that this would be really serve
the purpose, acceleration with very low power, you know. Like or not that when
we came to a, you know, global scale -- not an environment, then the power
customization become really big deal. In this method, we determine that very
effective.
Okay. So in addition to the low-level work, we also want to, you can just see,
whether or not there's any possibility that can rank images based on some kind
of a new way of looking at [indiscernible] level, that can we see this kind of
notion similarity based on stream matching concepts, because stream matching is
very fast. So but they're two different problems. Because, you know, we did
some other work related to constrained repetition type of stream matching.
So if you imagine that there's some sort of structure with some sort of
[indiscernible] of some sort of [indiscernible], because any kind of high level
object, image object analysis, you will be required to go down to negligence
process. And that means find a number of selections which exactly means a set
of finer symbols. So we see some parallel between the between. And the way I
did the problem from [indiscernible], hopefully there might be some clue about
can we map this problem from image of J into stream string, to this string
matching problem, which we don't even know exactly what the form and shape it
would look like, okay?
21
So that pretty much conclude my talk, and thank you for your attention.
[applause].
>> Alessandro Forin:
Any questions?
>>: How much time have you spent on the FPGA solution that you described? Is
that something that's been worked on for part of a semester or part of a year
or ten years?
>> Steve Liu:
month or two.
architecture.
In fact, not much time at all. It's very recent that, about a
The student that he's also very, very familiar with the FPGA
So --
>>: The reason I was asking, the curious what level of optimization has been
done. The algorithms that you described where you were calculating the
coefficients, it seems like for instance if calculations have already been done
on one column, the opposite of that calculation is going to happen on the next
column so you should be able to leverage the ->> Steve Liu:
Very true.
>>: That there are things like that, that [indiscernible] would be better
suited for than the processor would.
>> Steve Liu: You're talking about sharing the results, okay. The sharing
results between neighbors. And this is usually -- this is very true among all
kinds of image processing problem, because image, you know, they have a
locality. So therefore, that they don't change that as facts.
So, in fact, I agree with you. In fact this can be a really excellent vehicle
for major applications such as compression. Kodak, okay? You know, I believe
obviously somebody already did the work. But then the question here, you know,
as we did the work earlier, we ask some basic question. Really did they
capture the physical property so that we can really get some very robust method
would be the challenge.
>> Alessandro Forin:
[applause].
All right?
Questions?
22
>> Steve Liu:
Thank you very much.