>> Ran Gilad-Bachrach: We are very fortunate to have Erienne Olesh coming from West
Virginia and she's going to talk about rehabilitation using the Kinect in the rehabilitation
process. Thank you.
>> Erienne Olesh: Thanks. Thanks so much for having me here. I do live on the other side of
the country right now but I actually really did my undergraduate work in Washington just down
the road in Tacoma, so it's good to be back to my pseudo-home or what was my home for five
years not too long ago. Hopefully, I can kind of keep everybody's attention. I know we may all
be in that sort of post lunch lag right now. But I really want to share with you some of the
research we've been doing, in particular the research I've been doing for my thesis, which is
really directed at using the Kinect for two purposes in stroke, stroke motor improvements. And
that is really first the assessment of stroke patients and their initial motor deficits and in
tracking their improvements, or in some cases decline over time, but then also using the Kinect
in one step further as kind of a rehabilitation device. Just to give you an idea since I think most
of you are kind of familiar faces to each other and you probably all know what the University of
Washington campus looks like, but just to give you an idea, I am at West Virginia University. I
am technically in the center for neuroscience, although, we also are kind of interdisciplinary
with the physical therapy department, so that nice glass building is kind of where I call my
home right now. We're located within the subdivision of the Center for Neuroscience, which is
actually a clinical translational stroke center. This center was developed not too long ago, but
we scan everything from human clinical translational work all the way down to more molecular
and genetic mechanisms of stroke, so we have a really nice breadth and depth of research
groups that are kind of all involved in this stroke center. We like to say that we have the
opportunity to study stroke patients because we are one of the highest states at risk in
incidence of stroke, which gives us a great patient population to pull from. Good opportunity
for us, though, maybe not such a great opportunity some of the people who live there, but in
either case West Virginia University Hospital which we are actually fortunate enough to be
physically attached to, so it makes it really easy to get access to both patients, neurologists and
clinicians. Ruby Hospital, which is our West Virginia hospital, is the only stroke certified center
in West Virginia, so any time a patient has a stroke in West Virginia, upon entry into whatever
emergency department in whatever county or city or local hospital they have, if somebody
thinks they're having a stroke they're immediately Lifeflighted to our hospital. Again, a huge
population of patients for us to pull from, which is really has really allowed the research that we
are conducting to really be possible since we do have this great breadth of patient population.
We're also really fortunate that we have great access to our neurologists and physical
therapists. We have these monthly meetings. They're always available to help and kind of talk
and collaborate with me and my PI as well whenever we need them. This has really enabled us
to kind of drive our research to meet critical needs. We have no interest in developing another
tool that clinicians and physical therapists have no interest or no need to use, because that
wastes everybody's time and money and there's really just no point in it. To give you an idea,
just kind of a basic overview of what our lab does, I'm a graduate student in the neural
engineering and rehabilitation lab. In general, our focus is kind of twofold. One, to understand
the underlying mechanisms of motor control, so how do we go from wanting to make a
movement to our central and peripheral nervous systems working together to integrate,
develop a plan and then carry out that plan in an effective action. We're also really interested
in looking at how these mechanisms change with neurological injury. What are kind of the
underlying mechanisms that cause somebody who has had a stroke to kind of present with
these typical phenotypes that we see in these motor impairments we see. We'd like to take
what we learn from both healthy individuals and those who have suffered from stroke and kind
of combine that scientific evidence to really build and help improve rehabilitation practices.
And that's really what drives the majority of research that goes on in our lab. How do we do
this? I'm sure most of you are familiar with most of these tools, but motion and capture
equipment we use in the lab; we use an active LED system, which is our phasespace impulse
systems. We have 18 cameras set up in our lab. We can put anywhere from 36 to 48 LEDs on
an individual to get full body motion capture. Fantastic temporal resolution, so we can record
at 480 frames a second and we get submillimeter precision spatial accuracy. We also,
obviously, use the Kinect which is a lot of the research I'm going to be talking about today. For
electromyography or muscle activity recordings we have two systems. We use our Delsys
system which is a wireless system when we want to look at patients kind of interacting in an
environment or moving around a space. We also have our motion lab system which is a wired
system. The benefit of this system even though it is a little more cumbersome is the fact that
we can synchronize everything incredibly well, so between our kinematic and our dynamic
information, we have fantastic synchronization. We can also synchronize it with our
transcranial magnetic stimulator that we use. For those of you who are not familiar, TMS,
transcranial magnetic stimulation basically takes advantage of magnetic fields and when you
pulse of this thing, it activates a portion of the cortex over which you're hovering. We have two
types of coils. The coil I have here in the picture is more of a V-shaped coil, so it is actually
shaped like this and what that allows us to do is penetrate deeper into the sulci, so the areas of
the motor cortex that are more in control of your lower legs. We also have a flat coil which is
more shaped like this and that can stipulate the more peripheral areas of the motor cortex so
that we can target upper body function. The beauty of the TMS is that it allows us to really look
at how the motor cortex is directly interacting with specific movements, so as somebody is
moving we can use the TMS to stimulate the cortical spinal tract and really look at the
involvement there of the motor cortex in the movement. We also are kind of, our new fancy
piece of equipment we've been kind of integrating into our lab is we recently got a Bertec split
belt treadmill. This is integrated with AMTI force plates so we get fantastic force data while
being able to manipulate these belts individually of each other for speed. We're doing a lot of
the studies looking at kind of perception of difference in speed, speed difference between the
two legs. Again we can combine all these, or kind of attempt to as synchronously as possible
and record all this data at once. We also use the Oculus Rift which is a 3-D goggle set up, so it
actually looks like snowboarding goggles that are all enclosed and this is great for immersive 3D environments. We use the Vizard system as kind of our platform to build these virtual
environments involve having individuals reach for different targets, and so the way we do that,
instead of having to have this big contraption in front of them, is we actually projected targets
in 3-D space. The beauty in what we do is once we put the LEDs on somebody's arm, now they
can see their arm actually move in space and that 3-D helmet, so we kind of have this
representation stick figure so they actually feel like they're really touching and reaching the
targets. I think this makes it a lot more realistic. I probably don't have to go over this. Hearing
some of the talks today I think we are all comparable with what motor impairment is, but,
again, it's something just important to keep in mind when we are designing both assessment
and rehabilitation protocols. We have to understand that motor impairments are very variable
and so they can affect just one limb or multiple limbs. The etiology of these diseases can really
be different, so even though somebody, two individuals may be presenting with similar motor
deficits, how those motor deficits are really arising in the individual can be dramatically
different. I think that's really important to understand when you're trying to treat those motor
impairments. Okay. Since I'm going to be talking about stroke, I just want to give you a little
background on some of the statistics as well is what a stroke is. This is not just a problem in
America. Globally, there is roughly 15,000,000 individuals who will suffer from a stroke each
year. It's the second leading cause of death and disability globally. When we look at it within
the United States, there's roughly 800,000 strokes that occur each year. It's become the
leading cause of death and the leading cause of adult disability. It's projected or estimated that
we spend roughly $74 billion treating both the acute and long-term effects of stroke.
Obviously, this is a pretty pertinent problem right now. For those of you who don't know,
although I probably comfortable saying that most of us do know here what a stroke is, but we
can kind of think of it as a heart attack of the brain. Specifically, there's two types of strokes.
We see ischemic and hemorrhagic. Ischemic strokes are far more prevalent. They cover about
87 percent of all strokes and this is where there is a blockage of a blood vessel in the brain.
Again, when you develop a blood clot somewhere in your body and its thrown or carried to a
portion of the brain and it cuts off a portion of the blood supply to an area of the cortex.
Hemorrhagic strokes, on the other hand, are slightly less common but can be as devastating as
an ischemic and this is the rupture of a blood vessel that then leaks the blood into the
surrounding area of the tissue. Size and location of strokes obviously vary widely, but size does
not always mean severity. You can have a very small stroke, but if it's in a really crucial part of
your cortex, it could impact you just does severely as somebody who has had a very large or
broad stroke. One important thing that we talk a lot about as far as initial assessment or
tracking patients in the more acute phases of the strokes, the first 48 to 72 hours after their
stroke, is the difference between core and penumbra tissue. A core of a stroke is really the
immediate area or the immediate cells that have been damaged or killed off by that stroke.
The penumbra is more of this surrounding area here and this tissue is kind of the at risk tissue
for becoming kind of enveloped into that core area. A lot of the acute treatments that are
projected or research for stroke, acute stroke treatment, really focus on salvaging is much of
that penumbra tissue as possible and really limiting the size of that core. In the long term, the
less amount of tissue you can lose, the better prognosis and the better outcome the patient will
have. Acute treatment or assessment of a stroke, again, when somebody comes into an
emergency room and is presented with stroke like symptoms, they may have a very basic
neurological exam and if a neurologist or clinician thinks they are, in fact, having a stroke, the
most typical second step will be a radiological assessment. At West Virginia University, in
particular, we start with a CT scan and then move on to an MRI scan if it's deemed necessary.
This will really tell you the location of the stroke and perhaps the severity of the stroke if you
can get at that information. To treat a stroke acutely, so once somebody is in the emergency
room and you deemed that they are, in fact, having a stroke, you kind of have two treatment
options as of right now. You can use TPA, tissue plasminogen activator, which is given through
an IP injection, travels to the site of the clot and basically just worked to dissolve that clot up
and return blood flow to the area. The other thing that's being use right now is a mechanical
device. There's several different types, but just to give you an idea, this little cathode is snaked
up through a major artery. They basically find the site of the clot and then through either a
balloon or a kind of twisting mechanism attached onto the clot, retrieve it and, again, the idea
is to return blood flow to the area that is lacking blood. When we talk about initial stroke
assessment, one of the things commonly used in the emergency room setting is the NIH stroke
scale. This is a very broad scale. It covers everything from speech and language to ability to
swallow to very generalized motor functions, so they really just look at upper versus lower
extremity function. It doesn't give a lot of motor detail. But this can kind of tell a neurologist
may be where somebody is initially and they can track it in the 24, 48, 72 hours post stroke.
When we talk about specific deficits associated with a stroke, we kind of have two broad
phenotypes that we look at. We have this spasticity or this increased muscle tone and this
obviously leads to problems such as being able to grasp an object or release an object,
decreases fluidity in motion as well as range of motion. Hemiplegia is the other kind of typical
phenotype we see, and this is more of your flaccid paralysis. As we talked about earlier, in
older adults especially, who have this flaccid hemiplegia after stroke, this really leads to an
increase in susceptibility to falls after a stroke. Again, it's important to keep these phenotypes
in mind when we're developing both assessments and rehabilitation tools because certain
patients are going to have different levels of the kind of general motor impairment right after
their stroke. When we're talking about initial assessments, a lot of the tools that have kind of
been developed to initially assess motor functions of a stroke patient really have maybe been
develop more for research purposes are research tools to kind of look at effectiveness of a
treatment over time. Some of the problems associated with these assessments -- sorry that
that is cut off there. It's just the title. Some of the problems associated with these is the fact
that there are very wide ranges as to what they are actually looking at. Some of these
assessments focus more on ability to manipulate fine objects, whereas, other assessments by
look at more broad range functional activities of daily living. Some of the benefits of these
assessments is that, again, they can be a really useful research tool, so when we want to look at
the effectiveness of the treatment paradigm, we can use one of these assessments to kind of
get an initial idea of where the patient is at and then look at their progress over time. The
drawbacks to these is the fact that we're really not sure if we're capturing these underlying
motor deficits that they have after a stroke. Again, when you're asking a stroke patient to lift a
cup and you're measuring their ability to lift the cup, are you really seeing what is underlying
that motor deficit, or are you just looking at their ability to lift the cup? Also, the scores can
sometimes be vague and they are often not used consistently in a clinical setting. If you're not
tracking a patient's progress over time using kind of a standard assessment, how are you really
knowing if the treatment or rehabilitation paradigm that you're providing that patient is really
working? Also, a lot of these assessments can be lengthy and they can be difficult to perform
whether it's due to the size of area you need to perform the assessment or the specific tools
you need to perform the assessment, so again, benefits and drawbacks. Our idea is really to
kind of meet the needs of both clinicians and patients to develop some sort of assessment and
rehabilitation tools that may be better able to characterize some of these underlying motor
impairments. I kind of have the project that we're working on right now broken up into four
aims. In aims one and two I'm going to be focusing more on the assessment side of things,
whereas aims three and four are a little bit more geared towards long-term rehabilitation of the
stroke patients. The first thing we wanted to do was, again, we need to come up with a tool
that's low cost and can be used clinically. We talked about earlier today, we have these
fantastic motion capture systems, but to put them in every clinic seems a little bit not very
feasible and they're very expensive and the number of personnel and how well the personnel
have to be trained to get information out of those systems really can be quite cumbersome.
The idea we kind of had was let's find a better option. Let's find something that's easier to use
and is more low-cost. The Kinect obviously presents itself well to both of those options. Our
first idea was to compare how well the Kinect can capture kinematics compared to a state-ofthe-art system. That was really the first kind of general goal of this first aim. We also wanted
to determine if we could use in automated algorithm to actually score a stroke patient’s motor
ability. Again, this is kind of our phasespace system here, active LED system that we use to
compare against the Kinect. What we did is we had 10 stroke patients perform movements
that are based on clinical assessments. The movements we had them perform are illustrated
down here. This diagram just kind of shows the actual difference in positioning of the active
markers, the LED markers versus where the Kinect tracks the human body. We recorded as we
had our stroke patients perform these movements; we recorded them simultaneously with the
Kinect and with the phasespace system. After we recorded that data we calculated joint angles
of the shoulder, elbow and wrist and we basically compared kinematic or motion capture ability
between those two systems. If we look at the average error between the two motion capture
systems, this was across all of our patients in all 10 movements, we see that we get an average
difference between the phasespace system in the Kinect system for shoulder flexion extension
of 14 percent, shoulder abduction adduction of 12 percent, elbow flexion extension of 2
percent and wrist flex extension of 7 percent. 14 and 12 percent may seem, maybe on the
large side of things as far as an average difference between the two systems, but what was
really more intriguing was the fact that those averages were pretty consistent across all of our
patients. The benefit to this is as long as we know that the two systems have an average
difference that's fairly consistent, it really shouldn't actually affect the scoring that we did later
on with the kinematic data. Again, then we wanted to determine if we could use the phase
space and motion capture Kinect systems to automate the scoring of a stroke patient's motor
impairment. In plot A what we have here along this line here, along the x-axis are the scores
derived from the phase space motion capture system compared to the scores derived from the
Kinect system. Again, we had a pretty good linear relationship between the two suggesting
that the two systems basically the kinematics they were extracting more comparable to create
this quantitative score that we developed. We also compared the Kinect scores which are now
on the x-axis to the qualitative scores. These qualitative scores were generated from 30
physical therapy students rating each one of our patient's movements. We would be a record
each of our patients. De-identify them by basically blurring out their face and then we had 30
physical therapy students rate the patient's movements on a 0 to 2 scale and that's what we
used for comparing to our Kinect base scoring system. Another interesting thing we found from
this data was when we look at inner class correlation coefficient or basically how well the
human raters are performing compared to our system, we found that it was usually about an
average of three human raters that it took to get a comparable score to our computer-based
algorithm. This suggest that perhaps it could provide a more accurate score than one or two
physical therapy students and maybe it could perform better in a clinical setting. The second
kind of wave of this research was to determine if we could use this automated algorithm we
have developed to really automate a clinical assessment. For this we chose the Wolf Motor
Function Test. If you're not familiar with it this consists of roughly 17 movements that are
pretty typical of activities of daily living. We did omit two movements just for some feasibility
purposes, but we had 10 stroke patients perform the Wolf Motor Function Test with both of
their affected and unaffected limb and we recorded them while they were moving with the
Kinect. Again, we had physical therapy students rate their movements on the 0 to 5, now, scale
that the Wolf Motor Function Test uses and then we compared our automated algorithm to the
performance of those physical therapy students. To give you an idea of how our algorithm
actually works, what we do is once we have calculated the joint angle data for both the affected
and unaffected arm, we use a decomposition analysis and really what this does is it pulls out
the major trends or the major variants in the data. I will go through this step-by-step and
hopefully it will make as much sense as possible. The solid lines that I have represented in all of
these graphs are the actual kinematics recorded from the Kinect. The dashed lines are the
reconstructed movement that we get by extracting two principal components from the
unaffected arm. What we do is we extract these principal components here and then we
reconstruct the affected arm movement. What we use is we use correlation coefficients to
quantify how similar or in some cases how different that reconstruction was from the original
data set. What that allows us to do is really quantify differences in movement ability between
the two arms. So in the top plot here, this section here, we have a mildly impaired patient who
really didn't have a lot of difference in motor ability between their two limbs. This individual
would have received a high score on our quantitative algorithm. Whereas, if you compare it
with a more severely impaired patient, you can see here they have little to almost no
movement ability in their affected arm. In this patient the difference between the affected in
the unaffected limbs would be greater so they get a lower score on our quantitative algorithm.
If we look at the overall observations between our quantitative scores which are now here on
the y-axis and the scores from the physical therapy students, we actually get a surprisingly good
relationship between the two. One thing we did notice and through talking to some physical
therapist and actually talking with Doctor Wolf, who unbeknownst to him, that test was named
after him. He didn't actually develop that test which is kind of interesting. What typically
happens a lot is these floor and ceiling effects here. If I can get my pointer, so it's really easy to
get somebody a five which is perfect movement or a zero or one which is almost no movement,
but it's really hard to delineate in between. What we see is a lot of our patients were given
these scores that average shall somewhere between four and a half and five, but we see a lot
bigger difference in our quantitative algorithm. We don't think that that's because the
quantitative algorithm isn't working well. We actually think that's because this is based off of
kinematic recordings. It's actually picking up some of the minute differences that a human may
not be able to detect just through eyesight. The other important consideration for the Wolf
Motor Function Test is that there's a time domain associated with it, so all of the patients are
recorded on their time to perform each test. We wanted to make sure that our automated
algorithm could also detect onset and offset of movement and what we did for this was just
basically used a kind of simple velocity threshold detection method. Again, we saw pretty good
correlation between that and the time domain found by human physical therapists. Again,
those were kind of the two assessment based research tasks that we've taken on, and again, we
really want to kind of alleviate some of the costs and time constraints that cause these
assessments to not be used on a consistent basis in the clinic. We also want to provide this
more quantitative measure for motor impairment so really kind of make some sort of defining
numeric scale basically of which you can measure a stroke patient’s improvement over time.
Moving away from the assessment side and getting more into the rehabilitation side, aims
three and four are much more geared towards looking at long-term rehabilitation using the
Kinect. Some of the things we want to address in games three and four are to track patients’
improvements but also to improve specific joint angle scores. Right now we get one score for
the patient. We'd like to actually get scores for specific degrees of freedom at each joint. We'd
also like to eventually bring this care into the patient's home and develop this network for
rehabilitation, which I'll get into more in a second. I'm sure many of you know that for any kind
of rehabilitation the more often you can provide the rehabilitation and the earlier on in the
injury you can provide the rehabilitation will provide the best long-term outcomes. There has
been a lot of recent improvements in technology that we all have been discussing today and a
lot of these recent improvements in technology are now being incorporated into rehabilitation
efforts. Motion capture, accelerometers in gaming systems which we have already kind of gone
over. Some of the concerns associated with rehabilitation, again, I'm probably preaching to the
choir here, but time and cost. Home compliance is a big issue. In West Virginia, in particular,
we have a large rural population and it makes it really difficult for these patients to travel to
Morgantown which is where the university hospital is to get their treatment. Some of the
possible solutions may be to implement the Kinect type system in a home for home
rehabilitation or tele-health networking, and also to provide patients with interactive games
that might help boost this home compliance issue. Again, like I mentioned before, possible
ways to address some of these concerns, it really needs to be something that is easy to use and
is low-cost. We all kind of are facing this issue right now. Insurance companies are getting
stricter and stricter for what they want to pay, so costs associated concerns are really at the
forefront right now. One idea we had was to develop this wheel and spoke network which
basically would act as having a couple centralized physical therapists who could kind of monitor
patients in real-time in this kind of spoke like system. Aim three, this is our future direction.
We're kind of gearing up to get started right now, is to now take this automated system we
started studying in the lab and actually bring it into a clinical setting. Sorry that that's kind of
cut off there. What we would like to do is actually set up system in a clinic and actually assess
patients upon arrival prior to seeing the clinician and so when the clinician actually goes in to
meet with the patient; the clinician already has a printout of a score of the individual’s motor
impairment. We then want to record this information over visits and examine these long-term
outcomes from tracking their improvements over time. Aim four which is another future
direction we are moving in the direction of now is more of this home monitoring and
rehabilitation system. Again, using this kind of wheel and spoke approach, we want to set up
these Kinect type systems in individual’s homes so they could actually do this rehabilitation
while being monitored in real-time by physical therapist. We also want to provide them with
some interactive games and one idea we had too was to kind of boost home compliance,
maybe set up a network where these patients could actually play these games against each
other. We all kind of know that competition drives a lot of things, so if we could give these
individuals in these patients competition through playing with each other through these online
games maybe that would kind of help boost their compliance. This is kind of a random little
side note here, but recently our lab was visited by the Prime Minister of healthcare from
Estonia. Estonia is a very small country which I actually had no idea, to be honest, where was
geographically, but it's roughly almost the same in population size as West Virginia. They also
have an extremely high rural population. By the way, if you want to know one interesting fact
about Estonia that's where we got Skype from, so thank them for Skype. What they have really
moved to because of these high rural populations is that they are really now relying on telehealth care. In these more rural communities they are really relying on kind of this internet
network to provide healthcare to their individuals. We see this as kind of an interesting case
study and hopefully eventually this is where we could kind of move to in the United States,
providing patients with more in-home rehabilitation and therapy options. Kind of my last thing
I always like to remark especially when talking to clinicians is the fact that we are not trying to
replace physical therapists. We are simply trying to provide them with a tool that would help
assist them to do their jobs and provide better long-term outcomes to the patients. One of the
comments we get a lot from physical therapists is this system is going to put me out of a job.
Physical therapist will never be out of a job. We all know that. You always need a person to be
there to interact with a patient. We really just want to provide a tool that would help alleviate
a lot of the problems that they are having and the patients are having as well. With that, I just
want to acknowledge some members of my lab. My P.I., Doctor Gritsenko, the post doc and
our lab doctor, Doctor Talkington, as well as our lab tech who does a lot of the 3-D
programming for our virtual reality system, Brad Pollard, and our collaborators, Sergiy
Yakovenko, who works with us on our human projects as well as does some animal modeling
for us and then the funding of my fellowship. That's all. I'll take questions. [applause]. Yes?
>>: Is there an association between the programs that you can naturally show the Kinect like
standing in one place in some realities that you can do? This could [indiscernible] from the side
of the Kinect, so I don't know what the potential is.
>> Erienne Olesh: So you mean the ability to use different types of kinematic data in this
scoring algorithm?
>>: Yeah. I mean the Kinect is aiming, like people will study one place and other people other,
so you will drive into the wall, right?
>> Erienne Olesh: Yes.
>>: So an extension of the [indiscernible] itself?
>> Erienne Olesh: Right.
>>: How does it connect with [indiscernible], the analysis of different positions?
>> Erienne Olesh: That's one other aspect that we really want to eventually get into is trying to
see if we can expand, maybe not this algorithm in particular, but an algorithm like it to capture
a broader range of neurological injuries that create, or cause motor impairments. As far as the
clinical issues, some of the concerns and things right off the bat we had to address with putting
this in a clinic was, first of all you're going to have to be very specific as to where you set up the
Kinect and where you instruct the patient to stand, because we know there's kind of this sweet
spot with the Kinect. If you're to close it doesn't work; if you're too far away it doesn't work. If
you're moving a lot side to side, it may not work as well. Something else we have to think
about is what if we have an individual who comes in with a wheelchair. Is that now going to
interfere with the recording of the Kinect or will it be able to pick it up as well? We still have
some room to improve in some different tests that we need to look at and patient populations
that we need to look at.
>>: So the 17 trials of the Wolf tests do not include the legs at all?
>> Erienne Olesh: No. It's all upper body function.
>>: That makes it easy.
>> Erienne Olesh: That does make it easy. When you look at the upper body versus the lower
body, obviously gait is nice to study because it's, in a healthy individual it's very symmetric,
whereas, the upper body you kind of have many more degrees of freedom you can kind of
explore with your arms. On the other hand, when you are just looking at upper body, you can
have somebody be fairly stationary to assess their motor function of their upper body.
Whereas, if you are looking at lower leg motor function you need to kind of have them engage
in a larger area of workspace.
>>: That's what I'm pointing at, it's easier, the Kinect is targeting, I mean it would be easier to
look at the upper body, but if you try to look at lower body movements, then you run into
[indiscernible] limitation. What I'm essentially asking is whether this, is this going to limit you in
your therapy that you use for home-based [indiscernible]
>> Erienne Olesh: As of the way the device and the algorithm sits right now, yes. We are
limited to looking just at upper body function. I think there is absolutely room to improve on
that both on the technological and hardware and software sides of things, as well as the
algorithm that we are using to look at upper body function may not be the best algorithm to
look now at lower body function. I think we need to kind of expand some of our research ideas
to look at that. If we really want to integrate this in a home, it would make more sense if it
could assess both lower and upper body function at once, not just one or the other.
>>: Are you looking to collaborate with computer scientists?
>> Erienne Olesh: Yes, always. My PhD, in particular, is in neuroscience. I actually have a
pretty strong background in physical therapy. My P.I. doctor, Doctor Gritsenko and our
collaborator, Sergiy have been doing a lot of the programming to take some of the SDK
packages and develop them into these stand-alone systems, but as scientists you always run
into this problem of you want to do a lot of things, but certain things take a lot of time. If we
had somebody who maybe was a little more geared on computer science side of things, we
could probably develop and then test a lot of these algorithms a lot faster. Time constraints,
everything is time and money. Yes?
>>: You said you were going to have this set up in a clinic. Are you going to have a certain
room for that or are you just going to have it in the waiting room? And what do you expect the
challenges to be?
>> Erienne Olesh: Our idea is for the set up to basically have this in a separate waiting room
area where just one patient would go in anytime, and then you are going to eliminate a lot of
the issues with people walking behind and opening doors appear the idea is that, kind of the
perceptual view we have right now is that the patient would show up. They would check in. A
nurse would come and get them and escort them to this room. They would do the assessment.
The nurse could either sit and wait in the room while they were doing it to make sure that there
were no technical difficulties, and then as the patient was waiting for the clinician to have their
actual physical therapy appointment the scores would kind of be sent or printed out so that the
clinician could have those prior to seeing the patient.
>>: Sorry, just to follow-up, just for usability, do you plan to have any survey questions or
interview questions or have somebody observed them to see what their experience was like
and if they find it intrusive or weird or…
>> Erienne Olesh: Yeah. We need to do a lot of investigation into how patients are going to
perceive this, but also what we would like to do is kind of expand it so this could also be kind of
a surveys and motor assessment system, so that when they show up they could have questions
and answers for what is your average pain on a daily basis. Are you having trouble navigating
through your home? Have you fallen recently? Those kinds of things, so that we can kind of
build in some extra information into this generalized assessment.
>>: I wasn't quite sure of this algorithm that you used to create the score from the Kinect.
They came and you did this Wolf extremity battery and that's got a score through it, but then
you said you had this algorithm that the Kinect did the same score for how did, what was the
algorithm that the Kinect, the automatic, how did it arrive at its…
>> Erienne Olesh: Right. Okay. What we do is we have the physical therapy students rate our
subjects. Once we collected kinematic data from the Kinect -- I'm going the wrong way -- we
calculate joint angles and then we use that information and extract the principal components
for it. Right now this is all done kind of post analysis in Matlab. That's one of the other areas
we need to kind of focus on is to make this, make all of those analysis scripts automated into
the eventual software that will work with the hardware.
>>: Can you describe the, I didn't quite understand the PCA. So you going to [indiscernible]
PCA on shoulder abduction and shoulder flexion extension, so 4 degrees of freedom
[indiscernible]? And then you're pulling out two full components?
>> Erienne Olesh: Yeah. Because if we pulled out for principal components, we would probably
get a exact four angles that we were getting from the unaffected arm, so we're looking for the
two principal components that are explaining the most amount of variance in all four angles.
>>: And then, I guess the [indiscernible] is the rationale for then using those. I didn't quite
understand the leap then to applying correct those to the [indiscernible] arm and what that
shows and how it means something.
>> Erienne Olesh: When we extract the principal components we're assuming that these two
principal components and based basically on the mathematics used to do this decomposition
analysis, that these two principal components are representing the most amount of variance
from those for joint angles. When we take those two components and then we reconstruct
affected arm data, we want to see if there is a high correlation between the reconstructed data
in the original recorded data. Then we know that the two limbs were moving fairly similar.
>>: So it's kind of like your deviation?
>> Erienne Olesh: Yeah. And if we can't reconstruct the data well for the paretic arm then we
know that there was a large difference in motor function between the two limbs because the
variance that is explained for the unaffected limb is no longer explained in the variance for the
affected limb.
>>: And then the error is what you calculate?
>> Erienne Olesh: To get the score, yeah.
>>: This seems there is an easier way to compute the [indiscernible]
>>: So basically what you are trying to measures the amount of symmetry between the two
arms, right?
>> Erienne Olesh: Yes.
>>: So it's not as if you are saying there is a right way to perform this act. You are just saying…
>> Erienne Olesh: Correct. We're looking at the similarity of movement between the two
limbs.
>>: So it's like a database.
>> Erienne Olesh: Yes.
>>: [indiscernible] everyone's going to do it slightly differently and so you compare it to there
[indiscernible]?
>> Erienne Olesh: Yes, to the unaffected arm.
>>: You mentioned in your talk when you mentioned accuracy of angle compared to that.
What would be good accuracy?
>> Erienne Olesh: Zero percent difference, I guess. [laughter].
>>: That's easy [laughter]
>> Erienne Olesh: Again, I think maybe, I mean, obviously if you're having on average of 50
percent difference between a state-of-the-art system and a low-cost system like the Kinect,
that's going to raise a lot of red flags as to how well your motion capture’s actually working. I
think what maybe is though more important than the percentage is how consistent that
difference is. That really may speak more towards how the points of rotation are being actually
calculated by the Kinect and the actual skeleton fit of the Kinect. As long as that difference
between phasespace and Kinect joint angles is pretty consistent, then it's just basically a built-in
difference in how those joint angles are being calculated.
>>: Is it fair to say that you experience more of a bias than a variance? So it's kind of a bias
[indiscernible].
>> Erienne Olesh: Yes. That would be a good way to describe it. Uh-huh?
>>: So what quantitative measurements did you get the Kinect, using Kinect?
>> Erienne Olesh: The quantitative measures are these, so basically this quantitative score.
Since we're using correlation coefficients, it's a 0 to 1, basically an R squared value that we're
getting. When we're comparing those two reconstructed on original joint angles.
>> Ran Gilad-Bachrach: So let's thank Erienne again for the talk. [applause]