>> Ran Gilad-Bachrach: Thank you. So, really, thank you so much for coming over. I know
this was kind of a last-minute thing, and people came some from them even from out of the
States, and some of them even had to cross a lake to get here, and I appreciate that.
Traditionally, I should have been speaking about why is it important and so on, but I figure this
is just preaching to the choir, so I'll skip on that. We all agree it's important, so let's move on.
We have an interesting set of things to cover today. We're going to look at gait and balance from
different perspectives, and we have both about tools, about rehabilitation, usability, other cures,
and I'm very excited about that, because it sounds as if it's a small thing, but actually, there are so
many different perspectives to look at it and so many different challenges. And the people that
we have here today come from a really diverse set of disciplines, so we have a large
representation of physical therapists, but also people from engineering, computer science,
medicine, business, biomechanics. And the program, I think you've seen it. We had just a lastminute, small change, that the talk about Kinect will be at 1:45, and the talk about Parkinson's,
measurements for Parkinson's, will be at 3:30. If you have any questions or problems during the
day, please don't hesitate to ask me, and I'll help with any issue we might have. I'll just use the
couple minutes that I have to tell you a little bit about the kind of thing that we do here and my
motivation. Before I joined Microsoft Research, I used to work in Bing, and over there, we ran
experiments all the time. When you look at the Google, Facebook, all these guys, they run tons
of experiments, all the time. They don't change the size of the font in their page without running
a trial on that. The thing is that they can run trials really, really fast. They have millions of
people, and all of us are participating in these trials without even knowing. They just take a
small percentage of their visitors and send them to a slightly modified experience, and they get
the numbers. Once you have -- imagine, having a trial with N being 1 million. That's kind of
nice, right, compared to the trials that we are doing. And the question is, can we bring that to
kind of the fields of healthcare, wellbeing, medicine, because what it gives is there's a huge
acceleration to the innovation, to the rate of innovation that we can make. And one of the
bottlenecks in doing that is being able to run the measurements, because if we want to do
something -- for example, if we're talking about eldercare and we want to help people have better
balance, reduced risk of fall and so on and so forth, currently, we have to bring them to the lab
and do all sorts of measurements, and that limits how many people we can have in a trial. If we
can somehow bring the trial to the house, have the measurements being done all the time or
every day, that will allow us to increase the rate of innovation and, at the end of the day, bring
better care for people. So now technology allows us to do or at least try to do these kind of
things. We can use wearables, and there is a lot of work on that, and I think we're going to hear
some of it later today. I've been trying to look at the Kinect as an alternative, as an option, to do
that. It has some advantages and some disadvantages. One of the big advantages, that kind of
high-quality install and forget. You don't have to charge the batteries. You don't have to wear it.
You don't have to remember to do it. You don't have to wear it correctly. At the same time, it
has lower coverage, in the sense that it doesn't see you all the time, but we're trying to see what's
the pros and cons and how we can go about it. Let me -- in the interest of time, I'll just do a
quick demo, I hope it's going to work, of something that we are currently working on, so it's a
work in progress. It's not -- so, first of all, if you haven't seen the Kinect, this is actually the
older version of the Kinect. Over there, if you see just from the ceiling over here, this is a new
Kinect. You can have a look at it. I'm going to use the old Kinect for now. The Kinect is just -let me show you what it sees. It's basically just a camera like the webcam that you have and also
a depth camera. So if I'll stand over here, the colors represent how far away am I from the
sensor, and it uses an algorithm to extract the skeleton. So this is not an X-ray, so it doesn't
really see my skeleton. It kind of infers where the joints are, and this is very nice, so you have
all of that, but at the same time, we have to remember that it has limited accuracy, so we have to
take it into account whenever we use it. But one thing that we can do with it is, for example,
take the test that we do in standard tests that we do for gait and balance and digitize them. So,
for example, this one is sit to stand, so I'll try to demonstrate that. So what you see here, it
captured the height of my head, basically, and as I stand up and sit down, it just records that, and
if I do it multiple times, I feel a little silly, but this is what we are having people do. So I think
that will be enough. Now, I can ask it to analyze that. And what you see here is kind of the
results, and basically what we can do, we can get a much more detailed analysis of this exercise.
So as opposed to the standard way, in which we just measure how much time it takes to do five
repetitions of sit to stand, we get that, but we can also look, for example, how much time do I
spend sitting in this process? How much time do I spend standing up, between sitting and
standing and the different parts of it? And I can also look at the variation between the different
repetitions of this exercise, and on the kind of derivative, in a sense, the fatigue. If I repeat that
20 times, do I keep doing it at the same time rate, or do I get tired, as I do that. I can do the same
thing for gait. Let me try to do that quickly. So here I'm going to walk in place. So you can see
my knees, the two knees, the blue and the red line. Again, for the interest of time -- oh, I didn't
start it. But, anyway, I can get the same kind of analysis. I can look at how much time is spent
in double support, how much time I swing each one of the legs and all the statistics about it, so
am I becoming slower over time and so on and so forth. So we can do these kind of things but
with equipment that can be at everyone's home, or definitely at every clinic. So I think it brings
a lot of opportunities. Also, I have a lot of challenges. It's not clear. For example, one of the
challenges, if we want to look at things like angles of knee angles and stuff like that, that's not
trivial. This is one of the things that we're looking at with some of the people here in the
audience on how to do that. But I'm very interested to hear from all the people that are here
today about what should we look at and how should we do it and all sorts of lessons learned,
both on the technology side, from physical therapy and from usability. There is another
important aspect, which is do people want to use it? So let's get started. It won't allow me to do
that. So our first session for this morning -- sorry, I'll have to skip all these slides. We have Kat
Steele from the University of Washington that will talk about new tools and techniques for
clinical gait analysis, and then Eric Horvitz, who is the head of this lab, will welcome us. Eric is
both an expert in artificial intelligence, but actually, by training, he's a physician. So he's going
to have an interesting take about that. And then we have Shomir Chaudhuri talking about the
perceived usability of detection device for older adults, so I think that's pretty exciting, this
session for this morning. So do you want to take it from here?
>> Kat Steele: Should we go ahead and get started? I don't know about the rest of you all, but I
just want to play with the Kinect now for the rest of the morning. But I'm excited to be here. I'm
going to give you a little bit of a background on clinical gait analysis, how we use it, some of the
new tools and techniques. But before we begin, I'll just give you a little bit of background on me
and our group at the University of Washington. There's a lot of friendly faces here from the
University of Washington today. So I am in the Department of Mechanical Engineering, a new
assistant professor, and we have the Ability Lab over in Mechanical Engineering, and our real
focus is figuring out how we can empower human mobility through engineering, human-centered
design and engineering. And so we have a lot of different projects going on, some involving
imaging, ultrasound, MRI, and then a lot of our work does focus on characterizing human
movement using clinical motion analysis, as well as musculoskeletal modeling and simulation,
which I'll tell you more about today. My background, even though I am trained as a mechanical
engineer, I've spent a lot of time working in hospitals, working in the clinical gait analysis labs at
both Denver Children's Hospital and Lucile Packard Children's Hospital, and then most recently
for the last year, along with Nupur, I was at the Rehabilitation Institute of Chicago, as well. So
we're all kind of in this space where we're melding engineering and medicine and new
technology and design, and so I think it's awesome that we have so many people from different
perspectives interested in these topics, because that's the only way we can actually get things
translated into not only the clinic, but as Ran was saying, into the home, to hopefully improve
mobility. So as a mechanical engineer, I like to boast to my colleagues that I get to work with
the ultimate machine. It's a little corny, but we often talk about Big Dog as this amazing
machine. If you haven't seen it, go check it out on YouTube. I think they have like Big Cat now.
I forget what they actually called it, but some pretty amazing machines. But our machine has
even more degrees of freedom and more actuators. These antiquated motors, they just can't keep
up with our types of motors that can repair themselves, respond to the loads, really simplified
control systems we're dealing with in this technology and really challenging fuel sources. So, of
course, I see the ultimate machine as the human body, both as an engineer, as a source of
inspiration, but also as a really challenging design system to work with. Our ultimate motors are
muscles. Well, if you put more load on them, they'll get bigger. They can even repair
themselves over time. If we had an actual actuator that could respond as well as muscle, it would
transform a lot of engineering. And, of course, our control system, the wonderful central and
peripheral nerve system, and my favorite part is that we can power this whole system with a
really diverse array of fuel services. And so both at the Ability Lab and within what I'm going to
be talking about today, the challenges that this ultimate machine gives us a lot of flexibility and
versatility to pursue the diverse array of activities during daily life. However this same
complexity that makes our system and our machine so versatile also unfortunately makes it really
difficult to treat when things go wrong, whether due to age, injury, or what I'll be focusing on
today, neurological disorders, figuring out the exact mechanisms and appropriate treatments to
improve hindrances and performance with the ultimate machine is really a challenging problem
and why we need all the different brains in this room and beyond. So just a little bit of
background on what we're up to in the Ability Lab, and then I'll focus in on some clinical gait
analysis. So I like to divide it, once again, with our ultimate machine into three different areas.
First of all, we have our machine diagnostics. We spend a lot of time trying to figure out how do
we move, what are the underlying mechanisms, the neuromuscular control, the musculoskeletal
dynamics that are required for movement. In these individuals with neurological disorders, what
are those changes in control? How can we quantify? Every brain injury is unique, and how can
we quantify patient-specific changes in neuromuscular control? And also, there's a lot of
changes and adaptations that occur within the musculoskeletal system, so we use MRI and
ultrasound to provide some quantitative measures of patient-specific, subject-specific changes in
muscle property, such as contracture, spasticity, a lot of secondary disorders that develop after
brain injury. Those are our machine diagnostics. Probably more interesting to the rehab
medicine group in here is also our machine repair, so once the system breaks down, what are our
best options and patient-specific options to improve treatment, and we look a lot working with
orthopedic surgeons to figure out the best combination of treatments for individual patients, and
I'll actually give some examples of this today. Muscles are lengthened, transferred, bones are
rotated and realigned. Noninvasive treatments, we have quite a few programs looking at strength
training, especially in cerebral palsy and stroke, and then one of my favorites, optimizing
orthotic design, if you're familiar with ankle-foot orthoses. How can we customize the properties
of these orthoses to individual patients, and also how can we use new rapid prototyping
techniques, 3D printing, 3D scanning, to improve those? And, of course, with any good
machine, we also need our preventative maintenance, so we spend some time looking at how can
we avoid pain and degeneration, looking at cartilage thickness and how we can improve longterm mobility for individuals. And this all kind of relates to something that I think we'll be
talking a lot about today, what we've been referring to as ubiquitous rehabilitation, the idea that
we put enormous loads on our body every day, and there's a few physical therapists in the room,
but there's a reason why they often call PT pain and torture. It's not something -- and compliance
is a big issue, so how can we use these loads that we already are putting on our body every day
and all this new technology that we'll be seeing a lot to day to enable people to kind of
unconsciously do their rehabilitation and integrate it better into their daily life. So that's what
we're up to in the Ability Lab, and I'll be more than happy to chat with all of you later about any
of these things if any of them caught your interest. But today, I'm going to really kind of talk
about some of the things in our toolbox. In particular, we use clinical motion analysis,
musculoskeletal modeling and simulation and medical imaging in order to understand the
underlying pathology in individuals with neurological disorders. And, as engineers, this is really
built upon our fundamental principles as dynamics controls and biomechanics. And Ran today
really asked me to focus in on this kind of this clinical motion analysis side. What are the factors
and the tools that we use to analyze human movement, human gait and to work with clinicians
and therapists to improve treatment decisionmaking? So clinical motion analysis. If you're not
familiar with it, a large majority of Children's Hospitals now have clinical motion analysis
laboratories, where patients come in and both before and after treatment, will undergo a series of
tests to characterize and quantify how they move. A lot of other clinical centers, such as our own
VA hospital here in Seattle, also have motion analysis laboratories, with these goals of basically
quantifying movement disorders so that we can hopefully understand the underlying anatomic,
physiologic and functional causes of altered movement. Now, if you haven't seen it before,
here's just a few quick examples. One of the ones that we most commonly think of in motion
analysis is gait analysis, where individual patients come in, and this is from Gillette Children's
Specialty Healthcare in Minneapolis, and they walk back and forth across the lab, and they'll
wear those reflective markers that you see, and there's embedded force plates and
electromyography, and we characterize and quantify how they walk and compare it to
unimpaired individuals, so that we can hopefully identify patient-specific changes in movement.
Because you have to remember, every brain injury is unique, every patient is unique, and so part
of the challenge is figuring out the optimal set of treatments for each individual. So gait analysis
is the large majority of what a lot of these laboratories do, but they have extended their kind of
reach of services lately. This is an example from Lucile Packard Children's Hospital of some
upper extremity analysis, where they're analyzing reach and grasp for a child with cerebral palsy.
Other labs have also been starting to get into some of the sports area, especially with all the
shoulder injuries among Little League pitchers, characterizing throwing and pitching motions,
the few that do golf, and so there's a really wide diversity of activities and motions of daily living
that individuals in clinics examine in these laboratories. Now, these laboratories have been
around for decades. They're a fantastic resource. They've advanced a lot over the past few
decades in terms of allowing us to quantify and understand pathologic movement, but today I'm
going to focus in mostly on some of the challenges. And one of the biggest challenges with
these facilities is what I often refer to as the deluge of data. From these analyses, you get a wide
array of data. Some of the most basic are your kinematics, your joint angles, your kinetics, your
moments and powers, your electromyography, your muscle activity, so this is an example from
Gillette Children's Specialty Healthcare again, where you can see those joint angles, those joint
moments, those joint powers, overlaid with electromyography or muscle activity. And so
doctors are given this array of information and need to determine, well, here in their ankle
moment, we see they have an early moment, and they're always in plantar flexion. This is a
patient that's always walking on their toes. What are our best treatment options? But in addition
to this data, they also receive a full physical exam. They often participate in some metabolic
tests, whether heart rate or oxygen consumption. Sometimes, there's plantar pressure
measurements, sometimes there's balance measurements where they're standing on the force
plates. And so there's this huge slew of tests that thankfully produce a lot of quantitative data
now, but a big challenge is then figuring out how do you take all of this data and turn it into
actual treatment decisions? And that really comes down to the second big challenge of gait
analysis, that we're really trying to aim for patient-specific treatment, how can we improve
treatment for individual patients? And so some examples of the treatment options that they have
for these individuals are, first of all, one of the most common in these laboratories is multilevel
orthopedic surgery. So a large majority of children with cerebral palsy will receive multilevel
orthopedic surgery. In these surgeries, they do everything from lengthening tendons to relieve
some muscle contracture, transferring muscles to new locations. One of the most common
examples is your rectus femoris, the muscle on the front of your thigh, they'll physically remove
it from the front of the knee and move it to the back of the knee to try to remove some of the
excess knee extension. And then also since bone deformities are also common, there's a lot of
osteotomies and other procedures to realign bones. So these are some of the treatments on the
very invasive side of the spectrum. Other ones include selective dorsal rhizotomy, where they
actually will go in and cut some of the afferent nerves to try to reduce spasticity and some of the
excess muscle activity. There's also from the rehab side quite a few therapeutic treatments.
Most individuals receive some type of physical therapy, whether it's stretching. We have a
picture of a pediatric loco mat over there, where they can do some gait training. Electrical
stimulation, where they try to target individual muscles to improve either the gait pattern or
through therapy. And also quite a few neuromuscular toxins, so botulism toxin is common in
cerebral palsy and stroke, as well, to decrease muscle activity -- others, such as phenol and
baclofen. So you can see there's a deluge of data, and there's also a wide array of treatment
options. And so kind of the challenge that we're setting up here is that the clinicians have to
decide both what the specific causes of the pathologic movement are for each patient and then
kind of transpose that over their treatment options or their toolbox and identify the best
treatments. We could probably put a whole other level of future treatments along the bottom
here, and so there's always new things to be trying, but it's a very challenging space to work with.
And the bad news is that, unfortunately, we're not so great right now at making these treatment
decisions. This is a recent example that we just made with one of our collaborators. This is
looking at their gait or their walking pattern before and after a variety of treatments. So on the
X-axis here, we have their gait deviation index before treatment, so 100 is a normal gait pattern,
and then every 10-point increment is one standard deviation away from normal with the gait
deviation index. And so we're aiming to get everyone up to 100, and so this is where they start,
and then these were all subjects who had had two gait analyses separated by a year or two. And
you can see here their change in GDI. And so if you start off by looking at the no-treatment
option, that's the black line, you can see that, in general, even some of the more severe
individuals will get slightly better, but the no-treatment line is pretty stable, as what we would
expect. But the disappointing part is that after botulism toxin, single-level orthopedic surgery,
SEMLs, which is single-event multilevel orthopedic surgery or that selective dorsal rhizotomy
that I mentioned to you, where if you look at here's our zero line, unfortunately, we kind of want
to push this whole curve up, right? We want to be able to more consistently improve gait
outcomes. Now, you could argue that even an outcome of zero is good, because we know that
deterioration over time is common in these individuals, but a lot of our research and our goals is
to push this up, so that hopefully we can help empower clinicians and patients to have more
consistent positive changes in their gait pattern. And sometimes, you might say, well, we have
the Ministry of Funny Walks, if we think back to some of the old comedies, so maybe we should
just embrace the different walking patterns. But, unfortunately, a lot of these gait patterns do
contribute to bone deformities and pain over time, and so they can decrease mobility as they age.
And so we know that joint pain is a big problem, especially among adults with cerebral palsy and
that over 60% of individuals with cerebral palsy experience a decline in walking function by the
age of 40, so if they were walking unassisted before, they're now using crutches or other assistive
devices. If they were using crutches or assistive devices, they're now using a wheelchair often
for their primary mobility. So it's a challenging space to work in, but hopefully we have new
tools and new knowledge to help improve it. And another kind of tool that I'm going to
introduce that we have to improve this patient-specific treatment are also some musculoskeletal
modeling and simulation tools. So if we look at the current motion analysis environment, there's
kind of two big pieces of equipment that we use a lot. The first are these traditional motioncapture camera systems, where you have these reflective markers. You might have seen them
for video games or movies, as well, that they stick all over the body, and that gives us the 3D
position and trajectory of bodies through space. We also of course have force plates, which are
actually one of our most accurate sources of data. They can give us the external forces acting on
the world, from which we can back out all those joint moments, powers and a lot more
information about movement. But if we think about -- so these give us good ways to measure
position and force, and thankfully we can differentiate these to get our velocities, our
accelerations. We have a lot of equations of motion and dynamics for the human body, so we
can start to understand some of those multi-body dynamics, but to get beyond that point, we
actually need to know a lot more about the musculoskeletal and the neuromuscular system, so
with these tools, we're kind of limited in how far we can get back in this pipeline to understand
the underlying mechanisms and the impairments in muscle and nervous system that are actually
impairing motion. And so to step back into some of these more muscle-specific and neuralspecific factors, we need some other tools that will allow us to probe into some of the changes in
the musculoskeletal -- in our complex ultimate machine. So one of the tools that we now have
available is musculoskeletal modeling and simulation. This gives us basically there's models that
have been developed that are based upon cadaver and other data where we can really accurately
characterize the musculoskeletal geometry, the inertial properties and create full, dynamic
simulations of individual subjects' motion. So that lets us take our data on our positions and
forces from a clinical motion analysis laboratory, apply it to patient-specific models and then
work back to understand some of these changes in skeletal dynamics, what happens if you have a
bone deformity, femoral anteversion or tibial torsion? What if your muscle moment arms are off
or your muscle properties are different? What if you have a contracted muscle or a spastic
muscle? We can add those muscle models into the model. Or what happens if you want to
transfer a muscle to a new location? And then also probably still the most difficult part of this
equation is modeling neural control, and thankfully, there's also some new methods that we've
been helping develop with others so that we can start to understand how we can model
neuromuscular control and piece back some of this information to identify patient-specific
mechanisms that are contributing to pathologic movement. And so I'm going to give you some
examples today of how we can use musculoskeletal modeling and simulation to probe some of
these patient-specific factors and build upon that foundation from clinical gait analysis. So some
of the advantages of musculoskeletal modeling and simulation, first of all, it's really just a nice
tool to visualize complex movements. I don't know about you all, but I'm a very visual person,
and so when you can actually see where a muscle goes, how it actually accelerates the body -we're a connection of linked segments, so if you generate an acceleration at one joint, you're
going to generate accelerations all the way up the chain, and that can be really difficult to
visualize. And so this is especially nice for researchers, clinicians, students, as a fantastic way to
visualize this complex machine. We can also probe parameters that are difficult to measure. I
don't know how many of you would like to volunteer for me to measure your muscle force or
your contact between your bones, but I'm guessing probably not too many, but since this is a full
dynamic simulation, we can get estimates of a lot of those underlying parameters, how much
muscle force, what is your tibial-femoral contact force, that force at your knee. We can start
identifying some of those cause-effect relationships. Can the soleus or the gastrocnemius
contribute to crouch gait? Can a tight hamstring cause you to have excess knee flexion? And
you can start identifying the underlying mechanisms of patient-specific pathologic movement.
And then my favorite probably is that we can start performing what-if studies. We can only do
these surgeries often once, and they're irreversible, but you can start doing some of these things - well, what if we transfer a tendon to this location? What if we modeled their neural control in
this manner? And so you can basically kind of do hypothesis testing in silico, which gives you a
powerful platform to evaluate the system. So one example now of how this has been applied in
the clinic, so if we return to our example here of a young adolescent male with cerebral palsy
who's walking in a crouch gait, and this is one of the most common gait patterns among
individuals with cerebral palsy, one of the places where musculoskeletal modeling has been
integrated the most into the clinical motion analysis laboratory has been evaluating hamstring
length among these individuals with crouch gait. So if we look at a muscle like your hamstring,
it actually crosses two joints. It attaches to your pelvis, so it helps to extend your hip, and then it
goes all the way below your knee, so it also helps to flex your knee. So by just watching
someone's motion, it's difficult to actually tell how long that muscle is, how fast it's lengthening,
and if contracture or spasticity could be contributing to this crouch gait pattern. And it's often
theorized that a tight hamstring muscle is contributing to the excess knee flexion among these
individuals, and so some of them will receive hamstring lengthening surgery in order to try to
alleviate that. But once again, there's these inconsistent outcomes, and so hamstring length from
musculoskeletal modeling has been used in order to help kind of allow physicians to actually
probe and say, is the hamstring length of this individual short and slow? So to give a quick
example of that, so you'd have the kinematic, kinetic data from a motion analysis laboratory. In
this case, you just really need the joint angles. You have a scaled musculoskeletal model, which
includes the hamstring, and from that you can actually calculate the hamstring length and
velocity for a given patient's movement. So this is showing here what the hamstring length is
during normal unimpaired walking. And so then if we had a specific patient who was walking in
a crouch gait pattern, we can calculate their hamstring length, and you can see that right here,
when your heel strikes the ground in early stance, that's actually when your hamstring's at the
longest. Your hip is extended or is flexed, and your knee is extended, and that's really where you
need that maximal length. And so that's the point where we can look at it and say, do they have
adequate length in order to achieve an upright extended gait posture, and are they a good
candidate for this hamstring lengthening surgery. So, in this case, this patient does have a
shorter hamstring length and so probably would be a good candidate for surgery, while you
might have other subjects who, looking from a clinician's standpoint, if you just were to watch
them walk in the lab, probably look like they walk pretty similar. But if you actually plug in
their patient-specific kinematics, you might find that for the second patient, their hamstring is
actually operating at a near-normal length and velocity, and so maybe they would not be good
candidates for the hamstring-lengthening surgery. And so Allison Arnold and some others have
used this extensively and have integrated it into the clinical motion analysis laboratory and have
shown that calculating hamstring length and velocity can improve our ability to predict who are
good candidates for this surgery and also who are bad candidates. Because she also found that
among those subjects whose hamstrings were not short -- we call it short, slow and stupid -basically, they're on at the wrong time, as well. That for those subjects where that's not true
before surgery that they are actually contraindicated for this procedure, because afterwards, they
often would have excess anterior pelvic tilt because you've overlengthened hamstrings that didn't
really need to be lengthened. And so it provides a powerful tool to help clinicians make better
decisions, more informed decisions about muscle-specific and patient-specific outcomes. So
that's probably one example of where we've been able to integrate and probably the one that's
been integrated the most into clinical motion analysis laboratories. Some other examples from
some of our research is another case here, where we were looking at should the ankle plantar
flexors be surgically lengthened or targeted for strength training? And so if you look at your
ankle plantar flexors, you've got two muscles, and these are some of your most powerful muscles
on the back of your calf here, but they're complex, because they both attach through your
Achilles tendon, but your soleus muscle attaches to your tibia, and so it only acts to plantar flex
your angle, while your gastroc attaches above your knee, so it both flexes your knee and points
your toe, plantar flexes your ankle. And when you look at it, there's a wide variety of places
where surgeons will commonly target the gastroc-soleus, and you can see some of them, like A,
B, and C, will only impact the gastroc, while others that address more the Achilles tendon, like
D, E and F, will impact both muscles. And a lot of surgeons would prefer these, because you can
do it percutaneously, through the skin. It's a much easier procedure, while A, B and C require a
more invasive procedure. But here, you hit the soleus and the gastroc, since you're targeting the
Achilles, while here you can isolate just the gastroc, so is that important and worth the more
invasive surgery? So just like we were able to look at the length and velocity of muscles, using
simulations, we can also look at how individual muscles accelerate the body and how they
contribute to pathologic motion. And I won't go too much into the math here with this audience,
but it's all the equations of motion, force equals mass time acceleration, Newton's second law,
just this expanded. And thanks to knowing the dynamics of the system, we can calculate how
individual muscles accelerate the body so that you can anticipate and understand how that role
of, for example, the gastrocnemius and the soleus differ in accelerating the body, and so we can
calculate the potential of a muscle or what their acceleration per unit force would be, which is
only dependent upon their posture, your contact, and then if you have estimates from your
simulation of muscle force, you can actually look at individual contributions. So if we wanted to
look at some examples of this, if you're curious how you use some of your own muscles to walk,
when we look at unimpaired walking, these are the actions of our vastii, the quadriceps on the
front of your thigh, and the gastrocnemius, that big muscle on the back of your calf. And these
plots are showing over one gait cycle, so from when your heel strikes the ground to when your
heel strikes the ground again how these muscles are accelerating the body. And so it's really
quite beautiful in normal walking. You can think of each of these lines as an arrow pointing in
the direction that it's accelerating the body, whether it's supporting, breaking, propelling the
body. And so in unimpaired walking, the vastii supports you and brakes you in early stance so
that we're not constantly accelerating, and then they kind of ramp off. And then as they ramp
off, your gastroc ramps on to support and propel you. So you have this really nice where you're
using muscles for multiple purposes, to both support and propel your body, and it's a very
metabolically efficient motion, thankfully, when we look at unimpaired walking. But if we were
to compare that to our kids with crouch gait, we we'd see that for mild crouch, moderate crouch
and a severe crouch gait that this kind of beautiful synchrony starts to break down and
contributes to some of the inefficiencies of these gait patterns. So if we look over here at the
very severe crouch gait pattern, you can see that that vastii has to be on throughout the whole
gait pattern, which you can kind of imagine if you're down here, you're having to use those
quadriceps a lot. But that also means that you're braking the body throughout a crouch gait
pattern. And meanwhile, the gastroc is also on, helping to support the body throughout stance,
and so you kind of have now these two muscles, fighting against each other and contributing to
some of the inefficiencies of these pathologic gait patterns. So using these tools, we had looked
at this question of lengthening the ankle plantar flexors, and we saw that both of these muscles
really were important muscles for supporting the body and propelling the body forward, during
both unimpaired walking and crouch gait. But we can also look at how they accelerate
individual joints using the same equations of motion. And you can see that although they both
act to point the toes, plantar flex the ankle, they do have opposite effects at the knee. And so
looking at these results of muscle-specific actions with pathologic gait, we really saw that first of
all, the soleus is a critical crouch-countering muscle. It helps to support and propel the body and
extend all three of your lower-limb joints, because remember, we're a coupled system, dynamic
coupling. So even though this muscle only crosses the ankle, it's still one of the primary muscles
we also use to accelerate our knee and our hip into extension. Meanwhile, the gastroc presents a
little bit more of a tradeoff, where we saw that it's still important for supporting and propelling
our body, but it can contribute to some of this excess knee and hip flexion that you might see
during a crouch gait pattern. And so these are the kind of ways that we can probe the underlying
system. And now a quick example also, my favorite part of testing hypotheses in silico, so we
wanted to look a little further at how the gastroc could be contributing to crouch gait, because
some of our other research results had also suggested that if you were -- because we can also
model muscle weakness. So, for example, we can weaken muscles and see how it impacts the
gait pattern. And so one of our theories was that a weak gastrocnemius could be contributing to
a crouch gait pattern, similar to how we saw that in the last example of where you might want to
lengthen the muscle. But we were curious, what happens if you have a weak gastrocnemius. So
this is in collaboration with Jack Wang, who did this work. And so here you can see a
simulation of normal walking that he created, and it's -- in this case, we're not inputting motion
analysis data. It's just been given an objective function, largely minimizing metabolic energy,
and it can produce a pretty normal-looking walking function, and its actuators are muscles based
upon the same type that you saw in the previous models. And so we said, well, what happens if
the only change we make to the system is to weaken that gastrocnemius muscle. And so when
he went through and weakened the gastrocnemius muscle, you can see that this is the gait pattern
that resulted. It looks pretty much like one of our kids who's walking in a crouch gait. And so
just making that one change to the system provides further evidence that perhaps the gastroc can
be contributing to a crouch gait, and particularly that maybe strength training this muscle may
lead to more consistent outcomes after strength training for individuals with crouch gait.
Because right now, after crouch gait, they typically target the knee extensors and the hip
extensors, but some of our other research suggested those probably aren't the muscles that are
weak in these individuals and need to be targeted. So that was just a quick summary and kind of
peak into how we can use these new tools in musculoskeletal modeling and simulation in order
to probe the underlying factors and really start to understand the patient-specific factors that are
contributing to pathologic movement. It host a lot of advantages, but I think also one of the
biggest advantages is that most of this work is all done on an open-source platform, and so this
really gives us a common platform to both accelerate research and development but also to help
facilitate clinical translation. And so if you want to try it out yourself, it's called OpenSim. It's
an open-source musculoskeletal modeling and simulation software. You can download it here.
If you need money, they also have fantastic pilot projects and visiting scholars' programs, but it's
really become a nice platform where now you don't have one PhD student developing a whole
set of simulation software, and then he graduates and you have to redevelop it all or have
someone else go into the code, and you have this whole community who can accelerate our
understanding of pathologic movement, develop new tools and develop new methods. If you
were to go online and check it out, you'd see that there's quite a few different models that are
available that different groups have developed. From Berkeley, a group in Italy -- there's also a
lot of simulations. All of our simulations of crouch data are online. Others have done running.
If you're into animals, there's also T-rex models and several other animal models, as well. So it's
a great resource, and it's really fantastic for our community that we have that. And it has turned
into quite a community, too. So I was just curious and went online to look at the map this last
morning, and you can see the recent downloads of OpenSim over the last 180 days. So I think
this is a really promising way that we can start diving into some of our patient-specific
treatments. And it gives a lot of power, but it still restricts us to the laboratory environment.
We're still using this motion analysis data that comes from these highly controlled environments,
and there's quite a few limitations to our current labs. One is cost. These systems are on the
order of $100,000 for all the equipment we need, which makes it prohibitive not only for
insurance reimbursement, but also just for constructing these labs. Seattle Children's is one of
the few hospitals that doesn't have it. I think they should be the first Kinect-based lab, but we'll
see what happens. And also, it kind of assumes that we live in a linear world. If you look at all
these, we have people walking back and forth, and if we talk to Val Kelly over here and others,
they tell us that -- or Kevin McQuade, we turn a lot during the day -- in fact, probably the
majority of the day. And so we have all these other actions that we're really not capturing within
this controlled environment. And it's also important that we're analyzing a single time point, and
a lot of these kids have done, and it's in an observed environment. And a lot of these kids have
gotten really good at knowing when they're in the doctor's office, and so they have what we call
their clinic walk, so they may walk on their toes most of the time, but magically they can
somehow walk on their heels just during that time while they're in the clinic. And so you're
really just getting this one snapshot in time. Who knows, maybe they're tired, maybe they're
grumpy. Maybe they want their toy or their iPad, but it just doesn't give us a good, accurate
representation of how they're actually moving and interacting with the world on a daily basis. So
I think that's why it's time for us to step out of the lab and why I'm excited about some of the
projects here and looking forward. Just some of the things that people have done out there, let's
see -- well, my computer just froze. Bad Mac. But, basically, the big ones that I was going to
talk about was so a lot of this work started with some of the markerless motion capture with Tom
Andriacchi and Stefano Corazza's group down at Stanford, but there had been for a while this
movement to use normal cameras to produce visual halls, or basically you can do background
subtraction and get a visual projection of a volume of space of people. I don't even know how to
do Alt-Ctrl-Delete on a Mac. Oh well. Okay, but I'll pretty much finish up here. So we have the
visual halls that basically they would give us a three-dimensional representation of the world, but
even then, it required that you had four, eight high-definition cameras around the lab, and I think
that's where the Kinect really comes in, where one, it's much cheaper. It requires a lot less
computational power. We no longer have to perform background subtraction on multiple highdefinition cameras around the space. And it's a really exciting opportunity for us to be able to
now move this into the home, into the lab, in addition to having our RGB information, have that
depth information so that we can get some more accurate measurements. Also, most of us are all
carrying around at least one inertial sensor with us today, accelerometers, gyroscopes,
magnetometers, and so quite a few systems have been using those, as well, to start looking at gait
parameters. And so far, a lot of the work in the Kinect and the accelerometer community has
mainly been in the space of what I'd call temporal spatial parameters, so a lot of step length, step
width, how much time you're in stance, how much time you're in swing, similar to what Ran was
showing earlier. But especially for some of these other applications, we need to add some more
detail to it, so that we can start getting joint angles and get these more specific measures that we
really need to characterize pathologic movement. And so I'll see if I can get this back operating,
but I might not, so with that I'll close. Maybe I'll show you pictures if I get it back, but
otherwise, if there's any questions, I would love to answer them for you. Yes. So thank you,
guys. That has never happened to me. Does anyone any questions?
>>: So in terms of the crouch gait that you're looking at, have you started looking into some
mechanisms, also, as to what's affecting them other than the surgery?
>> Kat Steele: Yes, so we've done quite a few studies focused specifically on crouch gait,
looking at the different mechanisms, and some of Allison Arnold's work really focused on the
hamstrings and how those contribute. We did studies looking at how all the different muscles
contribute to crouch gait. We also have done some weakness studies, which motivated that
gastrocnemius work that you saw there, where we were looking at how can weakness of different
muscles contribute? And we saw that it probably isn't those hip extensors and hip extensors and
knee extensors. It is probably more likely to be your gastroc and your gluteus medius, if
weakness is the factor. And then also, most recently, I didn't share much of this today, but we've
been doing some neuromuscular modeling, and so we've been looking at synergies, or this idea
that we don't control our muscles individually, but we control them in these weighted groups, or
kind of a simplified control scheme for having this highly redundant system with many more
actuators than degrees of freedom. And we've seen that some of these synergies may be
contributing also to the underlying causes of crouch gait. We've looked at crouch gait for many
different angles, also their tibial-femoral contact force, and so we know that for different
patients, it's different causes, but we're getting better at picking out which causes are the most
important for each individual patient. Any other questions?
>>: Yes, so you had one slide in which you showed that, actually, so far, when we look at the
outcomes of different treatments, we still have a way to go.
>> Kat Steele: Yes.
>>: And on the other hand, you had as slide on which you showed the analysis of when does it
make sense to look at the -- I forget exactly what the treatment -- extending the ->> Kat Steele: The hamstrings, yes.
>>: The hamstrings. So was it successful in that sense? In this study, were they able to show
that the outcome is positive when you select the right patients?
>> Kat Steele: Yes. So the data that I showed was from over a decade with the changes, and so
a lot of that also included before we had the hamstring lengthening. But, still, that's only a small
segment of the individuals with the pathologic gait patterns. But a nice study by Jennifer Hicks
has looked retrospectively at if you look at hamstring lengths if it predicts better outcomes, and
they did find that it did improve basically that curve for specifically for the crouch gait patients,
where if you know that they're short, slow and stupid beforehand or if they're operating at a
normal length with normal activation timing, that you can do a much better job of predicting who
the better candidates are. But there hasn't yet been a prospective study, largely because, the
challenges of working within the clinical environment. But from the retrospective data, and so
most of the labs have included now hamstring lengths in their analysis of crouch gait patients.
>>: Do you have a plan if they do get data out of the lab? Suppose now you have been in the
[indiscernible] patients, for example, how are you going to proceed?
>> Kat Steele: So if we had millions of data, that would be awesome. But another nice thing
about a lot of our musculoskeletal modeling and simulation is that they're relatively fast. We can
simulate a gait cycle in just a few minutes, and so as computational power increases, it gives us a
lot -- looking at my PhD adviser, it would take him days in order to simulate gait, and so we can
now do a lot of these things a lot more quickly. If we did have a lot more individuals, it would
not only give us a better normal database in order to be able to compare to, but especially for the
individuals with pathologic movement, it would let us characterize a much broader percent of the
population. So we know a lot about cerebral palsy, because they've been going to these gait
analysis laboratories for decades. But when you start looking at Parkinson's, multiple sclerosis,
amputees, we don't know as much about a lot of how these different populations move, and I
think there's a lot that we're missing by not understanding the differences between a lot of these
disorders. So I'm not sure if that answers your question, or if it's more of a brainstorming, but
there's a lot of opportunities, I think. Do you have something more specific?
>>: Yes. I did not understand the plan, actually. So are you saying that you would like these
recordings to be labeled, so somebody with multiple sclerosis, you'll know that this is the one,
that they have this disease? And so you will be able to compare the recordings in some way?
>> Kat Steele: Yes.
>>: What's the plan?
>> Kat Steele: So I guess the big thing from my perspective is that these are tools, and so we
have really a lot more tools, but they all need to be driven by clear clinical needs. So we have
some very specific needs. We want to be able to empower the clinicians to be able to make more
specific patient-specific treatment outcomes, if you're working with Parkinson's disease or
others. But it's not like we have a hammer and we're going to go around with every problem and
address it. You have to start with the underlying clinical need, and for us, a lot of those clinical
needs are driven by cerebral palsy and individuals with stroke and other disorders. And it's being
able to understand what is the factors contributing to pathologic movement for these individuals
and what are the best treatment outcomes? In terms of stepping out of the lab, it will give us a
lot more power to understand both before surgery and after surgery their motion. So right now,
if we look at their motion, they come in -- if it's reimbursed by insurance, they'll come in one
year post op. There's nowhere else where you might get an analysis of how well your outcomes
are afterwards, and only at one time point, one year. But if you had a Kinect in someone's home,
for example, you'd be able to monitor, okay, how is little Johnny walking three months, six
months, actually on a daily basis, after their procedure? Do they need to increase the physical
therapy regime? Do we actually need to come back in and look at something, because their gait
isn't improving at the rate that we expect. Maybe one of their implants needs to be removed,
because it's impinging on some soft tissue and causing pain. And so a lot of this is to empower
the clinicians to be able to make more informed decisions, and decisions that are more insurance
reimbursable across the timeline and the course of treatment for these patients.
>>: I guess all these are very nice, but did you actually think about the limitations of the data
that you're going to receive? Most probably, these are going to be multiple recordings with no
labeling, with no knowledge about the people behind them and the clinical condition. And most
probably, these are not going to be like the things that you showed us, but simple snapshots of
1.5 steps every time, multiple of those twice a day. How are you going to use these data for the
targets that you mention?
>> Kat Steele: Well, one of the big goals, especially with the skeleton tracker, and I guess I can
go back up here. So with the skeleton tracker now, there are a lot of limitations, both in
resolution and accuracy. If you look at the skeletons that you get out of the old Xbox Kinect, it
was largely optimized and designed for the frontal plane, but you can still get estimates of joint
angle out. They're just not very accurate. But I think there's a lot of quick and easy
biomechanical tricks that we can integrate to make this a lot more accurate, so that when
someone does walk across their living room that you will get feedback of their joint angles. And
that's actually not that much different in terms of it being one to two gait cycles from what we do
analyze in the gait lab, especially when we think about we have these force plates in the ground,
and it's really those gait analysis cycles where they're over those force plates that we analyze.
And so if the kid walks back and forth like three or four times, then you have enough
information on their gait cycle to be able to capture their average joint angles and project it over
time. And say they then go in to get a Botox injection, you can then follow up, and actually
that's something that's not reimbursed for gait analysis right now. So you can go in and see what
are the specific changes for this individual patient. And I do see it more as feeding back joint
angles with the given level of resolution back to the clinical side of the lab.
>>: I think to address I think the issues that he's trying to bring, I think as a PT, as a physical
therapist, I think I'm more interested in looking at functional outcomes. And I would like to see
my patients getting better in what they want to do and how they can improve their quality of life.
So from the research perspective, I would worry about one or two standard deviations, but as a
physical therapist, I think I would like to see what's the functional outcome of the things that I'm
trying to improve in terms of probably the gait parameters. So I think in terms of that, the body
tracking thing is doing quite an impressive job for me understanding in terms of as a clinician.
>> Kat Steele: And I think that's also where a lot of the power comes in with some of these ones
that you can even wear outside of the home, so whether the XM systems or your inertial sensors
in your phone, where you can look at speed, number of steps across a whole day, so that you can
get a better -- what we're often aiming for is this community participation piece, how can we
better quantify community participation. But especially with these treatments that really target
specific muscles, specific joints, I think this can provide some of the level of the accuracy that
we need more accurately characterize the biomechanics and their pathologic movement, and then
we can also use these technologies to step out of the lab even further and get a good
understanding of their community participation. Does their AFO prescription have to be altered?
Are they gaining weight, and is that decreasing their functional ability? I think there's basically a
lot of potential within the space.
>>: And unfortunately the insurance companies, the requirements for disability evaluations and
things like that still require these basic range of motion metrics. And so they're still important
that we need to get in addition to the functional stuff. So I think the goal to get the kinematics is
still kind of necessary.
>> Kat Steele: And there's a huge range here too that we're going to be discussing today. There
are many applications where the temporal-spatial, these speed parameters, are actually excellent
and just what you need. I mean, look at the popularity of the Fitbit and some of those other
techniques. But then, especially in this clinical gait analysis world, we need better methods in
order to be able to quantify these joint angles, and so that's why it really is that clinical need and
what your specific researcher application is, where you need to fall within that spectrum. And I
think the good news is that we now have a lot of tools, so whether you want to know how many
times they're walking or what a specific muscle's function is, we're starting to have these tools to
pull that apart.
>> Ran Gilad-Bachrach: So I hate to be rude and cut this great discussion, but we have over
here our next speaker ready. Thank you.
>> Kat Steele: Thank you, guys.
>> Ran Gilad-Bachrach: So we are very lucky to have Eric Horvitz with us today. I won't do
the very lengthy introduction of his achievements. He's the head of this lab, and he's both an
expert in artificial intelligence and in this field but also a physician. So a very unique set of
skills here.
>> Eric Horvitz: Thanks, Ran. So there's an interesting nuance, history, as to how I ended up
getting an MD and then a PhD in the computer science area in my pursuit of cognition. But I
ended up later coming back to healthcare and doing a variety of things from the point of view of
computer science. I wanted to welcome you all here today. This is Microsoft Research
Redmond Lab. It's maybe you'd call it ground zero of the Microsoft Research larger
organization, which spans about 1,200 folks at seven sites throughout the world, and there are
some other actually advanced technology labs, as well, in addition to the main laboratories. We
do both basic long-term research, driven by curiosity, as well as pedal to the metal, let's ship
some new experiences and code in Microsoft's products. Often, those two goals are not at odds
with one another. They're actually quite complementary. And in fact, things we see on the
product side often stimulate our long-term thinking and vision. The Kinect, which I think I see
inspired many of you, folks who are interested in kinematics and gait and joints and recovery,
rehabilitation, that work came out of playful development in this lab and evolved later into a
very, very focused product effort to make the system more robust and reliable and inexpensive
and useful in consumer spaces or clinical spaces where the device could just be fielded. A little
bit of developing and programming, it could actually be useful, and it's actually changed the way
even our robotics colleagues in universities see some of these challenges with recognizing people
and gesture. In the healthcare space, we have done quite a few things. I would categorize them
as a classic electronic health record data mining and fielding predictive models that are now
running in hospitals across the world. For example, predicting the likelihood of readmission
given a discharge of a patient within 30 days, which actually turns out to be a Medicarepenalized metric. So this became a very interesting thing for hospitals. Some of the first
predictive models that are running live and doing machine learning from data in electronic health
records, we've worked on that and fielded those systems, and that work came out of Microsoft
Research. Another area that we're very active in is public health, and this includes work looking
at Twitter feeds and search and browsing behaviors in anonymized logs to, for example, identify
rare adverse side effects of medications, drug interactions, depression in large populations, even
nutrition. And those papers you can read on our websites what's happening there. And closer to
your interest area, we have several efforts looking at sensing, for example, new approaches to
sensing some of the physiologic indicators, heart rate, blood pressure, tracking sleep apnea with
new kinds of methodologies that might be less invasive than going off -- and less expensive than
doing formal sleep studies. And then the work that Ron's been leading up in looking at what we
can do to track gait and even discovering previously unknown implications of the dynamics of
gate over time for illness, long-term illness, tracking recovery or the progression of disease.
Healthcare is definitely -- we keep on reflecting at Microsoft more broadly about the percent of
GDP that goes into healthcare. We often think about well, can't Microsoft -- just on a business
side of things, be more involved in that segment of the economy, given how big it is, and help
with enhancing the efficiency, as well as the quality of healthcare delivery? And this is an
ongoing reflection. We actually had a Healthcare Solutions Group, a division, we set up in 2004
and a half or so that a year and a half ago we spun out into a joint venture with GE. This s called
Caradigm, Paradigm with a C, which it's moving ahead as a joint venture now, and that's
basically kind of an HIS company, doing a new kind of -- which was fielding this product called
Amalga, now called Caradigm Intelligence Platform, renamed. Anyway, I'd be happy to take -- I
wanted to just say hello and welcome you to this really interesting gathering today. I hope you'll
do this again, that this will be successful. The way we work, just so you get a sense for how we
work at Microsoft Research, is these kinds of efforts, the one you're a part of right now, which
can grow to become quite significant, are the children of passionate researchers, who like Ron in
this case decides I want to push on this area. I'd like to have some investment and some
enthusiasm from my management, but it's really -- it's the individual researchers who define what
it is that we do at Microsoft Research. There typically is not management edict going in this
direction. It's really from the ground up, typically. We often say that we -- people ask, what is it
that Microsoft Research does? Well, we typically hire the best and the brightest and then let
them tell us what we should be doing. That's how it works. So thanks for organizing this, and
you're actually feeling the tentacles of the passions of Ron here. So any questions or comments
about anything I said about Microsoft Research. We may give a few minutes for a few
comments, and then I'll get out of your hair so you can return to the normally scheduled contentfull program.
>>: I'm interested -- I'm a neuroscientist. I'm interested in are we at Microsoft doing something
related to motions or affect in terms of devices?
>> Eric Horvitz: So it's interesting. So when I think about what should I mention as some grand
swaths through some big areas of healthcare and wellbeing that there are many pieces that I left
out. So, for example, we have several efforts at several different of our labs. I would
characterize, I'll use the -- maybe it's jargon, but the phrase affective computing. That's a phrase
that came out of the MIT Media Lab, but the idea of using sensing and machine learning and
computation and inference, prediction and forecasting to understand mood, emotion, likelihood
of depression, recovery from depression, cost of interruption. Let's get more business oriented
now. What's the tolerance to disruption and what's the recovery from interruptions for, for
example, alerts on a desktop? The probability that someone will forget sets of things, that can be
very personalized to that human being, and that can include conditions on certain kinds of
degenerative disease processes. The notions of studying human cognition, what do people
consider a memory landmark, for example? That's an area that we've worked on. So as I'm
talking, I'm thinking of projects like we had a project called the Jogger Project, which referred to
Memory Jogger, and you can imagine, this project centered on -- was published actually in a
planning conference, but it focused on taking inputs from people who were training up systems,
where they would indicate what they had forgotten from their calendar in terms of the details of
the global aspects of their calendar. Just as an example, building a large data set and then
learning to predict when to alert somebody about something that they would likely forget based
on the nuances of the structure and the day and so on.
>>: To understand that, are you looking at some physiological responses that these people are
indicating?
>> Eric Horvitz: Not for that specific project, but Mary Czerwinski's in our affective computing
area certainly has strapped devices on people and looked at these feeds over time, over days and
days, to get a sense for a life log of what's been going up and down in their lives and trying to
associate that with other things, like grumpy meetings or meetings with grumpy bosses. That
might be personalities from the Exchange Server, who's showing up, and how does this affect
various kinds of biometric indications? So this has been an interesting area. I personally would
like to see more in this space and to invest more heavily in this space. On the neuroscience side,
I'll mention that Jeannette Wing, who is now the Worldwide Director of our labs, who I report to,
has mentioned wanting to push significantly into neuroscience. Now, what does this mean? She
leaves that to a small committee of five people -- I'm one of the five -- to figure out what should
Microsoft Research do in the neuroscience area. And so we have a variety of nascent efforts,
some of which have preexisted. Like, for example, we have a five to seven-year project
collaborating with Bill Kristan's team at UCSD on, ready for this? Leech, the operation of the
leech ganglia. It turns out the fluorescing dye data coming off 450 or so neurons in the ganglia
of a leech create this really interesting challenge problem to figure out just from observing what
you see in terms of the neuronal activity in leech ganglia how it links to the decisions the leech is
making, given stimulation, to infer the structure of that network. And it's a very interesting
problem. We've even had shared interns coming up here who are neurobiology postdocs and
graduate students who sit here for three weeks and work on leech. It's not like we're going to be
shipping Leech 2.0 anytime soon, or 2.1. I guess also the Microsoft Leech Project wouldn't
probably sound very good. But there are other projects, as well.
>>: Call it iLeech.
>> Eric Horvitz: With a small-I, right, right. But some of our teams have looked at fMRI data to
try to help with others. We do lots of really rich collaborations with academic partners.
Typically, we find it fairly easy to work out any kind of concerns with IP, to do open publication
with coauthored -- in kind of coauthored publications, for example, if it makes sense on both
sides and it's helpful to have a collaboration going. One other kind of point I'll make is a nice
way for academic folks and folks in other organizations to work with Microsoft researchers is at
times there'll be a shared student. So we have a rich intern program. I think it's kind of
surprising -- so we're about 1,100 to 1,200 folks worldwide, Microsoft Research is. And we
almost double in size every year, when we take on about 1,000 typically graduate student interns,
every year. And that's a significant investment in mentoring, as well as in collaboration. At
times, some of the students stick with the mentor at MSR in terms of staying in touch and
continuing a collaboration, becoming a focal point between the outside PI of another project or
researcher and Microsoft Research. I just was at -- just to give you an example, I was just at the
defense of one of these interesting people, who I stayed in touch with since her internship I think
five summers ago now, this summer it will be, Jenna Wiens, who was an MIT graduate student.
And typically, this is not always the case -- we don't necessarily push for this, but her intern
project became her thesis work, and it was on C. difficile, so predicting that patients in hospital
during their stay, for example, will come positive with an infection, diarrheal infection, C.
difficile. The work led to fielding models in hospitals that are running now, lots of interesting
studies on transfer learning, how do you transfer data from one hospital to another, work on
temporal reasoning, how do you reason across time with the model as data is coming in in a
hospital, and decisionmaking? And she, I just discovered yesterday, she's going off to Michigan
at the CS department there. So I'm not sure how our relationship will change now that we'll be
peers I guess, now. Any other questions or comments on how we work?
>>: So as an MD kind of looking at the technical space, what kind of challenges have you seen
taking technical students and technical individuals and translating the results to the clinic, or
translating working between the two?
>> Eric Horvitz: Well, there are several dimensions of that question. One is just the really hard
problem of what's called translation, translational biology, translational medical informatics.
There's so much that has to happen in translation. For example, we worked hard to get various
versions of our predictive modeling off electronic health record data running in hospitals, and in
many places, we just say, oh, by the way -- one of our internal folks working with us at the
hospital will say, we've just now, if you look at every single patient in the hospital, you'll see the
C. difficile risk, and it's a probability they'll get infected during this stay. And we're just going to
put it up on the display. We haven't designed any interventions. We're just going to display an
actual well-calibrated probability, and sometimes it takes really just displaying that number to
start discussions. What does this mean? What should I do about this? But in general, there's a
really big chasm. Even publishing in this space -- for example, I'll mention this machine
learning area. If you take work on showing how well machine learning works to make
predictions in a hospital, which very much the whole paper's on this general methodology that
takes specifics of specific hospital information and does a specific hospital-centric predictive
model. You put it in a major medical journal, you get these responses back, well, we want to
learn general things. We don't want to learn about this procedure. We want to learn about what
we know -- what if you learn about like, let's say congestive heart failure for medical knowledge
as opposed to telling the world we now have a procedure that does patient-specific and hospitalspecific things, which is a new way of thinking for old-fashioned healthcare that want facts that
medical students can study. So we see that kind of an impediment sometimes to understanding.
But I can go on and on. It's so hard to take some of the technical work sometimes, which is
viewed as geeky and rarified and get it into the actual standards of practice. And doctors will
also often say, especially in your areas, they're already busy and they have a whole team that's
tracking, with typically paper and in baskets. They don't necessarily want to take out the time,
and it's probably to take a risk at modifying things with new approaches, especially when they're
not in the literature yet, they're not certified in clinical trials, they're not accepted by let's say
your area, workman's compensation and insurance companies, necessarily. People are barely
keeping up with the old stuff. So it's challenging. I do think that one of the critical aspects, to
answer your question, is finding evangelists on the other side, finding clinical evangelists that get
it, that have the confidence of their colleagues, that can host meetings and say, I know what
you're thinking, but check this out kind of thing and can stay with it. Not just a flash in the pan,
but will stay with it. I ran here after hanging up on a phone call this morning, just now, on the
Infectious Disease Control Authority at one of the largest -- well, it's a top-10 urban hospital in
the United States in Washington, DC, and he's been working with me on evangelizing these
methods on minimizing hospital-associated infection. You can't say hospital-acquired infection
anymore, by the way -- hospital-associated infection. That's the politically correct way of saying
it, because of the community-acquired illness -- for several years now. And he is the director of
infectious disease. So I have him on my side, and then we can as a team communicate what we
might want to do in the hospital setting. Even a student like Jenna Wiens, who I mentioned
before, getting a really great technical person who happens to also be passionate about clinical
care and wants to really understand the nuances, that's pretty rare sometimes. It takes a certain
breed of person. I like to find those people and work with them very closely and groom them for
their future translation opportunities. Any other comments? Anything about Microsoft
Research?
>>: Sure.
>> Eric Horvitz: Yes.
>>: What are you most excited about coming down the horizon, I guess?
>> Eric Horvitz: In general?
>>: Sure, unless someone has a more specific question.
>> Eric Horvitz: So you mean in general. So I think, well, for me, I worry -- as Microsoft
Corporation, as kind of getting more into the executive level now, I worry a lot about the
company. So I think about what's coming down the horizon more strategically, but I can get into
it technically, as well, and think outside of MSR and Microsoft Research. But let me just say
that in 1995 or so -- so I've been here 21 years. In 1995 or so, at the company meeting, Bill
Gates put up a slide that said, here are our competitors. IBM, it was Lotus, WordPerfect I think
it was. And he said, but the people we really worry about, what really keeps you up at night, is
the black around those known entities. It's the entities that don't exist yet or that I don't know
about. This is before Google was formed, for example and before Amazon came to the fore as a
force. And certainly, I think Apple was already dwindling by that time into almost nonexistence,
as you all may recall. And so we do think broadly about trends and implications of basic science
research and advanced technology for where things are going in the world. The fact that a search
portal with machine-learning-powered ad matching and ad services would be this existential
threat to Microsoft someday is surprising. So we try to be on the lookout for those kinds of big
trends. On the lighter-weight things and maybe not on the lightweight, but on the technical
trends, let's say, areas like what's the rising role of intelligent assistance in life? Will we have
these services -- will Siri, Cortana and Google Now evolve into this competitive battlefield
where everyone's depending on different brands of ongoing decision support, including
healthcare decision support someday? Might there be competitive services someday that say
when you have -- when your glycemic levels just rise to the point where you're reaching up
against prediabetes or diabetes type II, adult onset, a service comes to the fore and says, join this
service where Dr. Krantz and 10,000 vendors will now provide for your lifelong engagement
with services, exercise programs and community as you travel through the world. You can
imagine that there are healthcare implications for new kinds of services coming to the fore that
might be life-spanning, sensing, reasoning, decisionmaking with a back end of commercialism
and commerce in a competitive way. People say they're going to keep you healthy, curated
through experts, for example, like that Dr. Krantz I just mentioned. I just made that name up.
But you could imagine trends like that could happen. We worry about that and reflect about that
and do projects in that space. We're very interested in where computation is going. The future
of datacenters, how to do those -- how to build systems that could do more efficient allocation
and processing of cloud services. It's unclear where the whole future of work is going. What's
the role of crowd work, for example, and how does that fit into traditional jobs you might have?
What happens when the automation of seemingly human-centric or intellectual tasks that require
humans become dominated by machine solutions? What happens to the economy and society?
Where's education going? How might we better share data with HIPAA rulings and taking into
consideration how might we learn from shared data sets about healthcare challenges? How do
physicians keep up with -- or researchers keep up with the exploding literature? Are there ways
we could automate syntheses that would track, for example, the relevance of single-nucleotide
polymorphisms that are coming to the fore and understand the implications of what a new one
means or a new finding means? And might there be breakthroughs that surprise us in healthcare
-- for example, computational breakthroughs in cancer, where we actually are looking at cancer
as a computational problem with control and feedback, with many touch points of detecting and
halting it.
>>: Thank you.
>> Eric Horvitz: That's a long paragraph. Okay, great. Thanks very much.
>> Shomir Chaudhuri: All right. Hi, everyone. My name is Shomir Chaudhuri, and I'm going
to be talking about the usability of fall-detection devices. Just a little bit of background about
myself. I'm a fourth-year PhD student at the University of Washington in the Biomedical and
Health Informatics Program, and the lab I work with is called the Health-E Lab, which I'd like to
talk about just a little bit. It's a collaboration between the School of Medicine and the School of
Nursing, started by Dr. George Demiris and Dr. Hilaire Thompson, and what we like to do is
build or think up informatics solutions for older adults. So some of the projects we've worked
on, we like to put sensors in people's homes and see what kind of data we can get out of it and
what kind of data they appreciate. So in one of our earlier projects, we put motion sensors and
hydro-sensors into people's homes and gave them back their data to see what they were
interested in, and also to see if anything was useful, maybe to their provider or to their caregiver.
We took a project similar to that, where we put a health kiosk in a retirement community and had
people take their own glucose data or heart rate data, and not only did we share back with them,
but we also tried to think up new ways to visualize this data, so that it would be more
understandable for them. And we did focus groups with both older adults and healthcare
providers using these. We also do stuff with gait, which is why I'm here. One of the first
projects I worked on, we took a gait mat to an older-adult community and had people walk on it,
and we compared their sense of their balance to the numbers that their gait mat produced, and it
was really interesting. It's actually kind of how I got into where I am now, so my dissertation
research is on fall-detection devices, which is what I'll be talking about today, and through that
gait study and a bunch of other studies I did early on, I found that falls are a big problem in older
adults. I'm sure many of you know. So just a little bit of outline. I'm going to a little
introduction into falls in older adults and how they match with older adults. I'm going to talk
about a literature review I did on fall-detection devices. I'm going to talk about a focus group
study I also did with a certain fall-detection device, and then I'm going to talk about an ongoing
pilot study I'm currently conducting. So in the instruction, I just want to go over our objectives.
I want us to understand the current state of fall-detection devices. I want us to explore the
perceived user needs -- how do people see these devices as helping them and what do they want
out of these devices? And I also want to explore the usability and feasibility of these devices in
the real world. So falls, as I'm going to define them, are from this definition, which is
unintentionally coming to the ground or some lower level, not because you received a hit or lost
consciousness, but because maybe you tripped, or because maybe you're having some
physiological problems that cause you to lose your balance. And the population I want to talk
about is older adults, and my definition of older adults is 65 and older, which is a rapidly rising
population, as you can see from this, going from about 56 million to 92 million in about 30 to 40
years. The reason for the rapid increase is both larger life expectancy, due to improved
healthcare, which is a great thing, and also the Baby Boomer generation is moving into this area,
so it's going to be a pretty burgeoning field pretty soon. So, in older adults, those 65 plus, about
one in every three fall at least once annually. This number increases as you grow older, with one
in every two falling for the 80-plus crowd at least once annually, so it's a pretty big problem.
And about 20% to 30% of older adults who experience a fall have some kind of injury, either
head trauma, laceration. I'm sure we're all familiar with hip fractures. These are all things that
can be caused by a fall, and if they're hospitalized, they can do well, but a lot of patients end up
dying either in the hospital or a year after, either from physiological problems or just from
mental problems of having experienced a fall for the first time and just realizing how maybe
alone they are or how fragile they are in terms of their health. In 2010, in the United States
alone, about 20,000 older adults died from a fall-related injury. This also has a huge economic
impact, with the US spending around $30 billion a year on medical costs for falls, which is
expected to rise greatly. So for in-hospital costs for a person who has experienced a fall versus a
person who has not experienced a fall, the difference is around 32,000, with the person who
experienced a fall obviously spending more time in the hospital, spending more money for the
hospital, so it has a big impact in that area. And, finally, there's a huge emotional impact, not
only for the faller but also for the relatives of the faller. This is an article from the "New York
Times," where this person's mom experienced a fall, and he's just writing about how all the next
week, all he could think about when's she going to fall next, how can I make sure she's okay?
And there's also something called post-fall syndrome. It's also called fear of falling. And it's just
a very debilitative syndrome in that people are just afraid to be alone now, and you can see them,
they're always wondering, where's my closest support, where am I going to be safe? It can be
pretty crippling to a person's activities of daily living, and especially if they experience a long
lie, which is remaining on the ground for more than an hour following a fall, which can have
incredibly severe complications -- dehydration, obviously, rhabdomyolysis, which is the
destruction of skeletal muscle, I'm sure most of you know that, and even death. They found that
half of those who experienced long lie will die within a year, and once again, that's probably
because they experienced some physiological effects, but also mentally, just being on the ground
for that long without anybody coming to help you, it can really be psychologically damaging. I
heard a story of a person being on the floor for about 36 hours, and it's a pretty awful fate. So
there's a lot of solutions to stop falls. One is, there's many different fall-prevention techniques.
Regular strength and exercise, Tai Chi is a big one, vitamin D supplementation. There's other
supplementations, as well. Fall-proofing one's house, a lot of people don't like it, but if you
come in and maybe remove their favorite throw rug, there'll be less things for them to lose their
balance on, and obviously assisted devices, canes, walkers, so on and so on. The topic I want to
get more closely towards is a fall-detection system, and the currently available ones, I'm sure
you've all heard of Life Alert or Philips has one called Lifeline. And the basic premise that a lot
of people said is that the faster you can respond to a fall, the better care they'll have, the less
complications they'll experience and the better mental health they'll experience. So here is an
example of one. This is from a person in one of my studies, and this pendant will just call the
front desk at the community they're living. It actually won't call. It'll send a signal, and the front
desk will then call them and make sure they're okay. Really useful devices. I absolutely love
these devices, and people seem to like them a lot, but there's a lot of cases where they might not
work, either because the person's unconscious, the person might not be able to reach them. They
also found that a lot of people are just so embarrassed that they have fallen and they can't do
anything about it that they don't press the button, because they don't want to be discovered like
that. A recent study found that 80% of people wearing the device did not use it after they had
fallen. So I guess a better -- I don't want to say better, but another solution would be automatic
fall detection, which is something that's been coming up quite recently, and I'll talk to you more
about that, but a system that understands that the person has fallen and automatically calls
someone or signals for help. And so that's kind of where I want to go with the literature review.
I really want to understand what has been done in this area, both with wearable devices and nonwearable devices to see what the problems are and how close we are to having something that
can automatically detect if a person's falling with high accuracy. So I was lucky enough to get
this published in the "Journal of Geriatric Physical Therapy," and I'm just going to go a little bit
into what I did and the results I got. I'm not sure if you guys are that interested in the
methodology, so I'll try to go over this quickly. I searched four different databases, and I looked
for articles that involved a system with the purpose of detecting when a person had fallen, but I
did not look at personal emergency response devices, and I kind of took away things that looked
at fall risk, which I think is something we're all talking about here and I'll touch on a little bit
later. There were 113 unique articles that matched my criteria. Most of them evaluated the
accuracy of the devices, whereas some looked at participant feedback, which is more of what I'm
interested in, but I'll talk more about that later. There was 57 projects that talked wearable
systems, systems either worn -- I'll go into that a little bit later -- and other 35 projects that talked
about environmental systems. So wearable system, as I'm sure the name implies, is placed upon
the person. I've usually seen it as a lanyard, but there are also hip ones. One of the funnier ones
I saw was it almost looked like a complete backpack, and it's really obtrusive, I would say, but
they tested it out anyways. And I think it's obtrusive because it has an airbag, in case they do
fall, an airbag, which is a really cool concept. So the most common location is around the trunk
of the body, which is right around here, because the closer it is to the center of mass, the easier it
is to understand if the person has fallen. Other locations, the ear, the arms. It's really hard to
compare these studies, but as you got farther and farther away from the center of the body, the
accuracy became lower. So good things about wearable devices, they're always with the person.
You can use them with multiple people. You can hand out different wearable devices, and
because of their proximity to the person, they experience the same acceleration, so it's easier to
understand if a person has fallen. Unfortunately, they have some cons. Obviously, they're
battery powered, so the person would have to charge them, usually. Sometimes, they can be
uncomfortable, especially in the case of that backpack, and maybe one of the bigger problems,
especially with this population, is that they require users to remember to use the device. And this
could definitely be a problem if you get into users who have Alzheimer's or dementia.
Environmental systems, something more like the Kinect, I would say, placed on the user's
normal device. There's a ton of them out there. They're usually cameras, but there's also
acoustic sensors and pressure sensors, which I think Dr. Steele talked about a little bit. Dr.
Steele? Yes. And I think you already touched on these when you talked about the Kinect, but
they don't rely on the user to remember to use the system, which is huge, and they usually are
plugged in, so they never have to be recharged or anything like that. Problems are, they're
limited to a specific space. A lot of people have privacy concerns, especially if you install a
camera in them. We had a study where we had infrared sensors that looked like cameras, and
even though we told the participants they're not cameras, they can't see you, participants would
still cover them up when going to the bathroom, or would make sure to cover themselves up after
getting out of the shower. So there's a lot of privacy concerns there. The occlusion is when the
camera can't see a person, maybe because a couch is in the way or something like that. And the
trouble with multiple people can be a big problem, because a lot of people do live with their
significant other or with friends or they have pets, and that could throw off your sensors. So this
is sort of -- I know it's a lot of information, but here's the median and range of accuracy,
sensitivity and specificity of wearable devices. You don't really need to take away any real
numbers. You just want to see that they're all in the upper 90s, which is pretty good. The same
with environmental, right about all in the upper 90s. I'm sure you're wondering why I didn't put
them all together so you could compare them. I'll do that now, but I kind of want to warn you,
it's really hard to compare these studies, because the way in which they were evaluated is so
variable that for me to say that 97 is better than 96 is completely false. I definitely cannot say
that. And I guess let me talk a little bit about the context of how they're done. So most of these
studies were done without older adults, either with young volunteers or there's fall dummies, that
kind of thing. All non-wearable devices were done without older adults. Of those done with
older adults, the majority was in the laboratory, so come in the laboratory, please fall down 20
times, and we'll see what we get. I guess that got a laugh. My favorite story of that was they
would have participants close their eyes for five minutes, and somewhere in those five minutes,
they'd push them, and that's how they got them to fall. And I was like, that sounds like a great
study. I was really excited about that one. But only about 17% were done with older adults in
the real world, and what I mean when I say real world is just us interacting as we normally
would, and real-world falls are found to be much different than laboratory falls, because when a
person falls, they usually try to grab onto something, where in a laboratory, you're told just to fall
backwards or forwards. So I think that will be something I'll talk about a little bit later as a gap.
So I talked about there were a few other studies that talked about participant feedback on these
devices. They're mainly interviews where they either gave a person a device for a little bit or
they were comparing a pendant to a device. So in terms of acceptance, participants really like
that they gave them a greater sense of security and allowed them to be independent.
Independence is a huge thing with this population, but they wanted more passive systems and
they wanted to know what the system was doing at all times. In terms of reduction, a lot of
times, they didn't feel they were in control of calling the call center. Sometimes, it would just go
of, or they didn't understand how it worked. And even through from the previous slide, 96% said
they liked the device, only 48% said they would actually use it. So this is kind of the thing I
want to get into, is how do we encourage more people to use it? So I already talked about
devices are becoming much more reliable. What we need now are more real-world tests of these
devices, and we need to understand what will make older adults actually use these devices?
What little things can we do, or big things, that can encourage them to have a device maybe
100% of the time, if they can. And that's kind of where my focus group studies went into, and
there has been one focus group study done. There has been one focus group study done on older
adults, but they focused more on fear of falling and such a device would affect their fear of
falling, which is really important. You guys should read that paper for sure. So my purpose was
to understand the perceived user needs, basically, understand what motivates or deters older
adults from using such devices, and what can we do to improve these devices? So I'm just going
to go briefly over the methodology real quick, and then we'll see a bunch of quotes, really. So I
did five focus groups in three older-adult communities. Focus groups consisted of a brief
presentation, just a PowerPoint slide, where I showed them the different kinds of wearable versus
environmental thing, and we had about a 20-minute discussion. And then I brought in a
prototype device, a wearable prototype device that has GPS capabilities, automatic fall detection
and can call directly to a call center. And I had them play around with that for a little bit, and
then we talked a little bit more on that. Focus groups were audio recorded, and then three
independent researchers thematically coded them. So we had 27 participants in total. There
were 22 female and five male. We didn't gather demographic data from them, which I'm really
regretting not doing, but in the flyers, we advertised for people 60 and older, and at the
retirement communities we were at, it was mainly people of that age, so we weren't too worried
about not getting older adults. Varying living situations. Two of the communities we went to
were definitely independent living. The other community was both independent and assisted, so
it was hard for us to really identify who needed assisted living, and very varying socioeconomic
levels. Six of the participants spent about -- had a monthly housing cost of about $400 to $600,
while the other 21 had a housing cost of about $3,000 to $5,000. So pretty different
socioeconomic levels there. A lot of the participants talked about experiencing previous falls.
This person in particular had a fall on the stairs and was pounding on the door, and luckily,
somebody came by. So another person also fell and thought they could have been there a long
time. Previous falls usually were motivators to have such a device, but not always. There was
one lady who fell on the sidewalk where a lot of people were around, so she didn't think she
would need this device at all, because what would it do that other people wouldn't do? So social
isolation was definitely a motivator for using fall-detection devices. People also had stories of
people falling, so this one, her friend fell and was just pressing a button and nobody ever came.
It was actually a really sad story to hear that one. Similarly -- well, actually, the opposite. This
person accidentally set off their device, which caused the police to come, and then the guy got
killed, because he didn't understand why the police were there. That was a news story they
heard. Pretty grim. But I don't know how truthful it might be, but it could be a very real
possibility, if someone with dementia or Alzheimer's maybe sets off this device and doesn't
understand why people are in their home. So independence I've been saying is a huge thing, and
especially in this population, anything that can encourage independence is seen as a great thing.
So this person really doesn't want to do be dependent on anything. She thinks of this device as
claiming her as dependent, and that's sort of a stigma that I'll talk about in a little bit. This
person's kind of sick and tired of just being told what to do by people, so when someone at the
retirement community says, look, you're in danger of falling, you need to wear this device, she
automatically shies away from it, just because she wants to be independent. So stigma was
definitely a big thing. Any kind of obtrusive device that was anywhere on their body was a
definite problem. This was a great quote, and actually, these are both from the same person, and
she's talking about hearing aids and how we've come to accept hearing aids, because they're kind
of stuck behind your ear and you can't really see them, or they've just become commonplace
now. But anything that really goes beyond that subtle thing is a real problem, because it just tells
people, I'm old. And she acknowledged that anyone who had been scared by a fall might still do
it, because they'd rather live than be old, but we really need to do something to hide these
devices, when they're in use. People really preferred wearable devices over environmental
devices. Most of the people at these communities went for walks daily, went to the KFC down
the street. And so any device that was only looking in their home, they really didn't want it. And
in terms of that, they really wanted a wrist watch. This was -- I guess it was surprising, but it's
only surprising because there hasn't been much done with wristwatches so far that I've seen, and
you can understand why, because it's really far away from the core of the body, and if somebody
is even just moving like this, that could be determined as a fall. But people thought wrist
watches were definitely better, because they'd be more apt to be worn in bed, and because a lot of
people don't like things around their neck, when they have necklaces already, or even just men
don't want to feel like they have a necklace, so that was a really interesting thing. So our
environmental devices were not as liked. They thought cameras or microphones would really
produce a lot of false alarms. They couldn't quite understand how a camera seeing you lying
down would identify you as fallen, and they thought it would be much more difficult to
implement. When they realized that I had to install a camera in every room, everybody was just
like, no, we're not doing that. That sounds more expensive. That sounds like it would be really
difficult. So I think environmental devices have a lot of pros. I think there needs to be more
understanding of how unobtrusive they can be, how accurate they can be and how inexpensive
they can be in the future. People really love the automatic fall detection. Everybody who had
one of the personal emergency response systems said, yes, it's useless. I mean, what if I fall and
I'm unconscious? It's not going to work. That occurred many times. They did have the caveat,
though, that they need a way to cancel it. A lot of people talked about I'm going out to the opera
today and I don't want this thing going off, and there's no way to turn it off, so that was the big
thing. They also still wanted a button they could manually press, just because they still want to
feel in control of the device. So these are some things to consider. They did like automatic fall
detection, but most people really were interested more in fall prevention. So when I started out
the talk, most people would say, this isn't about fall prevention. How are you going to stop me
from falling? I was like, no, this is about detection. I want to know when you've fallen. They're
like, well, what's the use in that? Some people -- yes. We got some snarky responses, but that's
fine. So they had some really interesting ideas. One person just wanted to know, hey, you're a
little off balance, balance up, something like that. Another person was saying the exact same
thing. One really interesting person wanted to be longitudinally tested, so if there was any
weirdness in his gait over time, maybe he would get a message that would say, go see your PT,
or go get tested for this. So I think there's a lot of promise in this area, and I think one way to
really encourage people to use fall-detection devices is to combine them with a fall-prevention
device, something that will say, look, this will definitely detect you when you fall. This will
most likely detect you when you fall, but it will also help to prevent falls, and that I think is the
real key that we're looking for here. People love the GPS system. Once again, they wanted to be
able to take walks wherever they want, and they didn't want to be confined to their home. The
current pendant that I showed you with that person was only confined to their community, so as
soon as they left the community, it was useless, and they just did not like that. Some people even
asked if they could use it outside the country. I think a lot of people past 65 like to travel, which
is a great thing, and so having something like this that maybe even like an OnStar service, maybe
you're stuck in Spain and need to get somewhere, just being able to press that button. It'll also
detect falls, but let me help you get where you need to go, that could definitely be something that
could encourage more use. Multiple functions. People did not like the idea of adding something
else to their routine. They already have a cellphone. They already have to keep track of a
necklace or something. They don't won't to have to keep track of one more thing, charge one
more thing. So the more functionality we can put into one fall-detection device, the more useful
it may become, but we also run the risk of making it too complicated, so it's a fine line that we
need to balance. And one of my ideas is to sort of have -- my next slide is actually
customization. It sort of has a customized piece for each person. A lot of people were like, look,
this fall-detection device is good for these kinds of people, but they're not good for me. I don't
want all this extra stuff. I just want this feature. So I think if we can roll out devices where we
can say, look, you want this feature, you don't want this feature, I think that would be the way to
go. Customization was also really important for GPS, because a lot of people don't have
Alzheimer's, don't have dementia, don't feel like they need to be tracked all the time. So they
said, look, if we can turn this off for non-wanderers, then that would be great, and that would
also save battery. Another concept of customization was who does the device call? A lot of
people like the idea of call center, but a lot of people just wanted to call 911 directly, because
they were like, what's the call center going to do, except delay the process? Some people wanted
their friends to be called. There was a big debate there, so I didn't quite get to who is the
number-one person to call, but I think this goes more to customization in terms of we need to be
able to adapt to anything the user might need. And I think I'm closing out of my quotes here.
Sorry there are so many. In terms of cost, cost was one of the big barriers. People knew they
had to pay for it, and then there was a monthly ongoing cost, and that was something that they
really did not want to do. A lot of people thought -- in both higher socioeconomic status and
lower socioeconomic status -- thought that it should be covered by Medicare. One person said,
look, this is exactly like a pacemaker. It's preventing me from dying. Why doesn't health
insurance pay for this? And we did a secondary analysis of the focus groups, comparing lower
socioeconomic to higher socioeconomic status, and we found very little difference. So that's a
good thing. At this point, right now, what we need is one fall-detection system, and maybe with
some customization, we can adapt it to a broader audience. So, in summary, participants' desire
for such a device was really related to their independence, the ability to control something, lack
of stigma. Automatic detection was desirable, but I think more functions, especially fall
prevention, are something that's necessary to encourage these devices. And most participants
preferred a device on their wrist that could track their location and, once again, could
automatically detect falls. So I'm just going to briefly go into the pilot study. So after the focus
groups, I think we can all agree that saying you want something versus actually using it and then
understanding what you really want is a little different, so we wanted to give people devices for a
longitudinal study, where we could test the feasibility of the device, real-world tests in case they
do fall and also understand the usability of the device -- what features actually work for them,
what features don't they like? Oh, I just went over that. Well, there it is again. This is the
device that we used. I'm trying to keep it anonymous, if I can, but it has automatic fall detection.
It has GPS capabilities. Unfortunately, it's not a wrist device, but it has the -- you can either
choose to wear it as a lanyard or a clip, which a lot of people are putting on their bras, actually.
And it has to be charged every 36 hours or so. So right now, we have eight subjects enrolled.
I'm trying to get 20. And in the event of a fall, the device will contact the front desk at the
location we're at. They will make sure they're okay, and then what I would like the participant to
do is record in their fall calendar that they've experienced a fall and what they were doing either
when the alarm went up, in case it's a false alarm, or when they fell. And I would also like the
participants to then call me. I realize now that that was a little far-fetched. So far, we've had
about 16 alerts. All of them have been false alarms, and I've gotten zero calls or anything. So
what's basically happening is I have a way to monitor the subjects, which is basically a map for
their GPS location, and then almost like a log of what's happening in terms of is it in the charger,
was there a fall detected? Was the classification -- it's a good system. It needs some work, but
I'm not focusing on that right now. But if I see a fall has been detected, I'll give them a call the
next day and see that they're okay and ask them what was happening. And a lot of the times, it's
just nobody knows that it even went off, or they're like, I heard a voice all of a sudden, and I
don't know why. So there's something going on there that we need to figure out. One of my
favorite stories -- I don't know how appropriate it is, but one of the ladies, she was wearing it on
her pants, and she said that every time she went to the bathroom and had to pull her pants down,
it would go off. So she was doing her thing when a voice would go off, and she'd be like, oh, oh
no. So, yes, the study is going well, I think. Once I find out more results, I'll probably come
back and tell you more. So in conclusion, for this whole thing, automatic fall detection is
improving, and it is very important. There still needs to be a lot of work done to understand
what the users really need to encourage people to use these devices. Right now, I'm pretty sure,
out of my eight subjects, maybe two are using it 100% of the time. And by 100% of the time,
we're asking them to charge it at night, so they're not even using it at night, which is when a lot
of falls occur, so that's a big problem. I think fall prevention is a great thing, and I think doing
something with longitudinal and gait analysis would be a great complement to fall detection. I'd
love to hear your ideas. I just need to thank a few people, my adviser, George Demiris and the
training grant I'm on and the Health-E team. And with that, I'd love to hear your ideas,
suggestions or if you have any basic questions. Thank you.
>>: So I think that there is in the making a line of fall-detection devices which are kind of in the
form factor of a Band-Aid.
>> Shomir Chaudhuri: Oh, really?
>>: You just stick it, and some of the claim that they could work maybe even for a week or so.
What do you think would be the reaction of people to this kind of device?
>> Shomir Chaudhuri: I think that would actually be really great, because the less they have to
do and they less they have to do and the less it can be seen, the better it would be. However, I
think you might run into problems with having to remember to put on a new Band-Aid,
especially if it's non-daily. From what I've seen, and this is a very small sample of people,
obviously, the older adults have a definite routine to their day. So if they put something on at the
start of the week and then have to forget about it, it may be a problem, but I think that's definitely
going in the right direction. The less obtrusive, the smaller it can be, the better, yes. And do you
know where that study's from?
>>: [Indiscernible].
>> Shomir Chaudhuri: Is it commercial?
>>: I've seen it at a conference. I don't remember what stage it was.
>> Shomir Chaudhuri: It's really interesting, yes.
>>: So what percentage of these older adults now have smartphones or other devices that they
help [indiscernible].
>> Shomir Chaudhuri: Yes, actually, there's a recent Pew study about that, and I don't remember
the exact numbers, but I think it's over 50%. I wouldn't say smartphones, but definitely have
phone.
>>: It was 40% or 50% had cellphones, but not necessarily smartphones.
>> Shomir Chaudhuri: So using a phone as a fall detector, is that what you're kind of getting to?
>>: Yes, and just what are the devices they're currently already carrying with them?
>> Shomir Chaudhuri: Through my literature review, there were I think four studies that tried to
use a phone as a fall detector, and the accuracy just isn't there yet. I think it's getting there, and I
think in the next five years, maybe, it'll definitely be there. But for right now, the phone just is
not as accurate as having a device solely devoted to fall detection. But I think that's where we
need to go, because the less devices we can give them to have, the better.
>>: Is it because of the sampling range in the accelerometers in the phone? What is it that
makes them not as accurate?
>> Shomir Chaudhuri: I'm not an algorithm person. I really have no idea. These are just from
the numbers I saw in the literature review, but could you think of any reasons why they wouldn't
be as accurate? Maybe there's a bunch of things ->>: The way that you wear them, right? You might be carrying them in a purse or in a pocket.
All of these people show that there's a lot of noise, where it's not just place it on your trunk.
>>: That makes sense.
>>: The suggestion would be like why not are they wearing them on their ankles, because in
terms of their posture correction and falling, if that's the first thing that we try to adjust in terms
of correcting our posture or our balance, so why [indiscernible]?
>> Shomir Chaudhuri: Yes, so I didn't say they should wear it on their wrists. They wanted to
wear it on their wrists. I think wearing something on an ankle is just very weird. Maybe just for
American culture. I'm not sure if that would be applicable anywhere else in the world, but an
ankle bracelet in my mind kind of signifies a probationary kind of thing, but yes, why not?
>>: It could be also that you just need more force to detect the signal, so at the ankle, you may
not see that large enough force, whereas on your trunk or on your risk or torso, you can see that
bigger ->> Shomir Chaudhuri: There were definitely studies that had ankle devices in combination with
the waist device, so there is work out there. From my focus group studies, nobody recommended
anything near the feet. That would be a good area to explore, though. I'm going to get you in in
one second.
>>: So the conclusions that I have from your talk is that you shouldn't incur a new device for fall
detection. You should find devices that they already use and just if somebody is using
something, his belt all the time, so you can adapt it to his belt. Actually, when I'm coming to
think about it, most of the people that you talk about will have glasses, right? So fall detection
using something like Google Glasses would be really trivial.
>> Shomir Chaudhuri: Really?
>>: So that's -- you shouldn't get them a new device all told, because that's basically what I'm
thinking.
>> Shomir Chaudhuri: Yes, and I think that's a really good -- if we could just make a shirt that
people wear or glasses like that, that would be great. But I guess what I'm trying to do is get to
the broader range of people who maybe don't wear glasses. I don't think a lot of people wear
glasses.
>>: How many people in those ages do not wear glasses?
>> Shomir Chaudhuri: I'm sure there's a good percentage. I just don't know the numbers, but
that'd be really interesting to find out. And there have been studies where they tried to just do a
vest or something that they'd have to wear underneath, and people like them. It gives some
complications of having to wash them and things like that. I think you're on the right track,
though. Already everyday-used things are what we need to be working on, along with a standard
fall-detection device for people that may not use glasses.
>>: I think there's usually a startle response when somebody falls, and I'm wondering if you had
some kind of way of detecting physiologic changes in that person, you could maybe think about
decreasing the number of false alarms?
>> Shomir Chaudhuri: Yes. There were very few devices that worked anywhere on the hands,
but there was one device that -- I can't remember exactly how it worked, but it looked at
neurological signals to see the sort of alarm that went off within the person's body, as well as
accelerometer.
>>: The Band-Aid situation, where the thing is actually held against somebody's skin, there are
simple -- just a change in electrical conductivity, for example.
>> Shomir Chaudhuri: Exactly. And the device that I'm prototyping, they also pick up five
seconds of audio, which if somebody's falling, they might yelp. That's really interesting, and I
think we're going to get to a point where we have too much data, and we need to figure out
what's most useful, but I think right now, yes, ideas like this are great. Did you have something?
>>: Yes, great talk. I was floored about the number of studies that actually use older adults to
validate the devices. It's quite low. From your experience, it sounds like you're getting a lot of
false positives, but those numbers, the sensitivity and specificity are up in the 90s, so what did
you think about -- why are these devices not really testing them out and -- I guess they're not
subject to regulation.
>> Shomir Chaudhuri: So the literature review wasn't on commercial projects, so they weren't
device managers so much as they were research labs, so I guess the regulation there was less.
But I think the thought is this is still a field that needs improvement. So I don't think that people
want to take the risk of putting it on someone in the field and have them fall with the device and
it not activate. So right now, I think lab tests are much easier, and you can get a lot more people
and a lot more data right away. I think people are going to the real-world tests, but real-world
tests are so difficult. There was one study where this person had 15 subjects on the device for it
must have been six months, seven months, and they got zero falls.
>>: That's a rarity.
>> Shomir Chaudhuri: Yes, that's a lot of -- what's the word I'm looking for? That's a lot of
money to put into something that doesn't reveal any kind of data. So for right now, just getting
the most data you can on 20 people falling, or 20 falls from one person, I think it's valuable. And
I'm not saying people need to just throw their devices out in the real world. I think we're getting
closer and closer to the point where real-world tests are going to be needed and usability will be
understood. Did I answer your question? I kind of -- sorry. Sorry.
>>: Yes, I thought it was really interesting. I actually saw one paper maybe last year where they
took acceleration data, which were recorded off some high-risk participant, and evaluated 13 or
14 different algorithms on it, and they compared the results on that with what was published.
You might try to find it. It was really interesting. But my question was, of all the ones that were
tested in the real world, were they all accelerometer-based sensors?
>> Shomir Chaudhuri: So there's the neurological sensor. They're mostly accelerometer based.
Some had magnetometers. I'm trying to think. Because, obviously, the environmental ones were
not accelerometer based. Let me get back to you. I can go look at the papers, or I can send you
my literature review, if you'd like.
>>: That'd be great.
>> Shomir Chaudhuri: Maybe we can talk at lunch, you can give me your card, or I can give
you mind.
>>: Getting back to the glasses, I think that's a good idea. If you think about mechanisms, so
getting back to the startle response, where people when they have a startle response, I think their
orbicularis oculi muscle also twitches. So if you can capture that, and then it goes back to the
startle and then coming back to the falls, I think that makes an interesting trajectory to study and
see how they can -- in terms of getting it for prevention, and that will be very interesting to see
how that happens, if they're wearing the glasses which can capture the muscle response of that
small muscle in the eye.
>> Shomir Chaudhuri: Absolutely, yes, that'd be really interesting. And in terms of fall
prevention, I think we were talking about maybe having the Kinect study gait patterns of people,
and if it's a longitudinal study, maybe see what happens to someone right before they fall. And
there's definitely already studies that do that, but maybe there's also neurological patterns that
predict a fall. I'm really sad that I took out fall risk from my literature review, but at some point,
it was just too many papers, so we had to cut it down.
>>: I did a small study of my own, a user study with my parents, so it's a small sample.
>> Shomir Chaudhuri: But the best sample.
>>: My father made an interesting point. He said 100 or 150 years ago, everyone used to walk
with a cane, and nowadays, no one uses a cane, and my parents don't use a cane, and they are
getting close to their 80s. He said the reason that he doesn't use a cane is because I'm not using a
cane. Back then, it was the fashion, everyone, people in our own age will use it and it was a
fashion thing, and you had the special carving and everything, and therefore, even as you get
older, you would use it. Now, once you have to use it, it's kind of a statement. I'm old, I'm
paralyzed or something like that. So I think one of the things -- one direction towards making
these devices more useful is hiding them, but the other thing is making them popular and not
necessarily just with the audience of people at risk. So a phone is such an example. If I carry a
phone and it has also the ability to detect falls, that's not a statement, right? If someone has a
phone, he has a phone, right? And it's a cool phone, whereas if you carry this thing on your neck
that only people that -- it doesn't cut it. So just for the fun of it, we had a kind of a brainstorming
session, how do you make a cane a fashion device, something that you'll be cool to have one.
>>: So, actually, let's take the glasses idea one step further. It can also act as a prevention
device, if it has a camera, right, so if this camera can be made to identify that you are now rolling
down staircases or something like that, it can warn you or it can make something that will make
you take greater care about what is going on. That's the kind of thing that can help prevent falls.
>> Shomir Chaudhuri: It would be really interesting. I think it definitely requires some kind of
cultural shift. Right now, most people that see the fall-protection device think about the Life
Alert ad, where the lady has fallen and can't get up, and it's almost become sort of a comedy
thing. And it's a really serious problem, and I think industry researchers, we really need to start
saying, look, this will help your independence. This is not here to restrict you in any way. It's
not here to victimize you, but yes, I'm just blabbering now. But it really requires some kind of
cultural shift and new innovations like that that will just make it fade into the background that
will help us identify falls. Yes.
>>: This is kind of a silly idea, but one of the things with studies like this is compliance, and in
some of the larger nursing homes or retirement communities now, which would be your
population target, I would think, they have benevolence funds. In other words, what they do is,
they build up money so that people who run out of money before they retire -- I mean, before
they die -- are able to stay in the place. And it's a pretty high cultural value, at least in the people
that I know that live in those places. And maybe if you rewarded the benevolence fund for the
number of people that participated and stayed compliant with your study, it would take the study
away from being about the person wearing the device and make it more noble, make it more
acceptable socially, they're actually doing something beneficial. It might be a way to motivate
them.
>> Shomir Chaudhuri: Like a Livestrong band almost, that kind of thing. I'm wearing this. It
means I'm contributing to your -- absolutely, absolutely. Yes. That's a great motivator. I think
we're -- I'd love to work on anyone with any kind of project that's related to this, so please come
talk to me, and thank you so much.