>> Vani Mandava: Hello and good afternoon. Welcome,
everyone. I'm pleased to welcome back here
University of Washington Center for Data Science
Team. We in the Microsoft Research Outreach Team
have been working with them for the past couple of
years on some other projects, as well.
We worked on CRONUS, Zoom, and Academic Search, and
more recently as part of the Azure for Research
program, they were given one of the awards for Azure
for Research, and they're here today to talk about
their work on predicting congestive heart failure
risk of readmission that they're collaborating with
MultiCare Hospital, which is a hospital in the South
Sound region. So there's several of the team over
here, and without further delay, I would welcome them
and ask them to come up and talk to us. Thank you.
>> Ankur Teredesai: Thank you, Vani. My name is
Ankur Teredesai. I'm a professor at the Institute of
Technology in US area, Washington, Tacoma, and I also
manage the Center for Data Science and its
activities.
It's a pleasure to come to Microsoft Research and
give a talk about an extremely important, if not
exciting, area that we are working on, which is
predicting congestive heart failure, risk of
readmission. I will start with a quote and let you
guys guess who might have said it: "When the cost of
health care devastates families and threatens to
bankrupt our enterprises, great and small," this is a
piece of an inaugural address by a president of the
United States of America. Any takers in the local
crowd here on who might have said it?
>>:
Obama?
>> Ankur Teredesai:
>>:
Huh?
So --
He's still trying to fix it, so --
>> Ankur Teredesai: Yeah, so we would all -- the
obvious sort of choice would be Obama, but it's not.
It's actually 1993 inaugural address by Bill Clinton.
And Bill Clinton was one of the first presidents to
mention what -- actually the first president to ever
mention the importance of health care and issues in
health care in an inaugural address. Well, that's as
far as addresses go, but something has to be done
about, you know, health care in our country and we
are all quite well aware of it.
Specifically our focus at the center and through this
initiative of Azure for Research has been to take the
mandates of Affordable Care Act, which translate into
affordable or avoidable costs. And what is important
for us to understand, that readmissions are
avoidable.
Readmissions are, as they're popularly known, as
rehospitalizations, is a term that is used frequently
within the health care community for explaining a
milieu of incidents that happen if the same patient
with one condition gets readmitted again to any
hospital system, they are considered largely
avoidable for chronic conditions. In particular,
30-day admissions contribute to the -- amongst all
readmissions that happen for patients, 20 percent of
them happen within 30 days, and that is the reason
why risk of readmission within 30 days is a very,
very important factor for affordable care.
The second important piece of data on this slide is
that 25 percent out of all readmissions are for
congestive heart failure alone. That means if we
attack this specific problem carefully, then we can
solve a big piece of the puzzle.
Today we'll be talking about several components. I
have several colleagues with me. I will first
present the problem. I will explain what is
predicting of risk of rehospitalization/admission,
how do we propose it as a machine-learning problem,
what did we do about it.
Then we will go deeper into the anatomy of a risk of
readmission model. We will look at what is the
state-of-the-art model looking like today. That will
be done by my colleagues, Senjuti Basu Roy. She will
also explain why it is important not to just predict
the re -- predict whether readmission will happen or
not.
Think about it. It's actually more important from a
patient perspective to predict or to understand what
will avoid the patient from coming back and what sort
of intervention should the clinical care team or the
protocol be followed inside a hospital system in
order to avoid readmissions in the first place
instead of just simply predicting the risk, which may
or may not be actionable. So Senjuti will talk about
actionable interventions.
From there on we will go on a journey to quickly see
what has Azure for Research and this award enabled us
to do which we otherwise would not have been able to
do in such a short amount of time. And we will show
you how our hard work has been accelerated by the
Azure for Research award. We will give you a neat
demo of a deployment that we have actually made in
the MultiCare Health System.
Now, if you think about it, academics are not usually
supposed to deploy anything. We are supposed to
prototype stuff, write some papers about it, and let
it go. But we are very proud to say that the models
that we have actually built are being used to make
clinical decisions in a real-world health care
system, and we feel that that's pretty cool. So we
are here to talk about it.
And towards the end I'll just present a very, very
brief -- I won't take much of your time between Q&A
and all the great work that we have been doing
through this award. Very lightly I will touch upon
our vision for 2015 and what we think is a great
opportunity for cloud-based health care machine
learning.
So congestive heart failure is a chronic condition.
It affects mostly elderly population, but the goal of
the research, somewhere in March 2012, somewhere
around two and a half years ago, we were asked this
question by Dr. Les Reed, who is here in this
picture, can you guys figure out if there is a
protective model that can be built for reducing the
risk of readmission, okay?
And we said, well, in order to build a predictive
model to reduce the risk, maybe we need to first
predict the risk. So we will set it up as a binary
classification problem. We will say will this
patient who has been diagnosed with congestive heart
failure come back within 30 days or not come back
within 30 days. So it's a binary classification
problem.
So as is classic in all academia, three of us who are
assistant professors and associate professors and
researchers set up this problem for our post-docs
[indiscernible] and for our students, and together as
a team we said we can solve this.
Just for those not initiated in machine learning,
here is a very simple flow of how all classification
in machine learning works. Essentially you have a
bunch of patient records. You extract features out
of those records. You also have with each patient
available a class label, readmission or no
readmission.
You train your machine-learning algorithms to look
into that data and features, and out comes a
predictive model. That predictive model can then be
used on a new patient by extracting the feature
vectors again from that new patient, and you assign a
new class label, readmission or no readmission.
So that's pretty simple. Top half tells you how to
build the model. Bottom half tells you how to score
the model or scoring a tuple. Now, remember, the
only reason I bring this up, and it's good to
remember, is today in our deployments on Azure for
Research, this is where it's helping us the most, in
scoring millions and millions and 1monies of events
on the cloud.
We are still doing the machine learning, the building
of the models on traditional, you know, single node
compute infrastructure, but that's where we want to
go next. So when I talk about the end of the
presentation and our slide on our plans, we will talk
about how we want to take the entire process to the
cloud.
And of course, what I said in the previous slide was
here is a nice machine-learning problem.
Unfortunately, the data is not so nice. It's, you
know, health care data. It's extremely messy.
The bulk of the problem in health care is the
complexity of the data, itself, and it took us, as
machine-learning scientists, a huge amount of time to
even extract and understand what congestive heart
failure means and how is it different from cardiac
heart failure, for example. Right?
And how do you extract -- I mean, within MultiCare,
there are 16 data marks, 200 views within each of
them. It's extremely difficult to get all that
information out of transactional systems.
Yes, question.
>>: [inaudible] on previous slide was there any
retraining of the model once you get new data?
>> Ankur Teredesai:
Yes.
>>: Okay. And then second question on this slide,
why would the HIPAA policy act like you guys ran into
any issues for ->> Ankur Teredesai: Very good question. Thank you.
I -- you know, we are -- anyone who touches the data
within the MultiCare Health System is HIPAA
certified. So we had to go through the training. We
had to sign a bunch of forms. It takes three weeks.
So if we onboard a student on to our team today, it
takes us almost three, three and a half weeks to get
them going with the data sets, yeah.
Not just that. We all live in a world of big data,
and I have to mention the word big data somewhere in
my presentation, so this is where it is, and, you
know, out of the three we's of popularly known as big
data, we have for the last three years dealt with
variety and volume, and this is a typical, and we
have not even added multiple other factors, but this
is a typical set of attributes that you are dealing
with when you say congestive heart failure risk of
readmission.
Most of our competitors and most of the other papers
that were published in this space before we started
this work dealt with one or the other siloed
machine-learning framework. So you would have a
model which was purely clinical. It would take a
look at only at the vitals of a patient and predict
the risk of readmission.
What is unique about our solution is that it's
integrative. It takes all these factors into account
and figures out, you know, in a hierarchal multilayer
approach, which Senjuti will talk about, to build a
predictive model.
Initial models were horrible. We spent a lot of
time. So by March 2012 to December 2012 we were just
getting to understand some of this space. We thought
we were great machine-learning people.
Unfortunately, or fortunately, we were humbled by the
amount of rejections we got from the academic
community. Every single paper we sent for almost a
year just got rejected. Nobody wanted to read it, or
at least nobody believed that this could be done in
this way.
Initial modeling [indiscernible] were also not that
exciting. There were other models which were better
than ours, but for some reason or the other,
MultiCare and our sponsors kept faith in us. So
thanks for that and a shout out to MultiCare. Yes.
>>: What is the feature size that we are looking at
and what is the data size when you mentioned big data
on the model?
>> Ankur Teredesai: Yeah. So in terms of
attributes, we have several variety of models. We
built some very simple models for patients to
understand and interact with. Those are like
10-attribute models. For clinicians, there are, you
know, 50-attribute models. We have 64 attribute
models.
And within those attributes, the reason that these
the reason that this attribute set is more than 64
because we also do some feature reconstruction, so
do reduce features and combine them and do a bunch
-is
we
of
things. But our biggest -- or I shouldn't say the
biggest, but the biggest big data model that we have
currently is a 64-attribute model.
>>: And what is the size of the dataset when you
train the mark?
>> Ankur Teredesai: So we have patient records for
the last three years, yeah. So on average you can
think of it like almost 40, 50,000 patients, all the
events, so ->>: And then when you do retraining, you do three a
window or ->> Ankur Teredesai: It depends on which of the
models you are trying to retrain. But we'll go into
the details of that. Okay. Thank you.
Okay. So finally in July 2013, on a nice sunny day,
though it was raining in Seattle, we did manage to
get a really good model. And of course, that led to
some very nice papers. Our current result is around
74 percent, 80 under the [indiscernible] accuracy for
the risk of readmission, and that's the model that we
are using to show you, you know, the demo, and that's
what's kind of deployed on Azure and so on. Okay.
>>:
[inaudible] accuracy, like what is the --
>> Ankur Teredesai: Yeah, we'll go into -- again,
you are just seeing the very, like 36,000 feet. We
are going to go into the deep and we'll explain how
we compute it, yes.
So the next question was formal health care, is this
a prototype, is this academic, or is this a product
that we need to, you know, build and ship and deploy.
And yes, some more papers.
So with that, I will hand off to Professor Senjuti
Basu Roy to take over and go deeper into the anatomy.
Thank you.
>> Senjuti Basu Roy: Thanks, Ankur.
everybody for attending the talk.
And thanks,
So as Ankur mentioned that our initial effort led us
nowhere, so in our post model, we started thinking
and digging deep, what went wrong, what we did not do
right.
And we realized that we are rather treating a very
specific problem in a very generalized way, which is
not fair and right. So instead of just doing the
integration and building a predictive model based on
off-the-shelf machine-learning tools, we started
investing a lot of energy and effort to the
pre-processing step which should go prior to the
predictive modeling. In particular, to the data
exploration and data pre-processing steps.
At the same time, we retweaked ourselves in the data
modeling, predictive modeling effort. We no longer
want to use the off-the-shelf machine-learning tools
like Naive Bayes model or sub [indiscernible] vector
machine, but we want to design our own predictive
model which is fair and appropriate for our specific
problem.
And in the next few minutes, I'm going to be touching
deep into some of those details. For brevity, I will
be skipping a lot as well, so please feel free to
stop me if you have any specific questions.
So one of our effort, as I mentioned, was significant
energy and effort was given to the exploration. The
objective of this tape is to understand different
variables. Many of these are numeric variables and
significantly really noisy, and how they contribute
to risk of readmission.
So we realize that it is not adequate for us just to
draw a graph and try to understand how this
particular variable relates to risk of readmission.
That's not fair. Because the variable, itself,
contain noise.
So as opposed to using the numeric variable as is, we
started discretizing the variable and we resorted to
statistical technique for discretization. For
example, in one of our effort we adopted the
Chi-Merge algorithm to discretize numeric variable to
discretize variable, and we wanted to visualize the
effect of discretization using odds ratio.
Obviously all of this goes nowhere unless we see that
these efforts leads to better predictive modeling,
and as you can see in the results of these two
following papers, we indeed observed that with data
exploration, with discretization based exploration,
we actually ended up predicting the risk of
readmission problem better. So that was one of our
successful process.
At the same time, we also re-tweaked ourselves a lot
in the predictive modeling. In addition to the
off-the-shelf predictive models like Naive Bayes,
logistic regression, or SVM, we started looking at
other techniques. We started investigating whether
existing ensemble methods like Random Forest or
AdaBoost gives us anything.
At the same time, we started designing our own
ensemble, and we started designing another very
interesting predictive model, we call it multilayer
classification. In the next few minutes in the talk,
I'm going to be talking in a greater detail about our
multilayer classification method.
>>: So question. Is it Random Forest, is it
gradient boosted trees?
>> Senjuti Basu Roy: Random Forest, yes, so gradient
boosted tree you can think of, that was one of the
model that we used for Random Forest, that's right.
>>:
And which [inaudible] fast train?
>> Senjuti Basu Roy:
right.
We used fast train, that's
So here, so I'm going to be talking a little more
little depth, little in depth about multilayer
classification, and the reason for choosing
multilayer classification was motivated from the
data, itself.
So when we took a deeper look into our data set, we
observed that the population of the patient that we
are dealing with is very, very heterogenous, and we
are not doing a right job of treating all of
similarly.
For example, we saw that only 32 percent of the
patient in our population ever gets readmitted. And
out of this 32 person, only five person comes back
within 30 days of readmission. So why treating all
of them in the same bucket? And that is really the
intuition of the designing our multilayer
classification.
So this is what we wanted to do. We wanted to
automatically design a hierarchal prediction model.
Then for each prediction, each layer, each layer in
this hierarchy, we wanted to design a predictive
model and we wanted to apply our appropriate
pre-processing steps such as feature selection or,
you know, factor analysis, whatever you name it.
Also, as you mentioned one from the audience, that
the objective of each of these layer is different.
In some layer we want to maximize recall. In some
other layer we want to maximize AUC or precision. So
that sort of is the intuition of our better break to
model. Okay.
>>: Is the pictures, are they [inaudible] or you can
also vary ->> Senjuti Basu Roy: They will also vary across the
layers, that's right. So probably this picture
clearly depicts what we want, intended to do. So the
root node of this picture is our original population,
which is all patient. From this entire set of
patients, we first wanted to filter out those who are
not coming back, and that's these branch of the tree.
Yes.
>>: So when you say not coming back, what kind of
data you have? Like you have data from the same
hospital, like the patient [inaudible] at the
hospital or maybe the patient died ->> Senjuti Basu Roy: So, yes, very good question and
I'm sorry that I haven't described the data set very
well yet. So, yes. So every -- the instance of a
data set is an encounter of a patient to hospital,
okay? And although the data is [indiscernible], we
know there is a unique ID given to each patient
record. So we know who came back and when. And of
course, every encountered ID is associated with a
data and time. So from there, we can calculate when
he got discharged, whether he got readmitted, and the
day between.
>>:
[inaudible].
>> Senjuti Basu Roy: To any hospital, to the four
hospital that MultiCare health system has.
>>:
Within the system?
>> Senjuti Basu Roy:
>>:
Within the system.
Within the system.
>> Senjuti Basu Roy:
Within the system.
>>: Do you have any information on patient data,
like the patient days and all that sort of
information [inaudible]?
>> Ankur Teredesai:
>> Senjuti Basu Roy:
didn't hear you.
Yes, we do.
What was the question?
I
>> Demographic.
>> Senjuti Basu Roy: Demography. Yes. So
psychosocial factor was one of the attributes, that
big list that you saw. So our mixed bag, that was
the set of factors that we used. Yes.
>>: It was [inaudible]. It's supposed to be
[inaudible]. Somehow [inaudible].
>> Ankur Teredesai: You can think of it as a team.
So if you throw a lot of data, this contains a lot of
patients who do not -- who never get readmitted, then
your classifier gets usually biased against the
second class state. So exporting time, you're still
going to predict readmit or not readmit, and you're
going to measure your accuracy only at that time.
>>:
[inaudible]
>> Senjuti Basu Roy: So readmitted, remember, we are
staying true our original problem definition, which
is whether the patient is going to be readmitted
within 30 days or not, okay?
>>:
[inaudible]
>> Senjuti Basu Roy: The problem we encountered if
we do not read this population differently is there
is a huge class imbalance, as I said, that only five
percent people gets readmitted within 30 days and
rest 95 percent don't.
So it is not fair to treat them -- to put them in the
same bag and just build something out of it. Rather,
we started seeing that -- we started treating this
different strata, as we call it, differently, and we
started applying different method and different
factors that are responsible for that particular
strata rather than treating all of the them together
at the same time.
And I can answer few more question offline maybe.
Anyway, so I still want to spend a few more minutes
on this slide because this sort of derives and leads
me to the next slide.
So this is what was our intention. We wanted to
treat the entire population in the beginning. We
wanted to filter out a set of people who don't come
back. Then our objective was to design a set of
internal layers, which are hierarchically related to
each other. And obviously, the lowest layer in our
system is the problem that we actually solved,
whether the person is going to be readmitted within
30 days or not.
So that was sort of our high level motivation.
So if you notice this slide, I just said that I
wanted to design an internal stage. Each internal
stage describes a time window of readmission. So the
obvious question that leads to, how do I decide those
time windows. Am I going to manually and in an ad
hoc way select those time window? Am I going to
consult the physician to select this windows?
am I going to do?
What
So this leads to this simple example that I'm going
to describe to [indiscernible] the settings. So
imagine that my objective for now is to create just
one single intermediate layer. So from my original
population of patients, I have eliminated everybody
who doesn't come back. Then I have calculated the
maximum number of day until readmission. For sake of
simplicity, let's assume that day is 100.
Then I am actually interested to predict whether a
person is going to come back within 30 days or not,
and I just said I want to create an intermediate
layer. That means I want to find out another day and
I want to create three layer where I will design
three different model, one for the patients who had
got readmitted between that K day to 100 day, K day
to 30 day, and less than 30 day. I hope I am making
myself clear. Okay.
So the question is how am I going to come up with
that K? And this is the simple grading algorithm
that we use to come up with that K. This algorithm
uses a very simple and classical intuition of strata,
meaning people who belongs to a strata, two different
strata should be significantly different from each
other. So that sort of our intuition of creating
this strata.
To do that, we write all this pseudo code, and I can
very -- I can easily explain it to you without
telling all of this what we are doing inside. What
we are doing inside is basically we are rounding a
grading process which is rippling through this entire
spectrum of possibility between hundred and 30 and
trying to find out that K value which stands for a
day which will create two bucket or two strata of
population which exhibit the highest difference
between each other.
In order to
each bucket
between two
compute the
compute the
do that, what did we do? We represented
as a probability distribution, okay? And
probability distribution, we wanted to
difference and we used KL divergence to
difference. We kept repeating this loop
until we find the point where we can converge, and
that's basically this pseudo code.
So these process creates, for the simple example of
three layers, this is the process that we adopt to
automatically generate this intermediate layers so
that -- yes.
>>: I'm just wondering, if I drew a [inaudible]
model [indiscernible], so I move one day ahead?
>> Senjuti Basu Roy: You can move one day ahead.
You can do ten day ahead. Depends on how you want to
move.
>>:
I see.
>> Senjuti Basu Roy:
pseudo code.
So that's our "i" in this
>>:
I see.
>>:
And how do you move [inaudible]?
>> Senjuti Basu Roy: Our experimentation moved in
different ways. We moved in five days, we moved in
ten days, and we moved in 15 days, yes.
>>: So the "i" is a fixed number rather than an
adaptive number?
>> Senjuti Basu Roy: "I" is a fixed number rather
than an adaptive number, that's right.
Okay. So my stratas are created. For each strata I
do factor analysis, I do all pre-processing in the
world that I want to do. I have built my model,
okay? But then I need to come up with a probability
of the patient being readmitted within 30 days.
How we do that? I'm not going to go to this
equation, but I'm going to use this picture to
describe that. So for the simple three-layer
example, the probability, the innermost box describes
the probability of a patient being readmitted within
30 days, given he got readmitted within that day a
layer above.
Similarly, the probability of the patient got
readmitted within K days, which is the layer above,
is conditioned on the probability that the patient at
all got readmitted. So this is how we compute the
entire probability, and that is what is summarized
here in the equation.
So when we showed this to MultiCare and, of course,
it came with better results, MultiCare was impressed
but not fully convinced. There were several baseline
that was going inside MultiCare system, and of
course, we had to build our baseline in order to
convince them.
So this is one of the very simple baseline that
MultiCare used. They call it a care management tool.
And as you can see, these are the ten or 12 factors
that use -- they use to score a patient in one of
these four categories, low, medium, high, or
intensive.
All these factors are actually psychosocial, whereas
we can consider a wide vast range of characteristics.
And obviously, we built them up easily and fast, and,
but we still have to compare ourselves to get
management tool every day we need. Believe me,
believe it or not. That's just the fact of the life.
>>: It's not so much the fact of the life. It's
also important to note that most hospital systems
rely on such simple computational tools that can be
derived, and in order to deploy a product or a lot of
them that with equal system, you have to convince
them that what you have is 20 times better than this,
and only then will they buy into that. So ->>: And who builds or imply that they should be able
to understand the decision?
>>:
Yes.
>> Senjuti Basu Roy:
>>:
Absolutely.
Of course.
I mean --
>> Senjuti Basu Roy:
You have no idea how many times
I have said what is the definition of precision and
what is the definition of recall.
>>: But I mean, understand
okay, this patient would be
probabilities for research,
them based on which feature
>> Senjuti Basu Roy:
>>:
like if you say that,
readmitted with the
that you can also show
--
Yeah, yeah, yeah.
[inaudible]
>> Senjuti Basu Roy: Yeah, they are very interested.
Actually, since they are the domain expert, they are
more interested to know why than how. They don't
care about the how. They really wanted to see why.
Okay. So this is the royal moment for us. All of
them came in a single package. When we designed our
prototype, which is now inside MultiCare Health
System, we call it Risk-O-Meter. Many of us call it
Risk-O-Meter. And it got accepted as a demonstration
paper in KDD 2013.
Risk-O-Meter has a wave-based UI, and it also has
a -- one of our students is going to demonstrate the
mobile app on Azure based on this application. It
has a very simple UI. You can see that even though
our actual model has 60 attributes or 64 attributes,
the UI accepts only ten or 12 of them. Even a user
can leave some of them blank.
That doesn't mean the model inside is going to use
only eight attributes if the user has entered eight
attributes. It's actually going to infer or impute
the rest of the attribute values that are not
entered.
So given this patient input, when once somebody hit
this button, calculate risk, Risk-O-Meter is going to
predict a real-time risk for this real-time 30-day
readmission risk for that patient.
And that's not the end of the story. At the same
time, it has two more interesting characteristics.
One is it is going to show the top associated risk
factor of this patient based on the input that is
provided in the UI. All right? And this is done -technically speaking, this is done by performing
associate rule mining and we have adapted a variant
of A Priori algorithm to do that.
Then, and more importantly, which was very well
received by the physician, Risk-O-Meter also tells
top three most important factor that the patient is
going to exhibit but hasn't entered to the UI as
input.
So it tries to discover hidden pattern, which was
very well received and excited by the physicians
because it alerts them some of the other
comorbidities or secondary diseases that that patient
have and the physician is not yet aware of.
So this is what Risk-O-Meter is, and Vivek is
actually will show the live demonstration of the
system.
So this is just some characteristics of Risk-O-Meter
which I already have explained.
So from then, all of this happened like eight months
ago. So what next? We have been wondering what
next. Should we just confine ourselves to risk
prediction? And obviously one immediate step that we
have started thinking that only prediction is not
adequate. We need to enable the physician support if
we really are interested to invest our energy in this
space.
So the last eight months we have been dealing to come
up with personalized intervention recommendation for
the patient. Intervention means a set of procedure
that the patient can adopt to minimize his
readmission risk. Obviously our objective is not to
replace the physician but to help them and aid them
in the decision support to improve the quality of
care, okay?
And this is our royal patient that we drew eight
months ago. So basically our idea is not to do
readmission prediction, but also to do readmission
intervention, predict intervention recommendation,
not only during hospitalization but any phase of the
patient life cycle, during pre-hospitalization phase,
during hospitalization, and post-discharge.
We call this intervention recommendation framework
because the solution that we design for this is
agnostic to the specific phase that the patient is
at. Depending on the appropriate variable that is
entered to that phase, the system will be able to
predict a set of -- best set of rules for this
patient to minimize his readmission risk.
So our effort in this space is now more on
intervention recommendation rather than risk
prediction.
And very briefly for the interest of time, I will be
going fast on this set of slides, the remaining set
of slides which is going to explain how we are doing
intervention recommendation, and this is our
architecture.
So we have formalized the intervention recommendation
problem as a network learning problem. We first -given the available set of data, we first learn
innate work which describes the complex interplay
between the multitude of factors that causes
readmission.
Then once this network is constructed, we do the
parameter learning to design the probability
distribution table on each node of this constructed
network.
From there, we generate a set of recommendation rule.
All of these are part of the pre-processing state and
is done Apriori.
Once a new patient comes, given a set of
characteristics, we pass him through with the set of
rules that are generated and we predict the rule. We
generate the rule which is most likely to minimize,
which is likely to minimize his risk maximum.
So that's the overall process. And now very, very
briefly, this is how it looks like. So this is the
output of one of our algorithm for the network
learning phase. So note that there are several
factors associated. The top are the demographic
factors; the second layer are the diagnoses; the
third layer are the procedures. We call them
interventions. And then how they relate to
readmission risk. These areas are being learned by
learning a Bayesian network, and we use several set
of algorithms like constrained based approached,
score-based approach, and hybrid approach to do our
experimentation.
>>:
[inaudible] network or --
>> Senjuti Basu Roy: No, we actually have used -- we
have used R and some of it we have coded ourselves.
So once this network is constructed, our job is to do
the parameter learning, and again, I'll be super fast
on this slide.
The objective is to create the probability
distribution table at each node of this constructed
network. We use Bayesian parameter estimation for
this purpose, and use MAP, Maximum A Posteriori, to
estimate the posterior probability.
And finally, once this network is created, the
probability distribution is generated, we create a
set of rules for -- based on our given data set.
Basically we want to see, given a set of
recommendations and for a patient who was [inaudible]
readmitted, if the set of recommendation reduces the
probability of readmission farther or not. If it
does, then we suggest that as a set of rule and we
store that in your database. Simple and clean. The
same could be done for the patients who got
readmitted and see how the recommendation rule for
him could be generated.
So that's sort of the brief overview of the large
processes that we followed over a period of two and a
half years. And I'm going to stop at this moment and
I'm going to hand it over to David and Vivek who are
going to talk about the engineering effort, and at
this point or later on I'm happy to answer any
specific question you may have on the machine
learning part. Thank you very much.
>> Vivek R. Rao: Hello, everyone. I'm Vivek. I'm
graduate student at University of Washington, Tacoma.
Today I'll be showing you the demo of Risk-O-Meter
app. Basically it's a clone of web UI, but it's
deployed on Windows Phone.
Presently we have an Android version, Windows Phone
version, and presently we are supporting two kinds of
profiles. One is the patient profile and another is
the doctor or physician profile. Today I'll be
showing you the demo of Windows Phone Risk-O-Meter
app and it is optimized for Windows Phone 8.1.
This is the screen shot or a screen share of my
phone. As you can see, here is the Risk-O-Meter app.
For the patient profile, let me take an example.
Mike. Mike is a very good friend of mine. His
father was admitted to hospital because of congestive
heart failure.
A couple of days back he was discharged, but Mike is
worried about his father being readmitted to the
hospital. So I said, Mike, hey, why don't you check
our app. It gives the probability of readmission to
the hospital.
So he went and checked this app. He opened this app.
In the welcome screen, you have a brief introduction
about the app, what it does, what are its
capabilities. As we are talking about the patient
profile, he goes to the patient profile and clicks
the start button.
In this UI, we are asking Mike about several inputs
or attributes of his father. Mike's father is 43
years old. He's a male. He belongs to
African-American community, and pre-hospitalizational
heart failure. Yes, he was admitted to hospital
because of heart failure. So I'll click yes. It
asks for period length of stay. He say a stay day in
the hospital for seven days. He has diabetes, but no
stroke. He's BP category is 201 and 219.
Mike's father is a pretty normal guy, so his BMI
would be around 25. And ejection fraction is 15.
Ejection fraction value, or EFV, is the volume of
blood which is pumped out of the heart every -- for
every heartbeat. Once he calculated this --
>>:
[inaudible]
>> Vivek R. Rao:
>>: No, no.
for that?
It's representing the report card.
I mean, how would they know the value
>> Ankur Teredesai: Yeah, so typically, you know, we
don't expect ejection fraction to be there. So the
model handles missing values and that's where
[inaudible].
>>:
So they can skip?
>> Ankur Teredesai: Yeah, I think, you know, we are
trying to really do the trial version.
>> Vivek R. Rao: So when I hit the calculate this
button, it takes all the attributes which Mike
entered and forms a post request and hits the API
sitting on the Azure.
Azure API takes the value and calculates the risk
profile and sends it back in a JSON format. As you
can see, in the results page it says that probability
of readmission is 37.79 percent and it's in a low.
It means that it's good, but not great.
And along with the risk, we give some of the
additional factors, like what are the top associated
risk factors. It says his BMI is less than 30.
Second is the male, and third is the diabetes. So
Mike's farther is 43 years old and he's a male with a
diabetes. So the risk score is around 37.79.
>>: I have a question. So when you display the risk
factors, does it tell you as to what's left or right?
It's showing you have a lower risk or a higher risk?
>> Vivek R. Rao:
Yeah.
>> Ankur Teredesai:
factors.
>>:
As you can see --
These are the top associated
Yes.
>> Ankur Teredesai:
So the idea is that given you or
patient at the instance, what are the other closest
instances and what are the top factors for it.
Essentially you're using a first stream model to find
the -- you take the patient, you put him or her in
one of the clusters, and then you say what are the
top five factors for that cluster.
>>: Yeah, I know that's a risk factor, but like for
me, I don't know what is a BMI. Is it less than 30
means I have higher risk of this disease or lower
risk?
>> Senjuti Basu Roy: So let me answer this. So I
think that's a good question in the sense that which
is high is very subjective also. I think the doctors
or the physicians may say that the risk of 50 -- 60
percent is high, 37 percent or 40 percent is a
moderate risk, but a patient like me might just, you
know, get annoyed if I see that I have a risk for
more than 20 percent.
So at this point, well, I really do see that what you
are asking is a very interesting question, and maybe
with the explanation with the help of the domain
experts, the physicians, we can provide some
explanations that which will say that if it is 40, is
it something to get alarmed. But we don't have that
yet.
>>: I guess you misunderstand my question, but that
was a really good idea, too. So say the first factor
is BMI larger than 30. What does a less than 30,
what does that mean to me? Is it showing that I
might have a higher risk or I might have a lower
risk?
>> Senjuti Basu Roy: No. So this patient, what they
have entered has a BMI of 25, if you remember.
>>:
Yes.
>> Senjuti Basu Roy: So his BMI was indeed less than
30. So in this case, higher BMI is worse, right? We
want the BMI ->>:
I just don't know that.
>> Senjuti Basu Roy:
BMI stands for what?
>>:
Body mass index.
>> Senjuti Basu Roy:
>>:
Okay. So the idea is higher -Base --
Body mass index.
Okay.
>> Senjuti Basu Roy:
So BMI, higher BMI means worse.
>>: Okay. So it's not like -- I would just suggest
for a user interface that that shows that this is
lowering my risk.
>> Ankur Teredesai:
having, you know ->>:
I think
It's confusing.
>> Ankur Teredesai:
>>:
Completely agree.
Yeah.
So BMI less than 30 is one of the decision --
>> Senjuti Basu Roy:
The decision factor.
>>: The decision factor, but it's not the risk
factor.
>> Ankur Teredesai:
>> Senjuti Basu Roy:
>>:
It's not a risk.
No, it's not a risk factor.
Okay.
>>: So you mentioned you would have some actionable
items, basically, for the patient. Are these the
actionable items?
>> Ankur Teredesai: Yeah, so you can put a filter on
top of these. Right now you're showing all
parameters that we have in the model. But the idea
is that you can put a filter on these based on the
domain expert and say vitals. I mean, there's very
little you can do about vitals, but there is a lot
you can do about sociodemographic factors or about
care management and health. So you can say regular
exercise would be a recommendation as a factor,
right?
Now, what does regular exercise mean is not very
clear to a non-domain expert, but each team will have
their own version of what that implies, so then that
would render the recommendation and the factor. We
are just showing like a very simple filtering
[inaudible].
>>: But so would these actionable items be displayed
here and then in this list?
>> Ankur Teredesai: They could be. Again, we are
not bound to the UI. This is just a demo UI to show
that this is the types of outputs you get from this.
Actual, like clinician and physician are not going to
use this UI. They're all going to custom design
their UI, because different apps will have different
UIs of what is needed. So an end patient app, for
example, will have things like, you know, what he
asked, you know, am I in the normal range of BMI for
this risk or am I in the higher range of BMI for this
risk?
Another app that we have in mind is something like
length of stay app. So, okay. So I am at high risk
for readmission, but, you know, when can I really go
home or when can my grandma go home. Like, so -- but
that's a completely different pivot on the predictive
model. So there will be lots of apps like these in
the system which people will develop on top of this
by just taking the score and the factors. So don't
think of it as just one app.
>>: Once [indiscernible] so you mentioned you had 64
attributes or so. And so this app you have just the
small subset [inaudible]?
>> Ankur Teredesai: Yeah, this is a 10-attribute
model, a 10-attribute model.
>>:
Okay.
>> Vivek R. Rao: So as I said earlier, we have two
profiles. First one was the patient profile. Now we
are going to the physician profile. So I have taken
an example of Dr. Albert. Presently he's at Sea-Tac
Airport heading for a conference in New York. Before
boarding the plane, he just wants to know how his
patients are doing, so he just grabs his mobile and
puts his credentials, and he'll be redirected to a
dashboard where he can monitor all alerts.
When he sees this page, it says that you have three
new alerts and four patients in your list.
Dr. Albert is concerned about the alert, so he taps
on the alerts. Now he gets a view of all of the
patient or alert which are fresh and new.
For example, take an example of John D. It says that
risk or is increased by 15 percent and pulse pressure
is increasing. Now Dr. Albert is worried, so he
wants to get more insights on this patient John D.
So he taps on John D, and it gives a brief or the
broad view of the causes which caused that alert.
It says the time since last change is 34 minutes and
changes EF values low, et cetera, et cetera.
So if the Dr. Albert wants to take some decisions
before boarding the plane, he can just call the
hospital, someone at the hospital and ask them to
take care of John D.
In the dashboard we have another view called monitor
all patients. Presently we have four patients. If
he clicks on the patient list, we'll get a broad
overview of all the patients and how they are doing.
Let's consider William Smith. William Smith has a
risk score of 60, but below 60 percent there is
another value with value 15 percent with the red mark
and an upward arrow mark. This means that 30 minutes
ago there was some event happened with William Smith
which increase his risk score by 15 percent.
So Dr. Albert is interested in William Smith, so he
taps and he gets a more broader view of William
Smith.
And it says top readmission contributors anemia,
pneumonia, cardiac referred failure, and
[indiscernible]. If he clicks the anemia, he get a
graphical view of that patient and all the causes
like the dialysis and the cardio respiratory failure.
Now Albert wants to monitor this patient, so he
clicks on monitor William Smith, and when he clicks
this check box, this William Smith will be added to a
list, and whenever there is a change in that patient,
William Smith, there will be an event and this Dr.
Albert will be notified by this.
Now, Albert is -- Dr. Albert is happy with this -with how his patients are doing, so he signs out and
takes a plane to the New York. Yeah.
>>: [inaudible] take care of the status problems,
suppose I am on the threshold and my levels are
changing pretty quickly. Do you [inaudible]?
>> Vivek R. Rao: I think we can monitor that.
>>:
So that's --
>> Ankur Teredesai: [indiscernible].
explain that. So just be patient.
>> Vivek R. Rao:
So we'll
Thank you.
>> David Hazel: Thanks for that. So I just want to
spend a few minutes sort of talking about the
architecture that's powering the app and the
integration in the MultiCare Systems. If we go back
to the very beginning of the effort, we had a lot of
students trying to build models on their laptops and
sort of, you know, completely underpowered and
dealing with information systems that wasn't really
designed for this modeling integration effort that
they're working on. So the result was the
experiments took a really long time to do, right?
So then we got this amazing email on November 7th,
2013, that said hey, congratulation, Ankur, you have
got your research award grant, which is just
incredible because it really opened up the
availability of infrastructure to do a lot of
experiments, really quickly.
And so, you know, we still
models inside of MultiCare
the full patient data, but
them to Azure and try them
had to build the early
because we're dealing with
then we could then deploy
out on other, you know,
data sets and just score millions of patient events
really, really quickly. Right?
So the result was that the tempo really accelerated,
our results improved, and the physicians started to
trust what we were doing, and that led to this
deployment in MultiCare in early 2014 where we can
now do real-time risk profile and scoring as
represented through the app or moving towards
integration with their EMR effort.
And then we can also compare the results of
MultiCare's patients to other patient populations,
whether that be another ACO or like a regional clinic
where we can sort of understand, you know, different
workloads, different protocols, how they interact to
build better models that then, you know, feed back in
and generate better results for MultiCare. Yes.
>>: So just wondering, how is the [inaudible] result
improvement coming from given that the model training
is still done [indiscernible]?
>> Ankur Teredesai: Very good question, and I'll
address that in my vision.
>> David Hazel: So real brief on the architecture.
I just want to point out we've got our Epic, which is
an EMR over here. HL7 messages send patient data
into a server through the API, you know, hit the
appropriate model, generate a risk profile, which is
the scores and contributing factors going back out
either to Epic or the mobile app, and the important
thing that all of this is running in Azure. The
whole modeling effort, you know, allows us to really
scale up. It's just amazing.
A little bit about the API that's sitting in the
backend. We've got models that are deployed to R
Apache, and those, when the URI comes in, you load
the libraries that we need, you know, hit the R
SCRIPS that we have, and then, you know, generate and
go through the model selector.
All this code is available on GitHub. If you want to
take a deep dive on the code or have conversation,
you know, just drop me a line. Happy to share that
out. So Ankur's just going to spend a few minutes
talking about the vision for next year.
>> Ankur Teredesai: Thanks, David. So where do we
go from here on Azure for Research? If you look at
the problem space, we have realized over the last
three years that it's extremely rich. Just having
access to this wonderful clinical data and the
partnership that we have, which is fairly organic in
itself, is one plus. But more than that, we are
launching something that we call readmission score as
a service which enables any, literally any app
developer to use the service to build apps.
Area hospitals, UDA medicine we are in conversations
trying to see if we can get them to start scoring and
using our models and any other hospitals or
providers, right? That's one.
But not just diagnostic recommendation. We also want
to get ground truth data. So we are, to answer your
question, it's not enough for the -- so, you know,
you have an API token. You use that token. You send
your attributes. You get the score back, but that
doesn't help us, because what happens after 30 days
is equally important in this case, right?
So what we are mandating in this service is you get
the result back today, but then now you are obligated
to send us whether that patient did come back or not
within 30 days. So we will send you a ping. You
send us back an acknowledgment within 30 days if our
ground truth, and that's how we start collecting
ground truth data. Make sense? That's how you
complete the loop. Of course, we want to integrate
with other sources, you know, with Azure. That's the
key sort of motivation to do that.
Because guess what, and after -- it's very humbling
experience to understand that machine learning really
had nothing to do with predicting risk, right? At
the end of the day, you know, in order to improve
quality of care, it's really about, you know,
literally a factor like whether there is someone at
home to take care of an elderly patient contributes
significantly to their readmission or no readmission,
right?
And no machine learning -- well, machine-learning
model can at least predict or find that out, provided
we have access to unstructured data. Because in the
CCDs and the clinical care documents and so on where
patients actually have conversations with doctors and
physicians and this gets noted, they do mention many
times that my doctor dropped me off, right?
Now the question is, are we in the legal gray area to
be able to mine that information. Even if we had
Hadoop and we had Spark and we had these nice, you
know, infrastructures, are we able to allow to touch
that piece of information which says that this
patient actually has a doctor who takes care of him
or her, right?
And that's sort of the vision where we want. So, but
big data infrastructure and things like Azure for
Research enable us to at least attempt and prototype
some of those efforts.
Not just that. A very interesting problem is
appearing and we sort of need help from this
community of Microsoft Research to attack it. We
sort of solved the problem to reduce readmission or
predict readmission risk.
There is another very interesting dual problem of
this, which is reducing length of stay. Because
guess what? The hospital can reduce readmissions by
just keeping the patient in the hospital for 30 days.
Right? That's no readmission, right?
But that's not how we want to solve the problem. We
do actually want to -- it's a multi [indiscernible]
Constrained optimization. And then we want to apply
intervention frameworks.
We have just been approached by MultiCare for yet
another year of, you know, collaboration across
pneumonia, COPD, AMI. So the task is to now start
building models for each of these.
Here is how sort of our partner landscape and the
machine learning on health care vision looks like for
2015. We have sort of on Azure we would like to
integrate multiple data sources. Some of these are
public. The state inpatient data said COPD, the H
cough, ill effect to the partner, it's a Bellevue
company that processes claims data and they are
donating data as well as resources so that we can
build predictive models, not just from the clinical
side but also from the claims side, which is
extremely powerful.
Now you can imagine that you have the holistic
picture of the patient because if the patient comes
to just MultiCare but gets readmitted somewhere else
in 30 days, we do not know that, but if we have
access to claims data, then we can do that.
On top that, we want to build some open source or we
want to use some open source ETL tools and then
start, you know, building models. And we have. Many
of these efforts are actually ongoing already. Here
is a fairly complex graph of what our organization
looks like today.
It consists of, you know, several people who work on
cost prediction models, a big team that's here today
presenting our results on MultiCare and risk of
readmission models. We have several professors who
work on the machine learning and big data side and
kind of so on.
So if any of these boxes interest people or
encourage, please drop us a note. We are looking for
adding more boxes in the machine learning bit. Of
course, the vertical is very fixed. It is for health
care analytics and we can add more algorithms.
Last but not the least, I want to sincerely express
my thanks and gratitude as the director of the center
and as the overt leader of this team here to all of
you for encouraging us to use this platform, Azure
for Research. Specifically I want to call out Vani
Mandava and Dennis Gannon for allowing us to share
this -- these results.
I also want to thank MultiCare for our data partners.
It's not just one US city effort, but the way. What
didn't go on the slide is there are people from
Granada in Spain who are building for this set models
on this. There are people from Kent University in
Belgium who are contributing to the open source
machine-learning efforts that we are doing. So it's
truly an international set of collaborators that has
come together to use this platform and solve some of
these problems, even though, you know, at the base we
are doing it over here.
With that, I want to thank all my co-speakers as well
today for the wonderful effort that they have done.
Hopefully, hopefully the hope is that what Bill
Clinton said in 1993 will one day come true and we
really will have an affordable care model for our
health care, and we hope that the steps that we are
taking today and the support that you guys are
providing will be, you know, one small step for
mankind towards that direction. So with that, thanks
a lot.
[applause]
>>: So would there be machine learning behind
[inaudible]? How easy is it to adapt to other
problem spaces and how much intervention [inaudible]
and so on?
>> Senjuti Basu Roy: Well, let me give you a shallow
answer. So I think the way we are ->> Ankur Teredesai:
question?
>> Senjuti Basu Roy:
I wear this?
>> Ankur Teredesai:
Senjuti, could you repeat the
Yeah, the question is -- should
No, no.
>> Senjuti Basu Roy: So the question she asked is
the multilayer machine learning model that we
developed, how easy it is to generalize for other
diseases, right?
>>:
Or for other problems.
>> Senjuti Basu Roy: Or for other problem spaces.
So my shallow answer to this question will be as long
as the other problems exhibit the characteristics
that we have, it will be possible. How successful it
will be, I don't know. But it's certainly worth
investigation.
>> Ankur Teredesai: I would add that, so the
multilayer approach and the solution that we designed
was not done as data scientists. It was done
collaboratively with physicians, right? So that's a
very important missing piece here.
>>:
[inaudible] domain expertise.
>> Senjuti Basu Roy:
Yes.
>> Ankur Teredesai: Exactly, exactly. So it took us
some time to understand that brute force, you know,
compute power is not really going to help us. We
really need to segregate it into multiple layers,
really need to understand what happens to a patient
inside a care pipeline, what is the care protocol
that they are following, and then at which points in
that care protocol can we really influence the
intervention, right? And that's why the multilayer
approach works, because, you know, their domain
knowledge then plays a big role in helping us solve
the problem.
One question, great talk.
>>:
Thank you.
Thanks everyone for coming.