Benchmarking the Emergency Department – Which “Best Practices” Really Drive Efficiency
Presenter: Dr. Harriet Nembhard, Penn State University
Recorded on: June 25, 2014
And we will now go to today's presenter, Dr. Nembhard.
>> Thank you very much, Sherri, I appreciate this opportunity to present our work to the group. Let's
just dive right in. This is a deep dive webinar. And start to put on the table some of the ideas that we are
dealing with.
Number one, according to the National Center for Health Statistics, in the past two decades, the number
of ED visits has skyrocketed in the US, from 105 million to 136 million. As a result of this increase, as well
as other factors, many ED's and hospitals have faced an overcrowding issue.
A recent survey by the American Hospital Association said that 35% of hospitals indicated that their EDs
were at or over capacity. This ED overcrowding has caused long wait times, sometimes ambulance
diversions. These things compromise patient safety, and, of course, as we know, lower patient
satisfaction. In a CHOT project conducted over the past year, our team has taken a deeper look at some
of the available data and we have tried to integrate our domain expertise to understand more about
what is really driving efficiency.
The data master on our team is Miss Hyojung Kang, who is pursuing her PHD in Industrial Engineering
with me as her adviser at Penn State. Our ED guru, is Dr. Chris DeFlitch, who brings much expertise,
experience, strategy, and perspective to this work. So, what I would like to do during this webinar today
is to discuss ways we can go about improving the ED and benchmarking the ED.
We will be using systems engineering methods, specifically, data envelopment analysis, or DEA, to
analyze efficiency. This will be a very high level and brief presentation. I expect to just use about 15 or
20 minutes. I have tried to keep the number of slides here to a minimum. I think I've got 10 more or 12
more after this one.
Because the main thing I would like to do in this webinar is to leave the bulk of our time for discussion.
This review is a preliminary study and we would like to benefit from the perspective of the participants
in helping us to think about some of the next steps and implications of this work.
So, there are three questions that we have at the end, that you can be mulling over as we are
proceeding through the talk. What would be other appropriate inputs and outputs to use? I will show
the ones that we have used in evaluations of the operational efficiency, but surely there may be others
and people may have some suggestions on what they may be.
We also found that smaller hospitals had fewer hospitals that were efficient by the method and
standards that I will describe in a little bit. But what is it about smaller hospitals that leaves them
perhaps to be less efficient? We can discuss this. And then lastly, what are some of the effective
strategies for adapting best practices as we are able to uncover them by our data detective work.
So, with that, let me turn to giving a little bit of brief background. We said that we're looking at ED
crowding. We know that there are many factors in ED crowding. And in response to these crowding
challenges, many EDs have tried to implement various improvement initiatives. Some of them have tried
to expand their physical plants.
Some have pursued IT solutions, such as computerized technician order entry and electronic medical
record implementation. In some cases, the processes may have been re-engineered to include things
like point of care testing, triage protocols, and separate care packs for low acuity patients. However,
only a limited number of EDs have successfully reduced overcrowding by these measures, as indicated in
the statistics that I cited at the outset.
In the meantime, healthcare organizations have pushed EDs to tackle the issues that result from
overcrowding and to improve the efficiency of care. For example, insurance providers have requested
hospitals to report a set of process metrics. They have also, many insurance providers, have offered
targeted based financial incentives for the level of efficiency.
The centers for Medicare and Medicaid services has developed performance measure sets for timely
care in the ED and has made the information publicly available, perhaps to some controversy in the
recent months. But what we want to do is to look at benchmarking as a tool, a tool that we can use to
learn from and hopefully to improve the performance of emergency departments.
There are a couple of ideas within benchmarking. Problem-based benchmarking efforts will talk about
specific concerns, perhaps such as the desire to improve an error rate or specific cycle times. But we
want to focus here on operational efficiency. So we are concerned with process-based benchmarking,
which is oftentimes tied to continuous policy improvement efforts.
There are four types of benchmarking. I will just say that functional, generic and internal benchmarking
all have a specific definition and appropriate uses, but here we want to focus on competitive
benchmarking. Competitive benchmarking, as the name implies, entails looking at your competitors, and
comparing your work processes to those of your best competitors.
Now, even though organizations can be competitive, hopefully they can still be collegial, especially in
healthcare where the overriding desire is for better health. And to that point, the Emergency
Department Benchmarking Alliance is a group of such competitors. Let me pause here for just a moment
to take a message from one of the participants.
Okay, I think everybody is able to join. So Was there another question? Okay, I'm looking at perhaps one
of my co-presenters. Might need to add a remark. Sherry, can you allow her audio?
>> She's on mute. Yes. Can you allow her audio?
>> Yes. She is on mute.
>> Oh. Okay. So. Go ahead.
>> Hello?
>> I have a text on my screen that says that you would like to speak. Do you have something to add
there on that slide or the previous one?
>> No.
>> Okay. I'm not sure if one of the other presenters, as I said has been working on the data analysis part
and Christoph Lych has been our emergency department guru.
We may have a little bit of technical difficulties to get them to add in at the moment. So what I will do is
proceed through these slides as I said I just have a few. And then hopefully at the end Sherry can unmute us all for a general discussion and I can even go back to some of the slides that they might want to
embellish.
Okay. So moving along as I said, the emergency department benchmarking alliance is a group of
colleagues. It's a not for profit organization which purports the people who manage emergency
departments across the country. It's to completely balance your organization. The organization is
created by the membership. It is for the use of the membership.
There's no commercial interest attached to it or government regulation attached to it whatsoever. They
said is there just to support the membership of those who manage emergency departments. And it is
through this database that has almost 1000 hospitals, that we were able to get a good clean set of data
to start looking at some of these bench marking ideas.
Here is the overall structure of the project. The objective is to develop a data-driven framework for
benchmarking the efficient emergency department. We're doing this in a specific way. To look at what
the emergency department efficiency frontier is. And the efficiency frontier is a part of the terminology
that in those systems engineering message.
We want to contrast those with the inefficient emergency departments in order to think about, of
course, how an inefficient emergency department may become a more efficient one. We are looking at
three areas in this study. We are looking at structural characteristics such as what kind of hospital it is.
We are looking at operational characteristics such as the volume of the emergency department, and
we're looking at advanced features, such as how do they index patients, how do they use
documentation in the emergency department? And perhaps whether they use a fast prep or the triage
of patients? So, in using data envelopment analysis.
As I said, this is a systems engineering tool that's used to evaluate the efficiency of each emergency
department, among a set of peer groups and compared their performance. Now again I'm not going into
a lot of the technical details of DEA. We do have a technical proceedings paper that we have submitted
for this work if people are interested in more of the details.
Let us say here that we're focusing on using DEA because it will show us the relative performance scores
of the decision making units, which is another term of DEA and it will incorporate multiple inputs and
outputs. While other commonly used metrics may provide absolute outputs and values that based on
just only the output of the system.
So I have here just one example, a little bit of a technical example, to show how it works. The idea is that
in a group of hospitals, so in this diagram labeled as H1 through H10, you want to find among some
factors, what is the efficient frontier. And on that efficient front here, the score that the hospital
receives for it's level of efficiency would be one.
So one would be the perfect score, that is again the hospital that is on the efficient frontier. So here in
this group, we see that hospitals one, eight, and seven are along the efficient frontier and the other
hospitals are not. The beauty of DEA is that it allows us to understand when a hospital that is not on the
efficient frontier by the factors and variables studied, what that hospital may do as a set of alternatives
to improve.
But for example, if we take a look at hospital H10 here, we can say that there are a couple of ways that
hospital H10 might become more efficient. One would be to increase admissions in other words to
become a larger hospital. The other route would be to decrease the nursing hours needed to take care
of the patients that it has.
Those two alternatives and the range in between are things that could, by this DEA approach, help the
hospital to become more efficient. In a DEA procedure, what is important is to establish what the
decision making unit should be or would be for the group of hospitals in this case, whose performance is
being benchmarked.
Again, we use the EDBA, the ED benchmarking alliance 2012 database. A small image is there to give
some idea of the types of questions that the participants in the alliance will answer in regard to their ED.
And as I said there are almost 1000 EDs involved in the database.
By looking at the data, and making sure that we had the needed data filled in in all of the cells for the
hospitals that would be Involved in this study, cleaning up that database left us with 449 EDs that had
enough information in the database for us to analyze.
So our analysis refers to about half of those hospitals. Then, the next critical step is to determine the
inputs and outputs that will be used in this study. Again, to look at a manageable scope in this first pass
of this study, our inputs are the number of And the number of hours the physicians and nurses involved
in the hospital.
Of course there are other care providers but our focus is these. And then with the output what is the
throughputs per day? What is the adjusted length of stay in the ED? So that is taking in account the
patient who might be transferred to other units. As well as the left without being seen rate, those
patients who essentially walked because there is too long of a wait in the waiting room.
Using a linear programming approach as a part of the DEA procedure on the 449 EDs, we are able to
solve the LP and I spared you the various equations and mathematical notations, but we solve the linear
program, a system of equations. To find out what are the optimal waits for the inputs and outputs that
we are using in this study and assign a score between zero and one according to the performance of that
particular hospital.
So doing that this slide then shows the initial result for all 449 EDs. It shows their efficiency score by
their volume. What you can see in this diagram is that there is a linear relationship between ED volume
and efficiency score. Or as my colleague Doctor Deflitch would say that's you know only obvious that the
volume will certainly have something to do with how well an ED can operate.
So with that, what we did next was to cluster the hospitals, or group the hospitals, by size and then
proceed to analyze their efficiency. So the six groups here, I'll just point out that as an artifact of how
the linear program is setup as a minimization problem this goes in reverse order.
So the volume of hospitals, this is up to 20,000, it should be units of thousands of patients. From up to
20,000 patients on an annual basis is our group six. Then those hospitals that see up to 40,000 patients
is our group five, up to 60,000 patients is our group four, and so on on through group one.
The largest hospitals that have in excess of 100,000 patients per year. So separating that out by group,
this slide shows their efficiency scores. So I'll start here at the bottom, since they started with group six,
the smallest hospitals before. You can see the efficiency scores for group six are from the volume, from
zero up to 20,000 here.
And the efficiency scores for those hospitals are from zero to one, as I indicated before. We can see a
certain segment of them are able to reach a fairly high efficiency in that case. But many many hospitals
are not operating at their maximal DEA efficiency. And so on is the story through the other groups, up
through the largest group of hospitals, with volumes from 100,000 patients and onwards.
We see here, by quick comparison. That many more of the large hospitals are able to achieve DEA
efficiency. I'll translate these results into a more simpler bar graph here. Again, by group of hospital. We
can see that there is a significant difference between the larger patient volume EDs.
Where about 45% to 55% of the hospitals are able to achieve EE, or I'm sorry, DEA efficiency as
compared to the smaller hospitals where only about 15% to 35% of them are able to achieve efficiency.
Certainly we would have to do a further investigation to determine the cause of these results.
We're certainly not saying that it's only because they are small that they are inefficient but we can say
there's a large efficiency gap between the small hospitals and the large hospitals that perhaps suggest
that a more rapid proliferation of such practices is needed among these small hospitals in order to
improve performance.
So last slide here before we move into discussion, I want to just highlight some of the further analysis
that we were able to do. So based on the efficiency score, what we did was to perform a logistic
regression to analyze which factors were associated with the efficient emergency departments.
So in this study we looked at three factors. Certainly there are other factors or advanced features, but
we looked at three here, and focused on these three for discussion purposes today. In the first set of
slides here, looking again across those 500 or so, 449 EDs. We do have from the EDBA information on
their intake model.
We can see from our study that those hospitals that have performance that is along the efficient frontier
use always nurses to do the intake. Whereas the less efficient groups, use nurses about half the time or
53% of the time and then the remaining time you position a mid level provider in LP for the intake of
patients.
We can also look at technology IT to improve the workflow. In our efficient hospitals, we found that 75%
of them use some type of computerized physician order entry system. With 25% of them did not. In
contrast, in the less efficient groups 88% of the hospitals used C.P.O.E and only 12% did not.
And then lastly, many ED's have adopted new patient flow models such as new triage protocols, a fast
track system where patients who have a low acuity might be tracked off differently than patients who
have a high acuity or high problem. And some of the implementation of these features as we looked
back retrospectively through observational studies don't necessarily contribute to a greater efficiency.
What we found in this among this set of hospitals is that 75% of them had a fast track, 25% of them did
not. Whereas among the less efficient groups, more of them had a fast track system and only 10% of
them Did not. So, with that, that lays out some of the ideas that we wanted to open for discussion here
today in looking at this data.
So I'll ask Sherry if she could perhaps unmute everybody. Maybe before we go to the specific questions
that I have here we might have general remarks of people or from the other people on my team.
>> Okay, I have now muted everyone who joined the webinar using their phone.
And for those who joined using the microphone from their computer. Please let me know if you have
any question because if I mute you we will hear a lot of noise. Dr. Dusilich, can you hear us? I can hear
you. Can you hear me?
>> Yes.
>> Very good.
>> Actually
>> Doctor Switch I can just barely hear you. Is there a way that you can, Sherri, is there a number that he
can call into?
>> I think you have to close a window and join the webinar again using your phone. Yeah, because I
don't think you can just switch to the film.
Dr. Desledge, do you mind coming back to the webinar again using your phone? Okay, Dr.Desledge said
he will rejoin it. Okay. If there's just a direct phone number that he can call, can he call this phone
number that's appearing on my screen? The 650-4? If he just calls that number, will he join this
conference?
>> I'm not sure. Well, maybe we give him a moment to try to re-join the call. Hi this is Bitta, Harriet,
thank you so much for a wonderful presentation. Great results to share here. I would recommend for all
the participants to, when you log into WebEx, to go ahead and go for the option of WebEx calling you on
your phone.
It's much easier if you're planning to participate in discussion, and we can hear you so much better. So
that would be a great way of assuring we can hear you. Harriet, would you expand on some of the
results that you showed us in the pie graph that are not as intuitive as we would expect them to be?
Yes, sure thing. Here we go. You're meaning here, right? That's perfect. One would assume that EPOE or
Fast Track protocols would help efficiencies.
>> Yes, so indeed that was some of the surprise actually in studying the data. Now as I said I want to
offer a disclaimer from the standpoint that this was a study on a certain set of almost 450 hospitals.
And this is information that is based on the group performance. There may also be interaction effects.
We did not do any adjustments or indexing of the data that might be done to smooth the programming
model of DEA but that said what this has suggested to us is that some of the assumptions on that yes,
Fast Track is good for example, might need to be looked at with finer detail to see.
Well what else is going on in that ED that might be contributing to whether Fast Tracks would be good
or not. I mean for those of us who understand the design of experiments or systems point of view. It's
basically saying that one factor in isolation may not necessarily tell the whole story, right?
So it could be, for example, that Fast Track is a better option for larger hospitals, but that it doesn't
necessarily have the same benefit for smaller hospitals, for example. It could be that really you get
better performance when you have Fast Tracks operating in conjunction with physicians at the intake
side.
That maybe having physicians there to help a fast-track model are two things that are needed together,
for example. So, this though is really where I was hoping that our ED guru, Dr. Chris Diflitch, might be
able to have some additional strategic insight. Chris are you able to join yet?
Well, while we wait on Chris, I'd like to say
>> Oh, here we go.
>> Hello?
>> Is this?
>> Yeah. You got a lot of technology, don't you?
>> Yeah.
>> Oh yeah.
>> Can you hear me okay? Yes.
>> Okay, very good. So I apologize for those technical glitches in between.
Hariet, I thought you did a really nice job of outlaying some of the work that we've done to try and put
some organization around what efficiency or inefficiency looks like in emergency departments. Some of
the things we've talked about as part of our work is what does efficiency look like and what does
efficiency not look like.
There are some practicalities as we have been working through and described in the different categories
of volume. One of the things we initially started talking about is lower volume emergency departments
are generally inefficient, mostly because they don't need to be efficient. And it's based upon location of
those emergency departments.
Most places where you have lower volume EDs, it's gonna be very rural, it's a one-stop shop. That type
of things. The other concepts based on the slide that's presented right now, is that we found it very
interesting that all of these different innovations of CPOE and Fast Tracks and arrival modalities that are
different are seen more frequently in less efficient emergency departments based upon those volumes.
You know, one way to look at it is Is these interventions make you less efficient. I think that's probably
not a correct interpretation. It is likely that efficient EDs don't need to be innovative in regards to
modalities because they don't have the flow issues that they have. So the more inefficient emergency
departments are trying something different to innovate, which is why you're seeing some of these new
model attempts in these historically less efficient emergency departments.
But we found it a helpful way to start categorizing or sectioning out the different types and models to
categorize efficiency versus inefficiency.
>> Yeah, precisely. Thank you so much, Chris, for giving that perspective as you always do. Another thing
that we had talked about amongst our team, as well as our colleagues is that, I guess I can say it as we
do see less efficiency in the smaller volume hospitals.
But sometimes we get the feeling that perhaps the smaller volume hospitals are just trying to take the
innovations, maybe one by one, that are used in larger hospitals, and try to recast them in the smaller
hospitals. Whereas perhaps this suggests that maybe we step back and that the smaller hospitals need
something else.
They need something that's not done or not offered in the larger hospitals. Of course, this is a theory. A
part of our future work is to try to develop ways that we can test these sorts of ideas. We definitely have
an interest in bringing the efficiency and the quality of care up in our smaller hospitals.
It's definitely needed for them to survive. But maybe, just perhaps, they need a different set of tools
that are customized for them specifically. This is one theory or idea that we have been bouncing around.
So with that, those few questions I'll put back up here to see if there's other discussion or comments.
Certainly we realize, as with any study, there would be multiple ways of picking it apart, that there might
be other inputs and outputs that other folks might like to see used in evaluating efficiency. If so, maybe
you can remark on what those might be. Maybe I'll just leave it as just that one question for now.
Is there any comment on using other factors as we think about efficiency?
>> Harriet, you did bring up the interaction, possibly. I think that's something that would be great to
follow up with. And this Vita Cash again. I believe the interaction between the staffing models and
certain innovations might be something that can be done quickly and would be very informative.
>> Yes, that's a good point. I'm just jotting that down, Vita, just so that we can come back to it when our
team reassembles in the fall. You know that's something we had indeed, as I mentioned, thought about.
The question then becomes, if you're really gonna study interaction effects, and you have some certain
statistical prudence that you want to adhere to, that maybe it's difficult to think about how to test for
the interaction effects.
Or at least for me, I can say it's perhaps difficult to think about how to test for the interaction effects
when the study design doesn't meet some of the fundamental assumptions. Which, maybe we can talk
about those technical details a little bit to the side, but the study design is not one of a trial, right, or a
comparative effectiveness approach.
It's really looking at past observational data, so it may be difficult to estimate a true interaction effect,
but it's certainly something that we've thought of. I see that Suzy has her hand raised. Sherry, can you
help her?
>> Suzy, I just un-mute you, can you say something? Suzy, can you hear us?
I don't think her microphone is connected.
>> Okay. Okay, well, maybe with the technology here being a little bit troublesome today for everybody
to discuss. Maybe we have to find another means for future exchange on this idea. But let me just
summarize, then, where we wanted to go with this and what drove this study.
As we've said, this has been a top project that's been conducted over the past year. And here at Penn
State, some of our industry partners, in particular Siemens Health Systems, has really pushed us to think
about, as I said in the title, which practices are really driving efficiency.
And the overall question is, how do we then proliferate those best practices? So I think that perhaps we
hope we've offered some food for thought about ways to use the available data on ED performance.
How do we understand this data in a framework that lets us think about efficiency at a performance or
operational level versus some of the more traditionally used metrics?
Not that they're not important, but some of the traditionally used metrics by themselves that might just
be things like readmissions or process type problems? What we hope to do, then, over the next year as
we continue this study is to really think more at the innovation level, at the management level, about
what makes effective strategies for adopting best practices.
And as I said, to think about how best practices might be customized for the type of emergency
department in which they need to be used. So>> Harriet, I have have a question from Suzy.
>> Oh, okay.
>> Okay, her question is, if you consider data for length of stay, time until treatment, mortality rate.
>> Oh, did we consider data for length of stay?
>> Yes.
>> Okay, now this is something interesting because in our approach we very much realized that there
could be certain quality metrics, if you will, such as mortality, that would certainly be of interest. But
from our framework, we really didn't want to get into thinking about how to trade-off on mortality, for
example.
That there's a overarching concern about health care quality and these sorts of things we did not
consider, for those reasons, in this study. So we didn't really look at things that we consider to be the
quality metrics or kind of outcome performance metrics of the patient. But rather, try to focus on the
operational efficiencies that might be driving costs.
>> Thank you.
>> Mm-hm.
>> Does anybody have any questions?
>> Well, again, let me thank everybody for their time in joining this webinar and hearing our results,
listening to our study. If people do have additional input that they would like to provide to us, our email
and contact information is right there on the CHOT website and we would love to hear from people at
another time.
But again, thanks, everyone, for today for joining this talk.
>> Thank you everyone for attending today's webinar. Please feel free to send us a email if you have any
question and then we will see you at our next webinar. Thank you.
>> Thanks again, Sherry.