Effectiveness of Evidence-Based CPOE Medication
Order Sets Measured by Health Outcomes
Jacob Krive, MBA, MS, CPHIMS; Joel Shoolin, DO, MBA; Mary Gagen, MBA
Information Systems & Systems Quality Departments at Advocate Support Centers
Abstract:
Introduction
Order sets offer promise of standardized care beyond convenience features
through evidence-based practices built upon powerful knowledge of clinical
professionals to achieve potentially more consistent health outcomes with patients.
Despite ongoing utilization of the order sets, there is a lack of quantitative evidence
behind their effectiveness. The study is aimed at quantitatively analyzing benefits of
the order sets applied to prevent VTE and treat pneumonia and CHF – testing
hospital mortality, readmission, and length of stay as health outcomes, supplemented
by comorbidity data to help explain or constrain results.
Methods
Statistical instruments and SPSS software are used to examine four years of
recent patient history from clinical applications. For the VTE study, prophylaxis focus
on prevention of acute VTE is tested between independent variables of order set
utilization and primary point occurrence, and the health outcome of length of stay,
mediated by Charlson Comorbidity Index (CCI), surgical status and gender of the
patients. For pneumonia and CHF, independent variable of utilization of sets is tested
against the health outcomes of mortality, 30-day readmission, and length of stay,
mediated by CCI. Statistical methods used for calculations include Pearson ChiSquare, binary logistic regression, Mann-Whitney test of independent samples, and
one-way ANOVA with Tukey, Bonferroni, and Dunnett t-test options.
Results
Results indicate that standardization of care practices by ordering via electronic
sets could be effective in decreasing acute VTE occurrence among non-surgical
patients, with no improvement in the length of stay. Order sets have potential to
assist with reduction in mortality, readmissions, and length of stay among pneumonia
patients. There is positive impact on mortality and length of stay among CHF
patients, with no improvement in the readmissions area.
Conclusions
Implementation of evidence-based practices has potential to improve health
outcomes. There are further research opportunities in this domain beyond VTE,
pneumonia, and CHF. Such effort would require buy-in from physicians necessary to
increase utilization and subsequently data samples to produce results of greater
statistical significance. Due to small experimental group data samples, the study
cannot currently produce complete and convincing results, thus it represents work in
progress in search of new partnerships with clinicians.
Methods:
Causal comparative (ex post de facto) research using CareConnection and EDW
patient data, 2007 – 2011 timeframe. Order set utilization averaged 2% - 7% of all
eligible patient records.
VTE
¾ Inclusions: adult patients over 18, surgical and non-surgical patients, VTE
prophylaxis filtered via 48-hour from admission medication order restriction.
¾ Exclusions: OB patients, Psych patients, pediatric patients, chronic VTE patients
¾ Order sets: IMMC DVT Prophylaxis Order Set, LGH VTE Prophylaxis Order Set
¾ Variables: utilization of the order sets (y/n), acute VTE (y/n) – independent;
comorbidity and length of stay - dependent.
¾ Categories: sex of the patient, surgical status, treatment facility
¾ Goals: test direct relationship between utilization of the sets and occurrence of
VTE; order sets and length of stay with acute VTE as a factor; order sets and
comorbidity with acute VTE as a factor; direct relationship between order sets length
of stay; direct relationship between order sets and comorbidity; acute VTE as
affected by factors of order set utilization, race, sex, and surgery status.
¾ Statistical Methods: crosstabs; binary logistic regression; Pearson Chi-Square;
Fisher’s Exact; One-Way ANOVA with Bonferroni, Tukey, and Dunnett post-tests;
Mann-Whitney Non-Parametric Independent Samples test.
Pneumonia
¾ Inclusions: adult patients over 18 with pneumonia diagnosis codes, community
and nosocomial
¾ Exclusions: OB patients, Psych patients, pediatric patients
¾ Order sets: CMC, GSAM, IMMC, SSUB, TRIN Pneumonia Inpatient or
Pneumonia Community Acquired sets
¾ Variables: utilization of the order sets (y/n) – independent, mortality, 30-day
readmission, comorbidity, and length of stay - dependent.
¾ Goals: test relationship between utilization of the sets and mortality, utilization of
the sets and readmissions, utilization of the order sets and length of stay, utilization
of the sets and comorbidity.
¾ Statistical Methods: crosstabs; binary logistic regression; Pearson Chi-Square;
Fisher’s Exact; One-Way ANOVA; Mann-Whitney Non-Parametric Independent
Samples test.
CHF
¾ Inclusions: adult patients over 18 with CHF diagnosis codes, CareConnection and
outpatient sets (paper and electronic)
¾ Exclusions: OB patients, Psych patients, pediatric patients
¾ Order sets: CMC, GSHP, IMMC, SSUB, and LGH Congestive Heart Failure or
Heart Failure Management or CHF Unit Admission sets
¾ Variables: utilization of the order sets (y/n) – independent, mortality, 30-day
readmission, comorbidity, and length of stay - dependent.
¾ Goals: test relationship between utilization of the sets and mortality, utilization of
the sets and readmissions, utilization of the order sets and length of stay, utilization of
the sets and comorbidity.
¾ Statistical Methods: crosstabs; binary logistic regression; Pearson Chi-Square;
Fisher’s Exact; One-Way ANOVA; Mann-Whitney Non-Parametric Independent
Samples test.
Results:
VTE
¾ Order Set == Acute VTE
IMMC Surgical Patients: odds ratio = 0.67 [0.452 and 0.992 at 95% CI], P = 0.044
LGH Surgical Patients: odds ratio = 0.497 [0.383 and 0.645 at 95% CI], P < 0.01
IMMC Non-Surgical Patients: odds ratio = 1.566 [1.354 and 1.812 at 95% CI], P <
0.01
LGH Non-Surgical Patients: odds ratio = 1.205 [1.043 and 1.393 at 95% CI], P =
0.011
¾ Order Set == Comorbidity via Acute VTE
• There is significant difference between Charlson Comorbidity Indexes for the IMMC
non-surgical groups of patients who received medications via orders sets and CCI for
those who received custom orders, F (3,28271) = 94.297, P < 0.05.
• There is significant difference between Charlson Comorbidity Indexes for the LGH
non-surgical groups of patients who received medications via orders sets and CCI for
those who received custom orders, F (3,38813) = 410.373, P < 0.05.
• There is no significant difference between VTE groups for IMMC and LGH surgical
patients, using Charlson Comorbidity Index as dependent variable.
• Due to combination of mixed and statistically insignificant results, comorbidity study
for VTE is inconclusive.
¾ Order Set == Length of Stay via Acute VTE
• There is significant difference between Length of Stay for the IMMC non-surgical
groups of patients who received medications via orders sets and LOS for those who
received custom orders, F (3,28271) = 172.499, P < 0.05.
• There is significant difference between Length of Stay for the LGH non-surgical
groups of patients who received medications via orders sets and LOS for those who
received custom orders, F (3,38813) = 410.373, P < 0.05.
• There is significant difference between Length of Stay for the IMMC surgical groups
of patients who received medications via orders sets and LOS for those who received
custom orders, F (3,10237) = 101.509, P < 0.05.
• There is significant difference between Length of Stay for the LGH surgical groups of
patients who received medications via orders sets and LOS for those who received
custom orders, F (3,17329) = 686.291, P < 0.05.
• All of the Dunnett t-tests showed statistically significant relationship between order
set and no order set groups relative to prevention of the VTE, but with LOS and CCI
higher for the experimental group.
¾ Order Set Î Length of Stay
IMMC Surgical: Total N: 10,259, Mann-Whitney U: 11,755,933.500, P < 0.01
VTEOrderSetUsed = 0: N = 7973, Mean Rank = 4798.53
VTEOrderSetUsed = 1: N = 2286, Mean Rank = 6286.08
IMMC Non-Surgical: Total N: 40,950, Mann-Whitney U: 189,937,128.500, P = 0.008
VTEOrderSetUsed = 0: N = 26252, Mean Rank = 20589.35
VTEOrderSetUsed = 1: N = 14698, Mean Rank = 20272.15
LGH Surgical: Total N: 17,359, Mann-Whitney U: 22,970,959.500, P < 0.01
VTEOrderSetUsed = 0: N = 15158, Mean Rank = 2201
VTEOrderSetUsed = 1: N = 8265.07, Mean Rank = 11537.60
LGH Non-Surgical: Total N: 56,799, Mann-Whitney U: 245,247,010.500
VTEOrderSetUsed = 0: N = 47629, Mean Rank = 8265.07
VTEOrderSetUsed = 1: N = 9170, Mean Rank = 31329.99
¾ Order Set Î Comorbidity
IMMC Surgical: Total N: 10,241, Mann-Whitney U: 10,860,731.500, P < 0.01
VTEOrderSetUsed = 0: N = 7969, Mean Rank = 4893.13
VTEOrderSetUsed = 1: N = 2272, Mean Rank = 5916.75
IMMC Non-Surgical: Total N: 28,275, Mann-Whitney U: 108,401,386.500, P < 0.01
VTEOrderSetUsed = 0: N = 14468, Mean Rank = 13549.01
VTEOrderSetUsed = 1: N = 13807, Mean Rank = 14755.19
LGH Surgical: Total N: 17,333, Mann-Whitney U: 19,295,760.00, P < 0.01
VTEOrderSetUsed = 0: N = 15140, Mean Rank = 8489.01
VTEOrderSetUsed = 1: N = 2193, Mean Rank = 9895.80
LGH Non-Surgical: Total N: 38,817, Mann-Whitney U: 143,818,946.500, P < 0.01
VTEOrderSetUsed = 0: N = 30060, Mean Rank = 19003.10
VTEOrderSetUsed = 1: N = 8757, Mean Rank = 20802.31
Pneumonia
¾ Order Set Î Mortality
GSAM, IMMC, SSUB, TRIN total: odds ratio = 1.79 [0.965 and 3.32 at 95% CI], P =
0.061
¾ Order Set Î Mortality (Hospice patients excluded)
GSAM, IMMC, SSUB, TRIN total: odds ratio = 1.822 [0.983 and 3.38 at 95% CI], P =
0.053
¾ Order Set Î Readmissions
GSAM, IMMC, SSUB, TRIN total: odds ratio = 1.362 [1.015 and 1.827 at 95% CI], P =
0.039
¾ Order Set Î Length of Stay
GSAM, IMMC, SSUB, TRIN combined: Mann-Whitney U: 1,148,309, P = 0.001
PNOrderSetIndicator=0: N = 4531, Mean Rank = 2568.57
PNOrderSetIndicator=1: N = 556, Mean Rank = 2343.80
One-Way ANOVA: There is significant difference between LOS for pneumonia
patients who received medications via order sets and those who has them prescribed
via custom orders, F (1, 5085) = 4.798, P < 0.05.
¾ Order Set Î Comorbidity
GSAM, IMMC, SSUB, TRIN combined: Mann-Whitney U: 1,153,767.500, P = 0.014
PNOrderSetIndicator=0: N = 4427, Mean Rank = 2509.38
PNOrderSetIndicator=1: N = 556, Mean Rank = 2353.62
One-Way ANOVA: There is significant difference between CCI for pneumonia patients
who received medications via order sets and those who has them prescribed via
custom orders, F (1, 4981) = 5.954, P < 0.05.
CHF
¾ Order Set Î Mortality
CHMC, IMMC, SSUB, GSHP, and LGH total: odds ratio = 1.854 [1.059 and 3.244 at
95% CI], P = 0.028
¾ Order Set Î Mortality (Hospice patients excluded)
CHMC, IMMC, SSUB, GSHP, and LGH total: odds ratio = 1.818 [1.039 and 3.181 at
95% CI], P = 0.034
¾ Order Set Î Readmissions
CHMC, IMMC, SSUB, GSHP, and LGH total: odds ratio = 0.913 [0.734 and 1.137 at
95% CI], P = 0.417
¾ Order Set Î Length of Stay
CHMC, IMMC, SSUB, GSHP, and LGH combined: Mann-Whitney U: 3,472,652, P =
0.013
HasOrderSet=0: N = 10219, Mean Rank = 5489.18
HasOrderSet=1: N = 719, Mean Rank = 5189.84
One-Way ANOVA: There is significant difference between LOS for CHF patients who
received medications via order sets and those who has them prescribed via custom
orders, F (1,10936) = 8.352, P < 0.05.
¾ Order Set Î Comorbidity
CHMC, IMMC, SSUB, GSHP, and LGH combined: Mann-Whitney U: 1,840,535.500,
P = 0.23
HasOrderSet=0: N = 7232, Mean Rank = 3887
HasOrderSet=1: N = 525, Mean Rank = 3768.78
One-Way ANOVA: There is no significant difference between CCI for CHF patients
who received medications via order sets and those who has them prescribed via
custom orders.
Outcomes:
Condition
Mortality
Readmission Length of Comorbidity Acute VTE
Stay
VTE
N/A
N/A
No
No
Pneumonia
Yes
Yes
Yes
Yes
Yes –
surgical
No – nonsurgical
N/A
CHF
Yes
No
Yes
No
N/A
Barriers and Issues:
1. Low order set utilization across Advocate leads to small experimental group
samples. Attempt to research effectiveness of AMI order sets produced no
results due to low utilization.
2. Order sets are not centrally managed and differ from site to site, so difficult to
compare due to differences in content.
3. Computer databases behind EMRs are not currently designed to perform
massive compares between orders, so custom selection within order sets is not
accounted for.
4. There is no good method to measure complications. Instead, comorbidity is
used to analyze length of stay outcomes.
Implications:
1. Early results indicate need to encourage physicians to increase utilization of the
evidence-based order sets to achieve consistent health outcomes.
2. Need larger utilization of sets to produce more convincing statistics.
3. Much of qualitative research published to date supports potential benefit of order
sets to improve quality of care and reduce cost. Need more quantitative support.
4. Advocate possesses sufficient EMR base for increasing data mining efforts
necessary to drive quality initiatives.
Literature:
Ballard, D. J., Ogola, G., Fleming, N. S., Stauffer, B. D., Leonard, B. M., Khetan, R., & Yancy, C. W.
(2010). Impact of a standardized heart failure order set on mortality, readmission, and quality
and costs of care. International Journal for Quality Health Care, 22(6), 437-444.
Observational study of the effects of CHF order sets on mortality, readmission, and cost of care at Baylor Health. Utilization
of order sets 73.1%. Increased compliance with core measures (1.51 & 1.08; 2.12 at the 95% confidence level) and reduced
in-patient mortality (0.49 & 0.28; 0.88 at the 95% confidence level).
Best, J. T., Frith, K., Anderson, F., Rapp, C. G., Rioux, L., & Ciccarello, C. (2011). Implementation of an
evidence-based order set to impact initial antibiotic time intervals in adult febrile neutropenia.
Oncology Nursing Forum, 38(6), 661-668.
Evaluated the impact of standardized medication order sets on the time interval in initiation of antibiotic therapy for adult
patients with cancer and febrile neutropenia. 30 patients in control group and 23 in experimental group. Order set usage was
31% in inpatient units and 71% in ED. Overall reduction of time intervals for initiation of antibiotic therapy observed
among experimental group of patients – presentation (t = 2.25, df = 37, p = 0.031) and antibiotic administration (t = 2.67,
df = 40.17, p = 0.012).
Formea, C. M., Picha, A. F., Griffin, M. G., Schaller, J. A., & Lee, M. R. (2010). Enhancing
participant safety through electronically generated medication order sets in a clinical research
environment: A medical informatics initiative. Clinical and Translational Science Journal,
3(6), 312-315.
Mayo Clinic introduced formal comprehensive order set review process that emphasized and increased utilization,
strengthened buy-in, and improved safety.
Khanna, R., Vittinghoff, E., Maselli, J., & Auerbach, A. (2011). Unintended consequences of a standard
admission order set on venous thromboembolism prophylaxis and patient outcomes. Journal
of General Internal Medicine, Springer Online First.
Study of unintended consequences of VTE order sets on certain groups of patients at UCSF Medical Center. There was an
overall rise in VTE cases after implementation at 58% after versus 51% before, p < 0.001, while patients with potential
harm from the sets exhibited larger increase in VTE – odds ratio = 1.58 (1.12 & 2.22 at 95% CI).
Thiel, S. W., Asghar, M. F., Micek, S. T., Reichley, R. M., Doherty, J. A., & Kollef, M. (2009). Hospitalwide impact of a standardized order set for the management of bacteremic severe sepsis.
Critical Care Medicine, 37(3), 819-824.
Severe sepsis order set study at 1,200-bed Jewish-Barnes hospital. Patients in the experimental group received more
intravenous fluids in the first 12 hours after onset of the hypotension (1627 +/- 1862 mL vs. 2054 +/- 2237 mL; p=0.04) and
were more likely to receive appropriate dose of the antibiotic therapy (53% vs. 65.5%, p=0.01). Mortality decreased in the
experimental group (55% vs. 39.5%, p<0.01), along with the hospital stay (28.7 +/- 30.1 days vs. 22.4 +/- 20.9 days,
p=0.02). The experimental group showed reduction in complications: (70.5% vs. 57%, p<0.01), cardiovascular failure
(70.5% vs. 57%, p<0.01), and less likelihood of the vasopressor administration after fluid resuscitation (68.5% vs. 52.5%,
p<0.01).
Wright, A., Sitting, D. F., Carpenter, J. D., Krall, M. A., Pang, J. E., & Middleton, B. (2010). Order sets in
computerized physician order entry systems: An analysis of seven sites.
Proceedings of
the American Medical Informatics Association Symposium, 892-896.
Harvard study of order set utilization and popularity at 7 hospitals: Brigham and Women’s in Boston, Faulkner Hospital in
Boston, Kaiser Permanente Northwest in Oregon/Washington, Massachusetts General Hospital in Boston, Sugar Land
Hospital in Sugar Land, TX, NSMC Union Hospital in Lynn, MA, and Providence Portlar Medical Center in Portland, OR..
Trend towards standardization of care, improved efficiencies, and increased core compliance. No quantitative data.