Clinical Information System Evaluation
A Framework for Estimating CIS Value and
Identifying Opportunities for Improvement
Why Evaluate?
Three purposes of the evaluation process:
1. To compare the results with the goals and expected effects of the system
2. To direct work towards the expected result with the help of evaluation during the
development of the system
3. To use the findings and outcomes as an experience base for the next project
(Peterson & Jelger, 1988)
Clinical Information System Evaluation
Framework
Does IT make a difference?
Literature Review
•
The earliest studies were done at the Regenstrief Institute in the late 1970s, evaluating the
effectiveness of CIS on adherence to influenza vaccination. Since then, there have been
hundreds of articles seeking to evaluate CIS in various domains: quality, efficiency, and
costs.
•
Neumann, Parente, and Paramore (1996) reviewed eleven studies and presented a
consolidated analysis of each. They found that fully automating administrative functions
could save between $5 and $8 billion annually.
•
The “Most Wired” report (Solovy, 2001) by Hospital and Health Networks and Deloitte
Consulting shows that “most wired” hospitals have better control of expenses and higher
productivity.
Clinical Information System Evaluation
Framework
Does IT make a difference?
Literature Review
• Chaudhry et al. conducted a systematic review on 257 articles evaluating health
information technology.
–
–
–
–
–
Quality improvement: increased adherence to guidelines, enhanced disease surveillance, and
decreased medication errors.
Major efficiency benefit: decreased utilization of care
Effect on time utilization is mixed
Empirically measured cost data is limited and inconclusive
Major limitation of the literature is its generalizability
Chaudhry B, Ann Intern Med. 2006;144:E-12-E-22.
Two Approaches to CIS Evaluation
Objectivist Approach
Quantitative
Important system attributes that can be
measured and interpreted
Subjectivist Approach
Qualitative
Observation results are dependent on
context and observer; different
individuals or groups may hold a
different opinion about a systems value
(Burkle et. Al, 2001)
Two Approaches to CIS Evaluation
Objective Approach
Has seldom produced positive evaluation results when applied to complex clinical
information systems
– Consumed large amounts of resources
– Expected and measured objective parameters, such as time saving, did not exceed those of the
control
Subjective Approach
Very often, the influence of such a system will not manifest in direct time savings, but
in:
– Improved cooperation between departments
– Increased quality of documentation
– Better patient care
(Burkle et. al, 2001)
Objectives of the CIS Evaluation Framework
Internal
• To provide a method for assessing CIS systems in order to
quantify value and identify opportunities for improvement
External
• To contribute to the broader healthcare community by
designing a generalizable methodology for value-rating CIS
system technologies in order to facilitate decisions on
technology implementations
Clinical Information System Evaluation
Framework
Is the CIS
used?
Does it make
a difference?
Extent of
technology
facilitation of
outcomes ?
Benefits Realization Number
Individual projects can be evaluated comparatively
for benefit realization
• Quantifies the benefits of the Clinical Information System
• Based on three questions:
–
–
–
•
Is the CIS being used?
Does it make a difference in quality of care, clinician workflow, or business
decision making?
Is the change related to the technology implemented?
Ranking value product of 3 numbers multiplied (highest 27)
–
–
–
Use 1-3
Difference 1-3
Sharing 1-3
Ranking Scheme – Benefits Realization
Rank
Population
Outcomes
CIS contribution
to outcomes
1
Limited use impacting
< 25% of population
No harm averted,
minor efficiency or
cost savings achieved
Small CIS
contribution
2
Moderate use
impacting 50% of
population
Minor harm averted,
moderate efficiency or
cost efficiency
CIS plus other
approaches
3
Maximum use
impacting 95 - 100%
of population
Transferability to
other centers, serious
harm averted, major
Entirely related to CIS
efficiency
improvement, major
cost savings
BRN = Population Rank x Outcomes Rank x CIS Rank
Benefits of CIS Improvements (08)
Benefit for Patient
Care
Problem
Addressed
Technology
Applied
Outcomes
BRN 1-27
Safe & Effective
Care
•72% of pregnancy
Screening
questions
standardized and
specified through
conditional logic
with automatic
ordering of a test as
appropriate
• 97% complete
2-3-2
Easy to use and
mandatory
documentation of
defined elements
for education
100% compliance
since Feb 08
screening
documentation is
complete
•90% of eligible
patients tested preoperatively
May 06
Safe & Effective
Care
75-80% compliance
with discharge
teaching for CHF
patients
screening
documentation
(35% improvement)
•1% improvement
(91%) in eligible
patients being
tested
April/May 08
12
2-2-3
12
Benefits of CIS Improvements
Benefit for Patient
Care
Problem
Addressed
Technology
Applied
Outcomes
BRN 1-27
Safe & Effective
Care
Future orders
available for
mistaken activation
on subsequent
admission
CCL script to
cancel future orders
2 weeks after order
place (discharged
pts only)
Negligible future
orders available for
inadvertent
activation on
readmission
3-3-3
Ordered care
potentially not
carried out because
of un-activated
future orders
Remove future
status from order
set orders where
possible
65% reduction in
chance for care
omissions
2-2-2
73% compliance
with pneumococcal
vaccine
Mandatory screen
and automated
order
100% compliance
with appropriate
vaccination
3-2-3
Safe & Effective
Care
Safe & Effective
Care
27
8
18
Benefits of CIS Improvements
Benefit for Patient
Care
Problem
Addressed
Technology
Applied
Outcomes
BRN
Safe & Effective
Care
Inappropriate doses
ordered based on
age, weight or
kidney function
Dose range
checking for single
and daily doses
28% of orders
change or adverted
secondary to alert
3-3-3
Increased risk for
error with manual
calculation of
mg/Kg doses for
neonates
Dose calculator
and 48 new order
sets and almost
700 order
sentences to guide
appropriate
prescribing
tbd
3-3-3
3.08 falls/1000
patient days
95% compliance
with care process
Enhanced
assessment tools,
prompts for
interventions, alerts
to physicians
2.2 falls/1000
patient days
98.8 compliance
with care process
Safe & Effective
Care
Safe & Effective
Care
27
27
3-2-2
12
The Benefits of CIS Improvements
Benefit Category
Problem
Addressed
Technology
Applied
Outcomes
BRN
Safe & Effective
Care
Less than 95%
compliance with
skin assessment
10% prevalence
New forms for
documentation,
rules for
interventions, MD
notification of issue
electronically
99% compliance
with skin
assessment
7.8 % prevalence
3-2-2
4520 free text
orders per month
causing potential
omissions in care
Order set with
discreet orders for
commonly used
orders
53% reduction in
use of free text
orders (2320);
subjective
improvement in
clinician
satisfaction with
orders
3-1-3
Safe & Effective
Care
12
9
The Benefits of CIS Improvements
Benefit Category
Problem
Addressed
Technology
Applied
Outcomes
BRN 1-27
Coordinated
Quality of shift
hand-offs
Automated report
that abstracts EMR
data – physician
sign out and
nursing SBAR pilot
40% improvement
in perceived
efficiency of nursing
report and 8 fold
improvement in
quality of reports
received.
1-3-2
1524/month
discharge
summaries
dictated by
residents
Electronic note
87% reduction in
dictated d/c
summaries (July
203) Annual cost
savings $191,650
2-2-3
Radiology report
turn-around-time
Radiology voice
recognition
20% reduction in
turn around time (2
days to 38 hrs. 7
mins) – adoption
only at 83% of
Attendings and
44% of reports
2-1-3
Coordinated
Timely & Efficient
6
12
6
The Benefits of CIS Improvements
Benefit Category
Problem
Addressed
Technology
Applied
Outcomes
BRN 1-27
Coordinated
.6 preventable
codes per 100
patient discharges
Mews report to
guide RRT on
proactive
evaluation of
patients who might
deteriorate
.09 preventable
codes per 100
discharges
3-3-2
Lack of direct way
to communicate
between patients &
providers
Relay Health
(patient access to
chart) for Internal
Medicine
30% patients
enrolled; patient
satisfaction results:
1-2-2
Lack of
measurement on
clinical decision
support
performance
relative to
benchmark
Dose Range and
drug/drug
interaction
checking
implemented
Passing score on
the Leapfrog
clinical decision
support test
27
Coordinated
Safe & Effective
Care
18
4