A methodology to
develop quality
indicators for Health IT
Nicolette de Keizer, Elske Ammenwerth,
Hannele Hyppönen
on behalf of the EFMI and IMIA WG on
Evaluation
Program
Welcome and introduction
eHealth indicator
development
Nicolette de Keizer
Hannele Hyppönen
Method for developing
quality indicators
Case study on CPOE
systems
Group work
Summary and next steps
Nicolette de Keizer
Elske Ammenwerth
Michael Rigby
Background



EU and WHO: ICT solutions in health care are
prerequisites for modern, patient-centered and
efficient health care services
Potential benefit of health ICT is widely
endorsed, but software used in health care is not
necessarily inherently safe
ICT interventions should therefore be thoroughly
evaluated before wide scale implementation
Thorough evaluation

Dissemination of successful ICT interventions is
only possible if quality and success of the ICT
intervention can be measured

-> Quality Indicator (QI)
a
measurable element of a system for which there is
evidence or consensus that it can be used to assess
a defined aspect of the system in question
Problem

There are so many health IT systems
 National
infrastructures for EMR
 Local EMRs
 Decision Support Systems
 CPOE systems
 Telemedicine applications
 PACS
 …….

Some QIs can be shared, others are specific
Aims of the workshop
to inform on and discuss a methodology
for developing QIs for health IT systems
 to draw up the interest within the
community to develop QI for health IT
systems
 to recruit groups that will develop QIs for a
specific type of health IT systems

Program
Welcome and introduction
eHealth indicator
development
Nicolette de Keizer
Hannele Hyppönen
Method for developing
quality indicators
Case study on CPOE
systems
Group work
Summary and next steps
Nicolette de Keizer
Elske Ammenwerth
Michael Rigby
eHealth indicator
development
Definitions
eHealth: A review in 2005 included 36
different definitions for the term e-health. Using, processing, sharing and controlling
health related information in electronic
format who, for what purpose, how?
 Indicator: a measurable element of
practice or system for which there is
evidence or consensus that it can be used
to assess a defined aspect of the practice
or system in question


Why are eHealth indicators needed?
 to
learn from initiatives, to see if they are worth
the money, to make informed decisions needed
in management of eHealth on local and national
(international) level

State of the art:
 eHealth
indicator-work is lagging behind,
indicators ambiguous or missing.
 Methodologies not transparent, gaps in existing
classifications used to group eHealth indicators,
gaps in indicators proposed by the experts need for a more formal generic methodology to
define indicators
Development of eHealth indicators
Generic
-
specific


Literature on indicator methodology
 Articles
discussing suitability of specific indicators as
measures in a specific field (e.g. drug treatment data
as an epidemiological indicator)
 Articles describing a methodology for defining indicators
for a specific assessment topics (e.g. care process
quality indicator)
 Articles describing a framework for defining indicators
for specific policy goals (e.g. eHealth or sustainable
development)
Category 3 was most suited as a starting point for
creation of a generic methodology (common for
different topics)
Two main approaches for indicator definition


Expert-led Top-down methodology: predominant in fields
focusing on monitoring implementation of policies and
their impact on society level (e.g. economic growth, main
aim also in European level eHealth indicator work). .
Bottom-up methodology: predominant in the fields
aiming to monitor or assess policy or strategy
implementation and impacts on micro level – e.g. on
local environment.

indicators tailored to the needs and resources of the indicator
users, still rooted in the policy in question (e.g. sustainable
development in environmental policy)
Common phases to proceed

Defining the context (human and environmental) for
measurement: key stakeholders and relevant systems. For
whom, about what, what for? Defining systems OR key
functionalities needed for core tasks?



Defining the goals. Top–down approaches rarely include
this, the goals are pre-determined by government offices
or policies. Whose goals, which goals for which stakeholders?
Defining methods for indicator selection and
categorization by reviewing existing indicator work,
expert knowledge, peer- reviewed literature, selection of
key indicators per purpose
Defining the data. This step tests the indicators by
applying them. Initiative data are collected, analyzed,
reported and feedback is acquired from different user
groups.
Questions
system vs functionality-based indicator
work (OECD eHealth indicators are
functionality-based, much of the existing
indicator data is collected system-based)
 Need for transparency of goals,
stakeholders and use purposes.

Program
Welcome and introduction
eHealth indicator
development
Nicolette de Keizer
Hannele Hyppöne
Method for developing
quality indicators
Case study on CPOE
systems
Group work
Summary and next steps
Nicolette de Keizer
Elske Ammenwerth
Michael Rigby
A methodology to
develop quality
indicators for Health IT
Background

RAND cooperation – methodology for
clinical QI development
 weak
reliability of the rating and consensus
procedures
modified by Van Engen et al [MIE2011]
 We adjusted this ‘modified RAND method’
to the context of quality of health IT

Methodology
1. Expert and
stakeholders panel
2. Literature
research
4. Inductive content
analysis: Draft set of QI
5. Individual rating
6. Group discussion
and anonymous voting
7. Define target standards
(repeat step 1-6)
3. Review of
guidelines
Expert and stakeholders panel


Search for representatives of stakeholders of the
system: developers, researchers, users
Clear invitation



goal of the questionnaire and a response date
their involvement in the rest of the project etc.
Web-based survey (LimeSurvey) on QI


subtasks or dimensions of the system
UMIT support development survey
Literature research
Search terms concerning the field of
interest (e.g. CPOE) combined with MeSH
terms and keywords referring to
‘effectiveness’, ‘assessment’, ‘outcome’,
‘quality assurance’ or ‘quality indicators’,
‘evaluation’ or ‘monitoring’.
 Use the evaluation inventory available
from http.//evaldb.umit.at to

Review of guidelines

Clinical guidelines important for clinical QI
 desirable
level of outcomes of care
 minimum procedures, standards and facilities
that services should include

(ISO/CEN) standards in the field of interest
Inductive content analysis


Combine results of the three sources
Indentify themes/concepts

“I like to know how much time it cost to enter a patient consult”
1. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982 - 2002.
Methods of Information in Medicine. 2005;44:44-56
Inductive content analysis



Combine results of the three sources
Indentify themes/concepts
Classify QIs:

http.//evaldb.umit.at 1





Delone and Maclean






structural quality
information logistics quality
effects on quality of processes
effects on outcome quality of care
Information quality
System quality
Use
User satisfaction
Further fine-grained classifications
UMIT support inductive content analysis
1. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982 - 2002.
Methods of Information in Medicine. 2005;44:44-56
Individual rating

Likert scale from 1 (total disagreement)
to 5 (total agreement):

Importance
 Actionability
 Easy to record or obtain

Rank on mean score
Group discussion and anonymous
voting

Consensus in expert panel through rounds
of debate, discussion and an anonymous
voting process
 Face
to face meetings
 web-based chat meetings
Define target standards

Repeat step 1-6 to obtain target values for
each of the review criteria
 QI:
time to enter one patient
 Value: < 1 minute
Program
Welcome and introduction
eHealth indicator
development
Nicolette de Keizer
Hannele Hyppöne
Method for developing
quality indicators
Case study on CPOE
systems
Group work
Summary and next steps
Nicolette de Keizer
Elske Ammenwerth
Michael Rigby
CPOE

Computerized Physian Order Entry
Systems

Typical functionality:
 Review
recently given drugs
 Modify drugs/order new drugs
 Decision support: Drugs overdosage, drugdrug-interaction, contraindications, allergies
etc.
CPOE: Benefit

CPOE systems have been found to be
effective to reduce medication errors and
- partly - ADEs
Risk ratio
(95% CI)
Study
Potts (2004)
Pestotnik (1996)
Cordero (2004)
Colpaert (2006)
Bates (1999)
Teich (2000)
Kim (2006)
Bizovi (2002)
King (2003)
Evans (1998)
Oliven (2005)
Gandhi (2005)
Kirk (2005)
Koide (2000)
Galanter (2005)
Mullett (2001)
Igboechi (2003)
Chertow (2001)
Peterson (2005)
Shulman (2005)
Tamblyn (2003)
Feldstein (2006)
Maurer (2003)
Mitchell (2004)
Spencer (2005)
0.01 ( 0.00,
0.02 ( 0.01,
0.04 ( 0.00,
0.13 ( 0.09,
0.14 ( 0.11,
0.27 ( 0.24,
0.27 ( 0.11,
0.30 ( 0.16,
0.33 ( 0.27,
0.34 ( 0.28,
0.37 ( 0.32,
0.39 ( 0.27,
0.45 ( 0.39,
0.46 ( 0.40,
0.53 ( 0.46,
0.59 ( 0.52,
0.61 ( 0.58,
0.65 ( 0.63,
0.66 ( 0.58,
0.72 ( 0.54,
0.80 ( 0.73,
0.85 ( 0.77,
0.87 ( 0.81,
1.02 ( 0.88,
1.26 ( 1.03,
.1
1
Risk ratio
Ammenwerth et al, JAMIA, 2008
10
0.01)
0.02)
0.67)
0.17)
0.20)
0.30)
0.64)
0.57)
0.41)
0.40)
0.43)
0.55)
0.51)
0.52)
0.61)
0.67)
0.64)
0.68)
0.74)
0.97)
0.88)
0.94)
0.94)
1.19)
1.55)
CPOE: Challenges

Increasing reports of problems when
introducing CPOE systems
 Unexpected
increased mortality after CPOE
implementation, YY. Han, 2005
 Unintended consequences of CPOE, Joan
Ash, 2007
CPOE: Needs for evaluation

Systematic monitoring and evaluation of
CPOE seems needed

What are good quality indicators to
monitor CPOE quality?
Workshop: CPOE QI indicators
1. Expert and
stakeholders panel
2. Literature
research
4. Inductive content
analysis: Draft set of QI
5. Individual rating
6. Group discussion
and anonymous voting
7. Define target standards
(repeat step 1-6)
3. Review of
guidelines
Group 1: The Expert Panel

Task:
 Collect
characteristics of excellent CPOE
performance
 What do you need to know about a CPOE
implementation in order to assess its quality?

Expected results:
 List
of proposed quality indicators
Group 2: The Literature Team

Task:
 Read
abstracts of CPOE studies from
http://evaldb.umit.at or from PubMed
 Identify CPOE quality indicators used in
these studies

Expected results:
 List
of identified quality indicators
Classifying the Quality Indicators
1.
QI concerning structural CPOE quality

2.
QI concerning impact on process quality

3.
E.g. system quality, information quality, usage
E.g. impact on clinical workflow,
communication, medication errors
QI concerning impact on outcome quality

E.g. mortality, length of stay, ADE
Classifying the Quality Indicators

You can modify these three main categories
by adding further sub-categories when
needed

Setting up the groups

Define time frame for groups

Working in groups

Result reporting
Next steps

Are you interested to lead a group on QI
for a special type of system (e.g. CPOE,
LIS, RIS, EHR)? Please contact us!

Evaluation WG provides support for
organizing group, conducting survey,
analysing intermediate results
Contact
Elske Ammenwerth, UMIT,
elske.ammenwerth@umit.at
 Nicolette de Keizer, Amsterdam Medical
Center, n.f.keizer@amc.uva.nl


Further information: http://iig.umit.at/efmi