Implementation of quality indicators:
barriers and facilitators
Peter HJ van der Voort, MD, PhD, MSc
Dept of intensive care
Onze Lieve Vrouwe Gasthuis
Amsterdam, The Netherlands
Content
•
•
•
•
•
•
Indicators and quality improvement
The Dutch Intensive Care Registry
Barriers to implementation of QI
Facilitators
Implementation strategies
From indicator to improvement of care
– InFoQi study with the Dutch indicator set
Dutch National Intensive Care
Registry (NICE)
Benchmark of outcome
• SMR
• length of stay
Benchmark of outcome
• SMR
• length of stay
• How to improve?
Dutch Society of Intensive Care
• 2003-2004 development of an indicator
set to analyse quality of care
• Based on quality domains of IOM
National Guideline On Intensive
Care Organisation
• 1993, revised 2005, implemented 2006
• Section on quality improvement:
• Quality Indicator set NVIC incorporated in guideline
• No indicators to follow implementation of the
guideline
How to use Quality Indicators
As a tool to measure the implementation
of a guideline
Guideline /
QI plan
Adjustment of
QI plan
implementation
Analysis*
* Indicators
• E.g. 2006 Guideline on organisation
advises to have regional partners for
collaboration (volume – outcome
relation). To discuss individual patients.
• Indicator: % patients discussed from
total admitted
How to use Quality Indicators
As a part of Total Quality Management
The Dutch Quality Indicator Set
• Cooperation between NICE and NVIC
• Set developed by NVIC
• Benchmark by NICE
• Pilot: registration workload, definitions
• Active promotion to Dutch ICU’s
Implementation of the indicator set
appeared to be a QI plan by itself
Implementation of the indicator set
appeared to be a QI plan by itself
Problems
– Create the sense of urgency to use indicators
– How to implement the registration of
indicators
– How to feed-back
– How to make conclusions
– How to implement changes
Definition of
quality
indicators
Quality
Domains
Adoption
Diffusion
Dissemination
AIRE
Implementation
of indicators
• Purpose, relevance and organisation where the indicator
appoints to
• Involvement of professionals
• Scientific evidence
• Additional reasoning and use
Definition of
quality
indicators
Quality
Domains
AIRE
Implementation
of indicators
Adjustment in
care process or
indicator
implementation
Feedback of
analysed data
Registration of
raw data
PDSA
Analyses
indicator data
Avedis Donabedian 1919-2000
o Structure:
o Organisation, resources and equipment
o Process:
o Process of care between caregiver and patient
o Outcome
o Results (at patient level)
Dutch indicators - internal
•Availability of intensivists
Safe
Timely
Patient
centred
Effective
Structure •Nurse to patient ratio
•Policy to prevent medication errors
•Registration of patient- and family
satisfaction
Process •Length of stay in the ICU
Equitable
Efficient
•Duration of ventilation
•Number of days with 100% beds
occupied
Outcome •Glucose regulation
•Mortality
•Incidence of pressure sores
•Unplanned extubation
Definition of
quality
indicators
Quality
Domains
AIRE
Implementation
of indicators
Adjustment in
care process or
indicator
implementation
Feedback of
analysed data
Registration of
raw data
PDSA
Analyses
indicator data
Decision to use a set of indicators
Organize and implement registration of
data
Data validity
Data export
Analysis and benchmark (NICE)
Feedback
Interpretation and conclusion
Plan for change
Implementation of changes / new
methods
Barriers on all levels
Decision to use a set of indicators
– Sense of urgency
– Intrinsic motivation to improve
– Legislation
– Hospital directors
– Society of Intensive Care
– Convince that using indicators
improve care
What we did:
Inform, offer tools
Decision to use a set of indicators
Organize and implement registration of
data
Data validity
Data export
Analysis and benchmark (NICE)
Feedback
Interpretation and conclusion
Plan for change
Implementation of changes / new
methods
• Intervention study: InFoQi to improve QI
using indicators
• 3 interventions:
– Extensive feedback
– QI team
– Educational Outreach
• To develop an optimal intervention
– Literature search on optimal feedback
– Search for barriers in literature, expert
groups, questionnaire
Definition of
quality
indicators
Quality
Domains
AIRE
Implementation
of indicators
Adjustment in
care process or
indicator
implementation
Feedback of
analysed data
Registration of
raw data
PDSA
Analyses
indicator data
Implementation strategies
using indicator data
Educational meeting
Educational outreach
Audit and feedback
Development of a quality improvement plan
Financial incentives
Supporting activities:
– Distribution of educational material
– Use of a local opinion leader
– Development of a quality improvement team
Educational meeting
Participation in conferences, seminars,
lectures, workshops or training
sessions.
During these meetings, feedback of
quality indicators is presented, and
study participants discuss how to
improve performance.
Educational outreach
A trained independent person or
investigator meets with health
professionals or managers in their
practice setting to provide information
(e.g. feedback of quality indicators).
Development of a quality
improvement plan
A plan based on indicator data to be
used to improve the quality of care.
Financial incentives
Rewarding individual health
professionals or institutions with higher
payments when they improve
performance.
Audit and feedback
Giving a report, including a summary of
clinical performance over a specified
period of time.
“any summary of clinical performance over a
specified period of time”
“It is striking how little can be discerned about the
effects of audit and feedback based on the 118 trials
included in this review.”
“Low baseline compliance with recommended practice
and higher intensity of audit and feedback were
associated with larger adjusted risk ratios (greater
effectiveness) across studies.”
Feedback of
analysed data
• Jamtvedt et al. reviewed information feedback
based on any health care data source, while we
focused on feedback based on data from medical
registries
• we aimed to include not only RCTs, but any peerreviewed paper on information feedback within the
context of a medical registry. Furthermore, where
Jamtvedt et al. only reported on the effectiveness
of information feedback, we also aimed to identify
the barriers and success factors to this
effectiveness as reported in the literature
• 53 papers; 50 feedback initiatives
• 24 analytic papers for 22 studies evaluating the
effect of a feedback method on one (n=8) or
more (n = 14) primary, clinical outcome
measures
• Positive effect on all outcome measures: 4
• Mix of positive and no effects: 8
• 10 not any effect.
• None of the 22 studies reported a negative
effect.
MFA = multifaceted approach
“To review the literature concerning strategies for implementing
quality indicators in hospital care, and their effectiveness in
improving the quality of care”
• 21 studies (9 RCT, 2 CCT and 10 B-A)
• 17 US; 14 cardiovascular care
• 1-379 participating hospitals
•
•
•
•
Indicators and hospital care
20 on process care
6 on patient outcomes
Follow up 6 months
de Vos et al. Int J Qual Health Care 2008;1-11
de Vos et al. Int J Qual Health Care 2008;1-11
de Vos et al. Int J Qual Health Care 2008;1-11
• Study design unrelated to effectiveness
• Results on outcomes:
– 4 studies indicators ineffective
– 1 partially effective
– 1 effective
Effective: > 50% sign improvement; partially effective:
+/- 50% improvement; ineffective: <50% improvement
• Results on process:
– 20 studies
–
–
–
–
3 no significant improvement at all
8 improvement in some
7 partially effective
2 significant improvement in all process
indicators
Most are effective on process of care
de Vos et al. Int J Qual Health Care 2008;1-11
de Vos et al. Int J Qual Health Care 2008;1-11
Successful implementation of
indicators
Feedback reports combined with
Educational implementation strategy
and/or Quality Improvement plan
de Vos et al. Int J Qual Health Care 2008;1-11
Barriers
Barriers should be identified before an
implementation strategy is launched
Facilitating factors
•
•
•
•
Supportive / collaborative management
Administration support
Detailed and credible data feedback
Ability of persons receiving feedback to
act on it
ICU
Barriers “knowledge”:
Factors limiting
adherence through a
cognitive component
Barriers “attitude”:
Factors limiting adherence through an affective
component
Barriers “behavior”
Factors limiting adherence
through a restriction of
physician ability
1) I am familiar with the use of quality indicators as a tool to improve quality of care (n= 142)
2) I am familiar with the Dutch set of ICU quality indicators (n=142)
3) I understand the importance of using quality indicators (n=142)
4) In general, I do not offer resistance towards working with quality indicators (n=142)
Strongly disagree
5) I am willing to implement quality indicators in daily practice (n=141)
Disagree
6) Feedback on quality indicators stimulates me to adjust my practice (n=142)
Neutral
Agree
7) M onitoring of quality indicators stimulates quality improvement (n=140)
Strongly agree
8) M onitoring of quality indicators leads to reliable benchmark data for ICUs (n=140)
9) M onitoring of quality indicators fits into the daily routines in the hospital setting (n=140)
10) M onitoring of quality indicators can be done without huge investments (n=140)
11) M onitoring of quality indicators does not take too much time (n=139)
0%
10%
20%
30%
40%
50%
60%
70%
Proportion of responding respondents
80%
90%
100%
Barrier 1 of 15
Barrier 1
How the barrier is targeted by the
feedback intervention
Lack of knowledge on how to interpret
the data
“people are not being taught how to handle the
results, how to interpret them.”
Barrier 1 of 15
Barrier 1
How the barrier is targeted by the
feedback intervention
Lack of knowledge on how to interpret During educational outreach visits the
the data
facilitators support the QI team in
interpreting their performance data in
the reports and in formulating a QI
action plan
“people are not being taught how to handle the
results, how to interpret them.”
Barrier 2 of 15
Barrier 2
How the barrier is targeted by the
feedback intervention
Lack of information to initiate QI
actions
“You want to improve the quality, but you don’t
know where to start or where the real problems lie”
Barrier 2 of 15
Barrier 2
How the barrier is targeted by the
feedback intervention
Lack of information to initiate QI
actions
The feedback reports contain
extended information on six of the
indicators; During educational
outreach visits the facilitators support
the QI team in further exploration of
data in the NICE registry
“You want to improve the quality, but you don’t
know where to start or where the real problems lie”
Barrier 3 of 15
Barrier 3
How the barrier is targeted by the
feedback intervention
Lack of trust in data
“The data are often regarded as unreliable. If you
put rubbish in, you will only get rubbish out. Trust in
the data is essential.”
Barrier 3 of 15
Barrier 3
How the barrier is targeted by the
feedback intervention
Lack of trust in data
During educational outreach visits the
facilitators discuss with the QI team
completeness and correctness of the
data sent to the NICE registry and -if
necessary- support them in
formulating actions to improve their
data quality.
“The data are often regarded as unreliable. If you
put rubbish in, you will only get rubbish out. Trust in
the data is essential.”
Barrier 4 of 15
Barrier 4
How the barrier is targeted by the
feedback intervention
Lack of statistical power for small
ICUs
Not targeted by the intervention
“small number of patients lead to broad confidence
intervals”
Barrier 5 of 15
Barrier 5
How the barrier is targeted by the
feedback intervention
Lack of case-mix correction
“the ‘my patients are sicker’ syndrome.”
Barrier 5 of 15
Barrier 5
How the barrier is targeted by the
feedback intervention
Lack of case-mix correction
Besides already available case-mix
corrected hospital mortality data, data
are stratified based on admission type
or on APACHE IV diagnosis. During
educational outreach visits the
facilitators support the QI team in
formulating additional case-mix
related analyses on data in the NICE
registry
“the ‘my patients are sicker’ syndrome.”
Barrier 6 of 15
Barrier 6
How the barrier is targeted by the
feedback intervention
Level of aggregation too high
“the care providers need data at practice
level, not only at the organisation level.”
Barrier 6 of 15
Barrier 6
How the barrier is targeted by the
feedback intervention
Level of aggregation too high
Besides data aggregated on ICU
level, the feedback reports contain
data on patient or shift level for six of
the indicators.
“the care providers need data at practice
level, not only at the organisation level.”
Barrier 7 of 15
Barrier 7
How the barrier is targeted by the
feedback intervention
Insufficient timeliness
“…the information might not have been presented
close enough to the time of decision making.”
Barrier 7 of 15
Barrier 7
How the barrier is targeted by the
feedback intervention
Insufficient timeliness
As the monthly reports do not contain
comparisons with other ICUs, it is
possible to decrease the time
between the end of a period and
reporting data on this period from ten
(for quarterly reports ) to six weeks
(for monthly reports).
“…the information might not have been presented
close enough to the time of decision making.”
Barrier 8 of 15
Barrier 8
How the barrier is targeted by the
feedback intervention
Lack of intensity
“…the care providers received prescriber feedback
letters only once.”
Barrier 8 of 15
Barrier 8
How the barrier is targeted by the
feedback intervention
Lack of intensity
In addition to the quarterly reports, the
QI team receives monthly feedback
reports containing their performance
data presented in a different way.
“…the care providers received prescriber feedback
letters only once.”
Barrier 9 of 15
Barrier 9
How the barrier is targeted by the
feedback intervention
Lack of outcome expectancy
“…the current rates were not considered a
problem.”
Barrier 9 of 15
Barrier 9
How the barrier is targeted by the
feedback intervention
Lack of outcome expectancy
During educational outreach visits the
facilitators discuss with the QI team
the opportunities for improvement
“…the current rates were not considered a
problem.”
Barrier 10 of 15
Barrier 10
How the barrier is targeted by the
feedback intervention
Lack of trust in QI principles
“It is difficult to convince staff to use continuous
quality improvement principles.”
Barrier 10 of 15
Barrier 10
How the barrier is targeted by the
feedback intervention
Lack of trust in QI principles
The facilitators discuss with the QI
team members the principles of
systematic QI during the educational
outreach visits.
“It is difficult to convince staff to use continuous
quality improvement principles.”
Barrier 11 of 15
Barrier 11
How the barrier is targeted by the
feedback intervention
Lack of dissemination of information
“…inadequate dissemination within the hospitals.”
Barrier 11 of 15
Barrier 11
How the barrier is targeted by the
feedback intervention
Lack of dissemination of information
Each QI team member receives the
feedback reports by e-mail. During
educational outreach visits and in
monthly reminders they are
encouraged to share their findings
with the rest of the staff
“…inadequate dissemination within the hospitals.”
Barrier 12 of 15
Barrier 12
How the barrier is targeted by the
feedback intervention
Lack of motivation
“As the intervention was unsolicited, the participants
had not agreed to review their practice.”
Barrier 12 of 15
Barrier 12
How the barrier is targeted by the
feedback intervention
Lack of motivation
The members of the QI team should
be selected based on their affinity and
experience with measuring and
improving quality of care and their
capability to convince staff to be
involved in QI activities
“As the intervention was unsolicited, the participants
had not agreed to review their practice.”
Barrier 13 of 15
Barrier 13
How the barrier is targeted by the
feedback intervention
Organizational constraints
“You will need a change of organizational
culture…That will take some time to achieve.”
“Patient care is the main task and [QI activities are]
just an extra”
Barrier 13 of 15
Barrier 13
How the barrier is targeted by the
feedback intervention
Organizational constraints
The QI team forms the organizational
basis for monitoring performance and
initiating QI activities. One of their
tasks is formulating a QI action plan
corresponding with the opportunities
for improvement within their own
organization. They are also asked to
discuss their performance during
monthly QI team meetings, using the
available reports and their QI plan as
a basis. They are encouraged to
report their findings during regular
existing staff meetings.
“Patient care is the main task and [QI activities are]
just an extra”
Barrier 14 of 15
Barrier 14
How the barrier is targeted by the
feedback intervention
Lack of resources
Not targeted by the intervention
“Monitoring of quality indicators takes too
much time and money”
Barrier 15 of 15
Barrier 15
How the barrier is targeted by the
feedback intervention
External barriers
Not targeted by the intervention
“…there is [a lack of] public awareness now of the
need to [improve the quality of care]”
Rules and policy (n=140)
Pay-for-performance (n=138)
Intrinsic motivation (n=140)
Encouragement from scientific society (n=137)
Social demand for transparency (n=140)
Strongly disagree
Social pressure from hospital management (n=140)
Disagree
Possibilities to improve care (n=141)
Neutral
Agree
Administrative support (n=141)
Strongly agree
Receiving feedback (n=140)
Reminders for registration (n=140)
Education (n=140)
Quality improvement team (n=140)
Opinion leader (n=140)
0%
20%
40%
60%
Proportion of responding respondents
80%
100%
Determinants of facilitating
factors
• Administrative support p=0.02
(physicians)
• Education p=0.01 (nurses)
• Feedback p=0.001 (managers)
• Opportunities to improve care p=0.003
(physicians)
Additional facilitators
• Patient Data Management System /
user friendly software
• Appointment of a Quality Manager
• Appointment of one person responsible
for coordination
Determinants
• Of knowledge:
– Being manager > health care prof (p=0.004)
– Intensivists > nurses (p=0.01)
– 40-49 yr
– Academic/teaching hosp > non-teaching
• Of attitude
– No significant differences
• Of behaviour
– >49 yr pos related to overall behaviour
– Non teaching neg related
Definition of
quality
indicators
Quality
Domains
AIRE
Implementation
of indicators
Adjustment in
care process or
indicator
implementation
Feedback of
analysed data
Registration of
raw data
PDSA
Analyses
indicator data
Decision to use a set of indicators
Organize and implement registration of
data
Data validity
Data export
Analysis (NICE)
Feedback
Interpretation and conclusion
Plan for change
Implementation of changes / new
methods
• Intervention study: InFoQi to improve QI
using indicators
• 3 interventions on identified barriers:
– Extensive feedback
– QI team
– Educational Outreach
• Multicenter cluster randomised trial
• From October 2008 – October 2010
• 30 ICU’s
Definition of
quality
indicators
Quality
Domains
AIRE
Implementation
of indicators
Adjustment in
care process or
indicator
implementation
Feedback of
analysed data
Registration of
raw data
PDSA
Analyses
indicator data
Conclusions (1)
Quality Indicators should meet strict
criteria
The implementation of Quality
Indicators should follow the PDSA cycle
The effect of implementation of QI has
not been studied in ICU’s
Conclusions (2)
To achieve effect a multifaceted
approach is needed
based on pre-defined barriers
Sabine vd Veer
Maartje de Vos
Nicolette de Keizer
Gert Westert
Nice participants
Implementation
of indicators
• Implementation policy
– National
– Hospital
– ICU
• Support
• QI implementation team
– multidisciplinary
Decision to use a set of indicators
Organize and implement registration of
data
Data validity
Data export
Analysis and benchmark (NICE)
Feedback
Interpretation and conclusion
Plan for change
Implementation of changes / new
methods
Registration of
raw data
• Who
• Complete
• Reliable
• Validation
– distribute
• When
– Part of the care process
– Daily workflow
– Responsibility
• How
– Paper / PDMS
Registration pitfalls
• Definitions
– NICE datadictionary
• Erroneous measurements
– Room temp
• Inappropriate data collected
– Wrong time period
• Extractions to export file
– Default values
• Decision to use a set of indicators
• Organize and implement registration of data
• Responsible persons for parts of the registration
on daily basis
• Overall responsibility for one person
• Data validity
– Data quality checks every year locally
– Data quality reports monthly
• Data export
– We do not offer a ICT tool for export
– Close collaboration with providers
• Decision to use a set of indicators
• Organize and implement registration of
data
• Data validity
• Data export
• Analysis (NICE)
– Clinical Information Department in an
academic hospital
• Decision to use a set of indicators
• Organize and implement registration of
data
• Data validity
• Data export
• Analysis (NICE)
• Feedback
Studies on barriers?
Problems in:
• gathering support
• personnel, management
• disseminating information
• registration / technical / political problems
“% of hospital staff and % of senior managers participating
in formally organized QI teams are associated with better
values on quality indicators. Percentage of physicians
participating in QI teams is not associated with better
values”
“41% felt that monitoring programs did not assist them in
improving care. Providers cited numerous barriers to
improving care processes.”
Am J Psychiatry 2004; 161:146–153