This work is supported by a grant from the CDC
to the Academic Pediatric Association and
in-kind contributions from
Children’s Mercy Hospital in Kansas City, MO.
QUALITY IMPROVEMENT
TRAINING:
Module #4
How will we know that a change is an improvement?
An introduction to QI measurement
Objectives
After viewing this segment, you will be able to:
1. Describe what makes up a complete set of
measures.
2. Name the 2 reasons to define measures and
collect data.
3. List at least 3 important features of data
collection for QI.
4. Describe the elements of a run chart
Effective
measurement
requires a complete
set of measures
3 Types of Measures
 Outcome measures
 Process measures
 Balancing measures
Outcome Measure
 The voice of the patient or other
stakeholder, e.g., parent or payer
 Describes where there’s value in the process
 Sometimes we use a proxy, for example:
 HPV vaccination rate
 Average hemoglobin A1c level
Process Measure
 Are the steps that are required in order to
reach the intended outcome being
followed?
 Describes the workings of the system
 Example
Date for next
dose on DC
paperwork
Our site has a
problem with
HPV vaccine
doses 2 & 3
Adolescent
immunization
rate (complete)
Balancing Measure
 Are other parts of the system being
disrupted by our changes?
 Describes unintended negative
consequences
 Example
Increased no
show rate
Date for next
dose on DC
paperwork
Our site has a
problem with
HPV vaccine
doses 2 & 3
Adolescent
immunization
rate (complete)
OR utilization
time
Change in
the steps of
the
operation
Step XX was
giving us
difficulty
Patient
survival rate
Why should you
define measures
and collect
data?
Nolan TW. Execution of Strategic Improvement Initiatives to Produce System level results. IHI
Innovations Series White Paper. Cambridge, Mass: Institute for Healthcare Improvement; 2007
Use data to investigate the problem
and to drive improvement -Establishing a learning system to increase
the chance of producing the intended results
Using Data to Investigate
 Do we truly have a problem?
 What exactly are we trying to accomplish?
 What issues are at the root of our problem?
 What factors cause the problem more or less
than others?
 Does the data confirm or disprove my theories?
 How frequently & to whom does the
problem occur?
 What is the baseline level of variation?
Using Data to Drive Improvement
 To focus our change strategies
Using Data to Drive Improvement
 To focus our change strategies
 To see variation over time
 To determine if an intervention is successful
 To see if our interventions resulted in
improvement
Some important
features of data
collection for QI
Langley, G., Moen, R., Nolan, K., Nolan, T., Norman, C., & Provost, L. (2009) The
Improvement Guide: A Practical Approach to Enhancing Organizational
Performance, 2nd Edition. San Francisco: Jossey-Bass.
Measurement Tips
 Collect data before and after a change
 Use clear operational definitions for your
measures
Operational Definition
 Establishes a shared understanding of a measure
 Removes ambiguity
 Includes:
 The method of measurement
 Criteria for judgment
 Example: Outcome measure-HPV vaccination initiation
 Population: As of 2/1/2015, pts at least 11 years of age &
seen for any visit during the past 2 years (per billing
data)
 Measure: Proportion of the target population that has
received at least 1 dose of any HPV vaccine (per EHR
extract) (irrespective of allergies, etc.)
Reliability & Validity
 Reliability: Can you repeat the
measurement process multiple times and
get the same result?
 Validity: Does the process really measure
what you are trying to measure?
Measurement Tips
 Collect data before and after a change
 Use clear operational definitions for your
measures
 Use a reliable and valid data collection system
 Keep measures to 6 or less
 Make sure that your data collection is easy
 Use qualitative data to give your quantitative
data depth
 Document your data collection plan
(who, what, when, how)
Displaying data over
time helps interpret it.
Constructing a
run chart
Traditional Research Approach
Same annual data,
more complete story
Same annual data,
very different story
To find the median, list your pre-cycle data in numerical order and find the
middle value. You need >12 pre-intervention values.
The run chart: a simple analytical tool for learning from variation in
healthcare processes. Rocco J Perla, Lloyd P Provost, Sandy K Murray
4 Elements in a Run Chart:
median, goal, time series, & notes
Reminder
sent
Start
standin
g orders
New
interns
The run chart: a simple analytical tool for learning from variation in
healthcare processes. Rocco J Perla, Lloyd P Provost, Sandy K Murray
4 Elements in a Run Chart:
median, goal, time series, & notes
Standing
orders start
Were the results as you predicted? (Your hypothesis)
•There was less of an increase than we expected
What did we learn from this PDSA cycle?
•The nurses need more training to feel confident with this
Summary
1. A complete set of measures has
outcome, process, and balancing measures;
include each in your project
2. Define measures and collect data to learn more
about your problem and to determine if your
change resulted in improvement.
3. Make sure your data collection system is
reliable, valid and uses clear operational definitions.
4. A run chart displays measurement changes over
time, the goal, the median of the pre-cycle data,
and notes on changes made to the process.
The End
For more on QI measurement, see
IHI. Science of Improvement: How to Improve
http://www.ihi.org/knowledge/Pages/HowtoImprove/ScienceofImproveme
ntHowtoImprove.aspx