Data is your Friend
Collecting, Charting, Analyzing, and
Interpreting Data to Support Quality
Improvement
Michael Campitelli and Ruth Croxford
QI Epidemiologists, Institute for Clinical Evaluative Sciences (ICES)
Doug Mitchell
Director Decision Support, Guelph General Hospital
Susan Taylor
Director, QI Program Delivery, Health Quality Ontario
www.ideasontario.ca
Faculty:
Ruth Croxford
• Relationships with commercial interests:
– Grants/Research Support: None
– Speakers Bureau/Honoraria: None
– Consulting Fees: None
– Other: None
www.ideasontario.ca
2
Faculty:
Michael Campitelli
• Relationships with commercial interests:
– Grants/Research Support: None
– Speakers Bureau/Honoraria: None
– Consulting Fees: None
– Other: None
www.ideasontario.ca
3
Faculty:
Susan Taylor
• Relationships with commercial interests:
– Grants/Research Support: None
– Speakers Bureau/Honoraria: None
– Consulting Fees: None
– Other: None
www.ideasontario.ca
4
Faculty:
Doug Mitchell
• Relationships with commercial interests:
– Grants/Research Support: None
– Speakers Bureau/Honoraria: None
– Consulting Fees: None
– Other: None
www.ideasontario.ca
5
Outline
• Review (45-50 minutes)
•
•
•
•
Bar charts and Pareto charts
Scatter plots
Run charts and SPC charts
Statistical testing between groups
• Break (10 minutes)
• Case Studies specific to the health care sector
you identify with the most (45-50 minutes)
• Primary care
• Long-Term care
• Acute care
www.ideasontario.ca
6
GUELPH GENERAL HOSPITAL –
HIP AND KNEE REPLACEMENT
IMPROVEMENT INITIATIVE
www.ideasontario.ca
7
Abbreviated, anonymized version of the data
www.ideasontario.ca
8
Tools to learn from variation in data
HC Data Guide, p 65
(fig 2.28)
www.ideasontario.ca
9
Bar Charts
www.ideasontario.ca
10
Bar Charts
100
90
80
70
60
% wait time 1 within target
50
% wait time 2 within target
40
30
20
10
0
A
B
www.ideasontario.ca
C
11
Bar Charts (fictional data)
www.ideasontario.ca
12
Statistics are like bikinis.
What they reveal is suggestive, but what
they conceal is vital.
Aaron Levenstein, Professor of Business, Baruch College
www.ideasontario.ca
13
Pareto Charts
www.ideasontario.ca
14
Pareto Charts
(fictional data)
Reason (fictious data)
Frequency
Referral missing X-rays
86
Referral missing other health
information
75
Patient weight loss required
35
Other medical
22
Patient refused offered
consult date
10
Patient requested specific
surgeon
43
Surgeon schedule
25
www.ideasontario.ca
15
www.ideasontario.ca
16
Scatter Plots
www.ideasontario.ca
17
Scatter Plots
www.ideasontario.ca
18
Scatter Plots
www.ideasontario.ca
19
Histograms
www.ideasontario.ca
20
Bikini #1
www.ideasontario.ca
21
Histograms
% within
target
Mean
Median
A
33%
176
163
B
24%
154
155
C
66%
68
71
Surgeon
www.ideasontario.ca
22
Tools to learn from variation in data
HC Data Guide, p 65
(fig 2.28)
www.ideasontario.ca
23
Run Charts
www.ideasontario.ca
24
Run Charts
www.ideasontario.ca
25
Run Charts
Four (probability-based) rules to identify non-random
signals of change in a run chart (Health Care Data Guide,
pgs 76 – 85)
• A trend
– Five or more consecutive points all going up or all going
down.
• A shift
– Six or more consecutive points either all above or all below
the median
• Too many or too few runs (crossings of the median)
– Depends on the number of points on the graph - requires a
table
• An astronomical data point
www.ideasontario.ca
26
http://www.qihub.scot.nhs.uk/knowledge-centre/quality-improvement-tools/runchart.aspx
www.ideasontario.ca
27
Run Charts
www.ideasontario.ca
28
The Run Chart as a Bikini
www.ideasontario.ca
29
Corresponding Shewhart Chart
www.ideasontario.ca
30
Statistical Process Control (SPC) Charts
www.ideasontario.ca
31
Shewhart Chart Selection Guide
HC Data
Guide p. 151
(fig 5.1)
www.ideasontario.ca
32
Types of Measures – Continuous Variables
Wait 2 (days)
Patient - decision to
ID
procedure
1
312
2
349
3
472
4
315
5
292
6
286
7
255
8
265
9
297
10
272
11
11
12
286
13
122
14
162
15
247
16
281
Month
Case 1 Case 2
Case 3
Case 4
Case 5
Case 6
May-11
312
349
Jun-11
472
315
292
Jul-11
286
255
265
297
272
11
Aug-11
286
122
162
247
281
288
Sep-11
211
289
391
226
272
Oct-11
121
299
243
160
240
278
Nov-11
122
129
164
110
110
138
Dec-11
112
101
104
139
251
87
Jan-12
70
73
79
154
178
173
Feb-12
153
190
344
359
21
286
Mar-12
85
272
202
206
210
182
Apr-12
184
287
208
260
31
122
www.ideasontario.ca
Average
Month
Wait 2
May-11
330.5
Jun-11
359.7
Jul-11
148.2
Aug-11
153.4
Sep-11
277.8
Oct-11
147.9
Nov-11
200.8
Dec-11
155.8
Jan-12
200.3
Feb-12
186.3
Mar-12
152.8
Apr-12
152.7
May-12
105.7
Jun-12
115.3
Jul-12
105.3
33
Types of Measures – Count Data
• Requires two columns of data for each time period: the
count and the number of “opportunities”
• The event being counted can occur more than once per
“opportunity”
Week
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Number of flash
sterilizations
42
47
51
45
36
34
37
49
39
46
28
46
34
44
41
Number of
surgeries
84
146
91
106
88
126
81
86
83
77
78
108
72
131
83
www.ideasontario.ca
Rate of flash
sterilizations (flash
sterilizations per
100 surgeries)
34
Types of Measures – Classification Data
• Requires two columns of data for each time period: the
total number of people or events that were observed,
and the number of “non-conforming” events.
Month
Apr-11
May-11
Jun-11
Jul-11
Aug-11
Sep-11
Oct-11
Nov-11
Dec-11
Jan-12
Feb-12
Mar-12
Apr-12
May-12
Jun-12
Jul-12
Number
within
target wait Total
time
number
0
1
0
2
0
3
0
6
1
12
0
5
1
16
9
29
6
21
7
25
15
39
9
31
7
22
15
28
9
31
5
24
Percent (percent
of patients seen
within the target
time)
www.ideasontario.ca
35
Learning from a Shewhart Chart
• Rules for detecting special cause variation.
• Annotation
• Setting and re-setting the baseline
www.ideasontario.ca
36
SPC Chart Rules
www.ideasontario.ca
37
Baseline Data
www.ideasontario.ca
38
First PDSA
www.ideasontario.ca
39
New Baseline
www.ideasontario.ca
40
PDSA 2
www.ideasontario.ca
41
Final Graph
www.ideasontario.ca
42
Rare Events (fictitious data)
date of
time from
complications previous event
1-Jul-11
29
30-Jul-11
24
23-Aug-11
19
11-Sep-11
23
4-Oct-11
28
1-Nov-11
43
14-Dec-11
29
12-Jan-12
33
14-Feb-12
38
23-Mar-12
42
4-May-12
40
13-Jun-12
54
6-Aug-12
40
15-Sep-12
50
4-Nov-12
43
17-Dec-12
56
11-Feb-13
50
2-Apr-13
49
21-May-13
83
12-Aug-13
88
8-Nov-13
69
16-Jan-14
62
19-Mar-14
76
3-Jun-14
51
24-Jul-14
82
www.ideasontario.ca
14-Oct-14
70
43
Rare Events (fictitious data)
www.ideasontario.ca
44
Data for Judgement vs.
Data for Improvement
• Measurements towards a target may hide or discourage
authentic and sustainable improvement
• Targets for accountability may focus on what is easily
measured rather than what has value (process rather
than outcome)
www.ideasontario.ca
45
Data for Judgement vs.
Data for Improvement
(fictitious data)
www.ideasontario.ca
46
Data for Judgement vs.
Data for Improvement
(fictitious data)
www.ideasontario.ca
47
Statistical Testing
www.ideasontario.ca
48
Statistical testing
• Statistical testing is a common form of analysis used in
clinical research and epidemiological studies
• Tests the hypothesis that the average/proportion/rate of
some outcome in one group of patients is equal to the
average/proportion/rate in another group of patients
• Statistical tests produce a P-value, which represents the
likelihood that the observed difference in the outcome
between the two groups is due to chance
• Studies often set the significance level at 0.05, meaning
if there is less than 5% chance the observed results are
due to chance, we deem the results `statistically
significant`
www.ideasontario.ca
49
Statistical testing versus QI analysis
• While used heavily in clinical research and epidemiology,
statistical testing is not the analytic method of choice
(e.g., the `Gold Standard`) for quality improvement
• QI involves conducting sequential tests of change over
time to some existing process; therefore, it is logical that
tracking outcome and process measures over time in an
SPC chart would be the preferred method of analysis
• Performing statistical tests, rather than tracking
measures over time, may cause us to claim
improvement when none has occurred, or miss
improvement when some has occurred.
www.ideasontario.ca
50
25.0%
P = 0.026
20.0%
20.0%
15.0%
12.5%
Readmissions
10.0%
5.0%
0.0%
Pre (Jan - Jun)
Post (Jul - Dec)
A medical unit has 40 COPD discharges per month.
On July 1, they implement a self-management training
program prior to discharge for all patients.
There is a statistically significant decrease (p=0.026) in
COPD readmission rates after the implementation.
www.ideasontario.ca
51
0.25
Readmissions1
0.2
0.15
0.1
0.05
0
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Here are the readmission rates plotted by month.
There is an apparent decrease happening throughout the
year, perhaps due to other quality improvement initiative.
Difficult to tie decrease to the July 1 initiative
www.ideasontario.ca
52
Is statistical testing forbidden in QI…
• All QI projects should strive to track measures over time
and use annotated run and SPC charts for analysis of
their data
• Having said that, sometimes it is not feasible to collect
data in any other fashion (e.g., satisfaction surveys
which are burdensome and time-consuming to
complete), and you are stuck with having to do a prepost comparison
• The following website has multiple online calculators to
help you perform basic statistical tests between 2 groups
for averages (means), proportions, and rates:
• http://www.socscistatistics.com/tests/
www.ideasontario.ca
53
Thank You
• michael.campitelli@ices.on.ca
• ruth.croxford@ices.on.ca
www.ideasontario.ca
54
Delivered in partnership and collaboration with:
Funding provided by the Government of Ontario
www.ideasontario.ca
55