Advanced Improvement Practitioner Programme Measurement Mike Davidge Issues and questions 2 WHAT TYPE OF PERSON ARE YOU? 3 Are you a plan-doer ? Are you an auditor? 6 Are you an improver? 5 Measurement Sins Having no baseline (or having a compromised one) Measuring the wrong thing Only collecting data at 2 points in time Inappropriate or mindless use of statistics Presenting results in a misleading way 7 7 Steps to measurement 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 7 Repeat steps 4-6 4 Collect data 5 Analyse & present 8 The Measures Checklist • Part One • • • • Why important? Who owns? Definitions Goals • Part Two • Collect • Analyse • Review 9 Step 1: Decide your aim 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 7 Repeat steps 4-6 5 Analyse & present 4 Collect data Specific Measurable Achievable Realistic Time-bound A worthwhile topic Outcome focused Measurable Specific population Clear timelines Succinct but clear 10 Step 2: Choose measures 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 7 Repeat steps 4-6 4 Collect data 5 Analyse & present 11 Useful Tools when choosing measures Process maps Driver diagrams 12 Using the process map Start ? Decision Point ? Handover ? End ? The ABCD of managing a service • • • • Activity: what we have actually done Backlog: what we should have done but didn’t Capacity: what we could have done Demand: what we should have done Listed in order of frequency of measurement Step 3: Define measures 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 7 Repeat steps 4-6 5 Analyse & present 4 Collect data An operational definition is a description, in quantifiable terms, of what to measure and the steps to follow to measure it consistently 15 Definitions Exercise Take a piece of A4 paper and follow my instructions to get an identical shape to mine 16 Types of calculation • • • • Counts Ratios or rates Percentages & Proportions Time between or cases between Types of calculation: When to use what • Counts – when the target population does not change very much – Example: Number of falls on an elderly ward (always full) • Percentages – when the numerator is a subset of the denominator – Example: Percentage of patients who fell • Ratios or rates – Numerator and denominator are measuring different things – Example: Falls per 100 bed days • Time between or cases between – When you are tracking a ‘rare’ event, say one that occurs less than once a week on average – Example: Days since a patient last fell on this ward Step 4: Collect your data 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 7 Repeat steps 4-6 4 Collect data 5 Analyse & present 19 Baselines 100 90 80 70 60 50 40 30 20 10 0 My Project - Linear trend F M A M A S O N D My Project - seasonal dip J F M A M 100 90 80 J 70J 60 Months 50 40 30 20 10 0 Something Important J 100 90 80 70 60 50 J 40J 30 Months 20 10 0 Something Important Something Important My Project - random fluctuation A S J F O M N A D M J J Months 20 A S O N D Analyse and present 1 Decide aim “The type of presentation you use has a crucial effect on how you react to data” 2 Choose measures 3 Define measures 6 Review measures 7 Repeat steps 4-6 4 Collect data 5 Analyse & present 21 21 Delayed transfers of care Indicator description Community Health Delayed transfers of care (NHS and Social Care) 2012 /13 Annual/Year end Target Dec Jan Feb 22 29 54 46 What you get presented with What did you decide to do? RAG l YTD Target 22 YTD Actual YTD RAG & 12 month Trend How we assess performance #1: 2 point comparisons Last This quarter quarter Change 80 96 +20% Why has the number of complaints gone up? Our service is getting worse. We need to do something! What decision are you going to make? UNDERSTANDING & DEALING WITH VARIATION IN ANALYSIS 25 “If I could reduce my message to management to just a few words, I'd say it all has to do with reducing variation”. W Edwards Deming 26 What’s a person’s normal body temperature? 27 “Data contains both signal and noise. To be able to extract information, one must separate the signal from the noise within the data.” Walter Shewhart RUN CHARTS 29 Run charts • Plot data in time order Run charts • Calculate and display median as a line • Analyse chart by studying how values fall around the median 30 Summarise – the median • Median defined – The median is the middle value of a finite list of numbers where the numbers are ordered from lowest value to highest value. If there is an even number of observations, the median is not unique, so one takes the mean of the two middle values. • Why do we use it? – Not affected by extremely large or small values – Half of values will always be below/ above the median • What do we need to beware of? – It tells us little about the spread of values – Tedious to calculate by hand if number of values is large 31 Creating your own run chart Using the complaints handout, graph paper, ruler and a pencil: • • • • • Draw and label the axes Plot the dots (each monthly value) Work out the median Add a title (with dates) Add a legend 32 Does yours look like this? Complaints during 2011 to 2012 60 50 40 30 20 2011 2012 Number 33 Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb Jan Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb 0 Jan 10 Rule #1: a shift Above centre Below centre Time Time At least 6 points above centre line At least 6 points below centre line 34 Rule #2: a trend Upward trend Downward trend Time Time At least 5 points all in upward direction At least 5 points all in downward direction 35 Rule #3: astronomical data point Time 36 Rule #4: runs 1st run 2nd run Time Time Too many or too few ‘runs’ Use table to determine Interpreting your run chart • Using the 4 rules listed below, see if you have any special causes in your data Rule #1: A shift in the process, or too many data points in a run (6 points above or below the median) Rule #2: A trend (5 all increasing or decreasing) Rule #3: An “astronomical” data point Rule #4: An unusual number of runs (use the table to determine) 38 Apply the rules Complaints during 2011 to 2012 60 50 40 30 20 2011 2012 Number Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb Jan Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb 0 Jan 10 SPC CHARTS 40 Control charts • • • • Control charts Plot data in time order Calculate and display mean as a line Calculate and display control limits as lines Analyse chart by studying how values fall around mean and between control limits Complaints during 2006 to 2007 70 60 50 40 30 20 10 2006 Number Dec Nov Oct Sep Jul 2007 Aug Jun May Apr Mar Feb Jan Dec Nov Oct Sep Jul Aug Jun May Apr Mar Feb Jan 0 An example SPC chart Complaints during 2006 to 2007 70 60 50 40 30 20 10 2006 2007 Number 42 Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb Jan Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb Jan 0 Summarise – the mean • Mean defined – the arithmetic mean (or simply the mean) of a list of numbers is the sum of all of the list divided by the number of items in the list. • Why do we use it? – Provides a useful estimate of the typical value – Easy to understand and calculate • What do we need to beware of? – A few extremely large values can inflate the mean – It tells us little about the spread of values Creating your own SPC chart • Use the SPC handout, ruler and a pencil. • This time the basic chart is drawn for you. • You will be calculating and drawing the mean and control or process limits • Then interpreting your chart 44 1. Plot individual values % weekly compliance with hand hygience April - Sept 2010 100 90 80 Percentage 70 60 50 40 30 20 10 0 Week 2. Calculate Mean & plot it % weekly compliance with hand hygience April - Sept 2010 Mean = 58 100 90 80 Percentage 70 60 50 40 30 20 10 0 Week In Excel use the formula =AVERAGE(range) 3: Derive moving range These are required to calculate the control limits The first row contains the chart data Use the second row to record the difference between successive data values 4 8 9 Moving 0 as a positive 5 value 1 The difference is always recorded Range 4 1 X Data 0 0 5 4: Calculate Average Moving Range R1 R2 R3 R4 R5 R6 R7 R8 7 23 25 25 16 20 33 35 R1 Add up the individual moving range values = 7+23+25+25+…+18 Divide by number of values = 463 24 Average moving range = 19.3 R23 R24 13 18 Now what does this mean? 5: Calculate the control limits First derive one measure of variation (referred to as 1 sigma) 1 sigma = Average moving range 1.128 = 19.3 1.128 1 sigma = 17.1 Calculate lower limit as Mean – 3 sigma Lower limit =58 – 3 x 17.1 Lower limit = 6.7 Calculate upper limit as Mean + 3 sigma Upper limit =58 + 3 x 17.1 Upper limit = 109.3 5. Plot limits % weekly compliance with hand hygience April - Sept 2010 Upper limit = 109 100 90 80 Percentage 70 60 50 40 30 Lower limit = 7 20 10 0 Week Constructing the chart: Summary There are 5 steps to constructing your chart: 1. Plot the individual values 2. Calculate the mean and plot it 3. Derive the moving range values 4. Calculate the average moving range (MR) 5. Derive upper and lower limits from this and plot them Control charts are like yoghurt ... They come in different flavours! The XmR chart (or I-chart) X – for individual values mR – for moving range Makes fewest assumptions about how the data values are distributed around the mean Will give you reliable results in almost all situations Some software to help you create SPC charts Limited version is FREE, full version costs £50 INTERPRETING SPC CHARTS 55 Source: Lloyd Nelson, Technical Aids: The Shewhart Control Chart – Tests for Special Causes SPC CHART RULES 56 Rule # 1: Any single point outside the control limits Upper control limit Mean Lower control limit Rule # 2: A shift At least 7 points consecutively either above or below the centre line Upper control limit Mean Lower control limit Rule #3: A drift At least 7 points consecutively ascending or descending Upper control limit Mean Lower control limit Note: the points can cross the centre line Rule #4: At least 14 points alternating up and down Upper control limit Mean Lower control limit Note: the points do not have to alternate above and below the centre line Other tests require you to plot the intermediate variation values Mean+3V Upper control limit Mean+2V Mean+1V Mean Mean Mean-1V Mean-2V Mean-3V Lower control limit Rule #5: 2 out of 3 points outside 2V on the same side of the centre line Mean+3V Upper control limit Mean+2V Mean+1V Mean Mean Mean-1V Mean-2V Mean-3V Lower control limit Rule #6: 4 out of 5 points outside 1V on the same side of the centre line Mean+3V Upper control limit Mean+2V Mean+1V Mean Mean Mean-1V Mean-2V Mean-3V Lower control limit Rule #7: Reduced variation 15 points within 1V either side of the centre line Mean+3V Upper control limit Mean+2V Mean+1V Mean Mean Mean-1V Mean-2V Mean-3V Lower control limit Rule #8: 2 processes 8 points in a row on both sides of the centre line with none within 1V Mean+3V Upper control limit Mean+2V Mean+1V Mean Mean Mean-1V Mean-2V Mean-3V Lower control limit Why 7 points in a row? • Toss a coin: Chances of getting a ‘head’? • Toss it twice: Chances of getting two ‘heads’? 66 How many options? 1 toss 2 tosses 3 tosses 1 in 2 1 in 4 1 in 8 4 tosses 1 in 16 67 Chances of ‘n’ heads in a row Number of tosses 1 2 3 4 5 6 7 8 9 Chance of all heads 1 in 2 or 50% 1 in 4 or 25% 1 in 8 or 12.5% 1 in 16 or 6.25% 1 in 32 or 3.13% 1 in 64 or 1.56% 1 in 128 or 0.78% 1 in 256 or 0.39% 1 in 512 or 0.20% 68 SAFETY CROSSES AND RUN CHARTS 69 Your Safety Cross Record the dates 04/07/2013 11/07/2013 25/07/2013 70 Calculate the days between falls Date 04/07/2013 11/07/2013 25/07/2013 Days between 11 – 4 = 7 25 – 11 = 14 What value do we get if we have 2 falls on the same day? 71 Plot the days between on a run chart The higher the value, the better it is Use the 3 run chart rules to interpret Plot X axis as text not dates 72 RECOMMENDED CHARTS 73 The Golden Rule when presenting data One picture, one message 74 Types of chart and when to use them Run & control charts • Used to display performance over time Complaints during 2011 to 2012 60 50 40 • Shows whether common causes or special causes (or both) are present in our data 30 20 2011 75 Dec Oct Nov Sep Jul 2012 Number Aug Jun Apr May Mar Jan Feb Dec Oct Nov Sep Jul Aug Jun Apr May Mar Jan 0 Feb 10 Types of chart and when to use them Pareto charts • Used to display the number of times distinct things happen such as reason for cancellation • To separate the ‘vital few’ from the ‘useful many’ (the 80/20 rule) 76 Types of chart and when to use them Bar charts • Used to display survey results • Display for each question – which questions look different? • Also display optionally for each respondent – are there types of respondent? I feel confident working on the Duty desk 7 6 5 4 3 2 1 0 Strongly disagree Disagree Neutral Agree Source: Call of Duty 77 Strongly agree QUALITATIVE ANALYSIS Surveys Issues to consider • Are all patients rating the same thing? • Is there a ceiling effect? • Do scales work in a linear fashion? – A score of 6 is twice as good as 3 – Strongly agree is twice as good as agree Collecting patient experience data Interviews Surveys Patient tracking Analysing qualitative data Thematic analysis: Look for the common themes Construct a story around typical findings The power of a good quote The power of a quote Staff Experience Quotes Before changes: “It felt unsafe, anxiety provoking, uncontained and filled with dread” “Scared, dreaded being on duty at times left feeling isolated, unsafe, unsure with no clear structures in place to support me” “Unclear, unsafe, unsupported” After Changes: “ I now enjoy working on the duty desk, it feels like you can make a quick difference to peoples lives when they are in distress” “I enjoy the challenges duty now presents, I feel more confident and safe in my practice, I feel it is more of a team approach.” “Challenging at times but stimulating, anxiety provoking but only at times and I feel supported” Source: Call of Duty Step 6 – Review measures 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 7 Repeat steps 4-6 4 Collect data It is a waste of time collecting and analysing your data if you don't take action on the results 5 Analyse & present 84 DTOCs part 2 Number Delayed transfers of care 80 70 60 50 40 30 Verdict 20 01 Feb 2013 01 Jan 2013 01 Dec 2012 01 Nov 2012 Months 01 Oct 2012 01 Sep 2012 01 Aug 2012 01 Jul 2012 01 Jun 2012 BaseLine 01 May 2012 Not Capable of achieving target 01 Apr 2012 Stable within 10 limits 0 (10 - 72) What decision do you make? Decision Because Do nothing Performance ok Contingency plans Special cause variation Process redesign Common cause variation Is your information presented in a way that allows you to confidently make one of these decisions? 86 Team Review Meetings Step 7: Keep going 1 Decide aim 2 Choose measures 3 Define measures You may not get it right first time! You may need several attempts. Remember PDSA 7 Steps now on YouTube 6 Review measures 7 Repeat steps 4-6 5 Analyse & present 4 Collect data http://youtu.be/Za1o77jAnbw Or put ‘Mike Davidge measurement’ into YouTube search box 88 References • The run chart basic reference – “The run chart: a simple analytical tool for learning from variation in healthcare processes”; Perla R, Provost L, Murray S; BMJ Qual Saf 2011;20:46e51. doi:10.1136/bmjqs.2009.037895 • A great introduction to variation and SPC – Understanding variation, Don Wheeler, www.spcpress.com, 1986 • A couple of useful websites/blogs – www.davisdatasanity.com – www.kurtosis.co.uk 89 Application to management • Fourth Generation Management – Brian L Joiner, McGraw-Hill, 1994 • The Leader’s Handbook – Peter R Scholtes, McGraw-Hill, 1998 • The New Economics (2nd edition) – W Edwards Deming, MIT Press, 1994 Issues and questions 91