Measurement for
Improvement
Hello & Welcome:
what are you here for?
Results of our pre-session survey…
1. I would describe myself as (tick one only):
• Early Years Practitioner but measurement beginner
• Tester (already collecting data on measures developed by others)
• Person involved in developing local measures
• Measurement advisor who provides support to improvement teams
• Other (please specify)
2. I feel comfortable with (tick all that apply):
• Testing
• Data collection when I am told what to count
• Developing measures
• Using Excel spreadsheets
• Run charts
• Statistical Process Control (SPC) charts
• Other (please specify)
Results of our pre-session survey…
3. I am most interested to learn about (tick one only):
• Testing
• Data collection when I am told what to count
• Developing measures
• Using excel spreadsheets
• Run charts
• Statistical Process Control (SPC) charts
• Something Else / In Addition (please specify)
4. Are there any triumphs/challenges you would be willing and interested to mention in the
session?
• No
• Yes (please feel free to provide your name if you would like to share)
Results of our pre-session survey…
5. Are you a member of the EYC Extranet
• Yes
• No
• Don't know
6. Which of the following mostly fits with what you do? (tick one only)
• I set up measures, enter data and produce leadership reports on the Extranet
• I enter data on the Extranet once the measure has been set up
• I collect data in a spreadsheet and give it to someone else to put on the Extranet
• I collect data in a spreadsheet and give it to someone else but don't know what they do with
it
• I only collect data in a spreadsheet locally
• I only collect data locally only in some other format
• I have not been involved in collecting any data
• Other
Turn to your neighbour
• What have been your biggest learnings or
challenges regarding data gathering and
measurement for your Early Years work?
– Developing measures
– Agreeing definitions
– Collecting data (who, when, frequency)
– Analysis
– Reporting/presenting data
– Other?
Model for Improvement
The three questions
provide the strategy
Our focus today
The PDSA cycle provides
the tactical approach to work
The Improvement Guide, API
10
Model for Improvement
Using data to
understand progress
toward the team’s aim
Using data to answer the
questions posed for each
PDSA cycle
The Improvement Guide, API
10
Without data, you are just another
person with an opinion
Andreas
Schleicher
11
With data, it’s hard not to
have an opinion
… it’s a pivot point for
conversation
12
The Quality Measurement Journey
(How good? By when?)
PDSA
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Case Study
•
•
•
•
Promoting Bedtime Reading
Lochrin and Grassmarket Nursery Schools
Edinburgh
Lead: Donna Murray
The Quality Measurement Journey
(How good? By when?)
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Project Aim Statements
• What, by when, by how much.
–“By the end of June, 90% of
children at Grassmarket
Nursery will receive a bedtime
story at least 3 times a week.”
Concepts: What does better look like?
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
WORKSTREAM 3 (30 months to start of primary school)
Theory of what actions will
ensure developmental
milestones are reached at
the start of primary school
Theory of what drives developmental milestones
Aim
2⁰
1⁰
Poverty
Quality Of Home Environment
Domestic Abuse &
Violence
Societal Issues
Children have all
the developmental
skills and abilities
expected at the
start of primary
school
Workforce Issues
Transport, Community
Capacity & Cultures
Access To Services
Employment
Drivers
Child’s physical &
mental health and
emotional
development
Drivers
Detailed Aim:
90% of all children
within each CPP
have reached all of
the expected
developmental
milestones at the
time the child starts
primary school, by
end-2017
Drivers
Carer’s physical &
mental health and
skills
Early Learning & Play
Health
Attachment
Improved teamwork,
communication and
collaboration
Improved uptake of
benefits
Improved child’s dental
health
Improving child nutrition
Improving brain
development and
physical play
Improved family centred
response
Improved stability /
permanence for LAC
Concept:
Bedtime
Reading
Improved identification
Improved joint working
Additional Support
Level of education
Misuse of alcohol &
drugs
Improved management,
planning and quality of
services
Improved sharing of
information
Nutrition
Disabilities & Mental
health
Parenting skills &
knowledge
Improved leadership,
culture & planning`
Identification & reasons
for current resilience
Version: 06/03/2013
From concept to measure
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
From concept to measure: not trivial
Aim
2⁰
1⁰
Poverty
Quality Of Home Environment
Domestic Abuse &
Violence
Societal Issues
Children have all
the developmental
skills and abilities
expected at the
start of primary
school
Workforce Issues
Transport, Community
Capacity & Cultures
Access To Services
Employment
Drivers
Child’s physical &
mental health and
emotional
development
Health
Drivers
Detailed Aim:
90% of all children
within each CPP
have reached all of
the expected
developmental
milestones at the
time the child starts
primary school, by
end-2017
Attachment
Improved uptake of
benefits
Improved child’s dental
health
Improving child nutrition
Improving brain
development and
physical play
Improved family centred
response
Improved stability /
permanence for LAC
Concept:
Bedtime
Reading
Improved identification
Improved joint working
Additional Support
Drivers
Carer’s physical &
mental health and
skills
Early Learning & Play
Improved teamwork,
communication and
collaboration
Level of education
Misuse of alcohol &
drugs
Improved management,
planning and quality of
services
Improved sharing of
information
These driver diagrams
Disabilities & Mental
healthshow lots of concepts,
Parenting skills &
but few
measures (numbers, %, rate)
knowledge
Nutrition
Improved leadership,
culture & planning`
Identification & reasons
for current resilience
Version: 06/03/2013
Our example: concept to measure
• (Driver) Concept is ‘Bedtime reading’
• Measures to provide feedback on the concept
– % receiving a story
– % of stories read at bedtime
– % of parents reporting improved bedtime routine
– % enjoying the bedtime story
– % reporting increase in bedtime story reading
Agreeing definitions
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
An operational definition…
… is a description,
in quantifiable terms,
of what to measure
and the steps
to measure it
consistently
•
Gives communicable
meaning to a concept
•
Is clear & unambiguous
•
Specifies
measurement
methods &
equipment
•
Identifies agreed
criteria for
recognition
© 2010 Institute for Healthcare Improvement/ R Lloyd
21
An operational definition…
•
Method of measurement?
•
Criteria for recognition?
22
Our example: definitions we need…
• What does reading a story mean?
• When is bedtime versus other time?
• What is improved bedtime routine?
• What is the definition of enjoyed?
Collecting data
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Planning data collection
• Who will collect the data?
• How often?
• How?
• Stratification?
• Will sampling be required?
What do we mean by stratification?
•
Does this differ by (known differences)...
•
Separation of data into mutually exclusive categories
•
To discover patterns obscured by aggregation
•
For example:
o
Teams
o
Seasons
o
Times of day
o
Days of week
o
Socioeconomic group
Sampling:
how many, how often?
Balance ease of collection
with
reduction in random variation
Data in PDSA cycles
•
Collect useful data not perfect data (learning not
evaluation)
•
Use pencil/ paper until info system is ready
•
Use sampling to reduce workload of data collection
•
Use qualitative data (feedback) rather than wait for
quantitative data
•
Record what went wrong during data collection
Integrate data into the day
•
Collect data within current work
•
Develop easy-use forms/information systems for
practitioners
•
Clearly define roles and responsibilities for data
collection
•
Set aside time to review data with those who collect it
So for our example…
• All children are included, no sampling. 7 days a week.
• Donna will inquire with each parent at drop off
Monday through Friday.
• Data will be captured on a paper form.
• Donna inputs daily data into spreadsheet following
drop off.
Analysis of data
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Analysis – Run Charts
Children receiving a bedtime story
120
Parents
survey
Percentage
of children
100
Goal
Median
.
80
60
Books
available at
collection
time.
Grassmarket
changes
40
introduced.
weekly
average
displayed for
parents
Research
information
handed to
parents.
20
0
M
T
W
TH
F
M
T
W
TH F
M
Day of the
w eek
T
W
TH
F
M
T
W
TH
F
The Quality Measurement Journey
(How good? By when?)
PDSA
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Action – Testing Change Ideas
Quality Measurement Journey example
Improve bedtime routines, attachment, literacy
Increase bedtime reading
% of children receiving a bedtime story
number read story/Total Children
daily; no sampling
teacher asks parents at drop off,
 spreadsheet
run chart
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
QUESTIONS?
Let’s build a run chart
Elements of a Run Chart
6.00
The centerline (CL) on a
Run Chart is the Median
5.75
5.50
Measure
Pounds of Red Bag Waste
5.25
5.00
4.75
Median=4.610
4.50
4.25
4.00
3.75
3.50
3.25
1
2
3
Time
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
Point Number
Month
Let’s build a run
chart
13-Jan-2013
13-Feb-2013
13-Mar-2013
13-Apr-2013
13-May-2013
13-Jun-2013
13-Jul-2013
13-Aug-2013
13-Sep-2013
13-Oct-2013
13-Nov-2013
13-Dec-2013
13-Jan-2014
13-Feb-2014
13-Mar-2014
13-Apr-2014
13-May-2014
13-Jun-2014
% of children
attending
Nursery
17%
15%
9%
23%
11%
12%
23%
49%
43%
78%
70%
55%
75%
72%
60%
55%
62%
78%
Let’s build a run chart in Excel.