Leveraging Data For Action
April 8, 2014
Background
Nonprofit organizations are no strangers to data, in fact, many are experts in
consolidating lots and lots of data points. We try to collect it, we try to
analyze it, and we try to respond to it. More often than not, the time and
effort is just that – time and effort with little to show for how our data can
be utilized to assess and improve our programs.
What’s on the horizon…
• The demand for data will continue to increase as organizations try to link
effective organizational and fiscal stewardship over:
–
–
–
–
Community impacts
Resources
Assets
Budgets
2
Learning Objectives
• Participants will learn how to develop a sustainable data management
strategy.
• Participants will learn how to set and evaluate data targets.
• Participants will increase their understanding of data analysis techniques
and tools that will help them make sense of their data.
• Participants will learn how to prioritize and apply a quality
improvement processes in response to their data findings.
• Participants will learn how to tell a compelling story utilizing NPI and
agency performance indicators.
3
Data Management Strategy
4
Where does Data Management Strategy belong?
• Who we are
• What we
are trying to
achieve
Vision,
Mission,
Values
Strategic
Aims /
Objectives
Performance
Measures
Goals
• How we will
measure
our success
• Approaches
we will take
5
What is a Data Management Strategy?
It is a …
• Strategic approach by which data is acquired, validated, stored,
protected, and analyzed, and by which its accessibility, reliability, and
timeliness is ensured to satisfy the needs of the data users.
When the data management strategy is working well it …
• Provides the right data at the right time, by ensuring data are regularly
collected, analyzed, reported, and effectively used to make
improvements.
6
Intentional data management begins below . . .
Acquisition: How is the data acquired?
Validation: How is the data validated?
Format: What format is the data in?
Accessibility: Is the data easily accessible?
Reliability: How reliable is the data?
Timeliness: How timely is the data?
Storage & Protection: How is the data stored & protected?
7
Acquisition
• What data is currently manually entered into the system?
– Interviews, observations, tracking
• Is the data automatically tracked by a system?
• Is the data coming from multiple sources?
– Community Action Agencies
– Public Sources
• Are staff able to to collect additional data about actions, perceptions, or
processes that they aren’t currently collecting?
• Are there consistent data templates?
Tip: Templates assist in consistent data delivery.
Data acquisition can be a challenge with different partner agencies, however there is
a shared goal – we all want to showcase the impact our program is having in our
communities.
8
Data Format
• Clear definition of the desired data format (%s, #s, $s).
– Percentages (%s)
– Numbers (counts, cumulative, ratios)
– Dollars (Budgets, Expenses, Revenues, Assets)
• Clear understanding of what comprises the numerator and denominator.
• Data is submitted in uniform format.
Tip: Develop a Measure Definition Sheet that details goals, performance
indicators, and format of the data. (see examples in Appendix)
A consistent data format is essential if we are going to compare years, counties, or
similar programs.
9
Validation
• Historical Trend
– Does it make sense?
• Outliers
– Is there just one? Or are there many?
• Spot Checking / Quality Control
– Small random sample to validate the accuracy of data findings.
– Do we need to spend significant amount of time cleaning it up?
Tip: Line graphs are your friends in visually assessing potential issues with
the data.
Different programs or partners will require more scrutiny until the validation process
is more consistent.
10
Accessibility
• Is the data readily available?
– Monthly, Quarterly, Bi Annually, Annually
• Is the data cloud-based? Or server-based?
• What type of system houses the data?
–
–
–
–
CAP 60, ServTraq
Microsoft Excel, Access
MySQL, Oracle
Attendance Logs
• Who runs the reports? Who is responsible for pulling the data?
– Do we have a backup that can run the reports?
Data accessibility is critical for tracking and measuring community impact. The
sooner you have the information the faster you can make the necessary course
corrections.
11
Timeliness
• What is the frequency in which I will be able to collect the information?
– Monthly, Quarterly, Bi Annually, Annually
• Do we have to consistently chase down the data?
– Same program
– Same individuals
Tip: Try to minimize the use of indicators where data are only available at
the end of the year. Find or develop a more timely source.
Utilize a balanced approach when dealing with data availability. Some data may be
available on a daily basis posing its own challenges, while other data may not be
available until the very end of the fiscal year which is truly too late.
12
Reliability
• How reliable is the data?
• Is the data consistently of good quality?
• Is there too much variability in the numbers to accurately tell the story?
Tip: Quality data is your strongest point of view in telling your story and
rebuffing critics. It’s unerring proof that the organization is making
progress and delivering on its mission.
Spend the necessary time ensuring the data is consistently of good quality. The old
adage is correct ….“Garbage in equals garbage out.”
13
Storage and Protection
• Can I access the historical record?
• Where do we store and backup the data?
• What types of protection protocols do we have in place?
– Do we have a Business Continuity Plan?
– What is the procedure if the data is inaccessible, erased, corrupted, or we’ve
experienced a natural disaster?
There is nothing worse than losing or not having access to all of the data and
corresponding analysis at a time when you need to. Plan for it.
14
Let’s Assess Our Data Management Strategy
• Assess and prioritize those measures with issues that need to be
improved first.
Quarterly Review
• The Key to Success in implementing a data management strategy is to
prioritize and focus on the metrics that are mission critical to your
community impact.
15
Setting and Evaluating Targets
16
Clearing Up Terminology Confusion
Term
Definition
Aim Statement
Strategic
Objective
Performance
Standard
Performance
Goal
Performance
Measure
Performance
Indicator
Performance
Targets
NPI Examples
• A clearly articulated statement
provides both the foundation and the
focus of the problem-solving effort.
“What are we trying to achieve?”
• Address causes of poverty by
implementing programs and services
that empower low-income families and
individuals.
• Objective standards or guidelines that
are used to assess an organization’s
performance.
“What approaches will we utilize?”
• Low income people become more selfsufficient.
• The conditions in which low income
people live are improved.
Setting and Evaluating Targets
• Quantitative measures of capacities,
• Number of participants pursuing postprocesses, or outcomes relevant to the
secondary education.
assessment of a performance
• Number of participants demonstrating
standard/goal.
ability to complete and maintain a
“What quantitative data do we have
budget for >90 days.
available to measure progress?”
• Specific and measureable goals related • Expected outcomes in reporting period.
to agency or system performance.
• 85% of participants will pursue
“What end of year performance
post-secondary education.
outcome do we want to achieve?”
17
Step 1: Setting Annual Targets
A target is the desired end of the year performance outcome which is
typically determined by past performance.
What needs to be in place before setting targets?
• Aim Statements
– Typically 4 to 5 statements that encompass community and organizational
needs.
• Goal statements
– Often range from 3 to 6 goal statements in support of 1 Aim
• Performance Indicators
– Typically 2 to 5 quantitative measures are selected in assessing progress
against defined goals.
18
Step 2: Setting Annual Targets
Scenario: We have defined our organizations aim statements, goals, and
have aligned on which performance indicators to track, measure, and report
on.
How do I determine what should be the target?
Funding Source
Sets Target
Yes
Apply target or set one
higher
No
Set interim targets that
bridge the gap
Yes
Evaluate your history and set
benchmark year
Is this doable?
Performance
Indicator
Organization
Sets Target
Do we have the
data?
Extrapolate to build initial
target
No
Leverage other program
targets as starting point
Leverage expert input
19
Step 3: Setting Annual Targets
In addition to annual targets, Red Line targets are just as important because
they help determine the minimum level of acceptability.
• Red line targets are unacceptable outcomes at any point and time that
require immediate program or organizational attention and intervention.
• Setting redline targets is essential, and often left out, to help your
organization gauge and prioritize your program improvement efforts.
Fell Short of Red
Line Target
Performed Above Red
Line & Below Annual
Target
Red Line Target
20
Achieved or
Exceeded Target
Annual Target
Example....
21
Data Analysis Techniques and Tools
22
Visual Analysis: Leveraging SQVID
• Series of 5 questions that we walk our initial idea through in order to
bring visual clarity.
• Helps us imagine what visual messages we’d like to convey before we
start worrying about which picture we’re going to use.
Simple
vs
Elaborate
Quality
vs
Quantity
Vision
vs
Execution
Individual Attributes
vs
Comparison
Delta (or Change)
vs
Status Quo
23
Source: Roam, Dan. The Back of the Napkin, Solving Problems and Selling Ideas with Pictures.
Visual Model
24
Source: Roam, Dan. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures.
Visual Model (cont’d)
25
Source: Roam, Dan. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures.
Visual Model (con’td)
26
Source: Roam, Dan. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures.
Rules of Thumb for Visual Modeling
•
Portrait
•
– Think simple.
– Illuminate lists.
– Visually describe.
•
Charts
– It’s the data that matters, so let it show.
– Pick the simplest model to make your
point.
• Bar charts: comparing absolute
quantities
• Lines & Area charts: For comparing
absolute quantities between two
different criteria or times.
– Time is a one-way street.
– Repeating timelines create life cycles.
– Round versus linear. (clock or ruler)
•
Flowchart
– Showcasing cause and effect.
•
Plots
– Multiple-variable plots aren’t hard to
make, but they require patience,
practice, and a point.
– Create a scale model of an entire
business universe or business
problem.
– The biggest danger of multiplevariable plots is that they invite the
layering of many data types.
– If you start with one model, stay with
one model.
•
Timelines
Maps
– Everything has a geography.
– North is a state of mind.
– Look beyond the obvious hierarchy.
27
Source: Roam, Dan. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures.
Quality Improvement
28
What is Quality Improvement?
• An integrative process linking ..
Knowledge
Structures
Processes
Outcomes
…throughout an organization.
• Allows us to ask, “given our resources and authority, are we improving the
lives within our communities in the most efficient way possible?
• Ultimately, it is a journey of continuous improvement, allowing
organizations to maximize their impact and outcomes.
“Doing the right thing for the right reason at the right place in the right
ways at the right time.”
29
Source: NNPHI, Proctor & Gamble
How does ROMA fit in a Performance Management System?
30
Source: Turning Point, From Silos to System, “Embracing Quality in Public Health: A Practitioner’s Performance Management Primer”
When Do I Launch a Quality Improvement Project?
QI is an essential piece of performance management.
• When your performance management system shows a gap that is not
closing between a performance target and your programs current
performance, you have an opportunity to implement QIP to improve
program and service impacts.
Popular QI Methodologies
• Plan-Do-Study-Act (PDSA)
• Lean / Six Sigma
• 5s
• FADE
31
Prioritize and Select Quality Improvement Projects
•
•
•
•
Review the measures that performed at the redline or below.
Review the measures that performed above redline and below the annual
target.
As a team, leverage a matrix exercise to help you prioritize and evaluate
which measures are candidates for a QIP. (See Appendix for QIP Matrix)
Out of the prioritization matrix, select a max of 3 potential measures that
may be candidates for a QIPs,
– Three may be too many if there are significant resource constraints or process
complexity, in which case target just 1 measure for a QIP.
32
Closing Gaps: Plan-Do-Study-Act (PDSA)
Iterative four-step management method used in business for the control and
continuous improvement of processes and products.
•
Establish objectives and processes necessary to deliver results in accordance
with the expected target or goals. When possible start on a small scale to
test possible effects.
•
Implement the plan, execute the process, deliver the service.
•
Study the actual results (measured and collected in "DO" above) and compare
against the expected results (targets or goals from the "PLAN") to ascertain any
differences or gaps.
•
Request corrective actions on significant differences between actual and
planned results. Analyze the differences to determine their root causes.
Determine where to apply changes that will include improvement of the
process, product, or service.
Plan
Do
Study
Act
33
Closing Gaps: Lean
Muri (Load)
Muda (Flow)
Mura (Waste)
• Concept of not overloading
the process
• Concept of keeping the
flow smooth and event
• Concept of removing the
non-value adding activities
Overproduction
Motion
Over Processing
Waiting
Transportation
Correction
Inventory
Unused Employee
Ideas & Talent
Photos www.alverdyan.com
34
Telling the story
35
Source: Non Profit Data Decision-Making Workbook, January 2013
Telling the story
“If a metric measures in a forest with no one around to hear it, it does not
make a sound…”
•
•
•
•
Who will be the recipients of the story?
In what meetings will they be used?
What decisions shouldn’t be made without them?
How frequently will you plan to adjust your actions based on what the
metrics say?
36
Source: Non Profit Data Decision-Making Workbook, January 2013
Effective Ways to Structure Your Story
Beginning
• Given the short attention spans and the need to move from one activity to the
next as quickly as possible, you want to highlight the momentous and game
changing results first.
Middle
• The body of your story should focus on those areas where we are improving and
making significant strides.
End
• The end of the story is where you will want to talk about the proactive steps that
will be taken to bring under performing metrics back on track.
Filter down the most important themes you wish to convey. Ask your self what do
we need to know to ensure we are making progress and report on those measures.
37
Leverage a combination of text and graphic to
reinforce your story making it more memorable.
38
Source: Ontario’s Health System Yearly Report, 2013
Resources
•
•
•
•
•
•
•
•
Getting Started with Data-Driven Decision Making: A Workbook.
http://www.nten.org/research/2013-data-workbook-download.
Organizational Research Services, A Handbook of Data Collection Tools: Companion to “A
Guide to Measuring Advocacy and Policy.”
http://www.organizationalresearch.com/publicationsandresources/a_handbook_of_data
_collection_tools.pdf.
Sample Size Estimator. http://www.surveysystem.com/sscalc.htm.
Roam, Dan. The Back of the Napkin, Solving Problems and Selling Ideas with Pictures.
Measuring Your Impact: Using Evaluation to Demonstrate Value.
http://nnlm.gov/evaluation/workshops/measuring_your_impact/DataCollectionHandout.p
df.
Turning Point, From Silos to System, “Embracing Quality in Public Health: A Practitioner's
Performance Management Primer.”
http://www.phf.org/resourcestools/Documents/silossystems.pdf.
“10 Steps to Creating the Perfect Infographic.” http://www.creativebloq.com/design/10steps-creating-perfect-infographic-3145672.
10 Free Tools for Creating Infographics. http://www.creativebloq.com/infographic/tools2131971.
39
Further Questions or Comments
• allison@hensleit-hss.com
This presentation was created by California/Nevada Community Action Partnership in performance of
the California Department of Community Services and Development CSBG Discretionary Grant #13F3122. Any options, findings, and conclusions, or recommendations expressed in this materials are those
of the author(s) and do not necessarily reflect the views of the U.S. Department of Health and Human
Services, Administration for Children and Families.
40
Appendix
41
Sample Measure Definition Sheet
42
Sample Measure Definition Sheet
43
Quality Improvement Project Prioritization Matrix
10
9
Community Impact
8
7
6
5
4
3
2
1
0
1
2
3
4
5
6
Process Complexity
44
7
8
9
10