Unlocking Impact: The
Five Key Questions
Malen Davies
RNIB Research Officer (Impact)
9 September 2014
1
Part One: Defining Success
What you do and what happens
when you do it
2
Part One: Defining Success
What you do and what happens when you
do it
Lunch club model
3
Part One: Defining Success
Five key questions
1. Why measure outcomes and impact?
2.What does success look like…exactly?
3. How are you going to make change
happen?
4. Who needs to know you are being
successful?
5. How are you going to know you're being
successful?
4
Part One: Defining Success
First Key Question:
Why measure outcomes and
impacts?
5
Part One: Defining success
Why measure outcomes and impact?
• Finding out progress against mission
• Learning how to improve
• Secure funding
• Inspiring staff and improving their work
• Raising profile
• Contributing to knowledge of ‘what works’
6
Part One: Defining Success
Second Key Question:
What does success look like?
7
OutPart One: Defining Successcome types
Outcome type
•
•
•
•
Knowledge & Skill - "I know more"
Emotional - "I feel different"
Behavioural - "I act different"
Opportunity - "I have a chance I didn't
before"
• Economic - "I am financially better off"
• Health - "My health has improved"
Part One: Defining Success
9
Part One: Defining Success
Second Key Question:
What does success look like…exactly?
(What are your intended outcomes?)
10
Part Two: Creating Change
Third Key Question:
How are you going to make change
happen?
11
Part Two: Creating Change
Theory of change
Inputs
Activities
Outputs
Outcomes
Impacts
12
Impact Map
Inputs
What are the
resources used
to run activities
and fulfil your
objectives?
Activities
What are the
projects or
processes that
allow you to
fulfil your
objectives?
Outputs
What is produced
as a direct result
of these actions?
Usually depicting
completion of
activity
Outcomes
Impacts
What benefit or
change is
accomplished
as a direct
result of the
output?
13
Part Two: Creating Change
Theory of change: Counselling service
Inputs
3 part time
counsellors
Activities
Counselling
up to 10
clients each
month
Outputs
360 people a
year provided
with
counselling
Outcomes
Impacts
Empower
more people
with sight
loss
14
Part Two: Creating Change
Theory of change: Counselling service
Inputs
3 part time
counsellors
Activities
Outputs
Outcomes
Impacts
Counselling
up to 10
clients each
month
360 people a
year provided
with
counselling
Empower
more people
with sight
loss
Challenging assumptions
External factors
Are they qualified?
Who is financing project?
Is there need for this much
support?
Is there other support delivered
outside of counselling?
15
Part Two: Creating Change
16
Part Two: Creating Change
Fourth Key Question:
How are you going to make success happen?
Building a theory of change group exercise
17
Part two: Creating Change
Exercise: building a theory of
change
Plan a theory of change for a new telebefriending
service for people with sight loss aged over 65
years old.
18
Part two: Creating Change
Theory of change
Inputs
Activities
-Staff time.
-Telebefriending groups set up
-Volunteer time
-Volunteers trained to become telebefriending
facilitators
- Grant funding
- Develop training guides and resources for groups
Outputs
Outcomes / Impact
-50 telebefriending groups set -Improve sense of well-being for customers
up
-Increased quality of life for customers
- 20 volunteers trained
- Volunteers improve knowledge and skills
- Information and resources
provided to telebefriending
groups
Part Two: Creating Change
Theory of Change:
• Help teams work together to achieve a shared
understanding of a project and its aims.
• Bring the process of change to the forefront.
• Make projects more effective
• Quickly communicate a project’s aims
• Encourage teams to engage with the existing evidence
base
20
Part Three: Planning Measurement
Fourth Key Question:
Who needs to know you are being
successful?
21
Part Three: Planning Measurement
Who wants to know and what do they want?
Who needs to know what do they need?
22
Part Three: Planning Measurement
Fourth Key Question:
Who needs to know you are being
successful?
(Identify your audience)
23
Part Four: Measuring Success
Fifth Key Question:
How are you going to know you're
being successful?
24
Part Four: Measuring Success
Impact Measurement Tools
and Tips
25
Part Four: Measuring Success
Principles of Impact Measurement
1.
2.
3.
4.
5.
6.
Clarity - The reader can quickly and easily understand the intervention
through a coherent narrative that connects charitable aims, plans, activities
and results
Accessibility - Relevant information can be found by anyone who looks for it,
in a range of formats suitable for different stakeholders
Transparency - Reporting is full, open and honest
Accountability - Reporting connects with stakeholders, partners and
beneficiaries to tell them what they need to know;
Verifiability - Claims about impact are backed up appropriately, allowing
others to review
Proportionality - The level and detail of reporting reflects the size of the
intervention
Charity Finance Group, ACEVO & NPC (2012)
26
Part Four: Measuring Success
Technical and Established
Methodologies
Example:
Social Return on Investment (free):
http://www.thesroinetwork.org/
27
Part Four: Measuring Success
Casework Measures
Examples:
Outcomes Star (cost):
http://www.outcomesstar.org.uk/
Rickter Scale (cost): http://www.rickterscale.com/
28
Part Four: Measuring Success
Validated Questionnaires
Example:
Warwick-Edinburgh Mental Well-Being Scale (free):
http://www.healthscotland.com/scotlandshealth/population/Measuring-positive-mentalhealth.aspx
29
Part Four: Measuring Success
Bespoke Tools
Examples:
Semi-Structured Outcome Questionnaires, Focus
Groups, Diary Studies, Social Media.
30
Part Four: Measuring Success
Deciding Which Measure to Use
1. Audience
2. Resource
3. Access
31
Part Four: Measuring Success
Tips for success
•Be as clear as possible about the outcomes you're trying
to measure - without this you will struggle measure them!
•Be flexible with you measurement methods - good
evidence needs good research principles, but it can be
done in flexible and 'natural' ways, through conversation
and existing opportunities to engage with customers
32
Part Four: Measuring Success
Tips for success
•Explain to customers your rationale - evaluation need not be
cold or abstract. Explain to your customers why you need to
collect impact information, and why it's important for you,
them and other customers
33
Part Five: Communicate Success
Communicate Your Evidence!
Example: Annual Report Format
Why we're doing this (Need)
What we did (Activity)
What difference it makes (Impact)
34
Unlocking Impact: The Five Key
Questions
1. Why measure outcomes and impact?
2.What does success look like…exactly?
3. How are you going to make change happen?
4. Who needs to know you are being successful?
5. How are you going to know you're being
successful?
35
Useful links
New Philanthropy Capital:
https://www.thinknpc.org/publications/the-principles-ofgood-impact-reporting-2/
CENI: http://www.ceni.org/evaluation-impactmeasurement
Impact HUB: http://inspiringimpact.org/listings/
36
Questions
Malen Davies
Research Officer (Impact)
Malen.davies@rnib.org.uk
37