Co-Producing Impact Evaluation: Making It Work
Sue Holloway, Director
July 2013
PBE aims to match professional economists with
charities;
providing pro bono help to measure performance
and understand charity impact, and
fostering a culture of volunteering within the
economics profession
© Pro Bono Economics
2
Approaches to Impact
Evaluation: what makes
cost-benefit analysis work?
Estimating impact
Measure outcomes
Best estimate of what would have happened anyway (the
counterfactual)
Outcomes minus counterfactual = impact
A measurement minus an estimate = an estimate
© Pro Bono Economics
4
Control Groups
National average
PBE: Foundation Training Company
Before
PBE: MEAM pilots – adults with multiple needs
Matched records in admin dataset
Peterborough Social Impact Bond/MoJ datalab
Natural experiment in charity data
PBE: Barnardos
RCT
Education Endowment Foundation/Social Research Unit
© Pro Bono Economics
5
Whose perspective?
Value to society
won’t include transfer payments e.g. tax and benefits
Value to Exchequer
Increase in tax receipts (+) or increase in benefit payments (-)
Potential costs avoided
Cashable savings
Value to individuals
change in income including benefits (+)
© Pro Bono Economics
6
Which benefits?
Hard outcomes
Qualifications → employment → lifetime earnings
Costs avoided by public services
Soft outcomes
Early stages of econometric work in this area
Related to income, so only for those who are earning
SROI uses stakeholders to identify proxies
© Pro Bono Economics
7
Which costs and values?
Input costs
Programme costs
Volunteer input
Other activities
Benefit values
Average costs
Marginal costs
© Pro Bono Economics
8
Conversation starters
Questions
When do I need a cost-benefit analysis?
What are the practical challenges?
© Pro Bono Economics
10