Workshop: Value for Money (VfM) in international development using Social Return on Investment (SROI) Natalie Nicholles nef (new economics foundation) 19th October 2010 © nef consulting Housekeeping • • • • Ground rules Fire exits Toilets Workshop materials © nef consulting Agenda 09:30 Big picture introduction to Value for Money and SROI 10:00 Crash course on SROI: understanding change 10:45 Tea/coffee break 11:00 Crash course on SROI: measuring change 11:30 Crash course on SROI: valuing change 12:30 Lunch 13:15 Crash course on SROI: attribution & partnership working 13:45 Crash course on SROI: forecasting long term benefits 14:45 Tea/coffee break 15:00 Worked-through example & organisational decision making 15:45 Answering key questions 17:00 Close © nef consulting Aims for the day 1. To present an overview of what Value for Money (VfM) means 2. To introduce ways of capturing Value for Money in international development through the Social Return on Investment approach 3. To share nef’s work on this subject to date 4. To facilitate discussion on how to further develop demonstrating value and impact of international development work © nef consulting new economics foundation (nef) • Founded in 1986 • Economics Think Tank working to promote: – Innovative solutions that challenge mainstream thinking on economic, environmental and social issues • nef consulting is the consultancy arm. It exists to promote and disseminate nef solutions – History of working with public sector on implementing value for money solutions © nef consulting Problems with measurement 1. Financial measurement: limited measure of value 2. We allocate resources only to the things we can measure 3: Stakeholders are left out of decision making © nef consulting The challenge • Measurement across the ‘triple bottom line’ The economy © nef consulting The environment People Value for Money (VfM) • VfM is about making sure that spending achieves as much as possible – A way of making decisions about how to use limited resources • Sometimes mistaken for lowest unit cost – Without a measure of quality or effectiveness, risk false economies • E.g. Children in care © nef consulting Treasury VfM Definition “VfM is defined as the optimum combination of whole-of-life costs and quality (or fitness for purpose) of the good or service to meet the user’s requirement. VfM is not the choice of goods and services based on the lowest cost bid.” HM Treasury (2006) Value for money guidance © nef consulting Treasury VfM Definition “Wider social and environmental costs and benefits for which there is no market price also need to be brought into any assessment. They will often be more difficult to assess but are often important and should not be ignored simply because they cannot easily be costed.” HM Treasury Green Book 1.Effectiveness, not lowest cost 2.Wider costs and benefits 3.Whole-of-life costs © nef consulting Value for Money Resources / Investment Often VfM is understood by comparing unit costs Money People Environment © nef consulting Service & Wider Outcomes Economic Inputs Outputs Real VfM is achieved by comparing outcomes with investment Social Environmental How does SROI fit in? • Framework for telling us how effectively money is spent – make VfM decisions • Adjusted form of cost-benefit analysis. SROI = value of positive + negative outcomes investment (or cost) • Essentially a measure of the efficiency of achieving outcomes • It considers triple-bottom-line benefits and investments – economic, social & environmental © nef consulting History of SROI Mid 1990s: REDF & Jed Emerson © nef consulting Early 2000s: new economics foundation Mainstreaming: nef consulting & others SROI today • • • • Third Sector: SROI Guide Local Government: commissioning Think tanks/public policy Central Government – Department of Health – National Audit Office: VfM – DFID: VfM • Pilot: HIV Aids Alliance • International interest – Europe, Australia, Canada, Asia © nef consulting Crash Course on SROI: Understanding change © nef consulting SROI Principles 1. Involve Stakeholders 2. Understand what changes 3. Value the things that matter 4. Only include what is material 5. Do not over claim 6. Be transparent 7. Verify the result © nef consulting SROI Process 1. Engage stakeholders to identify outcomes 2. Data collection • Outcomes • Deadweight, attribution, displacement • Benefit period and drop off 3. Model and calculate • Valuation of non-traded outcomes 4. Report © nef consulting Features that enhance VfM measurement • Focus on outcomes – Places quality and effectiveness at heart of analysis • Measures what matters, not what is easiest – Stakeholder-informed – Grapples with outcomes measurement • Values traded and non-traded outcomes – Failing to monetise non-traded outcomes effectively they gives these a value of zero • Rigour and transparency – Concerned with impact: deadweight, displacement, attribution, drop off © nef consulting Develop a theory of change Inputs Activities Outputs Outcomes Output: Tells you an activity has taken place and is usually quantitative (e.g. number of people trained) Outcome: The change that occurs as a result of an activity (e.g. improved well-being of training participants) © nef consulting Case study: African Community Development Foundation (ACDF) Background • ACDF started of as a U.K community based organisation, supporting socially isolated Africans in various ways. • In 2009, ACDF sent 28 volunteers as a pilot in Kisumu, Kenya known as Diaspora Development Associates (DDA). • ACDF specialise in team volunteering • Kenya has a preference for formal employment rather than selfemployment but ormal employment hard to get • High level migration from Kisumu to Nairobi to secure jobs • Due to preference for formal employment, very limited public interest for business training in Kisumu so organisations do not provide it. • The Government of Kenya in 2008 launched an ‘enterprise fund’ to encourage people to start their own business but there was hardly any uptake of these funds in Kisumu. © nef consulting Case study: African Community Development Foundation (ACDF) Activity • ACDF delivered Train the Trainer courses to infrastructure organisations and to direct beneficiaries. • This comprised of DDAs with skills covering- business set up, business management, book-keeping, organisational development, fundraising etc. © nef consulting Exercise 2: Impact Mapping • What changes do the stakeholders experience as a result of the project? I.e. what is the story of change? © nef consulting Impact map Stakeholders Kisumu’s unemployed resident X Kisumu’s infrastructure organisations ACDF’s volunteers (DDAs) State © nef consulting Inputs / Activities Train the trainer courses on: business set up, business management, book-keeping Outputs Outcomes Outputs vs. outcomes • Outputs do not always lead to the desired outcome • Key challenge in determining VfM is measuring outcomes – Outcomes measurement rarely takes place – Biggest obstacle in our work across sectors/organisations is the lack of outcomes data © nef consulting Crash Course on SROI: Measuring & valuing change © nef consulting Measuring outcomes • Some outcomes are ‘hard to measure’ – subjective, qualitative, diverse, far removed from the activity, long term • Systems and tools for capturing complex outcomes are improving – distance travelled – translating into quantitative format that enables aggregation – collecting complex information simply © nef consulting Evidencing outcomes • Distance travelled © nef consulting Example: National Accounts Model of well-being www.nationalaccountsofwellbeing.org © nef consulting Evidencing outcomes • Select ‘ways of knowing’ that an outcome (a change) has taken place = indicators Stakeholder Outcome Indicators Stakeholder (A): long term unemployed Kisumu resident Beneficiary (A) has sustainable, meaningful employment • (A) sustains employment for at least 6 months • (A) reports levels of job satisfaction • (A) improved financial situation © nef consulting Ways of measuring Stakeholder Outcome(s) Indicators (Data Collection Method) Stakeholder (B): Volunteers (DDAs) Increased employability • (B) increase in salary Increased well-being e.g. confidence, empowerment, aspiration • (B) reported Stakeholder (C): Kisumu State Improved economic situation • Increase in tax take Stakeholder (D): UK State Improved economic situation (from volunteering) • Increased tax take from salary increase © nef consulting improvements in wellbeing • Reduction in use of state services (if relevant) Valuing • SROI analyses value to all material stakeholders, not just the one funding the activity. • Key question: “what does this (change) mean to you?” Exercise • Imagine your employer gave you a day off. What will you do on your day off? What does this free time mean to you? • If you had to put a financial value on this, what would it be? I.e. what is this day off worth to you? © nef consulting Valuing and Pricing Value means different things to different people. • How much is your house/flat worth? • Who sets the price of fish? BUT • What does it mean to a Kisumu resident to have a job? • SROI uses financial proxies to estimate the social value of non-traded goods to different stakeholders. • Common currency © nef consulting Subjectivity of value Value is adjusted...not by any accurate measure, but by the higgling and bargaining of the market, according to that sort of rough equality which, though not exact, is sufficient for carrying on the business of common life. © nef consulting Subjectivity of value Can I put a value on anything? Can I find a value which, though not exact, is sufficient for helping me evaluate change? © nef consulting Financial proxies • Sometimes this is straightforward – mainly with outcomes to the State (e.g. value of health) • More difficult with non-traded outcomes – These do not have a ‘price’ and so require a proxy, or stand-in (e.g. emotional well-being) – Standard economic valuation techniques • Contingent Valuation – Willingness to pay (e.g. new park) or willingness to accept compensation (e.g. noise pollution) • Revealed Preference – Hedonic pricing (e.g. high risk job); Travel cost method (e.g. local service); Observed spending on related goods (e.g. leisure) © nef consulting Monetisation • Each outcome is then valued • Where no direct financial value is available, we use financial proxies to represent the social value created Stakeholder Outcome Possible proxy value Beneficiary (A): long term unemployed Kisumu resident Beneficiary (A) has sustainable, meaningful employment £ Value of increased earning potential OR £ Cost of fees for a skills development training course Stakeholder (B): Volunteers (DDAs) Increased well-being e.g. confidence, empowerment, aspiration Contingent valuation: what would (B) pay for to get similar level of well-being? Revealed preference – what do we observe (B) does to get a similar level of wellbeing? © nef consulting Valuation exercise Exercise: 1. In pairs, select 2 outcomes from the ACDF impact map 2. Think about options for financial proxies for the outcome. Remember, we are putting a value on the outcome for that particular stakeholder! © nef consulting Crash Course on SROI: Attribution & Partnership working © nef consulting Understanding impact • Deadweight: what would have happened anyway? • Attribution: how much is down to this project, and how much down to other factors? • Displacement: have we just moved an outcome to / from somewhere else? • Benefit period: how long does the outcome last, and does the effect ‘drop off’ over time? © nef consulting Impact: Attribution • Attribution – how much credit can your organisation take for the outcomes? – Expressed as a percentage – Not an exact science • Methods: – Based on your understanding of the journey of change – Involve stakeholders – interviews or surveys – Consult with other organisations © nef consulting Attribution stages 1. No account of attribution levels Effectively attribution = 100% 2. Informed estimate of attribution levels Qualitative research allows SROI practitioner to divide attribution between contributors 3. Quantitative research Quantitative stakeholder research gives average stakeholder’s view on contributors 4. Academic research Academic research can help refine (although not replace) stakeholder research © nef consulting Attribution Quantitative stakeholder research Time (years) © nef consulting Partnership working © nef consulting Projecting into the future • Benefit period – the length of time over which outcomes are expected to endure – e.g. benefits of an employment training programme may endure for some years after the course • Drop off – the rate at which benefits decrease over time – e.g. it is likely that benefit for training participants wears off as time goes on • Time value of money : discount rate • NB. Comparing value in different countries © nef consulting Inputs • Size of investment – If just looking at one project within organisation, need to work out its ‘share’ of inputs (Full Cost Recovery) – Where multiple funding streams, need to have identified the share of benefits when scoping if looking at return on just one of these investments – Include financial and non-financial inputs e.g. volunteer time © nef consulting Example & Organisational decisionmaking © nef consulting Hypothetical Example Outcome: capture the value of the influence on national policy-making i.e. changes for stakeholders that result from the national policy. INGO in Nigeria • Part of National Task Group on Sanitation • Successfully completed community-led total sanitation pilot (CLTS) • Advocated within the Task Group for drafting of National Strategy for Scaling-up Rural Sanitation and Hygiene • Key lesson: INGO’s advocacy on the National Strategy is not the outcome – it is a means to an end. © nef consulting INGO Impact Map Outcomes Value of improved health to the individual Value of increased economic activity to the State © nef consulting Value to the individual • No monitoring in place to track impact of the National Strategy • Forecasted using secondary research and stakeholder engagement Outcome Incidence Assumption 1: An additional 2% of the rural population is provided with improved access to sanitation. Disease Diarrhoea Schistosomiasis Ascariasis Hookworm © nef consulting Number of incidences avoided 29596 19319 85601 9111 Value to the individual Impact • Deadweight (population reached) -0.25% – baseline trend improvements in sanitation for rural communities (UNICEF/WHO) • Attribution to National Strategy 75% – significant change but concurrent factors included: International Year of Sanitation 2008 which saw increases in funding and other new initiatives • Attribution to INGO 5% – number of other actors involved in Task Group e.g. UNICEF, DFID. Credit for role in policy formation is 35% but implementation involved other actors including local government and this reduced attribution to 5%. © nef consulting Value to the individual Proxies – value of health to individual is made up of: • Increased income: fewer days lost to illness (working age population) – Based on secondary research on average rural wage in Nigeria • Direct costs associated with illness (all ages) – Based on WHO research into transport & medical expenses • Value of increased well-being (all ages) – Non-traded outcome. – (Limited) stakeholder engagement to find out what investments rural Nigeria might make to improve well-being. – Cost of generator divided by 5 for value to individual. • All proxies converted in US$ using World Bank PPP © nef consulting Value to the individual Benefit Period • Health improvement dependent on continued access to sanitation – WaterAid research suggests 5 years benefit period • Policy assumed to be in effect for 3 years Drop-off • WaterAid research suggests 16% drop-off over 3 years. – – – – Year 1: 2% Year 2: 4% Year 3: 10% Years 4 & 5: 20% per year (nef estimate) Discount rate: 10% (DFID) © nef consulting Value to the individual Social Value • Total value of improved health to individuals as a result of policy $22,561,910.66 • INGO share of value $1,128,095.53 • Cost incurred by INGO during policy influencing $14,863.00 Return on investment $1:$75.90 • Caution: benchmarks needed to understand whether ratio represents VfM © nef consulting Value to the individual Sensitivity Analysis ($1:$75.90) Assumptions in model systematically varied. • Decreasing outcome by 95% = $1:$11.81 • Well-being proxy is a sensitivity – Doubling proxy = doubling ratio – Removing proxy = $1:$1.21. – Proxy informed by limited stakeholder engagement therefore worth investigating further • Reducing attribution to 2% reduced ratio to $1:$30.36 © nef consulting Decision-making: organisational • Model to commission/plan/ design services or projects • Recommendation to DFID • Adoption by Local Authorities © nef consulting Organisational Decision-Making © nef consulting Sustainable Commissioning Commissioner & service user priorities 1. Activity 2. Output Community strategy & Corporate priorities 3. 4. Service level outcomes Camden Community outcomes –social –economic –environment National outcome frameworks © nef consulting 5. Value –Quantitative –Qualitative –Monetiseable Where value accrues: - To Service - Camden wide - central government Answering key questions • As a sector, how do we take this forward as a sector in terms of demonstrating the value and impact of our work? 1. Is this is an approach that can be useful? 2. In what circumstances can it be useful? 3. What are the limitations of this approach? 4. Where are the opportunities to for this to be adopted? 5. Any other reflections/thoughts on the approach? © nef consulting Summary: SROI as VfM measurement • Focus on outcomes – Places quality and effectiveness at heart of analysis as only outcomes tell you that change has occurred • Values traded and non-traded outcomes – Enables consideration of wider costs and benefits – Failing to monetise non-traded outcomes effectively they gives these a value of zero • Stakeholder informed – Ensures that you measure what matters to those experiencing the change – i.e. ‘quality of fit’ for service user • Rigour and transparency – Concerned with impact: deadweight, displacement, attribution, drop off © nef consulting Summary • SROI is a tool for assessing VfM and for achieving continuous improvement – Proving and improving • Not an ‘all or nothing’ approach, complement existing M&E systems • Next steps – from Bond or nef – Pilot? © nef consulting