Sub Section D2 - Methods for Impact Measurement
The methods that are concerned with impact measurement covered here are:
-
Change Check
Tell Your Story - Community Impact Mapping
Social Accounting and Audit
Social Return on Investment
Change Check – A Practical Guide to Assessing Impact
A systematic approach to looking at impact based on a review that considers survey information
collected for the purpose. Addresses four questions:
What do we want to achieve? - What is the core mission of the organisation?
What levels of impact are we having in our work now? - Social economic environmental cultural
Where do we want to strengthen our impact – and are there any areas of impact we are making
that we do not want? - Identifying potential improvements
How do we make the changes necessary to strengthen these impacts? - Prioritising and planning
action
Tell Your Story – Community Impact Mapping
A simple approach to basic impact mapping proposed as a five question conversation held with
stakeholders. The idea is to Tell a Story that can then be used where appropriate for a more formal
and systematic impact mapping exercise.
Starting out
Resources
Action
Nearly there?
Result!
Vision - what did
you want to
achieve for your
community?
What did you use
- Staff?
Volunteers?
Money?
Activities - what
did you do with
the resources?
What happened
because of what
you did?
What difference
did your actions
make to your
community
Source: Byrne K (2006) “Community Impact Mapping” DTA
Social Accounting and Audit
A detailed and thorough approach to impact mapping with its origins firmly in a critique of financially
focussed accounts of organisations activities and the need to include outcomes that are social,
environmental and cultural as well as financial and economic. Like other impact measurement tools
stakeholder views are important as are impacts that are difficult to quantify.
Social Accounting and Audit
Step 1 : Planning
The organisation clarifies its mission, objectives and activities as well as its
underpinning values. It also analyses its stakeholders through completing a
‘stakeholder map’. These exercises help the organisation to make explicit what
it does, why and how it does it, and who it works with and whom it seeks to
benefit.
Step 2 Accounting
In this phase, an organisation decides the ‘scope’ or focus of the social
accounts, especially if it will build a comprehensive picture over time. The
organisation then sets up ways of collecting relevant information over a period
of time to report on performance and impact against its values and its
objectives, encompassing both quantitative and qualitative. The information is
then brought together and analysed.
Step 3 Reporting
The information that was collected, collated and analysed in Step 2 is brought
and auditing
together in a single document, which serves as a draft of the social accounts.
People from outside the organisation (a Social Audit Panel) then review this
document to check that the report is based on information that has been
properly gathered and interpreted. When the Panel is satisfied with the report
it can then be made available to stakeholders.
Source: New Economics Foundation (2009)
Social Return on Investment
“Developed from traditional cost-benefit analysis and social accounting,
SROI is a participative approach that is able to capture in monetised form the value of a wide range
of outcomes, whether these already have a financial value or not. An SROI analysis produces a
narrative of how an organisation creates and destroys value in the course of making change in the
world, and a ratio that states how much social value (in £) is created for every £1 of investment.”
(NEF 2009)
There are two types of SROI –
Evaluative – This is a retrospective look at impacts and social value created over a given period. This
is possible only if relevant data is available which is often difficult if the need for its collection has
not been identified in advance.
Forecast – a projection of what social value may be created if forecast objectives are met.
There are six stages to a SROI calculation:
1. Establishing scope and identifying key
stakeholders.
2. Mapping outcomes
3. Evidencing outcomes and giving them a
value.
4. Establishing impact.
Clear boundaries about what the SROI will cover,
and who will be involved are determined in this
first step.
Through engaging with stakeholders, an impact
map, or theory of change, which shows the
relationship between inputs, outputs and
outcomes is developed
This step first involves finding data to show
whether outcomes have happened. Then
outcomes are monetised – this means putting a
financial value on the outcomes, including those
that don’t have a price attached to them. Also
have to consider drop off rates for outcomes –
how long they last.
Having collected evidence on outcomes and
monetised them, those aspects of change that
5. Calculating the SROI.
would not have happened anyway (deadweight)
or are not as a result of other factors
(attribution) are isolated.
This step involves adding up all the benefits,
subtracting any negatives and comparing them
to the investment.
6. Reporting, using and embedding
Easily forgotten, this vital last step involves
sharing findings and recommendations with
stakeholders, and embedding good outcomes
processes within your organisation.
Source NEF (2009) “Tools for You - Approaches to proving and improving for charities voluntary
organisations and social enterprises
Pros and Cons of the approaches
It is clear that the first two approaches are less resource intensive than the last two. They are also
less complex and require fewer skills and limited external verification.
This is in contrast to the two approaches which are very resource intensive and require special skills
and/or external verification. In the case of SORI there is also a cultural obstacle for most not for
profit organisations in the use of financial proxies to measure the value of activities and impacts that
many people consider are not measurable or where proxies used are arbitrary and subjective (e.g.
using the national minimum wage as a proxy for all labour inputs or the cost of a gym membership
as a proxy for the value of improved health). Many not for profit organisations that have used SROI
have reported the difficulties of getting all stakeholders around the table to agree what inputs (and
their value) can be attributed to them and agree the financial proxies for impacts that involve quality
of life, preventing people from turning to crime, getting off drugs etc.
Proponents of all the approaches do not consider them to be useful comparative tools so the SROI
ratio for one organisation for example should not be compared to the SROI ratio of another.
SROI proponents also recognise the “short termism” inherent in the use of a net present value
calculation:
“In order to calculate the net present value (NPV) the costs and benefits paid or received in different
time periods need to be added up. In order that these costs and benefits are comparable a process
called discounting is used. Discounting recognises that people generally prefer to receive money
today rather than tomorrow because there is a risk (e.g., that the money will not be paid) or because
there is an opportunity cost (e.g., potential gains from investing the money elsewhere). This is
known as the ‘time value of money’. An individual may have a high discount rate – for example, if
you would accept £2 in one year’s time, instead of £1 now, that implies a discount rate of 100%.This
is a controversial area and one where there is ongoing research and discussion. The main problem
with using discounting in SROI is that it encourages short-termism by discounting the future. This is
especially problematic for environmental outcomes” (Cabinet Office/OTS 2009)
All approaches however are considered important for improvement of performance and provide
potential for analysis of an organisation over time in much the way that financial accounts do.