Valuing the unquantifiable?
How VFM can be applied to complex projects
Increased public and political scrutiny of aid budgets has prompted governments and aid
agencies to place even greater emphasis on assessing the ‘value for money’ (VFM) of their aid
programmes and projects. However, the impacts of many programmes are not easily quantified.
This Briefing Note outlines a solution to this problem, based on a VFM analysis of a large,
multi-dimensional development project in Kenya.
Background
The UK’s Independent Commission on Aid Effectiveness (ICAI) neatly summed up the dilemma of measuring
value for money: “What matters is not always easy to quantify: we recognise that evaluating the achievement
of many objectives within the UK aid programme – such as increasing accountability and reducing conflict and
fragility – may rely more heavily on qualitative than quantitative analysis. We do not believe it is right to shy
away from the difficult-to-measure impacts, as these are often the ones that are most effective in the long-term.”
The ICAI added that the aim should be “to adopt a pragmatic approach, without being prescriptive or mechanistic”.
We agree, and found that such an approach can work, as we discuss in this Briefing Note.
One solution to the challenges of measuring VFM is to recognise two types of project output: those that can
be quantified with plausible assumptions and at reasonable costs for data gathering and analysis; and those that are
inherently difficult to quantify. It is suggested that VFM assessments should focus on the first of these types. This is a
more credible approach than one that seeks to put numerical values on things that are almost impossible to quantify.
Admittedly, this can work only in cases where some of the benefits from an aid project are large enough on their own
to justify the full costs of that project. But good and well-conceived aid projects will often have this characteristic.
Key points
•Many aid interventions, such as market development or community led approaches,
are designed to facilitate a process of change, rather than to deliver a well-defined
physical output. At the start of such a process, there are only informed assumptions.
So the projects in question cannot be expected to supply a detailed statement of the
precise activities, events and outcomes that will eventually emerge from that process;
•Moreover, this process involves other players, with the result that attributing
the credit for ‘success’ to any one or any group of players will invariably be
judgemental. In addition, the complexity of the direct and indirect impact
pathways makes it difficult to evaluate them;
•Despite these hurdles, it is often possible to demonstrate VFM of a programme
as a whole by concentrating on a few quantifiable elements, as OPM’s VFM
assessment of Kenya’s Financial Sector Deepening Programme illustrates.
While not all impacts could be quantified for this project, gains valued at
£103.9 million were measured – more than three times the cost of the project.
Briefing Note
2
Valuing the unquantifiable? – How VFM can be applied to complex projects
The challenges of VFM
In the past few years, VFM has become the catchphrase around which the response to the political pressures
on aid budgets has been organised by DFID in the UK and by some other aid organisations. Linked as it
invariably is to the terms ‘effectiveness’, ‘economy’ and ‘efficiency’ (the ‘3 Es’), VFM as a concept seems
difficult to argue against.
As the recent summary of the issues by the Independent Commission on Aid Effectiveness (ICAI) illustrates,
it is relatively easy to explain the usefulness of VFM and its supporting concepts in relation to a project with
a well-defined ‘hard’ development objective – such as the provision of anti-malarial bed nets. In such cases,
there will typically be some relatively simple metrics available to measure VFM.1 However, there are other
development objectives that are not so easily measured. Nevertheless, as the ICAI notes, such objectives
can often be “the most transformational, achieving a long-term, sustainable difference which empowers
intended beneficiaries”.
This suggests that, in some cases, the use of the VFM approach will inevitably involve a large degree of
subjective judgement – with evaluators free to utilise defendable combinations of known approaches to
evaluating aid projects2 that are viable given the availability of data, time and other resources to do the work,
and the point at which they enter the evaluation process. ICAI’s analysis is consistent with the view that
VFM can be seen as a convenient envelope that embraces these various different approaches rather than
being a wholly distinct alternative.
Specifically, the economic techniques such as cost-benefit analysis and results-based management are
referred to by ICAI as two particular approaches for integrating the concepts of VFM and effectiveness into the
planning, delivery and review of organisations’ activities. Other techniques, including randomised control trials
(RCTs) and community-based participatory methods, are referred to as two among several different ways of
addressing the numerous challenges involved in actually measuring VFM and aid effectiveness in practice.
Of course, not all of these techniques could be validly used in relation to all types of foreign aid project.
RCTs, for example, are inherently more useful for micro-based projects where it is possible to identify and
isolate ‘treatment’ and ‘control’ groups. This approach cannot be applied realistically in relation to meso- and
macro-level interventions (especially those that involve an influencing agenda). What would be the ‘control
group’ for the work of FSD Kenya in general, or even for measuring the impact of, say, FSD’s role in the
evolution of the policy and regulatory environment for branchless banking in Kenya?
The Financial Sector Deepening Trust (FSD) Kenya case is interesting not least because it shows that the
use of a very limited sub-set of recognised techniques applied to selected components of the project’s
delivered outputs is more than sufficient to demonstrate VFM.
Quantifying VFM in a complex programme
In 2001, the Department for International Development (DFID) initiated a project with the purpose of improving the
capacity of Kenya’s financial sector to meet the needs of poor rural and urban households and small and medium
sized enterprises – the Financial Sector Deepening Trust of Kenya (FSD). An evaluation of FSD’s activities in
2011 by Oxford Policy Management (OPM) concluded that FSD had fully achieved many of its objectives.
However, it also noted the inherent difficulties of assessing broad-based, multi-level initiatives such as FSD.3
In examining the detailed components of the FSD programme in Kenya, the OPM team concluded that the
impacts of four main components of the programme could be quantified. These impacts included:
1In this specific case, the metrics are: Economy – were the bed nets of reasonable quality purchased at the lowest price?
Efficiency – what proportion of the nets actually purchased were used for their intended purpose? Effectiveness – amongst
those persons provided with nets, how much did the incidence of malaria decrease? Equity – have the nets reached poorer
people and minority groups in reasonable numbers?
2Such approaches include cost benefit analysis, cost effectiveness analysis, randomised control trials, approaches linked
to results-based management, community scoring and other beneficiary assessments and related approaches.
3 See ‘Assessing Value for Money: The case of FSD Kenya’, OPM Working Paper, June 2012.
Oxford Policy Management
October 2012
Valuing the unquantifiable? – How VFM can be applied to complex projects
3
i.
Increased number and penetration of bank accounts. In the past few years for which there are firm
data, there has been a remarkable increase in the number of commercial bank accounts in Kenya.
ii.
Money transfers – lower costs and greater volumes. From a base of almost zero in 2007, the volume
of money transfers had risen by 2011 to some 30 million per month.
iii.
Increased availability of bank and non-bank financial outlets. This relates to the increased number
of both bank outlets and non-bank outlets (agents) and the consequences for this in terms of reduced
costs to make financial transactions.
iv.
The lower risks of failure of Savings & Credit Co-operatives (SACCOs). This relates to the benefits
that can accrue (and indeed have already accrued) from a reduction in the rate of loss of members’ funds
in SACCOs.
OPM assessed the total value of the benefits arising from the four main quantifiable components of FSD listed
above at £1,069 million in nominal terms (£1,039 million net present value). If a conservative 10% of these
benefits were attributed to FSD, that would give quantified benefits amounting to £103.9 million, as against
the overall programme cost of £29.4 million.
There were other non-quantifiable elements of the programme that were arguably more important to its overall
impact on financial markets, wealth creation and thus poverty reduction. However, the OPM team suggested
that the absence of any credible means to put values on these elements with the time and resources available
to the evaluation team effectively forced their omission from a fully quantified VFM analysis. Given more time
and budget resources, some of the non-quantified elements could probably have been quantified by using
additional assumptions.
Moving forward
In the case of the FSD Kenya project, this note illustrates that just four key contributions of the overall project
can provide a credible basis for assessing VFM. As better data becomes available, the authors expect that
some of the conservative assumptions made can be reviewed and some of the additional benefits quantified.
In the meantime, VFM assessment of FSD explicitly states all the assumptions used, so that these can be
subjected to closer scrutiny and further sensitivity testing, with the present analysis being seen as an
intermediate step to encourage better VFM analysis in the future.
For many aid projects it will be possible to demonstrate VFM of the programme as a whole by simply
concentrating on a few quantifiable elements. In marginal cases, the non-quantifiable elements can be brought
back into the analysis, while explicitly recognising the limitations of any quantification that is then attempted in
relation to those elements. It is suggested that this is a superior and more honest approach than one that
regards all elements as being equally amenable to quantification.
Find out more
OPM is pioneering new forms of VFM analysis and is active in numerous related projects for donors.
To find out more about the VFM analysis for the FSD project in Kenya, download our working paper
on this subject Assessing Value for Money: The case of donor support to FSD Kenya, from our
website (www.opml.co.uk).
Other VFM papers published by OPM include: Better Results? Value for money assessments of aid.
Information about the authors of this paper and details about OPM can be found on the next page.
Oxford Policy Management
October 2012
4
Valuing the unquantifiable? – How VFM can be applied to complex projects
About the authors
Sukhwinder Arora is a Principal Consultant at OPM, specialising in financial and private sector development.
Over the past 30 years, Sukhwinder has worked for and with a range of development organisations (including DFID
and ActionAid) to enable poor people to participate in and benefit from financial and other markets.
Alan Roe was until retirement in 2011 Principal Economist and Director at OPM. These appointments followed some
30 previous years of professional experience in international macroeconomics, industrial and financial sector policy
and economic statistics. Prior to joining OPM he was a Principal Economist at the World Bank, mainly engaged on
financial sector reform issues in the former Soviet Union. He also holds a part-time appointment at the University
of Warwick.
Robert Stone is an OPM Associate with 30 years’ experience of financial and private sector development, focusing
on the structure of markets within the financial sector and their impact on growth, investment and, particularly, poverty
reduction. He has developed methodologies for assessing and improving the reach and functioning of financial
markets and financial institutions.
Acknowledgements
This paper also benefited from peer review by David Hoole, Oxford Policy Management.
About OPM
Oxford Policy Management (OPM) is one of the world’s leading international policy development and management
consultancies. We enable strategic decision-makers in the public and private sectors to identify and implement
sustainable solutions for reducing social and economic disadvantage in low- and middle-income countries.
Supported by more than 100 full-time specialists on three continents, we have offices in the UK, Bangladesh,
India, Indonesia, Pakistan and South Africa. For further information, visit www.opml.co.uk
Contact us
If you would like to discuss how we might be able to help you with VFM assessments, or any other development
issue, please contact your nearest OPM office.
Africa
Asia (continued)
Europe
John Kruger
OPM Pretoria
Email: john.kruger@opml.co.uk
Abdur Rauf Khan
OPM Islamabad
Email: rauf.khan@opml.co.uk
Simon Hunt
OPM Oxford
Email: simon.hunt@opml.co.uk
Asia
Smita Notosusanto
OPML Jakarta
Email: smita.notosusanto@opml.co.uk
Aparna John
OPM New Delhi
Email: aparna.john@opml.co.uk
Asheq Rahman
OPM Dhaka
Email: asheq.rahman@opml.co.uk
ISSN 2042-0595 ISBN 978-1-902477-06-0
Oxford Policy Management
October 2012