Discussion Paper
Framework, key questions and criteria
for evaluating experience with results-based approaches
1. Introduction
In our first meeting on 26 November 2012 we agreed that we have a strong need both for synthesis
of the experiences and for a growing body of evidence and recommendations. Therefore we have
written this discussion paper with a concrete proposal for key questions and criteria for evaluating
experiences with results-based approaches.
2. Type of results-based approach
Results-based approaches is a generic term to describe any approach where development
aid is linked in any way to performance The distinguishing feature of these approaches is
that they link inputs (material goods and advisory services) and payments to the achievement of predetermined performance targets and/or results, which may involve the entire results chain (outputs, outcomes and impacts).
Results-based aid (RBA): RBA approaches are based on a contract between the donor and
partner government that ties the transfer of donor funds to the achievement of results. Examples include the MDG contracts of the European Commission and the CoD approach of
the Centre for Global Development.
Results-based funding (RBF): This type of approach involves agreements within a partner
country that are entered into between the partner government or another – usually governmental – institution (e.g. a ministry or public authority) and an implementing partner (e.g. a
public institution at the local level, a private business or an NGO) or a beneficiary of services.
It also involves the use of national funds of the partner country. Examples of RBF approaches include conditional cash transfer and voucher programmes.
Key Questions:
-
Does the approach in question involve results-based aid (RBA) or results-based financing (RBF)?
-
Which category (e.g. Cash on Delivery, output-based aid) does the approach fit into?
3. General conditions: country and sector context
“In the partner system, interdependent agents and levels manage the responsibility for outcomes - and for institutional change - in a way that is integrative of multiple policy management
threads and of social and political considerations. If a Results Based Approach with donors becomes an option, what matters, from the systemic perspective is less, how many or few indicators are on the list, but more, if the processes that have been found consensus on follow the internal logic, are coherent with each other and complete in terms of timings, stakeholders, results
and the internal allocation of resources.” (Meyer, 7f).
“Almost certainly, some types of PBR may “work” in some sets of circumstances, but not in others. This suggests a shift in emphasis, with less attention to “does PBR work?” and more on
identifying the circumstances under which it might be a more effective and appropriate strategy
As at: 27 June 2013
Authors: Jörg Freiberg and Michael Krempin (GIZ)
Page 1
to apply than alternatives. This is very much a question of impact, not merely of process.” (Perry, 20).
Key Questions:
-
Do the sectors offer good opportunities for applying results-based approaches?
Exists a result-oriented budgeting at country and at sector level?
Were the partner systems generally receptive to results-based approaches, e.g. through
their own experience with results-based management?
Who initiated the results-based approach: the donor or the partner?
By which process were the expected results defined? Which was the empirical evidence
for defining the indicators? Who participated in that process?
Are performance targets part of the result approach? Exists a baseline study characterizing options and barriers for implementing result-based approaches?
Does the partner country have the required financial resources and institutional capacities to implement results-based approaches, especially RBA?
How and on which terms have activities been financed by the partner side (partner’s own
funds, private banks, etc.)?
4. Impact
Result-based approaches link inputs (material goods and advisory services) and payments to
the achievement of predetermined performance targets and/or results. In order to assess expected and unexpected impacts, an understanding is needed regarding how incentives actually
work in practice and when they may or may not be sufficiently motivating to induce changes in
behaviour. This is not only an issue of financing, but also of institutional capacities, human resources, organizational culture, modes of cooperation, etc. Assessing the impact therefore requires an on-going monitoring and learning process observing the changes within the incentive
system, comparing alternatives in terms of cost-effectiveness and result-orientation.
Key Questions:
Incentives system: objectives and indicators
-
What was the proportion of funding for the results-based approach in relation to other
funding in the sector?
What was the design of the process used to devise the objectives and indicators?
What experience was gained with different systems of objectives and indicators (e.g. focus on a single indicator vs. focus on a more complex system of indicators)?
How important were unintended results and the risk of false incentives?
Were there any indications of gaming and the loss of intrinsic motivation in connection
with results-based approaches?
Monitoring and verification
-
Have the partners’ M&E systems proved suitable for monitoring results-based approaches? How great is the need to improve partners’ M&E capacities?
Was it possible to use partner country data to measure performance?
What experience was made with the different approaches used for third-party verification
of the results achieved?
Cost efficiency
As at: 27 June 2013
Authors: Jörg Freiberg and Michael Krempin (GIZ)
Page 2
-
Can statements be made on their cost-efficiency (because there were comparable projects or programmes with similar objectives, for instance)?
If statements can be made: How cost-efficient were results-based approaches as compared with other approaches?
How much cost/effort did the results-based approach entail (compared with a conventional project set-up)?
Results/impact
-
-
Which results were achieved, and to what extent?
What is the ratio of quantitative to qualitative results in connection with results-based approaches?
How sustainable are the results?
Were results also achieved at the impact level?
Did the results-based approaches focus in practice on target groups that were easy to
reach, or did they also reach poor population groups in remote areas?
Did the results-based approaches adhere to and strengthen the aid effectiveness principles (ownership, alignment, harmonisation, managing for results and mutual accountability)?
Is it possible to assess whether the results-based approach was more effective than other approaches?
5. Donor Contribution
Embedding results-based approaches into other DC approaches:
-
Were the results-based approaches part of more comprehensive development cooperation (DC) approaches?
Complementarity of financing and capacity development (CD) support within results-based
approaches:
-
How did the results-based approach combine financing and CD support? How did financial cooperation instruments interact with those of technical cooperation?
6. Summing up: challenges, opportunities and risks of results-based approaches
-
What are the main challenges, opportunities and risks in connection with results-based
approaches?
Under which conditions do results-based approaches offer comparative advantages over
other approaches?
What potential do the approaches offer for replication and scaling up?
As at: 27 June 2013
Authors: Jörg Freiberg and Michael Krempin (GIZ)
Page 3