The Paris Declaration Evaluation:
Some key issues for evaluators
Brief presentation To IPDET 2011
Prepared by Bernard Wood, Evaluation Team Leader
and
Niels Dabelstein, The PDE Secretariat
Topics covered
•
•
•
•
Basics on the Evaluation
Key Findings
“Shop talk” for evaluators
Sources of information and tools
The Paris Declaration and this report
• The Paris Declaration on Aid Effectiveness,
endorsed in 2005, is a landmark international
agreement and programme of reform – the
culmination of several decades of attempts to
improve the quality of aid and its impacts on
development.
• This Report is an independent global
evaluation of these efforts to improve the
effectiveness of international aid, especially
since 2005
3
Background and process for the Evaluation
Background
• The Declaration itself pledged an independent evaluation
- itself a tool for mutual accountability
• A fully joint (and transparent) evaluation conducted over
4 years (Phase 1: 2007-8; Phase 2: 2009-11)
Evidence base
• 22 Country-level evaluations led by partner countries and
managed in-country
• 18 Donor/agency HQ studies
• 7 Supplementary studies on key topics plus review of the
most significant global literature
4
Evaluation components
5
Country Evaluations & Donor Studies
6
Governance and management
• International Reference Group (50-plus reps. of governments,
international Organizations and CSOs. Co-chaired by Malawi
and Sweden)
• Management Group (Colombia, Malawi, Netherlands,
Sweden, US, Vietnam)
• Evaluation Secretariat at DIIS
• National/Agency Reference Groups and Evaluation
Coordinators
• National/Agency Evaluation Teams (with specified
recruitment criteria, and common generic ToRs)
• Core Evaluation Team (7 Members, from Canada, Denmark,
Malawi Nigeria, Peru, Sri Lanka and the UK)
• Peer Reviewers: Dr. Mary Chinery-Hesse and Mr. Mark
Malloch-Brown.
7
The Key Evaluation Questions
1. “What are the important factors that have affected the
relevance and implementation of the Paris Declaration and
its potential effects on aid effectiveness and development
results?” (The Paris Declaration in context)
2. “To what extent and how has the implementation of the
Paris Declaration led to an improvement in the efficiency
of aid delivery, the management and use of aid and better
partnerships?” (Outcomes for aid effectiveness)
3. “Has the implementation of the Paris Declaration
strengthened the contribution of aid to sustainable
development results? How?” (Development outcomes)
8
The Key Messages:
• The Paris Declaration has contributed to change of
behaviour – but unevenly so. Partner countries have
moved further and faster than donors. Some donors
more than others and some very little.
• The Paris Declaration has contributed to improve aid
effectiveness – but much remains to be done.
• The Paris Declaration has contributed to better
development results – but not across the board.
• The PD and AAA “campaign” remains relevant and
has gained momentum – but needs nurturing to
continue.
9
The Main Recommendations
Five to policy makers in both partner countries and
donor agencies and three to each of these groups
separately.
Some are not new – they may be seemingly obvious.
But key political actions must be pressed again –
simply and starkly – because they are so important and
the Paris and Accra commitments that have not yet
been met.
10
Strengths of Phase 2
• Evaluation Matrix developed for country studies, applying 11
outcomes of PD and AAA agreements.
• Integrated evaluation quality assurance and peer review
• Recognised the limits of aid in development and applied
“contribution analysis.”
• A targeted process of guidance & support, recognising the
primary importance of country studies.
• Good governance of the Evaluation at national and
international levels ( with 52 member International Reference
Group) ensured joint process and independence and
validated the framework and findings at key stages.
11
Key limitations
• Evaluating the effects of a political Declaration traditional ‘linear’ approaches were not relevant
• Limited time elapsed since 2005
• No comprehensive data from country studies on
multilaterals and donors
• Self-selection of participating countries / agencies –
some gaps but still a reasonably representative
“sample”
12
Challenges & lessons for complex, comparative evaluations
1. A fully participatory and independent approach is possible, but
demanding, and not cheap. It hinges on good governance
arrangements, synchronization, communications and support
2. Context is not just “background” – it is of the essence and should be
systematically analysed and featured
3. Finding the basic programme theory can be especially critical for very
complex evaluation objects – and it may be hidden in plain sight
4. Key common questions can be developed to reflect most cases, then
supplemented by special priorities for individual cases
5. Make the common evaluation framework feasible for all —a chain is
only as strong as its weakest link
6. Where new approaches are required, anticipate capacity differences
and needs for technical support to teams
13
Challenges & lessons ll
7.
Contribution analysis, properly applied, is indispensable for
evaluating many complex change processes
8. The pressure for rapid assessment and the necessary time to see
outcomes have to be balanced - evaluators must state the limits
9. Delays in practical arrangements for different participants’
mandates, structures, terms of reference, contracting, and approval
and release of reports can disrupt or derail the whole joint effort.
Clear expectations, peer pressures and deadlines ultimately need to
be backed by a readiness to move ahead without stragglers
10. For a multi-site evaluation, the development of ToRs and
procurement of teams should be concurrent and the Synthesis
process needs to be clarified from the start, in tandem with the
common framework
11. Visibly ensuring independence is critical, particularly where
potentially contentious findings are likely to emerge. This calls for
strong procedures and standards, and clear governance
14
Full reports and supporting materials
All documents from the Evaluation, including the full country
evaluations and donor studies, can be found
- in English, French and Spanish - on
www.busanhlf4.org
and
www.oecd.org/dac/evaluationnetwork/pde
Tools, templates and guidance materials (see Technical Annex to
the Report) will also be made available on the DAC site
Thank you for your attention
15
Supplementary slides for questions
16
Contributions to aid effectiveness
•
Pulled together and focused global attention on ambitious,
experience-based measures to improve development
cooperation and aid for better development results
•
Clarified the roles of ‘aid’ and ‘better aid’
•
Strengthened global norms of good practice
•
Helped progress toward 11 key outcomes set in 2005, but
very unevenly between actors and outcomes.
•
Improved the quality of aid partnerships - supported rising
aid volumes and hopes for better “North-South” relations
•
Ranked the degree of difficulty in objectives. Found better
progress among partner countries than among donors, who
(with some striking exceptions) have been too
uncoordinated and risk averse to play their full part
17
Examples of the range of performance
against each intended improvement (From Fig. 5)
18
Contributions to Development Results
Assessed in four key areas – health as a tracer sector, giving
priority to the poorest, strengthening institutions and social
capital, and a better mix of aid modalities.
Assessed through a three-question sequence:
– First, were development results achieved?
– Second, did aid contribute? Other influences always more
important.
– Third, did aid reforms plausibly strengthen the aid
contribution? (Contribution analysis a key tool)
Considered relevance, unintended consequences,
alternatives such as non-Declaration style aid, etc.
19
Contributions to Development Results 2
1. Results in specific sectors (health was the main case-study)
Declaration type measures have contributed to more focused,
efficient and collaborative aid efforts in health. These efforts have
already contributed to better development results since 2000-05,
and should be sustainable. The pathways of improvement are
indirect but clear. Not wide enough coverage of other sectors to
draw strong conclusions.
2. Priority to the needs of the poorest (especially women and girls)
Little progress in most countries in delivering on these
commitments. But evidence of some positive contributions by aid
and some value-added by Declaration reforms. A powerful national
commitment to change is a pre-requisite if aid is to help overcome
entrenched inequalities.
20
Contributions to Development Results 3
3. Strengthening institutional capacities and social capital
Insufficient capacity still a central obstacle to development - and
aid could help more with this than it does. Modest contributions
by aid and reforms to the long-term strengthening of
institutional capacities. Clearer evidence for contributions to
modest improvements in social capital.
4. Improving the mix of aid modalities
Evidence that employing a wider range of (especially joint)
modalities, has improved contributions to development results
in half the countries – especially at sector level. A mix of aid
modalities has continued to make sense for all actors.
21
Main Recommendations I
A. For decision-makers in both partner and donor
countries and agencies (at Busan and beyond):
1. Make the hard political choices and follow through
2. Focus on transparency, mutual accountability and shared risk
management
3. Centre and reinforce the aid effectiveness effort in countries
4. Work to extend the aid reform gains to all forms of
development cooperation
5. Reinforce the improved international partnerships in the next
phase of reforms
22
Main Recommendations 2
B. For policymakers in partner countries:
1. Take full leadership and responsibility at home for further aid
reforms
2. Set strategies and priorities for strengthening capacities
3. Intensify the political priority and concrete actions to combat
poverty, exclusion and corruption
C. For policymakers in donor countries and agencies:
1. Match the crucial global stakes in aid and reform with better
delivery on promises made
2. Face up to and manage risks honestly, admit failures
3. Apply peer pressure to ‘free-riders’ for more balanced donor
efforts
A dozen key areas are identified for work beyond the
Evaluation*
23