Bangladesh Joint Country
Assistance Evaluation: Assessing
Total ODA at the Country Level
Presentation to OECD DAC November 2006
Bruce Murray
Director General
Operations Evaluation Department
Asian Development Bank
Very Early Stages
Everything said is subject to change.
Benefits of Total ODA
Evaluation: Richard Manning’s
Challenge
•
•
•
•
•
•
Consistent with Paris Declaration: (i) mutual
accountability; (ii) partnering; (iii)
harmonization; (iv) alignment.
Reduced transaction costs for Government
Objectivity and legitimacy
Broader scope
Ability to address issues that are beyond
those addressed in evaluations undertaken
by one donor
October 2005 Paper “The effectiveness of
ODA – an evaluation proposal” focuses on
the impact of total ODA on policy choice and
implementation in a country.
Why Bangladesh as Pilot for
Total ODA Evaluation?
• Many donors active in country
• 4 major donors have a joint country
assistance program (WB; ADB; JBIC;
DfID)
• Some major donors require country
evaluations by 2009
• Use of SWAps.
Phasing of the Evaluation
• Phase 1 : joint country evaluation by
the World Bank, ADB, JBIC/JICA and
DFID (80% of ODA).
• Phase 2: integrate the 28 donors
delivering the other 20% of ODA.
• Daunting coordination problems. Need
for strong information sharing between
donors involved in Phases 1 and 2.
Mechanism not yet developed.
Issues Requiring Resolution:
Involving the Government
• Involving Government as partner in the evaluation work
– Paris Declaration on mutual accountability.
• Proposal to Government must be an open one, including
on approach and methodology (listen to Government’s
views)
• Identify issues that the Government wants addressed
(e.g., donor coordination; harmonization; use of country
systems)
• Work with Government evaluation units and local
research organizations.
• Involvement of NGOs that deliver much of the bilateral
aid.
Issues Requiring Resolution:
Approach and Methodology
• Identify issues that cannot be addressed in the country
assistance evaluations of individual donors.
• What is the counterfactual?
• Need for common approach and methodology for the
separate building blocks of the evaluation.
• Alignment with joint Country Strategy and PRSP. Note:
Joint Strategy assigns lead donors by sectors and
themes.
• Lead donors evaluate all interventions in the
sector/thematic area, regardless of funding (data base
being assembled to map available evaluation reports
and ongoing operations and strategy).
Issues Requiring Resolution:
Study Management
• Need a strong evaluation management mechanism
• Composition and role of steering committee –
Government, 4 major donors and 1 OECD DAC
representative.
• Coordination among task managers and teams, mission
planning, etc.
• How to balance internal needs of donors for evaluation
of their programs and joint evaluation (attribution to
total ODA, not individual donors)
• Reporting – one report on total ODA or do donors also
need separate reports on their operations for
accountability? Design methodology to be flexible
enough to produce single or multiple reports.
• Logistics, timing and cost sharing of the evaluation
Issues Needing Resolution:
Involving other Donors
• How to involve the donors delivering the other
20% of ODA?
• 1 representative from the OECD DAC Total ODA
task force on the steering group
• Involvement other donors up front in
identifying the evaluation questions.
• Sharing of information during the
sector/thematic evaluations
• Providing information during for the sector/
thematic evaluations
• Role of country offices and Headquarters
evaluation units.
Proposed Schedule
Present to end of 2006
Data collection and data sharing among main donors (on going);
decision on first sector/thematic evaluations among WB/ADB/JBICJICA/DFID. Nomination of steering committee for joint evaluation
and lead agencies for sectors/themes.
January-April 2007
Develop management structure for evaluation. Preparation of
approach paper for CAE, one sector/thematic evaluation and for
Jamuna Bridge/Access Road /Railway Link joint evaluation
April/May 2007
Proposal for joint country evaluation submitted to Government;
joint mission to Bangladesh to discuss proposal.
June 2007
Detailed decisions on joint country evaluation, costing and timing
and mechanism for coordination with the other donors.
July 2007
Final decision on allocation of other sector/thematic evaluations to
various evaluation partners
August-October 2007
Preparation of detailed evaluation approach papers for initial
sector/thematic evaluations, preparation of TORs, consultant
selection, detailed planning for joint missions, etc.
November 2007-June 2008
Field work
July – December 2008
Discussion and drafting of individual sector/theme reports
January – April 2009
Drafting and presentation of synthesis report;
Dissemination activities.
Challenges of Joint
Evaluations (DAC Guidelines)
• Donor need for accountability and attribution
often takes precedence over joint efforts to
assess interrelated impacts of multiple
agencies
• Finding subjects suitable for joint evaluations
(program support, multilateral, multidonor
funded projects) is difficult
• Processes for coordinating joint evaluations
can be complex and increase the cost and
duration of the evaluation
DAC Lessons:
Joint Evaluations
• Joint evaluations only reduce transaction costs
if they replace, not add to, individual
evaluations
• Identify key partners in evaluation;
involvement of government agencies as
evaluators is good practice
• Agree on the management structure early
(broad membership steering committee and
smaller management group; mixed
approaches, some areas of joint evaluation,
others to be delivered separately)
DAC Lessons:
Joint Evaluations
• Key areas of agreement:
• Plan for dissemination and follow
up