Update on the
Multilateral Effectiveness
Initiative
James Melanson
Director of Evaluation
CIDA
DAC Network on
Development Evaluation
June 2013
The Challenge
Gap in information on development effectiveness
• Great need for performance information of MOs
– Current climate for evidence-based decision making
regarding resource allocation
• Variable coverage, quality and reliability of reporting among
MOs does not provide clear picture of their performance
• Infrequent, lengthy and high cost of independent joint
evaluation of MOs
• Existing efforts focused primarily on
organizational assessment of MOs that
do not directly address development
effectiveness
2
The Response
New methodology
• 2009 – DAC EVALNET Task Team established to develop new
methodology that:
– Generates a body of credible information on a common set of
criteria that would provide a picture of the development
effectiveness of MOs
– Builds on evidence (evaluation reports) which is already available
– Uses methods that are modest in time and cost requirements and
with limited burden on MOs
• 2010 – Methodology developed and pilot tested (ADB & WHO)
• 2011 – DAC EVALNET endorses as an acceptable
methodology
3
Assessing Development Effectiveness
Common criteria
• Methodology focuses on a description of development
effectiveness that maps onto the DAC evaluation criteria
– Achievement of development objectives and expected results
– Crosscutting themes (environmental sustainability and gender
equality)
– Sustainability of results/benefits
– Relevance of interventions
– Efficiency
– Use of evaluation and monitoring to
to
improve effectiveness
4
Scenarios & Options
Preliminary Review
Establish Universe, Screen Reports
Scenario A
Scenario B
Scenario C
MO reporting on DE is
adequate
MO reporting on DE is not
adequate but evaluation
function is
MO effectiveness reporting
and available evaluations
inadequate for reporting on
DE
Option 1
Rely on MO
reporting systems
Option 2
Conduct a systematic
synthesis of
information from
available evaluations
Option 3
Implement actions
aimed at strengthening
MO evaluation system
and DE reporting
Apply the meta-synthesis of
evaluation results methodology
5
Methodology
Preliminary Review
• Establish universe of evaluation reports prepared by the MO
over a three- to four-year time-frame
• Select sample that provides reasonable coverage of MO
programming (geographic, thematic, objectives, sector,
technical focus)
• Screen reports from the sample for quality using DAC and
UNEG’s accepted quality standards
• Decide on Scenario A, B or C depending on screening results
– In order to validate if MO falls under Scenario A, it might still be
assessed under Scenario B
6
Methodology
Meta Synthesis (Scenario B)
• Review, analyze and classify evaluation findings for each
criterion (operational guidelines)
• Identify contextual factors contributing to or inhibiting
effectiveness for each criterion
• Prepare report that summarizes findings and context, and
establishes conclusions and recommendations
7
Experience to Date
• CIDA led, jointly with the Netherlands, the WFP and UNDP
reviews
 Successful use of the methodology – WFP and UNDP fall under scenario B
(reporting on DE not adequate but evaluation function is)
 Reviews provided good understanding of the organizations’ development
effectiveness
 Constructive conversations took place at the boards
• CIDA prepared reports based on the pilot tests of the ADB and
WHO reviews
 Successful use of the methodology
 Reviews provided good understanding of the organizations’ development
effectiveness, however, given the low number of evaluations available from
WHO, no generalization at the organization level was possible
8
Experience to Date
• CIDA led the AfDB review
 Report in process of finalization and will be published on CIDA and
EVALNET websites
• Netherlands led the UNICEF review
 Report presented to the board in May 2013
9
The Findings
• Organization is effective in achieving most of its objectives and
expected results
• Programs are highly relevant to the needs of the target groups
and developing country governments
• Sustainability and efficiency represent areas for improvement
• Challenges exist with gender equality and environmental
sustainability
• Good use of evaluation, but inadequate performance frameworks
and weak monitoring
Scenario B
10
The Findings
• Organization is effective in achieving most of its objectives and
expected results, and in supporting gender equality and
environmental sustainability
• Programs are highly relevant to the context in which they work
• Improving the sustainability of benefits remains a challenge
• Efficiency is an area for improvement
• Organization faces issues in strengthening decentralized
systems for evaluation, monitoring and results-based
management
• Evaluation Office produces high quality evaluations
Scenario B
11
The Findings
• Insufficient evidence available to make generalizable
conclusions
• The limited number of evaluation reports provide some insights
into the effectiveness of those programs
– Programs appear to be relevant to stakeholder needs and national
priorities and effective in achieving most of its development
objectives and expected results
– Programs appear to be sustainable, but there are challenges in
sustaining the capacity of its partners
– Evaluations have not regularly addressed effectiveness in
supporting gender equality or environmental sustainability
– Systems for evaluation and monitoring to be unsatisfactory
Scenario B
12
The Findings
• Most programs achieve their objectives and expected results
• Programs are relevant to stakeholder needs and national
priorities
• Improving the sustainability of benefits remains a challenge
• Efficiency represents an area for improvement
• Programs contribute to gender equality and environmental
sustainability, but improvements are needed with the latter
• Evaluation is effective and well used, but challenges are
highlighted in monitoring and results-based management
Scenario A
13
The Findings
• Programs are largely effective and highly relevant to the needs of
target groups
• A stronger focus on gender equality and environmental
sustainability is needed
• Continuity and sustainability of program benefits remain a
challenge
• Efficiency appears to be a challenge
• Effective use of evaluation through increased preparation of
management responses, but challenges remain with integration
and dissemination of evidence
• Challenges remain with respect to monitoring and results-based
management
Scenario B
14
MO Engagement
• Donors engage with multilateral organizations during the whole
review process
–
–
–
–
–
–
Launch of the review
Establishment of the evaluation universe
Confirmation of evaluation sample
Preliminary findings
Draft report
Final report
• Donor-neutral version prepared
each of the reviews and published
on the OECD EVALNET website
for
 For use by any partner donor
15
Utility of this approach?
• For CIDA
– Allowed demonstration of accountability for results in multilateral investments
– Improved the ability to substantiate positions in engagement at the Boards
• For Multilateral Organizations
– Optimizes resources and reduces transaction costs of reviews by utilizing
published and publicly available evaluation reports that encapsulate
evidence-based progress being made and lessons learned at country,
regional and global levels (UNDP)
– Report interesting and valuable, appropriately nuanced and reasonably
reflecting the evaluation reports and findings (WFP)
– Appreciate use of meta-synthesis methodology but need to ensure source of
information is comprehensive and current (ADB)
– Best practice and calls for its further emulation (UNDP and WFP)
– Constructively discussed at the Boards and provided impetus in areas of
needed improvement (anecdotes from Board participants)
16
Complementarity with
•
A study of complementarity with MOPAN was completed during the pilot
test phase (2010)
– Findings included:
• The two approaches focus on different aspects of multilateral
effectiveness and rely on different information sources
• Results are complementary rather than duplicative
• Together, they can provide a more complete picture of multilateral
organization's overall performance
•
However, does this still hold true with the new “results component” of
the MOPAN assessment?
– Both undertake document review
– Some similarities in criteria (relevance and achievement of outcomes)
– Differences in methodological approach
17
Criteria
•
•
•
•
•
•
Are objectives and expected
results at the national and local
level in developing countries
achieved?
Are interventions relevant to needs
of target groups and its members?
Are benefits and results achieved
sustainable?
Is programming delivered in a costefficient manner?
Does programming support
crosscutting themes (gender
equality and environmental
sustainability)?
Is evaluation and monitoring used
to improve development
effectiveness?
•
•
•
Extent of progress towards
organization-wide outcomes
Extent of contribution to countrylevel goals, priorities and MDGs
Relevance of objectives and
programme of work to stakeholders
18
Approach
•
Data sources
•
•
•
•
•
Evaluation reports
Other documents, including annual
reports (e.g., development
effectiveness, strategic plans),
COMPAS entries, DAC/UNEG Peer
Reviews
•
•
•
Methods
•
•
•
Preliminary review
Meta-evaluation to establish quality
and coverage of evaluation
performance information
Meta-synthesis of a sample of
evaluation reports to assess
development effectiveness and
identify conclusions and
recommendations
Data sources
•
Strategic plans, performance
reports, mid-term reviews,
evaluations, reviews
Country strategies, work plans,
reports, evaluations, reviews
MDG reports, national development
strategies and plans
Key informants from HQ, donor
countries and direct partners
Methods
•
•
Document review of all available
documentation
Survey of a sample of key
informants
19
Possible Next Steps
• Independent evaluation of MOPAN now in progress
• Comparison of MOPAN “results component” and EVALNET
development effectiveness review in the context of the 2013
IFAD assessment:
– Type/level of information obtain from both approaches
– Level of effort and resources required
– Ability to address information gap on development effectiveness
• Explore how EVALNET can
collaborate with MOPAN?
– Ensure complementarity
– Possibly integrate best components
of each methodology
20