Developing Consensus Principles and
Standards for Evaluating Global and
Regional Partnership Programs (GRPPs)
Progress Report to the Fifth Meeting of the DAC
Evaluation Network, November 16, 2006
Chris Gerrard
GRPP Coordinator, IEG
Outline of the Presentation
►Background
►Stakeholder consultative workshop held on
September 28-29, 2006
►Presentation of the revised Draft Principles and
Standards document
►Demand for a second, companion document
►IEG’s continuing role moving forward
2
Background to the Presentation
►IEG’s presentation to the Fourth Meeting on
March 31, 2006
►Broad support expressed for collectively developing
consensus principles and standard for evaluating
GRPPs
►Request that IEG play a leading role in developing
them
3
What are GRPPs?
►Programmatic partnerships in which:
• The partners contribute and pool resources (financial,
technical, staff, reputational, etc.) toward achieving
agreed-upon objectives over time
• The activities of the program are global, regional, or
multi-country (not single country) in scope
• The partners establish a new organization with a
governance structure and management unit to deliver
these activities.
►The majority of GRPPs are housed in existing
(host) organizations
4
Developing Consensus Evaluation
Principles
►Why?
• To improve the independence and quality of
program-level evaluations of GRPPs in order to
enhance their relevance and effectiveness.
►For Whom?
• Governing bodies and management units of
GRPPs
• Evaluators engaged in evaluating these programs
• High-level policy makers in aid agencies and
developing countries
5
Workshop Participants
September 28-29, 2006
17
6
6
3
2
5
6
3
3
6
Bilateral agencies and OECD secretariat
UN organizations
Multilateral development banks
Foundations
Developing country chairs of governing bodies
NGOs/private sector
GRPP management
GRPP evaluators
Evaluation associations
Objectives of the Workshop
To review an initial set of draft principles and
standards for GRPPs which IEG had prepared,
based upon:
►Extensive review of existing sets of related
principles
• DAC Principles and Standards
• UNEG Norms and Standards
• ECG, evaluation associations, etc.
►Evaluative work that IEG done on GRPPs over the
past few years: www.worldbank.org/ieg/grpp
7
A Self-Standing and
Comprehensive Document
► Evaluation Governance and Process Issues
• Enabling conditions, including M&E framework
• Independence and impartiality
• Participation and inclusion
• Transparency and disclosure
• Planning for scope and methodology
► Evaluation Content and Criteria Issues
• Relevance
• Effectiveness (or efficacy)
• Efficiency or cost-effectiveness
• Governance and management
• Sustainability, risk, and strategy for devolution or exit
8
Major Improvements Between
First and Second Drafts
►Enabling conditions, including M&E framework
►Independence and impartiality
►Participation and inclusion
►Planning for scope and methodology
►Relevance
►Effectiveness (or efficacy)
►Efficiency
►Governance and Management
►Sustainability
9
Demand for a Second
Companion Document
►Workshop participants who represented the
governing bodies and management of GRPPs liked
the structure and content of the present document.
►Evaluators working in bilateral agencies, UN
organizations and MDBs as well as GRPP
evaluators desired something more detailed and
specific to GRPPs, including guidance notes and
good practice examples on specific issues
►A number of participants also offered to help
contribute to the second document
10
Particular Requests for Guidance
Notes and Good Practice Examples
► Establishing good results-based M&E systems
► Developing generic indicators for generic types of GRPP
activities
► Ensuring independence of evaluation and mitigating
conflicts of interest in small programs
► Preparing terms of references, including scope and
methodology
► Assessing relevance, effectiveness, and efficiency
► Assessing legitimacy and effectiveness of governance and
management
► Assessing alternative strategies for devolution or exit
11
M&E Framework Often Lacking
Among 36 Programs IEG Has Reviewed
Clear and coherent
objectives and
strategies
Systematic and
regular processes
for data collection
A structured set of
indicators
0%
20%
40%
60%
80%
100%
Percent of Programs
12
High
Substantial
Modest
Negligible
Information on Evaluation Process
Also Lacking in Evaluation Reports
Yes
No
List of people
interviewed
Terms of reference
List of documents
consulted
Who commissioned
the evaluation
Who managed the
evaluation
To whom the
evaluators reported
0%
13
20%
40%
60%
80%
100%
Percent of Evaluations
Information on Evaluation Process in
Evaluation Reports (cont.)
Yes
No
How draft report was
reviewed
Who approved the
TOR
How evaluators were
selected
Response from
program
Planned dissemination
strategy
How much evaluation
cost
0%
20%
40%
60%
80%
100%
Percent of Evaluations
14
Difficult to Assess Degree of
Independence from Evaluation Reports
Yes
No
Don't know
Organizational
independence
Behavioral
independence
Protection from
external influence
Avoidance of
conflicts of interest
0%
20%
40%
60%
80%
100%
Percent of Evaluations
15
Most Evaluations Assessed Relevance;
Few Assessed Efficiency
Relevance
Effectiveness (or
efficacy)
Governance &
management
Resource mobilization
& financing
Partnership &
participation
Efficiency/costeffectiveness
0%
20%
40%
60%
80%
100%
Percent of Evaluations
16
High
Substantial
Modest
Negligible
Most Evaluations Used Interviews &
Document Reviews; Half Used Surveys
Yes
No
Interviews
Document review
Literature review
Site visits
Surveys
Case studies
Web-based feedback
Focus groups
0%
17
20%
40%
60%
80%
100%
Percent of Evaluations
Suggested Options for Moving Forward
► Disseminate the draft Principles and Standards widely, without
endorsing them at this time
► Encourage GRPP governing bodies & management to be guided
by these, and to provide feedback based on experience
► Revise the document in 1-2 years in the light of experience and
feedback received
► Start preparing the second companion document, based on:
• Reviews of existing GRPP evaluation policies, M&E
frameworks, and completed evaluations
• Consultations with partner representatives on governing
bodies, GRPP management, GRPP evaluators, evaluation
associations, and other evaluation experts
18