IDB’s Evaluation Architecture for
Public and Private Sector
Project Evaluation
Cheryl Gray, Director
Office of Evaluation and Oversight (OVE)
Inter-American Development Bank
Presentation at the meeting of Sub-Regional Development Institutions
Johannesburg, 28 April 2014
Some background on IDB
 The IDB Group includes IDB, the Inter-American Investment
Corporation (IIC), and the Multilateral Investment Fund (MIF)
 26 client countries and about $12b in lending per year
– Biggest borrowers: Argentina, Brazil, Colombia and Mexico
– Over 80% lending is to public sector (sovereign-guaranteed)
 About $150m in technical cooperation per year
– funded by ordinary capital and trust funds
 About 2000 employees and $600m administrative budget
IDB’s Accountability System
Board of
Executive Directors
Independent
Consultation and
Inspection Mechanism
Office of Evaluation
and Oversight
IDB Management
Management’s Internal
Accountability Mechanisms
•Self Evaluation
•Management oversight
•Institutional Integrity
•Internal Audit
•Ombudsperson
OVE is independent from IDB Management
 REPORTING: OVE Director selected by Board for 5-year term (renewable once);
reports directly to the Board; can be removed only by Board and only on performance
or ethical grounds
 BUDGET: “below-the-line” and approved by Board; currently 28 staff, 15 Research
Fellows
 EVALUATION PROCESS: formal protocol for management and Board review
 ACCESS TO INFORMATION: OVE has full access to all IDB information
 RECOMMENDATION FOLLOW-UP: Online system tracks adoption of OVE
recommendations
 CONFLICTS-OF-INTEREST: Evaluators follow conflict-of-interest rules
 OUTREACH: OVE handles its own external outreach
 IDB POLICIES: OVE follows all IDB financial management and HR policies and
procedures
OVE seeks to address both
accountability and learning
 Accountability for results:
– Is the Bank doing what it says? Is it achieving its objectives?
To what extent is it contributing to country outcomes? With what
results and development impact?
 Learning:
– What projects and programs achieve results – relevance,
efficacy, and efficiency – and why?
– How can M&E be strengthened in the Bank and in client
countries?
OVE produces a range of evaluation products
Notes,
summaries
Thematic, sectoral,
corporate evaluations
Country program evaluations
Project-level evaluations
Several tools are used for project
evaluation in IDB
 Development Effectiveness Matrix (DEM): evaluability ex-ante
 Project Monitoring/Supervision Reports (PMR/PSR): monitoring
during implementation
 Project Completion Reports (PCR) [public sector] or Extended
Project Supervision Reports (XPSR) [private sector]: selfevaluation (ex-post)
 Independent evaluation (OVE)
– DEM, PCR, XPSR validations
– Comparative project evaluations
To date public and private projects have been
evaluated using different criteria in MDBs
Public GPS Criteria
Private GPS Criteria
Relevance
Financial Performance and Fulfillment of
Project Business Objectives
Effectiveness
Economic Sustainability
Efficiency
Contribution to IFI Mandate Objectives
Sustainability
Environmental and Social Performance
Overall Project Performance
IFI Performance
IFI Investment Profitability
Borrower Performance
IFI Work Quality/Bank Handling
IFI Additionality
…and using somewhat different
approaches and standards
 Basis of the evaluation
– Public sector: objectives-based
– Private sector: commercial benchmarks
 Attribution and additionality
 Environmental and social safeguards
 Definitions of rating scales and what is required for a
satisfactory rating
IDB is moving toward more unified
criteria and guidelines
 Common basis of evaluation -- the project’s “objectives” (i.e.
intended results)
 Common criteria that are conceptually consistent, with some
sub-criteria specific to public and private sector operations
 Common guidance on evidentiary requirements, analytical
methods, and benchmarks for a positive rating
 Common approaches to attribution
The goals are comparability and a central focus on development
results in all projects.
Public Sector Sub-Criteria
ECG Harmonized Criteria
Private Sector Sub-Criteria
Relevance
Results
Achievement of Outputs
Operational Performance
Achievement of Outputs; Fulfillment of
Project Business Objectives
Contribution to Intended
Outcomes
Contribution to Corporate Goals
Unintended Outcomes
Economic Efficiency
Implementation Efficiency
Outcome Sustainability
Compliance with Safeguards
Quality at Entry
Quality of Supervision
Government and Implementing
Agency Performance
Efficient Use of Resources
Sustainability
IFI Performance
Client Performance
IFI Investment Profitability
Financial Performance
Economic Efficiency
Implementation Efficiency
Outcome Sustainability
Commercial Sustainability
Compliance with Safeguards
Quality at Entry
Quality of Supervision
Non-Financial Performance of the
Company
Other ongoing challenges to address
 The quality of PCRs has been uneven but is now improving (pilot program).
 The project monitoring system changed recently and may need further adjustments.
 The evaluation framework for non-lending work is less-developed than for projects
(in IDB and other MDBs).
Our goal is to continue to strengthen IDB’s focus on results.
Thank you
www.iadb.org/evaluation
Extra slides
Relevance
• Consistency of the project’s intended results with beneficiary needs, country
priorities, and IFI assistance strategy and corporate goals
• Justification for public sector involvement
• Evidence-based justification for market failures and targeting
• Logic linking project activities to intended intermediate and final outcomes
Results
Operational Performance
Contribution to Intended Outcomes
Contribution to Corporate Goals
Unintended Outcomes
Public Sector
Achievement of
Outputs
Private Sector
Results
Operational Performance
Achievement of Outputs
Fulfillment of Project
Business Objectives
• The extent to which the project achieved its targeted
outputs (appraisal projections or performance targets)
• The extent to which the project delivered on its process
and business objectives (private sector)
• Attribution analysis not needed
Results
Contribution to Intended Outcomes
• The project’s contribution (or expected contribution) to its intended
development, policy, or transition outcomes
• The project’s reach to intended beneficiaries
• The project’s success in reducing or compensating for market failures
• The project’s impact on all stakeholders (employees, suppliers, competitors,
etc.) (private sector)
Results
Contribution to Intended Outcomes
• Outcomes are assessed against a without-project
counterfactual, either using an impact evaluation or (more
commonly) a theory-based approach to establish plausible
causality
• The rating reflects the project’s incremental contribution to
observed outcomes, regardless of whether observed outcomes
moved in the “right” or “wrong” direction
• On an exceptional basis, “Not Rated” is possible when
evidence is missing or weak
Rating Based on Results Attributable
to the Project
Observed
decrease
in outcomes
indicator
Observed
What would have happened
without the project
(counterfactual)
Outcome
attributable to
the project
Counterfactual
Project Start
Project End
Results
Contribution to Corporate Goals
• The project’s contribution to broad corporate goals that were
not included in the project’s intended results (e.g., rural poverty
reduction, shared prosperity, gender equality)
• Uses a theory-based approach to establish plausible causality
• The rating reflects the project’s incremental contribution to
observed outcomes, regardless of whether observed outcomes
moved in the “right” or “wrong” direction
• “Not Rated” is a possible rating
Results
Unintended Outcomes
• Positive or negative results of the project that were not reflected
in its intended outcomes (e.g., environmental damage)
• Uses a theory-based approach to establish plausible causality
• Unintended outcomes must be at least as well evidenced as the
project’s intended outcomes
• “Not Rated” is a possible rating
Public Sector
Economic Efficiency
Implementation Efficiency
Private Sector
Financial Performance
Efficient Use
of Resources
Economic Efficiency
Implementation Efficiency
• The profitability of the project from the perspective of its financial
stakeholders (private sector) and society at large (public and private sector)
• The financial performance of the project/company is compared to a withoutproject counterfactual (private sector)
• The economic efficiency of the project (CBA or CEA) is compared to a
without-project counterfactual (public and private sector)
• Also considers implementation efficiency (e.g., delays)
Public Sector
Private Sector
Outcome Sustainability
Outcome Sustainability
Compliance with Safeguards
Sustainability
Commercial Sustainability
Compliance with Safeguards
• The sustainability of outcomes achieved or expected to be achieved (public
and private sector)
• The commercial sustainability of the company, financial institution, and/or
sub-borrowers/fund investees (mainly private sector)
• The environmental and social sustainability of project results, measured by
compliance with relevant safeguards (public and private sector)
Relevance
Results
Operational Performance
Contribution to Intended Outcomes
Contribution to Corporate Goals
Unintended Outcomes
Efficient Use of Resources
Sustainability
Overall Project Performance Rating
Public Sector
Quality at Entry
Quality of Supervision
Gov’t. and Implementing
Agency Performance
Private Sector
IFI Performance
Client Performance
IFI Investment
Profitability
Quality at Entry
Quality of Supervision
Non-Financial Performance
of the Company