Project Ratings:
Connects and Disconnects
Soniya Carvalho
Lead Evaluation Officer and ICR Review Coordinator, IEGPS
Independent Evaluation Group
How are Projects Rated at the World Bank?
• Project ratings are based on evaluation
criteria that are harmonized across IEG and
OPCS
• Main ratings are: Outcome, Risk to
Development Outcome, Bank Performance,
and Borrower Performance
• Evaluation methodology is objectives-based
What are the Harmonized Evaluation Criteria?
• Outcome = Relevance + Efficacy + Efficiency
• Risk to Development Outcome = Likelihood
of detrimental change x Impact on outcome if
that change materializes
• Bank Performance = Quality at Entry +
Quality of Supervision
• Borrower Performance = Government
Performance + Implementing Agency(ies)
Performance
Why do ICR and ICR Review Ratings Differ?
Reason #1: Disagreement over project objectives
or their weighting
• Objectives in ICR are not taken from PAD/Financing
Agreement
• ICR does not discuss how any poorly articulated
objectives have been interpreted
• “By” and “through” parts of objectives are treated as
the intended outcomes
• Relative weights given to different objectives in arriving
at ICR ratings are not made explicit and justified
Why do ICR and ICR Review Ratings Differ
(Cont’d)
Reason #2: Insufficient evidence on
achievement of objectives/efficacy
• Too much focus on outputs and inputs to the
neglect of outcomes
• Reported Key Performance Indicators do not fully
capture achievement of objectives
• Results chain neglected – attribution/plausible
association between outputs and outcomes not
adequately explained
Why do ICR and ICR Review Ratings Differ?
(Cont’d)
Reason #3: Differences in treatment of country
circumstances
• In rating Project Outcome, IEG makes no allowance
for difficult country context; objectives and design
are supposed to take that into account
• However, difficult country contexts do factor into
the Bank Performance rating
• Project Outcome rating is based on results not
effort
Why do ICR and ICR Review Ratings Differ
(Cont’d)
Reason #4: Lack of familiarity with the ratings
• ICR rates project outcome based only on
achievement of objectives, neglecting relevance
and efficiency
• Relevance in ICR may ignore current relevance
• Harmonized criteria for deriving Bank
Performance from its two constituent elements
not observed (Ditto for Borrower Performance)
How can the disconnect be reduced?
• Ensure clarity about the objectives being used as
the benchmark for evaluation
• Ensure that the ICR contains evidence on the
entire results chain, from outputs to outcomes,
including for institutional objectives
• Explicitly discuss attribution/plausible
association, providing contextual information on
overlapping government or other donor
activities in the sector
How can the disconnect be reduced? (Cont’d)
• Discuss the relevance of project outcomes to
the current CAS
• Acknowledge gaps in achieving the intended
outcomes, drawing insightful lessons
• Acknowledge gaps in M&E, indicating proxy
information used to underpin the ICR’s
judgments
Even an unsatisfactory project with poor
M&E can win an IEG Good Practice ICR
Quality Award!
Thank you