Preparing High-Quality
Implementation Completion and
Results Reports
Perspectives from the Independent
Evaluation Group
Martha Ainsworth and Soniya Carvalho
Part I: Tips for preparing a high-quality ICR
Martha Ainsworth
Part II: Project ratings:
connects and disconnects
Soniya Carvalho
Why is ICR quality important?
►The ICR is an opportunity to learn and to
improve effectiveness
• It’s the official (public) report on the project, its
achievements and lessons
►Low quality ICRs make it difficult for IEG to
rate the project accurately
• Unsatisfactory ICR quality is a good predictor of
ratings “disconnects” and disputes
• From 7/1/2006 onward, “when insufficient
information is provided by the Bank for IEG to
arrive at a clear rating, IEG will downgrade the
relevant rating”
What are the IEG criteria for ICR
quality?
►Results-orientation (ICR should be outcomeoriented, not an implementation narrative)
►Quality of evidence and analysis
►Lessons based on evidence and analysis
►Internal consistency
►Consistency with Bank guidelines
►Conciseness
1. Make the ICR results-driven
a. Organize evidence around achievement of objectives (not
implementation of components)
b. Document the “results chain”: Show the link between
inputs, outputs, outcomes, impacts
c. Explain the counterfactual (what would have happened
without the project), other factors operating, plausible
attribution
d. Don’t be constrained by the official project indicators
e. Show trends over the whole period, before and after the
project, as many observations as possible
f. Footnote sources of evidence, triangulate data.
 Provide the evidence necessary for someone disconnected
from the project to be able to rate it!
The Results Chain
Inputs  Outputs  Outcomes  Impacts
To assess efficacy using the results chain, you need:
• to be able to link these elements plausibly
• to understand the “counterfactual”— what would
have happened without these activities
Establishing a “counterfactual”
►Timeline of events
►Account for trends in other determinants
(For example, other donor support, weather,
changes in national policies, household
income, changes in prices that could affect
demand or incomes)
►Document trends before and after, in
project and non-project areas
Example #1: Eritrea HIV/AIDS, Malaria,
STI, and TB Control Project
Northern Red Sea
Distribution of Malaria
in Eritrea
Anseba
Gash barka
Maekel
Debub
Highest ranking
Lowest ranking
Southern Red Sea
120
45
40
Interventions
100
35
30
25
20
15
80
60
40
10
5
0
20
0
1999
2000
2001 2002
2003 2004 2005
2006
Incidence rate (per 100,000);
Rainfall ('0mm)
Annual outputs of anti-malaria
interventions increased
ITNs + bednets
reimpregnated (10,000)
Breeding sites filled and
treated (1,000)
Number of houses
sprayed (1,000)
Incidence rate (per
100,000)
Average annual rainfall
(10mm)
Nyarango et al. (2006) (incidence and rainfall data)
National Malaria Control Program (2004, 2006) (intervention data)
120
45
40
Interventions
100
35
30
25
20
15
80
60
40
10
5
0
20
0
1999
2000
2001 2002
2003 2004 2005
2006
Incidence rate (per 100,000);
Rainfall (10mm)
New cases of malaria declined
ITNs + bednets
reimpregnated
(10,000)
Breeding sites filled
and treated (1,000)
Number of houses
sprayed (1,000)
Incidence rate (per
100,000)
Average annual
rainfall (10mm)
Nyarango et al. (2006) (incidence and rainfall data)
National Malaria Control Program (2004, 2006) (intervention data)
120
45
100
40
35
80
30
60
25
20
40
15
20
10
5
0
0
1999
2000
2001
2002
2003
2004
2005
2006
Incidence rate (per 100,000);
Rainfall (10mm)
Interventions
Rainfall also declined, but malaria continued
to decline when rainfall recovered
ITNs + bednets
reimpregnated
(10,000)
Breeding sites filled
and treated (1,000)
Number of houses
sprayed (1,000)
Incidence rate
Average annual
rainfall
Nyarango et al. (2006) (incidence and rainfall data)
National Malaria Control Program (2004, 2006) (intervention data)
Example #2: Russia Health Reform Pilot Project
Russia’s abortion rate declined during the project
120
Abortions per 1,000 women aged 15-49
110
100
90
80
70
60
50
40
Health Reform Pilot Project
30
20
1993
1994
1995
1996
1997
1998
Kaluga
1999
2000
Tver
2001
2002
2003
2004
It declined even faster before the project
120
Abortions per 1,000 women aged 15-49
110
100
90
80
70
60
50
40
Health Reform Pilot Project
30
20
1993
1994
1995
1996
1997
1998
Kaluga
1999
2000
Tver
2001
2002
2003
2004
It declined all over Russia, without the project
and there were no data for the results chain linking project
activities to the decline in abortion rate
120
Abortions per 1,000 women aged 15-49
110
100
90
80
70
60
50
40
Health Reform Pilot Project
30
20
1993
1994
1995
1996
1997
1998
Russia
1999
Kaluga
2000
Tver
2001
2002
2003
2004
Example #3: Romania Roads 2 Project /road safety
Road fatalities declined in Romania
Vehicle registration (000)
Fatalities
Fatalities/1000 vehicles
4500
10
4,104
4000
8
3500
5.9
3,128
3000
2,863
2,436
2500
6
2000
4
1500
1000
2
500
0
0
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
Fatality rate/1000 vehicles
Fatalities & vehicle registration
9.2
They declined faster before the Traffic Safety Campaign
and there were no intermediate outcome data on safety
Vehicle registration (000)
Fatalities
Fatalities/1000 vehicles
4500
10
Year of traffic
safety 2002-3
4000
4,104
8
3500
5.9
3,128
3000
2,863
2,436
2500
6
2000
4
1500
1000
2
500
0
0
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
Fatality rate/1000 vehicles
Fatalities & vehicle registration
9.2
2. Show all indicators of efficiency
► ICRs for all investment-type projects (including TA
projects) must assess efficiency, not ICRs for DPLs
► Answers the question: Were costs in achieving the
objectives reasonable in relation to the benefits and to
recognized norms (“value for money”)
► Show evidence of:
•
•
Cost-benefit, cost-effectiveness, efficient use of project resources,
unit rate norms, service standards, least-cost analysis and
comparisons, other efficiency indicators
Aspects of design and implementation that contributed to or reduced
efficiency
► Cost-benefit analysis is not sufficient to assess
efficiency; other indicators must be shown
•
If presented, transparent discussion of assumptions
3. Document safeguards
►Safeguard policies not applicable to DPLs
►Applicable safeguard policy, project category
(A, B, C, FI), assessment/mitigation plan
►Whether the mitigation plan was
implemented
►Findings of third party safeguard reviews
►Updated assessment and revised mitigation
plans implemented if physical components
generating economic and social impacts
were modified.
4. Other quick tips
► Consistency: Make sure the ratings match the text and
the numbers are internally consistent
► Conciseness: More evidence is good, as long as it is
relevant and concise; if extensive, add an annex.
► Completeness: Ensure that the sections and annexes
are complete and accurate
► Candor: Be candid about shortcomings, data quality
► Lessons: Not too many; must be evidence-based and
come from the project’s experience; seek to explain
variability in outcomes
GOOD LUCK!