OFFICIAL USE ONLY
CODE2010-0038
June 9, 2010
For meeting of
Committee: Monday, June 21, 2010
COMMITTEE ON DEVELOPMENT EFFECTIVENESS
FROM: The Secretary, Committee on Development Effectiveness
Cost-Benefit Analysis in World Bank Projects
Attached is the Cost-Benefit Analysis in World Bank Projects, to be discussed at a meeting of the
Committee on Development Effectiveness scheduled for Monday, June 21, 2010.
Questions on the document should be addressed to Mr. M. Sundberg (ext. 34369),
or Mr. A. Warner (ext. 89696).
Distribution
Committee
Mr. Alzetta (MIGA)
Mr. Contreras
Mr. Hasan
Mr. Hofmann
Mr. Huber
Mr. Majnoni
Ms. Renteria
Mr. Shikibu
Mr. Yang
For Information
Other Executive Directors and Alternates
President
Bank Group Senior Management
Vice Presidents Bank, IFC and MIGA
This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents
may not otherwise be disclosed without World Bank Group authorization.
Cost-Benefit Analysis in
World Bank Projects
Report No. _____
June 4, 2010
This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents
may not otherwise be disclosed without World Bank authorization.
Abbreviations and Acronyms
ADB
ARDE
BP
CBA
DPL
ERR
FY
GDP
ICR
ICT
IDA
IEG
NPV
OP
PAD
SAR
Asian Development Bank
Annual Review of Development Effectiveness
Bank Procedure
Cost-benefit analysis
Development Policy Loan
Economic rate of return
Fiscal year
Gross domestic product
Implementation Completion and Results Report
Information and Communication Technologies
International Development Association
Independent Evaluation Group
Net present value
Operational Policy
Project Appraisal Document
Staff Appraisal Report
Contents
PREFACE ................................................................................................................... V
EVALUATION SUMMARY ....................................................................................... VII
1. WORLD BANK POLICY .......................................................................................... 1
Evaluation Managers
™ Vinod Thomas
Director-General, Evaluation
™ Cheryl Gray
Director, Independent
Evaluation Group
™ Mark Sundberg
Manager, IEG Corporate,
Global, and Methods
™ Andrew Warner
Task Manager
2. THE DECLINE IN COST-BENEFIT PRACTICE ...................................................... 5
3. THE SCOPE FOR COST-BENEFIT ANALYSIS ................................................... 11
4. QUALITY OF ANALYSIS ...................................................................................... 23
5. ACCURACY IN COST-BENEFIT CALCULATIONS ............................................. 29
6. USE OF COST-BENEFIT ANALYSIS FOR DECISION MAKING......................... 37
7. THE RISE IN RATES OF RETURN ....................................................................... 41
8. CONCLUSIONS .................................................................................................... 59
REFERENCES .......................................................................................................... 76
Tables
Table 1. Trends in Sector Composition and Proportion of Projects, by Sector,
Reporting Economic Rates of Returns .......................................................... 7
Table 2. The Shift from High-CBA Sectors to Low-CBA Sectors at the World Bank .... 9
Table 3. Sources of the Decline in ERR Reporting ...................................................... 9
Table 4. Numbers of Projects Reporting Economic Rates of Return at the Beginning
or at the Close of Projects, for Fiscal Year 2008 ......................................... 11
Table 5. Reporting of Rates of Return, Fiscal Year 2008, by Sector ......................... 12
Table 6. Reasons Offered in Project Completion Documents for Lack of Cost-Benefit
Estimates, Projects that Closed in Fiscal Year 2008 ................................... 14
Table 7. Trends in IEG Project Performance Ratings by Sector Board, 1993-2008 .. 47
Table 8. Country-by-Country Tests of the Impact of Economic Reforms on Project
Economic Returns ....................................................................................... 53
Table 9. The Rise in Economic Returns Correlates Strongly with the Rise in Market
Orientation in Client Countries, 1974-2007 .................................................. 55
i
CONTENTS
Boxes
Box 1. The Costs and Benefits of Cost-Benefit Analysis ........................................... 15
Box 2. Who Should Perform the Cost-Benefit Analysis? ........................................... 19
Box 3. Cost-Benefit Analysis and the Results Agenda .............................................. 26
Box 4. The Problem of Fragile Estimates .................................................................. 34
Box 5. The Difference between a 12 Percent and a 24 Percent Economic Rate of
Return—Example from Agriculture ............................................................. 42
Box 6. Recommendations of a 1992 Review of Cost-Benefit Analysis in the World Bank
.................................................................................................................... 60
Figures
Figure 1. Percentage of Bank Investment Projects with Estimates of the Economic
Rate of Return in the Appraisal Document, by Year of Approval .................. 5
Figure 2. Percentage of Bank Investment Projects with Estimates of the Economic
Rate of Return in the Final Completion Report, by Year of Completion ........ 6
Figure 3. Percentage of Investment Projects in the Five High-CBA Sectors with ERRs
in the Appraisal Report, by Year of Project Completion ................................ 8
Figure 4. The Shift toward Low-CBA Sectors Started in 1988 .................................. 10
Figure 5. Before-Project and After-Project Economic Rates of Return Have
Converged .................................................................................................. 30
Figure 6. Forecasting Bias in Predicted Economic Returns—Road Projects ............ 31
Figure 7. The Probability of Recalculation of ERRs at Closing is Higher in BetterPerforming Projects—High-CBA Sectors 1993-2007.................................. 32
Figure 8. The Probability of Recalculation of ERRs at Closing is Higher in BetterPerforming Projects—Low-CBA Sectors 1993-2007 .................................. 32
Figure 9. The Probability of Calculating a Final ERR Given that an Initial ERR Was
Calculated Has Always Been Lower for Lower-Rated Projects................... 33
Figure 10. Median Economic Rate of Return in Percent, All World Bank Projects,
1972-2008 ................................................................................................... 42
Figure 11. Median ERRs in Agriculture and Rural Development, in Percent ............ 44
Figure 12. Median ERRs in Energy and Mining, in Percent ...................................... 44
Figure 13. Median ERRs in Transport, in Percent ..................................................... 45
Figure 14. Median ERRs in Water and Urban Development, in Percent ................... 45
Figure 15. Median ERRs in all Other Sectors Combined, in Percent ........................ 46
Figure 16. Average IEG Performance Ratings—High-CBA Sectors—and Trend over
Time ............................................................................................................ 48
Figure 17. Average IEG Performance Ratings—Low-CBA Sectors—and Trend over
Time ............................................................................................................ 49
Figure 18. Average Economic Growth Rates Occurring during World Bank Project
Implementation ........................................................................................... 50
Figure 19. Median Economic Rates of Return in Bank Projects, IDA and Non-IDA
Countries .................................................................................................... 51
Figure 20. Percentage of Projects Completed in Countries that Implemented MarketOriented Policy Changes ............................................................................ 55
Figure 21. Economic Returns in Projects Correlate Positively with Economic Growth
1974-2007, Controlling for Reform .............................................................. 56
ii
CONTENTS
Figure 22. Economic Returns in Projects Correlate Strongly with Economic
Liberalization, Controlling for Growth........................................................... 57
Annexes
Annex A ..................................................................................................................... 67
Annex B. .................................................................................................................... 68
Annex C ..................................................................................................................... 69
Annex D. OP 10.04—Economic Evaluation of Investment Operations ...................... 73
Notes ......................................................................................................................... 79
iii
Preface
This report is prepared in the context of a major global effort in the
past eight years to better measure results in development assistance.
The agenda has been articulated and refined in a series of international conferences, beginning with the International Conference on Financing for Development in Monterrey in 2002 and continuing
through the Accra Agenda for Action in 2008. Cost-Benefit Analysis
entails measuring results, valuing results and comparing results to
costs, and hence is highly relevant to the results agenda. It can provide a comprehensive picture of the net impact of projects and help
direct funds to where their development effectiveness is highest.
The key documents from the international conferences cited above
rarely mention cost benefit analysis. This situation is mirrored at the
World Bank, where cost benefit analysis is rarely mentioned in recent
policy documents and where the application of cost benefit analysis
has been declining for the past three decades. The purpose of this report is to understand better why this trend is occurring and whether
the policies and practice of cost benefit analysis need to be revised.
This report was prepared by Andrew Warner (task manager). Management guidance and valuable comments were provided by Mark
Sundberg and Cheryl Gray. A background paper was prepared by
Pedro Belli and Pablo Guerrero that rated the analytical quality of
current cost benefit analysis at the Bank and compared it with a similar rating they conducted in the mid-1990s. Domenico Lombardi
conducted and analyzed staff interviews on the use of cost benefit
analysis at the World Bank. Diana Hakobyan provided research assistance and administrative support. Three peer reviewers provided insightful comments: Shantayanan Devarajan, Lyn Squire and Franck
Wiebe.
IEG colleagues provided helpful input and comments: Sanghoon
Ahn, Martha Ainsworth, Amitava Banerjee, Hans-Martin Boehmer,
Kenneth Chomitz, Ade Freeman, Roy Gilbert, Daniela Gressani, John
Heath, Monika Huppi, Ali Khadr, Marvin Taylor-Dormond and
Christine Wallich.
v
Evaluation Summary
Cost-Benefit Analysis in World Bank
Projects
Cost-benefit analysis used to be one of the World Bank’’s signature issues. It helped establish its
reputation as the knowledge Bank and served to demonstrate its commitment to measuring results and ensuring accountability to taxpayers. It was the Bank’’s answer to the results agenda
long before that term became popular. This report takes stock of what has happened to costbenefit analysis at the Bank, based on analysis of four decades of project data, project appraisal
and completion reports from recent fiscal years, and interviews with current Bank staff.
The percentage of projects that are justified by cost-benefit analysis has been declining for several
decades, due to both a decline in standards and difficulty in applying cost-benefit analysis. Where
cost-benefit analysis is applied to justify projects, there are examples of excellent analysis but also
examples of a lack of attention to fundamental analytical issues such as the public sector rationale
and comparison of the chosen project against alternatives. Cost-benefit analysis of completed
projects is hampered by the failure to collect relevant data, particularly for low-performing
projects. The Bank’’s use of cost-benefit analysis for decisions is limited because the analysis is
usually prepared after making the decision to proceed with the project.
This study draws two broad conclusions. First, the Bank needs to revisit the policy for costbenefit analysis in a way that recognizes legitimate difficulties in quantifying benefits while preserving a high degree of rigor in justifying projects. Second, it needs to ensure that when costbenefit analysis is done it is done with quality, rigor, and objectivity, as poor data and analysis
misinform, and do not improve results. Reforms are required to project appraisal procedures to
ensure objectivity, improve both the analysis and the use of evidence at appraisal, and ensure
effective use of cost-benefit analysis in decision-making.
Current Bank policy states that cost-benefit
analysis should be done for all projects at appraisal——with the single exception of projects for
which benefits cannot be measured in monetary
terms, in which case a cost-effectiveness analysis
should be performed. Ultimately, the requirement to conduct cost-benefit analysis stems from
the mandate in the Articles of Agreement that
the Bank should strive to increase the standard
of living in member countries. When countries
borrow——and have to repay——funds for projects
in which costs exceed benefits, the standard of
living of the country declines.
Using the presence of an ex-ante economic rate of
return estimate as an indicator of whether costbenefit analysis was performed, the percentage of
projects with such analysis dropped from 70 percent to 25 percent between 1970 and 2008. Further examination of project documents reveals this
to be a reliable indicator of the presence of costbenefit analysis. A little over half of this decline
vii
EVALUATION SUMMARY
was due to an increase in projects in sectors at the
Bank that tend not to apply a cost-benefit analysis
to their projects. About half of the sectors, which
have tended to decline as a share of activity in recent years, often apply cost-benefit screening to
their projects, while the other half, the growing
half, rarely do. In addition to this shift away from
sectors that apply cost-benefit analysis, there has
been a general decline in all sectors in the application of such analysis. Most of the improvement in
project performance ratings that has occurred at
the Bank in the past twenty years is in the five
sectors that tend to apply cost-benefit analysis.
World Bank policy notwithstanding, many appraisal documents for new projects in recent years
do not present cost-benefit analysis. How is this
omission explained? How are the projects justified? Of the 93 investment projects that closed in
2008 without reporting cost-benefit information
(either at appraisal or at closing), 60 provided no
explanation or asserted that efficiency considerations were not applicable. Eighteen cited inadequate data. Nineteen projects provided some relevant information, but the information tended to
be positive anecdotes, with no attempt to address
potential selection bias. Twenty-four project documents invoked cost-effectiveness as the standard
they were to be judged by, but of these, none actually applied cost-effectiveness analysis, which
entails a comparison between specific alternatives
on the basis of costs. One project claimed such
an analysis had been done but did not show the
results in the document.
Of projects that do provide cost-benefit analysis,
there are several examples of excellent analysis,
but often a lack of transparency. The most important data, the quantitative cost and benefit flows,
are rarely provided in a straightforward manner,
such as a simple table. Such a table could be provided along with a discussion of the main assumptions or empirical evidence that lies behind the
numbers. As was pointed out in a World Bank
report 20 years ago, ex-ante project analysis at the
Bank is usually based on the working assumption
that everything will go as planned. This imparts
an upward bias to the cost-benefit estimates because there are frequently disruptions along the
way. An alternative——more in line with Bank policy to present the expected economic return——
would be to employ the working assumption that
viii
new projects would achieve the average results
measured in previous similar projects, unless
changes are made to the project design that warrant revision.
The weak points in economic analysis of projects
are fundamental issues such as the public sector
rationale, comparison against alternatives, and
measurement of benefits against a without-project
counterfactual. Project justification rarely includes
a discussion of whether the project is producing a
public good, and if alternatives are considered,
they tend to be minor alternatives, such as alternative funding mechanisms, rather than truly alternative projects. Counterfactual analysis tends to be
good for projects in sectors in which this analysis
is hardwired into standard spreadsheets, such as
transport. Impact evaluations, which are designed
to address the counterfactual issue and thus are a
natural complement to cost-benefit analysis, have
rarely been used in the past, though their use is
now growing in some sectors. There is low usage
in cost-benefit analysis of shadow prices and other
technical adjustments to capture some of the social benefits and costs.
Projects that have easy-to-identify beneficiaries,
such as agriculture and community-based development projects (albeit ex-post), could provide
better poverty analysis. This often requires a special baseline household survey. Lack of baseline
data is a key weakness undermining ex-post costbenefit analysis in many projects. Overall, the
economic analysis in appraisal documents in 200708 is found to be acceptable or good in 54 percent
of the cases. This compares with 70 percent
found by a similar rating exercise in the 1990s.
This report also examines whether there is evidence of bias in the economic rates of return that
are reported. It finds that the ““everything goes
according to plan”” scenario is still the working
assumption underlying cost-benefit analysis at appraisal. The report also finds that the likelihood
that the economic rate of return is recalculated at
the close of projects is lower for projects with low
outcome ratings. Moreover, interviews with staff
indicate that project cost-benefit analysis is conducted after the decision to go ahead with the
projects, which puts the analysis under considerable pressure to reach conclusions consistent with
the decisions already taken.
EVALUATION SUMMARY
The lack of attention to cost-benefit information
is surprising given the positive story that emerges
on trends in the reported rates of return in the
declining subset of projects that apply this approach: reported economic rates of return have
doubled in 20 years, from a median of 12 percent
in the late 1980s to 24 percent in 2008. If reflective of the larger group of projects, this could signal a large rise in the effectiveness of these development projects.
Some discount this rise, believing that it indicates
nothing more than an increase since 1987 in the
upward bias in the measurement of economic returns. The available evidence does not confirm
this belief, but it cannot be dismissed because the
evidence is thin.
Another possible explanation for this large rise in
returns is growth-oriented reforms. Reforms -comprising both a retreat of anti-market approaches to projects and improvements in investments and institutional support in the economic environment -- could account for some of
the rise in economic returns for this subset of
projects. A review of project documents from the
pre-reform 1970s and 1980s suggests that project
execution was frequently frustrated by high transactions costs or unavailability of imported spare
parts and hampered by unresponsive state entities.
Examination of 47 countries where the available
data permit the impact of such factors to be tested
reveals that 43 had higher economic returns in
projects after reforms.
in Bank client countries, and project returns correlate with growth rates. But much of the growth
improvement occurred rather late in the 19872008 period, and thus is not sufficient to account
for the sustained rise in returns during the entire
period.
A review of economic analysis at the World Bank
20 years ago found many of the same shortcomings documented here in economic analysis in the
Bank. Yet that report’’s recommendations did not
go far enough in confronting underlying causes: a
decision-making process that often makes decisions before adequate evidence is provided, and
few institutional checks to counteract the influence of advocacy for projects that undermines
rigor in project appraisal, including cost-benefit
analysis.
The Bank needs reforms to ensure objectivity and
address conflicts of interest in ex-ante project
analysis. It needs to use cost-benefit analysis evidence to improve decisions in a context where
decisions are increasingly driven by borrowing
countries.
The policy for cost-benefit analysis needs to be
defined in a way that recognizes legitimate difficulties in quantifying benefits in some types of
projects while preserving a high degree of rigor in
justifying projects. This report closes with suggestions on how the Bank can address these institutional issues
External factors could also be responsible. Economic conditions facing countries have improved
ix
Evaluation Essentials
1. World Bank Policy
1.1
This evaluation is grounded in World Bank policy on costbenefit analysis, which in turn is grounded in the Articles of Agreement that established the International Bank for Reconstruction and
Development in 1944. The Articles state that: ““The purposes of the
Bank are: (i) To assist in the reconstruction and development of territories of members by facilitating the investment of capital for productive purposes……(iii)……thereby assisting in raising productivity, the
standard of living and conditions of labor in their territories.””1
1.2
The Articles state that the fundamental goal of World Bank
operations is to raise productivity, incomes, or welfare (““standard of
living””) and wages, employment, or working conditions (““conditions
of labor””) in the territories of member countries. Cost-benefit analysis
is the technique the Bank has used, since the early 1970s, to gauge for
itself, to verify for stakeholders outside the Bank and the Bank’’s
Board of Directors, that its operations are indeed having a net positive
effect on the standard of living of member countries. Cost-benefit
analysis is defined simply as any quantitative analysis to establish for
a given project whether the present value of benefits exceeds the
present value of costs. Such analysis usually also produces both a net
present value (NPV) calculation and an economic rate of return (ERR)
calculation.
™ The Articles of Agreement that
founded the World Bank
established the principle that
Bank projects should aim to
improve welfare in member
countries
™ Cost-benefit analysis is the
way the Bank has traditionally
sought to determine whether
its projects improve welfare.
Operations with a positive netpresent-value increase
welfare because discounted
benefits exceed costs
™ Bank policy requires a netpresent value calculation for
all projects except those for
which benefits cannot be
measured in monetary terms;
it also lists the elements of
good analysis
1.3
Improving the welfare of poorer countries is the fundamental
objective behind Bank policy on cost-benefit analysis. The policy directs the Bank to help borrowing counties select the highest net
present value project and to do nothing if the best alternative entails a
negative net present value. When counties borrow at commercial interest rates, and the (properly measured) net present value is negative, it is important to recognize that the country as a whole is thereby
becoming poorer as a result of the project. That is why the policy
against negative net present value projects is particularly important.
In the case of International Development Association (IDA) credits,
which contain a large grant element, a negative net present value does
not necessarily mean the receiving country is poorer, but nevertheless
wastes taxpayer funds from donor counties, because better alternatives are not pursued. Whenever countries fail to pursue the alternative with the highest net present value, global resources are wasted.
1
CHAPTER 1
WORLD BANK POLICY
1.4
Bank policy also serves as a safeguard against project choice
being captured by narrow political or sectional interests. Efficiency
considerations compete with other motives in project selection and the
policy is designed to give efficiency the upper hand in this competition.
The Bank’’s role as an honest broker is often appreciated by borrowers.2
1.5
The current version of Bank policy is Operational Policy 10.04,
entitled ““Economic Evaluation of Investment Operations,”” written in
September 1994. This statement of policy is reproduced in Annex D.
Three components of the policy deserve special comment. The first is
the decision criteria to be applied in approving investment operations. Bank policy is to choose the investment that maximizes the net
present value (NPV) of benefits from a list of alternatives and to do
nothing (that is, do not invest) if the NPV is negative. The clear line
against investments with negative net present values is rooted in the
desire to avoid lowering national welfare in member countries.
1.6
The second component of Bank policy establishes the scope of
cost-benefit analysis. The positive NPV test should apply to all Bank
investment operations, with a single exception: ““If the project is expected to generate benefits that cannot be measured in monetary
terms, the analysis (a) clearly defines and justifies the project objectives, reviewing broader sector or economy-wide programs to ensure
that the objectives have been appropriately chosen, and (b) shows that
the project represents the least-cost way of attaining the stated objectives.”” Hence the positive NPV criterion should apply to all operations except those in which benefits cannot be measured in monetary
terms, in which case the operation must be established as the leastcost way of attaining the stated objective. Thus, redistributive programs are not at odds with Bank policy as long as such redistribution
is accomplished at lowest cost.
1.7
The third component of Bank policy is guidance on what constitutes good cost-benefit analysis. The Bank stipulates what must be
analyzed to establish that an operation will achieve or has achieved a
positive net present value. These criteria will be covered in the section of this report that evaluates the quality of analysis in Bank
projects, but may be summarized here. The criteria stipulate that:
a. The main goal of the ex-ante project analysis is to estimate
the discounted expected present value of its benefits, net of
costs, since this is the basic criterion for a project’’s acceptability. This places a premium on accurate and unbiased estimates.
b. Benefits and costs should be measured against the situation without the project.
2
CHAPTER 1
WORLD BANK POLICY
c. All projects should be compared against alternatives, including the alternative of doing nothing.
d. Analysis should consider the sources, magnitude, and effects of the risks associated with a project by taking into
account the possible range in the values of the basic variables and assessing the robustness of the project outcomes with respect to changes in these values.
e. The economic analysis should examine the consistency
with the Bank’’s poverty reduction strategy.
f. The economic evaluation of Bank-financed projects should
take into account any domestic or cross-border externalities.
1.8
Basic principles of transparency and accuracy in the assumptions and estimates are also part of Bank policy. The analysis should
be clear, accurate, and sufficiently complete to support informed decisions.
1.9
The points mentioned in this chapter constitute the basic evaluative principles against which this evaluation will be conducted.
3
Evaluation Essentials
™ Cost-benefit analysis,
specifically the calculation of
ERRs, has declined at both
appraisal and project closure
2. The Decline in Cost-Benefit
Practice
™ There is a strong difference in
practice between sectors
2.1
World Bank policy (OP/BP 10.04) is stated in a manner that
offers little scope for exemption from a net-benefit test: ““For every investment project, Bank staff conduct economic analysis to determine
whether the project creates more net benefits to the economy than
other mutually exclusive options…….””3 In light of this policy, it is potentially significant that the use of cost-benefit analysis appears to be
declining, at least as indicated by the percentage of investment operations that contain an estimate of the economic rate of return (ERR) in
the initial project document.4 This percentage declined from a high of
over 70 percent during the early 1970s to approximately 30 percent in
the early 2000s (Figure 1).5
™ A shift in composition toward
sectors not using cost-benefit
analysis accounts for 23
percentage points of the 37
percentage point decline
™ But the decline is also evident
even in the sectors that do
traditionally use cost-benefit
analysis, which account for 15
percentage points of the
decline
Figure 1. Percentage of Bank Investment Projects with Estimates of the
Economic Rate of Return in the Appraisal Document, by Year of Approval
Percent with ERRs at Appraisal
80
50
20
1969
1980
1990
Year of Approv al of Projects
2001
Source: World Bank data.
2.2
The data in Figure 1 run only through 2001. This is because
information from the beginning of projects (such as the initial ERR)
gets recorded formally only once projects terminate, and this happens
on average seven years after commencement. But estimated ERRs are
also calculated upon project closing6 and are found in the final completed project report.7
5
CHAPTER 2
THE DECLINE IN COST-BENEFIT PRACTICE
2.3
Evidence of a decline in cost-benefit practice is also evident in
the percent of projects reporting ERRs in the final project report. Figure 2 reports the proportion of final project documents that contain an
estimated economic rate of return after the fact, displayed by the year
of project termination. The results in Figure 2 show similar evidence
of a long-term decline in calculation of economic rates of return.
There has been a slight increase in the proportion of projects with final ERRs in recent years.8
Figure 2. Percentage of Bank Investment Projects with Estimates of the
Economic Rate of Return in the Final Completion Report, by Year of
Completion
Percent with ERRs at Completion
80
50
20
1975
1980
1990
Year of Closing of Projects
2000
2008
Source: World Bank data.
2.4
The ERR coverage data in Figure 1 and Figure 2 provide convenient summaries of broad trends but are less––than-perfect indicators of some aspects of cost-benefit analysis. Strictly speaking, Bank
policy mandates calculations of net present values, not ERRs, and
Bank policy permits cost-effectiveness calculations in special circumstances, so ERR coverage is an imperfect indicator of adherence to
World Bank policy.9 Later in this report it will be argued that neither
of these issues is empirically significant. In particular, projects with a
cost-benefit analysis and an NPV calculation almost always report an
ERR. Second, the mere presence of an ERR calculation says nothing
about the quality of the analysis behind it or the accuracy of the data.
Both of these issues will be examined later in this report. And finally,
there is the coverage issue. Rarely do the reported ERRs cover 100
percent of expenditures of projects; the ERR coverage data are silent
on what proportion of a project’’s expenditures were covered by a
cost-benefit assessment. Nonetheless, having mentioned these limita-
6
CHAPTER 2
THE DECLINE IN COST-BENEFIT PRACTICE
tions, there remain important points to be learned from further examination of the coverage data.
2.5
To what degree does the decline stem from a change in the
composition of the projects that are being financed, or other factors?
The practice of calculating ERRs varies sharply by sectors. Table 1
shows data for major sectors at the Bank (Annex A provides the full
17 sectors). The last column on the right of the table shows a sharp
divide: 5 of the 11 sectors tend to produce the vast majority of the
ERR estimates, while another 6 hardly produce any. The 5 sectors
that tend to calculate ERRs are Agriculture and Rural Development,
Energy and Mining, Transport, Urban Development, and Water. The
six sectors that do not tend to calculate ERRs are Education; Environment; Finance and Private Sector Development; Health, Nutrition,
and Population; Public Sector Governance; and Social Protection. For
convenience these will be called high-CBA sectors and low-CBA sectors or simply H-sectors and L-sectors.
2.6
Table 1 shows the long-term shift in World Bank activity away
from the five sectors that tend to apply cost-benefit analysis. Comparing the two five-year periods, 1975-79 with 2003-07, the proportion of
project closings in such sectors covered 74 percent of all operations in
1975-1979 but only 46 percent in 2003-07.
Table 1. Trends in Sector Composition and Proportion of Projects, by Sector,
Reporting Economic Rates of Returns
Agriculture and Rural Development
Education
Energy and Mining
Environment
Financial and Private Sector
Development
Health, Nutrition, and Population
Public Sector Governance
Social Protection
Transport
Urban Development
Water
Other
TOTAL
Number of
operations
1975-79
165
50
96
0
60
Number of
operations
2003-07
190
133
83
82
97
Percent of
operations
1975-79
27%
8%
16%
0%
10%
Percent of
operations
2003-07
16%
11%
7%
7%
8%
4
7
0
162
9
22
35
610
109
78
80
121
73
71
45
1162
1%
1%
0%
27%
2%
4%
6%
100%
9%
7%
7%
10%
6%
6%
4%
100%
Percent
reporting
ERR at
entry
1970-2008
48%
1%
43%
8%
4%
1%
1%
3%
58%
32%
37%
13%
Notes: The “Other” category includes all sectors with less than 5 percent of operations in 2003-07.
"Financial Sector" operations are included in the category "Financial and Private Sector Development."
7
CHAPTER 2
THE DECLINE IN COST-BENEFIT PRACTICE
2.7
Nevertheless, despite the clear shift away from high-CBA sectors, the decline in ERR practice is not solely attributable to this shift.
Figure 3 shows that even in the high-CBA sectors the percentage of
projects with ERR calculations at entry has declined from approximately 90 percent in the early 1970s to just over 50 percent in the 2000s.
Figure 3. Percentage of Investment Projects in the Five High-CBA Sectors with
ERRs in the Appraisal Report, by Year of Project Completion
Percent with ERRs at Appraisal
100
90
80
70
60
50
1974
Year of Closi ng of Projects
2008
Source: World Bank data.
2.8
The decline in the percentage of projects reporting ERRs is
thus a result of two broad trends: a shift away from the high-CBA sectors and a decline in CBA even within such sectors. Table 2 shows the
figures when World Bank projects are grouped into the two broad
sectors: high-CBA and low-CBA. As the table shows, the percentage
of projects in the high-CBA sectors declined from 86 to 44 percent between 1975 and 2007 (specifically, 86 percent of projects that closed in
1975 had reported ERRs at entry). Over the same period the percentage of projects reporting ERRs declined even within the high-CBA
sectors from 79 percent to 61 percent.
8
CHAPTER 2
THE DECLINE IN COST-BENEFIT PRACTICE
Table 2. The Shift from High-CBA Sectors to Low-CBA Sectors at the World
Bank
High-CBA
sectors
Low-CBA
sectors
All
sectors
Percent in
High-CBA
sectors
9
109
65
194
86%
44%
Number of project closings:
1975
2007
56
85
Number reporting rates of return at entry (by year of project closing):
1975
44
0
44
2007
52
7
59
Percent reporting rates of return:
1975
79%
0%
68%
2007
61%
6%
30%
Source: World Bank data.
2.9
To calculate precisely the part of the overall decline due to declines within sectors versus shifts between sectors, Table 3 shows the
results of a mathematical decomposition. The full decline in the percentage of projects reporting ERRs at entry between 1975 and 2007
was 37 percentage points. Of this, 15 percentage points was due to
the decline in calculation of ERRs within the H-sectors, and 23 percentage points was due to a decline in the percentage of World Bank operations in the H-sectors. The very small rise in the percentage of Lsector operations that produced ERRs (from 0 to 6 percent) slightly
offset the contribution of these terms (contributing a positive 1 percentage point).10
Table 3. Sources of the Decline in ERR Reporting
Total change in ERR reporting (percentage points)
From change within High-CBA Sectors
From change within Low-CBA Sectors
From shift away from projects in High-CBA Sectors
-37
-15
1
-23
Source: World Bank data.
2.10
The evidence just shown confirms that there was a major
change in composition of World Bank projects away from high-CBA
sectors. Further evidence shows that this shift began to show up in
approvals after 1988, and reached a low in 2001 before staging a modest recovery. Figure 4 shows the evolution over the years in the fraction of projects in the high-CBA sectors.
9
CHAPTER 2
THE DECLINE IN COST-BENEFIT PRACTICE
Figure 4. The Shift toward Low-CBA Sectors Started in 1988
Percent of Projects
100
80
60
40
1970
1980
1988
Year of Approval
2000
2010
Source: World Bank data.
2.11
In sum, the prevalence of cost-benefit analysis, as indicated by
percentages of projects that contain ERR estimates, has declined over
the years. Of a total decline of 37 percentage points in the share of
projects with ERRs, 15 percentage points came from a decline in ERR
calculation within high-CBA sectors (that habitually report ERRs) and
23 percentage points came from a shift in operations from high-CBA
to low-CBA sectors. The shift away from high-CBA sectors started
after 1988 and reached a low in 2001. Since 2001 the percentage of operations in high-CBA sectors has risen slightly.
10
Evaluation Essentials
™ Of 166 investment projects that
closed in 2008, 93 had no
ERRs reported
3. The Scope for Cost-Benefit
Analysis
™ A common statement is that in
some way calculating an ERR
is not applicable
3.1
World Bank policy and World Bank practice regarding costbenefit analysis have been diverging for many years. World Bank
policy mandates a net present value calculation for all investment
projects and a cost-effectiveness analysis when the project is ““expected to generate benefits that cannot be measured in monetary
terms.”” World Bank practice increasingly lacks reporting of economic
rates of return at either the beginning of projects or at the end of
projects. What fraction of the non-reporting projects present costeffectiveness analysis?11 What fraction present cost-benefit information in a form other than an ERR? Of those projects with neither kind
of analysis, what reasons are offered for lack of analysis?
™ Many of reasons given for the
lack of ERRs are basic
problems with projects rather
than with cost-benefit
methodology
™ Non-quantifiable benefits is the
most legitimate but also the
most easily-abused reason for
not presenting cost-benefit
analysis
™ Of the 24 projects that claimed
justification through costeffectiveness analysis, one
appeared to use the concept
correctly
3.2
Of the 195 projects that closed in fiscal year 2008 and for which
records are available, 29 were development policy operations (which
rarely perform cost-benefit assessments) and 166 were investment operations. Of the 166 investment operations, 50 reported economic
rates of return at both the start and the close of the project, 93 reported neither, and 23 reported one or the other alone (Table 4).
Table 4. Numbers of Projects Reporting Economic Rates of Return at the
Beginning or at the Close of Projects, for Fiscal Year 2008
ERR AT
BEGINNING?
ERR at closing?
YES
NO
TOTAL
YES
50
9
59
NO
14
93
107
TOTAL
64
102
166
3.3
The focus of this chapter will be on the projects with no ERR
reporting. These projects are broken down by sector in the third column of Table 5. As can be seen, there are eight or more such projects
in Agriculture and Rural Development; Education; Environment;
Health, Nutrition, and Population; and Public Sector Governance.
11
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
Table 5. Reporting of Rates of Return, Fiscal Year 2008, by Sector
ERR reporting
Agriculture and Rural Development
Economic Policy
Education
Energy and Mining
Environment
Financial and Private Sector Development
Health, Nutrition and Population
Poverty Reduction
Public Sector Governance
Social Development
Social Protection
Transport
Urban Development
Water
Total
Percent
Total
projects
None
Complete
Start
only
End
only
30
2
14
14
11
10
19
0
10
2
7
27
7
13
166
100%
9
2
11
3
11
6
17
0
9
2
6
5
5
7
93
56%
13
0
0
7
0
0
2
0
1
0
0
21
1
5
50
30%
4
0
2
0
0
2
0
0
0
0
1
0
0
0
9
5%
4
0
1
4
0
2
0
0
0
0
0
1
1
1
14
8%
3.4
Why do such projects not present cost-benefit information? The
usual reason given is that cost-benefit analysis is not applicable for certain kinds of projects. This raises two issues. First, what lies behind
non-applicability; what does it mean exactly? What are the criteria by
which it is judged? What kinds of projects qualify? Second, if costbenefit analysis is not done, how can one know whether benefits exceed costs for such projects? Can alternative justifications be provided?
3.5
This chapter focuses on the first question: what is meant in
practice by non-applicable, and are the problems identified limitations
with cost-benefit analysis or rather limitations with the analysis or the
data collected by projects? As a first step, the chapter reviews what
project implementation completion reports (ICRs) say about the issue.
3.6
The ICRs contain a section on efficiency that is supposed to
summarize the cost-benefit estimates. In the projects with no ERR,
what is written in this section helps identify some of the issues. In several reports, the non-applicable thesis is asserted with little explanation:
x
x
12
““It is not possible to calculate quantitative measures of efficiency for the sub-projects””
““An economic and financial analysis was not undertaken given the institutional nature of the outputs””
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
x
x
““[The project’’s annual outputs were] not countable because it
was a social welfare-related adult non-formal education
project for poor people””
““The [project] did not include the implementation of civil works.
Therefore, conventional quantitative economic analysis which is
normally carried out for investment projects does not apply.””
3.7
For other projects it is simply stated that there was no analysis:
““An ERR was not carried out for the project;”” or ““No economic analysis was undertaken during project preparation.””
3.8
Still other project documents identify lack of prior data collection or analysis as the reason for no cost-benefit analysis:
x
x
x
““No economic analysis of the project or any of its components
or activities was attempted during appraisal and no economic
analysis was done as part of the [implementation completion
report] since there was not an adequate baseline available””
““Economic and financial analyses were not provided at appraisal and data were insufficient to carry out such analysis at
completion.””
““Appraisal documents did not include a cost-benefit or a costefficiency analysis. Consequently, it is not possible to assess if
project outcomes were produced in accordance with efficiency
benchmarks set at appraisal.””
3.9
One project document cites lack of data but asserts that the
data would not be meaningful even if collected due to diversity and
local conditions: ““As in most of the social development projects, it is
difficult to provide an overall project economic and financial analysis
because of the difficulty of quantifying the benefits in the absence of
the necessary data. Even if the data regarding the unit costs were
available, the relevance of comparison is not that meaningful due to
the diversity of micro-project types and the local conditions where
they are implemented.””
3.10
A few project documents refer to lack of time and other competing priorities: ““In the end, individual rates of return for projects
were not calculated, partly because of the competing demands that
(the project) faced.””
3.11
A few projects elaborate the reasons for no CBA by claiming
that the benefits are not quantifiable. The unquantifiable thesis is asserted most often for four kinds of projects: education, health, technical assistance, and environment: ““A formal cost/benefit analysis
was not done at appraisal because the project financed only technical
assistance activities.”” Furthermore: ““Cost-benefit analysis is not applicable to the education and health investments since benefits are not
13
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
fully quantifiable.”” This quote opens an important issue: by using the
qualifier ““fully”” to modify quantifiable the project document is hinting that benefits should not be quantified unless it can be done with
high accuracy, but what is the standard, ““fully”” meaning 100 percent
accurate, or something less?
3.12
Still other project documents cite the fact that the project activities are not known in advance: ““Given the demand-driven nature of
the [project] and the difficulty to predict in advance the types of loans
that would be financed, the PAD [project appraisal document] did not
attempt to predict an ……ERR……for the overall project.”” Other project
documents cite the fact that the project is emergency assistance that
requires quick disbursement: ““The project was an emergency response operation and no economic or financial analysis was undertaken during appraisal.””
3.13
Based on this review, it is possible to enumerate the major reasons cited in the documents for lack of cost-benefit analysis (Table 6).
Many projects simply leave the efficiency section blank or assert ““notapplicable”” with no further explanation. Of those that provide reasons, the reasons include unquantifiable benefits, lack of adequate data collected at inception or during implementation, inadequate analysis in the project appraisal document, or too little time.
Table 6. Reasons Offered in Project Completion Documents for Lack of CostBenefit Estimates, Projects that Closed in Fiscal Year 2008
Investment projects with no CBA
No information given or "not applicable" asserted
Number mentioning inadequate data
Number providing relevant information but no final CBA
Emergency programs
Analysis promised in PAD not done
Revolving funds
Number blaming lack of analysis in PAD
Number invoking cost effectiveness
Number performing cost effectiveness
93
60
18
19
3
4
3
3
24
1
Source: Author’s estimates using World Bank project documents.
Note: Some project documents cite more than one reason.
3.14
Of the projects without cost-benefit information, 60 asserted in
some manner that the efficiency issue was not applicable to the
project, either by leaving everything blank or explicitly writing ““not
applicable.”” At least 20 claim that the benefits of the project were not
quantifiable (the language is hard to classify in some cases).
3.15
There is a grey area in that some projects, even those claiming
that the benefits were not quantifiable, nevertheless provide information
14
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
relevant to determining whether benefits exceeded costs. Of the 19
projects that do so, however, most of the information portrays the project
in a positive light, with no effort to demonstrate that the results cited are
representative. A few projects, however, offer extensive information.
And some of these appear to have sufficient information to estimate an
economic rate of return but do not present the calculation.
3.16
A significant fraction of the project documents are apologetic
about the lack of information on efficiency. Nineteen project documents cite lack of prior data collection as the key problem; four mention that studies or analyses promised at appraisal were not carried
out; three mention inadequate analysis in appraisal documents; three
projects deal with emergency situations.
Box 1. The Costs and Benefits of Cost-Benefit Analysis
Consider the costs and benefits of spending time on analyzing costs and benefits. The
costs include the analyst’’s time and the costs of collecting data——the survey reported
later in this report indicates the median cost is currently $16,000. The benefits are a
higher chance of avoiding waste of money and selecting a better project. Without undertaking an exacting study, a few general points may be made:
ƒ
ƒ
ƒ
ƒ
ƒ
Costs tend to rise at a constant rate as more time is spent on the analysis (except for the one-time costs of data collection).
The benefits to additional time spent on the cost-benefit analysis tend to be
very high at first and to decline once the basic issues have become clear. This
is in part because one of the main benefits of CBA——forcing people to be concrete about the benefits they expect and to compare these with costs, are
achieved within the first days of work.
The benefits of performing a CBA and avoiding mistakes or selecting a better
project can potentially reach 10 percent of project costs. This means, with
loans on the order of $10-100 million, potential benefits can be on the order of
$1-10 million.
The limiting factor to doing a more accurate and thus better cost-benefit analysis will eventually become more accurate data. If one is not prepared to incur
additional costs of data collection, there will be a point beyond which additional time will not carry significant benefits.
The benefit of cost-benefit analysis is low if the conclusions reached will not
affect funding decisions.
Putting these points together:
ƒ
ƒ
It is almost certainly valuable to do some cost-benefit analysis. Neither extreme of doing no CBA or spending many months on the CBA is likely to be
the right answer.
Beyond a certain point, the CBA cannot be made much better without doing
something to overcome the data constraint.
If the Bank places little weight on cost-benefit results for decisions, staff will have little
incentive to invest effort in it. Mandating that staff conduct cost-benefit analysis while
giving it little weight in decisions imposes costs with few benefits.
15
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
3.17
Cost-effectiveness analysis is the major form of analysis most
often mentioned as an alternative to cost-benefit analysis. Twentyfour projects invoke cost-effectiveness as the method of justification,
either by checking the cost-effectiveness tab in the project appraisal
document or by asserting in the completion report that efficiency considerations would be addressed by cost-effectiveness analysis. Interestingly, of the 24, only 1 offers what appear to be a real costeffectiveness analysis.
3.18
A classic cost-effectiveness analysis starts by stating a specific
meaningful goal, such as reducing the incidence of a disease in a town
by 50 percent in four years, and then presents data on the expected
cost of two or more methods of achieving this goal, and finally selects
the least-cost alternative. Instead of this analysis, the 24 projects that
invoke cost-effectiveness do not mention a specific alternative project
to the project chosen. Second, project documents usually examine the
costs of doing the project rather than achievement of a meaningful
goal such as disease reduction. Several projects assert costeffectiveness on the grounds that some components of the project
spent less money than anticipated at appraisal. One project is considered cost-effective because costs are lower than the previous version
of the project. Other project documents assert cost-effectiveness on
the grounds that procurement was competitive. Still others asserted
cost effectiveness on the grounds that costs were kept ““within international norms”” or ““comparable to costs of other similar projects in the
region.”” One project claims that costs were low compared to the potential growth of fisheries in the region, implicitly invoking a costbenefit standard in an analysis that was claimed to be cost effectiveness. None of the projects present what is normally understood to be
a cost effectiveness analysis.
3.19
Of all the reasons offered above, unquantifiable benefits are arguably at once the most legitimate and most easily abused reason for
not applying cost-benefit analysis. The fact that current practice allows project managers broad scope to claim unquantifiable benefits
raises the risk that projects are being funded with negative net present
values. Some project documents treat this analysis as an all-or-nothing
proposition——benefits are either quantifiable or not——whereas often it is
a matter of degree. Assigning monetary value to benefits always entails some measurement error. What’’s missing is policy on which benefit streams entail sufficient uncertainty that a cost effectiveness analysis
is warranted. In addition, a calculation can always be provided giving
how large un-quantified benefits would have to be to justify the costs of
the project. The present review found no projects providing this information. If this calculation were standardized (for example nonquantified benefits per beneficiary) reasonable standards and case law
could be developed.
16
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
3.20
Also valuable would be outside information that establishes a
plausible case for the project short of quantification. Some projects do
this. An example is a recent completion report for an education loan
in Georgia which, while not presenting data to quantify benefits in
monetary terms, nevertheless provides some data to help the reader
judge the plausibility that the benefits exceeded the $26 million investment. Another example is a food and drug regulation project in
India.
3.21
A second underlying issue is the treatment of project components which, when considered in isolation or in the short run, may
have a low net present value. These take one of several forms. Some
components are necessary complements to the main intervention.
Administrative expenditures and monitoring and evaluation would
fall under this heading, as might capacity building and institutional
strengthening. If a component is necessary for the achievement of the
benefit flows it should be included as a cost in the cost-benefit calculation, but the practice varies widely across projects; often these are
treated as stand-alone components.
3.22
The second kind of intervention that falls under this category
includes interventions that are necessary but not sufficient to achieve
results and interventions that are really a gamble: they may achieve
large benefits but with a small probability. An example of the former
might be improving record keeping in a land registry: by itself it
probably will not increase productive rural investment, but it may if
several other factors are in place. Strictly speaking it is incorrect to
claim broadly that a cost-benefit analysis is not applicable for such
components. The right calculation, if the intervention will not yield
results over a specific horizon, is that the net benefit is negative for
that time horizon. Probabilities can be assigned to uncertain outcomes. Project managers may believe that the net benefit will be positive over a long time horizon, but the point is that that could be argued with the available evidence rather than asserting nonapplicability of efficiency considerations.
3.23
The fact that benefits may occur over long time horizons is not
in itself a reason that cost-benefit is inapplicable. Cost-benefit assessments routinely deal with benefit flows that occur over many
years. Literacy programs for children are an example. The incomeearning years for a child can start decades after primary schooling,
but since both plausible reasoning and solid evidence establishes that
benefits will materialize, there is no problem in applying a costbenefit calculation with a long time horizon.
3.24
Lack of data emerges as a major reason for lack of cost-benefit
estimates both at the start and end of projects. This is in part due to
17
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
the shortage of rigorous quantitative evaluations of closed projects
(which could be providing a wealth of information to guide CBA for
new projects), in part due to neglect to collect data during project implementation, and in part due to neglect to record and use the data
that is available from previous projects. The simplest area in which
individual projects fall short is failure to collect baseline data. Previous IEG evaluations (the 2009 Annual Review on Development Effectiveness and the evaluation on Health, Nutrition, and Population
in 2009) have documented the low level of baseline data collection in
World Bank projects.12
3.25
In other cases the country or the sector specialists could organize data collection in an aggregate form rather than for each individual project. It is efficient to organize supplements to the national
household income survey rather than conduct ad-hoc surveys project
by project. Similarly, background research on parameters such as expected yields or the average relation between GDP growth and traffic
growth require basic research or national or international data collection efforts. The Development Economics Research Group at the
Bank could be employed to organize this kind of background research
relevant to groups of projects.
3.26
Not knowing the activities to be financed poses another important obstacle to cost-benefit analysis. Budget support programs by
definition do not disclose the ultimate destination of the funds, nor do
many kinds of community-based development projects. The latter
commit to a process whereby the use of the money will be decided
later, by recipients. This poses a problem for any kind of ex-ante
analysis, not only cost-benefit assessment.
3.27
Emergency assistance projects are a further class of projects
that tend not to have cost-benefit assessments. The reason, presumably, is that the cost-benefit effort is believed to take too long. At issue
here is the time and cost required to perform a reasonable cost-benefit
assessment (Box 2). The learning curve in performing cost-benefit assessments is usually quite steep: high increments to learning early on
and lower increments as the analysis becomes more detailed. So it is
not obvious that a rapid-assessment cost-benefit analysis could not be
used in such cases.
3.28
The Bank is currently undertaking a major reform in its procedures for appraising projects (Investment Lending Reform) in which
the aim is to simplify and shorten approval requirements for low-risk
projects and devote more staff time to high-risk projects. A key question is what role cost-benefit analysis will have in the new procedures.
The 50-page document describing the new procedures, dated September 2009, does not mention cost-benefit analysis. The document states
18
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
that the main task in implementing a risk-based approach to lending is
to determine the ““risks to achieving development effectiveness as reflected in the results to be generated by Bank operations.””13 However,
positive results alone are not sufficient to justify projects: what counts is
the value of those results compared against the costs. And the document does not mention when this more relevant assessment will take
place. A risky project could be defined as one with a high probability
of achieving a negative net present value, and the determination of
which projects so qualify could be made on an objective empirical basis
by measuring and documenting the NPV of past projects. Instead it
appears that the determination of riskiness will depend on the subjective ratings of staff. In addition, the main remedy for dealing with
risky projects appears to be to devote more staff time and resources to
risky projects. A less costly alternative would be to require that such
projects pass pilot testing before being scaled up.
Box 2. Who Should Perform the Cost-Benefit Analysis?
Deciding who should perform the cost-benefit analysis for a new project
presents a dilemma in that the two desirable attributes, familiarity and objectivity, are unlikely to be found in the same group. Those who are most familiar with the project are those who are promoting the project and thus
have a potential conflict of interest when conducting cost-benefit analysis.
Some propose that borrowing countries should be carrying out the costbenefit analysis for their projects. Objectivity would still be critical, as
project promotion exists in both the client countries and the Bank. One variant of this proposal would be to entrust the cost-benefit analysis to an independent agency within the borrowing country, if such an agency exists
and functions effectively. Outside consultants hired by project promoters
are not necessarily the answer, as consultants may have an interest in securing the next consulting contract.
One question is whether entrusting the analysis to any single group is the
right way to ensure objectivity. The analogy with legal systems may be helpful, where the task of eliciting an objective verdict does not rest on the assumption that either the prosecution or the defense will be objective.
The accountability system in development agencies is typically built around
the assumption that those close to the project will provide objective information. There is oversight by independent groups in some agencies, such as the
World Bank, but it operates ex-post. Projects are assessed and rated after
they close. An independent voice to check the cost-benefit analysis and the
quality of monitoring and evaluation plans more generally before decisions
are made to proceed with the project could help strengthen accountability.
3.29
Based on this review of closed projects in 2008, the following
observations can be made about the scope for cost-benefit analysis.
First, the scope is greater than currently practiced if for no other reason
than the existence of projects that have ERRs at the beginning but not
19
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
the end or vice versa. If these gaps were filled the coverage of costbenefit analysis would rise by approximately 5 percentage points.
3.30
Second, data limitations emerge as a big reason for not conducting cost-benefit analysis. If these were addressed, the percentage
of projects with cost-benefit analysis at closing would rise substantially. And over time, accumulation of evidence would allow greater accuracy and coverage of costs and benefits at appraisal.
3.31
Current operational practice is usually that cost-benefit analysis is performed extensively or not at all. This all-or-nothing approach is worth examining. If the task were instead to present relevant information to help the reader assess whether benefits exceed
costs, several projects could present information short of a fullfledged analysis. The scope for cost-benefit analysis defined in this
way is substantially higher than current practice.
3.32
When emergency operations for war-torn areas or natural disaster relief entail reconstruction of structures such as bridges that were
previously heavily used, the case for positive net benefits is not difficult
to make. The Bank should pursue rapid cost-benefit assessments for
emergency operations, as a safeguard against expenditure decisions
being captured by narrow political interests in a crisis environment.
3.33
Projects such as community development operations, where
expenditures are committed before specific investments are identified,
pose a number of accountability issues. Cost-benefit analysis can be
done after the fact, at least for a representative sample of investments.
3.34
Any project that is implemented in specific communities or
regions of a country and not in others (as are many community investment projects) can conduct income and expenditure surveys before and after the project for affected regions and unaffected regions
and thus provide evidence for both poverty analysis and cost-benefit
analysis. A variety of options can be chosen for such surveys that
vary in terms of both analytical rigor and expense. The World Bank
has funded and promoted a number of these surveys at the national
level, but this evaluation has found few examples where such surveys
are extended and used for project-level cost-benefit analysis.
3.35
There appears to be scope for innovation in cost-benefit analysis, and this would probably raise the scope for CBA. Rapid appraisal
and greater use of household surveys have already been mentioned. A
further example could be credit programs that on-lend funds at known
interest rates to recipients who then make further investments. If repayment rates are close to 100 percent, as they often are, it can be inferred that recipients are likely investing the funds in activities for
which returns are at least as high as the interest rates charged by the
20
CHAPTER 3
THE SCOPE FOR COST-BENEFIT ANALYSIS
financial institutions. This observation could be used to develop lowerbound estimates of rates of returns on microcredit and other on-lending
operations. Further, there is probably greater scope for application of
cost-benefit analysis to social projects: Jimenez and Patrinos (2008) discuss the application of cost-benefit analysis to education projects and
Hammer (1997) discusses the application to health projects.
3.36
A general issue is the degree to which development assistance
has changed to render cost-benefit analysis more difficult or less applicable. Mentioned often in this connection is the increase in policy
reform and institutional reform projects, including budget support;
health and education projects; community-based development
projects; and greater country ownership in project choices.
3.37
Policy and institutional reform presents a mixed picture. It is
difficult to place a value on, as stated in a recent Bank project, ““strengthening the effectiveness of public resource management.”” But there
are well known methods of estimating the benefits of more efficient
pricing in the electricity sector or increased competition in the trucking sector.
3.38
Education and health also present a mixed picture. Raising
the educational attainment of children is a benefit that can be estimated and valued; curriculum reform is harder to value. Reducing
the incidence of disease can be valued; but improving administrative
records at health centers is more difficult to value.
3.39
As commented above, community-based development programs can be evaluated after the fact if not before the fact. The poverty reduction impact of such programs and a cost-benefit analysis of
such programs as poverty reduction tools can be assessed with
household income and expenditure surveys, particularly if the programs are rolled out in specific areas of the country and not in others.
3.40
Greater country ownership does not necessarily change the
need or scope for cost-benefit analysis, whether performed by the
Bank or the borrower. Borrowing countries are not necessarily less
interested in cost-benefit analysis than the Bank, given their direct interest in allocating funds efficiently. Even if this were the case, it
would not preclude the Bank from performing its own due diligence
for the borrower’’s information as well as for its own records, and for
the benefit of other clients contemplating similar investments.
21
Evaluation Essentials
4. Quality of Analysis
4.1
The criteria used to evaluate the quality of the analytical aspect of cost-benefit analysis are derived from Bank policy as outlined
in chapter 1. For ease of analysis, this policy is divided into six components: expected values, measure against counterfactual, alternatives, risk, poverty, and externalities.
a. Expected values: Since the expected net present value is
the main criterion to be applied in decisions, analysis in
appraisal documents should strive to estimate and present
the expected outcome rather than a best-case scenario.
b. Measure benefits and costs against counterfactual: Benefits and costs should be measured as the change compared
to what would be the case without the project.
c. Alternatives: All projects should be compared against alternatives, including the alternative of doing nothing.
d. Risk: Analysis should consider the sources, magnitude,
and effects of the risks associated with a project by taking
into account the possible range in the values of the basic
variables and assessing the robustness of the project outcomes with respect to changes in these values.
e. Poverty: The economic analysis should examine the consistency with the Bank’’s poverty reduction strategy.
f. Externalities: The economic evaluation of Bank-financed
projects should take into account any domestic or crossborder externalities.
™ The information in the
economic analysis section of
appraisal reports is usually of
high quality, but it is often
incomplete
™ The basic public sector
rationale for the projects and
the extent to which the project
provides a public good are
rarely discussed
™ ERRs at appraisal are rarely
the expected ERR as required
by Bank policy; rather, they are
rosy-scenario ERRs in which
downside risks are rarely
factored into the calculations
™ There is little evidence of a
systematic effort to compare
alternatives to the chosen
project
™ Poverty analysis could be
improved; the constraining
factor appears to be lack of
data collection
4.2
Many of the conclusions in this chapter draw on Belli and
Guerrero (2009), a separate review and rating of the economic analysis presented in staff appraisal reports for projects in calendar years
2007 and 2008 exceeding $100 million in commitments.14 The first
part of the chapter rates the analysis against the six priorities set out
in Bank policy. For an assessment of whether analytical quality has
improved since the 1990s, the second apart of the chapter compares
recent analysis with that of the 1990s, using criteria that replicate the
criteria used in previous ratings of 1994 and 1996/97.
4.3
The overall conclusion is that what is reported in the economic
analysis section of Bank project appraisal reports is frequently of high
quality, but the great weakness is in what is omitted. It is rare for
project analysis to cover all six items in the policy, and project justifi-
23
CHAPTER 4
QUALITY OF ANALYSIS
cations frequently omit crucial items such as comparison of alternatives, measurement of benefits against counterfactuals, and transparency about data and assumptions used. Compared to a similar review conducted in the mid-1990s, Belli and Guerrero found a lower
proportion of instances of poor analysis but also a lower proportion of
good and excellent analysis.
4.4
Expected values. In 1992, an important World Bank report assessing economic analysis in Bank project appraisal reports concluded
that ““no Staff Appraisal Reports (SARs) report truly expected economic
rates of return……, in the sense of their being the mean of the set of
possible outcomes. Downside risks are systematically ignored, and as
a result projected ERRs are biased upwards.””15 This report cited a
1990 report that had concluded that analysis was instead based on an
assumption of ““everything goes according to plan,”” dubbed EGAP
analysis.16
4.5
Seventeen years later, there is little evidence that this situation
has changed. Analysis of the economic sections of project completion
reports for calendar years 2007 and 2008 reveals that downside risks to
implementation identified in the institutional risk section of reports are
rarely factored into the net present value calculation. This review covered all such projects with net commitments that exceeded $1 million.
Further, there is little evidence from these completion reports or the
survey with Bank staff of systematic collection and use of data from
previous projects on items such as delays due to procurement problems, poor administration, or cost-overruns. In other words, there is no
evidence that the Bank systematically factors lessons from past costbenefit analysis into new rounds of cost-benefit assessment. To be sure,
this probably happens to some degree, but if so, it is highly dependent
on the person doing the analysis. There is no systematic effort to collect
and record information and factor it into future assessments to enforce
discipline and quality control across all cost-benefit analyses.
4.6
Counterfactual. Bank policy requires that ““Both benefits and
costs are defined as incremental compared to the situation without
the project”” and that ““The project is also compared with the alternative of not doing it at all.”” There exist both analytical tools and empirical tools to assist in this assessment, but the evidence is that these are
but sparingly used in the ex-ante analysis performed for projects.
4.7
Progress can be made in determining what would likely occur
without the project through analysis. For public goods such as a national system for registering property the situation without the project
is plausibly no investment. Similar reasoning can be applied to investments with high positive externalities. For such investments
24
CHAPTER 4
QUALITY OF ANALYSIS
there is a case based on the public good nature of the investment that
the without-project situation would be no investment.
4.8
Belli and Guerrero (2009) indicate that the extent to which what
the project provides is a public good is rarely discussed, and hence is
rarely used to support an assessment of what would happen without
the project. Furthermore, the way project appraisal reports treat the
public sector rationale for projects in general is an issue in itself, apart
from its use as helpful information to assess the without-project scenario.17 While it is no doubt true that some projects regard the public rationale as obvious and not requiring repetition, what is telling is that
““None of the projects that produced private goods provided a justification for public involvement.”” Indeed, in Bank appraisal reports the case
for doing projects is rarely grounded in first principles. Much more
common is for project documents to justify Bank involvement on the
basis of long-time involvement in the sector or accumulated experience.
4.9
When projects provide private goods or services there is always the possibility that private suppliers would have provided the
good anyway, and hence that the difference between with and without project benefit streams would be small. Determining and estimating what would have happened without the project in these cases can
be a difficult issue for the cost-benefit assessment. This is where empirical techniques such as impact evaluation can be a useful input to
cost-benefit assessments, but in fact there is little evidence of their use.
4.10
Alternatives. Comparison with alternatives is one of the core
features of the analysis recommended by Bank policy. Ideally, project
appraisal reports would simply present the alternatives considered
before decisions were taken, and the rationale for the decision would
be apparent from the net present value estimates associated with each
alternative. Instead what is found in appraisal reports is a single alternative presented——that of the project chosen. There is little evidence from the analysis that a systematic effort to compare and chose
from alternatives is a major part of decision making at the Bank.
Nevertheless, when rated against the less stringent criteria of whether
there was some evidence of consideration of meaningful alternatives,
about 50 percent of projects pass this test.
4.11
What emerges in the other roughly 50 percent of the cases is
that, while all documents have a section on alternatives, the discussion is not of alternative projects or alternative project designs, but
rather about less important choices such as alternative lending instruments. In other words, the alternatives considered do not bear on
fundamental issues of the choice and justification for the project.
25
CHAPTER 4
QUALITY OF ANALYSIS
Box 3. Cost-Benefit Analysis and the Results Agenda
The managing for results agenda in recent years has been dominated by discussions about measuring results, using logical frameworks to frame the
monitoring and evaluation efforts, and impact evaluation to measure impact
in a more accurate and rigorous way. These efforts complement each other
and also complement cost-benefit analysis. Yet in practice they are often
treated separately, leading to unnecessary fragmentation.
It is easy to find cost-benefit analysis that does not mention or use impact
evaluation results, despite the fact that measurement of benefits against the
counterfactual is integral to cost-benefit assessment. Similarly, it is rare to
find impact evaluation studies that embed the results they obtain in a costbenefit setting. For example, suppose that an intervention is designed to
raise rice yields. The value of the increase in yields would be part of the
benefit flow in the cost-benefit analysis, and the analyst typically would
make an informed estimate of what yields would have been in the absence of
the intervention and compute the value of the change in yields. An impact
evaluation would provide that figure more accurately. Similarly, an impact
evaluation that went no further than providing estimates of the increase in
yields would be an incomplete evaluation for decision––makers wanting to
know whether to repeat the project. What was the value of the increase in
yields, how does that compare with the costs? In this example the information from the impact evaluation and the cost-benefit framework complement
each other and provide better analysis.
Crucial to the success of the project is having good answers to basic questions. What is the problem being addressed? Will the activities of the
project lead to the desired outcomes? If the change desired is so beneficial,
why hasn’’t it been done already? What is the economic rationale——public
good, externality, information asymmetries, redistribution, or distortions in
other markets? It is sometimes said that such questions are more important
than the cost-benefit calculation. This may be the case, but there is no need to
pit the two against each other as though they are substitutes. A good costbenefit analysis requires a clear logic for how the benefits will be achieved.
And a clear logic in itself does not guarantee that benefits will exceed costs,
so project analysis should contain both features.
Furthermore, the effort of constructing a cost-benefit analysis clarifies the
indicators and data required to determine if benefits are being achieved at
the required levels. This can focus and simplify monitoring, indicator development, and indicator target-setting so the cost-benefit exercise can sharpen
the monitoring and evaluation effort.
In contrast, staff interviews revealed that the components above are often
undertaken by different groups at different times in the project development.
Individuals writing the results framework may not contact the economist or
consultant completing the cost-benefit analysis. And those determining the
indicators to use for monitoring may be yet a different team.
4.12
In some sectors, transport being the prime example, consideration of alternatives is hard-wired into the spreadsheets used for economic analysis. In the Highway Development and Maintenance
26
CHAPTER 4
QUALITY OF ANALYSIS
(HDM) model or the Roads Economic Decision model, alternatives are
incorporated directly into the models. Indeed, 24 of 28 transport
projects were assessed to have an acceptable discussion of alternatives.
Outside of the transport and energy sectors, only 18 of 58 projects were
judged to have an adequate or good consideration of alternatives. Sectors where discussion of alternatives is especially low are Health, Social
Protection, Social Development, and Public Governance.
4.13
Risk. Bank policy on risk analysis focuses on identification of
key variables upon which hinges the economic impact of the project.
Ideally this information is to be used by managers in setting priorities
and focusing their monitoring efforts. Technically, the steps required
are identification of key variables, sensitivity analysis of those key variables and calculation of switching values (critical values below
which the net present value would be negative).
4.14
Apart from its role in flagging sensitive variables and parameters, risk analysis is not expected to calculate the variability in returns
and feed that information into decision making. Bank policy is that
variability of the project’’s expected net present value around its mean
should not carry weight in decisions. But Bank policy does say that
risk analysis should be used to increase the expected return and reduce the risk of failure. Risk analysis should also be used to prioritize
what variables to monitor during implementation.
4.15
When assessed against these criteria, Belli and Guerrero (2009)
conclude that risk analysis is one of the weakest areas of the project
appraisal documents. The typical analysis of risk conducts sensitivity
analysis by simply varying aggregate costs and benefits by some percentage. Every document has a section identifying major risk, but
there is rarely a connection between these risks and the sensitivity
analysis. Monte Carlo analysis, which can be readily performed with
a spreadsheet, is done in less than 10 percent of the projects.
4.16
Poverty impact. In some projects it is possible to forecast beneficiaries and identify them by income group to gauge the impact of the
project in raising income of poor individuals. Agriculture technical assistance projects and education projects are examples. The appraisal
document identifies project beneficiaries by income in only 14 percent
of projects. In a further 26 percent project beneficiaries are identified
and there is some use of income data. In a further 47 percent the documents contain only general discussions of who would benefit, and in 6
percent of projects there is no discussion of beneficiaries.
4.17
Externalities. Just about every appraisal document discusses
the project’’s environmental impact and mitigating measures. The major issue appears to be lack of clarity about whether environmental
externalities are factored into the economic costs and benefits. In 13
27
CHAPTER 4
QUALITY OF ANALYSIS
percent of projects it is clear that environmental costs and benefits are
estimated and included in the economic analysis; in a further 34 percent environmental impact are quantified but there is no evidence of
inclusion. In 47 percent of projects environmental costs are discussed
in isolation.
4.18
There are further technical shortcomings in World Bank costbenefit work that are not listed in OP 10.04 but are nevertheless significant. When there are distortions to market prices, shadow prices
need to be used by the cost-benefit analysis to measure the real value
or costs of goods and services. Previous reviews of the Bank’’s economic cost-benefit analysis found that the use of shadow prices recommended by Little and Mirrlees (1969) and Squire and Van der Tak
(1975) were never effectively implemented.18 This has also been
found by Salop (1992), Garcia Garcia (2009), and Belli and Guerrero
(2009). It is clear that this issue with the analysis has not improved
over the years.
4.19
Belli and Guerrero found the economic analysis in appraisal
documents acceptable or good in 54 percent of the cases. An average
of the three similar reviews in the 1990s found economic analysis to
be acceptable or good in 70 percent of the appraisal documents, so
analytical quality appears to have declined. However, there is also a
lower percentage of very poor analysis. This lowest rating was given
to 7 percent of projects in the 1990s and only 1 percent in recent
projects. Hence there is a modest compression toward the middle.
When rated by sector, the best performance was in the Water sector,
followed by Agriculture. In contrast, only 20-40 percent of the appraisal documents in Urban, Education, and Health were rated acceptable or better.
28
Evaluation Essentials
5. Accuracy in Cost-Benefit
Calculations
5.1
This chapter assesses whether there is any evidence of bias in
the reporting of cost-benefit results at the end of projects, focusing
primarily but not exclusively on the ERRs. To what extent do the reported results provide what corporate accountants call a ““true and
fair view”” of the economic impact of the project?
5.2
The first indicator of possible bias is a comparison of the economic rates of return at project entry with the recalculated rates of
return at project closing: are they higher or lower? This ERR gap was
first discussed by the Board of the Bank in 1987,19 and a subsequent
report by the Operations Evaluation Department (now IEG) in 1989
concluded that two-thirds was due to actual benefits falling short of
expectations and one-third to project implementation delays.20 Further reports from OED cited unrealistic forecasts in appraisal estimates as a continued problem.21
™ The gap between ERRs at
appraisal and at close has
narrowed, but this is likely a byproduct of two separate and
unrelated forces rather than
enhanced consistency between
pre- and post-analysis
™ Projects with lower outcome
ratings are less likely to have
the ERRs calculated again at
closing
™ Few projects report negative
ERRs at closing
™ Growth forecasts from country
strategy reports are reasonably
accurate but usually miss
negative growth
™ Sustainability of benefits is
rarely checked beyond the
project life, which likely imparts
a positive bias to reported
ERRs
5.3
Interestingly, the ERR gap has virtually disappeared in recent
years. Figure 5 shows median ERRs, both at appraisal and closing,
since 1972, along with lines indicating the five-year moving average
of each data series. It shows that the moving averages became essentially the same by the mid 2000s. Figure 5 also shows that most of the
closing of the gap was due to the rise in median ERRs at closing, analyzed later in this report.
5.4
Closer analysis reveals that the closing of the ERR gap is likely
a byproduct of unrelated forces driving the two time series. Regarding the median ERR estimates at entry, constantly increasing forces
such as population and traffic density are likely responsible for the
steady positive trend over time. Examination of the way benefits are
calculated in the HDM-4 model for the transport sector reveals that
forecasted benefits will rise automatically with a rise in traffic density.
Regarding the ex-post ERRs, these reflect actual outcomes in projects,
and hence exhibit greater fluctuation in the time series, driven in part
by the forces discussed in chapter 7.
29
CHAPTER 5
ACCURACY IN COST-BENEFIT CALCULATIONS
Figure 5. Before-Project and After-Project Economic Rates of Return Have
Converged
Medi an ERR - end of project
Medi an ERR - start of project
5-year M oving Average
5-year M oving Average
24
20
15
12
9
1972
1980
1990
year
2000
2008
Source: World Bank data.
Note: Figures in percent.
5.5
Previous Bank reports have pointed to several manifestations
of an optimism bias at appraisal in the past. A report solicited by
President Preston in 1992 cited a Central Operations Department finding that only 22 percent of financial covenants in loan agreements
were in compliance, suggesting that what was written in loan covenants was chronically optimistic. Based in part on a workshop to solicit the views of borrowers, the same report concluded:
““In the eyes of Borrowers and co-lenders as well as staff, the
emphasis on timely loan approval (described in some assistance agencies as the ““approval culture””) and the often active
Bank role in preparation, may connote a promotional——rather
than objective——approach to appraisal. Borrowers allege that
loans feature conditions thought to be conducive to approval
by management and the Board, even when these may complicate projects so as to jeopardize successful implementation.””22
5.6
It has long been noted that projects frequently underestimate
the time required for implementation: the average length of projects is
seven years, compared with an average estimate at appraisal of five
years.23
5.7
It is rare to find reports that analyze and quantify bias in
project analysis at appraisal and especially rare to find ones that compare this across institutions. One exception is a study comparing forecasted rates of return for road projects in China done at the time
projects were under consideration with returns for the same projects
30
CHAPTER 5
ACCURACY IN COST-BENEFIT CALCULATIONS
estimated months later, just before the projects were to be implemented. Figure 6 shows a summary of the results. There was, on average, a positive bias in projects funded by the Chinese government,
the Asian Development Bank, and the World Bank. The re-estimated
returns were 2.5 percentage points lower than the initially estimated
returns for domestic-funded roads, 3 percentage points lower for
ADB––funded roads, and 5.5 percentage points lower for World Bankfunded roads.
Figure 6. Forecasting Bias in Predicted Economic Returns—Road Projects
6.0
5.0
4.0
3.0
2.0
1.0
0.0
Domestic Projects
Asian Development Bank
Projects
World Bank Projects
Source: Asian Development Bank (2006).
Note: Figures given are percentage point differences in ERRs calculated for feasibility studies and
revised ERRs just before projects commence.
5.8
Previous reports have been critical of the analysis and treatment of risks in Bank appraisal documents. Downside risks are
usually ignored, leading to an upward bias in the ERRs reported at
appraisal. A 1990 review of economic analysis in Bank projects concluded that ““project assumptions about government implementation
capacity, macroeconomic performance, availability of local cost financing, and other key operational variables are not factored into the
[ERR] calculations. Although some SARs refer to the macroeconomic
environment as being important for determining the project outcome,
the variables are not explicitly taken into account in calculating ERRs””
(emphasis in the original).24 The review of economic analysis conducted for the current report contains a similar conclusion: risks identified in the risk section of the appraisal reports are not factored into
the cost-benefit calculations in other parts of the reports.
5.9
In some projects, ERRs are calculated at appraisal but not
again at closing. The possibility of recalculation bias can be examined
by asking whether projects that received a low rating by the Independent Evaluation Group at the Bank were less likely to have their ERRs
recalculated than projects that received favorable ratings.
5.10
To determine if there is evidence that ERR recalculation was
lower for poorly performing projects, Figure 7 shows the probability
of recalculation of ERRs for projects in the high-CBA sectors (sectors
31
CHAPTER 5
ACCURACY IN COST-BENEFIT CALCULATIONS
that frequently calculate ERRs for their projects) since 1992. The results show that unsatisfactory projects were much less likely to get the
ERR recalculated at closing, suggesting that this kind of bias is indeed
present. Figure 8 shows the same evidence for low-CBA sectors. For
projects in these sectors the probability that an ERR will be recalculated for highly unsatisfactory projects is virtually zero.
Figure 7. The Probability of Recalculation of ERRs at Closing is Higher in
Better-Performing Projects—High-CBA Sectors 1993-2007
1.00
0.80
0.60
0.40
0.20
Highly Unsat Unsatisfactory Moderately
Unsat
Moderately
Sat
Satisfactory
Highly
Satisfactory
Source: World Bank data.
Figure 8. The Probability of Recalculation of ERRs at Closing is Higher in BetterPerforming Projects—Low-CBA Sectors 1993-2007
1.00
0.80
0.60
0.40
0.20
0.00
Highly Unsat Unsatisfactory Moderately
Unsat
Moderately
Sat
Satisfactory
Highly
Satisfactory
Source: World Bank data.
5.11
It was also checked whether this bias has changed over time.
Figure 9 shows that, among projects for which an ERR was calculated
before the project began, the probability that an ERR was calculated
after the project closed has always been lower for the lowerperforming projects. In this case a lower-performing project is defined as a project that received an unsatisfactory rating by the Independent Evaluation Group. The presence of an ERR has a minor
32
CHAPTER 5
ACCURACY IN COST-BENEFIT CALCULATIONS
weight in the project rating criteria. The recalculation probability has
been declining over time for both classes of projects. However, the
difference between the two recalculation probabilities, one possible
measure of the degree of bias, was higher after 1987, when it averaged
0.24, than before, when it averaged 0.18.25 This is one piece of evidence suggesting a small rise in the bias over time.
Figure 9. The Probability of Calculating a Final ERR Given that an Initial ERR
Was Calculated Has Always Been Lower for Lower-Rated Projects
High-rated Projects
Low-rated Proj ects
1
.5
1973
1980
1990
Year of Closi ng of Proj ects
2000
2006
Source: World Bank data.
Note: High-rated projects are defined as those that obtained a satisfactory rating by the Independent
Evaluation Group.
5.12
Apart from recalculation bias there is the possibility of calculation bias. Many projects do not report cost-benefit analysis at closing,
and some of these are due to correctable factors such as lack of data
collection. Another fact is that few ERRs reported at closing are negative. Of 1,299 projects that have closed since 1990 and report an estimate of the ERR at closing, 4 report returns below -5 percent. There
are some project completion reports that contain data suggesting negative returns, but the returns are not explicitly calculated. These facts
suggest that bias due to non-reporting is a potentially serious issue.
5.13
Further possible sources of bias for which evidence can be obtained are the growth forecasts in Country Assistance Strategy Reports. For example, ERRs for roads contain forecasts of traffic growth,
and these are usually assumed to be some fixed multiple of overall
forecasted economic growth. If the growth forecasts are biased in the
positive direction, then any ERR based on them will also have a positive bias.
33
CHAPTER 5
ACCURACY IN COST-BENEFIT CALCULATIONS
5.14
When checked, it emerges that this is a small source of bias.
The growth forecasts in 59 recent Country Assistance Strategy documents were examined and compared with actual growth outcomes.
In 51 cases, both forecasted and actual growth was positive. In these
51 cases the average forecast was 5.2 percent and the outcome was 5.0
percent, so there was a small upward bias. The weakness in forecasting, however, is that negative growth is rarely predicted. In a further
7 cases the forecast was positive and the outcome negative. In these 7
cases, the average forecast was 4.3 percent and the outcome, -4.6 percent. There was one case where growth of -5 percent was predicted,
and the outcome was positive growth of 1 percent. Combining all 59
cases together the average forecast was 5 percent, and the average
outcome was 3.8 percent. This evidence suggests there is a small positive bias in reported ERRs at entry stemming from the fact that forecasts rarely predict negative growth.26
Box 4. The Problem of Fragile Estimates
The conclusions of cost-benefit analyses are sensitive to parameter assumptions. Should this lead one
to avoid such analysis or to improve oversight and standards?
Consider, for comparison, the example and history of corporate accounting. If one thinks cost-benefit
analyses are easily manipulated, what about profit and loss statements of firms? The difference is
that with regard to corporate accounts, nations have developed their own Generally Accepted Accounting Principles (GAAP), and international bodies such as the International Accounting Standards Board have formed to develop and refine International Financial Reporting Standards. Many
nations have now adopted the IFRS, and the world is well on the path toward a single globally accepted set of accounting rules to discipline corporate accounts. An entire profession with accreditation standards exists to prepare corporate accounts.
But this is evidently not enough. On top of the accounting rules there are the auditors to independently verify the accounts. The auditors have their own Generally Accepted Auditing Standards
(GAAS) and the International Federation of Accountants has developed International Standards of
Auditing to move toward international standards and harmonized procedures.
Nevertheless, recent events have shown that even this superstructure of accounting standards is not
sufficient when conflicts of interest exist. Arthur Andersen is no longer in business after the ENRON
affair, and a court-appointed examiner has found that Ernst and Young were aware of but did not
question Lehman Brother’’s use and nondisclosure of an accounting procedure that understated their
leverage.27
The point is that when confronted with problems with corporate accounts, the action has usually
been to tighten accountability standards. Fragility in corporate accounting is rarely used to argue for
dispensing with corporate accounts altogether.
5.15
A final possible source of positive bias is the lack of reporting
on long-term outcomes. The typical ERR includes benefit flows that
last 25-30 years, yet the sustainability of such flows is rarely verified
after 7-10 years. Furthermore, ERR estimates virtually always assume
either that benefit flows remain at a constant high level or continue
growing. At the same time, the few studies on sustainability of
34
CHAPTER 5
ACCURACY IN COST-BENEFIT CALCULATIONS
projects that exist typically find that benefit levels fall short of forecasts. Although there is a shortage of formal documentation on this
subject, the evidence that does exist suggests that this is a further
source of upward bias in ERR estimates.
5.16
In summary, this chapter has reviewed eight possible areas
where there might be evidence for bias in calculation of ERRs. Of the
eight, one, the closing of the ERR gap is probably a byproduct of two
unconnected trends, and the other seven suggest a positive bias. On
balance this evidence, especially the non-reporting of negative outcomes, raises the likelihood of a positive bias in ERR reporting.
35
6. Use of Cost-Benefit Analysis for
Decision Making
6.1
The main purpose of cost-benefit analysis is to use it to improve decisions, choosing projects with higher net benefits over those
with lower net benefits and thereby maximizing the effectiveness of
development assistance. Using cost-benefit information for decision
making would reassure donors that the Bank is using their money to
the maximum effect.
6.2
This chapter examines whether the Bank’’s decision making
allows for effective use of cost-benefit analysis. The conclusions are
based on interviews with 51 Bank project leaders (task team leaders,
or TTLs), chosen randomly from the universe of all project leaders of
projects that closed in fiscal year 2006-07 and 2008-09.28 In most cases
(74 percent), sampled TTLs were not economists by background, although in the FY08-09 subsample there were more economists (31
percent) than in the other (21 percent).
Evaluation Essentials
™ Interviews with Bank staff
reveal that cost-benefit analysis
is prepared after major
decisions are made and thus
has little influence on important
decisions
™ Most cost-benefit analysis is
done by less senior staff or
consultants
™ Senior staff show little interest
in cost-benefit results and a lot
of interest in safeguards,
procurement and financial
management
6.3
The most significant finding is that project leaders report that
cost-benefit analysis is usually conducted after the decision has been
reached to pursue a project. Of the 51 project leaders, 5 report that costbenefit analysis is considered with significant weight at the identification stage. Eighteen of 51 report that cost-benefit analysis is given
significant weight at the preparation stage.
6.4
When probed for clarification on whether CBA has ever been
the key criterion in deciding to fund a project (Question 10), project
leaders overwhelmingly (82 percent) said it had not. The initial decision about a project occurs well before CBA is ever done; in those cases where it may already be available, it is still used as a designing tool
at best, to evaluate components or subcomponents of a project that
should be added or dropped based on their economic viability.
6.5
Moreover, the cost-benefit analysis is usually conducted by
less senior staff. Forty-one project leaders report that an outside consultant was hired for the task; only 10 project leaders did the analysis
themselves. Project leaders report that there is very little involvement
or interest in the cost-benefit analysis by staff above the level of the
project leader. Country directors or sector directors rarely comment
or show any interest.
6.6
One important question is the degree to which this situation
causes omission of potentially negative information. When asked
whether, in their experience, cost-benefit results had ever been altered
37
CHAPTER 6
USE OF COST-BENEFIT ANALYSIS FOR DECISION MAKING
at the request of a key decision maker, 14 project leaders said yes, 34
said no.
6.7
Does time devoted to improving the cost-benefit analysis enhance career prospects? Most TTLs agreed that contributing to the
CBA of a project did not result in a better personal performance evaluation (55 percent) or a better chance of being promoted (92 percent).
Thirty-eight percent of TTLs reported that contributing to the CBA of
a project did not result in greater recognition or approval by their
peers, while 42 percent noted that such recognition was only partly
significant. When it comes to the approval of the TTL’’s superior, 50
percent responded that recognition was only partly significant and 19
percent said that it was quite significant or significant. Some TTLs
added that when their superiors are economists, they tend to give
greater consideration to the quality of the CBA. The approval of outside stakeholders on the quality of CBA, on the other hand, is not significant at all for 54 percent of TTLs.
6.8
Time and money spent on cost-benefit analysis varies considerably. It is normally performed by a project economist, who might
either be a staff member or a consultant. He or she is allotted, as median values, 18 days and a budget of about $16,000. These values
were slightly higher in the subsamples of the projects ending in FY06
or 07 (18 days and $18,125) than in the subsample of projects starting
in FY08 and 09 (15 days and $14,375). For instance, a couple of TTLs
from Public Sector Governance reported that they allot only two days
to CBA——the lowest value in the whole sample. At the other end of
the spectrum, TTLs from the Transport sector allot about 20 days, the
highest value in the sample.
6.9
In their interviews, some TTLs asked the Bank to devote more
resources both to the training of staff and to the dissemination of methodologies in an effort to overcome some of the sector-specific challenges to meaningful use of CBA as a decision-making tool. As a case
in point they noted the heavy resources invested to implement the
guidelines on safeguards, procurement, and financial management,
which not only tend to absorb most of the already limited time at
TTLs’’ disposal, but also tend to be the areas where TTLs then expect
the Bank’’s senior staff and management to focus most when evaluating a project. Thus, in project design, TTLs will focus their efforts
heavily on the accurate implementation of the above guidelines, typically devoting little or no time to CBA, which rarely warrants comments from the upper echelon of the Bank's staff. As one TTL put it
““Management wants CBA for paperwork, and staff complies as it is
easier for them to go with the flow.””
6.10
Do negative results from a cost-benefit assessment impede or
block project approval? Overall, most TTLs (80 percent) concurred
38
CHAPTER 6
USE OF COST-BENEFIT ANALYSIS FOR DECISION MAKING
that when preparing a project proposal, it is sufficient to enclose a
reasonably specified CBA (Question 16a), alluding to the need to ““tick
a box”” in the template of the required documents for the project to be
successfully appraised. Many TTLs considered CBA to be just an ““input into the discussion,”” a ““factor in the conversation,”” a ““parameter,”” which might attract attention if it challenged the viability of a
project, though even then the project may still go forward. There are
no examples of appraisal documents that report negative ERR’’s or
NPV’’s. A small majority of TTLs (fifty-eight percent) think that the
existence of a CBA helps approval.
6.11
Bank staff are not unanimous in their support for basing
project decisions on cost-benefit analysis, so staff opposition is part of
the reason for poor adherence to Bank policy. But project leaders are
supportive on the whole. When asked about the usefulness of CBA
broadly defined, most TTLs responded that it does enable objective
performance assessment based on data (82 percent), although TTLs
from the Public Sector Governance, Health, and Education sectors
pointed out a lack of relevant data. Respondents also underscored the
potential of CBA to reveal factors crucial to better performance (77
percent), as well as its usefulness in diagnosing and modifying an
underperforming project midway through implementation, though
admittedly they have yet to see this happen (47 percent). Finally,
TTLs overwhelmingly agreed (82 percent) that CBA should be used to
accurately estimate economic returns at closing and to extrapolate lessons for amending future projects (88 percent). In fact, many of them
noted that CBA is done ex post precisely for this purpose, typically in
the context of implementation completion reports.
6.12
The overall picture that emerges is that the quality and accuracy of cost-benefit analysis is negatively affected by the decisionmaking structure and incentives at the Bank. Ex-ante cost-benefit information is available too late in the decision-making process, and the
incentives of staff or consultants to conduct careful cost-benefit analysis are affected by the fact that important decisions have already been
taken when they do their analysis. Project leaders reported that no
positive incentives for devoting time and energy to cost-benefit analysis, and report that senior management rarely takes notice of or comments on such analysis. There is also a potential conflict of interest in
the fact that the Bank places responsibility for the cost-benefit assessment in the hands of staff that are often responsible for guiding the
project through Board approval.
39
Evaluation Essentials
7. The Rise in Rates of Return
7.1
The World Bank has kept data on the economic rates of return
achieved by its projects for many years, but statements by Bank officials, major policy documents, or formal reviews of performance rarely mention the results. This lack of attention is both telling and surprising given that Bank data show that the median economic rate of
return on the subset of projects that conduct cost-benefit analysis has
approximately doubled over the past two decades——since 1987. Are
these data reliable? If so, what can we learn from them? What could
account for the rise?
7.2
This chapter presents tentative findings on the causes of this
rise in returns. One hypothesis that cannot be tested rigorously is that
bias has increased——specifically that since 1987 there have been gradual increases in the percentage of low-return projects that fail to report ERRs, causing the median ERR among the remaining projects to
drift upward. There is no objective evidence in favor of this, but it
cannot be ruled out. Note that most of the evidence in chapter 5 regarding an upward bias in returns concerns the level of median returns at all time periods, but this does not immediately cast doubt on
the rise in returns because errors may be constant over time. The one
piece of evidence that does cast light on the change in bias over time,
namely the difference in recalculation probabilities in Figure 9, does
suggest a slight increase after 1987, but the difference is not large and
drifts up and down over that period, whereas the rise in returns increases constantly.29
™ The median ERR reported for
Bank projects has doubled over
the past 20 years
™ An increasing bias in
measuring returns may explain
the rise, though this is difficult
to confirm
™ The trend in project outcomes
has been on a similar upward
trend
™ The rise in outcome ratings has
been driven by the five sectors
that routinely produce costbenefit analyses
™ The post-1987 period was
characterized by a shift toward
market-based approaches,
which correlates with the rise in
ERRs, as do economic growth
rates
™ Most (83 percent) of the
variation in rates of return
between 1974 and 2007 is
accounted for by the rise in
market orientation and higher
economic growth
7.3
Figure 10 shows the data on the median economic rates of return of all Bank projects for which ERRs have been calculated. These
rates of return are calculated at project closing rather than at appraisal. Hence they are the best information available on the actual return
achieved by Bank projects. The median rate of return among all
projects that closed in a given year has risen from a low of approximately 12 percent in 1987-88 to a high of about 24 percent in 200507——a doubling in 20 years.
41
CHAPTER 7
THE RISE ON RATES OF RETURN
Figure 10. Median Economic Rate of Return in Percent, All World Bank
Projects, 1972-2008
Medi an ERR - end of project
5-year M oving Average
24
20
15
12
9
1972
1980
1987 1990
year
2000
2008
Source: World Bank data.
7.4
These are not small differences. Consider the case of an agriculture project (Box 5) to understand the contrast between 12 and 24
percent rates of return. In the hypothetical example, real incomes of a
poor family increase by a factor of 5 after 20 years with a 24 percent
rate of return but by a factor of just 2 after 20 years with a 12 percent
rate of return. In this example a doubling of the economic rate of return is associated with more than a doubling of the underlying
growth in real agricultural incomes.
Box 5. The Difference between a 12 Percent and a 24 Percent Economic Rate of
Return—Example from Agriculture
The difference between a 12 and 24 percent economic rate of return is evident from a simplified example of an agriculture project that delivers technical assistance to families with income near the poverty line, with the objective of raising their incomes to enable them to escape from poverty. The
table below compares two projects. Both projects deliver technical assistance
to a family that begins with income of $2 per day——$730 per year.
Suppose that the technical assistance costs $2,000 per family in both cases.
The 24 percent ERR project requires income growth on the order of 8.9 percent per year, while the 12 percent ERR project requires income growth of
just 3.7 percent.
Continued on next page>
42
CHAPTER 7
THE RISE ON RATES OF RETURN
>Continued from preceding page
Base income (poverty line x2)
Income growth due to project
Cost of project
Income after 5 years
Income after 10 years
Income after 20 years
12% ERR
24% ERR
$730
3.7%
$2,000
$844
$1,012
$1,456
$730
8.9%
$2,000
$1,027
$1,572
$3,689
The full effect of the differing rates of return and differences in underlying
growth is apparent with the passage of time. After 10 years the 12 percent
ERR project would raise family income by only $282, from $730 to $1,012,
while the 24 percent ERR project would raise family income by almost three
times as much, to $1,572. After 20 years the 12 percent ERR project would
raise family income to $1,456, while the 24 percent ERR project would move
the family to near middle-class status, with an annual income of $3,689, more
than double the $1,456 figure. Hence the differences in income can be truly
significant over long horizons of 20-30 years.
Source: Author’s calculations.
7.5
Clearly the rise in rates of return is quantitatively important.
Yet the magnitude of the increase forces us to ask whether it is plausible and whether it can be corroborated by other performance measures. To investigate, this chapter first reviews data by sector to examine the plausibility of trends at the sector level. It then compares
the economic rate of return data with indicators of overall project performance, both by sector and in the aggregate. Are the changes over
time in median returns by sector positively associated with changes
over time in performance ratings? After finding that increases in median returns are associated with increases in the percentage of projects
given a satisfactory rating, the chapter considers what is driving the
increase in median returns.
7.6
Disaggregating the data by sector shows that the rise in median rates of return occurred in each of the five high-CBA sectors, but
the pattern over time was different in two of these sectors, Agriculture and Energy and Mining. In these two sectors economic returns
exhibited a V pattern with a bottom in 1987 rather than a steady increase over time.
7.7
Figure 11 shows median economic rates of return for Agriculture and Rural Development projects. The small circles in the figure
represent the median return of all investment projects that closed in
the indicated year; the line shows the average trend over time. The
downward trend during the 1970s and 1980s was broken in 1987,
when a positive trend set in and has continued to the present. The
43
CHAPTER 7
THE RISE ON RATES OF RETURN
year in which the trend changed was chosen by a statistical procedure
that selects the break point that best summarizes the data.
Figure 11. Median ERRs in Agriculture and Rural Development, in Percent
Median Rate of Return
Estimated Regression Line
25
20
15
9
1980
1972
1987 1990
Year of Project Completion
2000
2008
Source: World Bank data.
7.8
Figure 12 shows similar evidence for Energy and Mining. In
this case there was again a change in the trend from negative to positive with 1987 once again the critical year.
Figure 12. Median ERRs in Energy and Mining, in Percent
Median Rate of Return
Estimated Regression Line
25
20
15
9
1972
Source: World Bank data.
44
1980
1987 1990
Year of Project Completion
2000
2008
CHAPTER 7
THE RISE ON RATES OF RETURN
7.9
Median economic rates of return in the Transport sector exhibit a rising trend over time with no noticeable change in that trend
(Figure 13).
Figure 13. Median ERRs in Transport, in Percent
Median Rate of Return
Estimated Regression Line
40
35
30
25
20
15
9
1980
1987 1990
Year of Project Completion
1972
2000
2008
Source: World Bank data.
7.10
The same gradual rise in the median rates of return is apparent in the Water and Urban sectors (Figure 14). These two sectors are
not displayed separately due to small samples. In addition, the trends
in each turn out to be similar so a single graph provides a fair representation of trends in both sectors.
Figure 14. Median ERRs in Water and Urban Development, in Percent
Median Rate of Return
Estimated Regression Line
30
25
20
15
9
1972
1980
1987 1990
Year of Project Completion
2000
2008
Source: World Bank data.
45
CHAPTER 7
THE RISE ON RATES OF RETURN
7.11
The median rates of return in the remaining low-CBA sectors
show no significant positive trend (Figure 15).
Figure 15. Median ERRs in all Other Sectors Combined, in Percent
Median Rate of Return
Estimated Regression Line
35
30
25
20
15
9
1972
1980
1987 1990
Year of Project Completion
2000
2008
Source: World Bank data.
7.12
Hence the rise in median rates of return can be isolated to five
sectors. In two of these sectors, median returns exhibit a V pattern
centered on 1987. Before 1987, positive trends in the Transport, Water, and Urban sectors were offset by negative trends in Agriculture
and Energy and Mining, resulting in zero overall increase in Bankwide economic returns. After 1987, positive trends in all five sectors
combined to drive a large and significant increase in Bank-wide economic returns.
7.13
The low-CBA sectors show no positive time trend in median
economic rates of return (Figure 15), but they do show a mean return
of approximately 20 percent, indicating that such projects can have
high returns. The low-CBA sectors are not on average low-ERR sectors. Since these returns are not significantly different from those in
the high-CBA sectors, the evidence does not suggest that the shift toward low-CBA sectors raised overall economic returns. However,
note that the sample of projects behind Figure 15 is small.
7.14
Based on the foregoing review, there is nothing obviously implausible in the doubling of returns over 20 years. But can the rise in
median returns by sector be corroborated by other performance data?
The most commonly used performance ratings are the outcome ratings produced by IEG. These ratings have been applied to the vast
majority of projects since 1973. The ratings system was binary (satisfactory/unsatisfactory) until the early 1990s, when it switched to the
46
CHAPTER 7
THE RISE ON RATES OF RETURN
current six-point scale (highly satisfactory, satisfactory, moderately
satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory). All six categories of ratings were first widely applied by
IEG evaluators in 1993.
7.15
As a cautionary note, the performance ratings and the economic returns contain some common information, because the returns
are observed by the evaluator doing the rating. So these two pieces of
information are not entirely independent. But the overlap between
them is not huge either. Annex Table B 1 shows that there are numerous projects with low performance ratings and yet high economic
returns and vice versa. Overall, the simple correlation is approximately 40 percent.30
7.16
Using 1993 as the starting point for a comparison of the trends
in ratings reveals a pattern broadly similar to the positive trends in
median economic rates of return. Table 7 shows estimates of the average trends in project ratings by sector.
Table 7. Trends in IEG Project Performance Ratings by Sector Board, 1993-2008
Agriculture and Rural Development
Education
Energy and Mining
Environment
Financial and Private Sector
Development
Health, Nutrition, and Population
Public Sector Governance
Transport
Urban Development
Water
Time
trend
Standard
error
T-ratio
Number of
projects
0.053
-0.01
0.036
0.036
0.021
0.011
0.014
0.015
0.027
0.031
4.89
-0.68
2.36
1.32
0.69
692
357
421
172
221
-0.025
0.038
0.041
0.058
0.074
0.019
0.022
0.011
0.020
0.021
-1.29
1.74
3.62
2.87
3.46
269
192
410
206
190
Source: World Bank data.
7.17
The trends in performance ratings are positive and significant
for each of the same five sectors in which the trends in economic returns were positive. These five sectors are also the only sectors in
which the estimated trend is positive and statistically significant at the
5 percent level. Thus the same five sectors——Agriculture, Energy and
Mining, Transport, Urban Development, and Water——exhibit positive
trends in both performance measures. Trends in project ratings since
1992 broadly corroborate the positive trends in median economic
rates of return across sectors
47
CHAPTER 7
THE RISE ON RATES OF RETURN
7.18
The average performance ratings (scale of 1 to 6) of the five
high-CBA sectors are plotted over time in Figure 16, which shows the
strong increase in performance ratings over the 15-year period 19942008.
Figure 16. Average IEG Performance Ratings—High-CBA Sectors—and Trend
over Time
Perform ance - Hard Bank
T rend
4.8
4.6
4.4
4.2
4
3.8
1992
1996
2000
Year of Closi ng of Proj ects
2004
2008
Source: World Bank data.
Note: These are average ratings measured on a scale of 1-6, where highly unsatisfactory is assigned a
1, unsatisfactory a 2 and so forth. A rating of 4 corresponds to moderately satisfactory..
7.19
The average performance ratings of the low-CBA sectors are
plotted over time in Figure 17. Here there is no evident trend improvement. The lack of a positive trend is not due to an offset where
positive trends in some sectors cancel negative trends in others because, as Table 7 shows, none of the constituent sectors alone exhibited positive and statistically significant trends over time.
7.20
In conclusion, there is broad agreement between the ERR data
and the performance-rating data that there was an increase in project
performance since 1993 concentrated in the same five sectors. So,
what are the possible explanations for the increase in economic rates
of return?
48
CHAPTER 7
THE RISE ON RATES OF RETURN
Figure 17. Average IEG Performance Ratings—Low-CBA Sectors—and Trend
over Time
Perform ance - Soft Bank
T rend
4.8
4.6
4.4
4.2
4
3.8
1992
1996
2000
Year of Closi ng of Proj ects
2004
2008
Source: World Bank data.
Note: These are average ratings measured on a scale of 1-6, where highly unsatisfactory is assigned a
1, unsatisfactory a 2 and so forth. A rating of 4 corresponds to moderately satisfactory.
Explanations for the Rise in Rates of Return
7.21
The major explanations to be examined empirically are a rise
in the upward bias in returns; overall economic conditions as measured by growth; and the degree of market-orientation of the economic
regime and lack of war or civil conflict.
7.22
The possibility that an increasing positive bias might explain
the rise in returns was discussed at the beginning of the chapter. An
empirical proxy available for this is the difference in ERR recalculation
probabilities between low-rated and high-rated projects. The recalculation probabilities used in the calculation of the difference are those
shown in Figure 9.
7.23
An improvement in overall economic conditions in partner
countries is the second possible explanation for the rise in project returns. To test for this, the rate of GDP growth prevailing during project
implementation was calculated for all Bank projects and then averaged
over all projects in the year they closed.31 The resulting growth rates
are calculated separately for IDA and non-IDA countries.
49
CHAPTER 7
THE RISE ON RATES OF RETURN
Figure 18. Average Economic Growth Rates Occurring during World Bank
Project Implementation
IDA Countries
Others
6
5
4
3
2
1
0
1977
1987
Year of Closi ng
1997
2001
2007
Source: World bank data.
Note: Figures are the average annual growth rates that obtained during implementation of all projects
that closed in the year shown. The timing thus parallels that of the ERR calculations.
7.24
Based on the evidence in Figure 18, if growth raised returns
after 1987, its effects were most strongly felt late in the period. The
major increase in growth rates occurred late in the 1987-2007 period,
starting in 2001 for non-IDA countries and somewhat earlier for IDA
countries. The economic rates of return in Figure 10 show a long-term
increase starting in 1987; yet there is no obviously similar increase in
growth. The increase in growth since 1987 has been slightly higher
for IDA countries than non-IDA countries, which, holding constant
other things, should cause a higher increase in rates of return for IDA
counties.
7.25
It is worth noting that the absolute level of returns and the rise
in returns since 1987 are almost identical for IDA and non-IDA countries (Figure 19). This casts doubt on any explanation for the rise in
returns based on characteristics associated with IDA status, such as
level of income.
50
CHAPTER 7
THE RISE ON RATES OF RETURN
Figure 19. Median Economic Rates of Return in Bank Projects, IDA and NonIDA Countries
Non-IDA
IDA
24
20
15
12
9
1972
1980
1987 1990
year
2000
2008
Source: World Bank data.
7.26
Project completion reports and project performance audit reports from the years 1979, 1984, and 1989 were reviewed and compared with those for 2007-08 to better understand other factors that
could influence economic returns.32 Compared to projects from recent
years, projects in the earlier period contained more examples of administration by state enterprises and efforts to create, through
projects, private sector outcomes that the market was not creating.
Typical examples of problems that probably lowered returns include
the following. One credit project reported large cost increases experienced by borrowers due to delays in obtaining import licenses and
unanticipated increases in customs duties on imported equipment.
An agriculture project aimed to stimulate what was considered to be
disappointingly low private investment in livestock. However, after a
few years’’ effort, this did not occur to the extent anticipated. The
project then established a government-majority-owned development
company to help accomplish the task. The private sector response
continued to disappoint. Owners of small herds of livestock were
unwilling to organize themselves into cooperatives as desired by the
government and envisaged by the project. Further, the expected private financial co-investment in the development company was less
than anticipated, so the project switched to full government financing
of the development company. By the end of the project, production in
the formerly private livestock ranches controlled by the development
company was not significantly higher than before the project started.
7.27
Another agriculture program reported low returns stemming
from higher costs of vehicles and higher deterioration of equipment
51
CHAPTER 7
THE RISE ON RATES OF RETURN
than was anticipated at appraisal. The higher costs resulted from delays in obtaining import permits and from poor investment in maintenance by users. One root cause of the poor maintenance was the fact
that the vehicles were collectively owned by the project rather than the
farmers. Farmers did not have strong incentives to maintain any particular vehicle.
7.28
These examples illustrate four interconnected themes that can
influence economic returns. The livestock project illustrates a naïve
expectation that private investors would respond even if the incentives were not there and a failure to take private incentives seriously
in project design. The examples illustrate the extensive reliance on
public enterprises to implement projects. They also illustrate the effect of import quotas or licensing on the level and uncertainty of costs
of production. In 1997, the Bank’’s Operations Evaluation Department
(now IEG) summarized the environment under which agriculture
projects were implemented as follows:
““Throughout the 1970’’s, most developing countries followed
growth strategies that emphasized public production and direct controls on credit, foreign exchange and prices. Development programs relied heavily on Public Enterprises. Overvalued exchange rates, high tariffs, and quantitative
restrictions prevailed. Prices received by producers of export
crops were often less than half their world market value.””33
7.29
To test for the influence of such factors on economic returns in
projects, a simple indicator was developed to capture the broad orientation of economic policy in each country. The validity of this indicator rests on two facts. The first is that for a significant number of
countries, the fact that market-oriented reforms occurred is widely
known and uncontroversial. The second is that for many such countries, the date of reform is also widely known and uncontroversial.
The purpose of using nothing more than these two facts in measuring
the indicator is to ensure that the validity of the test does not hinge on
controversial measures of the degree of market orientation In a few
countries this indicator was modified.
7.30
A list of countries was developed where these two facts hold,
and tests were conducted using only such countries. Information on the
presence and timing of market-oriented reform was drawn largely from
a World Bank publication34 that documents and describes economic history in all developing countries. The publication is from 1996, which
guarantees that the judgments written about reform status of any particular country cannot possibly be influenced by either knowledge of
economic growth after 1996 in that country nor project performance of
any project in that country that closed after 1996, which covers most
projects approved after 1991. Countries were assigned reform status
and a year in which reforms commenced in cases where the evidence
52
CHAPTER 7
THE RISE ON RATES OF RETURN
was judged to be clear. Post-Soviet and Eastern European transition
countries were not assigned reform status until the first year in which
high inflation was stabilized in the 1990s or the cessation of a major
war.35 Bolivia was assigned reform status after 1985, the year in which
hyperinflation was stabilized. Other ostensible reform economies——
Chad, Côte d’’Ivoire, Ethiopia, and Lebanon——were excluded from the
tests if there was a major ongoing civil war or other conflict. El Salvador’’s reforms began in 1989, but it was not assigned full reform status
until after 1992, the year in which the peace accords were signed. The
full list of countries and the year assigned are presented in Annex Table
C1. For convenience this indicator will be referred to as the reform indicator even though in a few cases, such as El Salvador, reform is considered to be in effect only after reform and civil peace was achieved.
7.31
The impact of economic reforms on project rates of return is
first tested on a county-by-country basis: did returns rise in countries
after reforms occurred? To conserve space, the results are shown in
Table 8 for the first 20 countries (in alphabetical order). Annex Table
C2 contains results for all 47 countries.
Table 8. Country-by-Country Tests of the Impact of Economic Reforms on
Project Economic Returns
Mean rate of return for projects
completed:
Country
Algeria
Argentina
Bangladesh
Benin
Bolivia
Brazil
Bulgaria
Burkina Faso
Cape Verde
Chile
Colombia
Costa Rica
El Salvador
Gambia
Ghana
Guatemala
Guyana
Honduras
Hungary
India
Year of
reform
Before
reform
After
reform
Difference
1994
1991
1991
1989
1985
1990
1997
1994
1992
1976
1990
1982
1992
1985
1983
1994
1988
1992
1990
1991
19
12
21
41
9
19
24
21
14
15
15
17
14
18
21
28
4
15
35
18
23
25
36
33
30
46
49
39
41
30
36
65
19
23
34
27
18
24
20
24
4
13
16
-9
21
28
24
19
27
15
21
48
6
5
13
-1
14
9
-15
6
Statistically
significant
difference?
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Source: See text.
53
CHAPTER 7
THE RISE ON RATES OF RETURN
7.32
Nineteen of the 20 countries shown in Table 8 are found to
have higher average ERRs for projects conducted after reforms. Of
those 19 countries, the increase is also found to be statistically significant in 9 countries; and in the single country where returns were lower after reform, Bolivia, the decline is not statistically significant.
Some countries simply do not have enough projects with ERRs calculated to draw strong conclusions.
7.33
The full results (see Annex Table C2) indicate that of the 47
countries that have been identified as pursuing market-oriented reforms (and also have sufficient projects with economic return data), 43
are found to have had higher mean economic returns after reform. Only 4 exhibit lower returns, but in all such cases the sample sizes are
small or the differences are trivial and thus not statistically significant
(see Benin, Hungary, Lesotho, and the Russian Federation). In the 19
countries where there was sufficient data to draw statistically significant conclusions, all 19 showed higher returns after reform (see the final column in Annex Table C2). When all country results were pooled
together, mean returns rose by 14 percentage points after reform (17 to
31 percent) and the rise is easily statistically significant. Therefore this
country-by-country evidence supports the idea that the shift to market
orientation was a factor explaining higher mean returns.36
7.34
Did returns also rise in non-reforming countries? Clearly for
these countries there is no date of reform. Nevertheless, it turns out
that whatever year is selected for calculation yields similar conclusions. If 1991 is chosen to make the comparison, the mean returns
were 17 percent before this date and 22 percent after. So mean returns
were higher in the non-reform group, but by less than the reform
group. The difference-in-difference estimator in this case would ascribe an impact of 4.6 percentage points to liberalization ((27.5-17.4)——
(22-16.5)); lower than the before-after estimate of 10 percentage points
but still statistically significant.
7.35
A further test is to determine the influence of the explanations
against each other in a regression framework. Does market orientation account for the rise in returns over time even after controlling for
economic growth in each country? To conduct this test it was determined, for each World Bank project, the extent to which the project
was executed under market-oriented reform conditions. Projects that
spanned the reform year were assigned a fraction corresponding to
the percentage of the life of the project that was executed under
reform conditions. Then for each year between 1974 and 2007, the
fraction of all World Bank projects that closed in that year and were
implemented under reform conditions was calculated. The result is
the data series shown in Figure 20.
54
CHAPTER 7
THE RISE ON RATES OF RETURN
Figure 20. Percentage of Projects Completed in Countries that Implemented
Market-Oriented Policy Changes
Percent Reformed
1
.5
0
1974
1980
1987
1995
Year of Compl etion
2000
2007
Source: World Bank data.
7.36
As expected, this series increased strongly after 1987 and levels
off at approximately 80 percent after the year 2000.37 The impact of
this variable on project economic returns was tested in a regression
framework against the impact of economic growth and the bias indicator derived from the data in Figure 9. The results of the test indicate
(Table 9) that projects conducted in market-oriented environments ––
which typically have both appropriate policies and supporting regulatory institutions -- had economic returns that were 10 percentage
points higher than projects conducted in environments that were not
market-oriented. Projects conducted during faster economic growth
also had higher returns, with each percentage point increase in economic growth associated with a 1.5 percentage point rise in the economic rate of return. Together these three variables account for 83
percent of the variation in economic rates of return between 1974 and
2007.
Table 9. The Rise in Economic Returns Correlates Strongly with the Rise in
Market Orientation in Client Countries, 1974-2007
Variable
Economic growth
Market Orientation
Bias
R2 = 86 percent
Impact
1.5
10.0
3.1
N=33
S.E.
0.52
1.4
3.9
T-Ratio
2.9
7.7
0.8
Source: Author’s calculations using World Bank data
Note: The dependent variable is the median economic rate of return reported at the termination of projects
among all projects closing in a given year. The estimated impact of 1.5 associated with economic growth
suggests that a rise in growth of 1 percentage point was associated with a rise in economic returns of 1.5
percentage points. The estimated impact of 10.0 suggests that a shift to market-oriented policies was
associated with a rise in economic returns of 10.0 percentage points. The Bias variable is not significant.
55
CHAPTER 7
THE RISE ON RATES OF RETURN
7.37
Furthermore, focusing specifically on the explanation for the
large rise in returns between 1987 and 2007 (as opposed to the level of
returns), the rise in market orientation rather than rising growth accounts for most of the increase in returns, since the rise in the reform
variable was larger than the rise in growth during the period 19872007.38
7.38
The core evidence in the regression is summarized in Figure
21 and Figure 22. Economic returns in Bank projects correlate positively with the economic growth prevailing during project implementation, even after controlling for reform (Figure 21), but the positive
association between economic reform and project returns is strong,
even after controlling for the impact of economic growth (Figure 22).
Figure 21. Economic Returns in Projects Correlate Positively with Economic
Growth 1974-2007, Controlling for Reform
ERR (Dev iation f rom Mean)
Estimated Regression Line
2.9
-2.9
-1
Economic Growth
1.5
Source: Author’s calculations using World Bank data.
7.39
If it is correct that market-oriented reforms explain the rise in
returns, one might further expect to see little impact of reforms on exante returns (which are forecasts under best-case scenarios without
using actual observed results) compared with ex-post returns (which
use some data from actual results). The evidence does show little association between reform status and ex-ante economic returns. In the
period after 1987, mean economic returns at the beginning of projects
were virtually the same for reform countries compared with nonreform countries: 28 versus 27 percent. Hence the impact of reforms
shows up in the ex-post returns not in the ex-ante returns, consistent
with the idea that improved performance was driving higher returns.
56
CHAPTER 7
THE RISE ON RATES OF RETURN
Figure 22. Economic Returns in Projects Correlate Strongly with Economic
Liberalization, Controlling for Growth
ERR (Dev iation f rom Mean)
Estimated Regression Line
5
-5
-.5
Fraction Market-Oriented
.5
Source: Author’s calculations using World Bank data.
7.40
In summary, median economic rates of return have approximately doubled in a period of 20 years. Not only do the data on performance ratings corroborate the data on median economic rates of
return, they show further that most of the performance increases experienced by the World Bank are in the sectors that compute economic rates of return. A possible interpretation of this result is that sectors
that have a habit of quantifying results tend to learn to improve their
projects over time.
7.41
The chapter also runs tests to help determine whether and to
what degree three factors help explain the rise in median returns:
overall economic conditions (as measured by country growth), market-orientated approaches to project design and economic policies,
and possibly changing bias in measurement. The first two help explain the rise in project returns, accounting for 86 percent of the increase in returns. Statistical tests suggest that the quantitative impact
of market orientation is larger than that of growth; the shift to market
orientation that occurred in the late 1980s and 1990s is associated with
a 10 percentage point increase in economic returns of projects.
57
8. Conclusions
8.1
The World Bank has long required cost-benefit analysis in its
investment projects, but this review finds that Bank policy is not consistently implemented. Not only has the use of cost-benefit analysis
declined, even in sectors where it is typically applied, but numerous
shortcomings exist where it is applied, including upward biased information, poor compliance with Bank analytical standards, and limited use of cost-benefit results for learning or decision making.
8.2
A major review of cost-benefit analysis at the World Bank 20
years ago found many of the same shortcomings documented in this
report.39 That earlier report made numerous recommendations to
change Bank practice (see Box 6) noting problems such as lack of interest by higher management, low morale stemming from the belief
that better evidence would not alter decisions, poor incentives, and
lack of staff skills:
““Effectively implementing these recommendations will need
to go beyond the drafting of new guidelines. Ask any task
manager about project analysis, and the discussion quickly
turns to lack of management attention, staff incentives, and
perceived pressures to lend. Many staff feel that projects will
not be dropped even if the appraisal surfaces problems with a
project’’s viability. If the Bank is serious about improving
project quality: (1) managers will need to worry about the actual on the ground impact of investment operations; (2) the
Bank will need to provide effective support to project economists in securing appropriate skills, country parameters and
analysis, and the relevant lessons of experience for assigning
values to key parameters; and (3) chief economists, lead economists, and country economists will need to increase the attention they pay to project evaluation issues. ““
59
CHAPTER 8
CONCLUSIONS
Box 6. Recommendations of a 1992 Review of Cost-Benefit Analysis in the World Bank
An in-depth 1992 review of cost-benefit analysis concluded:
““This suggests that we have been lax in implementing some areas of the guidelines and that in some
areas the guidelines themselves need to be changed. The latter is easier. To this end, the Report’’s specific recommendations……are as follows:
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
Downgrade the prominence accorded to the theory of differential fiscal and distributional
weights, multiple conversion factors, and accounting rates of interest.
Upgrade the attention paid to realistic evaluations of project economic impact, based on the
lessons of experience with the country, the sector and the borrower.
Ensure that the macroeconomic, institutional, behavioral, and financial assumptions underlying the appraisal analysis are clearly spelled out.
Establish clear benefit standards——and success criteria——for the evaluation of projects not subject to an ERR test.
Ensure that a common methodological approach to evaluation obtains throughout the project
cycle——from identification through appraisal and implementation to completion and beyond.
Retain the expected ERR——or the alternative success measure——as the primary investment criterion, augmented by an assessment of the cumulative probability of an unsatisfactory outcome.
Use sensitivity analysis to test the impact of variations in key variables, and to identify appropriate proxy indicators for monitoring——and for reevaluating the project——during implementation.
Ensure that actions and parameters found to be critical for project viability are reflected in the
legal agreements.
Institute an indicator tracking system, with the indicators identified at appraisal used as a basis
for supervision ratings, which in turn trigger possible remedial action.
For sector investment operations, use the above approach for borrowers’’ evaluations of subprojects.
Launch studies to test the operational feasibility of (1) widening the coverage of economic costbenefit analysis of investment lending, to include the evaluation of policies and institutional reforms; (2) valuing (and including as project cost) deadweight losses associated with revenue
measures used to finance the project; (3) valuing the disutility of risk to be applied to the evaluation of large projects; and (4) calculating country-specific opportunity costs of capital.
Source: World Bank 1992.
8.3
The report stopped short of recommending institutional
changes, recommending instead:
x
x
x
Improved monitoring of portfolio quality
Providing institutional support for project economists
Involving the chief and lead economists.
8.4
Although the report’’s recommendation arguably contributed to
the establishment of the Quality Assurance Group (QAG), and may
have improved some quality dimensions, the evidence in this report is
that these recommendations were either not effectively implemented or
failed to solve the problems of biased analysis, poor compliance with
cost-benefit policy, and lack of feedback to and learning from costbenefit results in decisions on new projects. QAG’’s quality at entry rat60
CHAPTER 8
CONCLUSIONS
ings are performed after projects have been approved at the Board, and
therefore have no effect on approval decisions. Furthermore, the small
sample of projects reviewed by QAG is insufficient to draw conclusions
on quality by sector. As for the second two recommendations it is difficult to judge when they would be satisfactorily implemented.
8.5
Two institutional issues were not addressed by the recommendations of the earlier report. One is the finding in this study that management decisions to go ahead with projects are typically made before
cost-benefit information is available, which is likely to place pressure
on any subsequent analysis to conform to previous decisions. The
second is the potential conflict of interest, as cost-benefit analysis is
often conducted by, directed by, or commissioned by project managers with a professional interest in the outcome. They also select which
aspects of the analysis to report in project documents. Two further
findings in this study are the general lack of staff interest and, in some
cases, active staff opposition to the use of cost-benefit analysis for decision making.
8.6
Moving forward, the Bank needs to align actual practice with
policy in the area of cost-benefit analysis. To this end the evidence in
this report points to the need to change key practices at the Bank. IEG
also suggests that policy needs to be revised, but in specific ways so
as not to attenuate the clarity and strength of current policy.
8.7
Whatever reforms are adopted should strive to respect key
principles of evidence before decisions, transparency and unbiased
analysis:
x
x
x
Cost-benefit estimates should be available and used before decisions are made to go ahead with projects. They should influence decisions and be seen to influence decisions. This is critical, as the Bank should not fund projects with a negative net
present value and needs clear procedures in appraisal and supervision to address this risk.
The cost-benefit analysis should be summarized, preferably in a
single table, in each project appraisal document and implementation completion report. The table and corresponding text
should present the benefit and cost flows over time and the evidence or assumptions behind values for the benefit flows. The
spreadsheets behind the appraisal analysis should be saved for
review during implementation and final evaluation.
The cost-benefit estimates should follow high technical standards, use the best empirical evidence available, and represent
the expected or the most likely outcome. Responsibility for
funding, conducting, and directing the cost-benefit analysis
61
CHAPTER 8
CONCLUSIONS
should not lie mostly with those with vested interests in the
outcome.
8.8
The findings of this report point more specifically to the need
for reforms in the following four areas at the Bank.
Revisions to Bank policy
8.9
Bank policy requires some revision. It is important, however,
that any revision not overturn the basic strengths of current policy.
Current policy has a fundamental rationale, namely to determine
whether net benefits are positive in fulfillment of the Bank’’s mandate
in the articles of agreement to improve welfare. The wide application
of cost-benefit analysis also serves as a safeguard against the risk of
narrow political and sectional interests capturing project selection.
There is no alternative methodology on the horizon that can answer
the basic question whether benefits exceed costs other than quantitative cost-benefit analysis. Furthermore, the technical aspects of Bank
policy are basically sound. Any revision to Bank policy should not
diminish this basic strength and clarity.
8.10
At the same time, it is clear that Bank staff have difficulty implementing the policy as stated. This report has found that noncompliance with Bank policy is a function of several factors, including
lack of incentives, shortage of or neglect to collect appropriate data,
and lack of technical knowledge, all of which have nothing to do with
shortcomings in the policy. Part of the gap between policy and practice results from legitimate difficulty in implementing the policy in a
limited number of cases, as noted below.
The scope for cost-benefit analysis
8.11
The aspect of Bank policy that is most objectionable to the staff
is the expectation that either cost-benefit analysis or cost-effectiveness
analysis can be applied to 100 percent of interventions. The Bank policy cited at the beginning of this report applies to investment operations, not to policy loans, but some investment operations have policy
components where problems in applying traditional quantitative costbenefit analysis apply. Technical assistance operations are also often
not amenable to traditional cost-benefit analysis. The Bank needs to
define the scope for cost-benefit analysis in a way that recognizes the
legitimate difficulties in quantifying benefits while preserving a high
degree of rigor in justifying projects. Policy on which projects can
seek justification through cost-effectiveness rather than a cost-benefit
standard should be carefully defined and limited in scope, given that
cost-effective projects can still have negative net present values. Alternative standards for justifying projects may be required where traditional cost-benefit analysis cannot be applied. But these should be
62
CHAPTER 8
CONCLUSIONS
made explicit. Given improvements in technical methods and data in
recent years, there is likely to be scope for innovation.
Comprehensive analysis
8.12
Cost-benefit analysis is not a stand-alone, but part of a larger
analytical effort to appraise and evaluate projects. The crucial issue is
not solely whether a cost-benefit analysis is done but also whether the
reasoning motivating the project is analytically sound and supported
by credible evidence.
8.13
Current Bank policy does not contradict this point, but it does
not affirm it either. There are separate guidance documents for costbenefit analysis, developing results frameworks, defining development objectives, and monitoring and evaluation. And there is no formal policy on impact evaluation. This fragmentation in the guidance
documents is paralleled by fragmentation in project appraisal. In staff
interviews it became clear that groups perform these different aspects
of appraisal with little communication between groups. The revision
to Bank policy should recognize the mutually-reinforcing nature of
these appraisal activities, and move toward appraisal becoming one
complete and integrated analytical exercise.
8.14
It is important to clarify that comprehensive analysis does not
necessarily entail time-consuming analysis. The optimal amount of time
and resources to devote to cost-benefit analysis can range from hours to
many months. Sometimes the required information is readily available;
sometimes it is worth investing the time to collect better information. It
all depends on the nature of the project and the constraints to improved
information. Moreover, efficiencies can be achieved by engaging
groups like the Development Economics Group at the Bank (DEC) to
perform the background research and required household surveys and
to establish acceptable ranges for key parameters.
Objectivity and use in decision-making
8.15
Objectivity of the cost-benefit estimates is paramount for effective learning, decision-making, and improving results. Current Bank
policy is silent on procedures to ensure objectivity and who should
perform cost-benefit analysis and under what circumstances. There is
a potential conflict of interest when those responsible for cost-benefit
analysis at appraisal are also responsible for promoting a project
through Board approval.
8.16
Whatever policy is adopted needs to confront the related issues
of objectivity and client ownership. The Bank’’s clients propose and implement projects, but both the Bank and its clients have an interest in
ensuring objectivity. The issues in ensuring objectivity apply whether
63
CHAPTER 8
CONCLUSIONS
appraisal is done by the borrowing country or with the assistance of the
Bank. The three basic options for reinforcing objectivity range from (1)
an auditing model, where the project task team performs the analysis
but the conclusions are subject to independent and random audits, to
(2) a model in which the analytical work is separated from the project
task teams, to (3) a model in which different groups are explicitly given
adversarial roles promoting or criticizing projects, as in legal proceedings.40
8.17
Active use of empirical evidence can also be used to promote
objectivity. Audited or verified records of cost-benefit results actually
achieved from previous projects can be used to reinforce realism in
appraisal of new projects. One specific aspect of Bank practice requiring review is the finding, based on staff interviews, that results from
cost-benefit analysis at appraisal are frequently lost when the costbenefit analysis at closing is prepared. Furthermore, the cost-benefit
analyses for completed projects are rarely used as evidence in costbenefit analysis for new similar projects.
8.18
Disclosure and transparency are further methods to reinforce
objectivity. Currently, it is impossible to replicate the cost-benefit
analysis based on the information in appraisal and completion reports. Disclosing the spreadsheet used would immediately solve this
problem, particularly if disclosed in a way that highlighted the key
assumptions and parameters in the cost-benefit exercise.
8.19
It is crucial that the analysis and key evidence are available
before key decisions are made and be seen to influence those decisions. Current practice at the Bank often seems to be the opposite: decisions constrain the analysis. Bank operational policy currently has
no effective safeguards in place to prevent this from happening.
8.20
Ensuring objectivity and getting the right information available
at the right time in the decision process are interconnected issues. Costbenefit analysis needs to be produced earlier and quicker than is currently done, so that it can be reviewed before substantial resources and
time are committed to the project. Bank project teams already produce
a concept note for each project early in the decision process. To focus on
a serious discussion early in the project cycle, before a CBA can be developed, the Bank could clarify the required content of the note, ensuring that it sets out the public sector rationale for the project and a brief
description of the project that shows how it addresses the stated rationale. This would focus attention on the broader economics of the
project and not just the CBA. The concept note would also include the
proposed method of evaluation (CBA, cost-effectiveness, or other methods). Costs and benefits would have to be clearly described even if
they could not all be quantified, and the note would also include a de-
64
CHAPTER 8
CONCLUSIONS
scription of the data required for analysis before Board approval, including any need to collect new data.
8.21
In sum, this report suggests a number of complementary steps
to enhance the usefulness of CBA in the Bank: revising Bank policy on
use of cost-benefit analysis to clarify applicability and methods; better
integrating the various types of analysis (CBA, M&E frameworks, impact evaluation) undertaken as part of project appraisal; clarifying the
required content and process for review of project concept notes; and
considering institutional alternatives to ensure the objectivity and
transparency of CBA. Cost-benefit analysis can be a powerful tool
when appropriately applied, and taken together these steps could considerably enhance the overall results focus of Bank lending.
65
Annex A
Annex Table A1 shows the number and percentage of all investment projects at the World
Bank classified by Sector Board for two five-year periods: 1975-79 and 2003-07. As can be seen,
some Sector Boards have relatively few projects: Economic Policy, Gender and Development,
Private Sector Development, and Social Development. Another Sector Board, Global Information/Communications Technology, has few projects for the period 2003-07. The financial sector has been renamed and subsumed into Financial and Private Sector Development. The 11sector classification used in Table 1 of the text is obtained by grouping the small sectors into
““Other”” and combining the Financial Sector with Financial and Private Sector Development.
Annex Table A1. World Bank Projects by Sector Board, 1975-9 Compared with 2003-7
1975-79
Agriculture and Rural Development
Economic Policy
Education
Energy and Mining
Environment
Financial Sector
Financial and Private Sector Development
Gender and Development
Global Information/Communications Technology
Health, Nutrition and Population
Private Sector Development
Public Sector Governance
Social Development
Social Protection
Transport
Urban Development
Water
Total
67
2003-07
Number
Percent
Number
Percent
165
1
50
96
0
60
0
0
33
4
1
7
0
0
162
9
22
610
27.0%
0.2%
8.2%
15.7%
0.0%
9.8%
0.0%
0.0%
5.4%
0.7%
0.2%
1.1%
0.0%
0.0%
26.6%
1.5%
3.6%
100.0%
190
12
133
83
82
0
97
1
10
109
0
78
22
80
121
73
71
1162
16.4%
1.0%
11.4%
7.1%
7.1%
0.0%
8.3%
0.1%
0.9%
9.4%
0.0%
6.7%
1.9%
6.9%
10.4%
6.3%
6.1%
100.0%
Annex B.
Annex Table B 1. Relation between Project Ratings by the Independent Evaluation Group and Economic
Rates of Return, 1987-2008
Economic rate of return of project
IEG final rating:
Highly unsatisfactory
Unsatisfactory
Moderately unsatisfactory
Moderately satisfactory
Satisfactory
Highly satisfactory
Total
68
10
percent
or
lower
Between
10 and
14
Between
14 and
19
Between
19 and
25
Between
25 and
35
Higher
than
35
Total
number
of
projects
9
154
18
40
63
1
3
39
12
51
130
11
0
22
10
29
180
11
0
16
8
48
182
15
1
11
8
35
162
17
1
13
7
44
153
31
14
255
63
247
870
86
285
246
252
269
234
249
1,535
Annex C
Annex Table C1. List of Countries and Reform Dates Assigned
Country
Indonesia
Malaysia
Botswana
Chile
China
Belize
Costa Rica
Mauritius
Ghana
Thailand
Bolivia
Gambia, The
Lao People’s Democratic Republic
Tanzania
Tunisia
Guyana
Mexico
Benin
Jordan
Sri Lanka
Vietnam
Brazil
Colombia
Hungary
Mongolia
Peru
Philippines
Poland
Argentina
Bangladesh
Czech Republic
India
Nepal
Ethiopia
Albania
Cape Verde
El Salvador
Estonia
Honduras
Latvia
Mozambique
Paraguay
Uganda
Mauritania
Reform date
assigned
Before 1975
Before 1975
Before 1975
1976
1978
1981
1982
1982
1983
1983
1985
1985
1986
1986
1986
1988
1988
1989
1989
1989
1989
1990
1990
1990
1990
1990
1990
1990
1991
1991
1991
1991
1991
1991
1992
1992
1992
1992
1992
1992
1992
1992
1992
1992
69
ANNEX C
Country
Lebanon
Guinea-Bissau
Kazakhstan
Lithuania
Madagascar
Moldova
Nicaragua
Slovenia
Burkina Faso
Cambodia
Georgia
Guatemala
Kyrgyz Republic
Mali
Senegal
Slovak Republic
Côte d'Ivoire
Algeria
Armenia
Azerbaijan
Jamaica
Kenya
Romania
Russian Federation
Tajikistan
Ukraine
Uruguay
Chad
Lesotho
Bulgaria
70
Reform date
assigned
1992
1993
1993
1993
1993
1993
1993
1993
1994
1994
1994
1994
1994
1994
1994
1994
1994
1994
1995
1995
1995
1995
1995
1995
1995
1995
1995
1995
1996
1997
ANNEX C
Annex Table C2. Before-After Tests of Impact of Market-oriented Reforms on
Economic Returns
Average rate of return (%) for
projects:
Country
Algeria
Argentina
Bangladesh
Benin
Bolivia
Brazil
Bulgaria
Burkina Faso
Cape Verde
Chile
Colombia
Costa Rica
El Salvador
Gambia, The
Ghana
Guatemala
Guyana
Honduras
Hungary
India
Jamaica
Jordan
Kenya
Lao People's Democratic Rep.
Lesotho
Madagascar
Mali
Mauritania
Mauritius
Mexico
Mozambique
Nepal
Nicaragua
Paraguay
Peru
Philippines
Romania
Russian Federation
Senegal
Number of
projects:
Year of
reform
Before
After
Change
Before
After
1994
1991
1991
1989
1985
1990
1997
1994
1992
1976
1990
1982
1992
1985
1983
1994
1988
1992
1990
1991
1995
1989
1995
1986
1996
1993
1994
1992
1982
1988
1992
1991
1993
1992
1990
1990
1995
1995
1994
19
12
21
42
15
18
24
21
14
15
15
16
17
18
14
24
10
15
35
20
13
17
14
5
11
13
18
6
9
19
9
13
13
23
16
18
13
75
18
23
25
36
33
30
46
49
39
41
30
36
65
19
23
34
27
18
24
20
24
21
25
30
27
12
36
43
18
27
33
37
40
43
20
35
37
39
43
43
4
13
16
-10
15
28
24
19
27
15
21
48
2
5
20
3
7
9
-15
4
8
9
17
22
0
24
25
12
18
14
29
27
30
-4
19
18
26
-32
25
14
13
53
15
16
88
2
18
1
3
54
14
6
8
18
10
6
21
11
160
19
22
42
5
6
26
24
7
6
40
1
28
8
10
16
51
31
4
30
1
8
16
3
8
25
2
3
1
13
6
2
2
1
22
2
2
3
5
41
2
8
3
11
1
5
3
2
4
15
3
6
4
2
5
18
6
2
5
Statistically
significant
change?
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
71
ANNEX C
Average rate of return (%) for
projects:
Country
Slovenia
Sri Lanka
Tanzania
Thailand
Tunisia
Uganda
Uruguay
Vietnam
All Countries
Number of
projects:
Year of
reform
Before
After
Change
Before
After
1993
1989
1986
1983
1986
1992
1995
1989
-
7
15
7
18
21
15
16
8
17
28
30
21
22
25
40
27
34
31
20
15
14
4
4
25
11
26
14
2
26
35
43
38
16
15
1
1083
1
8
9
25
12
6
4
14
350
Statistically
significant
change?
Yes
Yes
Yes
Note: A project was coded before reform if some part of the implementation occurred before the reform date. A project was coded
after reform only if all of the implementation occurred after the reform date.
72
Annex D. OP 10.04—Economic
Evaluation of Investment Operations
These policies were prepared for use by World Bank staff and are not
necessarily a complete treatment of the subject.
OP 10.04
September, 1994
Note: OP 10.04 replaces the version dated April 1994. Please retain the April 1994 BP
10.04. OP and BP 10.04 are complemented by OP/BP10.00, Investment Lending: Identification
to Board Presentation. OP and BP 10.04 replace the Operational Memorandum Treatment of
Environmental Externalities in the Evaluation of Investment Projects, 10/4/93, and draw on the
following documents: OMS 2.20, Project Appraisal; OMS 2.21, Economic Analysis of Projects;
OPN 2.01, Investment Criteria in Economic Analysis of Projects; OPN 2.02, Risk and Sensitivity
Analysis in the Economic Analysis of Projects; OPN 2.04, Economic Analysis of Projects with
Foreign Participation; OPN 2.05, Foreign Exchange Effects and Project Justification; OPN 2.06,
Use of the Investment Premium and Distribution Weights in Project Analysis; OPN 2.07, Reporting and Monitoring Poverty Alleviation Work in the Bank; and OPN 2.09, Presentation of Project
Justification and Economic Analysis in Staff Appraisal Reports. Additional guidance on project
economic evaluation is provided in Handbook on Economic Analysis of Investment Operations. Questions may be addressed to opmanual@worldbank.org.
1. The Bank1evaluates investment projects to ensure that they promote the
development goals of the borrower country. For every investment project,
Bank staff conduct economic analysis to determine whether the project
creates more net benefits to the economy than other mutually exclusive options for the use of the resources in question.2
Criterion for Acceptability
2. The basic criterion for a project's acceptability involves the discounted
expected present value of its benefits, net of costs. Both benefits and costs
are defined as incremental compared to the situation without the project. To
be acceptable on economic grounds, a project must meet two conditions: (a)
the expected present value of the project's net benefits must not be negative;
and (b) the expected present value of the project's net benefits must be
higher than or equal to the expected net present value of mutually exclusive
project alternatives.3
Alternatives
3. Consideration of alternatives is one of the most important features of
proper project analysis throughout the project cycle. To ensure that the
project maximizes expected net present value, subject to financial, institutional, and other constraints, the Bank and the borrower explore alternative,
mutually exclusive, designs. The project design is compared with other designs involving differences in such important aspects as choice of beneficiaries, types of outputs and services, production technology, location, starting
date, and sequencing of components. The project is also compared with the
alternative of not doing it at all.
73
ANNEX D
Nonmonetary Benefits
4. If the project is expected to generate benefits that cannot be measured in
monetary terms, the analysis (a) clearly defines and justifies the project objectives, reviewing broader sectoral or economywide programs to ensure that
the objectives have been appropriately chosen, and (b) shows that the
project represents the least-cost way of attaining the stated objectives.
Sustainability
5. To obtain a reasonable assurance that the project's benefits will materialize as expected and will be sustained throughout the life of the project, the
Bank assesses the robustness of the project with respect to economic, financial, institutional, and environmental risks. Bank staff check, among other
things, (a) whether the legal and institutional framework either is in place or
will be developed during implementation to ensure that the project functions
as designed, and (b) whether critical private and institutional stakeholders
have or will have the incentives to implement the project successfully. Assessing sustainability includes evaluating the project's financial impact
on the implementing/sponsoring institution and estimating the direct effect on
public finances of the project's capital outlays and recurrent costs.
Risk
6. The economic analysis of projects is necessarily based on uncertain future events and inexact data and, therefore, inevitably involves probability
judgments. Accordingly, the Bank's economic evaluation considers the
sources, magnitude, and effects of the risks associated with the project by
taking into account the possible range in the values of the basic variables
and assessing the robustness of the project's outcome with respect to
changes in these values. The analysis estimates the switching values of key
variables (i.e., the value that each variable must assume to reduce the net
present value of the project to zero) and the sensitivity of the project's net
present value to changes in those variables (e.g., delays in implementation,
cost overruns, and other variables that can be controlled to some extent). The main purpose of this analysis is to identify the scope for improving
project design, increase the project's expected value, and reduce the risk of
failure.
Poverty
7. The economic analysis examines the project's consistency with the
Bank's poverty reduction strategy.4 If the project is to be included in the Program of Targeted Interventions, the analysis considers mechanisms for targeting the poor.
Externalities
8. A project may have domestic, cross-border, or global externalities.5 A
large proportion of such externalities are environmental. The economic evaluation of Bank-financed projects takes into account any domestic and crossborder externalities. A project's global externalities—normally identified in
the Bank's sector work or in the environmental assessment process—are
considered in the economic analysis when (a) payments related to the
74
ANNEX D
project are made under an international agreement, or (b) projects or project
components are financed by the Global Environment Facility.6 Otherwise,
global externalities are fully assessed (to the extent tools are available) as
part of the environment assessment process7and taken into account in
project design and selection.8
____________
1.
2.
"Bank" includes IBRD and IDA, and "loans" includes IDA credits and IDA grants.
All flows are measured in terms of opportunity costs and benefits, using "shadow prices,"
and after adjustments for inflation.
3. Although it has long been the Bank's policy to calculate the expected net present value,
standard practice has been to calculate the expected internal rate of economic return, that
is, the rate of discount that results in a zero expected net present value for the project. The
expected rate of return is not fully satisfactory (e.g., when comparing mutually exclusive
project alternatives); however, it is widely understood and may continue to be used for the
purpose of presenting the results of analysis.
4. See OP 1.00, Poverty Reduction.
5. "Cross-border externalities" are effects on neighboring countries (e.g., effects produced by
the construction of a dam on a river). "Global externalities" affect the entire world (i.e.,
emissions of greenhouse gases or ozone-depleting substances, pollution of international
waters, or impacts on biodiversity).
6. See OD 9.01, Procedures for Investment Operations under the Global Environment Facility
(to be reissued as OP/BP 10.20).
7. See OP/BP 4.01, Environmental Assessment.
8. The Bank's Environment Department provides guidance on analyzing, ranking, and physically quantifying environmental externalities—whether domestic, cross-border, or global—and on
taking them into account in project design and selection.
75
References
Adler, M.D., and E.A. Posner (eds.). 2001. Cost-Benefit Analysis: Legal, Economic, and Philosophical Perspectives. Chicago and London: University of
Chicago Press.
Beier, G. 1990. ““Economic Analysis in Project Appraisal.”” Discussion Paper.
Washington, DC: World Bank. Cited in Salop (1992).
Belli, P., and R. Pablo Guerrero. 2009. ““Quality of Economic Analysis in Investment Projects Approved in 2007 and 2008.”” Independent Evaluation Group. Washington, DC: World Bank.
Deininger, K., L. Squire, and S. Basu. 1998. ““Does Economic Analysis Improve the Quality of Foreign Assistance?”” World Bank Economic Review 12(3): 385-418.
Devarajan, S., L. Squire, and S. Suthiwart-Narueput. 1995. Reviving Project
Appraisal at the World Bank. Policy Research Working Paper 1496.
Washington, DC: World Bank.
Devarajan, S., L. Squire, and S. Suthiwart-Narueput. 1997. ““Beyond Rate of
Return: Reorienting Project Appraisal.”” World Bank Research Observer
12(1): 35-46.
Devarajan, S., A. S. Rajkumar, and V. Swaroop. 1999. What Does Aid to Africa
Finance? Policy Research Working Paper 2092. Washington, DC:
World Bank.
Dollar, D., and V. Levin. 2005. Showing and Preparing: Institutional Quality and
Project Outcomes in Developing Countries. World Bank Policy Research
Working Paper 3524. Washington, DC: World Bank.
Hammer, Jeffrey S. 1997. ““Economic Analysis for Health Projects””. World
Bank Research Observer 12(1): 47-72.
Garcia-Garcia, J. 2008. ““Economic Analysis in World Bank Financed
Projects.”” Washington, DC: World Bank. Mimeo.
Hirschman, A.O. 1967. Development Projects Observed. Reissue with a new
preface by the author, 1995. Washington, DC: The Brookings Institution.
Isham, J., D. Kaufmann, and L. Pritchett. 1997. ““Civil Liberties, Democracy,
and the Performance of Government Projects.”” World Bank Economic
Review 11(2): 219-42.
Isham, J., and D. Kaufmann. 1999. ““The Forgotten Rationale for Policy
Reform: The Productivity of Investment Projects.”” Quarterly Journal
of Economics 114(1): 149-184.
Jimenez, E., and H.A. Patrinos. 2008. Can Cost-Benefit Analysis Guide Education
Policy in Developing Countries? Policy Research Working Paper 4568.
Washington DC: World Bank.
76
REFERENCES
Kapur, D., J.P. Lewis, and R. Webb (eds.). 1997. The World Bank: Its First Half
Century, Volumes 1&2: Perspectives. Washington, DC: Brookings Institution Press.
Kaufmann, D., and Y. Wang. 1992. How Macroeconomic Policies Affect Project
Performance in the Social Sectors. World Bank Policy Research Working Paper 939. Washington, DC: World Bank.
Kaufmann, D., and Y. Wang. 1995. ““Macroeconomic Policies and Project Performance in the Social Sectors: A Model of Human Capital Production and Evidence from LDCs.”” World Development 23(5): 751-765.
Kilby, C. 1995. ““Supervision and Performance: The Case of World Bank
Projects.”” Center for Economic Research. Discussion paper 45. Tilburg University, Tilburg, the Netherlands, and Vassar College,
Poughkeepsie, NY.
Lee, J., and Zhao Guangbin. 2006. ““People’’s Republic of China: Toll Roads
Corporatization Strategy——Towards Better Governance.”” Meyrcik
and Associates, Australia. Winlot Consulting Ltd., Beijing, PRC.
Asian Development Bank Technical Assistance Consultant’’s Report.
Project Numbers: RSC-C51814 & RSC-C60519.
Little, Ian, and J. A. Mirrlees. 1990. ““Project Appraisal and Planning Twenty
Years On.”” In Stanley Fischer, Dennis de Tray, and Shekhar Shah,
(eds.), Proceedings of the World Bank Annual Conference on Development Economics. Washington, DC: World Bank.
Meerman, Jacob. 1997. Reforming Agriculture: The World Bank Goes to Market.
Washington, DC: World Bank.
Morley, S., and D. Coady. 2003. From Social Assistance to Social Development:
Targeted Education Subsidies in Developing Countries. Washington, DC:
Center for Global Development: International Food Policy Research
Institute.
Pohl, G., and D. Mihaljek. 1992. ““Project Evaluation and Uncertainty in Practice: A Statistical Analysis of Rate-of-Return Divergences of 1,015
World Bank Projects.”” World Bank Economic Review 6(2): 255-277.
Salop, J. 1992. ““Economic Analysis of Projects: Towards a Results-oriented
Approach to Evaluation.”” Discussion Draft. Washington, DC: World
Bank.
Squire and Van der Tak. 1975. Economic Analysis of Projects. Published for the
World Bank by John Hopkins University Press.
Verbeek, J. 1999. The World Bank’’s Unified Survey projections: how accurate are
they? An ex-post evaluation of US91-US97 (Policy Research Worlking
Paper Series, no 2071). Washington, DC: World Bank.
Ward, W.A., B.J. Deren, and E.H. D’’Silva. 1991. The Economics of Project Analysis: A Practitioner’’s Guide. Economic Development Institute Technical Materials. Washington, DC: World Bank.
World Bank 1988. Annual Review of Project Performance Results for 1987. Operations Evaluation Department. Washington, DC: World Bank.
77
REFERENCES
World Bank. 1988. Rural Development: World Bank Experience, 1965-86. Washington, DC: World Bank.
World Bank. 1992. Effective Implementation: Key to Development Impact. Report
11536. (Also known as the Wapenhans Report.) Washington, DC:
World Bank.
World Bank. 1996. Trends in Developing Economies. Washington, DC: World
Bank.
World Bank. 1997. ““A Review of the Quality of Economic Analysis in Staff
Appraisal Reports for Projects Approved in Fiscal Year 1996 and Preliminary Report for Projects Approved in the First Half of Fiscal Year
1997.”” Washington, DC: World Bank.
World Bank. 2009. Operations policy and Country Services: ““Moving Ahead
on Investment Lending Reform: Risk Framework and Implementation Support. September 3, 2009.
78
Notes
1. Articles of Agreement of the International Bank for Reconstruction and Development, July 22, 1944.
2. For example, a major conference of World Bank borrowers in 1992 concluded that borrowers ““want reduced Bank involvement in preparation and
design, but the same insistence on rigorous analysis that the Bank used to expect, since only in that way will they develop the capability for independent
project development.”” (World Bank 1992, Annex B, page 14.)
3. World Bank Operations Manual, section 10.04, September 1994.
4. To clarify, the elements outlined in the chapter on Bank policy define what a
complete cost benefit analysis would entail according to Bank policy. Although this evaluation will assess the degree to which Bank policy is followed
in its entirety as outlined in that chapter, it will also assess the extent to which
Bank project analysis contains any cost-benefit analysis, complete or not. For
this latter exercise cost-benefit analysis is defined simply as any quantitative
analysis to establish whether B>C, where B and C are the present value of benefits and costs. The presence of such analysis is indicated by the reporting of a
NPV or an ERR calculation in the Banks project database. Bank staff sometimes uses the terms cost-benefit analysis to refer to elaborate analysis that
employs shadow prices, or that follows techniques outlined by Squire and Van
der Tak (1975). The application of such techniques is not however the defining
characteristic of cost benefit analysis as the term is used in this evaluation.
5 A World Bank internal document claims that 44 percent of appraisal reports
for investment operations during 2008-2010 contain an ERR forecast covering
all components, and 10 percent contain one for some components. IEG has
not vetted this data because these projects are not completed, but if confirmed it would indicate a rise in the proportion of new projects with ERR
forecasts.
6. The economic rates of return at closing are called estimates rather than forecasts even though they still involve some forecasting. A typical ERR is based
on benefit flows lasting 20 or 30 years, so ERRs calculated just after projects
terminate, seven years after commencement on average, still contain a large
degree of forecasting.
7. The main document at the inception of projects was previously known as
the Staff Appraisal Report but is now the Project Appraisal Document. Similarly, the major document at project termination was previously known as the
Project Completion Report but is now the Implementation Completion Report.
Both World Bank and IEG information systems record project information at
completion.
8. Other multilateral donors such as the Asian Development Bank or the InterAmerican Development Bank and multilateral institutions such as the European Union provide instructions and guidelines for conducting cost-benefit
analysis, as does the World Bank. It is difficult to judge precisely from the web
79
NOTES
sites what the de-jure and de-facto policies are in terms of applying and using
cost-benefit analysis. The donor agency that has the most rigorous accountability standards and uses cost-benefit analysis most extensively is the Millennium Challenge Corporation, which makes its policy of requiring a CBA clear
on its web site, and also has a policy of conducting rigorous impact evaluations and using this information in the future to improve cost-benefit assessments. The Corporation’’s investment committee requires that ex-ante costbenefit evidence is presented to it before it makes funding decisions, and has
rejected projects on cost-benefit grounds. Three countries that are known for
using cost-benefit information in domestic policy decisions are China, Chile,
and South Korea.
9. A related study by IEG reports results of a search of Bank project documents
for words frequently associated with cost-benefit analysis, such as ““net present
value”” or ““discount rate”” and typically finds that less than 50 percent of the
documents contain such terms. This supports the picture of low use of costbenefit analysis. In contrast, 90 percent of documents contain the term ““environmental analysis.”” See Garcia-Garcia (2008) page 23.
10. The decomposition is chosen to ensure that the parts exactly sum to the
total. The contribution of the change within the high-CBA sectors is computed
as (.79-.61)*.86, the contribution of the change within the low-CBA sectors is
computed as (0.06-0.0)*(1-.86), and the contribution from the shift away from
projects in the high-CBA sectors is computed as (0.44-0.86)*(0.61-0.06).
11. The term ““cost-benefit analysis”” refers specifically to a cash flow analysis in
which the major benefit and cost flows are quantified over time. The results of
this analysis are typically summarized by the presentation of an ERR or an
NPV calculation. More generally the terms cost-benefit analysis or assessment
may be understood to refer to any attempt to estimate benefits in monetary
terms so that they can be compared with costs. Cost-effectiveness analysis
refers to analysis comparing the costs of alternative methods of achieving a
given goal.
12. In private correspondence, Bank management has claimed that newly approved projects in 2009-10 have both a higher rate of baseline data collection
and presentation of ex-ante ERRs than cited here. IEG has not yet had the
chance to independently verify these data. IEG normally evaluates only completed projects.
13. See World Bank: Operations policy and Country Services: ““Moving Ahead
on Investment Lending Reform: Risk Framework and Implementation Support. September 3, 2009. Paragraph 7 page 2.
14. See Belli and Guerrero 2009. The $100 million threshold was chosen to ensure that the sample of projects was one for which the monetary value of good
analysis would not be in doubt.
15. See World Bank 1992, page ii.
16. See Beier 1990.
17. The relative neglect of the public rationale for projects is stressed in Devarajan, Squire, and Suthiwart-Nareput 1997.
18. Little and Mirrlees 1990.
80
NOTES
19. See World Bank 1988.
20. ““The Gap between Economic Rates of Return (ERR) Estimates at Appraisal
and Completion, and Project Risk Analysis.”” SecM89-819, June 26, 1989.
21. Salop 1992, page 2.
22. World Bank 1992, page iii.
23. World Bank 1992, page 9.
24. Salop 1992, page 29.
25. A test of difference in means yields a t-ratio of 2.0, which is just statistically
significant at the 5 percent level.
26. Some evidence suggests that the World Bank’’s Unified Survey forecasts
(from 1991-1997) do not exhibit upward bias, but this is not inconsistent
with the points made here. (see Verbeek)
27. See http://lehmanreport.jenner.com/VOLUME%201.pdf, page 8.
28. This chapter draws on the results of the special survey written by Domenico Lombardi.
29. The four pieces of information suggest a constant but not increasing bias
over time. Looking at a sector, Agriculture, in which data are available for
long periods, it is evident that the likelihood of recalculating ERRs at closing
has remained fairly constant over time at slightly under 80 percent. These results do not suggest that ERR recalculation has deteriorated over time. The
review of the analytical quality of the Bank’’s ex-ante economic analysis conducted for this report finds that quality was roughly the same in 2007-08 as it
was 10 years earlier——again no indication of a significant change in standards
over time. Probably more importantly, the structural issues that serve to underpin bias——the facts that decision making proceeds analysis and that project
leaders are entrusted with funding the cost-benefit analysis——remain in force.
Finally, the finding earlier in this report that downside risks are systematically
ignored in ex-ante economic analysis of projects was reported 20 years ago,
again suggesting that the practice has continued uninterrupted over time.
30. The expectation is that the IEG ratings and the economic rates of returns
would be positively but not perfectly correlated. IEG ratings take into account
efficiency, effectiveness and relevance whereas the ERRs measure one of these.
31. Specifically, the rate of real economic growth prevailing in the country during the execution of the project was calculated for each World Bank project that
reported an economic rate of return at closing. Mean growth (unweighted)
across all projects closing in a given year was then calculated. Figure 8 depicts
the centered three-year average of these growth rates, for IDA and non-IDA
countries separately. Mean economic growth weighted by project commitments (in U.S. dollars) was also calculated and shows a similar pattern over
time.
32. A 10 percent random sample of projects that closed in 1979 and 1984 was
examined for this purpose.
33. See Meerman 1997, page 1.
81
NOTES
34. See World Bank 1996.
35. Notable countries so affected include Bulgaria, Romania, the Russian Federation, and Ukraine. Bulgaria achieved its inflation stabilization in 1997, Romania, Russia, and Ukraine in 1995. The reform years for Armenia (1995),
Azerbaijan (1995), and Georgia (1994) were delayed not only for hyperinflation
conditions in the immediate transition period but also due to war and civil
unrest.
36. Isham and Kaufmann (1999), using World Bank economic return data from
the period 1974-90, showed that rates of return were higher for projects undertaken in less-distorted economic policy environments. World Bank ERRs were
inversely associated with the premium on the exchange rate in the black market and international trade restrictions. Given that their data ended in 1990
they could not have known or analyzed the subsequent rise in returns, but
their result that ERRs rose in countries for which the black market premium
fell below 30 percent is consistent with the evidence shown here.
37. Several checks have been conducted on this data: varying the date of
reform one or two years and dropping countries in which the evidence could
be construed to be equivocal. These cause minor changes to the data but do
not alter the major fact of a large rise in the reform percentage of projects after
1987. The conclusions derived from the regression results are not materially
altered by these changes.
38. The product of the change in growth between 1988 and 2007 and its estimated coefficient is 3.75; the same calculation for economic liberalization is
6.70. Hence the economic liberalization variable accounts for approximately 70
percent of the rise in economic returns. There may be further causality from
liberalization to growth, which this calculation implicitly ignores, but allowing
for this in the analysis would only strengthen the estimated impact of liberalization.
39. Salop 1992
40. It may be helpful to recall that the independent auditor model has failed to
prevent misleading reporting in several important cases in the private sector.
On March 11 2010, a court-appointed examiner’’s report investigating the Lehman Brother’’s bankruptcy found that ““Lehman’’s auditors, Ernst & Young,
were aware of but did not question Lehman’’s use and nondisclosure of the
Repo 105 accounting transactions.”” This accounting procedure made Lehman’’s
leverage appear lower than was really the case. See
http://lehmanreport.jenner.com/VOLUME%201.pdf, page 8.
82