Estimates in Project Evaluation – Time for a New Approach ?
1
Introduction
1.1.1
This paper provides a review of the literature on project cost overruns and time
delays in major infrastructure projects.
1.1.2
The phenomenon of ‘optimism bias’ in which major infrastructure projects
consistently and significantly run over budget in terms of time and money has
been extensively examined. Much of the literature has focused on the explaining
the overruns in terns of individual cost assumptions such as inflation, raw
material costs or changes in project scope.
1.1.3
More recent work undertaken by Professor Flyvberg and colleagues has focused
on projects as ‘political’ events subject to a wide range of external factors such
as personal incentives which can drive over runs
1.1.4
This paper draws extensively on work undertaken by RGL, Frontier Economics
and Aecom consulting in their work for the European Commission on evaluating
ERDF projects.1
1.2 Time delays and cost overruns are a significant problem in infrastructure
projects
1.2.1
Cost overruns and delays occur because of inaccurate estimates leading to
higher actual costs, and longer construction periods, than anticipated. Numerous
studies have identified a systematic tendency for large infrastructure projects to
have significant overruns and delays. This tendency of projects to overrun has
been termed project optimism bias.
1.2.2
For example, Flyvbjerg et al (2003) found that 9 out of 10 projects, from a sample
of 258, across 20 countries and 5 continents, were subject to cost overruns.2 In a
recent study for the Commission, cost overruns in 51 projects were assessed
which showed cost overruns averaging 21%, as illustrated in Table 1 below.
Ex Post Evaluation Of Cohesion Policy Programmes 2000-2006, Work Package 10, ‘Efficiency: Unit costs
of major projects’ Final Report, 25 October 2009 RGL Forensics, Frontier Economics, Aecom
2 See Flyvbjerg, Bent, Mette Skamris Holm and Soren Buhl (2003), ‘How common and how large are cost
overruns in transport infrastructure projects?’ Transport Reviews, Vol. 23 No. 1
1
Table 1: Cost overrun summary by country and sector – average percentage differences
between estimated and actual costs
(%)/ (number
of projects)
Rail
Road
Germany
-4.3% (6)
-10.0% (3)
Spain
12.8% (6)
30.7% (1)
France
Urban
Transport
Water
17.4% (2)
Ireland
Italy
62.4% (5)
Poland
19.7% (8)
20.1% (2)
2.1% (5)
74.1% (1)
-5.0% (2)
19.7% (2)
Portugal
Weighted
average by
Member
State
15.8%
32.9% (1)
74.3% (2)
32.9%
9.4%
110.7%
(1)
110.7%
0.0% (1)
26.6%
14.1%
-0.9% (1)
37.6%
80.9% (2)
50.3%
9.0% (1)
26.9%
Weighted
average
by sector
-6.2%
Great Britain
Greece
Energy
45.4%
11.3%
3.3% (4)
4.4%
20.7%
21.2%
Source: RGL, Frontier Economics, Aecom3
1.2.3
Following Flyvbjerg (2005), explanations for time delays and cost overruns can
usefully be grouped together as technical, psychological and political-economic
explanations.4
1.3 Technical explanations
1.3.1
Technical explanations for delays and overruns are extensively examined in
previous evaluations. They include imperfect forecasting, inadequate data and
honest mistakes due, for example, to a lack of experience with infrastructure cost
forecasting or with forecasting costs for certain types of infrastructure. They
might also include inherent problems in predicting the future.
1.3.2
Technical explanations for cost overruns/delays in transport infrastructure are
also the subject of a growing academic literature inspired by Flyvbjerg et al
(2004) and continuing with Flyvbjerg (2008), the latter looking specifically at
urban transport projects.5
3
See footnote 1
See Flyvbjerg, Bent (2005), ‘Policy and Planning for Large Infrastructure Projects: Problems, Causes,
Cures’, World Bank Policy Research Working Paper 3781, World Bank, Washington D.C.
5 See Flyvbjerg, Bent et al (2004), ‘What Causes Cost Overrun in Transport Infrastructure Projects?’
Transport Reviews, Volume 24, No. 1, January and Flyvbjerg, Bent et al (2008), ‘Comparison of Capital
Costs per Route-Kilometre in Urban Rail’, EJTIR, 8, no.1.
4
1.3.3
DG REGIO’s Guide provides a set of key determinants of delays and cost
overruns. These are re-presented from the Guide in Figure 1 below.
Figure 1: Key determinants of infrastructure project delays and cost overruns
Source: European Commission DG XVI (1998), ‘Understanding and Monitoring the Cost-Determining Factors of
Infrastructure Projects’, A User’s Guide, Brussels
1.3.4
The Commission identified a lack of effective project management as the main
cause of cost overruns. Other important factors were design changes, and input
price inflation due to delays. Less important factors were land acquisition
problems, unexpected ground conditions and difficulties with contractors. The
Guide’s summary of the relative importance of each of the factors for different
categories of project cost is re-presented from the Guide in Figure 2 below. A
large circle indicates a major effect and a small circle indicates a minor effect.
Figure 2: Common causes of cost overruns and project time delays
Source: European Commission DG XVI (1998), ‘Understanding and Monitoring the Cost-Determining Factors of
Infrastructure Projects’, A User’s Guide, Brussels
1.3.5
ECORYS Transport’s (2005) study of 60 projects co-financed by the Cohesion
Fund, in the period 1993-2000, observed similar factors to the DG REGIO Guide
in explaining cost overruns.6 Their report notes that the predominant causes of
cost overruns, namely design changes, inflation impacts due to time delays and
site contractor issues, all point to inadequate project preparation. The report also
considers local community involvement in projects to be important.
1.3.6
The ‘Ex Post Evaluation of Objective 1 1994-1999’ by ECOTEC (2003) compares
water infrastructure projects with similar large-scale transport projects. They use
data from France, Germany, Greece, Ireland, Italy, Portugal, Spain and the UK.
They found that water projects were more susceptible to delay and cost overrun
– 70% of water projects go over budget, compared to 60% for transport projects.
Furthermore, water projects were found to be subject to delays of in excess of 12
months, greater than the delays for the transport projects.
1.3.7
The Nichols Report (2007) on 13 of the UK Highways Agency’s largest road
projects found that inflation, inaccurate cost estimation and inadequate project
definition were the most important causes of cost escalation.7 While the results
are not entirely representative, however, because only the 13 largest schemes
were examined, they do provide a good indication of the variety of aspects that
need to be accounted for to get more accurate cost estimates.
1.3.8
The UK National Audit Office (NAO, 2007) reported on a more extensive sample
of road projects under the Targeted Programme for Improvement (TPI) and Local
Transport Plan schemes.8 The study supports the findings of the Commission,
namely that poor project management, design changes and inflation are the most
significant causes of cost overruns.
ECORYS Transport (2005), ‘Ex post evaluation of a sample of projects co-financed by the Cohesion Fund
(1993-2002)’, Report to the European Commission, DG Regio, Brussels.
7 Nichols, Mike (2007), ‘Review of Highways Agency’s Major Roads Programme’, Report to the Secretary of
State for Transport, London,
8 UK National Audit Office (2007), ‘Estimating and Monitoring the Cost of Building Roads in England’, UK
Stationary Office.
6
1.3.9
Flyvbjerg et al (2004) conducted a statistical analysis of cost overruns and found
that they could be associated with the length of the project implementation
phase. They also found that the influence was not significantly different for rail,
fixed-link (bridges and tunnels) or road projects. The authors also found project
size to be significant for bridges and tunnels, but not for road or rail projects.
Weak accountability measures for delays and overruns were also a significant
influence and probably more significant than the effect of ownership incentives.
1.3.10 Bordat et al (2004) made the distinction between excusable delays (due to force
majeure) and non-excusable delays (attributable to contracting agencies and
contractors, and therefore preventable).9 Contracting agency errors included
planning and design deficiencies such as incorrect estimates of work quantities.
Contractor errors included unnecessary work, work that did not follow the design
plans, and work that did not meet contract specifications. Excusable delays due
to unforeseen circumstances included site conditions that differed from those
described in contract documents. Majid and McCaffer (1998), upon whose work
Bordat et al (2004) is built, identified 12 main causes of project time delays. They
pinpoint materials, equipment, and labour-related delays as the major cause of
contractors’ performance delays.
See Bordat, C. et al (2004), ‘An Analysis of Cost Overruns and Time Delays of INDOT Projects’, Joint
Transportation Research Programme, Indiana Department of Transportation.
9
1.3.11 In the recent evaluation of ERDF projects, design and legislation changes were
identified as the most common source of overruns, as shown in Figure 3 below.
Figure 3 ERDF Evaluation
1.6
Cost overrun score
Time delay score
1.4
1.2
Score
1.0
0.8
0.6
0.4
0.2
0.0
l
t
s t
r
s
y t
e c
es es ac ays ns ica nts tic en iod ms ers gy re ifi xity xit en tion ess on ur e ing tes cy he mic tes
ng ang imp del sio olit se r is em per ble mb olo eg pec ple ple em f la cc lati aj nn pu qua Ot no ra
a
n
n e
a
m a s
n d
o ge
ch ch tal ity pe P /Co act enc ion pro nu ch on r s om om nag In ite r e ce r pl r di ade
E c an
o
r
r
S blic or oo lie in
gn tion en ho sus
h
ts har m uct ing lde Te ati act n c ct c ma
i
i
c
F P p s
v r g a t
m c om tr d o
es la m ut k
Pu
Ex
no ont esi ntr jec
up nes
D gis ron A or
er ite e c ns un eh
n
S
i
P
i
I C D o ro
W
S at Co F tak
C P
us
Le nv
L
S
E
B
Source: RGL, Frontier Economics, Aecom10
1.4 Psychological explanations
1.4.1
Psychological explanations derive from the ‘planning fallacy’, the systematic
tendency to be overly optimistic about the outcomes of planned actions. In
infrastructure project planning, the planning fallacy leads to ‘optimism bias’, that
is, the systematic tendency to underestimate costs, completion times and risks
and to overestimate the benefits of planned actions.
1.4.2
This category of explanation was inspired by the work of Kahneman (1979, 2003)
and Kahneman and Lovallo (1993) inspired this type of explanation for overruns
and delays.11 They relate the planning fallacy to biases at the cognitive level, i.e.
‘errors’ in the processing of information that result from ‘delusional optimism’. By
examining the technical and political-economic explanations, one can, according
10
EX POST EVALUATION OF COHESION POLICY PROGRAMMES 2000-2006, WORK PACKAGE 10,
‘Efficiency: Unit costs of major projects’ FINAL REPORT, 25 October 2009 RGL Forensics, Frontier
Economics, Aecom
11 See Kahneman, D. (1973), Attention and effort, Englewood Cliffs, NJ: Prentice-Hall. See also Kahneman,
D. (2003), ‘Maps of bounded rationality: A perspective on intuitive judgment and choice’, in T. Frangsmyr
(Ed.), Les Prix Nobel 2002 [Nobel Prizes 2002]. Stockholm, Sweden: Almquist & Wiksell International. See
also Kahneman, D. and D. Lovallo (1993) ‘Timid choices and bold forecasts: A cognitive perspective on risktaking’, Management Science, 39, 17-31.
to the authors, better understand how to resolve or, rather, correct for optimism
bias.
1.4.3
Kahneman et al’s work on decision-making under uncertainty provided the
foundation for the concept of Reference Class Forecasting, which is based on
the presumption that past projects tend to be more similar to planned projects
than normally assumed and are, therefore, a means of increasing the accuracy of
forecasting. Rather than focusing only on the specific constituents of the planned
projects (the ‘inside view’), there should be equal if not more focus on the
outcomes of similar projects that have already been completed (an ‘outside
view’).
1.4.4
Flyvbjerg and Cowi (2004) developed an applied method of Reference Class
Forecasting for the UK Treasury.12 Their report advocates the use of explicit,
empirically based optimism bias uplifts to produce more realistic forecasts of
individual project capital expenditures. The study generated benchmark uplifts
by developing probability (or frequency) distributions of cost overruns for the
various classes of project under consideration.
1.4.5
Planners and promoters can choose the uplift that corresponds with the level of
risk of cost overrun that they are willing to accept. So, the authors state, they
should adopt the 50th percentile uplift if they were willing to accept a high risk of
cost overrun and the 80th percentile if such a level of risk was unacceptable.
(Note that the percentiles referred to here can be thought of as 1 minus the level
of risk that the project planner is willing to accept. Therefore, the 80 th percentile
corresponds with an acceptable risk of cost overrun of 20 per cent. The lower
the level of risk of cost overrun that is acceptable, the higher the optimism bias
uplift needs to be.)
1.4.6
The authors found that, for road projects, optimism uplifts range from 15% at the
50th percentile to 32% at the 80th percentile. For rail projects (including urban
transport), the uplifts range from 40% at the 50th percentile to 57% at the 80th
percentile. Fixed link uplifts range from 23% at the 50th percentile to 55% at the
80th percentile.
1.4.7
The report also asserts that few project planners have a direct interest in
avoiding cost overruns. As a result, they recommended that the UK Department
for Transport apply optimism bias uplifts that are supported by:13

establishing realistic budgeting as an ideal, and de-legitimising over-optimistic
budgeting as a routine;

the introduction of fiscal incentives to avoid cost overruns e.g. through
requiring local co-financing of project cost escalation where possible;

formalised requirements for high quality cost and risk assessment at the
business case stage; and

The introduction of independent appraisal.
Flyvbjerg, Bent in association with COWI (2004), ‘Procedures for Dealing with Optimism Bias in Transport
Planning: Guidance Document’, for the British Department for Transport, London.
13 The method is now a requirement for all large transport infrastructure projects seeking government
funding in the UK and has also been endorsed by the American Planning Association.
12
1.5 Political economy explanations
1.5.1
This is the most recent framework for explanaining overruns and delays and
addresses the possibility that delays and cost overruns may be the result of
deliberate misinformation. In other words, that project planners and promoters
misrepresent timescales and cost/benefit projections in order to win favour for the
project and get it started. Important work in this area includes Flyvbjerg et al
(2004), which deal with the cost side, and Flyvbjerg et al (2005), which deals with
the demand side (demand being the factor driving expectations about future
benefits).14
1.5.2
Flyvbjerg et al (2003), in their study of 258 transport infrastructure projects
completed between 1927 and 1998, observed that ‘no learning appears to be
taking place’ and that cost underestimation and subsequent cost overruns ‘are
allowed to continue unchecked decade after decade’.
1.5.3
The authors state that we may be in an equilibrium in which ‘strong incentives
and weak disincentives for cost underestimation and related escalation may have
taught project promoters that cost underestimation pays off.’ Flyvbjerg et al
(2004) tested this explanation of cost overruns and, as reported in Flyvbjerg et al
(2003), found that cost underestimation is ‘used strategically to make projects
appear less expensive than they really are in order to gain approval from
decision-makers to build projects’.
1.5.4
Most recently, Flyvbjerg, Garbuio and Lovallo have provided a useful framework
which seeks to explain optimism bias as a result of ‘delusion’ and deception.’15
Within the explanation of delusion, the authors include ‘planning fallacy’ and
‘anchoring and adjustment’. Planning fallacy is discussed in 1.4.1 above.
‘Anchoring and adjustment’ refers to the observation that planners find it difficult
to move away from initial estimates even if they know they are unrealistic. This
may perhaps be due to the negative connotations associated with increases in
estimates during the planning phase associated with a reduction in the probability
that the project will actually happen.
1.5.5
The idea of systematic ‘deception’ is used to explain political and agency sources
of cost overruns. Their analysis focuses on the ‘principal-agent problem’ and
‘sources of strategic deception.
1.5.6
The principal agent problem arises when funders of infrastructure project
(ultimately tax payers) are separated from the project by a chain of intermediaries
including contractors, consultants, local governments and departments and
national bodies. The authors identify a number of sources of deception which can
occur at each interface of agent and principal including differences in self
interest, the presence of asymmetric information, different risk preferences,
different time horizons and finally diffuse or asymmetric accountability. For
example politicians may be more concerned with securing approval for flagship
projects in the short term, rather than long term value for money for tax payers.
14
See footnotes Error! Bookmark not defined. and 5.
Delusion and Deception in Large Infrastructure Projects: Two Models for Explaining and Preventing
Executive Disaster, Bent Flyvbjerg, Massimo Garbuio, Dan Lovallo, California Management Review Winter
2009
15
1.6 Overcoming Project Optimism bias
1.6.1
Flyvberg et al have suggested two approaches to overcoming project optimism
bias: correcting incentives and reference class forecasting. 16
1.6.2
Minimising project cost over runs requires all parties to be incentivised to
accurately estimate projects and complete projects on time and to budget. For
example, all public bodies should be required to contribute to funding overruns.
Contractors can be incentivised using a range of project structures such as
public-private partnerships and Design, Build, Finance Maintain and Operate
(‘DBFMO’) contracts.
1.6.3
Reference class forecasting is based on an argument that the most reliable
estimate of what a project will cost is what previous projects have actually cost.
This approach requires cost distribution data to be gathered on similar projects.
The relative position of the project to be forecast within the distribution needs to
be considered and used to derive a wholly external view of what the project can
be expected to cost.
1.7 Conclusion
1.7.1
Most of the literature has focused on technical explanations of cost overruns and
time delays, typically cataloging a wide range of sources of cost overrun and
building a list of key assumptions which planners have got wrong.
1.7.2
The more recent work by Professor Flyvberg and his colleagues introduces a
new ‘political’ dimension into the analysis of project optimism bias which
suggests stakeholders incentives need to be structured to correct biases that
might otherwise contribute to delays or overruns. In addition the use of external
reference class forecasts is recommended as a means of cross checking the
strong forces which have been identified as stifling the preparation of robust
bottom-up forecasts.
1.7.3
Flyvberg’s new reference class forecasting approach has not yet had time to be
rigorously tested in practice. At the very least however, it does provide project
funders with a powerful analytical tool to use when appraising projects. It should
also provide project planners with useful information which will hopefully help
them build and develop more robust estimates as they consider why their own
bottom-up estimates are less than those developed using a reference class
approach.
16
See footnote 15