UNCERTAINTY IN COST
ESTIMATES
Joseph B. Kadane
Carnegie Mellon University
kadane@stat.cmu.edu
2
“Prediction is difficult, especially about the future”
Niels Bohr
Robert Storm Peterson
Yogi Berra
Samuel Goldwyn
Mark Twain
Outline:
1. Uncertainty in general
2. An example in another field
3. Uncertainty in cost estimation
4. Conclusion
3
Uncertainty in General
Suppose you are uncertain about the weather in Pittsburgh
tomorrow and in particular about whether it will rain, and
whether the high temperature will exceed 80° F. Then there
are 4 uncertain events:
•
•
•
•
A1 :
A2 :
A3 :
A4 :
rain and temp > 80°F
rain and temp ≤ 80°F
no rain and temp > 80°F
no rain and temp ≤ 80°F
One and only one of the Ai ' s will occur, i.e. they are
mutually exclusive and disjoint.
4
Imagine that you can buy or sell tickets that pay $1 if Ai
occurs, and $0 if Ai does not occur. At what price Pr{Ai }
would you be willing to buy or sell such a ticket?
1. If
Pr{Ai } > 1 you will lose for sure.
2. If
Pr{Ai } < 0 you will lose for sure.
3. If
Pr{Ai È Aj } ¹ Pr Ai + PrAj you will lose for sure.
4. If
Pr{A1 È A2 È A3 È A4 } ¹ 1 you will lose for sure.
5
Theorem 1 You avoid sure loss if and only if your prices
satisfy:
1)
Pr{A} ³ 0 for all events A.
2)
Pr{S} = 1 where S is the sure event.
3)
Pr{A È B} = Pr{A} + Pr{B} if A and B are disjoint.
Hence you avoid being a sure loser if and only if your
prices obey the laws of probability.
6
Notes:
1. Your prices need not be those of someone else. These
prices – probabilities – are personal, or subjective.
2. Avoiding sure loss does not make you a winner.
(Indeed, the most absurd beliefs can be expressed
using such prices.
3. This interpretation of probability does not require an
infinite (or long) sequence of independent and
identically distributed repetitions.
7
What about conditional events?
Suppose there are tickets that pay $1 if A and B both occur,
$0 if A occurs but B does not, and the price is returned if A
does not occur. Suppose you would pay Pr{B | A} for such
a ticket.
8
Theorem 2: You avoid sure loss if and only if your prices
satisfy
Pr{AB} = Pr{B | A}Pr{A}.
9
Notes:
1. This is the same as the usual definition of conditional
probability, except here it is a consequence of a desire
to avoid sure loss.
2. When
Pr{A} = 0, Pr{B | A} is unconstrained.
10
2. Example from another field: physics
Physics is touted as the most exact of sciences, one in
which uncertainty is best controlled. However, as we’ll see,
experimental physicists routinely assume that the
measurements they make have no systematic errors, and
are routinely wrong, by hindsight, in that assessment.
11
Systematic errors can arise from:
1) Undetected bias in experimental procedures
2) Computational approximations
3) Errors in auxiliary variables
4) Deficiencies in theoretical assumptions
None of these are apparent to the experimenter at the time.
12
Figure 1: The history of measurements of the speed of light, c.
13
Figure 2: History of reported uncertainty in other physical constants
14
Figure 2: (cont’d)
15
Repeating an experiment many times can reduce the
“random error” associated with an average, but does
nothing for systematic error. The culture of physics has not
traditionally encouraged a discussion of the likely
magnitude of systematic error.
16
3. Uncertainty in Cost Estimates
“A reported value whose accuracy is entirely unknown is
worthless”
Churchill Eisenhart
Why is it that cost estimates of almost anything are so
rarely too high, and very often too low? One simple
correction is to multiply the cost estimate by p = 3.14... .
If an interval is required, multiply by p ± e, where e = 2.72...
Perhaps it is possible to do better than this with a more
thoughtful approach.
17
What are some of the sources of uncertainly in estimating
the cost of large, long-term projects?
a) Scope creep: things change. New capabilities are
requested. Is the cost estimate supposed to estimate
1
2
The hypothetical cost of the project if no changes in scope are
required.
The actual cost, including possible future (now unforeseen)
changes in scope?
18
b) Technology risk: the technology to do the project may
not currently exist. How much may it cost to produce the
technology? What happens if the technological issues
cannot be resolved?
19
c) Economic and political changes: If cost estimates are in
current dollars, what effect may future inflation have?
What may be the effects of legislative or regulatory
changes? What assumptions underlie the project? What
are the consequences if those assumptions are wrong?
20
4. Conclusions
A. Give up the idea that cost estimates are anything but
somebody’s opinion.
B. Best practice is to express your opinion in terms of a
probability distribution.
C. Be clear about exactly what you are costing out,
particularly whether changes in scope are included.
D. Think about all the ways you might be wrong, and what
the costs under those scenarios look like.
21
References:
1. For a fast read, introduction:
D.V. Lindley, Understanding Uncertainty, J. Wiley & Sons.
2. For proof of theorems 1 and 2, and a more technical
discussion, my book
Principles of Uncertainty, Chapman and Hall, 2011, also free
on the web at my website http://www.uncertainty.com.
3. For the physics:
M. Henrion and B. Fischoff, “Assessing uncertainty in physical
constants,” American Journal of Physics, 54(9), September, 1986.