Understanding Uncertainty
Definitions and Tools for Risk
Analysts
Copyright © 2004 David M. Hassenzahl
Goals for this Lecture
• Understand sources and types of
uncertainty
• Evaluate quantitative and qualitative
representations of uncertainty
– Distributional forms
• Explore some typologies
• Consider how people interpret
uncertainty
Copyright © 2004 David M. Hassenzahl
Overview
• Uncertainty is unavoidable
• Uncertainty is information!
– Where to do further research
– Worst case scenarios
• Uncertainty is chronically understated
– Generally satisfied with statistical analysis
– Henrion and Fischhoff (1986), estimates of speed
of light (see Kammen and Hassenzahl 1999 page
125)
Copyright © 2004 David M. Hassenzahl
Measured speed of light (km/sec)
300000
299830
Expected value
with standard error
299950
299820
Recommended value
with reported
uncertainty
299900
299810
299850
299800
299800
299790
299750
299780
299700
299770
299650
299760
299600
1870
299750
1900
1880
1890
1900
Year of experiment
1984
value
1910
1920
1930
1940
1950
Year of experiment
From Henrion and Fischhoff
Copyright © 2004 David M. Hassenzahl
(1986)
1960
1970
Some thoughts
• “Research efforts in risk analysis should
be viewed as tools for understanding
uncertainties, not necessarily for
reducing them.” (Finkel, 1990)
• “Probability does not exist” (Morgan and
Henrion, 1990)
– Either you die from cause X or you don’t
Copyright © 2004 David M. Hassenzahl
Uncertainty Analysis Should
Be Thorough
• It should include quantitative measures
of uncertainty
• It should include qualitative discussion
of uncertainty
• Doing this well takes time and effort
(and is not always rewarded…)
Copyright © 2004 David M. Hassenzahl
Quantitative Measures
• Display the data set
• Use descriptive statistics
– Moments, ranges, etc
• Apply sensitivity analysis
• Eschew spurious precision!
Copyright © 2004 David M. Hassenzahl
Use of Point Estimates?
• “introducing confidence intervals…to
deal with uncertainty may be
pointless…unless the policy analyst
eliminates the point estimate itself.”
• “A specific number has a vividness and
simplicity which makes it an inevitable
focus of policy debate.”
• Camerer and Kunreuther, 1989
Copyright © 2004 David M. Hassenzahl
Point Estimate or not?
+100%
increas
e
Change
in risk
0
reduction
-100%
do
nothing
1
2
3
Option
Figure 10-6 from SWRI page 258
Copyright © 2004 David M. Hassenzahl
4
5
Qualitative discussion
• Discuss known sources of error
• Consider plausible sources of error
• Evaluate the importance of uncertainty
Copyright © 2004 David M. Hassenzahl
Challenges to ignoring
uncertainty
• Regarding a report on ozone depletion “…no
attempt was made to estimate the systematic
errors in evaluating rates or omission of
chemical processes. Without such estimates,
decision makers are free to make their own
judgments ranging from uncritical acceptance
of the current models to complete skepticism
as to their having any likelihood of being
correct.” Morgan and Henrion 1990.
Copyright © 2004 David M. Hassenzahl
Historical Mistakes
• Rasmussen Report (WASH 1400) on reactor safety
– Significant portions retracted by US government
– Still referenced
• Inhaber report on nuclear “safer” than other energy
technologies
– Major source (Holdren) responded “not so”
• Chauncy Starr
– Voluntary/involuntary
– Risk as f(benefit)
– True…but not precisely known
• Tengs et al?
Copyright © 2004 David M. Hassenzahl
Briggs and Sculpher (1995)
• Cost-effectiveness analyses from
medical literature
– Public Health origins of CEA
• Incomplete or inadequate attention to
uncertainty analysis in 86%
Copyright © 2004 David M. Hassenzahl
Starr (1969) interpretation

-3
10 
-4

-5

-6
10 
-7
10 
-8

-9

10
-10
Pf [Fatalities/person hr. exposure]
10
10
Voluntary
General aviation
R~B3
10
10
Hunting, skiing,
smoking
Railroads
Commercial
aviation Motor
vehicles
Involuntary
R~B3
Electric
power
10
R = Risk
B = Benefit
Natural
disasters

10
100
Average P f
due to disease
-11
200
500
1000
2000
5000
Average annual benefit/person involved [dollars]
Copyright © 2004 David M. Hassenzahl
10000
Otway and Cohen (1975)
Interpretation

-3
10 
-4

-5

-6
Pf [Fatalities/person hr. exposure]
10
R~B1.8
General aviation
10
10
10 
-7
10 
-8

-9

10
-10
Hunting, skiing,
smoking
Commercial
aviation Motor
vehicles
Voluntary
Average P f due
to disease
Railroads
Involuntary
R~B6.3
Electric
power
10
R = Risk
B = Benefit
Natural
disasters

10
100
-11
200
2000
500
1000
5000
Average annual benefit/person involved [dollars]
Copyright © 2004 David M. Hassenzahl
10000
Example: Amitraz on Pear
Orchards
• EPA decided to ban Amitraz for use on
pear orchards (US EPA 1979)
• Point estimate generated for Cost
effectiveness
– $2.6 million per life-year saved (Tengs et al
1995)
Copyright © 2004 David M. Hassenzahl
Expected Value of Ban
• Does Amitraz control pests?
– If not, ban has no economic implications
– cpyls ≤ $0
• Is Amitraz a carcinogen?
– If not, ban has major economic implications
– cpyls  $ ∞
• E(cpyls) ≠ $2.6 million
• E(cpyls) = uniform($0, $∞)
• Hassenzahl (2004)
Copyright © 2004 David M. Hassenzahl
Estimators
• Because we can’t always get the data
we want, we need to estimate data
• We can use
– Frequencies
– Distributions
– Curve-fitting
Copyright © 2004 David M. Hassenzahl
Frequency example: 500 people
•
•
•
•
•
•
495 have 10 toes
2 have 12 toes
1 has 9 toes
1 has 5 toes
1 has 0 toes
In next 1000: how many will have 6
toes?
Copyright © 2004 David M. Hassenzahl
Distributions as estimators
• We often use DISTRIBUTIONAL
FORMS to approximate data sets
• We then estimate missing or future
values using the distribution
– Extrapolate beyond data set
– Interpolate within data set
Copyright © 2004 David M. Hassenzahl
Example: 500 people
•
•
•
•
•
•
•
•
•
feet
0
0.5
1
1.5
2
2.5
3
3.5
number
0
0
0
0
0
0
1
1
•
•
•
•
•
•
•
•
•
feet
4
4.5
5
5.5
6
6.5
7
7.5
Copyright © 2004 David M. Hassenzahl
number
12
48
145
203
78
10
2
0
Triangular? Normal?
0.45
0.4
0.35
Frequency
0.3
0.25
0.2
0.15
0.1
0.05
0
-0.05
0
1
2
3
4
5
6
Height (feet)
Copyright © 2004 David M. Hassenzahl
7
8
9
Normal: Characteristics: Mean = 5.6
feet, Standard Deviation = 1.3 feet
Forecast: Normal
10,000 Trials
Frequency Chart
9,921 Displayed
.022
218
.016
163.5
.011
109
.005
54.5
.000
0
2.21
3.89
5.58
7.27
Copyright © 2004 David M. Hassenzahl
8.96
Triangular? Normal?
0.45
0.4
0.35
Frequency
0.3
0.25
0.2
0.15
0.1
0.05
0
-0.05
0
1
2
3
4
5
6
Height (feet)
Copyright © 2004 David M. Hassenzahl
7
8
9
Triangular? Normal?
0.45
0.4
0.35
Frequency
0.3
0.25
0.2
0.15
0.1
0.05
0
-0.05
0
1
2
3
4
5
6
Height (feet)
Copyright © 2004 David M. Hassenzahl
7
8
9
A Good Estimator Is
•
•
•
•
•
•
Consistent
Unbiased
Efficient
Sufficient
Robust
Practical
Copyright © 2004 David M. Hassenzahl
Depicting Uncertainty:
Distributions
• No consensus on how to depict
ignorance!
– More later under Monte Carlo Analysis
• Key forms
– Uniform / rectangular
– Triangular
– Normal, lognormal
– Many others!
Copyright © 2004 David M. Hassenzahl
Decision
• When should we hold the Founder’s
Day parade?
• Assume we want to avoid tornadoes
Copyright © 2004 David M. Hassenzahl
Kammen and Hassenzahl 1999
Uniform
Probability of tornado
0.003
0.002
0.001
Jan
Feb
M ar
Apr
M ay
Jun
Jul
Aug
Sep
Oct
Nov
Month
From SWRI page 106. Figure 3.5 Uniform distribution for daily
p(tornado) as described in problem 3-7a.
Copyright © 2004 David M. Hassenzahl
Dec
Kammen and Hassenzahl 1999
Rectangular
Probability of tornado
0.012
0.010
0.008
0.006
0.004
0.002
Jan
Feb
M ar
Apr
M ay
Jun
Jul
Aug
Sep
Oct
Nov
Month
From SWRI page 106. Figure 3-6 Rectangular distribution for
daily p(tornado) as described in problem 3-7a.
Copyright © 2004 David M. Hassenzahl
Dec
Kammen and Hassenzahl 1999
Triangular
Probability of tornado
0.025
0.020
0.015
0.010
0.005
0
Jun
Jul
Aug
Sep
Month
From SWRI page 107. Figure 3-7 Triangular distribution for
daily p(tornado) as described in problem 3-7a.
Copyright © 2004 David M. Hassenzahl
Oct
Policy Decision
• Radon is found in homes across the
country
• We might worry at 100 pCi/liter
• We can’t measure all the homes in the
country, but we have a decent sized
sample
• It matters how we model that
distribution
Copyright © 2004 David M. Hassenzahl
Normal Distribution
Number of homes
1
4
1
2
1
0
8
6
4
2
0
0
1
2
3
4
5
Radon level (pCi/liter)
Kammen and Hassenzahl 1999
Copyright © 2004 David M. Hassenzahl
6
7
8
Lognormal
20
Houses (%)
15
10
5
>8
0
0
Nero et al 1987
2
4
222Rn
6
(pCi/liter)
Copyright © 2004 David M. Hassenzahl
8
Exponential
% of houses with concentration > x
100
USAless6
pwr(1.25) %
10
pwr(1.75) %
LnNrm3.5%
1
0.1
0.01
0.001
0.0001
1
10
Goble and Socolow 1990
100
Rn Concentration
Copyright © 2004 David M. Hassenzahl
1000
Fitting distributions to data sets
• Compare data to predictions
– Least squares
– Maximum Likelihood
• We will explore these in context
– Binomial for animal toxicology data
Copyright © 2004 David M. Hassenzahl
Breaking Down Uncertainty
• Useful typologies for thinking about
uncertainty
• Can’t always reduce uncertainty
• Typologies have
– Internal overlaps
– Missing pieces (no perfect typology)
Copyright © 2004 David M. Hassenzahl
Four typologies
• Finkel
• Boholm
– Parameter Uncertainty
– Model Uncertainty
– Decision-rule Uncertainty
• Smithson
– non-quantifiable/holistic
aspects
– uncertainty as one
component of ignorance.”
– Situates Uncertainty
as the noncalculable part of risk
– appropriate coping
strategies:
• faith
• precaution and
• avoidance
Copyright © 2004 David M. Hassenzahl
Typology (after Morgan and
Henrion 1990)
1. Random error and
statistical variation
2. Systematic error
and subjective
judgment
3. Linguistic
imprecision
4. Variability
5. Randomness and
unpredictability
6. Expert Uncertainty
7. Approximation
8. Model uncertainty
 Normative
Uncertainty
Copyright © 2004 David M. Hassenzahl
Random error/statistical
variation
• We have a well defined set of tools
• These can be misleading!
– Often the ONLY thing that is done
– Often done…and ignored
• Z-scores, Chi-squared, p-values
• Meaning of 95% confidence interval?
Copyright © 2004 David M. Hassenzahl
Random Error: “Clusters”
Copyright © 2004 David M. Hassenzahl
Systematic error and
subjective judgment
• Example: speed of light, or energy
predictions
• Chronically understated
• Useful approach: bounding
Copyright © 2004 David M. Hassenzahl
Systematic Error: Predicted Year
2000 Energy Use
200
Quadrillion Btu Per Year
175
150
125
100
75
50
25
0
1972
1974
Goldemberg et al 1987
1976
1978
1980
Year of Publication
Copyright © 2004 David M. Hassenzahl
1982
Linguistic imprecision
• Inconsistencies in language and usage
can lead to problems
• Is “beyond a reasonable doubt” 95%?
What does that mean?
• “Rain is likely”…are you from Las Vegas
or Bangladesh?
• “A few thousand deaths”
Copyright © 2004 David M. Hassenzahl
Variability
•
•
•
•
Also called “dispersion”
Get the right population!
Describing variability can be a challenge
Monte Carlo analysis is a useful tool
Copyright © 2004 David M. Hassenzahl
Variability
• Height of individuals
– Deterministic
element
– Random element
• Susceptibility to
disease
– Predisposition
• Known
• Unknown
• Theoretical models
• Empirical data
• Who are we worried
about?
– “Average” person
– Most susceptible
subset
– Life-history and habit
Copyright © 2004 David M. Hassenzahl
Randomness and
unpredictability
• Inherent randomness is irreducible!
• Practical limitations and chaos
Copyright © 2004 David M. Hassenzahl
Expert uncertainty
• Multiple interpretations of a single data
set
• Norms of analysis
• Motivational bias (decision stakes,
reputation)
Copyright © 2004 David M. Hassenzahl
Expert Uncertainty
• Economists’ Conception
– Limited resources
– Resource substitution
– Adaptability
• Ecologists’ Conception
– Stable systems
– Long term impact of disruption
Copyright © 2004 David M. Hassenzahl
Climate Change: Economists
“versus” Ecologists
25
90th percentile
50th percentile
Loss of gross world product
20
10th percentile
15
10
5
0
-5
14 17 3 16 1 2 9 4 11 6 15 12 18 7 13 10 5 8
Individual respondents' answers
Nordhaus 1994
Copyright © 2004 David M. Hassenzahl
EMF’s and Expertise
• Biomechanists / physicists
– “Impossibility” theorems
• Epidemiologists
– Correlation and proposed causation
• Toxicologists
– Extrapolation
Copyright © 2004 David M. Hassenzahl
Approximation
• Never have complete data
• There is a tradeoff between efficient
computation and resolution or precision
– Sensitivity analysis
• Significant figures are important
Copyright © 2004 David M. Hassenzahl
Model uncertainty
• Getting the right model
– Does the model explain the data?
– Is the model consistent with theory?
• Getting the model right
– Is the model properly stated?
– Is the math done correctly
• Other uncertainties (drawn from the
typology above)
Copyright © 2004 David M. Hassenzahl
Model Uncertainty
% of houses with concentration > x
100
USAless6
pwr(1.25) %
10
pwr(1.75) %
LnNrm3.5%
1
0.1
0.01
0.001
0.0001
1
10
Goble and Socolow 1990
100
Rn Concentration
Copyright © 2004 David M. Hassenzahl
1000
Normative Uncertainty
• Often not asked: what is important to
us?
• Arguments about technical information
mask the true issues
• Leads to vitriol and claim of “ignorance”
and “antiscientific attitudes”
Copyright © 2004 David M. Hassenzahl
Normative UC and YMP
• Gore, Bush: “let the science decide”
• Secretary Abraham “technically suitable
site”
• LV residents “technically unsuitable site”
• Have we defined “suitability?”
Copyright © 2004 David M. Hassenzahl
Interpreting Uncertainty
• Very limited information on how people
interpret uncertainty
• Possible links
– Uncertainty and credibility
– Uncertainty and trust
Copyright © 2004 David M. Hassenzahl
References
• Briggs, A. and M. Sculpher (1995) Sensitivity analysis
in economic evaluation: a review of published
studies. Health Economics 4: 355-371.
• Camerer, C. F. and Kunreuther, H. (1989) “Decision
Processes for Low Probability Events: Policy
Implications,” J. Policy Analysis and Management 8
(4), 565 - 592.
• Goble, R. and Socolow, R. (1990), “High Radon
Houses: Implications for Epidemiology and Risk
Assessment,” Cented Research Report No. 5 (Clark
University, Worcester, MA).
Copyright © 2004 David M. Hassenzahl
References
• Goldemberg, J,. Johansson, T., Reddy, A and
Williams, R (1987). Energy for a Sustainable
World. Washington DC: World Resources
Institute.
• Hassenzahl, D. M. (2004). “The effect of
uncertainty on ‘risk rationalizing’ decisions.”
Journal of Risk Research.
• Henrion, Max and Fischhoff, Baruch (1986)
“Assessing uncertainty in physical constants”,
American J. of Physics,54, 791 - 798.
Copyright © 2004 David M. Hassenzahl
References
• Nero, A. V., Schwehr, M. B., Nazaroff, W.W. and
Revzan, K.L. (1986) “Distribution of Airborne Radon222 Concentrations in U.S. Homes,” Science (234)
992 - 997.
• Nordhaus, W. D. (1994) “Expert opinion on climate
change”, American Scientist, 82, 45 - 51.
• Otway, H. and J. J. Cohen (1975). Revealed
Preferences: Comments on the Starr Benefit-Risk
Relationships. Laxenburg, Austria, International
Institute for Applied Systems Analysis.
Copyright © 2004 David M. Hassenzahl
References
• Starr, C. (1969). "Social benefit versus
techological risk." Science 165: 1232-1238.
• Tengs, T. O., M. E. Adams, et al. (1995).
"Five-Hundred Life-Saving Interventions and
Their Cost-Effectiveness." Risk Analysis
15(3): 369 - 390.
• US EPA (1979) Determination pursuant to
40CFR162.11(a)(5) concluding the rebuttable
presumption against registration of pesticide
products containing amitraz. Federal Register
48: 2678-83.
Copyright © 2004 David M. Hassenzahl
Additional readings
• Kammen and Hassenzahl, 1999. Should We
Risk It? Exploring Environmental, Health and
Technology Problem Solving, Princeton
University Press, Princeton NJ.
• Finkel, Adam, 1990. Confronting Uncertainty
in Risk Management (Resources for the
Future, Washington DC)
• Morgan, M. Granger and Max Henrion
(1990). Uncertainty Cambridge University
Press, NY NY.
Copyright © 2004 David M. Hassenzahl