Carnegie Mellon University
Analyzing and Communicating
Uncertainty in the Context of
Weather and Climate
An overview talk for the 4th UCAR/NCAR
Junior Faculty Forum on Future Scientific Directions
2006 August 01
M. Granger Morgan
Head, Department of
Engineering and Public Policy
Carnegie Mellon University
tel: 412-268-2672
e-mail: granger.morgan@andrew.cmu.edu
Department of Engineering and Public Policy
1
Carnegie Mellon University
This morning I will talk about:
• Sources of uncertainty and the characterization of
uncertainty.
• Two basic types of uncertainty.
• Uncertainty about coefficient values.
• Uncertainty about model functional form.
• Performing uncertainty analysis and making decisions in
the face of uncertainty.
• Public perceptions and communication.
• Some summary guidance on reporting, characterizing
and analyzing uncertainty.
Department of Engineering and Public Policy
2
Carnegie Mellon University
Probability
Probability is the basic language of uncertainty.
I will adopt a personalistic view of probability
(sometimes also called a subjectivist or Bayesian view).
In this view, probability is a statement of the degree of
belief that a person has that a specified event will occur
given all the relevant information currently known by that
person.
P(X|i) where:
X is the uncertain event
i is the person's state of information.
Department of Engineering and Public Policy
3
Carnegie Mellon University
The clairvoyant test
Even if we take a personalist view of probability, the event
or quantity of interest must be well specified for a
probability, or a probability distribution, to be meaningful.
"The retail price of gasoline in 2008" does not pass this
test. A clairvoyant would need to know things such as:
• Where will the gas be purchased?
• At what time of year?
• What octane?
Department of Engineering and Public Policy
4
Carnegie Mellon University
Does a subjectivist view mean
your probability can be arbitrary?
NO, because if they are legitimate probabilities,
they must
• conform with the axioms of probability
• be consistent with available empirical data.
Lots of people ask, why deal with probability? Why not
just use subjective words such as "likely" and "unlikely" to
describe uncertainties? There are very good reasons not to
do this.
Department of Engineering and Public Policy
5
Carnegie Mellon University
The risks of using qualitative
uncertainty language
Qualitative uncertainty language is inadequate
because:
- the same words can mean very different things to
different people.
- the same words can mean very different things to
the same person in different contexts.
- important differences in experts' judgments about
mechanisms (functional relationships), and about
how well key coefficients are known, can be
easily masked in qualitative discussions.
Department of Engineering and Public Policy
6
Carnegie Mellon University
This figure shows the
range of probabilities that
people are asked to
assign probabilities to
words, absent any
specific context.
range from upper
to lower median
estimate
range of
individual
lower bound
estimates
Almost certain
Qualitative description of uncertainty used
Mapping
words to
probabilities
range of
individual
upper bound
estimates
Probable
Likely
Good chance
Possible
Tossup
Unlikely
Improbable
Doubtful
Almost impossible
1.0
0.8
0.6
0.4
0.2
0.0
Probability that subjects associated
with the qualitative description
Figure adapted from Wallsten et al., 1986.
Department of Engineering and Public Policy
7
Carnegie Mellon University
Key:
Ex Com of
EPA SAB
el
li k
t
no
en y
we i kel
t
be t l
n g d no
i
eth n
m ly a
o
s ke
li
y
Other meeting p articipants:
0.000001
0.00001
0.0001
0.001
0.01
0.05
0.1
0.3
0.5
0.7
0.9
Figure from Morgan, HERA, 1998.
y
SAB members:
1.0
The minimum probability
associated with the word
"likely" spanned four orders of
magnitude.
The maximum probability
associated with the word "not
likely" spanned more than five
orders of magnitude.
There was an overlap of the
probability associated with the
word "likely" and that
associated with the word
"unlikely"!
el
li k
Probability that the material
is a human carcinogen
Department of Engineering and Public Policy
8
Carnegie Mellon University
The bottom line
Without at least some quantification,
qualitative descriptions of uncertainty
convey little, if any, useful information.
The climate assessment community is
gradually learning this lesson.
Steve Schneider and Richard Moss have worked
hard to promote a better treatment of uncertainty in
the work of the IPCC.
At my insistence, U.S. national assessment
synthesis team gave quantitative definitions to five
probability words:
Department of Engineering and Public Policy
9
Carnegie Mellon University
BUT, in other fields…
…such as biomedical and health
effects, progress has been much slower.
A concrete example of this is provided
by the recommendations of
Presidential/Congressional
Commission on Risk Assessment and
Risk Management (1997) which
recommended…"against routine use of formal quantitative
analysis of uncertainty in risk estimation, particularly that related
to evaluating toxicology."
While analysts were encouraged to provide "qualitative descriptions of riskrelated uncertainty," the Commission concluded that "quantitative uncertainty
analyses of risk estimates are seldom necessary and are not useful on a routine
basis to support decision-making."
Slowly such views are giving way, but progress is slow.
Department of Engineering and Public Policy
10
Carnegie Mellon University
This morning I will talk about:
• Sources of uncertainty and the characterization of
uncertainty.
• Two basic types of uncertainty.
• Uncertainty about coefficient values.
• Uncertainty about model functional form.
• Performing uncertainty analysis and making decisions in
the face of uncertainty.
• Public perceptions and communication.
• Some summary guidance on reporting, characterizing
and analyzing uncertainty.
Department of Engineering and Public Policy
11
Carnegie Mellon University
We must consider two
quite different kinds of uncertainty
1. Situations in which we know the relevant variables
and the functional relationships among them, but we
not know the values of key coefficients (e.g., the
"climate sensitivity").
2. Situations in which we are not sure what all the
relevant variables are, or the functional relationships
among them (e.g., will rising energy prices induce
more technical innovation?).
Both are challenging, but the first is much more easily
addressed than the second. I'll talk more about the second,
when I talk about uncertainty analysis.
Department of Engineering and Public Policy
12
Carnegie Mellon University
Uncertainty about quantities
From Morgan and Henrion, Uncertainty, Cambridge, 1990/99.
Department of Engineering and Public Policy
13
Carnegie Mellon University
PDFs and CDFs
A number of examples I am about to show are in
the form of probability density functions (PDFs)
or cumulative distribution functions (CDFs).
Since some of you may not make regular use of
PDFs and CDF's, let me take just a moment to
remind you...
Department of Engineering and Public Policy
14
Carnegie Mellon University
Probability density
Probability density function
or PDF
V
V+
V, Value of the
uncertain quantity
Department of Engineering and Public Policy
15
Carnegie Mellon University
Cumulative probability
Cumulative
distribution
function
or CDF
1.0
0.5
median
p
NOTE: In asymmetric
distributions with long
tails, the mean may be
much much larger than the
median.
Probability density
0
mode
mean
p
V
V+
V, Value of the
uncertain quantity
Department of Engineering and Public Policy
16
Carnegie Mellon University
If I have good data...
...in the form of many observations of a random process, then I
can construct a probability distribution that describes that
process. For example, suppose I have the 145 years of rainfall
data for San Diego,
and I am prepared to
assume that over that
period San Diego's
climate has been
"stationary" (that is
the basic underlying
processes that create
the year-to-year
variability have not
Source: Inman et al., Scripps, 1998.
changed)…
Department of Engineering and Public Policy
17
Carnegie Mellon University
Then if I want…
…a PDF for future San Diego
annual rainfall, the simplest
approach would be to
construct a histogram from the
data, as illustrated to the right.
If I want to make a prediction
for some specific future year, I
might go on to look for time
patterns in the data. Even
better, I might try to relate
those time patterns to known
slow patterns of variation in
the regional climate, and
modify my PDF accordingly.
Department of Engineering and Public Policy
18
Carnegie Mellon University
In that way…
…I could construct a PDF and
CDF for future San Diego rainfall that would look roughly
like this.
However, suppose that what I
really care about is the
probability that very large
rainfall events will occur.
Since there have only been two
years in the past 145 years
when rainfall has been above 60
cm/yr over, I'll need to augment
my data with some model or
physical theory.
Department of Engineering and Public Policy
19
Carnegie Mellon University
In summary…
…one should use available data, and well-established physical
and statistical theory, to describe uncertainty whenever either
or both are available.
However, often the available data and theory are not exactly
relevant to the problem at hand, or they are not sufficiently
complete to support the full objective construction of a
probability distribution.
In such cases, we may have to rely on expert judgment. This
brings us to the problem of how to "elicit" expert judgment.
Department of Engineering and Public Policy
20
Carnegie Mellon University
Cognitive Heuristics
When ordinary people or experts make judgments about
uncertain events, such as numbers of deaths from chance
events, they use simple mental rules of thumb called
"cognitive heuristics."
In many day-to-day circumstances, these serve us very well,
but in some instances they can lead to bias - such as over
confidence - in the judgments we make.
This can be a problem for experts too.
The three slides that follow illustrate three key heuristics:
"availability," "anchoring and adjustment," and
"representativeness."
Department of Engineering and Public Policy
21
Carnegie Mellon University
Cognitive bias
from Lichtenstein et al., 1978.
Availability: probability judgment is driven by ease with
which people can think of previous occurrences of the event
or can imagine such occurrences.
Department of Engineering and Public Policy
22
Carnegie Mellon University
Cognitive bias…(Cont.)
from Lichtenstein et al., 1978.
Anchoring and adjustment: probability judgment is frequently
driven by the starting point which becomes an "anchor."
Department of Engineering and Public Policy
23
Carnegie Mellon University
Cognitive bias…(Cont.)
I flip a fair coin 8 times. Which of the following two
outcomes is more likely?
Outcome 1: T, T, T, T, H, H, H, H
Outcome 2: T, H, T, H, H, T, H, T
Of course, the two specific sequences are equally
likely...but the second seems more likely because it
looks more representative of the underlying random
process.
Representativeness: people judge the likelihood that an object
belongs to a particular class in terms of how much it resembles
that class.
Department of Engineering and Public Policy
24
Carnegie Mellon University
Overconfidence is an ubiquitous problem
21 different studies on
questions with known answers:
0%
10%
20%
30%
40%
50%
60%
Percentage of estimates in which the true value
lay outside of the respondent’s assessed
98% confidence interval.
For details see Morgan and Henrion,
Uncertatinty , Cambridge Univ. Press, 1990, pg 117.
Estimates of the
speed of light
Department of Engineering and Public Policy
25
Carnegie Mellon University
Expert elicitations we have done
Over the past three decades, my colleagues and I have
developed and performed a number of substantively detailed
expert elicitations. These have been designed to obtain
experts’ considered judgments. Examples include work on:
Health effects of air pollution from
coal-fired power plants.
•
M. Granger Morgan, Samuel C. Morris, Alan K. Meier and Debra L. Shenk, "A
Probabilistic Methodology for Estimating Air Pollution Health Effects from Coal-Fired
Power Plants," Energy Systems and Policy, 2, 287-310, 1978.
•
M. Granger Morgan, Samuel C. Morris, William R. Rish and Alan K. Meier, "Sulfur
Control in Coal-Fired Power Plants: A Probabilistic Approach to Policy Analysis,"
Journal of the Air Pollution Control Association, 28, 993-997, 1978.
•
M. Granger Morgan, Samuel C. Morris, Max Henrion, Deborah A.L. Amaral and
William R. Rish, "Technical Uncertainty in Quantitative Policy Analysis: A Sulfur
Air Pollution Example," Risk Analysis, 4, 201-216, 1984 September.
•
M. Granger Morgan, Samuel C. Morris, Max Henrion and Deborah A. L. Amaral,
"Uncertainty in Environmental Risk Assessment: A case study involving sulfur
transport and health effects," Environmental Science and Technology, 19, 662-667,
1985 August.
Department of Engineering and Public Policy
26
Carnegie Mellon University
Expert elicitations…(Cont.)
Climate science, climate impacts
and mitigation technology:
•
M. Granger Morgan and David Keith, "Subjective Judgments by Climate Experts," Environmental
Science & Technology, 29(10), 468-476, October 1995.
•
Elizabeth A. Casman, M. Granger Morgan and Hadi Dowlatabadi, “Mixed Levels of Uncertainty in
Complex Policy Models,” Risk Analysis, 19(1), 33-42, 1999.
•
M. Granger Morgan, Louis F. Pitelka and Elena Shevliakova, "Elicitation of Expert Judgments of
Climate Change Impacts on Forest Ecosystems," Climatic Change, 49, 279-307, 2001.
•
Anand B. Rao, Edward S. Rubin and M. Granger Morgan, "Evaluation of
Potential Cost Reductions from Improved CO2 Capture Systems,"Proceedings of the 2nd National
Conference on Carbon Sequestration, Alexandria, VA, May 5-8, 2003.
•
M. Granger Morgan, Peter J. Adams and David W. Keith, "Elicitation Of Expert Judgments of
Aerosol Forcing," Climatic Change, xxx.
•
Kirsten Zickfeld, Anders Levermann, M. Granger Morgan, Till Kuhlbrodt, Stefan Rahmstorf, and
David W. Keith, "Present state and future fate of the Atlantic meridional overturning circulation as
viewed by experts," in revision for Climatic Change.
Bounding uncertain health risks:
•
M. Granger Morgan, "The Neglected Art of Bounding Analysis," Environmental Science
& Technology, 35, pp. 162A-164A, April 1, 2001.
•
Minh Ha-Duong, Elizabeth A. Casman, and M. Granger Morgan, "Bounding Poorly
Characterized Risks: A lung cancer example," Risk Analysis, 24(5), 1071-1084, 2004.
•
Elizabeth Casman and M. Granger Morgan, "Use of Expert Judgment to Bound Lung
Cancer Risks," Environmental Science & Technology, 39, 5911-5920, 2005.
Department of Engineering and Public Policy
27
Carnegie Mellon University
Expert elicitation…(Cont.)
In all our elicitation studies, we've focused on creating a process
that allows the experts to provide their carefully considered
judgment, supported by all the resources they may care to use.
Thus, we have:
- Prepared a background review of the relevant literatures.
- Carefully iterated the questions with selected experts and run pilot
studies with younger (Post-doc) experts to distil and refine the
questions.
- Conducted interviews in experts' offices with full resources at hand.
- Provide ample opportunity for subsequent review and revision of the
judgments provided.
All of these efforts have involved the development of new
question formats that fit the issues at hand.
Department of Engineering and Public Policy
28
Carnegie Mellon University
Expert elicitation
takes time and care
Eliciting subjective probabilistic judgments requires
careful preparation and execution.
Developing and testing an appropriate interview
protocol typically takes several months. Each interview
is likely to require several hours.
When addressing complex, scientifically subtle
questions of the sorts involved with most problems in
climate change, there are no satisfactory short cuts.
Attempts to simplify and speed up the process almost
always lead to shoddy results.
Department of Engineering and Public Policy
29
Warming for
2x[CO2]
expert
Carnegie Mellon University
-10
-5
0
5
10
15
20
0
5
10
15
20
1
2
2 with state change
3
4
4 with "surprise"
5
6
7
8
9
10
11
12
13
14
15
16
-10
-5
Temperature response given 2x[CO
Source:
Morgan and Keith, ES&T, 1995.
2
] (K)
…and, lest you conclude that most of these
30
Department
Engineering
and Public Policy
experts
areofin
basic agreement…
Pole to equator
temperature
gradient for
2x[CO2]
expert
Carnegie Mellon University
1
2
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
30
35
40
35
40
w/climate state change
na
na
na
30
Meridinal temperature gradient given 2x[CO
2
] (K)
Source: Morgan and Keith, ES&T, 1995.
Department of Engineering and Public Policy
31
Carnegie Mellon University
Biomass in Northern Forests
w/ 2xCO2 climate change
B.
Change
in soil carbon
1
Expert
A.
Change
in standing biomass
2
Nort h A merica
Nort h A merica
E urasi a
E urasi a
"t ri vi al"
Nort h A merica
3
E urasi a
4
5
Nort h A merica
Nort h A merica
E urasi a
E urasi a
6
w/o perm afrost
w/permaf rost
w/o perm afrost
w/permaf rost
7
8
9
Nort h A merica
Nort h A merica
10
E urasi a
E urasi a
11
0.4
0.6
0.8
1.0
Nort h A merica and E urasia E of the Ural s
Nort h A merica and E urasia E of the Ural s
E urope west of t he Urals
E urope west of t he Urals
1.2
1.4
1.6
1.8
2.0
2.2
Change in st anding biomass in minimally dist urbed Nort hern F orest s
bet ween 45°N and 65°N under specif ied 2x[C O2] climate change.
Source: Morgan et al., Climatic Change, 2001.
2.4
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Change in soil carbon in minimally dist urbed
N orthern Forests bet ween 45°N and 65°N
under specif ied 2x[CO 2 ] climate change.
Department of Engineering and Public Policy
32
Carnegie Mellon University
Biomass in Tropical Forests
w/ 2xCO2 climate change
A.
Change in standing
biomass
Expert
1
B.
Change in soil carbon
2
3
4
5
6
7
8
9
10
11
0.4
0.6
0.8
1.0
1.2
1.4
Change in s tanding biomas s in minimally
dis turbed Tropic al Fores ts betw een
20°N and 20°S under spec ified
2x [CO 2 ] c limate c hange.
0.4
0.6
0.8
1.0
1.2
1.4
Change in s oil c arbon in minimally
dis turbed Tropic al Fores ts betw een
20°N and 20°S under spec ified
2x [CO 2 ] c limate c hange.
Source: Morgan et al., Climatic Change, 2001.
Department of Engineering and Public Policy
33
Carnegie Mellon University
Radiative Forcing by
Aerosols
We were asked to do this on a
rapid pace by Ron Prinn in
connection with the 4th IGCC
assessment.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Source: Morgan et al., Climatic Change, 2006.
Department of Engineering and Public Policy
34
Carnegie Mellon University
From IPCC 3rd Assessment
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Source: Climate Change 2001: The Scientific Basis, Working Group 1 of the Third IPCC Assessment,
Cambridge University Press, 881pp., 2001.
Department of Engineering and Public Policy
35
Carnegie Mellon University
Aerosol forcing
Direct aerosol effect: change in radiative flux by scattering and absorption of
unacti vated aerosol particles in the absence of any other climate changes
or feedbacks.
Semi-direct aerosol effect: change in radiative flux resulting from a c hange
in cloud formation because of local heating by black carbon aerosols.
First aerosol in direct effect (brightness): change in cloud reflectivity
resulti ng from a change in cloud condensation nuclei holding other cloud
properties constant (e.g. total liqu id water and cloud cover).
Second aerosol indirect effect (lifetime): change in cloud cover/lifeti me
resulti ng from a change in cloud condensation nuclei.
aerosols 
direct
effects
direct aerosol effect
(scattering and absorption
from aerosols)
semi-direct aerosol
effect
(change in cloud
formation due to local
heating from black
carbon absorption)
indirect
effects
first aerosol indirect
effect
(cloud w hiteness)
second aerosol indirect
effect
(cloud lifetime effect)
Department of Engineering and Public Policy
36
Carnegie Mellon University
The experts who participated
Name
Andrew Ackerman
Bruce Albrecht
Theodore Anderson
Meinrat O. Andreae
Mary Barth
Oli vier Boucher
Antony Clarke
Willi am Cotto n
Johann Feichter
Steve Ghan
Mark Z. Jacobson
Ralph Kahn
Yoram Kaufman
Jeff Kiehl
Stefan Kinne
Ulrike Lohmann
Surabi Menon
Dan Murphy
Athanasios Nenes
Spyros N. Pandis
Ronald G. Prinn
Phil Rasch
Steve Schwartz
John Seinfeld
Affili ation
NASA Ames Research
University of Miami
University of Washington
Max Planck Institute for Chemi stry
National Center for Atmospheric Research
Laboratoire dÕOptique Atmosphˇr ique, CNRS
University of Hawaii
Colorado State
Max Planck Institute for Meteorology
Pacifi c Northwest National Laboratory
Stanford University
NASA Š Jet Propulsion Laboratory
NASA Š Goddard Space Fli ght Center
National Center for Atmospheric Research
Max Planck Institute of Meteorology
ETH Zurich
Lawrence Berkeley National Laboratory
NOAA Š Aeronomy Laboratory
Georgia Institute of Technology
Carnegie Mell on University
Massachusetts Institute of Technology
National Center for Atmospheric Research
Brookhaven National Laboratory
Calif ornia Institute of Technology
Department of Engineering and Public Policy
37
Carnegie Mellon University
Scattering and absorption
(also called the direct aerosol effect)
Department of Engineering and Public Policy
38
Carnegie Mellon University
Change in cloud formation due to
local heating by black carbon
(also called the Semi-Direct Effect black carbon)
Department of Engineering and Public Policy
39
Carnegie Mellon University
Cloud Brightness
(First aerosol indirect effect)
Department of Engineering and Public Policy
40
Carnegie Mellon University
Cloud Lifetime
(Second aerosol indirect effect)
Department of Engineering and Public Policy
41
Carnegie Mellon University
Total
aerosol
forcing
(at the top of the
atmosphere)
Department of Engineering and Public Policy
42
Carnegie Mellon University
Surface
Forcing
Department of Engineering and Public Policy
43
Carnegie Mellon University
Comparison with IPCC
consensus results
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Sources: IPCC TAR WG1
Morgan et al, Climatic Change, in press.
Department of Engineering and Public Policy
44
Carnegie Mellon University
Climate impacts on the AMOC
Present state and future fate of the Atlantic meridional
overturning circulation as viewed by experts.
K. Zickfeld, A. Levermann, T. Kuhlbrodt and S. Rahmstorf
Potsdam Institute for Climate Impact Research
G. Morgan
Carnegie Mellon University
D. Keith
University of Calgary
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Climatic Change, in revision.
Source: UNEP
Department of Engineering and Public Policy
45
Carnegie Mellon University
Strength of
the AMOC
Department of Engineering and Public Policy
46
Carnegie Mellon University
2xCO2
Transient
response
4xCO2
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Department of Engineering and Public Policy
47
Carnegie Mellon University
Strength in 2100
w/ 2xCO2
w/ 4xCO2
Department of Engineering and Public Policy
48
Carnegie Mellon University
Probability
of >90%
collapse
Department of Engineering and Public Policy
49
Carnegie Mellon University
SOX health effects
…elicited subjective
judgments about oxidation
rates and wet and dry
deposition rates for SO2
and SO4= from nine
atmospheric science
experts.
Source: G. Morgan, S. Morris, M. Henrion, D. Amaral, W. Rish, "Technical Uncertainty in Quantitative Policy Analysis: A sulfur air
pollution example,” Risk Analysis, 4, 201-216, 1984.
Department of Engineering and Public Policy
50
Carnegie Mellon University
SOX health
effects…(Cont.)
Then for each expert we built a
separate plume model,
exposing people on the ground
using known population
densities.
Source: G. Morgan, S. Morris, M. Henrion, D. Amaral, W.
Rish, "Technical Uncertainty in Quantitative Policy Analysis:
A sulfur air pollution example,” Risk Analysis, 4, 201-216,
1984.
Department of Engineering and Public Policy
51
Carnegie Mellon University
SOX health
effects…(Cont.)
We elicited health damage
functions for sulfate aerosol
from seven health experts (of
whom only five were able to
give us probabilistic
estimates).
Finally, for each air expert,
we did an analysis using the
health damage function of
each health expert…
Source: D. Amaral, "Estimating Uncertainty in Policy
Analysis: Health effects from inhaled sulfur oxides,"
Ph.D. thesis, Department of Engineering and Public
Policy, Carnegie Mellon, 1983.
Department of Engineering and Public Policy
52
Carnegie Mellon University
SOX health effects…(Cont.)
The results showed that while there was great discussion
about uncertainty in the atmospheric science, the uncertainty
about the health damage functions completely dominated the
estimate of health impacts.
CDF of deaths/yr
from a new
(in 1984) supercritical 1GWe FGD
equipped coal-fired
plant in Pittsburgh.
Availability factor =
73%, net efficiency
= 35%.
Source: G. Morgan, S. Morris, M. Henrion, D. Amaral, W. Rish, "Technical Uncertainty in Quantitative Policy Analysis: A sulfur air
pollution example,” Risk Analysis, 4, 201-216, 1984.
Department of Engineering and Public Policy
53
Carnegie Mellon University
Multiple Experts
When different experts hold different views it is often best
not to combine the results, but rather to explore the
implications of each expert's views so that decision makers
have a clear understanding of whether and how much the
differences matter in the context of the overall decision.
However, sophisticated methods have been developed that
allow experts to work together to combine judgments so as to
yield a single overall composite judgment.
The community of seismologists have made the greatest
progress in this direction through a series of very detailed
studies of seismic risks to built structures (Hanks, T.C., 1997;
Budnitz, R.J. et al., 1995).
Department of Engineering and Public Policy
54
Carnegie Mellon University
Uncertainty versus variability
Variability involves random change over time or space (e.g.,
"the mid-day temperature in Beijing in May is variable").
Recently, in the U.S., some people have been drawing a sharp
distinction between variability and uncertainty. While the
two are different, and sometimes require different treatments,
the distinction can be overdrawn. In many contexts,
variability is simply one of several sources of uncertainty
(Morgan and Henrion, 1990).
One motivation people have for trying to sharpen the
distinction is that variability can often be measured
objectively, while other forms of uncertainty require
subjective judgment.
Department of Engineering and Public Policy
55
Carnegie Mellon University
Recently several investigators…
…have been combining models and the use of
subjective judgment to bound or estimate parameter
uncertainty. I will mention 4 examples:
Andronova and Schlesinger, 2001
Forest et al., 2006
Frame et al., 2005
Stainforth et al., 2005
Department of Engineering and Public Policy
56
Carnegie Mellon University
Andronova and Schlesinger, 2001
Used the Bayesian MonteCarlo simulation method,
specified the known
uncertainties in forcing with
a wide prior distribution for
climate sensitivity and tested
model outputs of the
evolution of global
temperature against observed
historic global temperature data. Each model run was weighted
(for likelihood of being true and accurate) according to its
"error" with respect to the data. This comparison permits the
re-weighting of the prior distribution used for climate
sensitivity to develop a posterior distribution consistent with
our knowledge of the different forcing functions and the
empirical data on temperature trends.
Source: Andronova, N, and M. E. Schlesinger, "Objective Estimation of the Probability Distribution
for Climate Sensitivity," J. Geophys. Res., 106 , D19, 22,605-22,612, 2001.
Department of Engineering and Public Policy
57
Carnegie Mellon University
Forest et al., 2006
Marginal posterior probability
density function obtained
when using uniform
probability distributions across
all relevant forcings and
matching outputs from the
ocean and atmospheric portion
of the MIT IGSM model.
Light contours bound the 10%
and 1% significance regions.
Similarly, the two dark contours are for an expert PDF on climate
sensitivity. Dots show outputs from a range of leading GCM’s all
of which lie to the right of the high-probability region, suggesting
that if Forest et al. are correct, these models may be mixing heat
into the deep ocean too efficiently.
Forest, Chris E, and Peter H. Stone and, and Andrei P. Sokolov, "Estimated PDF’s of Climate System Properties Including Natural and
Anthropogenic Forcings," Geophysical Research Letters, Vol. 3333, pp xx-xx, 2006.
Department of Engineering and Public Policy
58
Carnegie Mellon University
Frame et al., 2005
Relationship between
climate sensitivity (light
contours), effective ocean
heat capacity, and 20th
century warming for the
case of uniform sampling of
climate sensitivity (not
shown are similar results
for uniform sampling across feedback strength). The dark
contour shows the region consistent with observations at the
5% level.
Frame, D.J, B. B. B. Booth, J. A Kettleborough, D.A. Stainforth, J.M. Gregory, M. Collins, and M.R. Allen, "Constraining Climate
Forecasts: The role of prior assumptions," Geophysical Research Letters, 32, 2005.
Department of Engineering and Public Policy
59
Carnegie Mellon University
Stainforth et al., 2005
Histogram of climate
sensitivities found by in
their simulation of
2,578 versions of the
HadSM3 GCM model.
"We find model versions as realistic as other state-of-the-art climate models but with climate
sensitivities ranging from less than 2K to more than 11K. Models with such extreme sensitivities
are critical for the study of the full range of possible responses of the climate system to rising
greenhouse gas levels, and for assessing the risks associateds with a specific target for stabilizing
these levels…
The range of sensitivity across different versions of the same model is more than twice that found
in the GCM’s used in the IPCC Third Assessment Report…The possibility of such high sensitivities
has been reported by studies using observations to constrain this quantity, but this is the first time
that GCM’s have generated such behavior."
Stainforth, D.A., T. Aina, C. Christensen, M. Collins, N. Faull, D. J. Frame, J. A. Kett;eborough, S. Knight, A. Martin, J.M. Murphy, C. Piani, D.
Sexton, L. A. Smith, R. A. Spicer, A. J. Thorpe, and M. R. Allen, "Uncertainty in Predictions of the Climate Response to Rising Levels of
Greenhouse Gases," Nature, 433, pp. 403-406, 2005.
Department of Engineering and Public Policy
60
Carnegie Mellon University
This morning I will talk about:
• Sources of uncertainty and the characterization of
uncertainty.
• Two basic types of uncertainty.
• Uncertainty about coefficient values.
• Uncertainty about model functional form.
• Performing uncertainty analysis and making decisions in
the face of uncertainty.
• Public perceptions and communication.
• Some summary guidance on reporting, characterizing
and analyzing uncertainty.
Department of Engineering and Public Policy
61
Carnegie Mellon University
Uncertainty about model form
Often uncertainty about model form is as or more important
than uncertainty about values of coefficients. Until recently
there had been little practical progress in dealing with such
uncertainty, but now there are several good examples:
• John Evans and his colleagues at the Harvard School
of Public Health (e.g. Evans et al., 1994).
• Alan Cornell and others in the seismic risk (e.g.
Budnitz et al., 1995).
• Hadi Dowlatabadi and colleagues at Carnegie Mellon
in Integrated Assessment of Climate Change -ICAM
(e.g. Morgan and Dowlatabadi, 1996).
• Also on climate, Lempert and colleagues at RAND
(e.g. Lempert, Popper, Bankes, 2003).
Department of Engineering and Public Policy
62
Carnegie Mellon University
John Evans
and colleagues….
…have developed a method
which lays out a
"probability tree" to
describe all the plausible
ways in which a chemical
agent might cause harm.
Then experts are asked to
assess probabilities on each
branch.
For details see: John S. Evans et al., "A distributional approach to characterizing low-dose
cancer risk," Risk Analysis, 14, 25-34, 1994; and John S. Evans et al., "Use of probabilistic
expert judgment in uncertainty analysis of carcinogenic potency," Regulatory Toxicology and
Pharmacology, 20, 15-36, 1994.
Department of Engineering and Public Policy
63
Carnegie Mellon University
ICAM
Integrated Climate
Assessment Model
To run the model:
1 - Double click on INPUTS to set up the scenario inputs;
2 - Double click on STRUCTURE to set up the model;
3 - Double click on OUTPUTS and evaluate the indicators.
Demographics
& Economics
A very large hierarchically
organized stochastic
simulation model built
in Analytica®.
Impacts of
Climate Change
INTERVENTION
Atmospheric
Composition &
Climate
Energy &
Emissions
Inputs
Structure
Outputs
See for example:
Hadi Dowlatabadi and M. Granger Morgan, "A Model Framework for
Integrated Studies of the Climate Problem," Energy Policy, 21(3), 209221, March 1993.
and
M. Granger Morgan and Hadi Dowlatabadi, "Learning from Integrated
Assessment of Climate Change," Climatic Change, 34, 337-368, 1996.
Aerosol
Model
GHG
Models
RITS
Conc
Change in
Short Wave
Forcing
Change in
Long Wave
Forcing
Forcing
Elicited
Climate Model
Department of Engineering and Public Policy
Regional
²T
64
Carnegie Mellon University
ICAM deals with...
…both of the types of uncertainty I've talked about:
1. It deals with uncertain coefficients by assigning PDFs
to them and then performing stochastic simulation to
propagate the uncertainty through the model.
2. It deals with uncertainty about model functional form
(e.g., will rising energy prices induce more technical
innovation?) by introducing multiple alternative
models which can be chosen by throwing "switches."
Department of Engineering and Public Policy
65
Carnegie Mellon University
ICAM
There is not enough time to present any details from our work
with the ICAM integrated assessment model. Here are a few
conclusions from that work:
• Different sets of plausible model assumptions give
dramatically different results.
• No policy we have looked at is dominant over the wide
range of plausible futures we’ve examined.
• The regional differences in outcomes are so vast that few
if any policies would pass muster globally for similar
decision rules.
• Different metrics of aggregate outcomes (e.g., $s versus
hours of labor) skew the results to reflect the OECD or
developing regional issues respectively.
Department of Engineering and Public Policy
66
Carnegie Mellon University
These findings lead us...
...to switch from trying to project and examine the future, to
using the modeling framework as a test-bed to evaluate the
relative robustness, across a wide range of plausible model
futures, of alternative strategies that regional actors in the
model might adopt.
We populated the model's regions with simple decision agents
and asked, which behavioral strategies are robust in the face
of uncertain futures, which get us in trouble.
Thus, for example, it turns out that tracking and responding to
atmospheric concentration is more likely to lead regional
policy makers in the model to stable strategies than tracking
and responding to emissions.
Department of Engineering and Public Policy
67
Carnegie Mellon University
Our conclusion
Prediction and policy optimization are pretty silly analytical
objectives for much assessment and analysis related to the climate
problem.
It makes much more sense to:
• Acknowledge that describing and bounding a range of
futures may often be the best we can do.
• Recognize that climate is not the only thing that is changing,
and address the problem in that context.
• Focus on developing adaptive strategies and evaluating their
likely robustness in the face of a range of possible climate,
social, economic and ecological futures.
Recent work by Robert Lempert and colleagues takes a very
similar approach.
Department of Engineering and Public Policy
68
Carnegie Mellon University
Source: Lempert and Schlesinger, 2002
Lempert &
co-workers…
Qu i c k Ti m e ™ an d a
TIF F (LZ W) d ec om pres s o r
are n ee de d to s ee th is pi c tu re .
Qu i c k Ti m e ™ an d a
TIF F (LZ W) d ec om pres s o r
are n ee de d to s ee th is pi c tu re .
…have used a similar
decision making
approach to identify nearterm emissions mitigation
policies which are robust
over a wide range of
potential futures. They generate hundreds to millions of
plausible future states of the world, and then use statistical
methods, supported by interactive search and visualization to
help decision makers: design and choose robust strategies which
perform reasonably well across a very wide range of the
plausible futures; identify residual vulnerabilities of those
strategies: and assess the tradeoffs involved in addressing these
vulnerabilities.
See for example: Robert J. Lempert, Steven W. Popper and Steven C. Bankes, Shaping the Next One Hundred Years: N
New methods for quantitative long-Term term Policy policy analysis, RAND MR-1626-RPC, August 2003.
Department of Engineering and Public Policy
69
Carnegie Mellon University
This morning I will talk about:
• Sources of uncertainty and the characterization of
uncertainty.
• Two basic types of uncertainty.
• Uncertainty about coefficient values.
• Uncertainty about model functional form.
• Performing uncertainty analysis and making decisions in
the face of uncertainty.
• Public perceptions and communication.
• Some summary guidance on reporting, characterizing
and analyzing uncertainty.
Department of Engineering and Public Policy
70
Carnegie Mellon University
Propagation and analysis of uncertainty
Consider a basic model
of the form y = f(x 1, x 2)
The simplest form of
uncertainty analysis is
sensitivity analysis using
the slopes of y with
respect to the inputs x 1
and x 2
Source: Morgan and Henrion, 1990
Department of Engineering and Public Policy
71
Carnegie Mellon University
Nominal Range Sensitivity
Source: Morgan and Henrion, 1990
Department of Engineering and Public Policy
72
Carnegie Mellon University
Propagation of continuous distributions
There are many software tools
available such as Analytica,
@Risk, and Crystal Ball.
Source: Morgan and Henrion, 1990
Department of Engineering and Public Policy
73
Carnegie Mellon University
Tools for analysis
Tools for continuous processes:
Tools for discrete events:
• Exposure models
• Dose response functions
• etc.
• Failure modes and effects analysis
• Fault tree models
• etc.
To run the model:
1 - Double click on INPUTS to set up the scenario inputs;
2 - Double click on STRUCTURE to set up the model;
3 - Double click on OUTPUTS and evaluate the indicators.
Demographics
& Economics
Use of some of these tools used to
be very challenging and time
consuming. Today such analysis is
facilitated by many software tools
(e.g. Analytica®, @risk®, Crystal
Ball®, etc.).
Expenditures
to limit losses
Impacts of
Climate Change
INTERVENTION
Los ses due to
climate c hange
Regional GDP
in 1975
Gross
GDP
Total
Mitigation Costs
& Loss es
GDP net of
climate c hange
Atmospheric
Composition &
Climate
Energy &
Emissions
Produc tivity
Growth
Economic
Growth
Population
Growth
Population
Regional Pops
in 1975
Inputs
$ Per cap los s
due to CC
Disc. per cap
GDP Los s
due to CC
Structure
Outputs
Discounted
Los t GDP
Discount
Rate
Department of Engineering and Public Policy
74
Carnegie Mellon University
All that fancy stuff…
…is only useful when one has some basis to predict functional
relationships and quantify uncertainties about coefficient values.
When one does not have that luxury, order-of-magnitude
arguments and bounding analysis (based on things like
conservation laws, etc.) may be the best one can do.
The conventional decision-analytic model assumes that research
reduces uncertainty:
Research
$
Department of Engineering and Public Policy
75
Carnegie Mellon University
Research and uncertainty…(Cont.)
Sometimes this is what happens, but often, for variables or
processes we really care about, the reality is that uncertainty
either remains Research
unchanged…
$
Research
$
or even grows…
When the latter happens it is often because we find that the
underlying model we had assumed is incomplete or wrong.
Department of Engineering and Public Policy
76
Carnegie Mellon University
There are formal methods
to support decision making…
…such as B-C analysis, decision
analysis, analysis based on multiattribute utility theory, etc.
Things analysts need to remember:
Be explicit about decision rules (e.g., public
health versus science; degree of precaution; etc.).
Get the value judgments right and keep them
explicit.
Don't leave out important issues just because
they don't easily fit.
Remember that many methods become opaque
very quickly, especially to non-experts.
When uncertainty is high, it is often best to look
for adaptive as opposed to optimal strategies.
Source: Henrion, Lumina Systems
Department of Engineering and Public Policy
77
Carnegie Mellon University
A classic strategy for
solving problems:
Identify the problem
When uncertainty is high
(and perhaps irreducible) an
iterative adaptive strategy is
better:
Identify a problem
Do research
Do research
Learn what you can
and what you can’t
know (at least now).
Gain full understanding
of all relevant issues
Identify adaptive
policies and choose
one that currently
looks best
Identify policy
options
Inplement the
optimal policy
Solve the problem
Continue research
implement policy
and observe how
it works
Reassess policy
in light of new
understanding
Refine problem
identificaton as
needed
Department of Engineering and Public Policy
78
6
5
Expert 1
4
Probability of a change
in the climate state
Limited
domain of
model validity
Mean change in
global temperature,°C
Carnegie Mellon University
0.30
0.20
3
2
0.10
1
0.00
2000
2025
2050
2075
2100
Probability of a change
in the climate state
Mean change in
global temperature,°C
Year
6
5
Expert 8
0.30
4
0.20
3
2
0.10
1
0
1975
0.00
2000
2025
2050
2075
2100
Probability of a change
in the climate state
Year
Mean change in
global temperature,°C
Examples of warming
estimated via the
ICAM model (dark
curves) and probability
that the associated
climate forcing will
induce a state change
in the climate system
(light curves) using the
probabilistic judgments
of three different
climate experts.
0
1975
6
5
Expert 15
0.30
4
0.20
3
2
0.10
1
0
1975
0.00
2000
2025
2050
2075
2100
Year
Department of Engineering and Public Policy
79
Carnegie Mellon University
Model
switching
Schematic illustration of
the strategy of switching
to progressively simpler
models as one moves into
less well understood
regions of the problem
phase space, in this case,
over time.
Department of Engineering and Public Policy
80
Carnegie Mellon University
Illustration
of model
switching
Results of applying
the model switchover strategy to the
ICAM demographic
model (until about
2050) and an
estimate of the
upper-bound
estimate of global
population carrying
capacity based on
J. S. Cohen.
Department of Engineering and Public Policy
81
Carnegie Mellon University
This morning I will talk about:
• Sources of uncertainty and the characterization of
uncertainty.
• Two basic types of uncertainty.
• Uncertainty about coefficient values.
• Uncertainty about model functional form.
• Performing uncertainty analysis and making decisions in
the face of uncertainty.
• Public perceptions and communication.
• Some summary guidance on reporting, characterizing
and analyzing uncertainty.
Department of Engineering and Public Policy
82
Carnegie Mellon University
The traditional approach…
…to risk communication involves two steps:
1. Ask an expert what people should be told.
2. Get a "communications expert" to package it.
And, if you are being really fancy:
3. Run some tests to see how people like it.
However, if you think about it for a few minutes, this
approach ignores two critical issues:
•
What the people receiving the message already
"know" about the topic.
•
What they need to know to make the decisions they
face.
Department of Engineering and Public Policy
83
Carnegie Mellon University
This led us to
develop a new
approach
Cambridge University Press, 351pp., 2002
Department of Engineering and Public Policy
84
Carnegie Mellon University
Finding out what people know…
…is not simple! If I give people a questionnaire, I have to
put information in my questions.
People are smart. They will start making inferences
based on the information in my questions.
Pretty soon I won't know if their answers reflect what
they already knew, or the new ideas they have come up
with because of the information I have given them.
I need a better, less intrusive way to learn what they
know. The method we have developed to do this is called
the mental model interview.
Department of Engineering and Public Policy
85
Carnegie Mellon University
Five step process:
In work done over the past decade at Carnegie Mellon, we
have developed a five-step approach to risk communication,
based on people's mental models of risk processes:
2. Conduct an openended elicitation of
people's beliefs
about a hazard,
allowing the
expression of both
accurate and
inaccurate concepts.
Map the results to
the expert influence
diagram.
Cumulative number of concepts
1. Develop an "influence diagram" to structure expert
knowledge.
160
B2
120
S1
B3
S2
S3
S4
80
B1
A
40
0
1
10
20
30
Number of interviews
Department of Engineering and Public Policy
86
Carnegie Mellon University
Five steps…(Cont.)
3. On the basis of results from the open-ended interviews,
develop and administer a closed-form questionnaire to a much
larger group in order to determine the prevalence of these
beliefs.
4. Develop a draft communication based on both a decision
analytic assessment of what people need to know in order to
make informed decisions and a psychological assessment of
their current beliefs.
5. Iteratively test successive versions of those communications
using open-ended, closed-form, and problem-solving
instruments, administered before, during, and after the receipt
of the message.
In 1994, we applied these methods to study public understanding
of climate change. While the results are now getting old, I will
show you a few since they are unlikely to have changed much.
Department of Engineering and Public Policy
87
Carnegie Mellon University
Example of opening response
in interview
Interviewer: "I'd like you to tell me all about the issue of
climate change."
Subject: "Climate change. Do you mean global warming?"
Interviewer: "Climate change."
Subject: "OK. Let's see. What do I know. The earth is getting
warmer because there are holes in the atmosphere and this is
global warming and the greenhouse effect. Um... I really
don't know very much about it, but it does seem to be true.
The temperatures do seem to be kind of warm in the winters.
They do seem to be warmer than in the past.. and.. hmm..
That's all I know about global warming.
Department of Engineering and Public Policy
88
Carnegie Mellon University
Another example…
Interviewer: "Tell me all about the issue of climate
change."
Subject: "I'm pretty interested in it... The ice caps are
melting -- the hole in the ozone layer. They think
pollution from cars and aerosol cans are the cause of
all that. I think the space shuttle might have
something to do with it too, because they always
send that up through the earth, to get out in outer
space. So I think that would have something to do
with it, too."
Department of Engineering and Public Policy
89
Carnegie Mellon University
Another example…
Interviewer: "Tell me all about the issue of climate change."
Subject: "Climate change? Like, what about it? Like, as far as
the ozone layer and ice caps melting, water level raising,
rainforest going down, oxygen going down because of that?
All of that kind of stuff?"
Interviewer: "Anything else?"
Subject: "Well, erosion all over the place. Um, topsoils going
down into everywhere. Fertilizer poisoning. "
Interviewer: "Anything else that comes to mind related to
climate change?
Subject: "Climate change. Winter's ain't like they used to be.
Nothing's as severe. Not as much snow. Nothing like that."
Department of Engineering and Public Policy
90
Carnegie Mellon University
Studies of laypeople's understanding
We conducted both open-ended "mental model
interviews," and closed-form questionnaire studies
(N = 177).
Respondents regarded global warming as bad, and highly
likely. Many believed substantial warming has already
occurred.
Cumulative probability
10 0
T to d ate:
T in 1 0 years :
IPCC
=.45
=.08
IPCC
=.72
=.15
IPCC
=1.8
= .35
Our
respondents
=3.2
=4.3
median=1.7
Our
respondents
=2.7
=3.3
median=1.9
50
T in 5 0 years :
Our
respondents
=6.5
=9.1
median=4.2
0
0
10
0
10
20
30 -10
0
10
20
30
Temperature increase, °C
Department of Engineering and Public Policy
91
Carnegie Mellon University
Overall…
…our respondents had a poor appreciation of the fact that:
1) if significant global warming occurs, it will be primarily
the result of an increase in the concentration of carbon
dioxide in the earth's atmosphere; and
2) the single most important source of carbon dioxide
additions is the combustion of fossil fuels, most notably
coal and oil.
For details see:
Ann Bostrom, M. Granger Morgan, Baruch Fischhoff and Daniel Read, "What Do People
Know About Global Climate Change? Part 1: Mental models," Risk Analysis, 14(6),
959-970, 1994.
Daniel Read, Ann Bostrom, M. Granger Morgan, Baruch Fischhoff and Tom Smuts, "What
Do People Know About Global Climate Change? Part 2: Survey studies of educated
laypeople," Risk Analysis, 14(6), 971-982, 1994.
Department of Engineering and Public Policy
92
Carnegie Mellon University
A few general conclusions
on risk communication
There is no such thing as an "expert" in public
communication who can simply tell you what to do.
An empirical approach is essential.
Before developing a communication one must learn
what the public knows and thinks.
The mental models method offers a promising strategy
for doing this.
Uncertainty must be quantified.
Department of Engineering and Public Policy
93
Carnegie Mellon University
A recent
example of…
…of a study on public
perception that could turn
out to be very important for
climate abatement policy.
My colleagues and I
recently produced a report
for the Pew Center on
Climate Change which
addresses the problems of
the electricity industry and
climate change.
Available at: http://wpweb2k.tepper.cmu.edu/ceic/publications.htm
Department of Engineering and Public Policy
94
Carnegie Mellon University
The U.S. makes most if its
electricity from coal
Coal 51.2%
Nuclear 19.9%
Gas 16.6%
Hydro 7.2%
Oil 3.1%
Geothermal 0.34%
Wind 0.28%
Solar 0.01%
16000
Most
U.S. coal
plants
today
are
20 to 50
years
old.
12000
8000
4000
0
0
10 20
30 40 50 60 70
Age of coal plants in years
Many coal plants are old and
will soon need to be replaced.
Department of Engineering and Public Policy
95
Carnegie Mellon University
CO2 Capture and Sequestration (CCS)
There are several strategies.
The two closest to commercial use are:
1. Post-combustion
separation after
combustion in air.
electric power
air
coal
(or oil or natural gas)
power
plant
flue gas
N2 , SOx, NOx, etc.
separation
plant
CO2
To a deep geological formation
or the deep ocean.
2. Pre-combustion separation.
electric power
air
coal
(or oil or natural gas)
gasification
plant
hydrogen
power
plant
water vapor, NOx
CO2
other uses
sulfur and
other wastes
To a deep geological formation
or the deep ocean.
Department of Engineering and Public Policy
96
Carnegie Mellon University
Four examples of
existing facilities
Sleipner field in the
Norwegian North Sea
Great Planes
Coal Gasification
Plant
Source: Statoil
Shady Point, Oklahoma
Source: DoE
Salah gas project, Algeria: BP Amoco, Statoil and Sonatrach
Source: AES Shady Point, Inc.
Source: BP
Department of Engineering and Public Policy
97
Carnegie Mellon University
The U.S. already injects lots of fluid
The mass of current U.S. fluid injections is greater than the
mass of current power plant CO2 emissions.
10000
Large quantities
Long
Time
Frame
1000
100
~28Mt
Mt/year
Sub-seabed
Gases
CO2 from
all U.S.
power plants
10
1
FL Municipal Oilfield
Wastewater Brine
Hazardous
Waste
Acid
Gas
Compiled by E. Wilson with data from EPA, 2001;
Deurling, 2001; Keith, 2001,; DOE, 2001.
Natural Gas CO2 for OCS water
OCS
Storage
EOR injected for
gases
EOR and (e.g. NG)
brine
disposal
Department of Engineering and Public Policy
98
Carnegie Mellon University
Study design
Co-authors are Claire Palmgren, Wandi Bruine de Bruin and David
Keith.
Claire and I first ran an open-form mental model interview study
several years ago. On the basis of those results, we then developed a
closed-form survey with a Pittsburgh sample. N = 124.
1.
2a.
2b.
3.
4.
5.
5.1
5.2
6.
7.
8.
Background
Climate change
General issues including climate change
Options for limiting CO2
Energy systems that dispose of CO2
Places to dispose of CO2
Deep rock formations
Disposal of CO2 in the deep ocean
Final evaluation
Environmental issues (NEPS = New Ecological Paradigm Scale)
A few questions about yourself
Claire R. Palmgren, M. Granger Morgan, Wändi Bruine de Bruin, and David W. Keith, "Initial Public Perceptions of Deep
Geological and Oceanic Disposal of Carbon Dioxide," Environmental Science & Technology, 38(24), 6441-6450, 2004.
Department of Engineering and Public Policy
99
Carnegie Mellon University
Views on climate change
The continuing release of CO2 into the
earth's atmosphere during this century
may result in serious climate change.
Government regulation should begin to
significantly limit the amount of CO2 that
is released into the earth's atmosphere.
4.6
4.7
Government regulation will begin to
significantly limit the amount of CO2 that
is released into the earth's atmosphere at
some time in the next 20 years.
4.0
1
completely
disagree
4
7
completely
agree
Department of Engineering and Public Policy
100
Carnegie Mellon University
Options
"Whatever your own beliefs are about
climate change, imagine that the US
government has decided that we must cut
in half the amount of CO2 that is released
by generating electric power.
Suppose that your electricity supplier has
different methods of meeting the goal to
reduce CO2 emissions by 50%. Some of
these methods use a mixture of generation
systems that produce little or no CO2,
combined with regular coal-burning power
plants. These methods do not all cost the
same because some ways of making
electricity with less CO2 are cheaper than
others.
The supplier will be offering their
customers a choice of how they would like
them to meet this reduction. Some of the
options to reduce CO2 emissions will be
more expensive than the way we produce
electricity today, so we would like you to
tell us which method you would be willing
to pay the most for."
Options that will redu ce CO2 emissions by 50%
from your energy consumpti on
Rank
1 is what you
would pay the
most for.
Biomass Blend:
50% from bi omass, 50% from regular coa l
Half of the energy comes from regular coal, and the other half comes from bioma ss. Biomass
refers to harvesting and burning special fast growing grasses and trees to make energy. Biomass
releases CO 2 that plants have absorbed from the atmosphere so no new CO2 is produced.
Coal Blend with deep geological di sposal of CO2:
50% from coal with deep geological di sposal o f CO2, 50% from regular coa l
Half of the energy comes from regular coal. The other half also comes from coal, but won't let the
CO 2 enter the atmosphere. Instead they will separate the CO2 and dispose of it thousands of feet
underground.
Coal Blend with deep ocea n di sposal of CO2:
50% from coal with deep ocea n di sposal of CO2, 50% from regular coa l
Half of the energy comes from regular coal. The other half also comes from coal, but won't let the
CO 2 enter the atmosphere. Instead they will separate the CO2 and dispose of it deep in the ocean.
Energy Efficiency:
100% regular coa l
All of the energy comes from regular coal. This option will replace appliances and lights in your
house with much more efficient ones so that you use half as much electricity while enjoying using
your lights and appliances just as much as you do now. Using less electricity will result in less
CO 2 being produced.
Hydro-electric Blend:
50% from hydro-electric, 50% from reg ular coal
Half of the energy comes from regular coal, and the other half comes from hydro-electric. Hydroelectric refers to making electricity with water power from dams . Hyd ro-electric produces no
CO 2.
Natural gas Blend:
100% from natural gas
The energy comes from burning natural gas in highly efficient plants. Burning natural gas
efficiently produces about half as much CO2 as burning coal.
Nuclear power Blend:
50% from nuclear power , 50% from regular coal
Half of the energy comes from regular coal, and the other half comes nuclear power.
Nuclear powe r produces no CO 2.
Solar power Blend:
50% from solar pow er, 50% from regular coa l
Half of the energy comes from regular coal, and the other half comes from solar. Solar powe r
refers to making electricity with energy from the sun. Solar powe r produces no CO 2.
Wind power Blend:
50% from wind po wer, 50% from regular coa l
Half of the energy comes from regular coal, and the other half comes from wind. Wind power
refers to making electricity with energy from the wind. Wind power produces no CO 2.
Department of Engineering and Public Policy
101
Carnegie Mellon University
Rank order
of options
Options used different
generation mixes (base of
coal) in order to reduce
CO2 emissions by 50%.
Department of Engineering and Public Policy
102
Carnegie Mellon University
Summary evaluations
Geological disposal:
before details
after details
1
completely
oppose
4
7
completely
favor
Ocean disposal:
before details
after details
1
completely
oppose
4
7
completely
favor
Department of Engineering and Public Policy
103
Carnegie Mellon University
This morning I will talk about:
• Sources of uncertainty and the characterization of
uncertainty.
• Two basic types of uncertainty.
• Uncertainty about coefficient values.
• Uncertainty about model functional form.
• Performing uncertainty analysis and making decisions in
the face of uncertainty.
• Public perceptions and communication.
• Some summary guidance on reporting, characterizing
and analyzing uncertainty.
Department of Engineering and Public Policy
104
Carnegie Mellon University
CCSP Guidance Document
Along with several
colleagues I am currently
completing a guidance
document for the US Climate
Change Science Program.
The final slides reproduce
our draft summary advice.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
I've provided the meeting
organizers with a draft copy.
We'd welcome feedback and
suggestions.
Department of Engineering and Public Policy
105
Carnegie Mellon University
Doing a good job…
…of characterizing and dealing with uncertainty can never be
reduced to a simple cookbook. One must always think critically
and continually ask questions such as:
• Does what we are doing make sense?
• Are there other important factors which are as or more
important than the factors we are considering?
• Are there key correlation structures in the problem which
are being ignored?
• Are there normative assumptions and judgments about
which we are not being explicit?
That said, the following are a few words of guidance…
Department of Engineering and Public Policy
106
Carnegie Mellon University
Reporting uncertainty
When qualitative uncertainty words (such as likely and unlikely)
are used, it is important to clarify the range of subjective
probability values are to be associated with those words. Unless
there is some compelling reason to do otherwise, we recommend
the use of the framework shown below:
v ertually c ertain
> 0.99
v ery likely
~ 0.8 to < 0.99
likely
greater than an ev en c hanc e
> 0.5 to ~ 0.8
about an ev en c hanc e
~ 0.4 to ~ 0.6
unlikely
les s than an ev en chance
~ 0.2 to > 0.5
v ery unlikely
> 0.99 to ~ 0.2
v ertually imposs ible
> 0.99
0
0.25
0.5
0.75
1.0
probability that a statement is true
Department of Engineering and Public Policy
107
Carnegie Mellon University
Reporting uncertainty…(Cont.)
Another strategy is to display the judgment explicitly as shown:
0
0.25
0.5
0.75
1.0
probability that a statement is true
This approach provides somewhat greater precision and
allows some limited indication of secondary uncertainty for
those who feel uncomfortable making precise probability
judgments.
Department of Engineering and Public Policy
108
Carnegie Mellon University
Reporting uncertainty…(Cont.)
In any document that reports uncertainties in conventional
scientific format (e.g 3.5+0.7) it is important to be explicit about
what uncertainty is being included and what is not, and to
confirm that the range is plus or minus one standard deviation.
This reporting format is generally not appropriate for large
uncertainties or where distributions have a lower or upper bound
and hence are not symmetric.
Care should be taken in plotting and labeling the vertical axes
when reporting PDFs. The units are probability density (i.e.
probability per unit interval along the horizontal axis), not
probability.
Since many people find it difficult to read and correctly interpret
PDFs and CDFs, when space allows it is best practice to plot the
CDF together with the PDF on the same x-axis.
Department of Engineering and Public Policy
109
Carnegie Mellon University
Reporting uncertainty…(Cont.)
When many uncertain results must be reported, box plots (first
popularized by Tukey, 1977) are often the best way to do this in a
compact manor. There are several conventions. Our
recommendation is shown below, but what is most important is to
be clear about the notation.
m inim um
possible
val ue
0.05
m edian
val ue
0.25
m axim um
possible
val ue
m ean
val ue
0.75
0.95
cumulative probability values
m ovi ng from left to right
X, value of the quantity of interest
Tukey, John W., Exploratory Data Analysis, Addison-Wesley, 688pp, 1977.
Department of Engineering and Public Policy
110
Carnegie Mellon University
Reporting uncertainty…(Cont.)
While there may be circumstances in which it is desirable
or necessary to address and deal with second-order
uncertainty (e.g. how sure an expert is about the shape of
an elicited CDF), one should be very careful to determine
that the added level of such complication will aide in, and
will not unnecessarily complicate, subsequent use of the
results.
Department of Engineering and Public Policy
111
Carnegie Mellon University
Characterizing and analyzing uncertainty
Unless there are compelling reasons to do otherwise,
conventional probability is the best tool for
characterizing and analyzing uncertainty.
Department of Engineering and Public Policy
112
Carnegie Mellon University
Characterizing and analyzing…(Cont.)
The elicitation of expert judgment, often in the form of
subjective probability distributions, can be a useful way to
combine the formal knowledge in a field as reflected in the
literature with the informal knowledge and physical intuition of
experts. Elicitation is not a substitute for doing the needed
science, but if can be a very useful tool in support of research
planning, private decision making, and the formulation of public
policy.
HOWEVER the design and execution of a good expert elicitation
takes time and requires a careful integration of knowledge of the
relevant substantive domain with knowledge of behavioral
decision science.
Department of Engineering and Public Policy
113
Carnegie Mellon University
Characterizing and analyzing…(Cont.)
When eliciting probability distributions from multiple
experts, if they disagree significantly, it is generally better to
report the distributions separately then to combine them into
an artificial "consensus" distribution.
There are a variety of software tools available to support
probabilistic analysis using Monte Carlo and related
techniques. As with any powerful analytical tool, their proper
use requires careful thought and care.
Department of Engineering and Public Policy
114
Carnegie Mellon University
Characterizing and analyzing…(Cont.)
In performing uncertainty analysis, it is important to think
carefully about possible sources of correlation. One simple
procedure for getting a sense of how important this may be is
to run the analysis with key variables uncorrelated and then run
it again with key variables perfectly correlated. Often, in
answering questions about aggregate parameter values experts
assume correlation structures between the various components
of the aggregate value being elicited. Sometimes it is
important to elicit the component uncertainties separately from
the aggregate uncertainty in order to reason out why specific
correlation structures are being assumed.
Department of Engineering and Public Policy
115
Carnegie Mellon University
Characterizing and analyzing…(Cont.)
Methods for describing and dealing with data pedigree (see for
example Funtowicz and Ravetz, 1990) have not been developed to
the point that they can be effectively incorporated in probabilisitic
analysis. However, the quality of the data on which judgments
are based is clearly important and should be addressed, especially
when uncertain information of varying quality and reliability is
combined in a single analysis.
While full probabilistic analysis can be useful, in many context
simple parametric analysis, or back-to-front analysis (that works
backwards from an end point of interest) may be as or more
effective in identifying key unknowns and critical levels of
knowledge needed to make better decisions.
Funtowicz, S.O. and J.R. Ravetz, Uncertainty and quality in science for policy,
Kluwer Academic Publishers, 229 pp,1990.
Department of Engineering and Public Policy
116
Carnegie Mellon University
Characterizing and analyzing…(Cont.)
Scenarios analysis can be useful, but also carries risks. Specific
detailed scenarios can become cognitively compelling, with the
result that people may overlook many other pathways to the
same end-points. It is often best to "cut the long causal chains"
and focus on the possible range of a few key variables which can
most affect outcomes of interest.
Scenarios which describe a single point (or line) in a multidimensional space, cannot be assigned probabilities. If, as is
often the case, it will be useful to assign probabilities to
scenarios, they should be defined in terms of intervals in the
space of interest, not in terms of point values.
Department of Engineering and Public Policy
117
Carnegie Mellon University
Characterizing and analyzing…(Cont.)
Analysis that yields predictions is very helpful when our
knowledge is sufficient to make meaningful predictions.
However the past history of success in such efforts suggests
great caution (see for example Ch.s 3 and 6 in Smil, 2005).
When meaningful prediction is not possible, alternative
strategies, such as searching for responses or policies that will
be robust across a wide range of possible futures, deserve
careful consideration.
Smil, Vaclav, Energy at the Crossroads, MIT Press, 448pp, 2005.
Department of Engineering and Public Policy
118
Carnegie Mellon University
Characterizing and analyzing…(Cont.)
For some problems there comes a time when uncertainty is so
high that conventional modes of probabilistic analysis
(including decision analysis) may no longer make sense. While
it is not easy to identify this point, investigators should
continually ask themselves whether what they are doing makes
sense and whether a much simpler approach, such as a
bounding or order-of-magnitude analysis, might be superior
(see for example Casman et al., 1999).
Casman, Elizabeth A., M. Granger Morgan and Hadi Dowlatabadi, “Mixed Levels of
Uncertainty in Complex Policy Models,” Risk Analysis, 19(1), 33-42, 1999.
Department of Engineering and Public Policy
119
Carnegie Mellon University
The use of subjective probability
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Lynn Johnson, Thur. Nov 10, 2005
Department of Engineering and Public Policy
120