Are Decisionmakers at Home on the
Range? Communicating
Uncertainties in Regulatory Impact
Analyses
Alan Krupnick, Richard Morgenstern,
Peter Nelson
Expert Judgment Workshop
March 13, 2006
“A big part of my frustration was that scientists would
give me a range. And I would ask, ‘please just tell me at
which point you are safe, and we can do that.’ But they
would give a range, say from 5 to 25 parts per billion
(ppb). And that was often frustrating.”
Christine Todd Whitman,
quoted in Environmental Science and Technology
Online,
April 20, 2005
Why Introduce Uncertainty in
Regulatory Analyses?
• OMB says so: Circular A-4 calls for incorporating
uncertainty in RIAs ($1 billion or more in cost)
(Monte Carlo simulation)
• Permits more explicit judgment about risk
aversion
• Eliminates false sense of security with point
estimates
• Introduces more transparency to public
• Help guide research priorities (VOI)
State of Play
•EPA already active in this area: SAB, 812
studies, various RIAs, although significant
shortcomings remain
•But less activity in examining
communicating uncertainty to DMs
•NRC, SAB and others calling for more and
better treatment of uncertainty
Our Motivation
• Even though trends in analytical techniques and
software making it easier to incorporate
uncertainty in regulatory analyses…
• Better, more complete (and more complex)
information can confound and paralyze rather than
improve decisions.
• So improvements in capturing uncertainty
analytically must be matched with improvements
in communication
What we did
• Developed detailed analytical case study
pushing the “frontier” of RIAs
ƒ Addressed statistical uncertainties in
“standard” and new areas, including both cost
and benefit sides
ƒ Quantified model uncertainties
• Communicated results to seven high level
ex-EPA DMs
ANALYTIC SEQUENCE FOR RIAs
Scenario
Development:
Emissions Profile
Development
Cost Analysis
Benefits Analysis
Air Quality Modeling -Criteria Pollutants
Direct Cost
Estimation
Physical
Effects
Valuation
Comparison of Benefits
and Costs
ANALYTIC SEQUENCE FOR RIAs
Scenario
Development:
Emissions Profile
Development
Cost Analysis
Benefits Analysis
Air Quality Modeling -Criteria Pollutants
Direct Cost
Estimation
Physical
Effects*
Valuation*
* Standard statistical and model uncertainty
Comparison of Benefits
and Costs
ANALYTIC SEQUENCE FOR CASE STUDY
Scenario
Development:
2 tighter NOx
caps
Uncertain Sourcereceptor coefficients/
Models
NOx Emissions Profile
Development
Cost Analysis
Benefits Analysis
Air Quality Modeling -NOX to O3/PM2.5
Physical
Effects*
Uncertain
Population/
Models
Direct Cost
Estimation
Valuation*
* Standard statistical and model uncertainty
Comparison of Benefits
and Costs
Uncertain natural gas
Prices
ANALYTIC SEQUENCE FOR CASE STUDY
Scenario
Development:
2 tighter NOx
caps
Uncertain Sourcereceptor coefficients/
Models
NOx Emissions Profile
Development
Cost Analysis
Benefits Analysis
Air Quality Modeling -NOX to O3/PM2.5
Physical
Effects*
Valuation*
* Standard statistical and model uncertainty
Uncertain
Population/
Models
Direct Cost
Estimation
Expert
Elicitation
Comparison of Benefits
and Costs
Uncertain natural gas
Prices
Ozone Benefit Valuation ($US
Millions)
2025 – Benefits of Large NOx Reduction Policy From CAIR Baseline
Using Alternative Ozone Mortality Concentration Response Functions
1,600
1,400
1,200
1,000
µ
800
µ
600
µ
400
µ
µ
µ
200
l(
Sa
19
m
97
et
)A
et
al
(1
99
7)
B
Be
ll
(2
00
4)
95
)
Sa
m
et
e
ta
l(
19
6)
ar
et
a
(1
99
av
k
lg
oo
M
&
Ito
B
A
6)
s to
n
(1
99
Th
ur
s to
n
Th
ur
&
Ito
Ba
se
Ca
se
(W
ei
g
ht
ed
)
0
Ozone Short-Term Mortality CRF Study
µ
Literature Review on
Communicating Uncertainty to DMs
• Large body of research contradicts principle of
decision invariance (different representations of the
same problem should yield the same preferences)
(Tversky and Kahneman)
•Some research shows that presenting uncertainty
can cause people to discount the entire analysis
(“ambiguity avoidance”).
•Experts have been shown to be as susceptible to
these problems as the general public.
Literature on Visual
Presentations
• Bar graphs lead to less bias than pie charts
and symbols (USA Today)
• Graphics tend to make respondents more
risk averse than numeric presentations
(pdf’s: greater focus on the tails)
FIGURE 3
Probability Density Function
Comparison of Net Benefits of Tight NOx Cap & Intermediate NOx Cap
-1000
-500
0
Net Ben efits ($ US millions)
Net BenefitsTight NOx Cap
50 0
10 00
Net Benefits Interme di ate NOx Cap
Interviews with Former EPA
DMs
• 7 former DAs, AAs interviewed, using simplified version of
case study
•Each choice (tight cap, intermediate cap, do nothing) picked by
at least one DM
•Two favored tight option, one favored doing nothing; three
favored the intermediate option; one ruled out the do nothing
option and said to decide he would need more info.
•All said an uncertainty presentation was useful and helped them
get an idea of the confidence they should have in their decision
and better prepare them to defend against critics.
Findings
• Different DMs have different learning styles,
but…all wanted more info for a “real” decision Î
Present technical information in context
• Tables OK
• PDF a preferred graphic
- but may push to intermediate option
- permits thoughtful discussion
• Some felt in real life they got analyses with
pre-ordained outcome
• They often “don’t know what they don’t know”
TABLE 1
Mortality
Comparison of Tight NOx Cap and Intermediate NOx Cap Policies
Averted Physical Impacts in 2025
Tight NOx Cap
Intermediate NOx Cap
95% CI
95% CI
95% CI
95% CI
lower
upper
lower
upper
bound
bound
bound
bound
Mean
Mean
466
122
810
254
65
443
Cardiovascular
Hospital
Admissions
Admissions/Year
409
47
771
230
27
434
Non-Fatal Heart
Attacks
Cases/Year
995
338
1652
543
187
900
Respiratory
Hospital
Admissions
Admissions/Year
2611
1550
3672
841
512
1169
Cardiovascular
Hospital
Admissions
Admissions/Year
338
204
471
197
123
272
Asthma
Emergency Room
Visits/Year
598
358
838
265
150
380
Cost (millions $)
1340
710
TABLE 2
Comparison of Tight NOx Cap and Intermediate NOx Cap
Policies
Net Benefits in 2025
Tight NOx Cap
Mean
Total Benefits
($ US millions)
1351
Costs
($
US millions)
1340
Net Benefits ($
US millions)
11
Intermediate NOx Cap
95% CI
95% CI
lower bound upper bound
509
2194
Mean
720
95% CI
95% CI
lower bound upper bound
255
1184
-455
474
710
-831
854
10
FIGURE 1
Probability that Policies Produce Net Benefits in 2025
Comparison of Tight and Intermediate NOx Caps
Intermediate Option
Negati ve Net Benefits
Graphs by policy
Tight Option
Positive Net Benefits
FIGURE 2
Comparison of Net Benefits of Tight and Intermediate NOX Caps in 2025
-1,000
Millions of Dollars
-500
0
500
1,000
Box and Whisker Graph
Tight NOx Cap
Intermediate NOx Cap
FIGURE 3
Probability Density Function
Comparison of Net Benefits of Tight NOx Cap & Intermediate NOx Cap
-1000
-500
0
Net Ben efits ($ US millions)
Net BenefitsTight NOx Cap
50 0
10 00
Net Benefits Interme di ate NOx Cap
FIGURE 4
Cumulative Density Function
0
Cumulative Density
.1 .2 .3 .4 .5 .6 .7 .8 .9 1
Comparison of Net Benefits of Tight NOx Cap & Intermediate NOx Cap
-1000
-500
0
Net Benefits ($ US millions)
Net BenefitsTight NOx Cap
500
1000
Net Benefits Intermediate NOx Cap
FIGURE 5
Impact on Net Benefits Estimates in 2025
-1,000
Millions of Dollars
-500
0
500
1,000
Sources of Uncertainty
Mortality C-R
Bronchitis C-R
Population
Source Recep tor
VSL
Gas Prices
Heart Attack C-R
Findings
• Different DMs have different learning styles,
but…all wanted more info for a “real” decision Î
Present technical information in context
• Tables OK
• PDF a preferred graphic
- but may push to intermediate option
- permits thoughtful discussion
• Some felt in real life they got analyses with
pre-ordained outcome
• They often “don’t know what they don’t know”
To Do Better
• Develop internal processes for insuring unbiased
presentation of results
• Have SAB/OMB develop detailed approaches
ƒ Form for pros and cons
• Need research on data reduction step
• Need more research on communicating to DMs
(N>7, more systematic empirical treatment)
• Training DMs