Expert Judgment about Uncertainty in
PM2.5-Mortality:
What Have We Learned?
Katherine D. Walker, Henry Roman
Industrial Economics, Inc.
Patrick Kinney
Columbia University
Lisa Conner, Harvey Richmond, Bryan Hubbell, Bob Hetes,
Mary Ross, Zach Pekar
US Environmental Protection Agency
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
Overview
„
„
EPA PM2.5 –
Mortality Expert
Judgment
Elicitation Project
Promises and
Pitfalls Challenges
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
USEPA Pilot Expert Elicitation of
PM2.5 -Mortality Relationship:
Impetus for Study
„
Premature deaths avoided by reduction of PM2.5
constitute 85-95 % of monetized benefits
• $93 billion in reduced mortality (U.S. EPA Clean Air
Interstate Rule)
„
„
National Academy of Sciences (2002).
“Estimating the Public Health Benefits of
Proposed Air Pollution Regulations”
OMB Circular A-4
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
USEPA’s Primary Benefit Analysis
„
„
Uncertainty in the mortality estimate is
characterized by the confidence interval
from the standard error of one
epidemiological study, Pope et al. (2002).
Why might we need anything else?
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
Uncertainties in
PM2.5-Mortality Relationship
„
„
„
„
„
How strong is the likelihood of a causal
relationship?
What is the true shape of the dose-response
relationship? Threshold?
What is the impact of confounders and effect
modifiers?
How do potential errors in measuring
exposure influence results?
What is the impact of relative toxicity of PM
components or sources?
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
Overview of PM Expert Elicitation
Project Elements
Elements
Pilot Study Full Study
Elicitation team (Walker,
Kinney)
√
√
Selection of experts
5
12
Structured Protocol
√
√
Pilot testing of protocol
√
√
√
Pre-elicitation workshop
Elicitation and verification of
individual judgments
√
√
External Peer Review
√
√
Protocol Structure
Preview &
Assumptions
Examine
quantitative
question
Review
Assumptions
Conditioning Questions:
Issues and Evidence
Quantify C-R
Function
•Shape
•Mechanisms
•Conceptual
framework for
short- and longterm mortality
•Causality*
•Threshold
•Re-visit
causality
•Re-visit
threshold
•Other issues
•Epidemiology
evidence
•Confounding
•Effect
Modification
•Exposure
Assessment
*Red items involve quantitative responses
•Specify
percentiles of
C-R function
uncertainty
distribution
Re-visit
Pilot Elicitation Results:
Comparison to Studies Used in EPA Analyses
% Increase in Mortality per 1 ug/m3 PM2.5
2
1.5
1
0.5
0
A
B*
C --8 u g /m 3
C --1 0
u g /m 3
C --15
u g /m 3
C --2 0
u g /m 3
E xp ert
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
D
E
P o p e e t a l., D o c k e ry e t
2 0 02
al., 1 9 93 ;
K re w s ki,
2000
Primary Sensitivity
Estimate Analysis
Promises
„
„
„
„
„
Intellectual “cross-fertilization” in
uncertainty analysis
Well-defined questions
Individual expert opinions
Structured consideration of the
evidence
Comprehensive and explicit
characterization of uncertainties
typically left unquantified
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
Challenges
„
„
„
Practical
Methodological
Political
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
Practical
„
Cost
• This is not a low cost solution!
„
Logistics involving experts
• Overuse/Conflicts
• Limits on numbers of experts imposed
by federal regulations
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
Methodological
„
Expert selection
• No “one size fits all” methodologies
• Who is an “expert” for complex multi-disciplinary questions?
„
Assuring elicitation of “informed” but independent
judgments
•
•
•
Design of protocol and elicitation method
Role and influence of the “elicitors”
Identifying and eliminating motivational biases (real or
perceived)
• Role of pre- or post-elicitation workshops
„
How good are experts’ judgments about uncertainty?
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
Policy Challenges:
We’ve Characterized Uncertainty. Now what?
Non-road Diesel Rule Annual Change in Mortality Incidence in 2030
40,000
Reduction in Premature Mortality in 2030 .
35,000
Note: Distributions labeled Expert A - Expert E are based on individual
expert responses. The distribution labeled Combined Experts is based
on the averaged distributions of reduced incidence of premature mortality
across the set of experts. The distribution labeled Pope et al. (2002)
Statistical Error is based on the mean and standard error of the C-R
function from the study.
30,000
25,000
20,000
18,900
15,000
13,800
10,000
9,900
11,300
9,600
6,200
5,000
3,900
0
Expert A
Expert B
Expert C
Expert D
Expert E
Combined
Experts
Pope et al
(2002)
Statistical Error
Recommendations
„
„
Near term: Develop internal policies
for dealing with quantitative
measures of uncertainty
Longer term: Applied research on
expert judgment methodology on
complex problems
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls
Disclaimer
„
The opinions, findings, and
conclusions expressed are those of
the authors and do not necessarily
represent those of the U.S. EPA
March 13, 2006
RFF Workshop on Expert Judgment:
Promises and Pitfalls