Roger Cooke Resources for the Future Dept. Math, Delft Univ. of Technology

advertisement
UNCERTAINTY
AMBIGUITY
Roger Cooke
Resources for the Future
Dept. Math, Delft Univ. of
Technology
Dec 14, 2009
http://dutiosc.twi.tudelft.nl/~risk/
INDECISION
“Uncertainty from random sampling
...omits important sources of
uncertainty” NRC(2003)
All cause mortality, percent increase per 1
μg/m3 increase in PM2.5 (RESS-PM25.pdf)
Amer Cancer Soc.
(reanal.)
Six Cities Study
(reanal.)
Harvard
Kuwait,
Equal weights
(US)
Harvard
Kuwait,
Performance
weights (US)
Median/best
estimate
0.7
1.4
0.9657
0.6046
Ratio 95%/5%
2.5
4.8
257
63
UNCERTAINTY
AMBIGUITY
INDECISION
History Structured Expert
Judgment in Risk Analysis
•
•
•
•
•
•
•
WASH 1400 (Rasmussen Report, 1975)
IEEE Std 500 (1977)
Very Different Guidelines:
Canvey Island (1978)
The story you hear today is NOT the
only story
NUREG 1150 (1989)
T-book (Swedish Reliability Data Base 1994)
USNRC-EU (1995-1997)
Guidance on Uncertainty and Use of Experts.
NUREG/CR-6372, 1997
• Procedures Guide for Structured Expert
Judgment, EUR 18820EN, 2000
UNCERTAINTY
AMBIGUITY
INDECISION
Goals of an EJ study
• Census
• Political consensus
• Rational consensus
EJCoursenotes_review-EJ-literature.doc
UNCERTAINTY
AMBIGUITY
INDECISION
EJ for RATIONAL CONSENSUS:
RESS-TUDdatabase.pdf
Parties pre-commit to a method which satisfies necessary conditions
for scientific method:
Traceability/accountability
Neutrality (don’t encourage untruthfulness)
Fairness (ab initio, all experts equal)
Empirical control (performance meas’t)
Withdrawal post hoc incurs burden of proof.
Goal: comply with principals and combine experts’
judgments to get a Good Probability Assessor
“Classical Model for EJ”
UNCERTAINTY
AMBIGUITY
INDECISION
What is a GOOD subjective
probability assessor?
• Calibration, statistical likelihood
– Are the expert’s probability statements
statistically accurate? P-value of statistical
test
• Informativeness
– Probability mass concentrated in a small
region, relative to background measure
• Nominal values near truth
• ?
UNCERTAINTY
AMBIGUITY
INDECISION
Combined Score
Long run strictly proper scoring rule
• Calibration  information  cutoff
Requires that experts assess uncertainty for variables
from their field for which we (will) know the true values:
Calibration / performance / seed
variables
any expert, or combination of experts, can be regarded
as a statistical hypothesis
UNCERTAINTY
AMBIGUITY
INDECISION
The Ill Advised Bayesian
Calibration questions for PM2.5
RESS-PM25.pdf
In London 2000, weekly average PM10 was 18.4 μg/m3.
What is the ratio:
# non-accidental deaths in the week with the highest average
PM10 concentration (33.4 μg/m3)

Weekly average # non-accidental deaths.
5% :_______ 25%:_______ 50% :_______ 75%:________95%:________
UNCERTAINTY
AMBIGUITY
INDECISION
EJ
Bumperstickers
Expert Judgment is NOT
Knowledge
Observations – NOT EJ methods produces agreement among experts
EJ is for quantifying ....not removing.....
uncertainty.
UNCERTAINTY
AMBIGUITY
INDECISION
Experts CAN quantify
uncertainty as subjective
probability
UNCERTAINTY
AMBIGUITY
INDECISION
#
experts
#
variables
#
elicitations
Nuclear applications
98
2,203
20,461
Chemical & gas industry
56
403
4,491
Groundwater / water pollution / dike ring / barriers
49
212
3,714
Aerospace sector / space debris /aviation
51
161
1,149
Occupational sector: ladders / buildings (thermal physics)
13
70
800
Health: bovine / chicken (Campylobacter) / SARS
46
240
2,979
Banking: options / rent / operational risk
24
119
4,328
231
673
29079
Rest group
19
56
762
TOTAL
587
4137
67001
TU DELFT Expert Judgment database
45 applications (anno 2005):
Volcanoes / dams
Including 6700 calibration variables
Study
Variables of interest
calibration variables
Dispersion
Far-field dispersion
coeff’s
Near-field tracer exper’ts
Environmental
transport
Transfer coeff’s
Cumulative
concentrations
Dose-response
models
Human dose response
Animal dose response
Investment pricing
Quarterly rates
Weekly rates
PM2.5
Long term mortality
rates
Short term mortality
ratio’s
UNCERTAINTY
AMBIGUITY
INDECISION
Studies since 2006
UNCERTAINTY
AMBIGUITY
INDECISION
Experts confidence
does NOT predict
performance
Very informative assessors may be
statistically least accurate
PM25-Range-graphs.doc
UNCERTAINTY
AMBIGUITY
INDECISION
EJ is Not (just) an
academic exercise
Invasive
Species
in Great Lakes (EPA, NOAA)
•
•
•
•
Commercial Fish Landings
Sport Fishing Expenditures
Raw Water User Costs
Wildlife Watching Effort
16
Treat Experts as Statistical Hypotheses
P-Value Too Low for Acceptance
2006 Median Percent Reductions &
Costs
13%
Sport Fishing
35%
Commercial Fishing
23%
18%
21%
11%
Raw Water
33%
13%
18%
15%
Nuclear
Power Plants:
118K
/facility/year
Wildlife
Watching
0.8%
Other
facilities:
30K
/facility/year
Experts are sometimes well
calibrated
AMS-OPTION-TRADERS-RANGE-GRAPHS.doc
realestate-range graphs.doc
Sometimes not
GL-invasive-species-range-graphs.doc
Campy-range-graphs.doc
UNCERTAINTY
AMBIGUITY
INDECISION
We CAN do better than equal
weighting
UNCERTAINTY
AMBIGUITY
RESS-TUDdatabase.pdf
INDECISION
TU
D
d
TU isp
D er 1
d
TU isp
D er 2
d
TU epo
s
D
de 1
po
O
pe s2
rr
O isk
pe
1
rr
i
D sk2
ik
er
i
D ng1
ik
er
Th ing
2
er
m
Th bl d
1
er
m
R
ea bl d2
le
R stat
ea
e1
le
s
EU tat
N e2
R
EU CD
is
N
EU R 1
N CD
R
is
U CIn 2
EN
td
os
R
C
EU Int 1
d
N
R os2
EU CS
N OIL
R
1
G CS
as
O
IL
E
2
G nvi
as
ro
En n1
vi
ro
n2
AO
T1
AO
T
EU 2
W
D
EU 1
W
ES D2
TE
ES C1
TE
C
2
Out-of-sample Validation
Uncertainty
RESS_response2comments.pdf
Indecision
Ambiguity
0.1
0.01
0.001
13 studies with  14 seed vbls, split, initialize on one half, predict other half
3
1000
Ratios of combined scores: PW/Eq
100
10
1
Post hoc validation
UNCERTAINTY
AMBIGUITY
real estate risk
INDECISION
Real Estate Risk: Performance based DM
Real Estate Risk: Equal w eight DM
600
600
5%
500
50%
400
5%
500
50%
400
95%
95%
300
realiz
200
300
realiz
200
1
11
21
31
vbls 1-16 = seed; vbls 17-31 = vbls of
interest
1
11
21
31
vbls 1-16 =seed; vbls 17-31 = vbls. of
interest
Experts like performance
assessment
Ask them Aspinall_mvo_exerpts.pdf, Aspinall et al Geol Soc _.pdf ,
Aspinall & Cooke PSAM4 3-9.pdf, SparksAspinall_VolcanicActivity.pdf
Separate
scientific assessment of
uncertainty
from
decision making
UNCERTAINTY
AMBIGUITY
INDECISION
Univariate distributions is so 90’s
Now its all about Dependence
Elicit (conditional) copulae
Elicit Dependence Structure
IQ and fish consumption
Dioxin and dentition
Causal Model for Air Transport Safety
1400 nodes, functional and probabilistic
ESDs
Critical Accident
Human
Accidents types
Current Research Agenda
• Dependence elicitation
• Dependence modeling
• Computing large networks (eg wo normal
copula)
• Emerging features,
– micro  macro correlation
– Tail dependence
Thank You
Willy Aspinall
Tim Bedford
Jan van Noortwijk
Tom Mazzuchi
Dorota Kurowicka
David Lodge
Ramanan Laxminarayan
Abby Colson
Harry Joe
many students
god knows who else
Tail dependence and aggregation
EJ studies since 2006
UNCERTAINTY
AMBIGUITY
INDECISION
FAQ’s(1)
•
•
•
•
•
From an expert: I don't know that
–
Response: No one knows, if someone knew we would not need to do an
expert judgment exercise. We are tying to capture your uncertainty about this
variable. If you are very uncertain then you should choose very wide
confidence bounds.
From an expert: I can't assess that unless you give me more information.
–
Response: The information given corresponds with the assumptions of the
study. We are trying to get your uncertainty conditional on the assumptions of
the study. If you prefer to think of uncertainty conditional on other factors, then
you must try to unconditionalize and fold the uncertainty over these other
factors into your assessment.
From an expert: I am not the best expert for that.
–
Response: We don't know who are the best experts. Sometimes the people
with the most detailed knowledge are not the best at quantifying their
uncertainty.
From an expert: Does that answer look OK?
–
Response: You are the expert, not me.
From the problem owner: So you are going to score these experts like school
children?
–
Response: If this is not a serious matter for you, then forget it. If it is serious,
then we must take the quantification of uncertainty seriously. Without scoring
we can never validate our experts or the combination of their assessments.
Uncertainty
Ambiguity
4
Indecision
FAQ’s(2)
•
•
•
•
•
From the problem owner: The experts will never stand for it.
–
Response We've done it many times, the experts actually like it.
From the problem owner: Expert number 4 gave crazy assessments, who was
that guy?
–
Response: You are paying for the study, you own the data, and if you really
want to know I will tell you. But you don't need to know, and knowing will not
make things easier for you. Reflect first whether you really want to know this.
From the problem owner: How can I give an expert weight zero?
–
Response: Zero weight does not mean zero value. It simply means that this
expert's knowledge was already contributed by other experts and adding this
expert would only add a bit of noise. The value of unweighted experts is seen
in the robustness of our answers against loss of experts. Everyone
understands this when it is properly explained.
From the problem owner: How can I give weight one to a single expert?
–
Response: By giving all the others weight zero, see previous response.
From the problem owner: I prefer to use the equal weight combination.
–
Response: So long as the calibration of the equal weight combination is
acceptable, there is no scientific objection to doing this. Our job as analyst is
to indicate the best combination, according to the performance criteria, and to
say what other combinations are scientifically acceptable.
Uncertainty
Ambiguity
4
Indecision
Uncertainty
Ambiguity
3
Indecision
“ In the first few weeks of the Montserrat crisis there was
perhaps, at times, some unwarranted scientific dogmatism
about what might or might not happen at the volcano,
especially in terms of it turning magmatic and explosive. The
confounding effects of these diverging, categorical stances
were then compounded for a short while by an overall
diminution in communication between scientists and the
various civil authorities. The result was a dip in the
confidence of the authorities in the MVO team and, with it,
some loss of public credibility; this was not fully restored
until later, when a consensual approach was achieved. “
Aspinall et al The Montserrat Volcano Observatory: its evolution,
organization, rôle and activities.
Using Uncertainty to Manage
Vulcano risk response
Aspinall et al Geol Soc _.pdf
AMBIGUITY
UNCERTAINTY
1
INDECISION
British Airways Pilots study
UNCERTAINTY
AMBIGUITY
Range graphs
INDECISION
British Airways Pilots
Pilot nr
p-value
mean
normalized
information
weight
Pilot nr
p-value
mean
normalized
information
weight
1
0.1208
1.684
0
17
3.21E-05
2.127
0
2
0.000471
1.428
0
18
0.01041
1.98
0
3
0.007345
2.069
0
19
0.000173
1.868
0
4
0.0368
1.649
0
20
0.2846
1.348
0.3993
5
0.003861
1.628
0
21
0.00011
1.429
0
6
1.92E-06
1.846
0
22
0.01964
1.768
0
7
0.0368
1.712
0
23
0.01964
1.376
0
8
0.01964
1.522
0
24
0.000739
1.712
0
9
0.01891
1.239
0
25
0.1373
1.502
0
10
0.00348
1.517
0
26
0.09535
1.224
0
11
5.90E-05
1.681
0
27
0.003861
1.589
0
12
0.000739
1.829
0
28
0.1902
1.225
0.2802
13
5.90E-05
1.395
0
29
0.5707
1.866
0.3001
14
0.005704
1.246
0
30
6.39E-07
1.808
0
15
0.09535
1.305
0
31
0.000471
1.683
0
16
0.1208
1.798
0
Perf. Wght DM
0.7534
0.8221
17
3.21E-05
2.127
0
PerfDM wo 20,28,29
0.5882
1.005
equal wgt DM
0.64
0.2954
UNCERTAINTY
What Is?
AMBIGUITY
What means?
INDECISION
What’s best?
UNCERTAINTY
Experts’ job
AMBIGUITY
Analysts’ job
INDECISION
Stakeholder/problem owners’
job
Download