UNCERTAINTY AMBIGUITY Roger Cooke Resources for the Future Dept. Math, Delft Univ. of Technology Dec 14, 2009 http://dutiosc.twi.tudelft.nl/~risk/ INDECISION “Uncertainty from random sampling ...omits important sources of uncertainty” NRC(2003) All cause mortality, percent increase per 1 μg/m3 increase in PM2.5 (RESS-PM25.pdf) Amer Cancer Soc. (reanal.) Six Cities Study (reanal.) Harvard Kuwait, Equal weights (US) Harvard Kuwait, Performance weights (US) Median/best estimate 0.7 1.4 0.9657 0.6046 Ratio 95%/5% 2.5 4.8 257 63 UNCERTAINTY AMBIGUITY INDECISION History Structured Expert Judgment in Risk Analysis • • • • • • • WASH 1400 (Rasmussen Report, 1975) IEEE Std 500 (1977) Very Different Guidelines: Canvey Island (1978) The story you hear today is NOT the only story NUREG 1150 (1989) T-book (Swedish Reliability Data Base 1994) USNRC-EU (1995-1997) Guidance on Uncertainty and Use of Experts. NUREG/CR-6372, 1997 • Procedures Guide for Structured Expert Judgment, EUR 18820EN, 2000 UNCERTAINTY AMBIGUITY INDECISION Goals of an EJ study • Census • Political consensus • Rational consensus EJCoursenotes_review-EJ-literature.doc UNCERTAINTY AMBIGUITY INDECISION EJ for RATIONAL CONSENSUS: RESS-TUDdatabase.pdf Parties pre-commit to a method which satisfies necessary conditions for scientific method: Traceability/accountability Neutrality (don’t encourage untruthfulness) Fairness (ab initio, all experts equal) Empirical control (performance meas’t) Withdrawal post hoc incurs burden of proof. Goal: comply with principals and combine experts’ judgments to get a Good Probability Assessor “Classical Model for EJ” UNCERTAINTY AMBIGUITY INDECISION What is a GOOD subjective probability assessor? • Calibration, statistical likelihood – Are the expert’s probability statements statistically accurate? P-value of statistical test • Informativeness – Probability mass concentrated in a small region, relative to background measure • Nominal values near truth • ? UNCERTAINTY AMBIGUITY INDECISION Combined Score Long run strictly proper scoring rule • Calibration information cutoff Requires that experts assess uncertainty for variables from their field for which we (will) know the true values: Calibration / performance / seed variables any expert, or combination of experts, can be regarded as a statistical hypothesis UNCERTAINTY AMBIGUITY INDECISION The Ill Advised Bayesian Calibration questions for PM2.5 RESS-PM25.pdf In London 2000, weekly average PM10 was 18.4 μg/m3. What is the ratio: # non-accidental deaths in the week with the highest average PM10 concentration (33.4 μg/m3) Weekly average # non-accidental deaths. 5% :_______ 25%:_______ 50% :_______ 75%:________95%:________ UNCERTAINTY AMBIGUITY INDECISION EJ Bumperstickers Expert Judgment is NOT Knowledge Observations – NOT EJ methods produces agreement among experts EJ is for quantifying ....not removing..... uncertainty. UNCERTAINTY AMBIGUITY INDECISION Experts CAN quantify uncertainty as subjective probability UNCERTAINTY AMBIGUITY INDECISION # experts # variables # elicitations Nuclear applications 98 2,203 20,461 Chemical & gas industry 56 403 4,491 Groundwater / water pollution / dike ring / barriers 49 212 3,714 Aerospace sector / space debris /aviation 51 161 1,149 Occupational sector: ladders / buildings (thermal physics) 13 70 800 Health: bovine / chicken (Campylobacter) / SARS 46 240 2,979 Banking: options / rent / operational risk 24 119 4,328 231 673 29079 Rest group 19 56 762 TOTAL 587 4137 67001 TU DELFT Expert Judgment database 45 applications (anno 2005): Volcanoes / dams Including 6700 calibration variables Study Variables of interest calibration variables Dispersion Far-field dispersion coeff’s Near-field tracer exper’ts Environmental transport Transfer coeff’s Cumulative concentrations Dose-response models Human dose response Animal dose response Investment pricing Quarterly rates Weekly rates PM2.5 Long term mortality rates Short term mortality ratio’s UNCERTAINTY AMBIGUITY INDECISION Studies since 2006 UNCERTAINTY AMBIGUITY INDECISION Experts confidence does NOT predict performance Very informative assessors may be statistically least accurate PM25-Range-graphs.doc UNCERTAINTY AMBIGUITY INDECISION EJ is Not (just) an academic exercise Invasive Species in Great Lakes (EPA, NOAA) • • • • Commercial Fish Landings Sport Fishing Expenditures Raw Water User Costs Wildlife Watching Effort 16 Treat Experts as Statistical Hypotheses P-Value Too Low for Acceptance 2006 Median Percent Reductions & Costs 13% Sport Fishing 35% Commercial Fishing 23% 18% 21% 11% Raw Water 33% 13% 18% 15% Nuclear Power Plants: 118K /facility/year Wildlife Watching 0.8% Other facilities: 30K /facility/year Experts are sometimes well calibrated AMS-OPTION-TRADERS-RANGE-GRAPHS.doc realestate-range graphs.doc Sometimes not GL-invasive-species-range-graphs.doc Campy-range-graphs.doc UNCERTAINTY AMBIGUITY INDECISION We CAN do better than equal weighting UNCERTAINTY AMBIGUITY RESS-TUDdatabase.pdf INDECISION TU D d TU isp D er 1 d TU isp D er 2 d TU epo s D de 1 po O pe s2 rr O isk pe 1 rr i D sk2 ik er i D ng1 ik er Th ing 2 er m Th bl d 1 er m R ea bl d2 le R stat ea e1 le s EU tat N e2 R EU CD is N EU R 1 N CD R is U CIn 2 EN td os R C EU Int 1 d N R os2 EU CS N OIL R 1 G CS as O IL E 2 G nvi as ro En n1 vi ro n2 AO T1 AO T EU 2 W D EU 1 W ES D2 TE ES C1 TE C 2 Out-of-sample Validation Uncertainty RESS_response2comments.pdf Indecision Ambiguity 0.1 0.01 0.001 13 studies with 14 seed vbls, split, initialize on one half, predict other half 3 1000 Ratios of combined scores: PW/Eq 100 10 1 Post hoc validation UNCERTAINTY AMBIGUITY real estate risk INDECISION Real Estate Risk: Performance based DM Real Estate Risk: Equal w eight DM 600 600 5% 500 50% 400 5% 500 50% 400 95% 95% 300 realiz 200 300 realiz 200 1 11 21 31 vbls 1-16 = seed; vbls 17-31 = vbls of interest 1 11 21 31 vbls 1-16 =seed; vbls 17-31 = vbls. of interest Experts like performance assessment Ask them Aspinall_mvo_exerpts.pdf, Aspinall et al Geol Soc _.pdf , Aspinall & Cooke PSAM4 3-9.pdf, SparksAspinall_VolcanicActivity.pdf Separate scientific assessment of uncertainty from decision making UNCERTAINTY AMBIGUITY INDECISION Univariate distributions is so 90’s Now its all about Dependence Elicit (conditional) copulae Elicit Dependence Structure IQ and fish consumption Dioxin and dentition Causal Model for Air Transport Safety 1400 nodes, functional and probabilistic ESDs Critical Accident Human Accidents types Current Research Agenda • Dependence elicitation • Dependence modeling • Computing large networks (eg wo normal copula) • Emerging features, – micro macro correlation – Tail dependence Thank You Willy Aspinall Tim Bedford Jan van Noortwijk Tom Mazzuchi Dorota Kurowicka David Lodge Ramanan Laxminarayan Abby Colson Harry Joe many students god knows who else Tail dependence and aggregation EJ studies since 2006 UNCERTAINTY AMBIGUITY INDECISION FAQ’s(1) • • • • • From an expert: I don't know that – Response: No one knows, if someone knew we would not need to do an expert judgment exercise. We are tying to capture your uncertainty about this variable. If you are very uncertain then you should choose very wide confidence bounds. From an expert: I can't assess that unless you give me more information. – Response: The information given corresponds with the assumptions of the study. We are trying to get your uncertainty conditional on the assumptions of the study. If you prefer to think of uncertainty conditional on other factors, then you must try to unconditionalize and fold the uncertainty over these other factors into your assessment. From an expert: I am not the best expert for that. – Response: We don't know who are the best experts. Sometimes the people with the most detailed knowledge are not the best at quantifying their uncertainty. From an expert: Does that answer look OK? – Response: You are the expert, not me. From the problem owner: So you are going to score these experts like school children? – Response: If this is not a serious matter for you, then forget it. If it is serious, then we must take the quantification of uncertainty seriously. Without scoring we can never validate our experts or the combination of their assessments. Uncertainty Ambiguity 4 Indecision FAQ’s(2) • • • • • From the problem owner: The experts will never stand for it. – Response We've done it many times, the experts actually like it. From the problem owner: Expert number 4 gave crazy assessments, who was that guy? – Response: You are paying for the study, you own the data, and if you really want to know I will tell you. But you don't need to know, and knowing will not make things easier for you. Reflect first whether you really want to know this. From the problem owner: How can I give an expert weight zero? – Response: Zero weight does not mean zero value. It simply means that this expert's knowledge was already contributed by other experts and adding this expert would only add a bit of noise. The value of unweighted experts is seen in the robustness of our answers against loss of experts. Everyone understands this when it is properly explained. From the problem owner: How can I give weight one to a single expert? – Response: By giving all the others weight zero, see previous response. From the problem owner: I prefer to use the equal weight combination. – Response: So long as the calibration of the equal weight combination is acceptable, there is no scientific objection to doing this. Our job as analyst is to indicate the best combination, according to the performance criteria, and to say what other combinations are scientifically acceptable. Uncertainty Ambiguity 4 Indecision Uncertainty Ambiguity 3 Indecision “ In the first few weeks of the Montserrat crisis there was perhaps, at times, some unwarranted scientific dogmatism about what might or might not happen at the volcano, especially in terms of it turning magmatic and explosive. The confounding effects of these diverging, categorical stances were then compounded for a short while by an overall diminution in communication between scientists and the various civil authorities. The result was a dip in the confidence of the authorities in the MVO team and, with it, some loss of public credibility; this was not fully restored until later, when a consensual approach was achieved. “ Aspinall et al The Montserrat Volcano Observatory: its evolution, organization, rôle and activities. Using Uncertainty to Manage Vulcano risk response Aspinall et al Geol Soc _.pdf AMBIGUITY UNCERTAINTY 1 INDECISION British Airways Pilots study UNCERTAINTY AMBIGUITY Range graphs INDECISION British Airways Pilots Pilot nr p-value mean normalized information weight Pilot nr p-value mean normalized information weight 1 0.1208 1.684 0 17 3.21E-05 2.127 0 2 0.000471 1.428 0 18 0.01041 1.98 0 3 0.007345 2.069 0 19 0.000173 1.868 0 4 0.0368 1.649 0 20 0.2846 1.348 0.3993 5 0.003861 1.628 0 21 0.00011 1.429 0 6 1.92E-06 1.846 0 22 0.01964 1.768 0 7 0.0368 1.712 0 23 0.01964 1.376 0 8 0.01964 1.522 0 24 0.000739 1.712 0 9 0.01891 1.239 0 25 0.1373 1.502 0 10 0.00348 1.517 0 26 0.09535 1.224 0 11 5.90E-05 1.681 0 27 0.003861 1.589 0 12 0.000739 1.829 0 28 0.1902 1.225 0.2802 13 5.90E-05 1.395 0 29 0.5707 1.866 0.3001 14 0.005704 1.246 0 30 6.39E-07 1.808 0 15 0.09535 1.305 0 31 0.000471 1.683 0 16 0.1208 1.798 0 Perf. Wght DM 0.7534 0.8221 17 3.21E-05 2.127 0 PerfDM wo 20,28,29 0.5882 1.005 equal wgt DM 0.64 0.2954 UNCERTAINTY What Is? AMBIGUITY What means? INDECISION What’s best? UNCERTAINTY Experts’ job AMBIGUITY Analysts’ job INDECISION Stakeholder/problem owners’ job