Elicitation Protocol Page 1 of 21 Protocol for Probabilistic Characterization of Uncertainty in Mortality Response to Airborne Fine Particulate Matter Elicitation of European Air Pollution Experts Harvard-Kuwait Public Health Project Version 5-3, Printed: June 16, 2004 Elicitation Protocol Part I: Page 2 of 21 Introduction Thank you for participating in this expert judgment study of the concentration-response relationship between airborne fine particulate matter and population mortality rates. Objectives There are two main objectives of this study: 1. to probabilistically characterize the concentration-response relationship between airborne fine particulate matter (PM2.5) and population mortality rates. 2. to gain qualitative insight into factors influencing the PM2.5 mortality concentration-response relationship. Method The method employed is structured expert judgment. Experts’ uncertainty estimates for variables of interest (related to PM2.5 mortality concentration-response) are elicited via a structured protocol of questions. Included in the elicitation are variables whose true values will become known in the time frame of the study. These variables are used to measure and hopefully validate the expert performance uncertainty quantifications. A good uncertainty assessor is one who gives assessments which are statistically accurate, and gives assessments which are informative. Statistical accuracy means roughly the following. For each of a number of uncertain quantities the expert gives an interval which he is 90% certain will contain the (unknown) true value. Then the assessments are statistically accurate if (in an appropriate statistical sense) 90% of the true values fall within the respective intervals. A similar definition can be applied for other probability intervals. Informativeness means roughly that the probability intervals are concentrated, e.g. that the assessor is 90% certain of catching the true values in small intervals. Both performance indicators are important, but statistical accuracy is arguably more important than informativeness. Expert assessments may be combined in a number of different ways, including equal weighting or by one of several approaches designed to give differential weights to each expert. In a performance-based approach, differential weights are given to experts according to performance indicators. Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 3 of 21 Confidentiality/feedback Following the individual elicitation, we will summarize your responses in free text format. Each expert will receive a copy of his/her quantitative and qualitative assessments for review and will receive information about how the assessments will be used. The names and qualifications of the experts will be published in the final report, as will the individual expert assessments and all information relevant. The names of the experts will not be directly identified with individual assessments in the published reports. However, this information will be preserved with the project files and may be made available to in the case of an authorized peer review. Elicitation Format We ask you to probabilistically characterize the PM2.5 concentration-response in several different settings representing variations in: particle composition, baseline pollution conditions, age-structure and background health status of the exposed population, and exposure duration. We will ask you eight questions about the mortality concentration-response relationship in different settings. These are termed “variables of interest” for this elicitation project. In addition to these eight questions, we will ask you supplemental questions about the intersection of short-term and long-term exposure studies and about the potential for different particle constituents to have different toxicities. Throughout these questions, we will ask you to describe your reasoning for the uncertainty estimates that you provide. Following our elicitation of your uncertainty estimates for these variables, we will ask you, in a similar fashion, to answer several more questions relevant to the variables of interest. These are termed “seed questions” and they are designed to gauge the calibration and informativeness of each expert. These questions differ from variables of interest in that their answers are observable quantities. Therefore, performance on these questions may be assessed and used to determine expert weights, as previously explained. Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 4 of 21 Variables of Interest The questions for the seven variables of interest share a common form, with slight variations pertaining to different locations, exposure durations, and effect intervals for each setting. For example, the first question is presented as follows: What is your estimate of the true, but unknown, percent change in the total annual, non-accidental mortality rate in the adult U.S. population resulting from a permanent 1 μg/m3 reduction in long-term annual average PM2.5 (from a population-weighted baseline concentration of 18 μg/m3) throughout the U.S.? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. The underlined sections of text indicate elements of the general question which differ from one specific question to another. You are asked to express your uncertainty in terms of the 5th, 25th, 50th, 75th, and 95th percentiles of your uncertainty distribution, where: The 5th and 95th percentiles give an interval which you are 90% certain contains the true value, with equal probability of falling above or below. The 25th and 75th percentiles give an interval inside the 5th and 95th percentiles, which you are 50% certain contains the true value, with equal probability of falling above or below. The 50th percentile, or median, is that value for which you think it is equally likely that the true value fall above or below. These notions may be represented graphically in terms of a probability density as shown in the example question and hypothetical answer provided below. Because we do not want to use an example that might bias your responses in this elicitation, the example question is about benzene concentrations, rather than particulate matter concentrationresponse. Example Question: What is your estimate of the expected 6-day integrated benzene exposure concentration in ambient air to be experienced by subjects in a forthcoming exposure study (in ppm)? (Details of the study provided separately) To express the uncertainty associated with this exposure concentration, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. Example Answer: Elicited Values Percentiles 1.5 0% 5 5% Version 5-3, Printed: June 16, 2004 7 25% 8.8 50% 11 75% 14 95% 20 100% Elicitation Protocol Page 5 of 21 This table of elicited values corresponds to the following histogram: 25% 50% 0.14 75% 0.12 5% 0.1 0.08 95% 0.06 0.04 0.02 1.5 55 7 8.8 8.8 11 14 14 25 20 Note that although 0 and 100 percentiles are included in the example above, we will not ask you to estimate these percentiles in this elicitation. We will ask you to provide only the 5th, 25th, 50th, 75th, and 95th percentile estimates for each question. The following grid represents the eight questions for the variables of interest and the order in which they will be asked: Exposure duration Time of interest for effect US (Baseline: 18 µg/m3) Permanent Longterm 1 One day 1 week 3 One day 3 months 6 9 months Longterm Version 5-3, Printed: June 16, 2004 MCMA (Baseline: 35 µg/m3) Europe (Baseline: 20 µg/m3) Kuwait (Specific Exposure Scenario) 2 4 5 11, 12 Elicitation Protocol Page 6 of 21 Questions 7 – 10 are concerned with differences in toxicity among PM constituents and intersection of short and long term exposures. Definitions for each region of interest—including population profiles and baseline rates of relevant variables—are contained in region-specific background sheets included in an appendix to this protocol. 'US' in this context means the 48 states of the United States, excluding Hawaii and Alaska. 'Europe' means the 15 countries that belonged to the European Union before May 1st, 2004. 'MCMA' means Mexico City Metropolitan Area. Rationale Your opinions and interpretation of the evidence underlying these estimates are just as important as your quantitative assessments. As we elicit your quantitative responses, we will be discussing your reasoning and the evidence supporting your responses. The qualitative questions below represent important issues underlying the quantitative estimates that you give. They are meant to guide our discussion during the elicitation. You will not be asked to write responses to each question; rather, we will ask your opinion on these when they are relevant during the elicitation. Following the interview we will prepare a summary of your responses and send them to you for review. You will have 1 week to comment on these summaries before they are finalized for our project report. While we may not follow the questions below exactly, we want you to think about them as a general guide and express your reasoning before you give us your quantitative estimates. For each question, we would like you to incorporate the following questions into your rationale: What evidence suggests large values for this relationship? What is the highest plausible value? Tell us a little about your reasoning, the evidence, and theories that lead you to this value. Can you tell us of scenarios that would yield higher results? What evidence or theory suggests small values? What is the lowest plausible value? Tell us a little about your reasoning, the evidence, and theories that lead you to this value. Can you tell us of scenarios that would yield lower results? Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 7 of 21 In a few minutes we will begin asking the elicitation questions about the variables of interest. Before doing so, however, we would like to ask three general questions about PM epidemiology results and causality: What are the key properties of an ideal epidemiology study for measuring longterm mortality impacts of PM exposure? Similarly, what key properties can you identify for studies of short-term mortality impacts of PM exposure? What factors need to be considered to decide whether epidemiology results should be viewed as causal? Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 8 of 21 Part II: Elicitation of Uncertainty Distributions A. Variables of Interest Question 1 Setting US Exposure (Effect Interval) Long-term Change 1 μg/m3 Pollutant PM2.5 Composition Ambient Baseline 18 ug/m3 What is your estimate of the true, but unknown, percent change in the total annual, non-accidental mortality rate in the adult U.S. population resulting from a permanent 1 μg/m3 reduction in long-term annual average PM2.5 (from a population-weighted baseline concentration of 18 μg/m3) throughout the U.S.? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Question 2 Setting EU Exposure (Effect Interval) Long-term Change 1 μg/m3 Pollutant PM2.5 Composition Ambient Baseline 20 ug/m3 What is your estimate of the true, but unknown, percent change in the total annual, non-accidental mortality rate in the adult European population resulting from a permanent 1 μg/m3 reduction in long-term annual average PM2.5 (from a population-weighted baseline concentration of 20 μg/m3) throughout the EU? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Version 5-3, Printed: June 16, 2004 Elicitation Protocol Question 3 Setting US Page 9 of 21 Exposure (Effect Interval) Short-term (one week) Change 10 μg/m3 Pollutant PM2.5 Composition Ambient Baseline Current What is your estimate of the true, but unknown, percent change in non-accidental mortality in the total U.S. population over the one week following a 10 μg/m3 increase in PM2.5 levels on a single day throughout the U.S.? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Question 4 Setting MCMA Exposure (Effect Interval) Short-term (one week) Change 10 μg/m3 Pollutant PM2.5 Composition Ambient Baseline Current What is your estimate of the true, but unknown, percent change in non-accidental mortality in the total MCMA population over the one week following a 10 μg/m3 increase in PM2.5 levels on a single day throughout the MCMA? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Question 5 Setting EU Exposure (Effect Interval) Short-term (one week) Change 10 μg/m3 Pollutant PM2.5 Composition Ambient Baseline Current What is your estimate of the true, but unknown, percent change in non-accidental mortality in the total European population over the one week following a 10 μg/m3 increase in PM2.5 levels on a single day throughout the EU? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Version 5-3, Printed: June 16, 2004 Elicitation Protocol Question 6 Setting US Page 10 of 21 Exposure (Effect Interval) three months Change 10 μg/m3 Pollutant PM2.5 Composition Ambient Baseline Current What is your estimate of the true, but unknown, percent change in non-accidental mortality in the total U.S. population over the three months following a 10 μg/m3 increase in PM2.5 levels on a single day throughout the U.S.? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Intersection of Short-term and Long-term Effects One question that arises in the interpretation of cohort and time-series mortality studies is how to deal with the potential for double counting. We would like for you to explain your understanding of this issue, considering the following bulleted questions, and then answer a few quantitative questions. How does this potential arise? Which papers influence your views about this question? Now please answer the following questions: Question Setting 7 US Exposure (Effect Interval) Interaction (1 week) Change Pollutant Composition Baseline 1 μg/m3 PM2.5 Ambient 18 μg/m3 What fraction of the 50th percentile answer you gave in Question 1 (US, long-term) was due to effects which would express themselves within 1 week of the change in exposure? To express the uncertainty associated with this question, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Version 5-3, Printed: June 16, 2004 Elicitation Protocol 8 US Page 11 of 21 Interaction (3 months) 1 μg/m3 PM2.5 Ambient 18 μg/m3 What fraction of the 50th percentile answer you gave in Question 1 (US, long-term) was due to effects which would express themselves within three months of the change in exposure? To express the uncertainty associated with this question, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 12 of 21 Differential Toxicity Several authors have discussed the notion that different constituents of PM2.5 may have different toxic effects and magnitudes. This possibility is of significant importance to regulatory decision-makers because the control of different sources may remove different constituents of PM2.5 and lead to an overall change in the composition of ambient PM2.5. In this section, we would like to discuss your interpretation of the evidence for differential toxicity. We are particularly interested in what you believe can and cannot be said about quantitative estimates for this effect. We would like you to consider the bulleted questions and answer the question that follows them. Note that if you and other experts do not believe this is possible, all PM constituents will implicitly be treated as equivalently toxic. What kind of evidence would be ideal in this issue? What evidence do we actually have? How does this compare with the ideal evidence? What do you believe to be the important constituents of PM2.5 as regards mortality? For long-term exposures and ambient conditions in the US as defined in Question 1, how would your estimates change if the reduction in PM2.5 were solely composed of (separately) the following individual constituents? Size of Effect Lower Unchanged Primary (EC/OC) Ammonium Sulfate Ammonium Nitrate Crustal Fine PM Secondary Organics Diesel Particulate Burning Crude Oil Transition Metals Other (Specify) Version 5-3, Printed: June 16, 2004 Higher Less Uncertainty Unchanged More Elicitation Protocol Page 13 of 21 If you indicated that any of these composition categories were more toxic than the ambient mixture, then for the most toxic category, how would you revise your estimate of uncertainty for the long-term exposure conditions defined below? Question 9 Setting US Exposure (Effect Interval) Long-term Change 1 μg/m3 Pollutant PM2.5 Composition [Indicate] Baseline 18 μg/m3 What is your estimate of the true, but unknown, percent change in the total annual, non-accidental mortality rate in the adult U.S. population resulting from a permanent 1 μg/m3 reduction of [most toxic] in long-term annual average PM2.5 (from a population-weighted baseline concentration of 18 μg/m3) throughout the U.S.? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ If you indicated that any of these composition categories were less toxic than the ambient mixture, then for the least toxic category, how would you revise your estimate of uncertainty for the long-term exposure conditions defined below? Question 10 Setting US Exposure (Effect Interval) Long-term Change 1 μg/m3 Pollutant PM2.5 Composition [Indicate] Baseline 18 μg/m3 What is your estimate of the true, but unknown, percent change in the total annual, non-accidental mortality rate in the adult U.S. population resulting from a permanent 1 μg/m3 reduction of [least toxic] in long-term annual average PM2.5 (from a population-weighted baseline concentration of 18 μg/m3) throughout the U.S.? To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 14 of 21 For the following questions, please refer to the following plots and map. Question Setting Exposure (Effect Interval) Change Pollutant 11 Kuwait 9 months (plot) PM2.5 Composition Burning Crude Year 1991 What is your estimate of the true, but unknown, percent change in the non-accidental mortality rate in the Kuwaiti national population alive at the beginning of the 1991 oil fires resulting from the nine-month excursion in PM2.5 concentration patterns as given in the following plots? Note that the source of this excursion is from the combustion of crude oil. To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Question Setting Exposure (Effect Interval) Change Pollutant 12 Kuwait 9 months (plot) PM2.5 Composition Burning Crude Year 1991 What is your estimate of the true, but unknown, number of premature deaths in the Kuwaiti national population alive at the beginning of the 1991 oil fires resulting from the nine-month excursion in PM2.5 concentration patterns as given in the following plots? Note that the source of this excursion is from the combustion of crude oil. To express the uncertainty associated with the concentration-response relationship, please provide the 5th, 25th, 50th, 75th, and 95th percentiles of your estimate. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Additional task about question 12. We ask you to do the following: Make a calculation of a plausible “upper estimate” of the answer to question 12 and write your approach, reasoning and calculations down. Then, do a similar thing to calculate a plausible “lower estimate”, and a “central estimate.” We ask you to formalize this in a 2-3 page write up. We hope that you tell how you would weave together the answers from all the previous questions to approach this question. The "upper, lower, and central" correspond to 95th, 5th, and 50th percentiles, respectively, but are not necessarily equal to them. You may change your answers to question 12 after this additional task, if you think it is warranted. Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 15 of 21 Concentration Time-Series at 3 Populated Areas in Kuwait Time Series of PM10 at Jahra 1000 Ambient (Measured) 900 PM10 concentration (ug/m3) Ambient Average = 284 800 Contribution from Oil Fires (Modeled) 700 Oil Fire Average = 24.2 600 500 400 300 284 200 100 0 2/1/1991 24.2 3/23/1991 5/12/1991 7/1/1991 8/20/1991 10/9/1991 11/28/1991 Date Time Series of PM10 at Mansouria 1000 Ambient (Measured) 900 Ambient Average = 300 800 PM10 concentration (ug/m3) Contribution from Oil Fires (Modeled) 700 Oil Fire Average = 17.3 600 500 400 300 300 200 100 0 2/1/1991 17.3 3/23/1991 5/12/1991 7/1/1991 8/20/1991 Date Version 5-3, Printed: June 16, 2004 10/9/1991 11/28/1991 Elicitation Protocol Page 16 of 21 Time Series of PM10 at Riqa 1000 Ambient (Measured) 900 Ambient Average = 251 PM10 concentration (ug/m3) 800 700 Contribution from Oil Fires (Modeled) Oil Fire Average = 28.2 600 500 400 300 251 200 100 0 2/1/1991 28.2 3/23/1991 5/12/1991 7/1/1991 8/20/1991 Date Version 5-3, Printed: June 16, 2004 10/9/1991 11/28/1991 Elicitation Protocol Page 17 of 21 Map of Locations Corresponding to 3 Sites Jahra Mansouria Riqa Version 5-3, Printed: June 16, 2004 Elicitation Protocol B. Page 18 of 21 Seed/Calibration Variables The purpose of the following questions is to address empirical control on the quantitative uncertainty assessments for the variables of interest. They may be used to validate the expert assessments and to construct a set of weights for combination of these assessments. Each question below has an answer which either has been or soon will be observed and reported. You are asked to provide your best estimates for each answer in the form of percentiles representing a distribution of uncertainty. The following seed questions are based on PM10 data from London and Athens for the period 1997 – 2001. LONDON STATIONS HILLINGTON ELTHAM BRENT BLOOMSBURY BEXLEY MARUSSI THRAKOMAKEDONES ZOGRAFOU ATHENS STATIONS AGIA PARASKEVI LYKOVRISI Questions on PM10 Reference station exceedances 1. On how many days in 2001 did the daily average PM10 concentration exceed 50μg/m3 at least one of the above London stations (max 365)? 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ 2. On how many days in 2001did the daily average PM10 concentration fall below 30 μg/m3 at all of the above London stations (max 365)? 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 19 of 21 3. On how many days in 1997 did the daily average PM10 concentration exceed concentration exceed 50μg/m3 at least one of the above London stations (max 365)? 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ 4. On how many days in 1997did the daily average PM10 concentration fall below 30 μg/m3 at all of the above London stations (max 365)? 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ 5. On how many days in 2001 did the daily average PM10 concentration exceed 50μg/m3 at least one of the above Athens stations (max 365)? 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ 6. . On how many days in 2001did the daily average PM10 concentration fall below 30 μg/m3 at all of the above Athens stations (max 365)? 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Questions on Mortality LONDON The annual average for the year 2000 of daily PM10 concentration, averaged over the 5 stations in London was 18.4 g/m3. The highest weekly concentration averaged over the London stations was 33.4 g/m3 Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 20 of 21 7. What is the ratio: Number of non-accidental deaths in the week (7 days starting from January 1st) of 2000 with the highest average PM10 concentration / Weekly average number non-accidental deaths in 2000. 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ 8. What is the ratio Number of cardiovascular deaths (ICD10 Cause I) in the week (7 days starting from January 1st) of 2000 with the highest average PM10 concentration / Weekly average number cardiovascular deaths in 2000 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ ATHENS The annual average for the year 2001 of daily PM10 concentration, averaged over the 5 Athens stations was 45.5 g/m3; the highest weekly concentration averaged over the Athens stations was 63.2 g/m3; the lowest weekly concentration averaged over the Athens stations was 16.5 g/m3. 9. What is the ratio Number of non-accidental deaths in the week (7 days starting from January 1st) of 2001 with the highest average PM10 concentration / Weekly average number non-accidental deaths in 2001 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ 10 What is the ratio: Number of cardiovascular deaths in the week (7 days starting from January 1st) of 2001 with the highest average PM10 concentration / Weekly average number cardiovascular deaths in 2001 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Version 5-3, Printed: June 16, 2004 Elicitation Protocol Page 21 of 21 11. What is the ratio Number of non-accidental deaths in the week (7 days starting from January 1st) of 2001 with the lowest average PM10 concentration / Weekly average number non-accidental deaths in 2001 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ 12 What is the ratio Number of cardiovascular deaths in the week (7 days starting from January 1st) of 2001 with the lowest average PM10 concentration / Weekly average number cardiovascular deaths in 2001 5% :____________ 25%:____________ 50% :____________ 75%:____________ 95%:____________ Version 5-3, Printed: June 16, 2004