Science and Decisions:
Advancing Risk
Assessment
Risk Assessment Specialty Section (RASS) Monthly Telecon
Joseph Rodricks, Environ
Jonathan Levy, Harvard School of Public Health
May 13, 2009
Study motivation
• Risk assessment is at a crossroads, and its credibility is
being challenged.
• Science is increasingly complex.
• Risk assessment is being extended to address broader
environmental questions, such as life-cycle analysis
and issues of costs, benefits, and risk-risk tradeoffs.
• Stakeholders are often disengaged from the riskassessment process at a time when risk assessment is
increasingly intertwined with societal concerns.
• Disconnects between the available scientific data and
the information needs of decision-makers hinder the
use of risk assessment as a decision-making tool.
Committee’s charge (I)
• “An NRC committee will develop scientific and
technical recommendations for improving the risk
analysis approaches used by the U.S. Environmental
Protection Agency (EPA)…The committee will
consider analyses applied to contaminants in all
environmental media (water, air, food, soil) and all
routes of exposure (ingestion, inhalation, and dermal
absorption). The committee will focus primarily on
human health risk analysis and will comment on the
broad implications of its findings and
recommendations to ecological risk analysis. In
making recommendations, the committee will indicate
practical improvements that can be made in the near
term (2-5 years) and improvements that would be
made over a longer term (10-20 years).”
Committee’s charge (II)
• Increased role for probabilistic analysis in risk
analysis, including the potential expanded role for
expert elicitation
• Scientific bases for and alternatives to default
assumption choices made in areas of uncertainty
• Quantitative characterization of uncertainty resulting
from all steps in the risk analysis
• Approaches for assessing cumulative risk resulting
from multiple exposures to contaminant mixtures,
involving multiple sources, pathways, routes
• Variability in receptor populations, especially
sensitive subpopulations and critical life stages
Committee’s charge (III)
• Biologically relevant modes of action for estimating
dose-response relationships, and quantitative
implications of different modes
• Improvements in environmental transport and fate
models, exposure models, physiologically based
pharmacokinetic (PBPK) models, and dose-response
models
• How the concepts and practices of ecological risk
analysis can help inform and improve the concepts
and practices of human health risk analysis, and vice
versa
• Scientific basis for derivation of uncertainty factors
• Use of value-of-information analyses and other
techniques to identify priorities and approaches for
research to obtain relevant data to increase the utility
of risk analyses
Committee membership
Thomas Burke (Chair), Johns Hopkins Bloomberg School of Public Health
A. John Bailer, Miami University
John M. Balbus, Environmental Defense
Joshua T. Cohen, Tufts Medical Center
Adam M. Finkel, University of Medicine and Dentistry of New Jersey
Gary Ginsberg, Connecticut Department of Public Health
Bruce K. Hope, Oregon Department of Environmental Health
Jonathan I. Levy, Harvard School of Public Health
Thomas E. McKone, University of California
Gregory M. Paoli, Risk Sciences International
Charles Poole, University of North Carolina School of Public Health
Joseph V. Rodricks, ENVIRON International Corporation
Bailus Walker Jr., Howard University Medical Center
Terry F. Yosie, World Environment Center
Lauren Zeise, California Environmental Protection Agency
Evaluation strategy
• Committee concluded early on that risk
assessment can be “improved” in two
different ways
• Improving technical analysis (the
development and use of scientific knowledge
and information to improve characterizations
of risk)
• Improving utility (making risk assessment
more relevant and useful to risk management
decisions)
Structure of report conclusions
•
•
•
•
•
•
•
•
Design of risk assessment
Uncertainty and variability
Selection and use of defaults
Unified approach to dose-response assessment
Cumulative risk assessment
Improving the utility of risk assessment
Stakeholder involvement
Capacity-building
Design of risk assessment
• Design: The process of planning a risk assessment and
ensuring that it has the attributes desired by the decision
maker given various system constraints
• Analogy to product design: What is the “right” car to buy?
• From decision-support perspective, there are multiple
desirable attributes which may at times conflict with one
another
•
•
•
•
•
Use of best science and methods
Inclusiveness of scope
Inclusiveness of process
Transparency
Timeliness
Key conclusions
• Increased attention needed to the design
of risk assessment at its formative stages
• Planning and scoping and problem
formulation (as in EPA ecological and
cumulative risk guidance) should be
formalized and implemented.
Key design steps
• Planning and scoping: Discussion among
decision-makers, assessors, and stakeholders to
establish issue to be assessed and goals,
breadth, depth, and focus of assessment
• Problem formulation: Technical implications of
planning and scoping, with a conceptual model
and an analysis plan
• Early identification of decision-making options
Related concept –
Value of information (VOI)
• Decision makers face the tension between
acting now or delaying decisions while research
is conducted
• VOI analysis offers a framework for systematic
examination of this issue
• Part of committee’s charge, topic of interest to EPA
• Key point of emphasis:
• VOI analysis is not possible in the absence of a
structured analysis that includes information about
risk management options and detailed uncertainty
characterization
Conclusions about VOI
• VOI analysis is technically challenging for any
realistic problem structure, may place unrealistic
demands on science
• Needs knowledge of decision rules, well-defined
uncertainty characterization, assumes primacy of risk
assessment outputs
• Formal VOI may be impractical in most settings,
but the essential reasoning behind VOI can be
adopted
• Understanding the causal link between a specific
source of information, how a decision-maker would
change behaviors given this information, and how this
could improve decisions
• Similar concepts can be applied to consider the value
of methods rather than the value of information
Uncertainty
• A huge, cross-cutting topic…
• There have been substantial differences among EPA’s
approaches to and guidance for addressing uncertainty
in exposure and dose-response assessment.
• The level of detail for characterizing uncertainty is
appropriate only to the extent that it is needed to inform
specific risk-management decisions appropriately.
• Inconsistency in the treatment of uncertainty among
components of a risk assessment can make the
communication of uncertainty difficult and sometimes
misleading.
• Example: How do you interpret Monte Carlo analysis with
emissions assumed to be “known”, exposure model with limited
parametric uncertainty, and epidemiology with uncertainty only
related to study’s statistical power?
“Tiers” of uncertainty analysis
Example: WHO, 2007
• Tier 0: Default assumptions, single value for
result
• Tier 1: Qualitative but systematic identification
and characterization of uncertainties
• Tier 2: Quantitative evaluation of uncertainty
making use of bounding values, interval
analysis, sensitivity analyses
• Tier 3: Probabilistic assessments with single or
multiple outcome distributions reflecting
uncertainty and variability
Variability
• Variability in human susceptibility has not
received sufficient or consistent attention
in many EPA health risk assessments
• Committee encourages EPA to move
toward the long-term goal of quantifying
population variability more explicitly in
exposure assessment and dose-response
relationships.
General conclusions re uncertainty
and variability
• EPA should encourage risk assessments to characterize
and communicate uncertainty and variability in all key
computational steps.
• Uncertainty and variability analysis should be planned
and managed to reflect the needs for comparative
evaluation of risk management options.
• In the short term, EPA should adopt a “tiered” approach
for selecting the level of detail in uncertainty and
variability assessments
• This should be made explicit in the planning stage.
• EPA should develop guidance on the appropriate level of
detail needed in uncertainty and variability analyses
• Provide clear definitions and methods for identifying and
addressing different sources of uncertainty and
variability.
Related topic: Defaults
• Also called “inference options”, “default options”,
“science policy”, “risk assessment policy”
• The best choice for parameters/models on the basis
of risk assessment policy in the absence of data to
the contrary
• By definition, cannot be proven correct/incorrect,
but often has some scientific underpinning
• First formalized in the Red Book
Selection and use of defaults
• Established defaults need to be maintained for risk
assessment steps that require inferences
• EPA, for the most part, has not yet published clear,
general guidance on what level of evidence is needed to
justify use of agent-specific data instead of a default.
• Clear criteria should be available for judging whether, in
specific cases, data are adequate for direct use or to
support an inference in place of a default.
• There are a number of defaults (missing or implicit
defaults) that are engrained in EPA risk-assessment
practice but are absent from its risk-assessment
guidelines.
• EPA does not quantify uncertainty when default
assumptions are used.
Recommendations re defaults
• EPA should
• continue and expand use of the best, most
current science to support and revise default
assumptions.
• develop clear, general standards for the level
of evidence needed to justify the use of
alternative assumptions in place of defaults.
• work toward the development of explicitly
stated defaults to take the place of implicit
defaults.
Evidentiary standard for replacing
default
• “The committee recommends that EPA
adopt an alternative assumption in place
of a default when it determines that the
alternative is ‘clearly superior’, that its
plausibility clearly exceeds the plausibility
of the default.”
Showing uncertainty when using
defaults
• To the extent feasible, EPA should move beyond
qualitative description of uncertainty when
default assumptions are used
• Long term – improved probabilistic description of
uncertainty commensurate with risk
management needs
• Short term – criteria for listing alternative values
• Goal – Provide sensitivity analysis to illustrate impact
of alternative assumptions and hence characterize
robustness of risk estimates
• Limit attention to assumptions with plausibility
comparable to the default
• Goal is not an exhaustive presentation of plausible
estimates
Unification approach to doseresponse assessment
• Historically, dose-response assessments at EPA
conducted differently for cancer and noncancer
effects
• Methods have been criticized for not providing
the most useful results.
• A consistent approach to risk assessment for
cancer and noncancer effects is scientifically
feasible and needs to be implemented.
Current approach
Hazard Assessment
Sort by Cancer or Non-Cancer Endpoints
Non-Cancer Risk Assessment
Identify NOAEL or derive POD
Cancer Risk Assessment
Low Dose “Non-Linear”
Evaluate Mode of Action (MOA)
MOA not established
Select Uncertainty/Adjustment Factors
- Cross-species (U1)
- Human interindividual variability (U2)
- Other (U3)
Linear MOA
Animal to human dose conversion:
mg/kg3/4-d scaling or pharmacokinetic modeling
with pharmacodynamic adjustment
Derive Reference Dose
RfD = POD / (U1 x U2 x U3)
Derive POD (for example, LED01) and Slope Factor
For example, Slope Factor = 0.01 / POD
Risk Characterization: Hazard Index (HI) or MOE
HI = Σi (Exposure / RfDi)
MOE = POD / Exposure
Risk Characterization: Low Dose Risk
Extra Risk = Slope Factor x Exposure
Limitations and Issues
Limitations and Issues
1. Possibility for low dose linearity (for example,
due to background exposure) not assessed
2. No risk measure produced. HI, RfD, and MOEs
of limited utility for risk/benefit analyses
3. Uncertainty not distinguished from variability or
other adjustments
1.
2.
3.
Inter-human variability in risk either not
addressed at all (animal based) or incompletely
(epidemiology based)
For low dose non-linear carcinogens, no risk
measure produced. HI, RfD, and MOEs of
limited utility for risk/benefit analysis
Uncertainty is not characterized
Thoughts re current approach
• EPA has taken important steps to harmonize cancer and
noncancer approaches, but with many scientific and
operational limitations:
• Noncancer effects do not necessarily have threshold or low-dose
nonlinearity
• The mode of action of carcinogens varies.
• Background exposures and underlying disease processes
contribute to population background risk, which can lead to
linearity at the population doses of concern.
• RfDs and RfCs do not quantify risk for different magnitudes of
exposure but rather provide a bright line with limited use in riskmanagement decision-making
• Cancer risk assessments usually do not account for human
differences in cancer susceptibility (other than possible
differences in early-life).
Dose-response
relationship is
dependent on
heterogeneity in
background exposure
(endogenous and
xenobiotic), biological
susceptibility
Unification recommendations
• A consistent, unified approach for dose-response
modeling that includes formal, systematic assessment of
• background disease processes and exposures
• possible vulnerable populations
• modes of action that may affect a chemical’s dose-response in
humans.
• Redefine the RfD or RfC as a risk-specific dose
• provides information on the percentage of the population
expected above or below a defined acceptable risk (with specific
degree of confidence).
• Formal introduction of variability into cancer doseresponse modeling
• Will require implementation and development
• As new chemicals are assessed or old chemicals are
reassessed
• Of test cases to demonstrate proof of concept.
Assemble Health Effects Data
Endpoint Assessment
• Identify adverse effects, focusing on those of concern for exposed
populations
• Identify precursors and other upstream indicators of toxicity
• Identify gaps – for example, endpoints or lifestages under assessed or
not assessed
MOA Assessment
(for each endpoint of concern)
• Research MOAs for
endpoints observed in
animals and humans
• Evaluate the sufficiency of
the MOA evidence
• Evaluate endogenous
processes contributing to MOA
Vulnerable Populations
Assessment
Identify potentially vulnerable
groups and individuals,
considering endpoints, the
potential MOA, background
rate of health effect, and other
risk factors
Background Exposure
Assessment
• Identify possible
background exogenous and
endogenous exposures
• Conduct screening level
exposures and analysis focusing
on high end exposure groups
Conceptual Model Selection
Develop or select conceptual model:
• From linear conceptual models unless data sufficient to reject low dose linearity
• From non-linear conceptual models otherwise
Dose Response Method Selection
Select dose response model and method based on:
• Conceptual model
• Data availability
• Risk management needs for form of risk characterization
Dose-Response Modeling
and Results Reporting
Diagnostic questions to aid doseresponse assessment
• What is known or suspected to be the chemical’s MOA?
• What underlying degenerative or disease processes might the
toxicant effect?
• What are the background incidences and population distributions of
these processes?
• Are there identified sensitive populations?
• What environmental contaminants in air, drinking water, food or in
consumer products (e.g., in cosmetics) or endogenous chemicals
(e.g., natural hormones) are similar to the chemical?
• Could they potentially operate by MOAs similar to that of the
chemical in question?
• What chemicals might operate by a different MOA but have the
potential to affect the same toxic process as the chemical under
study?
• Can subgroups with particularly high exposures be identified?
• …more
Conceptual dose-response models –
Based on MOA, background exposure and
disease processes
General structures of the three
models
• Model 1: Estimate BMD, determine human POD
with uncertainty, extrapolate linearly
• Like current cancer framework, but applied to noncancer
• Model 2: Estimate BMD, determine human POD
with uncertainty, determine risk-specific RfD
given human heterogeneity
• Like proposals in the literature by Hattis, Evans
• Model 3: Estimate BMD, incorporate
interindividual variability factor, extrapolate
linearly while retaining variability
• Like current cancer framework with variability factor
Cumulative risk assessment
• EPA is increasingly asked to address broad public-health
and environmental-health issues that stakeholder groups
often consider inadequately captured by current risk
assessments
• multiple exposures
• complex mixtures
• vulnerability of exposed populations
• There is a need for cumulative risk assessments as
defined by EPA that include
• combined risks posed by exposure to multiple agents or
stressors
• aggregate exposure to a given agent or stressor
• all routes, pathways, and sources of exposure
• Chemical, biologic, radiologic, physical, and psychologic
stressors are considered.
Conclusions on cumulative risk
• Committee applauds the agency’s move toward the
broader definition, making risk assessment more
informative and relevant to decisions and stakeholders.
• However, in practice, EPA risk assessments often fall
short of what is possible and supported by agency
guidelines.
• Little consideration of nonchemical stressors, vulnerability, and
background risk factors.
• Because of the complexity of considering so many
factors simultaneously, there is a need for:
• Simplified risk assessment tools
• Orientation around pertinent risk management options to limit the
number of stressors under formal consideration
Step 1:
-Develop a conceptual model for the stressors of primary interest for the analysis
(stressors that would be significantly influenced by any of the risk-management options
under study).
- MOA assessment, assessment of background exposures to chemical and
nonchemical stressors that may affect the same health outcome, vulnerability
assessment that takes into account underlying disease processes in the population
-Identify the receptors and end points affected by these stressors.
-Review the conceptual model and stressors, receptors, and end points of interest with
stakeholders in initial planning and scoping.
Step 2:
-Use epidemiologic and toxicologic evidence and screening-level benefit calculations to
provide an initial evaluation of which stressors should be included in the cumulative risk
assessment. Gather stakeholder feedback and review and re-evaluate planning and
scoping for the analysis.
-Focus the assessment only on stressors that contribute to end points of interest for
risk-management options and are either differentially affected by different control
strategies or influence the benefits of stressors that are differentially affected.
Step 3:
-Evaluate benefits of different risk-management options with appropriate
characterization of uncertainty, including quantification of the effects of individual
stressors and bounding calculations of any possible interaction effects.
Step 4:
-If Step 3 is sufficient to discriminate among risk-management options given other
economic, social, and political factors, conclude the analysis; otherwise, sequentially
refine the analysis as needed, taking into account potential interactions.
Recommendations
• Draw on other approaches to incorporate interactions between
chemical and non-chemical stressors in assessments, including
those from ecologic risk assessment and social epidemiology
• Develop guidelines and methods for simpler analytical tools
• to support cumulative risk assessment
• to provide for greater involvement of stakeholders.
• In short-term, develop databases and default approaches to allow
for incorporation of key non-chemical stressors in the absence of
population-specific data, considering
• exposure patterns
• contributions to relevant background processes
• interactions with chemical stressors.
• In long-term, invest in research programs related to interactions
between chemical and non-chemical stressors, including
epidemiologic investigations and physiologically-based
pharmacokinetic modeling.
Improving the utility of risk
assessment
• Committee proposes a framework for risk-based
decision-making
• At its core are the “4 steps” as defined in the Red
Book
• Key difference from the Red Book in the initial and
final steps
• Framework asks implicitly:
• What options are there to reduce the hazards or
exposures that have been identified, and
• How can risk assessment be used to evaluate the
merits of the various options?
• Risk assessment as a means to an end
Improving the utility (II)
• Under this framework, the questions posed arise
from
• early and careful planning of the types of
assessments (including risks, costs, and technical
feasibility) and
• the required level of scientific depth needed to
evaluate the relative merits of the options being
considered.
• Risk management involves choosing among the
options after the appropriate assessments have
been undertaken and evaluated.
PHASE I:
PROBLEM FORMULATION
AND SCOPING
PHASE II:
PLANNING AND CONDUCT
OF RISK ASSESSMENT
PHASE III:
RISK MANAGEMENT
Stage 1: Planning
• For the given decision-context, what are the attributes of assessments necessary to characterize risks
of existing conditions and the effects on risk of proposed options? What level of uncertainty and
variability analysis is appropriate?
• What problem(s) are
associated with existing
environmental conditions?
• What are the relative health or
environmental benefits of the
proposed options?
Stage 2: Risk Assessment
• If existing conditions appear
to pose a threat to human or
environmental health, what
options exist for altering those
conditions?
• How are other decisionmaking factors (technologies,
costs) affected by the proposed
options?
• Hazard Identification
What adverse health or environmental effects
are associated with the agents of concern?
• What is the decision, and its
justification, in light of benefits,
costs, and uncertainties in each?
• Dose-Response Assessment
• Under the given decision
context, what risk and other
technical assessments are
necessary to evaluate the
possible risk management
options?
For each determining adverse effect, what is the
relationship between dose and the probability of the
occurrence of the adverse effects in the range of
doses identified in the exposure assessment?
• Risk Characterization
• How should the decision be
communicated?
What is the nature and
magnitude of risk associated with
existing conditions?
• Exposure Assessment
What risk decreases (benefits) are
associated with each of the
options?
What exposures/doses are incurred by each
population of interest under existing conditions?
Are any risks increased? What are
the significant uncertainties?
• Is it necessary to evaluate the
effectiveness of the decision?
• If so, how should this be done?
How does each option affect existing conditions and
resulting exposures/doses?
Stage 3: Confirmation of Utility
NO
• Does the assessment have the attributes called for in planning?
YES
• Does the assessment provide sufficient information to discriminate among risk management
options?
• Has the assessment been satisfactorily peer reviewed?
FORMAL PROVISIONS FOR INTERNAL AND EXTERNAL STAKEHOLDER INVOLVEMENT AT ALL STAGES
• The involvement of decision-makers, technical specialists, and other stakeholders in all phases of the processes leading to decisions should in no way compromise the technical assessment of risk, which is
carried out under its own standards and guidelines.
Phase I:
Problem Formulation and Scoping
• What is the problem to be investigated, and what is its
source?
• What are the possible opportunities for managing risks
associated with the problem? Has a full array of possible
options been considered, including legislative
requirements?
• What types of risk assessments and other technical and
cost assessments are necessary to evaluate existing
conditions and how the various risk-management options
alter the conditions?
• What impacts other than health and ecosystem threats
will be considered?
• How can the assessments be used to support decisions?
• What is the required timeframe for completion of
assessments?
• What resources are needed to undertake the
assessments?
Phase II:
Planning and conduct of risk assessment
Stage 1: Planning
• For the given decision-context, what are the attributes of
assessments necessary to characterize risks of existing conditions
and the effects on risk of proposed options?
• What level of uncertainty and variability analysis is appropriate?
Stage 2: Risk Assessment
Stage 3: Confirmation of Utility
• Does the assessment have the attributes called for in planning?
• Does the assessment provide sufficient information to discriminate
among risk-management options?
• Has the assessment been satisfactorily peer reviewed?
Phase III:
Risk management
• What are the relevant health or environmental
benefits of the proposed risk-management
options?
• How are other decision-making factors
(technologies, costs) affected by the proposed
options?
• What is the decision, and its justification, in light
of benefits, costs, and uncertainties in each?
• How should the decision be communicated?
• Is it necessary to evaluate the effectiveness of
the decision? If so, how should this be done?
The framework…
• Systematically identifies problems and options that risk
assessors should evaluate at the earliest stages of
decision-making.
• Expands the array of impacts assessed beyond
individual effects (e.g., respiratory effects) to include
broader questions (e.g., health status and ecosystem
protection).
• Provides a formal process for stakeholder involvement.
• Increases understanding of the strengths and limitations
of risk assessment by decision-makers at all levels, e.g.,
by making uncertainties and choices more transparent.
• Maintains the conceptual distinction between risk
assessment and risk management articulated in the Red
Book.
Framework recommendation
• EPA should adopt a framework for riskbased decision-making that embeds the
Red Book risk assessment paradigm into
a process with
• initial problem formulation and scoping,
• upfront identification of risk-management
options, and
• use of risk assessment to discriminate among
these options.
Stakeholder involvement
• Many stakeholders believe that the current process for
developing and applying risk assessments lacks
credibility and transparency.
• Greater stakeholder involvement is necessary to ensure
that the process is transparent and to ensure risk-based
decision-making proceeds effectively, efficiently, and
credibly.
• Stakeholder involvement needs to be an integral part of
the risk-based decision-making framework.
• It is important that EPA adhere to its own guidance on
stakeholder involvement particularly in the context of
cumulative risk assessment, in which communities often
have not been adequately involved.
Stakeholder recommendations
• EPA should establish a formal process for
stakeholder involvement in the framework
for risk-based decision-making with:
• time limits to ensure that decision-making
schedules are met
• incentives to allow for balanced participation
of stakeholders including impacted
communities and less advantaged
stakeholders.
Concluding thoughts
• The committee felt that, in spite of limitations,
risk assessment remains essential to EPA’s
mission to ensure protection of public health and
the environment.
• The committee hopes that the recommendations
and the proposed framework for risk-based
decision-making will provide a template for the
future of risk assessment in EPA and strengthen
the scientific basis, credibility, and effectiveness
of future risk-management decisions.