Models in Environmental
Regulatory Decision-Making
Report of the Committee on Models in the
Regulatory Decision Process
Presented at RASS- ISES Conference Call
Chris Whipple, Committee Chair
June 10, 2009
Origin of Study
Concerns over regulatory & judicial challenges
to models
Against a backdrop of
• Expanding reliance on models
• Increasing model innovations
• Increasing regulatory requirements
(Data Quality Act and others)
Origin of Study
Paul Gilman appointed EPA Science Advisor
•Revitalized the Council for Regulatory Environmental
Modeling (CREM)
•Moved CREM’s focus to model users
CREM tasked to
•Develop guidance document, web-based knowledge
base, regional workshops, stakeholder outreach
•Engage the National Academy of Sciences
NRC Task Statement
A National Research Council committee will provide
advice concerning the development of guidelines and a
vision for the selection and use of models at EPA. The
committee will consider cross-discipline issues related
to model use, performance evaluation, peer review,
uncertainty, and quality assurance/quality control. The
objective …. [is] to provide a report that will serve as a
fundamental guide for the selection and use of models in
the regulatory process. As part of its work, the
committee will need to carefully consider the realities of
EPA's regulatory mission so as to avoid an overly
prescriptive and stringent set of guidelines.
Committee Members
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Chris G. Whipple, Environ, Inc. (Chair)
M. Bruce Beck, University of Georgia
Clayton Clark, University of Florida
Robert T. Clemen, Duke University
Judith A. Graham, American Chemistry Council
Louis J. Gross, University of Tennessee
Winston Harrington, Resources for the Future
Philip Howard, Syracuse Research Corporation
Kimberly L. Jones, Howard University
Thomas E. McKone, University of California
Naomi Oreskes, University of California
Spyros N. Pandis, Carnegie Mellon University
Louise M. Ryan, Harvard School of Public Health
Michael L. Stein, The University of Chicago
Wendy E. Wagner, University of Texas School of Law
Fundamental Definition
What is an Environmental Regulatory Model?
“A computational model used to inform the environmental
regulatory process. Some models are independent of a
specific regulation …. Other models are created to provide
a regulation-specific set of analyses ….. The approaches
can range from single parameter linear relationship models
to models with thousands of separate components and
many billions of calculations.”
Implications of this definition
• Wide array of models under Committee’s
consideration
• Wide array of regulatory applications
Information Gathering
Workshop #1 – breadth of environmental regulatory
modeling at EPA and issues associated with these
activities
• Role of EPA regional offices
• Interactions of EPA, states, and consultants
• Variability in scale of model applications
• Environmental and business group perspectives
Workshop #2 – focus on 3 specific issues in-depth
• Proprietary models – motivations, proprietary
features, mitigation
• Peer review – types and elements of peer review
• Uncertainty analysis – methods and use in decisions
Information Gathering
Workshop #3 - new sources of information, future health
risk assessment modeling, and the role of models in
decision-making
• Genomics to environmental satellites
• Potential to mechanistically model human responses
• Decisions made by non-modelers
Study Approach
Committee considered broad patterns of models
usage in making recommendations
EPA has significantly advanced the science of
environmental modeling—report intended to help
provide an cross-agency vision and principles for
the use of models in the future
Never say “never” or “always”
Model Use in Regulatory
Processes
Models always constrained by computational limitations,
assumptions, and knowledge gaps
Environmental models can never be completely
“validated” in the traditional sense. But they can be
“evaluated”
Regulatory models are typically used to describe
important, complex, and poorly characterized problems
Models in the regulatory process best seen as tools
providing inputs, as opposed to “truth generating
machines”
Implications for Regulatory
Model Use
Evaluation of regulatory models requires different
tradeoffs than those for research models
Not simply how well do model estimates match
observations, but also how reproducible, transparent,
and useful a model is to the regulatory decision
Requires regulatory models be managed to be updated in
a timely manner and assist users and others to
understand conceptual basis, assumptions, input data
requirements, and life history
List and Structure of
Recommendations
MODEL EVALUATION
Life Cycle Model Evaluation
Peer Review
Assessing and Communicating Uncertainty
The Interdependence of Models and Measurements
Retrospective Analysis of Models*
PRINICIPLES FOR MODEL DEVELOPMENT, SELECTION, AND APPLICATION
Model Parsimony*
Extrapolation*
Proprietary Models
MODEL MANAGEMENT
Models and Rulemakings
Model Origin and History*
Improving Model Accessibility*
Life-Cycle Model
Evaluation
Evaluation of regulatory models is the process of
deciding whether and when a model and its
application is appropriate
Evaluation is a multifaceted activity—peer review,
corroboration of results with data and other
information, QA/QC checks, uncertainty and
sensitivity analyses, and other activities
Not a one time event—evaluation of regulatory
models should continue throughout regulatory
applications and revisions to the model
Life-Cycle of a Model
Problem
Formulation
Conceptual
Model
Constructed
Model
Model Use
Life-Cycle Model
Evaluation
All models should have a life-cycle evaluation plan of a
size and complexity commensurate with its regulatory
significance
Plan should address how model evaluation will occur
throughout a model’s life cycle
The committee did not make organizational
recommendations of how EPA should achieve this
A conceptual commitment to life cycle model evaluation
is needed
Peer Review
Peer review activities are an integral component of the
evaluation process
Some simple, uncontroversial models might require little
or no peer review
But many models require detailed and multiple peer
reviews that involve time and resources more extensive
than a report or journal paper—review of only model
documentation is not sufficient for important models
Some regulatory model peer reviews will involve
reviewing the model code and its documentation, and
comparing the model results against known test cases
Assessing/Communicating
Uncertainty
Two Approaches
Represent uncertainties probabilistically and
calculate the probability distribution of any
model result
 difficult to carry out
 obscures the sensitivities of the outcome to
individual sources of uncertainty
Scenario assessment and/or sensitivity analysis
 often more transparent
 may ignore important information corresponding to
other scenarios not included in assessment and
whatever is known about their relative likelihoods
Assessing/Communicating
Uncertainty
Many instances require probabilistic methods to
properly characterize uncertainties, propagate
them through the modeling exercise, and clearly
communicate the overall uncertainties
Recommend the use of case-specific hybrid
approaches in which some unknown quantities
are treated probabilistically, and others can be
manipulated in a scenario-assessment mode by
the decision makers
Requires communication between modelers and
decision-makers
Models and Data
Interdependence of measurements and modeling
must be considered during development of a
conceptual model
Requires development of adaptive strategies to
advance data collection and model development
Future will see vast new sources of informationto maximize the effectiveness of new data
collection efforts, models must be able to use
this information AND guide the design of data
collection
Proprietary Models
Use of proprietary models can produce distrust
among regulated parties and other groups
Recommends that EPA adopt a strong preference
for nonproprietary software
Only use proprietary models when a clear case can
be made for their advantages
Proprietary models should be subject to same
evaluation requirements as for public models
Models and Rulemaking
EPA may perceive that rigorous life cycle evaluation of
models and documenting such features in need of
improvement—may expose models to greater risk of
challenges and trigger lengthy regulatory process
Life cycle evaluation practices may be much easier if such
activities were considered to satisfy regulatory
requirements, such as those of the Information Quality Act
EPA could consider establishing a generic rule for the
process of evaluation and adjustment of models used in
rule-making to provide adequate opportunities for public
comment and revision of an individual model without
triggering the need for a separate rule-making for each
model revision
List and Structure of
Recommendations
MODEL EVALUATION
Life Cycle Model Evaluation
Peer Review
Assessing and Communicating Uncertainty
The Interdependence of Models and Measurements
Retrospective Analysis of Models*
PRINICIPLES FOR MODEL DEVELOPMENT, SELECTION, AND APPLICATION
Model Parsimony*
Extrapolation*
Proprietary Models
MODEL MANAGEMENT
Models and Rulemakings
Model Origin and History*
Improving Model Accessibility*