Rapid Risk Assessment
(and/or Rapid Risk Analysis)
Are they possible?
Villie Flari,
Food and Environment Research Agency, UK
Villie.flari@fera.gsi.gov.uk
JIFSAN / Fera annual meeting, June 2011
Trigger for this talk…
• … part of my work when at CFSAN, FDA (2007 – 2009)
– “Rapid Risk Assessment Framework” project: a tool to apply with CFSAN
emergency response procedures
– Ambitious and quite challenging project: aimed to identify whether there is
room for improving the existent strategies, employed at the time by
CFSAN/FDA during emergency events, and if yes to suggest and justify
alternative approaches
• Review FDA/CFSAN processes to assess public health threats during emergency events.
• Identify areas of opportunities for implementing risk assessment thought approaches
with the view to improving the current FDA/CFSAN process.
• Prepare a framework document to describe the recommended process.
• Develop templates for assessing information, characterizing unknown factors and for
communicating assessment results effectively to risk managers and/or risk
communicators.
Project orphan..
Are any changes to current
schemes and approaches truly
needed?
• What is the outcome we want to estimate – which
is our aim?
– Preventive approaches and emergency responses for ensuring food
safety and protecting human health
• Overall, not that bad
• Apart form regulatory processes a number of initiatives and
support systems nationally and internationally developed over the
last 30 years - currently in place (e.g. RASFF)
2008
2008-9
2009
2010
2011
Food safety challenges - Dynamic
constantly change…
Emerging sciences
& technologies
Source: Food Safety Risk Analysis: a guide for national food safety authorities, (FAO Food and Nutrition Paper 87, 2006)
Emerging risks –
emerging sciences /
technologies
?
Nano-sized
materials
Genetically
Modified
Organisms
?
?
Tactical and strategic
problems
Synthetic
Biology
Unknown, ?
thus far,
emerging
risk(s)
Designing rapid risk assessments
– are they possible?
• What is the outcome we want to estimate – which
is our aim?
– What do we mean precisely when we term a
document as a “risk assessment” or a “rapid risk
assessment”?
– How do we define the term “risk”?
• Safety assessment?
• Risk assessment?
• A collection of (background + epi) information?
Possible scenarios
Hazard
Chemical
Biological
Known
Scenario 1
Scenario 3
Scenario 5
Unknown
Scenario 2
Scenario 4
Scenario 6
Identified
vehicle (s)
Non identified
vehicles (s)
Looking into historical events &
following events “in vivo”
Hazard
Chemical
Biological
Known
Melamine in
infant formula –
2008 (previous
relevant case:
Melamine in pet
food – 2007)
Salmonella in
peanut butter –
2009 (previous
relevant case:
Salmonella in
Peter Pan)
Salmonella
saintpaul in fresh
produce –
2008/2009
Unknown
Melamine in pet
food - 2007
Scenario 4
Scenario 6
Identified
vehicle (s)
Non identified
vehicles (s)
Historical events
– “Melamine in pet food” - March to May 2007
Following events “in vivo”
– “Salmonella saintpaul in fresh produce” – May to August 2008
– “Salmonella typhimurium in peanut butter” - November 2008 to January 2009
What did we learn?
Incomplete
knowledge
Limited
time
Many
dimensions
International
Production
Historical
Data (Firm)
Distributors
Compliance
Processing
Plants
Packing
SOP
Kill steps
FIRMS’
PRACTICES
Distributors
RISK ASSESSMENT
Shelf life
of product
Questionnaires
Number of Ill
Source
Vehicle (s)
Complaints
Epi curve
Most illnesses
> 10/01
Illness onset
range
EPIDEMIOLOGY
Case control
Study results
Deaths related
to outbreak
Illnesses
Outside USA
Raw data
Trace back
2nd level
Best before date
LOT
Trace back
Trace back
3rd level
1st level
INVESTIGATION
PRODUCTS
FINDINGS
MANUFACTURED
Databases
Inspection reports
DISTRIBUTION
TRACEFORWARD
OUTBREAK
INVESTIGATION
FACTS
INVESTIGATIVE LABORATORY FINDINGS
CLINICAL
FDA
BACKGROUND
DATA
data
INFORMATION
International
samples
FIRM
data
STATE
Data
Background / rationale
Market share
(specific to firm (s)
Firm’s product
Trace back
TRACEFORWARD
General info
Specific to product
FOOD SUPPLY CHAIN
Domestic
RefDoc 11
Lines of investigation of emergency events: relevant information
needs
Recall
Problem i.e. Risk Management question (s)
See RefDoc1 and 2 for extended information
Assess problem against criteria for
Class I Recall1
See RefDoc 2 for extended information
Problem classified
As Class I Recall?
Risk Management
decision
Yes
No
Define RA endpoint (s)
Yes
See RefDoc 8 for extended information
Is Risk Assessment required by RM?
Continue investigation
No
Introduce RA as line of
investigation
Define all lines of investigation
See RefDoc11 for extended information
See RefDoc 8 for extended information
Uncertainty analysis
Decision making process
Monitor & Organize Data
See RefDoc12 for extended information
Evaluate data
Accept, Reject, Withhold data
See RefDoc 14 for extended information
Form hypothesis
See RefDoc 14 for extended information
See RefDoc 3 & 16 for extended information
Repeat for each line of investigation
Test hypothesis
See RefDoc 14 for extended information
Do not
accept
hypothesis
No
Risk Communication
Consider all lines of investigation
Data considered enough for
accepting hypothesis?
See RefDoc 14
for extended information
Yes
Hypothesis
accepted
Yes
Need to redefine
problem?
No
Risk Management
options
RefDoc 11
Risk Assessment’s value:
 Bringing all the information together – connecting all these lines of
investigation in order to reveal patterns, hypotheses.
 Gather information, direct research for the future.
TRACEFORWARD
Risk Assessment would allow one to be able to answer
Background / rationale
questions regarding “how risky it is to consume peanut
Risk Assessment would allow to address
SUPPLYe.g.
CHAIN
butterFOOD
products”,
“if I eat 200 crackers from 200
questions regarding the effect of risk
different packs which was the probability to have been
factors if certain mitigation methods would
contaminated by Salmonella typhimurium?”
FIRMS’
be followed; for example “If
the
PRACTICES
temperature during a kill step is always
RISK ASSESSMENT
above the threshold of 350 F what would
be the decrease in probable pathogen
Risk Assessment approach can be employed to ask information about the
concentration in a cookie?”
denominator of epidemiological and investigativeUsually
laboratorya results:
e.g.
presence/absence
of
“During the “Salmonella typhimurium in the peanutmicrobiological
butter” case howhazard
many
the
is
given. Colony
INVESTIGATION
samples have they been put under test?”
Forming Units (CFUs) informationFINDINGS
is not given.
Usually, a lot of attention is paid on the positive microbiological
samples,
As a result, Dose
Response information is
but not on the total of samples tested. The denominator
could give us
missing. Requesting
CFUs
could help risk
INVESTIGATIVE
information on EPIDEMIOLOGY
the actual risk posed by the particular
microbiological
assessments
in the futureLABORATORY
as well.
FINDINGS
hazard.
Is this what we need?
• Partially yes, but ….
Risk assessments / analyses in
emergency responses: why,
how?
The STEC outbreak reported from Germany is noteworthy
considering its magnitude: 276 HUS cases and two deaths due to
HUS reported in just a few weeks. The majority of cases reported
are adult women. Usually, about 15% of children with STEC infection
present with HUS, with this proportion being much lower among
adults. This means that several hundred STEC cases with diarrhoea
are likely to be occurring in the current outbreak.
Likely
47% sure?
71% sure?
83% sure?
The isolated outbreak strain STEC O104:H4 is very rare. Prior to
the current outbreak, only one case has been documented in
literature, and this case was a woman in Korea in 2005.
The case-control study in Hamburg revealed that raw tomatoes,
fresh cucumber and leafy salad are the likely vehicles of
infection. Samples of fresh cucumbers taken in Hamburg tested
positive for STEC, however, the exact time and place of
contamination remains unclear.
It is unclear whether the results from Hamburg can be extrapolated
to the whole of Germany. Furthermore, it cannot be excluded that
an alternative food item is the vehicle of infection. The definite
source of the infection remains to be confirmed.
There is currently no indication that either the source of the outbreak
or the vehicle of infection has been distributed outside of Germany.
The question then
becomes…
• Whether approaches and thought processes we
develop for and apply in innovative Risk
Assessments (or Analyses…), can help us
overcome difficulties such as those we encountered
recently in the food safety field.
– Chemical: Can we identify the true hazard quickly?
– Microbiological: Can we identify the true vehicle(s)/source(s) of an
outbreak quickly?
– Can we foresee possible future challenges relevant to new sciences
and technologies that are being developed?
– Can we produce reproducible, transparent, fully defensible
assessments / analyses that will withstand media pressure, public
criticism?
What did we learn?
Incomplete
knowledge
The work on this project highlighted the significance a
number of aspects that appear to be inherent in an
emergency investigations:
•
•
Limited
time
•
Unknown factors, and in particular the value of embracing
uncertainty - embracing uncertainties by recognizing them,
classifying them, characterizing them can only be beneficial for
all involved in the investigation, the scientists, the decision
makers and the public.
The art of eliciting expert judgment: subjective information
plays a major part in the science on which rapid decisions are
made. How do we ensure that we obtain the most informative
points of view, and how do we ensure that we capture experts’
uncertainty? How do we combine different opinions, if this is
deemed necessary?
Following a problem – solving approach: this type of work may
be quite different from simply following a process. Asking “what
is the problem, and how the problem could be defined?” is not
merely a philosophical question; instead, it is the most
important question to be settled in the start of an investigation.
Uncertainty in risk assessments,
risk analyses in emergency
events: why, how?
• What is the desired outcome – which is our aim?
• Rapid Risk Assessment
– Risk is the uncertainty of the outcome
• Outcome(s)  Risk Assessment endpoint(s)
– Which are the chances of my child dying if s/he consumes x amount
of contaminated food?
– Which is the proportion of Denmark population who may suffer from
severe adverse affects (i.e. >10 days of hospitalisation) if they
consume x amount of contaminated food?
• Uncertainty of outcome(s)
ANDY
HART
– Probability distributions - Quantitative (Hard data and/or
expert judgment)
– + Qualitative (acquired via expert judgment) information
What did we learn?
Incomplete
knowledge
The work on this project highlighted the significance a
number of aspects that appear to be inherent in an
emergency investigations:
•
•
Limited
time
•
Unknown factors, and in particular the value of embracing
uncertainty - embracing uncertainties by recognizing them,
classifying them, characterizing them can only be beneficial for
all involved in the investigation, the scientists, the decision
makers and the public.
The art of eliciting expert judgment: subjective information
plays a major part in the science on which rapid decisions are
made. How do we ensure that we obtain the most informative
points of view, and how do we ensure that we capture experts’
uncertainty? How do we combine different opinions, if this is
deemed necessary?
Following a problem – solving approach: this type of work may
be quite different from simply following a process. Asking “what
is the problem, and how the problem could be defined?” is not
merely a philosophical question; instead, it is the most
important question to be settled in the start of an investigation.
Expert judgment in risk
assessments in emergency
responses: why, how?
• Who is to be involved?
– Best expertise possible
• Lists of experts available
• Calibration of experts only in retrospect (time-consuming)
– Fit for purpose method
• Qualitative information?
• Quantitative information?
• Both?
Uncertainty of experts to be captured
 Trained risk analysts
 Table top exercises
ROGER
COOKE
What did we learn?
Incomplete
knowledge
The work on this project highlighted the significance a
number of aspects that appear to be inherent in an
emergency investigations:
•
•
Limited
time
•
Unknown factors, and in particular the value of embracing
uncertainty - embracing uncertainties by recognizing them,
classifying them, characterizing them can only be beneficial for
all involved in the investigation, the scientists, the decision
makers and the public.
The art of eliciting expert judgment: subjective information
plays a major part in the science on which rapid decisions are
made. How do we ensure that we obtain the most informative
points of view, and how do we ensure that we capture experts’
uncertainty? How do we combine different opinions, if this is
deemed necessary?
Following a problem – solving approach: this type of work may
be quite different from simply following a process. Asking “what
is the problem, and how the problem could be defined?” is not
merely a philosophical question; instead, it is the most
important question to be settled in the start of an investigation.
Problem i.e. Risk Management question (s)
See RefDoc1 and 2 for extended information
Assess problem against criteria for
Class I Recall1
See RefDoc 2 for extended information
Problem classified
As Class I Recall?
Risk Management
decision
Yes
No
Define RA endpoint (s)
Yes
See RefDoc 8 for extended information
Is Risk Assessment required by RM?
Continue investigation
No
Introduce RA as line of
investigation
Define all lines of investigation
See RefDoc11 for extended information
See RefDoc 8 for extended information
Uncertainty analysis
Decision making process
Monitor & Organize Data
See RefDoc12 for extended information
Evaluate data
Accept, Reject, Withhold data
See RefDoc 14 for extended information
Form hypothesis
See RefDoc 14 for extended information
See RefDoc 3 & 16 for extended information
Repeat for each line of investigation
Test hypothesis
See RefDoc 14 for extended information
Do not
accept
hypothesis
No
Risk Communication
Consider all lines of investigation
Data considered enough for
accepting hypothesis?
See RefDoc 14
for extended information
Yes
Hypothesis
accepted
Yes
Need to redefine
problem?
No
Risk Management
options
RefDoc 1
Defining the problem - question
Investigators, Analysts
Yes
Assessment of problem,
i.e. risk management question (s),
begins
Decision Makers,
Risk Managers
No
Background / rationale
Questions considered
unambiguous by
investigators, analysts?
1: Report trigger (s)
Template flow chart
2: Phrase question (s) of investigation
that answers provided fit the purpose.
“To identify the food vehicle responsible for causing an outbreak of
typhimurium
infections
with
the
following
PulseNet
strains1:JPXX01.1818, JPXX01.1825, JPXX01.0459”.
 Problem (i.e. risk management question (s)) is dynamic; thus, it may need to be
redefined in subsequent phases of the investigation.
“To a)
identify distribution patterns of Peanut Corporation of America
products, b) elicit expert opinion on manufacturing practices and c) review
epidemiological data associated with Salmonella typhimurium, particularly
with the view to prioritizing products according to the risk they may impose
to consumers”.
Advantages:
 Diminishing ambiguity in provided answers.
 Allowing investigators, analysts who participate in the emergency case to get
familiar with important information of the investigation irrespective of their role
and/or expertise and have the opportunity to look at the big picture of the
investigation.
Background / rationale
strains JPXX01.1818, JPXX01.1825, JPXX01.0459 in collaboration with CDC
Template flow chart
Salmonella
RefDoc 1
 It is best to define the problem (i.e. risk management question (s)) precisely to ensure
What did we learn?
•
Following a problem – solving approach: this type of work may
be quite different from simply following a process. Asking
“what is the problem, and how the problem could be defined?”
is not merely a philosophical question; instead, it is the most
important question to be settled in the start of an investigation
•
“Identifying, defining and then respecting the natural shape of
the problem does not come naturally to agencies, since these
are based on an organizational structure that was formed
years ago and most of the approaches they follow are
process-based” – Sparrow, 2008
•
Perhaps, such an approach becomes even more challenged
during an investigation of an emergency event as the
organization is under extra pressure to deliver results and
facilitate relevant decision making.
People
Perceptions
Biases
Following a problem-solving
approach
“Solving a problem simply means representing it so as
to make the solution transparent”
Herbert A. Simon: The sciences of the artificial, 1996
pet food” emergency event
Hypothesis:
Melamine responsible
for illnesses and deaths
Melamine
Toxicology findings:
Melamine not toxic
for mammals, in particular for
the recorded adverse effects
03/27/2007
Chemistry findings:
Further compounds identified
in contaminated wheat gluten
ureidomelamine, urea,
ammeline,
ammelide, cyanuric acid
03/30/2007
Pathology findings:
Acute/subacute kidney failure
in cats and dogs
03/15/2007
Ammeline
Urea
Ammelide
Hypothesis:
Other compound(s)
responsible
for illnesses and deaths
Hypothesis:
More than one compounds
responsible
in association with other
agents
for illnesses and deaths
Cyanuric Acid
Toxicology findings:
There may be some additional
links
between melamine and uric acid
04/03/2007
Ureidomelamine
Toxicology findings:
No compound, of those identified
in contaminated wheat gluten
is considered as toxic to mammals
04/04/2007
Melamine
Ammeline
Ammelide
Cyanuric Acid
Pathology findings:
Oral toxicology experiments indicate
formation of crystals due to
combination of chemicals, e.g.
melamine+ammeline+ammelide+
cyanuric acid
04/24/2007
Example
Chemistry findings:
Melamine present in
contaminated wheat gluten
03/25/2007
RefDoc 15
Building a conceptual model of the problem: example based on “Melamine in
ECRT COLLABORATES
WITH EXPERTS AND
RISK MANAGERS
TO DEFINE
STAKEHOLDERS,
CRITERIA
ECRT COLLABORATES
WITH EXPERTS TO DISPLAY
SENSITIVITY ANALYSIS,
GATHER INFORMATION
ECRT COLLABORATES
WITH EXPERTS,
ANALYSTS,
AND DECISION MAKERS
TO APPLY DECISION
MAKING TOOLS
ECRT decides that:
Hypothesis
Is accepted
Intervention
A
Intervention
B
Intervention
C
…
Stakeholders – Criteria - Units of measure for criteria
What if scenario
What if scenario
What if scenario
What if scenario
What if scenario
What if scenario
What if scenario
What if scenario
What if scenario
What if scenario
Aggregation of criteria in a single function in order to be
able to “compare” and prioritize between alternative
scenarios. This could be performed via a number of
Decision Making tools.
Default method to handle information
ECRT COLLABORATES
WITH EXPERTS TO DEFINE
OPTIONS OF
INTERVENTION
RefDoc 16
Decision making
models
Challenges in ameliorating risk
assessments / risk analyses:
Strategic versus tactical
• Similarities
–
–
–
–
Encompass uncertainties
Define problem in question precisely
Employ the best experts
Provide support tools for transparent, reproducible, fully
defensible assessments
• Further challenges when dealing with emergency
events
– All of the above become even more crucial when dealing
with emergency investigations!
Thank you!
• CFSAN / FDA colleagues
– Sherri Dennis
– Elisa Elliott
– Jack Guzewich
– David Hattan
– Faye Fenstein
– Karl Klonz
– Sherri McGarry
Villie.flari@fera.gsi.gov.uk
QUESTIONS
?
JIFSAN / Fera annual meeting, June 2011