EFSA supporting publication 2015:EN-663
EXTERNAL SCIENTIFIC REPORT
Uncertainty assessment using the NUSAP approach: a case study on the
EFoNAO tool1
Martijn Bouwknegta and Arie H. Havelaara,b,2
a
Centre for Infectious Disease Control, National Institute for Public Health and the Environment, Bilthoven, the
Netherlands; b Institute for Risk Assessment Sciences, Department of Veterinary Medicine, Utrecht University,
Utrecht, the Netherlands
ABSTRACT
Many of the current policy issues that have to be addressed are suffering from large uncertainties, while decision
stakes are high. Addressing the uncertainties explicitly in risk assessments becomes especially important when
addressing such issues. This report describes the evaluation of an uncertainty typology combined with a NUSAP
(acronym for Numeral, Unit, Spread, Pedigree and Assessment) approach. The uncertainty typology is used for
uncertainty characterization based on six dimensions, aiding the choice for subsequent approaches to dealing
with the uncertainties in e.g. decision making. One such approach is the NUSAP method, which is used for
prioritizing uncertainty sources based on the scientific strength and influence on results as judged by experts.
The case study selected was EFSA’s Food of Non-Animal origin risk ranking model, which aims to identify and
rank pathogen and food combinations based on public health concern. Sixteen uncertainty sources were
identified and characterized with the uncertainty typology. The 16 uncertainty sources were implicit or explicit
assumptions and assessed as such for scientific strength and influence on results during a NUSAP workshop.
Criteria for assessing the scientific strength included: influence of situational limitation, assumption plausibility,
choice space and peer agreement. Overall, the combination of uncertainty typology and NUSAP was found to be
very helpful as method for uncertainty assessment, as the procedure increases insight in uncertainty sources
related to model outcomes and in their impact on the end result. The attention to uncertainty was recommended
to be integrated in risk assessments, as it aids to formalise discussions about uncertainties. It was also
recommended that practicality and feasibility aspects should be considered when incorporating uncertainty
assessment in such assessments, that a clear and consistent terminology should be used and that training of
experts is required to integrate elements of uncertainty assessment in future risk assessment activities.
© National Institute for Public Health and the Environment, 2015
KEY WORDS
uncertainty, qualitative, quantitative, typology, NUSAP, expert opinion
DISCLAIMER
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried
out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s),
awarded following a tender procedure. The present document is published complying with the transparency principle to
which the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety
Authority reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present
document, without prejudice to the rights of the authors.
1
2
Question EFSA-Q-2014-00226.
Acknowledgement: The authors would like to acknowledge: the important contributions made by Roland Lindqvist in
identifying and characterizing the uncertainties and assumptions in the EFoNAO model; the participants in the workshop
for their inputs and included (in alphabetical order) Paul Cook, Maria Teresa Da Silva Felicio, Pablo Fernandez, John
Griffin, Kostas Koutsoumanis, Roland Lindqvist, Jim McLauchlin, Birgit Nørrung, Giuseppe Ru and Moez Sanaa; Jeroen
van der Sluijs for his support in developing the workshop protocols and data interpretation; and the EFSA staff for all
support provided to this external scientific report: Pablo Romero Barrios and Winy Messens.
Any enquiries related to this output should be addressed to biohaz@efsa.europa.eu
Suggested citation: Bouwknegt M and Havelaar AH, 2015. Uncertainty assessment using the NUSAP approach: a case study
on the EFoNAO tool. EFSA supporting publication 2015:EN-663, 20 pp.
Available online: www.efsa.europa.eu/publications
© European Food Safety Authority, 2015
Uncertainty assessment using the NUSAP approach
SUMMARY
The National Institute for Public Health and the Environment was awarded this project entitled
‘Uncertainty assessment using the NUSAP approach: a case study on the EFoNAO risk ranking tool’.
In a mandate of the EFSA BIOHAZ Panel for a scientific opinion on the development of a risk ranking
toolbox for EFSA BIOHAZ Panel (M-2013-0007), one of the objectives is to investigate
methodologies for introducing uncertainty and variability in the risk ranking models. To fulfil that
objective it was decided to identify and assess the uncertainties associated to one of the risk ranking
tools (that is the EFoNAO-RRT tool) using the NUSAP (acronym for Numeral, Unit, Spread,
Assessment, and Pedigree) methodology as case study.
Uncertainties related to the model were identified and listed as (explicit and implicit) assumptions, the
list of which was subsequently finalised by discussions with experts from the working group Risk
Ranking Tools. The sources of uncertainties (i.e., assumptions) were subsequently characterized with
an uncertainty typology using the following dimensions: location, nature, range, recognized ignorance,
methodological unreliability and value diversity. Applying the uncertainty typology helps to select
subsequent methods for dealing with the uncertainty sources. One such approach is NUSAP, which
allows for assessing the effect of uncertainty sources on the model outcome for both quantitative and
qualitative uncertainty sources.
NUSAP is an approach that simultaneously addresses quantitative and qualitative aspects of
uncertainty. The approach identifies the uncertainty sources that are most important for the total
uncertainty of a model outcome. NUSAP involves the assessment of the scientific strength of the
uncertainties based on multiple criteria and the impact of these uncertainties on the model outcome.
The criteria that were used for scientific strength in the current study included: influence of situational
limitations, plausibility, choice space and peer agreement. Each criterion was scored by 10 experts on
a discrete ordinal scale ranging from 0 to 4 and the median score per expert was taken as score for
scientific strength. The influence of assumptions on the model outcome was scored on a discrete
ordinal scale ranging from 0 to 3. The model outcomes under consideration were: 1) the identification
of important microbial hazards related to foods of non-animal origin, and 2) the ranking of these
hazards.
These uncertainty identification and characterization based on the typology yielded 16 general
assumptions relating to the EFoNAO-RRT (these assumptions are not described in detail in this report,
given the objective of the workshop to evaluate the use of the typology and NUSAP for EFSA rather
than a critical evaluation of the EFoNAO-RRT). Seven of the 16 uncertainty sources were scored
during the four-hour scoring part of the workshop. None of the assumptions were scored as
scientifically weak and largely influential on the model outcome, yet two uncertainty sources scored
notably lowest on scientific strength among those evaluated. All but one assumption were considered
to have no or negligible impact on the hazard identification. In contrast, 4 out of the 7 assumptions
were judged to have a moderate influence on the hazard ranking (median score 2) while the scientific
strength of these assumptions varied between 1.5 and 2.5. These assumptions are therefore identified
as the most important ones for further work among the seven assumptions evaluated.
The combination of uncertainty typology and NUSAP was found to be very helpful to gain insight in
the uncertainty sources related to model outcomes and to assess their impact on the end results.
Furthermore, the procedure aided in communicating models to non-modellers. The interpretation of
pedigree criteria to assess the strength and effect was found to be difficult by participants, partly due to
the unfamiliar terminology used. It was therefore recommended to use a clear terminology, which is
EFSA supporting publication 2015:EN-663
2
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
understood by all involved in the assessment and that preferably a short training session for experts is
conducted prior to the use of the evaluated approaches to increase the familiarity.
In conclusion, the combination of uncertainty typology and NUSAP was found to be very helpful by
all participants of the workshop. The procedure helped to systematically identify and evaluate the
uncertainty sources related to model outcomes and to assess their impact on the end results. A
framework encompassing uncertainty typology and evaluation (e.g. by NUSAP) was recommended to
be part of each risk assessment process to formalise discussions on uncertainties. Correct and efficient
implementation would require the use of clear terminology and training of experts to become
familiarized with the approaches.
EFSA supporting publication 2015:EN-663
3
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
TABLE OF CONTENTS
Abstract .................................................................................................................................................... 1
Summary .................................................................................................................................................. 2
Background as provided by EFSA ........................................................................................................... 5
Terms of reference as provided by EFSA ................................................................................................ 5
1. Introduction and Objectives............................................................................................................. 6
2. Materials and Methods .................................................................................................................... 6
2.1.
Uncertainty typology .............................................................................................................. 6
2.1.1. Location .............................................................................................................................. 7
2.1.2. Nature ................................................................................................................................. 8
2.1.3. Range .................................................................................................................................. 8
2.1.4. Recognized ignorance......................................................................................................... 8
2.1.5. Methodological unreliability .............................................................................................. 8
2.1.6. Value diversity.................................................................................................................... 8
2.2.
NUSAP ................................................................................................................................... 9
2.3.
NUSAP case study: the EFoNAO risk ranking tool ............................................................... 9
3. Results ........................................................................................................................................... 11
3.1.
Uncertainty typology ............................................................................................................ 11
3.2.
NUSAP workshop ................................................................................................................. 15
3.3.
Evaluation of the methodology ............................................................................................. 15
4. Discussion and Conclusions .......................................................................................................... 17
Recommendations .................................................................................................................................. 18
References .............................................................................................................................................. 19
Abbreviations ......................................................................................................................................... 20
EFSA supporting publication 2015:EN-663
4
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
BACKGROUND AS PROVIDED BY EFSA
In the context of an ongoing self-tasking mandate from the Panel of Biological Hazards (BIOHAZ
Panel) for a scientific opinion on the development of a risk ranking toolbox for EFSA BIOHAZ Panel
(M-2013-0007), and more specifically under ToR 2 (to investigate methodologies for introducing
uncertainty and variability in the risk ranking models), a workshop on the uncertainties in the EFSA
food of non-animal origin risk ranking tool (EFoNAO-RRT) is planned at EFSA premises for April
2014. This model supported the scientific opinion on the risks posed by pathogens in foods of nonanimal origin Part 1 (outbreak data analysis and risk ranking of food/pathogen combinations)
(www.efsa.europa.eu/en/efsajournal/doc/3025.pdf).
This workshop will aim at identifying and assessing the uncertainties associated to one of the risk
ranking tools (that is the EFoNAO-RRT tool) currently selected by the Working Group (WG) experts
dealing with the abovementioned mandate, using the NUSAP (acronym for numeral, unit, spread,
assessment and pedigree) methodology. This will serve as an example of how uncertainty and
variability arising from risk ranking models can be assessed. It is foreseen that the outcome of this
workshop will be used in the scientific opinion on the development of a risk ranking toolbox for EFSA
BIOHAZ Panel being developed. In order to be able to best capture the information gathered during
the workshop, we would like to seek the services of the workshop trainer to compile an external
scientific report analysing and summarising the findings of the workshop.
TERMS OF REFERENCE AS PROVIDED BY EFSA
To provide EFSA with an external scientific report analysing and summarising the findings of the
planned workshop on the uncertainties in the EFoNAO-RRT. This report will be used by the WG on
the development of a risk ranking toolbox as an input to answer one of the TORs of the mandate (to
investigate methodologies for introducing uncertainty and variability in the risk ranking models )
mentioned above. The report should cover at least the following areas:
−
−
−
−
Introduction
Approach/Methodology: describing the format of the workshop and the NUSAP methodology
Results: this should include the outcome of the assessment of the uncertainties related to the
different assumptions and parameters used in the EFoNAO-RRT
Conclusions and recommendations: these should focus not just on the outcome of the
EFoNAO-RRT assessment, but more generally about how this methodology can be applied to
risk ranking models or tools.
This contract was awarded by EFSA to:
Contractor: National Institute for Public Health and the Environment, Bilthoven, the Netherlands
Contract title: External scientific report analysing and summarising the findings of the NUSAP
workshop
Contract number: NP/EFSA/BIOCONTAM/2014/01
EFSA supporting publication 2015:EN-663
5
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
1.
INTRODUCTION AND OBJECTIVES
Many of the current policy issues that have to be addressed are suffering from large uncertainties,
while decision stakes are high. This means that assessments for policy support need to be adapted to
aid the management of such issues, requiring a different approach than the ‘normal’ fundamental and
applied scientific approaches that have proved useful in the past. Often driven by curiosity or restricted
problems, uncertainty in fundamental and applied study outcomes is considered provisional and
reducible by further research. The current societal problems are however more complex and cannot be
solved solely by these scientific approaches. A new approach, termed post-normal science (Funtowicz
and Ravetz, 1993), has been proposed, in which uncertainty is viewed as an intrinsic part of complex
systems that cannot always be quantified. One has therefore to deal openly with deeper dimensions of
uncertainty, including problem framing indeterminacy, ignorance, assumptions and value loadings.
Scientist, policy makers and stakeholders together need to find solutions with explicit discussions on
the imperfections and uncertainties. This means amongst others that uncertainty analysis is an explicit
part of the outcome. The focus therein extends beyond the classical interpretation of uncertainty, with
a central estimate and 95 % confidence interval, to an assessment of the larger part of the total
uncertainty, including also qualitative uncertainty sources (i.e., uncertainties for which the effect on
the model outcome cannot be quantified).
Different approaches exist to address different types of uncertainty, both qualitative and quantitative in
nature (van der Sluijs et al., 2004). These methods vary from quantitative uncertainty analyses using
model parameters and possibly Markov-Chain Monte Carlo (MCMC) methods to assessment of
qualitative uncertainty sources based on expert elicitation. In the Environmental Health Impact
Assessment field, the combination of an uncertainty typology to characterize uncertainty sources and
NUSAP (acronym for numeral, unit, spread, assessment and pedigree; see approach in section 2) has
been applied frequently to deal with large uncertainties. This approach was subsequently shown to be
useful in the field of foodborne diseases as well (Boone et al., 2010; Bouwknegt et al., 2014).
In a mandate on a Risk Ranking Toolbox (the EFSA BIOHAZ Panel), one of the objectives is to
evaluate methods and approaches to evaluate uncertainties associated with the outcomes of panel
assessments (opinions). As part of that work it was decided to evaluate the uncertainty typology and
NUSAP approach in a case study. The case study selected was EFSA’s Food of Non-Animal risk
ranking model (EFoNAO-RRT) used for identifying and ranking pathogen and food combinations of
most public health concern (EFSA BIOHAZ Panel, 2013). The working group identified and
characterised uncertainties (the typology) in that model and the uncertainties were evaluated in a
workshop setting with working group members and panel experts using the NUSAP approach.
Following the evaluation, experts discussed the potential use of this approach by EFSA. This report
describes the evaluation of the combined use of typology and NUSAP approach to evaluate
uncertainty based on this case study.
2.
MATERIALS AND METHODS
2.1.
Uncertainty typology
An uncertainty typology was used to characterize sources of uncertainty identified in the EFoNAORRT model. Uncertainties were identified by reviewing the approach as described in the opinion
(EFSA BIOHAZ Panel, 2013) and listing explicit and implicit assumptions and uncertainties. The list
of uncertainties was then finalised by discussions with experts from the working group on Risk
Ranking Tools and phrased as assumptions. The sources of uncertainties were subsequently
characterized based on the uncertainty typology from Knol et al. (2009). Sources of the uncertainties
are characterized in the following dimensions: location, nature, range, recognized ignorance,
EFSA supporting publication 2015:EN-663
6
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
methodological unreliability and value diversity (Table 1). Each dimension will be explained briefly in
the following paragraphs. Applying the uncertainty typology helps to select subsequent methods for
dealing with the uncertainty sources further. One such approach is NUSAP, which allows for assessing
the effect of uncertainty sources on the model outcome for both quantitative and qualitative
uncertainty sources.
Table 1:
Typology of uncertainty (obtained from Knol et al.(2009))
Uncertainty characterizations
Location: the location at which
the uncertainty manifests itself in
the assessment
Categories
Context: Definitions and boundaries of the system that is being assessed
Model structure: Structure and form of the relationships between the
variables that describe the system
Parameters: Constants in functions that define the relationships between
variables (such as relative risks or severity weights)
Input data: Input data sets (such as concentrations, demographic data,
and incidence data)
Nature: the underlying cause of Epistemic: resulting from incomplete knowledge
the uncertainty
Ontic (process variability): resulting from natural and social variability
in the system
Range: expression of the Statistical (range + chance) : specified probabilities and specified
uncertainty
outcomes
Scenario (range + “what if”) : specified outcomes, but unspecified
probabilities
Recognized ignorance: unknown outcomes, unknown probabilities – uncertainties are present, but no useful
estimate can be given
Methodological unreliability: Methodological quality of all different elements of the assessment; a
qualitative judgment of the assessment process which can based on e.g. its theoretical foundation, empirical
basis, reproducibility and acceptance within the peer community
Value diversity among analysts: Potential value‐ladeness of assumptions which inevitably involve – to some
degree – arbitrary judgments by the analysts.
2.1.1.
Location
The location at which an uncertainty source acts in a study can be characterized as one of three
mutually exclusive categories: contextual uncertainty, model uncertainty or data uncertainty. The latter
category was obtained by combining the parameter values and input data categories as originally
distinguished by Knol et al.(2009).
Contextual uncertainty refers to uncertainty that is related to the chosen study boundaries. For
instance, a choice to focus on a particular pathogen or pathogens due to e.g. data availability, whereas
all pathogens might be of interest from a public health point of view, leads to uncertainty due to an
incomplete focus. This contributes to the total uncertainty associated with the model outcome.
Model uncertainty refers to uncertainty in the modelling part of the study. This can be for instance the
choice for a specific regression technique, the choice for a specific type of dose-response model, and
the assumption for the type of relation (e.g. linear) between variables.
Data uncertainty (a combination of the categories ‘Parameters’ and ‘Input data’ from Table 1) refers to
uncertainty of all data and parameter values used in the modelling.
EFSA supporting publication 2015:EN-663
7
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
2.1.2.
Nature
The ‘nature’ dimension in our typology describes the underlying cause of the uncertainty. Two
categories were identified to describe the nature of an uncertainty source. First of all, uncertainty can
be due to incomplete knowledge, implying that the uncertainty would decrease when more information
would become available. This is called epistemic uncertainty. On the other hand, many processes are
variable in nature (such as the concentration of Campylobacter spp. on leafy greens), and ideally are
accounted for as such. Uncertainties related to variability were characterized as so-called ontic
uncertainties. The nature of an uncertainty source is thus characterized as either ‘epistemic’ or ‘ontic’.
Note that the epistemic and ontic uncertainties are generally interpreted as uncertainty and variability
in quantitative microbiological risk assessment studies.
2.1.3.
Range
The range of a source of uncertainty deals with the description of uncertainties, which can be either
statistically-based, described by a statistical distribution over a continuum, or category-based, using
different possible discrete scenarios. The former is presented for instance as 95% confidence interval
for a parameter estimated with regression models, whereas the latter is presented e.g. as different
results obtained by different models or alternative assumptions. The range of a source of uncertainty is
categorized as either ‘statistical’ or ‘scenario’.
2.1.4.
Recognized ignorance
Recognized ignorance deals with the realization that sources of uncertainty exist but have not been
considered in the modelling. The uncertainty sources are known to exist, but no useful estimate on its
likelihood and effect can be provided. This issue relates for instance to aspects of the system that have
been recognized to exist, but are currently insufficiently understood to provide a useful estimate of its
effect. The presence of recognized ignorance for an uncertainty source was indicated with plusses and
minuses.
2.1.5.
Methodological unreliability
The dimension methodological unreliability categorizes sources of uncertainty based on the way the
information is processed and the level of uncertainty introduced by the methods used. For parameter
values this dimension targets for instance the uncertainty introduced by the estimation approach: data
quality and statistical processing of data. For input data such as Campylobacter spp. concentrations on
chicken fillet, factors introducing uncertainty are for instance imperfect diagnostics tests. Modeling
aspects, such as the deviation between a fitted probability distribution and the original data, might also
occur. In general, methodological uncertainty can be interpreted as uncertainty related to the data or
method used. Methodological uncertainty was scored with plusses and minuses.
2.1.6.
Value diversity
Value diversity is a dimension used to describe the extent to which a source of uncertainty is due to
varying opinions among peers. It mainly deals with subjective choices between alternative models or
datasets. For instance, value diversity is lower when the majority of peers agrees on the procedure to
follow, than when the approach is subject to considerable debate. Also, value diversity might arise for
example when choosing how to deal with a number of non-representative datasets for parameter
estimation or when selecting the study area or cohort. The dimension of value diversity was scored
with plusses and minuses to indicate its presence or absence, respectively.
EFSA supporting publication 2015:EN-663
8
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
2.2.
NUSAP
NUSAP, an approach that simultaneously addresses quantitative and qualitative aspects of uncertainty
is an acronym for Numeral, Unit, Spread, Assessment, and Pedigree (Funtowicz and Ravetz, 1990).
The Numeral, Unit and Spread part refers to a central estimate (i.e., most likely value for a parameter),
its unit (e.g. ‘€’ or ‘per day’), and the spread of the estimate (e.g., a 95 % confidence interval).
NUSAP elaborates on this initial assessment by including expert judgment of the reliability of a
knowledge claim (the ‘Assessment’), and a systematic multi criteria evaluation of its knowledge base
(the ‘Pedigree’).
To identify the uncertainty sources that were most important for the total uncertainty of the model in
the case study of the EFoNAO-RRT model we used the assessment and pedigree component of
NUSAP. This approach involved the assessment of the scientific strength of the uncertainties
associated with the implicit and explicit assumptions of the EFoNAO-RRT model and the impact of
these uncertainties on the outcome of the model was assessed.
The scientific strength of each uncertainty source is scored according to four criteria (see Table 2 for
the criteria used and the scores in this study). The median of the strength scores per expert was firstly
assessed. The final score for scientific strength was subsequently taken as the median score of all
medians per expert. Experts then estimated the influence of the uncertainty on the model results. This
score, combined with the median score of the scientific strength, gives an impression of the
importance of an uncertainty source: sources with low accuracy and large influence on the final results
are the most important.
The model outcomes under consideration were: 1) the identification of important microbial hazards
related to foods of non-animal origin, and 2) the ranking of these hazards.
2.3.
NUSAP case study: the EFoNAO risk ranking tool
At the start of the workshop, the purpose and process of the meeting was explained to the participants.
Next, the scientific strength and influence on result for each assumption was assessed in the cycle
shown in Figure 1. The first step comprised the introduction and explanation of the assumption,
followed by a short period for clarifying questions. A first round of scoring was subsequently
conducted, during which the experts assessed the score per pedigree criteria (see Table 2) and noted
down the arguments for that score. Each expert then revealed the scores and the scoring was discussed
plenary. Especially large difference in scores were discussed to assess whether different views existed
due to e.g. different interpretations and judgments on evidence, or due to fallible arguments. In the last
phase, the experts were given the opportunity to adjust scores when the discussion led to a changed of
insight that altered the score.
The score and rationale for those scores were noted on a scoring card prepared in MS Excel (see
Figure 2 for an exemplary card). In addition to the scoring for different criteria and influence on
results, the rationale for assigning that score to specific criteria is an important output. Experts were
therefore also asked to provide the main motivation for their choice. After the prioritization of the
uncertainty sources, the motivation can be used to develop ideas to dealing with the uncertainty
sources (e.g., reanalysis of data, further data collection,)
The last hour of the workshop was devoted to discussing the usefulness of the approach as described
in this report for EFSA and if considered useful, what possible approaches could be considered for its
implementation.
EFSA supporting publication 2015:EN-663
9
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
Introduction of uncertainty source
Clarifying questions & answers
Scoring using the pedigree matrix
Collection of scores through plenary reporting
Discussion on the arguments for scoring
Possible score adjustment
Figure 1: The steps in the NUSAP approach that were completed in the workshop per uncertainty
source
Table 2: The pedigree matrix used in the NUSAP workshop to assess the scientific strength of each
uncertain assumption and its influence on the results (effect)
Scientific strength
Score
0
Influence of
situational
limitations
Choice assumption
hardly influenced
1
Limited influence
in choice
assumption
2
Choice assumption
moderately
influenced
3
Important
influence in choice
assumption
4
Totally different
assumption had
there not been
limitations
Plausibility
Choice space
Effect
Agreement among
peers
Influence on
results
The assumption is very
plausible (based on
established theory,
verified through peer
review)
Plausible (based on
model with theoretical
basis, empirically
verified data)
The assumption is
acceptable (based on a
simple model,
extrapolated data)
Assumption is doubtful
(based on not verified
empirical data)
Hardly any
alternative
available
A large majority (90100%) among peers
would have made the
same assumption
The assumption has
no or negligible
impact on the
results
Very limited
number of
alternatives
available
Limited choice
from alternative
assumptions
Many experts (75%)
would have made the
same assumption
The assumption has
little impact on the
results
Several experts
(50%) would have
made the same
assumption
Few experts (25%)
would have made the
same assumption
The assumption has
a moderate impact
on the end result
The assumption is
fictive or speculative
Ample choice
from alternative
assumptions
EFSA supporting publication 2015:EN-663
Average number
of alternatives
The assumption has
an important
impact on the end
result
Controversial
assumption, hardly
any expert (1%)
would have made the
same assumption
10
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
Figure 2: Example of score card used in the NUSAP workshop at EFSA
3.
RESULTS
3.1.
Uncertainty typology
A list of all assumption and parameter uncertainties was produced. These sources were grouped to a
general level and rewritten as assumptions, yielding 16 general assumptions relating to the EFoNAORRT (Table 3). The assumptions will not be described in detail in this report, given the objective of
the workshop (i.e., to evaluate the use of NUSAP for EFSA, and not to evaluate the EFoNAO-RRT).
The identification of the uncertainty sources based on the uncertainty typology from Table 1 is shown
in Table 3. The majority of uncertainty sources (11 out of 16) related to the parameter and input data
that was used. Furthermore, 14 of the 16 uncertainties were considered to relate to imperfect
knowledge (‘epistemic’), which could be reduced by further studies.
EFSA supporting publication 2015:EN-663
11
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
Table 3:
Characterisation of the 16 uncertainty sources by using the uncertainty typology of Table 1
Nature
Epistemic
Contextual uncertainty
Link between a pathogen and a type of FoNAO can be deduced from outbreak
data onlya(4)
The added value of considering pathogen inactivation to assess risk levels is
negligible for each food-hazard paira(5)
Contextual and model uncertainty
The risk of a pathogen/food combination can be estimated by a linear,
unweighted combination of scores on seven parameters, each divided in three
or four categories that are represented by arbitrary numbersa(2)
Model uncertainty
The risk of a pathogen/food combination can be estimated by a combination of
top-down and bottom-up approachesa(1)
Assuming a prevalence score of 2 to the category defined as “unknown
prevalence”, implies that the prevalence cannot be assumed to be zero for
Shigella spp., Yersinia spp., Staphylococcus aureus, Norovirus, HAV, and
Cryptosporidium spp.
Parameter and input data uncertainty
The estimated true number of illness by a specific pathogen in the EU, without
consideration of attribution to sources, is a valid indicator of the risk of a
specific pathogen in a specific food of non-animal origin
The prevalence of pathogens in all FoNAO samples is a valid estimate for the
prevalence in the FoNAO group under consideration
The relative degree of underreporting of outbreak cases is the same in the US
and EU and for each food-hazard pair
The incidence of norovirus and bacterial intoxications in the EU is similar to the
Netherlands
a
Range
Ontic
Statistical
X
X
X
X
Recognized
Method
Value
Scenario
ignorance
unreliability
diversity
X
+
+
+
X
+
-
+
X
+
-
+
+
-
+
X
X
X
+
-
+
X
X
+
-
+
X
X
+/-
-
+
X
X
+
+
+
X
X
+
-
+
assumption evaluated in the NUSAP workshop. The number between brackets indicates the assumption number used in Figures 3-6 for reference.
EFSA supporting publication 2015:EN-663
12
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between the European Food
Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document, without prejudice to the
rights of the authors.
Uncertainty assessment using the NUSAP approach
Table 3:
Continued
Nature
Epistemic
Parameter and input data uncertainty
The longest reported shelf life of food in a specific food group is representative
of all products in that group and pathogen growth is not affected by growth of
spoilage organisms
The available consumption data are representative of the whole EUa(12)
Low numbers of Salmonella spp., Shigella spp., pathogenic Escherichia coli (e.g.
STEC) and Yersinia enterocolitica can cause disease without growth during
storage in retail or consumer’s homes
Pathogen-specific DALY estimates published for the Netherlands are
representative for the whole EUa(14)
DALYs per case for Shigella spp. and Yersinia enterocolitica fall within the
same category as Salmonella spp. and are the same for STEC O157 and STEC
non-O157
a
Value
ignorance
unreliability
diversity
X
X
-
-
+
X
+
-
+
+
-
+
X
X
X
-
+
+
X
X
+
-
+
X
-
-
+
X
-
+/-
+
X
X
Method
Statistical
X
Scenario
Recognized
Ontic
X
a(16)
All products will be eaten at the end of their shelf life
With the exceptions of Bacillus cereus and Clostridium perfringens, the overall
prevalence of all pathogens in the different FoNAO groups, is assumed to be
either low (< 1%) or unknown
Range
assumption evaluated in the NUSAP workshop. The number between brackets indicates the assumption number used in Figures 3-6 for reference.
EFSA supporting publication 2015:EN-663
13
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between the European Food
Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document, without prejudice to the
rights of the authors.
Uncertainty assessment using the NUSAP approach
Figure 3: Scientific strength (‘rigor’) of the assumptions identified in the EFoNAO-RRT that yield
uncertainty in the model outcome
Legend: The white diamonds indicate the median score, the error bars the minimum (left) and maximum (right) score, and
the black rectangle indicates the interquartile range. Assumptions with higher scores (in the red zone) have lower scientific
strength compared to lower scores (green zone). Diamonds crossing the y-axis indicate the assumptions that have not been
scored.
Figure 4: Pedigree scores per criterion as provided by the experts during the NUSAP workshop
Legend: The x-axis displays the assumption that was scored according to the pedigree criteria explained in Table 2:
‘influence of the situational limitations’, ‘plausibility’, ‘choice space’, and ‘agreement among peers’. The y-axis shows the
median score for the criteria over all experts. The error bars indicate the interquartile range.
EFSA supporting publication 2015:EN-663
14
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
3.2.
NUSAP workshop
Seven of the 16 uncertainty sources were scored during the four-hour scoring part of the workshop,
needing on average about 35 minutes per assumption. There was intensive discussion on the NUSAPmethodology and on the interpretation of the criteria and the scores. This discussion led to a revision
of the definitions for the ‘influence on results’ categories (the final definition is shown in Table 2).
Figure 3 shows the scoring results for scientific strength for the seven assumptions. All interquartile
ranges covered 1 score class, thereby showing agreement among the experts. The uncertainty sources
with lowest scientific strength (i.e. those with the highest score) among those evaluated were
assumptions 2 and 4. None of the assumptions were scored as scientifically very weak and largely
influential on the model outcome. Notably, the median scores for scientific strength are concentrated
around the midpoint of the scale. Similarly, the scores for each of the pedigree criteria concentrated
around the mid-score (Figure 4). A score of ‘2’ was assigned most frequently (n=100), followed by a
score of ‘1’ (n=88) and ‘3’ (n=73). Scores of ‘0’ and ‘4’ were assigned 17 and six times, respectively.
Eighteen of the 28 scored criteria had an interquartile range ≤1 score unit, again showing agreement
among the experts regarding the pedigree criteria. The pedigree criterion ‘agreement among peers’
was scored consistently best for all assumptions; the criteria ‘influence of situational limitations’,
‘plausibility’ and ‘choice space’ scored in general lower than ‘agreement among peers’ and showed
larger variation.
Figures 5 and 6 show the strength and effect diagrams for the two model outcomes considered
(identification of hazards and ranking of these hazards, respectively). All but one assumption (nr. 4)
were considered to have no or negligible impact on the hazard identification. In contrast, 4 out of the 7
assumptions were judged to have a moderate influence on the hazard ranking (median score 2) while
the scientific strength of these assumptions varied between 1.5 and 2.5. These assumptions are
therefore identified as the most important ones for further work among the seven assumptions
evaluated.
3.3.
Evaluation of the methodology
NUSAP was found to be very helpful by all participants in the workshop. The procedure helps to gain
insight in the uncertainty sources related to model outcomes and to assess their impact on the end
results. Furthermore, the procedure aids in communicating models to non-modellers. The attention to
uncertainty was recommended to be a part of risk assessment, as it would help to formalise the
discussion about the uncertainties. By doing this structurally and integrated with the risk assessment
activities, experience grows and the process would eventually save time. Nevertheless, it was
recommended that practicality and feasibility aspects should always be considered when incorporating
uncertainty assessment in the risk assessment process.
The interpretation of pedigree criteria to assess the strength and effect (see Table 2) was found to be
difficult by participants. Part of the difficulty is caused by the difference in terminology between
scientists working in philosophical sciences, who developed the NUSAP methodology, and natural
sciences. It was recommended that a clear terminology is to be used, which is understood by all
involved in the assessment. Preferably, a short training session with dummy uncertainty sources
should be conducted prior to the NUSAP workshop. This training may be an on-site training given at
EFSAs premises, but could also be part of a web-based training program. All experts would be
required to follow this online training prior to their involvement in risk assessment activities.
Another cause for difficulty with the interpretation of the pedigree criteria related to the issue of
scoring the pedigree criteria independently from each other or conditionally. For instance, whether or
EFSA supporting publication 2015:EN-663
15
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
not ‘peer agreement’ for an assumption should be assessed conditional on e.g. a large influence of
situational limitations, had the latter been scored. Both approaches are valid, depending on the aims of
the assessment. When aiming for a general critical review as part of the uncertainty assessment, then
the criteria need to be assessed independently. The ‘influence of situational limitations’, for instance,
deals then with external factors influencing the scientific quality, whereas the ‘peer agreement’ is a
proxy for the proper interpretation and use of scientific knowledge by the analyst.
Much time during the workshop was devoted to discussions on how to describe the sources of
uncertainty. Ideally, consensus on the phrasing/wording is obtained before the scoring starts. This
should be an iterative process that involves both the principal analyst(s) of the study to be evaluated
and the experts that will participate in the NUSAP workshop.
No. 4
Nos. 12 & 16
No. 1
No. 2
Nos. 5 & 14
Legend: The x-axis displays the median scientific strength (‘rigor’, i.e., the white diamonds from Figure 2), the y-axis the
median score for the influence on results. Values with a high score on influence on results and scientific strength (the red
zone) are critical assumptions in the model.
Figure 5: Strength and effect diagram for the seven assumptions of the EFoNAO-RRT that were
scored during the workshop for the influence on hazard identification
EFSA supporting publication 2015:EN-663
16
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
No. 5
No. 12
No. 1
No. 14
No. 2
No. 16
No. 4
Legend: The x-axis displays the median scientific strength (‘rigor’, i.e., the white diamonds from Figure 2), the y-axis the
median score for the influence on results. Values with a high score on influence on results and scientific strength (the red
zone) are critical assumptions in the model.
Figure 6: Strength and effect diagram for the seven assumptions of the EFoNAO-RRT that were
scored during the workshop for the influence on hazard ranking.
4.
DISCUSSION AND CONCLUSIONS
The interpretation of pedigree criteria to assess the strength and effect (see Table 2) was found to be
difficult by participants. Part of the difficulty is caused by the difference in terminology between
scientists working in philosophical sciences, who developed the NUSAP methodology, and natural
sciences, who took part in the assessment workshop. It was recommended that a clear terminology is
developed, which is understood by all involved in the assessment. Preferably, a short training session
with dummy uncertainty sources should be conducted prior to the NUSAP workshop. This training
may be an on-site training given at EFSAs premises, but could also be part of a web-based training
program. All experts would be encouraged to follow this online training prior to their involvement in
scoring the assumptions.
Much time during the workshop was devoted to discussions on how to describe the sources of
uncertainty. Ideally, consensus on the phrasing/wording should be obtained before the scoring starts.
This should be an iterative process that involves both the principal analyst(s) of the study to be
evaluated and the experts that will participate in the NUSAP workshop.
EFSA supporting publication 2015:EN-663
17
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
The scoring by experts showed a tendency to the central value of ‘2’ for the scientific strength criteria.
At present we cannot provide the definitive cause for this tendency, but possible explanations include
the following. Firstly, the seven considered assumptions might all have a similar, intermediate level of
scientific strength according to the experts. Secondly, lack of experience of the experts might prevent
extreme choices to be made. Thirdly, we created a series of general assumptions based on the
EFoNAO-RRT model. The generalization might consider coincidentally several specific sources of
uncertainties that possibly counteract. Experts need to produce a judgment on the net strength,
possibly leading to intermediate scores. The central tendency would then be a characteristic of the
approach. An alternative approach to the NUSAP workshop is to prepare a longlist of specific
uncertainty sources and to prioritize this list by experts prior to the workshop to generate a shortlist.
The shortlist with seemingly important specific uncertainty sources can subsequently be considered in
the NUSAP workshop. Such an approach requires more time for preparation and more effort from
experts, but provides more divergent scores that aids in the distinction of critical and non-critical
uncertainty sources.
The aggregation of scores by all experts on all four pedigree criteria into a single median (and
interquartile range) that is subsequently used to judge the scientific strength of the assumptions was
considered to result in loss of information. The criteria considered in the scoring of scientific strength
are different in nature and addressing potential issues may require different strategies. A more detailed
summary description of the results of this analysis, and possibly the development of a multi criteria
analysis within the NUSAP approach was proposed in order to extract and use more of the information
obtained during scoring of all criteria.
In conclusion, the combination of uncertainty typology and NUSAP was found to be very helpful by
all participants of the workshop. The procedure helped to systematically identify and evaluate the
uncertainty sources related to model outcomes and to assess their impact on the end results.
Furthermore, the combination of uncertainty identification and characterization aided in the
understanding of the model by non-modellers.
RECOMMENDATIONS
A framework encompassing uncertainty typology and evaluation (e.g. by NUSAP) was recommended
to be part of each risk assessment to formalise discussions on uncertainties.
A clear and consistent terminology understood by all involved in the assessment should be used.
When using the uncertainty typology and NUSAP for the first time, training is recommended.
Consensus on the phrasing of assumptions and uncertainties should be reached among the experts and
principal analysts prior to the NUSAP scoring session in an interactive process.
Uncertainty sources should be described as specific as possible to allow for better consideration of the
pedigree criteria when scoring.
Nonetheless, it was also considered that practicality and feasibility aspects should always be
considered when incorporating uncertainty assessment in the risk assessment.
EFSA supporting publication 2015:EN-663
18
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
REFERENCES
Boone I, Van der Stede Y, Dewulf J, Messens W, Aerts M, Daube G and Mintiens K, 2010. NUSAP: a
method to evaluate the quality of assumptions in quantitative microbial risk assessment. Journal of
Risk Research, 13, 1-16.
Bouwknegt M, Knol AB, van der Sluijs JP and Evers EG, 2014. Uncertainty of population risk
estimates for pathogens based on QMRA or epidemiology: a case study of Campylobacter in the
Netherlands. Risk Analysis, 34, 847-864.
EFSA BIOHAZ Panel (EFSA Panel on Biological Hazards), 2013. Scientific Opinion on the risk
posed by pathogens in food of non-animal origin. Part 1 (outbreak data analysis and risk ranking of
food/pathogen
combinations).
EFSA
Journal
2013;11(1):3025,
138
pp.,
doi:10.2903/j.efsa.2013.3025
Funtowicz SO and Ravetz JR, 1990. Uncertainty and quality in science for policy. Kluwer Academic
Publishers, Dordrecht, the Netherlands, 229 pp.
Funtowicz SO and Ravetz JR, 1993. Science for the post-normal age. Futures, 25, 739-755.
Knol AB, Petersen AC, van der Sluijs JP and Lebret E, 2009. Dealing with uncertainties in
environmental burden of disease assessment. Environmental Health, 8, 21.
van der Sluijs JP, Janssen PHM, Petersen AC, Kloprogge P, Risbey JS, Tuinstra W and Ravetz JR,
2004. RIVM/MNP Guidance for Uncertainty Assessment and Communication: Tool Catalogue for
Uncertainty Assessment. NWS-E-2004-37. Available at: http://www.usf.uni-kassel.de/
ftp/user/mimler/uncertainty/vol_4_uncertaintyguidancetoolcat.pdf.
EFSA supporting publication 2015:EN-663
19
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.
Uncertainty assessment using the NUSAP approach
ABBREVIATIONS
DALY
FDA
EFoNAO-RR
FoNAO
HI
MCMC
MS
NUSAP
RA
RRT
RTE
STEC
WG
Disability adjusted life year
Food and Drug Administration
EFSA’s food of non-animal origin risk ranking tool
Food of non-animal origin
Health impact
Markov-chain Monte Carlo
Member states
Numeral, Unit, Spread, Assessment and Pedigree
Risk assessment
Risk ranking tool
Ready-to-eat
Shiga-toxin producing Escherichia coli
Working Group
EFSA supporting publication 2015:EN-663
20
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively
by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender
procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be
considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the
issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors.