APPROVED: 1 April 2022
doi:10.2903/sp.efsa.2022.EN-7339
Roadmap for actions on artificial intelligence for evidence
management in risk assessment
Final Report
Bersani C., Codagnone J., David L., Foiniotis A., Galasso G., Mancini S., Michieletti R.,
Orphanidou C., Pellegrino M.
PwC EU Services & Intellera Consulting Srl
Abstract
In May 2021, the European Food Safety Authority (EFSA) launched the “Roadmap for actions on Artificial
Intelligence for evidence management in risk assessment” project to develop an approach for the
implementation of Artificial Intelligence (AI) methods in the evidence management phase of its internal
risk assessment process. The main objective is to identify specific projects to be carried out to increase
– by 2027 - the accessibility and breadth of the body of evidence and enhance the trustworthiness of
the risk assessment process by applying human-centric AI in close coexistence with human expertise.
The roadmap, therefore, presents a full understanding of ongoing activities, market readiness,
knowledge gaps, and societal interests and concerns, as well as collaboration opportunities in the field
of AI application to the evidence management process. Subsequently, based on these findings, it
outlines a set of recommendations aimed at developing the Agency's capabilities to adopt and integrate
AI solutions in the evidence management process. Finally, the roadmap contains considerations aimed
at identifying potential communication and engagement opportunities in the area of AI applied to
evidence management in risk assessment, considering specific needs and concerns of EFSA's
stakeholders on the adoption of AI.
© European Food Safety Authority, 2022
Key words: Evidence Management, Risk Assessment, Artificial Intelligence (AI), Natural Language
Processing (NLP), Machine Learning (ML), EFSA
Question number: EFSA-Q-2022-00230
Correspondence: SPIDO@efsa.europa.eu
www.efsa.europa.eu/publications
EFSA Supporting publication 2022: EN-7339
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
EXTERNAL SCIENTIFIC REPORT
Disclaimer: This document has been produced and adopted by the bodies identified above as
author(s). This task has been carried out exclusively by the author(s) in the context of a contract
between the European Food Safety Authority and the author(s), awarded following a tender procedure.
This document is published complying with the transparency principle to which the Authority is subject.
It may not be considered as an output adopted by the Authority. The European Food Safety Authority
reserves its rights, view, and position as regards the issues addressed and the conclusions reached in
this document, without prejudice to the rights of the authors.
Acknowledgements: We would like to pay our special regards to Konstantinos Paraskevopoulos,
Ermanno Cavalli, Angelo Cafaro and the other members of the EFSA SPIDO team for their guidance and
support along the entire project, and to all EFSA staff members who provided insights and feedback to
help us raise the quality of our work. We are also grateful to the external stakeholders who provided us
with information during the data collection activities.
Amendment: This version was revised by adding information on the Terms of Reference of the
contract, as requested by the EFSA template for external scientific reports (page 5). This editorial
correction does not materially affect the contents or outcome of this scientific output. To avoid
confusion, the original version of the output has been removed from the EFSA Journal but is available
on request.
Suggested citation: PwC EU Services & Intellera Consulting, 2022. Roadmap for action on Artificial
Intelligence for evidence management in risk assessment. EFSA supporting publication 2022: EN-7339.
120 pp. doi:10.2903/sp.efsa.2022.EN-7339
ISSN: 2397-8325
© European Food Safety Authority, 2022
Reproduction is authorised provided the source is acknowledged.
www.efsa.europa.eu/publications
2
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried
out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s),
awarded following a tender procedure. The present document is published complying with the transparency principle to which
the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority
reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document,
without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Executive Summary
The European Food Safety Authority (EFSA) is the Agency of the European Union that provides independent scientific
advice and communicates on existing and emerging risks associated with the food chain. Evidence management is
at the core of the evidence-based risk assessment: it is a structured process of collecting and analysing data
generated through different types of studies and approaches, which are gathered by various sources. Evidence
management plays a crucial role as it contributes to the development of EFSA's scientific opinions. However, it is a
resource-intensive activity, where the amount of information processed is inherently limited by the accessibility to
the body of evidence and to available resources.
Given the relevance of the evidence management process within the organisation and the specific challenges faced
in terms of efficiency, quality and transparency, EFSA has recently presented the Theme (concept) paper on Artificial
Intelligence in risk assessment (2021) 1 with its strategic vision to build a harmonised approach on the implementation
of artificial intelligence (AI) methods in the evidence management phase of the risk assessment process. In this
theme paper, EFSA stated the objectives to be achieved by 2027: increasing the accessibility and breadth of the
body of evidence and improving the trustworthiness in the risk assessment process, while applying human-centric
AI in close coexistence with human expertise.
In order to reach these objectives, EFSA launched the project Roadmap for actions on artificial intelligence for
evidence management in risk assessment (specific contract No 35 – Rc 5 implementing Framework Contract No
OC/EFSA/FIN/2019/01), awarded to PwC EU Services EESV and the subcontractor Intellera Consulting Srl. The
purpose of the project is to deliver a roadmap for action, with recommendations for future multi-annual and multipartner studies or projects in the area of AI. This roadmap builds on EFSA´s vision and supports its preparedness
for future risk assessments, while simultaneously preventing possible discrepancies on sensitive matters.
In order to guide the activities for the roadmap development, the evidence management process was analysed and
split up into subsequent phases, which were defined as use cases on which to focus the following activities. Then,
to develop a complete and accurate roadmap, an analysis of the current market for AI solutions was conducted
through various data collection activities – including desk research, literature review, a survey, and several semistructured interviews. These preliminary findings highlighted the potential for applying AI to specific evidence
management use cases, given the availability of a relevant number of AI-based solutions in the market. Even though
the technology readiness level of these solutions varies, the majority of them are currently in a commercialised stage,
while others should be further developed and adapted to the agency's specific needs, and would also require an
upgrade of the agency's data capabilities. Most of the identified solutions were developed by small/medium software
companies, with a few solutions by academic organisations, such as universities and research centres. The AI-based
solutions were then categorised according to the previously identified use cases of the evidence management. The
categorisation made clear that the highest number of available AI tools can be integrated for the automation of the
primary source for data collection of evidence management, which is the literature review.
In order to explore the specific challenges faced by organisations similar to EFSA and draw upon best-practices with
regard to AI development, the analysis also focused on the strategies, roadmaps and projects launched by
institutional organisations. The data showed that only a few public organisations have applied AI within their evidence
management process. Nevertheless, some relevant institutions have demonstrated their interest in investing in this
area, leaving EFSA some ground for future collaborations. In general, the analysis showed that several opportunities
to collaborate with different stakeholders might be available in order to develop AI-related projects.
After obtaining a clear picture of the current state of the art, the next step was to identify the potential challenges
and blockers that public organisations face when applying AI-based solutions. What emerged from this analysis is
that the organisation should adopt a holistic organisational transformation, including for example the re-design of
specific process and workflows, to successfully adopt AI. In addition, a gap analysis conducted on EFSA’s data
1
European Food Safety Authority, Theme (concept) paper – on Artificial intelligence in risk assessment (2020) (unpublished
information)
www.efsa.europa.eu/publications
3
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
maturity assessment highlighted potential challenges related to both EFSA’s internal capabilities and technical
requirements.
The aforementioned activities led to the identification of a comprehensive set of potential collaboration opportunities
during the AI lifecycle – which involves various stages of the AI development and integration – with different
stakeholder categories from the industrial, academic and institutional arenas.
Finally, in order to specifically advise EFSA on the implementation of AI solutions in the evidence management
process, a set of horizontal and vertical recommendations has been provided.
On one hand, “horizontal recommendations” were developed to guide EFSA in building the technical, procedural and
governance capabilities to improve its preparedness towards the adoption of AI. These were focused on the need to
develop a sound data governance and infrastructure framework, together with Development and Operations
Management (DevOps), in order to increase the agency's ability to manage, control and deliver AI applications and
services at a high speed. In addition, one of the suggested recommendations is to adopt a Trustworthy AI framework
– including the development of a Code of Conduct and a Risk Framework - to ensure that EFSA is able to address
the ethical, social and legal challenges surrounding AI, in light of the potential impact of the AI Regulation proposed
by the Commission. Furthermore, domain-specific ontologies need to be developed to support the integration of
precise AI solutions for the automation and improvement of the evidence management process. Finally, the
acquisition of specific technical competences for the development and maintenance of AI-based systems is highly
recommended.
On the other hand, “vertical recommendations” were provided to link each of the prioritised use cases within EFSA’s
evidence management process with the optimal approach for procuring or developing AI solutions with key
stakeholders.
Furthermore, to specifically support EFSA in its decision-making and resource allocation processes, these
recommendations (or projects) have been first evaluated in terms of budget and duration, and then prioritised taking
into account their urgency and inter-dependencies. Based on this activity, projects have been distributed on a
timeline, from 2022 to 2027.
Finally, a communication and engagement plan has been developed to inform the Agency's stakeholders and its
surrounding ecosystem, strengthen current partnerships, and foster new collaboration opportunities.
Overall, this roadmap serves as a decision-making tool to identify and prioritise AI projects aimed to optimise EFSA’s
evidence management process through the introduction and use of AI. By implementing the
recommendations/projects devised in this roadmap, EFSA will be able to develop the fundamental competences that
enable the effective integration of AI within the organisation, both in terms of technical requirements to manage,
control, and use AI responsibly, and in terms of new skills and cultural change. These projects are particularly
relevant for building the Agency’s capabilities to integrate AI tools within evidence management and other processes,
thus promoting synergies with other roadmaps (e.g., the EU Partnership for next generation, systems-based
Environmental Risk Assessment - PERA - and the New Approach Methodologies - NAMs).
www.efsa.europa.eu/publications
4
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Table of Contents
Abstract ........................................................................................................................................ 1
Executive Summary ....................................................................................................................... 3
International Consumer Protection and Enforcement Network .......................................................... 9
1.
Introduction ..................................................................................................................... 11
2.
Background ...................................................................................................................... 11
2.1.
Policy Context .................................................................................................................. 12
2.2.
EFSA's specific objectives .................................................................................................. 13
2.3.
High level project phases .................................................................................................. 14
2.4.
Problem formulation ......................................................................................................... 15
3.
State of the art and analysis.............................................................................................. 20
3.1.
Market overview, map of relevant activities and further areas ............................................. 21
3.2.
Challenges and blockers related to the adoption of AI in evidence management ................... 29
3.3.
Cooperation/collaboration opportunities ............................................................................. 34
4.
Recommendations for projects .......................................................................................... 49
4.1.
Horizontal Recommendations ............................................................................................ 50
4.2.
Vertical Recommendations ................................................................................................ 75
4.3.
Prioritisation of Working Areas .......................................................................................... 89
5.
Communication and Engagement Plans.............................................................................. 96
5.1.
General Approach ............................................................................................................. 96
5.2.
Inform & Monitor/Listen Matrix Quadrant ......................................................................... 100
5.3.
Communication Plan ....................................................................................................... 101
5.4.
Engagement Plan ........................................................................................................... 107
5.5.
Final Considerations on Communication and Engagement – Timeline and dependencies ..... 117
6.
Conclusions .................................................................................................................... 118
www.efsa.europa.eu/publications
5
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
List of Figures
Figure 1 – EU’s approach to AI ........................................................................................................................ 12
Figure 2 - Main activities of the project ............................................................................................................ 15
Figure 3 - Evidence management process phases ............................................................................................. 16
Figure 4 – Evidence management process phases and use cases ....................................................................... 18
Figure 5 – MCDA criteria for use case prioritisation ........................................................................................... 19
Figure 6 - Revised prioritisation of the use cases............................................................................................... 20
Figure 7 - Data collection activities ................................................................................................................... 23
Figure 8 - Use case relevance .......................................................................................................................... 23
Figure 9 - Number of solutions per use case and readiness level ........................................................................ 24
Figure 10 - Summary of key insights ................................................................................................................ 24
Figure 11 - General Challenges ........................................................................................................................ 30
Figure 12 - Gap Analysis of the EFSA ecosystem ............................................................................................... 31
Figure 13 - AI lifecycle phases ......................................................................................................................... 34
Figure 14 - State of Play of EFSA AI ecosystem of stakeholders ......................................................................... 36
Figure 15 - AI lifecycle and collaboration models .............................................................................................. 39
Figure 16 - Vendor selection and procurement management process ................................................................. 42
Figure 17 - Prioritisation matrix of institutional stakeholders ............................................................................. 45
Figure 18 - Vertical and Horizontal recommendations link .................................................................................. 49
Figure 19 - Link of Recommendations with AI Lifecycle Phases .......................................................................... 50
Figure 20 - Dimensions and horizontal recommendations .................................................................................. 50
Figure 21 - Indicative Data Infrastructure for EFSA ........................................................................................... 55
Figure 22 - Indicative Data Infrastructure for EFSA based on the Azure cloud platform. ...................................... 56
Figure 23- Questions that a DG framework answers .......................................................................................... 59
Figure 24 - Indicative DG Framework incorporating Tools & Technology, Process & Operating Model and People &
Culture ........................................................................................................................................................... 59
Figure 25 - DG roadmap with increasing levels of maturity ................................................................................ 60
Figure 26 - DevOps Benefits ............................................................................................................................ 62
Figure 27 - ALTAI example of questions which refer to the human agency and oversight principles ..................... 69
Figure 28 - ETAPAS Indicator framework design and structure........................................................................... 70
Figure 29 - Steps for Data Science Competences Development .......................................................................... 73
Figure 30 - Six principles of People-centred Change and Adoption ..................................................................... 73
Figure 31 - Six pillars of change management approach .................................................................................... 74
Figure 32 - General Approach for Communication and Engagement ................................................................... 97
Figure
33
Communication/engagement
strategies
for
the
identified
stakeholder
segments
(Impact/Relevance/Easiness to collaborate matrix) ........................................................................................... 99
Figure 34 - Communication/Engagement strategies for the identified stakeholders segments ............................ 100
Figure 35 - Inform & Monitor/Listen Quadrant ................................................................................................ 101
Figure 36 - Ad-hoc Communication Quadrant ................................................................................................. 101
Figure 37 - Maintain & Reinforce Collaboration and Engage in the ecosystem ................................................... 108
List of Tables
Table
Table
Table
Table
Table
Table
1 - Risk Assessment Workflow and Evidence Management Process ........................................................... 16
2 - Top 10 prioritised use-cases .............................................................................................................. 20
3 - Extensive list of initiatives ................................................................................................................. 21
4 - Brief description of the development environments identified in the data collection .............................. 25
5 - Solutions for additional use cases within evidence management .......................................................... 26
6 - Additional features of tools ................................................................................................................ 28
www.efsa.europa.eu/publications
6
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
7 - Gaps and Recommendations per each dimension ............................................................................... 33
8 - Comprehensive overview of cooperation models ................................................................................. 36
9 - Development Levers ......................................................................................................................... 38
10 - Institutional organisation prioritisation ............................................................................................. 43
11 - Collaboration Opportunities per Prioritised Use Case ......................................................................... 45
12 - Mapping between AI Act requirements and recommendations and actions for EFSA ............................ 64
13 - Map of the HLEG’s Requirement with short description ...................................................................... 66
14 - List of ETAPAS principles with short description ................................................................................ 67
15 - Example of organisational governance ............................................................................................. 70
16 – AI Adoption Scenarios and connected Prioritised Use Cases of the Evidence Management .................. 77
17 - Prioritisation of Horizontal and Vertical Recommendation .................................................................. 90
18 - Roadmap for actions (Horizontal & Vertical Recommendations) ......................................................... 92
19 - Expected budget and time scale of the Horizontal Recommendations ................................................. 94
20 - Expected budget for Vertical Recommendations ................................................................................ 94
21 - Expected budget, time scale and man days for Programme Management ........................................... 95
22 - Expected Total Budget per Horizontal and Vertical Recommendations ................................................ 95
23 - List of EFSA stakeholders' categories and sub-categories ................................................................... 97
24 - Stakeholder value map .................................................................................................................. 105
25 - Concepts to integrate into Communication messages ...................................................................... 106
26 - Engagement plan.......................................................................................................................... 110
27 - Collaboration models and related tools ........................................................................................... 117
28 - Focus on synergies with other roadmaps ........................................................................................ 118
www.efsa.europa.eu/publications
7
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
List of acronyms
Acronym
Definition
ADRA
AI, Data and Robotics Association
AESGP
Association of the European Self-Care Industry
AI
Artificial Intelligence
ALTAI
Assessment List of Trustworthy Artificial Intelligence
AMU
Assessment and Methodological Unit
ANEC
European consumer voice in standardisation
API
Application Programming Interface
ArcGIS
Geospatial Information Management System
ASySD
Automated Systematic Search Deduplicator
AWS
Amazon Web Services
BEUC
European Consumer Organisation
CDT
Translation Centre for the Bodies of the European Union
CEF2
The Connecting Europe Facility 2021-2027
CoC
Code of Conduct
CNR
National Research Council
CPVO
Community Plant Variety Office
CV
Curriculum Vitae
DATA
Evidence Management Unit (EFSA)
DCF
Data Collection Framework
DEP
Digital European Programme
DevOps
Development and Operations
DG
Directorate General
DG
Data Governance
DIH
Digital Innovation Hubs
DOI
Digital Object Identifier
DTs
Disruptive Technologies
EC
European Commission
ECDC
European Centre for Disease Prevention and Control
ECHA
European Chemicals Agency
ECOTOX
Ecotoxicology Knowledgebase
EEA
European Environment Agency
EFA
European Free Alliance
EMA
European Medicines Agency
EP
European Parliament
EPA
United States Environmental Protection Agency
www.efsa.europa.eu/publications
8
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Acronym
Definition
EPHA
European Public Health Alliance
EFSA
European Food Security Authority
EKE
Expert Knowledge Elicitation
ENISA
European Union Agency for Cybersecurity
ETAPAS
Ethical Technology Adoption in Public Administration Services, Horizon 2020 project
ETL
Extract Transform and Load
EU
European Union
EUIPO
European Union Intellectual Property Office
FAO
Food and Agriculture Organization
FVE
Federation of Veterinarians of Europe
GO
Gene Ontology
H2020
Horizon 2020 programme
HLEG
High-Level Expert Group
ICPEN
International Consumer Protection and Enforcement Network
ICT
Information and Communications Technology
IUCLID
International Uniform Chemical Information Database
JRC
Joint Research Centre
KJC
Knowledge Junction Community
MCDA
Multi-Criteria Decision Analysis
ML
Machine Learning
MoU
Memorandum of Understanding
NAMs
New Approach Methodologies
NCA
National Competent Authorities
NCBO
National Center for Biomedical Ontology
NER
Named Entity Recognition
NGO
Non-Governmental Organisation
NLP
Natural Language Processing
OBO
Open Biological and Biomedical Ontology
OCR
Optical Character Recognition
OECD
Organisation for Economic Co-operation and Development
PA
Public Administration
PARC
European partnership for the assessment of risks from chemicals, Horizon Europe
PERA
Partnership for Environmental Risk Assessment
PCP
Pre-commercial Procurement
PoC
Proof of Concept
PPI
Public Procurement of Innovative Solutions
www.efsa.europa.eu/publications
9
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Acronym
Definition
R&D
Research & Development
RIS
Research Information Systems
RM
Roadmap
ROI
Return on Investment
SaaS
Software-as-a-Service
SDU
Smart Document Understanding
SEPAL
System for Earth Observation Data Access, Processing, and Analysis for Land Monitoring
SLR
Systematic Literature Review
SMEs
Small and Medium Enterprises
SNAPP
Science for Nature and People Partnership
SR
Systematic Reviews
SRA
Quantitative security risk assessment
SQL
Structured Query Language
TIM
Tools for Innovation Monitoring
TRL
Technology readiness level
UCL
University College London
VKM
The Norwegian Scientific Committee for Food and Environment
WHO
World Health Organization
www.efsa.europa.eu/publications
10
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
1.
Introduction
This report is an outcome of an EFSA procurement titled “Development of a Roadmap for Actions on artificial
intelligence for evidence management in risk assessment (specific contract No 35 –RC 5 implementing
Framework Contract No OC/EFSA/FIN/2019/01), awarded by the European Food Safety Authority to PwC EU
Services EESV and the subcontractor Intellera Consulting Srl.
This report is the final document of the “ Roadmap for actions on artificial intelligence for evidence
management in risk assessment” project (in brief “AI-RM”), conducted by PwC EU Services EESV and its subcontractor Intellera Consulting SLR for the European Food Safety Authority (EFSA). It presents all the activities
performed, together with the elaboration and the analysis of the main findings collected during the project.
The report is structured as follows:
•
Chapter 1 “Introduction” provides an overview of the content of each chapter of the report.
•
Chapter 2 “Background” includes the context in which the project was conducted, with a focus on the
European policy framework and on EFSA’s work on AI development. In addition, it presents the high-level
methodology and the problem formulation used to define the evidence management use cases that guided
the subsequent project activities.
•
Chapter 3 “State of the art and analysis” provides an overview of the state of the art of AI solutions
based on the results that emerged from the various data collection activities. In addition, a dedicated section
explores the potential challenges and blockers related to the adoption of AI, followed by the identification
and evaluation of collaboration opportunities with key players in the market.
•
Chapter 4 “Recommendations and prioritisation” outlines the roadmap’s recommendations (horizontal
recommendations and vertical recommendations) together with the prioritisation of working areas based on
the outcomes of the first activities of the project.
•
Chapter 5 “Communication and engagement opportunities” outlines potential communication and
engagement opportunities in the field of AI approaches to evidence management of risk assessments, taking
into consideration specific needs and concerns of EFSA’s stakeholders on the adoption of AI.
•
Chapter 6 “Conclusions” provides final considerations on the main outcomes of the project based on the
content provided in the chapters previously outlined.
2.
Background
This chapter focuses on the context in which the project was conducted, with a focus on the European policy
framework and on EFSA’s work on AI development. In addition, it presents the specific objectives of the project, the
high-level methodology and the problem formulation used to define the evidence management use cases that guided
the subsequent project activities.
The chapter is structured as follows:
•
Section 2.1 “Policy context” provides an overview of the European policy context that has been evolving
with regard to AI, taking into consideration regulatory actions from the European Commission.
•
Section 2.2 “EFSA’s specific objectives” provides context concerning the specific objectives to be
achieved through this project.
•
Section 2.3 “High level project phases” provides an overview of the various phases of the project.
•
Section 2.4 “Problem formulation” provides an overview of the main activities conducted in order to
frame the project.
www.efsa.europa.eu/publications
11
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
2.1.
Policy Context
Over the last years, growing computing power and increased data availability, have led to a significant rise in the
strategic importance of artificial intelligence (AI) in the European scenario. AI is revolutionising industry and society
worldwide, allowing for significant changes and opportunities, while also presenting new challenges. Indeed, AI is
considered a “disruptive technology”, with the potential to change the way products and services are created and
delivered across a wide range of industries. Using AI technologies in the public sector can promote openness,
accountability, quality and trustworthiness of given services, as well as saving costs and time, but it also entails
specific risks and challenges. In this context, the goal of the European Union (EU) is to make the EU a world-class
hub for AI, while ensuring that AI is human-centric, trustworthy and grounded in European values and fundamental
rights. To this end, in April 2018, in its communication “Artificial Intelligence for Europe”, the European
Commission (EC) reaffirmed the role of AI as one of the most strategic technologies of the XXI century, representing
a key driver of economic growth. Since then, the development of the AI market – and ensuing regulatory needs –
has always been high on the EC’s agenda, as demonstrated by the increasing amount of financial investments and
policy documents adopted.
Figure 1 below highlights the most relevant policy and regulatory initiatives launched by the European Commission
since 2018.
Figure 1 – EU’s approach to AI.
As highlighted in the previous figure, the EC published the proposal for an “AI Regulation” on 21 April 20212,
also called the AI Act, which aims to foster the development and application of AI in the internal market, while
addressing - and mitigating - the risks generated by specific uses of AI through a set of complementary, proportionate
and flexible rules (legal framework for trustworthy AI 2). The regulation’s primary objective is to “ensure the proper
functioning of the internal market by setting harmonised rules, in particular on the development, placing on the
Union market and use of products and services making use of AI technologies, or provided as stand-alone AI
systems”.3
In practice, a risk-based approach is put forward to address and manage risks associated with AI systems.
Specifically, risks are classified according to four clusters:
2
European Commission, Proposal for a Regulation Of The European Parliament And Of The Council Laying Down Harmonised
Rules On Artificial Intelligence (Artificial Intelligence Act) And Amending Certain Union Legislative Acts, COM (2021) 206 final.
www.efsa.europa.eu/publications
12
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
●
●
●
●
Unacceptable risk: All AI systems that are banned as they are considered a clear threat to the safety,
livelihoods and rights of people, e.g., from social scoring by governments to toys using voice assistance that
encourages dangerous behaviour;
High-risk: AI systems employed in critical sectors (e.g., transport, education, migration) and those that are
subject to strict obligations (i.e. in terms of quality, functionalities, transparency, security and robustness)
before they can be put on the market;
Limited risk: AI systems that are permitted but subject to specific transparency obligations (e.g. chatbots);
Minimal risk: AI systems with very low or no risk that are allowed without restrictions (e.g. AI-enabled video
games, spam filters).
The approach is aimed at preventing "high-risk" applications from entering the market. High-risk applications are
those relating to employment, transport, and law enforcement, for example. These systems must meet five
requirements: data quality, bias, explainability, human oversight, security, and accuracy. High-risk systems must be
registered with the EU and re-assessed if the system changes substantially. Applications that manipulate human
behaviour to evade users' free will and government social scoring are totally prohibited as they pose an intolerable
danger. With rare exclusions, biometric surveillance in public settings is likewise forbidden .
In addition, the legal framework aims to reduce administrative and financial barriers for businesses. In particular, it
fosters innovation through sandboxing programmes, aimed mostly at small and medium-sized enterprises (SMEs).
The proposal also includes the creation of the "European Artificial Intelligence Board".
In order to prepare EFSA to the disruptive impact that the regulation is likely to have on the way AI is used and
managed, in the context of this project, the trustworthiness of AI has been taken into consideration as a fundamental
aspect, therefore explored both during data collection activities and during the subsequent development of
recommendations.
2.2.
EFSA's specific objectives
EFSA's interest in Artificial Intelligence is in line with the EC's agenda. In the last years, together with other Agencies,
EFSA has launched the project “Joining forces at EU level on the implementation of Artificial Intelligence”,
in order to explore the benefits of AI technologies and develop a European virtual community around AI. The project
provided an overview on AI Trend Analysis, AI Maturity Assessment and the AI strategic roadmap. Moreover,
analysing EFSA internal processes, it emerged that one of the most relevant uses of AI should be applied to Evidence
Management Forecasting, upon which EFSA developed other interesting AI techniques.
This project was conceived also as an effect of the “Transparency Regulation”3 published by the EU, that in its
Article 32d states that, in exceptional circumstances of controversy or conflicting results, scientific studies may be
commissioned, with the objective of verifying evidence used in its risk assessment process, and having eventually a
wider scope than the evidence subject to verification. To this end, EFSA´s grant and procurement budget embed
verification studies on an annual basis from 2021 onwards, and in the absence of specific requests under Article 32d,
the Agency will dedicate part of its grants and procurement budget for the purpose of preparedness for verification
studies.
In addition, in line with its consideration on the future of science 4 and its 2027 strategic development 5, in 2021 EFSA
prioritised some scientific themes requiring multi-annual cooperative arrangements with Member States, EU
Agencies and where relevant, international partners. For each of them, a roadmap for action is being developed,
3
Regulation (EU) 2019/1381 of the European Parliament and of the Council of 20 June 2019 on the transparency and
sustainability of the EU risk assessment in the food chain and amending Regulations (EC) No 178/2002, (EC) No 1829/2003,
(EC) No 1831/2003, (EC) No 2065/2003, (EC) No 1935/2004, (EC) No 1331/2008, (EC) No 1107/2009, (EU) 2015/2283 and
Directive 2001/18/EC. OJ L 231, 6.9.2019, p. 1-28.
4
Bronzwaer, S., Kass, G., Robinson, T., Tazarona, J., Verhagen, H., Verloo, D., Vrobs, D., & Hugas. M (2019). Food Safety
Regulatory Research Needs 2030; EFSA Joural
5
https://www.efsa.europa.eu/sites/default/files/event/mb-82/mb191218-7-p.pdf
www.efsa.europa.eu/publications
13
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
in order to define specific projects or studies to carry out in the following years in order to attain the strategic
objectives.
One of these scientific themes concerns the use of AI in the evidence management process of risk
assessment. This roadmap provides recommendations for future multi-annual, multi-partner studies or
projects in the field of AI applications to the evidence management process, by building on EFSA’s vision
and supporting its preparedness for future risk assessments, while preventing possible discrepancies on sensitive
matters6.
The roadmap for action provides:
•
•
•
A full understanding of ongoing activities, knowledge gaps, and societal interests and concerns, as well as
collaboration opportunities and potential partners,
A set of recommendations for the adoption of AI in the evidence management process, with a proposed
timeline and budget estimate,
A communication and engagement plan in view of the adoption of AI.
By developing an approach for the implementation of AI methods in the evidence management phase of
risk assessment, EFSA will achieve its vision included in the Theme paper on AI in risk assessment 7 by:
●
increasing accessibility of the body of evidence,
●
increasing breadth of the body of evidence,
●
enhancing trustworthiness of the subsequent risk assessment process,
applying human centric artificial intelligence in close co-existence with human expertise.
2.3.
High level project phases
The roadmap was developed in 8 months, and was composed of three subsequent phases:
•
The Inception phase which aims at refining the scope, methodology and workplan of the entire project,
based on the Technical Offer and the input gathered at the Kick-off meeting by the client.
•
The Execution phase which foresees the data collection activities, data analysis and definition of interim
output needed to draft the Roadmap Report.
•
The Roadmap definition phase which includes the synthesis activities of the interim output and the design
of a communication and engagement strategy in the field of AI.
Figure 2 below shows the main activities carried out for each phase and related duration :
6
See also Regulation (EC) No 178/2002 of the European Parliament and of the Council of 28 January 2002 laying down the
general principles and requirements of food law, establishing the European Food Safety Authority and laying down procedures
in matters of food safety. OJ L 31, 1.2.2002, p. 1–24.; Regulation (EU) 2019/1381 of the European Parliament and of the
Council of 20 June 2019 on the transparency and sustainability of the EU risk assessment in the food chain and amending
Regulations (EC) No 178/2002, (EC) No 1829/2003, (EC) No 1831/2003, (EC) No 2065/2003, (EC) No 1935/2004, (EC) No
1331/2008, (EC) No 1107/2009, (EU) 2015/2283 and Directive 2001/18/EC. OJ L 231, 6.9.2019, p. 1-28.
7 European Food Safety Authority, Theme (concept) paper – on Artificial intelligence in risk assessment (2021)
www.efsa.europa.eu/publications
14
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 2 - Main activities of the project.
2.4.
Problem formulation
In order to clarify the scope of the roadmap and AI integration potential within the process, the following activities
have been performed:
•
High level mapping of the evidence management process, performed on the basis of the available
information and the inputs of the EFSA staff;
•
The identification of potential use cases, i.e. specific process steps or tasks which can be improved by
AI tools/approaches;
•
The assessment and prioritisation of use cases, aimed at defining a shortlist of key use cases to guide
the following project activities. The prioritisation was an iterative procedure, based on the different inputs
collected during project execution.
2.4.1.
High-level mapping of the evidence management process in EFSA
EFSA is the risk assessor, evaluating risks associated with the food chain. EFSA collects and analyses existing research
data to provide independent scientific advice, supporting the decision-making of risk managers. In addition, by
collecting and analysing available data, it communicates existing and emerging risks associated with the food chain.
Therefore, EFSA's evidence management process is at the core of the evidence-based risk assessment.
Specifically, evidence management is a structured process of collection and analysis of various streams of evidence
(e.g., scientific data, expertise literature, best practices, etc.) generated through multiple methods and reported by
multiple sources. It follows a specific and documented process with clearly defined phases managed by different
Units in EFSA – including AMU (Assessment and Methodological unit) and DATA (Integrated Data unit). The highlevel evidence management process steps are the following:
1. Evidence Collection: systematically gathering and validating the required streams of data through multiple
means and from various reliable sources (relevant data providers e.g. (EU Member States organisations’
databases, literature papers and from experts’ knowledge in the field)
2. Evidence Appraisal: assessing the quality, consistency and complementarity of all the data collected from
the different sources;
3. Evidence Synthesis and Integration: combining, summarising and finding key patterns within data
through the performance of multiple typologies of data analysis (e.g., statistical analysis, meta-analysis,
meta-regression and dose-response modelling) and consequently interpreting the results;
4. Evidence Visualisation and dissemination: the production of customised data visualisations - in the
form of reports, tables and graphs -and publication of evidence in the EFSA curated and open repository for
the exchange of evidence and supporting materials used in food and feed safety risk assessments
(Knowledge Junction Community)
www.efsa.europa.eu/publications
15
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 3 - Evidence management process phases.
The succession of these phases provides a fit for purpose and transparent risk assessment.
These macro-phases of evidence management are composed of single processes (see table 1), which in turn can be
broken-down into steps. For each process, key considerations on the current degree of automation and technology
architecture were put forward to identify relevant pain points and risks within the process, thus shedding light on
potential applications of AI technologies.
Table 1 - Risk Assessment Workflow and Evidence Management Process.
Macro phase
Process
Process description
Key Considerations
Data collection from
Member States, data
holders, and other
stakeholders
It consists in receiving, processing and validating
structured data through the DCF (Data Collection
Framework, i.e. EFSA Online Tool) from multiple
stakeholders
Process with low level of
automation
Systematic
Literature Review
It consists of the methodological extraction, appraisal,
analysis and reporting with respect to sources within
the existing literature (e.g. peer-reviewed articles)
Time-consuming process, with a
low level of automation and subject
to human error risk and bias
Expert Knowledge
Elicitation
It consists of eliciting relevant knowledge from one or
more experts in the field to complement the previous
processes and bridge limited, absent or conflicting
empirical evidence
Complexity in finding relevant
experts’ contacts with the required
competences and knowledge and
variety in their selection
Dossier assessment
It includes the submission of dossiers to EFSA as part
of the Application Management risk assessment
process
Continuous improvement of
digitalisation due to the
introduction of the Appian platform
Public consultations
It consists in the collection and management of
comments from key stakeholders in different cases
(e.g. comments on the list of intended studies and
their design for application renewal)
Highly labour-intensive process
with a low level of automation
B
Evidence Appraisal
Advice on Evidence
Appraisal
It includes the provision of advice and quality
evaluation of all types of integrated evidence streams
submitted or selected by EFSA
Time-consuming process execution
with a rather low level of
automatisation and prone to
possible human errors
C
Evidence synthesis
and integration
Evidence analysis
It consists of inspecting, cleansing, transforming,
modelling and interpreting data with the goal of
discovering useful information, informing conclusions,
and supporting decision-making.
Process with low degree of
technology development/maturity
and low level of automation
A
Evidence Collection
www.efsa.europa.eu/publications
16
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Macro phase
Process
Process description
Key Considerations
Uncertainty Analysis
It consists in the systematic identification and
characterisation of uncertainty and limitations in
questions of interest and/or quantities of interest in a
scientific assessment, and evaluating their implications
for scientific conclusions.
Process variant with low degree of
technology development/maturity
and low level of automation
Production of
Reports and Data
Visualisation
It consists in producing relevant data visualisations
and reports through statistical tools
Lack of automation in the
production of visual reports
concerning data collected from the
SLRs and the public consultation.
Publication on the
Knowledge Junction
Community
It consists of the publication of curated supporting
material/evidence (e.g. annexes of scientific outputs
containing raw data, specific documents of external
organisations during the public consultations of a
scientific output) on the Knowledge Junction
Community and the maintenance of the latter in terms
of requirements, permissions and metadata
Lack of automation in the
maintenance of the KJC repository
D
Evidence
visualisation and
dissemination
As previously mentioned, this process plays a crucial role within EFSA’s risk assessment, but it is a resource-intensive
activity where the amount of available information is limited both by the accessibility of the evidence itself and by
the available resources to undergo the different phases of the process. For this reason, AI has been considered a
key technology to improve the process in terms of costs, time and quality of the output. Artificial intelligence (AI)
approaches have a strong potential in reducing human-bias in the process and reducing the need to initiate
verification studies (as per Article 32d of Regulation (EU) 2019/1381). In case a verification study is required, AI
approaches will facilitate the integration of the new evidence generated by the verification study.
2.4.2.
Identification of the use cases
Through an internal consultation with EFSA, a clear understanding of the agency’s processes was achieved. The
information collected made it possible to identify an extensive list of use cases, i.e., specific tasks within the evidence
management process, which can potentially be automated through AI. In addition, data collection activities made it
possible to gather further information on the potential of the use cases in terms of impact and feasibility. The output
of these activities can be summarised in the extensive list of use cases shown in figure 4 below.
www.efsa.europa.eu/publications
17
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 4 – Evidence management process phases and use cases.
2.4.3.
Prioritisation of use cases
In order to guide the project activities, we performed a prioritisation exercise on the long list of use cases. This
prioritisation was an iterative procedure that occurred in multiple rounds, based on the additional inputs collected
during the project. We set out below the final output of this activity.
Our approach was based on a Multi-Criteria Decision Analysis, or MCDA, a method that is commonly applied to
face complex decisions and to solve problems involving a choice among alternatives. The approach had the
advantage of allowing us to focus on what is important in a logical, coherent, and proportionate way.
To apply MCDA for prioritising the AI use cases, the following steps were performed:
•
Criteria Selection for the purpose of dividing the decision into smaller, more understandable parts in the
form of criteria tailored to the subject at hand. The approach for selecting the relevant criteria was firstly
based on a standard impact/potential for adoption matrix approach. Within these broad categories, specific
criteria were selected to reflect EFSA's vision, stakeholder's opinions as registered during the working
sessions and a preliminary feasibility analysis of each use case's functionality. As a result of this analysis, the
dimensions and respective criteria used in the prioritisation exercise were the following:
Dimension 1: Impact, which included the following three criteria:
–
Time Efficiency/Effort Reduction: expressed as the reduced time duration and effort (i.e., hours
workers spent) to perform a process or a task;
–
Quality Improvement: expressed as an enhanced process performance (e.g. via an increase in the
accessibility of evidence, enhanced analysis capacity, etc) and delivery of better outputs in terms of
content and quality of conclusions;
–
Reduction of human bias: expressed as a lower risk of human errors and deviation from norm or
objectivity in any selection, appraisal and decision-making steps.
Dimension 2: Potential for adoption, which included the following two criteria:
–
Acceptance: expressed as the potential for acceptance of the integration of AI in the proposed use
case by the relevant users and stakeholders within the organisation;
www.efsa.europa.eu/publications
18
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
–
Feasibility: expressed as the degree of readiness in a functional level of the proposed use cases and
the potential integration of the AI solution into the current operations (e.g. in terms of process
framework).
•
Scoring of each use case according to each criterion. The approach was used to provide a score for each
criterion for each individual use case. The scoring approach assigned a score on a scale of 0-3, where 0
equals no impact/potential, 1 equals low impact/potential, 2 equals medium impact/potential and 3 equals
high impact/potential. For the Impact dimension criteria (time efficiency/quality improvement/reduction in
bias) we based our scoring on our understanding of the issues faced by EFSA at each stage of the evidence
management process, through the discussions with the various teams and stakeholders, and considering the
improvements that could be made via the use of AI. For the Potential for Adoption dimension criteria
(acceptance/feasibility) we based our score on the discussions with EFSA’s teams and stakeholders, and also
our own knowledge and understanding of the current state-of-the-art in AI-enabled solutions for providing
satisfactory solutions to each use case.
•
Integrating the scored criteria though a weighted average approach to produce a meaningful solution
and identify the final ranking. In order to reach an integrated scoring, we used a weighted average approach,
where the Impact and Potential for Adoption dimensions had an equal weight. The choice of weights for
individual criteria was based on our understanding of EFSA’s general purpose and objectives, taking into
consideration the opinion of EFSA’s internal stakeholders and our own understanding of the individual use
case complexities, as well as the current readiness level of AI applications on the market. The weighting
system has been tested through a sensibility analysis, and then validated by the project team.
Figure 5 – MCDA criteria for use case prioritisation
The output of the revised prioritisation is represented graphically in figure 6 below. The use cases highlighted in
green have the highest levels of Impact and Potential for Adoption, thus falling into the prioritised use cases list (also
listed in table 2 below).
www.efsa.europa.eu/publications
19
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 6 - Revised prioritisation of the use cases.
Table 2 - Top 10 prioritised use-cases.
Use Case Impact
Potential for
Adoption
Prioritisation Score
SLR Keywords Identification
2.5
3
5.5
2.
SLR Literature Findings Clustering
2.5
2.75
5.25
3.
Expert Selection
3
2.25
5.25
4.
Expert Pool Identification
2.75
2
4.75
5.
Comparable Appraisals Detection
2.5
2.25
4.75
6.
SLR Relevant Abstract Screening
1.75
3
4.75
7.
Automatic Text Summarisation
2.25
2.25
4.5
8.
Quality Improvement of the Scientific Output
2
2.25
4.25
9.
SLR Literature De-duplication
2
2.25
4.25
10.
Data Collection Terminology Assessment
2.25
2
4.25
#
Use Case
1.
In conclusion, according to our understanding of the state of the art combined with EFSA’s needs, the prioritised use
cases are the most feasible and impactful for the adoption of AI in EFSA’s evidence management process. Therefore,
prioritised use cases have been considered to develop the set of “vertical actions” for the roadmap (see chapter 4).
3.
State of the art and analysis
This chapter focuses on the current state of the art of AI in evidence management, taking into consideration existing
solutions and initiatives identified during the project. The main findings are analysed, providing information related
to potential challenges and blockers to be encountered when applying AI. Finally, potential cooperation and
collaboration opportunities, identified during data collection activities, are provided.
The chapter is structured as follows:
•
Section 3.1 “Market overview, map of relevant activities and further areas” provides a description
and some key results regarding the multiple AI initiatives and solutions and related organisations
investigated. These were identified with various activities performed during the project (desk research,
www.efsa.europa.eu/publications
20
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
survey, interviews, workshop), covering the prioritised evidence management use cases process. It also
focuses on further AI areas that fall outside of the scope of evidence management
•
Section 3.2 “Challenges and blockers” introduces general challenges and blockers for adopting AI in a
public organisation, together with specific challenges that EFSA would face for the adoption of AI in its
evidence management process in the form of a gap analysis of the AI and data maturity assessment of
EFSA’s ecosystem.
•
Section 3.3 “Cooperation and collaboration opportunities” provides an identification of the phases of
the end-to-end AI lifecycle, from strategy to operation, together with possible collaboration/cooperation
models to be adopted with multiple stakeholders. The link between AI lifecycle phases and collaboration
models will be further investigated in section 5.4.
3.1.
Market overview, map of relevant activities and further areas
The data collection phase was performed through multiple methods – desk research, literature review,
stakeholders’ survey, interviews with relevant stakeholders and workshop. Subsequently, the data collected during
the project has been analysed in order to map primarily key initiatives, tools and the respective organisations in the
area of AI approaches in evidence management, and secondly, to identify additional information such as the policy
background that is still relevant for the development of the roadmap. In addition, the same data collection activities
made it possible to further investigate AI applications that fall outside of the scope of the evidence management.
3.1.1.
Initiatives
The term “initiatives” refers to the activities of organisations – especially institutions – related to the use of AI in the
evidence management process (e.g., organisation of a webinar, pilot projects, tender procedures, development of
an internal algorithm). This was one of the most important topics explored through the data collection activities, as
it can be useful to understand whether there are other organisations with similar needs, and what has been the
impact, challenges and lessons learned from these experiences.
In general, only 8 initiatives related to the use of AI in the evidence management process have been identified (see
Table 3 below). Some other projects on the use of AI have been identified as well, but they fall outside of the scope
of this project. For example, we found some projects and roadmaps of AI related to the agricultural sector (e.g. FAO
introduced an Advanced Land Monitoring system used for earth observations, data access, processing & analysis for
land monitoring), and AI applications for the analysis of social media unstructured data (e.g. ECDC has launched a
free, open-source interactive tool called “Epitweetr” to automatise a more detailed early detection of public health
threats using Twitter data using the R programming language, allowing them to automatically monitor trends of
tweets by time, place and topic, thereby detecting signals such as an unusual increase in the number of tweets).
It is worth mentioning that, even though some institutions have projects related to evidence management, on the
basis of the evidence gathered, no detailed strategies, roadmaps or action plans on this topic have been developed
by them.
The following list of initiatives (Table 3) was drafted by combining the information collected by the different data
collection activities, including the literature review, desk research, survey and interviews. This table provides an
overview of the initiatives developed by institutional stakeholders (EU Agencies and Institutional, International
organisations and National competent authorities).
Table 3 - Extensive list of initiatives.
Organisation
Short description of the initiative
CDT
The organisation is currently working at adopting an AI-based tool to automatically summarise long texts, aiming to
significantly reduce costs on single translations and decrease the number of pages that need to be translated,
ultimately enabling a higher number of documents to be translated
www.efsa.europa.eu/publications
21
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Organisation
Short description of the initiative
The organisation has adopted AI for text extraction, where an open-source solution able to read letters and extract
relevant information is used. EUIPO’s AI model is internally developed.
EUIPO
The organisation, in collaboration with the IE University, presented through a webinar a detail review of AI’s
developments, both conceptually and in terms of its relevance to specific business situations. Among the most
relevant applications, AI can be employed for text extraction (e.g., more specifically anomaly detection, automated
table extraction), analysis and summarisation and natural language generation.
The organisation is leading on two recent pilot projects, each focusing on one use case of evidence management,
mainly Text data structuring (data extraction and structuring from summary product characteristics) and
Keywords/metadata identification.
The organisation has produced a report named ‘Evolving Data-Driven Regulation’ that outlines several
recommendations on leveraging Artificial Intelligence (AI) and new digital technologies, to improve efficiency in
processes and increase insights into data. These recommendations aimed to assist regulators and stakeholders in
capturing the opportunity for data-driven, evidence-based, rigorous decision-making that will underlie the
development, approval, and on-market safety and effectiveness monitoring of medicines, in a fast-expanding data
and analytics world
EMA
ENISA
Member of the Blueprint cooperation framework (EC), ENISA has opened a Tender procedure to further develop the
“Open Cyber Security Situation Awareness Machine,” an Open-Source Intelligence tool that aims to optimise the
production of cybersecurity situation awareness reports, by applying Natural Language Processing (NLP) and Machine
Learning (ML) techniques for the automatic classification and generation of documents.
OECD
The organisation has recently launched a workshop called “AI and the productivity of science”, which was pointed out
by a Senior Policy Analyst that we interviewed. The workshop addresses the critically important issue of the rate of
scientific progress, whether this is stagnating, as recently argued by a number of scholars, and how AI could raise the
pace of progress in science and discovery. The workshop brings together technical and policy experts to examine the
evidence on a purported productivity decline in science, as well as the ways that AI is currently used across different
fields of science – from neuroscience to materials science - and across all stages in the scientific process.
United States
Environmental
Protection
Agency (US
EPA)
The US Environmental Protection Agency (EPA)'s etc has developed a novel approach to article retrieval in the
PubMed Abstract Sifter. This approach has been implemented and made publicly available in the Dashboard and in a
Microsoft Excel version. The Dashboard version is a convenient and powerful way to find information quickly and
efficiently about a chemical of interest. For more in-depth literature tasks such as reviews, the Excel version of the
Abstract Sifter offers additional functionality. Results from a PubMed search are imported directing into an Excel
sheet, where the end-user can then use a novel “sifter” methodology for quick, agile relevance ranking of articles.
The tool also enables article triage capabilities through easy tagging and noting functionality. Triaged citations can be
exported to external software such as reference management tools. The Abstract Sifter can also provide a high-level
view of a corpus of literature for a defined set of entities such as chemicals.
3.1.2.
AI solutions and respective organisations
This paragraph provides an overview of the information related to the identified solutions through the different data
collection activities performed since the beginning of the project, namely desk research, literature review, survey,
interviews and workshop, as shown in Figure 7.
www.efsa.europa.eu/publications
22
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 7 - Data collection activities.
A total of 81 solutions were identified, which were then classified according to use cases, relevance and level of
readiness8.
Out of the 81 solutions identified, 35 are currently associated with a high readiness level, 40 with a medium readiness
level, and 7 with a low readiness level. It is worth mentioning that some solutions cover more than one use case:
out of the 81 solutions, 49 cover only one prioritised use case, 13 cover 2 prioritised use cases, and 5 cover 3
prioritised use cases (figure 8). 14 tools were associated to some non-prioritised evidence management use cases
(e.g., Public Consultation Automation, Confidentiality Dossier Assessment, Citations Screening, Information
Consolidation).
Figure 8 - Use case relevance.
In terms of use cases association, 23 solutions were related to Literature Clustering, 10 to Literature deduplication, 15 to Keywords identification, 13 to Experts selection, 4 to Expert Pool Identification, 9 to Improvement
of scientific output, 4 to Comparable appraisal detection, 8 to Terminology assessment, 12 to Text summarisation,
and 4 to Abstract screening (Figure 9).
The following graph (Figure 9) depicts the relationship between the solutions, use cases and readiness level.
8
Level of readiness definitions:
Low level: Refers to tools/solutions that are in a research stage
Medium level: Refers to tools/solutions that are in a development stage
High level: Refers to tools/solutions that are in a deployment or commercialisation stage
www.efsa.europa.eu/publications
23
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 9 - Number of solutions per use case and readiness level.
Overall, the Evidence Collection phase within the Evidence Management process represents the area with the
greatest room for AI-based automation and improvement possibilities, leading to multiple potential benefits
such as time and effort reduction and quality improvement. Within the Evidence Collection phase, Data Collection
from multiple stakeholders, SLR, and Expert Knowledge Elicitation are the areas that record the highest number of
developed or in development solutions, according to our analysis.
Furthermore, most solutions explored through the different data collection activities for this project can be
positioned into an advanced or medium phase of development (high or medium readiness), and this might
open opportunities for collaboration with key stakeholders (i.e., for the co-development of a tool, or co-execution of
research), resulting in possible purchases (in case of commercialised tools), and allowing the sharing of knowledge
and lessons learnt.
The following image (Figure 10) shows a summary of the main key insights, including the use case maturity &
relevance, the type of stakeholder which developed the solutions identified, and their locations.
Figure 10 - Summary of key insights.
www.efsa.europa.eu/publications
24
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
3.1.3.
AI Development environments and services
This section presents an overview of four cloud-based AI development services identified in the data collection phase
of the project, which can provide EFSA with a suitable suite of ready-made data preparation pipelines, ready-made
AI-based models and services and code development modules to enable the end-to-end development of AI solutions,
and in particular text analytics/natural language processing (NLP), to address the use cases relevant to evidence
management in risk assessment, and also EFSA's wider needs and plans with regard to the adoption of AI. The
identified services represent market leaders in NLP services and development environments: Microsoft Azure, IBM,
Amazon Web Services and Google cloud. Table 4 below provides a short description of the development
environments identified.
Table 4 - Brief description of the development environments identified in the data collection.
Service
Brief Description
Azure Cognitive
Services
Azure Cognitive Services are cloud-based services available to help users build cognitive
intelligence which can be customised. The development environment allows the inclusion
of cognitive features in AI applications without having artificial intelligence (AI) or data science
skills. Azure Cognitive Service for Language provides several Natural Language Processing (NLP)
systems, including Text Analytics, which is a collection of features from Cognitive Service for
Language that extract, classify and understand text within documents (relevant to several
use cases in evidence management, especially within the SLR activities). In addition, the service
includes a text summarisation module.
IBM Watson
IBM Watson Natural Language Understanding is a service provided by IBM cloud which uses deep
learning to extract meaning and metadata from unstructured text data. It offers services around
text analytics to extract categories, classify text and extract trends from text, all of
relevance to several of EFSA's use cases in evidence management. In addition, Watson
Knowledge Studio allows users to train Watson to understand the language of a specific domain
and extract customised insights.
Amazon Web Services
Amazon Comprehend is a natural language processing (NLP) service that uses machine learning
to extract key phrases, entities and sentiment from text for further analysis. This
particular service is, for the time-being, more geared towards customer service and front-office
applications, and is less mature than the Microsoft and IBM solutions for use in a scientific
context.
Google Cloud
Cloud Natural Language is the Google cloud service which helps users derive insights from
unstructured text using Google machine learning. One application offered which is of relevance to
EFSA is the custom entity extraction, which permits the identification of domain-specific
entities within documents – many of which don’t appear in standard language models –
without having to spend time or money on manual analysis. In addition to the ready-made
models, Google provides the AutoML Natural Language service which allows users to train a
custom machine learning model to classify documents, extract information, or
understand the sentiment based on a specific domain. These services are relevant to the
prioritised use cases since a classification model which analyses a document and returns a list of
content categories would fulfil, for example, the SLR Literature Findings Clustering.
3.1.4.
Overlaps with R&I programmes
With regards to the EU R&I funding programmes (e.g. Horizon Europe, Digital Europe Programme), these can be
helpful to identify whether other organisations do similar things, while exploring potential cooperation opportunities.
Although, currently, there are no specific calls or partnerships on the application of AI in evidence management
within the European ecosystem, there is some potential for future deployment of AI technology within organisations
across different sectors.
www.efsa.europa.eu/publications
25
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
For instance, it is worth mentioning that, in 2020, the EU Commission and ADRA 9 (AI, Data and Robotics Association)
signed the European Partnership on artificial Intelligence, Data and Robotics 10, which covers the deployment of AI
technology in a broad sense (without focusing specifically on the evidence management area). By 2030, the
partnership aims to boost European competitiveness, societal well-being and environmental aspects to lead the world
in researching, developing and deploying value-driven trustworthy AI, Data and Robotics based on fundamental
European rights, principles and values.
In addition, out of the scope of AI technology, some other relevant calls for EFSA have been designed in the 2021
Work Programme of Horizon Europe, such as, for instance:
-
HORIZON-HLTH-2021-ENVHLTH-03-01: European partnership for the assessment of risks from chemicals
(PARC)11
-
HORIZON-CL6-2021-FARM2FORK-01-16: Identification, assessment, and management of existing and
emerging food safety issues12
In light of this, it can be assumed that potential calls and initiatives focusing on AI adoption in evidence management
could be launched in the next few years, as a response to the above-mentioned calls and partnerships. Therefore, a
constant monitoring action should be planned to track their development and progress.
To conclude, in case of relevant calls for the use of AI in internal processes – and evidence management in particular
- it would be important for EFSA to participate as an interested party in order to be fully aligned on the outcomes of
the projects, to be involved within the community of stakeholders, and eventually to contribute to the development
or testing activities and benefit from the projects’ results.
3.1.5.
AI for additional use cases within the evidence management process
Even though the data collection phase focused on the prioritised use cases of the evidence management process,
and how the relevant tools can be implemented in one or more of these cases, additional solutions for the other
use cases have been identified. These solutions are presented in this section (Table 5), as they can potentially be
beneficial for EFSA and, if implemented, they could introduce additional innovation in the evidence management
process.
Table 5 - Solutions for additional use cases within evidence management.
Other Use Case
Anomaly
Detection
Readiness
level13
Tool
Tool Description
Logicmelt
Technologies SL
Logimelt offers product development support services to
technology companies that plan to use Artificial Intelligence in
their products. Among other services, they offer custom design
of Artificial Intelligence algorithms and optimised
implementation of these algorithms in Edge Computing
platforms.
High
9
Composed of several stakeholders: BDVA, CLAIRE, ELLIS, EurAI and euRobotics
https://ai-data-robotics-partnership.eu/
11 European Commission; Horizon Europe Work Programme 2021-2022; Health. (European Commission Decision C(2021)9128
of 15 December 2021)
12 European Commission; Horizon Europe Work Programme 2021-2022; Food, Bioeconomy, Natural Resources, Agriculture and
Environment. (European Commission Decision C(2021)9128 of 15 December 2021)
13 Level of readiness definitions:
Low level: Refers to tools/solutions that are in a research stage
Medium level: Refers to tools/solutions that are in a development stage
High level: Refers to tools/solutions that are in an already commercialised stage
10
www.efsa.europa.eu/publications
26
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Other Use Case
SLR Appraised
Papers Data
Extraction
SLR Review
Question
Evaluation
SLR Metaanalysis
Execution
Quantitative
Data Synthesis
and Integration
Tool Description
PhlexNeuron
The PhlexNeuron regulatory automation platform provides a
sophisticated machine learning framework that enables the
extraction of specific data from unstructured data sources.
High
Syras
Syras supports user to quickly review and categorise a large
body of references.
High
WebPlotDigitizer
WebPlotDigitizer is a web-based tool which extract numerical
data from plot images or maps.
High
SWIFT-Review
SWIFT-Review is an interactive workbench which provides
numerous tools to assist with problem formulation and literature
prioritisation.
High
EPPI - Reviewer
EPPI-Reviewer is a web-based software program for managing
and analysing data in literature reviews by identifying key
terms.
High
Syras
Syras support users to quickly review and categorise a large
body of references.
High
Synthesis
Synthesis allows user to automatically interact with network
quantitative synthesis built from extracted data.
High
RevMan
RevMan is software that supports the conduct of quantitative
synthesis of data and the graphical presentation of results.
High
CMA
CMA is a software package that allows the user to compute the
effect size for each study automatically and create highresolution plots.
High
Neo4j AuraDB
3.1.6.
Readiness
level13
Tool
Neo4j AuraDB is a fully managed cloud graph database service.
Built to leverage relationships in data, AuraDB enables lightningfast queries for real-time analytics and insights. AuraDB is
reliable, secure, and fully automated, enabling you to focus on
building graph applications without worrying about database
administration.
High
AI related to working areas outside of evidence management
This section presents solutions that were identified as particularly relevant for different working areas outside of the
evidence management process. Even though the literature review, desk research, survey and interviews focused on
the prioritised use cases of the evidence management process, and how relevant tools can be implemented in one
or more of these cases, additional functionalities and features have been highlighted. These features are presented
www.efsa.europa.eu/publications
27
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
in this section (Table 6) as they can be beneficial for EFSA and, if implemented, also foster additional innovation in
the risk assessment process.
Table 6 - Additional features of tools.
Tool
Organisation
Readiness
level14
Description
Main functionalities
SGS
Digicomply includes the tracking and analysis
of different discussions in the media, and on
social networks such as Twitter.
High
Sentiment analysis
social media
Google
It provides entity analysis to find and label
fields within a document—including emails,
chat, and social media—and then sentiment
analysis to understand customer opinions to
find actionable product and UX insights.
High
Sentiment analysis
AutoML
Natural
Language
Google
AutoML Natural Language enables you to
build and deploy custom machine learning
models that analyse documents, categorise
them, identify entities within them, or assess
attitudes within them. If you don't need a
custom model solution, the Cloud Natural
Language API provides content classification,
entity and sentiment analysis, and more.
High
Sentiment analysis
Crème Global
Crème Global
Crème Global includes a combination of a
data and scientific models to quantify and
understand risk, by implementing risk models
in a web-based user-friendly interface.
High
Risk assessment
analysis
TIM (Tools for
Innovation
Monitoring)
JRC
It functions as a monitoring system for
tracking the evolution of established and
emerging technologies.
Low
Monitoring emerging
technologies
Medium
Social media analysis
Digicomply
Natural
Language AI
Epitwitter
ECDC
Free, opensource interactive tool that helps
with the automatised early detection of public
health threats using Twitter data, and more in
detail using the R programming language,
allowing trends of tweets to be automatically
monitored by time, place and topic, and
detecting signals such as an unusual increase
in the number of tweets. It is designed to
support public health experts with the early
detection of threats from infectious diseases,
but can be extended to all hazards and other
fields of study by modifying the topics and
keywords.
14
Level of readiness definitions:
Low level: Refers to tools/solutions that are in a research stage
Medium level: Refers to tools/solutions that are in a development stage
High level: Refers to tools/solutions that are in an already commercialised stage
www.efsa.europa.eu/publications
28
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
These solutions, which fall outside of the evidence management working areas, should not be considered as relevant
for reaching the vision of EFSA (i.e., the adoption of AI for increasing the body of evidence within the evidence
management process by 2027). Nevertheless, they could be taken into account for other purposes such as
automating time- and effort-intensive activities.
3.2.
Challenges and blockers related to the adoption of AI in evidence
management
This sub-section provides insights concerning potential challenges and blockers on the basis of the information
collected during the data collection phase of the project - including the analysis of the literature review, desk research,
survey, interviews and workshop, as well as the information collected by internally consulting EFSA’s team for the
implementation of AI solutions within the EFSA ecosystem.
3.2.1.
General challenges and blockers
The evidence gathered allowed us to identify four challenges and blockers related to the adoption of AI in a public
organisation (See figure 11):
-
Adoption Challenge 1 – Impact on processes, governance and people: The first aspect includes
challenges that the public organisation might encounter when considering the impact of AI adoption in terms
of processes, governance, user acceptance and skills. This would require a more holistic organisational
transformation for AI to be successfully adopted. Furthermore, specific processes and workflows might need
to be re-designed to take into account the role of AI. In general, AI should not simply be viewed as the
addition of automation and decision support within existing workflows, but instead it should be viewed as a
transformational change that cannot only replace but also transform traditional ways of working.
-
Adoption Challenge 2 – Adopting solutions that can address specific problems: The second
challenge concerns the need to identify AI solutions that fit specific needs and requirements.
Organisations look for specific AI tools that can perform well on their own and that can provide answers to
specific solutions, which are not always available in market. Organisations seeking specific solutions need to
recognise that, for them to become truly AI-enabled, internal data science capabilities are needed, which
can bring desired results in combination with the organisation’s domain expertise.
-
Adoption Challenge 3 – Ethical Risks: The third challenge considers issues related to the ethical risks
of AI adoption, especially when used by public institutions, which must ensure the highest transparency and
fairness standards. In fact, the adoption of trustworthy and human-centric AI should be a priority both
internally and also in light of upcoming AI Regulation15.
-
Adoption Challenge 4 – Compliance: This challenge concerns the need for any use of data (within AI
or not) to address compliance considerations around data protection (e.g., Art. 24 of EU GDPR
"Responsibility of the controller").
These challenges - if not properly taken into consideration and addressed with dedicated measures - could hamper
the efficacy and/or the feasibility of the adoption of AI in the evidence management process. For this reason, they
are carefully considered during the roadmap development.
Concerning technical challenges - related to the potential AI tools to be implemented – four categories have been
identified as follows:
i)
Technical Challenge 1 – Readiness: The first technical challenge concerns the level of readiness of
the AI solutions to be adopted. In fact, several tools are still at a premature/experimental stage of
development and are not yet available in the market (i.e., not commercialised), therefore they would
need a certain degree of development and/or customisation before being fully integrated.
15
European Commission, Proposal for a Regulation Of The European Parliament And Of The Council Laying Down Harmonised
Rules On Artificial Intelligence (Artificial Intelligence Act) And Amending Certain Union Legislative Acts, COM (2021) 206 final.
www.efsa.europa.eu/publications
29
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
ii)
Technical Challenge 2 – Accessibility: The second technical challenge refers to the accessibility of
the tool. Many tools, in fact, require advanced (e.g., programming language) technical skills in the case
of customisable functionality.
iii)
Technical Challenge 3 – Applicability: The third technical challenge considers the applicability of
the tool to specific domains of the organisation. Many solutions, in fact, are currently applicable to
specific domains and, therefore, require customisation in order to be applied in different scientific
domains. To address this, in addition to the need for developing internal data science capability (see
Adoption Challenge 2 above), EFSA should invest in the development of subject-specific ontologies to
customise commercially available or internally developed AI solutions.
iv)
Technical Challenge 4 - Autonomy: The last technical challenge relates to the dependence of some
tools on human intervention. The level of maturity of commercial AI solutions still does not provide
sufficient confidence in their use without a human-in-the-loop.
The abovementioned technical challenges could hamper the feasibility and efficacy of the implementation of AI
solutions in the different evidence management processes.
Figure 11 - General Challenges.
Finally, market readiness can be considered as a potential blocker for the adoption of AI solutions within evidence
management in the next few years. The discontinuities and feasibility analysis highlighted those working areas (or
use cases) where AI can already be adopted or implemented due to the presence of mature solutions in the market,
and those where the AI market is still at an early stage .
3.2.2.
EFSA’s challenges and blockers: Results of the Gap Analysis
In order to identify technical or operational gaps that could hamper the feasibility or effectiveness of EFSA’s
actions with regard to the application of AI in the evidence management process, a Gap Analysis was performed
between EFSA current capabilities (AS-IS) and a target state which would support EFSA`s achievement of its vision
(TO-BE) related to the adoption of AI tools. From a methodological point of view, an AI and data maturity assessment
of EFSA has been performed, with emphasis on evidence management in risk assessment.
The Gap Analysis consists of the following phases:
www.efsa.europa.eu/publications
30
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
•
Reviewing of the current state of EFSA’s capabilities, resources, procedures and structure for adopting the
identified tools.
•
Describing the target state in terms of the technical requirements and skills required to customise and
integrate the identified AI tools in the evidence management process of the risk assessment.
•
Based on the results of the previous activities, drafting a list of recommendations in order to guide the
development of the full recommendations (see Chapter 4).
3.2.2.1.
Current State of EFSA capabilities and future plans
This section describes the current capabilities of EFSA's technical infrastructure (See Figure 12) with the aim of
evaluating the adoption of AI solutions for evidence management in risk assessment.
Figure 12 - Gap Analysis of the EFSA ecosystem.
The following list provides information on the Technical Capabilities of EFSA staff & Procedures within EFSA:
•
Programming capabilities: SAS Analytics, SQL, Python and R, and handling XML and Json file formats.
•
APIs and data management capabilities: APIs including GET and POST methods, transmitting and parsing
Json bodies, and data management and visualisation skills (MicroStrategy).
•
Data format procedure: data received from Member States are validated with rules implemented in SAS.
•
Scientific article storage procedure: scientific papers are downloaded from free and subscription-based
databases. Currently, all papers are stored into a simple data management system. However, recently EFSA
started to transfer them to Azure.
In the list below, an overview of Hardware & Software Components, Structure & Requirements within EFSA
is provided:
•
Cloud platform: EFSA has a contract to use Microsoft's Azure cloud platform and services. EFSA's vision is to
take full advantage of the capabilities that this platform offers.
•
Software within the SLR working area: Currently EFSA is using the DistillerSR AI module for the partial
automation of title and abstract screening phase of systematic literature reviews (SLRs), which approximately
saves EFSA 60-70 % of the effort needed to accomplish this task. Even though EFSA is constantly looking
for improvements of SLRs, replacing DistillerSR is not a desirable option as it would require a lot of effort
www.efsa.europa.eu/publications
31
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
due to users’ resistance against the change. In addition, training on a new tool might be required. On the
other hand, for full text screenings, no relevant tool is currently being used. Therefore, automating this
process would be of a great benefit for EFSA.
•
Evidence repository: The Knowledge Junction is a curated, open repository for the exchange of evidence and
supporting materials used in food safety risk assessments.
•
Software for statistical application: EFSA R4EU is a platform that is currently used to expose statistical
applications internally and to the public by providing a user-friendly interface to perform sample-size
calculations. The R4EU platform hosts a suite of different modules with different functionalities for modelling,
commonly used in assessments by EFSA. Existing models include benchmark dose, multi-drug resistance
analysis, risk assessment using Monte Carlo, risk-based surveillance systems, sample size calculator,
exploratory analysis for space-temporal epidemiology and automatic abstract and full-text screening using
machine learning.
•
Data collection framework: a data collection framework (DCF) is currently in place, responsible for
collecting, validating and storing scientific raw data from Member States in relation to contaminants,
pesticides, biological hazards and consumption data, as well as data regarding antimicrobial resistance,
microbiological contaminants and food-born outbreak data in the contexts of zoonoses. The system provides
support for all reference terminologies used in EFSA, such as the list of substances (used also for dossier
management) and FoodEx2 (international standard used for food coding).
•
Management data tools: Currently scientific data are accessed, managed and analysed through a
combination of systems, including ArcGIS (Geospatial information management system), MicroStrategy
(Reporting tool) and SAS (Analytics tool).
•
Computing environment: a personal computing environment that runs remotely on the cloud exists and is
accessible from any device for a limited number of scientific users that need access to the Scientific Data
Warehouse applications.
•
Data protection: EFSA can only use cloud-based tools if the cloud provider agrees to sign a contract
committing that they will host all the data within the European Union.
Some feedback on Future Plans for EFSA has been identified as follows:
•
DATA 2.0 project: special importance is given to a project that EFSA is working on, named DATA 2.0. The
project includes a tool which will be used to improve automation by converting data into standard formats
that are widely accepted.
•
Data governance framework: a Data Governance Framework is being considered as a future project to
support EFSA in terms of data accuracy, completeness, and consistency.
3.2.2.2.
Target State requirements
In this sub-section, a list of technical requirements and skills are provided with the aim of implementing the AI
solutions identified during the data collection phase of the project. “Technical Requirements” refers to different
terms, including infrastructure and tools needed for the development and training of AI tools, while “Skills” refers to
terms such as knowledge of operating systems, software engineering and development.
Technical Requirements
•
Data Infrastructure for Big Data and AI: The first technical requirement for EFSA's target state is a Data
Infrastructure which can facilitate end-to-end Big Data and AI analysis and development. While many AI
solutions are available in a Software-as-a-Service (SaaS) delivery model, and are thus easily accessible by
any users or organisations, it is evident that for EFSA to be able to adopt solutions that fit the organisation’s
requirements satisfactorily, and for moving into research activities, where data scientists need to develop or
adapt a given algorithm, the expansion of EFSA’s current infrastructure to one that can address the new
needs around Big Data and AI development is a pre-requisite.
www.efsa.europa.eu/publications
32
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
•
Programming Software: Common programming software for data preparation and data science includes SQL,
R, Python, C# and Scala.
•
Subject-specific ontologies: To customise commercially available AI tools, or to develop them from scratch
utilising Natural Language Processing (NLP) to address domain-specific applications, EFSA needs to adopt
existing, or create from scratch, domain-specific ontologies (concepts and categories in a subject area or
domain that shows their properties and the relations between them in the form of a table)
Skills
•
Programming: In addition to knowledge of at least one of R, Python, C# and Scala, a good knowledge of
fundamentals of Algorithms, Data Structures and Object-Oriented Languages is also needed, as well as the
ability to code Quantitative and Statistical Analyses.
•
SQL – Structured Query Language: knowledge of SQL, Structured Query Language, is fundamental for
extracting data from databases both in traditional data warehouses and big data technologies.
•
Data Visualisation: The ability to interpret the data visually (either within programming languages or via BI
tools) is necessary in order for insights via analytics and AI analyses to be successfully conveyed.
•
Mathematical/Analytical Skills: Problem solving skills around complex
mathematical/analytical background with an understanding of Big Data.
•
Knowledge of Natural Language Processing and Text Analytics: within the context of Evidence Management,
EFSA’s target state requires data scientists with specialised skills and experience in Natural Language
Processing (e.g., Named Entity Recognition, Text Summarisation, Topic Modelling, Semantic Search) and
text mining. These skills are also needed for the development of domain-specific ontologies using state-ofthe-art text mining methodologies.
•
Combination of Analytical and Domain Expertise: For EFSA to be able to use AI/NLP across domains, it is
important for data scientists to be able to acquire domain expertise so that the tools are programmed to
answer specific scientific questions within a specialised domain. It is thus necessary for data science
knowledge and skill to be dissipated within specific units rather than remain centralised.
3.2.2.3.
data,
require
a
solid
Addressing challenges and blockers identified by the Gap Analysis
In this sub-section, an analysis of possible solutions that might support EFSA in overcoming the challenges and
blockers identified during the previous activities, is presented. Via the analysis of the current vs target state, we have
recognised that for some of the identified gaps there is a good level of maturity within EFSA, however there is still
room for improvement given that a series of tools, functionalities and organisational changes can provide further
benefits. The identified gaps have provided the basis for the horizontal recommendations presented in section 4.1.
Table 7 - Gaps and Recommendations per each dimension.
Dimension
Infrastructure
Tools
&
Data
Information
Organisation
Governance
&
Process
Integration
&
&
Gap
While EFSA has adopted a cloud-based platform and uses multiple
programming languages and a BI tool, the current infrastructure
might not be sufficient for the use of Big Data and AI
EFSA does not have any domain-specific ontologies
While there are plans for a Data Governance framework, it is still
not in place.
Since AI has still not been widely adopted within EFSA, there are
no guidelines in place to ensure its ethical and transparent use.
Furthermore, for the same reasons, no ML model development
management framework is in place.
www.efsa.europa.eu/publications
33
Recommendation
Implement a Data Infrastructure
for Big Data and AI
Create (or adopt existing) domainspecific ontologies
Implement a Data Governance
framework that considers AI
Establish DevOps and adopt a
Trustworthy AI framework
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Skills & Culture
There is a plethora of data science skills within EFSA scattered
across some teams, however there are still some skill gaps,
especially related to big data technologies and NLP.
Develop data science skills around
NLP and Big Data and design
behaviour change mechanisms
The horizontal recommendations that can address these gaps are presented and elaborated in section 4.1.
3.3.
Cooperation/collaboration opportunities
This sub-section outlines and evaluates the cooperation/collaboration opportunities to be implemented with the
identified potential partners. It first provides an identification of the phases of the end-to-end lifecycle, from strategy
to operation and a list of possible collaboration/cooperation models to be adopted with multiple stakeholders.
3.3.1.
AI Lifecycle Phases
Third party collaboration/cooperation can happen in different phases of an AI adoption project, with different
objectives and categories of stakeholders. For this reason, we introduced a conceptual framework that breaks down
AI adoption into phases – we call it the “AI lifecycle” – and used this to better understand when
collaboration/cooperation can happen, for what objectives, through what modalities and with which type of
stakeholders.
Our previous data collection activities have been functional to this exercise because they shed light on the different
cooperation modalities available for EFSA, and on the composition of the stakeholder ecosystem.
The AI lifecycle, instead, is part of the output of the previous project, “Joining forces at EU level on the
implementation of Artificial Intelligence” (2019), and it includes all steps from strategy development to the monitoring
of AI-based solutions.
Specifically, the lifecycle can be broken down into 8 phases (as shown in Figure 13). The phases might occur in
parallel or be re-iterated, especially if an agile development approach is followed. Their description is provided below.
Figure 13 - AI lifecycle phases.
•
Strategy: consists of the design of EFSA’s AI strategy in line with the organisation's values, brand image,
morals and ethics (including several tasks such as the definition of its scope across the organisation,
ownership and accountability for its delivery and identification of impacted stakeholders’ groups), and
guideline to the related implementation such as Portfolio Management, including market research for
identifying and prioritising industry use cases and the definition of the desirable outcomes and risks for the
use cases. Furthermore, this phase includes the Policy & Regulation area, including tasks such as the
identification of relevant regulatory requirements and business policies relating to the use of AI, and
definition of the expected businesses compliance requirements for the solution.
•
Delivery: consists of the processes to manage the use cases’ development in terms of Organisational
Change, such as the development of plans and capabilities and definition of data science skills and business
functional knowledge within business functions, to enable the adoption and support of AI solutions. Also,
this phase includes the Programme Governance area, aimed at defining a clear program scope, roles,
www.efsa.europa.eu/publications
34
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
responsibilities, desired business outcomes, quality standards and risks to be monitored and managed
through several tools (e.g., quality assurance mechanisms). Finally, this phase foresees the adoption of an
Agile Delivery Approach to encourage fail-fast learning and minimum viable product releases, and also the
definition of a tailored AI development lifecycle.
•
Ecosystem: consists of processes for Vendor Management to evaluate different sourcing options (e.g.,
build vs buy options) and select the best vendor able to meet both functional and non-functional
requirements, and to demonstrate the quality of the solutions. In addition, this phase includes the definition
of a Technology Roadmap, involving the scouting for new technological options and their integration into
the enterprise IT estate for production support.
•
Furthermore, it foresees the definition of Standards and Practices (e.g. for the ethical use of AI, and
development of AI solutions through a consistent approach). A specific focus on vendor management within
this phase, applied in the context of the AI-RM project, is provided in sub-section 3.3.3.1
•
Data: consists of the definition of Data Requirements considering data type, quality, completeness, and
volume, and development of Data Gathering process from desired data sources, as well as Data
Preparation tasks to support AI models, leveraging on transformation functions, features selection, data
labelling, and data sampling. In addition, all of these activities within this phase need to follow the adopted
Data Governance framework.
•
Design: consists of the selection and development of a final AI Model (involving the also the definition of
appropriate performance measures and prior use of cross-validation to estimate the model performance by
means of training data), and its consequent Verification and Tuning to test the model in terms of
robustness, and consistency and compare it against functional and non-functional requirements.
•
Also, this phase includes the definition of Controls Framework (e.g., COSO framework to measure the
most prevalent risks), aligned with the ethics and governance guidelines and policies to ensure the quality
of the solution.
•
Implementation: consists of the implementation of the AI solution managing Business Readiness
through change management activities (i.e. making sure there are well-defined and shared use cases, and
that the organisation is aware of the central role of data for AI adoption), Deployment of the solutions in
the production environment following robust testing and errors management and Transition of the solution
in the daily operations without any negative disruption, and making sure it is provided with the necessary
hand over and/or training.
•
Operating: consists of the operating activities to guarantee Refinement & Improvement of the model’s
performance through feedback collection mechanisms and review of the model’s outputs. Also, it includes
the Resilience area, involving the definition of a business continuity plan and incident response procedures,
and Support for the AI solution, involving the effective implementation of incident and problem
management procedures, knowledge management procedures, and the change and release management
procedure.
•
Monitoring: consists of the monitoring activities to ensure Compliance, involving tasks such as the
implementation of checks on data protection, definition and use of a register with records of all processing
activities, and provision of evidence of compliance with policies and regulations to third parties. Also, this
phase includes the Cyber security area, including tasks such as the definition of effective methods to
identity, authenticate and check access permissions of every data source/user/application/device connecting
to data. Finally, this phase includes the Performance of the AI solution, consisting of the establishment of
tools and mechanisms to evaluate the benefits, costs, and performance anomalies of the solution.
Collaboration/cooperation with third parties can take place – and can be beneficial – in each one of these phases,
having specific objectives that are related to the nature of the activity and the types of stakeholders that could be
involved. The next paragraph adds another element to our reasoning, mapping the different
www.efsa.europa.eu/publications
35
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
collaboration/cooperation models that an organisation like EFSA can establish. The AI lifecycle is also mentioned in
the next section to contextualise the recommendations.
3.3.2.
Map of potential cooperation/collaboration models
As mentioned in the previous paragraph, this section provides an overview of the multiple collaboration/cooperation
models that EFSA can establish with different partners for the potential adoption of AI solutions.
According to the open innovation approach 16, multiple collaboration/cooperation opportunities can be established for
the development and implementation of AI solutions within the evidence management process, with existing and
potential stakeholders (from academia, industry and institutions etc.) composing the EFSA ecosystem (Figure 14).
Figure 14 - State of Play of EFSA AI ecosystem of stakeholders.
Some of these stakeholders have already built or are currently developing relevant solutions, capabilities or services
that can be used for EFSA’s internal processes, or even have the necessary knowledge and expertise in AI in evidence
management. However, other stakeholders are still in the research phase, or are designing an AI strategy or plan.
The possible cooperation/collaboration models that EFSA could involve various scouting instruments and modalities
to develop and deliver innovation. Table 8 provides a comprehensive overview of all cooperation models identified,
together with some elements such as the main functions of the model under scope, and the stakeholders that can
and may benefit from participating, together with a short description of the collaboration model.
Table 8 - Comprehensive overview of cooperation models.
Cooperation model
Stakeholder
Description
Strategic alignment
EU Agencies, EU
Institutions,
Refers to activities such as sharing ideas and lessons learned. Discussing common
challenges with other public organisations can pave the way to the definition of a
common vision and planning, and eventually to more concrete collaboration, avoiding
16
The open innovation approach is the use of purposive inflows and outflows of knowledge to accelerate internal innovation,
while taking into consideration the opportunity to expand the markets for use of innovation (Eelko K.R.E.,Huizingh; Open
innovation: State of the art and future perspectives; Technovation; 2011).
www.efsa.europa.eu/publications
36
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Cooperation model
Memorandum of
Understanding
Stakeholder
Description
International
Organisations
overlaps and waste of resources. Strategic alignment, therefore, is an important
cooperation modality and it can take place through the organisation of conferences,
workshops and ad-hoc meetings, such as those EFSA already does in the field of AI
with the EU Agencies as part of the “Join forces” initiative.
all
Refers to an agreement between two or more parties that defines the framework of a
negotiation, outlining the services to be exchanged and actions to be performed
within a specific period of time. It expresses a convergence of will between the
parties, indicating an intended common line of action, helping to explain any protocols
for communication, information exchange, reporting, confidentiality issues, and
modifications and conditions for terminating the agreement. MoUs are agreements in
the form of legal documents. However, they are not fully binding in the way that
contracts are, but they are stronger and more formal than a traditional gentleman's
agreement.
MODALITIES TO DEVELOP AND PROCURE INNOVATION: consisting in the launch of procurement procedures (of different kinds,
depending on the TRL of the solution to be acquired) or the direct participation in projects or cooperation programmes for the development
or the implementation
Procurement
Open (e.g.
Industry players
and academia)
Refers to the process by which public authorities purchase work, goods and services.
Usually results of the procured projects are owned by the contracting authority (unlike
in the grant procedure, where is can be shared with the contractor). In this context,
for instance, EFSA might launch a call for procurement in order to purchase off-the
shelf solutions, procure related services for integration or maintenance and outsource
internal activities.
Procurement can also be published on behalf of two or more contracting authorities
together (i.e. Joint procurement).
Refers to a specific kind of procurement that includes:
Innovation procurement
Open, usually
interesting for
Industry and
Academia e.g.,
startups,
innovative
SMEs, university
spin offs
i) the development of innovative solutions through the procurement of research and
development (R&D) services (Pre-Commercial Procurement). In this case, the
public buyer describes its need, prompting businesses and researchers to develop
innovative products, services or processes to meet the need.
ii) the procurement of innovative solutions that are not yet available or do not exist on
the market, or the procurement of innovative solutions that do exist but are not yet
widely available on the market (Public Procurement of Innovative Solutions). In
this second case, Research & Development (R&D) of the solution is not needed.
Different procedures could be applied to implement innovation procurement.17
Contest
Open to public
Refers to a recognition prize granted by a contracting authority to beneficiaries who
can most effectively meet a defined challenge. Specifically, contests consist of
acquiring a plan or design (e.g. of an AI prototype), to be selected by a jury, after
being put out to competition. The aim is to stimulate innovation and come up with
solutions to problems. The winner of a contest will receive cash, publicity coverage or
promotion as a reward.
Detailed Rules of contests are outlined in Art. 207 of the REGULATION (EU, Euratom)
2018/1046 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL. 18
Grant
Art 36
organisations
Refers to a financial donation awarded by the contracting authority to the grant
beneficiary. EFSA should consider awarding grants for projects and activities that
contribute to EFSA’s mission, which consists in developing, implementing and
17
https://ec.europa.eu/growth/single-market/public-procurement/innovation-procurement_en
Official Journal of the European Union; REGULATION (EU, Euratom) 2018/1046 OF THE EUROPEAN PARLIAMENT AND OF
THE COUNCIL of 18 July 2018 on the financial rules applicable to the general budget of the Union,
18
www.efsa.europa.eu/publications
37
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Cooperation model
Stakeholder
Description
adopting AI within specific processes of the evidence management19. According to
EFSA 20 there are several types of grants that the agency can launch: Specific grants,
Tasking grants, Thematic grants, Fellowship grants, Partnering grants.
Cooperation programme
EU Agencies
Refers to multi-agency programme that funds and manages various projects and joint
actions, with a shared budget and governance. Within a Cooperation Programme, call
for tenders and call for proposal can be launched with potential partners.
Horizon Europe
National food
safety
authorities,
Academia
Refers to participation to research & innovation projects carried out by consortia of
organisations working together on specific areas. EFSA can be an interested party in
those projects, therefore it can engage with the community, provide data for
development and testing of solutions, and other activities, depending on the scope of
the projects.
SCOUTING INSTRUMENTS and IDEAS GENERATION: instruments which can be used to collect innovative ideas and solutions on AI
Public consultation
Internet crowdsourcing
Hackathons
Open to the
public
Refers, according to EFSA sources, to the creation of an effective exchange on a draft
scientific output based on a decision of EFSA to seek comments from the public,
namely the non-institutional stakeholders, which include academics, NGOs, industry
and all other potentially interested and affected parties21.
All relevant
public and
private
stakeholders
Refers to a central scouting activity that can give access to a wider set of expertise by
engaging a broader spectrum of sources via the use of the internet and purpose-built
platforms to elicit knowledge and services. Depending on the request, there are
different types of crowdsourcing. A crowd contest is one of the most common types
that an actor can use by seeking multiple qualified companies or individuals who will
submit their own idea or solution (depending on the assigned task e.g., testing or
data analysis). The winner gets funds for producing this solution or idea22.
Open to the
public
Refers to the organisation of design sprint-like events together with other partners
such as universities or industries, which can have the focus on design of AI solutions
to be adopted for evidence management. Hackathons are scouting instruments that
have proven to be an excellent medium to generate ideas for new tech products and
services, and it can lead to the creation of innovative concepts, ideas, and prototypes
while, at the same time, it stimulates the innovative ecosystem.
According to the open innovation framework, several development levers could be taken into account to
accelerate the modalities to engage the AI community and establish cooperation. These levers include the items
reported in the table below (Table 9).
Table 9 - Development Levers
Development
levers
Description
19
https://ec.europa.eu/international-partnerships/grants_en
https://www.efsa.europa.eu/en/discover/infographics/yes-we-grant
21Https://www.efsa.europa.eu/sites/default/files/event/documentset/5_public_consultation_final_document_110609.pdf
22 EFSA https://www.efsa.europa.eu/en/press/news/190614
20
www.efsa.europa.eu/publications
38
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Digital innovation
hubs (DIHs)
DIHs are at the core of European digital policy as they function as one-stop shops that help companies become
more competitive with regard to their business/production processes, products or services using digital
technologies, by providing access to technical expertise and experimentation, so that companies can “test-beforeinvesting”. They also provide innovation services, such as financing advice, training and skills development that are
needed for a successful digital transformation. DIHs also support companies and public sector organisations in the
use of digital technology to improve the sustainability of their processes and products. A DIH as a multi-partner
cooperation can provide strong linkages with different actors inside and outside of its region promoting the
implementation of multiple digital opportunities in the industry and the public sector. A partnership with a DIH
focused on AI could help EFSA to engage the AI community and foster the uptake of digital technologies, including
AI solutions.
Technology
transfer office
Refers to an alternative approach to promote innovative solutions in line with the EFSA needs regarding the
development and implementation of AI solutions within the evidence management process23. Establishing a
partnership with a technology transfer office will promote the translation of recent technology advances into
marketable solutions by maintaining close and long-term relationships with the results creators.
Incubators and
accelerators
Refers to an organisation that helps start-up companies and individual entrepreneurs to develop their businesses
by providing a full-scale range of services, starting with management training and office space and ending with
venture capital financing. Incubators help many promising business ideas to be developed and take the first steps
to transforming them into marketable and scalable ideas). A partnership with an incubator is strategic in nature as
it will be a great source of innovative solutions that will bring new knowledge, skills and experience to the surface,
which are necessary tools for the implementation of EFSA's aspirations
Once the multiple cooperation/collaboration models and the related stakeholder category are defined (as mapped in
Table 9), they can be associated to the different phases of the end-to-end AI lifecycle (described in section 3.1.1)
based on the needs and added value for both EFSA and possible third parties in cooperation/collaboration, as shown
in Figure 15 below.
Figure 15 - AI lifecycle and collaboration models.
Aaboen, Lise & Holgersson, Marcus. (2016). "Technology Transfer Offices, Incubators, and Intellectual Property
Management". Academy of Management Proceedings. 2016.
23
www.efsa.europa.eu/publications
39
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
This mapping is further elaborated and discussed in section 5.2 “Engagement Plan”, where EFSA needs and the
reason why (main motives and incentives) a specific stakeholder category should be willing to collaborate with EFSA
through a specific model within a particular AI lifecycle phase are also defined.
3.3.2.1.
Vendor selection and procurement management
One of the key phases where collaboration with third parties is established is the “ecosystem” phase of the AI
lifecycle, where, in the vendor management activities, an organisation defines - based on its needs - the best
approach to procure or develop AI solutions to be integrated within its IT ecosystem. Being a crucial part, this
paragraph focuses on this procedure and its possible outcomes, in order to allow EFSA to choose the best vendor
selection procedure, and pave the way for the vertical recommendations (see section 4.2.).
The vendor selection procedure for AI solutions is made up of the following steps:
1. Definition of the need - Definition of the challenge or need in terms of scope, objectives (benefits, performance
and expected impact) and, if possible, requirements (functional and non-functional). In other words, the need
identifies the use case to be improved thanks to the adoption of AI technology.
2. Market scouting - Research, based on the identified need, of the market solutions that could be ready to be
purchased and integrated within the system to solve the challenge or need. The result of this step can be that
either there are many solutions (≥ 3 solutions, for instance), or that there are none/very few.
3. Coverage assessment - If there are many available solutions, an assessment of the coverage of the solutions
against the specific requirements should be carried out. This aims to understand whether they can cover fully
(>90%), partially (90% ≥ x ≥ 60%) or insufficiently (<60%) the requirements defined at step 1.
4. Scenario selection - Based on the outcome of the previous steps, there are 3 options for the selection of the
best vendor scenarios:
•
Scenario 1: If a high number (at least 3) of available solutions with a high coverage (higher than 90%)
have been identified, the AI market is mature enough for the organisation to purchase a good off-the-shelf
solution, without incurring in additional R&D costs. Therefore, a procurement procedure would be the
most suitable model of collaboration, with the aim to purchase solutions that are ready to be tested and
integrated into the production environment with little customisation.
Benefits
Launching a procurement procedure to buy off-the-shelf solutions is usually the most efficient – in terms of
costs and time – and effective way for an organisation to adopt AI. Although even in this case the integration
process could require some customisation activities, the project would not have high risks because the
solutions already cover very well the needs. In addition, procurement would allow internal resources to focus
on core organisational activity.
Challenges
Three activities are key to the launch of an effective procurement procedure for an AI solution to be
purchased off-the-shelf: first, market scouting, which has to reflect real market potential and be able to
assess with a high degree of certainty whether there are potentially fitting solutions to be bought; second,
the stakeholder engagement activities around the opening of the procedure have to ensure that a good
number of proposals are collected, in order to allow the organisation to choose the best one in terms of
quality and costs; third, the tender itself has to be clear, and allow the potential contractors to well
understand the needs, constraints and desired impact. For this reason, particular attention must be paid to
these activities, which should be allocated to internal staff with the right skills, experience and understanding
of the domain of application. In addition to the challenges related to the procurement itself, buying an offthe-shelf solution could entail the risk of a low degree of solution/vendor flexibility in case of interim changes
of need (scope, requirements), although this can be mitigated by adopting an agile project management
approach.
www.efsa.europa.eu/publications
40
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
•
Scenario 2: If from the market scouting and the coverage assessment at least 3 commercialised solutions
with a medium level of coverage (between 90% and 60%) are identified, it would be risky to try to buy
an off-the-shelf solution, as the procurement procedure may not be successful. With this in mind, the
organisation could launch a procurement procedure including a PoC (Proof of Concept)/Pilot phase to
demonstrate the quality and feasibility of the solutions before integration. Another option could be the launch
of a PPI (Public Procurement of Innovative Solutions) , a particular type of innovation procurement, to
purchase AI solutions that were not commercialised before or that still need some developments, or existing
solutions that need customisation phases to match perfectly the need, including the PoC/Pilot phase as well.
Benefits
Innovation Procurement can be a flexible way to develop and implement highly innovative solutions for
solving new problems, and to meet needs that are not fully covered by existing solutions in the market. A
procurement that includes a PoC in the foreseen tasks has a better probability of achieving the objectives.
Challenges
Launching an innovation procurement might result in being a more time-consuming and expensive process
in comparison to standard procurement, because of the need to perform a Proof of Concept/Pilot phase to
demonstrate the solution's performance and feasibility.
•
Scenario 3: If few solutions (less than 3) or no solutions have been identified by the market scouting, or
if the solutions identified cover insufficiently the needs (less than 60%), it will not be possible to buy any
solutions from the market, and so a new solution, tailored on the needs, must be developed from scratch. If
the organisation cannot/does not want to develop the algorithms internally, it can launch a grant to finance
R&D activities aimed at developing and implementing AI-based innovative solutions that respond to needs
and resolve problems that are not addressed by the solutions available on the market.
In addition, EFSA may launch a PCP (Pre-Commercial procurement) to buy research and development
(R&D) services from several competing suppliers, including the design, prototyping, original development
and validation/testing of a limited set of first products. In this regard, it should be noted that a PCP does not
include services related to the ‘commercialisation’ of the solution, intended as the full
deployment/implementation and diffusion of a large scale of end-products within the organisation. Hence,
following the PCP, EFSA could consider opening a PPI for the production and implementation of the solution
(in larger quantities) to be adopted widely by EFSA 24.
Finally, EFSA might consider opening a Contest procedure in order to develop, test and validate the
development of an innovative solution.
These scenarios would also involve a request to vendors to produce a PoC / Pilot Project, and the final
integration of this within the selected process to be automated.
Benefits
In general, outsourcing development activities with open competitions ensures the identification of highly
skilled teams at a convenient cost. With regard to EFSA grants, given that only organisations included in the
list of Competent Organisations as designated by a Member State can participate, we can assume that they
are aligned with EFSA's mission. In addition, opening a grant would also reinforce synergies among different
stakeholder participating under Article 3625.
Launching a PCP would facilitate access to new innovative players, such as start-ups and SMEs, and to the
public procurement market, sharing the risks and benefits of designing, prototyping, and testing new
products and services and creating the optimum conditions for wider commercialisation at a later stage. It
24
It should be noted that EFSA may engage the same vendor of the PPI by defining specific criteria within the PCP call, or by
only opening the PCP call to selected vendors.
25https://efsa.force.com/competentorganisations/s/recordlist/CompetentOrganisation__c/00B1v000009LqfIEAS
www.efsa.europa.eu/publications
41
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
is important to note that, in a PCP, suppliers usually retain IPR ownership rights, while procurers keep some
use and licensing rights.
Finally, opening a Contest would be an opportunity to reinforce synergies among the EFSA community. In
addition, it would enhance the reputation of EFSA about its mission and strategy, including key value such
as openness and innovation.
Challenges
In general, considering the low level of readiness of the solutions, risks related to the loss of control on the
quality and performance of the activities might emerge with a grant, PCP or Contest, together with a higher
investment in development activities and tests, which could require more time and effort than expected.
At the same time, grants would restrict the call to a limited number of stakeholders, considering, in fact, only
organisations within Art. 36, entailing the risk of not finding the right proposals of the solution to be adopted
by EFSA. Indeed, the estimation for consequently opening both the procedures would require at least a year
and half.
The entire process is represented graphically in Figure 16 below.
Figure 16 - Vendor selection and procurement management process.
In the context of this project, we have applied this approach to the 10 prioritised AI use cases of the evidence
management process, as output of the vertical recommendation, relying on the market scouting carried out during
the data collection activities (section 4.2). However, EFSA will be able to apply this methodological framework to
select the best vendor scenario in all use cases, including those that were not prioritised (see long list of use cases
in sub-section 2.4.2).
3.3.3.
Potential grounds for collaboration
As mentioned in the previous sections, collaboration opportunities throughout the AI lifecycle are multiple. In order
to better assess these opportunities and reflect on who could be the most relevant partners with the most promising
solutions, two exercises are presented in the following sections: firstly, the prioritisation of institutional partners
based on a set of criteria and, secondly, a presentation of an overview of solutions that are currently commercialised,
under development or in research phase, managed by academic, industrial and institutional stakeholders, for
each prioritised use case.
Going into more detail, the prioritisation of partnerships (sub-section 3.3.3.1) aims to assess, based on the
information collected in the data collection activities, the potential grounds for collaboration in the context of the
www.efsa.europa.eu/publications
42
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
adoption of AI in evidence management. We have performed this exercise with the institutional type of stakeholders,
as collaboration with them does not have to go through vendor management, but can be oriented and negotiated.
The approach can be applied to any institutions, and we have considered those that, according to data collection,
have an interest in adopting AI in the evidence management process. Article 36 organisations are not present
because the literature review and survey activities did not gather relevant/sufficient information.
On the other hand, the overview of solutions developed by industrial, academic and industrial players – clustered
per specific prioritised use case (sub-section 3.3.3.2) - aims to provide a comprehensive overview of stakeholders
that could collaborate with EFSA through different modalities such as procurement, innovation procurement and
grant.
3.3.3.1.
Assessment of collaboration opportunities with other Institutions
European Institutions, National Authorities and International Organisations, mainly engaged through the interviews,
have been prioritised based on two criteria with the same weight:
1) Common needs/strategy: This criterion takes into account whether the organisations under scope have
developed a strategy/roadmap that focuses on AI, whether they have the same needs and interest in relation
to the development of AI in Evidence Management, or whether they have the same processes in EM with
prioritised areas that require intervention.
2) Potential for collaboration: This criterion evaluates the ease of collaboration with the organisations,
taking into consideration many factors such as whether the organisation under scope has already
collaborated with EFSA, or if it is currently collaborating, and whether the organisation is an EU institution
or an Article 36 organisation.
Both criteria can have a 1-to-3 score (low, medium, high).
Table 10 presents the prioritisation of institutional stakeholders on the basis of the criteria previously mentioned. In
addition, feedback provided during the interviews and the workshop has been taken into consideration. Potential
collaboration models for each organisation are also provided. For more detailed information concerning the
description of the initiatives and solutions, see Table 3 in sub-section 3.1.1.
Table 10 - Institutional organisation prioritisation.
Institution
CDT
EUIPO
EMA
50%
50%
Common
Strategy/
Needs
Potential for
collaboration
3
3
3
www.efsa.europa.eu/publications
3
3
3
Ranking
Additional information
3
Based on the evidence presented in section 1.1.1, there are grounds for
collaboration for a specific use case (automatic text summarisation),
which could be carried out through joint procurement or a cooperation
programme. Feedback provided during the interviews and the workshop
was positive.
3
Based on the evidence presented in section 1.1.1, there are grounds for
collaboration for some use cases (automatic text summarisation and
automated table extraction). EFSA and EUIPO could cooperate through
joint procurement or a cooperation programme. Feedback provided
during the interviews and the workshop was positive.
3
Based on the evidence presented in section 1.1.1, there are grounds for
collaboration due to applications of AI for text data structuring. EFSA
and EUIPO could cooperate through joint procurement or a cooperation
programme. Feedback provided during the interviews and the workshop
was positive.
43
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Institution
50%
50%
Common
Strategy/
Needs
Potential for
collaboration
Ranking
Additional information
Based on the evidence presented in section 1.1.1, on past
collaborations between the two organisations, and on the fact they both
deal with similar fields, there is room for collaboration, which could take
place through a grant. Feedback provided during the interviews and the
workshop was positive.
VKM
3
3
3
ENISA
3
2
2.5
Based on the evidence presented in section 1.1.1, ENISA is quite
interested in developments concerning AI and ML, but a specific use
case of common interest has not been found. Further alignment would
be needed to work together in a joint AI project.
2
Based on the evidence presented in section 1.1.1, EPA has worked on a
tool that presents functionalities that resemble the use case abstract
screening. In addition, EPA and EFSA work in very linked fields
(environment and food safety). The challenge could concern the means
to cooperate with a non-European Agency in concrete joint projects,
therefore further strategic alignment would be needed.
2
Based on the evidence presented in section 1.1.1, FRA seems to focus
more on the impact of AI for fundamental rights than to the application
to specific use cases within the organisation's processes. Strategic
alignment is anyway important to allow EFSA to advance with AI
adoption taking into account FRA's guidelines. Feedback provided
during the interviews and the workshop was positive.
2
Based on the evidence presented in section 1.1.1, the JRC has different
research lines on AI, but seems not to be working on AI application
within its processes. Strategic alignment with this institution is anyway
important to eventually find grounds for cooperation in the future.
Feedback provided during the interviews and the workshop was
positive.
2
Based on the evidence presented in section 1.1.1, AMNH has a high
degree of alignment concerning AI in EM (a representative from the
organisation presented a tool that automates SLR and performs
keywords identification). On the other hand, being an organisation that
operates in the US, working on common projects would be more
difficult.
1.5
Based on the evidence presented in section 1.1.1, OECD has different
research lines on AI, but it has not developed any roadmap or solution
dealing with EM. Strategic alignment in the field of AI could be
established, to better evaluate if grounds for more specific collaboration
arise in the future.
1.5
Based on the evidence presented in section 1.1.1, FAO seems to focus
more on the use of AI in agriculture (for projects in the field) rather
than in internal processes. Strategic alignment could be established, to
better evaluate if grounds for more specific collaboration arise in the
future.
EPA
FRA
JRC
American
Museum of
Natural
History
OECD
FAO
3
1
1
3
1
1
1
3
3
1
2
2
In Figure 17 a prioritisation matrix with institutional stakeholders is presented, with organisations ranked according
to the criteria previously presented.
www.efsa.europa.eu/publications
44
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 17 - Prioritisation matrix of institutional stakeholders.
3.3.3.2.
Collaboration opportunities per prioritised use case
This section provides EFSA with a view on the relevant academic, industrial and institutional stakeholders with whom
EFSA could collaborate through different methods and means for the adoption of AI technology in Evidence
Management.
The table below lists all identified solutions during the data collection activities (desk research, survey, interviews
and the workshop) for each prioritised use case, together with their respective developer organisation’s name and
technology readiness level based on the available information. Low readiness solutions refer to those solutions
that are currently in a research phase, while medium readiness solutions refer to those that are currently under a
development phase and, finally, high readiness refers to those solutions that are already commercialised, or ready
to be used.
Table 11 - Collaboration Opportunities per Prioritised Use Case.
Prioritised use case
Data
Collection
Terminology Assessment
SLR Keywords identification
Organisation
Solutions offered
Octopai
Foundation for Research
Technologies - Hellas
Seven Past Nine
Universite de Tours
Uninova
University
of
Science
Technology Bannu
Arab Academy for Science
University of Manchester
Scipilot
SGS Digicomply
Iris.ai
Sciome
EMA
Risklick AG
University of Edinburgh
www.efsa.europa.eu/publications
45
Octopai
and
and
Readiness level of
solution
H
VisTA
H
Seven Past Nine
Heterotoki
Research Project
Homogeneous
Ontology
Mapping
Word Encoding Method
HYPHEN
Syras
Search Engine Results Page
Iris.ai
Swift Review
Internal tool
Risklick AG
Automated
Systematic
Search Deduplicator
H
H
M
M
M
L
H
H
H
H
H
H
M
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Prioritised use case
Organisation
Solutions offered
Agriculture and Agri-food Canada
TU München
SLR
Relevant
Screening
Abstract
Sciome
Evidence Partners
Seven Past Nine
University of Manchester
SLR
Literature
duplication
De-
Text Summarisation
Rayyan
AMNH
VKM
Eppi-Centre
Pico Portal
Sciome
Evidence Partners
Risklick AG
Iris.ai
ASReview
Research Screener
Robot Reviewer
CDT
EUIPO
University of Edinburgh
Agriculture and Agri-food Canada
Allegheny College
Macquarie University
Rockfeller University
SLR Literature
Clustering
Findings
Gujarat Technological University
Iris.ai
Innovamol Consulting
Eppi-Centre
Scipilot
SGS Digicomply
Sciome
Evidence Partners
Risklick AG
Crème Global
LibrAlry
Seven Past Nine
University of Edinburgh
Agriculture and Agri-food Canada
www.efsa.europa.eu/publications
46
Exploring
Identifiers
of
Research Articles Related to
Food and Disease Using
Artificial Intelligence
Integrative Approach for
Automated
Literature
Reviews
Swift Review
DistillerSR
Seven Past Nine
Systematic review workload
reduction through certaintybased screening
Rayyan
Colandr
Internal tool
Eppi-Reviewer
Pico Portal
Swift Review
DistillerSR
Risklick AG
Iris.ai
ASReview
Research Screener
Robot Reviewer
Internal tool
Internal tool
RegEX
Exploring
Identifiers
of
Research Articles Related to
Food and Disease Using
Artificial Intelligence
Automated
text
summarisation solution
Automated
text
summarisation solution
Automated
text
summarisation solution
Encoder-Decoder
Iris.ai
Innovamol
Eppi-Reviewer
Syras
Search Engine Results Page
Swift Review
DistillerSR
Risklick AG
Crème Global
LibrAlry
Seven Past Nine
RegEX
Exploring
Identifiers
of
Research Articles Related to
Readiness level of
solution
M
L
H
H
H
L
H
H
H
H
H
H
H
H
H
H
H
H
H
H
M
M
M
M
M
M
H
H
H
H
H
H
H
H
H
H
M
M
M
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Prioritised use case
Organisation
Solutions offered
University of Piraeus
Universität Siegen
University of California
University of Manchester
University of Manchester
Quality Improvement of the
Scientific Output
Expert Pool Identification
Expert Selection
GINGER
GradePro
ProWritingAID
National Taiwan University
University
Politehnica
of
Bucharest
Harbin Institute of Technology
Changchun University
Pomato
Department
of
Scientific
Research, Visibilia SP
Ramrao
Adik
Institue
of
Technology
University of Electronic Science
and Technology of China
Textkernel
MosaickTrack
Springer Nature
Ideal
Pomato
Leoforce
IMT Atlantique
Department
of
Scientific
Research, Visibilia SP
Ramrao
Adik
Institue
of
Technology
University of Electronic Science
and Technology of China
Agriculture and Agrifood Canada
Comparable
Detection
Appraisals
Wageningen
University
&
Research Information Technology
University of San Francisco
Food and Disease Using
Artificial Intelligence
Classification
of
dermatological images using
advanced
clustering
techniques
Clustering approach
Machine learning to advance
synthesis
RobotAnalyst
Systematic review workload
reduction through certaintybased screening
GingerSoftware
GradePro
ProWritingAID
DISA
ReadME
Readiness level of
solution
M
M
M
M
M
H
H
H
M
M
Write-righter
Writing Evaluation Model
Pomato
M
M
H
Job Recommendation
M
Job Recommender System
M
Machine Learned ResumeJob Matching
Textkernel
MosaickTrack
Reviewer Finder
AI for Recruiting
Pomato
Arya
Textkernel
M
H
H
H
H
H
H
M
Job Recommendations
M
Job Recommender System
M
Machine Learned ResumeJob Matching
Exploring
Identifiers
of
Research Articles Related to
Food and Disease Using
Artificial Intelligence
Automatic
Systematic
Literature Review
Critical Appraisal Tools
M
M
M
M
Concerning the prioritised use cases, Expert pool identification and Comparable Appraisals Detection, few developed
and commercialised solutions were identified. These were mainly developed by small software companies and
academic organisations. On the other hand, several solutions are being developed by industry and academic
organisations, whereas no solutions provided by institutional organisations can be associated to these use cases.
www.efsa.europa.eu/publications
47
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Taking into consideration the information provided above, and on the basis of the methodology provided in subsection 3.3.2.1, due to the low number of commercialised solutions and the high number of medium and low
readiness solutions for these uses cases, EFSA could launch a grant or a PCP to finance R&D activities for the design
and implementation of AI solutions that respond to its needs, thus resolving problems that are not addressed by
solutions available on the market.
On the other hand, for the use cases SLR Keywords identification, SLR Literature De-duplication, Text Summarisation,
SLR Literature Findings Clustering, Quality Improvement of the Scientific Output and Expert Selection, Data Collection
Terminology Assessment, SLR Relevant Abstract Screening, several developed and commercialised solutions were
identified, mainly developed by small software companies and academic organisations. In addition, for these use
cases, several solutions still under development or in their research phase have emerged, mainly managed by
academic organisations.
Taking into consideration the large number of commercialised solutions for these use cases, and following the
methodology provided in sub-section 3.3.2.1, EFSA could collaborate with these organisations through different
cooperation/models – procurement, innovation procurement or grants, based on the use case coverage level (i.e.,
how much the solution is able to fulfil the needs and requirements associated to the respective evidence management
step). The details and outputs of the use case coverage assessment are presented in section 4.2.
www.efsa.europa.eu/publications
48
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
4.
Recommendations for projects
This chapter outlines and prioritises a list of recommendations to develop a harmonised approach on the
implementation of AI methods in the evidence management phase of risk assessment. Two types of
recommendations have been developed:
•
Horizontal recommendations, presented in section 4.1, cover foundational elements enabling a
systematic and smooth adoption of customised AI-based solutions within the evidence management process,
building the capabilities to adopt and implement AI within both the risk management process and within the
organisation as a whole. They address several topics such as the infrastructure, governance and skills, that
need to be in place to unlock the adoption of AI in specific use cases (vertical recommendations) and achieve
EFSA’s vision and Strategy 2027 of adopting human-centric AI within the evidence management phase of
risk assessment.
•
Vertical recommendations, presented in section 4.2, propose the optimal vendor management scenario
and sourcing option (according to the approach described in sub-section 3.3.3.1) to acquire or develop AI
solutions for each of the prioritised use cases of evidence management.
These recommendations are designed as single projects that EFSA can decide to carry out – either in-house or
through outsourcing– to improve the evidence management process, achieve its 2027 strategic objectives and
prepare the organisation to embed AI technologies within its processes. In order to facilitate decision making, we
have carried out a time and budget estimation and a prioritisation exercise in section 4.3, aimed at understanding
which actions (both horizontal and vertical) can be performed in parallel, and which are instead dependent on the
completion of other actions for their proper initiation.
Figure 18 - Vertical and Horizontal recommendations link.
These recommendations can be associated to different phases of the AI lifecycle (sub-section 3.3.2), as shown in
Figure 19 below.
www.efsa.europa.eu/publications
49
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 19 - Link of Recommendations with AI Lifecycle Phases.
4.1.
Horizontal Recommendations
To achieve its vision of increasing the accessibility and the breadth of the body of evidence, as well as to apply
human-centric AI in close coexistence with existing human expertise by 2027, EFSA should bolster its preparedness
by taking steps towards better data management. In particular, data management should be appropriate, not only
for the types of data currently processed, but also for those data types planned to be processed in the future.
Adopting a fit-for-purpose data infrastructure, tools enabling Big Data and AI development, and a data governance
framework that addresses AI, while taking into consideration people, processes, and policies, are foundational
elements towards a more robust structure, aligned with EFSA’s organisational goals.
Horizontal recommendations include transversal considerations on which EFSA can build its readiness for an effective
AI adoption, not only in evidence management, but also across the organisation’s units and teams. While EFSA may
decide to adopt some ready-made solutions to address specific needs and use cases (see Vertical Recommendations),
a Design-to-Value approach is proposed, whereby quick wins are obtained in parallel with capability development,
such that EFSA is in a position, in the future, to build customised AI tools that will serve its own needs in a more
tailored way, ensuring maximum impact for the organisation.
Based on the Gap Analysis presented in section 3.2, recommendations are presented to address different dimensions
of AI capability development: Technology & Infrastructure, Data & Information, Organisation & Governance, Process
& Integration and Culture & Talent (see figure 20 below).
To this end, the following initiatives and projects to support EFSA in achieving its vision are recommended:
Figure 20 - Dimensions and horizontal recommendations.
www.efsa.europa.eu/publications
50
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Recommendation 1: Develop or adopt existing ontologies for domains of relevance
for EFSA
Description and scope
One of the key gaps that has emerged from the series of data collection activities conducted is the need to develop
(or adopt) tailor-made ontologies for the specific domains that are related to EFSA's work.
Several of the use cases examined rely on the use of Natural Language Processing (NLP) and text mining, and require
the use of knowledge sources for the purpose of understanding, extraction and retrieval from unstructured text.
Simpler tasks, such as named entity recognition (NER), which can be the basis of Automatic Text Summarisation,
and SLR Literature Findings Clustering, can be simply based on terminologies. However, more complex tasks that
rely on the identification of concepts (such as SLR Abstract Screening and Comparable Appraisals Detection), require
knowledge bases in the form of structured ontologies that encode multiple lexical representations in natural
language.
An ontology formally represents knowledge as a set of concepts within a domain, as well as the relationships between
these concepts (e.g., concepts that mean the same thing but may be expressed differently). In essence, an ontology
includes a vocabulary of terms and specifications of their meaning. This includes definitions and an indication of how
concepts are inter-related, which collectively impose a structure on the domain and constrain the possible
interpretations of terms26.
For EFSA, the development of accurate and complete domain-specific ontologies is an essential element to support
the implementation of AI technologies to automate and improve several steps within the evidence management
process. Indeed, the majority of AI-based tools/methods to be potentially adopted (even some of those
commercially available) require customisation, as they need to be trained on a comprehensive subjectspecific ontology. This will improve the effectiveness of both supervised and unsupervised AI tools.
Key considerations for ontologies development
A key consideration for NLP is that an ontology needs to be complete with respect to the entities represented, as
well as the relationships between them and natural-language synonyms27 For example, to automatically retrieve
documents that discuss genotoxic carcinogens, an NLP system requires an existing ontology that has an identifier
for genotoxic carcinogens that is correctly linked to the natural language terms (e.g., ‘genotoxic carcinogens’,
26
Gruber,T. (2008), Ontology, Encyclopedia of Database Systems
Kaihong Liu, William R. Hogan, Rebecca S. Crowley (2011),Natural Language Processing methods and systems for biomedical
ontology learning, Journal of Biomedical Informatics, Volume 44, Issue 1, pp 163-179.
27
www.efsa.europa.eu/publications
51
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
‘mutagenic’, ‘carcinogenic’, etc.), and also has relationships with additional identifiers for other entities, such as
genotoxic carcinogens interact with DNA. As a consequence, when an ontology lacks a representation of an entity,
a particular term associated with the entity, or some of its particular relations, the quality of an NLP system based
solely on such ontology will be negatively affected. Therefore, the lack of any representation of an entity inhibits the
detection of such entity. Equally, in the event that a given document uses a synonym to refer to a concept, the lack
of a synonym present in the ontology would prevent the recognition of the entity. Lastly, the lack of an entity’s
relationship might prevent and identification of the right answers to specific request as, for example, How do
genotoxic carcinogens cause cancer?
In more detail, an Ontology can be defined as a set of Individuals, Concepts (Classes), Attributes, Relations,
Instances and Axioms:
•
Individuals: represent the basic, "ground level" components of an ontology. The individuals in an ontology
may include concrete objects such as people, animals, tables, automobiles, molecules, and planets, as well
as abstract individuals including numbers and words;
•
Concepts (Classes): represent a set or class of entities, or `things/objects’ within a domain (e.g., Protein is
a concept within the domain of molecular biology), which can be of two types:
•
o
Primitive concepts: are those which only have necessary conditions (in terms of their properties)
for membership in a given class28;
o
Defined concepts: are those for which the use of a description is needed to be classified under a
class29.
Attributes: include aspects, properties, features, characteristics, or parameters that objects (and classes)
can have;
•
Relations: represent the interactions between concepts or concepts’ properties, and can be of two types:
o
Taxonomy: organising concepts into sub-concept tree structures. A common form of taxonomy is
the specialisation relationship (e.g., Enzyme is a kind of Protein, which in turn is a kind
of Macromolecule);
o
Associative Relationships: relating concepts across tree structures. A common form is the
nominative relationship that describes the names of concepts (e.g., Protein has Accession Number,
in the context of bioinformatics)
•
Instances: are describe as real-world examples represented by a concept.
•
Axioms: are considered as a set of rules used to limit the values within a class in a specific domain of
application.
The process of ontology development is still to a large extent manual. For example, identifiers, their synonyms and
relationships are still added one by one manually. The investment in ontology development in other areas has been
huge, with examples such as the Gene Ontology (GO), a Consortium funded by the National Human Genome
Research Institute30 , and the National Center for Biomedical Ontology (NCBO)31, amongst others.
28
For example, a globular protein is a kind of protein with a hydrophobic core, so all globular proteins must have a hydrophobic
core, but there could be other things that have a hydrophobic core that are not globular proteins.
29 For example, Eukaryotic cells are kinds of cells that have a nucleus. Not only does every eukaryotic cell have a nucleus, but
every nucleus also containing a cell is eukaryotic.
30 National Institutes of Health. Research Portfolio Online Reporting Tools (RePORT), 2010. Available from:
http://projectreporter.nih.gov/project_info_history.cfm?aid=7941562&icde=2611544.
31 BioInform. Stanford’s Mark Musen on the New National Center for Biomedical Ontology, 2005. Available from:
http://www.genomeweb.com/informatics/stanford-s-mark-musen-new-national-center-biomedical-ontology.
www.efsa.europa.eu/publications
52
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
It is worth noting that a significant set of resources is already available to provide a foundation for the development
of ontologies of relevance to EFSA’s work such as:
•
The Open Biological and Biomedical Ontology (OBO) Foundry32, an open-source initiative that aims to
develop a family of interoperable ontologies, which are both logically well-formed and scientifically accurate.
Through this initiative, participants contribute to the development of an evolving set of principles including
open use, collaborative development, non-overlapping and strictly-scoped content, and common syntax and
relations, based on ontology models that work well, such as the Gene Ontology (GO).
•
The ECOTOXicology Knowledgebase (ECOTOX)33, a comprehensive, publicly available Knowledgebase
providing single chemical environmental toxicity data on aquatic life, terrestrial plants and wildlife.
•
The OpenTox ontology34, an ontology of toxicological endpoints, used for the analysis and prediction of
toxic events.
•
The Gene Ontology (GO)35 is the world’s largest source of information on the functions of genes. This
knowledge is both human-readable and machine-readable, and is a foundation for computational analysis of
large-scale molecular biology and genetics experiments in biomedical research.
•
Other research efforts and initiatives that have been focused on the creation and testing of ontologies for
toxicologic data, by merging a wide range of public bio-ontologies relevant to toxicology36, including the
QSAR toolbox, eTOX, Pistoia Alliance, ToxWiz, Virtual Liver, EU-ADR, BEL, ToxML, and Bioclipse37.
In addition to the already available open-source ontologies that EFSA can take advantage of, Protégé38, a free,
open-source ontology editor and knowledge management system developed by the Stanford Centre for Biomedical
Informatics Research at the Stanford University School of Medicine that can be leveraged for user-friendly ontology
development should be considered too. Protégé, in fact, allows the definition of classes, class hierarchy’s variables,
variable-value restrictions, and the relationships between classes and the properties of these relationships. Overall,
it is considered as one the most widely deployed modelling tools, and the leading ontology editor. Protégé allows the
definition of classes, class hierarchy’s variables, variable-value restrictions, and the relationships between classes
and the properties of these relationships.
Recommendation Summary
Objectives
-
The recommendation suggests the development and adoption of well-structured ontologies within EFSA in order to
improve the performance of AI tools to be acquired/developed and used. The adoption of ontology ensures that EFSA
can make use of domain knowledge and scope, rather than the general operational knowledge that is currently being
used.
Challenges
-
32
33
34
35
Given that building a solid and functional ontology depends on the complexity of the problem and the knowledge of
the respective domain, experts within the organisation or externally outsourced must be consulted and involved when
developing the ontology
https://obofoundry.org/
https://cfpub.epa.gov/ecotox/
See http://old.opentox.org/dev/Ontology
http://geneontology.org/
36
Wang RL, Edwards S, Ives C. Ontology-based semantic mapping of chemical toxicities. Toxicology. 2019 Jan 15;412:89-100.
doi: 10.1016/j.tox.2018.11.005. Epub 2018 Nov 20. PMID: 30468866; PMCID: PMC6774258.
37Hardy B, Apic G, Carthew P, Clark D, Cook D, Dix I, Escher S, Hastings J, Heard DJ, Jeliazkova N, Judson P, Matis-Mitchell S,
Mitic D, Myatt G, Shah I, Spjuth O, Tcheremenskaia O, Toldo L, Watson D, White A, Yang C.
38 https://protege.stanford.edu/
www.efsa.europa.eu/publications
53
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
-
Since the ontology development is a challenging and time-intensive task, it should be ensured that domain experts
(potential ontology authors) are aligned about the usefulness and benefits of building and using an ontology, and
specifically, how it can positively affect the performance of the AI tools.
Expected Impact
-
An ontology can streamline the AI training process by automatically tagging, categorising and linking information to
improve performance.
An ontology can enable a smoother data integration and alignment if trained on different data sources.
The Development and adoption of ontologies are foundational elements for building the appropriate infrastructure
and tools to achieve improved data extraction, integration and standardisation to support the use of NAMs in chemical
risk assessment.
The following table provides the SWOT analysis for Recommendation 1
SWOT Analysis
Strengths
-
Opportunities
A number of ontologies are already available in
open-source databases that can be reused by EFSA
to perform pilots and assess the improvement of
the performance of AI tools. Some of these
ontologies have been built via multi-partner
consortia and are fairly complete for specific
domains.
-
Weaknesses
-
An ontology can be a decisive factor in improving
the algorithm of the AI tool, by providing a dynamic
knowledge-based that would be more preferable
than static frameworks.
Threats
No advanced expertise in the ontology engineering
field currently exists within EFSA, which is
responsible for setting the activities that concern
the ontology development process and lifecycle.
-
-
The purpose of the ontology could be
misunderstood, and it could be merely conceived as
a tool to structure data, like taxonomy. As a result,
this may affect the quality of the data within the
ontology, leading to inaccurate training of the AI
tool.
It might be difficult to develop ontologies to cover
the wide spectrum of EFSA’s domains of expertise.
There is a need for maintenance and systematic
updating of the ontology, which is recommended
every two years, depending on how quickly
scientific areas evolve (i.e., how fast new terms and
concepts are developed)
Recommendation 2: Implement a Data Infrastructure for Big Data Analytics & AI
development
Description and scope
In order to develop and adopt an efficient Data Governance framework, EFSA should consider building a consistent
Data Infrastructure. Designing and implementing a fit-for-purpose and scalable data infrastructure with the right
technology and tools should lay the grounds towards a more effective data accessibility, integration and analysis,
which might facilitate the use of AI in the evidence management phase of the risk assessment process and, at the
same time, would also support EFSA towards a holistic adoption of AI and Big Data analytics.
In this sense, this recommendation does not exclusively focus on the scope of evidence management, but it
addresses EFSA as a whole organisation, considering the inputs collected from other EFSA units and the needs
identified in other roadmaps, such as NAMs and PERA.
www.efsa.europa.eu/publications
54
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Going into more details of this recommendation, EFSA needs to implement a Data Infrastructure to enable Big Data
Analytics and AI for the following reasons:
•
The volume of data that EFSA will need to store and process in the future is likely to be too large for a traditional
database.
•
A large part of the data will consist of unstructured data (e.g., textual data) that need to be accurately
transformed for some key activities within the organisation, such as data analysis and reporting.
•
Real-time data collection and processing may be considered in EFSA’s future, especially in the context of an
evolving and more dynamic risk assessment process, and in relation to the future possibility to use external
unstructured data (e.g., from social media and news sites).
There are various data architecture frameworks of different complexities that can be used. Here, we present the
simplest form of data architecture, which contains three fundamental inter-connected layers, each characterised by
specific elements and procedures:
•
Storage Layer: in this layer, data is obtained and ingested from multiple internal and external sources and
then converted into a suitable format to enable a correct analysis. Depending on the type of data, the
appropriate storage model must be used. Furthermore, it is worth mentioning that all of the data coming
from the different data sources and available for analysis are appropriately selected, before moving into this
layer.
•
Process Layer: in this layer, data is read and analysed from the storage layer to produce meaningful
information. Two types of data processing can be distinguished: Batch processing, where processing
happens to a set of data that has already been stored over a period of time in order to obtain detailed
analytics and insights, and Stream processing, where processing happens while data is stored (continuing
flow of data), to quickly generate real time insights through analytics tools.
•
Visualisation Layer: in this layer, comprehensive reports and visualisations to support decision making are
created, based on the output of the process layer.
Indicative Data Infrastructure Blueprint
A blueprint of a data infrastructure is set out below (Figure 21), which is well suited to EFSA's needs in the area of
evidence management within the risk assessment process. The blueprint can also be used to fulfil EFSA's wider needs
around the domain of Big Data and AI across multiple units within the organisation. Specifically, it includes the
different logical components that fit into a big data infrastructure.
It is worth noting that EFSA's final layout may not contain every item in this diagram, on the grounds of specific
organisation's short-term and long-term goals, anticipated types and volumes of data, and the required types of data
analysis.
Figure 21 - Indicative Data Infrastructure for EFSA
www.efsa.europa.eu/publications
55
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Furthermore, depending on the vendor that EFSA may choose for the adoption of an AI-based solution, different
services of the infrastructure may be selected.
In Figure 22 below, the indicative data infrastructure is represented and integrated with the Microsoft Azure cloud
platform's components, which is EFSA's cloud platform currently in use.
Figure 22 - Indicative Data Infrastructure for EFSA based on the Azure cloud platform.
Expanding EFSA’s current cloud platform solution with additional components to fulfil the increasing demand would
lead to time and cost efficiencies.
Below, we provide some additional details and key considerations about the different components of the Data
Architecture that should be considered by EFSA:
-
Data Sources: an important topic of consideration for EFSA is the need to integrate multiple and possibly
disparate internal and external data sources. The topic of data integration has been recognised as a pain
point by multiple stakeholders within EFSA, including the NAMs project team. These data sources may consist
of structured data (e.g., scientific measurements) or unstructured data (e.g., text from papers, other
documentation or external unstructured data sources such as news and social media).
-
Storage Layer - Ingestion/Data Storage: it is important to ensure consistency in the data collection and
process step through the selection of an advanced tool that can store high volumes of large files in various
formats. This kind of storage is usually provided by a system/repository defined as a data lake. Possible
options for implementing this storage include Azure Data Lake Store, or blob containers in Azure Storage.
Most of the big data solutions involve various activities such as repeated data processing operations,
encapsulated in workflows, which transform source data, move data between multiple sources and sinks,
load the processed data into an analytical data store, or push the results straight to a report or dashboard.
To automate these workflows, an orchestration technology such as Azure Data Factory can be used.
www.efsa.europa.eu/publications
56
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
In this regard, the process of automating workflows to perform Extract Transform and Load (ETL) requires
particular attention and consideration from EFSA. ETL is a data integration process that combines, cleans,
transforms, and standardises data from multiple data sources and loads it into a data warehouse or other
target system, in a way that addresses specific business intelligence needs. In particular, multiple EFSA
stakeholders consulted in the previous phases of the project, highlighted the importance of data
standardisation as a pre-requisite and foundation for data analytics and machine learning workstreams.
Process Layer - Analytics/machine learning: it is recommended to select a big and advanced data
solution that will process the data files using long-running batch jobs to filter, aggregate, and prepare the
data for a proper analysis. Usually, these tasks involve reading source files, processing them, and writing
the output to new files. Therefore, possible options include running U-SQL jobs in Azure Data Lake Analytics,
using Hive, Pig, or custom Map/Reduce jobs in an HDInsight Hadoop cluster, or using Java, Scala, or Python
programs in an HDInsight Spark cluster. In addition, Azure Machine Learning Studio can be used to build
and deploy the machine learning models using two approaches: either Python scripts or with Hive tables on
an HDInsight (Hadoop) cluster.
-
Concerning, natural language processing capabilities, these are offered either by Azure HDInsight, with Spark
and Spark Mllib, or by Microsoft Cognitive Services. Regarding the latter, it contains pre-built NLP models
that can be used on free text, whereas the former gives the option to train customised models against a
large corpus of text data.
Microsoft Cognitive Services would be a more appropriate choice for EFSA, since it provides more high-level
NLP capabilities and, in addition, it can be customised to develop use cases like SLR (Systematic Literature
Review) information clustering and text summarisation, without the need for significant coding and
customisation.
Visualisation Layer - Reporting: the goal of most of the big data solutions is to generate insights from
the data stored, through accurate analysis and reporting. To empower users in analysing data, the
architecture may include a data modelling layer, such as a multidimensional OLAP cube or, tabular data
model in Azure Analysis Services. It might also support self-service Business Intelligence, using the modelling
and visualisation technologies in Microsoft Power BI or Microsoft Excel.
-
To conclude, the following table presents a summary of recommendation 2 (Implement a Data Infrastructure to
enable Big Data Analytics and AI), by outlining its objectives, challenges, expected impact, and a final swot analysis.
Recommendation Summary
Objectives
-
Design and adopt a Data Infrastructure that will fulfil EFSA’s current and future data analysis needs within the evidence
management process and across the entire organisation.
Effectively manage data and its flow throughout multiple infrastructural layers within EFSA.
Build the capability to store and process big data from different sources, including unstructured data.
Select a scalable and fit-for-purpose technological solution, such that EFSA’s changing needs can be met with minimal
disruption.
Challenges
-
Storing and processing big data from different sources, e.g., the collection of free-form text documents, is typically
computationally resource intensive, as well as being time intensive.
Without standardised document formats, it can be difficult to achieve consistently accurate results and generate
meaningful insights.
ESFA needs to build the relevant technical skills in-house to adopt and maintain a new potential Big Data architecture.
Data volumes to be stored are growing at a fast pace, and appear in different formats (i.e., structured and
unstructured). Therefore, frequent updates in the data infrastructure technologies and tools may be required.
Expected Impact
www.efsa.europa.eu/publications
57
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
-
Better data management and improved transparency and accessibility to data will empower EFSA users to experiment
with new advanced analyses that will have a positive long-term impact on EFSA operations and decision-making.
-
Foundations in place, thanks to a consistent Data Infrastructure, for the effective development of AI algorithms.
-
Lower maintenance cost and reduction of effort needed.
Reduction of IT-Workarounds.
Acceleration of the responsiveness to changes.
-
The Development and adoption of a Data Architecture for Big Data and AI is a foundational element for building the
appropriate infrastructure and tools to achieve improved data extraction, integration and standardisation to support
the use of NAMs in chemical risk assessment.
The following table provides the SWOT analysis for Recommendation 2
SWOT Analysis
Strengths
-
EFSA has already established a partnership with
Microsoft Azure, which offers the most cutting-edge
cloud platforms in the market.
-
Microsoft Azure offers various complementary and
additional services that can be integrated into the
designed Data Infrastructure.
Opportunities
-
Weaknesses
-
The design of a modern and agile data
infrastructure that can lead to the optimisation of
the data flows throughout the different data layers
from storage to visualisation.
Threats
Security is essential in a data infrastructure. The
development of a data governance framework for
EFSA is essential in order to set data policies and
controls over the data.
Currently, in EFSA, data are stored in multiple silos
(i.e., repositories of data that are controlled by one
department or business unit and isolated from the
rest of an organisation, which makes it hard for
users in other part of the organisation to access and
use the data). Therefore, to ensure the design of a
modern data infrastructure, these data silos should
be broken down.
-
The design of a modern data infrastructure should
not be performed independently, and before the
adoption of a solid AI governance framework.
Otherwise, there would be a risk of not selecting
and implementing the correct compliance and
security tools.
Collaboration with a third-party for the acquisition
of new solutions or services that compose a data
infrastructure is always associated with some
outsourcing risks. For instance, protecting personal
and sensitive data (hosted within the data
infrastructure) can be a barrier that hinders or
delays successful outsourcing. Failure to implement
privacy and data protection correctly can result in
severe legal, economic, and reputational penalties.
Recommendation 3: Implement a Data Governance framework which addresses AI
Description and scope
As outlined in its 2027 strategy, EFSA aims at becoming an AI-enabled organisation39, and challenges such as data
being scattered across the entire organisation in silos and not managed appropriately need to be addressed. Indeed,
a foundational element to the adoption of AI solutions is the management and utilisation of a large amount of
39
EFSA, Theme (concept) paper on Artificial Intelligence in risk assessment
www.efsa.europa.eu/publications
58
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
structured and unstructured data obtained from multiple sources and in multiple formats. AI encompasses the entire
cycle of data capture, data storage, data preparation, and advanced data analytics technologies. Therefore, to
successfully adopt AI, EFSA needs first and foremost to ensure that its data is accessible, relevant, reliable, and
protected. Within this context, the adoption of a Data Governance framework should ensure that data within EFSA
is managed and used correctly and responsibly while minimising data management effort and operational costs,
increasing transparency, and minimising risks, leading to better AI-enabled decision making.
In this sense, this recommendation does not exclusively focus on the scope of evidence management, but it
addresses EFSA as a whole organisation, considering the inputs collected from other EFSA units and the needs
identified in other roadmaps, such as NAMs. Indeed, the Development and adoption of a Data Governance framework
which address AI is a foundational element for building the appropriate infrastructure and tools to achieve improved
data extraction, integration and standardisation, in order to support the use of NAMs in chemical risk assessment.
Specific activity
A Data Governance (DG) framework essentially describes policies, roles, standards, and metrics needed to
continuously improve the use of data that ultimately enables an organisation to achieve its goals.
Some of the questions a DG framework should answer are set out in Figure 23 below:
Figure 23- Questions that a DG framework answers
Figure 24 below shows the structure of an indicative framework for the establishment of DG which can provide the
answers to the above questions:
Figure 24 - Indicative DG Framework incorporating Tools & Technology, Process & Operating Model and People &
Culture.
www.efsa.europa.eu/publications
59
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
By answering these questions, DG ensures the quality and security of an organisation's data reservoir since it clearly
defines who is responsible for what, and what actions should be taken (using what methods). In addition, addressing
these questions would be the basis for a good Responsible AI framework, as data bias and data management are
the keystones of the overall ethicality of an AI project (this is covered in more details in horizontal recommendation
5).
In the figure below (see Figure 25), a roadmap is set out including potential steps which EFSA should take for the
development and, consequently, adoption of a DG framework:
Figure 25 - DG roadmap with increasing levels of maturity.
Based on the current maturity of the DG development, EFSA should start with Technology-Based DG, focusing initially
on the aspects of governance affecting tools, technology and data management before advancing to a more complete
DG framework (the regulatory-based governance is an optional step), which would also cover organisational
structure, processes and people across the organisation.
It is noted that, for organisations like EFSA that are planning to leverage AI, further components need to be taken
into consideration and aligned with the DG framework. These are: Development & Operations Management (DevOps)
and Trustworthy AI. For DevOps, DG is its backbone, as it sets the processes, structures and roles to ensure that
qualitative data are provided for the model development, and that data lineage and continuity is preserved (see
Horizontal Recommendation 4). Moreover, setting up a DG framework that reflects the key tenants of Trustworthy
AI (see Horizonal Recommendation 5) should contribute to a fairer adoption and use of AI technology.
www.efsa.europa.eu/publications
60
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Recommendation Summary
Objectives
-
In general, it is recommended to EFSA to increase the control over its data assets to monitor, control and manage
them easily and more effectively, and to provide holistic, and qualitative data when needed.
EFSA should consider adopting a detailed data governance plan in order to establish roles, processes and policies for
improving EFSA's operational efficiencies.
As a parallel consequence for the adoption of a DG framework, EFSA will set the foundations for Trustworthy AI, and
will ensure that the AI decision-making process is explainable, transparent and fair.
Challenges
-
The establishment of a DG may require several organisation's units to get involved, hence it can cause misconceptions
over data governance.
-
Third party and other data that the organisation doesn't control.
Difficulties with access to data.
Fragmented or siloed teams.
Expected Impact
-
An effective DG framework adoption might support EFSA to effectively reduce risks, improve trust, obtain efficiencies
and adopt AI in a more trustworthy and transparent way.
-
An effective DG framework adoption supports a better understanding of the different data types that are available
and of their quality, leading to the possibility to reveal and realise the maximum value from data (e.g., identification
of new AI projects, or gaps to be bridged).
-
The Development and adoption of a DG is a foundational element for building the appropriate infrastructure and tools
to achieve improved data extraction, integration and standardisation to support the use of NAMs in chemical risk
assessment
The following table provides the SWOT analysis for Recommendation 3
SWOT Analysis
Strengths
-
Weaknesses
Set clear rules for managing the entire data lifecycle
(including data capture, data storage, data
preparation, and advanced data analytics
technologies).
-
Ensure the accuracy, availability, integrity, and
confidentiality of data by defining and following a
data governance framework.
-
Increase data processing efficiency by reducing the
lengthy coordination process.
-
-
Opportunities
-
Threats
Increase confidence in data, through qualityassured standard based on the Data Governance
Framework.
-
-
www.efsa.europa.eu/publications
Consider a time and effort consuming procedure to
tailor the Data Governance framework to the
specific needs and requirements of EFSA.
Resistance to change for such a pervasive action is
to be foreseen, change management activities must
be considered.
61
Potential difficulties in engaging EFSA stakeholders
through new practices and ways of working to
support the adoption of a Data Governance
Framework.
Misalignments among organisation units to establish
the Data Governance Framework.
DG would evolve with the organisation’s needs, so
continuous monitoring and maintenance to adapt to
change is required.
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Recommendation 4: Establish Development & Operations Management (DevOps)
Description and scope
An important component that can help the implementation of a robust Data Governance (DG) framework is the
adoption of a Development & Operations Management (DevOps). DevOps is a set of practices, tools, and cultural
philosophy that automate and integrate the processes between data science/analysis teams ( development) and IT
teams (operations),40 increasing an organisation’s ability to deliver applications and services at high velocity.
The function of DevOps is to enable teams to collaborate and communicate more effectively and efficiently, and to
ensure continuity and reproducibility in any model and product developed within EFSA.
We recommend that EFSA establishes a DevOps practice, and therefore adopts an ecosystem for building, testing,
deploying, and maintaining the enterprise data science and machine learning models, as well as other analyses and
tools, in production environments. The tangible benefits that DevOps can provide EFSA are included in Figure 26:
Figure 26 - DevOps Benefits.
Specific activities
Our experience has shown that people, processes and technology are all critical elements to establish an effective
DevOps practice. Therefore, for a successful DevOps implementation, EFSA should address all these three elements
in parallel as follows:
•
People: Design of an ad-hoc change management program for the whole organisation - or specific
department which would promote sufficient training, education programmes communications actions,
coaching sessions, and exemplary management, with a strong emphasis on supporting the DevOps delivery
team and management;
• Processes: Adherence to DevOps design principles;
• Technology: Design of an independent modular architecture, with uniform and fit-for-purpose automation
software;
Automation software for DevOps implementation allow tasks to be performed with reduced human assistance,
facilitating the tracking of the model development’s progress. In addition, this software provides quick feedback
loops between operations and development teams (data scientists in this case), so that iterative updates can be
deployed faster to applications in production. The basic components of these tools include logging (mainly model
and experiment parameters), version control and use of dashboards for monitoring activities and performance. Git,
Docker, SonarCube and Kubernetes are some of the most widely used open-source DevOps tools.
Within the Microsoft Azure environment, Azure DevOps provides services for allowing teams to plan work, collaborate
on code development, and build and deploy applications. Azure DevOps supports a culture and set of processes that
bring developers, project managers, and contributors together.
40
Under a DevOps model, development and operations teams are no longer “siloed.” Sometimes, these two teams are merged
into a single team, where the engineers work across the entire application lifecycle, from development and test to deployment
to operations, and develop a range of skills not limited to a single function.
www.efsa.europa.eu/publications
62
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Recommendation Summary
Objectives
The main objectives for Recommendation 4 can be summarised as follows:
-
It is recommended that EFSA adopts a DevOps, with the aim of building a series of practices and a culture to promote
collaboration between development and operations teams within EFSA.
-
In addition, EFSA should consider establishing a DevOps to increase trust, accelerate model development and
deployment, build traceability and continuity, and develop the ability to solve critical issues quickly, and better manage
unplanned work.
Challenges
-
Obstacles in establishing an efficient DevOps might be identified within its three main elements, as follows:
o
o
o
People: There is a need to build a team with various and advanced skills, such as knowledge of the machine
learning model lifecycle, and knowledge of the responsibilities that the data/machine learning/software engineer
must have. For this scope, there might be an unwillingness or inability to broaden scope and skill-sets to those
required.
Processes: Different employees may end up working for multiple teams, and teams may become too large when
trying to merge development and operations processes.
Technology: There might be issues related to interdependencies in responsibilities and Technology
interoperability and integration.
Expected Impact
-
Elimination of manual hand-offs to reduce time for deployment and human errors
Modernisation of systems to avoid bottlenecks
Increase in transparency and improved collaboration
Cost reduction, since no work is duplicated and errors are reduced
The following table provides the SWOT analysis for Recommendation 4
SWOT Analysis
Strengths
-
Weaknesses
EFSA already uses Microsoft Azure, which provides
the Azure DevOps.
Open-source tools are available, which can kickstart the DevOps activities with minimal investment.
-
Opportunities
-
The organisational challenges with bringing the
teams together may cause delays or resistance to
adoption.
Threats
Improve collaboration, reduce cost, reduce errors
and develop an all-around better culture.
Focusing only on the adoption of a DevOps tool
instead of tackling the issue from a people
perspective might lead to failure.
Recommendation 5: Adopt a Trustworthy AI Framework
Description and scope
With a view to EFSA's vision of scaling responsible and human-centric AI by 2027, the adoption of a Trustworthy AI
framework, which can ensure that AI challenges are addressed from an ethical, social and legal point of view, is
highly recommended. As we saw in section 3.1, the adoption and use of AI poses several challenges and risks to
EFSA regarding ethics and compliance with the data protection Regulation (EU GDPR), which need to be addressed
by EFSA to establish trust with their stakeholders. In fact, these risks and challenges represent great societal concerns
for EFSA's main stakeholders (e.g. NGOs). Furthermore, it is worth mentioning the guidelines that the European
Commission set in the Proposal for a Regulation of The European Parliament And Of The Council Laying Down
www.efsa.europa.eu/publications
63
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Harmonised Rules On Artificial Intelligence (AI Act) 41, which aim to strengthen AI update across private and public
organisations, while addressing AI risks, in order to ensure that AI systems are based on rules that safeguard the
public sector's operations and fundamental rights. To take the draft of the AI Act into consideration when developing
the Trustworthy AI framework would ensure that the AI systems adopted by EFSA are safe and legally compliant,
and respect fundamental rights, as well as European Union values. In addition, by adopting the framework, EFSA
would increase transparency towards both its internal and external stakeholders about how AI and data are used,
what are the fundamental decisional processes that are followed, and how the potential risks are identified and
addressed.
Trustworthy AI framework design and structure
The suggested Trustworthy AI framework consists of a series of rules, governance mechanisms and tools aimed at
guiding the design, development, and deployment of trustworthy AI tools. When efficiently adopted, the Trustworthy
AI framework empowers organisations to work with agility and enhance trust towards the use of AI solutions. As
organisations deploy AI to automate existing or new workflows (either with the aim of reducing time and/or effort
consumption and/or increase efficiency and results), the framework enables users to trust AI outcomes at every step
of the AI lifecycle (i.e., design, development, deployment, operating and monitoring phases), and also ensures
enhanced transparency on AI use to externals stakeholders.
In order to do so, the Trustworthy AI framework considers the requirements proposed in the draft of the AI act and
translates42 them into recommendations and suggested actions (see Table 22 below). The AI Act presents a balanced
and proportionate regulatory approach to AI, that tries to provide minimum necessary requirements to address the
risks and problems linked to AI through a risk-based approach. Indeed, the AI Act describes four risk levels for every
AI solution and provides different types of requirements for each of them.
Table 12 below illustrates how the most important requirements per risk level of the proposed AI Act map towards
the Recommendations for EFSA presented in this document, as well as the corresponding specific actions.
Table 12 - Mapping between AI Act requirements and recommendations and actions for EFSA.
AI Act Risk Level
AI Act Requirements
# Recommendation
Suggested actions
Minimal or no risk
Create and implement a Code of Conduct, which
may also include voluntary commitments related, for
example, to environmental sustainability, accessibility
for persons with disability, stakeholders’ participation
in the design and development of AI systems, and
diversity of development teams.
Recommendation N.1 Adopt a
trustworthy AI framework
Action 1 “Develop a
Code of conduct”
Limited Risk
Ensure transparency for system which interact with
humans, detect emotions or determine association
with (social) categories based on biometric data, or
(iii) generate or manipulate content (‘deep fakes’).
No specific recommendation as these requirements are very
specific and asked only for some type of AI applications.
Establish and
processes.
Recommendation N.1 Adopt a
trustworthy AI framework
High Risk
implement
risk
management
Adopt data and data governance requirements on
training, validation and testing data
Recommendation N. 3
Governance framework43
Action 2 “Create a Risk
Management
Framework”
Data
41
European Commission, Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial
Intelligence and Amending Certain Union Legislative Acts, (2021) available at
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206
42 It should be noted that the fourth risk concerns applications with an unacceptable risk, which are not allowed to be used
43 A data governance framework can be considered as a methodology to implement specific principles of Trustworthy AI
concerning data management. For more information, see Recommendation 3 “Data Governance framework”.
www.efsa.europa.eu/publications
64
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
AI Act Risk Level
AI Act Requirements
# Recommendation
Establish documentation and design
features (traceability & auditability)
•
•
•
logging
Ensure
appropriate
certain
degree
of
transparency and provide users with
information
Ensure human oversight
Ensure
robustness,
accuracy
and
cybersecurity
Suggested actions
Recommendation N. 4 Establish
Development
Operations
Management (DevOps)44
Recommendation N.1 Adopt a
trustworthy AI framework
Action 2 “Create a Risk
Management
Framework”
Recommendation N.4 Establish
Development
Operations
Management (DevOps) 45
As the table shows, some requirements from the AI Act are addressed by Recommendation N. 3 Data Governance
framework and Recommendation N. 4 Establish Development Operations Management (DevOps), because they
already implement some specific governance measures for those requirements. For example, the Data Governance
framework as described in Recommendation N. 3 already helps EFSA in the implementation of some AI principles
such as Privacy and Data Governance or Transparency, especially in terms of Traceability and Accountability. 46 In
addition, when developing this, additional questions related to ethical issues could be considered, such as:
o
o
o
o
o
What phenomena are represented in the data?
How was the data collected?
Is the data balanced?
Does the data include sensitive information or proxies?
Does the data hide some cultural or historical bias? Towards which groups?
Similarly, the adoption of Development & Operations Management (DevOps) (Recommendation N. 4) will address
several AI principles (such as traceability and auditability) and will integrate some requirements of the AI Act specific
for the development, deployment, operating and monitoring phases of AI solutions.
In addition to the actions stemming from the AI Act, a third action has been added to the Trustworthy AI framework
recommendation, which aims at ensuring an efficient adoption of the framework itself. This last action concerns
internal skills, capabilities and roles.
Specific activities
The complete list of the suggested actions under Recommendation 5 - Adopt a Trustworthy AI Framework, which
EFSA should follow to adopt a robust and efficient Trustworthy AI framework consists of:
•
•
•
Action 1 “Develop a Code of conduct”
Action 2 “Create a Risk Management Framework”
Action 3 “Organise internal skills, capabilities and roles”
Action 1 “Define organisation principles by developing a Code of Conduct”: Firstly, it is recommended that
the organisation adopts a Code of Conduct (CoC), which is a statement of principles aimed at guiding its own conduct.
These principles can serve both as affirmations of values and as the basis of an internal accountability mechanism
for the organisation. Codes of conduct usually focus on ethical requirements and recommendations that go beyond
what the law requires. Indeed, they can contribute significantly to an organisation’s ability to deal with difficult ethical
issues, especially if the code is developed in a process with broad participation in the organisation. For this reason,
44
Development Operations Management (DevOps) can be considered as another methodology to implement specific principles
of Trustworthy AI concerning model management. For more information see Recommendation 4 Establish Development
Operations Management (DevOps)
45 Development Operations Management (DevOps) can be considered as another methodology to implement specific principles
of Trustworthy AI concerning model management. For more information see Recommendation 4 Development Operations
Management (DevOps)
46 See the Requirements of Trustworthy AI from the Ethics Guidelines for Trustworthy Artificial Intelligence prepared by the
High-Level Expert Group on Artificial Intelligence https://ec.europa.eu/futurium/en/ai-alliance-consultation/guidelines/1.html
www.efsa.europa.eu/publications
65
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
EFSA’s board members and executives should participate in a collaborative process for establishing the CoC in order
to ensure that the entire organisation is pushing towards – and committed to – the same direction. After the code
has been adopted, EFSA might consider establishing an ethical committee to further contribute to the organisation’s
ethical culture and its ethical competence (see Action 3).
As a starting point for the development of a more detailed Code of Conduct, EFSA might consider, as a relevant
example, the Ethics Guidelines for the Trustworthy Artificial Intelligence developed by the High-Level Expert Group
on AI (HLEG)47.
This document defines Trustworthy AI: it consists of three aspects, which should be met throughout the AI system's
entire life cycle:
•
•
•
lawful - respecting all applicable laws and regulations
ethical - respecting ethical principles and values
robust - from a technical perspective, while taking into account its social environment.
Based on these three aspects, four main principles have been established to ensure that AI systems are developed,
deployed and used in a trustworthy manner: Respect for human autonomy, Prevention of harm, Fairness,
Explicability. To realise and implement these principles, the HLEG defined 7 key requirements, which are now
considered as a standard de facto48 across the world for how to implement Trustworthy AI. These requirements are:
Table 13 - Map of the HLEG’s Requirement with short description
HLEG’s
Requirement
Short description
Human agency
and oversight
AI systems should empower human beings, allowing them to make informed decisions and fostering their
fundamental rights. At the same time, proper oversight mechanisms need to be ensured, which can be achieved
through human-in-the-loop, human-on-the-loop, and human-in-command approaches.
Technical
robustness and
safety
AI systems need to be resilient and secure. They need to be safe, ensuring a fall-back plan in case something
goes wrong, as well as being accurate, reliable and reproducible. That is the only way to ensure that also
unintentional harm can be minimised and prevented.
Privacy and data
governance
Besides ensuring full respect for privacy and data protection, adequate data governance mechanisms must also
be ensured, taking into account the quality and integrity of the data, and ensuring legitimised access to data.
Transparency
The data, system and AI business models should be transparent. Traceability mechanisms can help achieving
this. Moreover, AI systems and their decisions should be explained in a manner adapted to the stakeholder
concerned. Humans need to be aware that they are interacting with an AI system, and must be informed of the
system’s capabilities and limitations.
Diversity, nondiscrimination
and fairness
Unfair bias must be avoided, as it might cause multiple negative implications, from the marginalisation of
vulnerable groups, to the exacerbation of prejudice and discrimination. Fostering diversity, AI systems should be
accessible to all, regardless of any disability, and involve relevant stakeholders throughout their entire life circle.
Societal and
environmental
well-being
AI systems should benefit all human beings, including future generations. It must hence be ensured that they
are sustainable and environmentally friendly. Moreover, they should take into account the environment,
including other living beings, and their social and societal impact should be carefully considered.
Accountability
Mechanisms should be put in place to ensure responsibility and accountability for AI systems and their
outcomes. Auditability, which enables the assessment of algorithms, data and design processes plays a key role
therein, especially in critical applications. Moreover, adequate and accessible redress should be ensured.
47
High-Level Expert Group on Artificial Intelligence (2019) “Ethics Guidelines for trustworthy AI.” European Commission.
Available at: https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai
48 Fjeld J., Achten N., Hilligoss H., Nagy A., Srikumar M., Principled Artificial Intelligence: Mapping Consensus in Ethical and
Rights-Based Approaches to Principles for AI, Berkman Klein Center Research Publication, 2020
www.efsa.europa.eu/publications
66
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Another relevant example for EFSA to take inspiration from is given by the general Code of Conduct 49 developed by
ETAPAS50, a project funded by the European Commission, which includes ten fundamental principles to guide the
correct implementation of Disruptive Technologies (i.e., Artificial Intelligence, Big/Open Data, Blockchain, Augmented
and Virtual Reality, Internet of Things and Robotics) by Public Sector organisations.
These principles are set out in Table 14 below.
Table 14 - List of ETAPAS principles with short description
Principle
Short description
1 Environmental
sustainability
It is important to use Disruptive Technologies (DTs) conforming to the principles of
environmental sustainability. The principle considers the production processes for the hardware
and the materials used to produce the new technology, including possible new materials and
their impact on the environment. Biodiversity is a precondition for ecological resilience, and it
must always have an important role in the evaluation of environmental effects of new
technologies.
2 Justice, equality,
and the rule of law
Equal treatment and the rule of law are fundamental values from which public administration
should never deviate. The introduction of DTs should promote equality of access and
opportunity to everyone, and aim at eliminating possible biases towards specific groups (e.g.
women, minorities, etc.) due to low quality or missing training data for algorithms used (e.g.
AI, machine-learning or decision algorithms). The input data for algorithms needs to be
carefully selected and evaluated in order to make sure that discrimination or other undesired
effects are not introduced. Automated decision-making that impacts on individuals should only
be used when there is reason to be confident that the algorithm does not discriminate against
any group of individuals.
3 Transparency and
explainability
Individuals who are affected by a public decision based on automated data processing should
have access to clear information that a layperson can understand, both on how the decision
was made (transparency), and on its justification (explainability). Information must also be
presented in an accessible way for disabled and elderly people.
4 Responsibility and
accountability
The public sector is subject to strict principles of accountability, and blaming a machine, an
algorithm or a decision support system for a decision affecting individuals or society is not a
feasible solution. There must always be sufficient human oversight and control of automatic
decision-making to ensure that human decision-makers can be held accountable. Adequate
procedures must be in place for communication with individuals, and for proper investigation
and mitigation of problems arising from the use of DTs.
5 Safety and security
Before the introduction of DTs, the risks of both intentional and unintentional harm should be
carefully evaluated. The technological and organisational measures need to prevent the harmful
effects DTs could have on individuals and any other specific security risks that may appear (e.g.
hacking, data misuse, data manipulation).
6 Privacy
Best-practice methods for data protection, including deletion mechanisms, should be used, and
these methods should be regularly updated. With access to large quantities of data, unintended
harmful consequences for individuals might occur. Thus, use of such data should be strictly
regulated, and efficient measures to prevent and discover unauthorised use should be
implemented.
7 Building an ethical
culture involving
employees
The participation in the creation of an ethical culture at the workplace is essential for the
functionality and credibility of the public sector. The DTs should not be invasive forms of
control of workplace behaviour, but tools to relieve public servants of routine work and make
better use of their competences. The DT introduction must be decided in a participative and
49
The Code of Conduct (CoC) of ETAPAS is available on the project website at the following link:
https://www.etapasproject.eu/resources/deliverables/
50 ETAPAS (Ethical Technology Adoption in Public Administration services) has received funding from the European Union’s
Horizon 2020 research and innovation programme under grant agreement No 101004594
www.efsa.europa.eu/publications
67
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Principle
Short description
co-creative process involving both the public servants and their organisations. Upskilling and
reskilling, including for the ethical use and management of new technologies, should be part of
the working conditions.
8 Retaining human
contacts
Ensuring the wellbeing of all residents must be a priority of the public sector, also in the
adoption of new technologies. While the new technologies might negatively impact the social
contacts, they should be used in ways that do not weaken the social networks and make
residents feel more isolated. The introduction of alternative human-based modes of contact
should be considered whenever needed to avoid the possible negative effects of DTs adoption.
9 Ethical publicprivate cooperation
The public sector should have enough competences of its own and ability to control and review
private sector involvement to ensure that such cooperation works efficiently in the public
interest. Private sector performance and accountability should be ensured through appropriate
contractual protections. Artificial decision support systems in the private sector that are subject
to public supervision must be sufficiently transparent to make sure that efficient supervision can
be performed.
10 Continuous
evaluation and
improvement
The social effects of disruptive technologies are difficult, often impossible, to foresee. When the
technology is introduced on a larger scale, evaluation should continue in order to detect
(positive and negative) effects that may not have been discovered in the small-scale trials.
These evaluations should include the perspectives of residents and employees, particularly
groups at risk of exclusion or discrimination from technologies. Adjustments, improvement and
when necessary, replacement of technologies should be made whenever needed.
Action 2 “Create a Risk Management Framework”: Once the Code of Conduct is drafted, it will be necessary
to develop practical guidelines which can guide the trustworthy adoption and monitoring of AI solution. Indeed,
principles within the Code of Conduct should be translated into a Risk Management Framework which would provide
a detailed mapping of all the risks to which EFSA might be exposed when adopting an AI solution and governance
mechanisms to control and mitigate such risks.
Starting from the good practices of risk management in various sectors (e.g. financial services where these activities
are very mature), it is important to define a taxonomy of possible risks to which EFSA could be exposed when
adopting and using AI.
For example, within the above-mentioned ETAPAS project, a risk framework 51 has been developed providing a
detailed mapping of all the risks to which public bodies might encounter when adopting DTs. In particular, it identifies
eight risk categories including 35 risks, which can be traced back to the 10 ethical principles of the Code of Conduct,
as follows:
a. Risks concerning direct interaction with humans, such as the risks of physical and psychological harm;
b. Legal Risk, including liability risk and law infringement risk;
c.
Governance risks, which comprises all the risks arising from mismanagement, weak supervision and lack of
standard procedures;
d. Enhanced inequality and discrimination, which covers risks related to possible perpetuation of discrimination
with DT-based solution and the unequal access to and benefit from the DTs, due to economic, cultural and
social differences;
e. Errors and misuse, including risks emerging from the DT lack of robustness, biased performances or security
branches, as well as the risk of the DT's being used as an autonomous weapon, for malicious surveillance
purposes or for spreading disinformation while limiting the decision-making power of citizens;
51
For more information, see the contribution to the OSAI21 workshop at the following link:
https://www.unive.it/pag/fileadmin/user_upload/progetti_ricerca/osai/img/grafica/OSAI21_paper_5.pdf
www.efsa.europa.eu/publications
68
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
f.
Security and data protection risks, such as the risk of the DT being hacked to get information access or to
change its behaviour;
g. Unsustainable use, stemming from high level of energy consumption or wide use of environmentally
unsustainable materials;
h. Workplace issues, including job displacement and lack of competences by public sector employees.
Once all of the risks have been identified, different governance mechanisms could be put in place, such as the
identification and definition of controls for each risk in a post-hoc evaluation of AI solutions, or the adoption of an
ex-ante assessment to drive the development of AI solution from the design phase. From our analysis, the latter
would suit EFSA’s context better.
The development and adoption of an assessment to evaluate and measure the AI impacts, and determine potential
risks and mitigations actions would constitute a practical method to integrate the risk management framework within
the entire AI lifecycle. The assessment should aim at ensuring an appropriate and certain degree of transparency,
and provide users with information, human oversight, robustness, accuracy and cybersecurity, while
adopting the AI solutions. Usually, these assessments have a twofold objective: provide practical actions and
guidelines to mitigate the possible risks, while assessing and monitoring the residual risk.
Also in this case, EFSA might take into consideration existing best practices, such as the Assessment List for
Trustworthy AI (ALTAI) developed by the High-Level Expert Group on AI 52. The ALTAI translates the principles and
the HLEG trustworthy AI requirements into an accessible and dynamic checklist aimed at implementing such
principles in practice. It consists of a series of questions - both technical and non-technical – which assess whether
the development process and the AI system address the selected risk. In the Figure below, an example of questions
referring to the principle of human agency and oversight is detailed.
Figure 27 - ALTAI example of questions which refer to the human agency and oversight principles
In addition, referring once again to the ETAPAS project, an assessment – called Indicator framework - has been built
as a way to track and measure relevant risks and their mitigation actions. This tool was developed specifically for
public sector organisations that want to adopt Disruptive Technologies in a responsible manner.
The ETAPAS Indicator Framework, made up of a set of indicators, has been defined and structured as a checklist.
Within the framework, two types of indicators have been selected: those assessing whether the risk is relevant for
the context of application (risk indicators), and those that evaluate how much the risks were mitigated based on
specific mitigation actions (mitigation indicators). If the risk is present, then the mitigation questions should be
addressed too. Considering Figure 28 below, an example of a risk question could be: “Does the DTA (Disruptive
Technology Application) interact with the decision-making process of human end-users?”. Then, mitigation follow52
https://digital-strategy.ec.europa.eu/en/library/assessment-list-trustworthy-artificial-intelligence-altai-self-assessment
www.efsa.europa.eu/publications
69
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
up questions are listed which suggest actions of a different type: e.g., “Will the relevant personnel be able to assume
control where necessary?”; or “How do users feel when taking decisions based on the DT application?”.
Figure 28 - ETAPAS Indicator framework design and structure.
Action 3 “Organise internal skills, capabilities and roles”: Designing and deploying trustworthy AI systems
should be an organisation-wide effort. It requires sound planning, cross-functional and coordinated execution,
employee training for the new required skills and capabilities, and significant investment in resources to drive the
adoption of responsible AI practices. Indeed, employees need to be trained to understand how risk manifests in their
contextual interactions with AI systems and, more importantly, how to identify, report and mitigate them. Significant
effort should be dedicated to tailor the training contents to the different roles in the organisations, in order to provide
the correct level of information and upskilling of all of the people53. In particular, the Code of Conduct should be well
known across the whole organisation to actually drive the behaviour of each person.
In addition, investment in resources for the establishment of new roles and/or integration of new responsibilities in
the current organisational structure would be fundamental. For example, the establishment of an ethical committee
might be considered to drive the development and monitoring of EFSA’s Trustworthy AI Framework. This ethical
board might become the central organism for the ethical decision-making concerning EFSA’s AI solutions, responsible
for the resolution of possible ethical issues and tensions. These considerations should be taken into account within
Recommendation N. 6 Develop data science competencies & design behaviour change mechanisms.
Within this context, different groups of stakeholders play different roles in ensuring that the practical guidelines are
met54. Some examples are provided the table below (Table 15) 55:
Table 15 - Example of organisational governance.
Role
Management and Board
Compliance/Legal
Short description
• Top management discusses and evaluates the AI systems’ development, deployment or
procurement, and serves as an escalation board for evaluating all AI innovations and uses,
when critical concerns are detected. It involves those impacted by the possible introduction
of AI systems (e.g. workers) and their representatives throughout the process via
information, consultation and participation procedures.
• The corporate responsibility department monitors the use of the assessment list and its
necessary evolution to meet the technological or regulatory changes. It updates the
53
For example, Recommendation 6. Development of data science competencies and design of behaviour change mechanisms
should also consider ethical AI related skills.
54 For information on the technical profiles needed to implement the Trustworthy AI framework, see Recommendation 6 “Data
Science Competencies & behaviour change mechanism”
55 This example refers to the proposed organisational governance for the ALTAI, available
herehttps://ec.europa.eu/futurium/en/ai-alliance-consultation/guidelines/2.html
www.efsa.europa.eu/publications
70
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
department/Corporate
responsibility
department
Product and
Development
equivalent
Service
or
standards or internal policies on AI systems and ensures that the use of such systems
complies with the current legal and regulatory framework and to the organisation's values.
• The Product and Service Development department uses the assessment list to evaluate
AI-based products and services, and logs all the results. These results are discussed at
management level, which ultimately approves the new or revised AI-based applications.
Quality Assurance
• The Quality Assurance department (or equivalent) ensures and checks the results of the
assessment list, and takes action to escalate an issue to a higher level if the result is not
satisfactory, or if unforeseen results are detected.
HR
• The HR department ensures the right mix of competences and diversity of profiles for
developers of AI systems. It ensures that the appropriate level of training is delivered on
Trustworthy AI inside the organisation
Procurement
• The procurement department ensures that the process to procure AI-based products or
services includes a check of Trustworthy AI.
Day-to-day operations
• Developers and project managers include the assessment list in their daily work, and
document the results and outcomes of the assessment.
It is important to note that the success of introducing the Trustworthy AI Framework in EFSA (and any other
organisation) is facilitated by smooth and incremental changes to the current organisation. For this reason, we
recommend defining the target processes, roles and responsibilities based on a thorough assessment and analysis
of the as-is situation.
Recommendation Summary
Objectives
The main objectives for Recommendation 5 can be summarised as follows:
-
In order to adopt the AI system in a responsible way, it is suggested that EFSA adopts a Trustworthy AI framework
that addresses the challenges around AI from an ethical, social and legal point of view.
Challenges
The main challenges for Recommendation 5 can be summarised as follows:
-
The establishment of a Trustworthy AI framework may require several of the organisation's units to get involved,
thereby possibly causing misconceptions over the framework.
Converting principles of the Code of Conduct into practice can be a difficult task. The technical complexity and scale
of people and process change might be underestimated.
The organisation could struggle to scale the Trustworthy framework within the lifecycle of the AI system.
Expected Impact
The main expected impact for Recommendation 5 can be summarised as follows:
-
An effective Trustworthy framework adoption might support EFSA to effectively improve trust, reduce risks, obtain
efficiencies, and adopt AI in a more trustworthy and transparent way
-
Once AI techniques are applied in NAMs for decision support in risk assessment, the adoption of a Responsible AI
framework will also be of paramount importance.
The following table provides the SWOT analysis for Recommendation 5.
SWOT Analysis
Strengths
www.efsa.europa.eu/publications
Weaknesses
71
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
-
Set clear rules and policies for the adoption of the
Trustworthy AI
Ensure the trustworthiness, transparency, and
fairness of the outcomes of AI solutions
-
Opportunities
-
-
Time and effort-consuming procedures due to the
organisation alignment to proper AI principles
Threats
Increase confidence in the outcome of the AI
solutions through transparency and a fairnessassured standard based on the AI governance
Framework
Anticipate some requirements of the future
European AI regulation
-
Misalignment among organisation executives while
adopting AI principle
Misalignment of different technical units while
adopting AI principle within the AI lifecycle
Difficult comprehension of terminology used in the
Trustworthy AI framework among BU of the
organisation due to lack of new skills
Recommendation 6: Expand data science competencies & design behaviour change
mechanisms
Description and scope
In order to efficiently adopt AI for evidence management and to support the actions around the previous 5
recommendations, it is necessary to expand data science competences (via recruitment and upskilling) such as
natural language processing (NLP), text mining, ethical AI and DevOps. The introduction of innovative tools and new
collaborative ways of working might require training and upskilling of EFSA internal staff. In addition, EFSA might
consider developing behaviour change mechanisms, which would facilitate the transition of EFSA towards an AIenabled organisation.
In more detail, EFSA competencies needed to effectively adopt and use the identified AI tools are:
•
NLP and text mining: Natural language processing (NLP) comes from a combination of computer science,
information sciences, AI, and linguistics. Therefore, current data science skills within EFSA are not sufficient
for moving towards the use of NLP. The necessary skills of a NLP data scientist/engineer are:
o
o
o
o
o
o
o
•
Knowledge of machine learning and deep learning frameworks and libraries
Knowledge of Big Data frameworks like Spark and Hadoop
The ability to understand text representation techniques, algorithms, statistics
Text classification & clustering skills
Programming skills – Python, Java and/or R
Familiarity with Syntactic & Semantic Parsing
Experience with ontologies and knowledge graphs
Domain Knowledge Expertise: While not a specific prerequisite, a general understanding of the field of
study in which AI solutions are applied would be beneficial to generate higher value for EFSA. A data scientist
with domain-specific knowledge can better support the transition towards the adoption of AI technology,
acting as a point of contact between the users and the AI-based tools implemented. Because of the difficulty
of recruiting data scientists with expertise in the subjects EFSA is working on, it might be beneficial to
consider recruiting individuals with NLP engineering skills to undertake research projects (e.g., PhDs in
collaboration with academic institutions) within specific EFSA departments in order to acquire the domain
expertise and apply their technical skills in a more targeted way.
•
Skills on Trustworthy AI: To move towards the adoption of the Trustworthy AI, it is fundamental that
EFSA data scientists acquire technical skills about this. These skills include:
o
Testing for bias in the data, model, and human use of AI algorithms to improve fair treatment across
groups.
www.efsa.europa.eu/publications
72
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
o
o
o
Adding transparency, explainability and provability to the modelling process to improve human
understanding of the model outputs.
Improving security and robustness of AI through rigorous validation, continuous monitoring and
maintenance, verification and adversarial modelling (this is addressed via the DevOps recommendation).
Understanding of moral and systemic implications of using AI during development and adoption (in
collaboration with business users).
Specific activities
In order to take full advantage of its data and to better understand the logic of the results when using AI-based tools
in the evidence management process, EFSA data science capabilities should be upgraded. The following five steps
(Figure 29) might help EFSA to achieve this aim, and, at the same time, create the basis for implementing the
mechanisms regarding the behaviour change.
Figure 29 - Steps for Data Science Competences Development.
Designing behavioural change mechanisms
It is important that - along with the introduction of new tools, practices, structures and ways of working -, EFSA
undertakes targeted change management activities in order to establish the behavioural changes that support the
success of the change.
In the light of this, for the implementation of any project related to both the horizontal and vertical recommendations,
we recommend EFSA to include dedicated change management actions to ensure the successful adoption and the
establishment of the appropriate behavioural changes
Figure 30 outlines the principles that should be taken into consideration in order to engage EFSA staff in the new
ways of working.
Figure 30 - Six principles of People-centred Change and Adoption.
www.efsa.europa.eu/publications
73
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
On the basis of the above-mentioned principles, the four pillars of a change management approach are shown below.
Figure 31 - Six pillars of change management approach.
Recommendation Summary
Objectives
-
-
EFSA should consider expanding data science competencies, either through new hires or upskilling of existing
personnel, mainly by introducing NLP and text mining skills. These new competencies will support the efficient
adoption and use of AI for evidence management in risk assessment, as well as the generation of new ideas on how
to manage and maximise value from data.
EFSA should consider change management actions which would enable critical behaviours supporting cultural
transformation and adoption of AI technology.
Challenges
-
Data science competencies are comprised of different specific skills including data preparation, modelling and analysis,
and visualisation and communication, as well as knowledge of NLP and text processing skills. Within this context, the
challenge consists in finding the perfect formula that would ensure the correct adoption and maintenance of AI
solutions.
-
Before recruiting, there is a need to accurately examine the current competences of internal technical profiles (e.g.,
solutions engineer, data architect, data scientist etc.), identify the gaps and, if deemed necessary, design a
personalised training programme or curriculum to upskill them.
www.efsa.europa.eu/publications
74
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Recommendation Summary
Expected Impact
-
The development of data science competencies and behavioural change mechanisms might support EFSA in gaining
autonomy with regard to the development, customisation (especially for those solutions with a low technology
readiness requiring specific domain expertise) and, mostly, monitoring of AI tools. In addition, considering the low
level of readiness of several solutions to automatise specific activities within the evidence management process,
developing these competences and mechanisms would support EFSA in the potential development or co-development
of AI solutions, and their customisation to EFSA needs.
The following table provides the SWOT analysis for Recommendation 6.
SWOT Analysis
Strengths
-
Weaknesses
EFSA has a sufficient number of experienced staff in
the field of Big Data and Data Analytics.
EFSA is willing to invest both in the data
infrastructure and data governance framework,
which play a key role in guiding the development of
data science competencies and cultural change of
the organisation.
-
-
Opportunities
-
4.2.
There is not an established and clear way for
providing rapid access to the data, while at the
same time ensuring a consistent data governance.
As a result, this slows down the rapid evolvement of
data science practitioners.
The organisation structure is not currently designed
in a way that it can leverage the collaboration and
full potential of the key data science capabilities.
Threats
There are still many use cases of the evidence
management process that are not associated with
any commercialised AI-based solution. The
combination of advanced data science capabilities
and domain specific knowledge within EFSA can be
the foundation for the potential internal
development or the co-development of AI solutions.
-
One risk might consist in the misinterpretation of
data due to lack of training/domain understanding
of data analysts, losing opportunities for high value
creation.
-
Regardless of the data science competencies
developed, there is the risk to draw poor
conclusions and wrong associations due to possible
misunderstanding and use of data.
Not enough work in niche areas (e.g., NLP) may be
available for specialists to be fully utilised.
Vertical Recommendations
In the previous section, a series of horizontal recommendations have been developed to enable a smooth and
effective adoption of customised AI-based solutions, and to allow EFSA to reach its Strategy 2027 objectives. 56
In fact, horizontal recommendations unlock the possibility to adopt AI in the use cases along the evidence
management process (more details on the relations between horizontal recommendations and adoption of AI in use
cases are outlined in section 5.2.3). The vertical recommendations that we present in this section make the last
step, by associating the prioritised use cases within the evidence management process with one of the 3
possible vendor selection scenarios, following the methodology presented in section 3.3.3.
More specifically, for each of the 10 prioritised use cases (resulting from the prioritisation exercise reported in
section 3.3.1), the optimal approach to either procuring or developing AI solutions to be adopted by EFSA (vendor
selection scenario) with its specific procedures and conditions, is suggested based on its assessment in terms of
56
EFSA, Theme (concept) paper on Artificial Intelligence in risk assessment.
www.efsa.europa.eu/publications
75
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
number of commercialised solutions - identified in the market scouting - and coverage level - through the
coverage assessment.
Table 16 below reports the results of this exercise, linking each prioritised use case of evidence management to the
corresponding vendor selection scenario.
According to the methodology, prioritised use cases with a high number (at least 3) of
available/commercialised solutions (high technology readiness level) and associated to a high coverage
level (higher than 90%) are placed into scenario 1, which considers the procurement process as the optimal
procedure to purchase off-the-shelf solutions.
Alternatively, for the use cases with a high number (at least 3) of available/commercialised solutions (high
technology readiness level) and associated with medium level of coverage (between 90% and 60%),
scenario 2 is recommended. Within this scenario, procurement or procurement of innovative solutions
(PPI), involving a Proof of Concept to test the solutions feasibility and performance, should be considered as the
optimal collaboration model for adopting the AI solutions.
Finally, use cases with either few available identified solutions (less than 3), no solutions available or with
a low level of use case coverage (< 60%) are associated with scenario 3. This scenario suggests launching a
grant, contest or PCP as the most suitable collaboration models to finance R&D activities, aimed at developing and
implementing AI-based innovative solutions that respond to needs and resolve problems that are not addressed by
solutions available in the market.
Table 16 below provides an overview of the three AI adoption scenarios in relation to the 10 prioritised use cases in
the evidence management process, also listing the potential partners and related AI solutions.
www.efsa.europa.eu/publications
76
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Table 16 – AI Adoption Scenarios and connected Prioritised Use Cases of the Evidence Management.
Prioritised Use Case
Data
Collection
Terminology
Assessment
No of available
Solutions
(>= 3 or < 3)
High
Market Solutions
Coverage
Vendor Management
Scenario
High
Scenario
1:
Procurement Procedure
to buy off-the-shelf and
advanced AI solutions
SLR
Keywords
Identification
High
High
SLR Relevant Abstract
Screening
High
High
Organisation
Solution
Octopai
Uninova
Seven Past Nine
Universite de Tours
Universiy
of
Science
Technology Bannu
Foundation for Research
Technologies - Hellas
Arab Academy for Science
University of Manchester
Scipilot
SGS Digicomply
Iris.ai
Sciome
Risklick AG
EMA
Sciome
Evidence Partners
Seven Past Nine
Octopai
Research project
Seven Past Nine
Heterotoki
Homogeneous
Mapping
University of Manchester
SLR
Literature
duplication
De-
www.efsa.europa.eu/publications
High
High
77
Rayyan
AMNH
VKM
Eppi-centre
Pico Portal
Sciome
and
and
Ontology
VisTA
Word Encoding Method
HYPHEN
Syras
Search Engine Results Page
Iris.ai
Swift Review
Risklick AG
Internal solution
Swift Review
DistillerSR
Seven Past Nine
Systematic review workload
reduction through certaintybased screening
Rayyan
Colandr
Internal solution
Eppi-Reviewer
Pico Portal
Swift Review
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Prioritised Use Case
No of available
Solutions
(>= 3 or < 3)
Market Solutions
Coverage
Text Summarisation
High
High
SLR Literature Findings
Clustering
High
High
Quality Improvement
of the Scientific Output
Expert
Identification
Pool
www.efsa.europa.eu/publications
High
Low
Medium
Low
78
Vendor Management
Scenario
Scenario
2:
Procurement
or
Innovation
procurement (PPI) to
develop a PoC (Proof of
Concept) and integrate AI
solutions or to integrate
the solution
Scenario 3: grant for
research
and
development services or
PCP to develop, test and
integrate the solutions or
Organisation
Solution
Evidence Partners
Risklick AG
Iris.ai
ASReview
Research Screener
Robot Reviewer
EUIPO
CDT
Iris.ai
Innovamol Consulting
Eppi-centre
Scipilot
SGS Digicomply
Sciome
Evidence Partners
Risklick AG
Creme Global
LibrAlry
Ginger
Grade Pro
DistillerSR
Risklick AG
Iris.ai
ASReview
Research Screener
Robot Reviewer
Internal solution
Internal solution
Iris.ai
Innovamol
Eppi-Reviewer
Syras
Search Engine Results Page
Swift Review
DistillerSR
Risklick AG
Creme Global
LibrAlry
Ginger Software
Grade Pro
ProWritingAID
ProWritingAID
Pomato
Department of Scientific Research,
Visibilia SP
Ramrao
Adik
Institue
of
Technology
Pomato
Job Reccomendation
Job Recommender System
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Prioritised Use Case
No of available
Solutions
(>= 3 or < 3)
Market Solutions
Coverage
Vendor Management
Scenario
Contest to develop, test
and validate innovative
solutions
Expert Selection
High
Low
Organisation
Solution
University of Electronic Science
and Technology of China
Textkernel
MosaickTrack
Springer Nature
Ideal
Pomato
Leoforce
Department of Scientific Research,
Visibilia SP
Ramrao
Adik
Institue
of
Technology
University of Electronic Science
and Technology of China
Machine Learned ResumeJob Matching
Texkernel
MosaickTrack
Reviewer Finder
AI for Recruiting
Pomato
Arya
Agriculture and Agrifood Canada
Comparable Appraisals
Detection
www.efsa.europa.eu/publications
Low
Medium
Wageningen
University
&
ResearchInformation Technology
University of San Francisco
79
Job Recommendation
Job Recommender System
Machine Learned ResumeJob Matching
Exploring
Identifiers
of
Research Articles Related to
Food and Disease Using
Artificial Intelligence
Automatic
Systematic
Literature Review
Critical Appraisal Tools
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
The following tables provide a SWOT Analysis to identify strengths, weaknesses, opportunities, and threats associated with the adoption of AI technology for
each prioritised use case that falls under the respective scenario:
Vendor Management Scenario 1 – Procurement
Swot Analysis – Data Collection Terminology Assessment
Strengths
-
-
-
Weaknesses
There are many available solutions related to this use case that EFSA
can evaluate through an effective and efficient procurement process to
ultimately choose the best solution in terms of quality and costs.
The cost of buying a commercialised tool that need to be customised
based on EFSA needs and requirements is less than the amount spent
on investing in research and development for a tool/service that does
not exist yet, or that is in its initial level of technology readiness.
The integration of an AI solution for this use case would have a low
impact in the organisation, and is not likely to face resistance to
adoption by EFSA staff.
Opportunities
-
-
-
Considering that the majority of the commercialised AI tools are
developed for a general purpose, EFSA may need a unique solution
tailored to its needs and context. However, most of the vendors have
ensured their solution is highly customisable, even though it may require
additional costs and time for implementation.
Threats
The adoption of AI technology would automate the alignment of the
content of data collected to EFSA vocabulary, leading to
time/efficiency gains and quality improvement.
The PERA team suggested that an AI tool that will be able to evaluate
how well an interview has been transcribed into text format could
really be a benefit in terms of time and workload reduction. PERA has
its own database of texts that have already been transcribed, and
therefore this can be used as “training data” for the AI tool, and to
further increase its accuracy. Modifying slightly the use case of Data
Collection Terminology Assessment would help address the challenge,
since the logic of assessing the text format and terms is the same.
-
There is a potential risk of incurring in unexpected costs if the
compatibility of the AI solutions within EFSA’s ecosystem is not properly
assessed and ensured.
There is a potential risk of a low degree of solution/vendor flexibility in
case of interim changes of need (scope, requirements), but this can be
mitigated by adopting an agile project management approach.
Swot Analysis – SLR Keywords Identification
www.efsa.europa.eu/publications
80
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Strengths
-
-
Weaknesses
Most of the tools have a good accuracy level, and therefore can
correctly fulfil EFSA needs and technical requirements in relations to
the use case.
There are many available solutions related to this use case that can be
evaluated to choose the best one in terms of quality and costs.
The cost of buying an already commercialised tool that needs to be
customised based on EFSA needs and requirements is less than the
amount spent on investing in research and development for a
tool/service that does not exist yet, or that is in its initial level of
technology readiness.
Opportunities
-
Considering that the majority of the commercialised AI tools are
developed for a general purpose, EFSA may need a unique solution
tailored to its needs and context. However, most of the vendors have
ensured their solution is highly customisable, even though it may require
additional costs and time for implementation. A well-defined ontology is
needed, which is currently not in place within EFSA. An ontology can
help in defining the similarities between terms.
Threats
-
The identification of the most relevant keyword within the literature
through an AI-based solution would represent a good opportunity for
EFSA to develop a well-defined domain specific ontology. The
expected impacts of creating an ontology can be found within the
horizontal recommendations section (5.2.1).
-
The adoption of AI technology would automate the identification of the
most appropriate keywords for the search strategy, leading to
time/efficiency gains and quality improvement.
-
One of the major issues of the PERA project is the resourceintensiveness when retrieving and analysing relevant information from
databases and websites. SLR Keyword Identification can introduce
time-efficiencies.
There is a potential risk of incurring unexpected costs if the compatibility of the
AI solutions within EFSA’s ecosystem is not properly assessed and ensured.
Swot Analysis – SLR Relevant Abstract Screening
Strengths
-
Weaknesses
EFSA already has experience in the use of the current tool, DistillerSR,
for Abstract Screening, Thus, there will be no significant resistance to
change.
www.efsa.europa.eu/publications
81
-
During the consultation with EFSA staff, it emerged that many
employees may be resistant to use a different tool for Abstract
Screening, since DistillerSR is widely and satisfactorily used.
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
-
-
There are many available solutions related to this use case, which
EFSA can evaluate through an effective and efficient procurement
process, to eventually choose the best solution in terms of quality and
costs.
The cost of buying a commercialised tool that need to be customised
based on EFSA needs and requirements is less than the amount spent
on investing in research and development for a tool/service that does
not exist yet, or that is in its initial level of technology readiness.
Opportunities
-
Considering that the majority of the commercialised AI tools are
developed for a general purpose, EFSA may need a unique solution
tailored to its needs and context. However, most of the vendors have
ensured their solution is highly customisable, even though it may require
additional costs and time for implementation.
-
Domain-specific ontologies would need to be developed to enable the
correct functioning of this solution.
Threats
-
For testing purposes, EFSA may consider purchasing an additional tool
for a trial period to compare its performance with that of DistillerSR.
-
The adoption of AI technology would automate the screening of
papers’ abstracts and their analysis against the research question,
leading to time/efficiency gains and quality improvement.
-
One of the major issues of the PERA project is the resourceintensiveness when retrieving and analysing relevant information from
databases and websites. SLR abstracts screening can introduce timeefficiencies.
In case EFSA decided to deploy an additional tool for testing and comparison
with the existing tool (DistillerSR), there is a risk of users’ resistance. As a result,
the new tool may be dropped, regardless of its effective performance, with all
the effects that may be associated in terms of cost and time.
Swot Analysis – SLR Literature De-duplication
Strengths
-
-
-
Weaknesses
Most of the tools identified do not separate the activities of data
clustering and de-duplication. Therefore, in these cases, EFSA would
purchase one solution that fulfils both the use cases.
There are many available solutions related to this use case, which
EFSA can evaluate through an effective and efficient procurement
process, to eventually choose the best solution in terms of quality and
costs.
The cost of buying an already commercialised tool that needs to be
customised based on EFSA needs and requirements is less than the
amount spent on investing in research and development for a
www.efsa.europa.eu/publications
82
-
Considering that the majority of the commercialised AI tools are
developed for a general purpose, EFSA may need a unique solution
tailored to its needs and context. However, most of the vendors have
ensured their solution is highly customisable, even though it may require
additional costs and time for implementation.
-
When performing data de-duplication, it is important to not set too strict
rules since this could lead to the risk of eliminating and losing relevant
data.
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
tool/service that does not exist yet, or that is in its initial level of
technology readiness.
Opportunities
-
-
-
Without a complete ontology, this use case would only be used as a
rule-based algorithm.
Threats
The adoption of AI technology would automate and make trustworthy
the detection of duplicated papers highlighted for removal from SLR,
leading to time/efficiency gains and quality improvement.
-
One of the major issues of the PERA project is the resourceintensiveness when retrieving and analysing relevant information from
databases and websites. SLR Literature de-duplication can introduce
time-efficiencies.
Risk of incurring unexpected costs in case the compatibility of the AI
solutions within EFSA’s ecosystem is not properly assessed and ensured.
If rule-based tools (already used within EFSA) perform to a satisfactory
level, it might be difficult to convince management to invest in an AIbased solution.
Swot Analysis – Text Summarisation
Strengths
-
-
-
-
Weaknesses
Most of the tools identified for this category, such as the IRIS AI,
Risklick AI or IBM Watson Discovery, offer additional services/functions
that can be used for other use cases.
There are many available solutions related to this use case, which
EFSA can evaluate through an effective and efficient procurement
process, to eventually choose the best solution in terms of quality and
costs.
The cost of buying an already commercialised tool that needs to be
customised based on EFSA needs and requirements is less than the
amount spent on investing in research and development for a
tool/service that does not exist yet, or that is in its initial level of
technology readiness.
-
As most of the AI tools identified for this use case offer multiple
services/functions, they may end up performing worse in comparison to
solutions specialised in just a single use case.
Although the implementation of the solutions may require some
customisation, there are low risks of project failure, since the solutions
already cover the main needs well.
Opportunities
www.efsa.europa.eu/publications
Threats
83
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
-
Most of the tools identified for this category, such as the IRIS AI,
Risklick AI or IBM Watson Discovery, offer an integrated set of
services/functions that can be used for other use cases as well,
improving the overall user experience.
-
The adoption of AI technology would automate and simplify the
summarisation of scientific evidence, leading to time and effort
efficiency, quality improvement and reduction of human bias.
The adoption of AI for this use case would be very relevant across
EFSA, and not only for evidence management in risk assessment.
-
-
There is a potential risk of incurring unexpected costs if the compatibility of the
AI solutions within EFSA’s ecosystem is not properly assessed and ensured.
One of the major issues of the PERA project is the resourceintensiveness when retrieving and analysing relevant information from
databases and websites. Text Summarisation can introduce timeefficiencies.
An additional pain point of PERA is the variability between the
outcomes of peer-reviews conducted by different risk assessors. AI
can help in this case through the application of topic modelling
approaches or Automatic Text Summarisation. These can simplify the
procedure and support PERA in producing informative summaries per
each peer-review. These summaries might contain key elements of the
review, which can be used in an AI tool (such as Comparable Appraisal
Detection) to identify similarities and discrepancies.
Swot Analysis – SLR Literature Findings Clustering
Strengths
-
-
Weaknesses
This use case reports the largest number of identified commercialised
tools in comparison with all the other prioritised use cases. Therefore,
EFSA may choose the one which best fit its needs from many relevant
solutions.
Most of the tools identified do not separate the activities of data
clustering and de-duplication. Therefore, in these cases, EFSA would
purchase one solution that fulfils two use cases.
www.efsa.europa.eu/publications
84
-
Considering that the majority of the commercialised AI tools are
developed for a general purpose, EFSA may need a unique solution
tailored to its needs and context. However, most of the vendors have
ensured their solution is highly customisable, even though it may require
additional costs and time for implementation.
-
A complete subject-specific ontology is needed for this solution to give
satisfactory results.
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
-
The cost of buying an already commercialised tool that needs to be
customised based on EFSA needs and requirements is less than the
amount spent on investing in research and development for a
tool/service that does not exist yet, or that is in its initial level of
technology readiness.
Opportunities
Threats
-
Currently, in EFSA, data clustering is performed for short texts via the
use of unsupervised learning techniques. Ideally, this should be
improved to capture the full text. Therefore, a potential collaboration
between EFSA (via providing data from its rich database) and a vendor
may occur in the future.
-
The adoption of AI technology would automate and simplify the
clustering of SLR findings in terms of relevance, leading to time and
effort efficiency and quality improvement.
The algorithm developed for the semantic analysis and reasoning,
required for the clustering of SLR, may perform worse than a simpler AIenabled application
-
Vendor Management Scenario 2 – Innovation Procurement (PPI)
Swot Analysis – Quality Improvement of the Scientific Output
Strengths
Weaknesses
-
The development of a solution aimed at improving the quality of
scientific output might be considered a great investment, since it might
also automatise other activities related to evidence management of risk
assessment (e.g., it might be used to summarise the literature review,
or it can syntheses evidence collected from other sources).
-
Although the implementation of the solutions will require some
customisation, there are low risks of project failure, since the solutions
already cover the needs well.
Opportunities
-
The customisation of the solution might be quite challenging for EFSA
considering the different features and functions that the solution could
support to improve the quality of the scientific output (e.g., AI might
support the grammar, synthesis, or syntax of the scientific output).
-
Launching an innovation procurement for this use case might result in a
time-consuming and expensive process because of the need to perform
a Proof of Concept/Pilot Project to demonstrate the solution’s
performance and feasibility.
Threats
-
www.efsa.europa.eu/publications
85
Even though a vendor may agree on customising its product to satisfy
EFSA needs, there is no guarantee that the end-product will produce
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
-
The adoption of AI technology would automate the proofreading and
qualitative assessment of reports, leading to time and effort efficiency
and quality improvement.
accurate results. As a result, an amount of time and money could be
invested without returning the expected result.
Vendor Management Scenario 3 – Grant, PCP or Contest
Swot Analysis – Expert pool identification
Strengths
Weaknesses
-
The development of an AI solution from scratch, or in its initial level of
readiness through a grant or PCP might help EFSA in reducing
overheads by building a tailored and customised solution.
-
The time required to build the proposed solution can extend far beyond
the desired outcome. The challenges in training, to reduce false
positives and increase quality production, may take years to overcome.
-
Even though the majority of the AI tools identified for this use case do
not match exactly its purpose, they can still be used as a point of
reference when developing a new tool.
-
To effectively develop this solution, there may be the need to integrate
different components, such as web-crawling.
-
There is the risk of selecting and awarding the grant to the wrong
candidate. This may result in investing a huge amount of time and
money without delivering the expected solution.
If experts’ information is not publicly available, the solution will not be
able to deliver the desired results.
Opportunities
-
Threats
This solution might be applied in different units within EFSA.
The adoption of AI technology would automate the identification of a
list of relevant experts, leading to time and effort efficiency, enhanced
trustworthiness, quality improvement, and the reduction of human bias.
-
The internal technical staff might not be prepared – from a technical
point of view - to develop such as sophisticated AI solution.
-
Beyond the AI utility, before building web-crawling tools, there are a
number of cybersecurity and GDPR issues to be taken into
consideration.
Swot Analysis – Expert selection
Strengths
Weaknesses
-
www.efsa.europa.eu/publications
86
The time required for the development of the solution can extend far
beyond the desired outcome. The challenges in training, to reduce false
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
-
-
The development of an AI solution from scratch, or in its initial level of
readiness through a grant or PCP might help EFSA in reducing
overheads by building a tailored and customised solution.
Developing an AI solution from scratch would enable EFSA to own the
intellectual property of the AI solution in this specific area of evidence
management.
Opportunities
-
-
positives and to increase quality production, may take years to
overcome.
There is the risk of selecting and awarding the grant to the wrong
candidate. This may result in investing a huge amount of time and
money without delivering the expected solution.
Threats
The adoption of AI technology would automate the ranking and
selection of relevant experts, leading to time and effort efficiency,
enhanced trustworthiness, quality improvement and reduction of
human bias.
-
No threats were identified for this use case.
Swot Analysis – Comparable appraisal detection
Strengths
Weaknesses
-
The development of an AI solution from scratch, or in its initial level of
readiness through a grant or PCP might help EFSA in reducing
overheads by building a tailored and customised solution.
-
Developing an AI solution from scratch would enable EFSA to own the
intellectual property of the AI solution in this specific area of evidence
management.
-
-
-
Opportunities
Threats
-
A research project in this area is likely to contribute important findings
to the wider NLP scientific community.
-
The adoption of AI technology would automate the appraisal of
evidence gathered from comparable research questions, leading to
www.efsa.europa.eu/publications
A specific functionality, which includes semantic analysis and reasoning,
for automating the use case might be difficult to develop as it will
require a lot of research activity, given that it is not adequately
developed.
Building and training an algorithm while reducing false positives and
ensuring quality production may be time and effort consuming (taking
years).
By launching a grant or a PCP for the development of a solution, EFSA
might invest a huge amount of money without receiving the expected
solution.
87
-
The internal technical staff might not be prepared – from a technical
point of view – for developing and maintaining the AI solution.
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
time and effort efficiency, enhanced trustworthiness, quality
improvement and reduction of human bias.
www.efsa.europa.eu/publications
88
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
4.3.
Prioritisation of Working Areas
In the previous sections, a list of horizontal and vertical recommendations was provided with the aim to: 1) Build
EFSA’s readiness to adopt AI for evidence management in risk assessment (and in a wider sense) and 2) Enable the
adoption of AI in the specific prioritised uses via the three proposed AI adoption scenarios. The two points together
will allow EFSA to reach its vision to increase in the accessibility and breadth of the body of evidence, thereby
enhancing the trustworthiness in the risk assessment process, and to apply human centric artificial intelligence in
close co-existence with human expertise by 2027.
However, to concretely support EFSA in its decision making and resources allocation, we present in this section a
prioritisation of these vertical and horizontal recommendations to enable their order and dependencies to
be understood.
To prioritise the horizontal recommendations, we have considered the following elements:
•
The impact on / need within evidence management of risk assessment (i.e., in terms of it being a prerequisite for the adoption of AI, or a “best-practice” recommendation),
•
The impact of the recommendation across EFSA (i.e., not only in the scope of evidence management
in risk assessment, but also in other key areas within EFSA, based on the information and needs collected
from internal stakeholders, and from other roadmaps. For example, input from the NAMs team has been
taken into consideration, including their need for automating the area of data standardisation and
integration, which might be covered by the infrastructure and governance recommendations.)
•
The dependence of each recommendation on other horizontal recommendations.
It is worth noting that the table below (Table 18) provides an account of the effort required to adopt each vertical
and horizontal recommendation. Regarding the horizontal recommendations, which are foundational to the
development of EFSA’s AI capability, the effort is not estimated, as the projects need to take place regardless of the
effort.
We then proceeded with incorporating in the prioritisation the vertical recommendations for the different use
cases, on the basis of:
•
The dependence on horizontal recommendations, and
•
The anticipated impact on evidence management in risk assessment (as per the impact assessment
set out in Table 17).
Finally, the two prioritisation exercises (for horizontal and vertical recommendations) were integrated to develop a
consistent and comprehensive roadmap aimed at delivering to EFSA quick wins, together with a sustainable and
scalable AI capability development. For example, although the creation of domain specific ontologies (Horizontal
Recommendation 1) is not a priority compared to other more foundational capacity development actions, the rapid
adoption of some market-ready solutions relies on the existence of domain specific ontologies. As a result, the
development of EFSA-specific ontologies was prioritised to consider this relationship. In addition, domain specific
ontologies will also support the scientific advancement of AI in those related areas.
Rationale behind ratings
For the horizontal recommendations, the rationale behind the ratings is the following:
•
Impact/Need within Evidence Management: Rated as “High” if it is a pre-requisite, and “Medium” if it is a
best practice for the adoption of AI;
•
Effort: rated in terms of the time and financial investment needed to adopt the recommendation;
•
Impact across EFSA: rated on the basis of the needs expressed by other EFSA stakeholders consulted and
roadmaps (e.g., NAMs and PERA roadmap) in terms of the impact that this recommendation would have on
www.efsa.europa.eu/publications
89
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
their own work. Indeed, some horizontal recommendations (e.g., Implementation of a Data Infrastructure)
are big projects that would be beneficial across other areas in EFSA, such as for the NAMs team, in order to
deal with data-intensive activities.
For the vertical recommendations, the rationale behind the ratings is the following:
•
Impact/Need within Evidence Management: Impact ratings based on the previous analysis.
•
Effort: rated as “Low” for use cases falling under Scenario 1, “Medium” for those falling under Scenario 2,
and “High” for those falling under Scenario 3.
•
Impact across EFSA: rated on the basis of the needs expressed by other EFSA stakeholders consulted and
roadmaps (e.g., NAMs team and PERA roadmap) in terms of the impact that this recommendation would
have on their own work.
First, the overall prioritisation was performed based on the recommendations’ dependencies. Then, in terms of the
impact ranking (both within evidence management and across EFSA) and, lastly, on the basis of the effort needed
(the latter only in the case of the vertical recommendations).
Table 17 - Prioritisation of Horizontal and Vertical Recommendation
No
1
2
3
4
5
6
No
UC 1
UC 2
UC 3
UC 4
Horizontal
Recommendation
Create (or adopt existing)
ontologies for domains of
relevance for EFSA 57
Implement
a
Data
Infrastructure
which
enables Big Data and AI
development
Design and Implement a
Data
Governance
framework
which
addresses AI
Adopt DevOps
Adopt a Trustworthy AI
framework 58
Develop
Data
Science
Competencies in NLP and
design behaviour change
mechanisms
Vertical Recommendation
Data
Collection
Terminology Assessment
SLR
Keywords
Identification
SLR
Relevant
Abstract
Screening
SLR
Literature
Deduplication
Capability
Development
Dimension
Impact/Need
within Evidence
Management
Effort
Impact
across EFSA
Dependencies
Data & Information
High
Medium
Low
6
Technology &
Infrastructure
High
High
High
3
Organisation &
Governance
High
High
High
None
Medium
Low
Medium
2,3,6
Medium
Medium
High
None
Skills & Culture
High
High
Low
None
Capability
Development
Dimension
Impact Within
Evidence
Management
Effort
Impact
Across EFSA
Dependencies
with horizontal
recommendatio
ns
Use Cases
2.25
Low
Medium
None
Use Cases
2.5
Low
Low
(1)59,6
Use Cases
1.75
Low
Low
6,1
Use Cases
2
Low
Low
1,6
Process &
Integration
Process &
Integration
57If
the ontology is to be developed in-house, it would have a dependency with horizontal recommendation 2, Implement a
Data Infrastructure which enables Big Data and AI development
58 An alignment with HR 3 is recommended as best practice.
59 It is suggested to have an ontology, although this is not mandatory as most of the solutions for this use case do not require
it.
www.efsa.europa.eu/publications
90
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
UC 5
UC 6
UC 7
UC 8
UC 9
UC
10
Text Summarisation
SLR Literature Findings
Clustering
Quality Improvement of
the Scientific Output
Expert Pool Identification
Expert Selection
Comparable
Appraisals
Detection
Use Cases
2.25
Low
Medium
(1)60, 6
Use Cases
2.5
Low
Medium
1, 3, 2, 6
Use Cases
2
Medium
Low
1,2,3, 6
Use Cases
Use Cases
2.75
3
High
High
Low
Low
2,3,6
2,3,6
Use Cases
2.5
High
Medium
1,2,3,6
Based on the above assessment, all of our recommendations (both horizontal and vertical) have been prioritised and
ordered as shown below in Table 18, along with the estimated time-frames. The approach used to distribute the
projects along the time-frame followed the dependencies and urgency of each project, as mentioned earlier, but also
in an attempt to avoid placing too many projects in the same time period, in order not to overcharge EFSA's effort.
The table allows the quick-wins that EFSA can obtain to be visualised, while developing its overall AI capability
(horizontal recommendations). This approach is fundamental to achieve EFSA’s 2027 vision 61.
The proposed time-frames do not take into account the time to prepare, launch and award the procurements/grants.
In addition, it should be noted that, in the event that horizontal recommendations become projects to be carried out
in house (and not outsourced via procurement), a 30%-time increase must be foreseen.
60
it.
61
It is suggested to have an ontology, although this is not mandatory as most of the solutions for this use case do not require
EFSA, Theme (concept) paper on Artificial Intelligence in risk assessment.
www.efsa.europa.eu/publications
91
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Table 18 - Roadmap for actions (Horizontal & Vertical Recommendations)
Recommendation
H2 2022
H1 2023
H2 2023
H1 2024
H2 2024
Programme
management
H1 2025
H2 2025
H1 2026
H2 2026
H1 2027
H2 2027
Programme management
Develop (or adopt existing) Ontologies
relevant to EFSA
Horizontal 1
Continuous maintenace required for this recommendation
Adopt a Data Infrastructure for
Big Data and AI
Horizontal 2
Horizontal 3
Design and Implement Data Governance
Training required for
this recommendation
Horizontal 4
Implement DevOps
Training and adoption
Adopt a Trustworthy AI framework - Risk
M gmt Framework
Adopt a Trustworthy
AI framework - CoC
Horizontal 5
Training required for this recommendation
Training required for this recommendation
First round of
recruitment
Assessment of Data
Science Skills and
recruitment/upskilli
ng plan
Horizontal 6
Second round of
recruitment*
Upskilling of current staff
Scenario 1 UC 1
Scenario 1 UC 2
Scenario 1 UC 3
Vertical scenario 1
Scenario 1 UC 4
Scenario 1 UC 5
Scenario 1 UC 6
Scenario 2 UC 7
Vertical scenario 2
Scenario 3 UC 8
Vertical scenario 3
Scenario 3 UC 9
Scenario 3 UC 10
www.efsa.europa.eu/publications
92
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Since there are multiple projects relating to the adoption of AI for evidence management, it is appropriate to also
consider some additional effort related to programme management.. The programme management may have
the following objectives:
•
•
•
•
•
•
•
Ensuring that AI projects are aligned with the overall EFSA strategy;
Planning, coordinating, and monitoring of progress, and managing of all the AI projects’ inter-dependencies,
including oversight of any risks and issues arising;
Facilitating EFSA governance by bringing together the internal key stakeholders from different units impacted
by the change e.g., through the organisation of bi-monthly progress meetings;
Providing strategic advice on the design of the communication and engagement plans with the interested
stakeholders (see chapter 6);
Managing third party contributions to the programme e.g., by organising an internal EFSA forum to connect
and align all involved project managers and contractors;
Ensuring that there is an efficient allocation of resources and skills within the programme's individual
projects;
Enabling effective change and endorsement of all AI projects by EFSA staff, by supporting the definition of
ad-hoc change management actions (e.g., design of a communication plan, training plan, resistance
management plan).
Budget and Time Estimates
This section aims to provide some key details for each of the six horizontal recommendations and for the three
vendor scenarios (vertical recommendations), such as their estimated budget, time and effort (expressed in man
days).
Initially, some general assumptions are presented, which relate to the estimates for all of the projects. Then we
show the estimated figures, together with some more specific assumptions that consider the peculiarities of each
horizontal or vertical project.
General Assumptions
As mentioned before, the following assumptions are valid for all projects:
•
•
•
•
•
The projects’ activities can be performed in-house or outsourced to contractors;
Standard market rates with no discounts were considered (500 EUR/Day for junior positions, 750 EUR/Day
for middle management positions, 1000 EUR/Day for senior positions). These costs should be adjusted with
EFSA rates, in case EFSA decides to undertake the projects in house.
Timings were estimated considering an outsourcing scenario. Alternatively, in an in-sourcing scenario, we
suggest counting an additional 30% of time required for implementing the projects, due to the current limited
availability of internal staff.
Procurement costs were not considered (i.e. for the definition of the proposal, selection and awarding, and
administrative costs).
For the implementation of the vertical recommendations, costs and times for the scouting activity and vendor
selection were not considered. In the future, if EFSA decides to apply the same approach to other use cases
(especially for those outside of evidence management), it should also count these cost and time factors for
scouting and vendor selection in order to obtain a complete estimate.
Estimates and specific assumptions
www.efsa.europa.eu/publications
93
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Programme Management
The table below sets out the expected budget, time scale, man days and specific assumptions for each
horizontal recommendation.
Table 19 – Expected budget and time scale of the Horizontal Recommendations.
Budget
1) Develop (or adopt
existing) ontologies for
domains of relevance for
EFSA
483K €
2) Implement a Data
Infrastructure for Big
Data Analytics & AI
development
Proposed time scale
1 year project + 4 years
maintenance
Man Days
(md)
Specific Assumptions
708 Days
• Estimated costs for the development of 3
ontologies
• Considered 20k EUR/Year*5 years of licence
costs, to be deleted in case of open-source tools
• Considered 200K to develop the ontologies in
case EFSA starts from scratch (to be reduced by
30% if you start with available ontologies)
764 Days
• 150K per year were estimated for the
infrastructure (Azure, *50 users, * 100 TB, pay as
you go)
• The logical and physical data modelling was
estimated for 10 subject areas
676 days
• The licence cost for the tool is not included (if not
already available, the licence for data quality +
stewardship/ownership
+
Master
Data
Management to be purchased)
608K €
1 year project + 1 year
training/upskilling
417K €
1 year project + 0.5
year training/upskilling
4) Establish Development
& Operations
Management (DevOps)
366K €
0.5 year project,
including training
504 days
• The licence cost for the infrastructure/tool is not
included (it is assumed to be included in the Azure
bundle)
5) Adopt a Trustworthy AI
Framework
394.5K €
4 months for the CoC +
1 year RMF + 4.5 year
training
630 days
NA
0.5 years + 5 years
training/upskilling of
current staff
270 days
NA
3) Implement a Data
Governance framework
which addresses AI
6) Expand data science
competencies & design
behaviour change
mechanisms
183K €
Total Estimated Budget
2022-2027
2.4Mln €
As regards the vertical recommendations – including scenarios 1,2,3 - their estimated budget and time is reported
in Table 20 below.
Table 20 - Expected budget for Vertical Recommendations.
Budget per Use
Case
Vertical
Recommendation
Scenario 1
58 K € (including
1 year tool
licence)
www.efsa.europa.eu/publications
Time
No of
cases
4 months
6
94
use
Use Case Titles
Data Collection Terminology
Assessment
SLR Keywords Identification
SLR
Relevant
Abstract
Screening
SLR Literature De-duplication
Text Summarisation
SLR
Literature
Findings
Clustering
Budget * No of use
cases
348K €
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Vertical
Recommendation
Scenario 2
215 K €
(including 1 year
tool licence)
Vertical
Recommendation
Scenario 3
367 K €
8 months + 1
year
maintenance
activities
18 months +1
year
maintenance
activities
1
Quality Improvement of the
Scientific Output
215K €
3
Expert Pool Identification
Expert Selection
Comparable
Appraisals
Detection
1.1Mln €
Total
Estimated
Budget 2022-2027
1.7Mln €
The following table sets out the estimates for the programme management.
Table 21 - Expected budget, time scale and man days for Programme Management.
Budget
Programme
Management
Proposed time scale
462K €
Man Days (md)
5.5 years
660 days
The following table summarises the estimated overall costs from 2022 to 2027.
Table 22 - Expected Total Budget per Horizontal and Vertical Recommendations.
Project Type
Budget
Programme Management
0.46Mln €
Horizontal Recommendations
2.4Mln €
Vertical Recommendations (Scenarios * 10 use cases)
1.7Mln €
Total Estimated Budget 2022-2027
4.56Mln €
www.efsa.europa.eu/publications
95
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
5.
Communication and Engagement Plans
This chapter aims to identify potential communication and engagement opportunities in the area of AI approaches
for evidence management of the risk assessment, and to outline the connected actions, taking into consideration
both EFSA’s objectives and potential stakeholders’ needs 62.
The main outputs are a communication and commitment plan, respectively designed to make EFSA's stakeholders
aware about the adoption of AI in the evidence management process, and to take on board key potential partners
to support the development and implementation of AI-based solutions.
The chapter is structured as follows:
•
•
•
•
•
5.1.
Section 5.1 presents the general approach to build a communication and engagement plan in view of the
AI adoption for evidence management. First, the approach consists of the mapping of EFSA stakeholders’
categories, their assessment through a multi-criteria methodology and the design of the communication and
engagement plans Then, focus on the results of the stakeholder's categories mapping is provided (5.1.1).
Finally, the criteria used for the stakeholder's assessment, together with the results, are described (5.1.2).
The results of the assessment are represented through a two-axis matrix with four quadrants, each
associated to a specific result, where stakeholders are placed.
Section 5.2 focuses on the lower-left quadrant of the matrix, which includes the stakeholders that are not
significantly valuable for the purpose of engagement activities, and that are not particularly sensitive to the
AI topic. In this scenario, the optimal strategy would be monitoring these stakeholders’ future actions around
AI and, in parallel, keeping them updated on EFSA's AI work.
Section 5.3 focuses on the upper-left quadrant of the matrix, where stakeholders are still not particularly
relevant for the purpose of engagement activities in view of AI adoption, but may be particularly sensitive
to the AI topic with specific concerns on the matter. For these organisations, the design of an ad-hoc
communication could be the optimal strategy to respond to their concerns.
Section 5.4 outlines the proposed engagement plan for those stakeholders’ categories that are particularly
relevant for the purpose of engagement activities in the field of AI adoption in the evidence management
process (lower-right and upper-right quadrants of the matrix).
Section 5.5 puts forwards some final considerations around the communication and engagement plans,
regarding their time for implementation, dependencies with other horizontal recommendations, and scope.
General Approach
Not all of the stakeholder groups are of equal importance to a public organisation. Knowing when, how, to whom
and why to send communications and/or collaborate is vital to any communication/engagement plan. Some
stakeholders may require immediate notification about a change or situation affecting the organisation, whilst others
that are less impacted or not involved at all do not need access to the same information.
To lay down a proper communication and engagement plan, we designed the following three-step approach:
1. Mapping of Stakeholders: consists of the mapping of EFSA’s relevant stakeholders to be targeted,
identifying them according to specific categories (e.g., Industry, Academia) and sub-categories (e.g., NGOs,
Business Operators). As mentioned before, we refer to both stakeholders within EFSA’s food safety ecosystem
for which EFSA is accountable, and potential partners that may collaborate with EFSA for the purpose of this
project around the AI adoption.
2. Assessment of Stakeholder’s Categories: consists of assessing each stakeholder category/sub-category
based on several criteria that are designed for the purpose of this project. The aim of this assessment is to
We refer to the different stakeholders’ types mapped through the previous project’s activities to be potentially engaged for
the purpose of the project (see sections 3.1 and 3.3), and also other actors within EFSA’s food safety ecosystem to be informed
about the AI developments (e.g. EFSA’s risk managers, NGOs etc.). More details are reported in the following sections.
62
www.efsa.europa.eu/publications
96
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
understand how better to approach and/or involve the different stakeholders in view of the implementation
of AI technology for evidence management.
3. Development of the Communication Plan and Engagement Plan based on the assessment of
stakeholder’s categories/sub-categories, including specific actions to be followed and methods to be to be
used. Specifically, stakeholders that are not relevant for collaboration opportunities, but that should be kept
informed on EFSA’s AI developments are targeted for the communication plan, while key stakeholders that
can contribute to the AI adoption in EFSA will be targeted for the engagement plan.
Further details on the methodology and results of the above-mentioned steps are provided in the following sections.
Figure 32 - General Approach for Communication and Engagement.
5.1.1.
Mapping of Stakeholders’ Categories/Sub-categories
As explained previously in the general approach, the first step consisted of the mapping of EFSA stakeholders,
clustering them according to specific categories (e.g., Industry, Academia, and sub-categories (e.g., NGOs, Business
Operators, Distributors, etc).
Table 23 provides an overview of EFSA’s stakeholder categories and sub-categories, developed by adapting and
extending the official categorisation of EFSA food chain stakeholders - adopted by the Management Board in 2016
63
and reflected in EFSA’s Stakeholder Engagement Approach – to the scope of this project around AI adoption for
evidence management. In particular, our categorisation also includes additional stakeholders identified during the
project’s data collection activities, which are relevant for the purpose of collaboration to foster AI adoption in EFSA.
Table 23 - List of EFSA stakeholders' categories and sub-categories.
Stakeholder Category
Stakeholder Sub-category
European Commission
Sister Agencies
Institutions
Other European Agencies
Other international organisations
National Competent Authorities (incl. Art. 36 org)
Industry
63
Business and food industry
IT/Software companies
https://www.efsa.europa.eu/en/partnersnetworks/stakeholder
www.efsa.europa.eu/publications
97
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
University spin-offs
AI start-ups
Distributors and HORECA [Hotels, Restaurants, Catering]
Farmers and primary producers
Advocacy groups
NGOs and civil society organisations
Consumer organisations
Practitioners’ associations
Academia
Universities and research centres (incl. Art. 36 org)
Digital Innovation Hubs
Development Levers64
Technology transfer offices
Incubators and accelerators
5.1.2.
Assessment of Stakeholder’s Categories
Once the stakeholder categories/sub-categories were identified, each of them was assessed based on three criteria
to understand how to involve and approach them in view of the adoption of AI for evidence management.
•
Relevance for the project, expressed as the strategic importance that stakeholders have in terms of
concrete cooperation/collaboration opportunities with EFSA in view of AI adoption for evidence
management. This reflection is based on the analysis performed in section 3.3, where the stakeholders
playing a prominent role are mapped throughout the phases of AI development lifecycle to fulfil EFSA's
different needs (from 0 to 5).
•
Impact in terms of effect – positive or negative – on EFSA’s accountability of stakeholders’ opinions
with respect to the implementation of AI projects, or in terms of effective contribution of their solutions
or actions to enable EFSA’s adoption of AI technology. Therefore, this criterion shows the importance of
having a stakeholder on board by establishing pertinent collaboration/cooperation models or properly
communicating with them on EFSA’s work and progress.
•
Easiness to collaborate with the stakeholders, based on current or past collaboration and the presence
of existing mechanisms and processes that facilitate collaboration in AI-related projects. In other words,
the easiness to collaborate depends on the kind of relationship / level of interaction of EFSA with a given
stakeholder, which can be categorised as established, evolving and emerging.
For example, stakeholders with an established relationship with EFSA are those that are that have an
active collaboration/exchange with EFSA; stakeholders with an evolving relationship are those that have
not extensively collaborated with EFSA in the past and are willing to strengthen the relationship, and
stakeholders with an emerging relationship levels are those organisations that have never collaborated
with EFSA with respect to AI development. Finally, stakeholders that are not able/interested to contribute
are classified as Not applicable (e.g., NGOs usually do not have the proper expertise to collaborate with
EFSA in this field). In fact, this criterion is only applied to stakeholders that can contribute to some extent
to the adoption of AI solutions, whether by commercialising them (e.g., software companies and startups), providing access to expertise (e.g., DIHs, Universities, Accelerators, etc), exchanging best practices
and lessons learned, or contributing with EFSA by co-developing AI solutions (e.g., EU and National
Institutions), etc, as mentioned in section 3.3.
64
Development levers are entities or programmes that accelerate the modalities to engage the AI community, and establish
cooperation. Therefore, they can connect EFSA with other stakeholders involved in innovation processes.
www.efsa.europa.eu/publications
98
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
It is important to highlight that the assessment was based on the information collected through the various methods
during the data collection and analysis phases of the project.
Based on the results of the assessment, stakeholder sub-categories were positioned in a matrix, as shown in the
figure 34, where the first criterion is represented in the horizontal axis, the second in the vertical axis, and the third
is represented through a colour code.
Figure 33 - Communication/engagement strategies
(Impact/Relevance/Easiness to collaborate matrix).
for
the
identified
stakeholder
segments
Once the stakeholders’ categories were positioned, the matrix was split into four quadrants, each grouping the
categories sharing similar results and the same outcome in terms of communication or engagement actions:
•
•
•
[LOWER-LEFT] Inform and Monitor/Listen: includes stakeholders associated to a low relevance
and low impact, such as farmers and producers, distributors and HORECA, and the business and food
industry. These stakeholders are not significantly valuable for the purpose of engagement activities in
the field of AI adoption in the evidence management process (i.e., they do not offer any AI solution, do
not undertake any relevant initiative around AI development, or have any solid AI knowledge). In
addition, as they are not sensitive to the AI topic, no concerns can be expected from their side in case
EFSA launches a public communication around the development of AI in the organisation.
In this scenario, the optimal strategy would be monitoring these stakeholders’ future actions around AI
and, in parallel, keeping them informed on EFSA’s progress on AI adoption through the available
institutional means and channels (e.g., press release, EFSA website), without the necessity to build
personalised communications messages.
[UPPER-LEFT] Design an ad-hoc communication: includes stakeholders associated to a low
relevance and high impact, such as practitioners associations, advocacy groups, and consumer
organisations.
Indeed, while these stakeholders are not significantly valuable for the purpose of possible collaboration
activities in view of AI adoption for evidence management, they are particularly sensitive to the AI topic.
This last aspect may entail possible concerns and claims from their side, in case of a public
communication of EFSA’s activities in the field of AI, with a potential negative impact on the
organisation’s reputation. Hence, it would be recommended to design an ad-hoc communication plan
that considers their concerns on AI, and mitigates the risk of potential claims and legal actions.
[LOWER-RIGHT] Engage in the ecosystem: includes stakeholders associated to a high relevance,
low impact organisations that have never collaborated with EFSA with respect to AI development (EFSA’s
AI ecosystem), such as international organisations (e.g. FAO), university spin-offs, and the so-called
development levers (DIH, Accelerators, and Technology Transfer Offices). These stakeholders are
www.efsa.europa.eu/publications
99
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
relevant for the purpose of engagement/collaboration activities in the field of AI adoption for evidence
management (e.g., development levers can accelerate the modalities to establish cooperation with other
actors, such as AI start-ups and universities with relevant AI expertise). On the other hand, these
stakeholders are considered less impactful - compared to those within the up-right quadrant - since they
do not specifically offer any AI solution at the moment, or otherwise given their mission. For instance,
on the one hand, development levers do not market or develop AI solutions, but do have the function
of accelerating cooperation with other stakeholders. On the other hand, with regard to International
organisations (e.g., FAO) and university spin-offs, according to our research they do not currently
commercialise any AI solutions for evidence management, or do not pursue any initiative/action around
AI adoption in such area.
In the light of this, the optimal strategy would be engaging these stakeholders within EFSA's AI
ecosystem (intended as those organisations that have already collaborated with EFSA in AI development)
to create synergies, open up opportunities for future concrete collaboration/cooperation, and boost the
innovation process.
•
[UPPER-RIGHT] Maintain & Reinforce Collaboration: includes stakeholders with a high relevance,
high impact and that are already part of EFSA's AI ecosystem, such as Universities and research centres,
Software companies, EFSA’s sister agencies and other EU agencies, Article 36 organisations, European
Commission bodies and National Authorities. These stakeholders are particularly relevant for the purpose
of engagement activities in the field of AI adoption in the evidence management (e.g., software
companies offer a high number of commercialised AI solutions; some EU Agencies are developing AI
solutions that fall within the area of evidence management and are open to collaborate in the future
with EFSA), and having them on board would generate a positive impact for EFSA. In the light of this,
the optimal strategy would be maintaining and reinforcing the collaboration actions with these partners
in the future for the adoption of AI.
Figure 34 - Communication/Engagement strategies for the identified stakeholders segments.
5.2.
Inform & Monitor/Listen Matrix Quadrant
As anticipated in the previous section, EFSA stakeholders with a low relevance and impact were positioned on the
lower-left portion of the matrix and are mainly composed of: Farmers and primary producers, Business and
food industry, Distributors and HORECA. These stakeholders are not relevant for the purpose of
collaboration/cooperation activities with EFSA in the field of AI adoption for evidence management, and they do not
have any particular concern related to the use of AI (with rare and isolated exceptions). This is proven by the fact
www.efsa.europa.eu/publications
100
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
that no position papers or articles, published by them, were found on this topic. Therefore, no concerns could be
expected from their side in the event of a public communication of EFSA’s activities in the field.
Hence, these stakeholders should be monitored and kept informed on the AI technology adoption within the
organisation through the available EFSA means and channels (e.g., press release, EFSA website), without the need
to build an ad-hoc communication plan.
It is important to keep in mind that, due to the emerging importance of AI, interests and concerns related to this
topic may change for these stakeholders. Thus, if during the monitoring activities it emerged that these stakeholders
have developed a higher regard and sensitivity towards the AI topic in the future, they would be moved to the upper
quadrant on the left (Higher Impact), and therefore targeted for an ad hoc communication plan.
Figure 35 - Inform & Monitor/Listen Quadrant.
5.3.
Communication Plan
As explained in section 5.1.2, Stakeholder sub-categories with a low relevance and high impact, falling into the upperleft quadrant of the matrix, were targeted for the development of an Ad-hoc Communication plan on the adoption
and use of AI technology.
Figure 36 - Ad-hoc Communication Quadrant.
To develop a sound and effective Communication Plan, three main steps can be followed:
www.efsa.europa.eu/publications
101
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
1. Identification of key stakeholders’ concerns related to the adoption of AI. This activity has been carried
out through a desk research to explore position papers, articles and reports published by EFSA’s stakeholders
with an interest in the Authority’s work, or in the wider food and feed sector;
2. Definition of potential risks associated with the communication to the targeted stakeholders of the
adoption of AI in the evidence management, based on their concerns.
3. Definition of mitigation actions for the risks identified, aimed to build proper communication messages –
in line also with some key horizontal recommendations described in chapter 4 - around EFSA’s AI work.
5.3.1.
Societal concerns on AI Adoption
To identify the concerns of EFSA’s stakeholders in relation to the potential adoption of AI, an ad hoc desk research
was conducted. During this task, position papers, articles and reports presenting stakeholders’ views on the
adoption of AI were investigated, per each stakeholder’s category/sub-category. Although it emerged that different
stakeholders share similar concerns, we managed to associate them to specific sub-categories.
The main needs and concerns on AI adoption are related to the following domains:
•
•
•
•
•
•
•
•
•
•
Ethics, trustworthiness and fairness in AI
Liability, accountability and right to compensation, if AI practices harm the public in some way
Transparency
Environmental impact
Bias and discrimination
Privacy and data protection
Manipulation
Legal compliance
Digitalisation
Access to justice
This section presents further details and some examples from our research.
Various position papers were found for the NGOs and civil society organisations category, in particular for
consumer organisations.
The European Consumer Organisation (BEUC), for instance, has published a position paper that focuses on how
Artificial Intelligence practices could harm the consumers.
The paper65, called “Regulating AI to protect the consumer”, provides insights regarding concerns and risks that
could be derived from AI applications. According to this, although AI technology could provide great improvements
and make consumers’ life more convenient, it also comes with great risks and could have major implications for
consumers’ autonomy and self-determination, their privacy, their capacity to interact with products and services and
ultimately, in the ability to hold businesses responsible if something goes wrong.
The paper refers to the proposed AI Act 66, as it should provide consumers with the rights and protections they need
to be at ease when using AI, ensuring that the EU’s fundamental rights and values are respected. Though the
organisation welcomes the attempt of the European Commission to regulate and provide a legal framework for
Artificial Intelligence, BEUC believes that the proposal requires substantial improvements to guarantee that
consumers have the protections they need, and can trust AI to respect their rights and freedoms.
In general, when applying AI, some basic principles should be taken into consideration:
65
BEUC, Regulating AI to Protect the Consumer, (2021). Available at: https://www.beuc.eu/publications/beuc-x-2021088_regulating_ai_to_protect_the_consumer.pdf
66 European Commission, Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial
Intelligence and Amending Certain Union Legislative Acts, (2021) available at
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206
www.efsa.europa.eu/publications
102
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
•
•
•
Fairness
Accountability
Transparency
In addition to these basic principles the paper provides a list of forbidden AI practices that could potentially harm
consumer rights:
-
AI practices should not manipulate consumers in a way that could cause them economic harm;
AI regulations should protect consumers made vulnerable through the use of persuasion profiles and
personalisation practices (digital asymmetry);
A ban on AI used by public and private organisations to evaluate individuals on the basis of their social
behaviour and personal attributes, in order to avoid consumer discrimination.
According to the paper, in order to be protected, consumers should have the right to be given clear explanations
about how an AI system could affect their work, and should have the right to object to algorithmic decisions. In this
sense, AI decisions should always be backed by human oversight, meaning that it should be a human-centric AI. In
addition to the information provided by BEUC, according to our research, organisations such as Consumer
International, ANEC, and ICPEN consider it relevant to protect consumers rights when applying AI to specific contexts.
Moreover, other organisations that do not belong to the Consumer organisation category consider it important to
address matters such as transparency, accountability, privacy and data protection, etc, for example advocacy groups
and practitioners’ associations that work in the health care sector (AESGP, EPHA, FVE and EFA).
These considerations are reinforced by those expressed by the report 67 of the European Union Agency for
Fundamental Rights (FRA), which illustrates some of the ways that companies and the public sector in the EU are
looking to use AI to support their work, and whether and how they are taking fundamental rights considerations into
account. Even though the Agency is positioned in the higher part of the relevance matrix, we mention it here since
its view is linked with those previously presented.
The main findings of the FRA report can be summarised as follows:
•
•
•
•
•
Using AI systems can affect a wide range of fundamental rights, regardless of the field of application. These
include privacy, data protection, non-discrimination and access to justice. So, when introducing
new policies and adopting new legislation on AI, the EU legislator and the Member States, acting within the
scope of EU law, must ensure that respect for the full spectrum of fundamental rights.
Prior impact assessment mainly focuses on technical issues, and rarely address potential effects on
fundamental rights, so the EU legislator should consider making mandatory impact assessment that covers
the full spectrum of fundamental rights before any AI system is used.
The EU and Member States should ensure that effective accountability systems are in place to monitor
and, where needed, effectively address any negative impact of AI systems on fundamental rights. They
should consider, in addition to fundamental rights impact assessments, introducing specific safeguards to
ensure that the accountability regime is effective.
EU Member States should consider encouraging companies and public administration to assess any
potentially discriminatory outcomes when using AI systems. The European Commission and Member
States should consider providing funding for targeted research on potentially discriminatory impacts of the
use of AI and algorithms.
To effectively contest decisions based on the use of AI, people need to know that AI is used, and how and
where to complain. Organisations using AI need to be able to explain their AI system and decisions based
on AI. The EU legislator and Member States should ensure effective access to justice for individuals in cases
involving AI-based decisions. To ensure that available remedies are accessible in practice, the EU legislator
and Member States could consider introducing a legal duty for public administration and private companies
using AI systems to provide those seeking redress information about the operation of their AI systems.
67
FRA, Getting the Future Right: Artificial Intelligence and Fundamental Rights, (2020). Available at:
https://fra.europa.eu/sites/default/files/fra_uploads/fra-2020-artificial-intelligence_en.pdf
www.efsa.europa.eu/publications
103
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Other papers published by NGOs and civil society organisations tackle different aspects on how AI could impact
their specific fields of competence.
For example, the European Environmental Bureau has shared on its website a paper68 published by the European
Parliament that focuses on the potential of AI to help to achieve a green transition, focusing on environmental
potential, characteristics and causes of environmental risks that could be derived from applying AI to this specific
field.
For instance, AI could have negative impacts on the environment, since the use of digital, and hardware and
infrastructure such as data centres and networks, could lead to an increased consumption of material and resources
of energy. Other risks could derive from systemic effects such a rebound effect, that could evolve as a consequence
of intended or unintended changes in the behaviour of consumers, users or producers. Such effects can be the
consequence of opaque dynamics of AI system learning, running contrary to an intended environmentally friendly
function of an application. For example, products designed to automatically manage and implement more efficient
energy consumption may actually have the effect of causing users to give up control over their energy consumption
and over-consume. Other adverse environmental impacts could result from AI applications that, as a consequence
of their intended employment, contribute to GHG emissions and nature destruction, for example the use of AI to
unlock oil and gas deposits, and to explore and develop new territories for fossil fuel extraction.
In addition, the paper highlights that the proposed AI Act of the European Commission does not include any hazards
related to the environment unless adverse environmental impacts pose a direct threat to human rights or interests.
For instance, rules related to data governance, transparency, human oversight and security do not yet provide a
governance system that will avoid adverse environmental impacts, so the EEB underlines the need for further
research and development of methods and institutional procedures for assessing and controlling environmental risks
caused by AI. Other organisations that deal with the environment such as Greenpeace have concerns similar to
those of EEB.
With regard to NGOs and civil society organisations that fall within the health and medical sector, such as
AESGP69 and EPHA70, the position that emerged from their papers are overall aligned. The use of AI is seen as
potentially beneficial, though it should only serve as a tool that could optimise day-to-day operations and data
analysis, and it should never substitute nor erase in-person healthcare services.
These organisations point out that supervision of AI should be proportionate to the intended use, and led by defined
risk categories with clear internationally defined criteria, focusing on topics such as the data protection of the
patients. In addition, as pointed in other papers and articles on the development and use of AI-powered tools,
accountabilities when using AI in the healthcare sector need to be clearly defined. Also, AI should be Ethical and
Trustworthy, meaning that AI systems need to be human-centric, with the aim of increasing individual and societal
well-being. In conclusion, since the healthcare sector is a highly regulated environment, regulators must ensure a
fair level playing field for all actors by developing and enforcing horizontal regulations across sectors.
Furthermore, EuroCommerce, which is an organisation that aims to favour European retail, wholesale and trade,
has published articles71 that overall consider AI as a powerful and useful tool that could have a huge potential for
the development of the world economy. They consider AI as a tool that will be crucial to better understand customer
needs, creating services that they want and making online and offline sales more efficient. In addition, they believe
that:
•
•
Having a positive narrative towards AI technologies is a prerequisite to unlock Europe’s tech sovereignty
The future EU framework for AI should be technology-neutral and focus more on achieving desirable
outcomes rather than regulating AI tools.
68European
Parliament, The role of Artificial Intelligence in the European Green Deal,(2021) available at:
https://www.europarl.europa.eu/RegData/etudes/STUD/2021/662906/IPOL_STU(2021)662906_EN.pdf
69 https://aesgp.eu/content/uploads/2021/08/AESGP-response-to-the-AI-regulation-proposal_FINAL.docx.pdf
70 https://epha.org/the-patients-voice-an-integral-part-of-digital-health-transformation/
71 https://www.eurocommerce.eu/search.aspx?q=Artificial%20Intelligence
www.efsa.europa.eu/publications
104
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
•
•
•
•
The future EU framework for AI should support the digital development of SMEs
The future European framework for AI should rely on a simple, narrow, clear and harmonised definition of
AI
Careful attention should be paid to avoid overregulation – especially considering recently adopted EU and
national legislation, support the use of existing AI technologies and bolster innovation.
High-risk AI applications should be defined and overseen in a way that provides legal certainty
So, all things considered, EuroCommerce has a favourable view on AI, with few concerns regarding regulations and
potential risks. One concern is however presented, since AI could significantly transform the job market, rendering
some jobs obsolete. There is also the need to provide public access to a strong supply of digital skills.
To sum up, the specific concerns of the stakeholders are summarised in the following table:
Table 24 - Stakeholder value map.
Stakeholder
Category
Stakeholder
category
Sub-
Specific Concerns
•
•
Advocacy groups
•
•
•
•
•
NGOs and civil
society
organisations
•
Consumer organisations
•
•
•
•
Practitioners’
associations
•
•
5.3.2.
Transparency, which would allow humans to see whether the AI systems have
been thoroughly tested, and provide explanations on why particular decision are
made
Ethics, trustworthiness and fairness when applying AI in EFSA’s processes,
in line with the guidelines of the European Commission and Recommendation 5
“Adopt a Trustworthy AI framework”.
Effective accountability systems that would ensure sufficient human oversight
and control of automatic decision-making to ensure that human decisionsmakers can be held responsible
Right for consumer complaints/legal action in case an AI system or practice
affects them, including a right to receive compensation for damages suffered
AI practices should not manipulate consumers in a way that could cause
them economic harm
Right for consumer complaints/legal action in case an AI system or practice
affects them, including a right to receive compensation for damages suffered
AI should not evaluate individuals on the basis of social behaviours or personal
attributes in order to avoid consumer discrimination
Transparency, which would allow humans to see whether the AI systems have
been thoroughly tested, and provide explanations on why particular decision are
made
Fairness, which would ensure that unfair bias is avoided, as it might cause
multiple negative implications, from the marginalisation of vulnerable groups, to
the exacerbation of prejudice and discrimination
Effective accountability systems that would ensure sufficient human oversight
and control of automatic decision-making to ensure that human decisionsmakers can be held responsible
Privacy and data protection of sensible information
Transparency, which would allow humans to see whether the AI systems have
been thoroughly tested, and provide explanations on why particular decision are
made
Human centricity, that must ensure that AI applications are centred on the
individual, and are always subject to some level of human oversight, especially
in decision making processes
Effective accountability systems that would ensure sufficient human oversight
and control of automatic decision-making to ensure that human decisionsmakers can be held responsible
Risks Analysis and Mitigation for Appropriate Communication
www.efsa.europa.eu/publications
105
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
On the basis of the main concerns, EFSA should take into consideration the main findings in order to lay down
effective communication messages towards its stakeholders, if it is to apply AI in its processes.
It is worth noting that these concerns reflect real risks that go hand in hand with the use of AI, which is the reason
why, at a policy and regulatory level, the European Union is making a lot of progress to ensure that organisations especially public organisations - adopt AI technology in a trustworthy way. Therefore, failing to address these risks
and failing to communicate this effectively to the stakeholders could have negative repercussion towards the agency.
Indeed, as a public institution, EFSA must be accountable towards its stakeholders, and must try to
avoid being perceived as untransparent in its processes and decisions, which could affect its reputation
among its stakeholders. Taking into consideration the main findings of our research, it clear that there are a wide
range of needs and concerns from external stakeholders on the use of AI, and therefore external communication of
these activities must be well conceived. Indeed, any miscommunication towards these stakeholders – especially to
NGOs and civil society organisations - could make them adopt specific actions against EFSA, undermining the
Agency’s reputation and perceived transparency, and lowering the accountability of the Agency’s risk
assessment activities and decisions.
In order to mitigate this risk and to be able to correctly communicate its AI activities externally, we would suggest
the following mitigation measures:
1. The correct implementation of the horizontal recommendations, and in particular, the adoption of a
Responsible AI framework (no 5) and Definition of Data Governance Framework (no 3).
2. The accurate design of customised and transparent messages that respond to the different specific
needs and concerns of the targeted audiences.
Table 25 below provides the match between stakeholder sub-categories, their concerns and the topics and concepts
that they need to address in order to develop effective communication messages, linking them with the horizontal
recommendations.
Table 25 - Concepts to integrate into Communication messages.
Stakeholders
Categories
Stakeholders
categories
Advocacy groups
Sub-
Stakeholders’
Subcategory Concerns
Topics/Concepts to integrate into Communication Messages
•
•
•
•
Transparency
Trustworthy/ Ethical AI
Accountability
Right to compensation
By following Horizontal Recommendation 5 “Adopt a
Trustworthy AI framework”, EFSA would ensure the
adoption of mechanisms, protocols and guidelines, with a
view to:
Ensure that humans are aware that they are interacting with an
AI system, and are informed of the system’s capabilities and
limitations;
Adopt AI in a responsible manner while addressing potential
ethical risks which might emerged in the maintaining and
monitoring of the AI system;
Ensure responsibility and accountability for AI systems and their
outcomes
• Consumer protection
• Right to compensation
• Consumer
discrimination
• Transparency
• Privacy
• Data protection
• Need
to
apply
regulations
By following Horizontal Recommendation 5 “Adopt a
Trustworthy AI framework”, EFSA would ensure the
adoption of mechanisms, protocols and guidelines, with a
view to:
Empower human beings, allowing them to make informed
decisions and fostering their fundamental rights;
Ensure proper oversight achieved through human-in-the-loop,
human-on-the-loop, and human-in-command approaches
Avoid unfair bias as it might cause multiple negative
implications, from the marginalisation of vulnerable groups, to
the exacerbation of prejudice and discrimination
Make accessible the system to all, regardless of any disability,
and involve relevant stakeholders throughout their entire life
circle.
NGOs
and
Civil society
Consumer
organisations
www.efsa.europa.eu/publications
106
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Stakeholders
Categories
Stakeholders
categories
Sub-
Stakeholders’
Subcategory Concerns
Topics/Concepts to integrate into Communication Messages
-
-
Ensure that humans are aware that they are interacting with an
AI system, and are informed of the system’s capabilities and
limitations;
Ensure full respect for privacy and data protection, taking into
account the quality and integrity of the data, and ensuring
legitimised access to data.
In addition, by following Horizontal Recommendation 3
“Implement a Data Governance framework which addresses
AI”, EFSA would be looking to:
Ensure that data is managed and used correctly and
responsibly, while minimising risks, leading to better AI-enabled
decision making in general
Describe policies, roles, standards, and metrics needed to
continuously improve the use of data that ultimately enables an
organisation to achieve its goals.
Ensure AI compliance with the law
Practitioners’
associations
•
•
•
•
Transparency
Human-centricity
Accountability
Trustworthy/ Ethical AI
By following Horizontal Recommendation 5 “Adopt a
Trustworthy AI framework”, EFSA would ensure the
adoption of mechanisms, protocols and guidelines, with a
view to:
Empower human beings, allowing them to make informed
decisions and fostering their fundamental rights;
Adopt AI in a responsible manner while addressing potential
ethical risks which might emerged in the maintaining and
monitoring of the AI system;
Design, develop, and deploy AI systems to learn from and
collaborate with humans in a deep, meaningful way
Ensure responsibility and accountability for AI systems and their
outcomes
It is worth noting that EFSA’s messages on trustworthy AI adoption should not be vague, but grounded by tangible
actions and results about the way AI processes are evaluated, approved and integrated within EFSA. For this reason,
the implementation of Recommendation 5 is particularly important.
Otherwise, EFSA could face a major risk of falling into “ethics washing” communication, which is the practice of
fabricating or exaggerating an organisation’s interest in an ethical AI system, without being compliant to its principles
in a second instance. This could send mixed messages to its stakeholders, and make it lose credibility.
5.4.
Engagement Plan
To allow a smooth adoption of AI technology in the EFSA evidence management process, a proper engagement
plan needs to be set up to target those stakeholders with the highest relevance, and therefore positioned on the
right side of the matrix. In this regard, we can distinguish two types of stakeholders:
•
•
[UPPER-RIGHT] Stakeholders that have already collaborated with EFSA in relation to the scope of
AI for this project (e.g., Institutions, Academia, Software companies), positioned in the upper right part of
the matrix. EFSA should aim to maintain and reinforce the relationship with them, or plan new collaboration
opportunities to devise and implement AI projects.
[LOWER-RIGHT] Stakeholders (e.g., international organisations, Digital Innovation Hubs), positioned
in the lower right of the matrix. They are associated with a high relevance, since collaborating with them
could be beneficial for EFSA in view of AI adoption projects. However, they have never collaborated with
EFSA on the topic of AI. Therefore, they should be targeted with engagement activities aimed at including
them into EFSA's AI ecosystem, in order to foster possible collaboration opportunities.
www.efsa.europa.eu/publications
107
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Figure 37 - Maintain & Reinforce Collaboration and Engage in the ecosystem.
•
It is worth noting that EFSA already has an established Engagement Approach72, in line with its strategy, which
can be used with a wide range of stakeholders. The approach follows specific goals, principles and steps, and foresees
several mechanisms and tools of engagement and consultation (e.g., discussion groups, information sessions). As
EFSA engagement approach provides a general framework to be followed, regardless of the specific domain, we
recommend customising this approach, to a certain extent, to the specific scope of this project around the adoption
of AI technology for evidence management, to reflect all peculiarities of the AI stakeholders’ ecosystem.
In this regard, we recall that section 3.3. describes a list of possible collaboration models to be potentially established
with each stakeholder category for each phase of the AI development lifecycle, including modalities to develop and
procure innovation, scouting instruments and ideas generation.
In this section, we make a step forward by mapping (see Table 26) for each different stakeholder category/subcategory the EFSA main objectives of collaboration for AI adoption (as a result of the Relevance/Impact/Easiness
to collaborate assessment), multiple optimal collaboration models to establish such collaboration, the
stakeholders’ specific needs (motivation to collaborate) and the tools73 that support EFSA in the engagement
model.
The analysis proposed in the table below provides the engagement tools that are more likely to be used and more
effective in relation to the objective and motivation for the stakeholder to collaborate. In order to choose the most
suitable method, EFSA must take into account aspects such as the budget available, the time constraints, the number
of resources available and the specificities of this project.
For example, Knowledge Fairs are proposed for collaboration models such as Strategic alignment, as they are
events designed to share large amounts of information from numerous sources. They are conducted at a common
venue and through fishbowls, world cafes or brainstorming in mind mapping exercises, and they are the perfect way
to encourage interaction, share experiences, disseminate and promote best practices, and understand peer
perspectives. Considering these points, Knowledge Fairs could be a good tool for Strategic Alignment activities,
although EFSA should also take into consideration that they would need a significant budget, not only for
implementing the event (logistics, travel, support personnel, among others) but also for the preparation phase
(experience documentation, promotion, and marketing), as well as for the monitoring phase (knowledge transfer,
follow up on agreements or statements, mobilisation of communities of practice, etc) 74. This argument can be applied
also to the remaining engagement tools provided in Table 26, meaning that EFSA must think about which will be the
most effective and appropriate tools for each specific occasion.
72https://www.efsa.europa.eu/sites/default/files/EFSA%20Stakeholder%20engagement%20approach_FINAL.pdf
73
To map the tools, the EFSA stakeholder engagement toolkit was consulted https://www.efsa.europa.eu/sites/default/files/documents/engagement-toolkit.pdf
74 https://www.shareweb.ch/site/Learning-and-Networking/sdc_km_tools/Documents/Knowledge%20Fair_Undp.pdf
www.efsa.europa.eu/publications
108
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
www.efsa.europa.eu/publications
109
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by
the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure.
The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an
output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and
the conclusions reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Table 26 - Engagement plan.
Stakeholder
Category
Stakeholder
Sub-Category
EFSA
Main
Objective
Collaboration/Coop
eration Models
Needs and Added value for EFSA to Collaborate
Motivation
collaborate
of
stakeholders
to
Strategy:
European
Commission
Maintain
Reinforce
Collaboration
&
o Align on existing and planned AI strategies, principles, and
policies of different actors
Ecosystem:
Strategic Alignment
o Sharing of plans, knowledge, lessons
learned and best practices
o Learn policies and standards for ethical use of AI from other
organisations
o Line up AI use cases and opportunities from different actors
o Set grounds for future cooperation along the following phases of
the AI lifecycle
Strategic Alignment
Ecosystem:
EU Agencies
Maintain
Reinforce
Collaboration
&
o Learn policies and standards for ethical use of AI from other
organisations
o Sharing of plans, knowledge, lessons
learned and best practices
o Opportunity for future collaboration for
AI adoption
o Scouting for new technologies options
o Alignment and/or collection of feedback on the Technology
roadmap development
Design:
o Participate in a joint action with shared budget, governance and
risks for the acquisition or co-development of AI solutions
o Implementation
o Participate in a joint action with shared budget, governance and
risks for the deployment and integration of AI solutions
• Public events (Workshops,
Webinars, Technical meetings)
• Discussion Groups and Forums
• Stakeholder Forum
• Focus Group
• Online Platforms
75
• Vision Factory
• PESTEL and SWOT analysis
• Open space technology76
Info-sessions,
• Envisioning the future 77
• Knowledge Fair
• Conferences
Strategy:
o Align on existing and planned AI strategies, principles, and
policies of different actors
Institutions
Collaboration/Cooperation Tools
Cooperation
programme
o Creation of synergies, sharing best
practices, lessons learned and resources
o Shared risk and costs
• Public events (Workshops,
Webinars, Technical meetings)
• Discussion Groups and Forums
• Stakeholder Forum
• Focus Group
• Online Platforms
• Vision Factory
• PESTEL and SWOT analysis
• Open space technology
• Envisioning the future
• Knowledge Fair
• Conferences
Info-sessions,
• Public events (Workshops,
Webinars, Technical meetings)
• Focus Group, Interviews
• Brainstorming
• Reverse Brainstorming
• Online platform
• Consensus building processes
Info-sessions,
75
It is a combination of World Cafés for the future development of complex topics, which can be used, for instance, to create engagement and consensus on a joint vision.
Open Space Technologies is a method to organise participation events, basically of large and medium scale. It can be used to support informal learning, brainstorming,
networking, deal making and collaboration within groups that have identified common goals; address highly complex central themes that no single person or small group can
understand completely; design action plans.
77 It is a scenario-building method that invites collective reflection about plausible futures. It works by imagining a time in the future (three to six years ahead), and assumes
that the organisation, section or field presence has achieved important goals.
76
www.efsa.europa.eu/publications
110
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
Stakeholder
Sub-Category
EFSA
Main
Objective
Needs and Added value for EFSA to Collaborate
Collaboration/Coop
eration Models
Motivation
collaborate
of
stakeholders
to
Collaboration/Cooperation Tools
• Ad hoc working group
• After action review (AAR)78
Design:
o Access to European funds to undertake R&D projects with other
organisations to design innovative solutions
Funding
Projects
(e.g., Horizon
Europe)
Design:
o Creation of synergies, sharing best
practices, lessons learned and resources
o Shared risk and costs
o Visibility
Contest for prize
o Contribution to the research and
development of innovative products and
services
o Line up AI use cases and opportunities from different actors
the
o Set grounds for future cooperation along the following phases of
the AI lifecycle
Strategic Alignment
Ecosystem:
o Learn policies and standards for ethical use of AI from other
organisations
o Sharing of plans, knowledge, lessons
learned and best practices
o Opportunity for future collaboration for
AI adoption
o Scouting for new technologies options
o Alignment and/or collection of feedback on the Technology
roadmap development
Design:
National
Competent
Authorities (incl.
Art. 36 org)
Maintain
Reinforce
Collaboration
&
o Delegate the design of innovative solutions through the release
of a grant, through effective and efficient partnership with
competent organisations that are active in the field of EFSA, and
are designated by the respective Members States
• Online platform
• Website
o Reward and/or profit
Strategy:
o Align on existing and planned AI strategies, principles, and
policies of different actors
Engage in
ecosystem
Info-sessions,
o Reputation
o Outsource to different third parties the design and testing of an
innovative AI solution in the market, for which the organisation
has not the resources and/or competences, aimed at improving
public services effectiveness and efficiency, and solving new
problems
International
Organisations
• Public events (Workshops,
Webinars, Technical meetings)
• Focus Group, Interviews
• Brainstorming
• Reverse Brainstorming
• Online platform
• Consensus building processes
• Ad hoc working group
• After action review (AAR)
o Creation of future opportunities for
collaboration
Grant
Implementation:
o Strengthen relationship with EFSA
o Easiness to collaborate (i.e., reduced
administrative burden)
• Public events (Workshops,
Webinars, Technical meetings)
• Discussion Groups and Forums
• Stakeholder Forum
• Focus Group
• Online Platforms
• Vision Factory
• PESTEL and SWOT analysis
• Open space technology
• Envisioning the future
• Knowledge Fair
• Conferences
Info-sessions,
• Technical Meetings, Workshops
• Design Thinking Project79
• Brainstorming
• Reverse Brainstorming
• Concept mapping80
78
It is a simple process to review a project, an activity, an event or a task. In an AAR, the individuals involved discuss what happened, why it happened, what went well, what
needs improvement, and what lessons can be learned from the experience, with a view to doing as well or better next time.
79 It is a design methodology that provides a solution-based approach to solving problems. It is extremely useful in tackling complex problems that are ill-defined or unknown,
by understanding the human needs involved, and by re-framing the problem in human-centric ways. This is pursued by stimulating the creation of innovative ideas in
brainstorming sessions, and by adopting a hands-on approach in prototyping and testing through the five stage of Design Thinking.
80 Concept mapping is a structured process, focused on a topic or construct of interest, involving input from one or more participants, that produces an interpretable pictorial
view (concept map) of their ideas and concepts, and how these are interrelated.
www.efsa.europa.eu/publications
111
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Stakeholder
Category
Stakeholder
Sub-Category
EFSA
Main
Objective
Needs and Added value for EFSA to Collaborate
Collaboration/Coop
eration Models
Motivation
collaborate
of
stakeholders
to
o Delegate the deployment and transition of innovative solutions
through the release of a grant, through effective and efficient
partnership with competent organisations that are active in the
field of EFSA, and are designated by the respective Members
States
Design:
o Access to European funds to undertake R&D projects with other
organisations to design innovative solutions
Collaboration/Cooperation Tools
User committee81
Online Platforms
After action review (AAR)
Participatory design
Interviews, Survey, Focus group, webinars (for
implementation
• Group interview with a co-design session82
•
•
•
•
•
Funding
Projects
(e.g., Horizon
Europe)
Design:
o Creation of synergies, sharing best
practices, lessons learned and resources
o Shared risk and costs
o Visibility
• Public events (Workshops,
Webinars, Technical meetings)
• Focus Group, Interviews
• Brainstorming
• Reverse Brainstorming
• Online platform
• Consensus building processes
• Ad hoc working group
• After action review (AAR)
Info-sessions,
o Reputation
o Outsource to different third parties the design and testing of an
innovative AI solution in the market, for which the organisation
has not the resources and/or competences, aimed at improving
public services effectiveness and efficiency, and solving new
problems
Contest for prize
o Contribution to the research and
development of innovative products and
services
• Online platform
• Website
o Reward and/or profit
Data:
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
definition of suitable data requirements) to gain efficiencies and
flexibility
Implementation
o Outsource to a third-party the deployment and transition of an
off-the-shelf AI solution that already meets the requirements and
needs, in a cost and time effective way with relatively low risks
Operating
o Funds to pursue their activities
Procurement
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
definition of services levels, incident, and problem management
procedures) to gain efficiencies and flexibility
o Establishment of a potential long-term
relationship
o Reputation
•
•
•
•
Technical Meetings, Workshops
After action review (AAR)
Online Platforms
Interviews, Survey, Focus group, webinars (for
implementation)
Monitoring
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
performance monitoring of the solution) to gain efficiencies and
flexibility
Industry
81
82
Software
companies
Maintain
Reinforce
Collaboration
&
Procurement
Data:
o Profit
• Technical Meetings, Workshops
• After action review (AAR)
• Online Platforms
This method involves users and other stakeholders in the formal monitoring and steering of the research and innovation process.
The group interview with a co-design session will provide feedback about the research scenarios presented.
www.efsa.europa.eu/publications
112
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Stakeholder
Category
Stakeholder
Sub-Category
EFSA
Main
Objective
Needs and Added value for EFSA to Collaborate
Collaboration/Coop
eration Models
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
definition of suitable data requirements) to gain efficiencies and
flexibility
Motivation
collaborate
of
stakeholders
to
o Establishment of a potential long-term
relationship
Collaboration/Cooperation Tools
• Interviews, Survey, Focus group, webinars (for
implementation)
o Reputation
Implementation
o Outsource to a third-party the deployment and transition of an
off-the-shelf AI solution that already meets the requirements and
needs, in a cost and time effective way with relatively low risks
Operating
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
definition of services levels, incident, and problem management
procedures) to gain efficiencies and flexibility
Monitoring
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
performance monitoring of the solution) to gain efficiencies and
flexibility
Design:
o Outsource to a third-party the design and testing of an innovative
AI solution in the market, for which the organisation has not the
resources and/or competences, aimed at improving public
services effectiveness and efficiency, and solving new problems
o Request for a demonstration of feasibility and quality of the
solution, through a Proof of Concept
o Profit
Innovation
Procurement
o Establishment of a potential long-term
relationship
o Reputation
Implementation:
Outsource to a third-party the deployment and transition of an
innovative AI solution in the market, aimed at improving public
services effectiveness and efficiency, and solving new problems
Design:
o Access to European funds to undertake R&D projects with other
organisations to design innovative solutions
Funding
Projects
(e.g., Horizon
Europe)
Design:
Contest for prize
www.efsa.europa.eu/publications
the
o Visibility
• Public events (Workshops,
Webinars, Technical meetings)
• Focus Group, Interviews
• Brainstorming
• Reverse Brainstorming
• Online platform
• Consensus building processes
• Ad hoc working group
• After action review (AAR)
Info-sessions,
o Outsource to a third-party for non-competitive use cases (for
which a start-up tends to perform better than a stand-alone
solution for one specific application in one company) the design
113
o Contribution to the research and
development of innovative products and
services
• Online platform
• Website
o Reward and/or profit
Design:
Engage in
ecosystem
o Shared risk and costs
Technical Meetings, Workshops
Design Thinking Project
Brainstorming
Reverse Brainstorming
Concept mapping
User committee
Online Platforms
After action review (AAR)
Participatory design
Interviews, Survey, Focus group, webinars (for
implementation
o Reputation
o Outsource to different third parties the design and testing of an
innovative AI solution in the market, for which the organisation
has not the resources and/or competences, aimed at improving
public services effectiveness and efficiency, and solving new
problems
AI startups
o Creation of synergies, sharing best
practices, lessons learned and resources
•
•
•
•
•
•
•
•
•
•
Innovation
Procurement
o Profit
o Establishment of a potential long-term
relationship
•
•
•
•
Technical Meetings, Workshops
Design Thinking Project
Brainstorming
Reverse Brainstorming
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Stakeholder
Category
Stakeholder
Sub-Category
EFSA
Main
Objective
Needs and Added value for EFSA to Collaborate
Collaboration/Coop
eration Models
and testing of an innovative AI solution in the market, for which
the organisation has not the resources and/or competences,
aimed at improving public services effectiveness and efficiency,
and solving new problems
Motivation
collaborate
of
stakeholders
to
o Reputation
o Growth and stability
o Request for a demonstration of feasibility and quality of the
solution, through a Proof of Concept
Implementation:
Collaboration/Cooperation Tools
•
•
•
•
•
•
•
Concept mapping
User committee
Online Platforms
After action review (AAR)
Participatory design
Innovation Jam
Interviews, Survey, Focus group, webinars (for
implementation
o Outsource to a third-party the agile deployment and transition of
an innovative AI solution in the market, aimed at improving public
services effectiveness and efficiency, and solving new problems
o Reputation
Design:
o Outsource to different third parties the design and testing of an
innovative AI solution in the market, for which the organisation
has not the resources and/or competences, aimed at improving
public services effectiveness and efficiency, and solving new
problems
Contest for prize
o Contribution to the research and
development of innovative products and
services
• Online platform
• Website
o Reward and/or profit
o Growth and stability
Design:
o Outsource to a third-party for non-competitive use cases (for
which a start-up tends to perform better than a stand-alone
solution for one specific application in one company) the design
and testing of an innovative AI solution in the market, for which
the organisation has not the resources and/or competences,
aimed at improving public services effectiveness and efficiency,
and solving new problems
o Request for a demonstration of feasibility and quality of the
solution, through a Proof of Concept
University
offs
Spin-
Engage in
ecosystem
the
o Profit
Innovation
Procurement
o Establishment of a potential long-term
relationship
o Reputation
o Growth and stability
Implementation:
•
•
•
•
•
•
•
•
•
•
Outsource to a third-party the agile deployment and transition of an
innovative AI solution in the market, aimed at improving public
services effectiveness and efficiency, and solving new problems
Technical Meetings, Workshops
Design Thinking Project
Brainstorming
Reverse Brainstorming
Concept mapping
User committee
Online Platforms
After action review (AAR)
Participatory design
Interviews, Survey, Focus group, webinars (for
implementation
o Reputation
Design:
Outsource to different third parties the design and testing of an
innovative AI solution in the market, for which the organisation has
not the resources and/or competences, aimed at improving public
services effectiveness and efficiency, and solving new problems
Contest for prize
o Contribution to the research and
development of innovative products and
services
• Online platform
• Website
o Reward and/or profit
o Growth and stability
Data:
Academia
Universities and
research centres
(incl. Art. 36 org)
Maintain
Reinforce
Collaboration
&
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
definition of suitable data requirements) to gain efficiencies and
flexibility
o Funds to pursue their activities
Procurement
Implementation
o Outsource to a third-party the deployment and transition of an
off-the-shelf AI solution that already meets the requirements and
needs, in a cost and time effective way with relatively low risks
o Establishment of a potential long-term
relationship
o Reputation
•
•
•
•
Technical Meetings, Workshops
After action review (AAR)
Online Platforms
Interviews, Survey, Focus group, webinars (for
implementation)
Operating
www.efsa.europa.eu/publications
114
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Stakeholder
Category
Stakeholder
Sub-Category
EFSA
Main
Objective
Needs and Added value for EFSA to Collaborate
Collaboration/Coop
eration Models
Motivation
collaborate
of
stakeholders
to
Collaboration/Cooperation Tools
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
definition of services levels, incident, and problem management
procedures) to gain efficiencies and flexibility
Monitoring
o (Partially) outsource to a third-party non-core activities or
activities for which skills are not fully developed internally (e.g.,
performance monitoring of the solution) to gain efficiencies and
flexibility
Design:
o Delegate the design of innovative solutions through the release
of a grant, through effective and efficient partnership with
competent organisations that are active in the field of EFSA, and
are designated by the respective Members States
Implementation:
o Creation of future opportunities for
collaboration
Grant
Design:
o Access to European funds to undertake R&D projects with other
organisations to design innovative solutions
o Strengthen relationship with EFSA
o Easiness to collaborate (i.e., reduced
administrative burden)
o Delegate the deployment and transition of innovative solutions
through the release of a grant, through effective and efficient
partnership with competent organisations that are active in the
field of EFSA, and are designated by the respective Members
States
Funding
Projects
(e.g., Horizon
Europe)
o Creation of synergies, sharing best
practices, lessons learned and resources
o Shared risk and costs
o Visibility
Design:
o Outsource to a third-party the design and testing of an innovative
AI solution in the market, for which the organisation has not the
resources and/or competences, aimed at improving public
services effectiveness and efficiency, and solving new problems
o Request for a demonstration of feasibility and quality of the
solution, through a Proof of Concept
o Funds to pursue their activities
Innovation
Procurement
o Establishment of a potential long-term
relationship
o Reputation
Implementation:
o Outsource to a third-party the deployment and transition of an
innovative AI solution in the market, aimed at improving public
services effectiveness and efficiency, and solving new problems
Design:
Technical Meetings, Workshops
Design Thinking Project
Brainstorming
Reverse Brainstorming
Concept mapping
User committee
Online Platforms
After action review (AAR)
Participatory design
Interviews, Survey, Focus group, webinars (for
implementation
• Group interview with a co-design session
• Public events (Workshops,
Webinars, Technical meetings)
• Focus Group, Interviews
• Brainstorming
• Reverse Brainstorming
• Online platform
• Consensus building processes
• Ad hoc working group
• After action review (AAR)
•
•
•
•
•
•
•
•
•
•
Info-sessions,
Technical Meetings, Workshops
Design Thinking Project
Brainstorming
Reverse Brainstorming
Concept mapping
User committee
Online Platforms
After action review (AAR)
Participatory design
Interviews, Survey, Focus group, webinars (for
implementation
o Reputation
o Outsource to different third parties the design and testing of an
innovative AI solution in the market, for which the organisation
has not the resources and/or competences, aimed at improving
public services effectiveness and efficiency, and solving new
problems
www.efsa.europa.eu/publications
•
•
•
•
•
•
•
•
•
•
115
Contest for prize
o Contribution to the research and
development of innovative products and
services
• Online platform
• Website
o Reward and/or profit
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Stakeholder
Category
Stakeholder
Sub-Category
EFSA
Main
Objective
Needs and Added value for EFSA to Collaborate
Collaboration/Coop
eration Models
the
stakeholders
to
o Alignment and/or collection of feedback on the Technology
roadmap development
o Widen the engagement outreach of EFSA towards various
stakeholders, whenever a procurement, contest for prize or grant
etc. is launched in the context of AI adoption, in order to create
synergies
Info-sessions,
Contact
• Public events (Workshops,
Webinars, Technical meetings)
• Online Platforms
• Open space technology
• Conferences
• Brainstorming
• Reverse Brainstorming
• Concept mapping
• E-mails
• Social Media/website
Info-sessions,
o Assist technology transfer by providing
networking and advocacy, and finding a
qualified commercial partnership to
successfully and diligently develop a
technology discovered and researched at
a university and research institutions
Contact
• Public events (Workshops,
Webinars, Technical meetings)
• Online Platforms
• Open space technology
• Conferences
• Brainstorming
• Reverse Brainstorming
• Concept mapping
• E-mails
• Social Media/website
Info-sessions,
o Point of contact by providing strong
linkages with different actors inside and
outside of its region, promoting the
implementation of multiple digital
opportunities in the industry and the
public sector, sustainability of their
processes and products
o Support to public sector organisations in
the use of digital technology
Contact
Consultation
and
o Point of contact by providing strong
linkages with different actors inside and
outside of its region, promoting the
implementation of multiple digital
opportunities in the industry and the
public sector, sustainability of their
processes and products
Design and Implementation
Development
Levers
Technology
transfer office
Engage in
ecosystem
the
o Widen the engagement outreach of EFSA towards various
stakeholders, whenever a procurement, contest for prize or grant
etc. is launched in the context of AI adoption.
o Establish a partnership with Academia that has discovered and
researched ground-breaking technologies
o Access to knowledge, bringing skills and experience to the
surface, which are necessary tools for the implementation of EFSA
aspirations
Incubators and
accelerators
Engage in
ecosystem
www.efsa.europa.eu/publications
the
Ecosystem
o Widen the engagement outreach of EFSA towards various
stakeholders, whenever a procurement, contest for prize or grant
etc. is launched in the context of AI adoption.
o Access to knowledge, bringing skills and experience to the
surface, which are necessary tools for the implementation of EFSA
aspirations
116
Collaboration/Cooperation Tools
• Public events (Workshops,
Webinars, Technical meetings)
• Online Platforms
• Open space technology
• Conferences
• Brainstorming
• Reverse Brainstorming
• Concept mapping
• E-mails
• Social Media/website
o Scouting for new technologies options
Engage in
ecosystem
of
o Provision of access to technical expertise
and experimentation needed for a
successful digital transformation.
Ecosystem:
Digital
Innovation Hubs
Motivation
collaborate
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between
the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is
subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions
reached in the present document, without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Stakeholder
Category
In addition, the following table shows other cross-cutting collaboration/cooperation models with a wide
outreach and related tools, which can be employed for any stakeholder category/sub-category:
Table 27 - Collaboration models and related tools
Collaboration/Cooperation
Models
Public consultation
Internet crowdsourcing
Hackathon
5.4.1.
Needs and Added value for EFSA to
Collaborate
Stakeholders’ Motivation to
collaborate
Strategy:
• Understand the opinion of the
ecosystem’s actors on the adoption
of AI strategies and principles
within organisations to guide
decision-making
Strategy:
• Access to a large audience with
different expertise to collect new
ideas and opportunities on the AI
topic
Ecosystem:
• Scouting for new technologies
options
Design:
• Generate ideas for new AI solutions
and services, stimulating the
ecosystem of actors
•
•
•
•
•
•
Contribution
to
the
technology transformation of
public sector
Opportunity
for
future
collaboration
for
AI
adoption
Contribution
to
the
technology transformation of
public sector
Monetary or non-monetary
awards
Contribution
to
the
technology transformation of
the public sector
Experience gain
Collaboration/Cooperation
Tools
•
IT
tools
Connect.EFSA)
•
Online Platforms
Innocentive)
•
•
Open Platforms
Design Thinking Project
(via
(e.g.
Communication on AI adoption
Although our approach suggests that EFSA should set up an engagement plan for the stakeholders
placed in the right part of the matrix (to engage new actors in the ecosystem or develop/strengthen
collaboration with existing actors), this does not exclude the parallel need to foresee some
communication activities (through the methods and channels EFSA has available) towards these
actors, regarding EFSA’s strategy, objectives, plans and activities on the adoption of AI for evidence
management. This is important to spread awareness among stakeholders, and to sow the ground for
future collaboration opportunities.
Moreover, to promote certain modes of collaboration it would be necessary to adopt more specific
communication activities, as in the case of the calls for the purchase of AI solutions, or the contests
for a prize, addressing specific stakeholders’ categories interested directly or indirectly (i.e., informed
through development levers, e.g. DIHs).
5.5.
Final Considerations on Communication and Engagement –
Timeline and dependencies
To conclude, we outline some final considerations on the correct time and existing dependencies of
the communication and engagement plans with other recommendations for the implementation of
the communication plan and engagement plan.
•
•
Communication Plan: it should be launched once Horizontal Recommendation 5 on the
Trustworthy AI framework is correctly implemented (or at least planned and launched), to avoid
the risk of ethics washing. For this reason, the first step of the Recommendation’s activities
(namely, the development of the Code of Conduct) should start quite soon, as shown in Table
18.
Engagement Plan: this can start immediately based on the needs, prior to and during the
adoption of AI technology, in order to reinforce, widen and cooperate with EFSA's AI ecosystem.
Therefore it is not subject to any particular dependency with other recommendations or
constraints.
www.efsa.europa.eu/publications
117
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried
out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s),
awarded following a tender procedure. The present document is published complying with the transparency principle to which
the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority
reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document,
without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
In addition, it is worth considering that these plans can be implemented both at a macro and a micro
level. In the first case, the plans relate to an entire programme, which contains multiple AI projects
(horizontal and vertical recommendations), and therefore said plans envisage more generic and strategic
communication and engagement activities, with a long-term strategic vision on the adoption of AI for
EFSA's evidence management.
In the second case, the activities of the communication and engagement plans are tailored to the specific
needs of the AI projects (e.g. organisation of a workshop with another EU agency for aligning on the
development of an AI trustworthy framework).
6.
Conclusions
The output of this roadmap will serve as a decision-making tool to identify and prioritise the AI projects
to be carried out from 2022 to 2027, and to optimise the performance of evidence management through
AI in order to achieve the objectives presented in EFSA's theme paper.
By implementing the projects devised in this roadmap, EFSA will have developed the fundamental
capabilities that will enable the integration of AI within the organisation, building not only the technical
instruments to manage, control, and use AI responsibly, but also in terms of skills and cultural change.
In addition, EFSA will have already integrated some AI solutions in several use cases, resulting in a
more efficient, performative and automated evidence management process 83. These projects will be
particularly relevant for building the Agency’s capabilities to integrate AI tools inside the evidence
management process, but also in other processes, thus forging synergies with the needs of other
roadmaps (e.g., the EU Partnership for next generation, systems-based environmental risk assessment
- PERA - and the New Approach Methodologies - NAMs), and of the organisation’s readiness towards AI
adoption as a whole.
Table 28 - Focus on synergies with other roadmaps.
New Approach Methodologies (NAMs roadmap)
A major element of the application of New Approach Methodologies (NAMs) is the need for integrated
approaches for testing and assessment (IATAs), defined approaches for data interpretation, and
performance-based evaluation of test methods. An issue identified by the NAMs project team is the
need for improved data-integration tools to facilitate the gathering of multiple disparate data sources
and the transformation of these data sources into standardised format for processing.
While no vertical recommendations were identified as relevant to the NAMs roadmap, Horizontal
recommendations around the Development and adoption of a Data Governance, Data Architecture
for Big Data and AI are both recognised as foundational elements for building the appropriate
infrastructure and tools to achieve improved data extraction, integration and standardisation to
support the use of NAMs in chemical risk assessment. In addition, once AI techniques are applied in
NAMs for decision support in risk assessment, the adoption of a Responsible AI framework will also
be of paramount importance.
EU Partnership for next generation, systems-based Environmental Risk Assessment
(PERA roadmap)
83
As already explained in the previous chapters, with regard to the 10 prioritised use cases, the optimal vender
scenario for developing or acquiring tailored AI-solutions (vertical recommendations) has been already selected
thanks to the market scouting performed during the data collection activities of the project. However, the vendor
selection approach developed in section 3.3.2.1 can be applied to all AI use cases, and therefore EFSA might use
it to identify the best way to adopt AI solutions also in other – not prioritised – use cases.
www.efsa.europa.eu/publications
118
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried
out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s),
awarded following a tender procedure. The present document is published complying with the transparency principle to which
the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority
reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document,
without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
One of the major issues of the PERA project is the resource-intensiveness when retrieving and
analysing relevant information from databases and websites. Use cases included in our vertical
recommendations that can introduce time-efficiencies in this field are the SLR Literature deduplication, SLR Relevant Abstract Screening, SLR Keyword Identification and Automatic Text
Summarisation.
An additional pain point of PERA is the variability between the outcomes of peer-reviews conducted
by different risk assessors. AI can help in this case through the application of topic modelling
approaches or Automatic Text Summarisation, which can help simplify the procedure by helping PERA
to produce informative summaries per each peer-review. These summaries will contain key elements
of the review, which can be used in an AI tool (such as Comparable Appraisal Detection) to identify
similarities and discrepancies. Lastly, PERA team suggested that an AI tool that will be able to evaluate
how well an interview has been transcribed into text format could really benefit in terms of time and
workload reduction. PERA has its own database of texts that have already been transcribed, and
therefore this can be used as “training data” for the AI tool, and to further increase its accuracy.
Modifying slightly the use case of Data Collection Terminology Assessment would help address the
challenge, since the logic of assessing the text format and terms is the same.
Thanks to the results that will be achieved by implementing these projects, it will be possible and
appropriate after 2027 to rethink the evidence management process as a whole, redesigning it in a
more integrated way to leverage the full potential of AI technology (with a technology-driven approach),
for example, eliminating steps and sub-processes that are no longer necessary, or merging steps that
were previously separated and could be carried out simultaneously. To fill the gap between “adopting
and integrating AI” and “embedding AI” in the organisation, thus making it not only an instrument of
automation but also a means of holistic innovation, can be considered as the objective to achieve in the
long run.
In the meantime, EFSA should monitor the key emerging technological trends that will shape the future
public sector innovation scenario, which might further boost a smooth adoption of AI within the
organisation. In addition, EFSA should also focus attention on its stakeholder ecosystem, keeping the
most relevant actors engaged and aligned on the future outcomes of the roadmap and on future
activities, with the aim of creating a flourishing AI community around EFSA, and building solid
partnerships and collaborations.
Concerning the engagement of the AI stakeholder ecosystem, it is worth considering that the
stakeholders that we called “innovation development levers” (i.e., accelerators, digital innovation hubs),
as presented in section 3.3.2, will play a more impactful and relevant role in the years to come, due to
an increase in EU and national funding for their development. In fact, building on and complementing
various national initiatives within the digital industry, the Commission is acting to trigger further
investments in the digitisation of industry, and thereby support the creation of better framework
conditions for the digital industrial revolution. For instance, the EC is further developing the network of
(European) Digital Innovation Hubs through the Digital Europe Programme, which will increase the
innovation potential of European SMEs, also with a focus on specific technologies such as AI [1]. At
national level, investments are being made in several EU Member States as well to develop accelerators
and incubators. These kinds of organisations can provide EFSA with strong linkages with innovative
companies inside and outside of its region, promoting public services innovation and providing an
exchange environment where calls for ideas and innovation procurement can be launched, drawing on
the knowledge, approaches and skills of startups, thereby contributing to EU digitalisation by driving
demand through open innovation. Therefore, although in the short run these types of stakeholders will
[1]
https://digital-strategy.ec.europa.eu/en/activities/edihs
www.efsa.europa.eu/publications
119
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried
out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s),
awarded following a tender procedure. The present document is published complying with the transparency principle to which
the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority
reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document,
without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence
probably have a minor role, in the middle term they will have built the capacity to be an important part
of EFSA's AI stakeholders’ ecosystem.
Another emerging trend to consider in the future - after 2027 - is the European AI On Demand Platform
and Ecosystem84, developed by the AI4EU consortium (established in January 2019) with the support
of the European Commission under the H2020 programme. The platform is a one-stop-shop for anyone
looking for AI knowledge, technology, tools, services and experts, and aims to increase innovation and
technology transfer, accelerate the growth of start-ups and SMEs, and fulfil the needs of the European
AI community. It will enable access to a catalogue of AI-based resources and a marketplace providing
central, easy and simple access to trustworthy AI tools developed in Europe. This will be another asset
that EFSA can leverage, supporting its own process towards the adoption of AI and the implementation
of the Adopt AI programme.
In the next 5 years leading up to 2027, EFSA can follow a structured guideline/roadmap for actions,
with key associated tools, to adopt AI for evidence management. Using the adoption of AI for evidence
management as a reference and best practice for other organisational processes, EFSA may build the
fundamental capabilities for an appropriate, organisation-wide AI implementation.
84
https://www.ai4europe.eu/
www.efsa.europa.eu/publications
120
EFSA Supporting publication 2022: EN-7339
The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried
out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s),
awarded following a tender procedure. The present document is published complying with the transparency principle to which
the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority
reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document,
without prejudice to the rights of the authors.
23978325, 2022, 5, Downloaded from https://efsa.onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2022.EN-7339, Wiley Online Library on [30/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Roadmap for actions on artificial intelligence