OECD Global Science Forum
Workshop on Methodologies and Tools for assessing Socio-Economic Impact
of Research Infrastructures
3 November 2015
Espace Saint-Martin
199 bis, rue Saint-Martin 75003 Paris
ANNOTATED AGENDA
Introduction
Shared Research infrastructures (RIs) play an ever growing role in scientific research. They are now
actively developed and used in most scientific domains, and allow for many new and breakthrough
research discoveries. Concomitantly, building and operating RIs also requires a growing share of
public research funding, and government and research funding institutions are therefore increasingly
concerned in the value for money that these infrastructures provide.
Governments and funding agencies could benefit from reliable studies on the socio-economic impact
of RIs, and indeed routinely request such information to inform the funding of new or up-graded RIs.
However, given that the primary objective of a RI is to support excellent science, traditional costbenefit analysis models (as used for industrial or transport infrastructures) can only be applied to a
limited extent and other models and appropriate indicators are necessary. This is complicated by the
fact that many of the most important impacts of research are long-term and unpredictable.
In 2014 The Global Science Forum (GSF) set up an Expert Group to examine potential priorities for
RI policy that should be addressed at the international (global) level. One of the highest priorities was
evaluation of the socio-economic impact of research infrastructures
The general agreement among members of the GSF Expert Group was that socio-economic impact
evaluation should not be conducted independently of scientific impact and that RI impact assessments
should be conducted in a comprehensive manner. Such assessments should include impact on
science, on economy (technology and knowledge transfer, innovation etc.), skills (training etc.) and
society at large. It was recognised that some categories of impact are difficult to assess quantitatively,
but that a multi-factorial combination of indicators should be able to give an informative picture for
assessing and comparing the impact of infrastructures over time.
The GSF secretariat has carried out a literature survey of existing reports and a preliminary analysis of
the models currently in use to evaluate the impact of research infrastructures worldwide. Although
there are clear overlaps in the strategies, models and tools developed and used by the various players,
this survey suggests that a number of very practical questions are as yet unanswered. The results of
this literature analysis suggest that a standardised reference framework could potentially be developed
to facilitate evaluation of the impact of RIs.
The objective of this workshop is to review and agree on the existing gaps and demands regarding
socio-economic impact assessment methodologies and tools, and to explore the feasibility of
developing and adopting a shared framework, with better standardised data and modelling processes,
for assessing the impact of RIs. Whilst significant efforts have already been made in this respect at
1
the European level, the aim is to explore what is desirable and feasible at a global level. This should
take into account needs of funding organisations as well as research infrastructure operators and user
communities for different types of research infrastructures.
Meeting organisation
The meeting will take place room Karnak at the Espace Saint-Martin 199 bis, rue Saint-Martin 75003
Paris
Information on the logistics are described in annexe.
Draft agenda
9:00
Welcome from French Research Ministry and OECD Secretariat
9:05
Introduction, objectives of the workshop (OECD secretariat)
9:25
Session 1 : Current landscape (Lucilla Alagna, OECD consultant)
09:55

RIs taxonomy and specificities

Increasing demand

Current diversity in impact assessment models and indicators

Identified challenges and remaining gaps
Session 2: Socio-economic impact of research infrastructures: current models
(moderator: Jean Moulin, BELSPO/ESFRI)
Examples of existing state-of-the-art models in the field
- Evaluating and Monitoring the Socio-Economic Impact of Investment in
Research Infrastructures (Frank Zuijdam, Jelena Angelis, Technopolis)
- Cost-Benefit Analysis Framework for evaluating RIs (Massimo Florio,
University Milan)
- The societal footprint of big science (Edwin Horlings, Rathenau instituut)
- EU new impact assessment policy applied to RIs (Reka Rozsavolgyi, EC)
10:55
Coffee break
11:15
Socio-economic impact of research infrastructures: current models (continued)
Open discussion
1. What are the methodologies and uses for ex ante and ex post analyses (costbenefit and impact assessments) ?
2. What are the advantages/disadvantages/challenges of different approaches ?
3. How are scientific, economic and social impacts integrated in overall
assessments ?
12:30-13:30
lunch
2
13:30
Session 3: Existing indicator categories and data (moderator: Peter van den
Besselaar, University of Amsterdam)

Examples of current practices
 Europe single site RI (ESRF, Christian Vettier)
 Europe distributed RI (PRACE, Philippe Segers)
 Japan single site RI (Visualising publication and patent clusters and
networks for assessing socio-economic impact of RIs, Sonoko
Watanabe, MEXT)
 International distributed RI (SKA, Daniel Adams, South African
Department of S&T)
 Open discussion
 What are the limitations and challenges in collecting and validating
data ?
 What is the right balance between qualitative and quantitative
indicators
 What are the current gaps between data collected and needs ?
15:10
Coffee break
15:30
Session 4: Desirability and feasibility of common indicator/data set (moderator:
Jean Moulin, BELSPO/ESFRI)
Developing a common framework for socio-economic impact indicators
1. Defining common needs
2. Can the same framework be developed to assess the diversity of RIs ?
16:30
Session 5: Next step
What action might be required by the different stakeholders (governments,
funding organisations, international organisations, RI administrations…).
17:15
Conclusions (OECD Secretariat)
17:30
End
3
Annotation to the agenda
Annotations to the Agenda (Sessions 1-4)
These annotations are based on empirical and analytical material that was compiled or developed
during the preparations for the workshop. This work included an extensive literature review and input
from experts with experience in assessing the socio-economic impact of research infrastructures.
Session 1 Current landscape

RIs taxonomy and specificities
The term Research Infrastructures covers a broad diversity of organisations with very different
objectives and missions (a notional taxonomy was proposed in the GSF report on distributed RIs1, but
there are many existing definitions, by ESFRI, GSO etc…). In this workshop, we consider RIs whose
primary objective is to perform or facilitate basic research, for which socio-economic impact
assessment is usually poorly developed (in contrast with RIs more focused on applied/dual purpose
research, such as wind tunnels, for which the business case usually integrates a clearer socioeconomic objective).

Increasing demand
Ex ante impact assessments are becoming standard practices in the establishment process of new RIs.
Most funding agencies now require that such assessment includes a detailed socio-economic
component beside the traditional description of the scientific added-value of the new infrastructure.
Ex ante impact assessment are also increasingly required for RIs to be included into national or
regional roadmaps, which play an important role in the prioritisation of future governments’
investments. Ex post evaluation is also increasingly demanded for RIs upgrades and even for
financing their operational costs. However, demand for such socio-economic impact assessments
rarely follows standardised or established methodologies. While it is clear that such analyses do
increasingly play a role in the funding decision-making process, the criteria that are actually used
remain heterogeneous, even though a common framework appears to exist at the European level.

Current diversity in impact assessment models and indicators
A number of reports and/or research papers have been devoted to analysing the impact of large
international research infrastructures (CERN is a typical example2). Recently, similar work has also
been focused on distributed and e-research infrastructures and to their particular problems. Socioeconomic impact is rarely studied as a single issue3. Often, reports or models address economic
impact, usually using a rigid/traditional economic analysis4, conducted with economic parameters
1.
International Distributed Research Infrastructures, GSF, OECD, 2014
http://www.oecd.org/sti/sci-tech/international-distributed-research-infrastructures.pdf
2 The Impacts of Large Research Infrastructures on Economic Innovation and on Society: Case Studies at
CERN, GSF, OECD, 2014 http://www.oecd.org/sti/sci-tech/CERN-case-studies.pdf
3
Some useful results in: Social Impact Assessment Methods for research and funding instruments through the
study of Productive Interactions between science and society,
http://www.siampi.eu/Content/SIAMPI_Final%20report.pdf
4
Hickling Arthur Low, 2013, TRIUMF, http://www.triumf.ca/sites/default/files/HAL-
ReturnOnInvestmentStudy-May-2013.pdf
Price Waterhouse, 2008, ESS, https://www.skane.se/upload/Webbplatser/Skaneportalenextern/dokument/ESSengPart_one.pdf
Matrix Insight, 2009, RIs by FP6, https://ec.europa.eu/research/infrastructures/pdf/csri.pdf
4
similar to those used for industrial or public infrastructures. Social impact, when taken into
account, is usually addressed in a more qualitative way, and there is a large heterogeneity among
social parameters to be considered. There is however a better defined framework for impact
assessment at the European level5, where a large number of studies had been conducted, both for ex
ante and ex post analysis.

Identified challenges and remaining gaps
Socio-economic impact assessments are regularly undertaken for RIs, both ex-ante and ex-post, but
remain extremely heterogeneous in their objective, performers and models used. There is indeed no
agreed model, shared between funding agencies and/or RIs’ organisations, to measure socio-economic
impact.
For economic impact assessments, an economic approach applying standard economic analysis
models developed for any non-scientific infrastructure is largely used and adapted for RIs (see
reference 4). But those are rarely fully validated and have to rely on uneven data input. Traditional
“direct” impact indicators are often available, although attribution (what is the exact share of the RI
impact into the overall economic impact) is more difficult to determine. Assessing more “indirect”
impact (for instance, on the R&D of companies involved in using or building RIs) also appears to be a
challenge.
Social impact assessments are usually far less developed, and usually rely on more qualitative
evaluations, although some metrics are available6. The definition of what constitute a social (or
societal) impact is not shared among countries and stakeholders, so can vary greatly between
assessments. This creates in itself a challenge for data collection.
RIs do collect already a wide range of valuable data that can be used for impact analysis. These data
are usually intended to describe infrastructure output. This is primarily related to scientific output
(given the primary objective of RIs) but economic/econometric data are also collected, although
following more heterogeneous criteria. Social impact data are sometimes available, but those are
usually very partial and address only a limited part of potentially valuable social impact. Good quality
historical time-series data is rarely available.
Better validated output data time series are required in a more standardised format to increase the
robustness of socio-economic impact assessment of RIs7.
Session 2 Socio-economic impact of research infrastructures: current models

Short overview of existing state-of-the-art models in the field
Economic models have been used worldwide to analyse the (potential) economic output of RIs in both
ex ante and ex post situation.
The Technopolis presentation, based on a 2013 report8, will provide a summary of the diversity of
economic and societal criteria that can be considered to assess the impacts of RIs, taking into account
5
Adrian Curaj et al: Foresight enriched Research Infrastructure Assessment Methodology (fenRIAM), ISBN:
978-973-711-318-4 http://www.fenriam.eu/overview.html
6
J. Britt Holbrook :Re-assessing the science – society relation: The case of the US National Science
Foundation’s broader impacts merit review criterion (1997 – 2011).
http://digital.library.unt.edu/ark:/67531/metadc77119
7
see for example:SFTC Impact Framework & Evaluation Strategy, 2014,
http://www.stfc.ac.uk/stfc/cache/file/B5D5D5C7-809A-44F3-990FBDC045835BAC.pdf
5
in particular both innovation and technology linked to RIs, and how they can be applied to a wide
diversity of RIs. In this regards, it might be interesting to compare the Technopolis classification with
that used in a recent review of the socio-economic impact of NASA activities analysed in a
comprehensive report9 from the Tauri Group.
The presentation from Massimo Florio will highlight the development of a specific economic model10
to help analyse the economic output of RIs, using the CERN Large Hadron Collider (LHC) as a case
study. This model shows that a quantitative evaluation of benefits can be made by determining their
value to users (scientists, early-stage researchers, businesses, visitors) and non-users (the general
public). Four classes of contributions to users are identified: knowledge output, human capital
development, technological spill-overs, and cultural effects. Benefits for non-users can be estimated,
by analogy to public goods with no immediate practical use (such as environmental preservation),
using the willingness to pay as an indicator. Applied to the LHC, this model suggests a 92%
probability that benefits would exceed costs during its lifetime.
The Rathenau Instituut contribution11 will refer to the societal impact of big science, illustrating the
difficulty of identifying adjunctive societal impact for big science facilities using traditional
methodologies. Although large scale research facilities do show evidence for creation of networks and
communities, quantitative social impact is less evident to identify.
The discussion should address how current models and tools can be used to assess RIs impact: there
are clearly limitations in identifying indirect impacts and there also is a very broad range of
social/societal impact, involving cultural and scientific advancement, technological and scientific
training, innovation, international links and exchanges, social awareness on various issues,
participation of society to science etc. How detailed or far ranging is the current demand for assessing
socio-economic impact of RIs both ex ante and ex post, and how can current models be adapted to
provide satisfactory responses?
The European Union has developed a new evaluation framework for its flagship research funding
programme Horizon 2020. The contribution from the European Commission will report on lesson
learned from previous Research Infrastructures ex post evaluation exercises and on the new model
and indicators being developed for the mid-term evaluation of H2020.
The discussion should address how new models and tools that are being developed can be used for
assessing the wider socio-economic impact of RIs, beyond traditional and direct impact evaluation.
Session 3 Existing indicator categories and data
8
Technopolis Group, Evaluating and Monitoring the Socio-Economic Impact of Investment in Research
Infrastructures,2015,
http://www.technopolis-group.com/wpcontent/uploads/2015/04/2015_Technopolis_Group_guide_to_impact_assessment_of_research_infrastructures.p
df
9
NASA Socio-Economic Impacts, The Tauri Group, 2013.
https://www.nasa.gov/sites/default/files/files/SEINSI.pdf
10
a) Massimo Florio, Emanuela Sirtori, The evaluation of research infrastructures: a Cost Benefit Analysis
framework http://wp.demm.unimi.it/tl_files/wp/2014/DEMM-2014_10wp.pdf
b) Massimo Florio, Stefano Forte Emanuela Sirtori, Cost-Benefit Analysis of the Large Hadron Collider to 2025
and beyond, arXiv:1507.05638v1 [physics.soc-ph] 20 Jul 2015 http://arxiv.org/abs/1507.05638
11
Edwin Horlings, Thomas Gurney, André Somers, Peter van den Besselaar; The societal footprint of big
science, Rathenau Instituut
http://www.researchgate.net/publication/266513583_The_societal_footprint_of_big_science
12
(Handsight study by DoD, USA, 1963)
http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=AD0495905
6
This session is intended to review which types of indicators and data are already used/collected, for
which use, what are the current challenges and needs, and whether common standards could be
developed.

Examples of current practices
Research infrastructures already collect a vast array of data and information to facilitate their
evaluation. Furthermore, in most cases, funding organisations also require some sort of ex ante socioeconomic impact assessment prior the establishment of any major new infrastructure. However,
current practices are very diverse, being a function of the type of research infrastructure, funding
institution practices, and of the complexity of the project.
Different cases will be presented to highlight current practices in the collection and use of indicators
for socio-economic assessments:


ESRF is typical of single-site international RIs. Christian Vettier will present a recently
conducted impact analysis of ESRF, which was carried out in particular to assess the
impact of its upgrade programme, and illustrate the type of information available for such
assessment.

PRACE is a good example of international distributed RIs. Philippe Segers will describe
socio-economic indicators, key performance indicators and impact assessment
methodologies used for PRACE, highlighting its specificity.

Using Spring-8 (the Japanese large synchrotron) as a case study, Sonoko Watanabe will
describe a new methodology to visualize the transition between scientific and socioeconomic impact through the analysis of patents and publications related to a research
infrastructure.

SKA is based on a unique new international partnership involving South Africa, Australia
and other partners. Daniel Adams will present the type of elements which have been used
to conduct the very detailed ex ante socio-economic impact assessment of this upcoming
large RI.
Discussion items:

Limitations and challenges in collecting and validating data
Robust ex post impact assessment requires validated data and information over the whole life
time of a given RI. What are the type of data that can be readily collected and verified which
can potentially provide most useful information for impact assessment studies, taking into
account the existing diversity of RIs ? It is possible to identify the minimum number of
shared indicators necessary to build a reliable and consistent impact assessment for different
RIs?

Qualitative vs quantitative
Econometric and scientometric indicators are widely used to evaluate RIs. However, whether
those are actually meaningful is contested, and they only provide a narrow picture of impact.
Furthermore, there is little consensus over the type of direct social/societal indicators to be
used. More qualitative indicators are probably also required, but may be more difficult to
validate. What is the range of validated indicators available that can be used to respond to the
current demand for socio-economic impact assessment of RIs ?

Existing gaps and needs
7
Life cycle and time scale is one of the most obvious challenges in RI impact assessment.
Whether you consider the concept, design, building, operations and decommissioning phases,
each of which can last several years, different set of indicators are required to evaluate RI
impact. For instance, research output typically needs to be evaluated over a long time scale
(> 20 years12) to grasp the full scale of impact.
Additional challenges relate to attribution (how to identify the specific impact of the RI
within a complex system, something already difficult to determine for scientific impact
alone), and to additionality factors (for instance, how to evaluate what additional benefit a
company is getting as a user or provider of a RI).
Session 4 Desirability and feasibility of common indicator/data set
Defining a set of agreed and shared indicators could help both funding institutions and RI
administrators in defining their RIs impact, improving decisions on funding at different stages and
helping comparison between investment in RIs and other types of investments. Many reports and
studies had been commissioned to provide a clear picture of the past and/or forecasted impact of RIs
before a funding or political decision has to take place and a commonly agreed and validated
methodology could help in providing objective input to the process. There are however several
challenges:

Defining common needs
Funding institutions, decision-makers, research infrastructure managers all have slightly different
needs for socio-economic impact assessments. Identifying those needs is a prerequisite before
defining agreed and shared indicators for impacts.

How to deal with difference between single-site research infrastructures, distributed and/or eresearch infrastructures?
The term Research Infrastructure actually encompasses a broad diversity of organisations. Their
impact, geographically, and on science, economy and society, is therefore extremely variable. Clearly,
specific indicators would need to be identified and used for each specific case. Impact assessment
might/will differ for scale (national mid-scale versus big international facilities), the type
(different impacts for different type of RIs: single sited versus distributed vs virtual) or
discipline (technical science versus social sciences and humanities). Could a set of more
generic tools be developed and relevant data be readily collected to assess some sort of baseline
impact ?
Session 5 Next step
The section will host an open discussion about possible improved international collaboration that
could help in developing a common framework of shared indicators for assessing social-economic
impact of RIs.
Participant contribution is requested in order to identify and agree on priority actions: what kind of
studies, expertise are needed to address the challenges? And which organisations could have
a role in taking it forward ? The expected results would be a consensus agreement on additional
efforts required to fill existing gaps and needs.
12
(Handsight study by DoD, USA, 1963)
http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=AD0495905
8