The European Research Council
ERC Monitoring & Evaluation
Strategy
European Research Council
Outline of the presentation
Established by the
European Commission
 Strategy for Monitoring and Evaluation of ERC funding
activities
 Implementing the strategy
 CSA projects
│2
European Research Council
Increasing pressure to account for public
funds
Established by the
European Commission
 Accountability in Principal/Agent Model
─
─
Principal : right to monitor (ensure that agent acts in the interest of
Principal)
Agent : obligation to account for resources
 Evolving “accountability”
─
─
─
─
Core question : “what have we done with the money”
Traditionally :
─ “Which projects do you fund?”
Currently : increased expectations
─ “which results have you achieved?”
─ “what is their significance- for science, economy, society” ?
Changing question
─ not if R&D investments produce benefits
─ but how are your investments produce benefits (or contribute to
producing benefits)
│3
European Research Council
ERC Monitoring and Evaluation (M&E) Strategy
Established by the
European Commission
• ERC implements the "Ideas" Specific Programme of the EU's 7th
Research Framework Programme. FP evaluated by EC.
• Roadmap of the Evaluation of the 7th FP by the EC
 Progress Report - 2009
 Interim Evaluation - 2010
 Ex-post Evaluation – 2015
• Coordination: EC - Unit A.5 Evaluation
│4
European Research Council
Role of ERC M&E
Established by the
European Commission
Inform decision making of the ERC council
and ERCEA senior management
Monitoring
Evaluation
Provide evidence base for the ERC Assessment
(mid-term review and the ex-post evaluation)
Inform stakeholders, partners and the
public on ERC activities, performance and impact
(incl. reporting obligations of the ERC/ERCEA)
│5
European Research Council
ERC Approach – I
Established by the
European Commission
Dimensions
Performance
Direct impacts
Accountability
& Information
Science
Management
Advancing
knowledge &
dissemination
Emerging
research
areas
Impact on
researchers
Impact on
research
organisations
Structural impacts
Derived impacts
Objectives
Components
Economic
benefits
Societal
benefits
Management
Efficiency
Stable, effective
administration
Training
Reinforce excellence,
dynamism and creativity
Impact on
research
policies and
funding
structures
Attract best researchers
and industrial research
investment
Better exploit research
assets and foster
innovation towards a
dynamic knowledge
based society
│6
European Research Council
ERC Approach - II
Established by the
European Commission

It takes time for results to be produced and years for the impact to unfold.
Only rigorous and carefully designed studies can help uncover the impact
External
Expertise
Call for proposals
Call for tenders
Expert Groups
Internal
capacities

Independent academics
(to be explored)
Good research information play a critical role
│7
European Research Council
Implementation of ERC M&E Strategy
Challenges
Established by the
European Commission
• Managing the limited resources the ERC can devote to M&E activities
• Securing the high quality data required to deliver the strategy
• “over-fishing“ and “evaluation fatigue”
• Data quality
• Adequate methodologies to measure impacts
• impact not easy to measure and/or to attribute
• impact of ERC funding will certainly need some time to
emerge into recognizable effects
│8
European Research Council
Data sources
Established by the
European Commission
•
•
Process-produced data (in-house data)
External data:
 Collected by CSA funded projects
 Collected from specialized databases (e.g. bibliographic systems such as
Web of Science, Scopus …)
 Data collected from other sources (Eurostats, Erawatch ...)
│9
European Research Council
Data Analysis Tools
Established by the
European Commission
•
AVEDAS – ERC research information system:
 Integration of different (internal & external) data sources through user
analytics module
 Enabling basic analysis in support of monitoring and evaluation of ERC
operations
 Flexible and open for new data and analysis developments, for example
integration of bibliometric tools developed by CSA projects
│ 10
European Research Council
Monitoring European Research Council‘s
Implementation of Excellence [MERCI]
•
Coordination: Professor Stefan Hornbostel (IFQ)
•
Aim:
to understand the impact of the grants in fours areas:
Established by the
European Commission
(1) career development
(2) institutional environments (e.g. on factors which favour or hinder
the establishment of independent research groups);
(3) structural effects in the European landscape (mobility) and
(4) contribution to academic advancement
(i.e. impact of publications of grant holders on the advancement of
their fields).
│ 11
European Research Council
Understanding and Assessing the impact and
outcomes of the ERC funding schemes
[EURECIA]
Established by the
European Commission
•
Coordination: Dr Maria Nedeva, Manchester Institute of Innovation
Research (MIoIR), University of Manchester, UK
•
Aim: develop and apply a methodology to measure and attribute impacts of
the two ERC funding schemes in three areas:
(1) on researchers (career path, skills, networking possibilities)
(2) on research organisations and
(3) on national as well as European research funding organisations.
│ 12
European Research Council
Development and Verification of a Bibliometric
model for the Identification of Frontier Research
[DBF]
Established by the
European Commission
•
Coordination: Katy Whitelegg, Austrian Institute of
Technology
•
Aim: compare the ranking of applicants made by the panels
(peer-review) and ranking based on the frontier research
indicators measured through bibliometrics (publication records
and citations from proposal submission)
│ 13
European Research Council
Emerging Research Areas and their Coverage by
the ERC-supported projects [ERACEP]
Established by the
European Commission
•
Coordination: Dr Thomas Reiss, Fraunhofer Institute Systems and
Innovation Research, Karlsruhe, Germany
•
Aim: to identify topically ‘emerging researach areas’ and assess to which
extend the ERC-supported projects cover and to contribute those emerging
scientific areas
* Identification of ‘emerging areas’ on basis of bibliometric analysis (SCI,
SSCI, A&HICI, years : 1998-2007)
│ 14