Multi-actor, multi-component
evaluation of the EU Framework
Programmes
EVALUATION 2009
Orlando, Florida, USA
2 November 2009
Neville Reeve
European Commission DG RTD
FP7 /1
EUROPEAN COMMISSION – DG Research – 12 November 2009
European
Commission
and Directorates General
European Union
Institutions
European Commission
Council of
the
European
Union
European
Managing the
Parliament
Framework Programme
European
Commission
Secretariat
General
DG Research
DG Information
Society and
Media
Court of
Justice
DG External
Relations
FP7 /2
DG External
Relations
DG Budget
DG Energy &
Transport
Joint Research
European
Centre
Court of
Auditors
DG Agriculture
& Rural dev’t
DG Enterprise
DG Regional
Policy
DG Competition
EU research
 Framework Programme
(FP) is a multi-annual
research funding
programme implemented
at EU level
 7th Framework
Programme (2006-2013) €50 billion – 27 Member
States – more than 100
participating countries –
tens of thousands of
transnational co-funded
projects
FP7 /3
EU Research Framework Programmes
Annual Budgets between 1984 and 2013
€bi l l i on
12
10
8
6
4
2
0
1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012
NB: budgets in current prices. Source: Annual Report 2003, plus FP7 revised proposal
FP7 /4
Budgets of the EU
Framework Programmes
2007-2013
2.7
50.521
2002-2006
17.500
1998-2002
14.960
1994-1998
13.120
6.600
1990-1994
1987-1991
5.360
3.270
1984-1987
0
10
20
30
Billion euros
FP7 /5
40
50
60
EU research: changing priorities
100
90
Other
80
Basic research
Coordination & development
%
70
Space
60
Science and society
Training of researchers
50
Dissemination & exploitation
40
International cooperation
Socio-economic
30
Transport
20
Energy
Life sciences
10
Environment
Industrial & materials technology
FP7 /6
FP7
FP6
FP5
FP4
FP3
FP2
FP1
0
IT and Communications
FP7 – Indicative breakdown (€ million)
FP7 /7
Multi component evaluation system
FP 5
FP 6
FP 7
Annual monitoring
FP6 ex post
5 year assessment
FP7 Ex ante Impact Assessment
FP8 Ex ante Impact Assessment
FP7 interim evaluation
Thematic level evaluations
National Impact studies
1998
FP7 /8
2000
2002
2004
2006
2008
Evaluation Actors, Networks &
Standards
 Actors
 Political level – Council, European Parliament
 Commission level – Secretariat General
 DG level – central units, operational level
 Member States evaluation
 Networks
 DG Research Evaluation Network
 European RTD Evaluation Network
 Commissions internal Evaluation Network
 European Commission evaluation standards
FP7 /9
Indicators for FP Monitoring (1)
Indicator / issue
Promotion of FP7
Performance of the calls
Performance of the proposal
evaluation and redress procedure
Sub-indicator
1.1
Number of information days
Annual NCP Survey
1.2
Number of attendees at information days
Annual NCP Survey
1.3
Commission organised meetings of NCPs
DG RTD
2.1
Success rate (overall) by priority area and funding scheme
CORDA
2.2 Success rate for different types of organisation by priority area and funding
scheme
CORDA
2.3 Success rate for different types of organisation by priority area and funding
scheme & success rates per country
CORDA
3.1 Overall quality assessment of the proposal evaluators on the FP proposal
evaluation process (evaluators survey)
Annual Evaluators'
Survey
3.2 Assessment of quality by the evaluators between the FP evaluation process and
other equivalent systems (evaluators survey)
Annual Evaluators'
Survey
3.3
Time to grant agreement
CORDA
3.4
Percentage of experts reimbursed within the specified 45 days
DG RTD/PMO
3.5
Redress cases upheld (i.e. leading to a re-evaluation) – numbers and percentages
DG RTD
4.1
Average results of independent project review process by priority area
Data from new reporting
system (not existing yet
for 2008)
4.2
Percentage of projects by priority area covered by reviews
Data from new reporting
system (not existing yet
for 2008)
Quality of on-going research
projects
Project performance by outputs
FP7 /10
Main Data Source
5.1 Average number of project publications per project by priority area and funding
scheme
Data from new reporting
system (not existing yet
for 2008)
5.2 Average number of other forms of dissemination activities per project by priority
area and funding scheme
Data from new reporting
system (not existing yet
for 2008)
5.3 Average number of different types of intellectual property protection per project by
priority area and funding scheme
Data from new reporting
system (not existing yet
for 2008)
Indicators for FP Monitoring (2)
6.1
Total number of active projects by priority area
CORDA
6.2
Average financial size of projects by priority area and funding scheme
CORDA
6.3
Participation by types of organisation by priority area funding scheme
CORDA
6.4
Participation totals per country
CORDA
7.1
Number of male and female coordinators in proposals
CORDA
7.2
Number of male and female coordinators in projects
CORDA
7.3
Gender breakdown (by seniority) of project participants
CORDA
FP activity
Achieving gender equality
Observing sound ethical
principles in FP research
Performance International
Cooperation activities
Simplification
FP7 /11
7.4 Percentage of male and female members in Advisory Groups and
Programme Committees
DG RTD
8.1 Number of projects going through the review process/ % by area/
programme
DG RTD
8.2 Number of ethics reviews where the result showed insufficient attention
had been given in proposal
DG RTD
8.3
Number of projects stopped as a results of the ethics review
DG RTD
8.4
Number of ethics screenings
DG RTD
9.1 Total numbers of participations of Third Countries by priority area and
funding scheme
CORDA
9.2
Success of Third Countries in calls by priority area and funding scheme
CORDA
9.3
EC contribution to Third Countries
CORDA
9.4
Number of international outgoing/incoming fellowships
DG RTD
10.1 Do stakeholders perceive that the FP is getting simpler to use in terms of
financial and administrative procedures?
Annual NCP Survey
Evaluation Roadmap
 FP6 Ex-post evaluation in 2008
 FP7 Progress report in 2009
 ERC review 2009
 FP7 interim evaluation in 2010
 Ex ante Impact Assessment (FP8) in 2011/12
 Ex evaluation FP7 in 2015
FP7 /12
Evaluation of the 6th FP
- Evidence and Analysis
EXPERT
GROUP
Impact studies by
Member States
Self-assessments;
interviews;
consultations
Studies at
Thematic level
Supporting Experts
analyses
Studies at FP level
Statistical analysis of FP implementation and
annual Monitoring Reports
FP7 /13
Evaluation study of FP structuring – (1)
 FP6 networks are characterised by a core periphery
structure dominated by a small number of closely-knit
organisations.
 Industry plays less of a coordinating role and is weakly
embedded in the FP6 networks with the exception of IST
 FP6 has resulted in more upstream than downstream
outputs
 Performance decreases as project size increases
 New Member State participation is dominated by a few
actors in Poland, Hungary and the Czech Republic.
FP7 /14
Evaluation study of FP structuring (2)
- Network structure of the FP6 thematic areas
FP7 /15
Evaluation study of FP structuring (3)
- Who coordinates whom in FP6 (Universities, Industry,
Institutes)
FP7 /16
Evaluation study of FP structuring (4)
- Who coordinates whom in FP6 (New Member
States)
FP7 /17
Using bibliometrics to assess
participation (1) - Relative citation rate of lead
scientists, by FP priority – year 2002, 2004, 2006
1_LIFE_SCIENCE_HEALTH
2_IST
3_NMP
4_AERO_SPACE
5_FOOD
6_ENERGY
6_GLOBAL
7_CITIZENS
EURATOM
INCO
2006
POLICY_SUPPORT
2004
2002
Total
0
FP7 /18
0.5
1
1.5
2
Using bibliometrics to assess
participation (2) - Lead scientists’ 2–year Relative
1,6
1,4
1,2
1,0
0,8
0,6
0,4
0,2
A
FR
A
N
LD
ES
P
BE
L
CH
E
D
N
K
SW
E
G
RC
FI
N
IS
R
PR
T
AU
T
IT
Lead Scientist RCR
x
FP7 /19
G
D
d share (%)
BR
0,0
EU
R
PR
T
AU
T
Relative Citation Ratio (RCR)
Citation Rate (RCR), all priorities taken together
2,0
1,8
WOS_RCR
http://ec.europa.eu/research/evaluations/
FP7 /20
FP7 /21
FP7 /22
Dr Neville Reeve
Directorate-General for Research
Unit for Evaluation and Monitoring of
Programmes
Phone +32-2-298 93 29
Fax +32-2-295 40 82
neville.reeve@ec.europa.eu
http://ec.europa.eu/research/evaluations
FP7 /23
EUROPEAN COMMISSION – DG Research – 2 November 2009