THE EVALUATION OF RESEARCH
Ten years after the launch of the Lisbon Strategy, the European Union is facing a global economic situation
that requires the adoption of extraordinary measures to move from the normal management of a crisis in a
shared policy of structural reforms. The commitment for Europe to become "the most competitive and
dynamic knowledge-based economy in the world" meant that EU countries should increase investment in
research up to about 3% of their GDP, but unfortunately this did not happened all over the continent. The
current crisis raises awareness that only research and development lays the foundation for a long term
strong economic growth. In presence of scarce resources, the identification of the correct mechanisms to
evaluate the outcomes of the research funding plays an even more crucial role.
INFN has always focused on control of its own research programs, thanks to the existence of different
structures that perform ex-ante, ongoing and ex-post evaluation of experiments and initiatives. With the
new Statuto, the Executive Committee and Board of Directors have, in addition to the Scientific
Committees, a Council, which oversees the technical-scientific decisions on new large-scale projects
proposed by such Committees and provides additional advisory opinion.
Since 1997 the Board has entrusted the overall evaluation of its activities to the judgment of an
International Evaluation Committee (CVI), which draws on an annual basis a report on the quality of
research INFN and provides guidance and recommendations to improve the overall performance. The CVI is
made of international experts of the highest authority, in the fields where the Institute conducts its own
research, and in sectors that are interested or related, such as industrial production or, more generally, the
economic one. No researcher, employee or associate INFN, is a member of CVI to guarantee the
impartiality of the work of the Committee.
The CVI meets annually the President of the Institute, the Executive Committee and the Presidents of the
Scientific Committees at a meeting lasting several days in which there is a thorough discussion of the
scientific activities of the Institute and lines of future planning. The meeting is attended by the Coordinator
of the Working Group on Evaluation (GLV). Since 2000 this working group build the process of selfassessment according to the criteria recommended by the Ministry through the CIVR (Committee for the
Evaluation of Research). It is composed by 5 subgroups, one per each line of activity, that have the task to
collect in a systematic way objective data describing the scientific performance of the INFN (included where
appropriate in an international context), together with elements useful to show both the activities of higher
education of young people carried out within the research institute, and the socio-economic and interdisciplinary activities of the organization. The Working group produces a summary document with the data
above mentioned, which is used by the CVI to prepare the final evaluation of research. The CVI is also a
reference point for the Ministry of Education, to which it annually submits its final report.
The approach used by the INFN is thus based on two components: one is expressed through objective data
indicators recognized by experts, and secondly the global analysis of their significance and scientific
activities of the Entity in its entirety by a peer group. It would be counterproductive to a proper evaluation
of a complex structure as the INFN, reducing the entire process to a mere list of indicators to be compared
with external data, ignoring the history of scientific advances (experiments and research that are in
different stages at different times) or added value resulting from the implementation of a major research
infrastructure, ahead of the publication of an article in a magazine.
After the Triennial Evaluation Research (VTR 2001-2003), 2012 marked the long-awaited step in the
completion of the year covering the seven years 2004-2010 (Quality Evaluation of Research, VQR),
managed by ANVUR. INFN, through its instruments of internal evaluation, has timely prepared and
transmitted the requested material. During the definition and transmission of data, the computerization of
most aspects of the life of INFN and of the information relating to employees and associated has played a
very positive role, allowing to obtain and certify the information required in a short time. In the future,
these tools will certainly refined and made more flexible.
It should be noted in this context that some parameters contained in the Guidelines for the VQR are
difficult to adapt to the nature of the research conducted by INFN and its modus operandi. For example,
the construction of large research facilities (in whole or in part) does not appear as a specific indicator.
In the following we will illustrate some of the elements that contribute to the process of evaluation of
research in INFN, in order to describe the process itself and to provide useful elements interpretation of
what is presented.
Fundamental research is the basis of the work of INFN , therefore it is obvious that scientific publications
are the main proxy for the productivity of the organization. During the 2001-2003 VTR, MIUR
recommended as a source of bibliometric data Web of Science (WOS), the Institute for Scientific
Information (ISI) (available by subscription). Since then ISI-WOS has been used by GLV as the source of all
primary data concerning publications: the criteria used by ISI-WOS also ensure that the items belong only
to journals of international level, where papers are accepted only after a rigorous process of peer review1.
Number of ISI Publicatons
2011
2010
2009
2008
2007
<04-06>
CSN1
300
277
195
256
280
296
CSN2
294
259
238
219
192
205
CSN3
276
258
223
206
266
255
CSN4
1112
1183
1099
1191
1236
1127
CSN5
361
320
326
333
325
264
Common
355
428
397
334
193
276
INFN
2700
2721
2478
2539
2492
2423
INFN/Italia
36
36
33
34
32
32
Tab. xx.1: Number of INFN publications in ISI database. Last line shows (ISI-Thomson Reuters data) the
percentage of INFN papers on the total number of Italian papers in Physics and Astrophysics.
However, it is also useful to recall how ISI publications are not the only channel used to disseminate
scientific findings in the fields of research proper to INFN activities. For example, INFN contributes
significantly to the preparation of reports for major international laboratories such as CERN or Fermilab, or
similar publishing projects, such as online publications, both in the context of collaboration with foreign
1
ISI WOS di Thomson Reuters is one of the two DB used by ANVUR in VQR 2004-2010
colleagues or on behalf of international organizations. In the coming years, also distribution of publications
in electronic format will become the most popular way to communicate scientific results and the editorial
policies of open access will play an increasing role. INFN is closely following this evolution, as a member of
the initiative SCOAP3 (Sponsoring Consortium for Open Access Publishing in Particle Physics). Financing
through a consortium seems to be the most promising model for the activities of the organization. This
approach is already operational for the great collaborations of the Large Hadron Collider (payment of a fee
by the Institute and corresponding on-line availability of all publications in international journals of
prestige). The recent (July 2012) choice of the British Government to make the results of all the publicly
funded researches open access by the end of 2014, certainly represents an acceleration in this direction.
Fraction of INFN Authors (%)
2011 2010
2009
Average Impact Factor
2008 2007 <04-06> 2011 2010 2009 2008 2007 <04-06>
CSN1
22
38
30
42
37
36
4,77
3.80
3.90
3.10
3.65
3.78
CSN2
51
51
53
64
64
75
3.8
4.08
4.40
2.80
2.89
2.15
CSN3
43
50
44
51
53
47
3.21
2.85
2.60
2.80
2.58
2.60
CSN4
61
55
56
63
58
59
3.71
3.73
3.73
3.47
3.62
3.44
CSN5
57
66
61
67
56
66
1.72
1.97
1.96
1.70
1.54
1.46
Tab. xxx.2: Some indicators of scientific productivity related to INFN-published papers.
Table xxx.1 shows the total productivity of INFN in 2011 (divided into the five scientific lines) compared
with the average result of the three year period 2004-2006 and subsequent years. The total in each column
exceeds the sum of the individual rows, since there are publications which are not directly attributable to a
single CSN, for example because they are made by the authors of different area (example: one theoretical
and one experimental).
It can be noted in particular the large number of publications in theoretical field (CSN4), since it reflects the
excellence of the Italian school in the field. It is also noted that in recent years the contribution of CSN1 has
increased, effect due to the publication of the first analysis of the data collected at the LHC while still the
experiments at SLAC and Fermilab publish their results.
Finally, it is clear that the Institute is has a very important role (last row) in the panorama of the Italian
research in Physics. The number of publications is only one of the parameters that can be considered in the
evaluation of scientific productivity. The use of the database ISI-WOS among other things allows direct
access to other bibliometric indicators, such as the Impact Factor (IF, designed precisely by ISI-Thomson), or
to perform more complex analyses related to the number of citations. The Impact Factor is derived from
the Journal of Citation Reports, published by ISI and characterize the quality of the corresponding journals.
The impact factor represents the average citations of articles published in a given journal in a period of two
years. In this context, therefore, it can at most be used to compare the journals between them, but this
indicator does not provide per se information on the quality of a single article published. Even in the first
sense, extreme caution must be used in the use of the IF, especially when comparing different disciplines,
whose researchers publish in journals with editorial policies that can be quite varied. Still remains an
important cross check between the feedback you get from the bibliometric indicators and those obtained
with a peer.
In table xxx.2 are summarized some of the parameters used to evaluate in a quantitative manner the
quality and characteristics of the scientific productivity of the organization. The mean value of Impact
Factor is constant over the years for each of the scientific lines. The average value of the articles of CSN5 is
typical of journals in technology and instrumentation, as compared to those which collect the results of
theoretical and experimental physics, and perfectly illustrates the caveats discussed above on the need to
differentiate the evaluation with respect to the characteristics of the specific scientific field . In fact, the
GEV (Evaluation Expert Group) for the group of physics appointed dall'ANVUR for VQR 2004-2010, working
on a mostly bibliometric process, started by dividing the articles presented in appropriate homogeneous
subsets.
The fraction of INFN authors is indicative of the level of internationalization that characterizes the research
activities of the Institute in each sector. Even in this case, as for the IF, it should be noted that the average
value is extracted from multi-modal distributions: in the case of CSN1, for example, it is obtained by
mediating articles with few authors with the articles of Collaborations at the LHC, which have about three
thousand authors each. In fact, the drop from 38 to 22% of the fraction of INFN authors for CSN1,
corresponds to the higher impact of articles published by the big LHC collaborations. Conversely, the mean
IF for publication increases because these collaborations have published their findings in physics journals of
prestige with high IF.
The complexity, the size and the time duration of major INFN projects in nuclear physics, subnuclear and
astro-particle requires a constant control at all stages of the experiments, by the construction to function
tests, until data taking and analysis . Here evaluation of research carries out two important roles: on the
one hand is to prevent long-term projects to run into difficulties that can jeopardize the success of the
experiment, on the other is a tool to assess the relevance given to the INFN in roles of responsibility in
Collaborations.
The first role is implemented through the National Scientific Committees that use referees (Italian or also
foreign) to examine the status of each project (usually twice a year). Each experiment at the time submits
budget requests for the following year, agrees with the referee also a set of milestones to be met in the
same time span. This mechanism allows a peer ex-ante, ongoing and ex-post evaluation of individual
projects and allows to timely intervene to solve problems whenever they could appear.
Milestone Achievemt (%)
Leadership roles(%)
2011 2010 2009 2008 2007 <04-06> 2011 2010
2009
2008 2007 <04-06>
CSN1
85
89
73
79
79
80
27
23
30
26
26
25
CSN2
78
63
56
68
70
79
56
55
57
43
39
51
CSN3
83
84
86
83
84
78
47
50
45
37
37
39
Tab. xxx.3: Some performance indicators for Collaborations and Research Groups
Table xxx.3 shows the degree of completion of the agreed milestones for experimental collaborations: as
you can see, a large percentage is respected. The mechanism allows, in a broader sense, to implement
actions where and when necessary. It should also be noted that, because of the complexity and
international scientific projects, sometimes delays in the realization of its objectives can also be induced by
motivations not directly related to the work of INFN groups.
Together with the high level of internationalization of the activities it is interesting to evaluate the fraction
of positions of responsibility (leadership) that are assigned to INFN personnel within Collaborations (the
definition of roles is usually provided by agreements approved by the management bodies of the
experiments). This is shown, still in Table xxx.3, for the three lines with experimental activities. The result is
indeed very good and better than the bare contribution/fraction of INFN personnel in the Collaborations,
an important recognition of the scientific and managerial skills of its researchers.
In this context, the most important highlight is that in 2011 all four International partnerships operating at
the Large Hadron Collider at CERN in Geneva were led by scientists of Italian nationality, three of which are
directly affiliated to the INFN and the fourth formed within INFN. We expect in the future to be able to
maintain this level, if at all possible to make an adequate generational change.
In this regard it is interesting to note that the INFN plays an important role in university and post graduate
education in Italy. Table xxx.4 reports the number of theses in the total area of in physics in Italy (font:
MIUR) and from our own data for INFN. It is clear that our activities have a high level of interest in young
people and, thanks to the presence of our facilities, we play –as such- an important role in the development
of the country. It also confirms the role of motivation and attraction that research in fundamental physics
has on the younger generation. This commitment to education and postgraduate, will be further
strengthened. Not only at the individual level, where many researchers INFN already put their knowledge
at the disposal of the University in a synergic way as supervisors of undergraduate and graduate students
as well as faculty for specific courses. The year 2012 indeed saw the foundation of the GSSI (Gran Sasso
Science Institute), an experimental international Doctorate School where INFN plays a leadership role.
Laurea Magistralis
Ph. D.
2011 2010 2009 2008 2007 <04-06> 2011 2010 2009 2008 2007 <04-06>
INFN
Total
386
288
302
368
333
332
174
141
139
163
153
180
808
868
907
859
990
n/a
n/a
381
374
342
388
Tab. xxx.4 Number of theses and PhDs in Physics (font: MIUR, Ufficio di Statistica) and in INFN (our data).
INTERNATIONAL COMPARISON
The Italian research in nuclear sub-nuclear and astro-particle (experimental and theoretical) physics is at
level comparable (or higher), in terms of both quantity and quality to that of similar countries inside and
outside Europe. This of course is also related to the international setting where INFN works.
The internationalization of research conducted by the INFN can be seen easily by looking at the number of
publications produced in collaboration with foreign colleagues. Table xxx. 5 shows for each line the
percentage of scientific publications in
international cooperation: the different
values for different CSN reflect the different
sociological, as well as financial, texture of
the different research lines.
CSN1 and CSN3 are examples where
substantially all of the publications are
produced in international collaboration but
also in the field of theoretical physics (CSN4)
you can see an increasing tendency to
collaborate with foreign colleagues.
2011
2010
2009
2008
2007
<04-06>
CSN1
99
96
96
96
95
95
CSN2
69
73
64
68
64
72
CSN3
94
93
85
91
92
95
CSN4
68
64
64
62
60
57
CSN5
25
21
24
21
23
20
Tab. xxx.5 Percentage of INFN pubblications in international
collaborations
It is also interesting to know what are the most important partners for papers published in international
collaboration. The table xxx.6 reports the analysis by considering all INFN publications for 2011 in journals
accredited by ISI. The global ranking INFN broadly reflects the one of its Scientific Committees, however,
the weights of the various countries are different. This is a sign of the different composition of
collaborations and also is related to the investments in large infrastructures in a given country. For CSN4
CSN5 and you may also notice the absence of a clear ranking, indicating that the type of collaboration with
foreign colleagues is geographically distributed. Another new feature for 2011 is the appearance of new
collaborations. For example, for CSN4, Netherlands (7%) and Poland (5%).
Recently (January 2012) Institute of Physics (IOP) commissioned a report (for United Kingdom) to Evidence
(a company directly linked to Thomson-Reuters) on "Bibliometric Evaluation and International
benchmarking of the UK's Physics research" in physics and its sub-disciplines This report represents a
source of independent international assessment. As the report is aimed at a comparison of the UK with the
rest of the world, provides a series of tables where, for each sub discipline, there are figures for various
countries (including Italy) for the period 2001-2010. It appears from these data that Italian researchers in
2010 have contributed to 5% of the total number of publications in physics (for comparison UK: 6.4, U.S. 22,
and France 7.6 Germany 10.5%). However, if we limit ourselves to two subcategories "Particle and Fields"
and "Nuclear Physics", the fraction of articles rises to 11.2 (third place after USA and Germany) and 8.7%.
respectively. Moreover, in recent years, these fractions are constant or slightly increasing for the sectors of
INFN interest, unlike the Italian general trend that has seen a decline from global maximum (6% in 2004) for
the decade to date (5 % in 2010)2.
The publication of an article in large international collaborations is often the result of a collective work and
that can take a long time. From the list of authors it is not simple to infer whether there were particular
contributions and to what extent by individual researchers. In large collaborations, due to the competitive
situation, however, is not simple even to have the opportunity to present scientific results at an
international conference. Therefore in order to assess the role of INFN researchers, we count the number
of presentations to a set of the most important (and recognized by the community) international
conferences and we compare to how many areassigned to researchers from other countries.
2
Font: Institute of Physics, UK. Available at http://www.iop.org/publications/iop/2012/page_53959.html
This comparison for 2011 (in parentheses the average 2007-2010) is shown in table xxx.7 for the three
scientific lines CSN1, CSN2 and CSN3 normalized to the total number of presentations. The result shows
that the INFN researchers are highly valued and that the work to educate, educate and integrate young
people into the great scientific experiments allows the Institute to create a robust generation of scientists
who will be the actors of the development of research and of future discoveries.
Italy
Germany
France
UK
USA
Japan
CSN1
10(13)
13(10)
9(7)
10(10)
25(29)
3(4)
CSN2
10(11)
11(13)
7(7)
5(3)
33(27)
8(11)
CSN3
10(10)
18(14)
7(8)
3(4)
30(24)
3(8)
Table xxx.7 Fraction (in %) of talks given by INFN personnel at a set of International Conferences.
xxx.3 A EUROPEAN PERSPECTIVE ON EVALUATION
INFN is a member of various European and international organizations and contributes to the definition of
the relevant multi-annual activities, with regard to the mission of the Institute in the field of fundamental
physics, both theoretical and experimental.
The European Science Foundation (ESF) is one of these organizations, and was born after the initiative of
several institutions and agencies (now 80 in 30 countries) that have set as its goal the cooperation and
collaboration in European scientific research, also considering the needs of providers of financial resources
(stakeholders). ESF is working in close collaboration with other institutions in Europe, such as the European
Commission itself as well as with partners outside Europe such as the NSF (National Science Foundation)
and the 'NIH (National Institutes of Health) from U.S. and OECD (Organization for Economic Cooperation
and Development). INFN is also a founding member of EuroHORCS (European Heads of Research Councils)
which is now evolved into Science Europe. Science Europe (hereafter SE), born in October 2011www.scienceeurope.org-in the coming years will assume an important role in the creation of the European
Research Area (ERA). ERA was launched in Lisbon in 2000 and relaunched with new momentum in 2007 . In
Science Europe sit together both research organizations and government agencies funding. The purpose is
to promote a coordinated development of European research that maintains the level of excellence
allowing national organizations to better coordinate their efforts. The foundation of SE (which was
attended by Italy INFN and CNR) has seen the simultaneous dissolution of EuroHORCS.
To strengthen and facilitate ERA in the coming years we expect a restructuring of organizations in the field
at European level, and the Institure is fully involved in this process. We Recall that the deployment of ERA
should be completed by 2014 and that the European Commission has launched an acceleration of this
process in view of Horizon 2020.
The European Science Foundation (ESF) has devoted some time to the harmonization of the instruments of
government research. INFN is part of ESF since its founding in 1974 (the only institution together with the
Italian National Research Council -CNR) and is represented in internal structures of the organization such as
Physics and Engineering Sciences Standing Committee (CFSP) and the Nuclear Physics European
Collaboration Committee (NuPECC ). For several years the role of ESF focused on the creation of the
European Research Area, which must be characterized by a dynamic science policy, loans based on the
quality and merit, the mobility of researchers, students and the financial resources and the creation of
adequate research facilities.
To identify common practices, using the experiences of its members, ESF has organized Fora discussion to
support the exchange of national experiences with the aim of identifying a shared set of practices that can
be transferred to different realities. All Fora will complete their reports by the end of 2012. Those reports
will be useful tools for the European Commission and, more generally, to various research actors to
strengthen, by sharing common practices, ERA. Recently (July 2012), the report on Science in Society was
presented. It provides a set of recommendations on the relationship between science and society to both
funding agencies and research organizations.