Research performance evaluation in the Czech Republic: current methodology and expected development Prof. Zdenek Nemecek Charles University, Prague Brief history and legal frame • The distribution of the government research support is governed by the law 130/2002 and its consecutive amendments • Evaluation of the research entities started in 2004 without any connection with financing • The amendment from 2005 states that the distribution of the support for the long-term conceptual development of research entities (SCD) among the support providers (ministries, Academy) should follow the evaluation of the results • Methodics 2008, 2010, 2013 • The providers are obliged to follow this distribution for their entities unless they have their own evaluation system R&D budget Government sources only, EU funds not included Total Grant agencies, international cooperation, etc. SCD Evaluation Methodics Keep in mind that: 1 - Methodics is dedicated for division of money among the research entities 2 – it is not supposed to be used on the level of research teams or individuals 3 - it is not intended for identification of excellence • Each research entity provides the list of their results to RIV • The results are divided into categories – journal paper, proceedings, patent, etc. • The results are further divided according topics – physics, social sciences, mathematics, chemistry, etc. • Methodics assigns a predefined number of points to each result • The money that a particular entity receives would be directly proportional to the sum of the points gained by its results Topical groups The number of points gained by a particular topical group is normalized to represent a given part of the points available (3 000 000) Medical & Health Sci. 15% 11% Biological Sci. 12% 17% Chemical Sci. Physics Mathematics 15% 5% 5% 5% 15% Earth Sci. Agri. Sci. Technical Sci. & IT Social Sci. & Humanities Categories of results The numbers of points assigned to publication and application results are normalized to represent a given part of the points available 3 281 983 points Journals (WoS) Journals (SCOPUS, ERIH) National journals 2% 3% 1% 4% 1% 4% Books 82.5% Proceedings 3% 9% Patents Utility models 5% 61% 7% Prototypes Certified methods Softw are Others 17.5% Evolution of the Methodics • The first Methodics 2008 provided a reasonable evaluation of research entities • However, the production of “results for points” started immediately • The only category of results not affected by this trend were papers in good journals (WoS) • The consecutive Methodis (2010) introduced several limitations: - percentage of applied results - division of results into topics (mathematics, social sciences, etc.) with predefined number of points These limitations brought a partial improvement but the main problem – equal number of points for all results in a given category - remains Evaluation Methodics 2013 • Three parts - Mainstays • Each mainstay has its own allocation of points • Mainstay I – old system for papers in journals indexed in WoS or Scopus, books and papers in the Czech journals come through peer review • Mainstay II - peer review evaluation of a limited number of results • Mainstay III – new system of evaluation of applications, the results are not weighted and the points are assigned to research entities. Mainstay weight • Mainstay (Pilíř) I – publication results • Mainstay (Pilíř) II – NEW – excellent results • Mainstay (Pilíř) III – applications 17.5% 10.0% Pilíř I Pilíř II Pilíř III 72.5% Mainstay I This mainstay is adopted and adapted from the previous Methodics 2008 and 2010 SHVa,bSci and Humanities Social SHVc Sci Social Technika aSci informatika Technical and IT ZemědělstvíSci Agriculture Vědy oScizemi Earth Matematika Mathematics Fyzika Physics Chemie Chemistry Biologie Biology Lékařství Medicine Points WoS SCOPUS Jimp Jimp Jimp Jimp Jimp Jimp Jimp Jimp Jimp Jimp Jsc Jsc Jsc Jsc Jsc Jsc (10 – 305) ERIH Domestic Books Jneimp Jneimp (10 – 30) Jrec 4 BC BC BC BC BC BC Proceedings D D D D D (4 – 120) (8 – 60) 82.5% of points is distributed according to this mainstay Mainstay II. • Each research unit submits one excellent result per CZK 10 mil. for evaluation • The submission is accompanied with reasoning why the result is excellent • The result is evaluated by the Expert panel of a given topical group and 20% of the best is chosen (category A) • 10% (60 000)points is allocated to this mainstay • These points will be distributed to the research entities in accord with the number of results in category A • Additional 2000 points will be allocated to the entity that received an ERC grant in course of past 5 years • Note that this mainstay will be applied from 2015 Estimated number of results submitted to Mainstay II celkem:999 999 výsledků Total: results os1 os2 51 110 os3 28 32 133 159 os4 os5 os6 68 149 151 37 81 os7 os8 os9 od10 os11 Mainstay III. • 105 000 points will be allocated according to this Mainstay • In the first step, a given number of points will be assigned to patents, novel races or cultivars • In the next step, the rest of points will be distributed in accord with the volume of money gained by a particular entity either as grants of applied research or by research contracts with private enterprises ∑Total = α∑Grants + β∑Contracted research (α = 0.9; β = 0.1) Present status of M 2013 • Data for Mainstays I and III are collected • The software tools for the data processing are almost ready • The members of Expert panels are nominated • Peer review of books and Czech journals will start in course of June • The excellent results will be collected in near future • Since the process according to M 2013 will be applied only on the data from the last year, it will influence only about 20% of the budget • Our modeling has shown that no dramatic changes in the money distribution can be expected