Evaluation in R&D sphere in Ukraine: Real practice and
problems of transition to new standards
Igor Yegorov
Centre for S&T Potential and Science History Studies, National Academy of
Sciences of Ukraine and
the Institute of Statistics, State Service of Statistics of
Ukraine
e-mail: igor_yegorov1@ukr.net
Paper presented at the
International workshop
Science evaluation as a prerequisite for promoting excellence in
research
6 November 2012
Chisinau, Moldova
Key objects of evaluation in R&D
and innovation sphere in Ukraine
a)
b)
c)
d)
Individual researchers
Research organizations (institutes,
design bureaus and so on)
Research and innovation projects
R&D programs and R&D sections of
development programs
Main sources of data







Expert opinion (individual experts or expert panels)
State statistics (State Service of Statistics of Ukraine)
Department of Intellectual Property
Institute of S&T and Economic Information
Internal statistics of different ministries and agencies (for
instance on functioning of technoparks)
Information from special surveys and databases
Foreign sources (internationally recognized publications,
awards and so on)
Problems with utilization of official statistics



Rigidity in approaches and difficulties in changes of
indicators and questionnaires
Deviation from international standards (examples:
system of scientific degrees, calculation of number of
publications and so on)
The willingness to serve ‘political’ interests (example:
number of research projects and other indicators, which
are working ‘in favour’ of old system)
Key regulatory acts






Law on Scientific and S&T Activities (1992)
Law on Innovation Activities (2002)
Law on S&T Expertise (2006)
Law on Higher Education (1993)
Law on State S&T Programs (2004)
Total number of laws and other regulatory acts on R&D
and innovation exceeds 80. Almost all of them contain
clauses on evaluation at different levels.
Important features of the Ukrainian
research system



High level of centralization in Ukrainian R&D system:
administrative power of rectors and directors of the
research institutes is ‘excessive’ (including full control
over all financial transactions)
System of the evaluation of research results with clear
domination of internal publications, not international
publications as indicators of success
The procedures of evaluation and selection of R&D
projects are often not transparent and fair for potential
participants.
Recent changes in R&D and Innovation
statistics




Introduction of the CIS-type survey in Ukraine (20092010, data for 2008-2009). New survey is expected in
2013
Switching (partial) to international classification of
sectors - since 2009
Implementation of new NACE-type classification (new
version of KVED) – since 2012
Implementation of indicators of international publications
in some universities and research institutes
Publications of Ukrainian authors in scientific
journals in 1996-2005, tous. (Thomson
Reuters)
4700
4600
4500
4400
4300
4200
4100
4000
3900
3800
3700
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Number of scientific publications according
to the State Statistical Service of Ukraine,
1991-2011, thous.
400000
350000
300000
250000
200000
150000
100000
50000
0
1991
1995
1999
2003
2007
2011
Evaluation of individual researchers



Method: Panels of experts from the same institute (as a
rule: at the level of department and at the level of the
institute) discuss the work of researcher and vote openly
for the decision. This decision has to be confirmed by the
scientific council of the institute.
Periodicity: Every five years
Indicators are used indirectly (number of publications,
participation in conferences and seminars, awards,
preparation of the thesis (if any), participation in a work
of different scientific councils and commissions and so
on)
Evaluation of research organizations



A) Panel of external experts from the same research
area prepares a report on activities (every four-six
years). Indicators include number of publications,
number of patents and so on.
B) Evaluation with implementation of the special metrics
(2012)
Evaluation of potential (within the projects or programs)
varies according to the goals of evaluation (academic
programs, STCU and so on)
Evaluation with implementation of the
special metrics (2012)
‘Objective’ indicators (60% of final assessment)
 Expert evaluation (40% of final assessment)
 ‘Objective’ indicators:
a) personnel (5 indicators)
b) financial situation and research equipment (8 indicators)
c) research in priority areas (3 indicators)
d) publication activities (2 indicators)
e) innovation activities (4 indicators)
f) international activities (3 indicators)
g)’visibility’ at the international level (2 indicators)

Example1: ’Visibility’ at the international
level


Number of references on the research organization in
the Google searching system
Number of publications in domestic and international
scholar journals with high impact-factor per one
employee of the research organization
Example 2: Publication activities


Number of publications in domestic scholar journals per
employee of research organization
Number of publications in foreign scholar journals from
international databases of scientific publications per
employee of research organization (data bases include
information from Scopus and some other databases on
medical publications)
Expert evaluation of research organization





Level of research activities
Relevance to the national priorities
Potential for further development
Indicators, which reflect specific features of research
area
Note: all indicators have their own weights, depending
on the type of organization
Evaluation of research projects






Ex- ante evaluation
Evaluation of the process (if the project has long duration
or it is divided into several stages)
Ex-post evaluation
Different types of indicators are used, depending on the
type of the project
Special forms are used by different agencies and
ministries, not less than two experts take part in the
evaluation procedure
Key idea is to reach the goal of the project (must include,
if possible, description of quantitative indicators, which
have to be achieved) within certain time and financial
constraints
Example of indicators for selection of the
project for the Ministry of Economic
Development and Trade (October 2012)
Proposed price (60% of the final assessment)
 Other indicators (40%):
a) Number of Doctors and Candidates of Sciences in the
research team (30 points, if more than 7: 3D+4C)
b)Number of articles in scholar journals (30 points, if more
than 10) on the topic of the project
c) Monographs on the subject (15points per monograph)
d) Participation in similar projects with the Ministry during
the last 3 years (10 points per project)

Evaluation of the programs


1.
2.
3.
In accordance with to the normative acts and existing
laws at the state and ministry level
Problems:
Lack of quantitative criteria and utilization of ‘artificially
constructed criteria’
Non-transparent expert procedures (as a rule, no
foreigners take part in the evaluation, excluding Key
Laboratory initiative and some joint programs with
foreign partners)
Lack of money for financing of R&D programs, which is
used as an argument for justification of the level of
results achieved within the program
Conclusion


Ukraine has not yet created relevant institutions and
proper instruments necessary for effective
transformation of R&D sector, including effective system
of research assessment. Internal practice of evaluation
differs substantially from the international one.
Unfortunately, every year the possibilities for the
implementation of effective transformation policy are
shrinking, as the number of researchers is declining and
the research centres are losing their ability to conduct
research.
Conclusion



With time running out, remaining research institutions will
face more problems with the creative capabilities needed
to compete internationally
Reforms of evaluation of R&D sector have to be an
integral part of general transformation of R&D system
EU could contribute to the reforms by stimulating
introduction of new standards of R&D project
assessment and implementation
Thank you for your attention !