Estonian experience in evaluating
research proposals
Madis Saluveer
Head, Department of Research Funding
Estonian Research Council
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research",
Chisinau 6 November 2012
1
Outline
• Institutional restructuring in Estonia in 1990s
and its outcomes
• International evaluations of Estonian R&D
Changes in the institutional financing system
• Investing into the development of R&D system
• Evaluating research - issues for discussion
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
2
R&D and higher education reform in 1990s
• Legislative acts to support the functioning of the
new system were adopted
• Research financing system reformed
• Institutional reform launched
• International evaluations of research areas and
the R&D funding system carried out
• Higher and post-graduate education
brought into accordance with international
criteria
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
3
Institutional reform in 1990s
• 1989-1990 - the Estonian Academy of Sciences was re-organised
into a classical (personal) type of Academy -> former Academy
research institutes were merged with universities
Nature 1 Oct 2009: “With the exception of Estonia, the basic
research in the region (CEE) is still controlled by the national
academies of science...”
• 1990-1991 All Soviet Union ministerial branch institutes dissolved
• 1990 - Estonian Science Foundation and Estonian Innovation
Foundation founded
• 1997 – Research Competence Council (=>Estonian Research Council)
established
• 1997 - Archimedes Foundation founded (management of EU
framework programmes + single NCP network)
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
4
Establishing the R&D legal „backbone“
• 1994 – „Organization of Research and Development Act“ adopted by
the Estonian Parliament; latest amendments approved by the
Parliament in February 2011 (researcher career path, financing
instruments, re-organisation of funding agencies)
• 1995 – Universities´Act adopted
• 1997 – Estonian Academy of Sciences Act
• 2001 - Estonian R&D Strategy 2002-2006 “Knowledge-based Estonia”
approved by the Estonian Parliament
• 2006 - Higher Education Strategy for 2006-2015 approved
• 2007 - Estonian R&D and Innovation Strategy 2007-2013 “Knowledgebased Estonia II” adopted by the Estonian Parliament
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research",
Chisinau 6 November 2012
5
Rational of performance-based funding
The rationale of performance funding is that funds should flow
to institutions where performance is manifest:
‘performing’ institutions should receive more income than lesser
performing institutions, which would provide performers with a
competitive edge and would stimulate less performing
institutions to perform.
Output should be rewarded, not input.
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research",
Chisinau 6 November 2012
6
Internationational evaluations of Estonian R&D
system 1
Evaluation of all R&D areas by Royal Swedish Academy of Sciences and the Swedish
Research Councils 1991-1992
• Approach: broad and deep evaluation of the whole system; no broad legislative
background existed
• Goals: to receive an unbiased opinion on the level of research according to the
standards used in Scandinavia; to introduce the international norms; to get advice
what to do
• Outcome: More than 400 reports presented by all research groups in all fields of
research; 8 expert panels paid site visits; 19 reports produced
Observations and recommendations:
• Restructure the policies for decision-making
• Integrate research and education
• Unequal quality across the fields
• Stop publishing in local papers without peer review-> promote peer review
• Lack of young scientists -> normalise age structure, increase the number of research
students
• Establish/ promote international collaboration
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research",
Chisinau 6 November 2012
7
Internationational evaluations of Estonian R&D
system 2
2000-2003 – the first systematic evaluation based on established legal
basis (ORDA)
Approach: full and deep evaluation by the fields of science – 46 expert
groups (approx. 200 foreign experts from 18 countries)
Outcome: Very good job done, a great number of substantial
recommendations put forward; however, the results were heavily
underused. Process costly, slow, involved a lot of bureaucracy;
assessment methodology changed/improved over the evaluation
period; units of assessment varied from small research groups to whole
institutes; rapid change of R&D landscape over the evaluation period
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
8
Internationational evaluations of Estonian R&D
system 3
2009 – new regulation in ORDA: introduction of 2 types of evaluations
2010 - regular evaluation ( “wide and shallow”, every 7 years):
Approach: voluntary, covers the whole R&D system, fast, low cost,
low bureaucratic burden - based mostly on existing information, full
usage of Estonian Research Information System ETIS; peer-review
carried out by foreign experts (16 peers); strict rules; simple outcome
(yes/no)
Consequences:
• financial: access to state research funding instruments – base-line
funding, targeted funding
• for HEI-s – prerequisite for the PhD studies
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
9
Internationational evaluations of Estonian R&D
system 4
Targeted evaluation (“narrow and deep”; „tailor-made“ : topic,
subjects of evaluation, terms of references, evaluators and the
detailed mechanism of evaluation may vary)
Goals may vary:
• gaining a profound insight of a research field
• gaining information for outlining new research policy
measures, e.g. R&D strategies, research programmes, major
research policy decisions, funding mechanisms, etc, and/or
for elaborating the existing ones
• may be carried out in single or in multiple recearch areas
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
10
Internationational evaluations of Estonian R&D
system 5
2011 – first targeted evaluation carried out (plant and soil science)
Approach: based on „tunnetatud vajadus“; self-assessment reports; international peer
review (5 experts from 5 countries), site visits; no grades but policy recommendations
Outcome: evaluation committee report with observations and recommendations
http://www.etag.ee/wp-content/uploads/2012/05/SEV_PMT_report_2012.pdf
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
11
Estonian R&D expenditure 2001-2010
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
12
Number of publications of Estonian researchers
indexed in Thomson Reuters WoS
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
13
Estonian participation in EU framework
programmes
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
14
FP7 budget per researcher (thousand €)
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
15
Estonian Research Information System ETIS:
some facts
• ERIS (predecessor of ETIS) 1999-2006
• Need for large scale developments (real-time multi-partner
dynamic system)
• Spring 2006: launch of ETIS with pre-installed data (users,
publications, projects, etc)
Evolution of ETIS as data repository:
• 2008 - 78 000 publication records
• 2010 - 100 000 publication records
• 2012 - 134 000 publication records
• 2012 - 39 different electronic forms in use
• September 2012: more than 76 000 visitors (Estonia 71 393, Finland
1 212, Germany 860, USA 581, UK 569, Sweden 547, Latvia 312,
Italy 300, France 273, Spain 243)
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
16
ETIS functionalities
• an information channel and a tool for researchers and
R&D institutions
• a gateway where researchers can submit grant
applications and reports
• an instrument for funding bodies to process
applications, reports and reviews
• a state register of R&D institutions, researchers,
financiers and research activity
• a gateway to Estonian science landscape
• a means to get fast and adequate R&D statistics
• a trusted data repository
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
17
ETIS classification schemes
• Research Classification Scheme
• Classification of publications in ETIS
• List of internationally acknowledged
publishers
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
18
Classification of publications in ETIS
Articles in journals
1.1. Scholarly articles indexed by TR Web of Science or published in journals indexed by ERIH categories INT1 and INT2;
1.2. Peer-reviewed articles in other international research journals with an ISSN code and international editorial board,
which are circulated internationally and open to international contributions; articles of ERIH category NAT;
1.3. Scholarly articles in Estonian and other peer-reviewed research journals with a local editorial board; scientific
articles in journals important for Estonian culture Akadeemia, Looming, Vikerkaar.
1.
2.
Books/monographs
2.1. High-level peer-reviewed research monographs;
2.2. Other monographs;
2.3. Dissertations published in a series of dissertations (excluding manuscripts).
3.
Articles in proceedings/a chapter in a book or in a collection
3.1. Articles/chapters in books published by the publishers listed in Annex (incl collections indexed by the TR Conf Proc
Citation Index);
3.2. Articles/chapters in books published by the publishers not listed in Annex;
3.3. Specific research publications (dictionaries, lexicons, sets of maps, (field) guides, text-critical publications);
3.4. Articles/presentations published in conference proceedings not listed in Section 3.1;
3.5. Articles/presentations published in local conference proceedings
4.
Editing scientific publications
5.
Published meeting abstracts
6.
Other publications
6.1. Full articles in encyclopedias; 6.2. Text books and other study materials; 6.3. Popular science articles; 6.4. Popular
science books; 6.5. Essays accompanying books of fiction; 6.6. Articles in other journals and newspapers; 6.7. Other
creative activities
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
19
Chisinau 6 November 2012
Classification of research fields in ETIS
Biosciences and Environment (12 sub-fiels): Biochemistry, Microbiology, Genetics, Ecology,
Biosystematics and –physiology, Forestry, Agricultural Sciences, Food Sciences, State of the
Environment and to Environmental Protection, Substances Hazardous to the Environment, Geography
and Regional Studies, Environmental Policy, Environmental Economy and Environmental Law,
Biotechnology, Molecular Biology, Cell Biology, Biophysics and Economic and Technological Research
relating to the above fields
Culture and Society (14 sub-fields): Philosophy, Theology, History and Archaeology, Cultures
Research, Aesthetics and Arts Research, Philology and Linguistics, Law, Psychology, Special Education,
Educational research, Social Sciences, Economics, Political Science and Administration, Communication
and Information Sciences
Health (11 sub-fields): Biomedicine, Veterinary Medicine, Pharmacy, Dental Science, Nursing
Science, Public Health Science, Clinical Medicine, Sport Sciences, Nutrition, Occupational and
Environmental Medicine, Biochemistry, Genetics, Microbiology, Biotechnology, Molecular Biology, Cell
Biology, Biophysics and Bioinformatics relating to the above fields
Natural Sciences and Engineering (17 sub-fields): Architecture and Industrial Design,
Geosciences, Space Research and Astronomy, Mathematics, Statistics, Information Processing Sciences,
Telecommunications, Electrical Engineering and Electronics, Medical Engineering, Physics and Technical
Physics, Chemistry and Chemical Technology, Process Technology and Materials Science, . Mechanical
Engineering, Automation Technology and Manufacturing Technology, Industrial Engineering and
Management, Construction and Municipal Engineering, Energetics, Biotechnology relating to the above
fields
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
20
Evaluating research proposals - issues for
discussion
•
•
•
•
Types, scope and frequency of evaluations
Peer-review and/or „metrics“
Scope of use of bibliometrics
Impact of research evaluations on funding
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
21
Evaluation of research proposals 2
1994-2012: Estonian Science Foundation (personal) grants
• Assessment scheme: set requirements for PI (presence of mandatory publications),
international peer review (2 peers), expert panels => low competition, very small
grants (~10,000 €/year), high number of grants (total 500 – 600)
2012 replaced by Estonian Research Council’s personal grants
• Assessment scheme: international blind peer review (2-3 peers), expert panels, final
assessment by the Evaluation Committee => very high competition, average grants
(50 – 70 000 €/year), small number of grants (2012:~20, total ~100)
1997 – 2012: targeted financing of research topics (multi-annual team funding)
• Assessment scheme: set requirements for PI and team (presence of mandatory
publications), international (blind) peer review (2-3 peers), expert panels, final
approval by MER => medium competition, small grants (~100 000 €/year), average
number of financed topics (~200)
2012 replaced by Estonian Research Council’s institutional funding (multi-annual team
funding of institutions)
• Assessment scheme: international blind peer review, expert panels, final approval by
MER => high competition, average grants (~200 000 €/year), low number of grants
(2012: 31; total ~100)
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
22
Evaluation of research proposals 3
MER Centres of excellence programme 2001-2007 (2 calls)
• Assessment scheme: peer review by local experts ; final
approval by MER => 4 – 11 research groups per CoE; average
competition, total 10 CoE, total programme 6.4 M€
Centres of excellence in research programme 2008-2013 (2 calls)
• Assessment scheme: international peer review (30 peers);
final approval by Programme Steering Committee => 4-8
research groups; high competition, total 12 CoE, total
programme 55.3 M€
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
23
Evaluation of national programmes
MER national programmes (limited domains, e.g Estonian language &
literature, history, folklore; national collections; language technology)
• Assessment scheme: peer review by local experts; final approval by
the Steering Committee => small teams, low competition, small
grants (15 -50 000 €/year)
EU-financed national thematic R&D programmes 2007-2013 (6
programmes, total 51.8 M€)
• Assessment scheme: peer review by local and international
experts; panel meeting and final approval by the Steering
Committee => complementary funding, targeted R&D domains,
low competition, grant sizes vary by domain (e.g. Health
technologies 200 -500 000 €, environmental technologies 130 –
400 000 €, materials and nanotechnology 200 – 850 000 €, etc.)
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research",
Chisinau 6 November 2012
24
OP 2 – Operational Programme for the Development of Economic Environment
Measure 2.1: Thematic R&D
programmes aimed at long-term
economic development
SUPPORT 51,8 mil eur
Priority 2: Improving the
competitiveness of Estonian R&D
activities through the research
programmes and modernisation of
higher education and R&D institutions
SUPPORT 346,7 mil eur
Measure 2.4: Modernising research apparatus
and equipment
SUPPORT 110,7 mil eur
Measure 2.2: Centres of excellence in
research and participating in
research cooperation
programmes of the EU and the Baltic
Sea region
SUPPORT 55,3 mil eur
Measure 2.3: Modernising the general
infrastructure of R&D institutions
SUPPORT 128,7 mil eur
Modernising the general
infrastructure for higher
education
SUPPORT 42,1 mil eur
research
Modernising the
educational environment
of institutions of
professional higher
education
and universities
SUPPORT 14 mil eur
Modernising the general
infrastructure for
research
SUPPORT 86,5 mil eur
higher education
research and higher education
Research infrastructures of
national importance
(Roadmap, investment plan)
SUPPORT 29,4 mil eur
Research equipment for R&D
institutions
(R&D institutions, open call)
SUPPORT 46,3 mil eur
Small-scale research equipment
for research groups
(Research groups, open call)
SUPPORT 14,5 mil eur
E-resources programme
SUPPORT 6,4 milCopyright
eur
25
Estonian RI Roadmap
• RI roadmap – a long term (10-20 years)
planning instrument that lists research
infrastructures of national importance,
either new or in need of upgrading
• The roadmap will be updated regularly (at
an interval of 3 to 5 years) to take into
account the changing circumstances and
opportunities
• Being part of the roadmap does not mean
that a RI will be funded but the roadmap
will be used as an input for the investment
decisions
• Budget 29.4 M € for 2010-2015
• Financed from EU ERDF (85%), state
budget (10%) and own contribution (5%)
• https://www.etis.ee/Portaal/infrastruktuur.
aspx
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
26
Evaluation of research infrastructure proposals
Support for small-scale research equipment (total: 14.5 M€)
• Assessment scheme: annual open calls; research groups; 2step review: local peer review and approval by ETAg
Evaluation Committee;
Support for R&D institutions (total: 46.3 M€)
• Assessment scheme: open calls; R&D institutions; 2-step
review: local peer review and approval by programme
Steering Committee;
RIs of national importance (total: 29.4 M€)
• Assessment scheme: call for expressions of interest; call
for full proposals (investment plans); R&D institutions;
main and reserve list, final approval by Estonian
Government;
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
27
Thank you for your attention!
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„
Chisinau 6 November 2012
28