Peer review and knowledge dynamics
Terttu Luukkonen
Research Institute of the Finnish Economy
ERC Workshop
Monitoring the performance and
quality of peer review systems
28-29 November, 2013, Brussels
Focus
• Evaluation of research proposals
• Groundbreaking/pathbreaking/frontier/
transformative research
Content
• Defining groundbreaking research
– potential differences between fields in the
understanding
• Phases in the emergence and growth of new
research areas
• Comparison of peer review with other
methods in its ability to select groundbreaking
research
• Conclusions
Categories of groundbreaking
• Discovery of a novel phenomenon
(serendipitous discoveries and others)
• New method or technique or their
combination as an enabler
• Access to new data
• General explanations (paradigms and other)
– unsolved ‘big questions’ (Laudel and Gläser, 2012)
Impacts of groundbreaking research
on
•
•
•
•
Own discipline or research field
Several other research fields
Opening up new research fields
Merging fields/interdisciplinary areas of
research
Different perspectives
• Selection of proposals - Forward look
–
–
–
–
–
Promise of opening up new avenues of research
Enabling new research directions
New perspectives
Paradigm shifting, revolutionary
Great uncertainty
• Backward
– Pin down what gave rise to the observed development
– often a longer process and several contributors
Difference between excellence and
groundbreaking
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
LS
Terttu Luukkonen:
interviews with 24 ERC peer
reviewers, Luukkonen, 2012
PE
SH
To a great or some extent
Different definitions of
groundbreaking and excellence
ERC peer review panelists (Luukkonen, 2012)
• Groundbreaking
– Synonyms: pathbreaking, cutting-edge, frontier
– Originality, novelty , revolutionary, paradigm shifting
• Excellence
– Originality, novelty, going beyond current state of the
art, making a difference for the development of
science, but also
– Robustness of the research, methodological rigour,
use of up-to-date methodology, coherent discussion
of the research problem and purpose
Development of ideas and scholars over time
NR OF IDEAS &
SCHOLARS
SCHOLARS
IDEAS
Early
adaptors
Early
Majorities
Late majorities
TIME
Brown, 2012
Latecomers
Project selection methods and cycles of ideas
NR OF IDEAS &
SCHOLARS
National RCs
National RCs
Bibl.
methods
National RCs
ERC
HHMI
Early
adaptors
Early
Majorities
Late majorities
TIME
Latecomers
ERC peer review system
Luukkonen, 2012
• Evaluation criteria
• Quality of the peers
• Panels have to consider feasibility and risks
– Capabilities of the investigator
– Instruments and equipment
– Contingency plans
– Avoid speculation and dilettantism
– Put in context, tradition
– Reasonable risk
Laudel and Gläser on ERC, 2012
• “ERC grants have impact on research because, at least,
some of them fund scientific innovations, the
exploitation of recent discoveries, or answers to ‘big
questions’ across all discipline groups”
• The funded research has epistemic properties not
usually met by grants from national funding agencies
–
–
–
–
–
–
Contradicting the mainstream
Addressing the community’s blindspots
Linking otherwise separate communities
Strategic & technical uncertainties
Complexities in equipment, approaches
Length of time it takes to conduct the research
Project selection methods and cycles of ideas
NR OF IDEAS &
SCHOLARS
National RCs
National RCs
Bibl.
methods
National RCs
ERC
HHMI
Early
adaptors
Early
Majorities
Late majorities
TIME
Latecomers
Bibliometric measures suggested
Hörlesberger et al., 2013
• Novelty
– Timeliness: how recent are the publications listed on
the reference list of application
– Similarity of proposal with emerging topics
• Risk
– Similarity of the proposed research to the
investigator’s previous research
– Interdisciplinarity
• There has to be some body of publications in the
field for the measures to be counted
Project selection methods and cycles of ideas
NR OF IDEAS &
SCHOLARS
National RCs
National RCs
Bibl.
methods
National RCs
ERC
HHMI
Early
adaptors
Early
Majorities
Late majorities
TIME
Latecomers
Varieties of peer review
• Robustness
– Use of international vs national experts
– Independence of the panels
– Their evaluation instructions and criteria
• Further features:
–
–
–
–
–
Fine-grained vs. rough marking (Langfeldt, 2001)
Remote reviews vs. or, in addition, panelists’ reviews
Degree of interdisciplinarity of panels
Panels rank or rate
Evaluation criteria: ground-breaking vs. excellence
Conclusions
• Peer review conservative?
• Not just one peer review, but many varieties in quality,
criteria, organisation
• The way peer review is organised and applied makes a
difference
• Peer review in combination with the conditions of the
scheme can make a difference for progress of science
• Quality of peer review provides legitimacy to the
scheme and affects quality of the applicants
• A risk that a thorough monitoring of peer review shifts
the system towards more conventional proposals –
short term indicators
Thank you for your
attention!
more information:
terttu.luukkonen@etla.fi
Literature
–
–
–
–
–
–
–
–
Braun, Dietmar. 2012. Why do scientists migrate? A diffusion model, Minerva, 50: 471-491.
Grant, Jonathan and Allen, Liz. 1999. Evaluating high risk research: an assessment of the Wellcome
Trust’s Sir Henry Wellcome Commemorative Awards for Innovative Research, Research Evaluation, 8:
201-204.
Hörlesberger, Marianne, Roche, Ivana, Basagni, Dominique, Scherngell, Thomas, Francois, Claire,
Cusax, Pascal, Schiebel, Edgar, Zitt, Michel, and Holste, Dirk. 2013. S concept for inferring ‘frontier
research’ in grant proposals, Scientometrics, 97: 129-148.
Langfeldt, Liv. 2001. The decision-making constraints and processes of grant peer review, and their
effects on the review outcomes, Social Studies of Science, 31/6: 820-841.
Laudel, Grit and Gläser, Jochen. 2012. The ERC’s impact on the grantees’ research and their careers
(EURECIA Work package 4 summary report). January 2012.
Luukkonen, Terttu. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices,
Research Evaluation, 21 (2012), No. 1, pp. 48–60.
Nedeva, Maria. 2012. Peer review and path-breaking research: selection practices of research
funding organisations. Unpublished.
Wagner, Caroline S. and Alexander, Jeffrey. 2013. Evaluating transformative research programmes: A
case study of NSF Grants for Exploratory Research Programme, Research Evaluation, 22: 187-197.
UNI project: Universities, funding systems, and the renewal of the industrial knowledge base – a
project funded by Tekes, 2012-2014; coordinated by Terttu Luukkonen; empirical data gathering
with research group leaders in universities in Finland and the UK.