Peer Review Survey 2009:
Preliminary Findings
8th September 2009
1
Reviewing Peer Review
Peer Review Survey 2009: Preliminary Findings
8th September 2009
Should peer review detect fraud and misconduct? What does it do for science and what does the scientific
community want it to do? Will it illuminate good ideas or shut them down? Should reviewers remain
anonymous? These questions are part of one of the largest ever international surveys of authors and reviewers
(over 4,000), the Peer Review Survey 20091, whose preliminary findings are released today.
Peer review now results in 1.3 million2 learned articles published every year. It is fundamental to integration of
new research findings in hundreds of fields of inquiry. It is the front line in critical review of research, enabling
other researchers to analyse or use findings and, in turn, society at large to sift research claims. It is growing year
on year with the expansion of the global research community, and with it has come a corresponding expansion of
concerns about getting the next generation of researchers to review in sufficient numbers: Can the peer reviewing
effort be sustained? Can the system be truly globalised and its integrity maintained? Some observers say that
peer review will be able to keep pace, following uptake of electronic technologies – from online processes to
programmes that help identify plagiarism; others have suggested that alternative metrics will play a greater role.
As a science education charity, Sense About Science2 sees peer review as vital to the transparency of scientific
reasoning, and the pressure of these questions led Sense About Science to find out more about what researchers
actually think about peer review and its future. The Peer Review Survey 2009 was developed by Sense About
Science in consultation with editors and publishers and administered with a grant from Elsevier. It repeated some
questions from the Peer Review Survey 20073 for comparison, and developed emerging questions about future
improvements, public awareness and new pressures on the system. Preliminary findings are presented in the
following pages.
2
1. Summary
1.1 Playing an active role in the community came top of reasons to review:
It is seemingly an unrewarding job with a few fringe benefits, so why do it? Reviewers indicate it is mainly
because they believe they are playing an active role in the community (90%), and quite simply, many just enjoy
just being able to improve papers (85%) Fig 2.1 . Only 16% of respondents said they agree to review because
they feel it will increase their chances of having future papers accepted.
1.2 Researchers want to improve, not replace peer review:
•
Most (69%) researchers are satisfied with the current system of peer review (64% in 2007) but only a third
(32% ) think that the current system is the best that can be achieved Fig 2.4, Fig 2.5.
•
Most (84%) believe that without peer review there would be no control in scientific communication (the
same as 2007) and only one in five researchers (19%) believe that peer review is unsustainable because
there are too few willing reviewers Fig 2.5.
•
Almost all researchers (91%) believe that their last paper was improved as a result of peer review Fig 2.6
& Fig 2.18; and the biggest area of improvement was in the discussion.
•
73% of reviewers (a sub-group in the study) believe that technological advances have made it easier to do
a more thorough job reviewing now than 5 years ago Fig 2.7. Whilst the majority of respondents enjoy
reviewing and will continue to review (86%) Fig 2.7, many think that more could be done to support
reviewers; 56% believe there is a lack of guidance on how to review while 68% agree that formal training
would improve the quality of reviews. Over the course of a year, on average, a reviewer turns down two
papers Fig 2.15.
•
Just 15% of respondents felt that ‘formal’ peer review could be replaced by usage statistics Fig 2.9.
3
1. Summary
1.3 High expectations:
•
79% or more of researchers think that peer review should identify the best papers, determine their
originality and importance, improve those papers, and though lower scoring, also determine
whether research is plagiarised or fraudulent Fig 2.10.
•
While 43% of respondents thought peer review was too slow Fig 2.13, 65% of authors reported that
they had received a decision on their most recent paper within 3 months Fig 2.12.
•
61% of reviewers have rejected an invitation to review an article in the last year, citing lack of
expertise as the main reason – this suggests that journals could better identify suitable reviewers
Fig 2.8.
1.4 Reviewers want anonymity:
More than half (58%) of the researchers say they would be less likely to review if their signed report was
published alongside the paper reviewed. Similarly, 51% would be discouraged from reviewing if their
name was disclosed just to the author and 46% would be discouraged if their name was published
alongside the paper as a reviewer Fig 2.2. Over three quarters (76%) favour the double blind system
where just the editor knows who the reviewers are, but some researchers questioned whether an
author’s identity can be truly anonymised Fig 2.9.
1.5 Understanding of peer review:
As might be expected researchers agree that peer review is well understood by the scientific
community,. However, this level of understanding is in sharp contrast to the research community’s
perception of the public’s awareness of peer review: just 30% believe the public understands the term
Fig 2.5.
4
1. Summary
1.6 Papers aren’t recognising previous work:
Most researchers (81%) think peer review should ensure previous research is acknowledged. However,
just over half think it currently does (54%) and or is capable of doing this (57%) Fig 2.10. This reflects
current discussions in the research community that there is a need for new studies to be set in the
context of existing evidence.4
1.7 Detecting plagiarism and fraud might be a noble aim but is not practical:
The vast majority of authors and reviewers think peer review should detect plagiarism(81%) or fraud
(79%) but only a small amount (38% and 33%) think it is capable of this Fig 2.10. It is the practicalities
involved that make it difficult; researchers point out that examining all raw data would mean peer
review grinds to a halt. When asked how peer review can be improved, very few mention fraud, clearly
indicating that is neither widespread nor a pressing issue in the minds of researchers.
1.8 Reviewers divided over incentives:
Just over half (52%) of reviewers thought receiving a payment in kind (e.g. subscription, waiver of their
own publishing costs etc.) would make them more likely to review. A large minority (41%) wanted
payment for reviewing, but this drops to just 2.5% if the author had to cover the cost. Acknowledgement
in the journal was the most popular of all the options, with 39% stating this would make them much
more or more likely to review Fig 2.2.
5
Notes
1.
2.
3.
2.
3.
4.
5.
The Peer Review Survey was an electronic survey conducted between 28 th July 2009 and 11th August 2009; 40,000
researchers were randomly selected from the ISI author database, which contains published researchers from over 10,000
journals. Altogether 4,037 researchers completed our survey. This represents a margin of error ± 1.5% at 95% confidence
levels , reviewers answered a subset of questions aimed specifically at reviewers (3,597 - a subset of the base) the error
margin for this group was ± 1.6% at 95% confidence levels.
The full findings and report are due to be published in November 2009 and will be available at
http://www.senseaboutscience.org.uk/index.php/site/project/29/.
Björk et al (2008) ‘Global annual volume of peer reviewed scholarly articles and the share available via different Open
Access options’ Proceedings ELPUB2008 Conference on Electronic Publishing – Toronto, Canada – June 2008
Sense About Science is a UK registered charity (No. 1101114) to equip people to make sense of science and evidence. It has
previously published ‘Peer Review and the Acceptance of New Scientific Ideas: a working party report’ 2004; and it is the
publisher of the public guide to peer review, ‘I Don’t Know What to Believe’. In 2008, Sense About Science established the
online education resource about scientific publishing and peer review, for schools www. senseaboutscience.net.
Publishing Research Consortium (2007) ‘Peer Review in Scholarly Journals: perspective of the scholarly community. An
international study’ www.publishingresearch.net/documents/PeerReviewFullPRCReport-final.pdf
Chalmers and Glasziou (2009) ’Avoidable waste in the production and reporting of research evidence’ The Lancet; 374: 86–
89.
The effects of rounding means that some of the % figures within the charts may be slightly different to the overall % total
found at the sides of the charts.
Further information and comment: contact Alice Tuff, Sense About Science
+44 (0)20 7478 4380 Atuff@senseaboutscience.org
6
2.1 Reasons for Reviewing
Question: Please indicate the extent to which you agree that the following reasons describe why you review
% agree
16%
33%
34%
30%
46%
85%
69%
72%
90%
n=3597
2.2 Incentives to review
Question: Please say whether the following would make you more or less likely to review for a journal
Payment by the journal
9% 8%
Payment in kind by the journal
5% 6%
43%
37%
37%
Accreditation (CME/CPD points) 5% 6%
55%
Acknowledgement in the journal
44%
8% 9%
Your signed report being published with the
paper
28%
27%
26%
32%
30%
31%
% less
likely
% more
likely
14%
15%
41%
14%
11%
51%
7%
11%
33%
8%
17%
40%
2%
9%
58%
11%
45%
18%
51%
8%
3%
Your name being published alongside the
paper as one of the reviewers
21%
Your name as the reviewer disclosed to the
author only
23%
0%
24%
36%
28%
42%
50%
15%
2%
6%
100%
n=3597
Much less likely
Less likely
No difference
More likely
Much more likely
8
2.3 Payment for Peer Review
Question: [Respondents who answered “much more likely” or “more likely” when asked whether payment by the
journal would make them more or less likely to review were asked the questions below:]
Would you still want payment if the following was true?
Author pays the fee
Funding body pays the fee
Publisher/Society covers the cost
n=1481
2.4 Overall satisfaction with Peer Review
Question: Overall, how satisfied are you with the peer review system used by scholarly journals?
(n=4037)
10
2.5 Principles of peer review
% Agree
Question: Please indicate the extent to which you agree with the following statements
*
* n=3964 (added after pilot stage so number is slightly lower)
2009
2007
68%
n/a
84%
83%
19%
n/a
82%
85%
32%
n/a
21%
19%
30%
32%
88%
n/a
32%
n/a
32%
32%
n=4037
11
2.6 Experience of peer review
Respondents were asked to consider the last peer-reviewed paper they had accepted for publication
Did you have to revise the paper?
If yes, do you think the review process
improved the quality of the paper?
n=3657
n=4037
How many journals (if any) rejected your last
paper before it was accepted?
n=4037
2.7 Reviewing Generally
Question: Please indicate the extent to which you agree with the following:
% agree
68%
73%
86%
56%
n=3597
2.8 Reasons for declining to review
Question: Thinking of the last time you declined an invitation to review, what were the main reasons for
declining: (Select up to a maximum of three reasons)
Paper was outside my area of expertise
58
Too busy doing my own research, lecturing etc
49
Too many prior reviewing commitments
30
Personal reasons (e.g. holiday, sickness)
20
Proposed deadline was too short to conduct a
thorough review
16
Poor quality English of the paper
10
Poor scientific quality of the paper
10
Conflict of interest
8
Journal was not on my preferred list of journals
6
Other
3
I have not declined a reviewing invitation
recently enough to recall
2
0
20
40
% agree
60
80
n=3597
14
2.9 Types of Peer Review thought effective
Question: For research papers published in your field, to what extent do you agree that the following
types of peer review are/would be effective?
*
* This is where the authors and reviewers are known to each other and additionally the reviewers’
signed reports are openly published alongside the paper
% agree
2009
2007
15%
5%
47%
n/a
25%
n/a
20%
27%
76%
71%
45%
52%
n=4037
15
2.10 Purpose of Peer Review
Questions: To what extent do you agree or disagree that the following objectives should be the purpose of peer review
To what extent do you agree or disagree that peer review is currently able to do the following?
And, to what extent do you agree that peer review currently fulfils the objectives below?
n = 4037
% agree
16
2.11 Number of articles respondents have published and reviewed
Questions:
Please indicate the number of academic peer-reviewed papers (articles):
a) You have published in your career to date
b) You have reviewed in the last year
No. of articles published in career to date
No. of articles reviewed in the last year
n=4037
2.12 Length of Peer Review
Question: Approximately, how long did the peer review process take:
from submission to first decision?
for any revision stages you undertook?
from submission to final
acceptance by the journal?
n=4037
2.13 Perception of length of peer review
Question: How did you feel about the reviewing time:
From submission to final acceptance by the journal?
From submission to first decision?
n=4037
2.14 Time taken to review
Question: Now thinking specifically about the last
paper you agreed to review. How much actual
time did you devote to the review?
No. of hours spent on last review
Modal time spent = 4 hours
Median time spent = 6 hours
Question: Approximately, what was the time
between your acceptance of the invitation to review
and the delivery of your report?
Time between acceptance of invitation to
review and delivery of report
86% returned their last review within
one month
2.15 Declined to review
Question: During the last 12 months, how many times have you declined an invitation to review?
Mean number of rejected papers is 2 per annum
n = 3597
21
2.16 Collaboration during review
Question: The last paper you agreed to review was:
2.17 Reviewer Frequency and Timeliness
What is the maximum number of papers that you
would be prepared to review in a year?
5% 1%
Do you submit your reviews on time?
7%
1% 0%
6%
7%
13%
36%
35%
57%
32%
Alw ays
1-2
3-5
6-10
11-15
16-20
21-50
Frequently
Sometimes
Rarely
Never
more than 50
57% of respondents always return their reviews on time with a further 35% doing so frequently.
Just 11 respondents admit to never returning their reviews on time.
36% feel that they can review 3-5 articles annually. A further 32% can manage 6-10 each year.
2.18 Aspects of improvement
Question: Respondents who agreed that peer review had improved their most recent paper were asked which aspects were
improved and to what extent?
The biggest area of improvement was in
discussion 91% felt that it had been improved to
some extent, with 11% specifying a substantial
improvement.
Only 50% saw an improvement to their paper’s
statistics (although the 50% who saw no
improvement is likely to include those whose
papers did not contain any statistical analysis).
n=3331
2.19 Demography
Organisation
Gender
Age
Region
2.20 Demography
Question: Which of the following best describes your field of research?
Agriculture
Position
3
What is your position?
Arts & Humanities
1
Astronomy
1
Biochemistry, Genetics and Molecular Biology
5
Biological Sciences
10
Chemical Engineering
1
Chemistry
6
Computer Sciences / IT
2
Earth & Planetary Sciences
3
Economics
2
Electrical/Electronic Engineering
4
Engineering & Technology
7
Environmental Sciences
4
Immunology
1
Microbiology
3
Materials Science
5
Mathematics
4
Medicine & Allied Health
15
Nursing
1
Neuroscience
4
Pharmacology, Toxicology and Pharmaceutics
2
Physics
7
Social Science
7
0
2
4
6
Percentage
8
10 12
14 16