Traditional Peer Review
ULS Scholarly Communications
Lunch and Learn #14
Office of Scholarly Communication and Publishing
University Library System
University of Pittsburgh
August 21, 2014
Traditional Peer Review
 What is peer review?
 Why should librarians care about peer review?
 How does it work? - the process
 Research ethics and peer review
 Problems and limitations of traditional peer review
NEXT MONTH: Innovations in peer review
What is peer review?
Peer review is expert, independent, unbiased
scrutiny of research to:
– self-regulate quality standards within an academic
discipline
– reinforce the scientific method
– provide credibility for research results
– determine suitability for scholarly publication
Scholarly peer review is different from:
– professional peer review
– clinical peer review
Why should librarians care about
peer review?
 essential to critical evaluation of information
sources
 adds value and credibility to resources we provide
 critical for the library as publisher
 librarians may serve as reviewers ourselves
 important to the clients we serve
 Major changes are underway - we need to
understand them!
History of peer review
1752: Royal Society of
London establishes
‘Committee on Papers’
 reviews papers for the
Philosophical Transactions
of the Royal Society
 journal already in existence
for 86 years
Some characteristics of peer review
 Conducted by independent experts
 Free from personal and professional bias
 Anonymous
– single blind: identity of reviewer is not known to author
– double blind: identity of author and reviewer unknown to
each other
 Norms and processes vary by discipline
What is being evaluated?
Research methodology
 Scientific soundness
 Statistical design and analysis
 General presentation of results
 Degree to which conclusions are supported by
evidence
 Can the research be replicated by another
researcher?
What is being evaluated?
 Originality of research
 Novelty and overlap with similar research
 Relevance: importance to the discipline
 Anticipated level of interest
 Appropriateness for scope of journal (or journal
section)
Traditional peer review:
THE PROCESS
Responsibility of the Author
 Removing identifying information from the submission
– Do not include authorship in the submission document
– Remove identifying information from the document
properties
– Do not attempt to discover the identity of the reviewer
and, if an author accidentally discovers the reviewer’s
identity, do not contact them regarding the submission
Responsibility of the Reviewer
 Disclose to the editor any conflicts of interest
 Only review submissions for which you have expertise
 Agree to review only if you can do so in a timely manner
 Decline reviews similar to your own current work in preparation or
submission
 Don’t share the submission or its details with others
 Support criticisms with evidence
 Do not attempt to discover the author’s identity, and if you
accidentally do so, inform the editor immediately
Reviewing: A Labor of Love?
 Reviewers typically do not get paid
 Reviewers are not publicly acknowledged for
individual reviews
 Some journals give an annual award for the most
active reviewer (Oh yeah! A plaque!)
 A single review can take several hours to complete
RESEARCH ETHICS and
PEER REVIEW
Test Your Review Ethics Knowledge
 You are an editor of a journal that practices doubleblind peer review. An author of a recently-reviewed
(but not yet published) article claims that the
rejection received from Reviewer #2 is due to
personal bias – the author alleges that Reviewer #2
discerned their identity from the very unique data
set that they were using and, due to a feud between
Reviewer #2 and the author’s dissertation advisor,
rejected the article for non-scholarly reasons. The
identity of Reviewer #2 is, in fact, the person alleged
to be in a feud with the author’s advisor. What do
you do?
Options:
 A. Nothing. Reviewers are allowed to reject the article for
any reason.
 B. Investigate the connection between Reviewer #2 and
the dissertation advisor in question, but leave the review
as it stands.
 C. Thank the author for their concern and recruit another
peer reviewer to take the place of Reviewer #2, throwing
out Reviewer #2’s comments. Choose this if the review is
pure vitriol
 D. Encourage the author to take any methodological or
theoretical concerns of Reviewer #2 into consideration,
but favor the responses of Reviewers 1 and 3 more
Choose this one if the review has substance
heavily.
Test Your Review Ethics Knowledge
As a reviewer for a double-blind peer reviewed
medical journal, you notice that the author of a paper
on different types of treatment for a disease
repeatedly cites works from a particular
pharmaceuticals group. You know that there are many
competing pharmaceuticals companies with different
treatments available and suspect that the author may
be affiliated with this group in some way. However,
due to the double-blind process, you do not know the
name or affiliation of the author, and no information is
given in the article. Is this a problem, and if so, what
do you do?
Options:
A. Report the issue to the editor with relevant evidence.
B. Conduct your own investigation, e-mailing the members of
the cited pharmaceutical group to inquire if any of them have
submitted a paper to the journal recently, then report your
findings to the editor.
C. Write your own paper responding to this article with
criticisms about Conflict of Interest, then submit it to the same
journal before you have completed your review.
D. Nothing. People from pharmaceuticals groups are allowed to
write papers too.
Part 2:
 As the editor in this instance, with the reviewer
presenting all relevant evidence, what do you do?
Options:
 A. Nothing. Conflicts of interest are not the domain
of your journal.
 B. Contact the author of the paper and request a
statement of any conflicts of interest.
 C. Immediately reject the paper for ethical reasons.
 D. Contact the author’s organization informing them
of ethical misconduct.
Test Your Review Ethics Knowledge
You are an author of a journal article investigating the
dialect of an unstudied Inuit fishing community. While your
article is being formatted for publication, you attend a
presentation at a conference about dialects of Inuit by a
Professor Richard Smith and, to your shock, one of his
slides contains the exact same data that you reported in
your unpublished article!
The community is small and you are pretty sure you would
know all of the other researchers who have done work in
the area, and Richard Smith is not one of them. You notice
that Professor Smith doesn’t cite the source of his data and
you suspect that he was a reviewer of your article. What
should you do?
Options:
 A. This is one of the hazards of publishing in academic
journals. There is nothing you can do.
 B. During the question and answer session, publicly
accuse Professor Smith of ‘scooping’ your data and
demand answers.
 C. Contact the editor of the journal you are publishing in,
presenting all of the evidence and your concerns.
 D. Call the U.S. copyright office and file a complaint
against Professor Smith.
Part 2:
 You are the editor of the journal who receives the
author’s complaint. You know that Professor
Richard Smith was one of the reviewers for this
article. What do you do?
Options:
Answer: all of the above, in this order, if
there is a breach of reviewer conduct!
 A. Say thank you to the author and promise to
investigate.
 B. Review the files and the timeline to see if
Professor Smith could have scooped the data.
 C. Contact Professor Smith directly to request an
explanation.
 D. Contact the reviewer’s institution requesting an
investigation.
 E. Remove the reviewer from your review database
and consider reporting the case in your journal.
Traditional peer review:
limitations and challenges
Traditional peer review:
limitations and challenges
 Labor intensive
 Reviewers are overtaxed and under-rewarded
 Time-consuming; slows the pace of research
 Process is open to bias, self-interest, and cronyism
 Valuable research can be lost through subjective
filtering for relevance and importance
 Limits transparency and accountability
NEXT MONTH
Innovations in Peer Review
 Open peer review
 The preprint publishing model
 Post-publication peer review (PubPeer)
 Filtering for methodology only (PLoS One and PeerJ)
 Rewarding and rating peer reviewers
 Independent peer review, pre-submission (Rubriq)