Alternative Peer Review:
Quality Management
for
21st Century Scholarship
http://www.public.iastate.edu/~gerrymck/APR.ppt
Gerry McKiernan
Science and Technology Librarian
and Bibliographer
Iowa State University Library
Ames IA
USA
gerrymck@iastate.edu
Workshop on Peer Review
in the
Age of Open Archives
International School for Advanced Studies
Interdisciplinary Laboratory
Trieste, Italy
May 23-24, 2003
THANK YOU!
Workshop Advisory Board [(Marco Fabbrichesi
(INFN/SISSA Italy), Stevan Harnad (University of
Southampton, UK), Stefano Mizzaro (University
of Udine, Italy) and Corrado Pettenati (CERN
Library, Geneva, Switzerland)]
Iowa State University, Faculty Senate, Committee
on Recognition and Development
European Commission
Iowa State University Library
Heike Kross, Ph.D.
DISCLAIMER (1)
The screen prints selected for this
presentation are for educational
purposes and their inclusion does
not constitute an endorsement of
an associated product, service,
place, or institution.
DISCLAIMER (2)
The views and opinions expressed
in this presentation are those of
the presenter and do not constitute
an endorsement by Iowa State
University or its Library.
NOTICE
No editors, authors, or referees
were harmed in the preparation of
this presentation.
http://www.sciencecartoonsplus.com/
Giuseppe De Nittis (1846-1884)
The Macchiaioli / Italian Impressionists
Campo di Biche (1875)
PEER REVIEW: DEFINITION
“Peer review is the assessment by an
expert of material submitted for
publication.”
Carin M. Olson, “Peer Review of the Biomedical Literature,” American
Journal of Emergency Medicine 8 no.4 (July 1990): 356-358.
PEER REVIEW: PURPOSES
Peer review helps to ensure that published research is:
Important
Original
Timely
Technically-reliable
Internally-consistent Well-presented
Benefited from guidance by experts
Carin M. Olson, “Peer Review of the Biomedical Literature,” American
Journal of Emergency Medicine 8 no.4 (July 1990): 356-358.
PEER REVIEW: STRENGTHS
The underlying strength of peer review
is“…the concerted effort by large numbers of
researchers and scholars who work to assure
that valid and valuable works are published
and conversely to assure that invalid or nonvaluable works are not published … .”
Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses.
(Medford, NJ: Information Today, 2001).
“Houston, We Have a Problem!”
PEER REVIEW: PROBLEMS
Subjectivity
Bias
Abuse
Detecting defects
Fraud and Misconduct
Delay
Fytton Rowland, “The Peer-Review Process,”
Learned Publishing 15 no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf
SUBJECTIVITY
Summary rejections by editor without
sending the paper to referees
Choice of referees by the editor (choosing
for example, a known harsh referee for a
paper the editor wishes to see rejected)
BIAS
Discrimination against authors because of
their nationality, native language, gender or
host institution
Situations where author and referee are
competitors in some sense, or belong to
warring schools of thought
ABUSE
Too many articles out of one piece of
research, or duplicate publication
Intellectual theft: omission or downgrading
of junior staff by senior authors
Plagiarism (stealing others yet unpublished
work that has been sent for review)
Delaying publication of potentially
competing research
DETECTING DEFECTS
Identification of factual errors within
submission
FRAUD and MISCONDUCT
Fabrication of results
Falsification of data
False claim of authorship for results
DELAY
“There is much muttering about publication delay, a real
enough problem, especially in paper publication, but peer
review itself is often responsible for as much of the delay as
the paper publication and distribution process itself.”
Stevan Harnad
Stevan Harnad, “Implementing Peer Review on the Net: Scientific
Quality Control in Scholarly Electronic Journals, in Scholarly
Publication: The Electronic Frontier, edited by Robin P. Peek and
Gregory B. Newby (Cambridge MA: MIT Press, 1996).
http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
http://www-marketing.wharton.upenn.edu/ideas/pdf/peerpaper.pdf
“Peer review is slow, expensive, profligate of
academic time, highly subjective, prone to bias,
easily abused, poor at detecting gross defects, and
almost useless in detecting fraud.”
Richard Smith
Editor, BMJ
Richard Smith, “Opening Up BMJ Peer Review,”
BMJ 318 (7175) (January 2 1999): 4-5
Stephen Lock, A Difficult Balance: Editorial Peer Review in Medicine
(Philadelphia, PA: ISI Press, 1986).
EXAMPLE
Jan Hendrik Schön
-
http://archive.salon.com/tech/feature/2002/09/16/physics/
Science
Nature
http://physicsweb.org/article/news/6/9/15/
http://agenda.cern.ch/fullAgenda.php?ida=a01193
RECOMMENDATIONS
Workshop on the Open Archives Initiative (OAI)
and Peer Review Journals in Europe,
CERN, Geneva Switzerland, March 22-24, 2001
“The participants were unanimous in their belief that the
certification of scholarly work remains a fundamental part
of a system for scholarly communication.”
“It was [also] generally believed that the electronic
environment allows for novel approaches to accord a
stamp of quality to scholarly works.”
Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert
Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful
Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.
http://www.arl.org/sparc/SPB21_OAI.pdf
“Let us be imaginative in exploring the
remarkable possibilities of this
brave new medium.”
Stevan Harnad, “Implementing Peer Review on the Net:
Scientific Quality Control in Scholarly Electronic
Journals, in Scholarly Publication: The Electronic
Frontier, edited by Robin P. Peek and Gregory B. Newby
(Cambridge MA: MIT Press, 1996).
http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
“Let us be more imaginative in exploring
the remarkable possibilities of this
brave new medium.”
With Apologies to Stevan Harnad
TM
http://lockss.stanford.edu/
LOCKSS
For centuries libraries and publishers have had
stable roles: publishers produced information;
libraries kept it safe for reader access. There is no
fundamental reason for the online environment to
force institutions to abandon these roles.
The LOCKSS model capitalizes on the traditional
roles of libraries and publishers. LOCKSS
creates low-cost, persistent digital "caches" of
authoritative versions of http-delivered content.
LOCKSS
The LOCKSS software enables institutions to
locally collect, store, preserve, and archive
authorized content thus safeguarding their
community's access to that content.
The LOCKSS model enforces the publisher's
access control systems and, for many publishers,
does no harm to their business models.
LAMPSS
Lots of Alternative Models Provide Sensible Solutions
DISCLAIMER
ON
The alternative peer review models
profiled are for informational and
educational purposes only and do not
necessarily constitute an endorsement.
ALTERNATIVE PEER REVIEW
Neo-Classical
Certificationbased
Open Peer Review
Commentarybased
Collaboratelyfiltered
Institution-based
Citation-based
Index-based
Metadata-based
Computer-assisted
NO Peer Review
Moderator-based
Tier-based
NEO-CLASSICAL PEER REVIEW
Neo-Classical Peer Review
Neo-Classical Peer Review
Neo-Classical Peer Review
CERTIFICATION-BASED
Certification-Based
“The process of pre-publication peer
review could be improved and become a
more reliable indicator of manuscript
quality if reviewers were trained in, and
subsequently applied systematically, critical
skills and use of a hierarchy of evidence to
classify submitted articles being reviewed.”
Stephen Pritchard , “Peer Review - a Proposal for Change,”
Paper presented at Thinking Globally - Acting Locally:
Medical Libraries at the Turn of an Era,
th
8 European Conference of Health and Medical Libraries,
September 16-21, 2002, Deutsche Zentralbibliothek für Medizin, Köln, Germany.
http://www.zbmed.de/eahil2002/abstracts/pritchard.pdf
OPEN PEER REVIEW
Open Peer Review
IDENTIFICATION OF REVIEWERS / SIGNED REVIEWS
BMJ
bmj.com
BioMed Central
biomedcentral.com
eMJA (Medical Journal of Australia)
www.mja.com.au/public/information/project.html
COMMENTARY-BASED
Commentary-based
Readers can comment before and/or after classic peer
review, or instead of classic peer review
Electronic Transactions on Artificial Intelligence
(www.etaij.org)
OPEN REVIEW / REFEERING
Journal of Interactive Media in Education
(www-jime.open.ac.uk)
PRE- AND POST- COMMENTARY
Psycoloquy
(psycprints.ecs.soton.ac.uk)
POST PEER REVIEW COMMENTARY)
Electronic Transactions on Artificial Intelligence
Journal of Interactive Media in Education
http://psycprints.ecs.soton.ac.uk/
COLLABORATIVELY-FILTERED
Collaboratively-Filtered
“DEFINITION: “Guiding people's choices
of what to read, what to look at, what to
watch, what to listen to (the filtering part);
and doing that guidance based on
information gathered from some other
people (the collaborative part)."
Paul Resnick
http://www.cni.org/Hforums/cni-announce/1996/0031.html
ResearchIndex / CiteSeer
http://www.researchindex.com
INSTITUTION-BASED
Institution-Based
INSTITUTIONAL INITIATIVES
DSpace™ (MIT)
www.dspace.org
eScholarship (University of California)
escholarship.cdlib.org
Glasgow ePrint Service (University of Glasgow)
eprints.lib.gla.ac.uk
CITATION-BASED
Citation-Based
Citations to Open Access / OAI-compliant
documents are indicators of document importance
http://citebase.eprints.org/
INDEX-BASED
Index-Based
INDEXING OF EPRINTS BY COMMERICAL ABSTRACTING AND
INDEXING SERVICE
Chemical Abstracts (American Chemical Society) (CAS)
indexes select appropriate e-prints from the arXiv.org
eprint server as well from the Chemical Preprint
Service (Elsevier)
Its “selection criteria for this kind of electronic
document are essentially the same as for the traditional
printed documents: they must report new information
of chemical or chemistry-related interest and must be
original publications. Also, the electronic publication
must be publicly available and have some relative
permanence ….”
Eric Shively / Chemical Abstracts Service
Index-Based
“CAB Abstracts doesn’t currently include
Eprints or Preprints, but we are looking at the
implications and possible mechanisms for
accessing and indexing Eprints and/or
Preprints related to the applied life sciences.”
Tracy Shaw / CAB International
METADATA-BASED
Metadata-Based
<oai-quality>
<category>internal</category>
<process>
peer review
</process>
<organization>
CERN
</organization>
<policies>
http://www.cern.ch/policies/review.html
</policies>
</oai-quality>
William Y. Arms, “Quality Control in Scholarly Publishing On The Web.
What Are the Alternatives to Peer Review?” PowerPoint presentation given at the Workshop on the Open
Archives Initiatives (OAI) and Peer Review Journals in Europe, March 22-24, 2003, CERN, Geneva, Switzerland.
http://www.cs.cornell.edu/wya/papers/CERN-2001.ppt
COMPUTER-ASSISTED
Computer-Assisted (1)
SOFTWARE THAT ASSISTS IN THE EVALUATION OF A
SUBMITTED MANUSCRIPT
A Software Program to Aid in Peer Review
Alvar Loria and Gladys Faba
Objective: To characterize a personal computer-based software
program developed as an aid to peer review of medical papers.
Design: The software is a Windows-based application that records
automatically a numeric score to a series of questions related to 8
sections of scientific papers (introduction, methods, results, and
discussion, plus 4 other sections). The questions and sections vary
according to type of paper (original reports, case reports, or reviews),
and the final output is a score with a maximum of 100 for a "perfect"
paper. The software was tested using a single reviewer to judge 289
papers (169 original reports, 50 case reports, and 70 reviews) from 44
Mexican medical journals. All statistical analysis of scores were done
with nonparametric tests.
Computer-Assisted
(2)
Results: The paper scores ranged from 29 to 97 with slightly higher
median and less dispersion of scores for reviews as compared with
original reports and case reports, but these differences did not reach
significance. Two observations suggest that the software operated
reasonably well: a) there were some differences in the section scores
by type of paper that agreed well with differences in their complexity;
b) the journal scores showed an association with their number of
original papers and their percentage of original papers (Kruskal-Wallis
test, P=.06 and 0.07, respectively).
Conclusions: The software operated reasonably well when used to
compare the relative quality of 289 papers. The validity of the program
is restricted in this study to the experience of 1 reviewer. An analysis
of the raw scores helped in detecting some ambiguous and redundant
questions that have been modified in an improved version. The
program has a potential as a training tool for inexperienced reviewers
or as a scorekeeper for experienced peer reviewers.
Alvar Loria and Gladys Faba, “A Software Program to Aid in Peer Review,” Abstract of paper presented at the
Third International Congress on Biomedical Peer Review and Global Communications,
September 18-20, 1997, Prague, Czech Republic.
http://www.ama-assn.org/public/peer/arev.htm
NO PEER REVIEW
NO Peer Review
http://xxx.arXiv.cornell.edu
MODERATOR-BASED
Moderator-based (1)
The intent of this model is to allow the widest range of
scientific manuscripts to be archived, searched, and
distributed electronically at the lowest possible cost.
This would be accomplished through very minimal
filtering and subsequent placement of eprints on a noncommercial archival server by a subject-specific
Moderator appointed by a society (or consortia of
societies).
A society-appointed Editorial Board (with double-blind
peer review approved by the non-profit Peer Review
Inc. organization) would then the identify the most
important materials from among these archived items,
and the stamp of approval for these items would be
included in a secondary Virtual Collection.
Moderator-based (2)
There are no direct submissions to the Editorial Board;
manuscripts would be directed to the Editorial Board
in one of three ways:
1. nominated by the eprint Moderator upon receipt
for the archival server,
2. notification sent to the Editorial Board when a
threshold number of hits are generated by any one
manuscript on the archive server, and
3. nominated by readers of material from the
archive; this process requires a letter of support
outlining the importance of the work to the Editorial
Board.
Moderator-based (3)
The Virtual Collection could be produced as a variety
of products:
enhanced abstracts
email threads (with comments)
virtual reviews of sub-disciplines
SDIs (selective dissemination of information) current
awareness tools
This process:
reduces the load on the Editorial Boards, which results in a
faster review process; differentiates those items worthy of
higher recognition from those worthy of archiving, making it
easier for a reader to filter material, based upon a society and
discipline authority (rather than commercial reasoning);
provides for search/browse/sdi from the Virtual Collection for
filtered info, reducing this more expensive option for only those
items recognized as of the highest quality.
2
3
1
David Stern, “The eprint Moderator Model,” Newsletter on Serials
Pricing Issues no. 214 (February 8, 1999).
http://www.lib.unc.edu/prices/1999/PRIC214.HTML#214.5
TIER-BASED
Tier-based
Two separate domains
Standard Tier
Any and all submissions would be accepted after a
cursory examination of or other pro forma
certification.
The review process could be “minimally laborintensive, perhaps relying primarily on an automated
check of author institutional affiliation, prior
publication record, research grant status, or other
related background; and involve human labor
primarily to adjudicate incomplete or ambiguous
results of an automated pass.”
Tier-based
Upper Tier
“At some later point (which could vary from article to
article, perhaps with no time limit), a much smaller set
of articles would be selected for the full peer review
process. The initial selection criteria for this smaller set
could be any of a variety of impact measures, to be
determined, and based explicitly on their prior
widespread and systematic availability and citability:
e.g., reader nomination or rating, citation impact, usage
statistics, editorial selection, ... .”
Paul Ginsparg, “Can Peer Review be Better Focused?,”
Science & Technology Libraries 22 No. 3/4 (In press).
http://arxiv.org/blurb/pg02pr.html
DISCLAIMER
OFF
Presented for Your Consideration
FREEDOM OF IDEAS
http://www.nrm.org/exhibits/current/four-freedoms.html
‘FUTURE OF IDEAS’
The explosion of innovation we have seen
in the environment of the Internet was not
conjured from some new, previously
unimagined technological magic; instead, it
came from an ideal as old as the nation.
Creativity flourished there because the
Internet protected an innovation commons.
‘FUTURE OF IDEAS’
The Internet’s very design built a neutral
platform upon which the widest range of
creators could experiment.
The legal architecture surrounding it protected
this free space so that culture and information –
the ideas of our era–could flow freely and
inspire an unprecedented breadth of expression.
Lawrence Lessig, The Future of Ideas: The Fate of the Commons in a Connected
World. (New York: Random House, c2001).
“IT’S
NOT
ABOUT
PUBLICATION;
IT’S
ABOUT
IDEAS.”
The Association of Learned and Professional Society
Publishers (ALPSP) Survey
Authors and Electronic Publishing
Scholarly research communication has seen farreaching developments in recent years.
Most journals are now available online as well as
in print, and numerous electronic-only journals
have been launched;
the Internet opens up new ways for journals to
operate.
Authors have also become conscious of alternative
ways to communicate their findings, and much has
been written about what they ought to think.
ALPSP felt that it would be timely to
discover what they actually thought and
what they actually did. This survey aimed to
discover the views of academics, both as
authors and as readers. Some 14,000
scholars were contacted across all disciplines
and all parts of the world, and nearly 9%
responded; their detailed comments make
thought-provoking reading.
Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors'
and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned and
Professional Society Publishers, 2002).
http://www.alpsp.org/pub5.htm
“When asked to predict what would
be the most common form of quality
control in five years time, only a
bare majority answered
‘traditional peer review’.”
Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15
no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf
FURTHERMORE …
16% said that the referees would no longer
be anonymous
27% said that traditional peer review would
be supplemented by post-publication
commentary
45% expected to see some changes in the
peer-review system within the next five
years
Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15
no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf
Importance of the Peer Review Process
Peer-reviewed
100
90
80
70
60
50
40
30
20
10
0
Refs' comments
published
Referees identified
Public commentary on
eprints
Post-publication public
commentary
Ability to submit
comments
http://www.alpsp.org/pub5.ppt
What is Gray/Grey Literature ?
Papers are often written to inform funding bodies
about the results of research projects, to
support grant applications, to inform rapidly a
specific scientific community, to present
preliminary results at conferences or as
dissertations.
Such material is disseminated quickly, often in
limited numbers, before or without the formal
publication process. Such documents are called
non-conventional or grey literature.
http://www.kb.nl/infolev/eagle/what_is_gl.htm
The Value of Grey Literature
Grey literature is really a type of informal
communication, which on a scale of formality,
fits in somewhere between conversation and
normal publication. A formal publication may
follow later but in many cases - contrary to the
common assumption - these papers may not
been made publicly available at all.
http://www.kb.nl/infolev/eagle/what_is_gl.htm
Nevertheless, grey publications may contain
comprehensive, concrete and up-to-date information on
research findings, and investigations have shown, that
even when grey documents are published officially at a
later stage, detailed information on techniques,
methods, measured values and details of experiments
are frequently omitted.
For these details of importance for further research, the
non-conventional literature is then the first and only
source of information.
http://www.kb.nl/infolev/eagle/what_is_gl.htm
Veterinary Medicine
12 Major Veterinary Medicine Journals
Overall, 6.38% of cited literature was
Gray/Grey Literature
The figures for individual journals ranged
from about 2.5 % to 10% gray/grey
literature
Research journals cited a higher percentage of
Gray/Grey Literature than did Clinical titles
William H. Weise and Nancy Pelzer, “Bibliometric Study of Grey
Literature in Core Veterinary Medicine Journals,” Journal of the
Medical Library Association 91 no. 4 (October 2003): In press.
Indexing and Abstracting Services
SIGLE: System for Information on Grey
Literature
National Technical Information Service
(NTIS)
PsychINFO (Psychological Abstracts)
SIGLE
System for Information on Grey
Literature
Grey literature documents covered
by SIGLE are technical or
research reports, preprints,
committee reports, working
papers, dissertations,
conference papers, discussion
and policy papers, government
reports, market surveys, etc.
http://www.kb.nl/infolev/eagle/frames.htm
SIGLE
System for Information on Grey
Literature
No. of Records | Category
4,158 | Aeronautics
17,044 | Agriculture, plant &
veterinary sciences
17,668 | Environmental pollution,
protection & control
256,657 | Humanities,
psychology & social sciences
81,269 | Biological & medical
sciences
25,089 | Chemistry
http://www.kb.nl/infolev/eagle/frames.htm
NTIS
(National Technical Information Service)
The NTIS Database produced by the National Technical
Information Service, is the preeminent resource for
accessing the latest U.S. government-sponsored research
and worldwide scientific, technical, engineering, and
business-related information.
http://www.csa2.com/csa/factsheets/ntis.shtml
NTIS Database provides bibliographic data
and abstracts of unclassified and publicly
available information from research reports,
journal articles, data files, computer programs
and audio visual products, from U.S. and nonU.S. governmental, organizational, and
commercial sources
Subject Coverage
Administration
and
Management
Aeronautics &
Aerodynamics
Agriculture
Behavior &
Society
Business
Chemistry
Communications Computer
Science
Education
Energy
Engineering
Environmental
Sciences
Health Care
International
Trade
Library &
Information
Science
Materials
Sciences
Mathematical
Sciences
Natural
Resources &
Earth Sciences
Nuclear Science
Physics
Regulations
Technology
Telecommunications
Transportation
PsycINFO
PsycINFO provides access to international literature in
psychology and related disciplines. Unrivaled in its depth of
psychological coverage and respected worldwide for its high
quality, the database is enriched with literature from an array of
disciplines related to psychology such as psychiatry, education,
business, medicine, nursing, pharmacology, law, linguistics, and
social work.
http://www.csa2.com/csa/factsheets/psycinfo.shtml
PsycINFO includes psychological research and its applications; the
database is of prime relevance to many industries and research
establishments worldwide. The sources include over 1,400
professional journals, chapters, books, reports, theses and
dissertations, published internationally.
Subject Coverage
Applied
Psychology
Communicatio
n Systems
Developmental Educational
Psychology
Psychology
Experimental
Psychology
Personality
Psychological
and physical
disorders
Professional
personnel
issues
Physiological
Psychology
and
Neuroscience
Psychometrics
And Statistics
Social
Psychology
Treatment and
Prevention
DATABASE COVERAGE AND SIZE
Database
SIGLE
NTIS
Coverage
1976 –
Present
1964 - Present
Size
781,410 records
(November 2002)
2,168,400
records
(October 2001)
PsycINFO
1872Present
1,870,180
records
(September 2002)
http://www.neci.nec.com/~lawrence/papers/online-nature01/
Conference papers are typical gray/grey literature!
EPrints are Gray/Grey Literature
Daniela Luzi (1998) “E-Print Archives: a New
Communication Pattern for Grey Literature,”
Interlending & Document Supply 26 no. 3 (1998):
130-139.
Gray/Grey Literature
“It’s good enough,
it’s smart enough,
and
doggone it, people
use it!”
With apologies to
Stuart Smalley
http://www.openarchives.org/Register/BrowseSites.pl
June 2003
http://software.eprints.org/
http://eprints.anu.edu.au/
http://caltechcstr.library.caltech.edu/
http://eprints.lub.lu.se/
NETWORKED DIGITAL LIBRARY OF THESES AND DISSERTATIONS
http://ndltdpapers.dlib.vt.edu:9090/
http://dspace.org/index.html
http://rocky.dlib.vt.edu/~etdunion/cgi-bin/browse.pl
http://www.ncstrl.org/
Quality Control in Scholarly
Publishing on the Web
"Most of the high quality materials on
the Web are not peer-reviewed and much
of the peer-reviewed literature is of
dubious quality.”
William Y. Arms, "What Are the Alternatives to Peer Review?
Quality Control in Scholarly Publishing On The Web."
Journal of Electronic Publishing, 8 no. 1 (August 2002).
http://www.press.umich.edu/jep/08-01/arms.html
http://www.update-software.com/Cochrane/MR000016.pdf
Cochrane Methodology Review
Despite its widespread use and costs, little
hard evidence exists that peer review
improves the quality of published
biomedical research.
There had never even been any consensus
on its aims and that it would be more
appropriate to refer to it as ‘competitive
review’.
Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”
BMJ 326 (February 1, 2003): 241
http://bmj.com/cgi/reprint/326/7383/241/a.pdf
Cochrane Methodology Review
On the basis of the current evidence, ‘the
practice of peer review is based on faith in
its effects, rather than on facts,' state the
authors, who call for large, government
funded research programmes to test the
effectiveness of the [classic peer review]
system and investigate possible alternatives.
Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”
BMJ 326 (February 1, 2003): 241
http://bmj.com/cgi/reprint/326/7383/241/a.pdf
Cochrane Methodology Review
The use of peer-review is usually assumed
to raise the quality of the end-product (i.e.
the journal or scientific meeting) and to
provide a mechanism for rational, fair and
objective decision-making. However, these
assumptions have rarely been tested.
Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager,
Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:
Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
Cochrane Methodology Review
The available research has not clearly
identified or assessed the impact of peer-review
on the more important outcomes (importance,
usefulness, relevance, and quality of published
reports)
… [G]iven the widespread use of peer-review
and its importance, it is surprising that so little
is known of its effects
Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager,
Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:
Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
Royal Society
http://www.biomedcentral.com/news/20030203/04/
UNCITEDNESS
David P. Hamilton, "Publishing by and for? -- the numbers,”
Science (New Series) 250 (4986) (December 7 1990): 1331-1332.
http://www.garfield.library.upenn.edu/papers/hamilton1.html
David P. Hamilton,“Research Papers: Who’s Uncited Now?,”
Science (New Series) 251 (4989) (January 4, 1991): 25
http://garfield.library.upenn.edu/papers/hamilton2.html
David P. Hamilton,“Research Papers: Who’s Uncited Now?,”
Science (New Series) 251 (4989) (January 4, 1991): 25
http://garfield.library.upenn.edu/papers/hamilton2.html
REJECTED CITATION CLASSICS
NOBEL PRIZE RESEARCH
Severo Ochoa
Polynucleotide phosphorylase
Hans Krebs
Citric acid cycle
Rosalind Yalow
Radioimmunoassay
Harmut Michel
Photosynthetic processes
Juan Miguel Campanario, “Commentary: On Influential Books and Journal Articles
Initially Rejected Because of Negative Referees’ Evaluations, Science Communication 16 no. 3
(March 1995): 306-325
PEER REVIEW: PURPOSES
Peer review helps to ensure that published research is:
Important ?
Original ?
Timely ?
Technically-reliable ?
Internally consistent ?
Well-presented ?
Benefited from guidance by experts ?
Carin M. Olson, “Peer Review of the Biomedical Literature,” American
Journal of Emergency Medicine 8 no.4 (July 1990): 356-368.
FILTERING (1)
UpStream / DownStream
“Researchers look at … [certain types] of
electronic publications because, despite
being tentative, may be relevant to their
work. Researchers are expected to do their
own ‘downstream-filtering’ of relevant
information, which in the electronic world
can be facilitated by providing metainformation.”
FILTERING (2)
UpStream / DownStream
Some have expressed the concern that having
non-peer reviewed documents with peerreviewed documents on the same server would
‘contaminate’ the latter and compromise its
quality:
Readers could have trouble in distinguishing
the different sections
‘Making non-peer-reviewed as well as peerreviewed material will confuse both scientists
and the public … .’
FILTERING (3)
UpStream / DownStream
“‘However, this perhaps belittles the ability of
scientists to recognize different levels of
evidence and to be able to interpret [quality]
labels that could make clear that certain
materials is non-peer-reviewed content’ after all
this is the age of transparency rather than
paternalism … .”
Gunther Essenbach, “The Impact of Preprint Servers and Electronic
Publishing on Biomedical Research,”
Current Opinion in Immunology 12 no. 5 (October 2000): 499-503.
http://yi.com/home/EysenbachGunther/scans/Eysenbach2000e_CurrOpImmunol_preprint_servers.pdf
INVISIBLE HAND OF
CLASSICAL PEER REVIEW
“The refereed journal
literature needs to be
freed from both
paper and its costs,
but not from peer
review, whose
‘invisible hand’ is
what maintains its
quality.”
Stevan Harnad
http://www.presidentmoron.com
INVISIBLE HAND OF
CLASSICAL PEER REVIEW
INVISIBLE HAND OF
CLASSICAL PEER REVIEW
“Human nature being what it is, it cannot be
altogether relied upon to police itself. Individual
exceptions there may be, but to treat them as the
rule would be to underestimate the degree to
which our potential unruliness is vetted by
collective constraints, implemented formally.”
Stevan Harnad, “The Invisible Hand of Peer Review,”
Exploit Interactive no. 5 (April 2000).
http://www.exploit-lib.org/issue5/peer-review/
INVISIBLE HAND OF
CLASSICAL PEER REVIEW
“The system is not perfect, but it is what has
vouchsafed us our refereed journal literature to
date, such as it is, and so far no one has
demonstrated any viable alternative to having
experts judge the work of their peers, let alone
one that is at least as effective in maintaining
the quality of the literature as the present
imperfect one is.”
INVISIBLE HAND OF
CLASSICAL PEER REVIEW
“Remove that invisible constraint -- let the authors be
answerable to no one but the general users of the
Archive [arXiv.org] (or even its self-appointed
"commentators") -- and watch human nature take its
natural course, standards eroding as the Archive
devolves toward the canonical state of unconstrained
postings: the free-for-all chat-groups of Usenet … , that
Global Graffiti Board for Trivial Pursuit -- until
someone re-invents peer review and quality control.”
Stevan Harnad, “The Invisible Hand of Peer Review,”
Exploit Interactive no. 5 (April 2000).
http://www.exploit-lib.org/issue5/peer-review/
INVISIBLE HANDS
INVISIBLE HANDS
Personal reputation
Institutional
reputation
Pride
Self-respect
Professional respect
Peer pressure
‘Critical Peer Response’
Invisible College
Self-Archiving-ProcessItself
Open access
Common Sense
Self-correcting dynamics
RECOMMENDATIONS
Workshop on the Open Archives Initiative (OAI)
and Peer Review Journals in Europe,
CERN, Geneva Switzerland,March 22-24, 2001
“It was [also] generally believed that the
electronic environment allows for novel
approaches to accord a stamp of quality to
scholarly works.”
Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert
Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful
Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.
http://www.arl.org/sparc/SPB21_OAI.pdf
Examples of new metrics that can be extracted
from a fully electronic communication system are:
Usage counts of a work
Automatically extracted citation information
with a scope beyond the ISI- core journals
Amount of discussion generated by a paper
submitted in a system with open peer review
and peer comment
Etc.
Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert
Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful
Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.
http://www.arl.org/sparc/SPB21_OAI.pdf
Scientific Publishing as Rhetoric
The problems with peer review become evident once the
fact that science has a rhetorical element is accepted.
On the one hand, the traditional mode of peer review
obscures the problems of reference and the rhetorical
dimension of science. The rhetorical process which is at
the heart of science and peer review conveniently
disappears with the final publication of the manuscript.
In its place is an ideal typical representation (the scientific
paper) of the realist assumptions about empirical reference.
All the academic world sees is a polished manuscript
where the personal involvement of the researcher and
reviewers has been systematically eliminated.
Mike Sosteric, “Interactive Peer Review: A Research Note,” Electronic Journal of
Sociology 2 no. 1 (1996).
http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html
‘IDEAL SPEECH SITUATION’
Jürgen Habermas
A theoretical construct that describes the
ideal type of interpersonal interaction
that should exist in a rhetorical situation.
IDEAL SPEECH SITUATION
the ideal speech situation permits each interlocutor an
equal opportunity to initiate speech;
there is mutual understanding between interlocutors;
there is space for clarification;
all interlocutors are equally free to use of any speech
act;
there is equal power over the exchange.
Applied in the context of peer, the Ideal Speech Situation
‘would permit unimpeded authorial initiative, endless rounds
of give and take, [and] unchecked openness among authors,
editors, and referees.’
Mike Sosteric, “ Interactive Peer Review: A Research Note,” Electronic Journal of
Sociology 2 no. 1 (1996).
http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html
CONTINUA
Continuum of PUBLICATION
(‘Scholarly Skywriting’)
STRONG
WEAK
MEDIUM
Continuum of REVIEW
(‘Scholarly Skyreading’)
OPEN ACCESS
and
OPEN RETRIEVAL
without
OPEN USE
Incongruent
Contradictory
Ironic
Paradoxical
Cognitively Dissonant
Three-Legged Stool
ACCESS -- RETRIEVAL -- USE
ACCESS
OPEN ACCESS INITIATIVES
RETRIEVAL
OPEN ARCHIVES INITIATIVE FOR
METADATA HARVESTING
USE
OPEN SCHOLARSHIP
ACCESS -- RETRIEVAL -- USE
Lots of Alternative Models
Provide Sensible Solutions
Four-Legged Stool
ACCESS -- RETRIEVAL -- USE -- NAVIGATION
INFORMATION OVERLOAD
OAIster
A search engine for
freely available,
difficult-to-access,
academicallyoriented digital
resources that are
OAI -compliant
http://oaister.umdl.umich.edu/
University of Michigan
Digital Library Production Service
institutional
repositories
departmental
repositories
e-Journal collections
technical reports
dissertations and
theses
discipline eprint
collections
working papers
Internet resources
audio
video
images
Cognitive Psychology
Open Archives Initiative Protocol for
Metadata Harvesting
(OAI-PMH)
ADD
METADATA ELEMENT for QUALITY
“Within the framework of OAI,
there is a need for a new protocol for certification.
There was strong support
for the extension of the usage of the
OAI protocol beyond discovery-related
metadata. Given the focus of the [1st OAI]
workshop on peer review, concrete
actions were suggested to address the
exchange of certification-related
metadata using the OAI protocol in a
trusted environment.”
Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de
Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European
Science Foundation Policy Briefing 21 (April 2003): 1-8.
http://www.arl.org/sparc/SPB21_OAI.pdf
QUALITY METADATA (1)
<oai-quality>
<category>internal</category>
<process>
peer review
</process>
<organization>
CERN
</organization>
<policies>
http://www.cern.ch/policies/review.html
</policies>
</oai-quality>
William Y. Arms, “Quality Control in Scholarly Publishing On The Web.
What Are the Alternatives to Peer Review?” PowerPoint presentation given at Workshop on the Open Archives
Initiatives (OAI) and Peer Review Journals in Europe, March 22-24, 2003, CERN, Geneva, Switzerland
http://www.cs.cornell.edu/wya/papers/CERN-2001.ppt
QUALITY METADATA (2)
<oai-quality>
<category>internal</category>
<rating>
*****
</rating>
<organization>
42
</organization>
<policies>
http://www.cern.ch/policies/review.html
</policies>
</oai-quality>
CERTIFICATION SERVICES
Faculty of
1,000,000
CERTICATION SERVICES
New roles for Indexing and Abstracting
Services
Expanded Role for Learned and
Professional Societies
Establishment of Formal/Commercial
Reviewing Services
PEER REVIEW: STRENGTHS
The underlying strength of peer review
is“…the concerted effort by large numbers
of researchers and scholars who work to
assure that valid and valuable works are
published and conversely to assure that
invalid or non-valuable works are not
published … .”
Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses.
(Medford, NJ: Information Today, 2001).
Faculty of 1000 / BioMed Central
BioMed Central (biomedcentral.com) publishes
Faculty of 1000 (F1000), the leading literature
evaluation service and “new online research
tool that highlights the most interesting papers
in biology, based on the recommendations of
over 1000 leading scientists.” F1000 is managed
“by scientists for scientists” …. [and] provide[s]
a rapidly updated consensus map of the
important papers and trends across biology.”
www.facultyof1000.com
Faculty of 1000 / BioMed Central
Among its many benefits, F1000:
systematically organizes and evaluates the mass of
information within scientific literature;
provides scientists with a continuously updated
insider's guide to the most important papers within any
given field of research;
highlights papers on the basis of their scientific merit
rather than the journal in which they appear;
offers the researcher a consensus of recommendations
from well over 1000 leading scientists; and,
offers an immediate rating of individual papers by the
authors' peers, and an important complement to the
indirect assessment provided by the journal impact
factor.
Faculty of 1000 / BioMed Central
Within the F1000, the entire field of biology
is
divided
into
16
subject
areas
(‘Faculties’)(e.g., ‘Biochemistry,’, Cell Biology,
‘Microbiology’). Each ‘Faculty’ is subdivided
into three (3) to twelve (12) ‘Sections,’ (e.g.,
Biochemistry:
Biocatalysis,
Molecular
evolution, Protein folding), with each section
comprised of between 10 to 50 faculty
members. F1000 seeks to invite the best
internationally known scientists in each
represented field and to involve both
experienced and younger investigators.
“Peer review is a quality-control and certification (QC/C)
filter necessitated by the vast scale of learned research
today. Without it, no one would know where to start
reading in the welter of new work reported every day, nor
what was worth reading, and believing, and trying to build
one’s own further research upon.”
Stevan Harnad, “Free at Last: The Future of PeerReviewed Journals,”
D-Lib Magazine 5 no. 12 (December 1999)
http://www.dlib.org/dlib/december99/12contents.html
SEIZE THE E!
Embrace the potential of the digital
environment to facilitate access, retrieval,
use, and navigation of electronic
scholarship.
OPEN NAVIGATION
New Age Navigation:
Innovative Interfaces for
Electronic Journals
Gerry McKiernan
The Serials Librarian
Fall 2003
SUMMARY. While it is typical for electronic journals to
offer conventional search features similar to those
provided by electronic databases, a select number of ejournals have also made available higher-level access
options as well. In this article, we review several novel
technologies and implementations that creatively exploit
the inherent potential of the digital environment to
further facilitate use of e-collections.
Gerry McKiernan, “New Age Navigation: Innovative
Interfaces for Electronic Journals,”
The Serials Librarian, Fall 2003.
http://www.coleonline.us/serialslibrarian/
http://www.highwire.org
Topic Map
Topic Map
Topic Map
Topic Map
Topic Map
Secondary Screen
Topic Map
Topic Map
TopicMap is based on the Hyperbolic Tree
SDK for Java, licensed from Inxight
Software, Inc., a spin-off company from
Xerox PARC, and leading provider of
Unstructured Data Management solutions
for accessing, analyzing and delivering
information.
http://www.inxight.com
Semio
Automated categorization software technology
http://www.entrieva.com/
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/milliondemo/html/root.html
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/milliondemo/html/1_cx5.html
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/milliondemo/html/2_gx10.html
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/milliondemo/getnd.cgi?32323
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/cgi-bin/getfile.cgi/comp.ai/39340
http://www.springer.de/books/toc/3540679219-c.pdf
“There are some excuses, but at the
bottom it will be seen to be the
sluggishness of human nature and its
superstitious cleavage to old habits.”
Stevan Harnad, “Free at Last: The Future of PeerReviewed Journals,”
D-Lib Magazine 5 no. 12 (December 1999)
http://www.dlib.org/dlib/december99/12contents.html
“… [In] the digital world, the
evaluation process stands ready to
be reinvented in a clear, rational
way by the relevant research
communities themselves.”
Jean-Claude Guédon,
In Oldenburg’s Long Shadow: Librarians, Research
Scientists, Publishers,
and the Control of Scientific Publishing.
(Washington, D.C.: Association of Research Libraries,
•2001), 54.
http://www.arl.org/arl/proceedings/138/guedon.html
</ENDQUOTE>
“The Medium is the
Message …
And
the Method.”
With apologies to
Marshall McLuhen
THANK YOU FOR YOUR
ATTENTION!
Alternative Peer Review:
Quality Management
for
21st Century Scholarship
Gerry McKiernan
Science and Technology Librarian
and Bibliographer
Iowa State University Library
Ames IA
USA
gerrymck@iastate.edu
OPEN MIND
REVISED VERSION 1.01
July 27, 2003
11:30 AM