Some Thoughts On Peer Review In The Global Internet
Context
Narayanan Komerath
Daniel Guggenheim School of Aerospace Engineering
Georgia Institute of Technology
Atlanta
http://farm4.static.flickr.com/3112/3157622458_b19d039158.jpg
IMETI 2010, Orlando, FL June 2010
“What is a scholarly or peer reviewed journal?”
http://knowledgecenter.unr.edu/help/using/peer.aspx
“Scholarly and professional journals feature articles written by researchers and
practitioners in a particular subject area.”
“The authors often have particular specialties. Peer groups of researchers,
scholars and professionals within a specific discipline are the audience for
scholarly literature.”
“Well-accepted indicator of quality scholarship.”
“Process by which an author's peers read a paper submitted for publication.”
“A number of recognized researchers in the field will evaluate a manuscript and
recommend its publication, revision, or rejection.”
“Articles accepted for publication through a peer review process implicitly meet
the discipline's expected standards of expertise.”
“Articles selected by an editor or board. Standards of scholarship in such journals
are often equal or comparable to those of peer-reviewed publications, although this
is not always the case.”
From West Virginia University Library
“What is a Peer-Reviewed Journal?
..Sends articles submitted by authors to a group of experts in the field for review
before publication. …The review process helps to ensure that research published
in the journal is of high quality and contributes new information to the field.
Characteristics of Peer-Reviewed Journals:
Purpose is to share the results of original research with the rest of the
scholarly world.
Usually have a plain cover with lengthy and in-depth articles that contain charts,
graphs, and tables. ..Minimal use of photographs and color. Any advertising is
geared towards the subject discipline.
Articles are written by researchers, scientists or scholars in the field.
Authors cite their sources in footnotes or bibliographies, which are often
extensive.
Uses the technical terminology appropriate to the discipline and assumes that the
reader will have a similar scholarly background.”
http://undsci.berkeley.edu/images/us101/p
eerreview.gif
http://www.whitehead.mit.edu/news/paradi
gm/spring_2008/img/peer_review.jpg
http://1.bp.blogspot.com
Reviewer’s S.O.P. According to n.k.
1.
2.
3.
4.
5.
6.
7.
8.
Read the paper back and forth at least 3 times. Relate conclusions to what is
shown. Put away for a couple of days and mull on the paper.
Consider originality. Search for related prior work if necessary.
Follow author’s train of thought and check for reasonableness.
Read carefully and note questions
Compose review and pose questions
Develop summary recommendations and ratings
Get the review back to the editor in a timely manner.
If requested to review again, do so quickly.
A good peer review should engage the brains of the authors and the editors.
Takes time to do it right, but your professional standards are at stake.
How would you like someone like yourself as reviewer when you are the author?
My Background
32 years, full-time researcher/ academic tenure track/ tenured professor in a major
technological institution
- Combustion diagnostics / propulsion experiments
- Turbulent combustion/ fluid dynamics
- DSP / Optical and other diagnostics development
- Experimental aerodynamics / wind tunnel operations / vortex flows / combat aircraft /
helicopters
- Guiding 15+ engineering PhDs.
- Institutional peer review panels.
- Proposal and paper reviewer for national and international organizations
~ 250 publications, ~90 of them peer-reviewed.
~ 14 years internet / interdisciplinary / digital library / resource building & maintenance
~ 14 years in formally multidisciplinary endeavors
~ 12 years in Strategic / International / Security affairs
~ 8 years in social sciences interactions in community vs. academics
Some Questions From the Cross-disciplinary Perspective
Prevailing Definition
Questions/Criticisms
“Journals feature articles written by researchers and
practitioners in a particular subject area.”
•Narrow?
•Are credentials relevant?
•Excuse for incompetent reviewing?
•Tactic to keep out “disruptive ideas”?
“The authors often have particular specialties. Peer groups Modify to accommodate crossof researchers, scholars and professionals within a specific disciplinary endeavor?
discipline are the audience for scholarly literature.”
“Well-accepted indicator of quality scholarship.”
Circular metric?
“Process by which author's peers read a paper submitted”
Do they really? CAN they?
“A number of recognized researchers in the field will
evaluate .. and recommend publication, revision, or rejection.”
Are researchers scholars?
“Articles accepted for publication implicitly meet the
discipline's expected standards of expertise.”
Circular logic?
If “Articles are selected by an editor or board: Standards of
scholarship in such journals are ..not always comparable to
those of peer-reviewed publications”
What is the absent process here?
Are Editors and the Board of a
journal experts? Researchers?
Scholars?
“Killing” (or sneaking in) a Paper is Easy if the Editor is Asleep (or Worse)
The dual role of quality control and motivation encounters conflicts of
interest, especially when there are underlying competitive aspects and
financial pressures.
•Delay
•Dismissive rejection, or glowing acceptance
•Unusually harsh or mild comments
•Demand for “additional work”
•Delay following rebuttal
•“Re-review” to think up new
excuses for rejection
http://2.bp.blogspot.com
So why have Peer Review?
Thoughtful review and comments from interested and knowledgeable persons is
more relevant than ever to motivating original thought, disciplined research and
progress.
.
What I have deleted is
“Accept only High-Quality Papers”
Observations on rejecting papers
1.
Some process for rejection is essential, if only to encourage other authors.
2.
High rejection rate is a low-quality metric of high quality.
3.
Professional organization’s responsibility to not publish nonsense is outweighed by the
risk of rejecting innovative work.
4.
There are better ways ….. 
Not possible in traditional model, to eliminate conflicts of interest.
•Reviewers are anonymous,
and do not have to defend their (in)actions
•Editors can delay publication
• Willing reviewers are hard to find
• Competent reviewers are harder to find
– and are usually competitors
Evolving technology and habits of researchers, and the attendant
fragmentation of time that many experts face, have driven traditional peer
reviewed paper journals to the verge of irrelevance.
•Adversarial / arrogant attitude (IP is
author’s!)
•Poor value addition by sloppy reviewing
•Cost (“paper charges”)
•Delay in review and publication
•Poor visibility: Libraries cannot afford to
subscribe to all, or most, journals
•Misses the Google Searcher Audience
• Citations will be more on papers that
are accessible!
Present generation does not follow
linear or hierarchial methods of
knowledge organization
“If u can’t b found on Google screens
1-3, u don’t exist”
www.columbia.edu/cu/21stC/issue-1.1/peer.htm
Casual adoption of technology in review systems poses its own
problems, especially when used in reviewing innovative work
•“Review Panels typically support the status quo” (priceless quote from an
experienced authority from this conference!)
•Adding technical convenience does not improve the amount or quality of time
and thought devoted to review.
http://francisthemulenews.files.wordpress.com/2009/04/dibujo20
090425_peer_review_not_always_useful_for_editors.jpg
Academic promotion/tenure
processes are relying more on
“prestige” and “percentile rankings
in a field” metrics along with
number counts of publications
Fragmentation & proliferation of
journals inhibits cross-disciplinary
innovation.
I reject this route as being
destructive of original thought and
cross-disciplinary
knowledge dissemination.
http://imvu.regnarts.com/news/wpcontent/uploads/2008/06/peer-review-tears.png
“Professor Komerath is one of the Top 3 Experts in the World
in measuring pressure-velocity correlations in turbulent premixed flames
using cross-spectral analysis of response-compensated Pitot microphone probes
with ceramic tips and a coherence metric”
(Great, but who else would want to join that club?)
Interim solution adopted by specialist communities to the lack of
competent “expert” reviewers in their narrow fields:
Creation of ever‐tighter “peer” circles,
leading to fragmentation of knowledge and the inability of any library to maintain
a comprehensive journal base.
Thus the most “prestigious” publications may be so exclusive that few can even
find them.
Scholarly publication and academic freedom
vs. responsible community behavior
Egregious examples of abuses in “religion” academia.
Refusal of the “Academy” to publish well-informed rebuttals and exposures from
lay practitioners destroys academy’s credibility.
“Journals feature articles written by researchers and practitioners in a particular
subject area…Peer groups of researchers, scholars and professionals within
a specific discipline are the audience for scholarly literature.”
Lesson: Scholarly communities that do not welcome and facilitate well-reasoned
disagreement from “lay persons”, are not worthy of respect
Technological capabilities and evolving trends in capturing
knowledge from prior work
•Internet Search Engines
•“Google Books” and “Google Scholar”
•Delivery to personal communication devices (Iphone, Ipad)
•Global access
• Moderated Discussion forum
• “Twitter”
Open, two-stage peer-review
From Nature (2006) | doi 10.1038/nature04988
Journal of Atmospheric Chemistry and Physics
Stage 1: Rapid pre-screening (access review) and immediate publication as
“discussion paper” on website. Public interactive discussion for 8 weeks, with
comments of designated reviewers & others are published alongside the
discussion paper. Reviewers sign or remain anonymous, but comments by
other scientists must be signed.
Stage 2: traditional manuscript revision and peer review. If accepted, final
papers are published in the main journal. Every discussion paper and
interactive comment remains permanently archived and individually citable.
Peer review model for cross-disciplinary innovation
Model
Rationale
Web-based journal with periodical
volume and date identifiers
Archival, with swift global search
enabled
Submitted on-line in specified format
For possible hardcopy publication
Assigned to 3 anonymous reviewers
by editors after editorial review.
Editor can reject. Review assignment
all over the world
Article posted on-line for comments
(BLOG model). Author can respond.
No censorship. Comments
moderated.
Anonymous review comments sent to
author only. Author has fixed period to
modify or withdraw article
Encourages reviewers to help author
improve article without penalty
Where disagreement remains, both
the anonymous comment and the
author’s rebuttal are published.
Main quality control: Archival reader
sees countering opinions – and
author’s rebuttal
Article plus discussion becomes
archival record.
Captures discussion and allows
reader to form an informed opinion
Further comments published as new.
Provides closure in a reasonable time.
Preliminary empirical observations
•Social-media forum to motivate experts to participate:
•Examples:
1.Strategic affairs discussion forum
2.Space Solar Power forum
3.Discussions on major news items (Gulf of Mexico oil leak)
•Most engineering journals are not likely to generate many comments from readers.
•Getting authors to submit articles, requires recognition of quality control processes
and archival reliability of “journal”
•Strong moderation is required
•Author ego is a problem with many authors unused to social
media
Notes:
J. Atmospheric Science reports excellent impact factor.
“Deters submission of low-quality papers: low rejection rate”
Conclusions
•
•
•
•
•
•
Purpose of peer review should be carefully re-emphasized
Abuses cannot be avoided with traditional paper journal review system
Swift & unobstructed publication essential.
Open but moderated, interactive discussion provides good quality
Archived reviews/rebuttal is a good experiment
Motivating participation requires thought and resources
http://www.the-funneled-web.com/images_2008/PeerReviewACS.gif
ACKNOWLEDGEMENTS
The work reported in this paper was made possible by resources being
developed for the “EXTROVERT” cross-disciplinary learning project under
NASA Grant NNX09AF67G S01. Mr. Anthony Springer is the Technical Monitor.
http://docinthemachine
.com/wordpress/wpcontent/uploads/2007/
02/gspz0625.jpg