Scholarly Communications
•
•
•
•
•
Bradley Hemminger
Assistant (Associate) Professor
School of Information and Library Science
University of North Carolina at Chapel Hill
bmh@ils.unc.edu
Big Questions
• What is scholarship?
• How do we measure the value of
scholarship?
• What should we preserve?
• How should we design an ideal model,
given that technology has freed us from
the constraints of physical mediums?
Public Storehouse of Knowledge
• Multiple open digital archives, holding all
the world’s knowledge. A single logical
universal archive, created by dynamic
federation of all public archives.
• Contains everything: archive holds grey
literature (publicly deposited) and gold
literature (refereed articles).
• No barriers to access. Knowledge is freely
available to anyone, any time, anywhere.
•
Access to information and knowledge correlates to society’s quality of life.
Archive Model (NeoRef)
• All material and metadata are author contributed to a
public OAI archive (author retains ownership).
• OAI archives have automated or manual moderator to
filter out “junk”.
• Everything--articles, reviews, comments, indexings, etc.,
are stored as digital content items on archive using the
same mechanism. Reviews contain quantitative score,
qualitative grade, qualitative comments.
• All materials universally available via search engines that
harvest metadata from OAI archives.
• Retrieval is through Google like one stop shopping
search interface, with dynamic filtering based on
metadata and reviews to limit hits to manageable
number to review.
Challenges are in Retrieval
• All material is archived (good and bad)
• Metrics (some new) are used to
differentiate type, content, and quality.
• Dynamic Searching allows quickly finding
material of most interest. Search on
– Type article=Review AND date > 1950
– Content (schizophrenia AND GeneX)
– Quality: Peer reviewed {journals}, citation rate
> XYZ
How Peer Review might work…
• Author submits article to her institution’s open archive
(DOI uncch:sils/0007548.pdf).
• Author “submits” to journal EMEDICINE by providing
DOI of article.
• Journal Editor schedules two reviewers. Reviewers
review article, and submit their reviews (cornell:0191.pdf,
ucb:0084.pdf).
• Author revises, and places revised article (DOI
uncch:sils/0007957.pdf) on archive, and submits this
final version to EMEDICINE.
• Journal submits review (EMED:0023424.pdf) which is
final statement from journal (editor), and indicates
acceptance of uncch:sils/0007957.pdf as EMED article).
Scholarly Communications Process
Today’s Example
Idea
V1
Referees
Revision
for journal
V5
Present to
colleagues
Present at
conference
V2
V3
Journal
Final
Revision
Revision to
update
analysis
V6
V7
Submit to
journal V4
Revision to
include
additional
new results
V8
Scholarly Communications
Process: What’s Captured Today
Journal
Final
Revision
V6
Only one version is captured, and the same
community then pays to buy back access to
article
Scholarly Communications Process
formulate
Idea
V1
discussion
Present to
colleagues
Present at
conference
V2
V3
comments
Author revision
Referees
Revision
for journal
V5
V6
Copyproofing
Two peer reviews
comments
comments
Journal
Final
Revision
discussion,
revision
Submit to
journal V4
Revision to
correct
analysis
V7
Criticisms, new
thoughts,
revision
Revision to
include
additional
new results
V8
new results,
revision
Scholarly Communications
Process:What
I’d
like
to
see
saved!
formulate
discussion
discussion,
Idea
V1
Present to
colleagues
Present at
conference
V2
V3
comments
Author revision
Referees
Revision
for journal
V5
V6
Copyproofing
Two peer reviews
Submit to
journal V4
comments
comments
Journal
Final
Revision
revision
Revision to
correct
analysis
V7
Criticisms, new
thoughts,
revision
Revision to
include
additional
new results
V8
new results,
revision
Change the Process!
• Think of scholarly communication as continuous process
instead of single product (journal publication).
• Capture significant changes/versions of a work.
• Include all criticisms and comments about work (all
stages).
• Support normal scholarly discourse, including authors
responses as well as others comments.
• Add reviewer’s quantitative rating of material to allow
better filtering based on absolute quality metric during
retrieval.
• Add machine (automated) reviews.
• Represent and store more than just text (datasets, stats,
multimedia, etc). Capture over time (Memento).
Challenges
• Searching
– DC metadata to allow coarse discovery.
– Specialized searching within domain after locating material
(based on metadata field indicating appropriate search
interface).
– Interactive searching to allow refinement to most desirable set
within a few seconds. Use reviews to help filter search
(Facultyof1000).
– Google searching on full text (covers all materials, but generates
large number of hits, lower specificity).
– Automated agents to bring material of interest to your attention
(California digital library).
• Example: article scores > 7.0, refereed, citation
count above 10, type=research article, search terms
= schizophrenia, geneX)
Challenges
• Knowledge Representation
– Extend DC to include “concepts” and “claims” to allow
higher level searching compared to simple indexing.
– Make OAI and DC representation more robust by
always supporting DOI to uniquely identify materials.
– Support unique identification of authors as well.
– Making all content items submitted permanent
– Use DC fields to link related items, new version of
paper to old version.
Challenges
• Rights Administration
– Support mechanisms to allow authors to set
permissions as they desire, and enforce this.
– Example is OAI recent support of rights
administration using Creative Commons.
Challenges
• Peer Review
– Make more public. Make available comments
on articles.
– Add quantitative scoring as well as qualitative.
Overview of Peer Review
(talk about single repository, PLoS One)
Qualitative
Quantitative
Grade
Score (1-10)
Quantitative
Filter
Published
Article
Review
Peer,
Open,
Machine
Article
submitted
Send
elsewhere
Reject
Accept,
reject,
revise
with respect to XYZ standards
Comments
to Author
General Review Model Parallels
• In general, you have sample (material)
which is judged/scored quantitatively and
qualitatively by an identified observer with
respect to some standard.
NeoRef for Movies, Products,…
• The same process used by NeoRef to support Scholarly
Communication could be used for most any
communication of information purpose. All that is
required is storage of Digital Content Items, and linking
of reviews, comments, etc to them.
• DocSouth: self cataloged and indexed items are Grey;
librarian/archivist cataloged and indexed items are Gold.
• Movies: Grey is everyone’s reviews; Gold is Siskel and
Ebert reviews.
• Consumer Products: product reviews by Consumer
Reports (gold), user reviews (grey).
Current Peer Review Options
• Human Judgement
–
–
–
–
Expert peer review (status quo)
Certified expert peer review
Open Peer Review BMJ, BioMed
Open comment review psycprints
Quantitative Qualitative
√
√
√
√
√ (relative)
√ (relative)
√ (absolute)
√ (absolute)
√
√
• Computer Judgement
– Computer peer review
• Human Usage
– Citation-based (CiteSeer)
– Usage counts (CiteSeer) Example
– Quantity of discussion
• Coarse Categorization
– Two Tier (grey/gold)
– Moderator (current arXiv)
– No review (old arXiv)
√
√
√
√
√
√
What do users want?
ALPSP survey was intended to discover the
views of academics, both as authors and as
readers. Some 14,000 scholars were
contacted across all disciplines and all parts
of the world, and with almost 9%
responding.
Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors'
and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned and
Professional Society Publishers, 2002).
http://www.alpsp.org/pub5.htm
Importance of journal features
90
80
70
60
50
40
30
20
10
0
Citation links
Additional
data
Addit/colour
images
Manipulable
content
Video/sound
Importance of the peer review
process
Peer-reviewed
100
90
80
70
60
50
40
30
20
10
0
Refs' comments
published
Referees identified
Public commentary on
eprints
Post-publication public
commentary
Ability to submit
comments
Additional Surveys
(Project Romeo)
• Authors want quick and convenient
dissemination of their work
– Free access to others papers
– Not overly concerned about copyright unless
it stops them from freely distributing their work
or accessing others.
Additional Surveys
(Zhang 1999 survey)
• Permanence and Quality of electronically
archived resources
• Better (faster, more accurate) searching
capabilities
Change the Mindest:
Open Commons
• Open Everything
– Journal Articles
– Datasets
– Scholarly resources (think humanities,
papers,maps,images, films, artwork, etc)
– Reviews/comments/annotations
What are the ramifications?
Who Holds the Materials?
•
•
•
•
Publishers
Universities
Libraries
Large private companies (like Google or
Microsoft Health)
• Government (NLM)
Ramifications?
What does search look like?
• Real time dynamic search
– Text search (Google)
– Faceted search (commercial webpages)
– Clustering of results
Main thing we’re missing is utilizing the
metadata in search algorithms.
How do we measure value of
scholarship?
• What is impact?
• For example, look at how research
councils in countries like Great Britain and
Sweden are trying to code this (formula for
evaluating individuals and departments
academic value, to tie to funding).
• Would this change in “open everything
world”? If so, how?
Think bigger
• We are concerned because of tenure and
promotion of academics in Research I
institutions.
Think bigger
•
We are concerned because of tenure and promotion of academics in research one institutions.
• There are many kinds of universities,
many don’t value original research.
Think bigger
•
We are concerned because of tenure and promotion of academics in research one institutions.
• There are many kinds of universities, many don’t value
original research.
• Scholarship isn’t limited to universities,
certainly not physical ones.
– University of Phoenix
– Virtual colleges
– Interdisciplinary, inter-institution collaborations
Think bigger
•
We are concerned because of tenure and promotion of academics in research one institutions.
•
•
There are many kinds of universities, many don’t value original research.
Scholarship isn’t limited to universities, certainly not physical ones.
–
–
–
University of Phoenix
Virtual colleges
Interdisciplinary, inter-institution collaborations
• Scholarship occurs in many forms
– Funded patrons (Carnegie,now it’s Gates
foundation)
– Free tools for Creation, Storage,
Dissemination can change what universities
are and how teaching occurs (khanacademy,
review, MIT Open courseware)
Think bigger
•
We are concerned because of tenure and promotion of academics in research one institutions.
•
•
There are many kinds of universities, many don’t value original research.
Scholarship isn’t limited to universities, certainly not physical ones.
•
– University of Phoenix
– Virtual colleges
– Interdisciplinary, inter-institution collaborations
Scholarship occurs in many forms
–
–
Funded patrons (now it’s Gates foundation, Carnegie)
Free tools for Creation, Storage, Dissemination can change what universities are and how teaching occurs
(khanacademy, review, MIT Open courseware)
• So, in this future, what scholarly value
metrics will we design to measure what
kinds of value?
(i.e. independent of shackles to the current world like tenure and promotion ).
Extra Material….
UNC Chapel Hill
psycprints
Harvester
(NeoRef)
Virginia Tech
ETD
arXix
University of California
Electronic Repository
Importance of publishers’ roles
Factor
Responses as authors
Responses as readers
Peer review
81
80
Gathering articles
together to enable
browsing of content
64
49
Selection of relevant and
quality-controlled
content
Content editing and
improvement of articles
71
54
60
39
Language or copy editing
50
34
Checking of
citations/adding links
46
28
Marketing (maximising
visibility of journal)
44
20
Can we save the Gold and Grey?
formulate
Idea
V1
discussion
Present to
colleagues
Present at
conference
V2
V3
comments
Author revision
Referees
Revision
for journal
V5
Copyproofing
Two peer reviews
discussion,
revision
Submit to
journal V4
comments
comments
Journal
Final
Revision
Revision to
correct
analysis
V6
V7
Criticisms, new
thoughts,
revision
Revision to
include
additional
new results
V8
new results,
revision
NeoRef Storage Model
Auto-indexing
Material
expressing
content
Revision to include
Author
Journal Final additional results and
Indexing
Revision V6 analyses V8
Comments on
V3
Journal
Submission V4
Comments on
Conference
V6
paper (v3)
Local powerpoint
Presentation v2
Filter (Moderate)
Machine Review
Author
Grey Literature
Top Tier (Keep
Forever)
Two peer reviews
Automated
Content
Item
Recognized Expert
Open (anyone)
Challenges
• Self Contribution
– Author indexed
– Author supplied metadata (Dublin Core)
– Archive file(s) must be in standard open
format
NeoRef: PDF/A with DC elements in tags for
automatic extraction of metadata
Challenges
• Archive Hosting
– Off the shelf computer system with lots of disk
space and public domain archiving application
(DSpace, Eprints).
– Who maintains the material? {Library (MIT
DSpace), Grad School, University (California),
Publisher (PLoS, BioMedCentral), Society
(arXiv)}
– Where are comments and reviews held (after
the fact content items that reference original)?
Challenges
• Make content universally available
– Export OAI items so they can be harvested
– Have public domain quality harvesters that
support quick and simple searching (i.e.
Google for metadata).
http://www.update-software.com/Cochrane/MR000016.pdf
The Association of Learned and Professional
Society Publishers (ALPSP) Survey
Authors and Electronic Publishing
• Scholarly research communication has seen
far-reaching developments in recent years.
• Most journals are now available online as well
as in print, and numerous electronic-only
journals have been launched;
• The Internet opens up new ways for journals
to operate.
• Authors have also become conscious of
alternative ways to communicate their
findings, and much has been written about
what they ought to think.
Importance of future
dissemination channels
Dissemination method
Very important
plus important
categories
Ranking
Traditional print + electronic journal
91
1
Discipline-based electronic reprint archive
78
2
Traditional print journal
77
3
Traditional electronic-only journal
66
4
Institution-based electronic reprint archive
60
5
New forms of electronic-only journal
49
6
Discipline-based electronic preprint
archive
44
7
Institution-based electronic preprint
archive
33
8
Cochrane Methodology
Review
• Despite its widespread use and costs,
little hard evidence exists that peer
review improves the quality of published
biomedical research.
• There had never even been any
consensus on its aims and that it would
be more appropriate to refer to it as
‘competitive review’.
Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”
BMJ 326 (February 1, 2003): 241
http://bmj.com/cgi/reprint/326/7383/241/a.pdf
Cochrane Methodology
Review
• On the basis of the current evidence,
‘the practice of peer review is based on
faith in its effects, rather than on facts,'
state the authors, who call for large,
government funded research
programmes to test the effectiveness of
the [classic peer review] system and
investigate
possible
alternatives.
Caroline
White, “Little Evidence
for Effectiveness
of Scientific Peer Review,”
BMJ 326 (February 1, 2003): 241
http://bmj.com/cgi/reprint/326/7383/241/a.pdf
•
Cochrane Methodology
Review
The use of peer-review is usually
assumed to raise the quality of the endproduct (i.e. the journal or scientific
meeting) and to provide a mechanism
for rational, fair and objective decisionmaking. However, these assumptions
have rarely been tested.
Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager,
Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:
Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
•
Cochrane Methodology
Review
The available research has not clearly
identified or assessed the impact of peerreview on the more important outcomes
(importance, usefulness, relevance, and
quality of published reports)
• … [G]iven the widespread use of peerreview and its importance, it is surprising
that so little is known of its effects
Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager,
Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:
Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
FURTHERMORE …
• 16% said that the referees would no
longer be anonymous
• 27% said that traditional peer review
would be supplemented by postpublication commentary
• 45% expected to see some changes in
the peer-review system within the next
five years
Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15
no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf
Provider Service Change
• What is worth paying for?
– Quality review (Faculty of 1000)
– Proofing, citation linking, professional
presentation (CiteSeer, Cite-base)
– Marketing
– Archival (JStor)
• Who hosts material:
– Society (arXiv)
– Commerical Publishers
(Elesiever,BioMedCentral)