COMMENTS ON
COCHRANE REVIEWS
Approaches to managing feedback
WORKSHOP
22 September 2013
Cochrane Colloquium, Quebec City
Introductions
We are:
• John Hilton
• Gavin Stewart
• Jo Garner
• Toby Lasserson
Editor, Cochrane Editorial Unit
Associate Editor, Wiley
Senior Business Analyst, Wiley
Senior Editor, Cochrane Editorial Unit
Feedback Editor, Airways Group + 19 other roles
Disclosures:
Management of comments is part our jobs as employees of our respective organisations
but otherwise we have no actual or potential conflict of interest in relation to this
presentation.
Who are you?
Introductions
We are:
• John Hilton
• Gavin Stewart
• Jo Garner
• Toby Lasserson
Editor, Cochrane Editorial Unit
Associate Editor, Wiley
Senior Business Analyst, Wiley
Senior Editor, Cochrane Editorial Unit
Feedback Editor, Airways Group + 19 other roles
Disclosures:
Management of comments is part our jobs as employees of our respective organisations
but otherwise we have no actual or potential conflict of interest in relation to this
presentation.
Who are you?
Introductions
We are:
• John Hilton
• Gavin Stewart
• Jo Garner
• Liz Dooley
Editor, Cochrane Editorial Unit
Associate Editor, Wiley
Senior Business Analyst, Wiley
Managing Editor, ARI Group
ME Support + 21 other roles
Disclosures:
Management of comments is part our jobs as employees of our respective organisations
but otherwise we have no actual or potential conflict of interest in relation to this
presentation.
Who are you?
Workshop
• Part 1: What’s the latest?
• Overview, data, guidance, support
• What do you need? What do you need to know?
• Part 2: Case studies
• Some real-life examples
• What would you do? And why? Any barriers, hurdles or brick walls?
• Part 3: The way ahead
• Improving the process and the display
• How can you help?
• Part 4: Is it worth it?
• Isn’t everyone using Twitter these days?
• What do you think?
Submitting comments
Submitting comments
Submitting comments
Submitting comments
Comments are also
submitted via other
channels:
• Direct contact to
CRG
• From OVID
• Direct contact to
Editor in Chief or
CEU
• Summaries website
(not at the moment)
Processing comments
Comment
submission
to Wiley &
CEU
Comment
triage by
Wiley
Comment
assessment
by ME & FE
Comment
management
What does CEU do?
•
•
•
•
•
Comments go to Editor in Chief, and two editors
EiC looking if intervention required (with CRG)
Editors looking for cross-CRG issues (e.g. multiple submissions)
Support
Tracking
Processing comments
Comment
submission
to Wiley &
CEU
Comment
triage by
Wiley
Comment
assessment
by ME & FE
Comment
management
What gets filtered out:
• Comments or enquiries about the website functionality, access,
display, etc
• Comments that are offensive, nonsensical, or inconsequential
• Comments that arise from system errors or system testing
• Duplicates
Processing comments
Comment
submission
to Wiley &
CEU
Comment
triage by
Wiley
 CRG-led process
•
•
•
•
Managing Editor
Feedback Editor
Co-ordinating Editor(s)
Feedback contributor(s)
Comment
assessment
by ME & FE
Comment
management
Processing comments
Comment
submission
to Wiley &
CEU
Comment
triage by
Wiley
 CRG-led process
•
•
•
•
•
•
Managing Editor
Feedback Editor
Co-ordinating Editor(s)
Feedback contributor(s)
Author(s)
Editor(s)
Comment
assessment
by ME & FE
Comment
management
Processing comments
Comment
submission
to Wiley &
CEU
Comment
triage by
Wiley
 CRG-led process
•
•
•
•
•
•
•
•
•
•
Managing Editor
Feedback Editor
Co-ordinating Editor(s)
Feedback contributor(s)
Author(s)
Editor(s)
Editor-in-Chief
CEU Editor(s)
Steering Group, CEO
Interfering third parties
Comment
assessment
by ME & FE
Comment
management
Acting on comments
Questions, questions, questions
•
•
•
•
•
•
•
•
•
•
•
•
•
Is it in English?
Is it coherent and not clearly wrong?
Is it about the content of the review?
Should it be published?
How much editing does it need?
Are the authors going to respond? And when?
If authors won’t or can’t respond, what then?
How long will the whole process take?
Is anyone in editorial base conflicted?
Has contributor declared conflicts?
Does it trigger an amendment?
Does it trigger an update?
Does it raise broader issues?
Acting on comments
Questions, questions, questions
•
•
•
•
•
•
•
•
•
•
•
•
•
Is it in English?
Is it coherent and not clearly wrong?
Is it about the content of the review?
Should it be published?
How much editing does it need?
Are the authors going to respond? And when?
If authors won’t or can’t respond, what then?
How long will the whole process take?
Is anyone in editorial base conflicted?
Has contributor declared conflicts?
Does it trigger an amendment?
Does it trigger an update?
Does it raise broader issues?
What have I missed?
Process
Judgement
call?
Workflow?
Comment outcomes
Validity
1.
1.
2.
Valid: take action
Not valid: take no further action
Usefulness
2.
1.
2.
3.
Translate
Edit
Drastically cut or ask for resubmission
Suitability for publication
3.
1.
2.
Publish
Don’t publish
Author response
4.
1.
2.
3.
Not needed
Needed but not forthcoming
Received and published
Action on review
5.
1.
2.
3.
4.
No action needed
Review amendment
Review update
Changes at next update
How many comments?
45
40
35
30
25
20
15
10
5
0
2011
Q1
2011
Q2
2011
Q3
2011
Q4
2012
Q1
2012
Q2
2012
Q3
2012
Q4
2013
Q1
2013
Q2
Which groups?
• In the first half of 2013…
Which groups?
• In the first half of 2013…
4
Anaesthesia Group
Heart Group
Pregnancy & Childbirth Group
Wounds Group
Which groups?
• In the first half of 2013…
4
6
Anaesthesia Group
Heart Group
Pregnancy & Childbirth Group
Wounds Group
Airways Group
Which groups?
• In the first half of 2013…
4
6
8
Anaesthesia Group
Heart Group
Pregnancy & Childbirth Group
Wounds Group
Airways Group
Acute Respiratory Infections Group
Which groups?
• In the first half of 2013…
4
6
8
Anaesthesia Group
Heart Group
Pregnancy & Childbirth Group
Wounds
Airways Group
Acute Respiratory Infections Group
And 32 other CRGs received at least 1 comment
Characteristics of comments
• Study inclusion/exclusion
• Interpretation of findings
• Conflict of interest
• Scope
• Methods
• Typos, errors
Characteristics of comments
• Study inclusion/exclusion
• Interpretation of findings
• Conflict of interest
• Scope
• Methods
• Typos, errors
One sentence
10 pages
Detailed, well-written referenced letter
Semi-coherent scribble
Focused on single aspect of review
Multiple issues, not all related
to review
Characteristics of comments
• Study inclusion/exclusion
• Interpretation of findings
• Conflict of interest
• Scope
• Methods
• Typos, errors
•
•
•
•
•
•
One sentence
10 pages
Detailed, well-written referenced letter
Semi-coherent scribble
Focused on single aspect of review
Multiple issues, not all related
to review
Retired academics
Cochranites
Students
Clinicians
Pharmaceutical companies or manufacturers
Persistent single-issue enthusiasts
Where do comments come from?
What about the unapproved comments?
Example 1
Review
Speed cameras for the prevention of road traffic injuries and deaths
Contributor
Maller
Comment
When there is accident happened, who will take responsibility? After
several day or long time, how to know the scene when it happened?
Car video recorder could record accident and store video data with gps
information on memory card, people could get accident reconstruction,
it is very easy. more view: http://www.cheapcarcamera.com
What about the unapproved comments?
Example 2
Review
Contributor
willywonka@gmail.com
Comment
What about the unapproved comments?
Example 3
Review
Probiotics for the prevention of Clostridium difficile‐associated diarrhea
in adults and children
Contributor
Rachel Bock
Comment
Test comment - please delete
Known unknowns?
• Who is submitting comments: students, clinicians,
•
•
•
•
Cochranites, pharma companies? (and why?)
How many comments are published, and how long after
submission?
How many reviews amended or updated as a result of
feedback?
What aspects are commentors focusing on? (study
inclusion, interpretation, outcomes, etc?)
Any more? What would it be useful to know?
• ANY VOLUNTEERS?
Cochrane Editorial & Publishing Policy Resource
http://www.cochrane.org/editorial-and-publishing-policy-resource
Cochrane Editorial & Publishing Policy Resource
http://www.cochrane.org/editorial-and-publishing-policy-resource
Cochrane Editorial & Publishing Policy Resource
http://www.cochrane.org/editorial-and-publishing-policy-resource
Cochrane Organisational Policy Manual
http://www.cochrane.org/organisational-policy-manual
Cochrane Handbook
http://handbook.cochrane.org
Cochrane Editorial Unit website
http://www.editorial-unit.cochrane.org/cochrane-library-feedback
So what’s in those new guidelines?
• Guidance on how feedback should be managed
• When to publish, when not to publish
• Guidance on reasonable timescales
• Roles of ME and FE
• Meeting expectations of contributors
• Where to get support
• What to do with non-English comments
Your feedback will be very welcome.
Feedback editors: neglected?
• Communication to and between FEs has been lacking
• Need Role description (purpose, expectations)
• Induction package?
• Page on Cochrane Training page?
• Independence of role
• Cochrane.org feature
• Forum
So what else do you/we need?
Case studies
• 1.
• 2.
• 3.
• 4.
What’s in that CEU proposal?
• Use the term ‘comments’
• Decouple comments from review, but link to version
• Display comments parallel to review version
• Make comments citable (DOI)
• Expectations about reasonable turnaround time
• Transparency and tracking
• Archie workflow for comments
So that was a while ago. What happened?....
Cochrane-Wiley Roadmap
•
•
•
•
A set of work streams fulfilling agreed contractual
projects
Next 18 months
Feedback scheduled for Q2 2014
Project will involve:
• CEU, Wiley, Cochrane Tech team
• CRGs
• You?
Goals for feedback project
• Make process more efficient and transparent
• Phased approach to first improve the user experience for:
• the Contributor
• the Reader of feedback
• De-couple publishing feedback from the review or protocol
• Make the individual comment citable in its own right
Requirements for user feedback
1.
2.
3.
4.
5.
6.
7.
A Contributor is able to submit a comment
The system shall save and acknowledge the comment
Wiley Editorial may moderate the comment
The system shall trigger an editorial workflow that may result
in publication of the comment along with a response
A Reader will be able to view comments and responses
A Reader will be able to cite the comment
A Search User will be able to search comments and
responses
Proposed approach – submission
Proposed approach – submission
Proposed approach – display
Improve processing
• A system for moderation
• Separate the publishing of feedback from the review, retain a
link between them:
• Feedback as a new Content type?
• Changes to Archie, through XDPS and to Online Library
• Add the ability to cite individual comments & author responses
• Improve the robustness of the feedback-response loop
•
Feedback items having their own workflow?
Why all the bother?
• So we can “…amend reviews in the light of new
evidence…to reflect the emergence of new data, valid
feedback, solicited or unsolicited, from whatever source”
• Chalmers I, Haynes B. Reporting, updating, and correcting systematic reviews of the
effects of health care. BMJ 1994;209:862-5.
But where are all the comments?
Not just Cochrane
http://jasonpriem.org/2011/01/has-journal-article-commenting-failed/
It can work…
Even then, will authors engage?
Peter Gøtzsche, Andreas Lundh and BMJ editors looked at BMJ
rapid responses
Even when comments could “invalidate research or reduce
reliability” over half the time authors couldn’t be bothered to
respond.
http://www.bmj.com/content/341/bmj.c3926
Are they using the ‘shadow system’ - scribbled marginalia, chats
in labs, peer review, discussions at conferences, journal clubs
And Twitter, of course.
So, why no comments?
“Why is it that comment forums provided by online scholarly
journals for post-publication peer review are so consistently
underutilized, even though online forums concerning countless
subjects, from culture to politics, are booming?”
• Technophobia? Time?
• Worry about appearing to be ‘unacademic’
• “Commenting on scientific articles does not advance my
career”
• Prefer a published letter (despite delays)
• Disincentives: fear of upsetting, being wrong, giving away ideas
Is the question not whether have comments failed, but
whether are they succeeding somewhere else?
What’s different about Cochrane?
• Updating! Comments are part of a continuous process of
improvement and updating
• Culture of acceptance of criticism
• Expectations of authors
So what could we or you do?
• Proactive solicitation
• More encouragement on website – expose comments,
prompt for comments
• Altmetrics
• Activate specific social networks and establish rapport