SYSTEMATIC REVIEWS: A PRIMER
FOR LIBRARIANS
Mark MacEachern, MLIS
Taubman Health Sciences Library, University of Michigan
Outline








Introductions
What is a Systematic Review?
Librarian Roles
Protocols & Registries
Databases & Search Strategies
Data Management
Reporting
Wrap Up & Other Stuff
Introduction



Name
Experience with systematic reviews
What do you want to get from this?
Introduction







Name
Experience with systematic reviews
What do you want to get from this?
Ask questions throughout!
I have no conflicts of interest to report
Some overlap with my MLA webinars and the UMich workshop
Discussion-based
What is a systematic review?
What is a systematic review?

From the Cochrane Handbook (1.2.2):
A systematic review attempts to collate all empirical evidence that
fits pre-specified eligibility criteria in order to answer a specific
research question. It uses explicit, systematic methods that are
selected with a view to minimizing bias, thus providing more reliable
findings from which conclusions can be drawn and decisions made
(Antman 1992, Oxman 1993).
What is a systematic review?

Common Elements:





A clearly stated set of objectives with pre-defined eligibility
criteria for studies
A reproducible methodology
A systematic search that attempts to identify all studies that
would meet the eligibility criteria
An assessment of the validity of the findings of the included
studies, for example through the assessment of risk of bias
A systematic presentation, and synthesis, of the characteristics
and findings of the included studies
Source: Cochrane Handbook, 1.2.2
What is a systematic review?
Source: Users’ Guides to the Medical Literature (JAMA Evidence)
What is a systematic review?

In short, it is likely a narrative review if:


The question is unclear from the title, abstract, or introduction
There is no methods section
Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.
What is a systematic review?

In short, it is likely a narrative review if:



The question is unclear from the title, abstract, or introduction
There is no methods section
Types of Systematic Reviews

Qualitative


Results summarized narratively, no stats, methods rigorous to minimize
bias
Quantitative

Results statistically combined, methods rigorous to minimize bias
Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.
What is a systematic review?
Steps
1.
Formulate a question
2.
Define inclusion/exclusion criteria
3.
Locate studies
4.
Select studies
5.
Assess study quality
6.
Extract data
7.
Analyze and present results
8.
Interpret results
Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.
What is a systematic review?
Source: Sackett DL. Evidence-based medicine: how to practice and teach EBM.
What is a systematic review?

Team Composition

One person


Two people


Possible, but more prone to bias
May want a third to help with disagreements
Three plus


Clinician(s), statistician, librarian
Too many can become problematic
Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.
What is a systematic review?

Team Composition
Source: Institute of Medicine. Standards for Initiating a Systematic Review.
What is a systematic review?

Rapid Reviews
Source: Khangura S. Evidence summaries: the evolution of a rapid review approach.
What is a systematic review?

Rapid Reviews
Methodological Approaches
Search fewer databases
Limited use of grey literature
Restrict types of studies included (eg., English only, most recent
5 yrs)
Relying on existing reviews
Limited full-text review
Limited dual review for study selection
See report for more
Source: EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews
What is a systematic review?
What is a systematic review?
What is a systematic review?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR work.
A researcher comes to you set on doing a systematic review for a
specific question. You don’t believe that the question lends itself to a
systematic review. How do you proceed?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR work.
A resident reaches out to you for help conducting a systematic
review search. She tells you that the project is done, the manuscript is
finished, and all she needs is a search that captures every study
included in her paper. Is this a problem? How do you proceed here?
Librarian Roles

IOM Standards for Initiating a Systematic Review



Standard 2.1 Establish a team with appropriate expertise and
experience to conduct the systematic review
 2.1.3 Include expertise in searching for relevant evidence
Standard 2.5 Formulate the topic for the systematic review
 2.5.1 Confirm the need for a new review
Standard 2.6 Develop a systematic review protocol
 2.6.4 Describe the search strategy for identifying relevant evidence
Source: Institute of Medicine. Standards for Initiating a Systematic Review
Librarian Roles

IOM Standards for Finding and Assessing Individual Studies

Standard 3.1 Conduct a comprehensive systematic search for evidence




3.1.1 Work with a librarian or other information specialist trained in
performing systematic reviews to plan the search strategy
3.1.2 Design the search strategy to address each key research question
3.1.3 Use an independent librarian or other information specialist to peer
review the search strategy
3.1.4 – 3.1.9 Search various databases and update searches
Source: Institute of Medicine. Standards for Initiating a Systematic Review.
Librarian Roles

IOM Standards for Finding and Assessing Individual Studies

Standard 3.2 Take action to address potentially biased reporting of
research results



3.2.1 Search grey literature databases, clinical trial registries, and other
sources of unpublished information about studies
3.2.4 – 3.2.6 Handsearch, web search, and non-English search
Standard 3.4 Document the search

3.4.1 Provide a line-by-line description of the search strategy
Source: Institute of Medicine. Standards for Initiating a Systematic Review.
Librarian Roles
Source: Institute of Medicine. Standards for Systematic Reviews (full report)
Common Errors in Published SR Search Strategies
Spelling errors
Misuse of MeSH and keywords
No spelling variants / truncations
Bad exploding
Incorrect use of Boolean and line
numbers
Redundancy in search terms
Translating strategies poorly
Source: Sampson, M. Errors in search strategies were identified by type and frequency.
Librarian Roles

Authorship
Librarian Roles

Authorship – ICMJE Authorship Criteria
Why Authorship Matters?
1.

Credit & responsibility for contributions
Who Is an Author?
2.




Substantial contributions to all aspects of article
Drafting and revising manuscript
Final approval of manuscript
Agreement to be accountable for all aspects of the work
Non Author Contributions
3.

If you don’t meet those 4 criteria, you should be listed in the
acknowledgements
Source: ICMJE. Defining the Role of Authors and Contributors
Librarian Roles

Authorship – ICMJE Authorship Criteria
Why Authorship Matters?
1.

Credit & responsibility for contributions
Who Is an Author?
2.




Substantial contributions to all aspects of article
Drafting and revising manuscript
Final approval of manuscript
Agreement to be accountable for all aspects of the work
Non Author Contributions
3.

If you don’t meet those 4 criteria, you should be listed in the
acknowledgements
Source: ICMJE. Defining the Role of Authors and Contributors
Librarian Roles

Authorship – ICMJE Authorship Criteria
Why Authorship Matters?
1.

Credit & responsibility for contributions
Who Is an Author?
2.




Substantial contributions to all aspects of article
Drafting and revising manuscript
Final approval of manuscript
Agreement to be accountable for all aspects of the work
Non Author Contributions
3.

If you don’t meet those 4 criteria, you should be listed in the
acknowledgements
Source: ICMJE. Defining the Role of Authors and Contributors
Librarian Roles

Acknowledgement (named)
Librarian Roles

Acknowledgement (unnamed)
Librarian Roles

Grants



Co-PI, Collaborator, Consultant
% effort, compensation
Good source for demonstrating value
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR work.
Is authorship on SRs appropriate? If so, in what instances? If not, why
not?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR work.
You have been asked to participate in a systematic review project
and would like to receive appropriate recognition for your
contributions. How would you broach this subject with the project
lead?
Protocols & Registries
A protocol is a plan or set of steps to be followed in a study. A
protocol for a systematic review should describe the rationale for
the review; the objectives; and the methods that will be used to
locate, select and critically appraise studies, and to collect and
analyse data from the included studies.
Source: Cochrane Community, http://community.cochrane.org/faq/general#t86n544
Courtesy: W Townsend
Protocols & Registries
Source: Institute of Medicine. Standards for Initiating a Systematic Review.
Protocols & Registries

Prevents post-hoc changes, but acknowledges that changes are
permissible.


Don’t feel like it sets everything in stone.
Prevents duplication of effort
Source: Liberati A. The PRISMA Statement…Explanation and Elaboration.
Protocols & Registries
Educate
• Ensure consults know what goes into an SR. Filters out those
who do not.
• Proper SR methodologies
• Realistic timelines, realistic questions
• Scoping review first
• Technology
• Reference IOM, Cochrane, PRISMA
Protocols & Registries
Educate
• Ensure people know what goes into an SR. Filters out those
who don’t.
• Proper SR methodologies
• Realistic timelines, realistic questions
• Scoping review first
• Technology
• Reference IOM, Cochrane, PRISMA
Inform
• Your search
• Helps elicit important info from project lead
• PICO
• Eligibility criteria
• Possible limits
• Appropriate databases
Protocols & Registries
Educate
• Ensure people know what goes into an SR. Filters out those
who don’t.
• Proper SR methodologies
• Realistic timelines, realistic questions
• Scoping review first
• Technology
• Reference IOM, Cochrane, PRISMA
Inform
• Your search
• Helps elicit important info from project lead
• PICO
• Eligibility criteria
• Possible limits
• Appropriate databases
Distribute
• Makes the project team accountable
• Facilitates team functioning
• Helps with the writing
Protocols & Registries
Source: Rethlefsen ML. I Want to do a Systematic Review.
Protocols & Registries
Source: Rethlefsen ML. Librarian co-authors correlated with higher quality reported...
Protocols & Registries

Realistic timelines
In 2008: 15 months
(Sampson M, 2008)
In 2013: 8 months
(Beller EM, 2013 )
Source: Cochrane Handbook, 2.3.b
Protocols & Registries

Why register?





Provides transparency
Reduce risk of multiple reviews on the same question
Reduce publication bias
Safeguard against reporting biases
Identify similar reviews in the process of being completed
Source: Liberati A. The PRISMA Statement…Explanation and Elaboration.
Protocols & Registries
Source: Sun G. Systematic review of quality improvement initiatives in the
intensive care setting
Protocols & Registries
Source: Sun G. Systematic review of quality improvement initiatives in the
intensive care setting
Protocols & Registries
Source: Sun G. Systematic review of quality improvement initiatives in the
intensive care setting
Protocols & Registries
Review title and timescale
Review title*
Review methods
Review question(s)*
General information
Type of review
Original language title
Anticipated or actual start date*
Anticipated completion date*
Searches*
URL to search strategy
Condition or domain being studied*
Language
Country
Other registration details
Stage of review at time of registration* Participants/ population*
Intervention(s), exposure(s)*
Review team details
Comparator(s)/ control*
Named contact*
Named contact email*
Named contact address
Named contact phone number
Review team members and their
organisational affiliations
Organisational affiliation of the review*
Funding sources/ sponsors*
Conflicts of interest*
Collaborators
Reference and/or URL for published
protocol
Dissemination plans
Keywords
Types of study to be included initially*
Details of any existing review of the
Context
same topic by the same authors
Primary outcome(s)*
Review status*
Secondary outcomes*
Any other information
Data extraction (selection and coding) Link to publication of final report
Risk of bias (quality) assessment*
Strategy for data synthesis*
Analysis of subgroups or subsets*
Source: PROSPERO, http://www.crd.york.ac.uk/PROSPERO/
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR work.
You and project lead have a discussion about protocols and are
trying to decide on whether to have one or not. She says that most
don’t have one, so why should we bother? This gets you thinking
about whether they’re worth the effort? Do you care whether the
project has one or not?
Databases & Search Strategies

Why is the search important?





A high-quality lit search is essential for a successful meta-analysis
It is from the search results that data is gathered for analysis
Failure to locate important studies can significantly affect results
Remember the goal is to capture every relevant study
Important to report search so your methodology can be
reproduced
Databases & Search Strategies

Expect:


1000s of results
To search multiple databases



Ovid MEDLINE, EMBASE, PubMed, Scopus, etc.
The search process to take awhile
To publish search strategy and search methodology
Databases & Search Strategies
Core
Secondary
Others
• MEDLINE
• PubMed
• Ovid MEDLINE
• EMBASE
• CENTRAL
•
•
•
•
•
• PsycINFO
• ERIC
• Sociological
Abstracts
• ABI Inform
• Proquest Theses and
Dissertations
• Etc.
Scopus
Web of Science
CINAHL
Google Scholar
ClinicalTrials.gov
• Cochrane Handbook (10.3.1)
– Search multiple sources
– Search unpublished studies
– Search trial registries
Databases & Search Strategies

Supplemental strategies


Examine the references of included studies and relevant reviews
Use ‘cited by’ features



Contact authors, companies, orgs, societies, etc.
Hand search important journals


Web of Science, Google Scholar
By consensus, Impact Factor, most frequently appearing journals in
Endnote/etc
Search for ongoing studies

Clinicaltrials.gov ; controlled-trials.com (ISRCTN); conference
abstracts
Databases & Search Strategies

Grey Literature


“It is usually understood to mean literature that is not formally
published in sources such as books or journal articles.” (Cochrane
Handbook, 6.2.1.8)
Types





Conference abstracts (Web of Science, Scopus)
Clinical trials (ClinicalTrials.gov)
Government reports, documents (.gov, CABI)
Dissertations (ProQuest Dissertations and Theses)
Unpublished manuscripts
Databases & Search Strategies

Grey Literature
Pros
Cons
• Combats publication bias
• Process can help find missed studies
• Strengthens methodology
• Search functionality of grey lit sources
tends to be basic
• Difficult sell value to team
• Increases retrieval (already large)
• Adds time to project (contact authors,
etc.)
• Difficult to export results
Databases & Search Strategies

Grey Literature
Pros
Cons
• Combats publication bias
• Process can help find missed studies
• Strengthens methodology
• Search functionality of grey lit sources
tends to be basic
• Difficult sell value to team
• Increases retrieval (already large)
• Adds time to project (contact authors,
etc.)
• Difficult to export results






Cook DJ, 1993 – Unpublished studies should not be systematically
excluded

Hopewell S, 2002 – Useful for RCTs. Combine with MEDLINE

Egger M, 2003 – Difference b/w comprehensive searches and less
comprehensive is small

Martin JL, 2005 – Study quality is important and grey lit might not
meet the standard

Armstrong R, 2005 – Handsearching is valuable to pick up
supplements

Benzies KM, 2006 – Grey literature is useful

Hopewell S, 2007 – Handsearching minimizes bias, but is costly and
requires much effort
Hopewell S, 2009 – Publication bias is a thing; consider methods to
minimize it
Craane B, 2012 – Handsearching contributes considerably
Mahmood Q, 2014 – Searching for grey lit has benefits and
challenges
Saleh AA, 2014 – Average time is 7 hours of librarian time to search
grey lit sources
Cochrane Handbook, 10.3.2 – Publication bias is a major threat.
Including unpublished trials appears to help
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR teams.
The project lead does not want to search the grey literature. Is grey
literature worth the effort? Is it something you would insist on doing?
Databases & Search Strategies

Harvesting Search Terms



PICO
Protocols
Sentinel and other relevant articles





Controlled terms
Read titles and abstracts
Find others via (***.ti AND ***.ti)
Entry terms
Use adjacency to cover variations

E.g. (breast* adj5 (cancer* or carcinom* or neoplas* or tumor* or
tumour* or malignan*)).tw
Databases & Search Strategies

Filters



A predefined search designed to target specific study
methodologies (RCTs, Cohort, Systematic Reviews)
Consider using a validated filter
E.g.: PubMed Clinical Queries – Specific, Therapy
Databases & Search Strategies

Filters
Databases & Search Strategies

Filters, Sensitivity & Specificity
Source: McKibbon, 2009
Databases & Search Strategies

Topic/Subject Filters
Databases & Search Strategies

Eligibility criteria can inform filters
Be careful applying them

RCT publication type vs Title searches

Databases & Search Strategies

Eligibility criteria can inform filters
Be careful applying them

Humans limit?

Databases & Search Strategies

Appraising filters
Source: Glanville, 2008
Databases & Search Strategies

Sources of validated filters

PubMed Clinical Queries


PubMed Special Queries



http://www.sign.ac.uk/methodology/filters.html
InterTASC


http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx
SIGN


http://www.nlm.nih.gov/bsd/special_queries.html
HIRU (McMaster University)


http://www.ncbi.nlm.nih.gov/books/NBK3827/#pubmedhelp.Clinical_Queries_Filters
http://www.york.ac.uk/inst/crd/intertasc/
Cochrane Groups
Individual studies

http://www.ncbi.nlm.nih.gov/pubmed/19712211 (eg)
Databases & Search Strategies

Supplemental strategies

Cited Reference Searching
Databases & Search Strategies

Supplemental strategies

Handsearching journals (see Cochrane 6.2.2.1)



Those selected by team
Impact Factor
Most frequently appearing journals in Endnote/Excel
Databases & Search Strategies

Supplemental strategies

Re-run the search near the end of the project
Databases & Search Strategies

Validate searches


Important to validate your search in each database
Make sure search captures all sentinel articles


If search doesn’t, why?
How:
Databases & Search Strategies

Cochrane recommendations
Source: Cochrane Editorial Unit, Methodological Expectations of Cochrane Intervention Reviews
Databases & Search Strategies

Cochrane recommendations
Source: Cochrane Editorial Unit, Methodological Expectations of Cochrane Intervention Reviews
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR teams.
The team tells you directly that they are only interested in searching
one database. What are the problems with this approach and how
would you discuss these problems with the project lead/team?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR teams.
I’m getting way too many results and I’m thinking of using NOT to
exclude editorials, letters, adults, and animals. What are possible
limitations to applying so many filters? Are there other approaches?
Other considerations?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenario and identify solutions
to this problem that commonly arises in SR work.
Let’s say you limited your search to RCTs, English language studies,
humans, and to the date range 2001-2015. How do you apply such
restrictions to databases with few filtering options (e.g., Scopus,
Web of Science, PsycINFO, ClinicalTrials.gov, etc.)?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenario and identify solutions
to this problem that commonly arises in SR work.
I’ve been working on a search that aims to identify trends within the
cancer literature. It’s not a systematic review, but requires a
significant search. I try applying PubMed’s Cancer Subset search,
but find it to be much too inclusive. Is it okay to create my own? Is it
okay to use an unvalidated filter?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR teams.
As you are building out your keyword searches, you are considering
whether to rely on title or title/abstract searches. Or, .mp in Ovid
MEDLINE? What do you factor in when making a decision?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR teams.
The project lead says that he doesn’t want to sift through more than
200 papers. What are the issues and how would you handle this?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR teams.
You’ve completed the searches and suggested to the team that they
check the reference lists and do a Times Cited check of their
included studies. As the team does this they notice that one of the
papers has been cited over 500 times. They don’t want to consider
these new papers. How do you handle this situation?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR teams.
The project lead is most familiar with PubMed and would like you to
run the search in that database. You feel Ovid MEDLINE is more
appropriate. What are some of the differences between the two
resources? How would you discuss these issues with the project lead?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR teams.
You’ve been focused on a specific systematic review search for a
long time and worry that there are minor mistakes. Would you
consider recruiting a peer-reviewer? What if you can’t find one? If
you were the peer-reviewer, how much time would you anticipate to
spend reviewing someone else’s search?
Data Management

An SR is an extensive search with greater emphasis on data mgmt

Programs


Endnote, Excel, DistillerSR, Abstrakr, other tools
Uses







Keeping track of citations from multiple databases
Eliminating duplicates
Sharing citations with team
Storing pdfs
Eligibility screening
Exporting to word, excel, and other formats
Creating in-text citations and bibliographies
Data Management

Important to document exact search terms
Source: Featherstone, R
Data Management
• What to note:
• Databases searched
• Dates searches were run
• All searches
• Result numbers (total & unique)
• Sentinel articles used for testing
• Decide on naming convention
Data Management

What to note:

All of this, plus the searches
Data Management


Data management strategy informs screening process
Cochrane recommends title/abstract review, then full-text
Source: Cochrane Handbook, 7.2.3
Data Management

Additional screening recommendations
Source: Institute of Medicine, Standards for Initiating a Systematic Review
Data Management

Endnote
Data Management

Excel
Data Management

Abstrackr (free)
Data Management

Abstrackr (free)
Data Management

EPPI Reviewer 4 (free trial)
Data Management

EPPI Reviewer 4 (free trial)
Data Management
Source: HLWiki, Software for systematic reviewing
Data Management
Data Management

Include rationale for exclusions

Per PRISMA (item 17)
RAPID FIRE SCENARIOS
In groups of 3, consider the scenarios and identify solutions to the
problems/issues that commonly arise in SR work.
The project lead and/or the team does not want to learn Endnote
for this project. What are possible solutions?
RAPID FIRE SCENARIOS
In groups of 3, consider the scenarios and identify solutions to these
problems/issues that commonly arise in SR work.
The screening team has identified 84 papers for full-text review and
has asked for help obtaining the pdfs. How would you handle this?
RAPID FIRE SCENARIOS
In groups of 3, consider the scenarios and identify solutions to these
problems/issues that commonly arise in SR work.
You’re working on an SR that includes members from multiple
institutions. How do you manage the citations with a team comprised
of folks who may or may not have the same access to resources?
Reporting

Standards



PRISMA
Cochrane Handbook
IOM Standards for Initiating a Systematic Review
Reporting

PRISMA: aims to ensure the transparent and complete reporting
Source: PRISMA, http://www.prisma-statement.org/statement.htm
Reporting

PRISMA: Item 7 – Information sources
Excerpts from the Explanation:
• Retrieval from databases may be
imperfect, even if done by
experienced searchers
• List database, platform, or
provider
• Include start and end dates
• Authors should report who
developed and conducted the
search
• Report supplementary approaches,
incl hand searching, reference lists,
trial registries, contacting authors
Source: Liberati, 2009
Reporting

PRISMA: Item 8 – Search
Excerpts from the Explanation:
• The search strategy is an essential
component of any SR
• Note if the search is peer-reviewed
• It’s not always feasible to include
the full strategy in the paper.
Include it as an appendix, online
supplement, etc.
• Archive searches
• Note limits and other limitations
• Database inaccessibility
• Financial / time restrictions
• No searching expertise
Source: Liberati, 2009
Reporting

PRISMA: Flow Diagram
Excerpts from the Explanation:
• Report total number of records
identified from all sources
• Include counts for each source, b/c
literature identified primarily from
references and experts can be
biased
• Note:
• Duplicates
• Unique records
• Levels of screening
• Layout of diagram may vary
Source: Liberati, 2009
Reporting

Flow diagram
template
Source: PRISMA, http://www.prisma-statement.org/statement.htm
Reporting

See Cochrane Handbook


http://www.cochrane-handbook.org/
 Section 6.6
“It should be borne in mind at the outset that the full search
strategies for each database will need to be included in an
Appendix of the review.”
Reporting

See Cochrane Handbook


http://www.cochrane-handbook.org/
 Section 6.6
In study flow diagram:

number of unique records identified by the searches;

number of records excluded after preliminary screening (e.g. of titles and
abstracts); and

number of records retrieved in full text
Reporting

See Cochrane Handbook


http://www.cochrane-handbook.org/
 Section 6.6
In abstract:

List all databases searched.

Note the dates of the last search for each database or the period searched.

Note any language or publication status restrictions (but refer to Section 6.4.9).

List individuals or organizations contacted.”
Reporting

See Cochrane Handbook


http://www.cochrane-handbook.org/
 Section 6.6
In methods:

List all databases searched.

Note the dates of the last search for each database AND the period searched.

Note any language or publication status restrictions

List grey literature sources.

List individuals or organizations contacted.

List any journals and conference proceedings specifically handsearched for the
review.

List any other sources searched (e.g. reference lists, the internet).
Reporting

See IOM Standards


http://iom.nationalacademies.org/Reports/2011/Finding-What-Worksin-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx
In methods:

Databases and other information sources used to identify relevant studies

Search strategy
Reporting

Example
Source: Sun GH, 2012
Reporting

Example
Source: Sun GH, 2012
Reporting

Example
Source: Sun GH, 2012
Reporting

Example
Source: Sun GH, 2012
Reporting

Example
Source: Sun GH, 2012
Reporting

Example
Source: Sun GH, 2012
Reporting

Example 1
Source: Donahue, 2007
Reporting

Example 2
Source: Everett, 2005
Reporting

Example 3
Source: Nuti, 2014
Reporting

Example 3
Source: Nuti, 2014
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR work.
A researcher meets with you to discuss the search. You realize that
she is pursuing the project individually. What are potential problems
with this? How do you advise her?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR work.
You were brought in to a project to run the search. You completed
this in Feb ‘14 and sent the project lead the results. In Dec ’14, you
are given a manuscript, which has you listed as an author. You notice
that the searches are not entirely yours and that you don’t like what
they’ve done methodologically. How would you handle this situation?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions
to these problems/issues that commonly arise in SR work.
Same scenario, but you notice that you made an error in the
searches. What would you do?
Wrap Up & Other Stuff
• Eligibility screening, from PRISMA Explanations document
Source: Liberati, 2009
Wrap Up & Other Stuff

Data Extraction forms
Source: Cochrane Review Group on HIV/AIDS
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
The publication or non-publication of research findings
Time lag bias
The rapid or delayed publication of research findings
Multiple publication bias
The multiple or singular publication of research findings
Location bias
The publication of research findings in journals with
different ease of access or levels of indexing in standard
databases
Citation bias
The citation or non-citation of research findings
Language bias
The publication of research findings in a particular
language
Outcome reporting bias
The selective reporting of some outcomes but not others
Source: Cochrane Handbook, 10
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Positive results are over-represented in published
literature. They are published and cited more
frequently and published more quickly.
Negative results are less likely to be published
and cited
What doctors don't know about the drugs they
prescribe (Goldacre, TED Talk)
Outcome reporting bias
Source: Cochrane Handbook, 10
Wrap Up & Other Stuff

No publication bias
Wrap Up & Other Stuff

Possible publication bias
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
Industry sponsored trials report “more favourable”
outcomes when published (Ross 2009), but more
industry-sponsored trials are left unpublished (Lundh
2012)
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
Multiple publication bias
Location bias
“Trials with positive results were published in
approximately 4 to 5 years. Trials with null or
negative results were published after about 6 to 8
years”
Our review shows that trials with positive results are
published sooner than other trials.”
Citation bias
Language bias
Outcome reporting bias
Source: Hopewell, S.
Hopewell S et al. Time to publication of results to
clinical trials. Cochrane Review. PMID: 17443632
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
Multiple publication bias
Location bias
“It is not always obvious that multiple publications
come
from a single study, and one set of study
participants may be included in an analysis twice.
The inclusion of duplicated data may therefore lead
to overestimation of intervention effects.”
Cochrane Handbook 10.2.2.1
Citation bias
Language bias
Outcome reporting bias
Source: Cochrane Handbook, 10.2.2.1
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
Significance of results and trial quality associated
w/ journal impact.
Multiple publication bias
Location bias
Citation bias
Language bias
“Choices regarding which databases to search may
bias the effect estimate in a meta-analysis.”
Cochrane Handbook 10.2.2.2
• Strategy
• Search multiple databases
• Search subject-specific resources
• No access? Partner w/ other librarians
Outcome reporting bias
Source: Cochrane Handbook, 10.2.2.2
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Bramer WM, et al. (2013) The comparative recall of
Google Scholar versus PubMed in identical searches for
biomedical systematic reviews: a review of searches used
in systematic reviews. Syst Rev. PMCID: PMC3882110.
Giustini D, et al. (2013) Google Scholar is not enough to
be used alone for systematic reviews. Online J Public
Health Inform. PMC3733758.
Sampson M, et al. (2003) Should meta-analysts search
Embase in addition to Medline? J Clin Epidemiol. PMID:
14568625.
Outcome reporting bias
Source: Cochrane Handbook, 10.2.2.2
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
“Perusal of reference lists….is far from objective.”
Cochrane Handbook 10.2.2.3
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
• Strategy
• Peruse reference lists (final safety net)
Source: Cochrane Handbook, 10.2.2.3
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
“Reviews have often been exclusively based on
studies published in English…. While the potential
impact of [excluding non-English studies] may be
minimal, it is difficult to predict in which cases this
exclusion may bias a systematic review.”
Cochrane Handbook 10.2.2.4
• Strategy
• Is it always feasible to include non-English
studies?
• Case-by-case basis
Outcome reporting bias
Source: Cochrane Handbook, 10.2.2.4
Wrap Up & Other Stuff

Limitations
Types of Reporting Bias
Definition
Publication bias
Time lag bias
Multiple publication bias
Location bias
“The choice of outcomes that are reported can be
influenced by the results, potentially making
published results misleading.”
Cochrane Handbook 10.2.2.5
Outcomes described in protocols are inconsistent
with those reported in publications (Dwan K, 2013)
Citation bias
Language bias
• Strategy
• Develop a protocol. Not infallible
Outcome reporting bias
Source: Cochrane Handbook, 10.2.2.5
Wrap Up & Other Stuff

Summary
Search Strategy
Considerations
Search multiple databases
Cochrane Handbook suggests MEDLINE, Embase, and Cochrane Register
Include subject specific databases
Fewer limits
MeSH terms
Screen for eligibility manually instead
Language bias
English?
Include grey literature sources
Trials registries / Conference proceedings / Protocols / Drug registries
and other info / Contacting authors
AHRQ on Finding Grey Literature (Balshem, 2013)
Cochrane Review on its value (Hopewell & MacDonald, 2007)
Librarian Involvement
Librarian co-authors correlated higher quality search strategy reporting.
(Rethlefsen 2015)
See also: Rethlefsen M. (2014). Engaging medical librarians to improve
the quality of review articles. JAMA. PMID 25203078
Useful Resources

Expert Searching Listserv


Cochrane Handbook


http://handbook.cochrane.org/
PRISMA (remember the elaboration article)


http://pss.mlanet.org/mailman/listinfo/expertsearching_pss.mlanet.org
http://www.prisma-statement.org/
Systematic Reviews: Opportunities for Librarians

http://www.lib.umich.edu/systematic-reviews-workshop
Acknowledgments



My colleagues involved in the Systematic Reviews: Opportunities
for Librarians workshop. (Nancy Allee, Rafael Becerra, Jane
Blumenthal, Marisa Conte, Emily Ginier, Nandita Mani, Chase
Masters, Anne Perorazio, Judy Smith, Whitney Townsend)
Joel Gagnier, Epidemiologist, University of Michigan,
MLA, for the Beyond the Search webinar series
References







Abstrakr, http://abstrackr.cebm.brown.edu/
Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. (1992). A comparison of results of meta-analyses of randomized
control trials and recommendations of clinical experts: Treatments for myocardial infarction. JAMA, 268. 240-248. PMID
1535110.
Armstrong R, Jackson N, Doyle J, Waters E, Howes F. (2005). It's in your hands: the value of handsearching in conducting
systematic reviews of public health interventions. J Public Health, 27(4).388-91. PMID 16311247.
Balshem H, Stevens A, Ansari M, Norris S, Kansagara D, Shamliyan T, Chou R, Chung M, Moher D, Dickersin K. (2013). Finding
Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions:
AHRQ and the Effective Health Care Program. PMID 24404628.
Beller EM, Chen JK, Wang UL, Glasziou PP. (2013). Are systematic reviews up-to-date at the time of publication? Syst Rev,
282. 36. PMID 23714302.
Benzies KM, Premji S, Hayden KA, Serrett K. (2006). State-of-the-evidence reviews: advantages and challenges of including
grey literature. Worldviews Evid Based Nurs, 3(2). 55-61. PMID 17040510.
Bramer WM, Giustini D, Kramer BM, Anderson P. (2013). The comparative recall of Google Scholar versus PubMed in identical
searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev, 23 (2). 115. PMCID
PMC3882110.

Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions. http://handbook.cochrane.org/

Cochrane Community. Glossary. http://community.cochrane.org/glossary



Cochrane Editorial Unit. Cochrane Editorial Unit, Methodological Expectations of Cochrane Intervention Reviews.
http://editorial-unit.cochrane.org/mecir
Cochrane Review Group on HIV/AIDS. Data Extraction and Assessment Form.
http://hiv.cochrane.org/sites/hiv.cochrane.org/files/uploads/Data%20extraction%20form_RCTs.docx
Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. (1993). Should unpublished data be
included in meta-analyses? Current convictions and controversies. JAMA, 269 (21). 2749–2753. PMID 8492400.
References










Craane B, Dijkstra PU, Stappaerts K, De Laat A. (2012). Methodological quality of a systematic review on physical therapy
for temporomandibular disorders: influence of hand search and quality scales. Clin Oral Investig, 16 (1). 295-303. PMID
21128088.
Donahue KE, Gartlehner G, Jonas DE, Lux LJ, Thieda P, Jonas BL, Hansen RA, Morgan LC, Lohr KN. (2008). Systematic review:
comparative effectiveness and harms of disease-modifying medications for rheumatoid arthritis. Ann Intern Med, 148 (2).
124-34. PMID 18025440.
Dwan K, Gamble C, Williamson PR, Kirkham JJ. (2013). Reporting Bias Group. Systematic review of the empirical evidence of
study publication bias and outcome reporting bias - an updated review. PLoS One, 8 (7). e66844. PMID 23861749.
Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. (2003). How important are comprehensive literature searches and the
assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess, 7 (1). 1-76. PMID 12583822.
EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews.
http://www.ncbi.nlm.nih.gov/books/NBK274092
EPPI Reviewer 4. eppi.ioe.ac.uk/cms/er4/
Everett CR, Shah RV, Sehgal N, McKenzie-Brown AM. (2005). A systematic review of diagnostic utility of selective nerve root
blocks. Pain Physician, 8 (2). 225-33. PMID 16850076.
Featherstone, R. Concept Map for Research Literature Reviews. http://www.slideshare.net/featherr/search-strategy-worksheet
Gagnier, JJ. (2015). Introduction to Systematic Reviews & Meta-Analyses.
http://open.umich.edu/education/sph/epid757/summer2011/materials
Giustini D, Boulos MN. (2013). Google Scholar is not enough to be used alone for systematic reviews. Online J Public Health
Inform. 5 (2). 214. PMCID PMC3733758
References










Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman ND, Foster L, Fraser C, Fry-Smith A, Golder S, Lefebvre C,
Miller C, Paisley S, Payne L, Price A, Welch K. (2008). So many filters, so little time: the development of a search filter
appraisal checklist. JMLA, 96(4). 356-61. PMID 18974813. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2568852/
Goldacre, B. (2012). What doctors don't know about the drugs they prescribe. TEDMED 2012.
http://www.ted.com/talks/ben_goldacre_what_doctors_don_t_know_about_the_drugs_they_prescribe?language=en
Grant MJ, Booth A. (2009). A Typology of reviews: an analysis of 14 review types and associated methodologies. Health Info
Libr J. 26 (2). 91-108. PMID 19490148.
Guyatt G, Rennie D, Meade MO, Cook, Deborah J. (2015). Summarizing the Evidence Worksheet. In: JAMAevidence, Users’
Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. 2nd Ed.
http://www.jamaevidence.com/resource/520
HIRU, McMaster University. http://www.nlm.nih.gov/bsd/special_queries.html
HLWiki International. Software for systematic reviewing.
http://hlwiki.slais.ubc.ca/index.php/Software_for_systematic_reviewing
Hopewell S, Clarke M, Lefebvre C, Scherer R. (2007). Handsearching versus electronic searching to identify reports of
randomized trials. Cochrane Database Syst Rev. 18(2). MR000001. PMID 17443625.
Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M. (2002). A comparison of handsearching versus MEDLINE searching to
identify reports of randomized controlled trials. Stat Med. 21 (11). 1625-34. PMID 12111923.
Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. (2007). Cochrane Database Syst
Rev, 2. MR000011. PMID 17443632.
Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. (2009). Publication bias in clinical trials due to statistical
significance or direction of trial results. Cochrane Database Syst Rev. 2009, 1. MR000006. PMID 19160345.
References









Hopewell S, McDonald S, Clarke M, Egger M. (2007). Grey literature in meta-analyses of randomized trials of health care
interventions. Cochrane Database Syst Rev. 18 (2). MR000010. PMID 17443631.
ICMJE. Defining the Role of Authors and Contributors. http://www.icmje.org/recommendations/browse/roles-andresponsibilities/defining-the-role-of-authors-and-contributors.html
Institute of Medicine. Standards for Initiating a Systematic Review. Standards for Systematic Reviews.
http://www.iom.edu/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx
InterTASC, http://www.sign.ac.uk/methodology/filters.html
Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. (2012). Evidence summaries: the evolution of a rapid review
approach. Syst Rev, 1. 10. PMC3351736.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis P, Clarke M, Devereaux PJ, Kleijnen J, Moher D. (2009).
The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions:
Explanation and elaboration. Annals of Internal Medicine, 151 (4). W-65-W-94. PMID 19622512.
Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. (2012). Industry sponsorship and research outcome. Cochrane Database
Syst Rev. 12. MR000033. PMID 23235689.
Mahood Q, Eerd DV, Irvin E. (2014). Searching for grey literature for systematic reviews: challenges and benefits. Res Synth
Methods, 5 (3). 221-34. PMID 26052848.
Martin JL, Pérez V, Sacristán M, Alvarez E. (2005). Is grey literature essential for a better control of publication bias in
psychiatry? An example from three meta-analyses of schizophrenia. Eur Psychiatry, 20 (8). 550-3. PMID 15994063.
References




McDonald S, Lefebvre C, Antes G, Galandi D, Gøtzsche P, Hammarquist C, Haugh M, Jensen KL, Kleijnen J, Loep M, Pistotti V,
Rüther A. (2002). The contribution of handsearching European general health care journals to the Cochrane Controlled Trials
Register. Eval Health Prof, 25 (1), 65-75. PMID 11868446.
McKibbon KA, Wilczynski NL, Haynes RB, Hedges Team. (2009). Retrieving randomized controlled trials from medline: a
comparison of 38 published search filters. Health Info Libr J, 26 (3). 187-202. PMID 19712211.
Nuti SV, Wayda B, Ranasinghe I, Wang S, Dreyer RP, Chen SI, Murugiah K. (2014). The use of google trends in health care
research: a systematic review. PLoS One, 9 (10). e109583. PMID 25337815.
Oxman AD, Guyatt GH. (1993). The science of reviewing research. Annals of the New York Academy of Sciences, 703. 125133. PMID 8192290.

PRISMA. http://www.prisma-statement.org/statement.htm

PROSPERO. Centre for Reviews and Dissemination. http://www.crd.york.ac.uk/PROSPERO/

PubMed Clinical Queries. http://www.ncbi.nlm.nih.gov/books/NBK3827/#pubmedhelp.Clinical_Queries_Filters

PubMed Special Queries. http://www.nlm.nih.gov/bsd/special_queries.html



Rethlefsen ML. (2013). I Want To Do a Systematic Review. Liblog: Newsletter of the Mayo Clinic Libraries.
http://liblog.mayo.edu/2013/05/01/i-want-to-do-a-systematic-review/
Rethlefsen M. (2014). Engaging medical librarians to improve the quality of review articles. JAMA. 312 (10). 999-1000. PMID
25203078.
Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. (2015). Librarian co-authors correlated with higher quality
reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol, 68 (6). 617-26. PMID:
25766056.
References










Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. (2009). Trial publication after registration in ClinicalTrials.Gov: a
cross-sectional analysis. PLoS Med, 6 (9). e1000144. PMID 19901971.
Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM.
2nd ed. Edinburgh: Churchill Livingstone, 2000.
Saleh AA, Ratajeski MA, Bertolet M. (2014). Grey Literature Searching for Health Sciences Systematic Reviews: A Prospective
Study of Time Spent and Resources Utilized. Evid Based Libr Inf Pract, 9 (3). 28-50. PMID 25914722.
Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P. (2003) Should meta-analysts
search Embase in addition to Medline? J Clin Epidemiol, 56 (10). 943-55. PMID: 14568625
Sampson, M, McGowan J. (2006). Errors in search strategies were identified by type and frequency. Journal of Clinical
Epidemiology, 59 (10). 1057-63. PMID 16980145.
Sampson M, Shojania KG, Garritty C, Horsley T, Ocampo M, Moher D. (2008). Systematic reviews can be produced and
published faster. J Clin Epidemiol, 61 (6). 531-6. PMID 18471656.
SIGN. http://www.sign.ac.uk/methodology/filters.html
Sun G, Gaines J, Perla R, MacEachern M, Davis M, Shrank W. (2013). Systematic review of quality improvement initiatives in
the intensive care setting. PROSPERO.
http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42012003303#.VD6ZomNDR8F.
Sun GH, Raji CA, Maceachern MP, Burke JF. (2012). Olfactory identification testing as a predictor of the development of
Alzheimer's dementia: a systematic review. Laryngoscope, 122 (7). 1455-62. PMID 22552846.
Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. (2014). Systematic review automation technologies. Syst Rev,
3. 74. PMID 25005128.