Grey Lit 101:
Shining a White Light on Grey Literature
Amanda Hodgson, MLIS
Sarah McGill, BSc, MLIS
April 5, 2011
About Us
CADTH
CADTH is the Canadian Agency for Drugs and Technologies in
Health, a national body that provides Canada’s federal,
provincial, and territorial health care decision makers with
credible, impartial advice and evidence-based information
about the effectiveness of drugs and other health technologies.
www.cadth.ca
About Us
Amanda Hodgson
•
Manager, Information Services
Sarah McGill
•
Information Specialist
Our department:
•
Literature searching for rapid reviews, health technology
assessments
•
Grey Matters – online resource for grey literature searching
http://www.cadth.ca/index.php/en/cadth/products/greymatters
Webinar Objectives
We will discuss:
•
Current definitions and assumptions
•
Importance of grey literature
•
Key grey literature resources
•
Grey literature search tactics
What is grey literature?
Definitions:
• Grey/gray
• Unpublished
• Non-commercial material
• In-house, vertical files
• Fugitive
• Information on the deep web
• “Hard to find”
What is grey literature?
“Information produced on all levels of government,
academics, business and industry in electronic and
print formats not controlled by commercial publishing
i.e. where publishing is not the primary activity of the
producing body.”
(International Conference on Grey Literature, Luxembourg,
1997 – Expanded in New York, 2004)
Source: GreyNet: grey literature network service. Amsterdam:
GreyNet; 2008.
What is grey literature?
Why does grey literature exist?

Mandate to produce information
Who creates it and what purpose does it serve?

Governments, research organizations, universities, industry

Informing stakeholders, public awareness, organization’s
own use
What are some examples?

Press release on an epidemic (government agency)

Clinical practice guideline (professional association)

Research report on aging population health needs (non-profit
think tank)
Grey literature definitions
How are grey literature definitions changing, and are these
definitions accurate and/or appropriate?
• Online publishing
 Open-access journals
• Grey literature as a field unto itself
 Importance of information community
• Consumers
 Focus on searching, assessment of materials
• Producers
 Transparency of methods, accessibility, impact
Grey literature definitions
Our experience
• Anything identified outside of biomedical bibliographic
databases such as PubMed/Medline, Embase
• Our search results may or may not include commercially
published materials
• We define our search separately from the material we find
Assumptions
What are assumptions about grey literature and are they
accurate/appropriate?
• Free source
• Current
• Unique
• “Hard to find”
• Alternate source
• Ephemeral
• Not credible, Not peer-reviewed
Importance of grey literature
Why is grey literature important in health services
research, public policy, and critical decisionmaking?
Short and long-term values, but primarily:

Comprehensive coverage

Overcomes bias of commercial publications

Part of evidence base
Importance of grey literature
“…. published trials tend to be larger and show an
overall greater treatment effect than grey trials.
This has important implications for reviewers who
need to ensure they identify grey trials, in order to
minimise the risk of introducing bias into their
review.”
Source: Hopewell S, McDonald S, Clarke MJ, Egger M. Grey literature in
meta-analyses of randomized trials of health care interventions. Cochrane
Database Syst Rev 2007;(2).
Importance of grey literature
“Published literature does not always accurately represent
trial results. Often, only articles with positive results are
published, while those with “null” or negative results are
not. And, even when studies are published, reporting can
be biased in many other ways. Systematic reviews
and meta-analysis based solely on published
literature that report positive results will exaggerate
any estimate of effectiveness.”
Source: Relevo R, Balshem H. Methods guide for comparative effectiveness reviews:
Finding evidence for comparing medical interventions. Rockville (MD): AHRQ; January
2011. http://www.effectivehealthcare.ahrq.gov/ehc/products/276/605/MethodsGuide-Relevo--1-5-2011.pdf
Importance of grey literature
• Aids in developing knowledge of a topic

How is this topic discussed internationally?

Find “hard to find” published literature
• Non-commercial/alternate source

Unique and exclusive information

Authority of producers
• Freely available, current information

Pre-publication study information, recruiting trials
Long-term value
As a source, it is part of the evidence-base:
• Patient information

Statistics, patient registries, clinical trials
• Unpublished trials

Conference abstracts, trial registries data
• Evidence synthesis

Systematic reviews, health technology assessments,
economic evaluations
Long-term value
For some health topic areas, it is an invaluable
source:
• Health policy
• Alternative and complementary medicine
• Rare diseases
Long-term value
Historical record
• Full reports vs. journal articles

A summary of a report or trial may be recorded in a
journal article, but the full report may be available
elsewhere
• Evidence supporting policy decisions

Health technology assessments

Lack of evidence may support further work (health
policy)
Grey literature as evidence
Back to assumptions…
Can grey literature be valid, appropriate material to use
as evidence? Yes!
What are the barriers to credibility:
• Quality of grey literature is informed by transparency of
methods
• Source of knowledge
• Authorship, agency/publisher, date info, etc.
Grey literature as evidence
Agency for Healthcare Research and Quality (AHRQ):
Relevo R, Balshem H. Methods guide for comparative effectiveness reviews: Finding
evidence for comparing medical interventions. Rockville (MD): AHRQ; January 2011.
http://www.effectivehealthcare.ahrq.gov/ehc/products/276/605/MethodsGuide-Relevo--1-5-2011.pdf
Institute of Medicine (IOM):
Institute of Medicine. Finding what works in health care: standards for systematic
reviews. Washington D.C.: National Academy of Sciences; March 2011.
http://www.iom.edu/Reports/2011/Finding-What-Works-in-Health-Care-Standards-forSystematic-Reviews/
Institute of Medicine. Clinical practice guidelines we can trust. Washington D.C.;
National Academy of Sciences; March 2011.
http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust
Grey literature as evidence
AHRQ methods guide and IOM standards:
• Librarian/expert searcher to develop search strategy
• Grey literature to be searched:
• Regulatory data
• Clinical trials registries
• Conference proceedings and abstracts
• Web searches
• Grey literature searches are to be recorded and sources
noted as part of the methods/search strategies reporting
Key grey literature resources
What are the key grey literature resources for health
professionals and health services researchers?
Main considerations relate to the information need at hand
and basic parameters:
• What type of search is needed
• Categories of information required
• Topic area
• Geography and politics
Key grey literature resources
• Grey literature producing bodies (sources)
 Foundations, government departments, think tanks,
manufacturers/industry
• Grey literature databases/repositories (resources)
 Databases (free/subscription-based), library
compilations, wikis, clearinghouses
• Grey literature categories
 Category of information (conference materials,
statistics) vs. type of producing body/source (research
institute, pharmaceutical company)
Key grey literature resources –
Who is out there?
• Librarians/Information specialists

Listservs for the information community
-
Medical Library Association http://www.mlanet.org/discussion/
• Industry


Clinical trials
Device and drug developments
• Experts in the field (clinicians, experienced researchers,
authors)


Conferences of interest
Key journals that may not be indexed in PubMed
Key grey literature resources –
What is out there?
•
•
Professional and membership associations, national
societies

Guidelines, standards

Conference proceedings, abstracts
Research institutes and think tanks

•
Forecasting reports, policy papers, working papers
Government departments and agencies

A wealth of information: statistics, white papers,
testimony/transcripts, regulatory approvals and
advisories
Key grey literature resources –
What is out there?
• Libraries and information community
 Subject guides and resources lists on library websites
 New York Academy of Medicine: Grey Literature report
(http://www.nyam.org/library/online-resources/greyliterature-report/)
 Grey Literature Network Service – GreySource Index
http://www.greynet.org/greysourceindex.html
 Grey literature producing agencies (AHRQ, CADTH,
ECRI Institute)
Key grey literature resources
Grey Matters:
a practical tool for
evidence-based
searching
http://www.cadth.ca/
Key grey literature resources
Some CADTH Grey Matters categories:










Health Technology Assessment (HTA) agencies
Health Economics
Clinical Practice Guidelines
Drug/Devices Approval
Advisories/Warnings
Drug Class Reviews
Clinical Trials
Databases
Statistics
Open Access Journals
Health Technology Assessment
• Agency for Healthcare Research and Quality (AHRQ)
http://www.ahrq.gov/
• U.S. Department of Veterans Affairs – VA Technology
Assessment Program http://www.va.gov/VATAP/
• ECRI Institute http://www.ecri.org
• HTAi – Health Technology Assessment International
http://www.htai.org
• International Network of Agencies for Health Technology
Assessment (INAHTA) http://www.inahta.org
• Centre for Reviews and Dissemination. University of York
http://www.york.ac.uk/inst/crd/
Regulatory and Safety Information
• U.S. Food and Drug Administration http://www.fda.gov/
 Medwatch: http://www.fda.gov/Safety/MedWatch/default.htm
 Advisory Committees:
http://www.fda.gov/AdvisoryCommittees/default.htm
 Center for Drug Evaluation and Research:
http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm
 FDA 510(k) Premarket Notification Database
http://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures
/DeviceApprovalsandClearances/default.htm#databases
• European Medicines Agency http://www.ema.europa.eu/
• Health Canada http://www.hc-sc.gc.ca
Clinical trials registries
• U.S. National Institutes of Health. ClinicalTrials.gov
http://www.clinicaltrials.gov/
• PhRMA Clinical Study Results Database
http://www.clinicalstudyresults.org
• Current Controlled Trials http://www.controlled-trials.com/
ISRCTN: http://www.controlled-trials.com/isrctn/
Meta-register: http://www.controlled-trials.com/mrct/
Statistics
• National Center for Health Statistics
http://www.cdc.gov/nchs/index.htm
• Center for Disease Control and Prevention (CDC) – data
and statistics page http://www.cdc.gov/DataStatistics/
• Institute for Health Economics (IHE) Database of Online
Health Statistics – U.S. list
http://www.ihe.ca/publications/healthdb/geo/North%20America/United+States/
Other resources for grey literature
• Open-access journals
 PubMed Central http://www.pubmedcentral.com
 Directory of Open Access Journals http://www.doaj.org
• News and media
 Standard news agencies
- Reuters, CNN, CBC
 Bulletins, newsletters, news alerts
• Internet searching
Search tactics
How would search tactics for grey literature differ for
systematic reviews, health technology assessments, and
general health policy questions?
• General health policy questions
• Systematic reviews
• Health technology assessments
Search tactics –
General health policy questions
• Rapid reviews

Inquiry service to provide quality assessments of
medical technologies

Looking for best evidence available
• Emerging issues

Environmental scanning reports
• Strategic planning

Organizational goals
Search tactics –
General health policy questions
Databases
•
•
An excellent place to find key sources FAST
Retrieve grey literature in addition to traditional
commercially published literature
Resources:

NHS CRD Centre for Reviews & Dissemination
http://www.crd.york.ac.uk/crdweb/

Cochrane Library ($) http://www.thecochranelibrary.com/

TRIP Database http://www.tripdatabase.com/

ECRI Institute ($) https://www.ecri.org/Pages/default.aspx
Databases – CRD
Databases – Cochrane Library
Databases – TRIP
Search tactics –
General health policy questions
• May limit to highest levels of evidence available if
necessary

•
Examples: Systematic reviews/HTAs only, Cost/economic
studies only
Internet search may be very focused

Example: limit by domain (.gov, .edu, .org)
• Some questions largely informed by web search

Example: Long-term care facility policies in a specific
geographic region
Search tactics – Systematic reviews
Attempting to retrieve information on all available clinical
trials
• Comprehensive, exhaustive search

Expanded categories of information
• Aim to be transparent and reproducible

Recording methods

Our experience: Recording search using Grey Matters tool
Search tactics – Systematic reviews
• Grey literature checklist (Grey Matters)
• May broaden scope in relation to electronic database
search strategies

Example: searching guidelines
•
Extensive Internet searching
•
Searching through bibliographies
•
Contacting experts, industry, and others for further
information
Search tactics –
Health technology assessments
Same approach as with systematic reviews, with added
components to consider:
• Health economics (economic evaluations)
• Economic models
• Quality of life information
• Ethics
• Assessment of measurements: validity of scales, questionnaires
• Health programs and health systems impact
• Comparative effectiveness
Search tactics –
Health technology assessments
• In searching for HTAs, we would broaden our focus to
include these added aspects of the report.
• Close correspondence with our clients is key to searching
for and compiling the results
• Alerts and updates
Search tactics for grey literature
General search tactics for grey literature:
• Organizing your search is key

PICO (Population/Intervention/Comparators/Outcome)
• Scope the higher levels of evidence to start

There may be evidence synthesis available on your topic

Online databases a good place to begin
• If you are not the recipient of the results, consultation with your
requestor is essential
• Internet searches important for all grey lit searches

Tricks to limiting Google searches

Librarian consultation can be helpful
The White Light…
Finally, some observations from our experience with grey
literature:
• Treat grey literature like any other category of literature
with respect to assessment and analysis of credibility
• Tap into the information community for up to date
resource lists and searching tips
For More Information
CADTH web site: www.cadth.ca