Content Analysis of Implications
for Practice in Empty Reviews
Joanne Yaffe, PhD, University of Utah
Lindsay Shepard, MSc, University of Utah
Paul Montgomery, PhD, University of Oxford
Sally Hopewell, PhD, UK Cochrane Centre
The Empty Reviews Project is funded by the Cochrane Opportunities Fund
Introduction: What We Knew
• 8.7% of reviews in The CDSR did not identify any eligible
studies.
• 25.3% of empty reviews did not list excluded studies.
• 90.2% of empty reviews did not list ongoing studies.
• 96.0% of empty reviews did not list studies waiting
assessment.
• 23.4% of empty reviews did not list any studies other than
what was discussed in their background section.
• Proportions of empty reviews differed somewhat across
topical areas, from 0% to 26.9%.
• CRG editors differed somewhat in their perspectives on
empty reviews and on editorial guidance provided to
authors of these reviews.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
What We Wanted to Know
Despite the lack of high quality, or any, evidence
included in empty reviews, how do authors
report implications for practice?
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Guidance from Handbook
Implications for Practice
12.7.2
“Authors of Cochrane reviews should not
make recommendations. If authors feel
compelled to lay out actions that clinicians
and patients could take, they should – after
describing the quality of evidence and the
balance of benefits and harms – highlight
different actions that might be consistent with
particular patterns of values and preferences.
Other factors that might influence a decision
should also be highlighted, including any
known factors that would be expected to
modify the effects of the intervention, the
baseline risk or status of the patient, costs and
who bears those costs, and the availability of
resources.”
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Conceptual Definition of
Recommendation
We defined recommendations for practice as
statements which instruct readers, including
consumers, practitioners, or policy makers to
do something.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Reporting of Empty Reviews
• Lang et al. (2007)
– Suggest guidelines needed.
– Further, reviewers could note observations from ineligible articles and
abstracts.
• Green et al. (2007)
– Specific structure could be helpful.
– Basing conclusions on studies which do not meet inclusion criteria
increases the risk of bias.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Objectives of the Present Analysis
To explore the following:
• How authors frame discussions of Implications
for Practice?
• Do authors incorporate information from
other studies? If so, how and where?
• Do authors make recommendations for
practice?
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
One Sentence Wonders
117 (31.1%) empty reviews contained a one
sentence Implications for Practice section.
These:
• May be extremely brief and contain almost no
information; and
• vary in how well they explain implications for
decision making.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Examples of One Sentence Wonders
• Very Brief:
– "None identified from this systematic review.”
• Mention lack of evidence and review topic:
– “No randomized controlled trials, which assessed the
effectiveness or safety of oral immunoglobulin
preparations for the treatment of rotavirus diarrhoea in
hospitalized low birthweight infants were found.”
• Absence of evidence is not evidence of absence:
– "No relevant trials appear to have been published so this
review concludes that there is no evidence of effect of
leukotriene antagonists in bronchiectasis rather than
evidence of no effect.”
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Vague Wording
26 (6.9%) empty reviews are written in such tentative
language that despite lengthy discussions between two of us,
we sometimes failed to come to agreement about whether
the section made recommendations and if so, for what.
• “It is likely that clinicians will continue with their current practice,
using clinical judgement and prescribing patterns to dictate
treatment because there is no RCT-based evidence to help guide
their choice of drug. It is difficult to know whether current practice
is justified outside of a well designed, conducted and reported RCT.
Currently policymakers have no RCT-based evidence upon which to
base guidelines for HIVAN. They are likely to continue to rely on
opinion and habit when making their recommendations. Funders of
studies may wish to make this important subgroup of people a
priority for future research.”
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Failure to Cite
Around half of implications for practice sections make
reference to evidence, but only 16% of these cite its
source.
• “The implications for fluid management in acute respiratory
infections have not been studied in any RCTs to date. There is
currently no evidence for or against increased fluids in acute
respiratory infections. Non-experimental (observational) data
suggests that there may be a risk of symptomatic
hyponatraemia due to increased antidiuretic hormone
secretion in lower respiratory tract infections, particularly in
children. The incidence in the primary care setting and the
clinical significance of this observational data needs to be
determined with further research, conducted as randomised
controlled trials.”
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Long and Detailed Discussions
• At least one CRG (Schizophrenia Group) makes
efforts to provide implications for multiple
audiences (consumers, clinicians, and policy
makers).
• The Implications for Practice sections from
these reviews provide rather lengthy
discussions, especially given the lack of
eligible studies.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Example of Long and Detailed
Discussion
"1. For people with severe mental illness
At present, the choice between dedicated supported housing schemes and outreach services is based on
a combination of personal preference, professional judgement and availability of resources. Decisions of
this nature should be made with the full understanding that no one intervention has been shown to be
more effective than another in making a difference to symptoms, future use of services, quality of life or
other measures of importance. Furthermore, the efficacy of supported housing remains untested.
Participating in trials that test the effectiveness of such services should be encouraged.
2. For practitioners
In the absence of evidence of their relative efficacy, decisions on the provision of alternative forms of
accommodation and continued support for people with mental illness can only be based on a
combination of professional judgement, patient preference and availability. This should be made clear to
the patient or client who has to make this important decision. Practitioners may wish to actively support
or participate in trials to test the effectiveness of supported housing schemes for people with severe
mental illness.
3. For managers and policymakers
Policies in favour of dedicated supported housing schemes should be viewed with some caution and
should not be implemented without plans for evaluation using rigorous methods or should be delayed
pending further evidence of their effectiveness. Forming alliances with researchers within this field may
result in a fruitful collaboration that would not only inform local policies on this issue, but would also
provide much needed evidence base on its effectiveness."
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Implications for Practice with
Recommendations
• 59 of 376 empty reviews (15.7%) appeared to
contain recommendations.
• Grouped as:
– generally supportive,
– recommending alternatives, or
– recommending either restricted use or non-use.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Generally Favorable Recommendations
• Represent 23 of 59 (39.0%) empty reviews
which appear to make recommendations.
– 14 of these appear to have been based largely on
other research cited in the background,
– 7 seem to not be based on evidence, but on
guidelines, theory, or absence of evidence of
harm.
– 2 seem to be based on excluded studies.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Example of Generally Favorable
Recommendation
• “There is a need to raise awareness of the likely risks of waterpipe tobacco
smoking worldwide. Particular attention should be devoted to examining
empirically the potentially false perceptions about levels of dependence,
disease and mortality associated with waterpipe tobacco smoking.
• Evidence-based information about waterpipe should be developed and
disseminated in order to deglamourise and denormalise its use.
• Young people, especially women and girls in the Eastern Mediterranean
Region, may be at higher risk of waterpipe use, and may need particular
attention from healthcare workers.
• In the absence of policy research on waterpipe, and given what we know
about its potential harmful effects, there is a need to include this tobacco
use method in all tobacco control policy initiatives (e.g. bans on
advertisement and sales to minors, taxation, warning labels, smokefree
indoor air).
• Treatments of proven efficacy should be provided to help waterpipe
smokers quit.”
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Recommendations to Use
Something Else
• 8 (13.6%) of 59 empty reviews making
recommendations.
• Appear to be based on existing guidelines,
practice standards, other research, or nothing
readily apparent.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Example of Recommendation
to Use Something Else
“As there were no randomised or quasirandomised controlled trials identified, we cannot draw
any conclusions on the benefits (or otherwise) of
regular administration of vitamin A in people with
cystic fibrosis. Until further data are available, country
or region specific guidelines (e.g. UK CF Trust Nutrition
Guidelines (CF Trust 2002)) on the use and monitoring
of vitamin A in people with cystic fibrosis should be
followed.”
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Recommendations for Restricted Use
• 13 (22.0%) of 59 empty reviews making
recommendations.
• Limits on use include use in some
circumstances (1), possible use in some
circumstances (1), minimal use (1), or use only
in the context of a controlled trial (10).
• Appear to be based largely on evidence of
harm, possibility of harm, or no readily
apparent basis.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Examples of Recommendations
for Restricted Use
• Restricted Use
“There is no evidence so far that subjective barriers reduce wandering, and the
possibility of harm (particularly psychological distress) cannot be excluded. If used,
then subjective barriers should form part of a diverse approach to problem
wandering, which may include the identification and definition of the problem in the
individual, preventative activities such as exercise classes or occupational therapies,
and improved communication between carer and wanderer.”
• “Restrict to Trials”
“…2. For clinicians In the absence of reliable evidence, clinicians prescribing calcium
channel blockers for people with tardive dyskinesia must balance the possible
benefits against the potential adverse effects of the treatment. Calcium-channel
blockers lower people’s blood pressure and may even cause symptoms of tardive
dyskinesia to increase. These drugs should really only be used in a situation where
their effects are closely monitored, i.e. within a randomised experimental design that
can be written up and disseminated in order to inform practice.”
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Recommendations to Not Use
• 15 (25.4%) of 59 empty reviews making a
recommendation.
• Appear to be based on lack of evidence,
evidence of harm, or ongoing research where
trials had been aborted.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Example of Recommendation
to Not Use
“Given the lack of evidence comparing hydralazine versus placebo for
primary (essential) hypertension, hydralazine should not be
recommended as monotherapy. Clinicians who wish to recommend
hydralazine for their patients should understand that while hydralazine
may reduce blood pressure (based on non-randomized cross-over trials),
to the best of our knowledge, there are no known clinical studies which
have associated the use of hydralazine with a reduction in all cause
mortality, myocardial infarction, or stroke. In addition, despite poor
reporting of treatment-emergent adverse effects, clinicians must weigh
the risks of potential serious side effects associated with the use of
hydralazine, such as reflex tachycardia, hemolytic anemia,
glomerulonephritis, vasculitis, as well as lupus-like syndrome against the
potential benefits of blood pressure reduction with no proven beneficial
effect on adverse cardiovascular outcomes.”
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Summary
• Implications for Practice sections differ widely
across empty reviews;
• some Implications for Practice sections are very
brief, while others are quite lengthy;
• some Implications for Practice sections are
difficult to interpret or vaguely worded;
• many Implications for Practice sections appear to
refer to research without a formal citation; and
• whilst the majority of Implications for Practice
sections do not make recommendations, some
do.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Conclusions
• Only 15.7% of empty reviews provide recommendations.
• Recommendations appear to be based on:
– Evidence from studies cited in Background,
– Evidence related to adverse effects of the intervention,
– Standard practice or practice guidelines,
– Theory, and/or
– Excluded or ongoing studies.
• A substantial proportion of implications for practice sections
provide very minimal information, even regarding the scope
of the review.
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com
Discussion Points Raised by Analysis
• What to report in Implications for Practice?
– Should evidence discussed in Implications for
Practice be specifically cited?
• What is the minimum that should be said?
• Should evidence of adverse events from
excluded studies or other research be
mentioned?
• Additional points?
The Cochrane Empty Reviews Project
cochrane.empty.reviews@gmail.com