Centre for Evidence Based Intervention
Evidence from Review Groups
Objectives
Identify policies & procedures on…
Determining
Practice
publication of empty reviews
of handling excluded studies across Cochrane
and other providers
Informing
practice based on empty reviews and
excluded studies
Methods

Survey with closed and open-ended questions

53 Cochrane Collaboration Review Groups

WHO, JBI, What Works Clearinghouse and SCIE

Coded & analyzed responses
Why might a review be empty?

“because it is a new or recently introduced intervention
which has yet to gather strength of evidence”

“the intervention does not lend itself to randomised
controlled trials”

“poor question?”

……..
Cochrane Groups publishing empty
reviews?
N=53
Rationale for publishing Empty Reviews
% of
responses
N=46
Page 6
Other providers

WHO - doesn't have a policy on empty reviews. In a way it isn't
relevant - they would search for the best available evidence,
but would need to make a recommendation regardless of what
they find.
 SCIE - As the evidence base in social care is very limited, we
may often find that a review with inclusion criteria specifying
high quality controlled (and/or randomised) studies of specific
interventions would be ‘empty’ – hence we would not
commission a review.
 WWCH - review focus is changed depending on the evidence
base, thus empty reviews are not generally produced.
Views about including empty reviews

Include to instigate further research


“Our aim with empty reviews is to raise the profile of
the question and hopefully generate research leading
to high quality evidence which we can then report
quickly in an updated review.”
Include if good case made

“There may be benefit in amending the TRF form to
include the line: ‘If you think this review might be
empty, please give your reasons why you feel it is
important to do the review.’”
Reasons for not publishing empty reviews
(Some groups have minimum # of studies stipulation - e.g., 2-3)
Not
worth time and resources

“a colossal waste of everyone’s time and energy”
Influence
on group’s impact factor
May be misleading - Empty reviews (< 3 studies) can put positive
supportive evidence from initial, low quality studies behind
treatment
BUT, question can still matter

“may be exceptions where review questions appear of great public
health relevance”
Handling excluded studies
Standard policy?
Unknown
4%
Permitted in review?
Yes
21%
No
31%
No
75%
Yes
58%
Unknown
11%
% of responses
Conditions for providing evidence based on
excluded studies
N=31
Further considerations for whether to allow
excluded studies

Excluded based on intervention or outcomes?

RCTs feasible/ethical for topic area?


If not, lower grade studies may be acceptable
Guidelines may be needed for application of evidence from
excluded studies

“We dissuade authors from providing evidence from excluded studies
because of inconsistency, which would lead to provision of evidence
that is arbitrary and unsystematic.”
Guidance for ‘Implications for practice’ section
For ‘Yes’ responses:
(n=7)
Extra
guidance but on case-by-case
basis (3)
General
Refer
advice (2)
to other empty reviews, EPICOT,
& extra advice (1)
N=53
Group
Methodologists give guidance (1)
Aim to update or withdraw empty
reviews faster than others?
For ‘Yes’ responses:
(n=4)
No
If
N=53
reason given (2)
there are recently published RCTs
(2)
Views about priority on updating empty
reviews

No, topic-based priority only


“Priority is only given on the basis of importance of the review topic, not on
whether there are included studies or not.”
Only if new study emerges

“Aim to update/review them around the same time as other reviews , but if
we became aware of a study relevant to an empty review, it would
probably prompt a more rapid update.”
Thank you!
paul.montgomery@spi.ox.ac.uk