Reducing waste in deciding
what research to do
Iain Chalmers
Coordinator, James Lind Initiative
NIHR Trainees Meeting
Leeds, 26 November 2013
Questions relevant
to clinicians &
patients?
Low priority questions
addressed
Important outcomes
not assessed
Clinicians and
patients not involved
in setting research
agendas
Appropriate design
and methods?
Accessible
full publication?
Unbiased and
usable report?
Over 50% studies
designed without
reference to
systematic reviews of
existing evidence
Over 50% of studies
never published in full
Over 30% of trial
interventions not
sufficiently described
Biased underreporting of studies
with disappointing
results
Over 50% of planned
study outcomes not
reported
Over 50% of studies
fail to take adequate
steps to reduce
biases, e.g.
unconcealed
treatment allocation
50
%
Most new research
not interpreted in the
context of systematic
assessment of other
relevant evidence
50
%
85% Research waste = over $85 Billion / year
50
%
Avoidable waste in producing and reporting
research evidence. Lancet series, 8 Jan 2014
Questions
relevant
to users of
research?
Appropriate
research
design, conduct
and analysis?
Low priority
questions addressed
Over 50% of
studies fail to take
adequate steps to
reduce biases
Important outcomes
not assessed
Over 50% studies
designed without
reference to
systematic reviews
of existing evidence
Studies with
inadequate
statistical power
Inadequate
replication of initial
observations
Efficient research
regulation
and delivery?
Hyper-regulation
of research
Inefficient delivery
of research
Poor re-use of
data
Failure to promote
evaluative
research as an
integral element
of good clinical
practice
Research waste
Accessible, full
research reports?
Over 50% of
studies never
published in full
Biased underreporting of
studies with
disappointing
results
Biased reporting of
data within studies
Unbiased and
usable reports?
Over 30% of trial
interventions not
sufficiently
described
Over 50% of
planned study
outcomes not
reported
Most new research
not interpreted in
the context of
systematic
assessment of other
relevant evidence
Avoidable waste in deciding what research to do.
Paper 1, Lancet series, 2014
Questions
relevant
to users of
research?
Appropriate
research
design, conduct
and analysis?
Low priority
questions addressed
Over 50% of
studies fail to take
adequate steps to
reduce biases
Important outcomes
not assessed
Over 50% studies
designed without
reference to
systematic reviews
of existing evidence
Studies with
inadequate
statistical power
Inadequate
replication of initial
observations
Efficient research
regulation
and delivery?
Hyper-regulation
of research
Inefficient delivery
of research
Poor re-use of
data
Failure to promote
evaluative
research as an
integral element
of good clinical
practice
Research waste
Accessible, full
research reports?
Over 50% of
studies never
published in full
Biased underreporting of
studies with
disappointing
results
Biased reporting of
data within studies
Unbiased and
usable reports?
Over 30% of trial
interventions not
sufficiently
described
Over 50% of
planned study
outcomes not
reported
Most new research
not interpreted in
the context of
systematic
assessment of other
relevant evidence
Iain Chalmers, James Lind Initiative, Oxford, UK
Michael B Bracken, Schools of Public Health
and Medicine, Yale University, New Haven, USA
Ben Djulbegovic, Moffitt Cancer Centre &
Research Institute, University of South Florida, USA
Silvio Garattini, Mario Negri Pharmacological
Research Institute, Milan, Italy
Jonathan Grant, RAND Europe, Cambridge, UK
Metin Gulmezoglu, WHO, Geneva, Switzerland
David Howells, National Stroke Research Institute,
Melbourne, Australia
John PA Ioannidis, School of Medicine and
Department of Statistics, Stanford University, USA
Sandy Oliver, Institute of Education, London, UK
1 Waste resulting from ignoring the
needs of potential users of research
2 Waste resulting from ignoring
what is already known or already
being researched
1 Waste resulting from
ignoring the needs of potential
users of research
Avoidable waste in deciding what research to do.
Paper 1, Lancet series, 2014
Questions
relevant
to users of
research?
Appropriate
research
design, conduct
and analysis?
Low priority
questions addressed
Over 50% of
studies fail to take
adequate steps to
reduce biases
Important outcomes
not assessed
Over 50% studies
designed without
reference to
systematic reviews
of existing evidence
Studies with
inadequate
statistical power
Inadequate
replication of initial
observations
Efficient research
regulation
and delivery?
Hyper-regulation
of research
Inefficient delivery
of research
Poor re-use of
data
Failure to promote
evaluative
research as an
integral element
of good clinical
practice
Research waste
Accessible, full
research reports?
Over 50% of
studies never
published in full
Biased underreporting of
studies with
disappointing
results
Biased reporting of
data within studies
Unbiased and
usable reports?
Over 30% of trial
interventions not
sufficiently
described
Over 50% of
planned study
outcomes not
reported
Most new research
not interpreted in
the context of
systematic
assessment of other
relevant evidence
Low priority questions addressed in research on
treatments for osteoarthritis of the knee
Tallon, Chard and Dieppe. Lancet, 2000.
The UK Database of Uncertainties
about the Effects of Treatments
To publish uncertainties about the effects of
treatments which cannot currently be answered
by referring to relevant and reliable, up-to-date
systematic reviews of existing research evidence
To promote Priority Setting Partnerships
involving patients and clinicians to identify
and promote their shared priorities for
therapeutic research
100%
20
89
90%
307
Education and training, service
delivery, psychological, physical,
exercise, complementary, diet,
other
80%
70%
74
60%
332
50%
689
40%
30%
29
20%
10%
Radiotherapy, surgery and
perioperative, devices, and
diagnostic
Drugs, vaccines and biologicals
397
23
0%
JLA patient-clinician
Priority Setting
Partnerships
Registered noncommercial trials
Registered
commercial trials
Interventions mentioned in research priorities identified by
James Lind Alliance patient-clinician Priority Setting Partnerships,
and among registered trials, 2003-2012. (Chalmers et al. 2013)
Important outcomes
not assessed
Priority treatment outcome from a
survey of patients with rheumatoid
arthritis was not pain
It was fatigue
Alessandro Liberati
"Studies that involved patients to a
greater extent were more likely to
have achieved recruitment targets
(χ2 = 4.58, P<0.05), defined as
reaching at least 90% of the target.“
Ennis L, Wyke T. Impact of patient involvement
in mental health research: longitudinal study. Br
J Psychiatr Published online ahead of print 12
Sept 2013. doi: 10.1192/bjp.bp.112.119818.
2 Waste resulting from ignoring
what is already known or
already being researched
Avoidable waste in deciding
what research to do, Lancet series, 2013
Questions
relevant
to users of
research?
Appropriate
research
design, conduct
and analysis?
Low priority
questions addressed
Over 50% of
studies fail to take
adequate steps to
reduce biases
Important outcomes
not assessed
Over 50% studies
designed without
reference to
systematic reviews
of existing evidence
Studies with
inadequate
statistical power
Inadequate
replication of initial
observations
Efficient research
regulation
and delivery?
Hyper-regulation
of research
Inefficient delivery
of research
Poor re-use of
data
Failure to promote
evaluative
research as an
integral element
of good clinical
practice
Research waste
Accessible, full
research reports?
Over 50% of
studies never
published in full
Biased underreporting of
studies with
disappointing
results
Biased reporting of
data within studies
Unbiased and
usable reports?
Over 30% of trial
interventions not
sufficiently
described
Over 50% of
planned study
outcomes not
reported
Most new research
not interpreted in
the context of
systematic
assessment of other
relevant evidence
"It is essential that existing sources of
evidence, especially systematic reviews, are
considered carefully prior to undertaking
research…
Research which duplicates other work
unnecessarily, or which is not of sufficient
quality to contribute something useful to
existing knowledge, is in itself unethical."
Department of Health 2001
Waste resulting from studies
designed without reference to
systematic reviews of existing evidence,
published and unpublished,
and ongoing studies
STUDIES IN ANIMALS
20 animal studies: “The results of this review did not show
convincing evidence to substantiate the decision to perform
trials with nimodipine in large numbers of patients.
Stroke 2001;32:2433-8.
STUDIES IN HUMANS
Horn J, Limburg M.
Calcium antagonists for acute ischemic stroke.
The Cochrane Database of Systematic Reviews, 2000
“46 trials were identified of which 28 were included (7521 patients).
No effect of calcium antagonists on poor outcome at the end of followup (OR 1.07, 95% CI 0.97/1.18), or on death at end of follow-up (OR
1.10, 95% CI 0.98/1.24) was found.”
Proportion (%) of clinical trials registered by
1999 and published by 2007
(from Ross et al. PLoS Med 2009;6(9): e1000144).
Country
Size
Phase
Funder
UK HTA program
Sense about Science’s campaign:
All trials registered; all trials published
If you haven’t already done so,
add your support at www.alltrials.net
Retrospective cumulative
meta-analyses demonstrate waste
In use-inspired basic research
Sena et al. 2010
In pure applied research
Cumulative meta-analysis showing effect of antenatal
corticosteroids on risk of neonatal death (Sinclair 1995)
The Cochrane Collaboration
Avoidable waste in deciding
what research to do, Lancet series, 2013
Questions
relevant
to users of
research?
Appropriate
research
design, conduct
and analysis?
Low priority
questions addressed
Over 50% of
studies fail to take
adequate steps to
reduce biases
Important outcomes
not assessed
Over 50% studies
designed without
reference to
systematic reviews
of existing evidence
Studies with
inadequate
statistical power
Inadequate
replication of initial
observations
Efficient research
regulation
and delivery?
Hyper-regulation
of research
Inefficient delivery
of research
Poor re-use of
data
Failure to promote
evaluative
research as an
integral element
of good clinical
practice
Research waste
Accessible, full
research reports?
Over 50% of
studies never
published in full
Biased underreporting of
studies with
disappointing
results
Biased reporting of
data within studies
Unbiased and
usable reports?
Over 30% of trial
interventions not
sufficiently
described
Over 50% of
planned study
outcomes not
reported
Most new research
not interpreted in
the context of
systematic
assessment of other
relevant evidence
An example of dividends from well
designed pure applied research
Ian Roberts and his
colleagues did the CRASH
trial to address uncertainty
about the effects of giving
systemic steroids for people
with acute traumatic brain
injury, a treatment that had
been in use for over three
decades.
A systematic review of
existing knowledge
Alderson P, Roberts I (1997). BMJ 314:1855-9;
and Cochrane Database of Systematic Reviews.
The review revealed important uncertainty
about whether systemic steroids
did more good than harm.
Addressing an important uncertainty
Because the systematic review and a
survey of clinical practice had revealed
important uncertainty,
a large, publicly-funded, multicentre
randomized trial was organised;
the trial was registered prospectively;
the protocol for the trial was published
Lancet 2004;364:1321-28
• The report of the CRASH trial is
exemplary because:
• it refers to current uncertainty about the effects of
a treatment, shown in a systematic review of all
the existing evidence, and in variations in
clinical practice
• it notes that the trial was registered and the
protocol published prospectively
• it sets the new results in the context of an
updated systematic review of all of the existing
evidence
• it provides readers with all the evidence needed
for action to prevent thousands of iatrogenic
deaths
Patients have suffered and died
unnecessarily because of:
● failure to address important
uncertainties about the effects of
treatments
● biased under-reporting of
research
● failure to review existing evidence
systematically when planning new
research
Alessandro Liberati