Coordinator, James Initiative www.jameslindlibrary.org
SysNet Annual Lecture, Cardiff University
23 April 2012

Questions relevant to clinicians & patients?
Appropriate design and methods?
Accessible full publication?
Unbiased and usable report?
Low priority questions addressed
Important outcomes not assessed
Clinicians and patients not involved in setting research agendas
Over 50% studies designed without reference to systematic reviews of existing evidence
Over 50% of studies fail to take adequate steps to reduce biases, e.g. unconcealed treatment allocation
50
%
Over 50% of studies never published in full
Biased underreporting of studies with disappointing results
50
%
Over 30% of trial interventions not sufficiently described
Over 50% of planned study outcomes not reported
Most new research not interpreted in the context of systematic assessment of other relevant evidence
50
%
85% Research waste = over $85 Billion / year

Questions relevant to clinicians & patients?
Appropriate design and methods?
Accessible full publication?
Unbiased and usable report?
Low priority questions addressed
Important outcomes not assessed
Clinicians and patients not involved in setting research agendas
Over 50% studies designed without reference to systematic reviews of existing evidence
Over 50% of studies fail to take adequate steps to reduce biases, e.g. unconcealed treatment allocation
50
%
Over 50% of studies never published in full
Biased underreporting of studies with disappointing results
50
%
Over 30% of trial interventions not sufficiently described
Over 50% of planned study outcomes not reported
Most new research not interpreted in the context of systematic assessment of other relevant evidence
50
%
85% Research waste = over $85 Billion / year

Patients have suffered and died unnecessarily, and resources for health care and health research have been wasted, because existing research evidence has not been reviewed systematically.

JAMA 1992;268:240-248.

The human costs of failing to cumulate evidence from research scientifically
“Advice on some life-saving therapies has been delayed for more than a decade, while other treatments have been recommended long after controlled research has shown them to be harmful.”
Antman et al. JAMA 1992;268:240-8.

2 nd edn, 1987

Sometimes very important advances in knowledge come from systematic reviews of existing evidence

"It is essential that existing sources of evidence, especially systematic reviews, are considered carefully prior to undertaking research. Research which duplicates other work unnecessarily, or which is not of sufficient quality to contribute something useful to existing knowledge, is in itself unethical."
Department of Health. Research Governance
Framework for Health and Social Care, 2001, para 2.3.1

TGN1412
TGN 1412

Cumulative estimate from trials of the effect of aprotinin on perioperative blood transfusion, 1987-2002.
Aprotinin better Aprotinin worse

New trials of aprotinin ignored previous trials

But was it the right question?
Is tranexamic acid better than aprotinin?
A comparison of aprotinin and lysine analogues in high-risk cardiac surgery.
Ferguson et al. N Engl J Med. 2008 May 29;358(22):2319-31.

*
*actually, the non-use
Only 11 of 24 responding authors of trial reports that had been added to existing systematic reviews were even aware of the relevant reviews when they designed their new studies.

STUDIES IN ANIMALS
20 animal studies: “The results of this review did not show convincing evidence to substantiate the decision to perform trials with nimodipine in large numbers of patients.
Stroke 2001;32:2433-8.
STUDIES IN HUMANS
Horn J, Limburg M.
Calcium antagonists for acute ischemic stroke.
The Cochrane Database of Systematic Reviews, 2000
“46 trials were identified of which 28 were included (7521 patients).
No effect of calcium antagonists on poor outcome at the end of followup (OR 1.07, 95% CI 0.97/1.18), or on death at end of follow-up (OR
1.10, 95% CI 0.98/1.24) was found.”

2005
“…there is a relative scarcity of systematic reviews…”
“…it would therefore be desirable to undertake further systematic reviews and meta-analyses to evaluate more fully the predictability and transferability of animal models.”

Can this policy be defended on scientific, ethical, or economic grounds?

No subsequent contact made.
Richard Hughes, personal communication,
30 Nov 2011

Patients have suffered and died unnecessarily, and resources for health care and health research have been wasted, because existing research evidence has not been reviewed systematically.

Four questions to which readers want answers when reading reports of
research.
1. Why did you start?
2. What did you do?
3. What answer did you get?
4. And what does it mean anyway?

Lord Rayleigh, 1842-1919
“One of the very few members of the higher nobility who won fame as an outstanding scientist.”

“If, as is sometimes supposed, science consisted in nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight… The work which deserves, but I am afraid does not always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out.”
Lord Rayleigh, 1884

Things that should be done:
Address uncertainties by reviewing existing evidence systematically.
Ensure that new research begins and ends by referring to systematic reviews of other relevant evidence.

How can we expect patients and the public to trust that we have their interests at heart if we and our professional institutions continue to acquiesce in our failure to make systematic use of the results of research that they have funded?

Promote research on the effects of treatments…

Promote research on the effects of treatments…
…but only if it meets scientific and ethical principles

Sir Paul Nurse , President of the Royal
Society, urged research funders to trust scientists to decide where
public funding should be directed.
In speaking about initiatives that arise from reviews to decide priorities Sir Paul warned that “such initiatives may attract less creative and effective scientists who simply follow where resources are being made available.”
The case for greater interaction between scientists and business was also made as being crucial to rebuilding an innovation-based economy.
De Havilland Alert, 1 Dec 2011.

The decline in costs of computer memory and gene sequencing
• Computer power: costs halve every 18
Months
• Gene sequencing faster!
• Why not systematic reviews?

The steps in a Systematic Review
• What is current time for each step?
• How can we make each easier/faster?
– Standardize
– Streamline
– Computerize

Steve
Pritchard
Alison
Weightman

• Highlight the search terms that had been used - yellow in example
BMC Fam Pract. 2010 Jul 29;11:54. Randomised controlled trial of local corticosteroid injections for carpal tunnel syndrome in general practice. Peters-Veluthamaningal C, Winters JC, Groenier KH,
Meyboom-de Jong B.
BACKGROUND: Carpal tunnel syndrome is caused by entrapment of the median nerve and results in pain, tingling and numbness in the wrist and hand. It is a common condition in general practice.
Effectiveness of treatment by intracarpal corticosteroid injection has never been investigated in general practice. The objective of this study was to determine if corticosteroid injections for carpal tunnel syndrome provided by general practitioners are effective.
• Highlight methodological terms
(from a supplementary list NOT used in the search) – pink in example.
METHODS: In this study 69 participants with a clinical diagnosis of carpal tunnel syndrome were recruited from 20 general practices. Short-term outcomes were assessed in a randomised, placebocontrolled trial. Long-term results were assessed in a prospective cohort-study of steroid responders.
Participants were randomised to intracarpal injections of 1 ml triamcinolonacetonide 10 mg/ml (TCA) or 1 ml NaCl (placebo). Non-responders to NaCl were treated with additional TCA injections. Main outcomes were immediate treatment success, mean score of the Symptom Severity Scale (SSS) and
Functional Status Scale (FSS) of the Boston carpal tunnel questionnaire, subjective improvement and proportion of participants with recurrences during follow-up. Duration of follow-up was twelve months.
• Allowing easy marking of definite, possible, and definitely not abstracts, and comparison of these markings across two reviewers.
RESULTS: The TCA-group (36 participants) had better outcomes than the NaCl-group (33 participants) during short-term assessment for outcome measures treatment response, mean improvement of
SSS-score (the mean difference in change score was 0.637 {95% CI: 0.320, 0.960; p < 0.001}) and FSSscore (the mean difference in change score was 0.588 {95% CI: 0.232, 0.944; p = 0.002}) and perceived improvement (p = 0.01). The number to treat to achieve satisfactory partial treatment response or complete resolution of symptoms and signs was 3 (95% CI:1.83, 9.72).49% of TCAresponders (17/35) had recurrences during follow-up. In the group of TCA-responders without recurrences (51%, 18/35) outcomes for SSS-score and FSS-score deteriorated during the follow-up period of 12 months (resp. p = 0.008 and p = 0.012).
[Sente 6]
CONCLUSIONS: Corticosteroid injections for CTS provided by general practitioners are effective regarding short-term outcomes when compared to placebo injections. The short-term beneficial treatment effects of steroid injections deteriorated during the follow-up period of twelve months and half of the cohort of steroid-responders had recurrences. PMID: 20670438

For any review step, do you have tips to:
•Standardize?
•Streamline?
•Automate?
Ideas, please, to: pglaszio@bond.edu.au
or colesbm@cardiff.ac.uk