Is it True: Evaluating Medical
Reviews
“The review article itself should be the
product of scientific investigation in which
the participants are original investigations
(research) rather than patients”
Usefulness

Work: Low, good source for POEMs

Relevance: If title and abstract or article
conclusion hold promise of POEM, continue
Validity: Uncertain

Reviews- Three Basic Types

Textbooks

Academic Reviews

“Translation” Journals
Textbooks

Collection of review articles

Minimal, if any, supporting evidence

Questionable validity, long lag time to publish
• Average 1-2 years

Most useful for retracing, less hunting and
foraging
Academic Reviews




Summary: “Broadly paint landscape”
Validity uncertain- begin with conclusions and
find supporting references
References often inaccurate and out of date
**Expertise of author varies inversely with
quality of review- Oxman/Guyatt**
• More later

Must confirm POEMs with original research,
increasing work
Academic Reviews

Synthesis: Systematic reviews

Answer one or two specific questions
Review primary literature with strict criteria



• Meta-analysis or overviews
Conclusions supported by available evidence
Meta-analysis: Achieve power not possible by
single study
Academic Reviews

Excellent source for hunting and foraging

The Cochrane Library - “Database of Systematic
Reviews”

Clinical Evidence (BMJ-BMA)

Clinical Inquiries (FPIN)- SORT

AFP EB Reviews- SORT

Dynamed - SORT

Essential Evidence Plus- SORT
Translation Journals

Quick reads for retracing and sporting

Low work, but with low validity, may be zero
usefulness
Hunting/foraging: Entering jungle on
starless night

Translation Journals

Common POEMs need original data for
verification, greatly increasing work

Watch for “weasel words”, based on DOEs
and anecdotes
• “it seems”, “may be effective”, “so one may assume”,
“it appears”, “in my experience”
Weasel Words
Translation Journals
“Buyer Beware: Unsystematic reviews
lead to unsystematic conclusions.
Readers looking for a shortcut to
understanding evidence about health
problems and patient care should at
least look for reviews by those who
have not taken shortcuts”
Determining the usefulness
of reviews
Onto the
worksheets!
Reviews: Determining Relevance
A. Addressing specific clinical question?
A. Patient-oriented evidence?
B. Common problem?
C. Change your practice?
Reviews: Determining Validity

Answer ALL worksheet questions

Stop = “fatal flaw”

Notice how hard this is! Average time for a
good systematic review- 2 years!
• Much different from your “usual review/CME talk”
Worksheet Qs: Finding the studies


Clearly stated?
•
Comprehensive?
•
•
•

Terms appropriate? MESH-linked? None missing?
Medline + another
• MEDLINE misses >50% of articles
• Cochrane registry is especially good source
• Science Citation Index
Bibliographic review
Unpublished literature
• conference abstracts, personal correspondence with
important investigators or pharmaceutical companies
Done by more than one person and compared
Worksheet Qs: Selecting the studies

Inclusion Criteria
•
•
•
•

Established a priori
Minimum factors: Population/problem;
intervention/comparison; outcomes; study design
Prefer no language restriction
Sometimes validity criteria incorporated (random, blinded,
appropriate follow-up, gold standard, etc.)
Best if done independently by 2 investigators
•
Possibly blinded to author/journal/study results
Worksheet Qs:Validity of included studies


Appropriate criteria?
•
•
Assurance that criteria specific to type of article employed
(therapy, diagnosis, prognosis, etc)
If therapy: randomization, blinding, concealed allocation,
follow-up
Process independent by > 2 authors?
•
•
Surprising differences!
Why blinding may be important:
• 2 sample articles, same study methods
• One finds benefit, other does not
• “serious flaws” in article without benefit
Worksheet Qs: Validity

Were the included studies valid?
• Garbage in = garbage out
• If yes, no problem
• If no, how did authors handle this?
• Exclusion/inclusion criteria for quality of study
• Subanalysis with comparison of results
• Need to consider how these flaws affect
results/conclusions
Worksheet Qs: Analyzing the data


Homogeneity vs Heterogeneity: just finding
the words and an explanation most important
If NOT homogeneous?
• Need qualitative explanation. Is it due to chance vs study
design, population, exposure, or outcome?
Worksheet Qs: Analyzing the data


Appropriateness of combining data:
•
•
•
“Vote” count not usually appropriate
Important to include ‘magnitude’ of the overall effect
Cannot be done without some common ground- outcome
Publication bias
•
•
•
Small, negative trials less likely to be published
Examined by funnel plot
Number needed to change results
Funnel plot examples
From: Cooper & Hedges: The handbook of research synthesis. 1994
From: Cooper & Hedges: The handbook of research synthesis. 1994
Reviews: Major Points

Validity traps to avoid
•
•


Assertions based on DOEs -- avoid perpetuating medical
gossip
Unassessed validity -- Personal experience unreliable as a
basis for therapeutic interventions
Missing pieces -- **Quality of the review varies
inversely with the expertness of the writer**
Failure to identify level of evidence – Look for
LOE’s/ SORT
Reviews- Three Basic Types
Effect on Patient-Oriented
Outcomes
Symptoms
Functioning
Quality of Life
Lifespan
Effect on Disease Markers
Diabetes (microalbuminuria,
GFR, photocoagulation rates)
Arthritis (ESR, x-rays)
Peptic Ulcer (endoscopic ulcers)
SORT
B
SORT
A
SORT
C
Effect on Risk Factors for
Disease
Improvement in markers (blood
pressure, HbA1C, cholesterol)
Uncontrolled Observations
&
Conjecture
Physiologic Research
Preliminary Clinical Research
Case reports
Observational studies
Validity of Evidence
Highly Controlled Research
Randomized Controlled Trials
Systematic Reviews