How to critically appraise a systematic review
185 (43%)
(27%)
Igho J. Onakpoya 116
MD
MSc
University of Oxford
Centre for Evidence-Based Medicine
Nuffield Department
of Primary Care Health Sciences
49 (11%)
Oxford
igho.onakpoya@phc.ox.ac.uk
Learning outcomes
 Principles of critical appraisal of SRs
 How to undertake critical appraisal of SRs
 How to appraise published SRs
 Assessing the relevance of published SRs
 2 sessions




Part I - Teaching (90 mins)
Part II - Class activity (90 mins)
1 short break in session 1 (5 mins)
15 minute break between sessions 1 and 2
Opening scenario
Ms X is a 35 year old banker with five
children. She lost her dad who was
hypertensive three years ago due to a
heart attack. She is worried about her
risk of having high blood pressure but
does not want to take any conventional
medicines because “they have lots of
side effects”. Her close friend has told
her to take green tea but she is not sure
whether “it works”. She therefore wants
your opinion on whether this a good
idea.
Get on the internet
Let’s see PubMed Clinical Queries!
Decide on which review/reviews meet your need .
Actually, we have done ours too!
What is a systematic review?
Systematic identification, appraisal
and synthesis of the evidence from
studies addressing the same
research question.
Why systematic reviews?
Traditional (narrative) reviews
Don’t accept the
conclusions of every
systematic review
hook, line and sinker!
How to read a systematic review
Start with the title!
Next step: Useful pneumonic
 Q: Was there a clear research question?
 F: Did they find all relevant studies?
 A: Was study quality assessed?
 S: Were the results adequately summarized?
Step 1: Is there a clear and focussed research Question?
Be sure of what you are looking for.
Clear Question?
 Participants
 Intervention
 Comparator
 Outcome
Based on the paper Onakpoya et al.
Q:
What is the PICO?
P: Normotensive or hypertensive adults
I: Green tea
C: Placebo/identical controls
O: Blood pressure
Step 2: How did they Find the studies?
1. Which databases were searched, how many?
2. Was the search period specified?
3. Was grey literature searched?
4. Did they contact study authors?
5. Did they search bibliography?
F: How were studies included?
 Independent determination of study eligibility
 How many were eligible?
 How many were excluded?
 Were the reasons for exclusion reported?
 Is there a list of excluded studies?
 Resolution of disagreements
Flow charts could be helpful
QF
Step 3: Did they Assess study quality?
Many quality assessment checklists available:
 Cochrane
 Jadad
 CONSORT
 STROBE
 GRACE
Quality rating can be subjective
Cochrane Assessing the risk of bias
Onakpoya et al. The effect of cactus pear (Opuntia ficus-indica) on body weight and cardiovascular risk factors: a systematic review and meta-analysis of randomized clinical trials. Nutrition. 2015 May;31(5):640-6
Step 4: Did they Summarize the evidence?
1. Did they extract results data from included studies?
2. Did they statistically combine the data?
3. Did it make sense to combine the data?
Statistical data combination
http://www.cochrane.org/
Basics of meta-analysis
 Statistically combines data from individual studies:
forest plot
Uses fixed or random effects model
 Can test for variations in trial designs and participants:
heterogeneity
 Can test for publication bias: funnel plot
What is a meta-analysis?
• A way to calculate an average
• Estimates an ‘average’ or ‘common’ effect
• Improves the precision of an estimate by using
all available data
What is a meta-analysis?
Optional part of a systematic review
Systematic reviews
Meta-analyses
At the bottom there’s
a horizontal line. This
is the scale measuring
the treatment effect.
The vertical line in the
middle is where the
treatment and control
have the same effect –
there is no difference
between the two
For each study
there is an id
The data for
each trial
are here, divided
into the experimental
and control groups
This is the % weight
given to this
study in the
pooled analysis
The data shown in
the graph are also
given numerically
The label above the graph
tells you what statistic
has been used
The pooled analysis is given a diamond shape
where the widest bit in the middle
is located at the calculated
best guess (point estimate),
and the horizontal width is the
confidence interval
Note on interpretation
If the confidence interval crosses the line
of no effect, this is equivalent to saying that
we have found no statistically significant difference in
the effects of the two interventions
Line of
no effect
trials
Confidence
interval
Overall
effect
Effect of green tea supplementation on systolic blood pressure (mmHg).
Did the authors interpret
the results correctly?
Beware of missing data!
Appendix 1a: Funnel plot of mean differences plotted against sample size. The vertical line represents the
mean difference for all trials.
Based on the review findings what
advice would you give Ms X?
And why?
Questions to ponder on
 Do the results appear valid and reliable?
 How widely applicable are the study findings?
 Are the results relevant to my local population?
 Do the results apply to this patient/individual?
Useful pneumonic
 Q: Was there a clear research question?
 F: Did they find all relevant studies?
 A: Was study quality assessed?
 S: Were the results adequately summarized?
Part II: Class activity
 Critically appraise the systematic review: 45 minutes
 2 groups will look at chlorogenic acids
 2 groups will look at steviol glycosides
 Nominate someone to present for your group
 We re-group at 4:30 pm
 5 minute presentation summary of your review
Class activity
Any Questions?
Useful resources
 Cochrane Handbook of Systematic Reviews of
Interventions [updated March 2011].
http://handbook.cochrane.org/
 Gough D, Oliver S, Thomas J. An Introduction to
Systematic Reviews. SAGE Publications Ltd. 2012
 PRISMA. Transparent Reporting of Systematic
Reviews and Meta-Analyses. http://www.prismastatement.org/index.htm