Assessing study quality for a
systematic review
Trudy Bekkering, PhD
Center of Evidence-Based Medicine &
Belgian Branch of the Dutch Cochrane Center
Centre for Methodology of Educational Research
Katholieke Universiteit Leuven, Belgium
Why is it so important?
• Meta-analysis aims to increase precision
• Meta-analysis of studies with bias in results
gives very precise but wrong results
• Garbage in, garbage out
Bias versus imprecision
Bias:
• A systematic error in the results or the
inferences
• Methodological flaw
• Overestimation or underestimation
Bias versus imprecision
Ideal study
BIAS
Bias versus imprecision
Imprecision:
• A random error in the results
• Sample variation
• Direction of error is random
Bias versus imprecision
Ideal study
IMPRECISION
Bias versus imprecision
BIAS + IMPRECISION
Bias versus imprecision
BIAS ? IMPRECISION?
Risk of bias versus bias
• Clear empirical evidence that particular flaws
in study design can lead to bias.
• Usually impossible to know to what extent
biases have affected the results.
• Key consideration = should the results be
believed
Tools for assessing study quality
Scales: discouraged
Checklist: between 3 and 57 items
Cochrane tool: “domain based”
Depends on study design
Tool for RCTs: Cochrane tool
Risk of bias on 6 domaines:
1.
Sequence generation
2.
Allocation concealment
3.
Blinding
4.
Incomplete outcome data
5.
Selective reporting
6.
Other sources of bias
Risk of which biases?
Selection bias
Differences between baseline characteristics of
the groups compared
Performance bias
Differences in the care that is provided, or in
exposure to other factors than the intervention
Attrition bias
Differences in withdrawals from a study
Detection bias
Differences in how outcomes are determined
Reporting bias
Differences between reported and unreported
outcomes
How do you assess?
Domaine
Sequence generation
Description
QUOTE: “patients were randomly allocated”
COMMENT: probably done, since earlier
reports of this study describe use of random
sequences
Judgement
YES (low risk of bias)
Allocation concealment
YES (low risk)
NO (high risk)
UNCLEAR (uncertain)
Blinding
YES
NO
UNCLEAR
Incomplete outcome data
YES
NO
UNCLEAR
Selective outcome
reporting
YES
NO
UNCLEAR
Other sources of bias
YES
NO
UNCLEAR
Sequence generation
Adequate

Random number table

Computer generated list

Coin tossing
Shuffling cards /
envelopes
minimization


Not adequate
Generated by:
 Odd/even date of birth
 Date/day of admission
 Hospital record number
Allocation by:
 Clinical judgement
 Participant’s preference
 Result of lab test
 Availability of intervention
Unclear
If insufficient
information about
sequence
generation!
Allocation concealment
Adequate



Central randomisation
(telephone, web-based,
pharmacy)
Sequentially numbered
drug containers of
identical appearance
Sequentially numbered,
opaque, sealed
envelopes
Not adequate

Open random allocation
schedule

Envelopes without
appropriate safeguards

Quasi-randomisation
Unclear
If insufficient
information about
sequence
generation!
(e.g. “sealed
envelopes”)
Blinding of intervention
• Participants (patients, clients)
• Care providers (doctors, nurses, teachers …)
• Outcome assessors
Blinding of intervention
Adequate



No blinding, outcome
(assessment) not
likely to be influenced
by lack of blinding
Blinding ensured and
unlikely to have been
broken
Either participants or
some personnel
unblinded, but unlikely
to introduce bias +
outcome assessment
blinded
Not adequate


No or incomplete
blinding and outcome
(assessment) likely to
be influenced by lack of
blinding
Blinding attempted but
likely that could have
been broken
Unclear

insufficient
information

Issue not
addressed in
the study
Incomplete outcome data
“Attrition” (drop-out): no data
• Withdrawal
• Do not attend follow-up appointment
• Failure to complete questionnaire / diaries
• Cannot be located (lost to follow-up)
• Decision by investigator to cease follow-up
• Data or records are lost
Incomplete outcome data
“Exclusion”:
data available, but excluded from analysis
• Participants found to be ineligible after enrolment
• An “as treated” (or per-protocol) analysis: participants
are only included if they received the intended intervention in
accordance with the protocol
Assessing risk of bias
Low risk of bias

Complete outcome data

Missing in both groups but reasons
are reported and balanced across
groups

Reason unlikely to be connected
with outcome (moved away)
High risk of bias

Difference in proportion of
incomplete data across groups
and related to outcomes (e.g.
adverse effects in experimental
group)

Differences in the reasons for
missing data (e.g. smoking
cessation)

“as treated” (per protocol)
analysis
Selective outcome reporting
=
selection of a subset of the variables recorded for inclusion in
publication,
on the basis of the results
For example:
• Omission of non-significant outcomes
• Choice of data for an outcome (e.g. osteoporosis)
• Choice of analysis (e.g. blood pressure)
• Reporting of subsets of data (e.g. sepsis)
• Under-reporting of data (e.g. only “not significant”)
Presentation in your review
“Risk of bias graph”
“Risk of bias summary”
Assessing risk of bias in NRS
•Selection bias
(how was group allocation?)
•Performance bias
(blinding, fidelity of interventions)
•Attrition bias
(completeness of sample & follow-up)
•Reporting bias
(selective outcome reporting)
•Confounding and adjustment
Confounding
Comparison intervention - control
Intervention versus
control
?
?
Imbalance in
prognostic factors
Difference in
outcome
Confounding
Association between 2 factors
Presence of risk
factor
Occurrence of
outcome
Confounding factor
Confounding & adjustment
• At the stage of protocol: list potential confounding factors
• Identify the factors the authors have considered and
omitted
• Assess balance between groups at baseline
• What did authors do to control for confounders (matching,
restricting to subgroups, stratification, regression modelling)
Tool for NRS
Downs and Black instrument
(J Epidemiol Community Health 1998;52:377-84)
27 items:
• Reporting (10)
• External validity (applicability) (3)
• Internal validity - bias (7)
• Internal validity – confounding (6)
• Power (1)
Downs and Black
instrument
http://www.nccmt.ca/
registry/view/eng/9.html
Tool for NRS
Newcastle-Ottawa Scale (NOS)
(Wells 2008)
8 items covering 3 perspectives:
•Selection of study groups
•Comparibility of groups
•Ascertainement of exposure (case-control) or
outcome (cohort)
http://www.ohri.ca/programs/
clinical_epidemiology/oxford.asp
Diagnostic studies
QUADAS tool
Whiting BMC Medical Research Methodology 2003;3:25
Cochrane version: 11 items (out of 14 original)
Diagnostic Test Accuracy Working Group: handbook
http://srdta.cochrane.org/
handbook-dta-reviews
Other risk of bias assessment
tools
SIGN (Scottish Intercollegiate Guidelines Network)
http://www.sign.ac.uk/methodology/
checklists.html
In summary
• Risk of bias assessment is essential for systematic reviews
• For RCT: use the Cochrane tool
• For NRS:
• Higher risk of bias (selection bias & reporting bias)
• Use the appropriate tool to assess risk of bias
• Consider how potential confounders are addressed