Systematic Reviews:
The Potential of
Different Methods
Steve Higgins
Henry Potts
James Thomas
Geoff Wong
ESRC Research Methods Festival
Oxford, 5 July 2012
(1)
Introduction
• Aims
– To outline major approaches to systematic
reviewing and research synthesis
– To demystify what on first glance looks like a
bewildering set of overlapping approaches
– To examine some of the assumptions that
underpin these approaches
– To discuss how these approaches might help in
my (your!) situation
(2)
Introduction
• Session structure:
– 20 Minute presentation + 10 minutes discussion
/ clarification of four approaches to synthesis
– Coffee break after the 3rd / 4th presentation
– Discussion in pairs
– Questions + panel discussion
(3)
What is a systematic review?
• Systematic: ‘done or acting according to a
fixed plan or system; methodical’
• Review: ‘a critical appraisal of a book, play,
or other work’
(OED)
(4)
What is systematic research
synthesis?
• Synthesis: ‘The process or result of building up
separate elements, especially ideas, into a connected
whole, especially a theory or system’ (OED)
• Not just a report of the findings of the individual studies in
a review
• Involves a transformation of the data from primary studies
in some way
• Synthesis of the findings of all the included studies in order
to answer the review question
(5)
Aims of synthesis reviews
• Answer questions: what do we want to know
and how can we know it?
• Bring together and ‘pool’ the findings of
primary research (i.e. clarify what we know)
• Any question so potentially any type of
evidence
• Driven by review users to answer relevant
questions in relevant ways
(6)
Introduction
• We will look at four main approaches to
reviewing:
– Meta-analysis
– The synthesis of ‘qualitative’ research
– Realist review
– Meta narrative review
• Each have different disciplinary
backgrounds and underlying assumptions
• They all bring together research using two
main logics (7)
Aggregation in reviews
Aggregation refers to
‘adding up’ (aggregating)
findings from primary
studies to answer a
review question…
… to indicate the
direction or size of effect
… and our degree of
confidence in that finding
(8)
Configuration in reviews
Configuration involves
the arrangement
(configuration) of the
findings of primary
studies to answer the
review question….
… to offer a meaningful
picture of what research
is telling us
... across a potentially
wide area of research
(9)
•
Most reviews contain elements of both,
but some patterns are common
Aggregation
(10)
Configuration
Approach to theory
Test
Research question
Closed
Concepts
Secure prior to review
Emergent in review
Timing of methods
decisions
A priori
Iterative / emergent
Search for studies
Representative /
unbiased / exhaustive
Quality appraisal
Avoiding bias
Explore
Generate
Open
Sufficient / purposive
Richness of data
Methods for the synthesis
of qualitative research
James Thomas1
ESRC Research Methods Festival
Oxford, 5 July 2012
The EPPI-Centre is part of the Social Science Research Unit at the Institute of
Education, University of London, UK1
(11)
Synthesis of qualitative research:
outline
• What can qualitative research tell us?
• Current developments/approaches in
qualitative review methods
• Examples of syntheses of qualitative
research
(12)
What can qualitative research tell us?
(13)
Qualitative research…
• aims to provide an in-depth understanding of
people’s experiences, perspectives and histories
in the context of their personal circumstances or
settings…*
• with the use of unstructured methods which are
sensitive to the social context of the study*
• the capture of data which are detailed, rich and
complex…*
• is generalisable on the basis of theory not
statistical probability **
* Spencer et al (2003); ** Popay (2006)
(14)
Methods used include*…
•
•
•
•
•
•
exploratory interviews
focus groups
observation
conversation
discourse and narrative analysis
documentary and video analysis
*Spencer et al (2003)
(15)
Is it simply the analysis of textual
data?
“Increasingly, the terms ‘quantitative research’
and ‘qualitative research’ came to signify
much more than ways of gathering data;
they came to denote divergent assumptions
about the nature and purposes of research
in the social sciences”
Bryman (1988) Quantity and Quality in Social Research. London: Unwin
Hyman. p.3.
(16)
Qualitative research can*
• help to define policy questions
• look in detail at how a programme or trial
was actually implemented
• help to determine appropriate outcome
measures by looking at ‘subjective’
outcomes
• help to clarify what counts as effective or
successful
(17)
* Davies (2000) ‘Contributions from qualitative research’ in Davies et
al (eds) What works? Evidence-based policy and practice in public
services. Bristol: Policy Press. Cited in *Spencer et al (2003). p.36
…continued
• identify and explore unintended consequences
• contribute to service delivery and policy
development by describing processes and
contexts
• inform and illuminate quantitative studies, e.g. by
contributing to the design of structured
instruments, assessing the fairness of
comparisons in experimental studies, or
unpacking variation within aggregated data
(18)
Davies (2000) ‘Contributions from qualitative research’ in Davies et
al (eds) What works? Evidence-based policy and practice in public
services. Bristol: Policy Press. Cited in *Spencer et al (2003). p.36
Current developments in
synthesising qualitative research
• Growing recognition of the value of
qualitative research to inform policy &
practice
• Methods developing to facilitate this
(19)
(20)
Try: http://www.wordle.net/
Methodological study
• We searched for methodological papers
concerning the synthesis of concepts or
theories
– Purposive, rather than systematic search
– Reference ‘chasing’
– Google scholar search
– Handsearching key journals
• 203 papers retrieved
• 9 distinct methods for synthesis
(21)
Methods for the synthesis of
qualitative research identified
•
•
•
•
•
•
•
•
(22)
Meta narrative synthesis
critical interpretive synthesis
meta-study
meta-ethnography
grounded theory
thematic synthesis
framework synthesis
ecological triangulation
Many methods: similar or different?
• Examined the methods across different
dimensions:
–
–
–
–
–
–
Epistemology
Approach to quality assessment
Attitudes towards problematizing the literature
Use of review question
How similar / different the included studies were
Characteristics of the synthetic product
• Found they fell into two broad camps: ‘idealist’
and ‘realist’
(23)
‘Realist’ approaches
Purpose
• To answer a policy relevant question
Methods
• Qualitative/ quantitative data analysed with
qualitative/ quantitative methods
• Searching linear or iterative
• Quality assessment of study methods
Product
• Directly applicable to policy and practice
decisions
(24)
‘Idealist’ approaches
Purpose
• To explore and construct concepts from the data
• For generating theory
Methods
• Qualitative data analysed with qualitative methods
• Searching iterative
• Qualitative assessment of study content > method
Product
• Complex, requiring further interpretation before
being used for policy or practice
(25)
Conclusions
• Operationally, many methods are very
similar
• Underlying principles differ
• Product differs in terms of the amount of
additional interpretation required. This may
reflect…
• (Barnett-Page and Thomas (2009))
(26)
Perspectives on how evidence
informs the policy process
Knowledge-driven model;
Problem-solving (policy
driven) model*
Idealist
Realist
Less suited
More suited
More suited
Less suited
(either researcher
disseminates research or
policy-maker commissions /
requests specific research)
Interactive model;
Enlightenment model*
(ongoing interactions
between researcher and
policymaker; ‘gradual
sedimentation of insights…’)
*Weiss (1979)
(27)
Two examples
• All this is a bit abstract… what do
systematic reviews of qualitative research
look like?
• Two examples:
– The ‘fine grain’ of a synthesis
– ‘Mixed methods’ synthesis
(28)
Example 1: Informing policy about
the ‘obesity epidemic’
What are children’s
views about:
• the meanings of obesity
or body size, shape or
weight (including their
perceptions of their own
body size)?
• influences on body size?
• changes that may help
them to achieve or
maintain a healthy
weight?
(29)
Views = attitudes,
opinions, beliefs,
understanding or
experiences
As distinct from
health/weight
status, behaviour,
factual knowledge
Rees et al 2009
‘Thematic synthesis’
• Similar to other methods of synthesising
qualitative research (e.g. ‘meta-ethnography’)
•
•
•
•
Source data = text (documents)
Source material = conceptual
Key method = translation
Final product = interpretation
• (Thomas and Harden 2008)
(30)
Stages of thematic synthesis
• Stages one and two: coding text and
developing descriptive themes
– Identifying the ‘findings’
– Line-by-line coding
– Developing descriptive themes
• Stage three: generating analytical themes
– In the light of the review question
(31)
Screenshot – line-by-line coding
(32)
Screenshot – descriptive codes
diagram
(33)
Example 2. adding insights to other
existing reviews
• How might a systematic review of qualitative
research support or add insight to a metaanalysis?
• Product of qualitative synthesis can form the
conceptual framework within which heterogeneity
can be explored in a meta-analysis
(34)
Review Question
e.g. ‘What is known about the barriers to, and facilitators of, outcome X (e.g. physical
activity) amongst population A (e.g. young people
Searching, screening mapping and user
involvement
Synthesis 1
Synthesis 2
e.g. Statistical meta-analysis of trials
a) Data extraction
b) Quality assessment
c) Effect sizes pooled
d) Narrative synthesis
e.g. Thematic synthesis of qualitative studies
a) Data extraction
b) Quality assessment
c) Descriptive themes
d) Analytical themes
Addresses sub RQ e.g. ‘which interventions are
effective?’
Addresses sub RQ e.g. ‘what are people’s perspectives
and experiences ?
Synthesis 3
Driven by overall review question
Integration of separate syntheses e.g.
a) Matches, mis-matches and gaps
b) Hypotheses generated in synthesis 2 tested
amongst trials in synthesis 1
Cross study synthesis: Findings
Children’s views
(36)
Outcome evaluations
Recommendation for
interventions
Good quality
Other
Do not promote fruit and
vegetables in the same way
No soundly
evaluated
interventions
No other
interventions
identified
Brand fruit and vegetables
as an ‘exciting’ or childrelevant product, as well as
a ‘tasty’ one
5 soundly
evaluated
interventions
identified
5 other interventions
Reduce health emphasis in
messages to promote fruit
and vegetables particularly
those which concern future
health
5 soundly
evaluated
interventions
identified
6 other interventions
identified
Cross study synthesis: an example of
sub-group analysis
Increase (standardised portions per day) in vegetable intake
across trials
0.6
0.5
0.4
Portions
Little or no
emphasis on
health messages
0.3
0.2
0.1
0
e
dl
ar
W
i
or
qu
i
L
y
H
r
en
rs
de
n
A
on
R
ds
ol
n
ey
Study
(37)
ld
Au
ld
Au
(b
ki
ws
o
an
)
r
Ba
rry
Pe
This method of synthesis across study
types:
• preserves the integrity of the findings of the
different types of studies
• allows us to integrate ‘quantitative’
estimates of benefit and harm with
‘qualitative’ understanding from people’s
lives
• allows the exploration of heterogeneity in
ways in which it would be difficult to
imagine in advance
– protects against ‘data dredging’
(38)
Assumptions / theoretical models
• Mixed methods synthesis can draw on thinking
relating to mixed methods in primary research
• In terms of a frequently used taxonomy of paradigm
stances (Creswell 2011) e.g.
– Incommensurability (cannot be mixed)
– A-paradigmatic (can be mixed and matched in different
ways)
– Complementary strengths (not incompatible, but are
different and should be kept separate)
– Dialectic (paradigms are important in different ways
leading to useful tensions & insights)
– Alternative paradigm (‘mixed methods’ paradigm;
foundation in e.g. pragmatism)
(39)
Conclusions
• Qualitative research offers valuable insights
that can inform policymaking
• Methods for synthesising qualitative
research are still being developed
• Selection of a particular method depends on
the type of answer required (and the means
by which the review will inform policy)
(40)
References
•
•
•
•
•
(41)
Barnett-Page E, Thomas J (2009) Methods for the synthesis of qualitative
research: a critical review. BMC Medical Research Methodology, 9:59.
doi:10.1186/1471-2288-9-59. (http://www.biomedcentral.com/14712288/9/59)
Creswell J (2011) Mapping the Developing Landscape of Mixed Methods
Research in Teddlie C, Tashakkori A: SAGE Handbook of Mixed Methods
in Social & Behavioral Research. New York: Sage.
Gough D, Oliver S, Thomas J (2012) An Introduction to Systematic
Reviews. London: Sage
Thomas J, Harden A (2008) Methods for the thematic synthesis of
qualitative research in systematic reviews. BMC Medical Research
Methodology, 8:45 doi:10.1186/1471-2288-8-45
(http://www.biomedcentral.com/1471-2288/8/45)
Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G,
Kavanagh J (2004) Integrating qualitative research with trials in systematic
reviews: an example from public health. British Medical Journal 328:
1010-1012. (http://www.bmj.com/cgi/content/full/328/7446/1010)
Systematic Reviews:
The Potential of
Different Methods
Steve Higgins
Henry Potts
James Thomas
Geoff Wong
ESRC Research Methods Festival
Oxford, 5 July 2012
(42)
Summary
(43)
Heterogeneity
Aggregation /
configuration
Meta-analysis
A problem in fixed effect
Allowance made for in
random effects
For ‘exploration’
Fixed & random effects
models are pure aggregation
Sub-group & metaregression both aggregate &
configure
Synthesis of qualitative
research
Expected. Key challenge is
judging when contexts /
concepts are too dissimilar
to be compared.
Similar concepts are
‘aggregated’; most synthesis
involves configuration
Realist review
Necessary to obtain different Aggregates within CMO
contexts etc
configurations; configures
between them
Meta narrative review
Necessary to obtain diversity Aggregation within
in paradigms
paradigms; configuration
between them
Your turn
Research
question
Research
question 1
Research
question 2
(44)
Heterogen- Metaeity?
analysis
Qualitative
synthesis
Realist
review
Meta
narrative
review