Version 1. 23 December 2010
Chapter 6: Supplemental Guidance on
Selecting a Method of Qualitative Evidence Synthesis, and
Integrating Qualitative Evidence with Cochrane Intervention
Reviews
This chapter should be cited as: Noyes J & Lewin S. Chapter 6: Supplemental
Guidance on Selecting a Method of Qualitative Evidence Synthesis, and Integrating
Qualitative Evidence with Cochrane Intervention Reviews. In: Noyes J, Booth A, Hannes
K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for
Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions.
Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group,
2011. Available from URL http://cqrmg.cochrane.org/supplemental-handbook-guidance
Key points

This document provides supplementary guidance to section 20.3.2.4 of the Cochrane
Intervention Handbook.

This Guidance is primarily for those authors wishing to synthesise qualitative
evidence to inform, enhance or extend a Cochrane intervention review.

The methods for qualitative evidence synthesis are still evolving and there are a
number of unanswered questions regarding which synthesis methods are most
appropriate:
o for a particular kind of review (e.g. to inform, enhance or extend a Cochrane
review)
o for particular types of studies (ethnographies, phenomenological studies etc.)
o for particular review questions (e.g. experiences of the intervention,
implementation, effectiveness, and broader issues beyond effectiveness).

There are blurred boundaries between the different qualitative synthesis methods,
with a proliferation of different approaches.

Qualitative evidence syntheses tend to take an iterative approach to the sampling,
extraction and synthesis of qualitative data. To some extent, this differs from the
more linear processes underlying the Cochrane intervention reviews of quantitative
data.

The choice of synthesis method should be informed by:
o whether the research question is more exploratory in nature or highly
focused and specific
o the nature of the evidence identified, for example whether evidence is
largely descriptive or highly theorized or conceptual
1
Version 1. 23 December 2010
o whether there is an established theoretical framework or model in place
for the phenomenon that is being explored. Where a well accepted
explanatory model is available it may not be necessary to develop one
through the synthesis and an aggregative approach to data synthesis may
be sufficient. In contrast, where no accepted explanatory framework exits,
a more inductive synthesis approach would be justified
o The expertise of the team and available resources.
Introduction
Chapter 20 of the Cochrane Intervention handbook outlines four ways that qualitative
research can potentially contribute to Cochrane Intervention reviews, including:
o Informing reviews by using evidence from qualitative research to help define and
refine the question and to ensure the review includes appropriate studies and
addresses important outcomes;
o Enhancing reviews by synthesising evidence from qualitative research identified
whilst looking for evidence of effectiveness;
o Extending reviews by undertaking a search to specifically seek out evidence from
qualitative studies to address questions directly related to the effectiveness
review; and
o Supplementing reviews by synthesising qualitative evidence within a stand-alone,
but complementary, qualitative review to address questions on aspects other than
effectiveness.
Although there are growing numbers of published high quality qualitative evidence syntheses, at
present the Cochrane Collaboration is not able to support stand alone ‘supplementary’ qualitative
reviews. Methodological developments within the Collaboration are currently constrained by the
focus on the ‘effectiveness’ of interventions and the associated limitations of the software
(RevMan), which has been designed as a platform for quantitative intervention reviews.
Currently review groups and authors may opt to use qualitative evidence to inform, enhance or
extend a Cochrane intervention review. It is up to individuals to decide which titles and topics
are most likely to gain significant additional value and utility by the addition of a qualitative
evidence synthesis. See chapter 3 for further guidance on question development and application
of qualitative evidence.
Choosing an appropriate synthesis method
2
Version 1. 23 December 2010
As outlined in section 20.3.2.4 of the Cochrane Intervention handbook, the choice of method
for inclusion of qualitative evidence in a Cochrane intervention review will depend on a number
of factors, including the:
• Type and scope of the review and review question(s);
• Pool of available evidence and the extent to which there are existing theoretical or conceptual
frameworks within this literature;
• Expertise of the team, and;
• Available resources.
In the Cochrane Intervention handbook, we previously recommended that any high quality
method of qualitative evidence synthesis may be used that is best suited to the type of
intervention review. In this additional guidance we signpost authors to named approaches that
may be appropriate to use for particular kinds of questions and evidence. The choice of approach
rests with the authors, but the Cochrane Qualitative Research Methods Group is happy to advise.
Different types of synthesis approaches
Methodologists tend to define and organise synthesis approaches on a spectrum from
integrative/aggregative/summative to interpretive and theory generating, as summarised
in Table 1.
Table 1: Different approaches to evidence synthesis (Adapted from Dixon-Woods et al.
2005b)
Integration/Aggregation/Summation
Interpretation and theory development
Aim
To assemble and pool data
Similarities and
differences
May use techniques such as meta-analysis, and
require a basic comparability between
phenomena so that the data can be aggregated for
analysis.
To develop concepts and theories that integrate
the themes described in the primary qualitative
studies
In an interpretive review, concepts are generally
not specified in advance of the synthesis.
Rather, these emerge as a product of the
interpretive analysis.
Their defining characteristics are a focus on
summarising data, and an assumption that the
concepts (or variables) under which those data
are to be summarised are largely secure and well
specified.
The output of the synthesis is not aggregations
of data, but theory that is grounded in the data
from the studies included in the review.
Key concepts are defined at an early stage in the
review and form the categories under which the
data from empirical studies are to be
summarised.
Examples
Cochrane intervention reviews of effect
JBI QARI approach of meta-aggregation
(Pearson 2004)
Meta-ethnography (Noblit and Hare 1988)
Critical Interpretive synthesis (Dixon-Woods et
al. 2006)
Realist review (Pawson et al. 2005)
3
Version 1. 23 December 2010
Factors to consider when selecting an appropriate qualitative evidence
synthesis method to inform, enhance or extend a Cochrane Intervention
review
Key messages:
 In choosing an evidence synthesis method it is important to consider what value
would be added by the method in terms of informing, enhancing or extending a
Cochrane review.
 The question(s) that the synthesis aims to address should inform the choice of
synthesis method – see figure 1 for a decision flow chart. If the synthesis aims to
develop new theory regarding, for example, why an intervention did not work, more
interpretive or constructivist approaches such as meta-ethnography may be
appropriate (Pound et al. 2005). The outputs of reviews using these methods tend to
be more conceptual, and perhaps less useful in answering specific practice questions
(Barnett-Page & Thomas 2009). In contrast, if the synthesis aims to define or refine a
Cochrane review question by summarizing the range of views expressed by
consumers regarding a health issue or intervention, more aggregative approaches such
as thematic analysis or meta-aggregation may be suitable.
 The type of data available for the synthesis should also inform the choice of synthesis
approach. For example, if the available qualitative studies include only thin
description then it may not be useful to attempt to use meta-ethnography. Metaethnography is better suited to interpretation than to aggregation and relies on primary
studies that include thick description. Similarly, synthesis approaches such as
thematic analysis would be more suitable than meta-ethnography if the evidence
synthesis was focused on aggregating the findings of process evaluations conducted
alongside trials included in a Cochrane review.
 Familiarity with the evidence synthesis method and its underlying techniques is
essential. If the review team has not used the synthesis method before, training and
support should be sought.
In summary, the evidence synthesis method chosen needs to be ‘fit for purpose’, i.e.
suitable for the research question; the available primary study data; and the review team’s
expertise and epistemological standpoint.
Figure 1. Decision flow chart
4
Version 1. 23 December 2010
Decision to conduct a qualitative evidence
synthesis
Purpose of the additional
qualitative synthesis
To aggregate / summarise /
integrate qualitative data to
address specific questions
in relation to a Cochrane
intervention review
Thematic analysis without
theory generation
Meta-aggregation
Meta-summary
Product
Aggregated findings from
source papers
To interpret synthesised qualitative
evidence and develop explanatory
theory or models
Meta-ethnography
Thematic analysis with theory
generation
Grounded theory
Primarily to integrate and interpret
qualitative and quantitative evidence
within a single approach or
integrated model
Can be used to develop
explanatory theory
Realist review
EPPI approach
Narrative synthesis
Bayesian synthesis
Critical interpretive synthesis
Product
Explanatory theory, analytical or
conceptual framework or
interpretative framework/ mechanism
Qualitative evidence synthesis approaches that ‘fit’ with the
Cochrane context.
Table 2 outlines the range of approaches to qualitative evidence synthesis viewed by the
Cochrane Qualitative Research Methods Group as most useful within the context of
Cochrane Intervention reviews
5
Version 1. 23 December 2010
Table 2: Potential qualitative evidence synthesis methods that ‘fit’ with a Cochrane Intervention review
Approach
Meta-ethnography
Adapted from primary method of
data analysis
Description/
Theoretical
framework
or
assumptions
Reciprocal translational analysis
(RTA) identifies key themes in
each study, then seeks to translate
these into context of each other
study. Themes with the best overall
fit/explanatory power are adopted.
The approach may then distil these
translations into a higher order
interpretation, which may be a "line
of argument" synthesis (a broader
picture of the phenomenon) or a
refutational synthesis, which
examines contradictions between
the findings of individual studies.
(Dixon-Woods et al. 2005b).
Synthesis
approach
Application
alongside
Cochrane
reviews
Thematic analysis
A variety of thematic
approaches adapted from
primary method of data
analysis
Identification of major /
recurrent themes in the
studies identified; summary
of findings of primary
studies under these
thematic headings
(Dixon-woods et al.
2005b).
A group of approaches all
use thematic analysis as the
basis for, or as part of, the
synthesis.
Interpretive
Interpretive / Integrative
Likely to be most useful in
supplementing, informing and
extending Cochrane reviews (see
introduction for an explanation of
Likely to be useful in
informing, enhancing,
extending and
supplementing Cochrane
Meta-aggregation
The web-based Qualitative
Assessment and Review Instrument
(QARI) includes a number of
processes that lead to the
development of synthesized findings
from the included studies:
• Translating the results from each
study into ‘QARI findings’ – a
conclusion reached by the
researchers and presented as a
metaphor or theme. These QARI
findings are illustrated using text
from the papers as quotes, or by
summarizing illustrative text.
• QARI findings are then grouped
into categories and combined to
create synthesized themes. These, in
turn, allow the recommendations for
practice to be generated. (Pearson
2004)
Integrative
Likely to be useful in informing,
enhancing, extending and
supplementing Cochrane reviews
Grounded theory
A variety of grounded
approaches adapted
from primary method
of data analysis
Constant comparative
method identifies
patterns and
interrelations in
primary data.
Sampling responds to
analysis, until
theoretical saturation
reached. (Dixon woods
2005a)
Interpretive
Likely to be useful in
informing, enhancing,
extending and
supplementing
6
Version 1. 23 December 2010
Available
guidance
Examples
Published
examples
within a
Cochrane
context
these different purposes)
Noblit & Hare 1988. More explicit
guidance in the application of this
method still needs to be developed.
There are now many published
examples, including: Campbell et
al. 2003; Merten et al. 2010; Munro
et al. 2007; Pound et al. 2005;
Schumm et al. 2010
Munro et al’s review on TB
adherence (2007,2008) was
undertaken to explore questions
raised by a Cochrane review on
directly observed therapy.
A review is underway on perceived
impacts and experiences of peer
support in chronic disease
(Embuldeniya 2010). This will
extend the update of a Cochrane
review on peer support strategies
for people with chronic diseases
(Doull et al. 2005).1
reviews
There are multiple
approaches (Pope et al.
2007). Some examples are
listed below.
Carlsen 2007
McInnes and Arskie 2004
Noyes and Popay 2007
Noyes and Popay (2007)
provide a worked example
of how qualitative synthesis
may be used to enhance,
extend and supplement a
Cochrane review on
directly observed therapy.
McInnes et al (2004)
examined the views,
preferences and
experiences of older people
in relation to falls
prevention strategies,
alongside a Cochrane
effects review on falls
prevention.
The Joanna Briggs Institute
Reviewer’s Manual, 2008 Edition
SUMARI User guide, 2008, Version
4.0
Briggs and Flemming 2007
Lamb et al. 2008
No published examples of the use of
this approach within the Cochrane
context.
Cochrane reviews
Glaser and Strauss
1967; Strauss and
Corbin 1998
Eaves 2001
Finfgeld 1999
Kearney 2001
No published
examples of the use of
this approach within
the Cochrane context.
1
A number of titles and protocols are in development for qualitative reviews within the Cochrane framework, including the following:
Protocol: Lins S et al. Telephone counselling for informal carers of people with dementia. Cochrane Dementia & Cognitive Improvement Group.
Protocol: Leiknes KA et al. Electroconvulsive therapy for depression. Cochrane Depression, Anxiety and Neurosis Group.
Title: Cracknell J. Factors which impact on the use of weaning protocols for reducing the duration of mechanical ventilation in adults and children: a
qualitative evidence synthesis. Cochrane Airways Review Group.

Title: Cassidy T et al. Psychosocial and cultural interventions to reduce alcohol consumption during lactation. Cochrane Drugs and Alcohol Group.



7
Version 1. 23 December 2010
Strengths and weaknesses of the different approaches to qualitative
evidence synthesis in the context of Cochrane reviews
Dixon-Woods et al. (2005b) and Pope et al. (2007) provide an overview of the
comparative strengths and challenges of different methods for evidence synthesis. This is
summarized in Table 3 below. Key points to note include:
 There are increasing numbers of published high quality qualitative and mixed method
syntheses, which provide descriptions of methods and processes used.
 It is particularly important that reports of reviews within a Cochrane context include a
detailed description of how the synthesis methods were applied; the key analytic and
methodological choices made; and the challenges experienced. This will improve the
transparency of the review process by providing an ‘audit trail’ for the reader.
 Approaches that aim to produce higher order interpretations of included evidence
require considerable expertise in qualitative methods; may also require a greater
investment of resources; and may not be suitable for syntheses in which large
numbers of primary studies need to be included.
 Most approaches do not provide detailed guidance on the sampling of studies for
inclusion in the review or on how the quality of included studies should be appraised.
Review authors will need to select from among the range of quality assessment tools
available, bearing in mind the strengths and weaknesses of the different quality
appraisal approaches (see Chapter 6).
8
Version 1. 23 December 2010
Table 3: Strengths and challenges of different methods for qualitative evidence synthesis (Dixon-Woods et al. 2005b; Pope et al. 2007:
Briggs and Flemming 2007)
Approach
Strengths
Meta-ethnography
Thematic analysis
Meta-aggregation
Grounded theory
Systematic approach combined
with the potential for preserving
the interpretive properties of the
primary data.
Allows clear identification of
prominent themes, and organised and
structured ways of dealing with a large
volume of literature under these
themes.
Can accommodate studies using a
wide range of methodologies and
philosophical perspectives
The generation of higher order themes
as a means of synthesis encourages
reflexivity on the part of the reviewer
while preserving the interpretive
properties of the underlying data.
Provides a transparent and auditable
pathway for the synthesis process
It allows some flexibility by
accommodating large volumes of
literature
Challenges
Requires high level of expertise
Grounded theory, and the notions of
theoretical saturation and theoretical
sampling, also offers a means of limiting
the number of papers that need be
reviewed, especially where the emphasis
is on conceptual robustness rather than
on completeness of data.
Is solely a means of synthesis
Thematic analysis can be either data
driven – driven by the themes
identified in the literature itself - or
theory driven – oriented to evaluation
of particular themes through
interrogation of the literature.
Includes all findings from a paper
regardless of whether they support a
specific viewpoint or whether they
are supported by direct quotes,
therefore can result in large numbers
of findings
It is demanding and laborious,
and might benefit from the
development of suitable software
or the use of existing software
packages
The failure of much writing on
thematic analysis to distinguish
adequately between these two
approaches has resulted in a lack of
transparency
Finding are rated based on linkage
with the text i.e. unequivocal,
credible, unsupported
The process of qualitative
synthesis cannot be reduced to a
set of mechanistic tasks, and
meta-ethnography thus runs into
There is lack of clarity about exactly
what thematic analysis involves and
the processes by which it can be
achieved; for example, there is a lack
of explicitness about procedures and
Offers no guidance on sampling
or appraisal
Progressing single findings through
categorisation to synthesised
findings
Does not generate third order
Requires a high level of expertise
As an interpretive method it inherently
lacks transparency
No advice is offered on how to appraise
studies for inclusion in a review
There are several important
epistemological issues to be resolved,
including the status of the accounts
offered in the studies and how to deal
with the varying credibility of these
accounts
Tensions exist between grounded theory
communities leading to development of
9
Version 1. 23 December 2010
the usual problems of
transparency
No steer on which order the
papers should be synthesised for
reciprocal translational analysis.
Other difficulties arise when a
large number of reports need to
be synthesised, because
reciprocal translational analysis
(RTA) appears to be most
suitable for small stable sets of
papers. Also, because RTA
provides summaries in terms that
have already been used in the
literature, there is a danger that it
will tend towards conservatism.
aims, including the extent to which
thematic analyses should be
descriptive or interpretive.
interpretations
convergent approaches
It is unclear whether the structure of
the analysis should reflect the
frequency with which particular
themes are reported, or whether the
analysis should be weighted towards
themes that appear to have a high level
of explanatory value.
If thematic analysis is limited to
summarising themes reported in
primary studies, it offers little by way
of theoretical structure within which to
develop higher order thematic
categories beyond those identified
from the literature.
10
Version 1. 23 December 2010
Using principles or adapting an approach
Whilst the CQRMG have made some recommendations in this guidance about
appropriate qualitative evidence synthesis methods to use within a Cochrane context, we
acknowledge that methods continue to be adapted for different contexts and uses.
For example Greenhalgh et al.’s realist review of school feeding programmes
(Greenhalgh et al. 2007) was undertaken to enhance a Cochrane review of school feeding
for improving health outcomes among disadvantaged elementary school children
(Kristjansson et al. 2007).
Although realist review methodology is designed to incorporate different types of
evidence, Greenhalgh et al. adapted the approach to only synthesise evidence from
process evaluations conducted alongside included trials in the published Cochrane
intervention review. They then applied the theoretical principles and processes of realist
synthesis and created context, mechanism and outcome chains to consider ‘what worked
for whom and in what contexts’. The product of this review provides a clear explanation
about the theoretical adequacy of interventions and a theory as to why the poorest
children did not benefit sufficiently from the intervention. Review authors may of course
work independently of Cochrane and choose this type of adapted approach and, like
Greenhalgh et al., publish the product in a high ranking journal, and include an explicit
link with the Cochrane intervention review.
CQRMG members will develop and amend this guidance as and when
qualitative synthesis methods evolve and more high quality exemplars are available
within a Cochrane framework.
Approaches to incorporating or juxtaposing qualitative evidence syntheses
with or alongside Cochrane intervention reviews of effects
At present authors who wish to enhance or extend a Cochrane intervention review are
compelled to undertake an approach that involves a separate qualitative evidence
synthesis rather than a truly integrated mixed-method approach to managing different
types of evidence. In the context of a Cochrane intervention review, the qualitative
evidence synthesis will at some stage need to be integrated with the quantitative
intervention effects review to explore and explain issues of interest, such as the
heterogeneity of trial results. Section 20.3.2.5 of the Cochrane intervention handbook
outlines two broad approaches that can be used to integrate qualitative and quantitative
findings:
1. Multi-level, simultaneous syntheses
Qualitative evidence (synthesis 1) and quantitative evidence (synthesis 2) are each
conducted as separate streams or separate, but linked, reviews. The product of each
synthesis is then combined (synthesis 3). This approach mirrors the EPPI approach for
integrating qualitative and quantitative evidence. For example, in a review of barriers and
11
Version 1. 23 December 2010
facilitators to healthy eating, Thomas et al. conducted both a synthesis of effectiveness
findings from outcome evaluations (2004) and a synthesis of children’s views studies
(2003) (See figure 2). A ‘cross study synthesis’ drew together the findings from these two
reviews by comparing interventions studied through outcome evaluations to the key
findings from the children’s views studies. In doing so, the cross study synthesis
identified both areas where issues raised in the views studies had been addressed by
outcome evaluations and areas where gaps between views and outcomes studies existed
(Thomas et al. 2003). Work underway by Leiknes et al. on integrated qualitative and
effectiveness reviews on electroconvulsive therapy for depression will use a similar approach (see
footnote 2).
Research Question(s)
Cochrane
Intervention
Review
Qualitative
Evidence
Synthesis
Synthesis of Qualitative
and Quantitative Studies
Figure 2. EPPI approach multi-layered syntheses
2. Parallel syntheses
Qualitative evidence (synthesis 1) and quantitative evidence (synthesis 2) can be
conducted as separate streams or separate but linked reviews. These reviews might be
undertaken by the same team or by two separate teams and may be undertaken simultaneously or
sequentially. The qualitative synthesis findings can then be used in parallel and juxtaposed
alongside the quantitative synthesis findings to aid the interpretation of the synthesized
trials. (See for example Noyes & Popay 2007; and the work underway by Embuldeniya on peer
support in chronic disease – Table 2).
2.1 Adding a post-hoc parallel synthesis of qualitative evidence to a published
Cochrane intervention review.
12
Version 1. 23 December 2010
Many published Cochrane reviews now call for a synthesis of qualitative evidence to help
explain why an intervention has or has not worked, or to incorporate stakeholders’ views
about the intervention or its impacts (see Figure 3). When adding to a published
intervention review, the qualitative review authors have the benefit of knowing the
quantitative review outcomes or issues of interest at the outset.
Published
Cochrane
Intervention
Review
New
Qualitative
Evidence
Synthesis
Figure 3. Integrating a new synthesis of qualitative evidence with a published Cochrane
Intervention review.
For example, Noyes and Popay (2007) selected a published Cochrane intervention review
on Directly Observed Therapy and TB (Volmink and Garner 2006), and set out to use
qualitative evidence to help explain the complex variations in trial outcomes and identify
the facilitators and barriers to accessing and complying with the various types of directly
observed therapy tested. In this post-hoc analysis, the authors of the qualitative review
noted that the ways in which directly observed therapy was delivered was important in
understanding the effects on cure rates of a particular type of observed therapy. The
qualitative review also suggested that social and economic factors, as well as the
considerable physical side-effects of the treatment, affected both treatment seeking
behaviour for tuberculosis and treatment adherence (Noyes and Popay 2007; Noyes et al.
2008).
13
Version 1. 23 December 2010
If considering a post-hoc qualitative synthesis, and the qualitative team has no previous
connection with the published Cochrane quantitative intervention review or its authors, it
may be courteous to contact the original authors at the outset to signal your intention to
add a qualitative synthesis to the quantitative review. This may also provide impetus or
an additional opportunity for the quantitative team to update their review at the same
time.
In addition, the processes for publishing qualitative evidence syntheses alongside, or
integrated with, published Cochrane intervention reviews have yet to be worked out.
Therefore establishing a relationship with the quantitative review team may be important
if additional qualitative evidence needs to be integrated with, and published alongside,
another team’s review.
2.2
Working with a quantitative review team to inform, enhance or extend a new
Cochrane intervention review
The CQRMG is increasingly receiving enquiries from quantitative and qualitative teams
who wish to work together and undertake a new Cochrane intervention review with an
additional qualitative evidence synthesis in a concurrent process. With this approach,
both review teams could potentially work together as appropriate and use qualitative
evidence at various stages to aid understanding of quantitative outcomes and phenomena
(see Figure 4).
Cochrane
Intervention
Review
Qualitative
Evidence
Synthesis
14
Version 1. 23 December 2010
Figure 4. Undertaking quantitative and qualitative reviews as a concurrent and interactive
processes
Multiple level and parallel syntheses both require a separate systematic review of
evidence from qualitative studies, which at some stage is synthesized with, or juxtaposed
alongside, the synthesis of trials. Guidance on the conduct of narrative synthesis (Popay
et al, 2006), contains a toolkit for bringing together findings from different study designs
within different methods and approaches. Further methodological work is required on the
processes by which evidence from studies using different qualitative methods and
generating a range of types of evidence can be synthesized and combined with
quantitative findings on effect without compromising the need to minimize bias (Lucas et
al. 2007).
Resources
Many software packages designed for the analysis of primary qualitative evidence can be
used for synthesis of secondary published qualitative evidence. Examples are listed in
Table 5.
Table 5: Examples of available qualitative synthesis software packages
Produced by
Free to
download
Nvivo
QSR
International
www.qsrinternati
onal.com/
QARI
Joanna Briggs
Institute
www.joannabrigg
s.edu.au/services/
sumari_info.php
ATLAS. ti
ATLAS.ti
Scientific
Software
Development
www.atlasti.de
No
Yes
No
EPPI-Reviewer
EPPI Centre, Institute
of Education,
University of London
eppi.ioe.ac.uk/cms/De
fault.aspx?tabid=1913
&language=en-US
No
Publishing the qualitative evidence synthesis in the Cochrane library
Processes have yet to be developed for publishing the product of the qualitative evidence
synthesis and/or the product of the integrated quantitative and qualitative synthesis in the
Cochrane library. Additional guidance on this will follow as these processes are
established.
Where conducting a qualitative synthesis within a Cochrane context does not seem
appropriate
For some review teams, the limitations of synthesizing and integrating qualitative
evidence within a Cochrane context may appear too constraining to address fully the
proposed review questions. After careful consideration, authors may feel that it is more
appropriate to continue developing their review outside of a Cochrane context with less
15
Version 1. 23 December 2010
focus on the effectiveness of interventions and the freedom to use a mixed method
approach. Outside of a Cochrane context, a number of synthesis approaches allow
qualitative and quantitative evidence to be considered within one process rather than only
in parallel. For example, Critical Interpretive Synthesis (Dixon-Woods et al. 2006),
Realist Synthesis and the comparative case study approach (Agranoff and Radin 1991).
For information, these approaches are summarized in Additional Table 1.
16
Version 1. 23 December 2010
References
Agranoff R, Radin BA (1991). The comparative case study approach in public
administration. Research in Public Administration. 1: 203-31.
Arai L, Roen K, Roberts H, Popay J. (2005) It might work in Oklahoma but will it work
in Oakhampton? What does the effectiveness literature on domestic smoke detectors tell
us about context and implementation? Injury Prevention. 11(3): 148–151.
Barnett-Page E, Thomas J. (2009) Methods for the synthesis of qualitative research: a
critical review. BMC Med Res Methodol. 9: 59
Briggs M, Flemming K. (2007) Living with leg ulceration: a synthesis of qualitative
research. Journal of Advanced Nursing. 59(4): 319–328.
Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J. (2003)
Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of
diabetes and diabetes care. Social Science and Medicine. 56: 671-684.
Carlsen B, Glenton C, Pope C. (2007) Thou shalt versus thou shalt not: a meta-synthesis
of GPs' attitudes to clinical practice guidelines. Br J Gen Pract. 57(545): 971-8.
Dixon-Woods M, Kirk D, Agarwal S, Arthur T, Harvey J, Hsu R, Katbamna S, Olsen R,
Smith L, Sutton AJ, Riley R. (2005a) Vulnerable groups and access to health care: a
critical interpretive review London: NHS SDO R&D Programme
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. (2005b) Synthesising
qualitative and quantitative evidence: a review of possible methods. J Health Serv Res
Policy. 10: 45-53.
Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R,
Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ. (2006) Conducting a critical
interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC
Med Res Methodol. 26: 6:35
Doull MJ, O'Connor AM, Robinson VA, Tugwell P, Wells G. (2005) Peer support
strategies for people with chronic disease to enhance health and promote health equity.
Cochrane Database of Systematic Reviews. Issue 3.
Eaves YD. (2001) A synthesis technique for grounded theory data analysis. J Adv Nurs.
35(5): 654-63.
Finfgeld D. (1999) Courage as a process of pushing beyond the struggle. Qual Health
Res. 9: 803-814.
17
Version 1. 23 December 2010
Flemming K. (2010) Synthesis of quantitative and qualitative research: an example using
Critical Interpretive Synthesis. J Adv Nurs. 66(1): 201-17.
Glaser BG, Strauss AL. (1967) The discovery of grounded theory: strategies for
qualitative research. New York: Aldine de Gruyter.
Greenhalgh T, Kristjannson E, Robinson V. (2007) Realist review to understand the
efficacy of school feeding programmes. BMJ. 335: 858-861.
Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A. (2004)
Applying systematic review methods to studies of people's views: an example from
public health research. J Epidemiol Community Health. 58(9): 794-800.
Kearney M. (2001) Enduring love: a grounded formal theory of women’s experience of
domestic violence. Research in Nursing and Health. 24: 270-282
Kristjansson EA, Robinson V, Petticrew M, MacDonald B, Krasevec J, Janzen L,
Greenhalgh T, Wells G, MacGowan J, Farmer A, Shea BJ, Mayhew A, Tugwell P. (2007)
School feeding for improving the physical and psychosocial health of disadvantaged
elementary school children. Cochrane Database Syst Rev. (1):CD004676.
Lamb M, Buchanan D, Godfrey CM, Harrison MB, Oakley P. (2008) The psychosocial
spiritual experience of elderly individuals recovering from stroke: a systematic review.
International Journal of Evidence-based Healthcare. 6(2): 173-205.
Lucas PJ, Baird J, Arai L, Law C, Roberts HM. (2007) Worked examples of alternative
methods for the synthesis of qualitative and quantitative research in systematic reviews.
BMC Medical Research Methodology. 7: 4.
McCormack B, Dewar B, Wright J, Garbett R, Harvey G, Ballantine K. A (2006) Realist
synthesis of evidence relating to practice development: Final report to NHS Education
for Scotland and NHS Quality Improvement Scotland. NHS Quality Improvement
Scotland, Edinburgh.
McInnes E, Askie L. (2004) Evidence review on older people's views and experiences of
falls prevention strategies. Worldviews Evid Based Nurs. 1(1):20-37.
Merten S, Kenter E, McKenzie O, Musheke M, Ntalasha H, Martin-Hilber A. (2010)
Patient-reported barriers and drivers of adherence to antiretrovirals in sub-Saharan
Africa: a meta-ethnography. Trop Med Int Health. 15, Suppl 1:16-33.
18
Version 1. 23 December 2010
Munro S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. (2007) Adherence to
tuberculosis treatment: a qualitative systematic review of stakeholder perceptions. PLOS
Medicine. 4(7): e238.
Munro S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. (2008) Conducting a
meta-ethnography of qualitative literature: lessons learnt. BMC Medical Research
Methodology. 8:21.
Noblit G, Hare R. (1988) Meta-ethnography: synthesising qualitative studies. Newbury
Park, CA: Sage.
Noyes J, & Popay J. (2007) Directly observed therapy and tuberculosis: how can a
systematic review of qualitative research contribute to improving services? A qualitative
meta-synthesis. J Adv Nurs. 57(3): 227-43.
Noyes J, Popay J, Pearson A, Hannes K, Booth A. (2008) Qualitative research and
Cochrane reviews. In: Higgins JPT, Green S (eds). Cochrane Handbook for systematic
reviews of interventions. Wiley Blackwell.
Pawson R, Tilley N (1997) Realistic Evaluation. London: Sage Publication.
Pawson R, Boaz A (2004) Methods Briefing 3: Evidence-based Policy, Theory-based
Synthesis, Practice-based Reviews. Centre for Census and Survey Research, University of
Manchester, Manchester.
Pawson R, Greenhalgh T, Harvey G, Walshe K (2004) Realist Synthesis: An
Introduction. RMP Methods Paper 2/2004. Centre for Census and Survey Research,
University of Manchester, Manchester.
Pawson R, Greenhalgh T, Harvey G, Walshe K. (2005) Realist review--a new method of
systematic review designed for complex policy interventions. J Health Serv Res Policy.
10, Suppl 1:21-34.
Pawson R (2006). Evidence-Based Policy: A Realist Perspective. London: Sage.
Pearson A. (2004) Balancing the evidence: incorporating the synthesis of qualitative data
into systematic reviews. JBI Reports. 2(2): 45–64.
Pearson A, Wiechula R, Court A, Lockwood C. (2005). The JBI Model of Evidencebased Healthcare. Evidence in healthcare Reports 3 (8).
(www.joannabriggs.edu.au/pubs/systematic_reviews.php)
Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., et al. (2006)
Guidance on the Conduct of Narrative Synthesis in Systematic Reviews, Version 3.
19
Version 1. 23 December 2010
Available from j.popay@lancaster.ac.uk. A summary is available at:
http://www.york.ac.uk/inst/crd/Posters/Guidance%20on%20the%20conduct%20of%20na
rrative%20synthesis%20in%20systematic%20review.pdf
Pope C, Mays N, Popay J. (2007) Synthesizing qualitative and quantitative health
research. Oxford: Oxford University Press.
Pound P, Britten N, Morgan M, Yardley L, Pope C, Daker-White G, Campbell R. (2005)
Resisting medicines: a synthesis of qualitative studies of medicine taking. Social Science
& Medicine. 6: 133-155
Roberts K, Dixon-Woods M, Fitzpatrick R, Abrams K, Jones D. (2002) Factors affecting
uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative
evidence. The Lancet. 360 (9345): 1596-1599.
Schumm K, Skea Z, McKee L, N'Dow J. (2020) 'They're doing surgery on two people': a
meta-ethnography of the influences on couples' treatment decision making for prostate
cancer. Health Expect. 13(4): 335-49.
Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR. (2000) Bayesian methods in health
technology assessment: a review. Health Technol Assess. 4(38).
Strauss A, Corbin J. (1998) Basics of qualitative research: techniques and procedures for
developing grounded theory. 2nd ed. Thousand Oaks, Ca: Sage.
Sutton AJ, Abrams KR. (2001) Bayesian methods in meta-analysis and evidence
synthesis. Stat Methods Med Res. 10: 277–303.
Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh F.
(2004) Integrating Qualitative Research with trials in systematic reviews: an example
review from public health shows how integration is possible and some potential benefits.
BMJ 328: 1010-12
Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J.
(2003) Children and Healthy Eating: A systematic review of barriers and facilitators.
London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of
London. Available at:
http://eppi.ioe.ac.uk/EPPIWebContent/hp/reports/healthy_eating02/Final_report_web.pdf
Thomas J, Harden A. (2008) Methods for the thematic synthesis of qualitative research in
systematic reviews. BMC Med Res Methodol. 8:45.
Volmink J, Garner P. (2006) Directly observed therapy for treating tuberculosis.
Cochrane Database of Systematic Reviews. Issue 4. Art No: CD003343.
20
Version 1. 23 December 2010
Additional Table 1: Mixed method approaches to synthesis of quantitative and qualitative evidence in common use outside of a
Cochrane context
Approach
EPPI approach
Narrative synthesis
JBI approach
Realist synthesis
Bayesian
Critical Interpretive
Inquiry
Description,
Theoretical
Framework or
Assumptions
Usually includes a
mapping or scoping
review followed by
refinement of the review
question.
Tool box of approaches
for the synthesis of
qualitative and
quantitative data.
Integration / aggregation
using comprehensive and
exhaustive search strategies, a
standardized approach to
critical appraisal, data
extraction and data synthesis
as described in an a priori
protocol. Data synthesis using
the QARI software is process
based, commencing with the
identification of findings,
categorization of findings and
development of synthesized
findings that are
representative of the primary
research they are drawn from.
Synthesised findings are used
as the basis for generalisable
statements related to an
intervention or phenomena of
interest, specifically to inform
practitioners.
Realist synthesis is a
model for synthesizing
plural forms of
evidence that are
generated through the
complex interactions
between processes
(mechanisms). The
model draws on the
methodological logic
of realistic evaluation
and applies it to the
synthesis of plural
forms of evidence.
(McCormack et al.
2006)
Bayesian analysis begins
with beliefs that are
temporally or logically prior
to the main data, formally
expressed as a probability
distribution, and updates
these beliefs using the main
data to produce an
assessment of current
evidence. It allows the
integration of qualitative and
quantitative forms of
evidence, and explicitly
allows qualitative evidence
to contribute to metaanalysis by identifying
variables to be included and
providing evidence about
effect sizes. (Dixon-Woods
et al. 2005a)
This method does not
take a
sequential approach
to review but rather
sees question
formulation,
searching, selection,
data extraction,
critique and synthesis
as iterative processes.
(Dixon-Woods et
al.2006)
Synthesis
approach
Guidance and
examples
Meta-analysis of the data
extracted from trials is
then conducted in
parallel with thematic
analysis to synthesise the
textual data extracted
from qualitative studies.
The findings from the
qualitative synthesis are
then integrated with
those from the
quantitative metaanalysis – i.e. one review
with three syntheses.
The framework includes:
 Developing a theory
of how the
intervention works,
why and for whom
 Developing a
preliminary synthesis
of findings of included
studies
 Exploring
relationships in the
data
 Assessing the
robustness of the
synthesis.
.
Interpretive
Interpretive
Aggregative
Interpretive
Interpretive
Interpretive
Thomas & Harden
(2008) describes the
approach used in EPPI
reviews, including data
extraction, coding and
synthesis
Popay et al - ESRC
guidance 2006
The Joanna Briggs Institute
Reviewer’s Manual, 2008
Edition,
SUMARI User guide, 2008,
Version 4.0
Pawson 2006;
Pawson and Boaz
2004; Pawson et al
2004; Pawson & Tilley
1997
Sutton et al. 2001;
Spiegelhalter et al. 2000;
Roberts et al 2002
Pearson et al (2005)
Greenhalgh et al.’s reinterpretation of a
Dixon-Woods et al
2006 on access to
healthcare by
vulnerable groups;
Flemming 2010 on
the use of morphine
to treat cancer-related
pain
Thomas et al. (2003)
Arai et al. (2005) review
on the use of domestic
smoke detectors
21
Version 1. 23 December 2010
review on children and
healthy eating
Several other examples
described in Harden et
al. 2004
Cochrane review of
children’s feeding
programmes (2007);
McCormack et al.
(2006) review on
practice development
22