Synthesising & Integrating
Qualitative Research Evidence
Methods @ Plymouth 2011
Friday 20 May
E. van Teijlingen, A. Avenell, F. Douglas & P. Hoddinott
www.bournemouth.ac.uk
Road map
• Introduction
• Rise of qualitative synthesis;
• Methodological conundrums (tensions &
questions raised);
• Tools use to address some of those;
• Our thinking around qualitative synthesis
for a HTA project on men’s health and
obesity prevention & treatment.
www.bournemouth.ac.uk
Introduction
• Research team
• Substantive topic
• HTA funding
Synthesising & Integrating Qual. Res.
• A growing methodological field
• Different approaches have grown
organically over past two decades
• Our ideas to attempt synthesis.
www.bournemouth.ac.uk
HTA Project I
Research team:
• Alison Avenell
• Flora Douglas
• Pat Hoddinott
• Edwin van Teijlingen
Topic areas:
• Men’s Health
• Obesity Prevention & Treatment
Funding body:
• HTA Project 09/127/01
www.bournemouth.ac.uk
HTA Project I
Multidisciplinary Research Team
Team member:
Expertise:
Alison Avenell
Health Service Research;
Nutrition/Obesity Research;
Quantitative Sys. Review.;
Flora Douglas
Health Prom.; Men’s Health;
Qualitative Research; Public Health
Nutrition;
Pat Hoddinott
Public Health Nutrition; General
Practice; Qualitative Research;
Edwin van Teijlingen Sociology; Health Promotion;
Systematic Review; Qualitative
Research; Men’s Health
www.bournemouth.ac.uk
Men’s Health I
Men die younger than women
Men generally have less healthy
lifestyles than women, more likely
to take risk, etc.
Men tend to seek help later in
disease stage (i.e. wait longer
before going to get medical care)
www.bournemouth.ac.uk
Men’s Health II
Perhaps a gender-sensitive
approach is needed to better
understand men's health seeking
behaviour.
If existing services were more “male
friendly”, i.e. convenient and
anonymous they might be better
used by men.
(Schofield et al., 2000; Banks, 2001; Lloyd & Forrest, 2001).
www.bournemouth.ac.uk
Weight
management
Obesity & associated diseases are
huge challenge to NHS and UK.
Lack of engagement of men in
services for obesity is a serious
hindrance to mitigating its effects.
Obesity interventions, especially
weight management interventions,
often seen as ‘for women’.
www.bournemouth.ac.uk
The funded study
Overarching objective is to integrate quantitative &
qualitative evidence base for engagement and
management of men with obesity in weight loss
services:
(1) Systematically reviewing the effectiveness and costeffectiveness of interventions for obesity in men, and
men in contrast to women
(2) Systematically reviewing the effectiveness and costeffectiveness of interventions to engage men in their
weight reduction
(3) Systematically reviewing qualitative
research with men about obesity
management, and providers of such
services for men
www.bournemouth.ac.uk
Road map
• Introduction
• Rise of qualitative synthesis;
• Methodological conundrums (tensions &
questions raised);
• Tools use to address some of those;
• Our thinking around qualitative synthesis
for a HTA project on men’s health and
obesity prevention & treatment.
www.bournemouth.ac.uk
Background I
• Field of synthesising evidence from
qualitative primary studies has witnessed
the emergence of several different
approaches, each is based on different
epistemological viewpoint.
• These approaches not only develop
research questions, but also conduct a
critique of the literature included in the
review.
www.bournemouth.ac.uk
Background II
BUT, end product of synthesis can be:
• too conceptual/ complex for policy-makers,
managers & practitioners;
• Too costly, some methods are time and
resource intensive.
Policy makers often / usually want a rapid
answer, which limits the choice of
analysis procedures.
www.bournemouth.ac.uk
Background III
• We reflect on approaching synthesis
from ‘realist’ perspective. We are trying
to find out not only “what works” for
men in terms of weight management,
but also “for which men, and under
what circumstances”.
• We are developing an eclectic study
design, drawing from several
methodological approaches, employing
deductive and inductive analytical
approaches throughout review process.
www.bournemouth.ac.uk
Why do we do it?
When to use Qualitative Methods:
•Focusing on the ‘why’ question.
•Preparing for quantitative studies
•In parallel with quantitative studies:
 explain / examine in-depth meaning
findings of quantitative studies.
 as part of triangulation.
www.bournemouth.ac.uk
Range of Methods
 In-depth interviews
 Focus groups
 Observation
 Content analysis
 Critical incident technique
 Case studies
www.bournemouth.ac.uk
Road map
• Introduction
• Rise of qualitative synthesis;
• Methodological conundrums (tensions &
questions raised);
• Tools use to address some of those;
• Our thinking around qualitative synthesis
for a HTA project on men’s health and
obesity prevention & treatment.
www.bournemouth.ac.uk
Can we compare? I
In order to be able to synthesise one
has to agree that we actually can
compare studies based on very
different qualitative research
methods in different populations
based on different methodological
approaches within qualitative
research.
www.bournemouth.ac.uk
Can we compare? II
Which leads to next step that one
has to agree that we can assess
the quality of qualitative research
across studies using different
methods and/or approaches .
www.bournemouth.ac.uk
Range of views
1. All research perspectives are unique, each valid
in its own right, i.e. no general quality criteria;
2. Qualitative and quantitative methods can be
assessed by the same (or very similar) criteria;
3. Qualitative and quantitative methods can be
assessed, but not by the same criteria
(checklists for qualitative research).
www.bournemouth.ac.uk
NO to comparing I
• Some people argue that we can’t
apply quantitative / Positivist ideas
of systematic reviewing and metaanalysis to methods as diverse as
qualitative ones.
www.bournemouth.ac.uk
NO to comparing II
SAMPLING
 Purposive / theoretical
 Criterion / quota
 Random
 Snowball
 etc
www.bournemouth.ac.uk
Purposive sampling:
•
Extreme Case
•
Maximum
Variation
•
Homogeneous
•
Typical Case
•
Critical Case
•
Etc.
No to comparing III
Can we really mix qualitative
methodologies?
Ethnographers study shared experiences
and their construction within a specific
environment and culture, whilst
Grounded Theorists uses the data
more to explain (i.e. build theory
around) a phenomenon.
www.bournemouth.ac.uk
No common criteria
• All research perspectives are unique
and each is valid in its own terms. This
view rejects the notion of establishing
general quality criteria;
• View of some purists.
Mays N, Pope C. Assessing quality in qualitative research. BMJ 2000;
320:50-52.
www.bournemouth.ac.uk
Same or similar
criteria
• Qualitative and quantitative methods
can be assessed by the same (or very
similar) criteria, such as:
• acceptability, cost, validity, reliability,
generalisability (or relevance) and
objectivity.
(Ryan et al. 2001)
www.bournemouth.ac.uk
Checklists
Predefined criteria in assessing quality of
qualitative studies: ‘checklists’
Blaxter, M. Criteria for qualitative research. Med. Sociol. News 2000; 26: 34-7.
Mays & Pope. Rigour and qualitative research. BMJ 1995; 311:109-112.
Mays & Pope. Assessing quality in qualitative research. BMJ 2000; 320:50-2.
Critical Appraisal Skills Programme (CASP) 10 questions to help you make
sense of qualitative research, www.sph.nhs.uk/sph-files/casp-appraisaltools/Qualitative%20Appraisal%20Tool.pdf
Malterud K. Qualitative research: standards, challenges & guidelines, The
Lancet 2001 358: 483-88
www.bournemouth.ac.uk
Typical checklist I
 Worth or relevance of research?
 Clarity of research question: able to set aside
preconceptions?
 Appropriateness of research design to question?
 Is the context or setting adequately described?
 Sampling: full range possible cases/settings so that
conceptual rather than statistical generalisations could
be made?
www.bournemouth.ac.uk
Cont.
Typical checklist II
 Analysis: analysis incorporating all observations? Could
it explain key processes or respondents’ accounts or
observations? Search for disconfirming cases?
 Reflexivity of account: Self-conscious assessment impact
of methods used on data obtained? Are sufficient data
included in report to assess whether analytical criteria
had been met?
www.bournemouth.ac.uk
(Mays & Pope, BMJ 2000: 50-52)
Case for checklists
•Checklist give outsiders a chance to evaluate the
quality of our work.
•At the same time it is important to stress that there a
many different ways to conduct qualitative research.
www.bournemouth.ac.uk
Against checklists I
•Are qualitative methods really different in general outline?
•Are there no common criteria for a good piece of research
using either qualitative or quantitative methods?
•There are different ways to conduct quantitative research:
• (e.g. parametric versus non-parametric statistics).
www.bournemouth.ac.uk
Against checklists II
•
Checklists focus on how research is conducted
rather than on the quality of the method itself.
•
There is a risk that criteria become ‘bureaucratic’
or ‘prescriptive’. Whilst flexibility is exactly the
strength of qualitative methods!
Barbour (2001)
www.bournemouth.ac.uk
Not same criteria
• Qualitative and quantitative methods can
be assessed, but not by the same criteria.
• E.g. Lincoln & Guba (1985) suggested
four criteria specifically for the evaluation
of qualitative methods:
• credibility, transferability, consistency
(or dependability) and confirmability.
www.bournemouth.ac.uk
Case for common
criteria I
• Main quality criteria: (1) acceptability; (2)
validity; (3) reliability; or (4) generalisability
(or relevance); (5) objectivity.
• Criteria apply to the valuation of quantitative and qualitative
techniques, perhaps some criteria require minor
modification for qualitative research.
www.bournemouth.ac.uk
Common criteria:
example ‘validity’
(2) Validity
• Triangulation
• Respondent validation
• Reflexivity (relevant to validity & reliability)
• Data analysis:
•Computer packages
•Grounded theory
•Analytical induction (e.g. identify ‘deviant’ cases)
www.bournemouth.ac.uk
Common criteria:
example ‘objectivity’
Methods employed to maximise objectivity:
•Reflexivity: a process where researchers
continually reflect upon how their own interests
and potential biases could alter the
interpretation of the results;
•intersubjectivity (interrater reliability).
www.bournemouth.ac.uk
Grading studies
No, or few flaws. The study credibility, transferability,
dependability and confirmability is high.
B: Some flaws, unlikely to affect the credibility,
transferability, dependability and/or confirmability of
the study.
C: Some flaws that may affect the credibility,
transferability, dependability and/or confirmability of
the study.
D: Significant flaws that are very likely to affect the
credibility, transferability, dependability and/or
confirmability of the study.
A:
(Downe et al. 2007)
www.bournemouth.ac.uk
Synthesizing
Table 1
Process of meta-synthesis
Steps in process of doing a qualitative meta-synthesis:
1. aim of study: establish appropriate research
question
2. set inclusion and exclusion criteria methodologies
3. identify and locate relevant papers/publications
4. appraise studies located through process above.
5. synthesize findings reported in papers above.
www.bournemouth.ac.uk
1. Research question
• Some of earlier meta-synthesis have been
criticised for being too broad, e.g. synthesis
by Sherwood (1997) on ‘clients perspective
of caring’
• Walsh and Downe (2005) concluded that
the debate of a broader more inclusive
stance versus a narrower more precise
stance is still ongoing, although they seem
to favour a narrower focus.
www.bournemouth.ac.uk
2 Inclusion/
exclusion criteria
• Include all qualitative research, all
methods and all approaches.
• Include only selected methods
• Include only selected approaches
• Etc.
www.bournemouth.ac.uk
3 Identifying and
locating studies
Qualitative researchers are less likely to
publish in journals. Most electronic
databases comprise mainly journal articles,
not reports and book chapters which often
accepts longer manuscripts needed for
reporting qualitative research in detail.
Hence need to search outwith the standard
electronic databases, and check references
of articles found in the initial trawl.
Methods sections in papers are often too short
for authors to give detailed information on
how the study was exactly conducted, this
creates the need to contact the original
authors for further details.
www.bournemouth.ac.uk
4 Appraising
qualitative literature
• Arguably the most crucial step in
meta-analysis, but most uncertain.
• Some meta-synthesists refer to
existing appraisal methods
(‘tools’), such as CASP (as
mentioned before) or the UK
Cabinet Office quality framework
(Spencer et al. 2003).
www.bournemouth.ac.uk
5 Approaches to
meta analysis
•
•
•
•
•
•
•
•
•
•
•
Meta narrative
Meta synthesis
Thematic synthesis
Grounded Theory
Critical interpretive synthesis
Meta ethnography
Ecological Triangulation
Qualitative Metasummary
Content Analysis
Framework synthesis
Etc.
(review of range of methods by Barnett-Page & Thomas 2009)
www.bournemouth.ac.uk
5a Meta-narrative
Narrative inductive method of analysis
Greenhalgh et al. (2005) on diffusion of
innovation in health services.
Bring together research which contains
many different theories from many
different disciplines & study designs.
“In the best narrative work, descriptive/
interpretative analysis is a story about
stories.” (Jones 2004: 107)
www.bournemouth.ac.uk
5b Thematic
synthesis
Adopts approaches from metaethnography and grounded theory.
Method developed to assist reviews
on interventions.
Thomas & Harden (2008) advocated
thematic analysis in three stages: (a)
coding ‘line by line’ of included
studies; (b) develop ‘descriptive’
themes and (c) generate analytical
themes.
www.bournemouth.ac.uk
5c Critical
interpretive synthesis
Adopts approaches from metaethnography and grounded theory.
Critical interpretive synthesis is
approach to doing the whole reviewing
process not just the synthesis part of it
(Dixon Woods et al. 2007). E.g.
iterative approach redefining research
question and searching and selecting
from literature.
www.bournemouth.ac.uk
5d Meta
ethnography
A comparative textural analysis using
three ways to order them:
• in terms of one another;
• set against one another;
• tied to one another.
Meta-ethnography aims to recover the
social and theoretical context in
which substantive findings emerge
(Noblit & Hare, 1988, p. 5-6). It is
driven by interpretation, not analysis.
www.bournemouth.ac.uk
But complex &
expensive
Barnett-Page and Thomas (2009)
argued that products of some
qualitative synthesis reviews can
be complex and conceptual and
(in their opinion), tend to require
further interpretation by policy
makers and practitioners.
• Some synthesis methods can be
very time and resource intensive
www.bournemouth.ac.uk
Road map
• Introduction
• Rise of qualitative synthesis;
• Methodological conundrums (tensions &
questions raised);
• Tools use to address some of those;
• Our thinking around qualitative
synthesis for a HTA project on men’s
health and obesity prevention &
treatment.
www.bournemouth.ac.uk
Realist Approach I
Realist evaluation has as its underlying
principles of what works best, for
whom, and under what
circumstances (Kazi 2003:1).
Realistic or Realist evaluation has
particular strengths in assessing
interventions delivered in context;
(Pawson & Tilley 1997).
www.bournemouth.ac.uk
Realist Approach II
Realist evaluation is particularly useful
“in the evaluation of practice in an
open system that consists of a
constellation of interconnected
structures, mechanisms and
contexts”
(Kazi 2003:158).
www.bournemouth.ac.uk
Realist Approach III
• We aim to uncover how effective
interventions work (if they exist),
and describe a recipe for the
intervention ingredients,
processes, environmental and
contextual factors that contribute
to effectiveness (Pawson 1997).
www.bournemouth.ac.uk
Realist Approach IV
• A ‘realist’ perspective (Pawson,
2006; Spencer et al., 2003), as we
are concerned with trying to find out
not only “what works” for men in
terms of weight management, but
also “for which men, and under what
circumstances”.
• Also we as researchers span a range
of clinical, sociological and public
health disciplines, which represents
a broad range of perspectives.
www.bournemouth.ac.uk
Realist Approach V
We shall try an eclectic study design,
drawing from range of methodological
literature and employ deductive and
inductive analytical approaches
throughout proposed review process.
We will define a priori research questions,
but also generate new research
questions inductively from the data to
build theory (about what works, for
whom, etc.) using for example, constant
comparative method, and searching for
disconfirming evidence to falsify any
emerging theory or theories.
www.bournemouth.ac.uk
The Plan: Step 1
Step 1: A priori research questions
We aim to uncover how effective interventions work
(if they exist), and describe intervention
ingredients, processes, environmental and
contextual factors that contribute to effectiveness
(Pawson, 2006). To identify barriers/ facilitators
men experience in weight management
interventions. Specific research questions are:
• What are the best evidence-based
management strategies for treating
obesity in men?
• How can men's engagement in obesity
services be improved?
www.bournemouth.ac.uk
Steps 2 & 3
Step 2: Literature search
Use multiple strategies to identify
qualitative research (Shaw et al.
(2004; Grant 2004).
Step 3: Apply inclusion /exclusion
criteria to study abstracts
Studies include males, must have
overweight and weight management
as the prime focus of study. Studies
in developed countries only, which
are relevant to the UK.
www.bournemouth.ac.uk
Step 4
Purposive sampling to group studies into three
categories
A. Qualitative & mixed method studies linked
to RCTs, which include any qualitative data
as part of reporting quantitative outcomes,
as recommended in critical interpretive
synthesis (Dixon Woods et al., 2007)
B. Qualitative & mixed method studies linked
to non-randomised intervention studies
including any qualitative data reported as
part quantitative outcomes.
C. Qualitative studies that are not linked to
any specific intervention.
www.bournemouth.ac.uk
Step 4 (cont.)
An iterative, sequential analysis will then proceed.
During qualitative stage A, qualitative data from
(effective & non-effective) RCTs will be analysed
using constant comparative method to illicit
aspects of intervention processes and contexts
that contributed to trial outcomes.
Using theoretical sampling, category B studies (nonrandomised intervention studies) will be
indentified and analysed in the same way. During
this stage emergent themes and patterns between
the two groups will be compared and contrasted,
and emergent theory modified accordingly.
In qualitative stage C, non-intervention studies will
be theoretically sampled and analysed to further
refine & validate our emerging analysis.
www.bournemouth.ac.uk
Step 5
Quality assessment of included studies
Apply Critical Appraisal Skills Programme
(CASP) quality appraisal criteria.
We will assess each article identified, but
which would be excluded on purely
quality grounds, on the basis of its
potential conceptual contribution to
review (‘sensitivity analysis’).
www.bournemouth.ac.uk
Step 6
Final thematic synthesis step is proposed
as a staged (but overlapping) process
which involves: coding data, developing
initial descriptive themes, with a final
stage that involves the development of
higher order analytical themes and
concepts.
This output will identify promising
‘ingredients’ of interventions more likely
to be effective in male weight reduction,
both in terms of essential and necessary
contextual/environmental variables, and
intervention processes.
www.bournemouth.ac.uk
Men’s involvement
Advisory group from Men’s Health Forum to:
• ensure review is conducted in systematic
and transparent manner;
• check for over-interpretation of data;
• provide expert opinion on credibility and
plausibility of thematic index and other
emergent issues; and potentially, be able
to provide sources for disconfirming data.
Will report our findings as a narrative synthesis
combining and juxtaposing quantitative and
qualitative findings.
www.bournemouth.ac.uk
• Figure 1 - Graphical
representation of the proposed
review process
www.bournemouth.ac.uk
References I
Banks, I. (2001) No man’s land: Men illness and the NHS, BMJ; 323: 1058-1060.
Barbour R. (2001) Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ
322:1115-17
Barnett-Page, E., Thomas, J. (2009) Methods for the synthesis of qualitative research: a critical review BMC Med
Res Meth, 9 (59) (www.biomedcentral.com/content/pdf/1471-2288-9-59.pdf)
Campbell, R., et al. (2003) Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of
diabetes and diabetes care. Soc Sci Med 56 (4), 671-684.
Critical Appraisal Skills Programme : www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdf
Dixon-Woods, M., et al. (2007) Appraising qualitative research for inclusion in systematic reviews: a quantitative
and qualitative comparison of three methods. J Health Serv Res Pol 12: 42–47.
Downe S,. et al. (2007) Expert intrapartum maternity care: a metasynthesis. J Adv Nurs 57: 127-40.
Downe S. (2008) Metasynthesis: a guide to knitting smoke. Evidence Based Midwifery 6(1): 4-8
Egger G, et al. (2001) Systematic reviews in health care: Meta-analysis in context. London: BMJ Publ
EPPI-Centre (2007) EPPI-Centre methods for conducting systematic reviews. London: Inst. of Educ.
Grant, MJ. (2004) How does your searching grow? A survey of search preferences and use of optimal search
strategies in identification of qualitative research. Health Info Libraries J 21: 21-32.
www.bournemouth.ac.uk
References II
Greenhalgh T, et al. (2005) Storylines of research in diffusion of innovation: a metanarrative
approach to systematic review. Soc Sci Med 2005, 61:417-30.
Jones, K., (2004) Mission drift in qualitative research, or moving toward a systematic review of
qualitative studies, moving back to a more systematic narrative review. The Qualitative
Report, 9 (1): 95-112.
Kazi, M.A.F. (2003) Realist Evaluation in Practice: Health and Social Work . London: Sage.
Lloyd, T, Forrest, S. (2001) Boys’ and Young Men’s Health. A Literature & Practice Review: An
Interim Report. London, UK: Health Development Agency.
Lincoln & Guba. Naturalistic enquiry, Sage, Beverley Hills, CA, 1985.
Noblit GW., Hare, RD. (1988) Meta-ethnography: synthesizing qualitative studies. London: Sage.
Pawson, R, Tilley, N. (1997) Realistic Evaluation. London: Sage.
Pawson, R. (2006) Evidence-based policy: A realist perspective. London: Sage.
Pope, K, et al. (2007) Synthesising qualitative and quantitative health evidence. Open Univ.
Press.
Rose, H.D., Cohen, K. (2010) The experiences of young carers: a meta-synthesis of qualitative
findings, J Youth Stud 13 (4): 473- 487.
Ryan M, et al. (2001) Eliciting public preferences for healthcare: a systematic review of
techniques. Health Technol Assess, 5(5). http://www.hta.nhsweb.nhs.uk/fullmono/mon505.pdf
Robertson LM, et al. (2008) BMC Health Serv Res; 8: 141.
www.bournemouth.ac.uk
References III
Schofield T. et al. (2000). Understanding men's health and illness: a gender-relations approach
to policy, research, and practice. J Am Coll Health. 48(6): 247-256.
Shaw, RL et al. (2004) Finding qualitative research: evaluation of search strategies. BMC Med
Res Meth 5; 4
Sherwood, G. (1997) Meta-synthesis: merging qualitative studies to develop nursing knowledge.
IntJ Human Caring 3 (1): 37-42.
Spencer, L. et al. (2003) Quality in Qualitative Evaluation: A Framework for Assessing Research
Evidence. Centre for Social Research. London: National Centre for Social Research/UK
Cabinet Office.
Strauss A., Corbin, J. (1994) Grounded theory methodology: An overview. In Handbook of
Qualitative Research. (Eds) Denzin, N.K. & Lincoln, Y.S. Sage: Thousand Oaks.
Thomas, J., Harden, A. (2008) Methods for the thematic synthesis of qualitative research in
systematic reviews. BMC Med Res Meth 8(45) http://www.biomedcentral.com/14712288/8/45.
Walsh D, Downe, S (2005) Meta-synthesis method for qualitative research: a lit. rev J Advanc
Nurs 50: 204-11.
www.bournemouth.ac.uk
Thank you!
Prof. Edwin van Teijlingen
Centre for Midwifery, Maternal & Perinatal Health Research,
Bournemouth University
vanteijlingen@bournemouth.ac.uk
www.bournemouth.ac.uk