Qualitative Evidence Synthesis
(QES):
Karin Hannes
Centre for Methodology of
Educational Research
Qualitative Evidence Synthesis: Agenda
PART 1
• Its context (and how I became triggered by QES)
• How to formulate questions and search for qualitative insights
PART 2
• Its popularity
• Its role in systematic reviews (practical examples)
• A list of developed approaches
• A comparison of characteristics of two commonly used
approaches
PART 3
• New developments
– Context specific versus multi-context reviews
– Mixed methods reviews
PART 1:
CONTEXT AND STEPWISE APPROACH
CONTEXT
Emma,
Born the 6th of October 2010
CONTEXT
CONTEXT
• Women who exercised did not lose significantly more
weight than women in the usual care group.
• Women who took part in a diet or diet plus exercise
programme, lost significantly more weight than
women in the usual care.
• There was no difference in the magnitude of weight
loss between diet and diet plus exercise group.
• The interventions seemed not to affect breastfeeding
performance adversely.
CONTEXT
A recent study in the Journal of the
America College of Nutritian found that
those who ate cereals where lower in
weight compared to those who ate meat
and eggs, bread or skipped breakfast.
CONTEXT
Logical, rational reasoning:
• IF a diet helps to loose weight after pregancy.
• IF cereals have proven to work well as a diet.
• THEN the consumption of cereals will lead to
weight loss after pregnancy!
• After having consumed cereals for several
months……………..
CONTEXT
What is evidence?
• Evidence of ‘effectiveness’: the extent to which an
intervention, when used appropriately, achieves the intended
effect.
• Evidence of ‘feasibility’: the extent to which an intervention is
practical and practicable, whether or not an intervention is
physically, culturally or financially practical or possible within a
given context.
• Evidence of ‘appropriateness’ the extent to which an
intervention fits with a situation, how an intervention relates
to the context in which it is given.
• Evidence of ‘meaningfulness’: the extent to which an
intervention is positively experienced by the population and
relates to the personal experience, opinions, values, beliefs
and interpretations of the population.
CONTEXT
Systematic Reviews
IF I am not interested in evidence of effectiveness,
BUT in feasibility, appropriateness, meaningfulness,…
Qualitative Evidence Synthesis:
A process of summarizing qualitative research findings, either aggregative
or interpretive, by comparing and analysing texts derived from multiple
accounts of an event or situation as reported in basic qualitative research
studies.
Explore questions such as
• how do people experience illness?
• why does an intervention work (or not), for whom and in what circumstances…?
• what are the barriers and facilitators to accessing health care?
• what impact do specific barriers and facilitators have on people, their experiences and
behavior?
CONTEXT
Systematic Reviews
• Could I still use the methodology outlined for
SR to answer these questions?
Could I use it in the same way?
1. Question formulation  PICO becomes SPI(C)E
2. Searching  Sensitive (all-inclusive) versus Specific
(Selective)
3. Critical appraisal  Proponents versus Opponents
4. Synthesis  Variety of Approaches
5. Recommendations  Not always the goal, could be
building theory as well
QES: Stepwise approach
PICO becomes SPICE
S
Setting:
Western
P
Perspective:
Mothers in a post-natal situation
I
(C)
E
Intervention/(topic of) Interest: diet
Comparison:
(training, placebo)
Evaluation:
For some qualitative
questions there isn’t
an intervention to be
evaluated.
Attitude, view, opinion on...
•Elements that hinder the diet
•Impact of the diet on the general welbeing of the mother
•Opinions on how to best integrate the diet in daily family life
 Targeted suggestions on how to make things work.
QES: Stepwise approach
Searching qualitative evidence:
problems
•
•
Little result from searching the major databases
– 30% database & handsearch
– 50% ‘snowballing’
– 24% personal contacts
(Greenhalgh, 2005)
Major problems:
– Bad indexing
– Less developed and tested methodological filters
– MeSH-term: qualitative research
• General rules:
–
–
–
–
Use methodological filters
Examine references
Use the related article features in major databases
Search for citations (backword and forward)
QES: Stepwise approach
Searching qualitative evidence: INTERTASC
www.york.ac.uk/inst/crd/intertasc/index.htm
Methodology
•
nursing methodology research OR (qualitative OR ethnolog* OR ethnog* OR
ethnomethodolog* OR emic OR etic OR phenomenolog*[Title/Abstract]) OR
(hermeneutic* OR heidegger* OR husserl* OR colaizzi* OR giorgi* OR glaser OR strauss
[title/abstract]) OR (kaam* OR manen OR participant observ* OR constant compar*
[title/abstract]) OR (focus group* OR grounded theory OR "narrative analysis" OR lived
experience* OR life experience* [title/abstract]) OR (theoretical sampl* OR purposive
sampl* OR ricoeur OR spiegelberg* OR merleau [title/abstract])Citations/authors
OR (metasynthes* OR
meta-synthes* OR metasummar* OR meta-summar* OR metastud* OR meta-stud*
[title/abstract]) OR (maximum variation OR snowball [title/abstract]) OR (field stud* OR
field note* OR fieldnote* OR field record* OR action research [title/abstract]) OR
(thematic analys* OR content analy* OR unstructured categor* or structured categor*
[title/abstract]) OR (participant observation* OR nonparticipant observation* OR non
participant observation* [title/abstract]) OR (tape recording OR "tape record*" OR
"video record*" OR "audio record*" OR taperecord* OR audiorecord* OR videotap* OR
videorecord*)
Techniques
http://www.library.ualberta.ca/subject/healthsciences/QualitativeFilters/index.cfm
QES: Stepwise approach
Searching qualitative evidence: debate
• We need a sensitive approach
to searching that includes all
potentially relevant studies.
• We need a search strategy that
is specific and purposeful,
including studies that are
relevant to our synthesis. This
might include working with a
saturation point for inclusion.
•  Related to the goal and/or
approach of a particular study.
Searching: addressed in upcoming presentation
QES: Stepwise approach
Critically appraising evidence: debate
• The more you appraise the
more you obstruct the creative,
interpretive nature of
qualitative research. The most
important criterion is
relevance!
• The more you appraise the
smaller the chance that you
would end up with statements
and conclusions that do not
represent the truth. The most
important criterion is
(methodological) quality!
‘Critical appraisal’ addressed in upcoming presentation
PART 2: Qualitative Evidence Synthesis (QES)
approaches
QES in the scientific literature
The role of QES (in SR)
A list of developed approaches
Comparing characteristics of two commonly used approaches
QES in the public health literature
Timetrend based on the reviews from:
Dixon-Woods & colleagues (2007)
Hannes & Macaitis (2011)
The role of QES in SR: different aims
Metaanalysis
Evidence-synthesis
Nature
Quantitative
Qualitative
Aim
Accumulating
Make sense of data
Strictly comparable
Basic comparability
More power
Added value in content
Through data
Through interpretation
Studies
Result
Synthesis
The role of QES in SR
• Inform: reviews by using evidence from qualitative research to help
define and refine the question, and to ensure the review includes
appropriate studies and addresses important outcomes (scoping
review)
• Enhance: reviews by synthesizing evidence from qualitative research
identified whilst looking for evidence of effectiveness (process and
implementation issues).
• Extend: reviews by undertaking a search to specifically seek out
evidence from qualitative studies to address questions directly related
to the effectiveness review (mixed method or multilevel synthesis).
• Supplement: reviews by synthesizing qualitative evidence within a
stand-alone, but complementary review to address questions on other
than effectiveness (stand-alone or parallel synthesis).
Note:These should be distinguished from a narrative report of a quantitative SR. When
individual studies cannot be pooled quantitatively (reason: heterogeneity), they may still
have useful qualitative information to be shared with the reader.
Example:Narrative report
Narrative synthesis because pooling is not possible?
Rather not labelled as QES…
Example: Extending Review
Which interventions match recommendations derived from children’s views and
experiences?
‘Mixed Method Approach’
Children & Healthy Eating - EPPI-centre: eppi.ioe.ac.uk
Children’s Views
Trials
Recommendation for interventions
Good quality
Other
Do not promote fruit and vegetables in the
same way
None
None
Brand fruit and vegetables as an ‘exciting’
or child-relevant product, as well as a
‘tasty’ one
5
5
Reduce health emphasis in messages to
promote fruit and vegetables particularly
those which concern future health
5
6
Example: Supplementing review
• Barroso J, Powell-Cope GM. Metasynthesis of Qualitative Research on Living
with HIV Infection. Qual Health Res vol 10, nr 3, 2000.:
–
–
–
–
Understand experience of adults living with HIV infection
21 articles
Method: constant comparative analysis
Themes:
•
•
•
•
•
•
Finding meaning in HIV: death, loss, surviving
Shattered meaning: fear, irreparable
Human connectedness: meaningfull relationships alienation, isolation
Focusing on the self: actions to enhance fysical and emotional health
Negociating care: active role
Dealing with stigma: personal, family, society
The role of QES in SR
• Provide evidence on the subjective experience of those involved in
developing, delivering and receiving an intervention
OR/AND
• Provide a research-based context for interpreting and explaining trial
results
• How to achieve change (more effectively)?
• How to improve interventions?
• How to ‘fit’ subjective needs?
• What other type of interventions might be needed?
OR/AND
• Reveal the extent to which effective interventions are actually adopted
in policies and practice (what are barriers and bridges?)
OR/AND
• Contribute to the understanding of heterogeneity in outcomes
A list of developed approaches
QES-approach
Developed
Meta-ethnography
Noblit & Hare, 1988
Meta-summary
Sandelowski & Barosso (2008)
Meta-study
Paterson et al (2001)
Realist synthesis
Pawson et al (2004)
Meta-narrative mapping
Greenhalgh (2005)
Critical Interpretive Synthesis
Dixon-Woods et al (2006)
Narrative Synthesis
Popey et al (2006)
Textual narrative synthesis
Lucas (2007)
Ecological triangulation
Banning (unknown)
Framework synthesis
Brunton et al (2006), Oliver et al (2008)
Meta-interpretation
Weed (2005)
Meta-aggregation
Joanna Briggs Institute (2001), Hannes and Lockwood (2010)
Bayesian meta-analysis
Roberts et al (2002), Voils et al (2009)
Content analysis
Evans and Fitzgerald (2002), Suikkala & Leino-Kilpi (2000)
Case Survey
Yin & Heald (1975), Jensen & Rodgers (2001)
Qualitative Comparative analysis
Cress & Snow (2000)
Thematic synthesis
Thomas & Harden (2008)
Cross-case analysis
Miles & Huberman (1994)
Grounded theory
Finfgeld (1999) Kearney (2001), Eaves (2001)
A list of developed approaches
• MAKING SENSE OF THE
MYRIAD OF
QUALITATIVE EVIDENCE
SYNTHESIS METHODS
• No clear guidance about
how to determine which
of the existing synthesis
methods best fits a
particular purpose.
• possible
considerations…
A list of developed approaches
Consider
Questions to be asked
The nature of the research
•Will the synthesis method result in the expected and desired outcomes?
•Is the method congruent with the goals of the synthesis project?
•Does the primary research support the method?
The nature of the researcher
•How tolerant is the researcher to the amount of structure and ambiguity
that is inherent in the method?
•Is his/her epistemological stance congruent with that of the synthesis
method?
The nature of the research team •Is there the necessary mix of disciplinary, methodological, and other
perspectives among the research team to enact this method?
• Is the expertise needed for this method (e.g., statistical analysis,
theoretical) available?
Resource requirements
•How much personnel, time and effort are required for this method?
•Is there adequate funding to support expenses incurred in implementing
this method?
Paterson, B. (2011). Introducing Qualitative Evidence Synthesis. In Hannes, K., &
Lockwood, C. (eds.). Qualitative Ev. Synthesis: choosing the right approach. WileyBlackwell, UK.
Qualitative Evidence Synthesis approaches
Decision to conduct a qualitative evidence
synthesis
Purpose of the additional
qualitative synthesis
To aggregate / summarise /
integrate qualitative data to
address specific questions
in relation to a Cochrane
intervention review
Thematic analysis without
theory generation
Meta-aggregation
Product
Aggregated findings
from source papers
To interpret synthesised
qualitative evidence and
develop explanatory theory
or models
Meta-ethnography
Thematic analysis with theory
generation
Grounded theory
Primarily to integrate and interpret
qualitative and quantitative evidence
within a single approach
or integrated model
Can be used to develop
explanatory theory
Realist review
EPPI (mixed) approach
(thematic)
Narrative synthesis
Product
Explanatory theory, analytical or
conceptual framework or
interpretative framework/ mechanism
Comparing characteristics of different approaches (developed for synthesis)
Meta-ethnography
Meta-aggregation
Purpose
Seek and reveal similarities
and differences, achieving a
degree of innovation
To aggregate findings of
included studies and aid
decision making
Epistemology
Idealism
Realism
Quality Assessment
Not discussed, relevance
argument
Required, using standardized
critical appraisal instrument
Synthesis
Refutational and reciprocal
Aggregation of findings into
translation, line of argument categories and of categories
synthesis
into synthesized findings
Outcome
Higher order interpretation
of findings, theory
Standardised chart informing
practice and policy
PART 3: Recent developments
Context-specific versus multi-context reviews
Mixed methods reviews
Context-specific versus multi-context syntheses
Quantitative Review
Qualitative Review
Context-specific versus multi-context reviews
Multi-context reviews
Context-specific reviews
Exhaustive search
Selective search
Little access to or knowledge of local databases
and experts
(related to context)
Access to and knowledge of local databases
and experts
Targets a broad audience (but no-one in particular)
Findings may be too general
Risk of downplaying important local characteristics
Context may get lost
Only relevant to the ‘happy few’.
Findings are less likely transferable to
other settings
Potential low level of acceptance in end-users
Wide ranging in scope
Targeted audience
Ability to cross compare different settings
Highly relevant to practice and policy
Works for topics were little heterogeneity between
settings is expected
Maintains integrity with the context
reported in original studies
Findings are more likely transferable to a broad range of
settings
Findings may induce a higher level of
acceptance in the end-users
Context-specific versus multi-context reviews
• Context-specific syntheses do well in responding to the needs
and policies of a targeted setting.
• Multi-context syntheses assist in building a cumulative
knowledge base and are an excellent choice when little
heterogeneity is expected.
• Integrating the best of both
– Umbrella reviews, in which insights in a particular phenomenon
generated from different settings could be summarized.
– Transcontextual adaptation, which means modifying insights in such a
way that they become relevant and reply to the needs and policies of
a targeted setting.
Mixed methods reviews: definition
•‘Mixed research synthesis’ (Sandelowski et al., 2006)
•‘Mixed studies review’ (Pluye et al., 2009)
•‘Mixed methods synthesis’ (Harden & Thomas, 2005)
• ‘Mixed methods research synthesis’ = A synthesis in which
researchers combine primary qualitative, quantitative, and
mixed methods studies, and apply a mixed methods
approach in order to integrate those studies, for the broad
purposes of breadth and depth of understanding and
corroboration (Heyvaert, Onghena & Maes, 2011).
•Promising utility for research and practice:
Combining strengths of qualitative and quantitative
techniques and studies
Paired: Concurrent but separate review example
– 1 team
Research Question(s)
Cochrane
Intervention
Review
Paired: A/S -QUAL + QUAN
–Equal status
-Concurrent
Qualitative
Evidence
Synthesis
Synthesis of Qualitative
and Quantitative Studies
•
•
Lins S, Rücker G, Motschall E, Langer G, Antes G, Meyer G. Efficacy and experiences of telephone counselling for
informal carers of people with dementia (Protocol). Cochrane Database of Systematic Reviews 2011, Issue 5. Art.
No.: CD009126. DOI: 10.1002/14651858.CD009126
– Qualitative thematic synthesis related to the efficacy of telephone counselling
Leiknes KA, Berg RC, Smedslund G, Jarosch-von Schweder L, Øverland S, Hammerstrøm KT, Høie B. Electroconvulsive
therapy for depression (Protocol). Cochrane Database of Systematic Reviews 2011, Issue 5.
– Meta-aggregative approach to qualitative synthesis
Paired: Concurrent but separate review
example – 2 teams
Research Question(s)
Paired: A/S -QUAL + QUAN
–Equal status
-Concurrent
Cochrane
Intervention
Review
Qualitative
Evidence
Synthesis
Synthesis of Qualitative
and Quantitative Studies
•
Embuldeniya, G. Et al. Perceived impacts and experiences of peer support in chronic
disease
– Qualitative meta-ethnography exploring issues related to an ongoing Cochrane effectiveness
review (Doull MJ, O'Connor AM, Robinson VA, Tugwell P, Wells G. Peer support strategies for people with chronic disease to enhance
health and promote health equity. Cochrane Database of Systematic Reviews 2005, Issue 3.)
Retrospective, stand alone review example
Published
Cochrane
Intervention
Review
New
Retrospective, stand
alone
-Equal status
-Sequential
Qualitative
Evidence
Synthesis
 Noyes, J., Popay, J (2007).Directly observed therapy and tuberculosis: how can a systematic review of
qualitative research contribute to improving services? A qualitative meta-synthesis. Journal of Advanced
Nursing, 57, 227-243.

Narrative Summary approach extending/enhancing a Cochrane intervention review Volmink J. & Garner P (1997)
(updated 2002, 2006) Directly observed therapy for treating tuberculosis. Cochrane Database Systematic Reiews 19(2),
CD003343.
 Jordan J, Rose L, Dainty KN, Noyes J, Clarke S, Blackwood B. Factors which impact on the use of weaning protocols for
reducing the duration of mechanical ventilation in adults and children.
 Meta-ethnography extending/enhancing a recent Cochrane intervention review
(Blackwood B, Alderdice F, Burns KEA, Cardwell CR,
Lavery G, O'Halloran P. (2010) Protocolized versus non-protocolized weaning for reducing the duration of mechanical ventilation in critically ill adult patients.
Cochrane Database of Systematic Reviews (in press).
QES: conclusion
• If you wish to embark in the field of qualitative
evidence synthesis you will soon realise that
–
–
–
–
–
There are no fixed standards
There is no general consensus (and will there ever be one?)
You shall have to travel slowly,
By your own means (but the CQRM- and CP&I-groups can help)
On small and potentially difficult roads…
• BE PREPARED FOR A POTENTIAL DELAY!