Qualitative Evidence Synthesis
(QES):
Karin Hannes
Centre for Methodology of
Educational Research
Qualitative Evidence Synthesis: Agenda
PART 1
• Its context (and how I became triggered by QES)
• How to formulate questions and search for qualitative insights
PART 2
• Its popularity
• Its role in systematic reviews (practical examples)
• A list of developed approaches
• A comparison of characteristics of two commonly used
approaches
PART 3
• New developments
– Context specific versus multi-context reviews
– Mixed methods reviews
PART 1:
CONTEXT AND STEPWISE APPROACH
CONTEXT
Emma,
Born the 6th of October 2010
CONTEXT
CONTEXT
• Women who exercised did not lose significantly more
weight than women in the usual care group.
• Women who took part in a diet or diet plus exercise
programme, lost significantly more weight than
women in the usual care.
• There was no difference in the magnitude of weight
loss between diet and diet plus exercise group.
• The interventions seemed not to affect breastfeeding
performance adversely.
CONTEXT
A recent study in the Journal of the
America College of Nutritian found that
those who ate cereals where lower in
weight compared to those who ate meat
and eggs or skipped breakfast.
CONTEXT
Logical, rational reasoning:
• IF a diet helps to loose weight after pregancy.
• IF cereals have proven to work well as a diet.
• THEN the consumption of cereals will lead to weight loss
after pregnancy!
• After having consumed cereals for several months……………..
• If it has proven itself to work for others…
• THEN WHY DOESN’T IT WORK FOR ME!
CONTEXT
What is evidence?
• Evidence of ‘effectiveness’: the extent to which an
intervention, when used appropriately, achieves the intended
effect.
• Evidence of ‘feasibility’: the extent to which an intervention is
practical and practicable, whether or not an intervention is
physically, culturally or financially practical or possible within a
given context.
• Evidence of ‘appropriateness’ the extent to which an
intervention fits with a situation, how an intervention relates
to the context in which it is given.
• Evidence of ‘meaningfulness’: the extent to which an
intervention is positively experienced by the population and
relates to the personal experience, opinions, values, beliefs
and interpretations of the population.
CONTEXT
Systematic Reviews
IF I am not interested in evidence of effectiveness,
BUT in feasibility, appropriateness, meaningfulness,…
Qualitative Evidence Synthesis:
A process of summarizing qualitative research findings, either aggregative
or interpretive, by comparing and analysing texts derived from multiple
accounts of an event or situation as reported in basic qualitative research
studies.
Explore questions such as
• how do people experience illness?
• why does an intervention work (or not), for whom and in what circumstances…?
• what are the barriers and facilitators to accessing health care?
• what impact do specific barriers and facilitators have on people, their experiences and
behavior?
CONTEXT
Systematic Reviews
• Could I still use the methodology outlined for
SR to answer these questions?
Could I use it in the same way?
1. Question formulation  PICO becomes SPI(C)E
2. Searching  Sensitive (all-inclusive) versus Specific
(Selective)
3. Critical appraisal  Proponents versus Opponents
4. Synthesis  Variety of Approaches
5. Recommendations  Not always the goal, could be
building theory as well
QES: Stepwise approach
PICO becomes SPICE
S
Setting:
Western
P
Perspective:
Mothers in a post-natal situation
I
(C)
E
Intervention/(topic of) Interest: diet
Comparison:
(training, placebo)
Evaluation:
For some qualitative
questions there isn’t
an intervention to be
evaluated.
Attitude, view, opinion on...
•Elements that hinder the diet
•Impact of the diet on the general welbeing of the mother
•Opinions on how to best integrate the diet in daily family life
QES: Stepwise approach
Searching qualitative evidence:
problems
•
Little result from searching the major databases
– 30% database & handsearch
– 50% ‘snowballing’
– 24% personal contacts
(Greenhalgh, 2005)
•
Major problems:
– Bad indexing
– Less developed and tested methodological filters
– MeSH-term: qualitative research
•
General rules:
–
–
–
–
Use methodological filters
Examine references
Use the related article features in major databases
Search for citations (backword and forward)
QES: Stepwise approach
Searching qualitative evidence:
keywords
• Technique:
• Software:
• Citations :
• Phenomenon:
• Methodology:
• Type of data:
focus group, interview, action
research, content analysis,
thematic analysis...
Nudist OR Nvivo OR Atlas-ti
Glaser and Strauss, Miles and
Huberman,...
perceptions, attitudes, user
views, standpoints, viewpoints,
experiences,...
ethnographic, phenomenologic,…
stories, narratives, descriptions,
themes, exploration,
findings (<--> results)
QES: Stepwise approach
Searching qualitative evidence: INTERTASC
www.york.ac.uk/inst/crd/intertasc/index.htm
Methodology
•
nursing methodology research OR (qualitative OR ethnolog* OR ethnog* OR
ethnomethodolog* OR emic OR etic OR phenomenolog*[Title/Abstract]) OR
(hermeneutic* OR heidegger* OR husserl* OR colaizzi* OR giorgi* OR glaser OR strauss
[title/abstract]) OR (kaam* OR manen OR participant observ* OR constant compar*
[title/abstract]) OR (focus group* OR grounded theory OR "narrative analysis" OR lived
experience* OR life experience* [title/abstract]) OR (theoretical sampl* OR purposive
sampl* OR ricoeur OR spiegelberg* OR merleau [title/abstract])Citations/authors
OR (metasynthes* OR
meta-synthes* OR metasummar* OR meta-summar* OR metastud* OR meta-stud*
[title/abstract]) OR (maximum variation OR snowball [title/abstract]) OR (field stud* OR
field note* OR fieldnote* OR field record* OR action research [title/abstract]) OR
(thematic analys* OR content analy* OR unstructured categor* or structured categor*
[title/abstract]) OR (participant observation* OR nonparticipant observation* OR non
participant observation* [title/abstract]) OR (tape recording OR "tape record*" OR
"video record*" OR "audio record*" OR taperecord* OR audiorecord* OR videotap* OR
videorecord*)
Techniques
http://www.library.ualberta.ca/subject/healthsciences/QualitativeFilters/index.cfm
QES: Stepwise approach
Searching qualitative evidence: debate
• We need a sensitive approach
to searching that includes all
potentially relevant studies.
• We need a search strategy that
is specific and purposeful,
including studies that are
relevant to our synthesis. This
might include working with a
saturation point for inclusion.
•  Related to the goal and/or
approach of a particular study.
Searching: addressed in upcoming presentation
Next step ‘critical appraisal’ addressed in upcoming presentation
PART 2: Qualitative Evidence Synthesis (QES)
approaches
QES in the public health literature
The role of QES (in SR)
A list of developed approaches
Comparing characteristics of different approaches
QES in the public health literature
Timetrend based on the reviews from:
Dixon-Woods & colleagues (2007)
Hannes & Macaitis (2011)
The role of QES in SR: different aims
Metaanalysis
Evidence-synthesis
Nature
Quantitative
Qualitative
Aim
Accumulating
Make sense of data
Strictly comparable
Basic comparability
More power
Added value in content
Through data
Through interpretation
Studies
Result
Synthesis
The role of QES in SR
• Inform: reviews by using evidence from qualitative research to help
define and refine the question, and to ensure the review includes
appropriate studies and addresses important outcomes (scoping
review)
• Enhance: reviews by synthesizing evidence from qualitative research
identified whilst looking for evidence of effectiveness (process and
implementation issues).
• Extend: reviews by undertaking a search to specifically seek out
evidence from qualitative studies to address questions directly related
to the effectiveness review (mixed method or multilevel synthesis).
• Supplement: reviews by synthesizing qualitative evidence within a
stand-alone, but complementary review to address questions on other
than effectiveness (stand-alone or parallel synthesis).
Note:These should be distinguished from a narrative report of a quantitative SR. When
individual studies cannot be pooled quantitatively (reason: heterogeneity), they may still
have useful qualitative information to be shared with the reader.
Example:Narrative report
Narrative synthesis because pooling is not possible?
Rather not labelled as QES…
Example: Extending Review
Which interventions match recommendations derived from children’s views and
experiences? Conclusion
‘Mixed Method Approach’
Children & Healthy Eating - EPPI-centre: eppi.ioe.ac.uk
Children’s Views
Trials
Recommendation for interventions
Good quality
Other
Do not promote fruit and vegetables in the
same way
None
None
Brand fruit and vegetables as an ‘exciting’
or child-relevant product, as well as a
‘tasty’ one
5
5
Reduce health emphasis in messages to
promote fruit and vegetables particularly
those which concern future health
5
6
Example: Supplementing review
• Barroso J, Powell-Cope GM. Metasynthesis of Qualitative Research on Living
with HIV Infection. Qual Health Res vol 10, nr 3, 2000.:
–
–
–
–
Understand experience of adults living with HIV infection
21 articles
Method: constant comparative analysis
Themes:
•
•
•
•
•
•
Finding meaning in HIV: death, loss, surviving
Shattered meaning: fear, irreparable
Human connectedness: meaningfull relationships alienation, isolation
Focusing on the self: actions to enhance fysical and emotional health
Negociating care: active role
Dealing with stigma: personal, family, society
The role of QES in SR
• Provide evidence on the subjective experience of those involved in
developing, delivering and receiving an intervention
OR/AND
• Provide a research-based context for interpreting and explaining trial
results
• How to achieve change (more effectively)?
• How to improve interventions?
• How to ‘fit’ subjective needs?
• What other type of interventions might be needed?
OR/AND
• Reveal the extent to which effective interventions are actually adopted
in policies and practice (what are barriers and bridges?)
OR/AND
• Contribute to the understanding of heterogeneity in outcomes
A list of developed approaches
QES-approach
Developed
Meta-ethnography
Noblit & Hare, 1988
Meta-summary
Sandelowski & Barosso (2008)
Meta-study
Paterson et al (2001)
Realist synthesis
Pawson et al (2004)
Meta-narrative mapping
Greenhalgh (2005)
Critical Interpretive Synthesis
Dixon-Woods et al (2006)
Narrative Synthesis
Popey et al (2006)
Textual narrative synthesis
Lucas (2007)
Ecological triangulation
Banning (unknown)
Framework synthesis
Brunton et al (2006), Oliver et al (2008)
Meta-interpretation
Weed (2005)
Meta-aggregation
Joanna Briggs Institute (2001), Hannes and Lockwood (2010)
Bayesian meta-analysis
Roberts et al (2002), Voils et al (2009)
Content analysis
Evans and Fitzgerald (2002), Suikkala & Leino-Kilpi (2000)
Case Survey
Yin & Heald (1975), Jensen & Rodgers (2001)
Qualitative Comparative analysis
Cress & Snow (2000)
Thematic synthesis
Thomas & Harden (2008)
Cross-case analysis
Miles & Huberman (1994)
Grounded theory
Finfgeld (1999) Kearney (2001), Eaves (2001)
A list of developed approaches
• MAKING SENSE OF THE
MYRIAD OF
QUALITATIVE EVIDENCE
SYNTHESIS METHODS
• No clear guidance about
how to determine which
of the existing synthesis
methods best fits a
particular purpose.
• possible
considerations…
A list of developed approaches
Consider
Questions to be asked
The nature of the research
•Will the synthesis method result in the expected and desired outcomes?
•Is the method congruent with the goals of the synthesis project?
•Does the primary research support the method?
The nature of the researcher
•How tolerant is the researcher to the amount of structure and ambiguity
that is inherent in the method?
•Is his/her epistemological stance congruent with that of the synthesis
method?
The nature of the research team •Is there the necessary mix of disciplinary, methodological, and other
perspectives among the research team to enact this method?
• Is the expertise needed for this method (e.g., statistical analysis,
theoretical) available?
Resource requirements
•How much personnel, time and effort are required for this method?
•Is there adequate funding to support expenses incurred in implementing
this method?
Paterson, B. (2011). Introducing Qualitative Evidence Synthesis. In Hannes, K., &
Lockwood, C. (eds.). Qualitative Ev. Synthesis: choosing the right approach. WileyBlackwell, UK.
Qualitative Evidence Synthesis approaches
Decision to conduct a qualitative evidence
synthesis
Purpose of the additional
qualitative synthesis
To aggregate / summarise /
integrate qualitative data to
address specific questions
in relation to a Cochrane
intervention review
Thematic analysis without
theory generation
Meta-aggregation
Product
Aggregated findings
from source papers
To interpret synthesised
qualitative evidence and
develop explanatory theory
or models
Meta-ethnography
Thematic analysis with theory
generation
Grounded theory
Primarily to integrate and interpret
qualitative and quantitative evidence
within a single approach
or integrated model
Can be used to develop
explanatory theory
Realist review
EPPI approach (thematic)
Narrative synthesis
Product
Explanatory theory, analytical or
conceptual framework or
interpretative framework/ mechanism
Comparing characteristics of different approaches (developed for synthesis)
Meta-ethnography
Meta-aggregation
Purpose
Seek and reveal similarities and
differences, achieving a degree of
innovation
To aggregate findings of included studies and
aid decision making
Epistemology
Idealism
Realism
Quality Assessment
Not discussed, relevance argument
Required, using standardized critical
appraisal instrument
Synthesis
Refutational and reciprocal translation,
line of argument synthesis
Aggregation of findings into categories and of
categories into synthesized findings
Outcome
Higher order interpretation of findings,
theory
Standardised chart informing practice and
policy
PART 3: Revent developments
Context-specific versus multi-context reviews
Mixed methods reviews
Context-specific versus multi-context syntheses
Quantitative Review
Qualitative Review
Context-specific versus multi-context reviews
Multi-context reviews
Context-specific reviews
Exhaustive search
Selective search
Little access to or knowledge of local databases
and experts
(related to context)
Access to and knowledge of local databases
and experts
Targets a broad audience (but no-one in particular)
Findings may be too general
Risk of downplaying important local characteristics
Context may get lost
Only relevant to the ‘happy few’.
Findings are less likely transferable to
other settings
Potential low level of acceptance in end-users
Wide ranging in scope
Targeted audience
Ability to cross compare different settings
Highly relevant to practice and policy
Works for topics were little heterogeneity between
settings is expected
Maintains integrity with the context
reported in original studies
Findings are more likely transferable to a broad range of
settings
Findings may induce a higher level of
acceptance in the end-users
Context-specific versus multi-context reviews
• Context-specific syntheses do well in responding to the needs
and policies of a targeted setting.
• Multi-context syntheses assist in building a cumulative
knowledge base and are an excellent choice when little
heterogeneity is expected.
• Integrating the best of both
– Umbrella reviews, in which insights in a particular phenomenon
generated from different settings could be summarized.
– Transcontextual adaptation, which means modifying insights in such a
way that they become relevant and reply to the needs and policies of
a targeted setting.
QES: conclusion
• If you wish to embark in the field of qualitative
evidence synthesis you will soon realise that
– There are no fixed standards
– There is no general consensus (and will there ever be
one?)
– You shall have to travel slowly,
– By your own means (but the CQRM-group can help),
– On small and potentially difficult roads…
• BE PREPARED FOR A POTENTIAL DELAY!