Workshop: Thematic Synthesis
and Framework Synthesis
Parts 1-4 – Data Extraction, Quality
Assessment, Synthesising Across
Studies, Completing the Analysis
Workshop: Framework Synthesis,
Meta-Ethnography and Realist
Synthesis
Shared Topic: Adherence to Antiretroviral
therapy (ART) for HIV in Zambia
• BACKGROUND: Antiretroviral therapy (ART) has
significantly improved morbidity and mortality of
individuals infected with HIV. However, lack of
adherence to highly active antiretroviral therapy
(HAART) remains a key challenge to successful
management of patients with HIV/AIDS. Adherence
rates lower than 95% are associated with development
of viral resistance to antiretroviral medications.
• ‘Efforts to sustain adherence in Africa and elsewhere
remain important goals to optimize outcomes for
individuals and global HIV treatment.’ (Mills, Nachega,
Buchan, Orbinski, Attaran, Singh et al., 2006).
Shared Topic: Adherence to Antiretroviral
therapy (ART) for HIV in Zambia
• Different Emphases
– Barriers and Facilitators to ART (Framework
Synthesis)
– Theory Explaining Adherence to ART (Thematic
Synthesis)
Reading and Data Extraction
Data extraction
 What is it?

An attempt to reduce a mass of material (your
included papers) to a much smaller body of text
and numbers, amenable to analysis and the
interpretation of findings
Data extraction form











Location
Setting
Sample (n)
Age
Gender
Ethnicity
Socio-economic status
Intervention (if any)
Quality assessment criteria
Results?
Further citations


See Handouts 1-3
Format?????
What results do you extract?
 What is your question?
Keep the question in mind as you read:
Are the data relevant to this question?
Is the question answered by the data?
Framework Synthesis
Thematic Synthesis
Data Extraction
Extracts data against
framework. Coding
framework with definitions
provided to increase
consistency. Data not
explained by framework is
“parked” for subsequent
inductive stage. Distinction
typically made between
original data extracts and
author’s analysis.
Key themes and concepts
extracted and reviewed for
inclusiveness. Distinction
preserved between original
(participant) extracts and
(author’s analysis) findings.
Findings coded in duplicate.
Discrepancy between codes
resolved by third person.
Quality Assessment
Andrew Booth, Reader in Evidence Based
Information Practice, Co-Convenor –
Cochrane Collaboration Qualitative
Methods Group
Before You Begin…
• Consider how you will use judgements of
quality (cp. 50% of published Cochrane
Quantitative Reviews performed quality
appraisal but did not make it clear how
judgements were used!)
– To exclude or to moderate?
• Will chosen instrument militate against certain
types of research?
• Quality of reporting or quality of study?
Variability in Practice - 1
21 papers did not describe appraisal of candidate
studies
6 explicitly mentioned not conducting formal appraisal
of studies
5 papers did a critical appraisal, but did not use a
formal checklist
7 described modifying existing instruments
1 used an existing instrument without modification
Dixon-Woods, Booth & Sutton (2007)
Variability in Current Practice - 1
23 papers did not describe critical appraisal
5 papers explicitly pleaded against quality assessment
of papers or provided valid reason for not conducting
quality appraisal.
Criteria used varied between detailed descriptions of
relevant items in existing or modified checklists to a
set of broad criteria evaluating, for example, rich
description of data, credibility or relevance of the
original study.
Hannes and Macaitis (2012)
Variability in Current Practice - 2
One team used overall judgement (Smith et al., 2005).
Five opted for self-developed assessment instrument
Three used previously developed checklists to create own.
Two mentioned critical appraisal, but did not specify tool.
Most used existing instruments/frameworks. 24 different
assessment tools identified:
Critical Appraisal Skills Programme (CASP)
(n = 18)
Mays and Pope criteria
(n = 6)
Popay criteria
(n = 6)
Joanna Briggs Institute
(n = 4).
Hannes and Macaitis (2012)
Appraising research quality
1. Epistemological criteria: Judgement of ‘trustworthiness’
requires criteria tailored to different research ‘paradigms’.
2. Theoretical Criteria: Explicit theoretical framework shaping
the design of the study and informing claims for
generalisability
3. Prima facie ‘Technical’ criteria: Used to assess ‘quality’
common to all research traditions e.g.:
Sufficient explanation of background;
Method appropriate to question;
Succinct statement of objectives/research questions;
Full description of methods include approach to analysis;
Clear presentation of findings including justification for interpretation of
data etc.
Noyes J (2005)
Two dimensional approach to appraising
qualitative research
Technical markers –
CASP
Epistemological and theoretical
markers – Popay et al
Technical Quality High
Description – thicker
•Privileges Subjective experience and
meanings
•Use of theory to build explanations
Technical Quality Low
Description - thinner
•Imposed pre-determined framework on
respondents narratives.
•Limited/no/inappropriate use of theory,
little explanatory insight
(Noyes, 2005)
Available Tools - 1
• CASP – 10 questions to help you make sense of qualitative
research http://www.casp-uk.net/wpcontent/uploads/2011/11/CASP_Qualitative_Appraisal_Check
list_14oct10.pdf
• Joanna Briggs Institute - Critical Appraisal Checklist for
Interpretive & Critical Research
http://www.jbiconnect.org/agedcare/downloads/QARI_crit_a
pprais.pdf
• National Centre for Social Research. Quality in Qualitative
Evaluation: A Framework for Assessing Research Evidence.
London: National Centre for Social Research/UK Cabinet
Office, 2003 http://www.civilservice.gov.uk/wpcontent/uploads/2011/09/a_quality_framework_tcm638740.pdf
Available Tools - 2
• Dixon-Woods M, Shaw RL, Agarwal S & Smith JA (2004) The
problem of appraising qualitative research. Quality & Safety in
Health Care, 13, 223-5.
• Hannes K, Lockwood C, Pearson A (2010). A comparative
analysis of three online appraisal instruments' ability to assess
validity in qualitative research. Qualitative Health Research.
20(12):1736-43.
• Popay J, Rogers A & Williams G (1998) Rationale & standards
for the systematic review of qualitative literature in health
services research. Qualitative Health Research, 8, 341-51.
• Seale C & Silverman D (1997) Ensuring rigour in qualitative
research. European Journal of Public Health, 7, 379-84.
CASP – Technical/Procedural Tool
JBI – Theoretical Tool
Key Issue
• How are you going to use the quality
assessment?
– From quantitative assessment we know authors
frequently say they do it – but they don’t
incorporate it into results
– Is it technical proceduralism gone mad?
– Or can we use the assessments to improve our
synthesis and subsequent interpretation?
Quality Assessment
Framework Synthesis
Thematic Synthesis (e.g. as first
stage of Meta-Ethnography)
Pragmatic so tends to include all
studies . Focuses explicitly on
quality of reporting. Qualitative
sensitivity analysis used to test
robustness of synthesis.
Quality Assessment as Hurdle
(often used when plenty of studies
to draw upon):
Studies using qualitative design and
analysis method included.
Studies assessed for relevance first
to continue to full-text review.
Studies passing quality appraisal
(are retained.
Data Synthesis
What is Data Synthesis?
• Process of moving from focus on single studies
(cp. Data Extraction and Quality Assessment) to
focus on cross-study analysis
• Requires identification of patterns across data,
including contradictory findings and data that
does not fit
• Iterative and requires ongoing refinement
• Acts as prelude to Analysis which seeks to explain
patterns, contradictions and differences
Methods of qualitative evidence
synthesis
 Thematic synthesis; Critical Interpretive Synthesis;
Meta-ethnography
1. Only include “good” qualitative studies (?)
2. Constant comparison; iterative; interpretations generated from the data
by reviewers
3. Create a theory
– Inductive (theory-generating)
 Examples:
 Thomas J, Harden A. Methods for the thematic synthesis of
qualitative research in systematic reviews. BMC Medical Research
Methodology 2008; 8.
 Campbell R et al. Evaluating meta-ethnography: a synthesis of
qualitative research on lay experiences of diabetes and diabetes
care. Social Science & Medicine 2003; 65:671-684.
Methods of qualitative evidence
synthesis
 Framework synthesis:
1. Only include “good” qualitative studies (?)
2. Map data from included studies onto an existing framework to test the
framework/theory (a role for theory)
3. Build a conceptual model or framework
– Deductive (theory-testing)
 Examples:
 Oliver S et al: A multidimensional conceptual framework for analysing
public involvement in health services research. Health Expectations 2008,
11:72-84.
 Brunton G, Oliver S, Oliver K, Lorenc T. A Synthesis of Research Addressing
Children’s, Young People’s and Parents’ Views of Walking and Cycling for
Transport London. London, EPPI-Centre, Social Science Research Unit,
Institute of Education, University of London; 2006.
Methods of qualitative evidence
synthesis
 “Best-fit” framework synthesis
1. Identify relevant pre-existing conceptual models or frameworks
2. Identify and extract all relevant qualitative studies satisfying review’s
inclusion criteria
3. Code data from included studies against framework
4. Use secondary thematic analysis/synthesis to generate completely new
themes to supplement the framework’s themes
5. Create new framework and conceptual model or theory
 Deductive and Inductive
 Framework and Thematic synthesis

Carroll C, Booth A, Cooper K. A worked example of “best-fit” framework
synthesis: A systematic review of views concerning the taking of potential
chemopreventive agents, BMC Medical Research Methodology 2011; 11: 29
Data Synthesis
Framework Synthesis
Thematic synthesis (may be taken
forward as Meta-Ethnography)
Original best fit framework is
expanded to include new themes.
Relationship between themes is
examined and the data is used to
reconstitute a new model. Particular
attention is directed at discrepant
cases.
Second-order constructs pertinent
to adherence identified and crosscompared and presented in results
section. Key themes consolidated
into line of argument (third-order
analysis), presented in the synthesis
⁄ discussion section.
Booth et al, 2011
Some Practicalities
• Tabulation of data – looking for and
explaining differences (e.g. majority…, split…,
exception…)
• Post-Its – arranging according to patterns or
clusters
• Mapping e.g. Mind Map, Process Maps (e.g.
Pathways of Care), Logic Models
• Integration (with quantitative) – congruence,
contradictions, gaps with explanation