Quality Appraisal of Qualitative
Studies
Alison Cooke
Midwife &
NIHR Doctoral Research Fellow
ESN Presentation 050214
Why appraise quality?
• A systematic review is a scientific exercise
which may influence healthcare provision
• The quality of a review will be influenced by
the methods used to minimise error and bias
• There is a plethora of published work on how
to appraise quality of research
• Whilst there are some differences in how to
appraise, there is a high level of agreement on
the key components of quality appraisal
What is metasynthesis?
“A systematic review of qualitative research”
(Booth 2001)
“Seeks to develop and refine theories while retaining
the uniqueness of individual studies”
(Jensen 2004)
“The science of summing up”
(Light & Pillemer 1984)
Types of qualitative synthesis
•
•
•
•
•
•
•
•
Meta-ethnography (Noblit & Hare)
Grounded Theory (Kearney / Eaves / Finfgeld)
Thematic Synthesis (Thomas & Harden)
Textual Narrative Synthesis (Lucas et al.)
Meta-study (Paterson et al.)
Meta-narrative (Greenhalgh et al.)
Critical Interpretive Synthesis (Dixon-Woods et al.)
Ecological Triangulation / Ecological Sentence Synthesis
(Banning)
• Framework Synthesis (Brunton et al. / Oliver et al.)
Noblit & Hare
• Meta-ethnographic approach: to reveal
analogies between studies, making sense of
what the collection of studies is saying
• Line-of-argument synthesis; the interpretive
synthesis is concerned with inference – what
can we say of the whole phenomenon based
on selective studies of the parts?
• Repeated comparison between studies
reveal similarities and differences
put
findings into new interpretive context
Quality Appraisal
• Best tool for assessment is often defined by type of
metasynthesis being conducted, e.g.
 specific 10-point tool for Framework synthesis
 DIAD tool for ecological triangulation
 JBI specific 10-point checklist
• Over 100 tools and frameworks are available
Are rigid checklists appropriate?
Role of expert judgement?
Insufficient evidence to inform judgement on rigour or
added value of various approaches (Noyes et al., 2008)
Which tool to use?
Rigid
yes/no
checklist
Expert
review
Advantages/Disadvantages
Rigid checklist
• Too restrictive for qualitative appraisal
• More suited to quantitative appraisal; more objective
• Validated tools; reader aware of robustness of
quality assessment process
Expert review
• Experienced reviewer reads the paper as a whole
and provides assessment of quality
• Not credible; reader not able to assess robustness of
quality assessment process
• Highly subjective
Optimum tool
• Somewhere along the continuum between
‘rigid checklist’ and ‘expert opinion’
• Best ‘fits’ the type of metasynthesis being
conducted
• Validated, or has been used many times in the
topic area
• Provides an organised and systematic
approach which can be replicated between
reviewers, but also by the reader
Worked example
• Quality assessment tool developed by Walsh &
Downe; 2005
• Grading tool developed by Downe et al. 2009, based
on work by Lincoln & Guba, 1985.
Stages
Essential Criteria
Scope and Purpose
Clear statement of, and rationale for, research
questions/aims/purposes
Study thoroughly contextualised by existing literature
Design
Method/design apparent, and consistent with
research intent
Data collection strategy apparent and appropriate
Sampling Strategy
Sample and sampling method appropriate
Analysis
Analytic approach appropriate
Interpretation
Context described and taken account of in
interpretation
Clear audit trail given
Data used to support interpretation
Reflexivity
Researcher reflexivity demonstrated
Ethical Dimensions
Demonstration of sensitivity to ethical concerns
Relevance and
Transferability
Relevance and transferability evident
Notes
Grading System
created by Downe et al based on the work by Lincoln & Guba
A
No, or few flaws. The study credibility, transferability,
dependability and confirmability is high.
B
Some flaws, unlikely to affect the credibility, transferability,
dependability and/or confirmability of the study.
C
Some flaws that may affect the credibility, transferability,
dependability and/or confirmability of the study.
D
Significant flaws that are very likely to affect the credibility,
transferability, dependability and/or confirmability of the
study.
Activity
• Read the extract from Dobrzykowski
et al., 2003
• Look at the explanatory notes for use
of relevant sections of assessment
tool
• Complete your quality assessment of
this extract
Completed assessment: 1
Design
Sampling
Strategy
Method/design
apparent, and
consistent with
research intent
Grounded theory: rationale provided.
Discussion of rationale for design choice.
Setting appears appropriate.
Data collection
strategy apparent
and appropriate
Face to face interview, telephone
conversation, email (first author).
Tape recorded and transcribed verbatim.
2-3 interviews; appears sufficient to
capture complexity/diversity.
Sample and
sampling method
appropriate
n=53
Data and emergent categories dictated.
Sample selection process, justification
explained.
Thickness of description likely.
Completed assessment: 2
Design
Sampling
Strategy
Method/design
apparent, and
consistent with
research intent
Clear rationale given for the qualitative
approach, used grounded theory in order
to explore processes and symbolic
meanings for study group.
Data collection
strategy apparent
and appropriate
In-depth interviews in person, by
telephone and follow up interview by
email.
Sample and
sampling method
appropriate
Yes, fully explained.
Consensus
• All 3 reviewers gave a grade A/B for this study
(Very few flaws. The study credibility, transferability,
dependability and confirmability is high)
• Important to have a meeting following
independent quality assessment to reach
consensus over grading and an agreed
decision over inclusion and exclusion of
studies for synthesis
Flaws to determine grading
• “No evidence of systematic approach to literature
review”
• “Reflexivity unclear: no discussion of researchers
influence on participants etc.”
• “Limitations of study not discussed”
• “No evidence of ethical approval”
• “No evidence of member checking”
• “Unlikely to capture thickness of data”
• “?Bias: only one researcher involved in data
collection and analysis process”
Should studies be excluded?
Grade D = “Significant flaws that are very likely
to affect the credibility, transferability,
dependability and/or confirmability of the
study”
Should we exclude in a metasynthesis any
studies found in answer to a research
question?
Argument to include all?
• Edwards et al., 1998
Advocate balancing an assessment of
methodological quality against the weight of its
message – the ‘signal to noise ratio’, rather than
excluding studies that fall below a certain quality
threshold
• Booth, 2001
If research is rejected on the basis of design
alone there is a high risk of denying valuable
insights that contribute to our interpretation of
a phenomenon
Challenges of qualitative
evidence synthesis
• Literature searching: poor indexing of qualitative
studies in databases (use SPIDER! Cooke et al. 2012)
• Journal word count may mean that points
appraised may be badly explained or not included
• Appraisal techniques are dominated by the
quantitative paradigm
• Structured approaches may not provide greater
consistency of judgements to include or exclude
• Subjective judgement
• Interpretive, rather than aggregating, intent
• Time consuming
In summary
• There is no ‘right’ way of
conducting quality
assessment of qualitative
research
• Having a structured
approach can provide the
reader with clarity of the
decision process used
• Multiple reviewers required
to review papers
independently, then meeting
to reach a consensus of
decision to include/exclude
Thank you
Any questions?
Alison.Cooke@manchester.ac.uk
Reference List










Banning J. undated. Design and Implementation Assessment Device (DIAD) Version 0.3: A response from a
qualitative perspective [Website] Available from:
http://mycahs.colostate.edu/James.H.Banning/PDFs/design%20and%20implementation%20assessment%
20device.pdf [Accessed: 03/02/14]
Banning J. undated. Ecological Triangulation [Website] Available from:
http://www.mychhs.colostate.edu/James.H.Banning/PDFs/Ecological%20Triangualtion.pdf [Accessed:
03/02/14]
Banning J. undated. Ecological Sentence Synthesis [Website] Available from:
http://www.mychhs.colostate.edu/James.H.Banning/PDFs/Ecological%20Sentence%20Synthesis.pdf
[Accessed: 03/02/14]
Booth A. 2001. Cochrane or cock-eyed? How should we conduct systematic reviews of qualitative
research? Qualitative Evidence-based Practice Conference, Coventry University, May 14-16
Brunton G, Oliver S, et al. 2006. A synthesis of Research Addressing Children’s, Young People’s and
Parents’ Views of Walking and Cycling for Transport EPPI-Centre, London
Cooke A, Smith D & Booth A. 2012. Beyond PICO: The SPIDER Tool for Qualitative Evidence Synthesis.
Qualitative Health Research 22(10) p1435-43
Dixon-Woods M, Cavers D, et al. 2006. Conducting a critical interpretive synthesis of the literature on
access to healthcare by vulnerable groups. MBC Medical Research Methods 6(35)
Downe S Finlayson K et al. 2009. ‘Weighing up and balancing out’: a meta-synthesis of barriers to
antenatal care for marginalised women in high-income countries BJOG 116 p518-529
Eaves YD. 2001. A synthesis technique for grounded theory data analysis. Journal of Advanced Nursing 35
p654-663
Edwards AG, Russell LT & Stott NCH. 1998. Signal versus noise in the evidence base for medicine: an
alternative to hierarchies of evidence? Family Practice 15(4) p319-322
Reference List













Finfgeld D. 1999. Courage as a process of pushing beyond the struggle. Qualitative Health Research 9
p803-814
Greenhalgh T, Robert G, et al. 2005. Storylines of research in diffusion of innovation: a meta-narrative
approach to systematic review. Social Science & Medicine 61 p417-430
Jensen LA. 2004. Extending meta-analysis. Qualitative Health Research 14(10) p1346-1347
Kearney MH. 2001. Enduring love: a grounded formal theory of women’s experience of domestic violence.
Research in Nursing & Health 24 p270-282
Light RJ & Pillemer DB. 1984. Summing up: the science of reviewing research. Havard University Press,
Cambridge, MA
Lincoln YS & Guba EG. 1985. Naturalistic Inquiry. Sage Publications, California
Lucas PJ, Arai L, et al. 2007. Worked examples of alternative methods for the synthesis of qualitative and
quantitative research in systematic reviews. BMC Medical Research Methods 7(4)
Noblit GW & Hare RD. 1988. Meta-Ethnography: Synthesizing Qualitative Studies (Qualitative Research
Methods Series 11). Sage Publications, California
Noyes J, Popay J, et al. 2008. Chapter 20: Qualitative research and Cochrane reviews. In: Higgins JPT &
Green S (Eds) Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons, Chichester
Oliver S, Rees R, et al. 2008. A multidimensional conceptual framework for analysing public involvement in
health services research. Health Expectations 11 p72-84
Paterson BL, Thorne SE, et al. 2001. Meta-study of Qualitative Health Research. A Practical Guide to MetaAnalysis and Meta-Synthesis. Sage Publications, California
Thomas J & Harden A. 2008. Methods for the thematic analysis of qualitative research in systematic
reviews. BMC Medical Research Methods 8 p45
Walsh D & Downe S. 2006. Appraising the quality of qualitative research. Midwifery 22 p108-19