Excellence in
Qualitative Research
Kathleen A. Knafl, PhD, FAAN
Associate Dean for Research
Frances Hill Fox Distinguished Professor
The University of North Carolina at Chapel Hill
Learning to Evaluate Qualitative
Research
 Striving for quality as a researcher
 Being evaluated – successful/unsuccessful
attempts to publish and secure funding
 Evaluating others – proposal and
manuscript reviewer
Overview of Presentation
 Nature and purpose of qualitative research
 Position statements and guidelines for
evaluating qualitative research
 Common expectations and indicators of
excellence
Overall Goal
 Avoid an overly simplified or overly
complex discussion of issues
 Present guidelines that address scientific
rigor while recognizing the distinct
qualities and contributions of qualitative
research
“Creativity must be preserved within
qualitative research, but not at the
expense of the quality of the science”
(Maxwell, 1990)
Nature and Purpose of Qualitative
Research
 Multiple approaches
 Terminological jungle
 Ethnography
 Phenomenology
 Hermeneutics
 Constructivism
 Participatory action research
 Grounded theory, etc…
Common Characteristics of
Qualitative Research
 Focus on subjective perspective of
respondents
 Emphasis on contextual understanding
 Design flexibility; emergent design
 Generation of narrative data
 Researcher as primary data collection
instrument
Differences Across Approaches
 Purpose
 Philosophical underpinnings
 Disciplinary roots
 Theoretical orientation
 Dominant methods for data collection
and analysis
Varying Purposes of Qualitative
Research
 Description
 Conceptualization and theoretical
understanding
 Empowerment and social change
Description (e.g. descriptive phenomenology,
ethnography, qualitative description)
“The purpose of this study is to explore how
spousal carers of people with MS interpreted
their lived experience with their partner, the
way in they assigned meaning to their being
in such a situation, and the skills and
knowledge they have developed to live with
their situation” (Cheung & Hocking, 2004, p. 155).
Conceptualization
(e.g. grounded theory, concept development)
“The author investigated decision-making
experiences of 20 surrogates who assisted
terminally ill family members for this
grounded theory study. Findings describe a
basic social process of Seeing them Through
with Care and Respect” (Meeker, 2004, p.204).
Empowerment and Change
(e.g., feminist, participatory action research)
“In this participatory research study, injured
workers and academics together participated in
setting the agenda, determining the research
questions and methods, gathering the data, and
interpreting the results. The process helped all
understand more deeply the complicated reality of
being an injured worker within a set of system and
societal norms (Beardwood, et al., 2005, 33-344).
The Challenge of Evaluating
Qualitative Research
 Evaluative stance – Should the same
criteria apply to all research? All qualitative
research?
 Reasonable expectations – What should
we expect to see included in a qualitative
report/proposal?
 Appropriate criteria – What standards
should we use to judge the merits of a
qualitative report/proposal?
Guidelines and Criteria
 Guidelines for reporting (Tong, et.all, 2007)
 Shared standards (Collingridge & Gantt, 2008;
Guba & Lincoln, 1985)
 Alternative standards (Cheek, 2007; Cohen &
Crabtree, 2008; Davies & Dodd, 2002; Fossey, et
al., 2002; Whittemore, et al., 2001)
Shared Standards (Lincoln & Guba, 1985)
 Truth Value (internal validity/credibility)
 Applicability (external validity/
transferability)
 Consistency (reliability/auditability)
 Neutrality (objectivity/confirmability)
Qualitative
standard
Credibility (internal validity)
 Defined as adequate representation of
group or situation studied
 Quality of the data
 Demonstrated by prolonged engagement,
persistent observation, triangulation,
external checks, negative case analysis,
member checking
Transferability (external validity)
 Defined as the degree of similarity between
research site/participants and others
 Demonstrated by thick description, reporting
of information for judgments about
transferability
 Joint responsibility of researcher and
consumer
Dependability (reliability)
 Defined as differentiating real
perspective/behavior of respondents from
reactive effects of research.
 Demonstrated by audit of data collection
processes
 Quality of interaction with subjects
 External review of data
Confirmability (objectivity)
 Defined as the adequacy of the results,
interpretations, and recommendations
 Demonstrated by audit of analytic
procedures
 Data processing and reduction
 Data review and interpretation
 External auditor
Alternative Standards (Davies & Dodd)
“Incorporating the notion of ethics, the authors
develop a cluster of terms around which they
argue that qualitative research can
meaningfully speak about rigor:
attentiveness, empathy, carefulness,
sensitivity, respect, reflection,
conscientiousness, engagement, awareness
and openness” (Davies & Dodd, 2002, p. 279)
Common Evaluative Criteria
(Cohen & Crabtree, 2008; Tong, et al., 2007;
Whittemore, 2001)
 Importance of the research
 Use of appropriate methods
 Coherence of the research
 Clarity of the presentation
 Ethical conduct of research
Application of Common Criteria
to Research Reports/Proposals
 Reasonable expectations
 Indicators of excellence
Importance of Research
Reasonable expectations
 General statement of purpose, research
questions, or aims
 Review of relevant literature
Importance of Research
Indicators of excellence
 Evidence of building on prior research;
addressing an important gap
 Pragmatically or theoretically useful
 Focus on the research problem, not the method
Making the Case for Importance
“The aim of this study was to identify approaches
primary care providers use to engage older men
in depression care. We focus specifically on
older men because previous work has shown
that they suffer from higher levels of undertreatment and that men differ from women in
gender-specific barriers to help seeking”
(Apesoa-Varano, et al., 2010, p. 587)
Use of Appropriate Methods –
Study Design
Reasonable expectations
 Varied approaches
 Emergent design
Indicators of excellence
 Rationale for selection of approach;
fit with study purpose
 Citations in support of the approach
Rationale for Study Design
“We chose grounded theory as the method for
data collection and analysis. It is a method for
conceptualizing patterns of behavior in which
people are engaged (Glaser,1978;1998). In
this study, the patterns of behavior are those in
which nurses engage while caring for palliative
cancer patients in hospitals” (Sandgren, et al., 2006,
p. 80).
Use of Appropriate Methods –
Role of Theory (Sandelowski, 2003)
Reasonable expectations
 Variation in the source, function, and temporal
placement of theory
 Different qualitative traditions specify different
roles of theory
 Prevailing (though misleading) belief that
qualitative research is atheoretical
Use of Appropriate Methods –
Role of Theory
Indicators of excellence
 Explicit statement of philosophical beliefs,
concepts, or theories that informed design
of study
 When a concept/theory is an outcome of the
research, comparison to related concepts or
theories
Example - Use of Theory
“The philosophical framework for this study
was based on interpretive phenomenology”
(Cheung, 2004, 155).
“The purpose of this study was to examine
older African American women’s perceptions
of social and environmental stress in relation
to their heart disease through the lens of the
weathering conceptual framework” (WarrenFindlow, 2007, 234).
Sample Size and Selection
Reasonable expectations
 Purposive approach; non-probability sample
 Relatively small sample
 Single case to 50+ participants
 Likely range of number of participants
Sample Size and Selection
Indicators of excellence
 Statement of purposive intent (e.g. maximum
variation, theoretical, intensity)
 Specification of unit of interest
(individual, group, setting)
 Rationale for sample size; invoking the principle
of saturation
Purposive Sample – Maximum
Variation
“The base line criterion for inclusion in the
sample was that all parents were employed and
had at least one child whom they identified as
disabled. The aim was to generate a sample that
included a range of situations in which parents
combined employment and care. This was not
intended to achieve generalizability, but to
enable examination of issues not related to a
specific homogenous group” (Lewis, et. al, 2000,
p.1035).
Purposive Sample – Theoretical
“ As data collection progressed, I conducted
theoretical sampling to provide data needed
to describe the categories thoroughly. For
example, because early participants were all
reporting that other family members had
been very supportive, I sought participants
who had experienced conflict (Meeker, 2004,
p.208) .
Data Collection –
Interviewing and Observation
Reasonable expectations
 Use of single or multiple data collection
strategies
 Description of:
• Setting for data collection
• Duration of data collection
• Identification of data collectors
• Description of observational or interview guides
• Method for recording data
Data Collection –
Interviewing and Observation
Indicators of excellence
 Overall explicitness and thoroughness
• Description of role of the investigators; nature of
interactions with the participants
• Options for follow-up contact with participants
• Efforts to assure and monitor data quality
• Evidence of sensitivity to incoming data;
emergent design
Example – Interviews
Each participant was told, “I’d like you to tell me
the story of your experience with a chronic health
condition. Start at the time your symptoms began
and describe the things that happened one after
another regarding your health condition until
today. I encourage details because whatever is
important to you is of interest to me” (Lee & Poole,
2005, 349).
Example - Observation
“I collected data through participant observation
in two cardiac rehabilitation programs. In both,
I assisted in the day-to-day work of the staff,
attended education sessions, and conversed
with participants on an ongoing basis. When I
introduced myself to clients, I indicated I was
there to learn about heart disease from the
participants’ point of view and that I was carrying
out research (Wheatley, 2005, p. 440-441).
Data Processing and Analysis
Reasonable expectations
 Overview of the steps the researcher took to
break the data into smaller units for the purpose
of analysis
 Methodological citations that support the
approach
 Variability in use of software programs to
support analysis
Data Processing and Analysis
Indicators of excellence
 Analytic process is consistent with the
qualitative approach
• Line by line coding for grounded theory
• Extraction of significant statements for
phenomenology
 The unit of study is preserved in the analysis
 Evidence of thoroughness and checks
on quality
 Explicitness
Results
Reasonable expectations
 Variation in organizing structure for
presenting results
• Process
• Essential structure
• Thematic description
• Typology
 Use of illustrative quotes or vignettes
 Variable use of numbers
Results
Indicators of excellence
 Vividness
• Compelling presentation
• A good read
• Creative
 Coherence
• Integration of the data; more than a description
of codes and themes
• Convincingly addresses all study aims
Discussion
Reasonable expectations
 Possibly integrated with results
 Linkages to the body of knowledge in the field
 Statement of applied and/or theoretical
implications of the findings
 Limitations and next steps
Discussion
Indicators of excellence
 Elaboration of pragmatic/theoretical implications
 Limitations that address the unique aspects of
qualitative research (e.g. non-probability sample
not a limitation)
 Explicit consideration of transferability
Ethics
Reasonable expectations
 Statement that IRB approval has been obtained
Indicators of excellence
 Evidence of sensitivity to human, social, and
cultural contexts
 Recognition of ongoing ethical issues and
decisions
Overall Coherence and Consistency
 Alignment across all aspects of the study
 Completeness and follow through
 Explicitness; thoroughness of presentation
 Creativity – thoughtful, innovative use of
methods; insightful linkages
Balancing Rigor and Creativity
“We can preserve or kill the spirit of qualitative
work; we can soften our notion of rigor to
include playfulness, soulfulness, imagination,
and technique we associate with more artistic
endeavors, or we can further harden it by the
uncritical application of rules. The choice is
ours rigor or rigor mortis” (Sandelowski, 1993).
The University of North Carolina at Chapel Hill