Achieving Appropriate Rigor in
Qualitative Research
Research Day, February 4, 2011
Mary Katherine O’Connor, Ph.D.
School of Social Work,
Virginia Commonwealth University
mkoconno@vcu.edu
Presentation Goals



Propose a multi-paradigmatic heuristic for
understanding variety in qualitative
research
Detail differential standards for research
quality depending upon paradigmatic
perspective
Aid in matching research questions,
analytical needs and computer-based
qualitative analysis packages
Two Dimensions of Analysis


Subjective/objective relates to
assumptions about the nature of
knowing (and, therefore, of
science)
Regulation/radical change relates
to assumptions about the nature
of society (and, therefore,
appropriate goals for science)
Burrell & Morgan’s
Framework
Sociology of Radical Change
Radical
Humanist
Radical
Structuralist
Subjectivity
Objectivity
Interpretive
Functionalist
Sociology of Regulation
Multiple Paradigms




Fundamentally different perspectives for
analysis of social phenomena
Generating different concepts and
analytical tools
Different standards for good research
practice
Four different types of qualitative research
building from different types of research
questions and research goals
Multiple Paradigms in
Qualitative Research
Radical Change
Critical Interpretive
Subjective
Critical Positivist
Objective
Positivist/
Post Positivist
Interpretive
Regulation
Research Goals
Radical Change
Subjective
Objective
Positivist/
Post Positivist
Regulation
Positivist Research:
UNDERSTANDING FOR THEORY BUILDING
THAT CAN RESULT IN THEORY TESTING



Assumptions: Rational, pragmatic methods
of natural science are appropriate to study
human affairs
Questions: Useful for theory building in the
early stages of theory development
Standards: understanding for
generalizability including random
assignment, control groups, valid and
reliable data collection, standardized data
analysis
Radical Change
Critical Positivist
Subjective
Objective
Regulation
Critical Positivist Research:
CONSCIOUSNESS RAISING AND CHANGE AT
THE SYSTEM LEVEL




Assumptions: Knowledge is for radical/transformative
change, emancipation and potentiality; realist but
concentrating on structural relationships to understand
and generate fundamental conflicts that will be the
basis for radical change at the class/structural level
Questions: Focus on structural relationships seeking to
provide explanations of the basic interrelationship
within the context of total social formation for the
purpose of consciousness raising
Designs: critical ethnographies, cooperative or
collaborative inquiry, participatory action research,
appreciative inquiry, empowerment evaluation
Standards: same as functionalist qualitative inquiry, but
in addition, measurable presence of increased
sophistication with change or change potential
Radical Change
Subjective
Objective
Interpretive
Regulation
Interpretive Research:
MEANING MAKING




Assumptions: Understanding at the level of
subjective experience from perspective of
participants, not observer. Relativist in that the
world, though ordered, is an emergent social
enterprise that is continually being created.
Questions: seeking explanation within individual
consciousness and subjectivity from standpoint
of the participants rather than the observer
Designs: ethnographies, case studies,
phenomenological studies, constructivist studies
Standards: sampling for maximum variation,
emergent design, multiple data collection
approaches, triangulation, reflexivity, inductive
data analysis, thick description of research
product
Radical Change
Critical Interpretive
Subjective
Objective
Regulation
Critical Interpretive Research:
CONSCIOUSNESS RAISING AND CHANGE AT
THE INDIVIDUAL LEVEL




Assumptions: Critical interpretive with radical
change focus, but from a subjectivist, individualistic
standpoint. Concentrating on human consciousness
and the social structures that inhibit true fulfillment.
Goal is release of constraints that hamper individual
human development.
Questions: how individual makes sense of life and
experiences of societal constraints
Designs: interview studies, narrative research, life
histories, autoethnographies, collective biographies
Standards: more artistic or aesthetic including
creativity, aesthetic quality, interpretive vitality, and
degree of stimulation for change
Research Goals: Implications



All qualitative research is not based on the same
ontological and epistemological assumptions
All qualitative research cannot be expected to be
held to a universal standard of rigor, quality or
worth
Key to justification of qualitative processes and
products is selection of appropriate research
design including data collection and analysis
techniques able to answer research question
lodged within specific paradigmatic perspective
QUESTIONS?
Research Design
Positivist Design:





An appropriate research design could include structured
interviewing, focus group, grounded theory, etc. and should be
identified and justified along with a data collection mechanism.
The sampling plan is preferably randomized or at least looking for
maximum variation along identifiable dimensions.
An “n” of at least 30 or a justification for a lesser number that is
similar to a justification for a lesser sample number in a
quantitative project.
A data analytic plan (thematic, content, constant comparison,
linguistic, etc.) that matches the question, including whether or not
computer based text analysis is being used. When computer
analysis is present, the selected program must be able to do the
analysis proposed. In most cases the variables for analysis should
be identified beforehand. The exception would be in grounded
theory designs.
Inter-rater reliability should also be addressed.
Critical Positivist Design:






An appropriate research design could include
empowerment, appreciative, focus group, etc. and
should be identified and justified along with a data
collection mechanism.
The sampling plan is preferably randomized or at least
looking for maximum variation along identifiable
dimensions.
An “n” of at least 30 or a justification for a lesser number
that is similar to a justification for a lesser number in a
quantitative project.
A data analytic plan (thematic, content, constant
comparison, linguistic, etc.) that matches the question,
including whether or not computer based text analysis is
being used. When computer analysis is present, the
selected program must be able to do the analysis
proposed. In most cases the variables for analysis
should be identified beforehand.
Inter-rater reliability should also be addressed.
Change resulting from the research will need to be
measured.
Interpretive Design:




An appropriate design could include ethnomethodological,
constructivist, phenomenological, etc. and should be identified
and justified. If the design is emergent, benchmarks for the
emergence should be clear including how sampling and data
collection will begin and might change.
Sampling for maximum variation is preferred and should
identify the dimensions of variation of interest, the
stakeholding groups to be sampled, and what might constitute
saturation in determining the end of sampling and data
collection.
A data analytic plan and the type of analysis should match the
question and if computer-based analysis is being used, the
program should be geared to interpretive analysis. Variables
for analysis will rarely be identified beforehand; therefore,
content analysis and constant comparison would be preferred.
Inter-rater reliability is not favored. Instead, member checking
with participants is preferred.
Critical Interpretive Design:





An appropriate design could include an interview study,
narrative research, life history, etc. and should be identified
and justified. If the design is emergent, benchmarks for the
emergence should be clear including how sampling and data
collection will begin and might change.
Sampling will tend to be for a single case; but if the “n” is
greater, maximum variation should identify the dimensions of
variation of interest, the stakeholding groups to be sampled,
and what might constitute saturation in determining the end of
sampling and data collection.
A data analytic plan and the type of analysis should match the
question and if computer-based analysis is being used, the
program should be geared to interpretive analysis. Variables
for analysis will rarely be identified beforehand; therefore,
content analysis and constant comparison would be preferred.
Inter-rater reliability is not favored. Instead, member checking
with participants is preferred.
Plans for documenting individual change as a result of the
process can be noted.
Data Analysis &
Reporting
Paradigmatic Implications


Content Analysis
Thematic Analysis
Linguistic Analysis
Semiotics
Constant Comparison
Reporting
Count
Case Study
Narratives
Content Analysis




Thematic Analysis - reading “chunks” of data for
meaning and attaching a meaning label (can
then move to word count/distribution analysis)
Linguistic Analysis - assessing word usage,
message style (can then move to word
count/distribution analysis)
Semiotics - reading for meaning through use of
metaphor (can then move to word
count/distribution analysis)
Constant Comparison - deconstruction and
reconstruction through unitization and
categorization (can then move to word
count/distribution analysis)
Reporting



Count- data display, distribution tables,
conceptual or causal maps
Case Study – to chronicle (to record temporally
and sequentially as in a history); to render (as in
a description or to provide a vicarious
experience); to teach (as instructional material);
to test (using case as a trial for certain theories
or hypotheses)
Narrative – factual (he said/she said);
interpretive (novel like); evaluative (elaborative
judgments for evaluative testing)
Computer Assisted Qualitative
Data Analysis Software
(CAQDAS)
Misconceptions about software:




A primary research instrument
A separate form of analysis
Enhances rigor
Packages have similar paradigmatic
assumptions
Data Analysis with CAQDAS
Practical Considerations



Available technology
Skills and abilities
Time
Political Considerations


Institutional expectations (IRB, etc.)
Peer review/professional/funding expectations
Paradigmatic Considerations


Basic design assumptions
Package similarities and nuanced differences
CAQDAS Disciplinary
Backgrounds



Atlas.ti – Interdisciplinary,
Psychology, Linguistics, Computer
Science
NVivo – Sociology
MAXqda – Political Science
CAQDAS Similarities







Unitizing
Coding
Annotating
Retrieving
Querying
Graphic representation
Writing Support
CAQDAS Differences



Types of data
Coding structure
Contextualization
However…
…When considering qualitative data
analysis software:
Access + Awareness 
Paradigmatic Congruence
Conclusion
When there is congruence between paradigmatic
perspective, research goal and selected processes,
articulation of what can and should be expected for
accountability standards are possible and usefulness
can be asserted.




Positivist work cannot provide deep,
individualized meaning, but can be basis for
appropriately targeted quantitative work or
provide evidence to support causal hypotheses.
Critical Positivist work must engender class or
structural changes from objective, generalizable
data.
Interpretive work is not generalizable, but can
provide complex, context-based deep
understanding.
Critical Interpretive work must engender
individual, subjective changes at the intersection
of art, spirituality and scientific ways of knowing.
Protocol Reviews
(or…How do you defend what you are
doing? Or assess student work?)



Try to determine underlying assumptions in
order to establish the paradigmatic location
of the research by evaluating the aims or
goals.
Look for mixing of assumptions that suggest
an incorrect understanding that qualitative
research is always interpretive research
Apply the standards for design and analysis
quality consistent with the paradigm.
Words to the Wise
for the Researcher



Be aware of underlying assumptions in project
that establish the paradigmatic location of the
research, generally best articulated by project
aims or goals.
Avoid mixing of assumptions that suggest an
incorrect understanding that qualitative research
is always interpretive research.
Articulate and demonstrate the standards for
design and analysis quality consistent with the
paradigmatic perspective of the project…then
choose your software