Qualitative Data Analysis
Chapter 11, Research Methods for Public Administrators
Dr. Gail Johnson
Revised & expanded for PADM 522, Designing Applied Research
Dr. Mario Rivera
Dr. G. Johnson,
www.ResearchDemystified.org
1
Analyzing Qualitative Data
A process of making sense of non-numeric data. Data from:





Narrative documents (speeches, newspapers, diaries, reports, etc)
Open-ended interviews
Open-ended questions on a survey
Case study (as the principal method or as embedded in a larger
complex of qualitative data and forms of analysis).
Often, qualitative evaluation research incorporates cross-case
analysis: For instance in evaluating two sites of the same program,
or in assessing a program site against another without the
intervention. An example might be a school with an extended
learning program to others without it.
Dr. G. Johnson, with additional material by
Dr. M. Rivera
2
Qualitative Data Analysis
Data From




Focus groups transcripts; interview transcripts and notes
Unstructured observations
Document review
Videos, etc.
The goal is to identify common themes



Requires a solid plan, attention to detail, good organization and
sufficient time
Whether the analysis is done by computer or by hand, it is often
necessary to develop a coding scheme so the data can be
systematically organized and analyzed
Computer software can help locate and organize data according
to the coding scheme created
Dr. G. Johnson & Dr. M. Rivera
3
Qualitative Data Analysis





But the analysis—that is, making sense of the data, discovering
the story the data reveals—is done by the researcher(s)
The greatest concern is bias/paradigm blinders: hard to
recognize things you do not expect, or to avoid finding things
you are inclined to find.
Maintaining the Thread
 Review the data, make notes as you go along
 Read again once all the data collection is completed
Organize the data—according to research questions, by date, by
geographic location—what makes sense given the situation
Begin to look for recurrent themes. In one cross-case evaluation
of a family-based drug education program, for instance, the
instructor (Rivera) found clear commonalities and divergences
in program-development in two Navajo communities in New
Mexico.
Dr. G. Johnson, Dr. M. Rivera
4
Basic Activities: Preliminary Analysis



Identify common words, ideas, themes
Develop spreadsheet or write on index cards
Identify “quotable quotes”—that is, quotes that highlight
the key issues: general views, divergent views, a range
views. In evaluating an afterschool science program for
middle-school students, Dr. Rivera conducted short
interviews with students and with their parents, on various
occasions. These exchanges produced quotes pertaining to
favorite and least favorite parts of the program, parents’
views on program impact on students, etc. In another
program evaluation, short quotations from participants
supplemented survey results.
Dr. G. Johnson, Dr. M. Rivera
5
Qualitative Data Analysis: Coding Process

When applicable (focus groups, content analysis of documents),
code the data
 Identify common word, issues, themes and go through the
material and label them according to that coding scheme




You may go through the material several times
Sometimes a major theme emerges at the end and you will need
to go back through to see if it was present in the earlier data
Software is available that may help. Major software are
ATLAS.ti and NUD*IST, others like NVivo are specialized for
various uses. There is growing literature on the use of computerassisted qualitative data analysis software (CAQDAS).
Researchers might also use spreadsheets, index cards, color
coding of documents, or other devices or combinations of these.
Dr. G. Johnson, Dr. M. Rivera
6
Ensuring Quality in Qualitative Analysis

Work to ensure inter-rater reliability
Like content analysis, the researchers should review the same
material and apply the coding scheme
 Then review and determine if there are difference
 Discuss, revise the coding scheme if necessary and retrain the
coders
 Repeat this test until there is agreement in the coding

Compare results, work out differences, then code all the material
 Inter-rater reliability was a major concern in the assessment of
teacher dossiers for advancement in tier, in Dr. Rivera’s
evaluation of Three-tier Teacher Licensure in New Mexico in
2003-2004. Several trial (but real) assessments of dossiers by 3person rater teams, of teacher volunteering to be thus evaluated,
were tried during 2004-05, a transitional year for the new system.

Dr. G. Johnson, Dr. Mario Rivera
7
Qualitative Data Analysis: General Process

Interpreting the data




Making sense of the data: Review salient themes
Are there any seeming relationships between the themes or
characteristics?
What are the major points that emerge from the analysis?
What about minor points?




Sometimes an uncommon theme stands out, or is noted by one or two
investigators
The researchers need to consider that perspective; always indicate
when it is a minority view and be careful that it does not trump
majority views
When evaluating programs, principal investigators and program
managers are often engaged in conjoint analysis by or with the
evaluator. Eliciting a variety of views is important.
These are stakeholders, and stakeholders, when really engaged, make
us smarter. Cf. Chen on “stakeholder validity” vs. scientific validity.*
*Chen, H. (2005). Practical program evaluation: Assessing and improving planning,
implementation, and effectiveness. Thousand Oaks: Sage Publications.
One stakeholder engagement technique: the
Affinity Diagram

A group process for analyzing qualitative data:






Research team reviews the material
In silence, write down each idea, key work theme on a sticky
note
In silence, post on wall
In silence, sort into similar categories
Once the notes are organized, discuss; look for consensus
themes
Interpret the data with stakeholders

Reality check: do others involved in program or project agree?
 Share preliminary drafts with stakeholders or a small group of
the participants to explore the issues you have discovered
 Share final draft with experts and cold readers
Affinity Diagram




Identify common themes
The Affinity Diagram tool is a quick way to set up a coding
framework
It is also a great way to identify the major themes and topics in
the final article or report
 It has the advantage of capturing all ideas in a room
 Everyone’s ideas have equal value when posted
Focus on major themes:
 “Some said X, while others said Y . . .”
 Avoid generalizations: if not everyone was asked to comment
about a particular practice, you do not know if the five who
commented represented the views of others or were just the
only ones that thought this practice was important.
Leading Discussion and Writing Up Results—
Could function as a facilitated focus group


Provide a range of perspectives.
Highlight interesting perspectives even if only said by one
or two people.



A few offered unique views: “….”
These might be important because “…..”
Do not try to report numbers or percents unless everyone
was counted in exactly the same way


For example, 10 people out of the 20 who you interviewed
commented that they liked the training program.
But unless you asked all 20 people, you do not know what the
other 10 people thought.
Dr. G. Johnson, Dr. Mario Rivera
11
Limitations of Qualitative Data Analysis
It can be like a Rorschach test –an inkblot that some see as a
butterfly and others see as maple leaf
 Reasonable people can read the same material and have very
different interpretations. It is important to explore those
differences to gain a deeper understanding rather than to win
the argument
 It is much more difficult to generalize from qualitative analysis
than otherwise. Questions of validity, reliability, replicability,
etc., inevitably arise with qualitative data. These may be
overweighed by the richness of insight gleaned from the data
(“thick description”), but these challenges need to be taken into
account and addressed with some directness in research reports.
 Best when combined with quantitative analysis in mixedmethods research, which allows for triangulation of data and
methods alike.

Creative Commons



This PowerPoint is meant to be used and shared
with attribution
Please provide feedback
If you make changes, please share freely and send
me a copy of changes:


Johnsong62@gmail.com
Visit www.creativecommons.org for more
information
Dr. G. Johnson,
www.ResearchDemystified.org
13