Qualitative Methods in
Institutional Research
Workshop
CAIR Annual Meetings
November 12, 2003
Judith Richlin-Klonsky, Ph.D.
UCLA Student Affairs Information &
Research Office
jrichlin@saonet.ucla.edu
http://www.sairo.ucal.edu
1
The goals of the workshop
o
o
o
To outline the issues that need to be
considered when undertaking a
qualitative research project.
To describe experiences with specific
qualitative research techniques.
To respond to questions about using
qualitative methods in institutional
research.
2
Our agenda






Introductions
Overview of qualitative methods in
institutional research (JR-K)
Quick & Easy Focus Groups (WW)
Blitzkrieg Ethnography (BB)
“Fishbowl” Discussions (JR-K)
Q&A/Discussion
3
What do we mean by “qualitative
research?”
o
o
o
o
o
o
o
Examples of qualitative data-collection
techniques:
Individual interviews
Group interviews
Focus groups
Ethnographic observation
Participant-observation
Archival and/or documentary research
Journaling
(see handout for descriptions)
4
What do we mean by “qualitative
research?” (2)

Not just a set of techniques.

Not just “non-quantitative” research
methods.
5
What do we mean by “qualitative
research?” (3)


Research design and methods that help us
fulfill specific goals as researchers:
By gathering data through participants’ own
words and/or (inter-)actions, we seek to
learn about the phenomenon being studied
from their point of view.
By spending time with experts (those
directly involved), we seek to learn in depth
about the phenomenon being studied.
6
What do we mean by “qualitative
research?” (4)
o
o
o
Unlike most quantitative institutional research,
qualitative research typically doesn’t:
Measure. (Ex: “What is the rate of graduation?”)
Produce a quantified product (Ex: “86% of our
students are satisfied with their learning
experience.”)
Seek to establish a causal relationship. (Ex:
“Summer attendance shortens time to degree.”)
7
“But it’s not scientific…”
“Whether qualitative techniques are
considered scientific description or
explanation, or whether they offer merely a
prelude to scientific inquiry, depends on the
philosophical stance taken with respect to
the nature of social science. Indeed…some
have argued that qualitative data provide
the only empirical foundation on which
social science can be built” (Walker, p. 3).
8
Research design: When is
qualitative research most useful?
When the research topic is:
o Sensitive.
o Complicated.
o Unmeasurable.
o Concerned with interaction and/or
process.
9
Research design: When is
qualitative research most useful? (2)
When the research objective is:
o To learn about research subjects who are
few in number.
o To brainstorm (generate as many ideas as
possible, not reduce to a single number).
o To identify important issues to be explored
more broadly through quantitative methods.
o To interpret, illuminate, or illustrate.
10
Examples of data-collection
issues
Individual & group interviews, Recruitment
focus groups
Representativeness
Ethnographic observation
Site selection & access
Participant-observation
Recruitment & training
of participants to be
effective observers
Access
Archival &/or documentary
research
Journaling
Data management
11
Data collection issues
All:
How to maintain subject privacy and
demonstrate to IRB that you are
prepared to maintain anonymity or
confidentiality?
12
Data analysis
Analysis is aimed at
• Answering the research question.
• Identifying themes, categories, or
types.
13
Data analysis (2)
o
“Analysis of qualitative material is
more explicitly interpretive, creative,
and personal than in quantitative
research, which is not to say that it
should not be equally systematic and
careful” (Walker, p. 4).
14
How do we know when we have
“valid results?”
Saturation
When not hearing new types of experiences,
observations, roles, etc.
Triangulation within qualitative methods
Data gathered at different times or places, at
different levels
Between multiple observers of the same
phenomenon.
Triangulation between methods
Quantitative and qualitative data
15
Reporting issues
“The report on a qualitative study will be
completely different than that on a
quantitative study. The structure for the
report will have emerged from that used in
the analysis: the findings will be presented
in terms of impressions gained, as
hypotheses rather than as firm conclusions.
[continued]
16
Reporting issues (2)
…There will be no tables… …and no references to
numbers or percentages; at most the researcher
will use such phrases as ‘most respondents’ or
‘around half’ or…‘a minority view was….’ The
researcher will be mainly concerned to identify and
describe the range of behavior and options rather
than to indicate…how many hold each view. In all
cases the description of beliefs, attitudes, and
motivations should be supported by evidence in the
form of verbatim quotations form the interviews and
discussions” (Morton-Williams, in Walker, p. 41).
17
Reporting issues (3)
o
o
How to encourage the audience to
give appropriate attention and weight
to the data?
Not too little: avoid being dismissed
because of “small N.”
Not too much: avoid emotional effects
of direct participant voices.
18
Selected resources: qualitative
methods
o
Sharon B. Merriam, Qualitative
Research and Case Study Applications
in Education, John Wiley & Sons, 1998.
o
Sharan B. Merriam & Associates,
Qualitative Research in Practice:
Examples for Discussion and Analysis,
Jossey-Bass, 2002.
o
Jean J. Schensul and Margaret D.
LeCompte, Editors, Ethnographer’s
Toolkit, Altamira Press, 1999.
19
Selected resources: qualitative
methods (2)
o
o
Robert Walker, Editor, Applied
Qualitative Research, Gower Publishing
Company, 1985.
Elizabeth Whitt, “’Making the Familiar
Strange’: Discovering Culture,” in
Cultural Perspectives in Student Affairs
Work, George D. Kuh, Editor, American
College Personnel Association, 1993.
20
Transition

The following slides describe use of
student “fishbowl” discussions.
21
UCLA Student “Fishbowl”
Discussions
Judith Richlin-Klonsky, Ph.D.
UCLA Student Affairs
Information
& Research Office
jrichlin@saonet.ucla.edu
http://www.sairo.ucla.edu
22
The “Fishbowl” Discussions:
1999-2003



Format
Background
Permutations
23
Five “Fishbowl” permutations
1999
2000
2001
2002
2003
First yrs
First yrs
Soph.
Junior
Seniors
Senior
Senior
Seniors
Grad
students
--
24
Recruitment and planning
(see appendixes of reports at
http://www.sairo.ucla.edu, “Recent reports”)





Campus leaders only as general participants.
Invitations to be considered for participation:
 “ Cold calls” based on lists generated from student
records, using parameters related to gender,
race/ethnicity, class standing, source direct/transfer.
 Referrals from Student Affairs directors.
 Ad in Daily Bruin
Offered incentive as “thanks” for participation
Made clear it would be videotaped
Topic areas identified in dialogue with Student Affairs
leadership.
25
Selection
o
o
o
o
Participation not automatic, not
promised.
Each candidate contacted personally
for a short conversation.
Selection based on perceived
contribution to group “mix,” ability and
interest to share experience, energy.
Frequent follow-up contact prior to
event.
26
Room set up



Varied by year, depending on
anticipated size of audience and
location: classroom; stage; in law
school classroom “well”; tiered lecture
hall; meeting room.
Goal is to arrange students so that they
can easily conduct conversation, but
that audience can see and hear them
Facilitator and assistant sit with
27
Types of subjects addressed







Academic experience
Co-curricular activities
Personal and academic development
Perceptions of campus climate
Access to/use of student services
Residential life (especially crowding)
Use of technology
28
Afterwards


Refreshments
Mingling
29
Reviewing and summarizing the
data




Review notes taken during discussions.
View duplicated copy of videotape.
Make notes from videotape,
transcribing comments that concisely
capture main points being made.
Use notes to summarize student
comments in major areas covered in
discussion.
30
Summaries

Full reports and executive summaries
posted on the SAIRO web site at
http://www.sairo.ucla.edu (“Recent
Reports”).
31
Examples

Campus Diversity
“I came from a very heterogeneous background,
mostly minorities… My thinking was when I left high
school, oh cool, I’m going to UCLA and there’s going
to be all these other people, but this time it won’t be
cliqued, meaning that all the pilipinos will be together
and all the [shrug] will be together. But when I went
to college, it was more of the same… and I was
surprised especially with all of the diversity issues
coming up and especially with the abolition of
affirmative action… I kinda find it weird that each of
their ethnic group clubs have outreach for their
ethnic group and it is cliqued even more.” (ud,
32
21:24:32)
Examples

Campus diversity
“It’s such a diverse campus, but people
[of the same background] still
congregate together.” [ld]
“When I walk around campus, the only
ones who are smiling at you are the
ones that look like you.” [ld]
33
Examples

Transition to UCLA
“I’ve been told for the last two years [in hs],
‘College is going to be such a different
experience for you.’ And then I got here and
it was more different than I ever thought.
You can’t describe it. You can’t explain it.
It’s just a transition. And it’s just a transition.
And it’s a different part of life that you have to
go through…. Just be prepared to change
your outlook on academics and everything.”
[ld]
34
Examples

Learning to persevere
A member of the senior group used the metaphor of
UCLA as “an emotional roller coaster” and the image
resonated with other participants as well. Part of the
bumpy ride they described was the discouragement
they felt after arriving as someone who had been
very “impressive” in high school and then feeling
“thrown in a pile” with other high-achieving students.
Another student agreed that at UCLA you have to
“cope with ups and downs, have to deal with
discouragement regularly.”
35
Examples

Learning to persevere
Other seniors described in various ways their
personal developments in light of the challenges
they faced in a large, complex, and competitive
setting:
UCLA does grind you into a tough person. In
applying for jobs, you have that toughness going for
you – created by adversity.
UCLA has taught me to persevere.
36
Examples

Student culture: time to degree
“Never once during this fifth year have I
regretted that decision. It was the best
thing. It helped me mature and I think I
would really encourage first year
students…to consider a fifth year…”
(21:14:57)
37
Reports


Full report and executive summary
distributed to all those invited to the
event: Student Affairs AVCs and
directors, Chancellor and his executive
team, deans.
Posted on web site.
38
Uses




Some in audience moved and/or energized
by hearing from students, especially in early
years and in 2003 “good-bye” discussion.
Concrete changes made, e.g., changes in
hours of operation for some Student Affairs
offices
Contribution to planning for various areas
(e.g., technology development, external
affairs).
Format also used in session for student loan
counselors and planned as a forum for staff
experiences.
39