How to Design a Mixed
Methods Study
by John W. Creswell, Ph.D. and Vicki L. Plano Clark, M.S.
University of Nebraska-Lincoln
Andrews University, July, 2004
2
How would you combine two
types of data?
Qualitative
Text Data
This is a sample of a
text file of words that
might be collected on
transcripts through
interviews, fieldnotes
from observations, or
from optically-scanned
documents.
Quantitative
Numeric Data
2342543112232132
23322543
3122432432132433
32334441
2222111432143213
22111555
2331432432132433
32135432
3
Objectives of the workshop:




Let’s design a mixed methods study
Let’s study how people learn mixed
methods research in this room?
(or you can work on your own project
and follow along at each step)
Let’s start with a title. Write a title.
What data will we collect?
4
What are types of quantitative and
qualitative data?

Quantitative data
 Close-ended scales
 Attitudinal/behavioral
scales
 Behavioral checklists
 Census, attendance
records

Qualitative data
 Open-ended
responses
 Semi-structured
interviews
 Semi-structured
observations
 Records/documents
 Videotapes
5

Let’s identify our quantitative and
qualitative data collection
6
Now let’s consider some reasons for
why we are collecting (and mixing)
both forms of data





Together quantitative and qualitative data provide both precise
measurement and generalizability of quantitative research and
the in-depth, complex picture of qualitative research
To validate quantitative results with qualitative data
We do not have an adequate instrument. Thus, we need to
explore views and develop an instrument
Our quantitative data provide a general explanation and we
need to follow-up with participants and have them explain the
quantitative results
In our experiment, outcomes to be measured are not enough;
they need to be complemented by understanding the process of
participants
7

Let’s identify our reason for mixing
8
So…


There are good reasons for gathering
both forms of data
But…there are certain requirements for
this to work best
9
Requirement #1: Now let’s consider
whether we have the skills, time, and
resources?


We need minimum skills in both
qualitative and quantitative data
collection. What do we need?
We need time and resources for
extensive data collection and analysis.
How much time and resources do we
need?
10

Write down the skills, time, and
resources we will need
11
Requirement #2: The
audience(s)




Does our audience appreciate both
numbers and stories?
Are they familiar with this design?
Do they need to be educated?
Are examples of published studies
available in our content area?
12

Let’s identify the audiences
13
But audiences may not recognize
it yet because it is so new



Increased use and acceptance of qualitative
research from 1990’s to present
The complexity of our research problems
today requires understanding trends,
differences, as well as individual stories,
setting
Individuals advocating for and writing about
mixed methods research as a distinct, new
procedure (e.g., books)
14
They may think that it is analyzing
data separately
Quantitative Data
Qualitative Data
Mixing: converging the data or connecting
the data
15
But how do we mix?
Converge data:
Qual
Results
Quan
Connect data:
Qual
Quan
Results
16
Why our audience may
recognize it

The evidence





Books
Methodological articles
Many published research studies using it
Federal agencies
Private foundations
17
18
Other writings, initiatives on
mixed methods research:






Research studies reported in journals
Methodological articles exploring issues and
procedures
Website for bringing mixed methods writers
together
Conference sessions
Handbook of Mixed Methods in Social and
Behavioral Research
Private foundation interest; federal agency
interest
19
NIH Guidelines
- Mentioned several
approaches for combining
qualitative and
quantitative research
- Considerations for deciding
what model to use (literature
available, prior studies,
realistic design, expertise)
- Need to describe each
method thoroughly
20
Quotes:

“Combining qualitative and quantitative
methods has gained broad appeal in public
health research. The key question has
become not whether it is acceptable or
legitimate to combine methods, but rather
how they will be combined to be mutually
supportive and how findings achieved
through different methods will be integrated.”
(NIH, Office of Behavioral and Social Science
Research, 1999).
21
National Academy of Sciences
Three major research questions in
quality educational research:
• What is happening? (qualitative designs)
• Is there a systematic effect?
(a quantitative experiment)
• Why or how it is happening?
(a qualitative followup)
22
But even if they recognize it, they may
not appreciate or understand how to
design a mixed methods study


“We are interested in a randomized control
trial with a non-experimental approach
embedded within it.” (a private foundation
officer)
“We accept multi-method studies, but
investigators mostly do not sort out the
complexity of these projects so that we can
understand them.” (a federal projects officer).
23
We need to define mixed methods
research for our audiences


Mixed methods research is a design for
collecting, analyzing, and mixing both
quantitative and qualitative data in a
single study or series of studies to
understand a research problem.
The purpose of this form of research is that
both qualitative and quantitative methods, in
combination, provide a better understanding
of a research problem or issue than either
method alone.
24
Now we could mix within single studies or
multiple studies
Single Study:
Quan
Qual
Results
Multiple Studies (called multimethod research):
Quan
Qual
Study 1
Study 2
Qual
Quan
Study 3
Study 4
25
So how do we design a mixed
methods study? The model
Worldviews, theoretical frameworks, problem and research
question, skills, resources
Type of mixed methods design
Procedures for:
•designing the title
•writing the introduction to a study
•writing the purpose statement and research
questions/hypotheses
•data collection
•data analysis
•writing the mixed methods report
•evaluating the mixed methods research
26
What is a worldview?




Philosophy about your preferences for how
you learn about something through research
You prefer the quantitative worldview:
you are the expert, you decide what needs to
be learned, you build in objectivity
You prefer the qualitative worldview:
participant is the expert, participant helps you
build knowledge, you bring personal bias in
You prefer both the quantitative and
qualitative worldview
27
The next steps in planning our
study

Let’s write the overall research question
for our study
28
Then let’s choose a type of mixed
methods study to conduct

What designs are possible?
29
Types of mixed methods
designs
I. Triangulation Mixed Methods Design
QUAN
Data and Results
+
QUAL
Data and Results
Interpretation
II. Nested Mixed Methods Design
QUAN
Pre-test
Data and Results
Qual Process
QUAN
Post-test
Data and Results
30
Types of mixed methods
designs
III. Explanatory Mixed Methods Design
QUAN
Data and Results
Follow-up
qual
Data and Results
IV. Exploratory Mixed Methods Design
QUAL
Data and Results
Building
quan
Data and Results
31
Triangulation Design: Characteristics





Collecting both quantitative and qualitative data
Collecting these data at the same time in the
research procedure
Analyzing the quantitative and qualitative data
separately
Comparing or combining the results of the
quantitative and qualitative analysis
Example: collect survey data (quantitative) and
collect individual interviews (qualitative) and then
compare the results
32
Triangulation Design: When is
it used?



When you want to combine the advantages
of quantitative (trends, large numbers,
generalization) with qualitative (detail, small
numbers, in-depth)
When you want to validate your quantitative
findings with qualitative data
When you want to expand your quantitative
findings with some open-ended qualitative
data (e.g., survey with closed- and openended data)
33
Nested Design: Characteristics




Collecting both quantitative and qualitative data
Collecting both types of data at the same time
Having ONE form of data play a smaller role in the study
than the other form of data
Also,



Using one form of data to answer one question; the other form
another question
Collecting one form of data at one level of analysis and another at
another level of analysis
Example: You conduct an experiment and during the
experiment you gather qualitative interview data. The
outcomes of the experiment assessed quantitatively
address different questions than the process of the
experiment explored qualitatively.
35
Nested Design: When is it
used?



When you do not have time or
resources to commit to extensive
quantitative and qualitative data
collection
When you want to study the process of
an experiment as well as the outcomes
When you want to examine different
levels in an organization
36
Nested Research Design
Experiment
Quan
Data collection
Pre-test
Intervention
Quan
Data collection
Post-test
Process – collection
and analysis of qualitative
data
37
Explanatory Sequential
Design: Characteristics




Viewing the study as a two-phase project
Collecting quantitative data first followed by
collecting qualitative data second
Typically, a greater emphasis is placed on the
quantitative data in the study
Example: You first conduct a survey and then
follow up with a few individuals who
answered positively to the questions through
interviews
38
Explanatory Sequential
Design: When do you use it?


When you want to explain the
quantitative results in more depth with
qualitative data (e.g., statistical
differences among groups, individuals
who scored at extreme levels)
When you want to identify appropriate
participants to study in more depth
qualitatively
39
Here is an example of an explanatory design:
Quantitative
Data Collection
(quan)
Quantitative
Data Analysis
(quan)
Case Selection
Qualitative
Data Analysis
(QUAL)
Quantitative Analysis
Case Selection
Qualitative Analysis
Interpretation –
based on quan
and QUAL results
+
Qualitative
Data Collection
Quantitative Data*
Number of cigarettes
CES-D6
Qualitative Data*
Semi-structured
interviews, audio
recorded and
transcribed
Graphic plot of CES
D6 scores over time
for each participant
Graphic plot of
cigarettes/day values
over time for each
participant
* Data collected 10 times
over the course of a
calendar year for 40
participants
Creswell et al. (in progress)
Selected 5 cases
maximally varying
Identified critical
months in which
smoking varied
Description of each
case
Identification of life
events occurring
during critical
months where
smoking increased or
decreased
Thematic analysis of
life events for each
case
Cross-case thematic
analysis
Interpretation
Why did changes in
smoking occur?
Exploratory Sequential Design:
Characteristics




Viewing the study as a two-phase project
Qualitative data collection precedes
quantitative data collection
Typically, greater emphasis is placed on the
qualitative data in the study
Example: You collect qualitative diary entries,
analyze the data for themes, and then
develop an instrument based on the themes
to measure attitudes on a quantitative survey
administered to a large sample.
41
Exploratory Sequential Design:
When do you use it?



To develop an instrument when one is
not available (first explore, then
develop instrument)
To develop a classification or typology
for testing
To identify the most important variables
to study quantitatively when these
variable are not known
42
Phase I Qualitative Research - Year 1
Qualitative Data Collection
Qualitative Data Analysis
Qualitative Findings
Phase II Quantitative Research - Year 2
Quantitative Instrument Development
Unstructured Interviews 50 participants
8 observations at the site
16 documents
Text Analysis: Using QSR N6
Development of codes and themes
for each site
Create approximately a 80-item
instrument plus demographics
Administer survey to 500 individuals
Quantitative Test of the Instrument
Quantitative Results
Determine factor structure of items and
conduct reliability analysis for scales
Determine how groups differ
using ANOVA test
Sequential Exploratory Mixed Methods Design
How will we analyze the quantitative and
qualitative data (within the design types)?
Types of analysis:
 Quantitative analysis



Numeric data
Descriptive trend
analysis
Hypothesis testing,
effect size, interval
estimates

Qualitative analysis





Text/image data
Coding
Themes
Description
Interrelated themes
44
Triangulation data analysis
QUAN
data collection
• Separate QUAN
and QUAL data
analysis
QUAL
data collection
QUAN
data analysis
• Two options
• Data transformation (change
QUAL to QUAN or QUAN to QUAL)
• Comparison (keep separate and
compare/contrast)
QUAL
data analysis
Results
45
Table. Example of Data Transformation of Text Units into Numeric Data
Count
Adj.Count**
Row Pct
Column Pct
Patients
N=2
Physicians
N=4
Medical Assistants
N=4
Familiarity
With the
Form
13
6.5
41.9
5.8
17
4.25
27.4
3.0
19
4.75
30.7
5.3
49
15.5
100.0
Reactions to
the Form
23
11.5
22.2
10.2
100
25.0
48.3
17.4
61
15.3
29.5
16.9
184
51.75
100.0
Use for
Managing
Depression
67
33.5
38.6
29.8
177
44.25
51.0
30.7
36
9.0
10.4
10.0
280
86.75
100.0
Changes to
the Form
115
57.5
37.5
51.1
196
49.0
32.0
34.0
187
46.8
30.5
51.7
498
153.3
100.0
Situational
Use of the
Form
7
3.5
8.9
3.1
86
21.5
54.4
14.9
58
14.5
36.7
16.1
151
39.5
100.0
225
112.5
100.0
576
144.0
100.0
361
90.3
100.0
Themes
Nested data analysis
Quantitative Experiment
Quan
Data collection
Pre-test
Intervention
Quan
Data collection
Post-test
Qualitative Process
Data
Analysis
Pre-test
scores
Themes/Codes/
Interrelated Themes
Post-test scores
or gain scores
Compare/Describe Results
47
Explanatory sequential data analysis
QUAN
data analysis
Qual
data collection
(purposeful sampling)
• Statistical results
• Outlier cases
• Extreme cases
•
•
•
•
Select
Select
Select
Select
cases
cases
cases
cases
based on s.d. variables
to represent outliers
to represent extreme cases
to make group comparisons
Qual
analysis
• codes
• themes
• cases
48
Exploratory sequential data analysis
QUAL data
analysis
Quan data analysis
instrument development
Quotes
Items on a survey
Codes
Variables on a survey
Themes
Scales on a survey
49

Let’s identify how we will analyze the
data
50
Drawing our Design






Let’s draw a picture of our design
Identify the type of design
Add in data collection
Add in data analysis
Show the flow of activities
Add in “products” for our audiences
51
Helpful tips for creating this visual:
1.
2.
3.
4.
5.
6.
7.
8.
9.
Give a title to the visual model.
Choose either horizontal or vertical layout for the model.
Draw boxes for quantitative and qualitative stages of data
collection, data analysis and interpretation of the study
results.
Use capitalized (QUAN) or small letters (quan) to designate
priority of quantitative and qualitative data collection and
analysis.
Use single-headed arrows to show the flow of procedures in
the design.
Specify procedures for each quantitative and qualitative data
collection and analysis stage.
Specify expected products or outcomes of each quantitative
and qualitative data collection and analysis procedure.
Make your model simple.
Size your model to one page.
52
Now let’s rework our purpose
statement using some scripts
53
Sample Script for a Concurrent Design
(Triangulation or Nested)
“The purpose of this concurrent mixed methods study is to
better understand a research problem by converging both
quantitative (numeric) and qualitative (text or image) data.
In this approach, ___________ (quantitative instruments)
will be used to measure the relationship between the
________ (independent variables) and __________
(dependent variables). At the same time in the study, the
__________ (central phenomenon) will be explored using
_____________ (qualitative interviews, documents,
observations, visual materials) with _________ (participants)
at ____________ (the research site).”
54
Sample Script for a Sequential Exploratory Design
“The purpose of this two-phase, exploratory mixed methods
study will be to explore participant views with the intent of
using this information to develop and test an instrument with
a sample from a population. The first phase will be a
qualitative exploration of a _______(central phenomenon) by
collecting ___________(data) from ____________
(participants) at _______ (research site). Themes from this
qualitative data will then be developed into an instrument (or
survey) so that the __________ (theory and research
questions/hypotheses) can be tested that ________ (relate,
compare) ____________ (independent variable) with
__________ (dependent variable) for _________(sample of a
55
population) at _________ (research site).”
Sample Script for a Sequential Explanatory Design
“The purpose of this two-phase, explanatory mixed methods
study will be to obtain statistical, quantitative results from a
sample and then follow-up with a few individuals to probe or
explore those results in more depth. In the first phase,
quantitative research questions or hypotheses will address the
relationship or comparison of __________ (independent) and
________ (dependent) variables with ___________
(participants) at ___________(the research site). In the
second phase, qualitative interviews or observations will be
used to problem significant _______(quantitative results) by
exploring aspects of the ________ (central phenomenon) with
_______ (a few participants) at ____________ (research
56
site).”
Criteria for evaluating our plan:




Use appropriate
terminology for title and
design
Provide a rationale for
mixing and include it
early in the study
(“when you use…”)
Create a mixed methods
purpose statement
Identify types of qual
and quan data to be
collected and qual and
quan data analysis
steps


Include a
visual/procedural
diagram of methods
with timeline
Use rigorous procedures
for the quantitative data
collection and analysis
57
Let’s share our drawings of our
mixed methods procedures
58
How to Design a Mixed
Methods Study
by John W. Creswell, Ph.D. and Vicki L. Plano Clark, M.S.
University of Nebraska-Lincoln
Andrews University, July, 2004