Qualitative Research Design

Qualitative vs. Quantitative Research
Two types of observational study
 Nonrandom case selection


Selecting Cases on the Independent
Variable
Most similar systems
 Most different systems


Income Inequality and Civil War
Qualitative vs. Quantitative
Research

Qualitative and quantitative studies are
both types of observational studies.

Quantitative research measures differences
in number for variables, and usually studies
a large number of cases (Large “N”).

Qualitative research measures differences
in kind for variables, and usually studies a
small number of cases (Small “N”).
Qualitative vs. Quantitative
Research

Because it covers a broad range of
cases, quantitative research yields
conclusions that can be generalized (it
has the strongest external validity).

Because it looks closely at a few cases
and traces causal pathways, qualitative
research often outperforms quantitative
research in its measurement validity and
internal validity.
Qualitative vs. Quantitative
Research

When selecting cases for your quantitative
research sample, it is imperative that you
use random selection.

In qualitative research, “selection must be
done in an intentional fashion, consistent
with research objectives and strategy.”
(King, Keohane, and Verba, 1994, p.139)
Selecting Cases on the Independent
Variable

“Selecting on the independent variable”
means “selecting your cases according
to the values of the independent variable
that they take on.”
In order to do this, you have to know a little
bit about all of your potential cases.
 In order to do this right, you cannot act as if
you also know the values that the
dependent variable takes on.

Selecting Cases on the Independent
Variable

The Most Similar Systems method
selects cases that take on similar values
of confounding variables, but different
values of a key independent variable.
This “holds constant” the confounds
because they take on the same values in all
of the cases.
 This is the design recommended by King,
Keohane, and Verba.

Selecting Cases on the Independent
Variable

Most Similar Systems is:

A NEGD with a treatment and comparison
group.
NOXO
NO O
Selecting Cases on the Independent
Variable

The Most Different Systems method
selects cases that take on very different
values for multiple independent variables.

If it turns out that these cases all take on the
same value of a dependent variable, then
we can rule out the independent variables as
causes of the dependent variable.

Less useful since it can only disprove a
hypothesis.
Income Inequality and Civil War
Income
Inequality
Poverty
Colonial Past
External Threat
Civil
War
Income Inequality and Civil War
Case
Costa
Rica
Income
Poverty Colonial External
Inequality
Past
Threat
Moderate Yes
Yup
Nope
El
High
Salvador
Yes
Yup
Nope
Cuba
Yes
Yup
Nope
High
Income Inequality and Civil War
Case
Civil War?

We can hold the
confounds constant
by selecting these
similar cases from
Latin America.

It appears that
income inequality
does lead to civil
war.
Costa Rica No
El
Salvador
Yes
Cuba
Yes
Can be a powerful research design when it
is difficult or costly to study a large number
of cases. When carried out correctly, can
be internally valid. Do not need a large
number of cases for a proper test.
 Implicit foundation for “area studies.” Belief
that regions share many similarities, and
that these similarities are related to similar
outcomes (weak test) and not related to
dissimilar outcomes (stronger test).

How Similar is Similar?
In most similar designs, which covariates
should you try to match?
 Similar but irrelevant covariates do not add
anything to the test. Likewise, dissimilar
but irrelevant covariates do not detract
from the test. Both reduce degrees of
freedom.
 Covariates that are related to both the
treatment and outcome variables must be
included whether similar or not –
otherwise, omitted variables bias.

Qualitative Research Design II

Selecting on the Dependent Variable
Mills’ Method of Agreement
 Mills’ Method of Difference


Example

Dreze and Sen
Selecting on the Dependent Variable

Selecting cases according to the value of
the dependent variable that they take on
is more controversial than selecting on
the independent variable.
It allows you to look at extreme values or
divergent cases.
 “However, if this design is to lead to
meaningful … causal inferences, it is crucial
to select observations without regard to
values of the explanatory variables. K.K.V.”

Selecting on the Dependent
Variable: Method of Agreement

When you use Mills’ Method of Agreement,
you select cases that take on the same
values of the dependent variable.

This helps you to rule out possible causes,
because independent variables that vary over
these cases can’t cause the dependent var.

This method can only disprove a hypothesis,
because it can’t find a correlation.
Selecting on the Dependent
Variable: Method of Agreement

This design could help us rule out “early
industrialization” as a cause of whether
a country has a viable socialist party.
Case
France
Early
Industrialization?
No
Viable Socialist
Party?
Yes
Britain
Yes
Yes
Selecting on the Dependent
Variable: Method of Difference

When you use Mills’ Method of
Difference, you select cases that take on
different values of the dependent
variable.
After you have selected your cases, you
determine what values they take on for
some independent variables.
 Perhaps one independent variable will vary
across your cases, and explain the D.V.

Selecting on the Dependent
Variable: Method of Difference

Adding a country that has no viable
socialist party can add causal leverage
to our early investigation.
Case
Early Indust.? Feudalism? Viable Social
Party?
France
No
Yes
Yes
Britain
Yes
Yes
Yes
USA
Yes
No
No
Example: Dreze and Sen

Both countries began a new political
regime at mid-century with large
populations and little wealth.

They have diverged since then: “There is
little doubt that as far a morbidity, mortality,
and longevity are concerned, China has a
large and decisive lead over India. (p. 205)”

“What has brought about that lead is a matter
of very considerable interest. (p. 206)”
Eckstein’s Crucial Case Studies

Most likely case studies: if theory holds
anywhere, it should hold in this case.


Least likely case studies: if theory works
in this case, it should work in all cases.


Failure to support theory counts
disproportionately against theory.
Support for theory counts disproportionately in
favor of theory.
If a case is more or less likely, theory
contains unstated premises.
Potentially Valid Single Case
Designs
That are nonetheless seldom used
in political science.
Non-equivalent Dependent
Variables Design
Theory predicts treatment effect on one
outcome variable but not on another
similar outcome variable.
O1 X O 1
O2 O 2
 Other outcome serves as “comparison
group.” Strength comes from “pattern
matching” across different outcomes.

Interrupted Time Series

Single case observed over time, pre- and posttreatment (aka: regression point displacement).
OOOOOXOOOOO
 Analogous to a most similar design, with the
case as its own “comparison group.”
 The narrower the window around the treatment
effect, the more powerful the test.
 When to start and stop the series can be
problematic. Need good estimate of functional
form.
Problems Frequently
Encountered in Case Study
Research
Threats to Internal Validity
History: Many case studies are conducted
over time. Need to consider other
variables/events that may affect outcome.
 Maturation: Again, change over time within
the cases selected is likely to confound
results. Need to model functional form.
 Testing: Since case studies are typically
given by nature, not likely to be a threat;
but if pre- and post-tests are administered,
testing threats may exist.

Threats to Validity, continued
Instrumentation: Often a problem. Cases
selected because data is costly or difficult
to obtain. Typically present verbal
descriptions of the variables.
Operationalization is even more important
than in large-n designs.
 Regression: Possible, especially if case
selected is a prominent “outlier.”
 Mortality: unlikely, since cases intentionally
selected.

Selection Bias
In case studies, we intentionally select
some (small) number of cases for our
sample.
 If we select cases from limited ranges of
the outcome variable, we “truncate” that
variable and introduce selection bias.
 Truncating the outcome variable produces
(on average) an underestimate of the
treatment effect.

Example of Selection Bias

KKV give example of business school student
who wants a high paid job and selects for his
study sample only those graduates earning high
salaries. He then relates salary to number of
accounting courses.
 By excluding graduates with low salaries, he
paradoxically underestimates the effect of
additional accounting courses on income.