Natural Experiments
as Multi-Method Research
Thad Dunning
Yale University
Prepared for a short course on Multi-Method Research, American Political
Science Association meetings, Seattle, Washington, August 30, 2011
The Growth of Natural Experiments (1960-2009)
90
Number of Published Articles
80
70
60
50
40
30
20
10
0
1960-1989
1990-1999
Political Science
2000-2009
Economics
Articles published in major political science and economics journals with “natural experiment” in the title or
abstract (as tracked in the online archive JSTOR).
A “quantitative” technique?
•
Natural experiments are often regarded as quantitative:
• Standard Natural Experiments
• Regression-Discontinuity Designs
• Instrumental-Variables Designs
•
Gathering systematic data on outcomes, exposure to
treatment, and pre-treatment covariates for each case—
a.k.a. Data-Set Observations (DSOs)—is indeed critical
•
One advantage of strong natural experiments: the
quantitative analysis may be simple and transparent
• Often, a simple difference of means suffices to estimate causal effects
•
Another advantage: models of the data-generating process
are often credible (e.g., the Neyman model)
The crucial role of qualitative methods
•
However, qualitative methods are invaluable and
often prerequisite for natural experiments.
•
As Angrist and Krueger (2001) put it,
“Our view is that progress in the application of instrumental
variables methods depends mostly on the gritty work of finding or
creating plausible experiments that can be used to measure
important economic relationships—what the statistician David
Freedman (1991) has called “shoe-leather” research. Here the
challenges are not primarily technical in the sense of requiring new
theorems or estimators. Rather, progress comes from detailed
institutional knowledge and the careful investigation and
quantification of the forces at work in a particular setting. Of course,
such endeavors are not really new. They have always been at the
heart of good empirical research” [emphasis added].
What Role Do Qualitative Methods Play?
•
A “Causal-Process Observation” (CPO) is “an insight or
piece of data that provides information about context,
process, or mechanism” (Collier et al. 2010)
•
Dunning (book manuscript) describes the contribution of five
types of CPOs to natural experiments (see Mahoney 2010):
1. Treatment-Assignment CPOs
2. Independent-Variable CPOs
3. Mechanism CPOs
4. Auxiliary-Outcome CPOs
5. Model-Validation CPOs
•
The idea is to put the role of “detailed institutional knowledge
and careful investigation…of the forces at work in a
particular setting” on a more systematic foundation
• Chapter 6 of Dunning, Natural Experiments in the Social Sciences
1. Assessing as-if random
•
Relative to conventional observational studies, the claim
that treatment assignment is as good as random is the
defining feature of a natural experiment
•
Definition: A natural experiment is an observational study (that is,
a study without an experimental manipulation) in which
assignment to treatment and control groups is done at random—
or “as if” at random
•
Since there is (usually) no true randomization, this
assertion is, to some extent, not testable
•
Yet evidence and a priori reasoning, informed by
substantive knowledge, can help validate a natural
experiment
Quantitative Tools for Assessing As-If Random
•
Standard Natural Experiments
• Balance tests: Treatment and control groups are statistically
balanced on pre-treatment covariates—just as they should be in
true (randomized controlled) experiments
•
Regression-Discontinuity Designs
• Balance tests, plus histograms or “conditional density” tests:
•
units do not bunch on one side of the RD threshold (e.g., effects
of anti-discrimination laws)
Placebo tests: no apparent effect away from the RD threshold
• Instrumental-Variables Designs
• Balance tests; specification tests (note many applications include
covariates in specifications—not so good)
•
These are all based on Data-Set Observations (DSOs)
Treatment-Assignment CPOs
•
These are nuggets of information about the process by
which cases ended up in treatment or control groups
• Analyst must explain why this process plausibly leads to an as-
•
if random allocation of units to treatment and control.
This is quite different from saying that one cannot think of any
potential confounders, and thus assignment must be as good
as random (or worse, that confounders have been “controlled”)
•
Knowledge of this process does not typically come by
knowing the values of independent and dependent
variables for each case (i.e., DSOs)
•
Treatment assignment CPOs provide crucial leverage
for causal inference
Example: Impact of Land Titling
•
Squatters with land titles seem to do better than those
without them (De Soto 2000). Do titles have a causal
effect?
•
•
Maybe—but squatters who obtain titles are different from those
who don’t.
They may have, say, different family backgrounds
•
Differences in outcomes could be due to the effect of titles, due to
the confounders, or both.
•
We might try to control for confounders we can measure—
e.g., compare titled and non-titled squatters with similar
family backgrounds.
•
But within these strata, more determined or motivated
squatters may obtain titles—and determination is hard to
measure
A Natural Experiment on Property Rights
•
In 1981, squatters organized by the Catholic church
occupied an urban wasteland in the province of
Buenos Aires, dividing the land into similar parcels
•
After the return to democracy, a 1984 law expropriated
the land, with the intention of transferring title to the
squatters
•
Some of the original owners challenged the
expropriation in court, leading to long delays, while
other titles were ceded and transferred to squatters
•
The legal action created a “treatment” group –
squatters to whom titles were ceded – and a control
group – squatters whose titles were not ceded
A Natural Experiment? Quantitative Evidence
•
Galiani and Schargrodsky (2006, 2007) use DSOs to argue
that whether title was ceded is “as if” random
•
For example, they show “pre-treatment equivalence” of treated
and untreated units:
• Titled and untitled parcels are side-by-side (reminiscent of Snow)
• Pre-treatment characteristics of the parcels (such as distance from
•
•
•
polluted creeks) are similar in both groups
The compensation offered by the government (in square meter terms)
was very similar across parcels
Pre-treatment characteristics of squatters (age, sex, etc.) do not predict
whether they received title
Treatment and control groups are statistically
indistinguishable—just as they usually would be after
randomization
Qualitative evidence
•
Yet, just as important is qualitative evidence – for
example, on the process by which the squatting took
place
• Squatters and Catholic Church organizers did not appear to know
•
•
•
the land was privately owned at the time they occupied it
They did not anticipate the later expropriation of land by the state
They had no basis for predicting which particular plots of land
would be expropriated, and thus could not assign plots to
particular squatters with this in mind
These are Treatment-Assignment CPOs:
• They do not come in the form of systematic data on the values of
•
independent and dependent variables (DSOs)
They help the analysts understand the process by which units
were eventually allocated to treatment and control groups—and
to validate the claim that this assignment was as good as random
The effects of property rights
•
Galiani and Schargrodsky (2006, 2007) find significant
differences in housing investment, household structure,
and educational attainment of children (but not in
access to credit markets, contradicting De Soto)
•
Squatters who received titles also believed more in
individual efficacy (!)—e.g. “people get ahead in life
through hard work, rather than luck” (Di Tella et al.)
•
Without the natural experiment, such a finding could be due to
confounding: people who acquired land titles might have had
stronger ex-ante faith in their efficacy than those who didn’t
Snow on Cholera
•
Nineteenth-century
London: Against the
predominant theory—
miasma—John Snow
hypothesized that
cholera was a water- or
waste-born disease
The predominant theory
• Snow’s natural experiment
• In 1852, the Lambeth water company moved its intake pipe upstream on the
Thames, to a purer water source
• The Southwark & Vauxhall company left its intake pipe in place.
Death rate from cholera in London,
by source of water supply
Southwark &
Vauxhall
Lambeth
Rest of London
Rate per 10,000
houses
315
37
59
(adapted from Snow 1855, Table IX)
Snow on Cholera: DSOs
“The pipes of each Company go down all the streets…A
few houses are supplied by one Company and a few by
the other… In many cases a single house has a supply
different from that on either side. Each company
supplies both rich and poor, both large houses and
small; there is no difference either in the condition or
occupation of the persons receiving the water of either
company…
“It is obvious no experiment could have been designed
which would more thoroughly test the effect of water
supply on the progress of cholera than this.”
-- John Snow (1885: 74-75)
• The quote speaks to external validity, but also to the “pretreatment equivalence” of treatment and control groups
•
Snow on cholera: CPOs
Households typically did not self-select into sources of
water supply; absentee landlords often took the decision
regarding water supply source, thereby dividing
“more than three hundred thousand people of all ages and social
strata…into two groups without their choice, and, in most cases,
without their knowledge; one group being supplied with water
containing the sewage of London, and, amongst it, whatever might
have come from the cholera patients, the other group having water
quite free from such impurity” (Snow 1855: 75; italics added).
•
Moreover, water pipes were laid down and water-supply
source was determined
“according to the decision of the owner or occupier at that time when
the Water Companies were in active competition” (ibid),
long before the cholera outbreak
•
Such Treatment-Assignment CPOs play a critical role in
making the claim of as-if random persuasive
Exercise: Treatment-Assignment CPOs
Snow on cholera. The impurity of water in the Thames was a source
of concern to public authorities before the cholera outbreak, and the
Metropolis Water Act of 1852 made it unlawful for any water
company to supply houses with water from the tidal reaches of the
Thames after August 31, 1855. Yet, while the Lambeth’s move of its
intake pipe upstream was planned in the late 1840s and completed
in 1852—before the cholera outbreak of 1853-54—the Southwark
and Vauxhall company did not move its pipe until 1855.
In other words, the Lambeth Waterworks Company chose to move
its pipe upstream before it was legally required to do so, while
Southwark & Vauxhall left its intake pipe in place.
Questions: Could this fact pose a threat to as-if random? How might
Snow’s discussion of the process by which water was purchased
and supplied counter some potential threats to the validity of the
natural experiment? Are these Treatment-Assignment CPOs?
Validating As-If Random, In Sum
•
Some key questions to ask of any study:
• Are treatment and control groups unbalanced with
respect to variables other than the treatment?
• Do units “self-select” into groups?
• Are policy interventions applied in a way that
anticipates the behavioral response of units?
•
“No” to all questions may be necessary, if not sufficient,
for a plausible natural experiment
•
Thus, both CPOs and DSOs play a critical role
•
Similar points may be made about regressiondiscontinuity and instrumental-variables designs
2. Independent-Variable CPOs
•
These nuggets of information provide information about the
presence or values of an independent variable (a treatment);
they can contribute both to natural experiments and in
exploratory or confirmatory research undertaken in
conjunction with a natural experiment.
• They can also sometimes be useful for investigating what aspect or
component of a treatment is plausibly responsible for an estimated
causal effect.
•
Snow on cholera:
• Early in Snow’s research, he found infected water or waste was
•
•
present during cases of cholera transmission
The Broad Street pump: brewers get water from private pumps
Verifying source of water supply in his natural experiment—a case of
“shoe leather” epidemiology
3. Mechanism CPOs
•
In many natural-experimental contexts, using selected units
in treatment and control groups to “extract… ideas at close
range” (Collier 1999) can be useful for explaining effects
•
There may be rich opportunities for “experimental
ethnography” (Sherman and Strang 2004), in which
qualitative data shapes the interpretation of causal effects
•
Mechanisms may be understood as invariant processes, not
necessarily mediating variables:
• E.g., “deterrence,” “empowerment,” or “solidarity”
• Close range fieldwork may be quite important for identifying such
processes that might account for an effect (e.g. the Argentina land
titling study)
4. Auxiliary-Outcome CPOs
•
These provide “information about particular occurrences that
should occur alongside (or perhaps as a result of) the main
outcome of interest if in fact that outcome were caused in the
way stipulated by the theory under investigation” (Mahoney
2010: 129).
• They may thus be closely linked to theory testing; the metaphor of a
criminal detective searching for clues is useful here (Collier, Brady
and Seawright 2010).
•
Using a natural experiment, Dunning and Nilekani (2011) find
that caste-based quotas in India have very weak distributive
effects
• They attribute this to patterns of party competition engendered by the
rotation of quotas across electoral jurisdictions (village councils)
5. Model-Validation CPOs
•
These are nuggets of information about causal process that
can support or invalidate assumptions of causal models
• Causal inference depends on maintained assumptions—but these
can sometimes be validated, at least partially
•
In principle, experiments and natural experiments are
“design-based” rather than “model-based” methods
• Inferential problems (such as confounding) are confronted through
•
•
•
research design, not ex-post statistical adjustment
Causal leverage comes (or should come) from the design and not
from modeling (e.g., multivariate regression analysis, matching
Thus, a simple difference-of-means test can suffice
Yet, the assumptions of models—e.g., the Neyman
model—matter here as well
The Effects of Land Titles on Children’s Health
Source: Galiani and Schargrodsky (2004). Notice that this is intention-to-treat
analysis.
In the first two rows, data for children ages 0-11 are shown; in the third row, data for
teenage girls aged 14-17 are shown. The number of observations is in brackets.
Is the Model Credible?
•
The Neyman causal model—which defines the estimands
that difference-of-means tests estimate—is very general
•
However, there are assumptions that may or may not be
credible, depending on the setting
•
One unit’s potential response is assumed not to depend on the
treatment assignment of other units
•
Rubin referred to this “no interference between units” as the
Stable Unit Treatment Value (SUTVA) assumption
•
Thus the model may not be an adequate description of
happens in certain natural experiments
•
•
E.g., in Argentina, the fertility decisions of squatters in the control
group may be influenced by the fertility decisions of their
neighbors in the treatment group
Model-validation CPOs can be helpful here
•
E.g., how do squatters interact with each other?
Regression-Based Inference
•
With conventional regression analysis, the need to
validate the model may be even greater
•
Quantitative studies using conventional
observational data often rely on regression
modeling to approximate an experimental ideal
•
“The power of multiple regression analysis is that it allows
us to do in non-experimental environments what natural
scientists are able to do in a controlled laboratory setting:
keep other factors fixed” (Wooldridge 2009: 77).
•
Yet, the assumptions behind the models play a key
role—and if the assumptions aren’t valid, the results
shouldn’t be trusted
•
I won’t thoroughly discuss the limitations of
conventional regression analysis here, but this is
important background
Model-Validation CPOs: Regression
•
Question: How does affluence affect attitudes
towards economic policy?
•
Doherty, Green, and Gerber (2005, 2006) take a
survey of lottery winners and ask questions
about political attitudes, e.g., towards
redistributive taxation or social spending
•
An innovative natural experiment: lottery
winners are randomized to the level of winnings
Doherty et al. as a natural experiment
•
•
•
•
Findings: lottery winnings influence attitudes
about narrow redistributive issues (the estate
tax) but not broader issues (role of government)
A key merit: there really is a random mechanism
determining assignment to treatment status
However, our interest may lie in the effects of
overall income, not lottery winnings
Instrumental-variables regression may help—if
the assumptions of IV model are correct
Lottery Winnings as an Instrument

The impact of income on political attitudes?
Regress the latter on the former:
ATTITUDES i =



INCOME i   i
The problem: Income and  i are not independent,
so OLS estimates of  will be biased.
The solution: Use lottery winnings as an
instrument for income.
Notice that
INCOME = LOTTERY_WINNINGS + EARNED_INCOME
i
i
i
Thus, the model says:
ATTITUDES =i


(LOTTERY_WINNINGS + EARNED_INCOME )
i
i
 i
•
According to the model, a marginal increment in
either lottery winnings or earned income is associated
with the same expected marginal increment in attitudes
• This could be called the assumption of “homogenous
partial effects” (Dunning 2008)
• Yet, windfall earnings associated with the lottery may
have different effects than money earned through work
Suppose:
ATTITUDES = i LOTTERY_WINNINGS
+i
1
EARNED_INCOME
2
i
 i
• To estimate  2 consistently, we would need another
instrument for earned income—and getting rid of the
confounding of earned income is supposed to be the point
• The point is that as-if random is not enough; the model of
the data-generating process must be credible as well
• In principle, model-validation CPOs could help here
• Do lottery winners say they treat windfall income differently?
Exercise: Model-Validation CPOs
• Horiuchi and Saito (2009), in their study of the effect of turnout on fiscal
disbursements to municipalities in Japan, use election-day rainfall as an
instrumental variable for municipal turnout. They argue that in this context,
the assumption of “homogenous partial effects” (Dunning 2008)—that is,
the assumption that variation in turnout related to rainfall has the same
effect as variation unrelated to rainfall—is likely to be valid.
• Why do these authors use instrumental variables here?
• What does the assumption of “homogenous partial effects” imply in this
substantive context?
• What sorts of model-validation CPOs might validate the assumption?
• What model-validation CPOs would undermine it?
Other Model-Validation CPOs
•
In instrumental-variables analysis, model-validation CPOs
can help assess whether the instrument affects the outcome
through channels other than the treatment variable
• Does rainfall affect conflict in Africa through channels other than
•
•
economic growth (Miguel et al. 2004)?
Does draft lottery number affect later earnings through channels other
than service in Vietnam (Angrist 1990)?
Validation of the model is crucial in any study, and qualitative
methods can play a central role
• However, there are fewer examples of successful ModelValidation CPOs than the other types of CPOs
• This may be because model validation is too-infrequently discussed…
Conclusions: Natural Experiments
as Multi-Method Research
•
Quantitative analysis is critical and has the potential
advantage of simplicity and transparency
•
Reliance mainly on differences-of-means tests gives analysts more
space to discuss the substantive context, research design and results
•
Qualitative methods can bolster the credibility of quantitative
analysis
•
Qualitative research is critical in other ways
•
•
Substantive knowledge and close attention to context are crucial for
recognizing the potential existence of a natural experiment
Thus, natural experiments are pre-eminently a form of multimethod research
•
Yet, the quantitative analysis of natural experiments has received
much more methodological attention
Conclusions: Next Directions
•
•
What distinguishes a good (convincing, compelling) CPO
from a bad (misleading) one?
•
Knowledge of context is critical, and only a few experts may possess
this
•
A possible tendency towards confirmation bias—ignore disconfirming
CPOs while highlighting helpful ones?
•
Does the adversarial tradition (e.g. debates among case experts)
help?
How can qualitative methods best bolster the strength of
natural experiments along many dimensions?
Strong
Research
Design
Most
Least
Credibility of the Models
Typology of Natural Experiments
Weak Least
Research
Design
Most
Plausibility of As-If Random