Lecture 23:
Experiments
(Chapter 15.1–15.5)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
Agenda
• Review
• Negative Income Tax Experiments
• Estimating Means (Chapter 15.1)
• Randomized Experiments (Chapter 15.2, 15.5)
• Natural Experiments (Chapter 15.3)
• Differences-in-Differences (Chapter 15.4)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-2
Review
• Under the Gauss–Markov assumptions, OLS
is consistent, unbiased, and efficient.
• Under heteroskedasticity or serial correlation,
OLS is still consistent and unbiased, but
inefficient (and the OLS formula for estimated
standard errors is incorrect)
• When X is correlated with e, then OLS is
inconsistent and biased.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-3
Review (cont.)
• There are many possible reasons why
X could be correlated with e :
– Omitted Variables
– Simultaneity
– Measurement Error
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-4
Review (cont.)
• Solutions to contemporaneous
correlation must break the link between
X and e.
• Instrumental Variables breaks this link
ex post, isolating part of the variation
observed in X that is known to be
uncorrelated with e.
• Experiments break the link ex ante.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-5
Example: Negative Income Tax
• In the 1970s, the government conducted
extensive research on a guaranteed
minimum income.
• Individuals with incomes lower than
the minimum would have their incomes
supplemented by the tax system
(a negative income tax).
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-6
Example: Negative Income Tax (cont.)
• What would be the effects of an NIT?
• Would the guaranteed income lead to mass
emigration from the labor force to public
assistance, as low-wage workers lost the
incentive to work?
• Policy makers wanted a clear answer, free of
biases, and were willing to pay for a massive
research program.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-7
Example: Negative Income Tax (cont.)
• In four separate NIT experiments, over eight
thousand households were randomly
assigned to receive guaranteed incomes.
• Because the assistance rate was assigned
randomly, researchers knew it was
uncorrelated with e.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-8
Example: Negative Income Tax (cont.)
• The NIT experiment showed that a
guaranteed minimum income had moderate
effects on labor effort.
• Heads of households who faced the highest
effective tax rate (because if they earned
more money they would reduce their subsidy)
decreased hours worked by about 11%-15%
• Second earners cut their hours of work more
dramatically, by 20-30%
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-9
Example: Negative Income Tax (cont.)
• Randomized social policy experiments
have also been used to study:
– Class size (the Tennessee
STAR experiment)
– Housing subsidies (HUD’s Moving to
Opportunity study)
– Job training programs (the Department of
Labor’s Job Training and Partnership
Act assessments)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-10
Example: Negative Income Tax (cont.)
• Randomized social policies are very
expensive, and so are conducted infrequently.
• Laboratory studies are becoming increasingly
common (using undergraduate subjects in
simulated settings, almost always playing for
real but small amounts of money). Laboratory
studies test economic theory but do not
estimate the effects of social programs.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-11
Example: Negative Income Tax (cont.)
• The basic methodological principle of the NIT
research is quite simple: if we assign X
directly, then we can eliminate its relationship
to e (at least at the point of assignment).
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-12
Estimating Means (Chapter 15.1)
• In principle, experimental economists can
assign a wide range of X values and use OLS
methods to estimate coefficients.
• Small-scale laboratory experiments can easily
assign a range of X values.
• However, for larger studies, it is often
practical to work with only a few X values.
The NIT studies are an exception.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-13
Estimating Means (cont.)
• When X takes on only a few values,
the parameter of interest shifts from
estimating a slope to estimating
a mean.
• We have already seen that dummy
variables can be used to estimate
different means for different groups.
Now we will look more closely at
estimating means.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-14
Estimating Means (cont.)
• Typically we are interested in knowing the
effect of one or more treatments.
• We want to know what outcomes are caused
by the treatment/s.
• For example, what effect does the Head Start
program have on participants’ grades?
• What effect would moving to a better
neighborhood have on residents of
impoverished neighborhoods?
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-15
Estimating Means (cont.)
• We really want to know the causal effect
of the treatment T.
• We want to know what would happen on
average to a person randomly chosen
from the population if we gave him/her
the treatment, as opposed to NOT
giving him/her the treatment.
E(Yi | Ti  1) - E(Yi | Ti  0)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-16
Estimating Means (cont.)
• Unfortunately, we do not get to observe the
same individuals in each state.
• A naïve analyst might simply compare the
outcomes for those observed in the
treatment state to those not observed in the
treatment state.
• In an observational study, we cannot
DIRECTLY compare groups that receive the
treatment to groups that do not.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-17
Estimating Means (cont.)
• Unfortunately, we do not get to observe the
same individuals in each state.
• In an observational study, we cannot
DIRECTLY compare groups that receive the
treatment to groups that do not.
• Families that choose to join programs like
Head Start differ systematically from families
that do not join such programs.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-18
Estimating Means (cont.)
• Whenever selection into a treatment is
non-random, researchers must worry
about unobserved heterogeneity
among subjects.
• Some subjects have greater ability,
motivation, resources, etc., that make
them more likely to seek out and gain
access to helpful treatments (and to
avoid unhelpful ones).
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-19
Estimating Means (cont.)
• Treatments also tend to attract
individuals who derive the most benefit
from them.
• Public housing is highly unattractive.
People in public housing tend to
be those with the worst access
to alternatives.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-20
Estimating Means (cont.)
• Selection Bias: individuals are sorted into
the treatment/non-treatment group on the
basis of some underlying characteristic, such
as motivation. This underlying characteristic
has its own effect on outcomes.
• More motivated parents enroll their children
in Head Start.
• Children of motivated parents do better
academically.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-21
Estimating Means (cont.)
• A naïve comparison would not estimate
E(Y | T  1) - E(Y | T  0)
but would come closer to estimating
E (Y | T  1 & M  1) – E (Y | T  0 & M  0)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-22
Estimating Means (cont.)
• The selection bias is a special case of
omitted variables bias.
• The goal of experiments is to break the
selection bias.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-23
Estimating Means (cont.)
• “Breaking” the selection bias requires the
experimenter to intervene in the agent’s world
in some way.
• The greater the intervention, the greater
the control the economist possesses, and the
more certain the economist is to eliminate
selection biases.
• The greater the intervention, the greater the
danger that the results will not generalize to a
more authentic situation.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-24
Randomized Experiments
(Chapter 15.2, 15.5)
• The experimenter randomly divides the
subjects into two groups, a treatment group
and a control group.
• The treatment group receives the
treatment (T = 1).
• The control group does not receive the
treatment (T = 0).
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-25
Randomized Experiments (cont.)
• Individual subjects will vary in myriad
unobservable ways.
• By randomly assigning the treatment,
the experiment ensures that on average
these differences will “cancel out.”
• More motivated subjects are as likely
to be in the control group as in the
treatment group.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-26
Randomized Experiments (cont.)
• Because the treatment has been
randomly assigned, the treatment and
control groups are on average the same
(within the bounds of sampling error).
• The only systematic difference between
the two groups is the treatment.
• The effect of the treatment can be
estimated by comparing the two
groups’ means.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-27
Randomized Experiments (cont.)
• Example: the Moving to Opportunity
Program
– HUD randomly assigns a sample of
low-income participants to receive a
special housing voucher (the treatment
group) or
to receive no voucher (the control group)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-28
Randomized Experiments (cont.)
• The vouchers are only valid for housing
in “good” neighborhoods.
• Receipt of the voucher is not correlated
with any other traits of the participants.
• Actual use of the voucher will be
correlated with such traits as motivation,
income, and existing connections in a
qualifying “good” neighborhood.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-29
Randomized Experiments (cont.)
• Researchers must be very careful about
which effects have been successfully isolated
from e, and which have not.
• The Moving to Opportunity experiment DOES
provide an unbiased estimate of the effect of
giving low income families a voucher to move
to a “good” neighborhood.
• Such estimates are valuable for assessing
proposals for such a voucher program.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-30
Randomized Experiments (cont.)
• The experiment does NOT provide an
unbiased estimate of the effects of living in a
good neighborhood vs. a poor neighborhood.
• Not every subject assigned to the treatment
group will take advantage of the treatment.
• Treatment subjects will self-select into those
who use the treatment, and those who do not.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-31
Randomized Experiments (cont.)
• Suppose we define the “treatment” as
moving from a poor neighborhood to a
good neighborhood.
• If the participants in the program are
representative of the population of
interest, and if everyone who receives a
voucher uses it, then we can estimate
the average effect of the treatment.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-32
Randomized Experiments (cont.)
• Suppose we define the “treatment” as
moving from a poor neighborhood to a
good neighborhood.
• If the participants in the program are
NOT representative of the population of
interest, and if everyone who receives a
voucher uses it, then we can estimate
the average effect of the treatment on
the treated.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-33
Randomized Experiments (cont.)
• Suppose we define the “treatment” as
moving from a poor neighborhood to a
good neighborhood.
• If NOT everyone who receives a
voucher uses it, then we can estimate
the average effect of the treatment on
those we intend to treat.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-34
Randomized Experiment (cont.)
• Biases in Randomized Experiments:
– Non-Response Bias: participants do not
provide their data to the researchers
– Attrition Bias: participants drop out of
the study
– Sample Selection Bias: individuals who
agree to participate in a randomized study
differ from the population of interest
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-35
Randomized Experiments (cont.)
• General Equilibrium Effects: the
experiment shows the results of a
small-scale program.
• Implementing the program on a larger
scale might change the environment
in ways that a smaller scale study
does not.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-36
Randomized Experiments (cont.)
• Impoverished neighborhoods are not
greatly affected by the emigration of a
few of their most talented residents from
a small study.
• If HUD decides to implement the
program on a wide scale, poor
neighborhoods could be greatly affected.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-37
Randomized Experiments (cont.)
• Another example of General Equilibrium
Effects: training programs
• If a few individuals randomly receive
extra job training, their wages increase
because (i) they are more productive
and (ii) they have a competitive edge.
• If everyone receives the extra training,
no one gains a competitive edge.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-38
Natural Experiments (Chapter 15.3)
• Researchers must choose how much control
to exert over their experiment.
• Greater control decreases any possible
correlations between the treatment and e.
• For example, HUD could have (at great
expense) moved everyone in the treatment
group to a good neighborhood, instead of
relying on participants to choose how to
use vouchers.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-39
Natural Experiments (cont.)
• In a laboratory experiment, experimental
economists can exert great control over every
aspect of the subjects’ environment.
• Greater control increases the artificiality of
the experiment.
• At some point, economists wish to generalize
from the subjects and environments studied
to a real program in the population of interest.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-40
Natural Experiments (cont.)
• Internal Validity: the ability of the
economist to attribute differences
between the treatment and control
groups to the treatment itself (X and
e are uncorrelated).
• External Validity: the ability of the
economist to generalize from the
experiment to the setting and population
of interest.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-41
Natural Experiments (cont.)
• Often economists face a trade-off
between internal and external validity.
• The more they break the natural
connections between X and e, the
more danger arises that the results will
not generalize.
• Natural experiments often offer
greater external validity (and are
much cheaper!)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-42
Natural Experiments (cont.)
• A natural experiment is an observational
study of a natural setting that appears
to assign a treatment in a reasonably
random manner.
• Natural experiments have great external
validity, at least to the particular setting and
population affected.
• Natural experiments have weaker internal
validity; the experiment does not control the
randomization process directly.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-43
Natural Experiments (cont.)
• Example: medical residency clearinghouses
• The American Medical Association runs a
centralized clearinghouse to assign medical
school graduates to hospitals for residency
training programs.
• Hospitals and applicants each submit
ranked preference lists, and the AMA
assigns matches.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-44
Natural Experiments (cont.)
• The AMA hired economist Alvin Roth to
redesign their matching procedures.
• Roth knew from game theory that a
matching algorithm would work better
if it were “stable,” i.e. if no
residency/hospital pair would be better
off by abandoning their assigned
matches to match with each other.
• But is this property really important?
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-45
Natural Experiments (cont.)
• Roth observed a natural experiment in
the United Kingdom.
• In the UK, each region conducts its own
regional clearinghouse.
• Some regions adopted matching
procedures that were “stable” (i.e.
that had desirable game theoretic
properties); others did not.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-46
Natural Experiments (cont.)
• The Treatment: using a “stable” matching
procedure
• The Treatment Group: regions in the
UK that had adopted stable procedures
• The Control Group: regions in the
UK that had NOT adopted stable
procedures
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-47
Natural Experiments (cont.)
• The Outcome: regions in the Treatment
group were still using their matching
procedures several years later. Regions
in the Control group had abandoned
their clearinghouses.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-48
Natural Experiments (cont.)
• However, Roth could only HOPE that the two
groups differed only in their choice of
procedures. Perhaps the Treatment group
regions were simply better at running
clearinghouses, or had cultural differences?
• The experiment’s internal validity was suspect.
• Roth’s solution: a laboratory experiment.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-49
Natural Experiments (cont.)
• Roth designed a simple “matching
market” that could be run in a few
minutes by undergraduate subjects
on computers.
• Some undergraduates pretended
to be hospitals; others pretended to
be applicants.
• Undergraduates were paid based on
their ability to make a “good” match.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-50
Natural Experiments (cont.)
• In some randomly chosen laboratory
markets, the clearinghouse used the stable
rules adopted by the Treatment group in the
natural experiment.
• Other laboratory markets used the bad
rules adopted by the Control group in the
natural experiment.
• The laboratory subjects reached the same
outcome as the real UK regions.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-51
Natural Experiments (cont.)
• In Roth’s randomized laboratory
experiment, the Treatment and Control
groups differed ONLY in the rules used.
• However, the laboratory study used
a very abstract computer game and
undergraduate subjects. External
validity is suspect.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-52
Natural Experiments (cont.)
• The laboratory experiments, with their strong
internal validity, suggest that good game
theoretic rules are a major factor in the
success of clearinghouses.
• The natural experiments, with their strong
external validity, suggest that the rules are
important in real markets similar to the one
the AMA wished to design.
• The two studies reinforce each other.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-53
Natural Experiments (cont.)
• Another example: the Minimum Wage
– Policy makers have long argued over the
effects of minimum wage laws
– Benefits: some workers receive
higher wages
– Costs: businesses must pay higher
wages; some workers become unemployed
when employers decide not to pay the
higher wage
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-54
Hit Figure 15.1
The Nominal and Real Minimum Wage
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-55
Natural Experiments
• Card and Krueger uncovered a natural
experiment to study the effect of a
minimum wage increase on the
employment of low-skill teenagers in the
fast food industry.
• In 1992, New Jersey rose its statemandated minimum wage from $4.25 to
$5.05. Fast food workers in NJ naturally
became a treatment group.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-56
Natural Experiments (cont.)
• Card and Krueger needed a control group.
• They chose to focus on fast food workers in
neighboring Pennsylvania, which did not
increase its minimum wage law.
• NJ and PA are very similar states. Teenagers’
families decisions to live in one or the other
are very unlikely to be correlated with NJ’s
decision to raise its minimum wage in 1992.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-57
Natural Experiments (cont.)
• After the change in the minimum wage,
fast food restaurants in NJ employed on
average 30.0 workers.
• In our control state of PA, restaurants
employed 30.9 workers.
• At first glance, it appears that the
higher minimum wage state has
slightly smaller employment in fast
food restaurants.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-58
Natural Experiments (cont.)
• One problem: NJ and PA are similar, but
not quite identical.
• Before NJ’s law changed, fast food
restaurants in NJ employed on average
29.8 workers.
• PA restaurants employed 33.1 workers.
• We need to take these baseline
differences into account.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-59
Natural Experiments (cont.)
• PA is not quite identical to NJ
• Perhaps we should look for a different
control group.
• We could use NJ restaurants before the
law changed.
• These are the exact same restaurants
as in the treatment group, so they
should be identical.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-60
Natural Experiments (cont.)
• Problem: there are other differences
between early 1992 and late 1992 than
just the change in the law.
• Perhaps the macroeconomy improved.
• Perhaps restaurants increased
employment because of seasonal
variation in the consumption of fast food.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-61
Natural Experiments (cont.)
• The Solution: look at the change in
employment in both NJ and PA when
the NJ law was enacted.
• Employment per restaurant in NJ
increased from 29.8 to 30.0
• Employment per restaurant in PA fell
from 33.1 to 30.9
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-62
HIT TABLE 15.1 Diff-in-Diffs Estimate of the
Employment Effect of New Jersey’s Minimum
Wage Increase
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-63
Natural Experiments
• NJ and PA do have some differences,
but we are HOPING those differences
do not change from early 1992 to
late 1992.
• Conditions for fast food restaurants do
change from early 1992 to late 1992,
but we are HOPING those differences
are the same in both NJ and PA (except
for the NJ minimum wage increase).
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-64
Natural Experiments (cont.)
• In effect, to interpret our natural experiment,
we have used TWO control groups.
• Neither control group is perfect, but using
the two together HOPEFULLY eliminates
the imperfections.
• This procedure of using two control groups
is called Differences-in-Differences
(or “Diffs-in-Diffs”).
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-65
Differences-in-Differences (Chapter 15.4)
• Let us construct a more general DGP.
• The outcome, Yi , is the result of
three factors:
1. Receiving the treatment (e.g. a minimum
wage increase)
2. Being in the group that receives the
treatment (e.g. NJ)
3. Being in the time after the treatment has
occurred (e.g. late 1992)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-66
Differences-in-Differences (cont.)
Y  0  1 D
Treatment
 2 D
After
 3 D After ¥DTreatment  e
where DTreatment = 1 if the observation is of a
subject assigned to the treatment group,
either before or after the treatment is received
and DAfter = 1 if the observation is of a subject in
either group after the treatment has been
received by the treatment group
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-67
Differences-in-Differences (cont.)
E (Y )   0  1D
Treatment
 3 D
After
 2 D
After
•D
Treatment
• 1 captures the underlying difference between
the treatment and control groups.
• 2 captures the underlying difference
between the two time periods.
• 3 captures the effect of the treatment.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-68
Differences-in-Differences (cont.)
E (Y )   0  1DTreatment   2 D After
 3 D
After
After
•D
Treatment
Treatment Group
 0  1   2   3 -
Control Group
0  2
 0  1
0
----------------------------------------------------------------- 2  3  2  3
- Before
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-69
Differences-in-Differences (cont.)
After
- Before
Treatment Group
 0  1   2   3 -
Control Group
0  2
 0  1
0
-
------------------------------------------------------------------ 2  3  2  3
• One method for estimating 3 is to
calculate the means for each group at
each time and subtract twice.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-70
Differences-in-Differences (cont.)
• However, we want to calculate an
estimated standard error for the
treatment group. This calculation is
not transparent when subtracting
group means.
• We also might want to add other
explanators.
• Can we express Diffs-in-Diffs in a
regression framework?
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-71
Differences-in-Differences (cont.)
• We can estimate group means using
dummy variables.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-72
Checking Understanding
Yi   0  1 Di
Assigned
  2 Di
After
  3 Di
Assigned
Di
After
 4 X i  ei
Question: which coefficient estimates
the treatment effect, controlling for X ?
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-73
Checking Understanding (cont.)
Yi   0  1 Di Assigned   2 Di After   3 Di Assigned Di After
 4 X i  ei
E (Y ), Controlling for X
After
Treatment Group
 0  1   2   3

Control Group
0  2
 1   3
Before
 0  1
0

 2  3

 1
2

3
ˆ3 estimates the treatment effect.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-74
Differences-in-Differences (cont.)
• The Great Weakness of Diffs-in-Diffs
– Some other difference may arise between
the Treatment and Control groups at the
same time that the Treatment occurs.
– Example: suppose at the same time NJ
increased the minimum wage, PA
introduced a new food labeling law that
reduced the demand for fast food.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-75
Review
• We are often interested in knowing the
effect of one or more treatments.
• We want to know what outcomes are
caused by the treatment/s.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-76
Review (cont.)
• We really want to know the causal effect
of the treatment T.
• We want to know what would happen on
average to a person randomly chosen
from the population if we gave him/her
the treatment, as opposed to NOT
giving him/her the treatment.
E (Yi | Ti  1) - E (Yi | Ti  0)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-77
Review (cont.)
• Whenever selection into a treatment is
non-random, researchers must worry
about unobserved heterogeneity
among subjects.
• Some subjects have greater ability,
motivation, resources, etc., that make
them more likely to seek out and gain
access to helpful treatments (and to
avoid unhelpful ones).
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-78
Review (cont.)
• Treatments also tend to attract
individuals who derive the most benefit
from them.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-79
Review (cont.)
• Selection Bias: individuals are sorted
into the treatment/non-treatment group
on the basis of some underlying
characteristic, such as motivation. This
underlying characteristic has its own
effect on outcomes.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-80
Review (cont.)
• The experimenter randomly divides the
subjects into two groups, a treatment
group and a control group.
• The treatment group receives the
treatment (T = 1).
• The control group does not receive the
treatment (T = 0).
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-81
Review (cont.)
• Individual subjects will vary in myriad
unobservable ways.
• By randomly assigning the treatment,
the experiment ensures that on average
these differences will “cancel out.”
• More motivated subjects are as likely
to be in the control group as in the
treatment group.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-82
Review (cont.)
• Because the treatment has been
randomly assigned, the treatment and
control groups are on average the same
(within the bounds of sampling error).
• The only systematic difference between
the two groups is the treatment.
• The effect of the treatment can be
estimated by comparing the two
groups’ means.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-83
Review (cont.)
• If our subject pool is the same as the
population of interest, and the entire
Treatment group responds to the Treatment,
we can estimate the average effect of
the Treatment.
• If our subject pool differs from the population
of interest, but the entire Treatment group is
responsive, we can estimate the average
effect of the Treatment on the Treated.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-84
Review (cont.)
• If not all of our Treatment Group
responds to the Treatment, then we
estimate the average effect of the
treatment on those we intend to treat.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-85
Review (cont.)
• Biases in Randomized Experiments:
– Non-Response Bias: participants do not
provide their data to the researchers
– Attrition Bias: participants drop out of
the study
– Sample Selection Bias: individuals who
agree to participate in a randomized study
differ from the population of interest
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-86
Review (cont.)
• General Equilibrium Effects: the
experiment shows the results of a
small-scale program.
• Implementing the program on a larger
scale might change the environment
in ways that a smaller scale study
does not.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-87
Review (cont.)
• Internal Validity: the ability of the
economist to attribute differences
between the treatment and control
groups to the treatment itself (X and e
are uncorrelated).
• External Validity: the ability of the
economist to generalize from the
experiment to the setting and population
of interest.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-88
Review (cont.)
• Often economists face a trade-off
between internal and external validity.
• The more they break the natural
connections between X and e, the
more danger arises that the results will
not generalize.
• Natural experiments often offer
greater external validity.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-89
Review (cont.)
• A natural experiment is an observational
study of a natural setting that appears to
assign a treatment in a reasonably
random manner.
• Natural experiments have great external
validity, at least to the particular setting and
population affected.
• Natural experiments have weaker internal
validity; the experiment does not control the
randomization process directly.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-90
Review (cont.)
• Suppose the outcome, Yi , is the result
of three factors:
1. Receiving the treatment (e.g. a
minimum wage increase)
2. Being in the group that receives the
treatment (e.g. NJ)
3. Being in the time after the treatment has
occurred (e.g. late 1992)
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-91
Review (cont.)
Y   0  1DTreatment   2 D After
  3 D After • DTreatment  e
where DTreatment = 1 if the observation is of a
subject assigned to the treatment group,
either before or after the treatment is received
and DAfter = 1 if the observation is of a subject in
either group after the treatment has been
received by the treatment group
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-92
Review (cont.)
E (Y )   0  1DTreatment   2 D After
 3 D After • DTreatment
• 1 captures the underlying difference
between the treatment and control groups.
• 2 captures the underlying difference
between the two time periods.
• 3 captures the effect of the treatment.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-93
Review: Diffs-in-Diffs
E (Y )   0  1DTreatment   2 D After
 3 D
After
After
•D
Treatment
Treatment Group
 0  1   2   3 -
Control Group
0  2
 0  1
0
----------------------------------------------------------------- 2  3  2  3
- Before
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-94
Review
• The Great Weakness of Diffs-in-Diffs
– Some other difference may arise between
the Treatment and Control groups at the
same time that the Treatment occurs.
– Example: suppose at the same time NJ
increased the minimum wage, PA
introduced a new food labeling law that
reduced the demand for fast food.
Copyright © 2006 Pearson Addison-Wesley. All rights reserved.
23-95