Department of Economics, SUFFOLK UNIVERSITY
Jonathan Haughton
ECONOMICS 724: COST-BENEFIT ANALYSIS & IMPACT EVALUATION
ASSIGNMENT 9 and 10: Impact Evaluation for Real
These exercises are quite difficult and will take some time, which is why they may be considered as constituting
two exercises. Answers to question 1 are due back by April 7, 2015, and to the remaining questions by April 21. All
the datasets are available via http://web.cas.suffolk.edu/faculty/jhaughton/ by following the link near the
Handbook on Poverty and Inequality picture.
1.
Bangladesh: Evaluating the Impact of Agricultural Extension
Download hh98big7bs.dta. This file has data from Bangladesh that should be adequately labeled. There is a
variable called agextend that indicates whether a household was chosen to participate in a program of
agricultural extension that provides advice and support. [NB: The rest of the dataset is real; this variable is
invented.] We now want to ask a basic question: what was the impact of the agricultural extension program.
a.
First, let us look at the raw numbers.
i. Load hh98big7bs.dta, sort by the variable nh, and save.
ii. Now load consume98v7s.dta (or equivalent), sort by nh, and merge nh using
hh98big7bs.
iii. Check that the merge worked correctly by looking at the _merge variable.
b.
Now compare per capita income and consumption levels for households that did, and did not, get agricultural
extension help.
i. Hint 1. First create measures of total income per capita, and total consumption per capita.
ii. Hint 2. Sort by agextend and then use the syntax by agextend: sum hh* or equivalent.
Specifically, are households that got agricultural extension poorer? Richer? Larger? Are they more
reliant on farm income?
Assume for now that agricultural extension was provided perfectly randomly.
iii. Compute the unconditional single difference in per capita expenditure and test whether it is
statistically significantly different from zero using a t-test.
iv. An alternative approach is to regress per capita expenditure on the binary treatment variable. Try
this, and determine whether the treatment effect is statistically significant. Compare the test
statistic here with that found in b(iii).
c.
Next, let us assume that agricultural extension was provided randomly, once other variables are held constant
(i.e. “partial randomization”), and then ask what effect the program had.
i. Create dummy variables for each district (“thana”). The tab thana, gen(than) command
will do this nicely.
ii. The common impact model: Run a regression of per capita income (or consumption or farm income)
on the agextend, individual variables (such as gender, age, education, family size, etc.), and district
dummy variables. The coefficient on the agextend variable measures the impact of the program.
You will probably want to run a few regressions, one for each output variable (such as income per
capita) that is of interest.
iii. Are the effects measured in c.ii larger or smaller than in b?
iv. Explain why the effects are larger or smaller than in c. For this you might want to check the
correlations among the right hand variables.
v. Do as in c.ii, but this time do not include the district dummy variables.
vi. Do as in c.v, but this time include, on the right hand side, interactive terms with the treatment
dummy variable. This is a more general specification than the dummy variable model and is more
difficult to program. Comment on the size of the impact in this case relative to that in c.vi.
Ec 724 (Cost-Benefit Analysis and Impact Evaluation), Assignments 9 and 10
Page 1 of 3
2.
Using Panel Data for Impact Evaluation
a.
b.
c.
d.
e.
f.
Open consume98.dta, keep nh and hhexpfd , rename this latter by appending 98, sort by nh,
and save under a new name such as rconsume98.dta.
Open consume91.dta, keep the same variables, sort by nh, merge with rconsume98, check
that the merge has worked (using tab _merge), drop the _merge variable, sort by nh, and save
as rconsume9198.dta.
If you have not already done so, open hh98big7bs.dta and rename each variable (except nh) by
suffixing 98. Example:
rename vill vill98.
This file has information on income. Sort using nh and save under a new name such as
revhh98.dta.
Now open hh91.dta, sort by nh, and merge using revhh98.dta. As usual, check that the two
files have merged, by examining _merge , and then delete this variable.
Sort by nh and merge using rconsume9198.dta. Save this file, which is the file with which you
will now work.
Note that prices in 1998 were 47% higher than in 1991, so before food expenditures can be compared, they
must be adjusted for the price difference. We will do this below.
g.
h.
i.
j.
k.
3.
Construct a measure of household food expenditure per capita for 1991 and multiply it by 1.47 to get
the equivalent in 1998 prices. Call it pce91in98.
Construct a measure of household food expenditure per capita for 1998. Call it pce98.
Use an unconditional double difference estimator to determine whether agricultural extension had
an impact on food expenditure per capita.
Use a regression of the change in (real) food expenditure per capita to test whether agricultural
extension (which we assume was introduced in 1995) had an impact on per capita food expenditure.
Include appropriate control variables in the equation.
Justify your choice of variables in j. and write a 100-word commentary on the results.
Propensity Score Matching
For this exercise, I assume that you already have a satisfactory copy of rconsume9198.dta. If not, please
follow steps a-f of question 2 above (or go to the web site!).
Open Stata and read in rconsume9198.dta. If you have not already created a measure of household food
expenditure per capita for 1998, do that now and call it pce98.
Now let us run a propensity score analysis. The idea is first to create a “propensity score” that measures the
probability that a household will get agricultural extension; and then to use this score to match each “treated”
household (i.e. household that gets agricultural extension) with an untreated household that is otherwise very
similar (i.e. has a very similar propensity score). Here is how it might work:
i.
ii.
iii.
iv.
From within Stata, use the search command to find “pscore” and “attnd” and download the relevant
*.ado files. This is mainly an issue of following the instructions.
Estimate the propensity score equation. This will look something like this:
pscore agextend sexhead … [other variables, including district
dummies] … [iw=weight], pscore(fhat1) comsup
Now make the comparison, using nearest-neighbor matching, using
attnd xxx agextend [iw=weight], pscore(fhat1) comsup
where xxx refers to outcomes variables (e.g. per capita income, per capita expenditure, per capita farm
income, etc.) that are of interest. Do the exercise for at least three different outcome variables.
Interpret the results: how big is the effect of agricultural extension on outcomes? Is the balancing
property met? How large are the results compared to those found by double differencing? By single
differencing?
Ec 724 (Cost-Benefit Analysis and Impact Evaluation), Assignments 9 and 10
Page 2 of 3
v.
vi.
4.
Repeat iii but now use Gaussian matching.
Repeat iii, but use nearest neighbor matching with 5 matches.
Direct covariate matching [Optional (but Ph.D. students have no excuse for avoiding it!)]
Repeat the exercise in question 1 but use direct matching instead of propensity score matching. To get started,
from within Stata use the search command to find nnmatch, download the relevant files, read the help screen,
and do it!
5.
Instrumental variables
For this exercise, I assume that you already have a satisfactory copy of rconsume9198.dta. If not, please
follow steps a-f of question 2 above (or go to the web site).
Open Stata and read in rconsume9198.dta. If you have not already created a measure of household
expenditure per capita for 1998, do that now and call it pcex98.
(i)
First try a straightforward regression of the outcome (pcex98) on plausible explanatory variables.
As a first pass try using sex, age, family size, whether household has electricity, and whether
household has a sanitary toilet. How large is the apparent impact of agricultural extension?
(ii)
The approach in (i) is only reasonable if there is randomization in the design of the experiment
(perhaps conditional on the variables in question). In the absence of randomization, we might want
to use instrumental variables. Assume, for the sake of this exercise, that educated people are more
likely to get agricultural extension visits, but that education does not directly influence pcex98 [yes,
this is not very plausible, but let’s accept it for now]. Do an instrumental variable regression for this.
Hint: use the ivreg2 command. You will need (agextend=educhead) somewhere, and it is
a good idea to use ,first in order to generate the first-stage regression. [Optional: You might
consider using the treatreg command here.]
(iii)
Redo (ii) but account for weights and clustering. Use [aw= … ] for the weights, and for the
clustering append gmm cluster(vill) to the command.
(iv)
Are your results in (ii) and (iii) plausible? Why?
6.
Pulling it all together
We have used the Bangladesh example consistently throughout these assignments. Now it is time to pull all the
results together. Complete the following table (and keep your Stata code, either in a *.log or *.do file):
Method
Impact of agricultural extension on per capita
expenditure
Difference
p-value of difference
1.
Show the mean levels of per capita expenditure for
 households receiving agricultural extension
 households not receiving agricultural extension
2. Assume randomization: simple difference (D)
3. Assume randomization: common impact model
4. Assume randomization: more general regression model
5. Covariate matching model
6. Propensity score matching model
7. Instrumental variable model
8. Instrumental variable model accounting for clustering
9. Simple double difference (DD)
10. Double difference: regression model
11. Reflexive comparison
Which results do you prefer? Why?
Ec 724 (Cost-Benefit Analysis and Impact Evaluation), Assignments 9 and 10
Page 3 of 3