The brass standard?
The scope for RCTs in impact evaluation
Jan Willem Gunning
VU University Amsterdam
University of Sussex, December 5, 2014
Assessing policy impact in development economics
• growing concern about endogeneity (growth regressions)
• scepticism about IV-estimates
• the example of medical research (testing drugs): RCTs
But drugs testing is fundamentally different!
• extraordinary claims: development economis is RCTs
The randomista´s
• gold standard: only RCTs can identify causal effects
• don’t waste time on questions not suitable for RCTs
• RCT results have (unlimited) external validity
Abhijit Banerjee en Esther Duflo
RCTs the gold standard?
“experiments have no special ability to produce
more credible knowledge than other methods”
Angus Deaton, JEL, 2010
Ignore the big questions?
“Instead of discussing how best to fight diarrhea or
dengue, many of the most vocal experts tend to be
fixated on the “big questions”:
What is the ultimate cause of poverty?
How much faith should we place in free markets?
Is democracy good for the poor?
Does foreign aid have a role to play?
And so on.”
Banerjee en Duflo, Poor Economics, 2011
RCTs under fire ..
• not suitable for important questions
Rodrik (2008), Ravallion (2012)
• doubts about external validity
Banerjee and He (2008) vs. Banerjee and Duflo (2011)
• RCT may affect the control group
Deaton (2010)
• upscaling changes the intervention
Pritchett and Sandefur (2013), Bold et al. (2013)
• RCT misses selection on the gain
Elbers and Gunning (2013)
Who benefits? selection on the gain
assignment and treatment effect correlated
program officer bases assignment in part on private
information on treatment effect
examples:
• NGO drinking water and sanitation program: local
staff select program villages
• credit program: loan officer selects clients
RCTs in trouble ..
Can RCTs deal with this?
•
•
•
•
•
•
•
no, the problem is fundamental
RCT (randomisation over beneficiaries) does not mimic the actual
assignment process
hence the RCT gives a correct answer to an irrelevant question
randomising instead over program officers:
loss of statistical power
internal validity undermined: characteristics of program
officers correlated with controls
rehabilitation for observational data?
such data became suspect because of endogeneity concerns ..
but in this case they can do the trick ..
if data from a reprentative sample: external validity assured
Estimating the total program effect (TPE)
regression with observational data
Does it matter?
• example: impact evaluation of a health insurance
program in Vietnam
• accounting for treatment heterogeneity doubles the
health effect
• simple test: joint significance of the additional
regressors in the TPE regression
Conclusion
• in development programs assignment often neither
universal nor random, but correlated with
treatment effects
• IV-regressions and RCTs then both wrong
• the impact evaluation literature focuses on an
irrelevant parameter: Eβ
• the relevant parameter EβP can be estimated using
observational rather than experimental data
Complication: P and X correlated
• examples: educational policies induce changes in parents’
contribution (Dercon); local governments reduce activities in
response to central government programs (Deaton)
• unidirectional P caused by X: no adjustment
• unidirectional X caused by P: estimate TPE without the
relevant X terms
– this gives direct and indirect effect of P
• bidirectional case: if effect of P on X can be identified then
indirect effect can be estimated and added to the direct effect
13