Power and Pitfalls of Experiments in
Development Economics:
Some Non‐random Reflections
Christopher B. Barrett
Michael R. Carter
Presented at the annual meetings of the
Agricultural and Applied Economics Association (AAEA)
July 2010
Denver, Colorado
Overview
Experiments now wildly popular in economics overall.
So what role do they play in development economics?
Subject of considerable dispute currently (Deaton and
Heckman vs. Banerjee, Duflo and Imbens)
We study human beings as agents whose choices,
conditioned by the external environment, result in
behaviors that matter not just to their own well-being
but also, due to externalities and general equilibrium
effects, to the aggregate experience of their
communities.
Overview
Ultimately development economists
interested in welfare and behavioral
outcomes:
y=f(s,p,b,e,α,ε) and b= g(s,p,y,e,θ,φ)
s = structural variables
p = policy variables
e = elicitable behavioral characteristics
α/θ = unobservable characteristics
Estimation challenges: (i) simultaneity of b and y,
(ii) unknown true functional form
(iii) the common non-orthogonality of (α+ε)
and the b,p,s and e variables of primary interest.
Two different threads of experimental
work to address these challenges:
1) Behavioral field experiments to elicit e or to
identify ∂b/∂p cleanly.
2) Randomized controlled trials (or social
experiments) aim to identify ∂y/∂p cleanly.
2. The Power of Experiments:
Behavioral field experiments …
1) Substantially reduce unobserved heterogeneity and
endogeneity concerns. Ex: key risk, time and trust
preference parameters. Carter and Castillo (2005)
study of Hurricane Mitch in Honduras.
2) Can replicate realistic choice settings to test key
hypotheses. Ex: BDM auctions to identify
effectiveness of free distribution of insecticide treated
bednets in Uganda (Hoffmann et al. 2009).
… remain underemployed in development economics.
2. The Power of Experiments:
RCTs… in contrast to BE, these are now dominant in
empirical dev’t economics
1) May resolve econometric problems associated with
program placement and selection effects as well as
the endogeneity of key p or s variables.
2) Use random assignment to evaluate pilots and do
more rigorous program evaluations. Huge donor
demand for more evidence of near-term impacts.
2. The Pitfalls of Experiments:
A) Internal Validity
The supposed trump claim of experiments, esp. RCTs,
often not as strong as claimed.
1) Randomization bias
2) Faux exogeneity where true treatment effects are
subjective and unmeasurable
3) Ideal designs commonly compromised in field
implementation, esp. by non-research partners
2. The Pitfalls of Experiments:
B) External Validity
Unobservable and observable features inevitably vary
at community level and cannot be controlled for in
experimental design because context matters.
1) Non-random implementation partner
2) Essential heterogeneity – experimental results are
unknown weighted average of heterogeneous
responses … may be no external population that
matches the sample mean from the RCT!
2. The Pitfalls of Experiments:
C) Asking The Right Questions?
Only a few relevant topics are amenable to randomization. Omits
macro, political, GE, negative shocks questions.
1)
Limited ability to study phenomena with lagged effects (e.g.,
early childhood interventions)
2) RCTs only identify the mean treatment effect. Often want other
characteristics of dist’n of effects: conditional effects, proportion
+/-, etc.
3) Crucial distinction between efficacy (the study of a treatment’s
capacity to have an effect under controlled conditions) and
effectiveness (real world impact). Overcorrection for
endogeneity can render findings consistently and unbiasedly
irrelevant.
Serious risks of distorting research agendas … economics
reduced to evaluation … often of points obvious to laypeople.
2. The Pitfalls of Experiments:
D) What Opportunity Cost?
1) Behavioral experiments aim to identify and explain
variation in behaviors. Good.
2) RCTs compare against the “no intervention” counterfactual.
Infeasible to do multi-factorial randomized block design of
high order dimensionality. Leads to distorted
recommendations (example: deworming).
2. The Pitfalls of Experiments:
E) Ethical Concerns
Class 1: Predictably violate basic “do no harm” obligation. IRBs
not sufficient. Example: Bertrand et al. (2007 QJE)
Class 2: Ignore responsibility to secure informed consent.
RCTs commonly blind subjects to intervention to avoid
endogenous behavioral responses.
Class 3: Suspend targeting principle, compromising the
expected effectiveness of resources expended in addressing
social ills. Explicitly ignore local information.
3. Access to Capital as Central Issue in
Development Economics
1.
Evolution of capital access theory and policy
•
Monopolistic perspective  public banking & interest rate regulation
•
Laissez faire in theory & practice
•
Imperfect information & the search for collateral substitutes
2.
Evaluation of credit markets & interventions is difficult
•
Double selection in credit markets heightens concerns over separating
the impact of capital access from the impact of the characteristics of
those with credit
•
Fundamental Identification problem of (potentially) disequilibrium
market: Observed transactions (loan or no loan) do not allow complete
sorting of observations into correct behavioral regime:
  u p   c x  [e   ] if D  S

i
i
i
i
i
i
bi  
u
u

r


zi  [ei   i ] if Di  S

i
3. Access to Capital as Central Issue in
Development Economics
1.
Key empirical questions regarding capital access:
•
•
•
•
2.
3.
Does the financial market work such that we find households in the non-price
rationed regimes or do markets work in an efficient, price-rationed manner for all?
If there is non-price rationing, is it systematically biased against any particular set of
households (e.g., low wealth households) such that the operation of the competitive
economy tends to reinforce initial levels of poverty and inequality?
How costly is non-price rationing and how much would household input use and
income increase if liquidity constraints could be relaxed and non-price rationing
eliminated?
Are there contractual or institutional innovations that can change the rules of access
to capital, lessen non-price rationing and decrease its cost?
Econometric analysis of observational data has long struggled to
answer these questions:
•
Control for latent characteristics (panel and Heckmanesque methods)
•
Getting the regime sorting (regression heterogeneity) right by getting beyond
naïve approaches
RCT and behavioral experiments seem potentially valuable—
indeed success claimed for RCTs in realm of credit.
RCTs & Existence of Credit Rationing
1.
Theory suggests that under asymmetric information, interest rate
increases can cause lenders to decline because these price increases
induce adverse selection & moral hazard. Would therefore expect
lenders to self-impose interest rate ceilings, resulting in excess
demand at that rate & market clearing via quantity rationing.
2.
If correct, this would imply that higher interest rates should result in
more default and lower profit via either adverse selection or moral
hazard.
3.
In an innovative experiment, Karlan & Zinman (Econometrica, 2010)
worked with a South African paycheck lender to randomize the
interest rate to see if higher rates had this impact
•
Existing clients invited to borrow at an announced rate
•
Some were then offered a lower rate when they applied
•
Find very little evidence that higher rates influence default
RCTs & Existence of Credit Rationing
1.
At best this strategy would only take us part way towards
understanding the existence and incidence of non-price rationing as
it only operated on an existing base of borrowers and would say
nothing about those that already rationed out
2.
While no study can do it all, there is another level at which we may
wonder about the meaning of such a price variation experiment:
–
In a companion paper, Karlan & Zinman (AER 2008) use this data to the
price elasticity of demand for credit
–
Find a kink in the relationship around the ‘normal’ market price: demand
responds to prices above that price, but does not respond to prices below
that price
–
One explanation could be that the some of the treated did not find the
announced lower price credible (there is no free lunch!)
–
From this perspective, we have to ask if we can really randomize things
like prices that have a social meaning and depend on the subject’s
perceptions. Was this a case of faux randomization where the true
treatment received was endogenously determined by the subject’s
education and sophistication?
Randomized Liquidity Injections 1: Santa
Claus Treatment Effects
1.
Normal credit selection processes inexorably make access to capital
correlated with hard to observe borrower characteristics, leading to
identification problems
2.
De Mel, MacKenzie & Woodruff (QJE, 2009) suggest solving this
problem by simply randomly distributing liquidity injections to small
scale entrepreneurs in Sri Lanka.
3.
While solves one of the identification problems noted above, it does not
deal with the regime switching which has been a central preoccupation of
the empirical literature for 20 years.
4.
Similar to some of the older naïve econometric literature, this approach
only identifies the impact of capital under the assumption that all
businesses were in an excess demand regime
5.
If not true, then estimated treatment effects are an unknown, dataweighted average of multiple regression regimes
6.
From a policy perspective, does not tell us what to expect if credit access
were expanded through normal banking, as opposed to ‘Santa Claus’
mechanisms
Randomized Liquidity Injections: Take 2
1.
One seeming way to circumvent this problem of the de Mel et al.
study would be to give liquidity injections only to those who reveal
excess demand by applying for loans
2.
Working with the same paycheck lender mentioned above, Karlan
and Zinman (Rev of Fin Studies, 2009) created a design to de-ration
some randomly selected applicants whose credit scores deemed them
credit unworthy
3.
However, because this study was creating real debt (unlike Santa
Claus liquidity gifts), several problems ensued:
•
Loan officers failed to comply with random assignment almost 50% of the time
•
Experiment exposed borrowers to real harm if they could not repay the debt—a nontrivial concern given that the lender’s scoring model in fact predicted repayment
problems for these de-rationed agents
•
Under standard human subjects protection protocols, experiments like this would require
full disclosure (no deception) and procedures to compensate for any harm caused by the
treatment
•
However, implementing such standard protections would likely invalidate the results of
the study (telling borrowers that their experimentally induced debt would be repaid in
case of default would clearly alter repayment incentives and behavior)
•
Again see how economic experimentation is very different than, say, drug trials
Purging the Error Term with
Behavioral Experiments
1.
While the RCTs just discussed try to break the problematic
correlation between error term and capital access by randomizing the
latter, an alternative approach is use behavioral experiments to make
unobservables like risk aversion, business acumen and time
preferences observable and thereby purge the error term of its most
problematic part
2.
Binswanger’s early and influential field experiments on risk aversion
(EJ, 1980; AJAE, 1981) suggest that this can be done (though his own
results were problematic)
3.
A few studies are moving in this direction, but still much to be done
on credible measurement of these kinds of characteristics
Changing the Structural Determinants of
Capital Access
1.
Theory offers a number of insights into the structural conditions
(risk, asymmetric information, etc.) that create non-price rationing
2.
An ambitious approach seen in some recent studies is to innovate and
implement new contracts and institutions designed to change these
fundamentals and induce new rules of credit access:
•
McIntosh, de Janvry & Sadoulet (EJ 2010) on credit bureaus in Guatemala
•
Gine and Yang (JDE?)on biometrics in credit markets in Malawi
•
I4 Index Insurance Innovation Initiative projects in Peru (Boucher, Trivelli &
Carter), Ethiopia (McIntosh, Sarris & Ahmad) and Mali (Bellemer, Guirkinger &
Carter)
3.
All of these rely on real institutions (hard!) and spatially randomized
rollout
4.
Stay tuned!
Conclusions
1.
2.
Challenge for contemporary development economics is to
keep its balance of rigor and relevance:
•
Experiments rightly play a powerful role in modern development
economics in complementing theory and observational data to
understand underlying structure and enable descriptive,
predictive and prescriptive analysis.
•
Behavioral experiments, in particular, can play a still-greater role
in teasing out credible estimates of otherwise-unobservable
parameters that matter to estimating behavioral and welfare
response.
Some rollback of the RCT obsession would be welcome.
There is no ‘gold standard’ of perfect identification, and
we need to beware of the blind pursuit of exogenous
variation lest it crucify development economics on a cross
of golden irrelevance.
Thank you for your comments