How do we Randomize?
Jenny C. Aker
Tufts University
Course Overview
• Introduction to ATAI and J-PAL
• Using randomized evaluations to test adoption
constraints
• From adoption to impact
• Alternative strategies for randomizing programs
• Power and sample size
• Managing and minimizing threats to analysis
• Common pitfalls
• Randomized evaluation: Start-to-finish
Why are we here?
• What is the impact of the intervention?
o What is the impact of NERICA on rice yields?
o What is the impact of improved bags on cowpea
losses?
• Was this (observed) impact due to the
program or something else?
o Unbiased treatment or program effect
o Attribution
Measuring Impact
“Treated” farmers
yields
“Control” farmers
yields
Farmers use seeds
Farmers don’t use
seeds
Offer farmers
improved quality
seeds
10 kg
Is this
unbiased?
Too big or
too small?
Measuring Impact
“Treated” farmers
yields
“Control” farmers
yields
Farmers in treated
villages use seeds
Farmers in control
villages don’t use
seeds
Choose villages away
from a paved road to
get seeds
10 kg
Is this
unbiased?
Too big or
too small?
Experimental
QuasiExperiment
Randomizati
on
RDD
Non—Experimental
DD
Matching
PSM
IV
High
internal
validity
Lower
internal
validity
Lower
external
validity
Higher
external
•5
validity
What is randomization?
• Randomization involves randomly assigning a
potential participant (individual, household or
village) to the treatment or control group
• It gives each potential participant a (usually
equal) chance of being assigned to each group
• The objective is to ensure that the only
systematic difference between the program
participants (treatment) and non-participants
(control) is the presence of the program
Basic setup of a randomized evaluation
Not in
evaluation
Target
Population
Evaluation
Sample
Random
number
generator,
calebasse,
Excel, STATA
Random
Assignment
Treatment
group
Participants
No-Shows
Control group
NonParticipants
Cross-overs
7
How can randomization be useful
to measure a program effect?
• It gives each potential participant a (usually
equal) chance of being assigned to each group
• On average, and if the sample size is large
enough, observable and unobservable
characteristics between the program
participants (treatment) and non-participants
(control) is the same
• The only difference is the presence of the
program
• (But we need to check that it worked)
Possible Randomization Designs
•
•
•
•
•
Simple lottery
Randomization in the “bubble”
Randomized phase-in
Rotation
Encouragement design
These are not mutually exclusive.
Lecture Overview
•
•
•
•
Unit and method of randomization
Real-world constraints
Revisiting unit and method
Variations on simple treatment-control
Lecture Overview
•
•
•
•
Unit and method of randomization
Real-world constraints
Revisiting unit and method
Variations on simple treatment-control
Unit of Randomization: Options
1. Randomizing at the individual level
2. Randomizing at the group level
“Cluster Randomized Trial”
• At which level should we randomize?
•12
Unit of Randomization: Individual?
Unit of Randomization: Individual?
Unit of Randomization: Clusters?
“Groups of individuals”: Cluster Randomized Trial
Unit of Randomization:
Farmers’ Group?
Unit of Randomization:
Farmers’ Group?
Unit of Randomization: Village?
Unit of Randomization: Village?
How do we choose the level?
• What unit does the program target for
treatment?
• What is the unit of analysis?
How do we choose the level?
• Nature of the Treatment
o How is the intervention administered?
o What is the catchment area of each
“unit of intervention”
o How wide is the potential impact?
• Aggregation level of available data
• Power requirements
• Generally, it is best to randomize at the level at
which the treatment is administered.
Example: Tree-Planting in Malawi
(Jack 2011)
• Intervention: Tree-planting contract for farmers
(same subsidy, different payment schemes)
• Treatment Level: Farmers (or farm households)
• Catchment level: Agricultural extension agent
• Randomization level: Individual (500 farmers
across 23 villages)
• Oversubscription, so additional public lottery
Example: Fertilizer Tree Adoption
in Zambia
• Intervention: Promotion of fertilizer “trees”
• Treatment level: Farmers’ group and
individual farmers
• Randomization level:
o Farmer group (each group randomly offered
different price for tree)
o Individual: Reward cards for maintaining trees
randomized across farmers within the group
Example: SMS-Based Price
Information in India
• Intervention: SMS-based price information in
India
• Treatment level: Farmers within the village
• Randomization level: Village
o Ten farmers selected in each village
o Farmers received subscription to a SMS-based
price information service (Reuters Market Light)
o Intensity of treatment varied across villages (ie,
number of farmers)
Lecture Overview
•
•
•
•
Unit and method of randomization
Real-world constraints
Revisiting unit and method
Variations on simple treatment-control
Real-World Constraints
•
•
•
•
•
•
Fairness and ethical issues
Political Concerns
Resources
Crossovers/spillovers
Logistics
Sample size
Fairness
• Randomizing at the individual level within a
farmers’ association
o Non-treated farmers might be unhappy
• Randomizing at the household-level within the
village
o Non-recipient households or the village chief might
be unhappy
• Randomizing at the village or farmers’
association level
o Ministry of Agriculture might be unhappy
Political Concerns
•
•
•
•
Lotteries are simple and common
Randomly chosen from applicant pool
Participants know the “winners” and “losers”
Simple lottery is useful when there is no a
priori reason to discriminate
• Can be perceived as fair
• Transparent
•How to Randomize, Part
I - 28
Resources
• Many programs have limited resources
o Vouchers, Subsidies, Training
o More eligible recipients than resources
• How will program recipients be chosen?
o Clear-cut criteria
o Arbitrary criteria
o Random process
o Some combination of the above
•How to Randomize, Part
I - 29
Spillovers/Crossovers
• Contamination of the control group can be
due to:
• Spillovers – positive or negative
• Crossovers – movement to treatment (or control)
group
Logistics
• Is it possible or feasible for staff to implement
different programs in the same catchment area?
• Agricultural extension agent provides training in
improved NRM techniques
– Training is one of many responsibilities of the agent
– The agent might serve farmers from both treatment
and control villages within his/her catchment areas
– It might be difficult to train them to follow different
procedures for different groups, and to keep track of
what to give whom
Sample Size
• The program is only large enough to serve a
handful of communities
• Might not be able to survey (or implement the
program in) enough communities to detect a
(statistical) effect
Lecture Overview
•
•
•
•
Unit and method of randomization
Real-world constraints
Revisiting unit and method
Variations on simple treatment-control
Possible Randomization Designs
•
•
•
•
•
Simple lottery
Randomization in the “bubble”
Randomized phase-in
Rotation
Encouragement design
These are not mutually exclusive.
Randomization in “the bubble”
• A partner may not be willing to randomize among
eligible people.
• However, a partner might be willing to randomize
in “the bubble.”
• People “in the bubble” are those who are
borderline in terms of eligibility
– Just above the threshold  not eligible, but almost
• What treatment effect do we measure? What
does it mean for external validity?
Randomization in “the bubble”
Within the
bubble,
compare
treatment
to control
Treatment
Non-participants
> .25 ha
Participants
<=.25 ha
Control
Randomization in the Bubble
Must receive the program
Randomized assignment to the
program
Ineligible
Randomization in “the bubble”
• Program still has discretion to treat
“necessary” groups
• Example: Agricultural grant program in Niger
(PRODEX)
• Example: Expansion of consumer credit in
South Africa
Randomized Phase-In
• Takes advantage of the program expansion (ie,
the NGO cannot implement in all villages the
first year)
• Everyone gets program eventually
• If everyone is eligible for the program, what
determines which villages, schools, branches,
etc. will be covered in which year?
Randomized Phase-In
3
Round 1
1
2
Treatment: 1/3
Control: 2/3
3
3
1
3
3
2
3
2
Treatment: 2/3
Control: 1/3
2
3
1
2
1
3
3
3
2
3
1
2
1
2
2
2
1
3
3
Round 3
Treatment: 3/3
Control: 0
2
1
Round 2
Randomized
evaluation ends
2
2
3
2
1
3
1
3
1
2
1
1
2
3
1
Randomized Phase-In
Advantages
• Everyone gets something eventually
• Provides incentives to maintain contact
Concerns
• Can complicate estimating long-run effects
• Be careful with phase-in windows
• Do expectations of change actions today?
Rotation
• Groups get treatment in turns
– Group A gets treatment in the first period
– Group B gets treatment in the second period
•How to Randomize, Part
I - 42
Rotation design
Round 1
Treatment: 1/2
Control: 1/2
Round 2
Treatment
from Round 1
 Control
——————————————————————————
Control from
Round 1 
Treatment
Rotation
• Advantages:
– Might be perceived as fairer, therefore easier to get
accepted
• Disadvantages:
– If those in Group B anticipate treatment, they might
change their behavior
– Cannot measure long-term impact because no pure
control group
•How to Randomize, Part
I - 44
Randomized Encouragement
• Sometimes it’s not possible to randomize
program access (vaccines, savings program,
etc)
• But many programs have less than 100% takeup
• Randomize encouragement to receive
treatment
Encouragement design
Encourage
Do not encourage
participated
did not participate
compare
encouraged to not
encouraged
These must be correlated
do not compare
participants to nonparticipants
Complying
Not complying
adjust for non-compliance in
analysis phase
What is “encouragement”?
• Something that makes some individuals more
likely to use program than others
– Not in itself a treatment
– E.g., vouchers, training, visit from agent, etc
• For whom are we estimating the treatment
effect?
– Think about who responds to encouragement
(compliers)
Summary of Possible Designs
•
•
•
•
•
Simple lottery
Randomization in the “bubble”
Randomized phase-in
Rotation
Encouragement design
These are not mutually exclusive.
Methods of randomization - recap
Design
Most useful
when…
•Program
oversubscribed
Basic Lottery
Advantages
•Familiar
•Easy to understand
•Easy to implement
•Can be implemented
in public
Disadvantages
•Control group may not
cooperate
•Differential attrition
•How to Randomize, Part
I - 49
Methods of randomization - recap
Design
Phase-In
Most useful when…
•Expanding over
time
•Everyone must
receive treatment
eventually
Advantages
•Easy to understand
•Constraint is easy
to explain
•Control group
complies because
they expect to
benefit later
Disadvantages
•Anticipation of
treatment may impact
short-run behavior
•Difficult to measure
long-term impact
•How to Randomize, Part
I - 50
Methods of randomization - recap
Design
Rotation
Most useful when…
Advantages
•Everyone must
•More data points
receive something at than phase-in
some point
•Not enough
resources per given
time period for all
Disadvantages
•Difficult to measure
long-term impact
•How to Randomize, Part
I - 51
Methods of randomization - recap
Design
Most useful
when…
•Program has to
be open to all
comers
•When take-up is
Encouragement low, but can be
easily improved
with an incentive
Advantages
•Can randomize at
individual level
even when the
program is not
administered at
that level
Disadvantages
•Measures impact of
those who respond to
the incentive
•Need large enough
inducement to improve
take-up
•Encouragement itself
may have direct effect
•How to Randomize, Part
I - 52
Grants to Farmers’ Association in
Niger
• Intervention: Provide capacity-building grants
to farmers’ association (World Bank, GoN)
• Treatment level: Farmers’ association
• Randomization level: Farmers’ association
• Randomization design:
– Partial lottery
– Randomized phase-in
– Encouragement scheme
• But huge implications for sample size!
Lecture Overview
•
•
•
•
Unit and method of randomization
Real-world constraints
Revisiting unit and method
Variations on simple treatment-control
Variations on Simple Treatment
and Control
•
•
•
•
•
Multiple treatments
Crossing or interacting treatments
Varying levels of treatment
Stratified randomization
Multiple-stage randomization
Multiple treatments
• Sometimes the core question is deciding
among different possible interventions
– Ie, in-person extension agent visits versus a call-in
hotline
• You can randomize these interventions
• Does this teach us about the benefit of any
one intervention?
• Do you have a control group?
•How to Randomize, Part
I - 59
Multiple treatments
Treatment 1
Treatment 2
Treatment 3
Cross-cutting treatments
• Test different components of treatment in
different combinations
– Improved seeds only, improved seeds plus
training, training only, no treatment
• Test whether components serve as substitutes
or complements
• What is most cost-effective combination?
• Advantage: win-win for operations, can help
answer questions for them, beyond simple
“impact”!
Varying levels of treatment
• Some villages are assigned full treatment
– All households receive access to SMS-based price
information
• Some villages are assigned partial treatment
– 50% of households receive access to SMS-based
price information
• Testing subsidies and prices
– Vary the price of seeds, inputs or access to market
information via mobile phone
Stratified Randomization
• Randomization should, in principle, ensure
balance in the treatment and control groups if
the sample size is large enough
• What happens when it is small?
• Stratified randomization can help to ensure
balance across groups when there is a small(er)
sample
– Divide the sample into different subgroups
– Select treatment and control from each subgroup
• What happens if you don’t stratify?
•63
Stratified Randomization
• Stratify on variables that could have important
impact on outcome variable (bit of a guess)
• Stratify on subgroups that you are particularly
interested in (where may think impact of program
may be different)
• Stratification more important when small data set
• Can get complex to stratify on too many variables
• Makes the draw less transparent the more you
stratify
• You can also stratify on index variables you create
•64
Mechanics of Randomization
• Need sample frame
• Pull out of a
hat/bucket/calabasse
• Use random number
generator in
spreadsheet program to
order observations
randomly
• Stata program code
• What if no existing list?
Source: Jenny Aker
•65
Mechanics of Randomization