TRANSLATING RESEARCH INTO ACTION
How to randomize?
Rachel Glennerster
J‐PAL
povertyactionlab.org
1
Course Overview
•
•
•
•
•
•
•
Why evaluate? What is evaluation?
Outcomes, indicators and measuring impact
Impact evaluation
l i – why
h randomize
d i
How to randomize
Sampling and sample size
Analysis and inference
RCTs: Start to Finish
2
Lecture Overview
•
•
•
•
Unit and method of randomization
Real‐world constraints
Revisiting
i i i unit
i and
d method
h d
Variations on simple treatment‐control
3
Unit of Randomization: Options
1.
1 Randomizing at the individual level
2. Randomizing at the group level
er Randomized Trrial”
Cluster
ial
“Clust
• Which level to randomize?
4
Unit of Randomization: Considerations
• What unit does the program target for
treatment?
• What is the unit of analyssis?
5
Unit of Randomization: Individual?
6
Unit of Randomization: Individual?
7
Unit of Randomization: Clusters?
“Groups of individuals”: Cluster Randomized Trial
8
Unit
of Randomization: Class?
Unit of Randomization: Class?
9
Unit
of Randomization: Class?
Unit of Randomization: Class?
10
Unit
of Randomization: School?
Unit of Randomization: School?
11
Unit
of Randomization: School?
Unit of Randomization: School?
12
How to Choose the Level
• Nature of the Treatment
– How is the intervention administered?
– What is the catchment area of each
“unit of intervention”
– How wide is the potential impact?
• Aggregation level of available data
• Power requirements
• Generally, best to randomize at the level at
which the treatment is administered.
13
Lecture Overview
•
•
•
•
Unit and method of randomization
Real‐world constraints
Revisiting
i i i unit
i and
d method
h d
Variations on simple treatment‐control
14
Constraints: Political Advantages
Not as severe as often claimed
Lotteries are simple, common and transparent
Randomly
d l chosen
h
ffrom applicant
li
pooll
Participants know the “winners” and “losers”
Simple lottery is useful when there is no a
p
priori
reason to discriminate
• Perceived as fair
• Transparent
•
•
•
•
•
•How to Randomize, Part
I - 15
15
Constraints: Resources
• Most programs have limited resources
– Vouchers, Farmer Training Programs
• Re
ts than
Results
sults in more eligible recipien
recipients
resources will allow services for
• Are often an evaluator’s best friend
– Key is to get around mechanism implementers use
to match recipients to treatment
– Recruit until constraints met
– May distinguish recipients by arbitrary criteria
•How to Randomize, Part
I - 16
16
Constraints: contamination
ll
/
Spillovers/Crossovers
• Remember the counterfactual!
• If control group is different from the
counterfactual, our
our results
resu s can
can be biased
• Can occur due to
• Spillovers
• Crossovers
17
Constraints: logistics
• Need to recognized logistical constraints in
research designs.
workers
• Ex: de
de‐worm
ng trea
eatment by health wor
ers
worming
– Suppose administering de‐worming drugs was one
worker
many re
responsibilities of aa health wor
of many
er
– Suppose the health worker served members from
both treatment and control groups
– It might be difficult to train them to follow
different procedures for different groups,
groups and to
keep track of what to give whom
18
Constraints: fairness, politics
• Randomizing at the child
child‐level
level within classes
• Randomizing at the class‐level within schools
• Randomizing
d i i at the
h community‐level
i l l
19
Constraints: sample size
• The program is only large enough to serve a
handful of communities
an issue of statistical
• Primarily an
s ca power
• Will be addressed tomorrow
20
Lecture Overview
•
•
•
•
Unit and method of randomization
Real‐world constraints
Revisiting
i i i unit
i and
d method
h d
Variations on simple treatment‐control
21
What if you have 500 applicants for
l
500 slots?
• Consider non
non‐standard
standard lottery designs
• Could increase outreach activities
• Is this
hi ethical?
hi l?
•How to Randomize, Part
I - 22
22
Sometimesscreening
screeningmatters
matters
Sometimes
• Suppose there are 2000 applicants
Suppose there are 2000 applicants
• Screening of applications produces 500 worthy candidates
“worthy”
candidates
• There are 500 slots
• A simple lottery will not work
• What are our options?
23
Consider the screening rules
• What are they screening for?
• Which elements are essential?
• Selection procedures may exist only to reduce
eligible candidates in order to meet a capacity
constraint
constraint
• If certain filtering mechanisms appear
arbitrary (although not random),
random),
“arbitrary”
randomization can serve the purpose of
filtering and help us evaluate
24
Randomization in “the
the bubble”
bubble
• Sometimes a partner may not be willing to
randomize among eligible people.
• Partner might be willing to randomize in “the
the
bubble.”
• People “in
in the bubble
bubble” are people who are
borderline in terms of eligibility
– Just above the threshold Æ not eligible
eligible, but almost
• What treatment effect do we measure? What
does it mean for external validity?
25
Randomization in “the bubble”
Treatment
Within the
bubble,
b bbl
compare
treattmentt
to control
Non-participants
participants
Control
26
When screening matters: Partial
Lottery
• Program officers can maintain discretion
• Example: Training program
• Example:
l Expansion
i off consumer credit
di iin
South Africa
27
Phase‐in: takes advantage of
expansion
• Everyone gets program eventually
• Natural approach when expanding program
faces resource constraints
• What determines which schools, branches,
etc. will
ill b
be covered
d iin which
hi h year??
28
Phase‐in design
3
1
Round 1
2
3
3
3
1
3
3
2
3
2
Treatment: 2/3
Control: 1/3
Control:
2
3
1
2
1
3
3
3
2
3
1
2
1
2
2
2
1
3
3
Round 3
Treatment: 3/3
Control: 0
2
1
Round 2
Randomized
evaluation ends
2
2
Treatment: 1/3
Control: 2/3
2
1
3
1
3
1
2
1
1
2
3
1
29
Phase‐in designs
g
Advantages
• Everyone gets something eventually
• Provides incentives to maintain contact
Concerns
• Can complicate estimating long‐run effects
• Care required with phase‐in windows
• Do expectations of change actions today?
30
Rotation design
• Groups get treatment in turns
• Advantages
• Concerns
C
•How to Randomize, Part
I - 31
31
Rotation design
Round 1
Tr
T
Treatment:
Treatment
reatment
eatment: 1/2
Control: 1/2
Round 2
Treatment
from Round 1
Æ Control
Control from
Round 1 Æ
Treatment
——————————————————————————
32
“Want
Want to survey me? Then treat me”
me
• Phase-in
ase may not
ot provide
de eenough
ou be
e t to
ate
benefit
to late
round participants
p
from control group
g p mayy be critical
• Cooperation
•
•
•
•
Consider within‐group randomization
g , balsakhi p
program
g
E.g.,
All participants get some benefit
Concern: increased likelihood of contamination
33
Encouragement design: What to do
h you can’t randomize
d
when
access
• Sometimes it
it’ss practically or ethically
impossible to randomize program access
ve less than 100% takee‐
• But most programs have
up
• Randomize
R d i encouragement to receive
i
treatment
34
Encouragement design
Encourage
Do not encourage
participated
did not participate
compare
encouraged to
not encouraged
These must be correlated
do not compare
participants to
non-participants
Complying
Not complying
adjust for non-compliance
in analysis phase
35
What is “encouragement”?
encouragement ?
• Something that makes some folks more likely
to use program than others
eatment
• Not itself aa “tr
trea
tment”
• For whom are we estimating the treatment
effect
ff ?
• Think about who responds to encouragement
36
To summarize: Possible designs
•
•
•
•
•
Simple lottery
Randomization in the “bubble”
Randomized
d i d phase‐in
h
i
Rotation
Encouragement design
– Note: These are not mutually exclusive.
37
Methods of randomization ‐ recap
p
Design
Most useful
when…
•Program
oversubscribed
Basic Lottery
Advantages
•Familiar
•Easy to understand
•Easy to implement
•Can
C be
b implemented
i l
t d
in public
Disadvantages
•Control group may not
cooperate
•Differential attrition
•How to Randomize, Part
I - 38
38
Methods of randomization ‐ recap
p
Design
Phase‐In
Most useful when…
•Expanding over
time
•Everyone must
receive treatment
eventually
Advantages
Disadvantages
•Easy to understand
•Constraint is easy
to explain
•Control group
complies because
they expect to
benefit later
•Anticipation of
treatment may impact
short‐run behavior
•Difficult to measure
long‐term impact
•How to Randomize, Part
I - 39
39
Methods of randomization ‐ recap
p
Design
Rotation
Most useful when…
Advantages
•Everyone must
•More data points
receive something at than phase‐in
some point
•Not enough
resources per given
time period for all
Disadvantages
•Difficult to measure
long‐term impact
•How to Randomize, Part
I - 40
40
Methods of randomization ‐ recap
p
Design
Most useful
when…
•Program has to
b open to allll
be
comers
•When take‐up is
Encouragement
g
low,, but can be
easily improved
with an incentive
Advantages
•Can randomize at
i di id l level
individual
l l
even when the
program is not
administered at
that level
Disadvantages
•Measures impact of
h
who
h respond
d to
those
the incentive
•Need large enough
inducement to improve
p
take‐up
•Encouragement itself
may have direct effect
•How to Randomize, Part
I - 41
41
Lecture Overview
•
•
•
•
Unit and method of randomization
Real‐world constraints
Revisiting
i i i unit
i and
d method
h d
Variations on simple treatment‐control
42
Multiple treatments
• Sometimes core question is deciding among
different possible interventions
• Youu can
programs
can randomize
ra
e these programs
• Does this teach us about the benefit of any
one intervention?
i
i ?
• Do you have a control group?
•How to Randomize, Part
I - 43
43
Multiple treatments
Treatment 1
Treatment 2
Treatment 3
44
Cross‐cutting
Cross cutting treatments
• Test different components of treatment in
different combinations
es
• Tesst whether componen
componentss serve as substitutes
or compliments
• What
Wh iis most cost‐effective
ff i combination?
bi i ?
• Advantage: win‐win for operations, can help
answer questions for them, beyond simple
“impact”!
45
Varying levels of treatment
• Some schools are assigned full treatment
– All kids get pills
• Some schools are assigned partial treatment
– 50% are designated to get pills
• Testing subsidies and prices
46
Stratification
• Objective: balancing your sample when you
have a small sample
• What is it:
– dividing the sample into different subgroups
– selecting treatment and control from each
subgroup
• What happens if you don't stratify?
47
When to stratify
• Stratifyy on variables that could have important
p
impact on outcome variable (bit of a guess)
• Stratify on subgroups that you are particularly
i
interested
d in
i (where
( h
may think
hi k impact
i
off program
may be different)
ant when small data
• Stra
atificaation more
more importan
a se
set
• Can get complex to stratify on too many variables
• Makes the draw less transparent the more you
stratify
• You
Y can also
l stratify
if on index
i d variables
i bl you create
48
Mechanics of randomization
• Need sample frame
• Pull out of a hat/bucket
• Use random
random number
generator in
spreadsheet program to
order observations
randomly
• Stata program code
• What if no
no existing
ex s ng list?
Courtesy of Chris Blattman. Used with permission.
49
MIT OpenCourseWare
http://ocw.mit.edu
Resource: Abdul Latif Jameel Poverty Action Lab Executive Training: Evaluating Social Programs
Dr. Rachel Glennerster, Prof. Abhijit Banerjee, Prof. Esther Duflo
The following may not correspond to a particular course on MIT OpenCourseWare, but has been
provided by the author as an individual learning resource.
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.