Individual vs. Group
Randomized Trials
Jens Ludwig
University of Chicago,
Brookings Institution and NBER
Three things to consider
• Realizing randomization
• Power
• Nature of the intervention
Realizing randomization
• “Should I randomize at the individual or
the group level?”
Realizing randomization
• “Should I randomize at the individual or
the group level?”
• Sort of like asking:
– Should I take my $30 million Powerball prize
all at once, or spread it out in installments?
Why are so many American public
schools not performing better?
• Four major hypotheses:
–
–
–
–
Inadequate resources
Inefficient production technologies (curriculum, etc.)
Unmotivated teachers
Unmotivated students
• Education researchers usually can only
randomize what they can pay for
– Severely limits which of these we can study with
randomized experimental designs of any sort
Keep our eye on the prize
• Our real goal has to be to convince governments
to do more randomization
– Tennessee STAR
– Progressa
• Even Roland Fryer w/ Eli Broad at his back is
taking on just a modest slice (pay for grades)
• First consideration: Choose unit to randomize
you need to in order to be able to randomize
The rhetoric of randomization
• Me: “Can we randomly assign
the intervention?”
• City official in very large
Midwestern city:
The rhetoric of randomization
• Me: “Can we randomly assign
the intervention?”
• City official in very large
Midwestern city: “No way.”
The rhetoric of randomization
• Me: “Can we randomly assign
the intervention?”
• Me: “Well, could we do a pilot
program?”
• City official in very large
Midwestern city: “No way.”
The rhetoric of randomization
• Me: “Can we randomly assign
the intervention?”
• Me: “Well, could we do a pilot
program?”
• City official in very large
Midwestern city: “No way.”
• City official: “Sure. We do pilot
programs all the time.”
The rhetoric of randomization
• Me: “Can we randomly assign
the intervention?”
• Me: “Well, could we do a pilot
program?”
• Me: “How do you decide who
gets the pilot program if there
is excess demand?”
• City official in very large
Midwestern city: “No way.”
• City official: “Sure. We do pilot
programs all the time.”
The rhetoric of randomization
• Me: “Can we randomly assign
the intervention?”
• Me: “Well, could we do a pilot
program?”
• Me: “How do you decide who
gets the pilot program if there
is excess demand?”
• City official in very large
Midwestern city: “No way.”
• City official: “Sure. We do pilot
programs all the time.”
• City official: “Good question.”
The rhetoric of randomization
• Me: “Can we randomly assign
the intervention?”
• Me: “Well, could we do a pilot
program?”
• Me: “How do you decide who
gets the pilot program if there
is excess demand?”
• Me: “Could we flip a coin,
which would be the fair thing to
do?”
• City official in very large
Midwestern city: “No way.”
• City official: “Sure. We do pilot
programs all the time.”
• City official: “Good question.”
The rhetoric of randomization
• Me: “Can we randomly assign
the intervention?”
• Me: “Well, could we do a pilot
program?”
• Me: “How do you decide who
gets the pilot program if there
is excess demand?”
• Me: “Could we flip a coin,
which would be the fair thing to
do?”
• City official in very large
Midwestern city: “No way.”
• City official: “Sure. We do pilot
programs all the time.”
• City official: “Good question.”
• City official: “Ah, now I get it.”
The rhetoric of randomization
• Never use term “randomized experiment”
The rhetoric of randomization
• Never use term “randomized experiment”
• Acceptable talking points:
– “Pilot program”
– “Excess demand”
– “Fair, random lotteries”
• If there would be a natural unit for doing
“regular” pilot program, randomize that
– I.e., we’d just be implementing an unusually
informative pilot program
– For many education interventions would seem to
argue for group randomization (ex: pay for grades)
Second consideration:
Power
• There is the standard statistics version:
– More power from 1,000 kids distributed
across 1,000 schools than 1,000 kids
distributed across < 1,000 schools
• Due to non-independence of student observations
within schools
Then there is the real-world version
of this issue
• We live in a resource-constrained world
• Are there economies of scale in data
collection?
– Cluster randomization could reduce data
collection costs for same reason that
population surveys use two-phase sampling
• Are there economies of scale in delivering
the intervention itself?
For a given budget, cluster randomization
could in principle generate more power
• Imagine rapidly declining marginal costs of
“treating” and studying kids w/in a school
– Suppose school-based self-administered
student and teacher surveys plus use of
student administrative school records
– Suppose we have a DARE-like intervention
(“Don’t do drugs kids, look what happened to
me!”)
Power considerations in a
resource-constrained environment
• It’s conceivable you could get more power out of
a clustered sample of given size N, even with
non-trivial intra-class correlations (ICCs)
• Significant information requirements:
– Need to know ICCs for your sample & outcome
– Need school system to help you think about average
cost & marginal cost schedules for intervention
– Need very good survey subcontractor to help you
think about economies of scale in data collection
Power considerations
• On the other hand, you can run out of
observational units quickly in clustered
experiments
• Imagine randomizing schools:
– Even in Chicago, “just” 116 high schools, 483
elementary schools
– Imagine half of schools meet your eligibility criteria,
then half of principals agree to cooperate in
experiment, then you randomize half to T and C
• That would be 14 treatment high schools, 14 controls
• Or 60 treatment elementary schools, 60 controls
Nature of the intervention
• Previous two issues are shamelessly
pragmatic
• Individual vs. group choice could also
hinge on substantive considerations
Spillover effects
• Stable unit treatment value (SUTVA):
– Your treatment effect is independent of how many
others get treated
– If violated, then your treatment effect estimates
generates only to situations with similar take-up rates
• Sometimes you want to study intervention
independent of these spillovers, while
sometimes spillovers key part of treatment
Spillover effects example:
Moving to Opportunity (MTO)
• Initial concerns:
– Generic SUTVA concern
– Groups of public housing families moving in to
new areas might generate backlash
– Also didn’t want families to recreate any
unproductive baseline social ties
• New concerns:
– Families lose access to productive social ties,
so should we have randomized in groups?
Spillover effects example number 2
• Roland Fryer, paying kids for grades in
Chicago, DC, NYC
– Wants to change whole school climate around
academic achievement
• Generic adolescent (American?) antiintellectualism
• Plus specific “acting white” concerns
– Only by paying everyone (or offering to pay
everyone) can you change peer norms (or try
to change peer norms)
Interventions as public goods
• MTO suggests neighborhood safety key
factor for parental mental health
– Maybe for kids, too
– Could also affect learning in other ways, too
• If you reduce crime in neighborhood, every
kid in neighborhood will benefit
– This is sort of another way of talking about
economies of scale in providing intervention
Bottom lines
• Clustered experiments might help realizing
randomization
• Power considerations complex in a resourceconstrained environment
– Research community needs to develop infrastructure
to meet informational requirements for decisions
• Substantive considerations about role of
spillovers and “public good” interventions
• ‘Tis better to have randomized at the wrong level
than to have never randomized at all?