ANNUAL LECTURE
21ST MARCH 2012
Heterogeneous Agents, Social Interactions and
Causal Inference
Stephen W. Raudenbush
Lewis-Sebring Distinguished Service
Professor in the Department of Sociology at the University of Chicago and
Chairman of the Committee on Education
This talk is based on “Heterogeneous Agents, Social Interactions, and Causal Inference” by Guanglei Hong and Stephen W.
Raudenbush, to appear in Morgan, S. (Ed.) Handbook of Causal Analysis for Social Research (Springer 2012) and draws on
two examples originally reported in:
Savitz-Verbitsky, N. and Raudenbush, S.W. (in press). Evaluating community policing program in Chicago: A case study of
causal inference in spatial settings. To appear in Epidemiologic Methods; and
Raudenbush, S.W., Reardon, S. and Nomi, T. (in press). Statistical analysis for multi-site trials using instrumental variables. To
appear in Journal of Research and Educational Effectiveness.
The research reported here was supported by a grant from the Spencer Foundation entitled “Improving Research on
Instruction: Models Designs, and Analytic Methods;” and a grant from the W.T. Grant Foundation entitled “Building Capacity for
Evaluating Group-Level Interventions.”
Abstract
This talk will focus on two pervasive features of social interventions
designed to increase human health, skills, or productivity. First, the
interventions are usually delivered by human agents – physicians, teachers,
case workers, therapists, police officers, or workplace managers - who tend
to be ‘heterogeneous’ in the sense that they differ in their beliefs, training,
and experience. These agents enact the intervention and shape its effects.
Second, the participants in these interventions – patients, pupils, employees
or offenders - are typically clustered in organizational settings, and social
interactions among these participants influence the success of the
intervention. In this presentation, Stephen will argue that causal models
conventionally used in medical research are not well suited to study these
interventions. Instead, he proposes a model in which the heterogeneous
agents and social interactions among participants shape participants’
response to an intervention. Stephen will illustrate this model with studies
of community policing and high-school curricular reform.
Outline
Counter-Factual Account of Causation
The “drug-trial paradigm” for causal inference
An alternative paradigm for social interventions
Heterogeneous agents
Social interactions among participants
Examples
Community policing
High School Curricular Reform
Conclusions
Counter-factual Account of
Causality
In statistics (Neyman, Rubin, Rosenbaum)
In economics (Haavelmo, Roy, Heckman)
Potential Outcomes in a Drug Trial
Y(1): Outcome if the patient receives Z = 1
(the “new drug”)
Y(0): Outcome if the patient receives Z = 0
(the “standard treatment”)
Y(1) – Y(0): Patient-specific causal effect
E (Y(1) – Y(0)) =  : Average causal effect
Stable Unit Treatment Value
Assumption (Rubin, 1986)
• Each patient has two potential outcomes
• Implies
– Only one “version” of each treatment
– No “interference between units”
• Implies the doctor and the other patients have
no effect on the potential outcomes
Formally…
Y1( z1, z2 ,...,zn ; d )  Y1( z1)
Failure of SUTVA in Education
• Teachers enact instruction in classrooms
– Multiple “versions of the treatment”
• Treatment assignment of one’s peers
affects one’s own potential outcomes
– EG Grade Retention
– Hong and Raudenbush, Educational Evaluation and Policy
Analysis, 2005
– Hong and Raudenbush, Journal of the American Statistical
Association, 2006
Group-Randomized Trials
Potential outcome
Y1 j ( z1 j , z2 j ,..., znj ; t j )
 Y1 j (1,1,...,1; t j ) if j is assigned to treatm ent
 Y1 j (0,0,...,0; t j ) if j is assigned to control
Thus, each child has only two potential outcomes
– if we have “intact classrooms”
– if we have “no interference between classrooms”
Limitations of cluster randomized trial
Mechanisms operate within clusters
* Example: 4Rs
teachers vary in response
classroom interactions spill over
We may have interference between clusters
* Example: community policing
Alternative Paradigm
Treatment setting (Hong, 2004):
Yij ( z1 j , z2 j ,..., zn j ; t j )
A unique local environment for each treatment composed of
* a set of agents who may implement an intervention and
* a set of participants who may receive it
Each participant possesses a single potential outcome within each
possible treatment setting
Causal effects are comparisons between these potential outcomes
Example 1: Community Policing
(joint work with Natalya Verbitsky-Shavitz)
• Let Zj=1 if Neighborhood j gets
community policing
• Let Zj=0 if not
• Under SUTVA
 j  Y j (1)  Y j (0)
Relaxing SUTVA
Potential outcome for any unit depends on
the treatment assignment
 of ALL units in
the population, Y j ( Z j , Z  j )
Individual Causal Effect:


 j  Y j (Z j , Z  j )  Y j (Z *j , Z * j )
Population Average Causal Effect:

*
*
  E[ j ]  E[Y j (Z j , Z  j )  Y j (Z j , Z  j )]
“All or none”


 j  Y j (1,1)  Y j (0,0)
1
0
1
1
0
1
1
0
0
0
“Shall we do it in my neighborhood?”


 j  Y j (1, Z  j )  Y j (0, Z  j )
1
1
1
0
0
0
0
1
1
0
Do it only in high-crime areas:
effect on those areas


 j  Y j (1, Z  j )  Y j (0,0)
1, HC
1, HC
1, HC
0, HC
0, LC
0, HC
0, HC
0, LC
0, LC
0, LC
Do it only in high-crime areas:
effect on low-crime areas


 j '  Y j ' (0, Z  j ' )  Y j ' (0,0)
1, HC
1, HC
1, HC
0, LC
0, HC
0, HC
0, LC
0, HC
0, LC
0, LC
Spatial Causal Assumptions (1)
Functional Form:


Y j ( Z j , Z  j )  Y j ( Z j , F ( Z  j ))

Ex : F ( Z  j )  fraction of neighbors with z  1
1, #1
1, #3
1, #2
0, #5
0, #4
1 / 4
 
1/ 4
F3     1 / 2
 0 
 
 0 
Longitudinal Design: 25 districts, 279 “beats”
91 92 93 94 95 96 97 98 99
No community
policing
25 25 25 20 20 0
Community
policing
0
0
0
5
5
0
0
0
25 25 25 25
Results
Having community policing was especially good if
your surrounding neighbors had it
Not having community policing was especially bad
if your neighbors had it
*** So targetting only high crime areas may fail***
Example 2: Double-dose Algebra
Requires 9th-graders to take Double-dose Algebra if
they scored below 50 percentile on 8th-grade math
test
1200 students in 60 Chicago high schools
Enrollment Rates
Double-dose Algebra enrollment rate
by math percentile scores (city wide)
ITBS percentile scores
Conventional Mediation Model
(T, M,Y model)
Cut off (T)
Γ
Double-Dose
Algebra (M)
Δ
Algebra
Learning (Y)
• Assume no direct effect of T on Y (exclusion restriction)
•
•
Δ= Effect of double dose on the “compliers”
Δ Γ= Effect of assignment to double dose (“ITT” effect)
Nomi, T., & Allensworth, E. (2009)
Effects of Double-dose Algebra:
District-wide average
Effect of cutoff on taking DD (average compliance rate):
Increase prob by .72
District-wide average ITT effect on Y:
Average effect≈0.15
District-wide average Complier-Average Treatment Effect
Average ≈0.21
double-dose algebra effects varied across schools
But the policy changed classroom
composition!!
Classroom average skill levels by math percentile scores
Pre-policy
(2001-02 and 2002-03 cohorts)
Post-policy
(2003-04 and 2004-05 cohorts)
Implementation varied across schools in--• Complying with the policy
• Inducing classroom segregation
Exclusion Restriction Revised
T-M-C-Y model
Double-Dose
Algebra (M)
Algebra
score (Y)
Cut off (T)
Classroom
Peer ability (C)
Research Questions
1) What is the average effect of assignment to DD? (“ITT
effect”)
2) What is the average effect of taking double-dose
algebra? (effect “on the compliers”).
3) How much do these effects vary across schools?
4) What is the effect of taking double-dose Algebra,
holding constant classroom peer ability?
5) What is the effect of classroom peer ability, holding
constant taking double-dose Algebra?
Results
The effect of double-dose algebra on algebra scores by
the degree of sorting
Degree of sorting
Low
Average
High
ITT
0.21
0.15
0.11
Complier effects
0.29
0.23
0.14
School N
19
19
22
We now estimate the effect of taking DDA and classroom
peer composition
Statistical Models
Stage 1: the effect of Cut-off on Double Dose
and Peer Ability
E (DDij )   0 j  1 j (Below Cut )ij   2 j X ij   3 j X ij2
E (PEERij )   0 j   1 j (Below Cut )ij   2 j X ij   3 j X ij2
Stage 2: the effect of M and C on Y
E(Yij )   0 j  1E(DDij )   2E(Peerij )   3 j Xij   4 j Xi2
31
Stage 1 Results:
the average effect T on M and C
The effect of the cutoff score (T) on double-dose algebra
enrollment (M) and peer composition (C)
Double-dose algebra
enrollment
Coeff
SE
Peer composition
0.72***
-0.24***
0.03
0.03
Note: *** p<.001, **p<.01, * p<.05
32
Context specific effects:
The effect of cut off score on peer ability
The effects of cutoff score on double-dose algebra
enrollment and peer ability
The effect of cut off score on double-dose algebra enrollment
Stage 2 results:
The effect of M and C on Y
The average effect of taking double-dose algebra (M) and
peer ability (C) on Algebra test scores
Double-dose algebra
enrollment
Coeff
SE
Classroom Peer
composition
0.30***
0.06
0.40***
0.12
34
5. Conclusions
The reform enhanced math instruction for lowskill students, and that helped a lot
The reform also intensified tracking and that
hurt
On balance the effect was positive, but much
more so in schools that implemented double
dose with minimal tracking
Final Thoughts
Conventional causal paradigm:
* a single potential outcome per participant under each
treatment
Alternative paradigm
* a single potential outcome per participant in each
treatment setting
- aims to avoid bias
-open up new questions
Policy implications are potentially large