Addressing the Don’t Ask, Don’t Tell Practice in
Observational Studies: Using Interviews to
Understand the Assignment Mechanism
Jordan H. Rickles
Social Research Methodology Division
Graduate School of Education & Information Studies
University of California, Los Angeles
SREE Annual Meeting | Washington, DC | March 6, 2010
Presentation Outline
I.
II.
III.
IV.
V.
VI.
VII.
VIII.
IX.
Overview of Study
Brief Overview of Causal Inference
The Assignment Mechanism
Literature on Course Assignment
Data and Methods
Interview Findings
Example at Three Schools
Implications for Estimating Effects
Conclusions
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 2
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions
How many legs does a dog have if you call the tail
a leg?
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 3
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Two Objectives
 Draw attention to the importance of studying the
assignment mechanism

Statistical inference for causal effects “requires the
specification of a posited assignment mechanism describing
the process by which treatments were assigned to units”
(Rubin, 1991, p. 403).
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 4
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Two Objectives
 Draw attention to the importance of studying the
assignment mechanism


Statistical inference for causal effects “requires the
specification of a posited assignment mechanism describing
the process by which treatments were assigned to units”
(Rubin, 1991, p. 403).
“Knowledge of the selection process can significantly reduce
selection bias provided the selection process is valid and
reliably measured” (Cook, Shadish & Wong, 2008, p. 740).
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 5
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Two Objectives
 Draw attention to the importance of studying the
assignment mechanism



Statistical inference for causal effects “requires the
specification of a posited assignment mechanism describing
the process by which treatments were assigned to units”
(Rubin, 1991, p. 403).
“Knowledge of the selection process can significantly reduce
selection bias provided the selection process is valid and
reliably measured” (Cook, Shadish & Wong, 2008, p. 740).
In randomized controlled trial and regression discontinuity
designs assignment mechanism is usually known
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 6
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Two Objectives
 Draw attention to the importance of studying the
assignment mechanism




Statistical inference for causal effects “requires the
specification of a posited assignment mechanism describing
the process by which treatments were assigned to units”
(Rubin, 1991, p. 403).
“Knowledge of the selection process can significantly reduce
selection bias provided the selection process is valid and
reliably measured” (Cook, Shadish & Wong, 2008, p. 740).
In randomized controlled trial and regression discontinuity
designs assignment mechanism is usually known
In observational study the assignment mechanism is usually
unknown
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 7
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Two Objectives
 Provide example for using interviews to study the
assignment mechanism

Focus on study of taking algebra vs. pre-algebra in 8th grade
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 8
Causal Inference
Overview |
| Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

The Potential Outcomes Framework
 Causal effects defined by potential outcomes at the unit
of analysis


δi = yt(i) – yc(i)
Average Treatment Effect: E[δATE] = E[Yt] – E[Yc]
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 9
Causal Inference
Overview |
| Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

The Potential Outcomes Framework
 Causal effects defined by potential outcomes at the unit
of analysis



δi = yt(i) – yc(i)
Average Treatment Effect: E[δATE] = E[Yt] – E[Yc]
Fundamental problem of causal inference
(Holland, 1986)


Can only observe yt for units assigned to the treatment and can
only observe yc for units assigned to the control
Under selection independence can estimate ATE as
E[δATE]=E[Yt|D=1] – E[Yc|D=0]
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 10
Assignment Mechanism
Overview | Causal Inference |
| Literature | Data &
Methods | Interview Findings | Example at Three Schools | Implications | Conclusions

Selection Independence
 Treatment assignment is independent of the potential
outcomes


Holds for a randomized experiment
Not likely to hold for an observational study where treatment
assignment depends on factors (S) that are associated with
potential outcomes
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 11
Assignment Mechanism
Overview | Causal Inference |
| Literature | Data &
Methods | Interview Findings | Example at Three Schools | Implications | Conclusions

Selection Independence
 Treatment assignment is independent of the potential
outcomes



Holds for a randomized experiment
Not likely to hold for an observational study where treatment
assignment depends on factors (S) that are associated with
potential outcomes
Assumption of strongly ignorable treatment assignment
(Rosenbaum & Rubin, 1983)


Treatment assignment is independent of the potential
outcomes conditional on observed factors (S)
Under conditional independence can estimate ATE as
E[δATE]=E[Yt|S, D=1] – E[Yc|S, D=0]
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 12
Assignment Mechanism
Overview | Causal Inference |
| Literature | Data &
Methods | Interview Findings | Example at Three Schools | Implications | Conclusions

Validity of the assumption of strongly ignorable treatment
assignment depends on the assignment mechanism
 If methods for conditioning on S accurately capture the
true assignment mechanism then assumption holds
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 13
Assignment Mechanism
Overview | Causal Inference |
| Literature | Data &
Methods | Interview Findings | Example at Three Schools | Implications | Conclusions

Validity of the assumption of strongly ignorable treatment
assignment depends on the assignment mechanism
 If methods for conditioning on S accurately capture the
true assignment mechanism then assumption holds
 If methods for conditioning on S do not fully capture the
true assignment mechanism then assumption may not
hold
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 14
Assignment Mechanism
Overview | Causal Inference |
| Literature | Data &
Methods | Interview Findings | Example at Three Schools | Implications | Conclusions

Validity of the assumption of strongly ignorable treatment
assignment depends on the assignment mechanism
 If methods for conditioning on S accurately capture the
true assignment mechanism then assumption holds
 If methods for conditioning on S do not fully capture the
true assignment mechanism then assumption may not
hold
 How do we know if assumption holds?


Short answer: we don’t.
Better answer: we can investigate the assignment mechanism
for a more informed determination
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 15
Literature
Overview | Causal Inference | Assignment Mechanism |
| Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

What does the literature say about the assignment of 8th
graders to algebra vs. pre-algebra?
 Need to rely on related literature on ability grouping,
tracking, and high school course taking
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 16
Literature
Overview | Causal Inference | Assignment Mechanism |
| Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

What does the literature say about the assignment of 8th
graders to algebra vs. pre-algebra?
 Need to rely on related literature on ability grouping,
tracking, and high school course taking
 Rational choice (or human capital) theory: students are
matched with courses to efficiently accommodate
differences in student ability
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 17
Literature
Overview | Causal Inference | Assignment Mechanism |
| Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

What does the literature say about the assignment of 8th
graders to algebra vs. pre-algebra?
 Need to rely on related literature on ability grouping,
tracking, and high school course taking
 Rational choice (or human capital) theory: students are
matched with courses to efficiently accommodate
differences in student ability
 Nonacademic factors can mediate or constrain the
optimality of the rational choice theory


e.g., social class, student expectations, teacher & parent input,
school assignment practices
See Hallinan (1994) and Oakes, Gamoran & Page (1992)
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 18
Literature
Overview | Causal Inference | Assignment Mechanism |
| Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

How have past researchers estimated the effect of 8th
grade algebra?
 Mostly through regression-based modeling adjustments
based on a variety of covariates



OLS regression (Gamoran & Hannigan, 2000)
Path analysis (Smith, 1996)
Linear growth modeling (Ma, 2005)
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 19
Literature
Overview | Causal Inference | Assignment Mechanism |
| Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

How have past researchers estimated the effect of 8th
grade algebra?
 Mostly through regression-based modeling adjustments
based on a variety of covariates




OLS regression (Gamoran & Hannigan, 2000)
Path analysis (Smith, 1996)
Linear growth modeling (Ma, 2005)
Studies generally find positive effect of 8th grade
algebra on subsequent mathematics achievement


Findings depend on assumption of strong ignorability
Studies do not examine plausibility of this assumption
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 20
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

Data for estimation of causal effects
 Student-level data for a cohort of 2006-07 8th graders


From large urban California school district
Academic & demographic data from district’s administrative
files, covering 2002-03 through 2007-08 school years
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 21
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

Data for estimation of causal effects
 Student-level data for a cohort of 2006-07 8th graders



From large urban California school district
Academic & demographic data from district’s administrative
files, covering 2002-03 through 2007-08 school years
Data to examine the assignment process
 10 interviews of key school-level decision makers


From middle schools with 8th graders in both algebra and prealgebra courses
Interviews conducted in early 2009
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 22
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

The interview protocol
 Part I: six semi-structured, open-ended questions
about:



The decision-making process
The types of students typically assigned to algebra
The general philosophy regarding 8th grade algebra
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 23
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

The interview protocol
 Part I: six semi-structured, open-ended questions
about:




The decision-making process
The types of students typically assigned to algebra
The general philosophy regarding 8th grade algebra
Part II: questions about two student scenarios

Designed to get more standardized summary of assignment
mechanism across schools
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 24
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

The interview protocol
 Part I: six semi-structured, open-ended questions
about:




Part II: questions about two student scenarios


The decision-making process
The types of students typically assigned to algebra
The general philosophy regarding 8th grade algebra
Designed to get more standardized summary of assignment
mechanism across schools
Part III: rate importance of different information sources
in assignment decision

5 category Likert scale
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 25
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

The interview protocol
 Part II, Scenario 1: Martin
In 7th grade, Martin got a C in his first semester math class and
a D the second semester. He received C’s and B’s in his other
classes. He also received a mix of satisfactory and unsatisfactory
marks for work habits and cooperation. In 6th grade, Martin’s
math grades were a little higher, with a C the first semester and
a B the second semester. Similarly, he scored Basic on the 6th
grade math CST and Below Basic on the 7th grade math CST.
You heard a couple of Martin’s 7th grade teachers mention that he
started slipping behind and became more of a disruption in
class as the year progressed.
 On a scale of 1 to 5, how likely is it that you would enroll Martin
in Algebra in 8th grade?
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 26
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

The interview protocol
 Part II, Scenario 2: Maya

Maya moved to California from Mexico during her 6th grade year
and started attending this school in 7th grade. She is an English
learner and is struggling to keep up in most of her classes. She
received mostly D’s in 7th grade, but got a C in her second
semester math class. Her work habits and cooperation marks are
all satisfactory and there is no mention of any disciplinary problems
in her records. She scored Far Below Basic on her 6th grade
math and ELA CST tests but scored Basic on her 7th grade
math test. You do not know much else about her except what is in
her official record.
On a scale of 1 to 5, how likely is it that you would enroll Maya in
Algebra in 8th grade?
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 27
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

Analysis of interviews
 Coded each interview based on four assignment
process characteristics




Whether weight is given to objective (e.g., test scores) or
subjective (e.g., teacher recommendations) criteria
Whether decisions are primarily based on data systematically
collected by the district (i.e., observable) or non-systematic
data (i.e., unobservable).
Whether well defined inclusion/exclusion decision rules are
used or not
Whether the school’s course placement philosophy is more
protectionist or laissez faire
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 28
Data & Methods
Overview | Causal Inference | Assignment Mechanism | Literature |
Interview Findings | Example at Three Schools | Implications | Conclusions

Analysis of interviews
 Coded each interview based on four assignment
process characteristics





Whether weight is given to objective (e.g., test scores) or
subjective (e.g., teacher recommendations) criteria
Whether decisions are primarily based on data systematically
collected by the district (i.e., observable) or non-systematic
data (i.e., unobservable).
Whether well defined inclusion/exclusion decision rules are
used or not
Whether the school’s course placement philosophy is more
protectionist or laissez faire
Combined the codes with the close-ended responses
for analysis
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 29
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Schools draw from a common battery of information
sources in the decision-making process
Teacher Recommendation
7th Grade Math Course Marks
7th Grade Math CST
Individualized Education Plan (IEP)
MDTP Algebra Readiness Test
6th Grade Math CST
6th Grade Math Course Marks
School Attendance Record
Parent Pref erence
Student Pref erence
1
2
3
4
5
Importance of Inf ormation in Decision-Making Process
(1=not at all important; 5=very important)
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 30
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Schools differ in weight given to specific information
sources and the benchmarks (or cut-points) used to
determine algebra placement
 Differences exemplified in responses to scenarios
Favor Pre-Algebra
Scenario 1
(Martin)
n=5
Scenario 2
(Maya)
10%
20%
Favor Algebra
n=1
n=3
0%
Uncertain
n=4
n=4
30%
40%
50%
n=3
60%
70%
80%
90% 100%
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 31
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Underlying philosophy associated with strictness of
algebra placement benchmarks and decisions for
“borderline” students
 Three schools had a protectionist philosophy


Do not want to “program a kid for failure”
“We don’t want students to be in a class where they’re going to
struggle so much that they’re not going to be successful.”
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 32
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications | Conclusions

Underlying philosophy associated with strictness of
algebra placement benchmarks and decisions for
“borderline” students
 Three schools had a protectionist philosophy



Do not want to “program a kid for failure”
“We don’t want students to be in a class where they’re going to
struggle so much that they’re not going to be successful.”
Five schools had a laissez faire philosophy


“Kids have a right to fail.”
A student is “still better off failing the on-grade-level class than
if he’d taken the other one.”
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 33
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings |

Example at Three Schools | Implications | Conclusions
Can exemplify school differences in the assignment
mechanism by contrasting practices at three schools
 Haverbrook Middle School (HMS)





Half of the 600 8th graders in algebra
Teacher recommendations, math course grades & MDTP most
important sources of information
Soft benchmark of 70% on MDTP, C in 7th grade math class &
Basic on CST
Not strongly protectionist or laissez faire
Martin in algebra, need more info to place Maya
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 34
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings |

Example at Three Schools | Implications | Conclusions
Can exemplify school differences in the assignment
mechanism by contrasting practices at three schools
 Ogden Middle School (OMS)





Half of the 900 8th graders in algebra
Teacher recommendations, CST & weekly school-developed
math quizzes most important sources of information
No formal benchmarks but look for 60% correct on quizzes
laissez faire philosophy
Martin in pre-algebra, need more info to place Maya
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 35
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings |

Example at Three Schools | Implications | Conclusions
Can exemplify school differences in the assignment
mechanism by contrasting practices at three schools
 Shelby Middle School (SMS)





One-third of the 500 8th graders in algebra
CST, math grades & MDTP most important sources of
information
Strong benchmarks of 80% on MDTP, Advanced on CST, B in
7th grade math class
Protectionist philosophy
Martin and Maya in pre-algebra
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 36
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings |

Example at Three Schools | Implications | Conclusions
Distributions of prior achievement for algebra and prealgebra students at the three schools are consistent with
7th Grade Mathematics Standardized Test
6th Grade Mathematics Standardized Test
the general selection process described by each school
Shelby Middle School
PRE-ALG
PRE-ALG
ALGEBRA
ALGEBRA
8th Grade Math Course
8th Grade Math Course
Shelby Middle School
Ogden Middle School
PRE-ALG
ALGEBRA
Haverbrook Middle School
Ogden Middle School
PRE-ALG
ALGEBRA
Haverbrook Middle School
PRE-ALG
PRE-ALG
ALGEBRA
ALGEBRA
200
300
400
500
6th Grade Math CST Scale Score
600
200
300
400
500
600
7th Grade Math CST Scale Score
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 37
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings |
Observed probability of algebra placement given CST
performance and course grades also reflects school’s
reported selection process
 Shelby Middle School (SMS)
7th Grade Math Course Grade
6th Grade Math CST
Performance Level

Example at Three Schools | Implications | Conclusions
FBB
A
B
C
D
F
1.00
.
0.00
0.00
0.00
0.00 to 0.24
0.25 to 0.49
BB
0.80
0.13
0.06
0.00
0.00
0.50 to 0.74
0.75 to 1.00
Basic
0.76
0.19
0.12
0.19
0.07
Pro
0.93
0.76
0.57
0.25
0.09
Adv
0.88
0.87
1.00
0.83
0.75
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 38
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings |
Observed probability of algebra placement given CST
performance and course grades also reflects school’s
reported selection process
 Ogden Middle School (OMS)
7th Grade Math Course Grade
A
B
C
D
F
6th Grade Math CST
Performance Level

Example at Three Schools | Implications | Conclusions
FBB
.
0.00
0.16
0.10
0.18
0.00 to 0.24
0.25 to 0.49
BB
0.50
0.61
0.40
0.29
0.17
0.50 to 0.74
0.75 to 1.00
Basic
0.89
0.90
0.82
0.72
0.55
Pro
0.92
0.89
0.90
0.64
0.92
Adv
1.00
1.00
1.00
1.00
.
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 39
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings |
Observed probability of algebra placement given CST
performance and course grades also reflects school’s
reported selection process
 Haverbrook Middle School (HMS)
7th Grade Math Course Grade
A
B
C
D
F
6th Grade Math CST
Performance Level

Example at Three Schools | Implications | Conclusions
FBB
0.20
0.43
0.30
0.19
0.19
0.00 to 0.24
0.25 to 0.49
BB
0.84
0.55
0.42
0.37
0.33
0.50 to 0.74
0.75 to 1.00
Basic
0.88
0.89
0.65
0.83
0.53
Pro
1.00
0.94
1.00
0.71
0.71
Adv
1.00
1.00
1.00
1.00
1.00
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 40
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools |

Implications | Conclusions
Investigating the assignment mechanism through
interviews aides the causal effect research design
 Provides current & localized information about which
factors are associated with treatment assignment

What confounders should be controlled for?
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 41
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools |

Implications | Conclusions
Investigating the assignment mechanism through
interviews aides the causal effect research design
 Provides current & localized information about which
factors are associated with treatment assignment


What confounders should be controlled for?
Provides information about how the factors are
associated with treatment assignment

How should confounders get included in a statistical model?
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 42
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools |

Implications | Conclusions
Investigating the assignment mechanism through
interviews aides the causal effect research design
 Provides current & localized information about which
factors are associated with treatment assignment


Provides information about how the factors are
associated with treatment assignment


What confounders should be controlled for?
How should confounders get included in a statistical model?
Provides information about heterogeneity in the
assignment process across schools

Can a single fixed-effects model control for confounders at all
sites?
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 43
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools |

Implications | Conclusions
Example: specifying a logistic regression model for
assignment to algebra vs. pre-algebra
 Four different model specifications





Model 1: naïve model with school homogeneity
Model 2: informed model with school homogeneity
Model 3: informed model with random intercept
Model 4: informed model with random intercept & slopes
Want to use predicted probabilities for propensity scorebased causal effect estimates, but need to use most
appropriate propensity score model
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 44
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools |

Implications | Conclusions
Different models result in different propensity scores
 For the average student …
0.80
SMS
OMS
HMS
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Model 1
Model 2
Model 3
Model 4
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 45
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools |

Implications | Conclusions
Different models result in different propensity scores
 For a student like Martin …
0.80
SMS
OMS
HMS
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Model 1
Model 2
Model 3
Model 4
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 46
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools |

Implications | Conclusions
Different models result in different propensity scores
 For a student like Maya …
0.80
SMS
OMS
HMS
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Model 1
Model 2
Model 3
Model 4
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 47
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools |

Implications | Conclusions
Bias resulting for unobserved key factors should be tested
through sensitivity analysis
 MDTP not observed


Probably correlated with observed CST scores
Teacher recommendations not observed

Probably correlated with observed course grades
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 48
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications |

Conclusions
When the assignment mechanism is unknown,
investigating the assignment process can help control for
selection bias and communicate potential sources of bias
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 49
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications |


Conclusions
When the assignment mechanism is unknown,
investigating the assignment process can help control for
selection bias and communicate potential sources of bias
An investigation of the assignment mechanism does not
have to be extensive or resource intensive
 Short interviews conducted in a small sample of
schools provides a lot of information
 Interviews could be part of a pilot study
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 50
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications |

Conclusions
Complicated, heterogeneous assignment process found
for 8th grade algebra probably applies to many major
topics in education research
 Ability grouping, tracking & curricular intensity
 Intervention programs, tutoring services & school
choice
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 51
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods
Interview Findings | Example at Three Schools | Implications |


Conclusions
Complicated, heterogeneous assignment process found
for 8th grade algebra probably applies to many major
topics in education research
 Ability grouping, tracking & curricular intensity
 Intervention programs, tutoring services & school
choice
Observational studies can play an important role in
educational policy making
 But researchers must address the assignment
mechanism complexities
 And honestly communicate those complexities for
informed research consumption
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 52
Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview
Findings | Example at Three Schools | Implications |
Conclusions
How many legs does a dog have if you call the tail
a leg?
Four. Calling a tail a leg doesn’t make it a leg.
- Abraham Lincoln
Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 53
Addressing the Don’t Ask, Don’t Tell Practice in Observational Studies:
Using Interviews to Understand the Assignment Mechanism
Jordan H. Rickles
jrickles@ucla.edu
Social Research Methodology Division
Graduate School of Education & Information Studies
University of California, Los Angeles
SREE Annual Meeting | Washington, DC | March 6, 2010
Part of this research is made possible by a pre-doctoral advanced quantitative methodology training grant
(#R305B080016) awarded to UCLA by the Institute of Education Sciences of the US Department of Education. The
views expressed in this paper are the author’s alone and do not reflect the views/policies of the funding agencies or
grantees.