Disruptive Peers and the Estimation of Teacher ValueAdded
Irina Horoi
Department of Economics
University of Illinois at Chicago
Ben Ost
Department of Economics
University of Illinois at Chicago
Draft version: 03-05-2014
Abstract
Using matched student-teacher data from North Carolina, we find that the arrival
of an emotionally disabled transfer student substantially reduces the academic
performance of his/her peers. Since standard value-added models fail to account for
these peer effects, we show that teachers have lower estimated value-added in years when
an emotionally disabled transfer student enters their grade. Importantly, since the peer
effects we document are highly non-linear, even value-added models that control for
average peer characteristics do not address this issue. In addition to illustrating an
important limitation of value-added modeling, our study provides credibly identified
estimates of the impact of non-cognitive peer characteristics on cognitive outcomes.
These estimates use within-school cohort variation in exposure to transfer students to
address concerns regarding reflection, homophily and the non-random placement of
students into classrooms.
Keywords: Teacher Quality, Value-Added, Disruptive Peers, Cognitive Achievement,
Transfer Students
1
Disruptive Peers and the Estimation of Teacher Value-Added
I.
Introduction
Understanding classroom peer effects is important both for determining optimal
grouping patterns and for generally understanding the educational production
function. While classroom peer effects have been studied extensively, most research
has focused on how the existence or absence of peer effects influences whether
students should be tracked or placed in heterogeneous classrooms. While these
considerations are first-order, the existence of peer effects also implies that the
educational production functions typically estimated in the literature omit an
important input. To the extent that these unmeasured peer inputs are correlated with
other school and classroom inputs, estimates of non-peer inputs will be biased. This
point is illustrated theoretically by Lazear (2001) in the context of estimating the
returns to class size, but little research has examined the issue empirically.
In this study, we consider the extent to which peer effects bias the estimated
impact of other inputs by showing how disruptive peers influence the estimation of
teacher value-added. While teachers are just one input whose estimated impact could
be biased by peer effects, the use of value-added estimates in high stakes personnel
decisions makes it particularly important to correctly estimate teachers’ impact.1 We
consider classroom disruption both because a single disruptive student has the
potential to substantially impact an entire class, and because relatively few studies
have compellingly documented the impact of disruptive peers on their classmates.
As of 2013, 40 states require that a teacher’s annual evaluation is based in part on her valueadded, and none of the models used to estimate this value-added control for the existence of
disruptive students (Doherty and Jacobs 2013).
1
2
We expand the literature on classroom peer effects in several ways. First, we
provide carefully identified evidence that peer non-cognitive attributes have a strong
influence on academic achievement. Second, we use matched data on students and
teachers over a six-year period to show that the existence of these non-cognitive peer
effects systematically influences the estimation of teacher value-added. Because most
VA models emphasize individual controls rather than peer controls, it is not
surprising that teachers have lower estimated VA in years when their students
generate negative peer effects. Unfortunately, we show that even value-added models
that attempt to control for peer effects do little to address this bias, because the peer
effects we document are highly non-linear.
While there is a large literature on classroom peer effects, most of this research
has focused on how peer academic performance impacts one’s academic
performance, with a much smaller literature exploring how the non-cognitive
attributes of one’s peers impact one’s academic performance. Our study contributes
to this latter literature by showing that transfer students who were previously
diagnosed with an emotional disability have a substantial negative impact on the
academic performance of students at their new school. Unlike some past work, we
are able to address the possibility that these students are non-randomly placed into
classrooms by aggregating peer groups to the school-grade-year level and including a
school-by-year fixed effect. Second, our focus on transfer students who were
previously diagnosed as emotionally disabled addresses concerns regarding reflection
and common shocks (correlated unobservables). Finally, we test for non-random
sorting into grades and find that the arrival of an emotionally disabled transfer student
3
is uncorrelated with all observable pre-determined characteristics, suggesting that
homophily is unlikely to bias estimates of the peer effects we document.
To study how peer effects influence the estimation of teacher value-added, we
examine how teacher effectiveness differs in years when teachers are in grades with
an emotionally disabled transfer student. To address concerns that school inputs are
correlated with the probability that an emotionally disabled transfer student enters a
school, we include teacher-by-school fixed effects when estimating the impact on
teacher value-added. To avoid conflating teacher value-added with student sorting
we aggregate the value-added analysis to the grade level. We find that teachers’
estimated value-added is reduced by 0.016 standard deviations when teaching in a
grade with an emotionally disabled transfer student. This effect is robust to
controlling for classroom factors, including average peer characteristics.
Educational production functions invariably omit important inputs and we do not
argue that this incompleteness necessarily leads to biased estimates of teacher quality.
That said, inputs such as parental time and neighborhood characteristics are likely to
be highly correlated over time and thus controlling for a lagged test score or student
fixed effect plausibly addresses most concerns regarding omitted family or
neighborhood inputs.
Compared to omitting family or neighborhood characteristics, failing to control
for peer effects presents a potentially more serious issue for value-added modeling for
several reasons. First, since classmates change each year, peer effects will be timevarying, and thus lagged test score will not control for current peer effects. Second,
the majority of value-added models emphasize individual rather than peer controls,
4
and these individual controls are unlikely to be good proxies for peer characteristics.
Finally, value-added models that control for peer composition invariably do so in a
fairly crude fashion by controlling for average peer characteristics such as test scores
or gender.
II.
Related Literature
A growing body of literature in the field of education focuses on the impact of
non-cognitive peer characteristics on cognitive outcomes (Neidell and Waldfogel
2010; Hoxby 2000; Lavy and Schlosser 2011;Fletcher 2009a, 2009b; Friesen, Hickey,
and Krauth 2010; Imberman, Kugler, and Sacerdote 2012; Carrell and Hoekstra 2010;
Figlio 2007). Of this literature, only a few studies have focused on disruptive students
at the elementary level (Fletchera,b 2009; Carrell and Hoekstra 2010; Figlio 2007;)
with one study looking at the middle school level (Friesen , Hickey, and Krauth
2010).
There are several obstacles to studying the impact of disruption on peer outcomes.
First, disruption is likely endogenous to teacher and peer quality and thus it is
necessary to use pre-determined or exogenous determinants of disruption to
instrument or proxy for disruptive students. Second, as in all peer effects studies, it is
necessary to address the possibility of common shocks (correlated unobservables) and
homophily (sorting).
In a series of related papers, Fletcher proxies for disruption using the diagnosis of
an emotional disability and uses school fixed effects and student fixed effects to
address concerns of non-random student placement. While these controls likely
address part of the sorting of students to classrooms, because he has access to only a
5
single cohort of data, he is unable to aggregate the analysis to the grade level and
therefore the systematic placement of ED students has the potential to bias estimates.
Figlio (2007) likewise uses student fixed effects to address sorting into classrooms.
He addresses the reflection problem by using data on males with female sounding
names as an instrument for disruptive behavior as these boys have higher propensities
to act out.
As in our paper, Carrel and Hoekstra (2010) address the possibility of sorting of
students to classrooms by aggregating peer groups to the school-grade-year level, and
use variation across grade cohorts within a school over a time. Their paper solves the
reflection problem using data on parental domestic disputes as a novel instrument for
disruptive behavior. Using cohort variation similar to that of Carrel and Hoekstra
(2010) Friesen, Hickey, and Krauth (2010), explore spillovers of various disabled
peers on a given student’s academic achievement. Unlike the Carrel and Hoekstra
article, Friesen, Hickey and Krauth are not able to address the possibility of reflection
since they only observe students after they have had many years of interaction.
While there are several studies of the impact of student disruption, none of these
studies relate the peer estimates to teacher value-added and only Carrel and Hoekstra
(2010) and Figlio (2007) are able to address the reflection problem.
III.
Institutional Background
Serious emotional disturbance or emotional disability as we refer to it is one of
the disabilities covered by the Individuals with Disabilities Education Act (IDEA), a
law passed in 1990, which governs how states provide interventions and services to
6
students with these disabilities at a minimum2. This legislation defines an emotional
disability as “a condition exhibiting one or more of the following characteristics over
a long period of time, and to a marked degree that adversely affects a child’s
educational performance:
a) An inability to learn that cannot be explained by intellectual, sensory, or
health factors.
b) An inability to build or maintain satisfactory interpersonal relationships with
peers and teachers.
c) Inappropriate types of behavior or feelings under normal circumstances.
d) A general pervasive mood of unhappiness or depression.
e) A tendency to develop physical symptoms of fears associated with personal or
school problems.”
While it is likely that students diagnosed with this disorder are heterogeneous,
many of the behaviors can be directly linked to classroom disruption. The screening
and evaluation for emotional disability guidelines provided by the North Carolina
Department of Public Instruction, for example, lists these behaviors corresponding to
characteristics (a)-(e) defined under IDEA: “aggressive and authority challenging
behaviors, overreaction to environmental stimuli, markedly diminished interest in
activities, agitated, and physical manifestation to fear that have psychosomatic
origin.”3
2
For additional information see: http://idea.ed.gov/explore/home
This is not an exhaustive list of the behaviors that fit into one or more of the federally defined
characteristics. For more examples and information see:
http://ec.ncpublicschools.gov/instructional-resources/behavior-support/resources/screening-andevaluation-for-serious-emotional-disability.
3
7
IV.
Data
This study uses restricted access student-teacher-matched data from the North
Carolina Education Research Data Center (NCERDC). The NCERDC manages data
on North Carolina’s public schools. The North Carolina’s public school data contains
rich information on students, classrooms, teachers, schools, and districts. It includes
this information for all public school students in the state of North Carolina from
1995-2012. However, course membership information necessary for matching
students accurately to classrooms is unavailable before 2006 and incomplete for that
year. Therefore, we use data from 2007-2012 for the present analysis
To create our estimation sample we start with student test score data from 20062012. We restrict it to 4th and 5th graders in the years 2007-2012, using 2006 to obtain
baseline test scores. Additionally, we include only students who have taken
comparable standardized tests in Math and Reading referred as end-of-grade tests,
and who have a baseline test score. To simplify the analysis, we focus our attention
on primary school grades where classrooms are generally self-contained. Finally, to
determine math and reading classrooms and their associated teachers, we use the
2007-2012 course membership data. These data allow us to match students to all their
subject specific classrooms and teachers, which we then merge with the test score
data for those same years.
Table 1 shows the descriptive characteristics for the entire student sample, the
sample of emotional disabled (ED) students, and the ED transfer students that we use
to identify disruptive peer treatment. Emotionally disabled transfer students appear
different than the average student in our sample, but are similar to the average
8
emotionally disabled student. For example ED transfer students are thirty-four
percentage points more likely to be male, twenty-eight percentage points more likely
to be African American, thirty percentage points more likely to come from
economically disadvantaged homes, lower test performers (on average scoring .9 of
standard deviation lower on their baseline math score and .84 of a standard deviation
on their baseline reading score).4 They are on average placed in smaller classes and
have marginally less experienced teacher than the average student. While different
from the average student, ED transfer students share more similarities with the
average ED student, as can be seen by comparing columns 3-4 to 5-6. For example
ED transfer students are only two percentage points more likely to be male, six
percentage points more likely to be African American, perform only .1 of a standard
deviation lower on their baseline math and reading tests, and tend to be placed in
similarly small classroom (on average about one student more) in comparison with
the average ED student. The descriptive statistics in Table 1 convey that ED transfer
students may not be randomly assigned to classrooms, as they appear to be different
from the average student, and consequently may have distinctly different needs.
In order to directly examine the degree to which ED transfer students are nonrandomly placed into classrooms, we examine the characteristics of classrooms with
and without an ED transfer student. Comparing columns one and three in Table 2
shows that on average classrooms with an ED transfer student are considerably
different than classrooms without an ED transfer student: about six percentage points
4
Test scores including pretests have been normalized such that grade-by-year test score means
are zero with a standard deviation of one.
9
more male, ten percentage points more African American, and eleven percentage
point more economically disadvantaged.
While students in classrooms with an ED transfer student on average score 0.33 of
a standard deviation lower on their math achievement test, it would be wrong to
interpret this difference as the causal effect of the ED transfer student. This
difference must be partly driven by student sorting since students who are in
classrooms with an ED transfer student also scored poorly on their math test in the
previous year (premath score is -.31).
Simple empirical models using classroom variation will fail to control for the nonrandom selection of students in classrooms. Furthermore, even models that include
school fixed effects will be biased since within school sorting will still bias these
estimates. For this reason we use grade variation in exposure to an ED transfer
student combined with school-by-year fixed effects to identify estimates. In Section
VI we test for and fail to find evidence that grade variation in ED transfers is driven
by student selection.
V.
Empirical Approach
A. Peer Effect on Student Achievement
Identifying and estimating the impact of high-needs children on their peers is
complex due to issues of correlated unobservables, reflection, and homophily, which
concern all peer effect studies (Manski 1993; Moffitt 2001). To illustrate these issues
in our context, consider estimating the naïve peer effects model shown in (1).
(1) 𝑌𝑖𝑐𝑔𝑠𝑡 = 𝛼𝐶𝑙𝑎𝑠𝑠𝐸𝐷𝑇𝑆𝑐𝑠𝑡 + 𝜔𝑌𝑖𝑐𝑔𝑠(𝑡−1) + 𝑋𝑖𝑡 𝛽 + 𝐶𝑐𝑠𝑡 𝛿 + 𝜀𝑖𝑐𝑔𝑠𝑡
10
Yicgst is a subject-specific test score of individual i in classroom c in grade g in school
s at time t, ClassEDTScst is an indicator set to one if the subject specific class in
school s at time t has an emotionally disabled transfer student, Yicgs(t-1) is the lagged
subject specific test score , Xit is a vector of student demographic information at time
t, Ccst is a vector of classroom level characteristics of classroom c in school s at time
t, and εicgst is an error term. The parameter of interest in equation (1) is , which
reflects the mean difference in achievement between classrooms with and without an
ED transfer student conditional on observable student and classroom characteristics.
The implicit assumptions required for
to be an unbiased estimate of exposure to an
emotional disabled student are:
1) Conditional on student and classroom controls, students who are
exposed to an ED transfer student are comparable to those who are not
in classrooms with an ED transfer student (homophily).
2) Emotional disability status, i.e. behavior associated with this
classification, and peers’ student achievement is not simultaneously
determined (reflection).
3) Conditional on student and classroom controls, the error term is
orthogonal to ClassEDTScst : i.e. there exists no correlated
unoberservables with exposure to an ED transfer student and student
achievement.
11
In our context, the three assumptions required for
to be unbiased are unlikely to
hold with the specification used in equation (1). Assumption one requires that
students and teachers are randomly assigned to classrooms. As highlighted in Section
IV, classrooms with an ED transfer student look quite different on observables both
across and within schools. This evidence suggests that unless we have an instrument
that creates random variation in classroom formation, classroom variation should not
be used to identify the effect of exposure to an ED transfer student on student
achievement. We address the endogenous peer formation by aggregating exposure to
an ED transfer student to the grade rather than the classroom within a school and
year. Past studies have used this same aggregation strategy to address homophily
(Friesen, Hickey, and Krauth 2010; Carrel and Hoekstra 2010). In Section VI we test
directly for endogeneity of peers at the grade level, and find no evidence of it.
Assumption two requires that a transfer student’s emotional disability is not
simultaneously determined with their peers’ achievement. Specification (1) uses the
contemporaneous measure of an ED transfer student to generate ClassEDTScst ,
which will lead to spurious peer effects if low-achieving students in a class cause a
transfer student to become emotionally disabled. To satisfy assumption two, we
follow past work (Hoxby and Weingarth 2005; Lavy and Schlosser 2011; Neidell and
Waldfogel 2010), and we identify exposure to ED transfer student by classifying an
ED transfer student as a student that was diagnosed as emotionally disabled in the
previous year. Transfer status is additionally useful to overcome reflection, because
transfer students are highly unlikely to have ever had exposure to their current peers.5
5
Student relocations have also been used in the peer effect literature to resolve reflection
(Imberman, Kugler, and Sacerdote, 2012).
12
Lastly, the simple specification in equation (1) is unlikely to satisfy assumption
three. For instance, if parents transfer their ED students to schools because of a
particularly effective new principal, estimated peer effects will be biased since this
new principal will also directly impact the performance of all students in the school.
We address this concern by controlling for school-by-year fixed effects. Since we
also control for grade-by-year fixed effects, common shocks will only bias our
estimates if parents transfer their ED students to particular schools due to school-bygrade-by-year specific factors.
In light of the above issues, our preferred model is:
(2) 𝑌𝑖𝑔𝑠𝑡 = 𝛼𝐺𝑟𝑎𝑑𝑒𝐸𝐷𝑇𝑆𝑔𝑠𝑡 + 𝜔𝑌𝑖𝑔𝑠(𝑡−1) + 𝑋𝑖𝑡 𝛽 + 𝐺𝑔𝑠𝑡 𝛿 + 𝜃𝑠𝑡 + 𝜆𝑔𝑡 + 𝜀𝑖𝑔𝑠𝑡
where Yigst is a subject specific test score of individual i in grade g in school s at time
t, GradeEDTSgst is an indicator set to one if the grade in school s at time t has an
emotional disabled transfer student, Yigs(t-1) is the lagged subject specific test score ,
Xit is a vector of student demographic information at time t, Ggst is a vector of grade
level peer characteristics of grade g in school s at time t, θst is a school-by-year fixed
effect, λgt is a grade-by-year fixed effect, and εigst is an error term. Empirical
specifications of education production functions with student lagged test scores on the
right hand side are widely used and preferred in the economics of education literature
because they more fully control for past inputs (Kane and Staiger 2008). We estimate
four variations of equation (2). First, we estimate a base model that includes the
variable of interest, GradeEDTSgst , and student controls. Then, that same
13
specification along with school fixed effects, and another one with school-by-year
fixed effects. Finally, we include grade peer controls in the specification with schoolby-year fixed effects.6 In all our models we cluster the standard errors at the schoolby-year-by-grade level, as this is the level of identifying variation.
B. Peer Effect on Teacher Value-Added
To evaluate how disruption impacts estimated value-added, we implement a twostep procedure. First, we first estimate value-added for every teacher in each year that
they teach. We allow value-added to be time varying to mimic the value-added
models typically used in policy. We then use these teacher-by-year value-added
estimates as the dependent variable to assess whether teacher value-added fluctuates
in years when the teacher teaches in a grade with an ED transfer student.
Specifically, to estimate teacher-by-year value we run a regression as described by
equation (3):
(3) 𝑌𝑖𝑗𝑐𝑔𝑠𝑡 = 𝜔𝑌𝑖𝑗𝑐𝑔𝑠(𝑡−1) + 𝑋𝑖𝑡 𝛽 + 𝐶𝑗𝑐𝑠𝑡 𝛿 + 𝜇𝑗𝑡 + 𝜀𝑖𝑐𝑗𝑔𝑠𝑡
where Yijcgst is subject specific test score of individual i matched to subject specific
teacher j in classroom c in grade g in school s at time t, Yijcgs(t-1) lagged subject
specific test score of individual i matched to subject specific teacher j in classroom c
in grade g in school s, Xit is a vector of student demographic information at time t,
6
It is important to note that we cannot disentangle the endogenous and exogenous portions of the
peer effect.
14
Cjcst is a vector of classroom level characteristics of classroom c with teacher j in
school s at time t, μjt are teacher-by-year fixed effects, and εicjgst is an error term. 7
Using the estimated teacher-by-year value-added estimates as the dependent
variable we examine the impact of exposure to an ED transfer student. Analogous to
equation (2), we estimate (4):
(4) 𝜇𝑗𝑠𝑡 = 𝛼𝑇𝑒𝑎𝑐ℎ𝐺𝑟𝑎𝑑𝑒𝐸𝐷𝑇𝑆𝑗𝑠𝑡 + 𝜃𝑗𝑠 + 𝛾𝑡 + 𝜀𝑗𝑠𝑡
where μjst is the predicted subject specific teacher effectiveness for teacher j in
school s at time t, TeachGradeEDTSjst is an indicator equal to one if the teacher j in
school s at time t teaches test specific subject in a grade with an ED transfer student,
θjs is a teacher-by-school fixed effect, γt is a year fixed effect, and εjst is an error
term. For equation (6) to yield a causal estimate, teachers need to be randomly
assigned to grades with an ED transfer student conditional on being in a school.8 All
standard errors are clustered at the school-by-year-by-grade level.
An important question is whether the impact of peer interactions on value-added
calculations depends on the observable characteristics controlled for in the value7
The classroom composition effect , is capture through a three-step procedure as described by
Isenberg and Walsh (2013). First a specification is run as described by (3.1):
(3.1) 𝑌𝑖𝑗𝑐𝑔𝑠𝑡 = 𝜔𝑌𝑖𝑗𝑐𝑔𝑠(𝑡−1) + 𝑋𝑖𝑡 𝛽 + 𝐶𝑗𝑐𝑠𝑡 𝛿 + 𝜇𝑗𝑠 + 𝜀𝑖𝑐𝑗𝑔𝑠𝑡
This specification is similar to (3), with the exception of teacher-by-year fixed effects, which are
exchanged with teacher-by-school fixed effects. This allows us to compare multiple classrooms
on a teacher over time in a school. In step two, we calculate an adjusted subject specific test score
that nets out the contribution of the classroom characteristics: (3.2)𝑌̂𝑖𝑗𝑐𝑔𝑠𝑡 = 𝑌𝑖𝑗𝑐𝑔𝑠𝑡 − 𝐶𝑗𝑐𝑠𝑡 𝛿
̂ijcgst in place of the actual test
In the final step, we use the adjusted subject specific scores, Y
scores, and estimate (3), omitting classroom variables, Cjcst , from the specification.
8
While not shown in this paper currently, conditional on school, teacher experience is not a
predictive measure of being assigned to teach in a grade with an ED transfer student within a
school and year.
15
added model. To investigate this question, we show how the results from equation (4)
change when the value-added estimates are calculated using various individual and
peer controls.9
VI.
Results
A. Peer Effect on Student Achievement
Table 3 reports the effects of grade exposure to an ED transfer student on Math
and Reading achievements of regular students. The first four columns show the effect
for math achievement, and the last four columns report the estimates for reading
achievement. Column 1 report OLS estimates controlling for student level
demographics, lagged student achievement in math, and grade-by-year dummies.10 In
column 2 school fixed effects are added, in column 3 school-by-year fixed effects are
included in place of school fixed effects, and in column 4, grade-level peer
characteristics are added.11 Columns 5-8 show the same specifications for reading
achievement.
The results shown in Table 3 provide strong evidence that student disruption
impacts math scores. Estimates in math drop by about 40% when school fixed effects
9
The specifications include one base model with only student level controls; the second
specification adds on a student level dummy for predetermined emotionally disabled student; the
third exchanges dummy for predetermined emotionally disabled with a dummy for emotionally
disabled transfer student; the fourth includes onto specification three the percent of students in a
classroom that are emotionally disabled along with other classroom controls; the fifth exchanges
% of student in classroom that are emotionally disabled with % that are ED transfer students; the
sixth exchanges % that are ED transfer students in classroom with the nonlinear format of a
dummy indicator of whether a student is in a classroom with an ED transfer student, and lastly we
use in place of classroom the grade level nonlinear format of the peer effect.
10
The student level demographics include: a male gender dummy, race dummies (African
American, Hispanic, and White), a dummy for limited English proficiency, and a dummy for
being economically disadvantaged.
11
The grade level peer controls include: percent male, percent African American, percent
Hispanic, percent White, percent limited English proficiency, percent economically
disadvantaged, and average pretest.
16
are added (column 2) but remain fairly stable (and statistically indistinguishable)
when subsequently adding school-by-year and peer characteristics. The fact that the
estimated peer effect falls considerably when adding school fixed effects suggests
that emotionally disabled students tend to transfer to schools with lower achieving
students and failing to account for this selection biases estimates downward. The
relative stability of the estimates in columns 2-4 is reassuring.
In reading the estimates drop by more than half off the base specification when
school fixed effects are included (column 6), and stay statistically the same in
magnitude but become insignificant across more controlled specifications.
In our preferred specification a single emotionally disabled student causes the average
math performance of other students in the grade to be reduced by 0.016 standard
deviations.12 This finding is consistent with the literature on academic externalities
associated with disruptive peers that consistently finds negative effects (Carrel and
Hoekstra 2010; Figlio 2007; Friesen, Hickey, and Krauth 2010; Fletchera,b 2009). The
effect sizes we find are moderate, particularly for using grade variation, and are in
line with the literature.13 In our preferred specification, which includes school-by-year
fixed effects, grade-by-year dummies, and student and grade level controls, the
12
Estimating peer effects at the grade level is similar to using grade-level variation to instrument
for having an ED transfer student in one’s classroom. In fact, our preferred specification is
simply the reduced form from that IV specification. To estimate the IV specification, one simply
scales up our estimate of 0.016 based on the inverse probability that a student is in classroom with
an ED transfer student, given that they are in a grade with an ED transfer student. We opt not to
estimate the peer effects using the IV specification because this specification assumes that
students have no interaction with students in their grade outside of their classroom, an assumption
we find implausible. That said, when the IV specification is used, the estimates are on par with
the literature.
13
Fletcher b(2009) finds a 5% of standard deviation reduction in math and reading scores from
exposure to a classmate with emotional problems. In comparing our estimates to his studies, it is
important to keep in mind that we use grade-level variation whereas those studies use class-level
variation.
17
exposure effect for math and reading respectively is - 1.6% of standard deviation and
- 0.63% of standard deviation. While the reading result is insignificant, the math
result is significant at the 1 percent level.14
B. Disruption vs. Low-Ability
In addition to the fact that emotionally disabled transfer students have higher
tendencies to be disruptive, they also have a tendency to be low-achieving. On
average, ED transfer students score about one standard deviation lower on math and
reading pretests relative to the average student in the sample. As such, it is possible
that what we refer to as disruption is actually driven by low-achieving knowledge
spillover. To explore this, we estimate the preferred specification, but instead of using
grade exposure to ED transfer student, we use grade exposure to a low-achieving
transfer student that is not emotionally disabled. We characterized students as lowachieving if they scored more than one standard deviation lower than average in their
math or reading pretests respectively. It is important to note that if low-achieving
transfer students also contribute to disruption, these specifications do not isolate the
pure achievement spillover.
Table 4 shows side by side results for the effect of exposure to an ED transfer
student, and the effect of exposure to low-achieving non-emotionally disabled
transfer student for both math (columns 1 and 2) and reading (columns 3 and 4)
student outcomes. Being exposed to a low-achieving transfer student lowers own
student achievement on math and reading: 0.89% of a standard deviation in math and
0.59% o a standard deviation in reading. While these are both significant, they are
14
Finding a smaller impact on reading scores is a consistent finding in the literature. The most
likely explanation is that school is where most math learning occurs, whereas reading is more
likely to be learned both at school and at home.
18
smaller in magnitude than grade exposure to an ED transfer student for math and
reading respectively. As a result we believe that the effect of exposure to an ED
transfer student is at least partly driven by classroom disruption rather than cognitive
spillovers. It is important to note that for the analysis of teacher value-added, it is
unimportant whether the peer effects are driven by cognitive spillovers or classroom
disruption so long as a peer effect exists.
C. Falsification Tests
To test for whether ED transfer students endogenously enter particular grades in a
school, we regress exogenous predetermined characteristics of students on whether or
not the student was in a grade with an emotionally disabled transfer student
conditional on school-by-year and grade-by-year fixed effects. In addition we also
test for student sorting into classes with an ED transfer student using the same
approach, but measure ED transfer students at the classroom level rather than the
grade level. The idea behind these tests is to see whether ED transfer students enter
grades or classes that were going to be performing badly in any case.
These results are presented in Table 5. Column 1 shows results for the classroom
level regression, and column 2 shows the results for the grade level specification. We
find that a number of predetermined student characteristics such as gender, Hispanic,
previously emotional disability, and pretests scores are predictive of the probability of
being assigned a classroom with an ED transfer student within a school and year. For
example a student classified with an emotional disability in a previous year is 7.5
percentage points more likely to get assigned to a class with an ED transfer student.
These results suggest that ED transfer students are systematically placed into certain
19
classrooms and thus, across class variation cannot be used to identify the impact peer
effects.
Column 2 of Table 5 shows that when we use grade variation within a school and
year, we find that none of the student characteristics predict assignment to grades
with an ED transfer student. This suggests that conditional on school-by-year fixed
effects, ED transfer students are not systematically placed into grades with certain
types of students. We view this as strong evidence that selection is not a problem
with this aggregated variation.15
D. Heterogeneous Effects by Student Gender, Race, and Economic Disadvantage
Status
Disruption can potentially have different effects on student achievement
depending on students’ characteristics. This heterogeneity is important because to the
extent that certain students are less or more impacted, this helps to provide a policy
prescription with respect to how to group students. Table 6 shows the extent of
heterogeneity along the dimensions of gender, race, and socio-economic background.
Throughout Table 6 we use our preferred specification that estimates equation (2)
using the full set of student controls, grade peer controls, as well as school-by-year
and grade-by-year fixed effects.
Panel A of Table 6 shows that there is little heterogeneity across groups for math
achievement. While the point estimate is slightly smaller for economically
disadvantaged students, the difference is statistically indistinguishable from the
15
The series of regressions use the sample of students in math classes. When we run these
regressions using the reading sample the results are qualitatively and quantitatively the same.
These separate results are available upon request from the authors.
20
aggregate specification shown in Table 3. More generally, all 6 estimates shown in
Panel A are statistically indistinguishable from one another and the estimates are
fairly close in magnitude.
Panel B of Table 6 shows that there is similarly little heterogeneity in the results
for reading scores. Males face on average a 0.89% of standard deviation reduction in
achievement and females face a 0.29% of standard deviation reduction, however
these effects are not statically distinguishable from one another. In fact, none of the
estimates in Panel B are statically distinguishable from those reported in Table 3 for
the entire sample.
While there appears to be little heterogeneity in the impact of disruptive peers,
Table 6 does demonstrate that the basic results from Table 3 are very robust across
sample choices. Math achievement appears to be substantially impacted by disruptive
peers whereas estimates for reading achievement are smaller and statistically
insignificant.
E. Peer effect and Teacher Value-Added
Table 7 presents the results of equation (6): the effect of teacher exposure to an
ED transfer student on teacher-by-year value-added. Separate results are presented for
math and reading teachers shown in Panel A and Panel B respectively. All regressions
include year dummies and teacher-by-school fixed effects. Each column, however,
differs with respect to the specifications used to calculate the teacher-by-year valueadded. Column one includes only student level controls (lagged test score, race,
gender, economic disadvantaged) in the estimation of valued-added. Column two
includes an additional student level indicator for whether student was diagnosed as
21
emotionally disabled in a previous year, and column three exchanges the indicator of
an emotionally disabled student with an emotionally disabled transfer student
indicator. Importantly, columns 1-3 only include individual level controls in the
value-added model.
Columns 4-7 add peer controls to the value-added model in an attempt to control
for the peer effects. Theoretically, even if peer effects exist, if the value-added model
controls for these peer effects, there should be no relationship between estimated
teacher value-added and whether the teacher is exposed to an ED transfer student.16
The fourth column adds classroom level controls (mean lagged test score; percent
African American, White, Hispanic, male, economically disadvantaged and ED). The
fifth through seventh columns modify the functional form through which we control
for the ED transfer student peer effect. Column 5 uses percent ED transfer in class,
column 6 controls for an indicator for whether there is an ED transfer student in the
class, and column 7 controls for whether there is an ED transfer student in the grade.
Since column 7 controls for the exact functional form that we use to relate ED
transfer student to value-added, we expect that there should be no relationship
between ED transfer student exposure and value-added using this specification.
Panel A of Table 7 shows that in years when a math teacher is exposed to an ED
transfer student, their value-added is lower by 2.1% of a standard deviation. As we
add individual controls to the specification that calculates value-added, the effect on
teacher-by-year value-added falls slightly. This estimate falls slightly more when
16
As we have documented, classroom peer controls are endogenous. Nonetheless
estimating models with theses peer controls is instructive as it allows us to observe their
ability to account for the peer effect at hand.
22
adding controls for peer effects, but it does not diminish substantially until we add a
control for whether there is an ED transfer student in the grade. The results from
Panel A, column 5 and 6 are particularly noteworthy: teachers have lower estimated
value-added in years that there is an ED transfer student in their grade, even when the
value-added model controls for whether they are teaching an ED transfer student.
For reading teachers, the magnitudes are smaller, though the basic results are
similar. The one important difference is that for reading teachers, the value-added
model that controls for an ED transfer student in the class yields value-added
estimates that are unrelated to whether the teacher has an ED transfer student in the
grade.
These results raise several points. First teachers who are exposed to an ED
transfer student in a given year have lower value-added measures compared to other
years, even conditional on student characteristics. In addition, the functional form of
the peer effect matters. The peer effects we document are non-linear. Consequently
controlling for average peer characteristics does little to remove the effect on teacher
value-added. Importantly, even though the specific issue we document is solved in
the final specification from Table 7 (column 7), even the value-added estimates from
column 7 are likely biased by other types of peer effects.
VII.
Conclusion
The landscape for high stakes teacher evaluation policies has changed
dramatically over the last five years. Since 2009, 25 states and the District of
Columbia (D.C.) have adopted policies that require teacher evaluation to include
objective measures of student achievement. This is a 173% increase. More strikingly,
23
the number of states which require student growth to be the major criterion in teacher
evaluation increased by 500%, going from 4 to 20 states including D.C. in 2013
(Doherty and Jacobs 2013). As evaluations of teachers continue to rely more heavily
on teacher value-added estimates, it is necessary that the science is transparent about
the limitations and strengths of these estimates.
In this paper we have estimated the effects of being in a grade with an
emotionally disabled transfer student on student achievement outcomes, and on
teacher value-added. We find that student achievement suffers in grades with an
emotionally disabled transfer student. However, we do not find heterogeneous effects
by gender, race, or socioeconomic status. Further our results convey that teachers,
who teach in a grade with an emotionally disabled transfer student, have lower
teacher value-added, relative to years when they are not teaching grades with an
emotionally disabled transfer student. More strikingly, as we control for average peer
controls, including the percent of emotionally disabled students, the result does not
dissipate.
Our research raises several points about the role of peer effects on student
achievement, and teacher value-added. First, while it is important to document this
“disruption” peer effect, further research is required to unpack the mechanisms
through which the peer effect operates on student outcomes. As far as we know Lavy,
Paserman, and Schlosser (2011) and Lavy and Schlosser (2011), are the only papers
to explore empirically, what they call the “black box” of peer effects. Second, while
this paper documents one specific type of peer effect and functional form that
negatively impact teacher value-added, this point follows more general to other peer
24
effects. Since teacher-value-added is calculated based on unexplained classroom
performance, controlling appropriately for classroom peer effects is an important
consideration.
25
References
Carrell, Scott and Mark L. Hoekstra, 2010. “Externalities in the Classroom: How
Children Exposed to Domestic Violence Affect Everyone’s Kids,” American
Economic Journal: Applied Economics, 2(1): 211-228.
Doherty, Kathryn M. and Sandi Jacobs, The National Council on Teacher Quality. 2013.
State of States 2013 Connect the Dots: Using Evaluations of Teacher
Effectiveness to Inform Policy and Practice (Research Report). Retrieved from
http://www.nctq.org/dmsView/State_of_the_States_2013_Using_Teacher_Evalua
tions_NCTQ_Report
Figlio, David N. 2007. “Boys Named Sue: Disruptive Children and their Peers.”
Education Finance and Policy, 2(4): 376-394.
Fletcher, Jason M. 2009a. “The Effects of Inclusion on Classmates of Students with
Special Needs: The Case of Serious Emotional Problems.” Education Finance
and Policy, 4(3): 278-299.
Fletcher, Jason M. 2009b. “Spillover Effects of Inclusion of Classmates with Emotional
Problems on Test Scores in Early Elementary School.” Journal of Policy Analysis
and Management, 29(1): 69-83.
Friesen, Jane, Ross Hickey, and Brian Krauth. 2010. “Disabled Peers and Academic
Achievement.” Education Finance and Policy, 5(3): 317-348.
Hoxby, Caroline M. 2000. “Peer Effects in the Classroom: Learning from Gender and
Race Variation.” NBER Working Paper No. 7867.
Hoxby, Caroline M. and Gretchen Weingarth. 2005. “Taking Race Out of the Equation:
School Reassignment and the Structure of Peer Effects.” Mimeo, Harvard
University.
Imberman, Scott A., Adriana D. Kugler, and Bruce I. Sacerdote. 2012. "Katrina's
Children: Evidence on the Structure of Peer Effects from Hurricane Evacuees."
American Economic Review, 102(5): 2048-2082.
Isenberg, Eric and Elias Walsh. 2013. Measuring Teacher Value Added in DC, 20122013 School Year (Report No. 06038.502).Washington, DC: Mathematica Policy
Research.
Kane, Thomas J., and Douglas O. Staiger. 2008. “Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation.” National Bureau of Economic
Research.” NBER Working Papers 14607.
Lavy, Victor, and Analia Schlosser. 2011. "Mechanisms and Impacts of Gender Peer
Effects at School." American Economic Journal: Applied Economics, 3(2): 1-33.
26
Lavy, Victor, Daniele M. Paserman, and Analia Schlosser. 2011. “Inside the Black Box
of Ability Peer Effects: Evidence from Variation in the Proportion of Low
Achievers in the Classroom.” The Economic Journal, 122(559): 208-237.
Lazear, Edward. 2001. “Education Production,” Quarterly Journal of Economics, 116(3):
777-803.
Manski, Charles. 1993. “Identification of Endogenous Social Effects: The Reflection
Problem.” Review of Economic Studies, 60(3): 531-542.
Moffitt, Robert. “Policy Interventions, Low-Level Equilibrium, and Social Interactions,”
in S. Durlauf and P. Young (Eds.), Social Dynamics. Cambridge, MA: MIT Press,
2001.
Neidell Matthew, and Jane Waldfogel. 2010. “Cognitive and Noncognitive Peer Effects
in Early Education.” The Review of Economics and Statistics, 92(3): 562-576.
27
Table 1: Summary Statistics of Students in Analysis
All Students
Male
African American
Hispanic
White
Limited English proficiency
Economically disadvantaged
Reading score
Math score
Reading pretest
Math pretest
Class size
Teacher experience
Observations
mean
sd
0.50
0.50
0.26
0.44
0.12
0.32
0.54
0.50
0.07
0.25
0.51
0.50
0.00
1.00
0.00
1.00
0.04
0.97
0.04
0.97
23.28
8.34
11.50
9.17
1311485
Emotionally
Disabled Students
mean
0.82
0.48
0.02
0.43
0.01
0.76
-0.76
-0.83
-0.70
-0.75
16.92
11.52
sd
0.39
0.50
0.15
0.49
0.11
0.43
1.02
0.99
1.00
0.95
11.04
9.02
3902
Emotionally
Disabled Transfer
Students
mean
sd
0.84
0.37
0.54
0.5
0.03
0.16
0.35
0.48
0.02
0.13
0.81
0.39
-0.87
0.97
-0.98
0.93
-0.8
0.96
-0.85
0.89
15.34
11.44
10.99
8.79
1128
28
Table 2: Summary Statistics of Classrooms With and Without an Emotionally Disabled Transfer Student
Percent male
Percent African American
Percent Hispanic
Percent White
Percent limited English proficiency
Percent economically disadvantaged
Teacher experience
Class size
Premath score
Preread score
Math score
Reading score
Observations
Classroom without
EDTS
mean
sd
0.51
0.14
0.28
0.27
0.10
0.14
0.55
0.31
0.07
0.12
0.53
0.28
11.57
9.18
22.99
11.12
-0.03
0.61
-0.08
0.64
-0.05
0.63
-0.10
0.64
51155
Classroom with
EDTS
mean
sd
0.57
0.21
0.38
0.32
0.09
0.13
0.46
0.33
0.06
0.11
0.64
0.28
10.93
8.76
20.28
11.60
-0.31
0.70
-0.32
0.72
-0.38
0.71
-0.38
0.72
642
29
Table 3: Estimates of Exposure to an Emotionally Disabled Transfer Student on Math and Reading Test Scores
(1)
Math
(2)
Math
(3)
Math
(4)
Math
(5)
Reading
(6)
Reading
(7)
Reading
(8)
Reading
Grade has an EDTS
-0.0344***
(0.0064)
-0.0191***
(0.0056)
-0.0156**
(0.0062)
-0.0161***
(0.0061)
-0.0241***
(0.0044)
-0.0097**
(0.0039)
-0.0061
(0.0039)
-0.0063
(0.0039)
Student controls
Grade controls
School FE
School-by-year FE
Yes
No
No
No
Yes
No
Yes
No
Yes
No
No
Yes
Yes
Yes
No
Yes
Yes
No
No
No
Yes
No
Yes
No
Yes
No
No
Yes
Yes
Yes
No
Yes
Observations
1164357
1164357
1164357
1164357
1156750
1156750
1156750 1156750
Notes: All models include student controls (male, African American, Hispanic, White, limited English proficiency, economically
disadvantaged, and the subject specific pretest score), grade peer controls (percent male, African American, Hispanic, White, limited
English proficiency, economically disadvantaged, and the average subject specific pretest score), and grade-by-year dummies. Standard
errors clustered at the school-by-year-by-grade level are shown in parentheses.
* p<0.10 ** p<0.05 *** p<0.01
30
Table 4: Mechanisms Driving Spillover Estimates of Exposure to Emotional Disabled Transfer Student
on Test Scores
(1)
Math
Grade has an EDTS
-0.0161***
(0.0061)
Grade has a low performing transfer student
Student controls
Grade controls
School-by-year FE
(2)
Math
(3)
Reading
-0.0063
(0.0039)
-0.0089**
(0.0044)
Yes
Yes
Yes
(4)
Reading
Yes
Yes
Yes
-0.0059**
(0.0027)
Yes
Yes
Yes
Yes
Yes
Yes
Observations
1164357
1164357 1156750 1156750
Notes: All models include student controls (male, African American, Hispanic, White, limited English
proficiency, economically disadvantaged, and the subject specific pretest score), grade peer controls
(percent male, African American, Hispanic, White, limited English proficiency, economically
disadvantaged, and the average subject specific pretest score), and grade-by-year dummies. Standard
errors clustered at the school-by-year-by-grade level are shown in parentheses.
* p<0.10 ** p<0.05 *** p<0.01
31
Table 5: Predicting Probability of Exposure to an Emotionally Disabled Transfer
Student
Estimated at
Class
Estimated at
Grade
(1)
(2)
Male
0.0006***
(0.0001)
-0.0001
(0.0002)
African American
-0.0003
(0.0004)
-0.0006
(0.0005)
Hispanic
-0.0009*
(0.0005)
0.0004
(0.0006)
White
-0.0002
(0.0004)
-0.0003
(0.0004)
Previous emotional disability
0.0747***
(0.0073)
-0.0035
(0.0030)
Math pretest
-0.0009***
(0.0002)
0.0000
(0.0003)
Reading pretest
-0.0004**
-0.0001
Yes
Yes
School-by-year FE
Observations
1154588
1154588
Notes: All models include grade-by-year dummies. The samples refer to students in
math classes. Standard errors clustered at the school-by-year-by-grade level are shown
in parentheses.
* p<0.10 ** p<0.05 *** p<0.01
32
Table 6: Estimates of Grade has EDTS by Gender, Race, and Economically Disadvantaged Status
Panel A. Math scores
Grade has an EDTS
(1)
(2)
(3)
(4)
(5)
Male
Female
African American
White
Hispanic
-0.0156**
(0.0065)
-0.0163**
(0.0065)
-0.0170**
(0.0074)
-0.0180**
(0.0077)
-0.0210*
(0.0111)
-0.0120*
(0.0066)
585576
578781
306918
653526
115954
582206
-0.0089*
(0.0048)
-0.0029
(0.0044)
-0.0075
(0.0055)
-0.0072
(0.0048)
-0.0087
(0.0084)
-0.0025
(0.0048)
579653
577097
305139
650060
114232
576508
Observations
Panel B. Reading score
Grade has an EDTS
Observations
(6)
Economically
Disadvantaged
Student controls
Yes
Yes
Yes
Yes
Yes
Yes
Grade controls
Yes
Yes
Yes
Yes
Yes
Yes
School-by-year
Yes
Yes
Yes
Yes
Yes
Yes
Notes: All models include student controls (limited English proficiency and the subject specific pretest score), grade peer controls (percent male, African
American, Hispanic, White, limited English proficiency, economically disadvantaged, and the average subject specific pretest score), and grade-by-year
dummies. In addition models in columns 1-2 control for student's race (African American, White, Hispanic) and economic disadvantage status; columns 3-5
control for student's gender (male) and economic disadvantage status; and column 6 has controls for student's gender (male) and race (African American, White,
Hispanic). Standard errors clustered at the school-by-year-by-grade level are shown in parentheses.
* p<0.10 ** p<0.05 *** p<0.01
33
Table 7: Estimates of Exposure to an Emotionally Disabled Transfer Student on Teacher Value-Added
Panel A. Math Teachers
Teaching in a grade with EDTS
Observations
Panel B. Reading Teachers
Teaching in a grade with EDTS
(1)
(2)
(3)
(4)
(5)
(6)
(7)
-0.0210***
-0.0051
-0.0191***
-0.0051
-0.0176***
-0.0051
-0.0163***
-0.0049
-0.0166***
-0.0049
-0.0162***
-0.0049
-0.0061
-0.0049
50382
50382
50382
49410
49410
49410
49410
-0.0102**
-0.0044
-0.0091**
-0.0044
-0.0086**
-0.0044
-0.0078*
-0.0042
-0.0075*
-0.0042
-0.004
-0.0042
0.0032
-0.0042
Observations
56824
56824
56824
54899
54899
54899
54899
Controls in calculation of teacher-by-year VA
Student controls
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Indicator for emotionally disabled
No
Yes
No
No
No
No
No
Indicator for EDTS
No
No
Yes
Yes
Yes
Yes
Yes
Classroom controls
No
No
No
Yes
Yes
Yes
Yes
% Emotionally disabled in classroom
No
No
No
Yes
No
No
No
% EDTS in classroom
No
No
No
No
Yes
No
No
Indicator for classroom has EDTS
No
No
No
No
No
Yes
No
Indicator for grade has EDTS
No
No
No
No
No
No
Yes
Notes: All columns include teacher-by-school fixed effects, and year dummies. Under “Control in calculation of teacher-by-year VA”, student controls
refer to indicators for male, African American, Hispanic, White, limited English proficiency, economically disadvantaged, and the subject specific
pretest score, and classroom controls refer to percent male, African American, Hispanic, White, limited English proficiency, economically
disadvantaged, and the average subject specific pretest score. Standard errors clustered at the school-by-year-by-grade level are shown in parentheses.
* p<0.10 ** p<0.05 *** p<0.01
34
35