October, 2012
Under Pressure:
Job Security, Resource Allocation, and
Productivity in Schools under NCLB
Randall Reback
Barnard College, Columbia University
Jonah E. Rockoff
Columbia Business School
Heather L. Schwartz
RAND
Excellent research assistant provided by Tamara Lalovic Cox & Elizabeth Davidson
Funding provided by the Spencer Foundation and the U.S. Institute of Education Sciences
Policy Background: No Child Left Behind
●
●
●
●
First took effect during spring of 2003
Requires states to adopt school accountability systems that
determine whether public schools satisfy Adequate Yearly
Progress (AYP)
Schools’ AYP status based on
– % of students taking and demonstrating proficiency on
statewide exams in math and reading
– both overall and student subgroups pass rates
– state-specific exams, standards, and rules
Consequences of failing AYP
– Escalating sanctions, including
• Inter-district public school choice
• Funds redirected to after-school tutoring for students from
low income families
– States are also required to publish annual school report cards,
so schools’ AYP status can also affect school prestige and local
property values.
1
Overview of Our Study
●
Assemble NCLB related data and outcomes for all
schools nationwide for first 2 years of NCLB
●
Find schools that were at substantial risk of failing
AYP and therefore faced pressure under NCLB
– Considerable variation across states
●
Use several external data sets to investigate the
impact of NCLB pressure on teachers and students
2
Preview of Main Results
●
Teacher-level results: Accountability pressure
– increases teachers’ concerns about their job security and
decreases their expected career length
– decreases frequency of instruction in low-stakes subjects
(e.g., science) and decreases time on whole-class instruction
●
Student-level results In schools facing the strongest short term
accountability incentives, students…
• score higher on low-stakes readings exams
• perform at least as well on low-stakes math and science
exams
• do not experience any negative effects in terms of their
enjoyment of math & reading or their anxiety about testing
3
NCLB Policy Variation
Interaction of four features significantly influences the
likelihood that a school fails AYP:
(1) state rules for the numerical significance of student
subgroups
(2) within-school heterogeneity, which influences how
many student subgroups are numerically significant
(3) the generosity of the state’s confidence intervals
(4) the generosity of the state’s safe harbor provisions
Several other idiosyncratic policies also matter
Methodology (Part 1)
●
Predicting which schools were on the margin of
making AYP
– Use 2002 data on test scores and demographics (prepolicy) to predict probability of making AYP in 2003 and
2004 using separate probit regressions for each state
●
Define schools as on the AYP margin if…
…at least one group moderate probability of passing
…no group has a very low probability of passing
●
Define schools as below the AYP margin if any group
has a very low probability of passing
5
Difference-in-differences, An Example
●
Take two pairs of schools, from NJ and PA
School name
Hamilton School
State
PA
Title I
eligible
eligible
yes
Richland Elementary
PA
no
18%
18%
94%
94%
1%
1%
Bradley Elementary
NJ
yes
87%
87%
4%
4%
Upper Pittsgrove
NJ
no
16%
16%
93%
93%
●
Percent
Percent Percent
Percent Percent
Percent Percent
Percent
poor
white
black
poor
white
black Hispanic
Hispanic
89%
5%
83%
10%
89%
5%
83%
10%
Size
Size
454
454
On margin of
failing AYP?
no
3%
3%
472
472
no
81%
81%
15%
15%
418
418
yes
4%
4%
3%
3%
419
419
no
Even though pairs are observably similar,
differences in state rules create arguably
exogenous variation in NCLB pressure
6
Difference-in-differences
Regressions control for
– state fixed effects
– various school-,child-,household- level
controls
• including student-level and school-level
pre-NCLB test performance
– simulated % of states where school would be
on AYP margin
– simulated % of states would be below margin
7
Evidence from RAND-NCLB Survey of Math Teachers
I search for more effective teaching
methods
I spend more time teaching content
I focus more on the state standards
I focus more on topics emphasized
in the state test
I spend more time teaching general
test-taking strategies
I focus more effort on students who
are close to proficient
0
10
20
30
40
50
60
70
80
% Agreement Among Math Teachers
High (N=1074)
Marginal (N=224)
90
100
Low (N=19)
8
Evidence from RAND-NCLB Survey of Principals
My school’s AYP status accurately reflects
the overall performance of our school
My school can attain the AYP targets for
the next five years
My school can attain the AYP targets for
2003-04
Encouraged or required teachers to spend
more time on tested subjects and less time
on other subjects
Distributed released copies of the state test
or test items
Distributed commercial test preparation
materials
0
10
20
30
40
50
60
70
80
90
100
% Agreement Among Principals
Marginal (N=21)
High (N=104)
9
Data Used in Second Stage Analysis
●
Early Childhood Longitudinal Survey (ECLS):
students in the kindergarten cohort of 1998-99,
followed up at 1st, 3rd, 5th (and 8th) grade
– Nationally representative, but not from all 50 states
– Rich data set with low stakes exams. Used adaptive
testing to capture wide range of abilities.
●
●
SASS 2003-04: nationally representative sample of
schools and teachers within schools
Restricted-use data allow use to link ECLS/SASS
observations with variables on NCLB pressure
10
Effects of NCLB on Teachers
Job Security
Concerns for
Relatively
Inexperienced
Teachers
Work Hours in
a Typical Week Work Hours in a
Plan to Teach
for
Typical Week for
until
“Generalist”
“Specialist”
Retirement
Teachers
Teachers
Main Sample: NCLB Sample Wave
On AYP Margin
0.101*
(0.057)
-0.057*
(0.033)
-1.84**
(0.84)
4.15**
(1.81)
Below AYP
Margin
0.170**
(0.076)
-0.154***
(0.048)
-2.65**
(1.13)
4.89**
(2.31)
0.28
(0.66)
0.79
(0.86)
0.63
(2.16)
-0.40
(2.73)
Falsification Sample: Pre-NCLB Sample Wave
On AYP Margin
-.071
0.039
(.068)
(0.036)
Below AYP
-.025
0.059
Margin
(.093)
(0.049)
11
Effects of School being on AYP Margin on
Student Learning
Reading Score
0.08 **
(0.04)
Reading
Enjoyment
0.03
(0.07)
Math Score
Science Score
0.00
(0.04)
0.04
(0.03)
Anxiety
About
Testing
-.09***
(.04)
Math Enjoyment
0.13 *
(0.08)
12
ECLS: Other Findings
●
Effects are not very different for
– Students in the subgroups under pressure
– “Bubble students”
– Students from low-income families
●
Stronger positive effects for math score gains in
states without strong accountability prior to
NCLB (related to Dee and Jacob, 2011)
13
Conclusions
●
States vary widely in rates of making AYP
– Cross-state variation in student academic
aptitude or in exam difficulty explains relatively
little of this variation
●
Short term NCLB pressure…
– Threatens perceived job security of teachers
– Influences teachers’ time use
– Has positive net effects on average student test
score growth on low-stakes reading exams
– Does not have any short-term negative effects
on outcomes for the average student!
14
No Data Left Behind!
Our NCLB data is publicly available from our
“No Data Left Behind” website:
http://www7.gsb.columbia.edu/nclb/
15
16
First stage: Pr(Made AYP in ’03 & ’04)
●
●
1
in 5 marginal;
1 in 10groups
low probability
Large
variation across
in
contributions
to risk
AYP AYP
failures
Large
differences
in of
actual
outcomes
Panel
Outcomes
Panel B:
B: Subgroup
Subgroup Outcomes
Outcomes
Conditional
on Numerical
Significance
Conditional
onNumerical
Numerical
Significance
Conditional
on
Significance
Numerically Predicted
Predicted
Marginal
Chance
Predicted
Chance
Numerically
Numerically
Predicted
Marginal
Chance
Predicted
Low Low
Chance
Marginal
Chance
Table 3: First Stage Predictions
of
AYP
Outcomes
Significant
Subgroup Math
Reading MathMath Reading
Reading
Significant
Subgroup
Significant
Subgroup
MathMath Reading
Reading
Overall
Population
9.3%
2.6%
Population
92.5%92.5%
7.5%7.5%
9.3%Below
2.2%2.2%
2.6%
PanelSchool
A: School-wide
Outcomes
On7.5%
Above
92.5%
9.3%
Actually made AYP in
and in
'04
52.0% 52.7%
52.7% 11.0%
11.0%
in '03
subject
in '03
'03 and
and '04
'04
52.0%
52.7%
8.6%8.6%
subject
52.0%
Percent
of
Schools
22.9%
10.1%
67.0%
Economically Disadvantaged
14.9% 18.5%
18.5% 3.7%3.7%
Disadvantaged
61.5%61.5%
14.9%
18.5%
4.6%4.6%
61.5%
14.9%
Actually
AYP in
'03
and in
'04
54.2% 53.7%
53.7%
13.1%
inMade
subject
in '03
'03 and
and
'04 and 2004
54.2%
53.7%
subject
'04
54.2%
Percentmade
Actually
AYP
2003
36.4%
6.9% 12.6%12.6% 13.1%
86.0%
Limited English Proficient
19.4% 37.6%
37.6% 4.9%4.9%
10.5%
Proficient
24.0%24.0%
19.4%
37.6%
10.5%
24.0%
19.4%
Actually made AYP in
and in
'04
58.3% 49.8%
49.8% 14.2%
14.2% 19.2%
19.2%
in '03
subject
in '03
'03 and
and '04
'04
58.3%
49.8%
subject
58.3%
Disabled
31.0% 33.3%
33.3% 14.6%
14.6% 17.6%
17.6%
32.0%32.0%
31.0%
33.3%
32.0%
31.0%
Actually made AYP in
and in
'04
51.0% 51.7%
51.7% 14.5%
14.5% 12.7%
12.7%
in '03
subject
in '03
'03 and
and '04
'04
51.0%
51.7%
subject
51.0%
White
71.8%71.8%
1.4%1.4%
0.9%0.9%
0.1%0.1%
0.0%0.0%
71.8%
1.4%
0.9%
Actually made AYP in
and in
'04
56.7% 61.4%
61.4% 15.8%
15.8% 33.3%
33.3%
in '03
subject
in '03
'03 and
and '04
'04
56.7%
61.4%
subject
56.7%
Black
27.4% 24.9%
24.9% 9.8%9.8%
33.9%33.9%
27.4%
24.9%
7.8%7.8%
33.9%
27.4%
Actually made AYP in
and in
'04
51.6% 53.5%
53.5% 16.5%
16.5% 14.6%
14.6%
in '03
subject
in '03
'03 and
and '04
'04
51.6%
53.5%
subject
51.6%
Hispanic
11.5% 19.9%
19.9% 1.3%1.3%
30.6%30.6%
11.5%
19.9%
2.7%2.7%
30.6%
11.5%
Actually made AYP in
and in
'04
55.8% 54.7%
54.7% 12.3%
12.3% 15.8%
15.8%
in '03
subject
in '03
'03 and
and '04
'04
55.8%
54.7%
subject
55.8%
Asian/Pacific Islander/Filipino
11.2% 6.6%6.6%
Islander/Filipino
11.7%11.7%
5.6%5.6%
11.2%
9.1%9.1%
11.7%
5.6%
11.2%
Actually made AYP in
and in
'04
39.2% 47.3%
47.3% 43.6%
43.6% 11.5%
11.5%
in '03
subject
in '03
'03 and
and '04
'04
39.2%
47.3%
subject
39.2%
Native American
11.0% 12.9%
12.9% 11.3%
11.3% 10.4%
10.4%
5.2%5.2%
11.0%
12.9%
5.2%
11.0%
Actually made AYP in
and in
'04
53.23% 44.74%
44.74% 8.06%
8.06% 14.75%
14.75%
in '03
subject
in '03
'03 and
and '04
'04
53.23%
44.74%
subject
53.23%
17
Which Children Have Been Left Behind?
Total Number of Schools
Average Enrollment
Student/Teacher Ratio
Percent of Schools…
School Title I Eligible
Located in City
Located in Urban Fringe
Located in Town or Rural Area
Serving Primary Grades
Serving Middle Grades
Serving High School Grades
Ungraded/Other
Percent of Students…
Eligible for Free/Reduced Lunch
White
Black
Hispanic
Asian
Unweighted
Failed AYP Made AYP
19,483
65,332
727
500
17.4
16.1
Weighted by Enrollment
Failed AYP Made AYP
19,483
65,332
1,216
800
18.4
17.2
39.9%
36.5%
31.9%
31.5%
36.0%
27.4%
27.9%
8.5%
29.4%
23.1%
33.3%
43.5%
64.1%
15.5%
17.1%
3.2%
34.8%
39.2%
37.8%
23.0%
27.1%
29.4%
39.1%
4.4%
27.6%
27.7%
41.1%
31.2%
56.3%
17.0%
24.5%
2.2%
54.2%
46.5%
25.6%
20.8%
3.4%
39.9%
67.7%
13.2%
13.6%
3.4%
49.3%
46.2%
24.1%
23.7%
4.3%
37.0%
63.6%
14.1%
16.3%
4.5%
Note: Includes data from 50 states. Data on school and student characteristics taken from the Common
Core of Data, 2003-04. For schools in T ennessee, data on student ethnicity taken from 1998-99 and data
on free/reduced price lunch eligibility is unavailable.
18