Does School Choice Work?
Effects on Integration
and Student Achievement
Julian R. Betts, Lorien A. Rice,
Andrew C. Zau, Y. Emily Tang, and
Cory R. Koedel
PPIC
1
Overview of Entire Report
 Entire book is downloadable from
www.ppic.org
 Today will focus on lottery-based
assessment of impact of choice on math and
reading achievement
 Will briefly highlight other aspects of report:
• “Who chooses”
• Effects on integration
• Non-lottery study of charters and
achievement
PPIC
2
Specific to this School Choice Study (I)
 Many of the previous studies of effect of choice on
achievement lack convincing statistical
identification
– Compare switchers to those students left behind
– Compare switchers to local students at current
school
 This study uses lottery data to study the three main
choice programs. Compares lottery winners to
losers
– Student fixed effects models for charter schools
only that compare individual student to herself in
PPIC
years before switch
3
Choice Programs in San Diego
 Busing available
– VEEP (Voluntary Ethnic Enrollment Program)
– Magnets
• Both are legacies of 30-year-old court
desegregation order
– NCLB-ordered busing from “failing schools”
• Not studied: minor program, tiny enrollments
 No busing
– Choice (State-mandated open enrollment program)
– Charter Schools (1 charter pays district for busing)
PPIC
4
Between 2001 and 2003, School Choice
Enrollment Share Rose from 25% to 28%
Traditional (non-choice)
Charters
NCLB Choice
Magnet (Non-resident)
Choice (Open Enrollment)
VEEP
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
2003
2001
PPIC
5
Outline
 Introduction
 Who Chooses to Leave and Why?
 The Effect of School Choice on Integration
 The Effect of Choice on Achievement
 Policy Implications
PPIC
6
Who Exercises Choice Personal Characteristics
 Non-whites
– Sometimes, but not always, more likely to apply
 Blacks
– Always more likely to apply
• Twice the average probability of applying to VEEP high
schools
 English Learners
– Generally less likely to apply
 Are students with higher test scores, GPA or higher parental
education being skimmed off?
– Mixed evidence, weak effects
PPIC
7
Who Exercises Choice –
School Characteristics
 Weak evidence that students base application
decisions on academic characteristics of local and
option schools
 Distance is typically a barrier
– At high school level, distance is strongest
deterrent in Choice (open enrollment) program,
which provides no busing
PPIC
8
Outline
 Introduction
 Who Chooses to Leave and Why?
 The Effect of School Choice on Integration
 The Effect of Choice on Achievement
 Policy Implications
PPIC
9
Integration Results
 Overall, choice programs…
– Integrate along race/ethnicity and parental education lines
– Segregate slightly along test score and English Learner (EL)
lines
 Decreasing Rank of integrating efficacy
– VEEP, Magnets, Choice (no busing)
• Choice actually lowers integration along race, parental
education, test score and language status lines
 Limited supply of places sharply reduces actual integration
PPIC
10
Outline
 Introduction
 Who Chooses to Leave and Why?
 The Effect of School Choice on Integration
 The Effect of Choice on Achievement
 Policy Implications
PPIC
11
Overview
 Lotteries represent a true quasi-experimental
way of measuring impact of choice on
choosers because selectivity bias is removed
 Model reading and math test scores one, two
and three years after the admissions lotteries
for Fall 2001
– Two measures of math achievement per
year, and three measures of reading
achievement per year
– Three types of school choice
PPIC
12
How Lotteries Work in San Diego
 Each application is assigned a random number
 Applications to a given school and grade are
sorted into priority groups (e.g. top priority goes
to siblings), so potentially multiple lotteries per
school/grade.
– Table C.1 lists priority groups
 Central administration admits in descending
order of priority. Typically one priority group per
grade undergoes a lottery
PPIC
13
Preliminary Analysis for the Fall 2001
Admissions Lotteries
 First, check which potential lotteries are
“true” lotteries, in sense that proportion p
winning admission obeys 0 < p < 1
– Generally a greater proportion of lotteries
are “true” in grades above grade 3
• Elementary VEEP in particular: very
small sample of students in true lotteries
relative to other cases
• Table C.2
PPIC
14
Preliminary Analysis for the Fall 2001
Admissions Lotteries (II)
 Second, for students in the subsample of
lotteries that are “true”, need to check that
lotteries are “fair”.
– Should be no statistically significant
difference in initial (spring 2001) test
scores between lottery winners and losers
– A regression of initial test scores on lottery
dummies and a dummy for “winning”
showed the latter to be insignificant
PPIC
15
Preliminary Analysis for the Fall 2001
Admissions Lotteries (III)
 Additional tests that lotteries are “fair”.
– Repeated for 178 regression samples and
null is rejected at roughly the size of the
test, as expected (e.g. 0.6% of time is
rejection at the 1% level)
– Table C.3
PPIC
16
Lottery Regression Models
 Will focus on models of the “intent to treat”.
– Let J denote number of true lotteries.
– We will model the test score for student i
in year t, where t is one of the post-lottery
years 2002, 2003 and 2004. This student
applies to lottery j, and in year t attends
school s, so his score is denoted by Sijst.
PPIC
17
Lottery Regression Models
 Simplest model:
Sijst    j  WIN ijt   st   ijst 
J

j 1
 Model includes:
– a set of dummy variables αj for the lotteries
– a dummy variable WINijt and corresponding
coefficient β indicating whether the student
i, whose test score is modeled in year t,
won lottery j, and
– an error term consisting of an error
component for school s in year t, ηst, and a
white noise error term εijst:
PPIC
18
Robustness Check: Estimate 4
Specifications
 In addition to the previous model (1), add:
– (2) Same as (1) but add spring 2001 test
score
– (3) Same as (2) but add square of spring
2001 test score
– (4) Same as (3) but add personal
characteristics
– (5) Same as (4) but add classroom
characteristics (teacher qualifications etc.)
to see if we can explain any of the
differences accruing to lottery winners
PPIC
19
Preview of Results
 Few differences across specifications
– Will focus on model (2) which adds lagged
test score to improve precision
– Results for all four specifications
appear in Appendix C
PPIC
20
Effects of Winning a VEEP Lottery: CST Reading
Achievement
VEEP Re ading
3500
1.2
1
3000
0.8
2500
0.6
Sample 2000
Size
1500
0.4
0.2
0
Proportion
of a
Standard
Deviation
-0.2
1000
-0.4
500
-0.6
0
-0.8
0102
0203
0304
0102
Elementary
0203
0304
Middle
0102
0203
High
0304
0102
0203
0304
All Grades
PPIC
21
Effects of Winning a Magnet Lottery: CST
Reading Achievement
M agne t Re ading
3500
1.2
1
3000
0.8
2500
0.6
0.4 Proportion
of a
0.2
Standard
0
Deviation
-0.2
Sample 2000
Size
1500
1000
-0.4
500
-0.6
0
-0.8
0102
0203
0304
Elementary
0102
0203
0304
Middle
0102
0203
High
0304
0102
0203
0304
All Grades
PPIC
22
Effects of Winning an Open Enrollment (Choice)
Lottery: CST Reading Achievement
Choice Re ading
3500
1.2
1
3000
0.8
2500
Sample
Size
0.6
0.4
2000
0.2
1500
0
Proportion
of a
Standard
Deviation
-0.2
1000
-0.4
500
-0.6
0
-0.8
0102
0203
0304
Elementary
0102
0203
0304
Middle
0102
0203
High
0304
0102
0203
0304
All Grades
PPIC
23
Effects of Winning a VEEP Lottery: CST Math
Achievement
VEEP M ath
3500
0.8
3000
0.6
0.4
2500
0.2
Proportion
of a
0
Standard
-0.2 Deviation
Sample 2000
Size
1500
1000
-0.4
500
-0.6
0
-0.8
0102
0203
0304
Elementary
0102
0203
0304
Middle
0102
0203
High
0304
0102
0203
0304
All Grades
PPIC
24
Effects of Winning a Magnet Lottery: CST Math
Achievement
Magnet Math
3500
0.8
3000
0.6
0.4
2500
Sample
Size
0.2 Proportion
2000
of a
Standard
-0.2 Deviation
0
1500
1000
-0.4
500
-0.6
0
-0.8
0102
0203
0304
Elementary
0102
0203
0304
Middle
0102
0203
High
0304
0102
0203
0304
All Grades
PPIC
25
Effects of Winning an Open Enrollment (Choice)
Lottery: CST Math Achievement
Choice Math
3500
0.8
3000
0.6
0.4
2500
Sample
Size
0.2 Proportion
2000
of a
Standard
-0.2 Deviation
0
1500
1000
-0.4
500
-0.6
0
-0.8
0102
0203
0304
0102
Elementary
0203
0304
Middle
0102
0203
High
0304
0102
0203
0304
All Grades
PPIC
26
Results of Choice on Achievement
 In general, winning a lottery has no statistically
significant effect on reading or math achievement
one to three years after
 Important exceptions
– In some cases lottery winners fare slightly worse
in first year, but recover to identical achievement
– Students winning lotteries for magnet high
schools seem to do better in math 2-3 years later
• Effect sizes are meaningful: 0.18 and 0.23 in
years 2 and 3
– But only for CST and not CAT/6
PPIC
27
Charter Schools
Effect on Achievement Gains
 Relative to regular public schools
– Charters do not systematically under- or
overperform
 A few exceptions. Most interestingly…
– Often, elementary startup charters underperform
in math and reading for 1 to 3 years, then
perform at same level
PPIC
28
Outline
 Introduction
 Who Chooses to Leave and Why?
 The Effect of School Choice on Integration
 The Effect of Choice on Achievement
 Policy Implications
PPIC
29
Conflicting Achievement Results
 Choice affected some achievement results, but the more
general finding was of no effect even three years after winning
a lottery
 Yet, school choice programs extremely popular in San Diego
with 28% student share by 2003-2004
 Possible explanations:
– Parents care about math and reading but lack information
– Parents consider factors other than math and reading, such
as socioeconomic climate, other achievement outcomes
PPIC
30
Conclusions and National Implications
 NCLB mandates that students in “failing” schools
be given option to be bused to other schools
 Question: Is the best way to help students at
“failing” schools to move students from the
school, or to reform the schools in question?
 In San Diego, the NCLB Choice program
piggybacked on VEEP and magnets
– Effects likely similar, and small
PPIC
31
Conclusions and National Implications
 Additional aid to students in low-performing
schools may be a better solution
– Betts, Zau and King (PPIC, 2005) report some
evidence that “within-school” reforms can boost
achievement
 Choice may provide better option in certain
situations
 But presumption that choice automatically confers
large achievement gains is probably incorrect
PPIC
32
Does School Choice Work?
Effects
on Integration
and Student Achievement
Julian R. Betts, Lorien A. Rice,
Andrew C. Zau, Y. Emily Tang, and
Cory R. Koedel
PPIC
33
Supplementary Slides
PPIC
34
Who Exercises Choice - Methodology
 Look at applications to VEEP, magnet and Choice
enrollment programs for 2001-2002
– Gives truer picture of demand for school choice
than actual school switchers, because demand
exceeds supply
 Statistically model the probability that a student
applies to VEEP; separate models for magnets and
Choice (open enrollment)
PPIC
35
P
si
an
d.
C
re
de
nt
ia
l
ar
en
ta
lE
Fu
ll
H
ig
h
L
la
ck
E
iz
e
I
pa
ni
c
A
B
H
is
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
S
Lo
ca
l%
Lo
ca
lC
la
ss
Lo
ca
lA
P
Percentage Change in Students Applying
Change in the Probability of Application for
Local School Characteristics (Elementary)
Elementary School
8
6
4
2
0
-2
-4
-6
-8
-10
-12
VEEP
Magnet
Choice
PPIC
36
P
L
C
re
de
nt
ia
l
d.
pa
ni
c
si
an
la
ck
E
iz
e
ar
en
ta
lE
Fu
ll
H
ig
h
A
B
H
is
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
S
Lo
ca
l%
Lo
ca
lC
la
ss
I
ta
nc
e
Lo
ca
lA
P
Lo
ca
lD
is
Percentage Change in Students Applying
Change in the Probability of Application for
Local School Characteristics (Middle)
Middle School
8
6
4
2
0
-2
-4
-6
-8
-10
-12
VEEP
Magnet
Choice
PPIC
37
P
L
C
re
de
nt
ia
l
d.
pa
ni
c
si
an
la
ck
E
iz
e
ar
en
ta
lE
Fu
ll
H
ig
h
A
B
H
is
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
Lo
ca
l%
S
Lo
ca
l%
Lo
ca
lC
la
ss
I
ta
nc
e
Lo
ca
lA
P
Lo
ca
lD
is
Percentage Change in Students Applying
Change in the Probability of Application for
Local School Characteristics (High)
High School
8
6
4
2
0
-2
-4
-6
-8
-10
-12
VEEP
Magnet
Choice
PPIC
38
%
O
pt
io
n
O
pt
io
n
H
A
B
E
L
iz
e
si
a
C
re
de
nt
ia
l
al
Ed
.
c
n
la
ck
%
S
y
is
pa
ni
%
%
Pa
re
nt
%
Fu
ll
ig
h
%
H
O
pt
io
n
O
pt
io
n
O
pt
io
n
O
pt
io
n
ss
m
ta
nc
e
PI
D
um
D
is
C
la
A
O
pt
io
n
O
pt
io
n
O
pt
io
n
Percentage Change in Students Applying
Change in the Probability of Application for
Option School Characteristics (Elementary)
Elementary School
8
6
4
2
0
-2
-4
-6
-8
-10
-12
VEEP
Magnet
Choice
PPIC
39
Percentage Change in Students Applying
Change in the Probability of Application for
Option School Characteristics (Middle)
Middle School
8
6
4
2
0
-2
-4
-6
-8
-10
-12
n
t io
p
O
VEEP
Magnet
Choice
e
nc
a
t
s
Di
n
t io
p
O
m
Du
I
AP
m
y
n
t io
p
O
Cl
Si
s
as
ze
n
t io
p
O
%
EL
n
t io
p
O
%
k
ac
l
B
n
t io
p
O
%
l
.
t ia
Ed
n
l
a
de
nt
e
r
e
r
%
ll C
Pa
n
u
F
t io
gh
%
Hi
Op
n
%
t io
n
p
o
O
ti
Op
ia
As
n
i
an
p
s
Hi
c
PPIC
40
%
O
pt
io
n
O
pt
io
n
H
A
B
E
L
iz
e
si
a
C
re
de
nt
ia
l
al
Ed
.
c
n
la
ck
%
S
y
is
pa
ni
%
%
Pa
re
nt
%
Fu
ll
ig
h
%
H
O
pt
io
n
O
pt
io
n
O
pt
io
n
O
pt
io
n
ss
m
ta
nc
e
PI
D
um
D
is
C
la
A
O
pt
io
n
O
pt
io
n
O
pt
io
n
Percentage Change in Students Applying
Change in the Probability of Application for
Option School Characteristics (High)
High School
8
6
4
2
0
-2
-4
-6
-8
-10
-12
VEEP
Magnet
Choice
PPIC
41
Who Exercises Choice –
Percent Change in Probability of Applying
High School
VEEP
Magnet
3.2
Choice
2.7
2.2
1.7
1.2
0.7
0.2
-0.3
-0.8
-1.3
GPA
Stanford 9 Female
Score
Black
Asian
Hispanic
English
Learner
High
Parental
Education
Relative to White, Male, non-English Learner, Low Parental Education
PPIC
42
Black VEEP Applicants’ Change in % White (School Population)
Potential versus Actual
50
45
40
35
30
25
20
15
10
5
0
Applications
(potential
exposure)
Lottery
Winners
(potential)
Lottery Winners
Who Switch
(actual)
PPIC
43
Charter School Enrollment Share (%) in San
Diego
7
6
5
4
startup
conversion
3
2
1
0
1996-97
1997-98
1998-99
1999-00
2000-01
2001-02
2002-03
2003-04
PPIC
44
Charter Schools
Background
 Charters vary widely in academic focus and remediation
programs
 Exempt from many district regulations and state laws
 Other general characteristics
– Students
• 83% non-white (non-charters: 73%)
• 66% eligible for free/reduced-price meals
(non-charters: 56%)
– Draw from a wider geographic area
– Fewer teachers with long experience and advanced
education
– Often have additional costs such as rental of space,
rental of school buses
PPIC
45
Effects of Attending a Charter on Stanford 9
Scores
0.1
Proportion of a Standard Deviation
0.05
0
Math
-0.05
Reading
-0.1
-0.15
-0.2
Elementary
Middle
Gradespan
High
PPIC
46
Effect of Attending a Startup or Conversion Charter School
on Stanford 9 Test Score Gain
0.15
Proportion of a Standard Deviation
0.10
0.05
0.00
Math
-0.05
Reading
-0.10
-0.15
-0.20
-0.25
Startup
Conversion
Startup
Elementary
Conversion
Middle
Gradespan, Charter Type
PPIC
47
Policy Implications
Integration vs. Resegregation
 Overall, school choice improves racial/ethnic and
socioeconomic integration
 Weak evidence that students with higher test
scores or more highly educated parents more
likely to apply
– Effects small
– Less clear for GPA
– Does not support notion that choice programs
“skim off the cream”
PPIC
48
Qualifications of Integration Results
 School choice programs could integrate more if
supply constraints less severe
 Notable integrative differences between Choice—
which reduced integration along various lines—and
VEEP and magnet programs
– VEEP and magnets feature
• Publicly provided busing
• Regulation through geographic matching or
preferences
PPIC
49
Policy Question
Does Competition Force Schools to Improve?
 Although not directly studied, student applications
respond only weakly to average test scores at
schools
 If actual choices do not typically boost
achievement, how can student departures induce
schools that lose students to boost achievement?
PPIC
50
Other Result Implications
 Limited results (one cohort) prevent effective
argument against any of the choice programs
 Relative cost effectiveness implications
– Choice—no public busing cost
– VEEP and magnet—public busing cost
• Positive results for some magnets an offsetting
factor
– Charters—Often appear to receive less funding
yet perform about as well as regular public
schools
• Urgent need for studies of charter funding PPIC
51