THE EFFECT OF CHARTER SCHOOLS ON
STUDENT ACHIEVEMENT: A META-ANALYSIS
OF THE LITERATURE
CAMPBELL COLLOQUIUM
EDUCATION PANEL, MAY 2012
Julian R. Betts and Y. Emily Tang,
University of California, San Diego
(jbetts@ucsd.edu, yetang@ucsd.edu)
We are grateful to the Center on Reinventing Public
Education, University of Washington, Bothell, for funding this
research
OUTLINE
Introduction and Motivation
 Selecting Studies to Include
 Assessment of Alternative Methods of
Evaluating the Impact of Charter Schools
 Challenges in Study Collection/Review Process
 Description of Methods Used in Review
 Results
 Future Research and Policy Implications

2
SOME BACKGROUND ON US EDUCATION

Persistent concern over the performance of US
public schools at the elementary and
secondary levels

Elementary
 Grades

K-5 (ages 5-11)
Secondary
 Middle:
Grades 6-8 (ages 11-14)
 High: Grades 9-12 (ages 14-18)
3
THE US SPENDS A LOT (PER PRIMARY SCHOOL
PUPIL) ON EDUCATION, OBTAINS AVERAGE
EDUCATIONAL OUTCOMES
Source: Gruber (2010)
4
THE US SPENDS ABOUT AVERAGE (% OF GDP)
ON EDUCATION, OBTAINS AVERAGE
EDUCATIONAL OUTCOMES
Source: OECD (2011)
5
IN THE US THE SCHOOL THAT A STUDENT
ATTENDS IS PRIMARILY DETERMINED BY
WHERE HE/SHE LIVES
San Diego Unified School District
Elementary School Boundaries 2011-12
6
WHAT IS A CHARTER SCHOOL?

Charter schools are a relatively new alternative
to traditional neighborhood public schools


~ 20 years, substantial growth in the 2000s
A succession of U.S. presidents has named
charter schools as important agents of school
reform
7
APPROXIMATELY 5% OF PUBLIC SCHOOLS ARE
CHARTER SCHOOLS, THIS NUMBER IS GROWING
Source: Lake and Gross (2011)
8
WHAT IS A CHARTER SCHOOL?

Charter schools are publicly funded, governed
by organization under contract with the state

Charter schools are exempted from parts of the
state education code, freeing them to innovate
with respect to curriculum, pedagogy and hiring
of teachers
9
CHARTER SCHOOLS ARE DIFFERENT FROM
EACH OTHER, EXAMPLES FROM SAN DIEGO

Albert Einstein Academy:
 “independent
charter school that would have a dual
instructional focus of German-English immersion
within the context of a rigorous academic
instructional model”

Charter School of San Diego:
 initially
developed from a state bill “designed to
reduce the dropout rate by recovering students who
had been out of school for more than 45 days”
10
SELECTING STUDIES FOR THIS LITERATURE
REVIEW

Scope: Include studies of US elementary and
secondary charter school performance
US public K-12 education is decentralized
 Most data on student performance are collected
at the level of a US state, or the level of a school
district (smaller than a US state)


Outcomes: Include studies that use student
performance on math and reading
standardized tests as an outcome measure

Methods: Include studies that use credible
approaches to address selection bias
11
SELECTION BIAS: MAIN CONCERNS WITH
ALTERNATIVE APPROACHES LEADING TO
EXCLUSION
 Snapshots of average student
achievement at one point in time can be
misleading as they do not account for selfselection into schools
 US
school attendance based largely on
geographic residence.
 Students choosing to attend charter schools
are likely different in observable and
unobservable ways
12
UNOBSERVED CHARACTERISTICS CORRELATED
WITH CHARTER SCHOOL ATTENDANCE




Negative selection (downward bias)
Example: An underprivileged, disadvantaged
student without family support is at high risk of
dropping out of school. She is advised by her high
school counseling staff to transfer to a charter
school, and she chooses to transfer.
Problem: Underprivileged, disadvantaged
students without family support are not likely to
obtain high test scores in any school, traditional or
charter.
The estimate of charter school effectiveness
based on comparison of charter school student
performance and traditional school student
performance would be biased downwards.
13
UNOBSERVED CHARACTERISTICS CORRELATED
WITH CHARTER SCHOOL ATTENDANCE




Positive selection (upward bias)
Example: An active, concerned, involved parent is
dissatisfied with the traditional public school in
his/her neighborhood. The parent decides to optout of the traditional school and enroll his/her
child in a charter school.
Problem: Students with active, concerned,
involved parents are likely to obtain high test
scores in any school, traditional or charter.
Implication: The estimate of charter school
effectiveness based on comparison of charter
school student performance and traditional
school student performance would be upwardly
biased.
14
SELECTING STUDIES FOR THIS LITERATURE
REVIEW
National Charter School Research Project
issued a White Paper (drafters: Betts and
Hill, 2006) arguing that lottery-based
studies and student-level longitudinal
“value-added” studies were the two most
credible approaches
 These methods more convincing than
other methods.

15
METHODS MATTER
Source: Hill (2006)
16
4 COMMONLY USED METHODS OF ANALYSIS IN
THE INCLUDED STUDIES
In the set of studies we include, there are four
approaches used
 1) Lottery-based studies
 2) Fixed-effect studies, that compare a
student’s gains in achievement in years
attended a charter to his or her gains in years
attended a traditional public school
 3) Propensity score matching
 4) Other types of matching (e.g. CREDO)

17
LOTTERY-BASED ANALYSIS
Source: Waiting for Superman movie (2010)
18
LOTTERY-BASED ANALYSIS
Obvious benefit: expected outcomes identical
for lottery winners and losers if lottery
conducted fairly
 But several weaknesses to this “gold standard”
 External validity

 Most
charter schools not oversubscribed
 Mathematica
study of charter middle schools: only
130/492 oversubscribed

Could be bias from attrition
19
PROPENSITY SCORE MATCHING
Assumes “selection on observables”
 If students in charter schools have unobserved
variations in ability or motivation, will be biased
 Two major studies of KIPP (Knowledge is Power
Program) schools have used this approach
 CREDO at Stanford has produced string of
influential state-level studies. Uses a unique
matching process. Not propensity score but
has similar issue with “selection on
observables”

20
STUDENT FIXED-EFFECTS


Benefit: Avoids need to compare one student with another,
instead comparing individual students’ trajectories in charter
schools and traditional public schools
But many elementary students never switch between the two
types of schools – external validity issue

Zimmer et al (2009) compare test-score gains of charter “stayers”
and switchers and do not get clear-cut result. But in some cases
“stayers” have higher test-score gains



Suggests downward bias from using this method
Zimmer et al (2009) also raise concerns about reversibility –
are the effects of attending a charter dependent on the
order in which a student attends the charter and the
traditional public school? Find some evidence that this is
the case.
Unobserved heterogeneity may change over time. Fixed
effects cannot solve
21
INCLUDED STUDIES
40 reports now available, with just under 100
estimates of effects for each of math and
English Language Arts (reading)
 Lottery-based studies still quite rare: still only 8
papers that use lotteries, covering 90 charter
schools
 We exclude studies using less rigorous
methods, specifically, those that do not use
student-level test score gains as outcomes.

22
CHALLENGES IN STUDY COLLECTION/
REVIEW PROCESS

Handling large weight (large number of students and
large number of schools) studies


Handling the different methods used in different studies


Solution: Analyze with and without large weight studies
Solution: Investigate whether method of analysis matters
Some reports omit important information, e.g. number
of schools in the sample

Solution: Email exchange with authors
23
Introduction and Motivation
 Assessment of Alternative Methods of
Evaluating the Impact of Charter Schools
 Selecting Studies to Include
 Challenges in Study Collection/Review Process
 Description of Methods Used in Review
 Results
 Future Research and Policy Implications

24
OUR METHODS OF ANALYSIS



Fisher test – Is there evidence that no study finds
negative effects; conversely, evidence of no positive
effects?
Formal meta-analysis provides overall estimated effect,
its statistical significance and measures of how much
true underlying variation there is across studies
Histograms


Show variability and the influence of weighting of studies
Vote-counting as a way of assessing variation in results
25
HETEROGENEITY IS AN UNDERLYING THEME

Look for variations in effect by:
 Subject
area tested (math vs. reading)
 Grade span (E, M, H)
 Geographic location
 KIPP vs. non-KIPP
 Is there a systematic difference in results based on
the method researchers use?
26
METHODS USED IN REVIEW
Testing Whether Charter Schools in Any Study
Increase or Decrease Achievement Relative to
Traditional Public Schools
 Meta-Analysis of Effect Size
 Histograms and Vote Counting as Measures of
Variation

27
METHOD #1: EVIDENCE OF NO POSITIVE
EFFECTS, OR NO NEGATIVE EFFECTS?

Fisher’s combined test
k
S  2ln( pi )
i1
S is distributed  with df=2k
 Null hypothesis: No positive effects

 Null hypothesis: No negative effects
2


28
METHOD #1: EVIDENCE OF NO POSITIVE
EFFECTS, OR NO NEGATIVE EFFECTS?
We conduct this analysis 12 times: 6 ways of
combining grades, and two subjects (math and
ELA)
 First sign of heterogeneous effects of charter
schools: in 9/12 cases there is clear evidence of
BOTH negative and positive effects
 Three exceptions with evidence of positive effects
but no evidence of negative effects: elementary
and middle school ELA scores, and middle school
math scores

29
PROBABILITY OF NO POSITIVE EFFECTS IN
ANY OF THE STUDIES: ALMOST ZERO
Grade-Span
Reading Tests
Math Tests
Elementary
<0.001
<0.001
Middle
<0.001
<0.001
High
<0.001
0.001
<0.001
<0.001
All
<0.001
<0.001
Studies of All Grades or Largest
Grade Span(s) If An All-Grade Study
Not Available
<0.001
<0.001
El’y, Middle, and Combined
El’y/Middle
30
PROBABILITY OF NO NEGATIVE EFFECTS IN
ANY OF THE STUDIES: ALMOST ZERO IN
MOST CASES, AND QUITE HIGH IN 3 CASES
Grade-Span
Reading Tests
Math Tests
Elementary
0.987
<0.001
Middle
0.994
0.978
High
<0.001
0.001
<0.001
<0.001
All
<0.001
<0.001
Studies of All Grades or Largest
Grade Span(s) If An All-Grade Study
Not Available
<0.001
<0.001
El’y, Middle, and Combined
El’y/Middle
31
METHODS USED IN REVIEW
Testing Whether Charter Schools in Any Study
Increase or Decrease Achievement Relative to
Traditional Public Schools
 Meta-Analysis of Effect Size
 Histograms and Vote Counting as Measures of
Variation

32
METHOD #2: FORMAL META-ANALYSIS
Assume charter school estimates are randomly
distributed
 Therefore it is important to estimate both the
mean and the variation
 Underlying “true” variation across studies is the
extent to which variation cannot be explaining by
sampling error (“uncertainty”) in individual
estimates
 Omitted many studies of individual KIPP schools
as they would have disproportionate influence


Include KIPP schools in subsidiary analysis
33
THE MEAN EFFECT IS A WEIGHTED AVERAGE

In a random effects meta-analysis, we take a
weighted average of the effect sizes across
studies. If Yi is the effect size for the ith of k
studies, and Wi is the weight for each study, our
overall estimated effect size M is :
k
M

 (1)
W Y
i 1
k
i i
W
i 1
i
34
WEIGHTS DEPEND ON WITHIN-STUDY VARIANCE
AND ESTIMATED ACROSS-STUDY (TRUE)
VARIANCE




The weight for each study is the inverse of the sum of the
within-study variance (based on the standard error) and an
estimate of the true between-study variance, T2:
1
(2) Wi 
2
VYi  T
T2 based on a method of moments estimate of the
variance of true effect sizes.
Note that as T2 becomes large relative to the average withinstudy variance estimate, then we will tend toward equal
weighting across studies; whereas as T2 becomes relatively
small, the weights can become highly unequal with heavier
weight given to studies with the lowest sampling variance.
35
AN ESTIMATE OF WHAT % OF THE VARIANCE
ACROSS STUDIES IS TRUE

Use the I2 statistic (Higgins et al., 2003)
 Provides
estimate of the percentage of variation
across studies that reflects true underlying
variation
36
SAMPLE OF OUR RESULTS ON EFFECT SIZES

Grade Span
Reading Tests
Math Tests
E (Elementary)
0.022*
(9-7), 77.7%
0.049*
(10-8), 94.7%
* Indicates statistically significant (5% level)
37
SAMPLE OF OUR RESULTS

Grade Span
Reading Tests
Math Tests
E (Elementary)
0.022*
(9-7), 77.7%
0.049*
(10-8), 94.7%
* Indicates statistically significant (5% level)
“On average, attending a charter school is
associated with an increase in test scores in
reading equal to 0.022 of a standard deviation
per year.”
38
SAMPLE OF OUR RESULTS

Grade Span
Reading Tests
Math Tests
E (Elementary)
0.022*
(9-7), 77.7%
0.049*
(10-8), 94.7%
* Indicates statistically significant (5% level)
Nine studies covering
7 geographic areas
77.7% of the variation across studies
represents true variation in charter school
effects, rather than “noise”
39
OVERALL EFFECT SIZE ESTIMATES
Grade Span
Reading Tests
Math Tests
E (Elementary)
0.022*
(9-7), 77.7%
0.049*
(10-8), 94.7%
M (Middle)
0.011
(9-7), 85.7%
0.055*
(10-8), 92.0%
H (High)
0.054
(7-5), 98.3%
-0.015
(8-6), 98.6%
Combined E/M
-0.009
(15-12), 93.4%
-0.012
(15-12), 97.9%
E, M, and
Combined E/M
0.002
(31-17), 90.3%
0.020*
(33-18), 96.8%
0.008
(17-14), 98.4%
0.014
(18-15), 97.7%
All
40
ELEMENTARY/MIDDLE SCHOOL MATH EFFECTS:
MEANINGFUL BUT NOT HUGE
Enough to move a student at the 50th
percentile to the 52nd percentile after attending
a charter for one year
 Elementary school reading impact is smaller:
enough to boost a student from 50th to about
percentile 50.8

41
ELEMENTARY SCHOOL READING EFFECT SIZES
Study
%
ID
ES (95% CI)
Weight
Boston
0.06 (0.01, 0.10)
8.73
California
-0.00 (-0.01, 0.00)
25.00
Chicago
0.10 (0.03, 0.18)
3.70
Delaware
0.03 (0.00, 0.07)
12.45
NYC
0.04 (0.01, 0.07)
12.83
NYC
0.19 (0.02, 0.35)
0.88
National
0.01 (0.01, 0.01)
27.01
San Diego
-0.08 (-0.17, 0.01)
2.61
San Diego
0.04 (-0.01, 0.09)
6.80
Overall (I-squared = 77.7%, p = 0.000)
0.02 (0.01, 0.04)
100.00
NOTE: Weights are from random effects analysis
-.3
-.2
-.1
0
.1
.2
.3
42
ELEMENTARY SCHOOL MATH EFFECT SIZES
Study
%
ID
ES (95% CI)
Weight
Boston
0.02 (-0.03, 0.07)
11.44
California
-0.03 (-0.04, -0.02)
16.26
Chicago
0.12 (0.04, 0.19)
8.51
Delaware
0.04 (0.01, 0.07)
14.25
Idaho
0.33 (0.03, 0.63)
1.10
NYC
0.09 (0.06, 0.12)
14.29
NYC
0.19 (0.02, 0.36)
2.89
National
-0.00 (-0.00, 0.00)
16.50
San Diego
-0.19 (-0.30, -0.08)
5.89
San Diego
0.29 (0.22, 0.37)
8.88
Overall (I-squared = 94.7%, p = 0.000)
0.05 (0.02, 0.08)
100.00
NOTE: Weights are from random effects analysis
-.3
-.2
-.1
0
.1
.2
.3
.4
43
MIDDLE SCHOOL READING EFFECT SIZES
Study
%
ID
ES (95% CI)
Weight
Boston
0.17 (0.07, 0.27)
5.65
Chicago
-0.06 (-0.14, 0.01)
8.49
Delaware
0.08 (0.04, 0.12)
13.58
NYC
0.04 (-0.02, 0.10)
10.00
National
-0.10 (-0.23, 0.03)
4.21
National
0.02 (0.02, 0.02)
17.37
San Diego
-0.08 (-0.12, -0.04)
13.69
San Diego
0.01 (-0.04, 0.06)
10.94
Texas
0.01 (-0.01, 0.04)
16.06
Overall (I-squared = 85.7%, p = 0.000)
0.01 (-0.02, 0.04)
100.00
NOTE: Weights are from random effects analysis
-.3
-.2
-.1
0
.1
.2
.3
44
MIDDLE SCHOOL MATH EFFECT SIZES
Study
%
ID
ES (95% CI)
Weight
Boston
0.54 (0.39, 0.69)
4.81
Chicago
-0.09 (-0.16, -0.02)
10.32
Delaware
0.09 (0.05, 0.13)
13.10
Idaho
-0.05 (-0.18, 0.08)
5.88
NYC
0.24 (0.16, 0.31)
9.70
National
-0.08 (-0.20, 0.04)
6.31
National
0.02 (0.02, 0.02)
14.66
San Diego
0.06 (0.03, 0.10)
13.15
San Diego
0.01 (-0.09, 0.11)
7.90
Texas
-0.00 (-0.02, 0.02)
14.17
Overall (I-squared = 92.0%, p = 0.000)
0.05 (0.01, 0.10)
100.00
NOTE: Weights are from random effects analysis
-.3
-.2
-.1
0
.1
.2
.3
45
HIGH SCHOOL READING EFFECT SIZES
Study
%
ID
ES (95% CI)
Weight
Boston
0.16 (0.02, 0.31)
11.60
Delaware
0.21 (0.16, 0.26)
16.18
National
-0.02 (-0.02, -0.02)
16.98
San Diego
0.04 (-0.24, 0.33)
6.09
San Diego
0.03 (-0.01, 0.07)
16.33
San Diego
0.15 (0.10, 0.20)
15.97
Texas
-0.16 (-0.18, -0.14)
16.84
Overall (I-squared = 98.3%, p = 0.000)
0.05 (-0.03, 0.14)
100.00
NOTE: Weights are from random effects analysis
-.3
-.2
-.1
0
.1
.2
.3
46
HIGH SCHOOL MATH EFFECT SIZES
Study
%
ID
ES (95% CI)
Weight
Boston
0.16 (0.02, 0.31)
11.60
Delaware
0.21 (0.16, 0.26)
16.18
National
-0.02 (-0.02, -0.02)
16.98
San Diego
0.04 (-0.24, 0.33)
6.09
San Diego
0.03 (-0.01, 0.07)
16.33
San Diego
0.15 (0.10, 0.20)
15.97
Texas
-0.16 (-0.18, -0.14)
16.84
Overall (I-squared = 98.3%, p = 0.000)
0.05 (-0.03, 0.14)
100.00
NOTE: Weights are from random effects analysis
-.3
-.2
-.1
0
.1
.2
.3
47
READING EFFECT SIZES FOR STUDIES THAT
COMBINE ELEMENTARY AND MIDDLE SCHOOLS
Study
%
ID
ES (95% CI)
Weight
Arizona
-0.01 (-0.02, -0.01)
7.96
Arkansas
0.02 (0.00, 0.04)
7.00
Chicago
-0.04 (-0.06, -0.02)
6.85
Chicago
0.00 (-0.01, 0.01)
7.64
DC
-0.01 (-0.02, 0.01)
7.10
Georgia
0.01 (-0.00, 0.01)
7.86
Massachusetts
0.00 (-0.01, 0.02)
7.50
Minnesota
-0.02 (-0.03, -0.01)
7.46
Missouri
0.03 (0.01, 0.05)
7.05
NYC
0.09 (0.02, 0.16)
2.57
North Carolina
-0.09 (-0.12, -0.07)
6.00
Ohio
-0.08 (-0.12, -0.04)
4.74
Ohio
-0.00 (-0.01, 0.00)
7.80
Texas
0.09 (0.06, 0.12)
5.61
Texas
-0.08 (-0.10, -0.06)
6.85
Overall (I-squared = 93.4%, p = 0.000)
-0.01 (-0.02, 0.00)
100.00
NOTE: Weights are from random effects analysis
-.3
-.2
-.1
0
.1
.2
.3
48
MATH EFFECT SIZES FOR STUDIES THAT
COMBINE ELEMENTARY AND MIDDLE SCHOOLS
Study
%
ID
ES (95% CI)
Weight
Arizona
-0.04 (-0.05, -0.04)
7.60
Arkansas
0.05 (0.03, 0.07)
7.20
Chicago
0.02 (-0.02, 0.06)
6.23
Chicago
0.02 (0.01, 0.03)
7.53
DC
0.01 (-0.00, 0.03)
7.36
Georgia
-0.01 (-0.02, -0.00)
7.59
Massachusetts
0.06 (0.05, 0.07)
7.48
Minnesota
-0.03 (-0.04, -0.02)
7.44
Missouri
0.03 (0.01, 0.04)
7.24
NYC
0.12 (0.03, 0.21)
3.43
North Carolina
-0.16 (-0.20, -0.12)
6.11
Ohio
-0.18 (-0.26, -0.10)
4.01
Ohio
-0.06 (-0.07, -0.05)
7.56
Texas
0.08 (0.06, 0.11)
7.00
Texas
-0.12 (-0.16, -0.08)
6.23
Overall (I-squared = 97.9%, p = 0.000)
-0.01 (-0.03, 0.01)
100.00
NOTE: Weights are from random effects analysis
-.3
-.2
-.1
0
.1
.2
.3
49
METHODS USED IN REVIEW
Testing Whether Charter Schools in Any Study
Increase or Decrease Achievement Relative to
Traditional Public Schools
 Meta-Analysis of Effect Size
 Histograms and Vote Counting as Measures of
Variation

50
METHOD #3: HISTOGRAMS
Another way of displaying the variation across
studies
 Tried weighting each study equally and
weighting studies by number of observations

 Latter
approach gives heavy weight to CREDO
studies

Our formal meta-analysis is closer to weighting
studies equally than weighting by observation
51
52
53
METHOD #4: VOTE COUNTING
Categorize studies by sign of effect and whether
statistically significant
 Method is problematic because it ignores fact that
many statistically insignificant results all in the
same direction may, taken together, suggest a
statistically significant result
 We use mostly to highlight the heterogeneity
 Typically find that for most grade spans >50% of
studies show positive effects, but this weakens
and sometimes reverses if we weight studies by
number of observations

54
RESULTS VARY BY METHOD
Lottery results yielded the most positive
results, followed closely by propensity score
matching.
 These were followed by fixed effects and other
matching methods (which are fairly similar
with mixed positive and negative results)
 But it may not be the method that matters
quite so much as the specific schools studied

 Example:
Propensity score results are large but
focus on KIPP schools
55
RESULTS VARY BY METHOD
56
RESULTS VARY BY METHOD
57
REPLICATION OF RESULTS USING DIFFERENT
METHODS
There are 3 studies/pairs of studies that replicate
lottery results using more traditional “valueadded” methods
 They generally suggest that value-added models
can get close to the lottery results (but in a few
cases estimates slightly to meaningfully lower):

Boston (Abulkadiroglu et al.)
 New York (Hoxby et al. and CREDO)
 San Diego Preuss School (McLure et al., Betts, Tang
and Zau)

58
REPLICATION OF RESULTS USING DIFFERENT
METHODS
There are 3 studies/pairs of studies that replicate
lottery results using more traditional “valueadded” methods
 They generally suggest that value-added models
can get close to the lottery results (but in a few
cases estimates slightly to meaningfully lower):

Boston (Abulkadiroglu et al.)
 New York (Hoxby et al. and CREDO)
 San Diego Preuss School (McLure et al., Betts, Tang
and Zau)

59
Introduction and Motivation
 Selecting Studies to Include
 Assessment of Alternative Methods of
Evaluating the Impact of Charter Schools
 Challenges in Study Collection/Review Process
 Description of Methods Used in Review
 Results
 Future Research and Policy Implications

60
IMPLICATIONS FOR RESEARCH

Evaluate individual schools
 Charters
are meant to innovate; unlikely that all
charters will have the same impact
Charters should obtain permission from
applicants to gather student records
 States and chartering authorities should
regularly receive lottery data

61
IMPLICATIONS FOR RESEARCH

Focus on successful schools to identify
characteristics that may be working
 E.g.
longer day/time, student population targeted,
discipline policies, teacher management

Obtain more details about charter school
heterogeneity and study them

Obtain more details about charter school
closures
62
IMPLICATIONS FOR RESEARCH
Probably important to examine more than
results on math and ELA achievement.
 A handful of studies point to positive charter
effects on graduation, college attendance and
behavior.

 Expand
focus to include outcomes other than
math/reading test scores
63
WHAT WORKS CLEARINGHOUSE (WWC) FOR
CHARTER SCHOOL RESULTS

In the long run it would be good to have a nonpartisan group that collected and interpreted
school-level charter results.
64
IMPLICATIONS FOR POLICY

Status as a charter school vs. traditional public
school unlikely to be (on its own) meaningful
Promoting charter schools for sake of charter schools
probably not productive path to comprehensive reform
 Continue expansion (no particular reason not to)
 Still only ~5% of traditional public school sector
 Renew focus on traditional public school reform


Exploit flexibility of charter schools by using them
as laboratories to learn what works
65
THANK YOU!




Published version available at:
http://www.crpe.org/cs/crpe/view/csr_pubs/467
Executive summary at:
http://www.crpe.org/cs/crpe/view/csr_pubs/468
66
SUPPLEMENTARY SLIDES
67
ADDING KIPP STUDIES BACK IN HAS BIG EFFECT
Grade Span
Reading Tests
Math Tests
0.070*
(38-33), 88.3%
0.180*
(39-34), 96.8%
0.034*
(60-43), 90.8%
0.105*
(62-44), 98.6%
0.096*
(29-unknown),
82.7%
0.223*
(29-unknown),
93.7%
Including KIPP
Schools
M
E, M, and
Combined E/M
Results
Including Only
KIPP Estimates
M
68
SENSITIVITY TO EXCLUSION OF CREDO STUDIES
CREDO (Stanford) has produced impressive string
of mostly state-wide longitudinal student studies.
Match each charter student to an average of
several similar demographics and test scores
 Many charter students are matched based on
their test scores AFTER they enter charter schools
 potential bias
 Hoxby (2009) has concerns about measurement
error that may bias charter coefficient down


CREDO offers partial rebuttal
69
RESULTS SOMEWHAT STRONGER IF OMIT
CREDO STUDIES
Grade Span
Reading Tests
Math Tests
E
0.034*
(8-6), 79.5%
0.072*
(9-7), 95.2%
M
0.010
(8-7), 87.2%
0.068*
(9-8), 92.8%
H
0.072
(6-4), 98.5%
-0.002
(7-5), 97.5%
Combined E/M
-0.023
(6-5), 95.5%
-0.041
(6-5), 96.9%
E, M, and
Combined E/M
0.008
(22-10), 92.0%
0.038*
(24-11), 95.0%
0.016
(10-9), 86.6%
0.041*
(11-10), 67.7%
All
70
EFFECTS FOR URBAN DISTRICTS AND SCHOOLS
LARGER THAN FOR ALL DISTRICTS
Grade Span
Reading Tests
Math Tests
E
0.046*
(6-4), 61.8%
0.085
(6-4), 92.2%
M
0.009
(5-4), 87.0%
0.139
(5-4), 94.8%
H
0.101*
(4-2), 78.2%
0.019
(4-2), 42.7%
Combined E/M
-0.003
(4-3), 86.2%
0.021*
(4-3), 47.7%
E, M, and
Combined E/M
0.016
(15-5), 84.1%
0.077*
(15-5), 92.4%
All
0.008
(8-6), 63.2%
0.045*
(8-6), 74.8%
71
VARIATIONS BY RACE/ETHNICITY
Surprisingly few studies test for variation by
race/ethnicity. CREDO studies an important
exception
 Patterns not uniform, but overall, charter
effects decline in the following order: black >
Hispanic > Native American > white

 Results
for whites typically negative, not always
significant. Biggest exception is high school
reading, with a positive and significant effect
72
VARIATIONS BY ENGLISH LEARNER, SPECIAL
EDUCATION, MEAL ASSISTANCE
Effects often insignificant, perhaps due to
smaller sample sizes
 But some evidence of positive effects of
charter schools on EL and special education
students in both math and reading from
studies of all grades and studies of middle
schools

73