Arkansas’ K-12 Achievement
& NSLA Funding
September 4, 2013
AAEA
1
•AR Education Reports
• Policy Briefs
• Report Cards
• Newsletters
• Data Resources
OEP is a research center within the College of Education and Health Professions at the University
of Arkansas that specializes in
Education Research and Policy.
Officeforeducationpolicy.org
2
Accessing Data Resources through the OEP
Arkansas School Data
OEP Homepage
Refer to menu bar at
the top left of the OEP homepage.
http://www.officeforeducationpolicy.org/
Click on Arkansas School
Data
Arkansas School Data has multiple
databases at both school and district
levels.
3
Accessing Report Cards, Education
Reports and Policy Briefs through the OEP
Refer to menu bar at the top left
of the OEP homepage.
www.uark.edu/ua/oep
OEPublications leads to
options such as Report
Cards, Education
Reports and Policy
Briefs.
Remember to sign up for our weekly e-mail, OEP Web Links (OWL), to get updated on
current education news across the state and nation. Please e-mail oep@uark.edu to sign
up.
Also, sign up for the OEP Blog at www.officeforedpolicy.com to receive alerts when the
latest OEP Blog posts are published.
4
OEP Outreach
• We at the OEP believe that teacher quality is important and that all
Arkansas classrooms should be lead by a qualified teacher.
• The Arkansas Teacher Corps (ATC) program is a collaborative
partnership between the University of Arkansas, school districts, and
local community organizations that aims to provide an accelerated
path to teaching for the highest-performing and most talented
individuals to have a lasting impact on students and communities
in Arkansas.
arkansasteachercorps.org
5
Outline
1. Overall Achievement: Are we 5th or 49th?
- Benchmark
- NAEP
2. The NSLA Funding Question
- Has NSLA funding produced gains for FRL
students?
- How have districts spent NSLA funding?
3. Our Recommendations for NSLA Funding
6
Overall AR Achievement:
How was Arkansas’ performance on
the Benchmark and End-of-Course
Exams in 2012-13? Over time?
7
Benchmark Performance
• Growth over
time, until slight
decrease in
2012-13 in
literacy and
math
• Slight decrease
can be attributed
to many factors,
including ceiling
effects and CCSS
“implementation
dip”
• Grade-level
trends: lower
grades perform at
higher levels than
upper grades
Benchmark, Grade 3 – 8, % Proficient/Advanced, Over time
59%
55%
2005-06
Literacy
Math
64%
2007-08
68%
75%
2010-11
77%
81%
78%
2011-12
79%
75%
2012-13
0%
20%
40%
60%
% Proficient/Advanced
80%
100%
8
Benchmark Performance, By Region
Literacy Benchmark, Grades 3-8
Math Benchmark, Grades 3-8
81%
79%
Arkansas
85%
83%
Region 1 (NW)
Region 1 (NW)
Region 2 (NE)
81%
78%
Region 2 (NE)
Region 3 (CN)
80%
78%
Region 3 (CN)
Region 4 (SW)
79%
77%
Region 4 (SW)
0%
20%
40%
60%
%Proficient/Advanced
2011-12
2012-13
80%
70%
67%
Region 5 (SE)
75%
73%
Region 5 (SE)
78%
75%
82%
80%
77%
74%
76%
73%
73%
72%
Arkansas
100%
0%
20%
40%
60%
80%
% Proficient/Advanced
2011-12
20012-13
• Higher-performing regions: Northwest and Northeast
9
1
EOC Performance
• In 2012-13,
slight
decreases
in Algebra
&
Geometry
scores
• Steady
increases
in Grade 11
Literacy
and
Biology
scores over
time
66%
78%
81%
77%
Algebra
60%
Geometry
51%
Literacy
29%
Biology
0%
10%
20%
65%
68%
70%
41%
42%
44%
30% 40% 50% 60%
% Proficient/Advanced
2007-08
72%
75%
72%
2010-11
2011-12
70%
80%
90%
2012-13
10
How was Arkansas’ performance on
the NAEP in 2011? Over time?
11
NAEP
• National Assessment of Education Progress –
Nation’s Report Card
• Administered to random sample of 4th and 8th
grade students
• Most recent data from 2011
– New 2013 NAEP data to be released this fall
12
NAEP Math, 2011
Grade 4
50
245
42
39
40
39
37
34
235
30
30
25
26
20
225
10
230
231
233
237
238
240
240
241
MS
LA
TN
OK
AR
US
MO
TX
0
215
Mean Scale Score
% Proficient & Advanced
• Grade 4 in math: Slightly below national average
13
NAEP Math, 2011
Grade 8
50
295
40
40
285
34
32
29
30
27
24
275
22
20
19
265
10
269
273
274
279
279
282
283
290
MS
LA
TN
OK
AR
MO
US
TX
0
255
Mean Scale Score
% Proficient & Advanced
• Grade 8 in math: Below national average
14
NAEP Reading, 2011
Grade 4
40
225
34
32
30
30
26
22
220
29
26
23
215
20
210
10
205
209
210
215
215
217
218
220
220
MS
LA
OK
TN
AR
TX
US
MO
0
200
Mean Scale Score
% Proficient & Advanced
• Grade 4 in reading: Below national average
15
NAEP Reading, 2011
Grade 8
40
270
36
32
265
30
27
21
28
27
27
260
22
20
255
10
250
254
255
259
259
260
261
264
267
MS
LA
TN
AR
OK
TX
US
MO
0
245
Mean Scale Score
% Proficient & Advanced
• Grade 8 in reading: Below national average
16
NAEP Performance, 2011
AR %
Prof
37%
US %
Prof
39%
Diff
Grade 4 Reading
30%
32%
-2%
Grade 8 Math
29%
34%
-5%
Grade 8 Reading
28%
32%
-4%
Grade 4 Math
-2%
Surrounding
States
AR > TN,
OKA, LA, MS
AR > TN, OK,
LA, MS
AR > OK, TN,
LA, MS
AR > TN, LA,
MS
17
NAEP Performance, Over time
Math, Grade 4
Math, Grade 8
50
40
39
35
40
38
40
25
20
17
34
37
36
20
10
27
15
22
26
13
34
28
20
29
24
19
10
10
33
23
37
20
27
26
30
31
30
31
13
9
10
1990
1992
13
14
1996
2000
0
0
1992
1996
2000
2003
AR
2005
U.S.
2007
2009
2011
AR
2003
2005
2007
2009
2011
U.S.
• In math, in grades 4 and 8, Arkansas’s students have
decreased the gap between Arkansas and the nation on the
NAEP.
• However, Arkansas still performs less well than the nation in
math and grades 4 and 8 on the NAEP. (Closer in Grade 4)
18
NAEP Performance, Over time
Reading, Grade 4
Reading, Grade 8
40
30
20
40
27
23
28
24
29
30
30
30
32
32
32
31
31
30
29
29
27
27
26
25
2002
2003
2005
2007
30
32
30
28
26
30
29
29
30
20
23
23
27
28
2009
2011
10
10
0
0
1992
1994
1998
2002
AR
2003
2005
U.S.
2007
2009
2011
1998
AR
U.S.
• In literacy, in grades 4 and 8, Arkansas’s students have
decreased the gap between Arkansas and the nation on the
NAEP.
• However, Arkansas still performs less well than the nation in
literacy and grades 4 and 8 on the NAEP. (Closer in Grade 4) 19
5th or 49th?
• Two stories are out there today:
1. AR is backwards … “Thank goodness for
Mississippi” … falling way behind in school
quality
2. AR is rapidly climbing … 6th in national rankings
on the 2012 Quality Counts report and now 5th in
2013!! AR has better schools than in
Connecticut, Florida, and Texas.
• Let’s look at comparable data to do a fair
comparison of AR scores to US totals.
20
NAEP Math, 2011
“Apples to Apples” Comparisons – Positive Results for AR
60
50
40
30
57
57
FRL Eligible
47
44
20
26
10
FRL Not Eligible
24
19
18
0
AR
US
Grade 4
AR
Grade 8
US
• In Grade 4, Arkansas’ FRL students were slightly ahead of the nation’s average.
21
• In Grade 8, Arkansas’ FRL students were slightly below the nation’s average.
NAEP Reading, 2011
“Apples to Apples” Comparisons – Positive Results for AR
60
50
40
30
FRL Eligible
48
FRL Not Eligible
48
20
10
44
41
20
18
18
18
0
AR
US
Grade 4
AR
Grade 8
US
• In Grade 4, Arkansas’ FRL students were slightly ahead of the nation’s average.
• In Grade 8, Arkansas’ FRL students were on par with the nation’s average. 22
NAEP v. Region, 2011
Math and Reading, Grade 4: Comparison to Region/US by Income
260
252
240
252
248
250
241
238
235
230
230
233
228
229
235
231
221
217
220
215
207
210
205
207
200
Math
Reading
Math
Overall
Reading
Low Income
Arkansas
Surrounding
Math
Reading
Higher Income
National
• Arkansas compares well to surrounding states and to the nation when
scores are compared by poverty level.
• Our state suffers in the overall category because more of our students are in
the low income group than in other states.
23
Careful with these results…
• When comparing performance of FRL students across
states, it is important to keep in mind cost of living.
– Income level of for a family of four at ~$30,000 (free lunch
threshold) looks different in Little Rock than in Los Angeles
• Therefore, FRL is an imperfect measure when
examining poverty levels and comparing data across
states.
• This might generate a positive BIAS for AR
• E.G. LR FRL = $30K ~= $22K in Seattle; thus comparing a
“wealthier” set of AR kids to WA kids.
24
NAEP: Ranking States by
Achievement
Achievement Measure
4th Grade
Math
4th Grade
Reading
8th Grade
Math
8th Grade
Reading
2011 NAEP Scaled
Score
238
217
279
259
Scaled Score Rank
36
38
39
43
+2.6
+2.6
+2.6
+0.6
14
11
12
21
(50 States + DC)
Difference Score
(Achieved – Expected)
Difference Score Rank
(50 States + DC)
• Above is Arkansas’ rank when comparing simple NAEP scores and a
ranking for when each state’s demographics are taken into consideration
(Difference Score Rank)
• Although Arkansas’ scores are lower than other states, the state as a whole
does well when our demographics are taken into consideration.
25
OEP Similar Schools Database
• Allows for
comparisons to
districts with
similar or the
same SES
characteristics,
including % FRL,
% household
bachelor degrees,
median income,
and district
enrollment
growth.
Find on our website (Officeforeducationpolicy.org), under Arkansas Schools Data
26
Back to the Question at Hand …
What do we think we know so far?
• AR students have been improving:
-
Benchmark and EOC growth over time (until 201213)
-
-
Slight NAEP overall growth over time
-
•
But test scores generally increase with time due to test
familiarity...so it’s important to compare AR to the US
Slight decrease in AR/US gap in 4th grade math/reading
Relates to question: Has NSLA funding for FRL
students helped?
27
The NSLA Question
1. How does NSLA funding work?
2. How do we know if it works?
- If it were working, what changes might we expect
to see?
3. So, what did we find about possible
effectiveness?
4. Given the uncertainty, could we have expected
great gains? (How were funds used?)
5. After all this, what would we suggest?
28
NSLA: How does it work?
• In the 2013 Quality Counts report, Arkansas received
a B+ on equity funding, ranking it as one of the top
states in the nation in distributing equity funding to
districts.
• Arkansas should be commended for its focus on
students in poverty, as the formula does channel
more resources toward students in poverty,
particularly those in very poor districts.
29
NSLA: How does it work?
Math (GPA Measure), Districts By % FRL
Literacy (GPA Measure), Districts By % FRL
• We know that districts with
70%4or more FRL students
see 3.8
a drop in achievement.
3.6 funding seeks to
• NSLA
3.4
allocate
more funding to
3.2
those districts.
4
3.8
3.6
3.4
3.2
3
3
2.8
2.8
2.6
2.6
2.4
2.4
2.2
2.2
2
2
2011-12
2009-10
2007-08
2011-12
2009-10
2007-08
30
NSLA Funding: How does it work?
$1,800
$1,600
$ per FRL Pupil
$1,400
$1,200
$1,000
$800
$600
$400
$200
$0%
10%
20%
30%
40%
50%
60%
% FRL of School
70%
80%
90%
100%
The tiered system creates two
“cliffs.”
• “Cliffs” cause districts with very similar demographics
to be treated differently in the funding system.
• For example, a district with 69% FRL receives less
funding per FRL pupil than a district with 70% FRL;
however, student bodies with 69% and 70% FRL look
relatively similar.
31
The Big Q – How would we know if
NSLA funding worked?
• We might observe …
– Hypothesis 1: Increased scores for FRL students
(relative to non-FRL students) … this may be the
most important!
– Hypothesis 2: Districts just above the “cliffs”
performing better relative to those just below the
“cliffs.”
– Hypothesis 3: Districts with influxes in NSLA
funds performing better than in past.
32
Hypothesis 1:
FRL Students vs Non-FRL Students
• If NSLA Funding were working, we might
expect to see increase in achievement for FRL
students relative to non-FRL students.
33
Benchmark Achievement
Math, 2005-06 to 2011-12
Math
Non-FRL students
FRL students
2005-06
2011-12
62nd
40th
66th
40th
Percentile
Point Growth
+4
0
Literacy, 2005-06 to 2011-12
Literacy
Non-FRL students
FRL students
2005-06
2011-12
63rd
39th
66th
43rd
Percentile Point
Growth
+3
+4
In math, the gap
between FRL and nonFRL students has
widened over time.
In literacy, FRL
students have
slightly closed the
gap; but FRL
students still
perform less well.
34
NAEP Growth, 2003 to 2011
Math and Reading Score Gains, 2003 to 2011
15
+13
12
+9
+9
+8
9
+7
+6
+7
+7
+6
+6
+6
+6
6
+4
+3
+3
+4
+3
+2
3
0
Math
Reading
Math
Overall
Reading
Low Income
Arkansas
Surrounding
Math
Reading
Higher Income
National
• Over the past decade, Arkansas scores have grown by leaps and bounds,
but that statistic is padded by lower baseline scores.
• The greatest gains come in math and for higher-income students.
35
Hypothesis 1:
FRL Students vs Non-FRL Students
• Achievement gap between FRL and non-FRL
students continues to exist.
– Benchmark
• Gap is widening in math performance
• Gap is slightly shrinking in literacy
– NAEP
• Non-FRL produced higher gains than FRL students
over time
36
Hypothesis 2:
“Cliff” Districts
• “Cliffs” cause districts with very similar demographics to
be treated differently in the funding system.
• For example, a district with 69% FRL receives less funding
per FRL pupil than a district with 70% FRL; however,
student bodies with 69% and 70% FRL look relatively
similar.
• The “cliffs” allow us to compare the performance of relatively
similar districts (e.g. 69% to 70%) that receive different
amounts of funding.
• Thus, if NSLA were working, we would see greater
performance for districts “above the cliffs”
37
Hypothesis 2:
“Cliff” Districts
Achievement Comparisons at the 70% “Cliff”*
Benchmark Math GPA, 2007-08 to 2012-13
Benchmark Literacy GPA, 2007-08 to 2012-13
4.00
4.00
3.80
3.80
3.60
3.60
3.40
3.40
3.20
3.20
3.00
3.00
2.80
2.80
2.60
2.60
2.40
2.40
2.20
2.20
2.00
2007-08 2008-09 2009-10 2010-11 2011-12 2012-13
64%-69%
70% - 75%
State Average
2.00
2007-08
2008-09
64%-69%
2009-10
2010-11
70% - 75%
2011-12
2012-13
State Average
• On the math and literacy benchmark exams, the districts just above and
below the cliff (thus, districts who are socio-economically “equal”)
perform nearly identically.
38
Hypothesis 2:
“Cliff” Districts
Achievement Comparisons at the 90% “Cliff”*
Benchmark Math GPA, 2007-08 to 2012-13
Benchmark Literacy GPA, 2007-08 to 2012-13
4.00
4.00
3.80
3.80
3.60
3.60
3.40
3.40
3.20
3.20
3.00
3.00
2.80
2.80
2.60
2.60
2.40
2.40
2.20
2.20
2.00
2.00
2007-08
2008-09
84%-89%
2009-10 2010-11
90%-95%
2011-12 2012-13
State Average
2007-08
2008-09
84%-89%
2009-10
2010-11
90%-95%
2011-12
2012-13
State Average
• On the math and literacy benchmark exams, districts just below the
90% cliff outperformed the districts above the cliff.
39
Hypothesis 3:
Increased Funding
• When a district “moves up a tier” by having a higher % of
FRL students, FRL students may perform at higher levels
after the district has received more funding.
• Thus, if NSLA were working, we would see greater
performance for districts after the new funds
• Since 2004-05, some districts have moved into a higher
tier of poverty funding. The achievement of these districts
was compared and at both the 70% and 90% cliffs, no
district showed an increase in achievement as a result
of a financial windfall.
40
So, what do we know about
NSLA?
•
It is important to note that we do not have the counterfactual
to examine how districts would perform without poverty
funding. Nevertheless, we do know that:
1.
Most agree that additional resources should be provided to
schools with higher concentrations of poverty (to help students
overcome additional challenges associated poverty).
2.
No research indicates exact $$ amount needed to create equal
opportunities for poor students.
3.
From data presented thus far, no justification for funding “cliffs”
(theoretical or empirical).
So, how do districts use NSLA funding?
41
So, how do districts use NSLA funding?
Year Coded
as Exp.
Percent of NSLA Funding in
2011-12
Literacy, Math, and Science Specialists and Coaches
2003
16.51%
Other activities approved by the ADE
High Qualified Classroom Teachers
Transfer to ALE Categorical Fund
School Improvement Plan
Counselors, Social Workers, Nurses
Teachers’ Aides
Curriculum Specialist
Pre-Kindergarten
2003
2003
2003
2003
2003
11.56%
9.42%
8.63%
8.62%
8.30%
8.17%
4.69%
3.27%
Before and After School Academic Programs
2003
2.76%
-
2.77%
Tutors
Transfer to ELL Categorical Fund
2003
2.35%
2.28%
Professional Development in Literacy, Math, and Science
2003
2.02%
Summer Programs
Early Intervention
Transfer to Special Educations Programs
2003
2003
-
1.28%
1.22%
0.93%
-
0.87%
District Required Free Meal Program
Parent Education
ACT Fees for 11th Graders and Operating/Supporting a Post-Secondary Preparatory Program
Scholastic Audit
2011
2003
2011
-
0.70%
0.52%
0.10%
0.37%
Districted Reduced-Lunch Meal Program
2011
0.05%
Remediation activities for college
2011
0.05%
Teach For America professional development
2011
0.03%
Implementing Arkansas Advanced Initiative for Math and Science
2011
0.01%
Hiring Career and College Coaches
2011
0.00%
Materials, supplies, and equipment including technology
2003
-
Expenses related to a longer school day
Expenses related to a longer school year
2011
2011
-
Expenditure Categories
Supplementing Salaries of Classroom Teachers
Transfer to Professional Development Categorical Fund
Shaded box
denotes a
coded use
originally
set in 2003.
42
How do districts use NSLA funding?
• The majority of districts distribute funding among 8 or more
expenditure codes.
• Districts seldom focus the money in one or two specific areas; therefore, it
seems if many districts use the funding to plug gaps in budgets.
• It is unclear as to whether all districts are specifically pinpointing the
funding towards students in poverty (or schools serving these students).
• For example, a district may spend a large portion of funding on
Highly Qualified teachers or Specialists – these teachers may or may
not work specifically with the low-income students.
• Furthermore, districts do not use all the funding – many have balances at
the end of the year.
43
How do districts use NSLA funding?
• Given the uncertainty, could we have expected
great gains?
– Funds have spent across the board by most districts
– No clear evidence that funding has been focused
for students in poverty
– Money is allocated to district offices and not even
to schools with high levels of poverty
44
What do we recommend?
Two main discussions this year:
• Distribution of funds
– “Smooth sliding” scale to replace the current tiered system
– Distribute more funding for districts with higher
concentrations of FRL students
– Weighting the funding to differentiate between poverty
levels by factoring in the difference between “free” and
“reduced” lunch students
– Leftover balances by districts
• Use of funding: more or less prescriptive?
45
Example of a Smooth Distribution
$1,800
$1,600
$1,400
$1,200
$1,000
$800
Problem:
Several “affluent”
districts would
lose $$
$600
$400
$200
$0%
20%
40%
Accel Smooth Option
60%
80%
100%
Status Quo
• “Smooth” sliding scale
• Weighted to account for differences in “free” and “reduced”
• Weights are 75% for Reduced-Lunch Students and 100% for
Free-Lunch Students.
46
Smoother … but prescriptive?
Should the use of NSLA funding be more prescriptive?
Long debate over extent of “mandating the spending matrix”
• Arguments for prescriptive use:
– Current lack of focus of funds
– Pinpoint only to students in poverty
– Use prescriptive manner as a way to figure out what works
• Arguments against prescriptive use:
– Flexibility is necessary: State-wide policies may not fit for
all.
– What do you prescribe? Research isn’t conclusive on what
works best
47
Concluding Thoughts
• Arkansas scores on the Benchmark and EOC have improved in the
past 10 years, but… much of the NAEP increases occurred before
2003 (slides 18 and 19)
• On the NAEP, Arkansas students have only slightly increased
scores in 4th grade and 8th grade
– FRL and non-FRL students have produced gains; but non-FRL
students have experienced greater gains
• It is difficult to determine the effectiveness of NSLA funding over
the past 10 years.
– The gap between FRL and non-FRL students has not shrunk.
• Policymakers and districts need to continue to strategically think
about how NSLA funding can be pinpointed so that students in
poverty can achieve at higher levels.
48
Comments? Questions?
Thank you for your time and input!
49