Review and Discussion
of AMAO 2 Criteria & Targets
NC Department of Public Instruction
With
WestEd & Wisconsin Center for Education Research
Statewide Web Conference
August 30, 2010
Purpose
Review and comment on recommendations
for changes to NC State Board of Education
policy GCS-A-012, Annual Measurable
Achievement Objectives for NCLB Title III
 NCDPI Recommendations in August
 Policy Revisions to SBE in September
NCDPI
AMAO 2 Meeting
2
Session Agenda
 Introduction and Purpose (Ground Rules)
 Historical Perspective and 1-year Recap
 AMAO 2 Proficiency Criteria
(Comprehensive Objective Composite,COC)
Review and Comparison (2009 & 2010)
 AMAO 2 Targets for Consideration:
2009-10 and Beyond
 Next Steps & Meeting Wrap Up
NCDPI
AMAO 2 Meeting
3
Introductions
New NCDPI Staff
 Scott Beaudry, Testing Policy & Operations
Special Guests
 Robert Linquanti, WestEd
 Gary Cook, Wisconsin Center for Education
Research
 Shirley Carraway, Appalachia Regional
Comprehensive Center
NCDPI
AMAO 2 Meeting
4
Historical Perspective
In 2007-2008:
1. Determined revisions to AMAOs 1 & 2 needed
 Criteria were too loose or too stringent
 Targets set without federal guidance
 Targets not based on empirical data
2. Determined new standards and assessments needed
 ACCESS for ELLs would replace IPT
3. Determined that AMAO 1 criteria and targets would be
revised after two years of ACCESS for ELLs data was
gathered.
NCDPI
AMAO 2 Meeting
5
Historical Perspective: AMAO 2
In 2007-2008, cont’d.:
 Decided to keep the criterion for
proficiency the same for 2007-08
 2007-08 target was set at 17% to account
for differences in using Form A and Form B
of the IPT
 Targets for 2008-09 and beyond removed
as they needed to be based on empirical
results from new ELP assessment
NCDPI
AMAO 2 Meeting
6
Last Year Recap: AMAO 2
In 2008-09:
 Per USED Notice of Final Interpretations (2008):
 Only one data point needed to calculate AMAO 2 for
each ELL
 All ELLs (K-12) must be included in calculation
 2008-09 ACCESS results used to define COC
and new, one-year target for 2008-09 (14.7%)
 Decision made to set future targets after
examining another year of ACCESS results
NCDPI
AMAO 2 Meeting
7
Last Year Recap: AMAO 2
In 2008-09, cont’d.:
 Stakeholders endorsed state-recommended
COC derived from analyses of student
performance on 2008-09 ACCESS and
state’s reading and math assessments
 Overall 4.8, R & W each 4.0 minimum
NCDPI
AMAO 2 Meeting
8
AMAO 2 Proficiency Criteria (COC)
Review & Comparison (2009 & 2010)
 Purpose: Replicate analysis performed in
2008-09 to validate COC criteria chosen
 Apply same decision consistency method
to 2009-10 ACCESS and EOG/EOC reading
and math assessments
NCDPI
AMAO 2 Meeting
9
Decision Consistency Method
 These analyses identify language
proficiency level that optimally
classifies students as true-positives or
true-negatives on both NC EOG/EOC
Reading & Math Assessments and
ACCESS
NCDPI
AMAO 2 Meeting
10
Decision Matrix
Criterion A
Criterion B
Below
Above
True
False
Below
Negative Positive
False
Above
Negative
True
Positive
 True Positives  True Negatives 

Correct  
Total number in sample


NCDPI
AMAO 2 Meeting
11
Decision Matrix
ELP Assessment
Below
Above
Content
Assessment
Below
43
14
Above
18
25
Correct = 68%
NCDPI
AMAO 2 Meeting
12
Decision Matrix
ELP Assessment
Below
Above
Content
Assessment
Below
51
5
Above
10
34
Correct = 85%
NCDPI
AMAO 2 Meeting
13
Reading to ACCESS: 2009
NC EOG Reading to ACCESS Decision Consistency
across Clusters
Percent Classified Correctly
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<2.0
2.0-2.4
2.5-2.9
3.0-3.4
3.5-3.9
4.0-4.4
4.5-4.9
5.0-5.4
5.5-6.0
WIDA Composite Proficiency Band
Cluster 3-5
Cluster 6-8
NCDPI
AMAO 2 Meeting
Cluster 9-12
14
Mathematics to ACCESS: 2009
NC EOG Mathematics to ACCESS Decision Consistency
across Clusters
Percent Classified Correctly
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<2.0
2.0-2.4
2.5-2.9
3.0-3.4
3.5-3.9
4.0-4.4
4.5-4.9
5.0-5.4
5.5-6.0
WIDA Composite Proficiency Band
Cluster 3-5
Cluster 6-8
NCDPI
AMAO 2 Meeting
Cluster 9-12
15
Comparing DC Analysis, 2009 to 2010:
Reading to ACCESS, Grades 3-5
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<2.0
2.0-2.4
2.5-2.9
3.0-3.4
G35 Read 2009
3.5-3.9
4.0-4.4
4.5-4.9
5.0-5.4
5.5-6.0
G35 Read 2010
NCDPI
AMAO 2 Meeting
16
Comparing DC Analysis, 2009 to 2010:
Reading to ACCESS, Grades 6-8
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<2.0
2.0-2.4
2.5-2.9
3.0-3.4
G68 Read 2009
3.5-3.9
4.0-4.4
4.5-4.9
5.0-5.4
5.5-6.0
G68 Read 2010
NCDPI
AMAO 2 Meeting
17
Comparing DC Analysis, 2009 to 2010:
Reading to ACCESS, Grades 9-12
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<2.0
2.0-2.4
2.5-2.9
3.0-3.4
G912 Read 2009
3.5-3.9
4.0-4.4
4.5-4.9
5.0-5.4
5.5-6.0
G912 Read 2010
NCDPI
AMAO 2 Meeting
18
Comparing DC Analysis, 2009 to 2010:
Math to ACCESS, Grades 3-5
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<2.0
2.0-2.4
2.5-2.9
3.0-3.4
G35 Math 2009
3.5-3.9
4.0-4.4
4.5-4.9
5.0-5.4
5.5-6.0
G35 Math 2010
NCDPI
AMAO 2 Meeting
19
Comparing DC Analysis, 2009 to 2010:
Math to ACCESS, Grades 6-8
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<2.0
2.0-2.4
2.5-2.9
3.0-3.4
G68 Math 2009
3.5-3.9
4.0-4.4
4.5-4.9
5.0-5.4
5.5-6.0
G68 Math 2010
NCDPI
AMAO 2 Meeting
20
Comparing DC Analysis, 2009 to 2010:
Math to ACCESS, Grades 9-12
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<2.0
2.0-2.4
2.5-2.9
3.0-3.4
G912 Math 2009
3.5-3.9
4.0-4.4
4.5-4.9
5.0-5.4
5.5-6.0
G912 Math 2010
NCDPI
AMAO 2 Meeting
21
AMAO 2 Criterion Confirmed
Comprehensive Objective Composite (COC)
Current English language proficiency definition
on the ACCESS test holds:
 Composite score of at least 4.8 and at least 4.0 on
Reading subtest and 4.0 on Writing subtest.
Note: Students who attain the COC as defined
above exit LEP identification. Those who do
not remain identified LEP.
NCDPI
AMAO 2 Meeting
22
REMINDER:
2009-10 AMAO 2 Cohort Definition
Cohort definition required by federal law:
 ALL LEP students (K-12) must be included in AMAO
2 calculation
Numerator = # of LEP students attaining COC
Denominator = #of LEP students required to test
NCDPI
AMAO 2 Meeting
23
Historical AMAO 2 Target Data
Year
Target
Met
Not Met
Missing data
Total
2003-04
2004-05
20.0%
25.0%
86
81
6
1
0
1
92
83
2005-06
2006-07
2007-08
30.0%
35.0%
17.0%
4
76
3
83
38
47
0
85
2008-09
2009-10*
14.7%
11.8%
25
68
63
20
0
0
88
88
*All 2009-10 results are unofficial
NC LEA and State AMAO 2 Performance Using
Current AMAO 2 Criterion:
AMAO 2 Criterion
Overall 4.8 (R&W GE 4.0)
2010 Analysis
% LEP meeting AMAO 2 Criterion by LEA Percentile Rank
STATE
P10
P15
P20
% LEP
9.5%
10.7%
11.5%
P25
P50
P75
P80
P90
11.8% 13.8% 16.8% 17.4% 19.7%
14.5%
Shows percentage of LEPs meeting COC performance
level for LEAs at that ranking and Statewide
NCDPI
AMAO 2 Meeting
25
2010-2019 Annual Targets
for LEAs & State
using 2010 as Base Year
Recommended Targets for Consideration
Percent of LEP attaining English
proficiency
25.0
23.0
21.0
19.0
17.0
15.0
13.0
11.0
9.0
7.0
5.0
25 %ile
2009 2010 2011 2012 2013 2014 2015 2016
-10 -11 -12 -13 -14 -15 -16 -17
AMAO 2
11.8 12.4 12.9 13.5 14.0 14.6 15.1 15.7
Target
NCDPI
AMAO 2 Meeting
AMAO 2
• Proposed target
for 2009-10 is
11.8% of LEP
students in an
LEA attaining
English
language
proficiency.
• Proposed end
point in 2018-19
75 %ile is 16.8% of LEP
students in an
LEA attaining
2017 2018
English
-18 -19
language
16.2 16.8
proficiency.
26
Target Recommendation: 2010-2019
 Set the 2009-10 target at 11.8%
(25th %ile)
 Set the 2018-19 target at 16.8%
(75th %ile)
 Structure targets to increase by equal
increments each year
(0.55 percentage points)
NCDPI
AMAO 2 Meeting
2009-10 AMAO Analysis
 AMAO 1 Criteria and targets are the
same
 improve at least one proficiency level in
at least one of the subtests of reading,
writing, speaking, or listening
 Target = 70%
2009-10 Analysis (continued)
Sanctions for AMAOs Not Met
 In 2008-09, Title III status based on
whether or not the LEA failed to make
progress toward meeting the same
AMAO
 Starting in 2009-10, Title III status
based on failure to meet the
AMAOs
GCS-A-012 Revisions
 Show Draft Policy
 Policy to SBE in September as Action
on First Read
Next Steps (for 2009-10 data)
 Updated GCS-A-012 sent to USED for Title III
Workbook and Title III Plan submission
 Preliminary AMAO report sent to districts for
review in September
 AMAO report presented to SBE in November
NCDPI
AMAO 2 Meeting
31
Next Steps (2010 -11 and beyond)
 Analysis of potential changes to AMAO 1
progress definitions and targets to occur
during Fall 2010
 AMAO 1 analyses and draft
recommendations vetted with
stakeholders during 2010-11 school year
 AMAO 1 policy approval in 2010-11
 Updated policy sent to USED for
submission with Title III Workbook and
Title III Plan
 Federal Title III Audit in Spring 2011