Accountability Services
North Carolina Department of
Public Instruction
September 23, 2014









Tom Tomberlin
Director, District HR Support

• The North Carolina Educator Effectiveness System
(NCEES) has six standards of performance for teachers and eight standards for principals.
• NC has a conjunctive model, meaning that teachers and principals must be proficient (or better) on all standards in order to receive an overall effectiveness rating. We do not average or index these standards.
• Unlike the observational standards, student growth
(standard 6 for teacher, standard 8 for principals), requires three years of valid data in order to generate a rating.

Teachers
Demonstrate
1
Establish
2
Know
3
Facilitate
4
Reflect on
5 Contribute 6
Success
Principals (and other Administrators)
Strategic
1
Instructional
2
Cultural
3 Human 4
Managerial
5
External
6
Micro-
7 8
Leadership
Academic
Achievement
Leadership

Rating from
2 years ago
Rating from
1 year ago
Rating from this year
Standard Standard
6 6
Standard
6
1.0
Met
Expected
Growth
-2.5
Did not meet
Expected
Growth
1.2
Met
Expected
Growth
1.0 + (-2.5) + 1.2
3
= -0.3
Met Expected Growth
3- year average rating on standard 6 for determining status
Note: A similar methodology applies to principals as well.

Standards 1-5
In the year
Demonstrate
1
Establish
2
Know
3
Any rating lower than proficient
Facilitate
4
Reflect on
5
Standards 6
Three year rolling average
)
2 years ago
6 + 1 year ago
6 + This 6
) / 3
Does Not Meet Expected Growth

Standards 1-5
In the year
Demonstrate
1
Establish
2
Know
3
Facilitate
4
Reflect on
5
Proficient or Higher on Standards 1 - 5
Standard 6
Three year rolling average
)
2 years ago
6 + 1 year ago
6 + This 6
) / 3
Meets or Exceeds Expected Growth

Standards 1-5
In the year
Demonstrate
1
Establish
2
Know
3
Facilitate
4
Reflect on
5
Accomplished or Higher on Standards 1 - 5
Standard 6
Three year rolling average
)
2 years ago
6 + 1 year ago
6 + This 6
) / 3
Exceeds Expected Growth

• For all teachers (and principals) the first status for
Standard 6 will be generated from the best two out of three valid Standard 6 ratings.
• School-level growth that has been assigned to a teacher as a result of a waiver (from NCFEs or ASW) will function as a valid Standard 6 rating.
• School-level growth that has been assigned as a result of a lack of data for a teacher (i.e., not from a waiver) will not count as a valid Standard 6 rating.

Rating from
2012-13
Rating from
2013-14
Rating from
2014-15
1.0
Met
Expected
Growth
6 6 6
-2.5
Did not meet
Expected
Growth
1.2
Met
Expected
Growth
• Teacher has individual-level data for three years.
• Standard 6 from the 2013-14 school year is the lowest of the three ratings.
• Teacher’s Standard 6 status is 1.1 – “Meets Expected
Growth ”.

Rating from
2012-13
Rating from
2013-14
Rating from
2014-15
1.0
Met
Expected
Growth
6 6 6
-2.5
Did not meet
Expected
Growth
1.2
Met
Expected
Growth
• Teacher has individual-level data for the first two years.
• The 2014-15 data is schoollevel growth from a waiver.
• Standard 6 from the 2013-14 school year is the lowest of the three ratings.
• Teacher’s Standard 6 status is 1.1 – “Meets Expected
Growth ”.

Rating from
2012-13
Rating from
2013-14
Rating from
2014-15
1.0
Met
Expected
Growth
6 6 6
-2.5
Did not meet
Expected
Growth
1.2
Met
Expected
Growth
•
Teacher has individual-level data for the final two years.
• The 2012-13 data is schoollevel growth because teacher did not have individual-level data.
• The teacher does not receive a status in the fall of 2015 because teacher does not have 3 years of valid data.
• First status in Fall 2016
(provided teacher has valid data in SY 2015-16).

Rating from
2012-13
Rating from
2013-14
Rating from
2014-15
Rating from
2015-16
1.0
Met
Expected
Growth
6 6 6
-2.5
Did not meet
Expected
Growth
1.2
Met
Expected
Growth
4.0
Exceeded
Expected
Growth
6
• Teacher receives second status in fall of 2016.
• Rating from 2012-
13 “rolls off”.
• Rating from 2013-
14 returns to the rolling average
(even though it was dropped from prior year’s calculation).
• Teacher’s status is
“Meets Expected
Growth” with an average of 0.9.

• The six standards (eight for principals) are weighted equally in the determination of teachers’ effectiveness ratings.
• In practice, however, student growth carries much more weight in differentiating teachers in terms of effectiveness.
• Nominal vs. Effective Weighting

Status (Observation) →
Status (Obs + Growth) ↓
Needs
Improvement
Effective
Highly
Effective
Needs Improvement 1416 4658 2648
Effective
Highly Effective
0
0
14049
0
10106
5383
Total
8722
(22.8%)
24155
(63.1%)
5383
(14.1%)
Total
1416
(3.7%)
18707
(48.9%)
18137
(47.4%)
38260

• Standard 6, student growth, plays a greater role in determining teacher effectiveness ratings than observational data.
• The disproportional effect of student growth is an artifact of the lack of variation in observational data, not a value judgment.
• More accurate assessment of teacher performance can improve this phenomenon.

SY 2011-12 SY 2012-13

Student Growth, as measured using value-added data, and ratings on
Standards 1-5 continue to be very, very loosely correlated. Additionally, it appears that we are measuring one, holistic element of “good teaching” rather than five discrete standards of practice.

70
60
50
40
90
80
30
20
10
0
Results from Fall READY
Principals
Ms. Brown - Area and Perimeter
ND Dev Prof Acc Dist
Novice
10 year
20 year

Student Raw Score
(26/50)
Deviation from the mean
(50th NCE) for each student is aggregated at the teacher (or school, or district) level.
The mean difference is the teacher effect which has an associated standard error
Conversion to Scale
Score (250)
Difference Between
Expectation and Actual
Scale Score (240 vs. 250)
Percentile Rank is converted to NCE
(~ 58 NCE)
Positive 10 Scale Score
Points is compared to distribution at the state level for that grade and subject.
(e.g., 65th %tile)
The teacher effect is divided by the standard error to create the index.
Index>=2 – Exceeds
2>=Index>2 – Meets
Index <-2 – Does Not Meet

Student A
Student B
Student C
Student D
Student E
Mean
Std Dev
Std Error
Index
Teacher 1
0.5
-0.9
0.9
0.5
-1.7
-1.9
-1.2
-0.7
-1.0
Teacher 2
-1.5
-1.0
1.5
0.5
-2.0
-2.2
-1.6
1.6
-1.2
Teacher 3
-1.2
-0.9
0.5
0.5
-1.8
-0.4
-0.5
-1.6
-0.7

Student A
Student B
Student C
Student D
Student E
Mean
Index
Teacher 1
-1.9
-1.2
-0.7
-1.0
0.5
-0.9
Std Dev
School A
Teacher 2
-2.2
-1.6
1.6
-1.2
-1.5
1.0
-3
Std Error
Teacher 3
-0.4
-0.5
-1.6
-0.7
-1.2
0.3

• How can a school increase proficiency rates by X percentage points but not meet growth?
Student A
Student B
Student C
Student D
Student E
Student F
Student G
Student H
Student I
Student J
Proficiency = 230
Prior Year (Expected) Score
229
250
255
226
228
243
225
231
227
235
Prior Year NCE
Prior Year
% Proficient = 50%
54
62
64
53
54
59
52
55
53
56
Current Year Score
Current Year
%Proficient = 90%
231
238
236
230
232
235
230
230
220
230
Current Year NCE
Mean
Std Dev
Std Error
Index
55
57
56
54
55
56
54
54
50
54
Growth
2
-1
-3
-2
-1.7
3.2
0.8
-2.1
1
-5
-8
1
1
-3

Does Growth Matter?
• Two groups of students captured:
• Younger Cohort: began Grade 5 in SY 2010-11
• Older Cohort: began Grade 6 in SY 2010-11
Younger
Older
Math
Reading
Math
Reading
Low 1
1935
184
1560
183
High 2
1825
221
1547
301
1 Students in the “Low Growth” category have consecutive years of teachers in the Does Not Meet Expected Growth category
2 Students in the “High Growth” category have consecutive years of teachers in the Exceeds Expected Growth category

Evaluation and Change in Growth

Thomas R. Tomberlin
Director, District HR Support thomas.tomberlin@dpi.nc.gov
919-807-3440

Reducing the minimum length of reading camps
• Camps now referred to as reading camp
• Reading camp shall:
- Offer at least 72 hours of reading instruction to yield positive reading outcomes for participants
- Allow volunteer mentors to read with students at time other than during the 72 hours of reading instruction
-72 hours of reading instruction shall be provided over no less than 3 weeks for students in schools using calendars other than year-round calendars
-Funding reverts on October 31, 2014

Student Attendance in Summer Reading Camps
• parents or guardians of students not demonstrating reading proficiency shall be encouraged to enroll their student in a reading camp
– Parent/guardian shall make the final decision regarding student reading camp attendance
– LEAs shall provide a least one opportunity for students not participating in a reading camp to demonstrate reading proficiency appropriate for third grade students

Revises Good Cause Exemptions
• Limited English Proficient: students with less than two school years of instruction in an ESL program
• Students who demonstrate reading proficiency appropriate for third grade students on an alternative assessment approved by the SBE (took out wording on when to administer)
• Multiple retention language stays the same
• Portfolio: Student reading portfolio and review processes used by LEAs shall be approved by the SBE

Revises Good Cause Exemptions
• Students with disabilities whose IEPs indicate:
- the use of NCEXTEND 1
- at least a two school year delay in educational performance (guidance from EC division)
- receipt of intensive reading interventions for at least two school years

• BOG
• EOG
• EOG retest
• RtA Alternative Test
• Reading 3D at Level P (with comprehension)
• SBE approved Local Alternative Assessment
• Completed Portfolio (state or local)

Retained Student Placement
• Retained in third grade Accelerated reading class – third grade standards and curriculum
• Placed in 3/4 Transition Class with Retained Reading label – fourth grade standards and curriculum
• Placed in 4 th Grade Accelerated reading class with
Retained Reading label – fourth grade standards and curriculum

• SBE approved policy on process for submission in Sept.
• Proposals for local alternative portfolios must follow requirements of policy
• Reviewed by Portfolio Advisory Committee
• Recommendations to SBE by PAC

• Must be reported to SBE by Sept 1 and posted on district website
• Results will be included in the October state board report
• This report goes to the General Assembly on October 15

• Demonstrating and not demonstrating proficiency on EOG
• Exempt for Good Cause
• Passed Alternative Assessment (state or local)
• Students retained (placement in third grade, 3/4
Transition or 4 th grade accelerated)
• Charter – retained students who did not return

• November 1(SBE policy)
• RtA alternative test, local alternative test, completed portfolio, Reading 3D at Level P (can use all, if needed)
• After Nov. 1, principal can grade and classify
• MUST still show proficiency of third-grade standards (local alternative, portfolio, or Reading
3D at Level P)

• After Nov. 1, transition and accelerated classes continue
• No matter the results
• 90 minutes of uninterrupted reading instruction
• Recommended 30 more of interventions outside of the 90 minutes

 “…but shall continue to receive instructional supports and services and reading interventions…”
 Definition – means intentional strategies used with a majority of students to facilitate reading development and remediate emerging difficulty with reading development
 Include but not limited to – small group instruction, reduced teacher/student ratios, frequent progress monitoring, and extended learning time

Tammy Howard, Director
Accountability Services

“The State Board of Education shall (i) provide several valid and reliable alternative assessments to local school administrative units upon request, (ii) approve valid and reliable alternative assessments submitted by local school administrative units, and (iii) establish achievement level ranges for each approved alternative assessment. The State Board of Education shall annually review all alternative assessments to ensure ongoing relevance, validity, and reliability ."

 Reviewed assessments submitted by LEAs for the
2013-14 school year
 Reviewed technical information
 Identified Lexiles as a meaningful way to compare the assessments
 Linking studies completed for most of the assessments
 Allows comparison to the Level 3 cut point on EOG (439) with a Lexile of 725

 Recommending assessments with Lexiles
 Recommending assessments without Lexiles as pending the completion of a linking study
 Case Assessments by TE21
 Discovery Education
 Recommending not including item banks
 SchoolNet, ClassScape and Study Island
 However, districts may arrange a Lexile linking study and submit for approval
 Also, selections may be used to build a local portfolio (must be approved)

Garron Gianopulos, Psychometrician
Test Development

*The total count excludes records that did not have an EOG3 score and students who transferred schools.

(2013)
BOG3 Score Distribution
• All third graders with BOG3 scores and EOG3 scores within a school were included.
• Level 1 was divided into three roughly equal sized groups.
• Total count of 103,394.

Highlighted cells indicated most likely achievement level given a student’s
BOG3 scale score.

69% of students from the scale score range 408 to 421 on the BOG3 stayed within
Achievement Level 1 on the
EOG.

23% of students from the scale score range 408 to 421 on the
BOG3 reached Achievement
Level 2 on the EOG.

8% of students from the scale score range 408 to 421 on the BOG3 became grade level proficient.

25% of students from the scale score range
422 to 427 on the BOG3 became grade level proficient.
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

52% of students from the scale score range
428 to 431 on the BOG3 became grade level proficient.
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

83% of students from the scale score range
432 to 438 on the BOG3 became grade level proficient.
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example
Questions:
1. What was the most likely
EOG achievement level for a student scoring a level 2 on the
BOG3?
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example
Questions:
1. What was the most likely
EOG achievement level for a student scoring a level 2 on the
BOG3?
Answer: Level 4
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example
Questions:
2. What was the most likely
EOG level for a student scoring at level 3 on the
BOG3?
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example
Questions:
2. What was the most likely
EOG level for a student scoring at level 3 on the
BOG3?
Answer: Level 4
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example
Questions:
3. What was the likelihood that a randomly chosen student who scored a level 2 on the
BOG3 reached a level 4 or 5 on the EOG3?
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

3. What was the likelihood that a randomly chosen student who scored a level 2 on the
BOG3 reached a level 4 or 5 on the EOG3?
Answer: 62%
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

Example
Questions:
4. What is the probability that a randomly chosen student who scored a level 2 on the
BOG3 in 2014 will reach level
4 or 5 on the
EOG3 in 2015?
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

4. What is the probability that a randomly chosen student who scored a level 2 on the BOG3 in 2014 will reach level 4 or 5 on the EOG3 in 2015?
Answer: If instruction remains unchanged, then the expectation remains the same as 2013-14: 62%
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

Summary:
1. The percentages under the
EOG3 achievement levels sum to 100 and describe only that row. Do not sum the columns.
2. BOG3 Level 1 was disaggregated to help differentiate levels of instructional need.
3. Expectations for 2014-15 defined by this table assume students are randomly chosen within a BOG3 score range and that instruction remains unchanged from 2013-14.
4. If schools reduce effort for level 2 students, the 83% proficiency may drop for those students. Therefore, we need to maintain or increase instructional support for level
2 and above while at the same time increasing instructional support for students at level 1.
*The total count excludes records that did not have an EOG3 score and students who transferred schools.

Hope Lung, Section Chief
Test Development

 English II online mandated
 NCFEs available online Fall 2014: Eng III, Math
II, Math III, G7 Sci, Phy Sci, Earth/Env, G8 SS,
C&E, USH, WH
‒ Spring 2015: All
 EOG Grade 7 ELA/Reading & Math available online
 Projected Online Summative Assessment
Participation Survey (9/29)

 Required for online tutorials, released test forms/items, and summative assessments
 Options include:
‒ Secure browser for MS Windows & Apple OS X
(version same as 2013–14)
‒ Native app for iPad
‒ Chromebook (Chrome OS) version with special restrictions
‒ NEW NCTest Chrome App for Chromebooks,
Windows, and Macintosh workstations/laptops
(alternative to secure browser)

 Spring 2014 End-of-Year Survey
(Thank you!)
 Test Window Scheduler completed by LEA
TC/TA
‒ Enter start/end test window dates
‒ Select test(s) by “test type”: EOC, NCFE ELA,
NCFE math, NCFE science, NCFE social studies or
EOG
‒ Generates enrollment
‒ School TC can review but not edit

(continued)
 SIQ Tab
‒ All student SIQs visible on SIQ tab
‒ Reminder: Schools must check SIQ and record
SIQs for LEP and 504 only
 Start a test
‒ Select test type
‒ Filter by alphabetical range (e.g., A–C, D–F)

 Webinar being scheduled for week of
October 13
 LEA Technology Coordinator
 NCEducation Guides (screen shots for completing various tasks within NCEd)
 Regional Accountability
Coordinators/Regional Computer
Consultants







Curtis Sonneman, Education Consultant
Accountability Services

School Performance Grades
Assignment of Grades
North Carolina General Assembly legislation requires assignment of School Performance
Grades beginning with 2013–14 results
– Presented to State Board of Education for first time
February 5, 2015
• Legislation (budget bill) states “no earlier than January
15, 2015.”
– Available publicly in new NC School Report Card

School Performance Grades
Highlights of Legislation
 Calculated using 80%
Achievement and 20%
Growth
– Schools with no growth use 100% achievement
– Schools with no data receive no designation
 15 Point Scale 2013–14
(2014–15 and beyond uses 10
Point Scale)
 A separate achievement score and grade for reading and math must be reported for schools serving grades K–8
 Regional and charter schools assigned a “D” or “F” SPG must inform parents via a letter
( § 115C-238.66 and § 115C-
238.29F)

School Performance Grades
Highlights of Legislation
“If a school has met expected growth and inclusion of the school’s growth score reduces the school’s performance score and grade, a school may choose to use the school achievement score solely to calculate the performance score and grade.”
– May need to provide an opt in/out document for
Superintendents/Charter Directors to sign

School Performance Grades
Achievement Indicators
EOG Mathematics
EOG ELA/Reading
EOG Science
Math I (when applicable)
Biology (when applicable)
Math I
English II
Biology
The ACT
Math Course Rigor
ACT WorkKeys
Graduation Rate

School Performance Grades
Achievement Indicator Standards
Summative Assessment
Scores:
Percent of students who score at or above Level 3
(Grade Level Proficiency)
End-of-Grade Tests
End-of-Course Tests
Graduation Rate:
Percent of students who graduate in four years
(4-Year Cohort Rate)
Passing Math III:
Percent of graduates who successfully complete Math
III
The ACT:
Percent of 11 th grade students who score 17 or above (UNC System’s minimum composite score requirement)
ACT WorkKeys:
Percent of CTE concentrator graduates who achieve a
Silver Certificate or above

Growth Indicator and Reporting
Exceeds Expected
Growth
EVAAS School
Accountability
Growth
Composite Index
Score
Assigned an EVAAS
Growth Status reported with School Performance
Grades
Meets Expected Growth
Does Not Meet Expected
Growth
Generated Using End –of-
Grade and End-of-Course
Assessments
Converted to 100 Point
Scale for School
Performance Grades
50-100 Point Scale
Composites Capped

School Performance Grades
Calculating the Grade (Middle School Example)
School Achievement Score
Add all indicator numerators (met indicators standard) and divide by all indicator denominators (students included in indicators) to determine School Achievement Score
Indicator
EOG Reading (3–8)
EOG Math (3–8)
EOG Science (5 & 8)
EOC Math I
EOC English II
EOC Biology
The ACT (UNC System 17)
ACT WorkKeys (Silver or Better)
Passing Math III (Math Course Rigor)
4-year Graduation Rate
Total
Numerator
200
180
60
30
470
Denominator Score Total Achievement Score
300 66.7
300
100
35
60.0
60.0
85.7
735 63.9

School Performance Grades
Calculating the Grade (Middle School Example)
Growth Score
EVAAS School Accountability Growth Index score is converted to a 100 point scale and given a designation
Converted Score and Designation are reported on School
Report Card
Growth Score
Composite
Index Score
10.00
Converted
Score
100.0
Status
Exceeds Expected
Growth

School Performance Grades
Calculating the Grade (Middle School Example)
Final Score and Grade
• 80% Achievement Score and 20% Growth Score
• Set on a 15 Point Scale
• Calculated using data rounded to the nearest 10 th . Prior to assignment of grades and for reporting purposes scores are rounded to the nearest whole (except for growth, kept at 10 th for reporting.)
School Achievement
Growth
Score Multiply by Input for Final Grade
63.9
100.0
.80
.20
51.1
20.0
Final Score and Grade Reported
Reading Score and Grade Reported
Math Score and Grade Reported
71
67
60
B
C
C

School Performance Grades
Calculating the Grade (High School Example)
School Achievement Score
Add all indicator numerators (met indicators standard) and divide by all indicator denominators (students included in indicators) to determine School Achievement Score
Indicator
EOG Reading (3-8)
EOG Math (3-8)
EOG Science (5 & 8)
EOC Math I
EOC English II
EOC Biology
The ACT (UNC System 17)
ACT WorkKeys (Silver or Better)
Passing Math III (Math Course Rigor)
4-year Graduation Rate
Total
Numerator
175
223
198
162
50
244
238
1,290
Denominator Score Total Achievement Score
230
240
229
254
56
250
250
1,509
76.1
92.9
86.5
63.8
89.3
97.6
95.2
85.5

School Performance Grades
Calculating the Grade (Middle School Example)
Growth Score
EVAAS School Accountability Growth Index score is converted to a 100 point scale and given a designation
Converted Score and Designation are reported on School
Report Card
Growth Score
Composite
Index Score
-1.99
Converted
Score
70.0
Status
Meet Expected Growth

School Performance Grades
Calculating the Grade (Middle School Example)
Final Score and Grade
• 80% Achievement Score and 20% Growth Score
• Set on a 15 Point Scale
• Calculated using data rounded to the nearest 10 th . Prior to assignment of grades and for reporting purposes scores are rounded to the nearest whole
(except for growth, kept at 10 th for reporting.)
• School meets (exceeds) growth and growth lowers final score and grade, use achievement score (school option)
School Achievement
Growth
Score Multiply by
85.5
.80
70.0
.20
Input for Final Grade
68.4
14.0
Final Score with Growth (Grade)
Final Score no Growth (Grade)
Final Score and Grade Reported
82 (B)
86 (A)
86/A
93

School Performance Grades
Additional Notes
All accountability business rules are applied to school performance grades
– Banked scores for high school EOC as part of achievement calculation
After 2013–14 reporting in January 2015, future reporting is expected to occur at beginning of school year in conjunction to release of accountability results.
SPG will be one piece of the reporting. All accountability results will be reported through NC School Report Card.

Curtis Sonneman, Education Consultant
Accountability Services

Alternative School Accountability
Current Status
 According to General Statue for alternative schools
– “As part of its evaluation of the effectiveness of these programs, the State Board shall, through the application of the accountability system developed under G.S. 115C-83.15 and G.S. 115C-105.35, measure the educational performance and growth of students placed in alternative schools and alternative programs. If appropriate, the Board may modify this system to adapt to the specific characteristics of these schools.”

Alternative School Accountability
Current Status
 Current model for alternative schools is Local
Options Model for use in ABCs (GCS-C-013 and
GCS-C-022)
– With elimination of ABCs model current policy is out of alignment with READY and School
Performance Grades
– With creation of School Performance Grades a new model better suited for providing designations to alternative schools is warranted

Alternative School Accountability
Timeline of model development
 August State Board Meeting Discussion Item
– Gathered feedback from SBE regarding the intent for evaluating alternative schools
• Agreed in principle that these schools should have a different evaluation other than A–F letter grades
• Acknowledged that another model should be designed for alternative schools
 August-September
– Gathered input on an optimal model

Alternative School Accountability
Timeline of model development
 Present model ideas
– Accountability Coordinators Fall Meeting
• September 23, 2014
– North Carolina Association of Alternative Educators
Annual Conference
• September 26, 2014
– Alternative Schools Webinar
• September 29, 2014

Alternative School Accountability
Timeline of model development
 November State Board Meeting
– Propose new alternative schools accountability model effective with 2014–15
– Receive approval to remove alternative schools from A–F model for 2013–14
 December-January
– If approved, Local Options will not be collected
– Data collection processes will be reviewed and any changes will be made and provided to field

Alternative School Accountability
New Model Proposal
 New model will create three options for alternative schools to be evaluated
1. Participate in A–F system in exact same way as all other schools
2. Return all results back to sending schools and receive no designations
3. Participate in Alternative Progress Model

Alternative School Accountability
New Model Proposal
1. Participate in A–F system
– Schools elect to participate in A–F system and are evaluated in the same way as all other
“traditional” schools in letter grade system
• School/Superintendent/Local Board decides at beginning of each school year (designated date)
2. Return all test scores back to base school(s)
– School receives no designations, reporting notes all data sent back to base school
• School/Superintendent/Local Board decides at beginning of each school year (designated date)

Alternative School Accountability
New Model Proposal
3. Participate in Alternative Progress Model
‒ School/Superintendent/Local Board decides at beginning of each school year (designated date)
‒ Alternative Progress Model measures performance of school compared to itself in the previous year
‒ School given a designation for reporting purposes
• Progressing
» School increases score by at least specified amount over previous year
• Maintaining
» School maintains score between a specified amount +/- over previous year
• Declining
» School decreases score by at least specified amount over previous year

Alternative School Accountability
Alternative Progress Model Components
Student Persistence (20%)
‒ Student still in school at the end of the school year after attending an alternative school any time during the school year
• Measured as percent of students still enrolled in a school anywhere in
North Carolina at the end of the school year after appearing in any data collection at the alternative school
Student Achievement (20%)
‒ Percent students proficient based on 3-year average
• Total students numerator and denominator for 3-year
• Must be assessed at alternative school to be included in calculation
• Includes all available data used for calculating School Performance
Grades

Alternative School Accountability
Alternative Progress Model Components
Growth (60%)
 EVAAS Growth
– Working with EVAAS team to develop an alternative growth model
• Remove 140 day membership rule and calculate as a percentage for all students taking assessments at alternative school
• Use student enrollment data to provide percentage of student growth to alternative school if student is enrolled in another school in same LEA (remaining percentage is not attributed to other school)
• Use 3-years of or 3-year average growth measure to create index
» First 2 years will likely need to be 1-year result then 2-year average, before becoming 3-year rolling average

Alternative School Accountability
Calculating Total Score
In order to find the baseline and to calculate future numbers use following methodology
Component Component
Student
Calculation of Data for Current Year
Persistence
Student
Achievement
Growth
Score
85.6
14.7
64.8
Multiply
By
.20
.20
.60
Adjusted
Points
17.1
2.9
38.9
Year
2014
Calculation of Change for
Designation
2015
Points
38.4
41.8
Change +3.4
Rating
*
Improving
Total 41.8

Tammy Howard, Director
Accountability Services

 School Performance Grades
– Based on EOGs/EOCs, The ACT, ACT WorkKeys,
4-Year Cohort Graduation Rate, Passing Math III
– Based on 80 % Achievement and 20 % Growth
– Reported on NC Report Card
 Schools with Grades 3–8 will also have a reading and mathematics grade reported
 NC Report Card will be released

Released Data Format
(September 4)
2013–14 State, District, School Level EOG and EOC Report
LEA Name School Name
School
Code Subject Percent Level 1 Percent Level 2 Percent Level 3 Percent Level 4 Percent Level 5
2013–14 State, District, School Level Other Indicators Report
(Schools without grades 9–12 will only have data on EVAAS Growth Status)
LEA Name School
Name
School
Code
Percent
AMO
Targets
Met
EVAAS
Growth
Status
The ACT
Percent
ACT
WorkKeys
Percent
Passing
Math III
Percent
4- Year
Cohort
Graduation
Rate
Percent
5- Year
Cohort
Graduation
Rate
Percent
Graduation
Project
2013–14 State, District, School Level AMO Targets Report
LEA
Name
School
Name
School
Code Subject Subgroup Target Denominator Percent
Met
Target
Used
Alternate
Used
Improvement
Used
Confidence
Interval

Implementation of Five Achievement
Levels: Effective with 2013–14 Data
 Two Standards for Reporting
– Grade Level Proficiency (Level 3 and above)
• Used in School Performance Grades
• Reported on NC School Report Card except Annual
Measurable Objective Targets (AMOs)
– College and Career Readiness (Level 4 and above)
• Used for AMO targets (Baseline 2012–13)
• Reported on NC School Report Card

 Graduation Rate: 83.8
– Continuing to increase
– Also increased for all subgroups
 Comparing CCR (2012–13 to 2013–14)
– Increases for 10 of the 14 EOGs
– Math I and English II increased; Biology decreased
 The ACT and ACT WorkKeys increased

 Growth Increased
– A 3.2 increase; 71.5 to 74.7 met and exceeded growth
 Questions about growth
– How can a school have high achievement and not meet growth?
– Is it a given that a certain number of schools will have to be designated as not meeting growth

 NC School Report Card and Data Reports will include banked scores (Math I and Biology) for schools starting with 9 th grade
 Inclusion of Beginning-of-Grade 3 assessment in accountability reporting if achievement level is higher than End-of-Grade 3 ELA/Reading
 Credit by Demonstrated Mastery for EOC courses are included in accountability reporting if student successfully completes program in entirety (Phase 1 and Phase 2 in same accountability year)

 Per 115C-238.29F(l)
“A charter school shall ensure that the report card issued for it by the State Board of Education receives wide distribution to the local press or is otherwise provided to the public. A charter school shall ensure that the overall school performance score and grade earned by the charter school for the current and previous four school years is prominently displayed on the school Web site. If a charter school is awarded a grade of D or F, the charter school shall provide notice of the grade in writing to the parent or guardian of all students enrolled in that school.”



Testing Window Waiver: “For the 2014-2015 school year only…”
Exclusion of Growth: “If a school has met expected growth and inclusion of the school's growth score reduces the school's performance score and grade, a school may choose to use the school achievement score solely to calculate the performance score and grade.”

 Information on Blueprints
 Technical Documentation
 Alignment Study

 http://www.ncpublicschools.org/accountability/co mmon-exams/released-items/
 Items previously administered to students
 May not reflect all assessed standards or range of difficulty
 Includes correct answer, percent correct and standard/clarifying objective
 Includes list of standards/clarifying objectives addressed by the released items
Note: MSL Spring 2013 Released Forms to be reviewed and revised as needed
117

Item
Number
Example
MC
Example
CR
Type
MC
Key Percent
Correct
D
Standard
55% CCSS.ELA-Literacy.RL.11-12.2
CR Rubric 47% CCSS.ELA-Literacy.WHST.11-12.9.A
• Percent correct for multiple-choice (MC) item is the percentage of students who answered the item correctly.
• Percent correct for a constructed response (CR) item is the percentage of students who scored a 1 or 2.
118

 EOG, EOC and NCFE test specifications to be updated to include blueprint information (number or percent of items at each standard)
 http://www.ncpublicschools.org/accountability/ testing/generalinfo
 http://www.ncpublicschools.org/accountability/ common-exams/specifications/
 Technical reports will be posted to http://www.ncpublicschools.org/accountability/tes ting/technicalnotes
119

 Content analysis of assessments (fall 2014)
 EOG Grades 3-8 ELA/reading and math
 EOG Grades 5 & 8 science
 EOC English II, Math I, Biology
 Instructional alignment (survey)
 Early spring 2015
 Recruit approximately 700 teachers
 Professional development
 Summer 2015
120

 Security Audit
 Administration protocols
 Security processes
 Assessment Briefs
 http://www.ncpublicschools.org/accountabili ty/policies/briefs/
 Banked Scores
 Understanding the Five Achievement Levels
 Beginning-of-Grade 3 ELA/Reading Test

Iris Garner, NAEP Coordinator
Testing Policy and Operations

NAEP Overview
 Known as "the Nation's Report Card"
 Only nationally representative and continuing assessment of what America's students know and can do in various subject areas
 Authorized by Congress
 Administered by the National Center for Education
Statistics in the U.S. Department of Education

NAEP Overview
 Since 1969, assessments have been conducted periodically in reading, mathematics, science, writing, U.S. history, civics, geography, and the arts.
 Starting in 1990, NAEP assessments have also been conducted to give results for participating states.
 Since 1990, North Carolina has participated in the state-level assessments of NAEP.

NAEP Overview
State Board of Education Policy
HSP-A-001 (16 NCAC 6D .0302):
“To ensure adequate representation and generalizability of the data used to develop tests and to conduct evaluation studies, selected LEAs and schools, determined through stratified random samples, shall participate in field testing and other sample testing such as NAEP and other national or international assessments as designated by the department or the State Board of Education.”

NAEP Overview
 "No Child Left Behind" legislation stipulates that states, districts, and schools within districts that receive Title I funds must participate in NAEP if selected.
 Reading and mathematics must be administered every two years in grades 4 and 8.
 Students and schools that are selected to participate are kept confidential.

NAEP 2015
NAEP
Assessment
Window:
Jan. 26 –
March 6

National Assessment of Educational Progress (NAEP)
2013 Statistic Summary
North Carolina Performance Compared to the Nation’s Public Schools
Math
=
Grade 4
Reading
=
Grade 8
=
Scale Score
Grade 4
Reading
=
Grade 8
=
% At or Above Basic
Grade 4
Reading
=
Grade 8
=
% At or Above Proficiency
=
Math
=
=
Math
=
=
Legend
Above Nation
Same as Nation
Below the Nation
Score/percentage higher than previous administration (2011)
Score/percentage lower than previous administration (2011)
= Score/percentage same as previous administration (2011)

Curtis Sonneman, Education Consultant
Accountability Services

Participation Requirements
Current Information
 Memo sent to all superintendents/charter
–
–
– school directors
TNN of memo posted September 15, 2014 by Jaime Kelley
(Use Revised Memo)
Includes sample letter for Year 1 participation requirements
Explains LEA/charter school requirements for missing participation in each of 3-years in cycle
 NCDPI finalizing plan for Year 2 monitoring

Participation Requirements
Yearly Consequences Overview
 The minimum participation rate for all subgroups is 95%; must have a subgroup of
30 students to be applied
–
–
–
Year 1: Letter to parents with plan for improving participation
Year 2: School labeled as “consistently lowparticipating school” and develops an intensive action plan
Year 3: non-participating students (to meet
95%) counted as not-proficient

Participation Requirements
Year 1 Explanation

–
–
–
–
Year 1 Requirements
•
Within 30 days after accountability results are approved school must send letter to parents
Includes plan of action to ensure full participation for all subgroups targeting those missing participation
Within 30 days a copy of letter from each school, including plan of action, submitted to Regional
Accountability Coordinator (RAC)
RAC submits to NCDPI with in 35 days
Upon receipt NCDPI will contact schools not in compliance

Participation Requirements
Year 1 Explanation

–
Year 1 Timeline
October 2, 2014, Final results approved by SBE
– November 3, 2014, Letters due to parents and
RACs
– November 7, 2014, Letters due to NCDPI
– November 14, 2014, NCDPI begins sending notifications regarding non-compliance

Participation Requirements
Year 2 Explanation

–
–
–
Year 2 Requirements
•
Schools designated “consistently lowparticipating school”
In School Report Card and documentation on
Accountability website
•
•
Schools create and implement an intensive intervention plan
Plan submitted to LEA Testing Coordinator/RAC
(charters) on or before November 1, 2014
LEA Testing Coordinators/RACs will verify submission with Accountability Services
NCDPI implements state monitoring plan


Participation Requirements
Year 2 Explanation
NCDPI Monitoring Plan
–
–
•
•
Desk Monitoring
Ensure that all schools have submitted a plan to LEA Testing Coordinator/RAC
Randomly selected schools will be asked to submit plans to NCDPI
•
•
Pre-site review
Plans examined and schools identified for onsite review
Additional documentation may be requested

Participation Requirements
Year 2 Monitoring Plan

–
NCDPI Monitoring Plan (cont.)
•
•
•
On-site review
Schools selected will be visited by NCDPI staff
Visit will occur no later than 1 month prior to testing
Review/Discuss documentation submitted
» Strengths and weaknesses
–
•
•
State Response and Targeted Assistance
Provide recommendations and targeted assistance
Letter summarizing visit mailed to LEA superintendent/charter director within 90 days of onsite visit

Participation Requirements
Schools in both Year 1 and Year 2
 Schools with subgroups in both years must complete both requirements
‒
‒
Are not required to include information about Year 2 in Year 1 letter, but are encouraged to do so
Are not required to include Year 1 subgroups in intensive intervention plan for Year 2, but are encouraged to do so

Participation Requirements
Year 3 Explanation

–
–
Year 3 Requirements
•
•
NCDPI counts non-participating students as not proficient
Number added will be the number of students needed to raise participation to 95%
These students will be added to the denominator of the AMO performance target
Within 30 days after accountability results are
• approved school must send letter to parents
Includes plan of action to ensure full participation for all subgroups targeting those missing participation

Participation Requirements
Year 3 Explanation

–
–
–
Year 3 Requirements (cont.)
Within 30 days a copy of letter from each school, including plan of action, submitted to
Regional Accountability Coordinator (RAC)
RAC submits to NCDPI with 35 days
Upon receipt NCDPI will contact schools not in compliance

Ken Barbour
George Stubblefield
Analysis and Reporting

Accountability Information Technology Team
 Update on 20 th day files for 2015
 Setting up the New computers
 Moving Winscan to new computers
 Moving data to new computers
 When will we get Office?
 Who uses MYSQL in ReadyTools?
 Do you need training for Ready Tools
 Training other
 Questions on SSH
 Other questions
142

Accountability Information Technology Team
 A new WinScan32 release for 2014-15
 CCRAA Grade 10 – PLAN Alternate
 October testing window
 No scores available at this time (January 2015?)
 Scanned locally
 Note that the CCRAA Grade 11 (March 2015) will be sent to TOPS for processing
143

Accountability Information Technology Team
 Flexible Testing Waivers
 EOC
 Modify the “TestDate” to differentiate from Summer
 NC Final Exams: use Spring 2014 tests but scan using files in the NCFE14Fall folder
 Can use 2014-15 grade conversions
 Need to keep separate from the Dec. NCFE tests which will be new and have different answer keys and need new grade conversions
144

Accountability Information Technology Team
 For the EOC Flexible Testing Waivers and EOC Credit by
Demonstrated Mastery
 Continue to download V14 ODF files to \Scan1314\ODF
 English II Delay in Scoring
 When the new EOC and NCFE tests are available in December
(perhaps mid-November), the downloaded online files will be stored in \Scan1415\ODF and have a V15 file name extension
 Users might need to update the settings in READYTools
145