(A)(2) RttT Evaluation Part B Narrative, North Carolina, January 2014
Page 1 of 8
Race to the Top Progress Update
Sub-criterion (A)(2)/(A)(3)
Part B: In preparation for monthly calls, States must update applicable questions (i.e., those for which
there is new information) and provide appropriate documentation to substantiate its responses for all
relevant application sub-criterion (e.g. (A)(2) and (D)(4)). 1 All responses in this section should be
tailored to the goals and projects associated with this sub-criterion.
Application sub-criterion:2 (A)(2) Building strong statewide capacity to implement, scale up &
sustain proposed plans; (A)(3) Demonstrating significant progress in raising achievement and
closing gaps
STATE’s goals for this sub-criterion:
North Carolina’s self-described goals for this sub-criterion

Conduct ongoing evaluations to inform continuous improvement of Race to the Top
initiatives as well as to inform future program, policy, and funding decisions including
potential removal of policy barriers and the development of policies that support reforms.
Relevant projects:

Evaluation of Race to the Top Initiatives
Questions:
1. Is the State on-track to implement the activities and meet the goals and performance
measures that are included in its approved scope of work for this sub-criterion? If so,
explain why. If not, explain why not.
The North Carolina Race to the Top Evaluation Team has met all major targets through January
2014 and remains on target to meet all goals by the end of the grant period. Completion of two
2013 deliverables has been delayed (see Table 1, below), but both are on track to be completed
in spring 2014.
Since its last Progress Update (June 2013), the Team has started to wind down its formative
evaluation activities for each Race to the Top (RttT) initiative, as well as state and local-level
data collection activities (e.g., site visits, formal observations of activities and of individual
teachers, and initiative-specific and overall survey administrations). The Team has moved more
resources to final summative evaluation activities, many of which will be in initial internal draft
form in Spring 2014, with work on more comprehensive summative reports (for the overall,
teacher and leader effectiveness, professional development, District and School Transformation,
and local spending evaluations) extending through 2014 (pending approval of the Team’s no-cost
1
Note that States will only be required to submit documentation for the on-site program review, not for monthly calls. States
should work with their Program Officers to determine relevant state-specific documentation.
2
All highlighted fields will be pre-populated by the Department Program Officer prior to State completion.
(A)(2) RttT Evaluation Part B Narrative, North Carolina, January 2014
Page 2 of 8
extension request; see below), per the Evaluation work plan that is updated and submitted
monthly.
In addition, the Team continues to improve its data collection procedures to better meet the goals
of this sub-criterion. Revised data collection procedures include improvements in the approach to
collecting responses from educators from across the state for the Team’s fourth and final
statewide survey of a representative sample of schools (with 365 schools again expected to
participate this spring). Most important for participating schools will be the Team’s
implementation of a streamlined process for establishing school-level participant rosters that not
only will reduce administrative burdens on schools but also will improve data quality.
By the end of this final administration, the data from the survey—which already supports several
initiative-level evaluations (e.g., the Teacher and Leader Effectiveness evaluation, the District
and School Transformation evaluation, the Professional Development Initiative evaluation,
etc.)—will provide critical information for the final series of reports that will address the overall
impact of the RttT initiatives.
The Team also continues to maintain an online database for tracking interactions with schools
and LEAs (in an effort to minimize the evaluation burden placed on any given school or LEA).3
Finally, in addition to the 23 scheduled reports already submitted to the North Carolina
Department of Public Instruction (NCDPI) before the previous review in Summer 2013, the
Team has submitted thirteen more reports (Table 1, following pages), all of which have been
shared with stakeholder groups, and nine of which have been approved and finalized. To date
(January 2014), the Evaluation Team has finalized 32 reports, four are under review at DPI, and
one more is to be submitted to DPI for review by early March. The Team is on schedule to
complete an additional 16 reports4 by the end of the RttT period. All finalized reports continue to
be made available on the Team’s website.5
Of particular note, one deliverable that experienced multiple delays in finalization in 2012 and
early 2013—the baseline study of the statewide distribution of teacher quality—now is complete
and available publicly.6 The baseline it established will serve as an important component of the
series of final evaluation reports.
3
https://s3.amazonaws.com/evaluation.tools/evaluationdb/index.html; please be aware that there is an occasional delay when the
page initially loads; the Team also continues to update and maintains a dynamic list of contacts for each LEA and school, which
is accessible to all Evaluation Team members: https://s3.amazonaws.com/evaluation.tools/contactsdb/index.html
4
Listed at: http://cerenc.org/rttt-evaluation/executive-summaries/
5
http://cerenc.org
6
http://cerenc.org/wp-content/uploads/2013/12/Baseline-TQ-Report_FINAL_12-05-2013.pdf
(A)(2) RttT Evaluation Part B Narrative, North Carolina, January 2014
Page 3 of 8
Table 1. Submitted North Carolina Race to the Top Evaluation Reports (as of January 2014)
North Carolina
Teacher Corps
Supply and
Distribution of
Teachers and
Leaders
Strategic Staffing
Regional Leadership Academies
Teacher
Quality
Incentives
Distribution
Effective
Teachers and
Leaders
Report Title
Teacher and Leader Evaluation
Evaluation Strand
Comparing Value Added
Models for Estimating
Teacher Effectiveness:
Technical Briefing
Comparing Value-Added
Models for Estimating
Individual Teacher Effects on
a Statewide Basis: Simulation
and Empirical Analyses
Release Date
Available at:
February 2012
http://cerenc.org/wpcontent/uploads/2011/1
0/EE_VAM_Briefing_2
-7-12.pdf
October 2012
http://cerenc.org/wpcontent/uploads/2011/1
0/FullVAM_report_FINAL_8
-27-12.pdf
Measures of Student Growth
in the North Carolina
Educator Evaluation System:
Formative Evaluation Report
September 2013
Teacher Performance
Incentives in North Carolina
September 2013
The Distribution of Effective
Teachers in North Carolina
December 2013
Regional Leadership
Academies: Cost-Effectiveness
Framework
North Carolina Regional
Leadership Academies: Final
2012 Activity Report
North Carolina Regional
Leadership Academies: Final
2013 Activity Report
March 2012
March 2013
February 2014 (Anticipated)
Local Strategic Staffing in
North Carolina: A Review of
Plans and Early
Implementation
September 2012
State Strategic Staffing:
Recruitment Incentive for
Lowest-Performing Schools
September 2013
North Carolina Teacher
Corps Start-Up and Teach for
America Expansion: Initial
Findings on Recruitment,
Training, and Placement
North Carolina Teacher
Corps: Year One
Implementation Report
http://cerenc.org/wpcontent/uploads/2011/1
0/FINAL-Formativereport-TLEE-measuresof-student-growth-9-x13.pdf
http://cerenc.org/wpcontent/uploads/2011/1
0/FINAL-BonusIncentive-ProgramReport-8-29-13.pdf
http://cerenc.org/wpcontent/uploads/2013/1
2/Baseline-TQReport_FINAL_12-052013.pdf
http://cerenc.org/wpcontent/uploads/2011/1
0/RLA_cost_effectiven
ess_framework_3-112.pdf
http://cerenc.org/wpcontent/uploads/2011/1
0/RLA_First-YearReport-03-04-13.pdf
http://cerenc.org/wpcontent/uploads/2011/1
0/Strategicstaffing_1stYearReport_-FINAL-09-242012.pdf
http://cerenc.org/wpcontent/uploads/2011/1
0/FINAL-StateStrategic-Staffing-8-2913.pdf
October 2012
http://cerenc.org/wpcontent/uploads/2011/1
0/NCTC_PreliminaryR
eport_10-29-2012.pdf
November 2013
http://cerenc.org/wpcontent/uploads/2011/1
0/FINAL_NCTC_Seco
nd-Annual-Report_1107-2013.pdf
Supply and
Distribution of
Teachers and
Leaders (cont.)
Release Date
Available at:
New Teacher Support
Program
Evaluation Strand
Report Title
Page 4 of 8
North Carolina New Teacher
Support Program: First
Annual Race to the Top
Evaluation Report
April 2013
http://cerenc.org/wpcontent/uploads/2013/0
4/NTSP-First-YearReport-FINAL.pdf
North Carolina New Teacher
Support Program: Interim
Evaluation Report
March 2014 (Anticipated)^
NC Virtual Public School Blended
STEM Courses
(A)(2) RttT Evaluation Part B Narrative, North Carolina, January 2014
North Carolina Virtual Public
School Blended Learning
STEM Courses: A Formative
Assessment of Initial
Implementation, Part I
North Carolina Virtual Public
School Blended Learning
STEM Courses: A Formative
Assessment of Initial
Implementation, Part II
Prospects for Using Digital
Recording Systems for
Evaluation: An Overview
April 2013
Oct-13
October 2011
http://cerenc.org/wpcontent/uploads/2011/1
0/NCVPS-blendedcourseimpact_FINAL.pdf
http://cerenc.org/wpcontent/uploads/2011/1
0/NCVPS-blendedcourse-impact_Spring2013-follow-upreport_FINAL-10-042013.pdf
http://cerenc.org/wpcontent/uploads/2011/1
1/Report_ClassroomObservation-CameraTechnology_10-312011.pdf
First Annual Race to the Top Professional Development Evaluation Report
(in Three Parts):
A. Building LEA and Regional
Professional Development
Capacity: First Annual
Evaluation Report
Professional Development
B. Distinguished Leadership
in Practice (DLP): First
Annual RttT Evaluation
Report
C. Race to the Top Online
Professional Development
Evaluation: Year 1 Report
January 2012
http://cerenc.org/wpcontent/uploads/2012/0
1/PD_1st_Year_Report
_1-5-121.pdf
September 2012
http://cerenc.org/wpcontent/uploads/2011/1
1/NC-RttT_TDLPReport_-9-3-12.pdf
November 2012
http://cerenc.org/wpcontent/uploads/2011/1
1/OPD_Y1_FINAL_11
_9_12.pdf
Second Annual Race to the Top Professional Development Evaluation Report
(in Two Parts):
A. Statewide Face-to-Face
Professional Development
Formative Evaluation
March 2013
B. Local Outcomes Baseline
Study
March 2013
http://cerenc.org/wpcontent/uploads/2011/1
1/2nd-Annual-PDReport_statewide_0201-13.pdf
http://cerenc.org/wpcontent/uploads/2011/1
1/2nd-Annual-PDReport_localoutcomes_02-01-13.pdf
(A)(2) RttT Evaluation Part B Narrative, North Carolina, January 2014
Evaluation Strand
Report Title
Release Date
Page 5 of 8
Available at:
Third Annual Race to the Top Professional Development Evaluation Report
(in Two Parts):
Professional Development
(cont.)
A. Distinguished
Leadership in Practice (DLP):
Second annual Evaluation
Report
B. Third Annual Race to the
Top Professional Development
Evaluation Report
Turning Around LowestAchieving Schools
Turning Around North
Carolina’s Lowest Achieving
Schools (2006-2010)
STEM Anchor and Affinity Schools
District and
School
Transformation
Local Spending
Productive Connections:
Intervention in Low
Performing School Districts
by the NCDPI District and
School Transformation
Division, 2011-12
Initial Findings on the School
Leader Professional
Development Series
Turning Around North
Carolina's Lowest Achieving
Schools: Third Annual Report
North Carolina’s STEM High
Schools: An Overview of
Current Data
STEM Affinity Networks: Year
1 Report
STEM Affinity Network:
Second-Year Report
November 2013
http://cerenc.org/wpcontent/uploads/2011/1
1/FINAL_DLP-201213-Evaluation-Report11-7-13.pdf
February 2014 (Anticipated)
September 2011
http://cerenc.org/wpcontent/uploads/2011/1
1/DST_1st-yearReport_FINAL_12-052011.pdf
April 2013
http://cerenc.org/wpcontent/uploads/2013/0
4/DST-District-LevelReport-Y2-FINAL.pdf
April 2013
http://cerenc.org/wpcontent/uploads/2013/0
4/DST-PD-FINAL.pdf
May 2014 (Delayed; Revised
Anticipated Date)*
December 2011
April 2012
November 2012
STEM Affinity Network: Year
3 Report
December 2013
Local Education Agency Race
to the Top Expenditures: An
Initial Analysis
September 2012
Local Education Agency Race
to the Top Expenditures: An
Analysis of Fund Use and
Expenditure Patterns
June 2013
http://cerenc.org/wpcontent/uploads/2011/1
1/STEM_BaselineReport_11-02-2011.pdf
http://cerenc.org/wpcontent/uploads/2012/0
4/STEM_Affinity_Net
works_Year_1_report__4_4_12.pdf
http://cerenc.org/wpcontent/uploads/2011/1
1/STEM_SecondYear_Report_FINAL_1
1_13_12.pdf
http://cerenc.org/wpcontent/uploads/2013/1
2/RttT-STEM_Y3Report_FINAL_12-052013.pdf
http://cerenc.org/wpcontent/uploads/2011/1
1/NC-RttT_Localspending-baseline_9-412.pdf
http://cerenc.org/wpcontent/uploads/2011/1
1/LocalSpending_Y2_05-2913_Full-Report.pdf
(A)(2) RttT Evaluation Part B Narrative, North Carolina, January 2014
Evaluation Strand
Report Title
Summary: Formative
Assessments of Promising
Practices, 2010-2012
Overall Evaluation
Release Date
March 2013
North Carolina’s Race to the
Top Initiative: An Evaluation
Update
November 2013
A Preliminary Cost-Effectiveness Analysis of North
Carolina’s Race to the Top
Initiatives: Race to the Top
Formative Evaluation Report
April 2014 (Delayed;
Revised Anticipated Date)*
Page 6 of 8
Available at:
http://cerenc.org/wpcontent/uploads/2013/0
2/RttT-OverallFormativeSummaryMemo_02-01-13.pdf
http://cerenc.org/wpcontent/uploads/2013/0
2/Fall-2013-summaryof-RttT-evaluation-116-13.pdf
^NTSP: Due date for this second report (August 2013) was extended after delays in finalization of the first report
prevented completion of the second report by the original due date.
*DST: Initial draft anticipated for February 2014; Cost-Effectiveness: Initial draft anticipated for January 2014.
2. Does the State have evidence indicating the quality of implementation for this subcriterion? What is/has the State doing/done as a result of this information?
To ensure timely completion of all evaluation tasks, the state continues to rely on a monitoring
system that includes bi-weekly meetings with the Evaluation Team’s Steering Committee, a
monthly report from the Evaluation Team on its progress toward meeting all activities described
in the Scope of Work, and regular contacts among various RttT managers, initiative
implementers, and evaluators. These contacts are characterized by a regular flow of information
from the implementers about their programs and from the evaluators about evaluation progress
and informal findings, as well as cross-organizational collaborations, when appropriate. As noted
in previous reports, since fall 2012, the Evaluation Team has made significant changes to this
communications flow to ensure greater awareness on the part of all stakeholders by expanding
the list of contacts who are included in both formal and informal information exchanges about
each initiative’s evaluation activities and processes.
In addition, as noted above, the Evaluation Team continues to follow closely its four-year plan
for providing formal, periodic reports of formative and summative evaluation results, both at the
individual initiative level and at the overall RttT program level. Each report continues to go
through a series of reviews and revisions at various levels of leadership at NCDPI before
transmission to the final reviewing entity, the State Board of Education, to ensure that it is of the
highest quality before release. Once reports are finalized, they continue to be posted on a public
website7 that also includes information about planned content and expected completion dates for
all remaining reports; a link on the NCDPI website8 also directs readers to this page. The Team
continues to add one-paragraph summaries and separate executive summary documents for every
finalized report, as well as a separate page9 with a full list of all completed and upcoming reports
(also with links to executive summaries and full reports), to facilitate distribution of all findings
to a wider array of stakeholders.
7
http://cerenc.org/
http://www.ncpublicschools.org/rttt/reports/
9
http://cerenc.org/rttt-evaluation/executive-summaries/
8
(A)(2) RttT Evaluation Part B Narrative, North Carolina, January 2014
Page 7 of 8
To continue to fulfill the Team’s obligation to share RttT-related findings and to ensure that
those findings are thoroughly vetted, the Evaluation Team, NCDPI, and other initiative
implementers continue to work together not only to maximize the impact the reports have on
initiative implementation, but also to ensure those findings are shared with the State Board of
Education and the North Carolina General Assembly. For example, as in the prior year, the
Evaluation Team presented a summary of all findings to date to the State Board of Education in
November 2013. At that same meeting, the Team worked with implementers of the RttT-funded
Regional Leadership Academies to prepare a presentation for the Board that paired updates on
implementation with findings from the most recent evaluation report. Also in November, Team
members followed up on an April 2013 meeting with interested legislators (co-hosted by the
General Assembly’s Research division) about local-level RttT-funded strategic staffing
initiatives across the state by holding a second meeting for additional legislators. The Team is
scheduled to present several more reports to the State Board throughout the spring of 2014.
3. What obstacles and/or risks could impact the State’s ability to meet its goals and
performance measures related to this sub-criterion?
The state remains on track to meet all goals associated with the RttT evaluation by the dates
indicated in the current scope of work. However, approval of the state’s bid for a no-cost
extension through June 2015 would allow the Evaluation Team to extend completion of the
larger impact evaluations in order to include student achievement data—an important primary
outcome metric—from the 2013-14 school year. These include the impact evaluations of changes
to educator evaluation standards, the New Teacher Support Program, the state’s professional
development initiatives, efforts to turn around low-achieving schools, final assessments of local
spending, and overall impact of the RttT grant.
As always, timely and thorough completion of evaluation goals and performance measures are
dependent upon the implementation schedules for the RttT initiatives being evaluated. It is
important to continue to bear in mind that, on the current deliverables schedule, evaluation
activities for initiatives whose expected full implementation was extended to Y3 (2012-13) or Y4
(2013-14) will be limited due to the availability of, at most, one year of student outcome data,
though a no-cost extension could help to alleviate this limitation to some extent. The Team is
aware that it will need to continue to be flexible in its timelines over the closing months of the
grant, to ensure that the most up-to-date data are included in summative evaluations. By the same
token, the Team will benefit from ongoing support from NCDPI and the other implementers with
respect to providing the Team with essential data as quickly as possible—particularly student
outcomes data—as well as concordance tables that align current test items and results under the
state’s new standards with items and results related to the previous standards
Evaluation: Based on the responses to the previous question, evaluate the State’s performance
and progress to date for this sub-criterion (choose one)
Red (1)
10
Orange (2)
Yellow (3)
Green (4)10
Red – substantially off-track and/or has significant quality concerns; urgent and decisive action is required; Orange –off-track
and/or there are quality concerns; many aspects require significant attention; Yellow –generally on-track and of high or good
quality; only a few aspects require additional attention; Green – on-track with high quality.
(A)(2) RttT Evaluation Part B Narrative, North Carolina, January 2014
Page 8 of 8
Paperwork Reduction Act Statement
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such
collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to
average 74 hours (annually) per response, including time for reviewing instructions, searching existing data sources, gathering
and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this
collection is required to obtain or retain benefit (34 CFR 75.720, 75.730-732; 34 CFR 80.40 and 80.41). Send comments
regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this
burden, to the U.S. Department of Education, 400 Maryland Ave., SW, Washington, DC 20210-4537 or email
ICDocketMgr@ed.gov and reference the OMB Control Number 1894-0011.