(A)(2) RttT Evaluation Part B Narrative, North Carolina, June 2013
Page 1 of 8
Race to the Top Progress Update
Sub-criterion (A)(2)/(A)(3)
Part B: In preparation for monthly calls, States must update applicable questions (i.e., those for which
there is new information) and provide appropriate documentation to substantiate its responses for all
relevant application sub-criterion (e.g. (A)(2) and (D)(4)). 1 All responses in this section should be
tailored to the goals and projects associated with this sub-criterion.
Application sub-criterion:2 (A)(2) Building strong statewide capacity to implement, scale up &
sustain proposed plans; (A)(3) Demonstrating significant progress in raising achievement and
closing gaps
STATE’s goals for this sub-criterion:
North Carolina’s self-described goals for this sub-criterion
•
Conduct ongoing evaluations to inform continuous improvement of Race to the Top
initiatives as well as to inform future program, policy, and funding decisions including
potential removal of policy barriers and the development of policies that support reforms.
Relevant projects:
•
Evaluation of Race to the Top Initiatives
Questions:
1. Is the State on-track to implement the activities and meet the goals and performance
measures that are included in its approved scope of work for this sub-criterion? If so,
explain why. If not, explain why not.
The North Carolina Race to the Top Evaluation Team has met all targets through June 2013 and
remains on target to meet all remaining goals. Since its last Progress Update (November 2012),
the Team has continued its formative evaluation activities for each Race to the Top initiative, as
well as state and local-level data collection activities (e.g., ongoing site visits, formal
observations of activities and of individual teachers, and initiative-specific and overall survey
administrations); additionally, the Team has begun final summative evaluation activities,
including completion of a detailed scope of work for the final series of summative evaluation
reports (described in the November 2012 Progress Update and listed in the Evaluation Workplan
that is updated and submitted monthly).
Also, the Team continues to expand its data and communications resources to better meet the
goals of this sub-criterion. New data resources include responses from educators from across the
state who participated in the Team’s third statewide survey of a representative sample of schools
1
Note that States will only be required to submit documentation for the on-site program review, not for monthly calls. States
should work with their Program Officers to determine relevant state-specific documentation.
2
All highlighted fields will be pre-populated by the Department Program Officer prior to State completion.
(A)(2) RttT Evaluation Part B Narrative, North Carolina, June 2013
Page 2 of 8
(with 365 schools participating this Spring). These data continue to support several initiativelevel evaluations (e.g., the Teacher and Leader Effectiveness evaluation, the District and School
Transformation evaluation, the Professional Development Initiative evaluation, etc.), and, with
several administrations of the survey now complete, are now able to inform the development of
the scope of work for the evaluation of the overall impact of RttT initiatives.
In addition, the Team has followed up on its successful Fall 2012 launch of an online database
for tracking interactions with schools and LEAs (in an effort to minimize the evaluation burden
placed on any given school or LEA)3 with a series of presentations to school representatives in
each of the state’s eight education regions that not only explain the availability and functionality
of this tracking resource but also highlight findings from all finalized Evaluation Team
reports4—thus fulfilling the Team’s obligation to share Race to the Top-related findings not only
with state-level implementers but also with local-level educators who are directly impacted by
the various grant activities. As a result of these presentations, the Team was invited to conduct
several focused, follow-up meetings with other interested groups, including superintendents and
state representatives.
Earlier this year, in an effort to be more responsive to the information and formative feedback
needs of initiative implementers at the North Carolina Department of Public Instruction (NCDPI)
and other implementing organizations, each initiative-level evaluation group on the Team
expanded its list of contacts for all communications directly related to evaluation, from simple
day-to-day interactions to more significant discussions of evaluation methodology and findings.
Finally, in addition to the 20 scheduled reports already submitted to NCDPI before the previous
review in Fall 2012, the Team has submitted nine more reports (Table 1, following pages), all of
which have been shared with stakeholder groups, and three of which have been approved and
finalized. To date (June 2013), the Evaluation Team has finalized 23 reports, 3 are under review
at DPI, and 3 more are to be submitted to DPI for review in the first week of July, and the Team
is on schedule to complete an additional 21 reports5 by the end of the Race to the Top period. All
finalized reports continue to be made available on the Team’s website (http://cerenc.org).
One deliverable that continues to experience a delay in finalization is the baseline study of the
statewide distribution of teacher quality. Drafted and originally scheduled for completion by
November 2011, the report’s release was delayed in order to accommodate a request by NCDPI
to update the findings with value-added measures calculated using the state’s preferred valueadded model (the selection of which had not been finalized when the first draft of the report was
constructed). Data from the organization contracted by the state to provide official value-added
estimations were delivered in late fall 2012, and the revised draft was re-submitted to Race to the
Top leadership in June 2013.
3
https://s3.amazonaws.com/evaluation.tools/evaluationdb/index.html; please be aware that there is an occasional delay when the
page initially loads
4
The Team also continues to update and maintains a dynamic list of contacts for each LEA and school, which is accessible to all
Evaluation Team members: https://s3.amazonaws.com/evaluation.tools/contactsdb/index.html
5
Listed at: http://cerenc.org/rttt-evaluation/executive-summaries/
(A)(2) RttT Evaluation Part B Narrative, North Carolina, June 2013
Page 3 of 8
Table 1. Submitted North Carolina Race to the Top Evaluation Reports (as of June 2013)
Supply and
Distribution of
Teachers and
Leaders
Teacher
Incentive
Quality
s
Distribution
Regional Leadership
Academies
Strategic Staffing
Supply and
Distribution of
Teachers and
Leaders
North Carolina
Teacher Corps
Effective
Teachers and
Leaders
Teacher and Leader Evaluation
Evaluation Strand
Report Title
Comparing Value Added
Models for Estimating
Teacher Effectiveness:
Technical Briefing
Comparing Value-Added
Models for Estimating
Individual Teacher Effects
on a Statewide Basis:
Simulation and Empirical
Analyses
Measures of Student
Growth in the North
Carolina Educator
Evaluation System:
Formative Evaluation
Report
Teacher Performance
Incentives in North
Carolina
The Distribution of
Effective Teachers in North
Carolina
Regional Leadership
Academies: CostEffectiveness Framework
North Carolina Regional
Leadership Academies:
Final 2012 Activity Report
Local Strategic Staffing in
North Carolina: A Review
of Plans and Early
Implementation
State Strategic Staffing:
Recruitment Incentive for
Lowest-Performing Schools
North Carolina Teacher
Corps Start-Up and Teach
for America Expansion:
Initial Findings on
Recruitment, Training, and
Placement
North Carolina Teacher
Corps: Year One
Implementation Report
Submission Status and Release
Date (Anticipated)
Available at:
February 2012
http://cerenc.org/wpcontent/uploads/2011/10/EE_VA
M_Briefing_2-7-12.pdf
October 2012
http://cerenc.org/wpcontent/uploads/2011/10/FullVAM_report_FINAL_8-2712.pdf
(Submitted; September 2013)
(Submitted; September 2013)
(Revised version submitted;
Summer 2013)
March 2012
http://cerenc.org/wpcontent/uploads/2011/10/RLA_co
st_effectiveness_framework_3-112.pdf
March 2013
http://cerenc.org/wpcontent/uploads/2011/10/RLA_Fi
rst-Year-Report-03-04-13.pdf
September 2012
http://cerenc.org/wpcontent/uploads/2011/10/Strategi
c-staffing_1stYear-Report_FINAL-09-24-2012.pdf
(Submitted; October 2013)
October 2012
(Submitted; October 2013)
http://cerenc.org/wpcontent/uploads/2011/10/NCTC_
PreliminaryReport_10-292012.pdf
(A)(2) RttT Evaluation Part B Narrative, North Carolina, June 2013
New Teacher
Support Program
Supply and
Distribution of
Teachers and
Leaders
Report Title
North Carolina New
Teacher Support Program:
First Annual Race to the
Top Evaluation Report
NC Virtual Public School Blended
STEM Courses
Evaluation Strand
North Carolina Virtual
Public School Blended
Learning STEM Courses: A
Formative Assessment of
Initial Implementation,
Part I
North Carolina Virtual
Public School Blended
Learning STEM Courses: A
Formative Assessment of
Initial Implementation,
Part II
Prospects for Using Digital
Recording Systems for
Evaluation: An Overview
Submission Status and Release
Date (Anticipated)
Page 4 of 8
Available at:
April 2013
http://cerenc.org/wpcontent/uploads/2013/04/NTSPFirst-Year-Report-FINAL.pdf
April 2013
http://cerenc.org/wpcontent/uploads/2011/10/NCVPS
-blended-courseimpact_FINAL.pdf
(Submitted; October 2013)
October 2011
http://cerenc.org/wpcontent/uploads/2011/11/Report_
Classroom-Observation-CameraTechnology_10-31-2011.pdf
First Annual Race to the Top Professional Development Evaluation Report
(in Three Parts):
Professional
Development
A. Building LEA and
Regional Professional
Development Capacity:
First Annual Evaluation
Report
B. Distinguished
Leadership in Practice
(DLP): First Annual RttT
Evaluation Report
C. Race to the Top
Online Professional
Development Evaluation:
Year 1 Report
January 2012
http://cerenc.org/wpcontent/uploads/2012/01/PD_1st_
Year_Report_1-5-121.pdf
September 2012
http://cerenc.org/wpcontent/uploads/2011/11/NCRttT_TDLP-Report_-9-3-12.pdf
November 2012
http://cerenc.org/wpcontent/uploads/2011/11/OPD_Y
1_FINAL_11_9_12.pdf
Second Annual Race to the Top Professional Development Evaluation Report
(in Two Parts):
A. Statewide Face-toFace Professional
Development Formative
Evaluation
B. Local Outcomes
Baseline Study
March 2013
http://cerenc.org/wpcontent/uploads/2011/11/2ndAnnual-PDReport_statewide_02-01-13.pdf
March 2013
http://cerenc.org/wpcontent/uploads/2011/11/2ndAnnual-PD-Report_localoutcomes_02-01-13.pdf
(A)(2) RttT Evaluation Part B Narrative, North Carolina, June 2013
Turning Around LowestAchieving Schools
Evaluation Strand
STEM Anchor and Affinity Schools
District and
School
Transformation
Local Spending
Overall Evaluation
Report Title
Turning Around North
Carolina’s Lowest
Achieving Schools (20062010)
Productive Connections:
Intervention in Low
Performing School
Districts by the NCDPI
District and School
Transformation Division,
2011-12
Initial Findings on the
School Leader Professional
Development Series
North Carolina’s STEM
High Schools: An Overview
of Current Data
Submission Status and Release
Date (Anticipated)
September 2011
Page 5 of 8
Available at:
http://cerenc.org/wpcontent/uploads/2011/11/DST_1s
t-year-Report_FINAL_12-052011.pdf
April 2013
http://cerenc.org/wpcontent/uploads/2013/04/DSTDistrict-Level-Report-Y2FINAL.pdf
April 2013
http://cerenc.org/wpcontent/uploads/2013/04/DSTPD-FINAL.pdf
December 2011
http://cerenc.org/wpcontent/uploads/2011/11/STEM_
Baseline-Report_11-02-2011.pdf
April 2012
http://cerenc.org/wpcontent/uploads/2012/04/STEM_
Affinity_Networks_Year_1_repor
t_-_4_4_12.pdf
November 2012
http://cerenc.org/wpcontent/uploads/2011/11/STEM_
SecondYear_Report_FINAL_11_13_12.
pdf
Local Education Agency
Race to the Top
Expenditures: An Initial
Analysis
September 2012
http://cerenc.org/wpcontent/uploads/2011/11/NCRttT_Local-spending-baseline_94-12.pdf
Local Education Agency
Cost Assessment
June 2013
http://cerenc.org/wpcontent/uploads/2011/11/LocalSpending_Y2_05-29-13_FullReport.pdf
March 2013
http://cerenc.org/wpcontent/uploads/2013/02/RttTOverall-FormativeSummaryMemo_02-01-13.pdf
STEM Affinity Networks:
Year 1 Report
STEM Affinity Network:
Second-Year Report
Memo: Formative
Summary*
*Initial release anticipated for Fall 2012, when it was approved, but release was delayed until Spring 2013, when all
reports that informed this summary were finalized
2. Does the State have evidence indicating the quality of implementation for this subcriterion? What is/has the State doing/done as a result of this information?
To ensure timely completion of all evaluation tasks, the state continues to rely on a monitoring
system that includes bi-weekly meetings with the Evaluation Team’s Steering Committee, a
monthly report from the Evaluation Team on its progress toward meeting all activities described
in the Scope of Work, and regular contacts between various Race to the Top managers, initiative
implementers, and evaluators. These contacts are characterized by a regular flow of information
from the implementers about their programs and from the evaluators about evaluation progress
(A)(2) RttT Evaluation Part B Narrative, North Carolina, June 2013
Page 6 of 8
and informal findings, as well as cross-organizational collaborations, when appropriate. As noted
above, since Fall 2012, the Evaluation Team has made significant changes to this
communications flow to ensure greater awareness on the part of all stakeholders of evaluation
activities and processes by expanding the list of contacts who are included in both formal and
informal information exchanges about each initiative evaluation.
In addition, as noted above, the Evaluation Team continues to follow closely its four-year plan
for providing formal, periodic reports of formative and summative evaluation results, both at the
individual initiative level and at the overall Race to the Top program level. Each report continues
to go through a series of reviews and revisions at various levels of leadership at NCDPI before
transmission to the final reviewing entity, the State Board of Education, to ensure that it is of the
highest quality before release. Once reports are finalized, they continue to be posted on a public
website (http://cerenc.org/) that also includes information about planned content and expected
completion dates for all remaining reports, as well as on a separate site hosted by NCDPI
(http://www.ncpublicschools.org/rttt/reports/). Since the fall of 2012, the Team has added oneparagraph summaries and separate executive summary documents for every finalized report, as
well as a separate page (http://cerenc.org/rttt-evaluation/executive-summaries/) with a full list of
all completed and upcoming reports (also with links to executive summaries and full reports), to
facilitate distribution of all findings to a wider array of stakeholders.
The Evaluation Team, NCDPI, and other initiative implementers continue to work together not
only to maximize the impact the reports have on initiative implementation, but also, in recent
months, to ensure that findings and implementation adjustments related to those findings are
shared with the State Board of Education and the North Carolina General Assembly. For
example, the Evaluation Team contributed a summary of all findings to date to supplement a
presentation made by NCDPI to the House Education Committee in February 2013 on progress
towards implementing the Race to the Top initiatives, and in April 2013 team members cohosted with the General Assembly’s Research division an information session for legislators on
current local-level strategic staffing initiatives (many of which are RttT-funded) across the state.
In addition, the Team worked with professional development implementers to craft a two-part
presentation to the State Board of Education in March 2013 that paired updates on
implementation with findings from the second annual evaluation report on Race to the Topfunded professional development efforts.
3. What obstacles and/or risks could impact the State’s ability to meet its goals and
performance measures related to this sub-criterion?
The state remains on track to meet goals associated with Race to the Top evaluation. To ensure
timely completion of all deliverables, NCDPI and the Evaluation Team recently negotiated
earlier submission dates for several deliverables that are scheduled for finalization in the fall of
2013 and the fall of 2014, to minimize the possibility of delays in report finalization due to the
high volume of reports to be submitted for review and revision in both of these periods. For
example, under the original schedule, the Evaluation Team would have submitted as many as 12
report drafts in May 2014 for review and approval by the end of the grant period. Under the
revised schedule, the Team will submit six of those report drafts in March and April of that year
instead.
(A)(2) RttT Evaluation Part B Narrative, North Carolina, June 2013
Page 7 of 8
As always, timely and thorough completion of most evaluation goals and performance measures
continue to be dependent upon the implementation schedules for the Race to the Top initiatives
being evaluated. It is important to continue to bear in mind that evaluation activities that require
data on student outcomes for initiatives whose expected full implementation has been extended
to Y3 (2012-13) or Y4 (2013-14) will be limited due to either a reduction in (Y3) or absence of
(Y4) the student outcome data available before the end of the grant period. In recent months, the
Team has redesigned components of the evaluations for the Teacher and Leader Effectiveness,
District and School Transformation, and Professional Development initiatives to ensure that the
Team’s reports accommodate changes in implementation and also meet requests from the
implementation teams and from Race to the Top implementation leadership for new or additional
formative and summative feedback. The Team is aware that it will continue to need to provide
such flexibility in the closing months of the grant; by the same token, the Team will benefit from
ongoing support and flexibility on the part of implementers as it makes these adjustments and
begins its work on the final round of impact evaluations.
Evaluation: Based on the responses to the previous question, evaluate the State’s performance
and progress to date for this sub-criterion (choose one)
Red (1)
Orange (2)
Yellow (3)
Green (4)6 6
Red – substantially off-track and/or has significant quality concerns; urgent and decisive action is required; Orange –off-track
and/or there are quality concerns; many aspects require significant attention; Yellow –generally on-track and of high or good
quality; only a few aspects require additional attention; Green – on-track with high quality. (A)(2) RttT Evaluation Part B Narrative, North Carolina, June 2013
Page 8 of 8
Paperwork Reduction Act Statement
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such
collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to
average 74 hours (annually) per response, including time for reviewing instructions, searching existing data sources, gathering
and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this
collection is required to obtain or retain benefit (34 CFR 75.720, 75.730-732; 34 CFR 80.40 and 80.41). Send comments
regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this
burden, to the U.S. Department of Education, 400 Maryland Ave., SW, Washington, DC 20210-4537 or email
ICDocketMgr@ed.gov and reference the OMB Control Number 1894-0011.