Consortium for
Educational
Research and
Evaluation–
North
Carolina
NC Race to the Top Evaluation
October 18, 2011
Gary T. Henry, Carolina Institute for Public Policy, UNC-CH
Julie Marks, Carolina Institute for Public Policy, UNC-CH
Trip Stallings, Friday Institute, NCSU
Laura Simpson, SERVE Center, UNCG
NC Race to the Top Evaluation
NC RttT is designed to be a “game changer”
A coordinated set of innovative activities and policy reforms designed to
collectively improve the performances of students, teachers, leaders,
and schools
Evaluation contributes to NC RttT “changing the game”
in two ways:
1. Program Evaluation: Provide formative information on the
implementation of NC RttT initiatives
Inform decisions to
improve implementation
2. Policy Evaluation: Assess – from the perspective of students,
teachers, leaders, and schools – the improvements that have
occurred as a result of NC RttT initiatives collectively and, to the
extent possible, the contributions of specific individual initiatives
Inform decisions about sustainability and impacts
Consortium for Educational Research and Evaluation–North Carolina
2
Focus of the Evaluation
• External evaluation provides objective analysis
of the activities described in NC’s RttT grant
proposal:
 implementation fidelity
 short-term outcomes
 collective/overall impact
• It is not an evaluation of specific teachers,
leaders, or schools
Consortium for Educational Research and Evaluation–North Carolina
3
NC RttT Evaluation: Team
Consortium for Educational Research and Evaluation-NC (CERE-NC):
SERVE Center, Carolina Institute for Public Policy, and Friday Institute
Steering Committee: Gary Henry, Ludy van Broekhuizen, & Glenn Kleiman
Principal Investigator: Gary Henry
Management Committee: Trip Stallings, Laura Simpson, and Julie Marks
Team Leaders:
Teacher and Leader Effectiveness - Laura Simpson
Equitable Supply and Distribution of Teachers and Leaders - Trip Stallings
Professional Development - Jeni Corn and Karla Lewis
Turnaround of LEAs and Schools - Charles Thompson
Local-Level Implementation and Spending - Eric Houck
Overall Impact - Gary Henry and Julie Marks
Other Leadership Roles:
LEA Coordinator - Alexa Edwards (aedwards@serve.org; 336 315-7436);
State Liaison - Trip Stallings (dtstalli@ncsu.edu; 919 513-8576)
Consortium for Educational Research and Evaluation–North Carolina
4
NC Race to the Top Evaluation:
Key Milestones in 2011
Spring 2011: Evaluation designed, staffed, and baseline data collection
initiated (e.g. interviews with RttT implementation leaders, Teacher &
Principal Survey, local expenditures, field work in schools that have
undergone turnaround)
Summer 2011: Scopes of work for evaluation of initiatives finalized;
Developing, piloting, & training for instructional observation instrument;
ongoing fieldwork
Fall 2011: Complete draft reports for: Baseline analysis of School
Turnaround, STEM, and Teacher Quality; formative assessment of PD
and RLA cost-effectiveness; briefings on analysis of teacher value-added
models; Begin analysis of baseline Teacher & Principal Survey
Winter 2011-12: Draft reports finalized and presented to SBE; continue Y2
SOW activities
Consortium for Educational Research and Evaluation–North Carolina
5
NC RttT Initiatives: Evaluation Organization
1. Teacher and leader effectiveness
Integration of value-added student achievement measures into
educator evaluation system
2. Equitable supply and distribution of teachers and leaders
Teach for America, NC Teacher Corps, Regional Leadership Academies,
Teacher Induction Program, Virtual Public School, Incentives
3. Professional development
All professional development activities in support of RttT initiatives,
including: PD for standards and assessment, IIS, and data use; and PD
delivery capacity-building efforts
4. Turnaround of LEAs and schools
Low-achieving LEAs and schools; STEM schools
5. Local-level implementation and spending on RttT
Cloud computing, allocation of RttT funds, cost savings
6. Overall impact of RttT on students, teachers, and school leaders
7. Omnibus Survey of Teachers and Principals
Consortium for Educational Research and Evaluation–North Carolina
6
1. Teacher and Leader Effectiveness
Purpose of this Evaluation Project
• To ensure quality, consistency, and fairness of
new and ongoing teacher and principal
evaluation processes through examination of
validity and reliability across multiple
observational perspectives
• To examine educators’ perspectives on new
evaluation standards and the effect of these
standards on educators’ practices
Consortium for Educational Research and Evaluation–North Carolina
7
1. Teacher and Leader Effectiveness
Evaluation Questions
• Have valid and reliable measures of student growth been
identified for inclusion in the teacher and administrator/principal
evaluation process?
• Does the revised evaluation process allow for/make meaningful
distinctions between teachers’ and administrators’ effective and
ineffective performance?
• How do educators view the implementation/rollout of the
evaluation process? Does the new evaluation process change
educators’ attitudes? Does it change educators’ practices?
• Do performance incentives for teachers in low-performing schools
have positive effects on student and teacher outcomes?
Consortium for Educational Research and Evaluation–North Carolina
8
1. Teacher and Leader Effectiveness
Evaluation Approaches
• Data sources: existing scholarship; quantitative data (including
longitudinal DPI data and survey data collected from principals,
teachers, and students); and qualitative data (including teacher and
principal interviews, focus group data, and observations of teachers).
Key Deliverables and Estimated Delivery
A. Teacher and Leader Effectiveness: Evaluation
Report: Preliminary evaluation of contractor's proposed approach to
measuring educator impact on student achievement (based on existing data)
3/2013
Final Report: Evaluation of new EES elements and their implementation
9/2014
B. Teacher and Leader Effectiveness: Incentives
Report: School-level bonuses
8/2013
Report: School-level and individual teacher bonuses
Final Report: Summative evaluation
6/2014
9/2014
Consortium for Educational Research and Evaluation–North Carolina
9
2. Equitable Supply and Distribution
of Teachers and Leaders
Purpose of this Evaluation Project
• To provide both summative and formative
information about RttT efforts to increase the
overall supply and to ensure the equitable
distribution of effective educators statewide
Evaluation Strands
• Baseline
• Regional Leadership
Academies
• NCTC & TFA Expansion
• Strategic Staffing
• New Teacher Induction
• NCVPS Blended STEM
Courses
Consortium for Educational Research and Evaluation–North Carolina
10
2. Equitable Supply and Distribution
of Teachers and Leaders
Overall Evaluation Questions
• What is the nature and quality of the experiences provided
by each of the six initiative programs?
• Are students affected by each of these programs better off
than students in schools and districts not served by these
programs?
• Are these initiatives cost-effective and sustainable?
• To what extent did the initiatives further the goal of having
an effective teacher in every classroom and an effective
principal in every school?
Consortium for Educational Research and Evaluation–North Carolina
11
2. Equitable Supply and Distribution
of Teachers and Leaders
Baseline
RLAs
NCTC/TFA
Strategic Staffing
Induction
NCVPS
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Consortium for Educational Research and Evaluation–North Carolina
X
X
X
X
X
X
Accounting Data
Quant Analyses
Artifact Review
Interviews
Surveys
Focus Groups
Observations
Evaluation Approaches
X
X
X
X
12
2. Equitable Supply and Distribution
of Teachers and Leaders
Key Deliverables and Estimated Delivery (I)
A. Evaluation of overall changes in distribution of higher-quality & effective teachers & leaders
Develop baseline estimates of the distribution of higher-quality teachers
and school leaders, revised from initial RttT proposal estimates.
Develop estimates of changes and trends in the distribution of higherquality teachers and school leaders.
2/2012
6/2014
B. Evaluation of Regional Leadership Academies
Cost-effectiveness analyses
Final 2012 activity report
Final 2013 activity report
Final Report
2/2012
2/2013
2/2014
9/2014
C. Evaluation of TFA Expansion & NC Teacher Corps (NCTC)
Report: Characteristics and placement of TFA and NCTC candidates
12/2012
Interim Report: NCTC impact on teacher retention
9/2013
Final Report: Impact, qualitative assessment, and policy recommendations
9/2014
Consortium for Educational Research and Evaluation–North Carolina
13
2. Equitable Supply and Distribution
of Teachers and Leaders
Key Deliverables and Estimated Delivery (II)
D. Evaluation of Induction Program for Novice Teachers
Preliminary report/briefing on 1st year implementation
12/2012
Report: Impact and implementation of the 1st year of the 2-year cycle
Final Report: Program effectiveness, implementation, and sustainability
7/2013
10/2014
E. Evaluation of Strategic Staffing Efforts
Report: Local SS plan and implementation review
9/2012
Report: State SS Y1 & Y2 review
9/2013
Final Report: Summative evaluation of local and state SS
9/2014
F. Evaluation of NCVPS Blended Courses
Initial Report: Estimates of blended course impact on teachers, students
4/2013
Report: Qualitative assessment of Y1 and Y2 course offerings
9/2013
Final Report: Impact, qualitative assessment, and policy recommendations
9/2014
Consortium for Educational Research and Evaluation–North Carolina
14
3. Professional Development
Purpose of this Evaluation Project
• To conduct ongoing analysis of the delivery
and quality of state- and local-level
professional development, with the goal of
analyzing the impact of the PDI on local
capacity, teacher practices, and student
achievement. We will examine longitudinal
education data combined with data collected
using a sample of schools approach.
Consortium for Educational Research and Evaluation–North Carolina
15
3. Professional Development
Major Evaluation Questions
• State Strategies: To what extent did the state implement and
support proposed RttT PD efforts?
• Short-Term Outcomes: What were direct outcomes of State-level
RttT PD Efforts?
• Intermediate Outcome: To what extent did RttT PD efforts
successfully update the NC Education Workforce?
• Impacts on student performance: To what extent are gains in
student performance outcomes associated with RttT PD?
Consortium for Educational Research and Evaluation–North Carolina
16
3. Professional Development
Evaluation Approaches
•
•
•
•
•
Interviews and focus groups
Observations
PD, teaching practices, PLCs, local PD Sessions
Research-based review of PD content and delivery, including reviews of the
NCDPI Online Repository and eLearning Portal diagnostics
Data reviews and analyses
administrative data, PDI-specific data (PDI
Participation Database data, survey data including PD Exit Surveys, leadership
inventory, reflection), NCEES summary data, student data including EOG/EOC,
graduation rates; LEA PD expenditure data
Document reviews
LEA PD Action Plans, other PDI-specific documents
Key Deliverables and Estimated Delivery
Baseline Report
1/2012
Annual Report: Status of PDI
12/2012
Annual Report: Status of PDI
12/2013
Final Report: Impact
Consortium for Educational Research and Evaluation–North Carolina
9/2014
17
4. Turnaround of LEAs and Schools
Purpose of this Evaluation Project
• To understand the extent to which and the ways in
which interventions by the District and School
Transformation division (DST) improve outcomes for
students in the state’s lowest-performing schools and
districts
• To explore the fidelity of implementation of the STEM
Schools initiative and examine its impacts on students,
teachers, principals, schools, and school networks.
Consortium for Educational Research and Evaluation–North Carolina
18
4. Turnaround of LEAs and Schools
Turnaround of LEAs and Schools Evaluation Questions
• What problems are identified in the low-performing schools and
districts?
• What are the main intervention strategies that the District and
School Transformation unit employs to improve low-performing
schools?
• What are the intended mechanisms of improvement?
• How do the strategies work? Do the strategies and mechanisms
play out as intended?
• What is the impact of the intervention strategies on intermediate
outcomes as well as student achievement and graduation rates?
Consortium for Educational Research and Evaluation–North Carolina
19
4. Turnaround of LEAs and Schools
STEM Anchor Evaluation Questions
• To what extent have the network of STEM anchor and cluster
schools been implemented as intended?
• What are the impacts of the network of STEM anchor and
cluster schools on student and on school-level outcomes and
how do these impacts compare with the impacts of other
transformation models?
• Can the impacts on student performance be disaggregated by
student and school characteristics?
• What mechanisms are put in place for the sustainability and
scaling up of the model, or its most successful elements?
Consortium for Educational Research and Evaluation–North Carolina
20
4. Turnaround of LEAs and Schools
Evaluation Approaches
•
DST: 30+ site visits, multiple interviews (over the next three years with
multiple people), observations, focus groups, surveys, artifact review,
document analysis, and quantitative analyses.
•
STEM: Predominantly qualitative analyses (observations of
professional development, site visits to STEM schools, interviews with
providers), with quantitative analyses (student and school staff surveys,
administrative data).
Key Deliverables and Estimated Delivery
A. Evaluation of District and School Transformation RttT Work
Report: Formative Assessments of the Efforts to Transform the Lowest-Performing Schools
12/2012
Annual Report
12/2013
Final Report: Quantitative and qualitative findings
9/2014
B. Evaluation of STEM Anchor School System Development
Baseline Scan and Year 1 report
12/2011 & 3/2012
Year 2 Report
12/2012
Year 3 Report
12/2013
Year 4 Report
9/2014
Consortium for Educational Research and Evaluation–North Carolina
21
5. Local-Level Implementation & Spending
Purpose of this Evaluation Project
• To determine how Race to the Top funding
is being allocated and used across districts
and schools throughout NC
Consortium for Educational Research and Evaluation–North Carolina
22
5. Local-Level Implementation & Spending
Cloud Questions
• To what extent does the Cloud reduce state & local expenditures
for technology?
• To what extent does the Cloud provide reliable, secure, accessible,
and efficient service?
• How satisfied are LEAs with the Cloud Computing infrastructure?
Local Spending Questions
• How do local districts spend RttT funds?
• Are some local RttT spending patterns associated with higher
student performance in schools and districts?
Local Efficiencies and Savings Questions
•
Do RttT funds alter costs incurred by the state and districts?
Consortium for Educational Research and Evaluation–North Carolina
23
5. Local-Level Implementation & Spending
Evaluation Approaches
• Mixed-method approach, combining document review, interviews,
surveys, and quantitative analysis of administrative data
Key Deliverables and Estimated Delivery
Conduct baseline analysis of local education expenditures in preparation for cost
analyses, which will specifically include the technology initiative
4/2012
Site reports on local RttT spending; survey findings; baseline productivity report
4/2013
Follow-up to 2012 report; interval and summative productivity report
9/2014
Consortium for Educational Research and Evaluation–North Carolina
24
6. Overall Impact
Purpose of this Evaluation Project
• To provide estimations of the overall impact of RttTfunded initiatives
• To explore under what conditions and circumstances
the initiatives collectively and in various combinations
appeared to be most effective, and for whom
• To consider sustainability options beyond the life of the
grant
• To track and compare the metrics/goals defined in the
proposal
Consortium for Educational Research and Evaluation–North Carolina
25
6. Overall Impact
Evaluation Questions
• Was each RttT initiative implemented as intended?
• What are the overall impacts of RttT on increasing student
performance, such as achievement, engagement, attendance,
graduation?
• Are the impacts of RttT on student performance larger in some
schools/districts than others (for example, high-poverty or lowperforming schools)?
• Are some RttT initiatives more effective in increasing student
performance than others?
• How can the successful RttT initiatives be sustained after 2014?
Consortium for Educational Research and Evaluation–North Carolina
26
6. Overall Impact
Evaluation Approaches
• The Overall Impact evaluation will consider selected quantitative
and qualitative data and results from all initiative-level
evaluations, as well as cross-initiative and all-inclusive data
Key Deliverables and Estimated Delivery
Overall cost and sustainability analysis of the RttT initiatives
Final Report: Synthesis and policy recommendations across initiatives
Consortium for Educational Research and Evaluation–North Carolina
12/2013
9/2014
27
7. Omnibus Teacher and Leader Survey
• Administered to probability sample of 358
schools across the state
• Assessing “Instructional Climate”

23 dimensions on leadership and
organizational conditions affecting instruction
• Baseline Fall 2011
• Next administration Feb/Mar 2012 and
annually throughout RttT Evaluation
Consortium for Educational Research and Evaluation–North Carolina
28
School Level Activities Report
Consortium for Educational Research and Evaluation–North Carolina
29