The New Superintendents Induction Program
Annual Evaluation Report FY 2012
The New Superintendents Induction Program (NSIP) is a joint initiative of the Massachusetts
Association of School Superintendents and the Massachusetts Department of Elementary
and Secondary Education’s Office for Targeted Assistance
November 2012
NSIP Annual Evaluation Report: FY 2012
Table of Contents
Table of Contents
Executive Summary .................................................................................................................. i
Introduction .............................................................................................................................. 1
Profile of Participating Superintendents and their Districts .............................................. 1
Evaluation Goals and Data Sources................................................................................ 3
Report Organization ........................................................................................................ 4
Curriculum and Content Days ................................................................................................. 5
Key Findings ................................................................................................................... 5
Overall Value .................................................................................................................. 5
Curriculum ...................................................................................................................... 7
Assignments and Work Products ........................................................................................... 9
Findings .......................................................................................................................... 9
Superintendents’ Overall Perception ............................................................................. 10
Assignment-Specific Responses ................................................................................... 11
Coaching Support .................................................................................................................. 14
Key Findings ................................................................................................................. 14
Communication Frequency ........................................................................................... 15
Focus of Coaching Time ............................................................................................... 15
Coach Knowledge and Support..................................................................................... 17
Impact on Superintendent Capacity and Performance ........................................................ 19
Key Findings ................................................................................................................. 19
View of the Superintendency ........................................................................................ 19
Focused on Instructional Quality ................................................................................... 21
Superintendent’s Baseline Abilities ............................................................................... 22
Impact on Superintendent Abilities ................................................................................ 22
Impact on District Improvement ............................................................................................ 25
Key Findings ................................................................................................................. 25
District Baseline Conditions .......................................................................................... 25
Impact on District Practice and Capacity ....................................................................... 26
Participant Satisfaction and Engagement ............................................................................ 29
Conclusion .............................................................................................................................. 31
UMass Donahue Institute
Research and Evaluation Group
NSIP Annual Evaluation Report: FY 2012
Table of Contents
Appendix A – Assignments and Work Products .................................................................. 32
Report of Entry Findings ............................................................................................... 32
Theory of Action ............................................................................................................ 33
Strategy Development .................................................................................................. 34
Strategy Implementation ............................................................................................... 37
Classroom Visits and Principal Debriefings ................................................................... 38
Leadership Team Development .................................................................................... 39
Appendix B – Impact on Superintendents ............................................................................ 41
Instructional Leadership ................................................................................................ 41
Strategic Thinking ......................................................................................................... 41
Root Cause Analysis ..................................................................................................... 42
Strategic Resource Management .................................................................................. 43
Developing School Leaders .......................................................................................... 44
Leadership Team Development .................................................................................... 45
Relationships with Key Constituencies .......................................................................... 45
Supervision and Evaluation Systems ............................................................................ 47
Appendix C – Impact on Districts .......................................................................................... 48
District-level Focus on Instructional Improvement ......................................................... 48
District-level Strategy for Instructional Improvement ..................................................... 49
School-level Focus on Instructional Improvement ......................................................... 50
Principal Readiness to Lead Instructional Improvement ................................................ 50
Overall Quality of Instruction ......................................................................................... 51
Alignment of Budget to District Strategy ........................................................................ 52
Productivity of School Committee Meetings .................................................................. 53
Productivity of District Leader Team Meetings .............................................................. 54
Administrator and Teacher Evaluation Systems ............................................................ 55
UMass Donahue Institute
Research and Evaluation Group
NSIP Annual Evaluation Report: FY 2012
Executive Summary
Executive Summary
Introduction
The Massachusetts Association of School Superintendents (MASS) and the Massachusetts Department of
Elementary and Secondary Education (ESE) have entered into an innovative partnership intended to develop and
sustain district leadership and, by extension, enhance district capacity to support school improvement. The New
Superintendent’s Induction Program (NSIP) seeks to enhance superintendents’ effectiveness through an induction
process centered on “The Massachusetts Way.” The Massachusetts Way is a specific approach to practice
emphasizing five core objectives that reflect program designers’ understanding of how to enhance district capacity
to catalyze school improvement:

Building instructional leadership

Developing effective leadership teams

Developing collaborative relationships with key constituencies

Managing resources strategically

Developing and implementing effective systems for supervision and evaluation
The program delivers its curriculum through a series of Content Days and supports implementation of key
practices through related work assignments and intensive coaching by a team of former superintendents who are
themselves receiving extensive professional development in relation to the NSIP curriculum. The intended role of
the coach is primarily to support superintendents’ implementation of that curriculum.
NSIP is conceptualized as a three-year induction program designed to meet the needs of the Commonwealth’s
new1 superintendents, with a cohort of newly hired, first-year superintendents entering the program each July. The
first group of 25 superintendents (cohort 1) entered the program in July 2010. Of them, 22 continued into a second
year of the program2. A new group of 26 (cohort 2) entered the program in July 2011. As this report was being
developed a third cohort of 22 superintendents was entering the program. While Cohort 1 was limited to districts
that were in Level 3 or Level 4 accountability status at the time of program entry, later cohorts were also opened
to districts classified as Level 1 and Level 2, on a fee basis.
The University of Massachusetts Donahue Institute (UMDI) serves as NSIP’s external evaluator. The FY12 goals
of the evaluation reflect a shift toward measuring and understanding NSIP outcomes, while continuing to meet
program managers’ needs for timely formative feedback. This report summarizes key findings and lessons learned
through the second year of NSIP operation. It focuses on core findings emerging from surveys and group
interviews as well as ongoing informal observation of coaching sessions and content days.
Curriculum and Content Days
During the 2011-2012 academic year, cohort 1 met for five content days and cohort 2 met for eight content days.
While program designers started the year with curriculum content mapped to particular content days, the specifics
were subject to a considerable amount of adjustment and fine-tuning as the year progressed. Changes were based
both on experience built through previous content days as well coaches’ input which was informed by their
ongoing interactions with the participating superintendents. Adaptations were also made from year-to-year based
1
For the purposes of NSIP, new superintendents are defined as individuals with no previous experience as a superintendent or experienced
superintendents who are new to Massachusetts.
2 One of the initial 25, one chose to discontinue the program and the other two were no longer working as superintendents. A fourth
(included in the 22) resigned her position late in the 2011-2012 school year.
UMass Donahue Institute
Research and Evaluation Group
i
NSIP Annual Evaluation Report: FY 2012
Executive Summary
upon the experience of working with the previous cohort. The following bullets offer a summary of the
evaluation’s key findings with relation to curriculum and content days:
 Respondents from both cohorts generally rated the overall value of attending content days as excellent or
good. On the whole, cohort 1 superintendents were more positive in their responses than were cohort 2
superintendents. It is evident that introducing a focus on implementation of the new educator evaluation
system as a theme for cohort 1 superintendents made the content days feel more relevant for this group.
 While superintendents from both cohorts expressed the need for more support in management and operations,
program managers stress the innovative nature of the focus on strategy not operational issues.
 The diversity of preparation and previous experience of participants reinforces the need to continue to
differentiate content and instruction. It may also be effective to provide additional opportunities for
participants with strengths in particular areas to share with their peers.
 Additional work is needed to effectively integrate the readings and assignments into the content days. While
there is an acknowledged need for some flexibility, there should be a conscious effort to follow the planned
agenda as closely as possible and to make certain that assignments are explicitly addressed during the session.
Assignments and Work Products
In addition to the readings and assignments for each content day session, NSIP superintendents are expected to
complete a number of key “work products” through their participation in the program. They are: report of entry
findings, theory of action, district improvement strategy (development and implementation), classroom visits and
principal debriefings, and leadership team development. The following bullets offer a summary of the
evaluation’s key findings with relation to assignments and work products:
 Consistent with the findings for curriculum and content days, cohort 1 superintendents were generally more
positive than their cohort 2 counterparts. Most superintendents from both cohorts offered favorable ratings to
coaching support related to the assignments, the value of the resulting products to district improvement work,
and the value of the assignments for their own professional development. Comparisons to spring 2011
responses from cohort 1 suggest that the rationale for and value of NSIP assignments may become more
evident as individuals progress within the program and gain more experience in their positions. Responses of
cohort 2 superintendents further reinforce the previous observation that the program needs to improve the
explicit linkage and integration of assignments to the curriculum and content days.
 As a whole, superintendents perceived greater progress on these work products than did coaches. Yet with
only one exception (cohort 2 strategy implementation), at least half of all respondents described progress on
each product as either substantial or completed/refining. While not widespread, disagreements about the
extent of individual progress highlight the need to establish clearer shared understandings of progress between
superintendents and their coaches.
Coaching Support
In their first year, superintendents are allotted eight hours of one-on-one coaching time each month. In the second
year, this is scaled back to four hours per month. The following bullets offer a summary of the evaluation’s key
findings with relation to coaching support:
 Slightly more than half of coaching time was focused on work directly related to the NSIP curriculum and
about one-third was devoted to other issues of concern to the superintendents. Results for cohort 2 show a
marked increase in the percentage of time focused on work directly related to the NSIP curriculum and
assignments over the course of their first year in the program. This is consistent with several coaches
comments that they need to more closely adhere to the NSIP curriculum during their coaching sessions and
more effectively tie NSIP to the day-to-day work of the superintendent.
UMass Donahue Institute
Research and Evaluation Group
ii
NSIP Annual Evaluation Report: FY 2012
Executive Summary
 Superintendents expressed how highly they value the opportunity to address other issues through coaching as
a critical means of obtaining help dealing with concrete day-to-day issues that they are facing in their districts,
but are not addressed as part of the formal content day curriculum.
 Cohort 1 superintendents and coaches reported a substantial focus on developing effective systems for
supervision and evaluation – a finding that is consistent with the integration of educator evaluation
implementation as part of the year 2 curriculum. In contrast nearly one-quarter of cohort 2 superintendents
and more than one-third of their coaches reported spending little or no time discussing supervision and
evaluation systems. Although not directly responsive to supervision and evaluation systems, many
superintendents cited using coaching time to discuss staffing and personnel issues including addressing
conflict, hiring new staff, supporting building principals and improving leadership team effectiveness.
 Both superintendents and coaches expressed a need to focus coaching time in the following areas:

Spending more time in school buildings and classrooms

Continuing/increasing support for implementation of the new educator evaluation system

Focusing more time on strategy development, implementation and refinement

Establishing priorities among multiple initiatives and competing responsibilities

Increasing the focus on working with school committees

Spending more time developing effective leadership teams including issues related to hiring and
integrating new staff members
 Superintendents offered overwhelmingly positive ratings of their coaches’ knowledge and support. While it
would be inaccurate to describe the feedback regarding coaches’ performance as anything less than positive, it
is notable that cohort 2 ratings of support for implementation of NSIP practices and knowledge of specific
practices promoted by NSIP were somewhat lower than their ratings for other aspects. Although coaches have
undoubtedly become more familiar and comfortable with NSIP practices, it is important to acknowledge that
they require continued and ongoing support to improve their understanding of these practices, which represent
a new approach to the superintendency.
Impact on Superintendent Capacity and Performance
The evaluation plan for NSIP’s second year of implementation reflects an increased attention to outcomes
measurement and is intended to provide a foundation for evaluation over the next several years. One aspect of this
effort is gauging the extent to which superintendents are improving their own abilities to implement key aspects
of program in their districts. It is important to recognize that the survey data analyzed for this report reflect
interim measures of impact for a three-year program and thus reflect progress toward desired program outcomes.
The following bullets offer a summary of the evaluation’s key findings with relation to impact on superintendent
capacity and performance:
 Most superintendents indicated that NSIP has had a moderate or substantial impact on their view of the role of
the superintendent. However, about 40 percent of the superintendents from both cohorts reported that the
program has had little or no impact in this regard. Comments suggest that many superintendents feel that their
views of the role were already well-aligned with the perspective promoted by NSIP.
 About two-thirds of responding superintendents indicated that at least 60 percent of their time was focused on
activities and initiatives that had a direct impact on the quality of instruction in their district. Some continued
struggling to balance the time they spent on instructional leadership compared to management and operations.
Still, most acknowledged that NSIP has helped them focus more on instruction, though many felt that they
weren’t spending as much time as the program expects.
UMass Donahue Institute
Research and Evaluation Group
iii
NSIP Annual Evaluation Report: FY 2012
Executive Summary
 Nearly all cohort 1 superintendents reported that NSIP had at least modestly improved their ability to create
effective leadership teams, serve as an effective instructional leader, and think strategically about district
improvement. For most dimensions superintendents reported higher levels of impact than their coaches. The
exceptions are the superintendents’ ability to conduct root cause analysis of instructional problems and
establish effective systems for supervision and evaluation. Both cohort 1 superintendents and their coaches
reported the least impacts on the ability to manage resources in a strategic manner and forge collaborative
relationships with key constituencies.
 Cohort 2 superintendents and coaches generally reported the most impact in the ability to think strategically
about district improvement, conduct root cause analysis of instructional problems and serve as an effective
instructional leader. Superintendents reported relatively little impact on their ability to manage resources in a
strategic manner compared to coaches who rated this as one of the areas where the program had the most
impact. Both superintendents and coaches offered relatively low impact ratings for the ability to establish
effective systems for supervision and evaluation and forge collaborative relationships for key constituencies.
Impact on District Improvement
In addition to improving superintendents’ practice, NSIP is expected to positively impact districts as participating
superintendents influence system capacity and practice through application of new knowledge, skills and tools.
The following bullets offer a summary of the evaluation’s key findings with relation to district improvement:
 At least half of the cohort 1 superintendents reported their participation had resulted in modest or substantial
district improvement. They reported the most improvement for district-level strategy and principal readiness
to lead instructional improvement. Their lowest impact ratings were for teacher evaluation systems, school
committee productivity, and overall instructional quality. In general, the coaches’ perceptions of impact on
cohort 1 districts were well-aligned with superintendents’ perceptions. Compared to superintendents, coaches
indicated more improvement in district-level focus on instructional improvement and somewhat less
improvement in principal readiness, alignment of budget to strategy and administrator evaluation systems.
 For nearly all areas cohort 2 superintendents reported less impact than their coaches. The one exception was
productivity of district leadership team meetings, which nearly three-quarters of cohort 2 superintendents
reported had improved at least modestly as a result of their participation in NSIP. Cohort 2 superintendents
also gave relatively high impact ratings for district-level focus on and strategy for instructional improvement.
Impact in these areas was also highly rated by their coaches, who also rated alignment of budget to district
strategy as an area of relatively strong impact.
 Superintendents and coaches from both cohorts reported the least impact on teacher evaluation systems.
Cohort 2 superintendents and their coaches also reported relatively little impact on administrator evaluation
systems. This is reflective of the fact that the cohort 1 curriculum has included a deliberate focus on
implementation of the new educator evaluation system, which was not a strong component of the cohort 2
curriculum in 2011-2012. It is anticipated that the associated impact ratings for cohort 2 districts will increase
on the spring 2013 survey as they will be focusing more on educator evaluation in their second year of NSIP.
Participant Satisfaction and Engagement
Survey and interview data reveal high levels of satisfaction among cohort 1 superintendents with more mixed
reviews from those in the second cohort. Coaches perceived that about half of the superintendents were very
committed to the program. Despite cohort 1 superintendents’ highly positive feedback, coaches indicated that
nearly a third of them were only slightly committed to the program. Conversely, although cohort 2
superintendents tended to be more critical of the program, coaches reported that only 16 percent were less than
moderately committed participants.
UMass Donahue Institute
Research and Evaluation Group
iv
NSIP Annual Evaluation Report: FY 2012
Executive Summary
Conclusion
As the New Superintendents Induction Program completed its second year, participant and coach feedback
suggest that it was increasingly providing critical support to new superintendents’ development and district
improvement efforts. Most notably, participating superintendents continued to place considerable value on the
coaching provided to them. They offer overwhelmingly positive ratings of their coaches’ knowledge and support.
Although coaches have undoubtedly become more familiar and comfortable with NSIP practices over the past two
years, it is important to acknowledge that they require continued and ongoing support to improve their
understanding of these practices, which represent a new approach to the superintendency.
In service of that new approach, coaches are making a concerted effort to more closely adhere to the NSIP
curriculum during their coaching sessions and more effectively tie NSIP to the day-to-day work of the
superintendents. Yet, superintendents also view the coaching as a critical means of obtaining help with
challenging operational issues that are not being addressed through the formal content day curriculum. Evaluation
data suggest that many new superintendents struggle to balance the time they spend on instructional leadership
compared to management and operations. The lack of attention to these matters has been particularly frustrating
for a number of cohort 2 superintendents and has negatively colored their overall satisfaction with NSIP.
It is also evident that introducing a focus on implementation of the new educator evaluation systems as a theme
for cohort 1 superintendents made the content day sessions feel more relevant for this group. Maintaining some
focus on using educator evaluation as a driver of the district improvement strategy should remain a focus of their
participation in year 3 of the program. Those linkages should also be made for both cohort 2 and cohort 3
superintendents as a way to ground their learning in practical challenges that they are facing in their districts.
Although respondents generally rated the overall value of content days favorably, superintendents and coaches
identified a number of opportunities to improve those sessions including: more effectively integrating the readings
and assignments into the content day itself; improving on past efforts to differentiate content and instruction given
the diversity of preparation and previous experience among the participants; and strengthening coaches’ role in
content delivery.
As NSIP matures it will be important to develop and implement systems to evaluate the extent to which the
program is achieving its intended impact on superintendents and their districts. To date, most superintendents
indicate that the program has had a moderate or substantial impact on their view of the role of the superintendent.
Most also credit the program with yielding at least a modest improvement in various aspects of their own practice.
While many superintendents also reported that their participation has resulted in some improvements in district
capacity and practice, it is clear that more time is required to see NSIP principles effectively integrated throughout
the organizations in service of the ultimate goal of creating and sustaining improvements in student achievement.
UMass Donahue Institute
Research and Evaluation Group
v
NSIP Annual Evaluation Report: FY 2012
Introduction
Introduction
The Massachusetts Association of School Superintendents (MASS) and the Massachusetts Department of
Elementary and Secondary Education (ESE) have entered into an innovative partnership intended to develop and
sustain district leadership and, by extension, enhance district capacity to support school improvement. This
strategy aligns with ESE’s theory of action for district-wide school improvement and manifests in the form of the
New Superintendent’s Induction Program (NSIP). The program is grounded in the NSIP theory of action:
If, new superintendents are supported during the first three years of their tenure by an intensive
professional development curriculum focused on specific practices shown through research to support
school improvement . . .
And if, superintendents’ adoption of those research-based practices is supported by coaches whose role
is to support superintendents’ strategic thinking and implementation of the NSIP curriculum . . .
And if, those coaches are highly experienced former superintendents who have themselves received
rigorous training in relation to coaching techniques and the NSIP curriculum . . .
Then, participating superintendents will focus on and effectively address issues understood to have the
greatest impact on teaching and learning,
Resulting in, improved student achievement throughout district schools.
NSIP seeks to enhance superintendents’ effectiveness through an induction process centered on “The
Massachusetts Way.” The Massachusetts Way is a specific approach to practice emphasizing five core objectives
that reflect program designers’ understanding of how to enhance district capacity to catalyze school improvement:

Building instructional leadership

Developing effective leadership teams

Developing collaborative relationships with key constituencies

Managing resources strategically

Developing and implementing effective systems for supervision and evaluation
Annual survey data suggest that NSIP objectives are generally well aligned with superintendents’ priorities for
district improvement, with the strongest alignment found in relation to building instructional leadership,
developing effective leadership teams, and developing systems for supervision and evaluation.
The program delivers its curriculum through a series of Content Days and it supports implementation of key
practices through related work assignments and intensive coaching by a team of former superintendents who are
themselves receiving extensive professional development in relation to the NSIP curriculum. The intended role of
the coach is primarily to support superintendents’ implementation of that curriculum.
Profile of Participating Superintendents and their Districts
NSIP is conceptualized as a three-year induction program designed to meet the needs of the Commonwealth’s
new3 superintendents, with a cohort of newly hired, first-year superintendents entering the program each July. The
first group of 25 superintendents (cohort 1) entered the program in July 2010. Of them, 22 continued into a second
3
For the purposes of NSIP, new superintendents are defined as individuals with no previous experience as a superintendent or experienced
superintendents who are new to Massachusetts.
UMass Donahue Institute
Research and Evaluation Group
1
Introduction
NSIP Annual Evaluation Report: FY 2012
year of the program4. A new group of 26 (cohort 2) entered the program in July 2011. As this report was being
developed a third cohort of 22 superintendents was entering the program. While Cohort 1 was limited to districts
that were in Level 3 or Level 4 accountability status at the time of program entry, later cohorts were also opened
to districts classified as Level 1 and Level 2, on a fee basis.
As shown in Table 1, roughly half of the participating superintendents described their districts as serving
suburban communities. Among cohort 1, nearly one-third described their districts as urban, which is generally
consistent with the fact that entry into cohort 1 was limited to those districts in level 3 or 4 accountability status.
Even so, as of 2011 more than half of those cohort 1 districts had moved from level 3 to level 2, while all four of
the districts entering as level 4 remained as such. In both cohorts about 80 percent of districts can be characterized
as traditional K-12 districts serving one community. The majority of cohort 1 districts have at least six schools
whereas most cohort 2 districts have five or fewer.
In terms of their experience, most participants were completely new to the role of superintendent though several
have served in interim or acting roles or are experienced superintendents who are new to Massachusetts. About
half of the cohort 1 participants and two-thirds of cohort 2 participants were new to their district.
Table 1: FY12 NSIP Participant Profile
Cohort 1
Cohort 2
22
26
Urban
32%
19%
Suburban
55%
54%
Rural
5%
8%
Mixed
9%
19%
Traditional
82%
81%
Regional
14%
8%
Vocational
5%
12%
Level 1 or 2
55%
73%
Level 3 or 4
45%
27%
1-5
41%
62%
6-10
50%
31%
11 or more
9%
8%
Brand new superintendent
73%
92%
Was interim or acting
14%
4%
Experienced, new to Massachusetts
14%
4%
New to district
55%
65%
Previously worked in district
45%
35%
Number of Participants
Type of Communities
Served*
District Type
2011 District
Accountability Status
Number of Schools in
the District*
Experience in Role
Experience in District
* These data are taken from superintendent survey responses
4
One of the initial 25, one chose to discontinue the program and the other two were no longer working as superintendents. A fourth
(included in the 22) resigned her position late in the 2011-2012 school year.
UMass Donahue Institute
Research and Evaluation Group
2
NSIP Annual Evaluation Report: FY 2012
Introduction
Evaluation Goals and Data Sources
The University of Massachusetts Donahue Institute (UMDI) serves as NSIP’s external evaluator. The FY12 goals
of the evaluation reflect a shift toward measuring and understanding NSIP outcomes, while continuing to meet
program managers’ needs for timely formative feedback. This report summarizes key findings and lessons learned
through the second year of NSIP operation. It focuses on core findings emerging from surveys and group
interviews as well as ongoing informal observation of coaching sessions and content days.
Following is a brief description of individual data sources contributing to the development of findings highlighted
in this report:

Content Day Surveys: Participant surveys were administered in conjunction with four cohort 1 content days
and three cohort 2 content days. These surveys gathered superintendent and coach perspectives on each
session’s effectiveness in relating key concepts and on the relevance of described concepts and tools to
superintendent practice.

Group Interviews: During the January and March5 coaching and content days, UMDI conducted separate
group interviews with all NSIP coaches, eight cohort 1 superintendents and ten cohort 2 superintendents. The
primary purpose of these interviews was to obtain formative feedback to inform program improvement
largely by seeking to probe further about findings from the superintendent survey conducted in November
2011.Selected content from those interviews is integrated into this report. A full analysis of the data was
presented in a May 2012 management brief.

Surveys: UMDI administered several web-based surveys as described below. All survey responses are treated
as confidential, but not anonymous so as to allow the evaluation to link parallel measures over time as well as
to link individual coach and superintendent responses. Cohort 1 superintendents also completed a survey in
the spring of 2011 at the end of their first year in the program. Ultimately response rates of 100 percent were
achieved for all surveys.
Fall Superintendents Survey - In November 2011 each superintendent received a cohort-specific survey
focused on acquiring program feedback and describing baseline conditions (cohort 2) or early progress
(cohort 1) related to NSIP objectives. The surveys included ratings of superintendents’ own knowledge
and practice as well as district practice, teams and systems targeted by NSIP.
Spring Superintendents Survey – In June 2012 each superintendent received a cohort-specific year-end
survey that revisited measures from the fall survey as well as the spring 2011 survey for cohort 1. As with
the fall survey the objective was to gather program feedback and gauge progress relative to NSIP
objectives.
Coaches Survey – Also in June 2012 each NSIP coach was asked to complete a brief survey about each
of the superintendents with whom they worked. The survey questions addressed the coach’s perspective
of each superintendent’s level of engagement with and commitment to NSIP, performance relative to
NSIP work assignments and objectives, and identification of outside factors impacting the
superintendent’s participation or progress. In one case where a cohort 1 superintendent had recently been
assigned to a new coach, that coach respectively declined to respond about that particular superintendent
due to the relatively short time that they had been working together.

Participant Observation: Throughout the year, UMDI observed and participated in coach training and
content days, thereby gaining a rich understanding of how these program components function from a
logistical, curricular, and cultural perspective. Coach training days, in particular, provided significant
exposure to reflective discussion and strategizing in relation to program implementation.
5
The second round of group interviews for coaches and cohort 2 superintendents were originally scheduled for the February sessions, but
were postponed due to illness of the UMDI project manager.
UMass Donahue Institute
Research and Evaluation Group
3
NSIP Annual Evaluation Report: FY 2012
Introduction
Together, these sources offer substantial insight into superintendents’ experiences with NSIP and resulting early
impacts of those experiences. Although self-report data are inherently subjective, they are also potentially
accurate and informative, with confidence in findings increasing as sources are triangulated across informants and
data sources.
Report Organization
The NSIP FY12 Annual Evaluation Report utilizes the framework of the year-end superintendent surveys as a
basis for its organization. These surveys consist of seven distinct topical sections, as listed below. The report also
features program improvement suggestions and a conclusion. Data are presented at the program level with
breakouts by cohort as relevant to differences in their tenure or notable disparities in response.
Topical sections:

Curriculum and Content Days

Assignments and Work Products

Coaching Support

Impact on Superintendent Capacity and Performance

Impact on District Improvement

Participant Satisfaction and Engagement
Note that survey frequencies presented in this report are presented without decimal places. As a result of rounding
error there are tables where the reported frequencies do not sum precisely to 100 percent.
UMass Donahue Institute
Research and Evaluation Group
4
NSIP Annual Evaluation Report: FY 2012
Curriculum and Content Days
Curriculum and Content Days
As noted in the introduction, NSIP delivers its curriculum through a series of content days held throughout the
year. During the 2011-2012 academic year, cohort 1 superintendents met for five content days (October-May) and
cohort 2 superintendents met for eight content days (July-May). While program designers started the year with
curriculum content mapped to particular content days, the specifics were subject to a considerable amount of
adjustment and fine-tuning as the year progressed. Changes were based both on experience built through previous
content days as well coaches’ input, which was informed by their ongoing interactions with the participating
superintendents. Adaptations were also made from year-to-year based upon the experience of working with the
previous cohort.
Key Findings
 Respondents from both cohorts generally rated the overall value of attending content days as excellent or
good. On the whole, cohort 1 superintendents were more positive in their responses than were cohort 2
superintendents. Introducing a focus on implementation of the new educator evaluation system as a theme for
cohort 1 superintendents made the content days feel more relevant for this group.
 While superintendents from both cohorts expressed the need for more support in management and operations,
program managers stress the innovative nature of the focus on strategy not operational issues.
 The diversity of preparation and previous experience of participants reinforces the need to continue to
differentiate content and instruction. It may also be effective to provide additional opportunities for
participants with strengths in particular areas to share with their peers.
 Additional work is needed to effectively integrate the readings and assignments into the content day itself.
While there is an acknowledged need for some flexibility, there should be a conscious effort to follow the
planned agenda as closely as possible and to make certain that assignments are explicitly addressed during the
session.
Overall Value
As presented in Table 2, respondents from both cohorts generally rated the overall value of attending content days
as excellent or good. Consistent with other results to be presented in this report, cohort 1 superintendents were
more positive in their responses than were cohort 2 superintendents. It is not clear to what extent this trend is
related to underlying differences between the groups, the nature of the curricula for the two years, and/or the
perspective that cohort 1 superintendents may gain from making it through their first year in the role.
Interview data and comments to open-ended survey questions indicate that introducing a focus on implementation
of the new educator evaluation system as a theme for cohort 1 superintendents made the content day sessions feel
more relevant for this group. This is reinforced when looking at the spring 2011 survey responses of cohort 1
superintendents which show the proportion responding excellent or good increased from 82 percent to 95 percent.
Examples of some relevant survey comments include:
o
The goal setting assignment for the new educator evaluation was very helpful in allowing us
to begin to comprehend at least one step of the new evaluation system. Focusing on one item
at a time reduced the anxiety from the enormity of the whole undertaking.
o
Creating SMART goals under the new framework for evaluation was valuable in letting me
reflect on my own practice in a safe environment.
UMass Donahue Institute
Research and Evaluation Group
5
Curriculum and Content Days
NSIP Annual Evaluation Report: FY 2012
o
Sitting down with the coaches and taking the time to write my own SMART goal and receive
feedback was worthwhile.
Maintaining some focus on using educator evaluation as a driver of the district improvement strategy should
remain a focus of their participation in year 3 of the program. Those linkages should also be made for both cohort
2 and cohort 3 superintendents as a way to ground their learning in practical challenges that they are facing in
their districts.
Table 2: Overall value of attending Content Days
N
Excellent
Good
Fair
Poor
Unsure/No
Response
Spring 2012 Cohort 1
22
50%
45%
0%
0%
5%
Spring 2012 Cohort 2
26
35%
35%
27%
4%
0%
Spring 2011 Cohort 1
22
41%
41%
14%
0%
5%
* Spring 2011 data reported here include only responses for superintendents completing the most recent survey. As a result, they differ
somewhat from those previously reported.
Most notably, among cohort 2 respondents nearly one-third rated the overall value of attending content days as
fair or poor. Several cohort 2 superintendents expressed their feelings that being out of the district for eight fullday content sessions was an overwhelming and unreasonable expectation of a new superintendent. Additionally
three of those superintendents who rated the value as fair offered the following comments:
o
While the curriculum was coherent and carefully thought through, I’m not sure it was the
right curriculum. I’m not convinced that the tremendous focus on strategy is well-placed . . .
I don’t think the strategy work should be CUT, I just think it should be pared back to allow
for other foci. There are some very difficult, very important aspects of being new to the
superintendency that I feel got too little attention on the agenda like: how to work with a
School Committee; various approaches to budget development; school/town politics and
power relationships.
o
I didn’t find the content new or different from what we have experienced in any programs
leading to administrative roles. I would have benefitted far more by talking through
situations and on-the-job training kinds of discussions.
o
First year superintendents could use more time discussing everyday topics such as dealing
with school committees, mayors or town managers, budget issues and other areas relating to
the position.
It should be recognized that comments about the desire to shift some focus toward more “practical” topics was not
limited those superintendents quoted above. In two additional cases superintendents who rated the value more
positively made similar suggestions. Both were from cohort 2. Although not evident in the survey data, similar
comments were made by cohort 1 superintendents during the group interviews.
o
I think there are so many facets to being a superintendent that it would have been beneficial
to spend some portions of each day focusing on some of those other activities. For example:
difficult situations with community and/or parents; laws that are sometimes difficult to
implement; case studies; etc. (Respondent rated overall value of the content days as “good.”)
o
There was not enough practical information on specific responsibilities of superintendents
addressed during the content days . . . Perhaps a portion of the content days in the future
could be spent on the discussion of some of these issues and strategies for solving them. Tom
Scott does a one-day orientation for new superintendents, but it is not long enough to
UMass Donahue Institute
Research and Evaluation Group
6
Curriculum and Content Days
NSIP Annual Evaluation Report: FY 2012
address this concern. It would be better to address some of the realities of the job through
cohort discussion. (Respondent rated overall value of the content days as “excellent.”)
Program managers are quite aware of this desire among participating superintendents. From their perspective
training and support for management and operations issues are readily available through MASS and other
providers. In alignment with the expectations that programs funded through the federal Race to the Top program
should be innovative, they deliberately chose to focus NSIP’s curriculum on developing participants’ abilities to
develop and implement strategies focused on instructional improvement and student academic achievement.
Curriculum
Respondents from both cohorts generally agreed that the NSIP content day curriculum was effectively integrated
as part of a coherent curriculum, sequenced in a manner that supported learning and clear in its connection to
NSIP’s goals. As shown in Table 3, it is interesting to compare spring 2012 responses from each of the cohorts to
the spring 2011 responses from cohort 1. Clearly perceptions among cohort 1 superintendents improved from the
end of their first year in the program to the end of their second year in the program. This was especially true with
regard to curriculum sequencing for which agreement levels increased from 73 percent to 91 percent. Notably, the
perceptions of both cohorts were similar at the end of their first year in the program (spring 2011 for cohort 1 and
spring 2012 for cohort 2). The higher level of agreement related to sequencing for cohort 2 (77%) compared to
cohort 1 (73%) suggests that changes made by the curriculum design team resulted in some improvement in the
organization of year 1 content.
Table 3: Presentation of NSIP Curriculum
Percent Responding Strongly Agree or Mostly Agree
Cohort 1
Spring 2012
Cohort 2
Spring 2012
Cohort 1
Spring 2011*
Integrated effectively as part of a coherent curriculum
91%
81%
86%
Sequenced in a manner that supported learning
91%
77%
73%
Clear in its connection to NSIP’s goals
96%
80%
86%
NSIP content was . . .
* Spring 2011 data reported here include only responses for superintendents completing the most recent survey. As a result, they differ
somewhat from those previously reported.
Table 4 provides additional insight regarding superintendents’ reaction to Content Days and selected approaches
to delivering content. As noted previously, cohort 1 responses were more favorable than responses from cohort 2.
Nevertheless respondents from both cohorts were generally positive about the quality of facilitation.
Table 4: Content Day Ratings
Percent Responding Excellent or Good
Cohort 1
Spring 2012
Cohort 2
Spring 2012
Cohort 1
Spring 2011*
Quality of facilitation
91%
81%
82%
Effectiveness of panel discussions as a format for learning
91%
77%
59%
Connection of Strategy in Action readings to Content Days
82%
65%
73%
* Spring 2011 data reported here include only responses for superintendents completing the most recent survey. As a result, they differ
somewhat from those previously reported.
** A careful reader may notice that the spring 2012 frequencies in the first two rows of this table are identical to those in the previous table.
This is merely a coincidence. The figures for both tables have been checked against the survey data and they are accurate.
UMass Donahue Institute
Research and Evaluation Group
7
NSIP Annual Evaluation Report: FY 2012
Curriculum and Content Days
Comparing the spring 2012 results related to the effectiveness of panel discussions, reveals that cohort 1
superintendents found them to be more valuable than cohort 2 superintendents. However, it is noteworthy that
both groups’ responses were substantially more favorable than the spring 2011 cohort 1 responses reflecting
strong improvement in the effectiveness of panel discussions in the second year of the program. Interview data
reinforce the value of allowing participants with strengths in particular areas to share with their peers.
Finally, data from the spring 2012 survey show that both cohorts offered their least positive ratings for the
connection of Strategy in Action readings to content days. This is quite consistent with data from the content day
surveys administered throughout the year. It is clear that additional work is needed to effectively integrate the
readings and assignments into the content day itself.
The coaches’ interviews afforded an opportunity to gather their perspectives on the problem of integrating
assignments and readings into the content days. There was clearly some recognition that frequently work is
assigned and then not referred to during the content day. Although that assignment may have inherent value as a
resource for the day, coaches perceive that participants feel the value of their limited time is not being respected.
As a result those that complete the assignment resent that it wasn’t addressed and others who anticipate this
pattern simply don’t bother to do the work. The latter often exacerbates the problem in that the facilitators, who
may have planned an activity around the assignment, will come to realize that many people haven’t completed it
and rework the agenda “on the fly” to minimize or exclude that activity. There are other cases where the agenda is
modified to follow another interesting issue that arises during the day. While coaches acknowledged the need for
some flexibility in the content day agenda, most felt that there should be a conscious effort to follow the planned
agenda as closely as possible and to make certain that assignments and readings are explicitly addressed during
the session. There was also a sense that “last minute” adjustments in the expectations around assignments are
frustrating for both superintendents and coaches who have been diligently working between sessions to address
the assignment as originally conceived.
An issue not directly probed by the survey instruments, but evident in the interview data, was the need to continue
to differentiate content and instruction given the diversity of preparation and previous experience among the
participants. Experienced superintendents, and to some extent those who previously served assistant
superintendents, continued to express some frustration that, in their perception, the program assumes that
everyone is brand new to their role. As a result, they have found some of the content redundant. While reinforcing
the need for differentiation, this also calls for ongoing acknowledgement that participants bring different
backgrounds and strengths with them into their roles and that some components of the program will be more or
less relevant depending on each individual’s situation.
During the interviews, superintendents from both cohorts recognized the value of working with other
superintendents and the opportunity to consult with their own coach as well as the larger coaching contingent.
Based on year 1 feedback, the program did make an effort to increase the role of coaches in the content days so it
is interesting to note that cohort 2 superintendents who participated in the interviews expressed a desire from them
to share even more during the content days.
UMass Donahue Institute
Research and Evaluation Group
8
NSIP Annual Evaluation Report: FY 2012
Assignments and Work Products
Assignments and Work Products
In addition to the readings and assignments for each content day session, NSIP superintendents are expected to
complete a number of key work products through their participation in the program. Those products are: report of
entry findings, theory of action, district improvement strategy (development and implementation), classroom
visits and principal debriefings, and leadership team development.
Findings
 Consistent with the findings for curriculum and content days, cohort 1 superintendents were generally more
positive than their counterparts in cohort 2. Most superintendents from both cohorts offered favorable ratings
to the support that their coach provided in relation to the assignments, the value of the resulting products to
district improvement work, and the value of the assignments for their own professional development.
Comparisons to spring 2011 responses from cohort 1 suggest that the rationale for and value of NSIP
assignments may become more evident as individuals progress within the program and gain more experience
in their positions. Responses of cohort 2 superintendents further reinforce the previous observation that the
program needs to improve the explicit linkage and integration of assignments to the year 1 curriculum and
content days.
 As a whole, superintendents perceived greater progress on these work products than did coaches. Yet with
only one exception (cohort 2 strategy implementation), at least half of all respondents described progress on
each product as either substantial or completed/refining. While not widespread, disagreements about the
extent of individual progress highlight the need to establish clearer shared understandings of progress between
superintendents and their coaches.
 Since entry is conceptualized as a first-year activity, it is interesting that fewer than half of cohort 1
superintendents described themselves as having completed their Report of Entry Findings. In contrast,
nearly three-quarters of cohort 2 superintendents described themselves as completing their entry reports. It is
important to note that cohort 2 had the benefit of a more refined and better articulated first year curriculum. In
this context, the greater rate of entry report completion for cohort 2 may be understood as stemming from an
improvement of the entry planning curriculum as that group began the program. Many superintendents from
both cohorts cited entry planning as a beneficial work product. However, comments from several cohort 2
superintendents suggest that, despite adjustments to the curriculum, the program has not been completely
successful in addressing the lack of clarity around this task.
 About 70 percent of superintendents and slightly more than half of coaches described progress on the Theory
of Action as substantial or completed/refining. This is generally conceived as an activity to be completed
within the first year of NSIP. As such, it is noteworthy that nearly half of the coaches described their
superintendent’s progress as moderate at best – including three coaches who reported little or no progress on
the Theory of Action.
 About 90 percent of superintendents described progress on strategy development as substantial or
completed. Given the program’s emphasis on strategy, it is surprising that 38 percent of cohort 1 coaches
described their superintendents as making only moderate progress on this activity at the end of their second
year of NSIP. Several respondents commented on the benefits of components of strategy development,
including Root Cause Analysis, SWOT analysis, logic models, strategic objectives and initiatives.
 Most superintendents reported that building principals and central office administrators had been substantially
involved in their strategy development. In contrast only 20 percent reported that the school committee was
UMass Donahue Institute
Research and Evaluation Group
9
Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
substantially involved and 20 percent reported that the school committee had little or no involvement in the
process.
 More than half of the cohort 2 superintendents perceived themselves making at least substantial progress in
strategy implementation – a level of progress that would seem premature given their tenure in both the
program and their positions. Indeed, cohort 2 coaches reported that fewer than 30 percent of cohort 2
superintendents had made substantial progress in strategy implementation and that none had reached the stage
of completion.
 About two-thirds of superintendents from both cohorts perceived themselves as making substantial progress
or completing their work related to classroom visits and principal debriefings. Only two superintendents
(one from each cohort) indicated that they have made little or no progress on this effort. From a coaches’
perspective somewhat larger numbers of superintendents (four from cohort 1 and three from cohort 2) failed
to make much progress on this activity.
 Most superintendents reported that they had made either substantial progress or completed their work on
leadership team development. As might be expected given their longer tenure in their positions, larger
percentages of cohort 1 superintendents indicated that they had reached the completion stage for this task.
Superintendents’ Overall Perception
The survey asked superintendents to reflect on and rate NSIP work assignments on several dimensions, as
presented in Table 5. Again, spring 2012 responses from cohort 1 superintendents were generally more positive
than their counterparts in cohort 2. For both cohorts, the dimension with the most positive ratings was the support
provided by their coach in relation to assignments. This is quite consistent with overall positive perception of
coaching support, which will be presented in the next section of the report.
The value of the resulting products to district improvement work was rated favorably by most respondents. In
fact, if rank-ordered based on the percentage of favorable responses this dimension would be within the top two
for both cohorts. However it is also important to note that there is a 14 percentage point gap between them (96
percent for cohort 1 compared to 81 percent for cohort 2). The similarity of responses of cohort 2 when compared
to the spring 2011 responses from cohort 1 suggests that the value may become more evident as individuals
progress within the program and gain more experience in their positions.
Table 5: Work Assignment Ratings
Percent Responding Excellent or Good
Cohort 1
Spring 2012
Cohort 2
Spring 2012
Cohort 1
Spring 2011*
The rationale for NSIP assignments
82%
73%
73%
Clarity of expectations for assignments
90%
58%
68%
Connection of assignments to Content Days
95%
69%
86%
Support provided by your coach in relation to assignments
95%
92%
91%
Value of assignments to your professional development
86%
77%
68%
Value of resulting products to district improvement work
96%
81%
77%
* Spring 2011 data reported here include only responses for superintendents completing the most recent survey. As a result, they differ
somewhat from those previously reported.
UMass Donahue Institute
Research and Evaluation Group
10
NSIP Annual Evaluation Report: FY 2012
Assignments and Work Products
Results from both cohorts were also generally favorable with regard to value of the assignments to individual
professional development and for both cohorts the spring 2012 results were more positive than the spring 2011
responses from cohort 1. While the spring 2012 pattern of more favorable cohort 1 responses holds for the
rationale for NSIP assignments, we do see that cohort 2 results were similar to those from cohort 1 when
measured at the end of one year of program participation. This suggests that, as with the value to district
improvement work, the rationale may become more evident as superintendents gain more experience in the
program and their positions. A cohort 1 superintendent made the following supportive comment:
o
I found all the assignments to be helpful. My ability to utilize them is not always timely, but I
can put them into my repertoire and use them later.
Responses for the connection of assignments to content days and clarity of expectations for assignments
follow similar patterns. Spring 2012 survey results indicate that most cohort 1 superintendents responded
positively reflecting an improvement over their responses from the prior year. However, considerably fewer
cohort 2 superintendents rated these dimensions favorably and their ratings were much less positive than those of
cohort 1 superintendents at the end of their first year in the program. This reinforces the previous observation that
the program needs to improve the explicit linkage and integration of assignments to the year 1 curriculum and
content days. Related comments include:
o
It isn’t that any assignment or work product was not helpful it was more that we completed
assignments and then we did not spend sufficient time discussing them after. It set the stage
for my not valuing assignments.
Assignment-Specific Responses
Superintendents and coaches were asked to rate individual progress on each of the key work products using the
following categories: Little or No Progress, Moderate Progress, Substantial Progress, Completed and/or Refining6.
This question was a new addition to the spring 2012 survey and thus prior year data are not available for
comparison. Table 6 summarizes the percentages of superintendents and coaches describing progress as
substantial or completed for each key work product. Data were included only for those superintendents where
both self-report ratings and matched coaches’ ratings were available. Responses from both cohorts reveal that, as
a whole, superintendents perceived greater progress on these work products than did coaches. Yet with only one
exception (cohort 2 strategy implementation), at least half of all respondents described progress on each product
as either substantial or completed.
Cohort 1 superintendents’ self-reports show that they perceived the most progress in the areas of strategy
development, strategy implementation and leadership team development and the least progress on classroom
visits and principal debriefings. Cohort 2 superintendent self-reports show that they perceived the most progress
on their entry reports and strategy development efforts and the least progress on strategy implementation.
It is noteworthy that for half of the work products – Theory of Action, strategy development and classroom visits
– superintendent self-reports as well as coach perceptions suggest that cohort 2 superintendents made similar
progress to their cohort 1 counterparts. An interesting exception is the Report of Entry Findings. Both
superintendents and coaches reported that cohort 2 has made more progress on this activity, which is presented
among the very first elements in the year one curriculum. As such, this result in isolation seems counterintuitive.
However, it is important to note that cohort 2 had the benefit of a more refined and better articulated year one
curriculum resulting from the lessons learned through piloting the material with the first cohort. During the first
year of the program (2010-2011), the lack of clarity regarding the entry planning and reporting components of the
curriculum was quickly surfaced as a concern by the evaluation and through the coaches’ meetings. In this
context, the greater rate of entry report completion for cohort 2 may be understood as stemming from an
improvement of the entry planning curriculum as that group began the program. Another exception is leadership
6
In the interest of presenting a more readable narrative, “completed and/or refining” will be shortened to “completed” in this report.
UMass Donahue Institute
Research and Evaluation Group
11
Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
team development where coaches perceived similar progress across the cohorts, but cohort 1 superintendents selfreported more progress than cohort 2 superintendents. The final exception was strategy implementation where
both coaches and superintendents perceived more progress for cohort 1 – a difference that seems quite logical
given the cohorts’ relative tenures in the program and their positions.
Table 6: Summary of Progress for Key Work Products – Substantial or Completed/Refining
Percent Responding Substantial or Completed/Refining
Matched
Pairs
Cohort 1
Superintendents
Coaches
Matched
Pairs
Cohort 2
Superintendents
Report of Entry Findings
19
74%
63%
26
92%
77%
Theory of Action
21
71%
52%
26
69%
54%
Strategy Development
21
90%
57%
26
88%
62%
Strategy Implementation
21
86%
52%
26
54%
27%
Classroom Visits/Principal
Debriefings
21
62%
52%
25
68%
60%
Leadership Team
Development
21
86%
57%
26
69%
58%
Work Product
Coaches
* Data reported only for cases where both self-report ratings and matched coaches’ ratings are available. For all items this excludes one
cohort 1 superintendent in a new coaching relationship. There are other excluded cases where either the superintendent or coach did not offer
a response about a specific work product.
Table 7 summarizes the level of disagreement between superintendent self-report and coach perception of
individual progress for each of the key work products. The data were generated by comparing responses to
parallel survey questions for matched pairs of superintendents and their coaches. When comparing the perceptions
of individual superintendents and their coaches, it is important to note that there are no standardized definitions or
progress markers for the categories being used in this question. As such, there is considerable subjectivity.
Therefore, for the purposes of this report, disagreement is being defined as a difference in perception of at least
two levels. To illustrate, in the context of this analysis, matched ratings of “Little or No Progress” and “Moderate
Progress” are not coded as “Disagree,” but “Little or No Progress” and “Substantial Progress” are coded as
“Disagree.” Such differences are most likely to represent a true difference in perception as opposed to simple
nuances in individual interpretations of the category definitions.
Table 7: Disagreement Between Superintendent Self-Report and Coach Perception of Progress
Percent Disagreement (2 or more levels)
Cohort 1
Cohort 2
Report of Entry Findings
21%
8%
Theory of Action
29%
8%
Strategy Development
14%
12%
Strategy Implementation
10%
23%
Classroom Visits/Principal Debriefings
10%
8%
Leadership Team Development
10%
12%
UMass Donahue Institute
Research and Evaluation Group
12
NSIP Annual Evaluation Report: FY 2012
Assignments and Work Products
For cohort 1 pairs, the highest levels of disagreement related to progress on the Theory of Action (29 percent) and
the Report of Entry Findings (21 percent). For cohort 2 the main area of disagreement was progress on strategy
implementation (23 percent), which (as shown later in the report) is likely an unrealistically favorable perception
of progress among superintendents. While not widespread, these disagreements highlight the need to establish
clearer shared understandings of progress between superintendents and their coaches.
Appendix A provides more detailed views of the survey responses for each of the identified assignments and work
products.
UMass Donahue Institute
Research and Evaluation Group
13
NSIP Annual Evaluation Report: FY 2012
Coaching Support
Coaching Support
NSIP participants are supported in their implementation of the curriculum through content-focused coaching by
experienced former superintendents. In their first year, superintendents are allotted eight hours of one-on-one
coaching time each month. In the second year, this is scaled back to four hours per month. As they begin their
third year in the program, cohort 1 superintendents will participate in small group sessions led by an NSIP coach.
Key Findings
 Slightly more than half of coaching time was focused on work directly related to the NSIP curriculum and
about one-third was devoted to other issues of concern to the superintendents. Results for cohort 2 show a
marked increase in the percentage of time focused on work directly related to the NSIP curriculum and
assignments over the course of their first year in the program. This is consistent with several coaches
comments that they need to more closely adhere to the NSIP curriculum during their coaching sessions and
more effectively tie NSIP to the day-to-day work of the superintendent.
 Superintendents expressed how highly they value the opportunity to address other issues through coaching as
a critical means of obtaining help dealing with concrete day-to-day issues that they are facing in their districts,
but are not addressed as part of the formal content day curriculum.
 Cohort 1 superintendents and coaches reported a substantial focus on developing effective systems for
supervision and evaluation – a finding that is consistent with the integration of educator evaluation
implementation as part of the cohort 1 curriculum for year 2. In contrast nearly one-quarter of cohort 2
superintendents and more than one-third of their coaches reported spending little or no time discussing
supervision and evaluation systems. Although not directly responsive to supervision and evaluation systems,
many superintendents cited using coaching time to discuss staffing and personnel issues including addressing
conflict, hiring new staff, supporting building principals and improving leadership team effectiveness.
 Both superintendents and coaches expressed a need to focus coaching time in the following areas:

Spending more time in school buildings and classrooms

Continuing/increasing support for implementation of the new educator evaluation system

Focusing more time on strategy development, implementation and refinement

Establishing priorities among multiple initiatives and competing responsibilities

Increasing the focus on working with school committees

Spending more time developing effective leadership teams including issues related to hiring and
integrating new staff members
 Superintendents offered overwhelmingly positive ratings of their coaches’ knowledge and support. While it
would be inaccurate to describe the feedback regarding coaches’ performance as anything less than positive, it
is notable that cohort 2 ratings of support for implementation of NSIP practices and knowledge of specific
practices promoted by NSIP were somewhat lower than their ratings for other aspects. Although coaches have
undoubtedly become more familiar and comfortable with NSIP practices, it is important to acknowledge that
they require continued and ongoing support to improve their understanding of these practices, which represent
a new approach to the superintendency.
UMass Donahue Institute
Research and Evaluation Group
14
Coaching Support
NSIP Annual Evaluation Report: FY 2012
Communication Frequency
Through the survey, superintendents and their coaches were asked to report how often they had substantive
communication with each other. Most cohort 1 superintendents reported that they had substantive communication
with their coach once or twice a month. More than half of cohort 2 superintendents reported the frequency of
communication as every other week, 20 percent indicated monthly and 16 percent indicated that they had
substantive communication with their coach at least once a week. These findings are consistent with the decreased
time budgeted for coaching as superintendents progress through the program.
Table 8: Frequency of Substantive Communication between NSIP Coach and Superintendent
Less than Once
a Month
Monthly
Every other
week
Weekly (or more)
Self-Report
10%
40%
45%
5%
Coaches’ Perspective
10%
45%
45%
0%
Self-Report
8%
20%
56%
16%
Coaches’ Perspective
0%
36%
60%
4%
Cohort 1 Superintendents (N=20*)
Cohort 2 Superintendents (N=25*)
* Data reported for individuals with both superintendent self-report and coach ratings
While superintendent responses were fairly consistent with overall reports from the coaches, there were
discrepancies among individual coaches and their superintendents. For cohort 1, 45 percent of coaches and
superintendents agreed on the frequency of communication. The pairs where the reports differed were fairly
evenly split between cases were the coach reported more contact and the coach reported less contact. However,
there was less agreement (24%) among cohort 2 coaching pairs and the disagreement was actually skewed to
cases where the coach reported less contact with the superintendent. While the rating categories are well defined,
this may suggest some difference in interpretation of what constitutes “substantive” communication.
Table 9: Frequency of Communication - Comparison of Superintendent and Coach Reports
Cohort 1
Cohort 2
20
25
Coach reports more contact than superintendent
30%
32%
Coach and superintendent report same contact
45%
24%
Coach reports less contact than superintendent
25%
44%
Number of Superintendent/Coach Pairs
* The accuracy of the responses may be influenced by different recollections or interpretations of the term “substantive communication”
Focus of Coaching Time
Table 10 summarizes superintendent responses about the proportion of scheduled coaching time focused on
several types of activity – work directly rated to NSIP, issues of concern the superintendent, issues of concern to
the coach, and other issues. On average, slightly more than half of coaching time for both cohorts was focused on
work directly related to the NSIP curriculum, about one-third was devoted to other issues of concern to the
superintendents, and about 10 percent to other issues. For cohort 1, this distribution of time was quite consistent
with the results reported on previous surveys. Results for cohort 2 show a marked increase in the percentage of
UMass Donahue Institute
Research and Evaluation Group
15
Coaching Support
NSIP Annual Evaluation Report: FY 2012
time focused on work directly related to the NSIP curriculum and assignments over the course of their first year in
the program – from a mean of 42 percent in the fall to 58 percent in the spring.
Table 10: Focus of Coach-Superintendent Meeting Time – Mean Percentages by Cohort
Cohort 1
Cohort 2
Work directly related to the NSIP
curriculum and related assignments
54%
58%
Issues of concern to you that were not
directly related to NSIP work
36%
32%
Issues of concern to your coach that were
not directly related to NSIP work
9%
7%
Other focus
1%
3%
* Data reported for individuals with both self-report ratings and coaches’ perspectives.
These findings are consistent with information obtained through the group interviews and observation of various
discussions occurring during the coach training days. It reflects the coaches’ overall commitment to their
primarily role, which is to support understanding and implementation of the NSIP curriculum, while
acknowledging the need to allocate some proportion of coaching time to address other issues of concern. In most
cases those concerns are raised by the superintendents themselves. However, in some cases they may be raised by
the coaches in response to what they are observing in superintendent practice. Through interviews and openresponse survey items superintendents expressed how highly they value the opportunity to address other issues
through coaching as a critical means of obtaining help dealing with concrete day-to-day issues that they are facing
in their districts, but are not addressed as part of the formal content day curriculum.
The previous section on assignments and work products addressed the extent to which coaching time was focused
on each. The survey also asked respondents to characterize how much coaching time was spent working on issues
related to developing effective systems for supervision and evaluation. Cohort 1 superintendents and coaches
reported more time focused on this activity – a finding that is consistent with the integration of educator
evaluation implementation as part of the cohort 1 curriculum for year 2. Nearly one-quarter of cohort 2
superintendents and more than one-third of their coaches reported spending little or no time discussing
supervision and evaluation systems. Although not directly responsive to supervision and evaluation systems, many
superintendents cited using coaching time to discuss staffing and personnel issues including addressing conflict,
hiring new staff, supporting building principals and improving leadership team effectiveness.
Table 11: Focus of Coaching Conversations – Developing Supervision and Evaluation Systems
Little or No Focus
Moderate Focus
Strong Focus
Cohort 1 Superintendents (N=20*)
Self-Report
5%
55%
40%
Coaches’ Perspective
15%
45%
40%
Self-Report
23%
46%
31%
Coaches’ Perspective
35%
54%
12%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
UMass Donahue Institute
Research and Evaluation Group
16
Coaching Support
NSIP Annual Evaluation Report: FY 2012
Superintendents were asked to reflect on the extent to which the distribution of coaching time met their needs.
Most of them indicated that the way coaching time was used met their needs to a great extent (90% for cohort 1
and 73% for cohort 2).
Table 12: Focus of Coaching Conversations – Meeting Superintendent Needs
To what extent has the distribution of coaching time met your
needs as a superintendent?
N
Little or No
Extent
Moderate
Extent
Great
Extent
Cohort 1 Superintendents
20
0%
10%
90%
Cohort 2 Superintendents
26
4%
23%
73%
Both superintendents and coaches were asked to comment on what changes they would make if given the
opportunity to go back. In a separate question they were asked to think ahead to the next year and indicate what
coaching supports would be most useful. Although approaching the issue from slightly different angles, responses
to both questions address similar themes including:

Spending more time in school buildings and classrooms

Continuing/increasing support for implementation of the new educator evaluation system

Focusing more time on strategy development, implementation and refinement

Establishing priorities among multiple initiatives and competing responsibilities

Increasing the focus on working with school committees

Spending more time developing effective leadership teams including issues related to hiring and
integrating new staff members
Several coaches, in some cases with reluctance, commented on the need to more closely adhere to the NSIP
curriculum during their coaching sessions as well as more effectively tie NSIP to the day-to-day work of the
superintendent.
Coach Knowledge and Support
Through both formal data collection activity and informal conversations, superintendents offered overwhelmingly
positive reviews of their coaches’ knowledge and support. As shown in Table 13, spring 2012 survey responses
show that more than 90 percent of superintendents from both cohorts rated their coaches as excellent or good on
each aspect of knowledge and support – with substantial majorities offering excellent ratings.
While it would be inaccurate to describe the feedback regarding coaches’ performance as anything less than
positive, it is notable that cohort 2 ratings of support for implementation of NSIP practices and knowledge of
specific practices promoted by NSIP were somewhat lower than their ratings for other aspects. Similarly, at the
end of their first year, cohort 1 superintendents offered comparatively lower ratings of their coaches’ knowledge
of NSIP practices. Although coaches have undoubtedly become more familiar and comfortable with NSIP
practices, it is important to acknowledge that they require continued and ongoing support to improve their
understanding of these practices, which represent a new approach to the superintendency.
UMass Donahue Institute
Research and Evaluation Group
17
Coaching Support
NSIP Annual Evaluation Report: FY 2012
Table 13: Coach Knowledge and Support
Cohort 1
Cohort 2
Excellent
Good
Excellent
Good
Overall knowledge of the superintendency
95%
5%
96%
4%
Knowledge of specific practices promoted by NSIP
82%
18%
62%
38%
Past experience with issues that confront your district
82%
14%
81%
12%
Ability to translate her/his experience into insight or
feedback that you find helpful
86%
14%
81%
15%
Support for your implementation of NSIP practices
86%
14%
73%
23%
Support for your district improvement work, overall
86%
14%
84%
8%
Although not directly posed as a question on the survey, many superintendents took advantage of other openresponse questions to comment on the knowledge and support offered by the coaches. The following are some
examples:
o
It has also been good to work with other coaches during our NSIP workshops. It is good to
get other perspectives.
o
The coaching sessions have been very effective this year. We have been able to meet
individually with the coach and as a small group.
o
My coach’s positive feedback and questioning skills encourage me to make sound decisions
o
Working with my coach has been the most effective impact on my leadership abilities. Her
support and strategies have been the reason for my survival.
o
My coach was fabulous, a perfect match for me. He pushed my thinking, yet at the same time
he gave me confidence in my ability to do the job well.
o
My coach had a keen sense of when to support me and when to guide me.
o
My coach was most effective when she shared concrete suggestions for wading through both
NSIP assignments and items of concern for me in my district.
UMass Donahue Institute
Research and Evaluation Group
18
NSIP Annual Evaluation Report: FY 2012
Impact on Superintendent Capacity and Performance
Impact on Superintendent Capacity and Performance
The evaluation plan for NSIP’s second year of implementation reflects an increased attention to outcomes
measurement and is intended to provide a foundation for evaluation over the next several years. One aspect of this
effort is gauging the extent to which superintendents are improving their own abilities to implement key aspects
of program in their districts. It is important to recognize that the survey data analyzed for this report reflect
interim measures of impact for a three-year program and thus reflect progress toward desired program outcomes.
Key Findings
 Most superintendents indicated that NSIP has had a moderate or substantial impact on their view of the role of
the superintendent. However, about 40 percent of the superintendents from both cohorts reported that the
program has had little or no impact in this regard. Comments suggest that many superintendents feel that their
views of the role were already well-aligned with the perspective promoted by NSIP.
 About two-thirds of responding superintendents indicated that at least 60 percent of their time is focused on
activities and initiatives that will have a direct impact on the quality of instruction in their district. Focus
group participants continued to struggle to balance the time they spend on instructional leadership compared
to management and operations. Still, most acknowledged that NSIP has helped them focus more on
instruction, though many felt that they weren’t spending as much time as the program expects.
 Superintendents from both cohorts reported relatively strong baseline abilities in the area of instructional
leadership, strategic thinking and strategic management of resources. Cohort 1 superintendents also reported
strong baseline abilities in the area of creating effective leadership teams. Cohort 2 superintendents reported
relatively high baseline ability in developing instructional leadership skills among school leaders. With little
exception superintendents reported better baseline abilities than their coaches. Both superintendents and their
coaches reported relatively weak baseline abilities in the area of root cause analysis.
 Nearly all cohort 1 superintendents reported that NSIP had at least modestly improved their ability to create
effective leadership teams (95%), serve as an effective instructional leader (90%), and think strategically
about district improvement (90%). For most dimensions superintendents reported higher levels of impact than
their coaches. The exceptions were the superintendents’ ability to conduct root cause analysis of instructional
problems and establish effective systems for supervision and evaluation. Both cohort 1 superintendents and
their coaches reported the least impacts on the ability to manage resources in a strategic manner and forge
collaborative relationships with key constituencies.
 Cohort 2 superintendents and their coaches generally reported the most impact in the ability to think
strategically about district improvement, conduct root cause analysis of instructional problems and serve as an
effective instructional leader. Superintendents reported relatively little impact on their ability to manage
resources in a strategic manner compared to coaches who rated this as one of the areas where the program has
had the most impact. Both superintendents and coaches offered relatively low impact ratings for the ability to
establish effective systems for supervision and evaluation and forge collaborative relationships with key
constituencies.
View of the Superintendency
It is a common refrain that NSIP is intended to “transform the face of the superintendency” in Massachusetts by
instilling a greater imperative and set of skills in relation to the program’s core objectives. Superintendents were
UMass Donahue Institute
Research and Evaluation Group
19
Impact on Superintendent Capacity and Performance
NSIP Annual Evaluation Report: FY 2012
asked to rate the extent to which the program has changed how they think about the role of the superintendent.
Coaches were also asked to share their perspectives on changing views of the superintendents with whom they
work. Table 14 shows that this impact is most commonly regarded as moderate. However, it is interesting to note
that one-third of the cohort 1 superintendents reported that their view had changed only slightly as a result of their
participation in the program. Furthermore, among cohort 2 superintendents nearly 20 percent reported that NSIP
has had no impact on their view of the superintendency and nearly one-quarter reported only a slight impact.
Comments suggest that many superintendents feel that their views of the role were already well-aligned with the
perspective promoted by NSIP and thus don’t perceive that the program has had a strong impact on their views. In
general, coaches for both cohorts reported stronger impacts than did the superintendents.
Table 14: NSIP Impact on Superintendents’ View of their Role
No
Change
Slight
Change
Moderate
Change
Substantial
Change
Self-Report
5%
33%
48%
14%
Coaches’ Perspective
5%
19%
52%
24%
Self-Report
19%
23%
42%
15%
Coaches’ Perspective
8%
27%
54%
12%
Cohort 1 Superintendents (N=21*)
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
Both superintendents and their coaches were asked to elaborate on how their perspective on the role of the
superintendent changed as a result of NSIP. Most comments cited the program’s impact on their view of the role
as a strategic leader focused on the instructional core and student achievement.
o
My view is that the superintendency needs to focus on being an effective instructional leader
and how to plan and implement the strategies necessary to make change happen. I believe
that the role of the superintendent is to build capacity of all of the leaders around him or her
so that they can carry out the mission and vision of the district. (Cohort 1 Superintendent)
o
It has taught me the importance of having a focus for the work. It has helped me to maintain
the focus of student achievement. (Cohort 1 Superintendent)
o
I understand more fully how the Office of the Superintendent can impact instruction at the
core. (Cohort 1 Superintendent)
o
The program has broadened my thinking on how to implement strategy and involve more
constituents.(Cohort 1 Superintendent)
o
The takeaway concept from year 1 = “The role of the superintendent is to focus the attention
of the district.” This is such an empowering and different perspective. (Cohort 2
superintendent)
o
NSIP has consistently emphasized the importance of strategic thinking and planning . . . As
a superintendent, never have I had to be so cognizant of so many stakeholder groups and
interests, thus strategy is of the utmost importance.(Cohort 2 superintendent)
o
I am thinking ‘strategically’ thanks to the program. I am constantly using some of the skills I
learned (almost automatically) and staying focused on how this work impacts student
learning. (Cohort 2 superintendent)
UMass Donahue Institute
Research and Evaluation Group
20
Impact on Superintendent Capacity and Performance
NSIP Annual Evaluation Report: FY 2012
o
[The superintendent] has realized that the superintendent is a leader and human capital
developer. (Cohort 1 coach)
o
I believe NSIP has altered the superintendent’s thinking regarding the significant influence
he brings to bear on the quality of instruction that is carried out in the district. (Cohort 2
coach)
o
[This superintendent] came to the job expecting to lead the charge for improvement in
student achievement. I think NSIP has given her a greater understanding about how this has
to be accomplished through others. (Cohort 2 coach)
A few cohort 1 coaches also specifically noted a change from a school-level to a district level-focus.
o
He has moved from a predominantly school based perspective to district based.
o
Thinking strategically from a district perspective.
o
I believe [this superintendent] now has the ability to look at issues from a district view.
Focused on Instructional Quality
About two-thirds of responding superintendents indicated that at least 60 percent of their time was focused on
activities and initiatives that will have a direct impact on the quality of instruction in their district. About 20
percent spent between 40 to 59 percent of their time on such activities. A small minority (three from each cohort)
spent only 20 to 39 percent of their time focused on instructional improvement. None of the respondents reported
spending less than 20 percent of their time on such tasks. Overall cohort 2 superintendents and coaches were
fairly well aligned in their perceptions. In contrast, coaches’ offer a somewhat more favorable perspective for
cohort 1 reporting that 86 percent of cohort 1 participants spent at least 60 percent of their time focused on quality
of instruction.
Table 15: Proportion of Time Focused on Quality of Instruction
0-19%
20-39%
40-59%
60-79%
80-100%
Self-Report
0%
14%
19%
43%
24%
Coaches’ Perspective
5%
10%
0%
48%
38%
Self-Report
0%
12%
23%
50%
15%
Coaches’ Perspective
4%
8%
27%
46%
15%
Cohort 1 Superintendents (N=21*)
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
Focus group participants expressed reservations about their ability to effectively balance the time they spend on
instructional leadership compared to management and operations. Though all recognize the importance of
focusing on instruction they continue to find their efforts imposed upon by administrative issues that arise. Even
so, superintendents do indicate that the program has helped them to focus more on instruction, though many feel
that they aren’t spending as much time as the program expects.
UMass Donahue Institute
Research and Evaluation Group
21
Impact on Superintendent Capacity and Performance
NSIP Annual Evaluation Report: FY 2012
Superintendent’s Baseline Abilities
Survey respondents were asked to rate superintendent baseline abilities in several key areas listed below in Table
16. Superintendents were asked to reflect back on their abilities at the time they entered the superintendency and
coaches to reflect back on when they first began working with those superintendents.
Superintendents from both cohorts reported relatively strong baseline abilities in the area of instructional
leadership, strategic thinking and strategic management of resources. Cohort 1 superintendents also reported
strong baseline abilities in the area of creating effective leadership teams. Cohort 2 superintendents reported
relatively high baseline ability in developing instructional leadership skills among school leaders. With little
exception superintendents reported better baseline abilities than their coaches. Both superintendents and their
coaches reported relatively weak baseline abilities in the area of root cause analysis.
Table 16: Summary of Superintendent Abilities – Baseline
Percent Responding Excellent or Good
Matched
Pairs
Cohort 1
Superintendents
Coaches
Matched
Pairs
Cohort 2
Superintendents
Serve as an effective
instructional leader
20
95%
60%
25
100%
84%
Think strategically about district
improvement
21
91%
43%
26
84%
62%
Conduct root cause analysis of
instructional problems
21
43%
19%
26
46%
16%
Manage resources in a strategic
manner
21
81%
77%
26
92%
69%
Develop school leaders’
instructional leadership skills
21
43%
29%
26
88%
54%
Create effective leadership
teams
21
81%
33%
25
72%
40%
Forge collaborative relationships
with key constituencies
21
67%
66%
26
76%
69%
Establish effective systems for
supervision and evaluation
20
55%
35%
26
73%
31%
Superintendent ability to . . .
Coaches
* Data reported only for cases where both self-report ratings and matched coaches’ ratings are available. For all items this excludes one
cohort 1 superintendent in a new coaching relationship. There are other excluded cases where either the superintendent or coach did not offer
a response about a specific ability.
Impact on Superintendent Abilities
Survey respondents were asked to indicate the extent to which NSIP improved superintendent abilities in those
same areas. They were explicitly instructed to report only impact that they felt was attributable to NSIP
participation and to select “no impact” if ability improved for other reasons.
More than half of the cohort 1 superintendents reported their participation had resulted in at least modest
improvement in each of the areas listed. Nearly all reported that NSIP had at least modestly improved their ability
to create effective leadership teams (95%), serve as an effective instructional leader (90%), and think strategically
about district improvement (90%). For most dimensions superintendents reported higher levels of impact than
UMass Donahue Institute
Research and Evaluation Group
22
Impact on Superintendent Capacity and Performance
NSIP Annual Evaluation Report: FY 2012
their coaches. The exceptions are the superintendents’ ability to conduct root cause analysis of instructional
problems and establish effective systems for supervision and evaluation. Both cohort 1 superintendents and their
coaches reported the least impacts on the ability to manage resources in a strategic manner and forge collaborative
relationships with key constituencies. These are both areas that superintendents, coaches and program developers
acknowledge have not received a great deal of attention in the NSIP curriculum.
Cohort 2 superintendents and their coaches generally reported the most impact in the ability to think strategically
about district improvement, conduct root cause analysis of instructional problems and serve as an effective
instructional leader. Superintendents reported relatively little impact on their ability to manage resources in a
strategic manner compared to coaches who rated this as one of the areas where the program has had the most
impact. Both superintendents and coaches offered relatively low impact ratings for the ability to establish
effective systems for supervision and evaluation and forge collaborative relationships for key constituencies.
Table 17: Summary of Impact on Superintendent Abilities – Substantially or Modestly Improved
Percent Responding Substantially or Modestly Improved
Matched
Pairs
Cohort 1
Superintendents
Coaches
Matched
Pairs
Cohort 2
Superintendents
Serve as an effective
instructional leader
20
90%
60%
25
64%
64%
Think strategically about district
improvement
21
90%
81%
26
69%
88%
Conduct root cause analysis of
instructional problems
21
71%
90%
26
69%
62%
Manage resources in a strategic
manner
21
57%
43%
26
27%
65%
Develop school leaders’
instructional leadership skills
21
81%
71%
26
54%
58%
Create effective leadership
teams
21
95%
62%
25
60%
56%
Forge collaborative relationships
with key constituencies
21
52%
43%
26
35%
50%
Establish effective systems for
supervision and evaluation
20
80%
80%
26
35%
38%
Superintendent ability to . . .
Coaches
* Data reported only for cases where both self-report ratings and matched coaches’ ratings are available. For all items this excludes one
cohort 1 superintendent in a new coaching relationship. There are other excluded cases where either the superintendent or coach did not offer
a response about a specific ability.
Superintendents were asked to describe anything outside of NSIP that had a significant impact on their abilities in
any of the above areas. Many cited their previous professional experience as a superintendent, central office
administrator or building principal. Other relevant comments were:
o
I have been involved as an early adopter for the new educator evaluation system at both the
state and federal level which has improved my knowledge base as an effective instructional
leader. In addition, I have been working with my leadership team on Patrick Lencioni’s
work in creating effective leadership teams.
UMass Donahue Institute
Research and Evaluation Group
23
NSIP Annual Evaluation Report: FY 2012
Impact on Superintendent Capacity and Performance
o
The NISL program has had a strong influence on my abilities as a superintendent.(2
respondents)
o
Summer MASS Executive Program and Alan November technology conferences.
o
Involvement with MASS and involvement with the International Society for Performance
Improvement Personal Learning Network
o
Collaboration with area superintendents through formal and informal networks. Private
leadership coaching.
Appendix B provides more detailed views of the survey responses for each area including superintendent and
coaches perspective on baseline ability and degree to which NSIP has had a positive impact on those abilities.
UMass Donahue Institute
Research and Evaluation Group
24
NSIP Annual Evaluation Report: FY 2012
Impact on District Improvement
Impact on District Improvement
In addition to improving superintendents’ own practice, NSIP is expected to positively impact districts as their
leaders influence system capacity and practice through the application of new knowledge, skills and tools.
Superintendents and their coaches were asked to report on baseline conditions in the district and their perceptions
of NSIP impact on district capacity and practice in the following areas: instructional leadership capacity,
developing effective leadership teams, developing relationships with key constituencies, strategic management of
resources, and developing systems for supervision and evaluation.
Key Findings
 For most aspects cohort 1 superintendents and their coaches offered positive ratings of baseline conditions for
fewer than 40 percent of the districts giving the least positive ratings for administrator evaluation systems.
With the exception of administrator and teacher evaluation systems, cohort 2 superintendents offered far more
favorable ratings of district baseline conditions with at least two-thirds offering positive ratings for most of
the aspects listed.
 At least half of the cohort 1 superintendents reported that their participation had resulted in at least modest
district improvement. They reported the most improvement for district-level strategy and principal readiness
to lead instructional improvement. Their lowest impact ratings were for teacher evaluation systems, school
committee productivity, and overall instructional quality. For cohort 1 coaches’ perceptions of impact on
districts were generally well-aligned with superintendents’ perceptions. Compared to superintendents,
coaches indicated more improvement in district-level focus on instructional improvement and somewhat less
improvement in principal readiness, alignment of budget to strategy and administrator evaluation systems.
 For nearly all areas cohort 2 superintendents reported less impact than their coaches as well as cohort 1. The
one exception was productivity of district leadership team meetings, which nearly three-quarters of cohort 2
superintendents reported improved at least modestly as a result of their participation in NSIP. Cohort 2
superintendents also gave relatively high impact ratings for district-level focus on and strategy for
instructional improvement. Impact in these areas was also highly rated by their coaches, who also rated
alignment of budget to district strategy as an area of relatively strong impact.
 Superintendents and coaches from both cohorts reported the least impact on teacher evaluation systems.
Cohort 2 superintendents and their coaches also reported relatively little impact on administrator evaluation
systems. This is reflective of the fact that the cohort 1 curriculum included a strong focus on implementation
of the new educator evaluation system, which was not a strong component of the cohort 2 curriculum in 20112012. It is anticipated that the associated impact ratings for cohort 2 districts will increase on the spring 2013
survey as they will be focusing more on educator evaluation in their second year of NSIP participation.
District Baseline Conditions
Survey respondents were asked to rate district baseline capacity and practice in several key areas listed below in
Table 18. Superintendents were asked to reflect back on their abilities at the time they entered the superintendency
and coaches to reflect back on when they first began working with those superintendents7.
7
Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come from the fall 2011
superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
UMass Donahue Institute
Research and Evaluation Group
25
Impact on District Improvement
NSIP Annual Evaluation Report: FY 2012
For most aspects cohort 1 superintendent’ and their coaches offered positive ratings of baseline conditions for
fewer than 40 percent of the districts giving the least positive ratings for administrator evaluation systems. Their
perspectives were also well-aligned for district strategy, school-level focus on instructional improvement,
principal readiness to lead instruction, and alignment of budget to strategy. On aspects where they diverged,
coaches were more favorable about district baseline conditions for teacher evaluation systems and district-level
focus on instructional improvement. Superintendents offered more positive views of the overall quality of
instruction as well as the productivity of school committee and district leadership team meetings.
With the exception of administrator and teacher evaluation systems, cohort 2 superintendents offered far more
favorable ratings of district baseline conditions compared to their own coaches’ ratings as well as cohort 1 ratings.
At least two-thirds of the cohort 2 superintendents offered positive ratings for most of the aspects listed.
Table 18: Summary of District Capacity and Practice – Baseline**
Percent Responding Excellent or Good
Matched
Pairs
Cohort 1
Superintendents
Coaches
Matched
Pairs
Cohort 2
Superintendents
District-level focus on
instructional improvement
20
35%
45%
26
81%
39%
District-level strategy for
instructional improvement
21
29%
29%
26
67%
12%
School-level focus on
instructional improvement
21
38%
38%
26
70%
42%
Principal readiness to lead
instructional improvement
20
30%
30%
26
69%
43%
Overall quality of instruction in
the district
21
48%
29%
26
85%
50%
Alignment of budget to overall
district improvement strategy
20
35%
30%
26
43%
15%
Productivity of school committee
meetings
21
39%
15%
26
65%
23%
Productivity of district leader
team meetings
21
38%
24%
26
77%
15%
Administrator evaluation
systems (as implemented)
20
15%
20%
26
16%
12%
Teacher evaluation systems (as
implemented)
18
17%
28%
25
24%
12%
Coaches
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
Impact on District Practice and Capacity
Survey respondents were asked to indicate the extent to which NSIP improved district capacity and practice in
those same areas. They were explicitly instructed to report only impact that they felt was attributable to NSIP
participation and to select “no impact” if ability improved for other reasons.
UMass Donahue Institute
Research and Evaluation Group
26
Impact on District Improvement
NSIP Annual Evaluation Report: FY 2012
At least half of the cohort 1 superintendents reported their participation had resulted in at least modest district
improvement in each of the areas listed. They reported the most improvement for district-level strategy and
principal readiness to lead instructional improvement. Their lowest impact ratings were for teacher evaluation
systems, school committee productivity, and overall instructional quality. In general, coaches’ perceptions of
impact on cohort 1 districts were well-aligned with superintendents’ perceptions. Compared to superintendents,
coaches indicated more improvement in district-level focus on instructional improvement and somewhat less
improvement in principal readiness, alignment of budget to strategy and administrator evaluation systems.
Table 19: Summary of Impact on District Practice and Capacity – Substantially or Modestly Improved
Percent Responding Substantially or Modestly Improved
Matched
Pairs
Cohort 1
Superintendents
Coaches
Matched
Pairs
Cohort 2
Superintendents
District-level focus on
instructional improvement
20
75%
95%
26
50%
77%
District-level strategy for
instructional improvement
21
86%
86%
26
62%
73%
School-level focus on
instructional improvement
21
71%
71%
26
35%
54%
Principal readiness to lead
instructional improvement
20
85%
60%
26
38%
58%
Overall quality of instruction in
the district
21
57%
48%
26
23%
46%
Alignment of budget to overall
district improvement strategy
20
60%
55%
26
27%
65%
Productivity of school committee
meetings
21
57%
43%
26
31%
42%
Productivity of district leader
team meetings
21
71%
71%
26
73%
54%
Administrator evaluation
systems (as implemented)
20
65%
50%
26
19%
23%
Teacher evaluation systems (as
implemented)
18
50%
44%
25
8%
16%
Coaches
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact. For all items this excludes
one cohort 1 superintendent in a new coaching relationship. There are other excluded cases where either the superintendent or coach did not
offer a response about a specific ability.
For nearly all areas cohort 2 superintendents reported less impact than their coaches as well as cohort 1. The one
exception was productivity of district leadership team meetings, which nearly three-quarters reported has
improved at least modestly as a result of their participation in NSIP. This is somewhat higher than the 54 percent
of coaches reporting that degree of impact for cohort 2 districts, and well aligned with the 71 percent positive
impact reported by both cohort 1 superintendents and coaches. Cohort 2 superintendents also gave relatively high
impact ratings for district-level focus on and strategy for instructional improvement. Impact in these areas was
also highly rated by their coaches, who also rated alignment of budget to district strategy as an area of relatively
strong impact. As with cohort 1, superintendents and coaches reported the least impact on teacher evaluation
systems. They also reported relatively little impact on administrator evaluation systems. Again this is reflective of
UMass Donahue Institute
Research and Evaluation Group
27
NSIP Annual Evaluation Report: FY 2012
Impact on District Improvement
the fact that the cohort 1 curriculum included a deliberate focus on implementation of the new educator evaluation
system, which was not a strong component of the cohort 2 curriculum in 2011-2012. It is anticipated that the
associated impact ratings for cohort 2 districts will increase on the spring 2013 survey as they will be focusing
more on educator evaluation in their second year of NSIP participation.
Superintendents were given the opportunity to comment on the impact of their NSIP participation on their
district’s improvement. Although the focus of the question was district improvement, most commented about their
own capacity and practice.
o
I think NSIP assisted me in creating a plan for improvement. We now need to see if the plan
can have a positive effect on the instructional practice of the district. It is too early to tell.
o
The NSIP program has helped me refine my thinking and approach to structuring leadership
team meetings and other strategically important meetings.
o
Participating in NSIP has helped me understand how to identify the right priority to focus
on regarding district improvement and how to develop an effective plan for implementation.
o
NSIP has had an impact on my overall leadership and on my ability to think and act
strategically to move the district towards a plan for improvement.
As articulated in the following coaches’ comments, this likely reflects the additional time needed for changes in
superintendents’ approach, practice, and planning to translate into specific and observable changes in the district.
o
It has been a slow process to implement many of the concepts from NSIP into the district.
o
There has been significant impact on the district’s plans, but minimal implementation. If and
when the plans go forward then improvement should follow.
It should be acknowledged that one superintendent offered a more critical comment about the impact of NSIP
participation on the district.
o
The program took too much time away from actually doing the work in the district. . .In
districts that have few administrators it is a real burden to take so many days away and to
have so much additional reading and paperwork to complete during the first year as a
superintendent. I feel that I could have accomplished much more without these distractions.
Superintendents were also asked to describe anything outside of NSIP that had a significant impact on their
district. Several cited DESE support for educator evaluation implementation. Other relevant comments were:
o
The district’s alignment with their improvement plan has had an impact on the advancement
of the district. Also supports from DSAC have been a factor for improvement in the
instructional practices for the district.
o
We have taken an aggressive and proactive stance on instructional improvement by
leveraging a number of opportunities through other entities (i.e. innovation schools and the
Gateway Cities initiative, a major STEM initiative through the STEM Education Center at
WPI, resources and on-going support through DSAC, and partnerships with colleges and
universities in our area. In addition, the district had the benefit of participating in the MASC
Governance Project.
Appendix C provides provide more detailed views of the survey responses for each area including superintendent
and coaches perspective on baseline ability and degree to which NSIP has had a positive impact on those abilities.
UMass Donahue Institute
Research and Evaluation Group
28
Participant Satisfaction and Engagement
NSIP Annual Evaluation Report: FY 2012
Participant Satisfaction and Engagement
As described throughout this report, survey and interview data suggest high levels of satisfaction among cohort 1
superintendents with more mixed reviews from those in the second cohort. All cohort 1 superintendents reported
that they were satisfied with the NSIP program, reflecting some improvement over the prior year when 14 percent
of respondents described themselves as mostly dissatisfied. Furthermore, the proportion reporting that they were
highly satisfied increased from 41 percent to 59 percent. Similarly, all cohort 1 superintendents reported that the
program was a good fit for them and that they would recommend it to other new superintendents. Although not as
strong, most cohort 2 superintendents also responded positively to these items with 85 percent indicating that they
were satisfied with the program, 89 percent reporting that it was a good fit for them, and 81 percent saying that
they would recommend the program to other new superintendents.
Table 20: Overall Program Satisfaction
Cohort 1
Spring 2012
Cohort 2
Spring 2012
Cohort 1
Spring 2011*
Highly Satisfied
59%
35%
41%
Mostly Satisfied
41%
50%
46%
Mostly Dissatisfied
0%
12%
14%
Highly Dissatisfied
0%
4%
0%
Cohort 1
Spring 2012
Cohort 2
Spring 2012
Cohort 1
Spring 2011*
Yes, it is a good fit for me
77%
54%
68%
Yes, but only to a point
23%
35%
23%
No, it is not a good fit for me
0%
12%
9%
Table 21: Is the Program a Good Fit for You
Table 22: Would you recommend this program to other new superintendents?
Cohort 1
Spring 2012
Cohort 2
Spring 2012
Cohort 1
Spring 2011*
Yes
100%
81%
100%
No
0%
4%
0%
Unsure
0%
15%
0%
* Spring 2011 data reported in these tables include only responses for superintendents completing the most recent survey. As a result, they
differ somewhat from those previously reported.
Coaches perceived that about half of the superintendents were very committed to the program. Despite cohort 1
superintendents’ highly positive feedback, coaches indicated that nearly a third were only slightly committed to
the program. Conversely, although cohort 2 superintendents tended to be more critical of the program, coaches
reported that only 16 percent are less than moderately committed participants.
UMass Donahue Institute
Research and Evaluation Group
29
Participant Satisfaction and Engagement
NSIP Annual Evaluation Report: FY 2012
Table 23: Superintendent Commitment to NSIP Goals and Practices – Coaches’ Perspective
Percent of Coaches Responding
N
Not at All
Committed
Slightly
Committed
Moderately
Committed
Very
Committed
Cohort 1 Superintendents
21
0%
29%
19%
52%
Cohort 2 Superintendents
26
4%
12%
38%
46%
UMass Donahue Institute
Research and Evaluation Group
30
NSIP Annual Evaluation Report: FY 2012
Conclusion
Conclusion
As the New Superintendents Induction Program completed its second year, participant and coach feedback
suggest that it was increasingly providing critical support to new superintendents’ development and district
improvement efforts.
Most notably, participating superintendents continued to place considerable value on the coaching provided to
them. They offered overwhelmingly positive ratings of their coaches’ knowledge and support. Although coaches
have undoubtedly become more familiar and comfortable with NSIP practices over the past two years, it is
important to acknowledge that they require continued and ongoing support to improve their understanding of
these practices, which represent a new approach to the superintendency.
In service of that new approach, coaches are making a concerted effort to more closely adhere to the NSIP
curriculum during their coaching sessions and more effectively tie NSIP to the day-to-day work of the
superintendents. Yet, superintendents also view the coaching as a critical means of obtaining help with
challenging operational issues that are not being addressed through the formal content day curriculum. Evaluation
data suggest that many new superintendents struggle to balance the time they spend on instructional leadership
compared to management and operations. In particular, superintendents expressed the need for more support in
these areas. The lack of attention to these matters has been particularly frustrating for a number of cohort 2
superintendents and has negatively colored their overall satisfaction with the NSIP program.
It is also evident that introducing a focus on implementation of the new educator evaluation system as a theme for
cohort 1 superintendents made the content day sessions feel more relevant for this group. Maintaining some focus
on using educator evaluation as a driver of the district improvement strategy should remain a focus of their
participation in year 3 of the program. Those linkages should also be made for both cohort 2 and cohort 3
superintendents as a way to ground their learning in practical challenges that they are facing in their districts.
Although respondents generally rated the overall value of content days favorably, superintendents and coaches
identified a number of opportunities to improve those sessions including: more effectively integrating the readings
and assignments into the content day itself; improving on past efforts to differentiate content and instruction given
the diversity of preparation and previous experience among the participants; and strengthening coaches’ role in
content delivery.
As NSIP matures it will be important to develop and implement robust systems to evaluate the extent to which the
program is achieving its intended impact on superintendents and their districts. To date, most superintendents
indicate that the program has had a moderate or substantial impact on their view of the role of the superintendent.
Most also credit the program with yielding at least a modest improvement in various aspects of their own practice.
While many superintendents also report that their participation has resulted in some improvements in district
capacity and practice, it is clear that more time is required to see NSIP principles effectively integrated throughout
the organizations in service of the ultimate goal of creating and sustaining improvements in student achievement.
UMass Donahue Institute
Research and Evaluation Group
31
Appendix A – Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
Appendix A – Assignments and Work Products
Report of Entry Findings
Majorities of superintendents and coaches from both cohorts described progress on the Report of Entry Findings
as substantial or completed. Since entry is conceptualized as a first-year activity, it is interesting that fewer than
half of cohort 1 superintendents indicated that they had completed this assignment and also to note that cohort 1
coaches generally had a more favorable view of those individuals’ progress (63 percent completed) than did the
superintendents themselves. In contrast, nearly three-quarters of cohort 2 superintendents described themselves as
having completed their entry reports, while their coaches had a somewhat less favorable view of their progress (50
percent completed).
Table 24: Progress on Report of Entry Findings
Little or No
Progress
Moderate
Progress
Substantial
Progress
Completed
and/or Refining
Self-Report
5%
21%
26%
47%
Coaches’ Perspective
16%
21%
0%
63%
Self-Report
0%
8%
19%
73%
Coaches’ Perspective
4%
19%
27%
50%
Cohort 1 Superintendents (N=19*)
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
It is notable that five superintendents were described as making little or no progress on this activity – one selfreported and four by their coaches. In two of the cases where coaches reported little or no progress, the
superintendents described their own progress as substantial and in a third the superintendent self-reported as
completed. Two other superintendents who self-reported as completed have coaches who described their progress
as moderate.
Through the survey, respondents were asked whether they found certain assignments or products particularly
beneficial or unhelpful in two separate questions. Among those who offered comments, eleven specifically cited
entry planning and/or the Report of Entry Findings as a beneficial work product. For example:
o
The entry plan was beneficial to target the focus on initial work and outcomes. The timetable
for the entry plan was helpful because the program facilitators suggested allowing more
time than the school committee would have liked.
o
The entry plan and development of strategy – as someone who had been in the district, it
helped to collect that information from a fresh lens. I think it also helped the public to see
that we were using the opportunity to really look closely at what would move the district
forward. My school committee and admin team found it helpful to document where we are
and where we’re trying to go.
o
Developing an Entry Plan and Entry Plan Findings Report were critical to having a clear,
systematic way to integrate myself into the district. More exemplars of entry plan findings
reports are needed!
UMass Donahue Institute
Research and Evaluation Group
32
Appendix A – Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
Four respondents offered more critical comments about the way that entry planning and the Report of Entry
findings were approached through the program. It is noteworthy that these comments all came from cohort 2
superintendents and thus suggest that, despite adjustments to the curriculum, the program has not been completely
successful in addressing the lack of clarity around this task that was identified when it was first delivered to
cohort 1 superintendents.
o
The entry plan is a critical deliverable of the NSIP, however, I experienced (as did others) a
sense of frustration with the general lack of direction provided by facilitators of the
program. With no real guidance, many of us just did what was right for our districts. I
received some guidance from my coach, but if there is some expectation of format or
essential content for the entry report, this has not been identified or communicated as part of
the NSIP curriculum.
o
The entry plan is a good idea. However, the report of entry findings, as it was described
during the sessions, did not follow from the entry work that I did. I believe this was because
we were not given direction as to the kinds of questions that should drive our inquiry from
the start and therefore the assignment lacked cohesiveness. It was very clear when we got to
the “report of entry findings” that we were supposed to have a report that was focused on
academic achievement but this was not given as the intention from the start which made it
very difficult to complete the report with any degree of integrity in cases where questions
that drove inquiry were more open-ended and not necessarily only about achievement.
o
All of the content was helpful, but I felt that much of the time the facilitators were still
struggling to figure out how to present the material. There was frustration around the
format and content of the entry reports.
o
I did not find the entry plan process clear. It led me to put together a document that really
did not clearly define the work. This was more because of my own understanding. The
pressure to present something to my committee in a timely fashion did not give me enough
time to truly understand what it should have looked like.
Theory of Action
A question in the coaching section of the survey probed on the extent to which several topics, including refining
the theory of action, were the focus of coaching conversations8. As shown in Table 25, the majority of
superintendents and coaches reported that the Theory of Action (TOA) was at least a moderate focus of their
coaching conversations. For each cohort there were seven cases where the superintendent and/or the coach
indicated that few, if any, of their conversations addressed this issue.
Table 25: Focus of Coaching Conversations – Refining the Theory of Action
Little or No Focus
Moderate Focus
Strong Focus
Cohort 1 Superintendents (N=20*)
Self-Report
10%
60%
30%
Coaches’ Perspective
30%
50%
20%
Self-Report
20%
40%
40%
Coaches’ Perspective
16%
56%
28%
Cohort 2 Superintendents (N=25*)
* Data reported for individuals with both superintendent self-report and coach ratings
8
This question addressed most of the work products cited, but not the Report of Entry Findings or Leadership Team Development.
UMass Donahue Institute
Research and Evaluation Group
33
Appendix A – Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
For both cohorts about 70 percent of superintendents and slightly more than half of coaches described progress on
the Theory of Action as substantial or completed. This is generally conceived as an activity to be completed
within the first year of NSIP. As such, it is noteworthy that about 30 percent of superintendents and nearly half of
the coaches described progress on this activity as moderate at best – including three coaches who reported little or
no progress on the Theory of Action.
Table 26: Progress on Theory of Action
Little or No
Progress
Moderate
Progress
Substantial
Progress
Completed
and/or Refining
Self-Report
0%
29%
33%
38%
Coaches’ Perspective
10%
38%
19%
33%
Self-Report
0%
31%
35%
35%
Coaches’ Perspective
4%
42%
31%
23%
Cohort 1 Superintendents (N=21*)
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
Five respondents cited the Theory of Action as a beneficial work product. Favorable comments included:
o
Theory of Action helped greatly with establishing strategy for district change.
o
Developing a Theory of Action for our strategy helped to focus me on the Big Understanding
of why we were doing something.
o
Theory of Action enabled me to articulate a cohesive and systematic response to the issues
surfaced through the entry process.
Two other respondents cited the Theory of Action as a product that was not helpful. A third recognized the
potential benefit, but offered the following critical comment about its timing within the curriculum:
o
The rich discussion around theory of action did not happen until the last content day. The
pacing of the curriculum still remains a bit disorganized after two cohorts.
Strategy Development
Nearly all superintendents and coaches reported that creating a strategy for district improvement was at least a
moderate focus of their coaching conversations. For each cohort there were three cases where the superintendent
and/or the coach indicated that few, if any, of their conversations addressed this issue.
Table 27: Focus of Coaching Conversations – Strategy Development
Little or No Focus
Moderate Focus
Strong Focus
Cohort 1 Superintendents (N=21*)
Self-Report
5%
48%
48%
Coaches’ Perspective
10%
48%
43%
Self-Report
12%
31%
58%
Coaches’ Perspective
0%
46%
54%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
UMass Donahue Institute
Research and Evaluation Group
34
Appendix A – Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
For both cohorts about 90 percent of superintendents and about 60 percent of coaches described progress on
Strategy Development as substantial or completed. Consistent with the expectations for the timing of this activity
(late in year 1 and/or early in year 2) both superintendent self-report and coaches’ ratings indicate that, compared
with cohort 2, a larger proportion of cohort 1 superintendents had completed or were refining their strategy.
Surprisingly, 38 percent of cohort 1 coaches described their superintendents as making only moderate progress on
this activity at the end of their second year of NSIP participation.
Table 28: Progress on Strategy Development
Little or No
Progress
Moderate
Progress
Substantial
Progress
Completed
and/or Refining
Self-Report
0%
10%
43%
48%
Coaches’ Perspective
5%
38%
19%
38%
Self-Report
0%
12%
54%
35%
Coaches’ Perspective
8%
31%
42%
19%
Cohort 1 Superintendents (N=21*)
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
Seven respondents commented on benefits of one or more components of strategy development, including Root
Cause Analysis, SWOT analysis, logic models, strategic objectives and initiatives. Their comments included:
o
Root Cause Analysis. We have many things to fix/improve in our district. Being able to get to
the root cause of the challenges will allow us to target our steps.
o
Strategy assignments are directly applicable to my work and contributed to my success in
my first year.
o
The development of a logic model for the implementation of the new educator evaluation
system was extremely helpful.
Two respondents offered more critical comments on topics related to strategy development, including:
o
I thought that the SWOT came out of sequence. Unofficially our team conducted an activity
quite similar in our first weeks of work together. The SWOT being presented after the entry
plan was odd to me.
o
Some overkill on the strategic pieces.
Additionally, superintendents and their coaches were asked to characterize the involvement of key stakeholder
groups in their efforts to develop a strategy for district improvement. Superintendents’ responses are summarized
in Table 29. In general, superintendents and coaches were well aligned in their perceptions of stakeholder
involvement. Superintendents reported the lowest level of involvement for the school committee with equal
proportions from each cohort (20%) characterizing them as having substantial or little to no involvement. For
cohort 1 this marks a slightly increased level of involvement over what they reported on the fall survey9. There is
only one case of disagreement10 between the superintendent and coach. Interestingly in this case a cohort 2
superintendent reported little or no school committee involvement, but the coach characterized the involvement as
substantial.
9
This question was not asked on the fall 2011 survey administered to cohort 2
For this report disagreement describes cases where the superintendent and coach offering ratings that are at least two levels apart. In
terms of this specific question that means that one reported substantial involvement and the other reported little or no involvement.
10
UMass Donahue Institute
Research and Evaluation Group
35
Appendix A – Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
Table 29: Strategy Development – Involvement of Key Stakeholder Groups
Substantial
Involvement
Moderate
Involvement
Little or No
Involvement
School Committee (N=20*)
20%
60%
20%
Central Office Administrators (N=21*)
81%
19%
0%
Building Principals (N=20*)
85%
15%
0%
School Committee (N=26)
19%
62%
19%
Central Office Administrators (N=25*)
76%
20%
4%
Building Principals (N=25*)
72%
24%
4%
Cohort 1
Cohort 2
* Data reported for individuals with both superintendent self-report and coach ratings
Among these districts, 14 had some base level of involvement in the District Governance Support Project
(DGSP), a companion program of NSIP designed to support effective practice among school committees and
foster stronger collaboration between those committees and superintendents. In most of the cases, superintendents
from those districts reported moderate school committee involvement in strategy development. Three reported
little or no involvement and only one reported substantial involvement.
These results are consistent with interview findings that building good working relationships with school
committees is an ongoing struggle for these new superintendents. While it is not substantially addressed through
the content day curriculum, for most respondents it was a moderate or strong focus of conversations with their
NSIP coaches. As above, for cohort 1 this marks a slight increase compared to their responses on the fall survey 11.
Table 30: Focus of Coaching Conversations – Building School Committee Relationships
Little or No Focus
Moderate Focus
Strong Focus
Cohort 1 Superintendents (N=21*)
Self-Report
14%
48%
38%
Coaches’ Perspective
19%
67%
14%
Self-Report
4%
35%
62%
Coaches’ Perspective
8%
50%
42%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
In contrast, strong majorities of superintendents from both cohorts reported substantial involvement of central
office administrators and building principals. All cohort 1 superintendents indicated that these stakeholders
were at least moderately involved in strategy development. There were three cohort 1 cases where superintendents
and their coaches disagreed about the level of central office and/or principal involvement – all were cases where
the superintendent reported substantial involvement and the coach reported little or none. One cohort 2
superintendent reported that central office administrators and building principals had little or no involvement,
while the coach indicated substantial central office involvement and moderate principal involvement. In two
11
This question was not asked on the fall 2011 survey administered to cohort 2
UMass Donahue Institute
Research and Evaluation Group
36
Appendix A – Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
additional cohort 2 cases – one for central office and the other for principals – superintendents reported substantial
involvement in contrast to coaches’ perceptions of little or no involvement.
Finally, superintendents were provided the opportunity to comment on significant challenges they faced in
developing their strategy. The comments offered clustered around the themes of difficulties working with the
school committee, district leadership issues, and the challenge of dealing with existing strategic plans.
Relevant comments included:
o
The school committee has been the greatest challenge. Even though they agreed to sign on
to [DGSP]it has done nothing to assist in the development and strategic improvement. It has
only added additional work that is not connected to the strategic planning that has occurred.
o
The school committee is not a cohesive group. Rather, they have their own separate
identities and frequently send multiple signals. Additionally, I had several long-entrenched
administrators who were counterproductive to the process, inhibiting positive outcomes.
o
We are struggling with building administrator capacity in terms of time and commitment to
growth on the part of some district leaders.
o
Entering a district with a strategic plan already in place creates a problem. I find myself
attempting to complete a crosswalk of the two documents.
o
We were completing a district strategic plan at the same time I was completing this
assignment, so at times it was difficult to manage the process and to be sure my entry
findings were informing the district strategic plan.
Strategy Implementation
All cohort 1 superintendents and all but one coach reported that implementing the strategy for district
improvement was at least a moderate focus of their coaching conversations with most indicating that it was a
strong focus. While most cohort 2 superintendents and coaches reported at least moderate focus on strategy
implementation, there are seven cases where either the superintendent or the coach reported spending little or no
time on this topic.
Table 31: Focus of Coaching Conversations – Strategy Implementation
Little or No Focus
Moderate Focus
Strong Focus
Cohort 1 Superintendents (N=21*)
Self-Report
0%
33%
67%
Coaches’ Perspective
5%
38%
57%
Self-Report
12%
35%
54%
Coaches’ Perspective
15%
58%
27%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
Both superintendent self-report and coach ratings suggest that, as would likely be expected, cohort 1 participants
made more progress in strategy implementation than their cohort 2 counterparts. These results seem fairly logical
given each cohort’s progress within the program. It is also not particularly surprising that relatively few
superintendents from either cohort perceived themselves as having completed their strategy implementation. In
fact, the most surprising thing may be that 19 percent of cohort 1 respondents reported that strategy
implementation has been completed. Further, more than half of the cohort 2 superintendents perceived themselves
UMass Donahue Institute
Research and Evaluation Group
37
Appendix A – Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
making at least substantial progress in strategy implementation – a level of progress that would seem premature
given their tenure in both the program and their positions. Indeed, coaches reported that fewer than 30 percent of
cohort 2 superintendents had made substantial progress in strategy implementation and that none had reached the
stage of completion.
Table 32: Progress on Strategy Implementation
Little or No
Progress
Moderate
Progress
Substantial
Progress
Completed
and/or Refining
Self-Report
0%
14%
67%
19%
Coaches’ Perspective
5%
43%
33%
19%
Self-Report
8%
38%
46%
8%
Coaches’ Perspective
35%
38%
27%
0%
Cohort 1 Superintendents (N=21*)
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
None of the respondents specifically mentioned strategy implementation in response to the open-ended questions.
Classroom Visits and Principal Debriefings
Most superintendents and coaches reported that visiting classrooms to observe instruction was at least a moderate
focus of their coaching conversations. There were a total of 11 cases where the superintendent and/or the coach
indicated that there was little or no focus on this topic – five from cohort 1 and six from cohort 2. These results
reflect a marked decrease in focus for cohort 1 when compared to their responses on the fall survey when more
than 60 percent indicated that classroom visits was a strong focus of their coaching conversations. At the same
time, it appears cohort 2 increased its focus on this throughout the year. In the fall nearly one-quarter of cohort 2
superintendents reported little or no focus on classroom visits compared to only 4 percent on the spring survey.
Table 33: Focus of Coaching Conversations – Visiting Classrooms
Little or No Focus
Moderate Focus
Strong Focus
Cohort 1 Superintendents (N=21*)
Self-Report
10%
57%
33%
Coaches’ Perspective
19%
48%
33%
Self-Report
4%
46%
50%
Coaches’ Perspective
19%
50%
31%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
As shown in Table 34, 62 percent of cohort 1 and 68 percent of cohort 2 superintendents perceived themselves as
making substantial progress or completing their work related to classroom visits and principal debriefings. Only
two superintendents (one from each cohort) indicated that they had made little or no progress on this effort. From
a coaches’ perspective somewhat larger numbers of superintendents (four from cohort 1 and three from cohort 2)
failed to make much progress on this activity.
UMass Donahue Institute
Research and Evaluation Group
38
Appendix A – Assignments and Work Products
NSIP Annual Evaluation Report: FY 2012
Table 34: Progress on Classroom Visits and Principal Debriefings
Little or No
Progress
Moderate
Progress
Substantial
Progress
Completed
and/or Refining
Self-Report
5%
33%
29%
33%
Coaches’ Perspective
19%
29%
29%
24%
Self-Report
4%
28%
36%
32%
Coaches’ Perspective
12%
28%
36%
24%
Cohort 1 Superintendents (N=21*)
Cohort 2 Superintendents (N=25*)
* Data reported for individuals with both superintendent self-report and coach ratings
Only one superintendent cited the classroom visits in the open response comments:
o
I would have done walkthroughs anyway, but I’m not sure all of my colleagues would have,
so I think asking us to do that was great.
Leadership Team Development
Most superintendents reported that they had made either substantial progress or completed their work on
leadership team development. As might be expected given their longer tenure in their positions, larger percentages
of cohort 1 superintendents indicated that they have reached the completion stage for this task. Only one
superintendent reported making little or no progress in leadership team development. From a coaches’
perspective, two cohort 1 superintendents and three cohort 2 superintendents made little progress on leadership
team development. These include one cohort 1 superintendent who self-reported as completed and two cohort 2
superintendents who self-reported as making substantial progress.
Table 35: Progress on Leadership Team Development – Superintendent Self-Report
Little or No
Progress
Moderate
Progress
Substantial
Progress
Completed
and/or Refining
Self-Report
0%
14%
48%
38%
Coaches’ Perspective
10%
33%
43%
14%
Self-Report
4%
27%
50%
19%
Coaches’ Perspective
12%
31%
50%
8%
Cohort 1 Superintendents (N=21*)
Cohort 2 Superintendents (N=26)
* Data reported for individuals with both superintendent self-report and coach ratings
Five superintendents offered comments acknowledging the value of work related to developing effective
leadership teams. However, two of them also noted that they would have liked more guidance and support in this
area. Relevant comments included:
o
I really liked the skill and knowledge Principal Matrix assignment. It was helpful to ground
myself in something concrete and then use it as a way for administrators to rate their own
skill sets.
UMass Donahue Institute
Research and Evaluation Group
39
NSIP Annual Evaluation Report: FY 2012
Appendix A – Assignments and Work Products
o
It was great to think through the leadership team evaluation rubric and to process that with
my own group. This helped us grow as a group.
o
I feel more time should be spent on this component of the program. The development of a
solid leadership team is critical to a district’s success. . . this is one area in which I believe
people always need to refine and adjust their style/skills.
o
Tools for assessing the functioning of the leadership team were helpful, but there was little
guidance as to the interventions for specific types of team dysfunction
UMass Donahue Institute
Research and Evaluation Group
40
Appendix B – Impact on Superintendents
NSIP Annual Evaluation Report: FY 2012
Appendix B – Impact on Superintendents
Instructional Leadership
Most superintendents reported that they came into the superintendency with good or excellent abilities to serve as
an effective instructional leader. Even with this strong starting point, most reported that those abilities improved
at least modestly as a result of NSIP participation. In contrast, coaches’ perspectives of superintendent baseline
abilities were somewhat less generous, rating 40 percent of cohort 1 and 12 percent of cohort 2 superintendents’
baseline abilities as fair or poor. Furthermore, on the whole coaches perceived that NSIP had less impact than
perceived by the superintendents themselves. Coaches reported that NSIP has had little or no impact on
instructional leadership abilities for 40 percent of cohort 1 superintendents and 36 percent of cohort 2
superintendents.
Table 36: Superintendent Ability to Serve as an Effective Instructional Leader – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
20%
75%
5%
0%
0%
Coaches’ Perspective
30%
30%
30%
10%
0%
Self-Report
20%
80%
0%
0%
0%
Coaches’ Perspective
32%
52%
8%
4%
4%
Cohort 2 Superintendents (N=25*)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Table 37: Superintendent Ability to Serve as an Effective Instructional Leader – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
30%
60%
10%
0%
0%
Coaches’ Perspective
25%
35%
30%
10%
0%
Cohort 2 Superintendents (N=25*)
Self-Report
36%
28%
12%
24%
0%
Coaches’ Perspective
4%
60%
28%
8%
0%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Strategic Thinking
Most superintendents reported that they came into the superintendency with good or excellent ability to think
strategically about district improvement. As with instructional leadership, most superintendents also reported that
their abilities in this area have improved at least modestly as a result of NSIP participation. However, it is notable
that more than one-quarter of cohort 2 superintendents reported that NSIP has had little or no impact in this area.
UMass Donahue Institute
Research and Evaluation Group
41
Appendix B – Impact on Superintendents
NSIP Annual Evaluation Report: FY 2012
Again, coaches’ perspectives of superintendents’ starting points were somewhat less favorable, rating 52 percent
of cohort 1 superintendents’ baseline abilities as fair and 39 percent of cohort 2 superintendents’ baseline abilities
as fair or poor. For both cohorts most coaches reported that superintendent abilities to think strategically about
district improvement improved at least modestly as a result of NSIP. However, coaches reported only slight
improvement for nearly 20 percent of cohort 1 superintendents and 12 percent of cohort 2 superintendents.
Table 38: Superintendent Ability to Think Strategically About District Improvement – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
10%
81%
10%
0%
0%
Coaches’ Perspective
14%
29%
52%
0%
5%
Self-Report
46%
38%
15%
0%
0%
Coaches’ Perspective
12%
50%
31%
8%
0%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Table 39: Superintendent Ability to Think Strategically About District Improvement – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
48%
43%
5%
5%
0%
Coaches’ Perspective
52%
29%
19%
0%
0%
Cohort 2 Superintendents (N=26)
Self-Report
50%
19%
15%
12%
4%
Coaches’ Perspective
23%
65%
12%
0%
0%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Root Cause Analysis
Superintendents were less positive about their baseline ability to conduct root cause analysis of instructional
problems. Most superintendents rated their baseline ability as fair or good. Nearly one-quarter of cohort 2
superintendents rated their initial skills level as poor. According to coaches’ ratings about 60 percent of the
superintendents from each cohort started out with fair or poor ability to conduct root cause analysis. Most
superintendents and coaches reported that superintendent abilities to conduct root cause analysis improved at least
modestly as a result of NSIP participation. However, between one-quarter and one-third of all respondents
(superintendents and coaches) report that NSIP has had little or no impact in this area.
UMass Donahue Institute
Research and Evaluation Group
42
Appendix B – Impact on Superintendents
NSIP Annual Evaluation Report: FY 2012
Table 40: Superintendent Ability to Conduct Root Cause Analysis of Instructional Problems – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
5%
38%
57%
0%
0%
Coaches’ Perspective
0%
19%
33%
29%
19%
Self-Report
4%
42%
31%
23%
0%
Coaches’ Perspective
4%
12%
31%
31%
23%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Table 41: Superintendent Ability to Conduct Root Cause Analysis of Instructional Problems – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
29%
48%
19%
5%
0%
Coaches’ Perspective
19%
52%
10%
19%
0%
Cohort 2 Superintendents (N=26)
Self-Report
35%
35%
23%
8%
0%
Coaches’ Perspective
19%
42%
23%
4%
12%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Strategic Resource Management
Most superintendents reported that they started with good or excellent abilities to manage resources in a strategic
manner. About 58 percent of cohort 1 superintendents reported at least modest improvement in this area. In
contrast, only 27 percent of cohort 2 superintendents reported this level of improvement with 31 percent
indicating that their abilities improved slightly and 42 percent reporting that NSIP had no impact in this regard.
Coaches’ perspectives of cohort 1 baseline abilities were similar. However, they reported that nearly one-quarter
of cohort 2 superintendents entered with less developed abilities in this area. According to coaches, 43 percent of
cohort 1 and 66 percent of cohort 2 superintendents realized at least modest improvement in this area.
Table 42: Superintendent Ability to Manage Resources in a Strategic Manner – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
33%
48%
14%
0%
5%
Coaches’ Perspective
10%
67%
14%
0%
10%
Self-Report
23%
69%
8%
0%
0%
Coaches’ Perspective
4%
65%
23%
0%
8%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
UMass Donahue Institute
Research and Evaluation Group
43
Appendix B – Impact on Superintendents
NSIP Annual Evaluation Report: FY 2012
Table 43: Superintendent Ability to Manage Resources in a Strategic Manner – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
10%
48%
29%
14%
0%
Coaches’ Perspective
5%
38%
14%
38%
5%
Cohort 2 Superintendents (N=26)
Self-Report
12%
15%
31%
42%
0%
Coaches’ Perspective
8%
58%
23%
8%
4%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Developing School Leaders
Cohort 1 superintendents were somewhat less positive about their initial ability to develop school-level
instructional leaders with 57 percent describing their baseline ability as fair. As they completed their second year
in the program, a strong majority indicated they had realized at least modest improvement as a result of their
participation in NSIP. The perspectives of cohort 1 superintendents and their coaches were well-aligned.
In comparison, most cohort 2 superintendents perceived themselves as having relatively strong baseline abilities
in this area. While slightly more than half indicated that NSIP participation had caused at least modest
improvement in these skills 23 percent reported only slight improvement and 23 reported that the program had no
impact. Coaches’ perceptions of the starting point for cohort 2 were somewhat less favorable rating 43 percent of
them as fair or poor. Although they were also somewhat more positive about NSIP impact in this area, coaches
indicated that NSIP resulted in little or no improvement for nearly 40 percent of cohort 2 superintendents.
Table 44: Superintendent Ability to Develop Instructional Leadership Skills among School Leaders – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
19%
24%
57%
0%
0%
Coaches’ Perspective
10%
19%
52%
10%
10%
Self-Report
23%
65%
12%
0%
0%
Coaches’ Perspective
4%
50%
35%
8%
4%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Table 45: Superintendent Ability to Develop Instructional Leadership Skills among School Leaders – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
14%
67%
19%
0%
0%
Coaches’ Perspective
24%
48%
14%
10%
5%
Cohort 2 Superintendents (N=26)
Self-Report
15%
38%
23%
23%
0%
Coaches’ Perspective
12%
46%
31%
8%
4%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
UMass Donahue Institute
Research and Evaluation Group
44
Appendix B – Impact on Superintendents
NSIP Annual Evaluation Report: FY 2012
Leadership Team Development
Most superintendents reported that they came into the superintendency with good or excellent ability to create
effective leadership teams. Most superintendents also reported that their abilities in this area improved at least
modestly as a result of NSIP participation. However, it is notable that 40 percent of cohort 2 superintendents
reported that NSIP had little or no impact in this area.
Coaches’ perspectives of superintendent starting points were significantly less favorable, rating the baseline
ability of 62 percent of cohort 1 superintendents and 56 percent of cohort 2 superintendents as fair or poor. For
both cohorts most coaches reported that superintendents’ ability to think strategically about district improvement
improved at least modestly as a result of NSIP participation. However, coaches reported little or no improvement
for 38 percent of cohort 1 superintendents and 40 percent of cohort 2 superintendents.
Table 46: Superintendent Ability to Create Effective Leadership Teams – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
19%
62%
19%
0%
0%
Coaches’ Perspective
14%
19%
48%
14%
5%
Self-Report
20%
52%
20%
4%
4%
Coaches’ Perspective
4%
36%
48%
8%
4%
Cohort 2 Superintendents (N=25*)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Table 47: Superintendent Ability to Create Effective Leadership Teams – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
24%
71%
5%
0%
0%
Coaches’ Perspective
24%
38%
24%
14%
0%
Cohort 2 Superintendents (N=25*)
Self-Report
12%
48%
28%
12%
0%
Coaches’ Perspective
4%
52%
32%
8%
4%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Relationships with Key Constituencies
Superintendents and coaches were asked to rate baseline abilities and improvement in superintendents’ ability to
forge collaborative relationships with key constituencies specifically school committees, unions, business and
community leaders.
Most superintendents reported that they started with good or excellent abilities to establish collaborative
relationships with these constituencies. However, one-third of cohort 1superintendents indicated that their
baseline ability in this area was fair and 23 percent of cohort 2 superintendents indicated that their baseline ability
in this area was fair or poor. Overall coaches responses were reflective of superintendent self-reported abilities in
this area.
UMass Donahue Institute
Research and Evaluation Group
45
Appendix B – Impact on Superintendents
NSIP Annual Evaluation Report: FY 2012
More than half of cohort 1 superintendents reported at least modest improvement in this area. However, coaches’
perspectives on cohort 1 improvements were less generous with only 43 percent reporting at least modest impact
and 33 percent reporting that NSIP had no impact on cohort 1 superintendents’ abilities to forge collaborative
relationships with key constituencies.
In contrast to the first cohort, only 35 percent of cohort 2 superintendents reported modest or substantial
improvement with 33 percent reporting that NSIP had no impact in this regard. However, coaches’ perspectives
on improvement for cohort 2 were more favorable with half reporting at least modest improvement and only 15
percent reporting that the program had no impact in this regard.
Table 48: Superintendent Ability to Forge Collaborative Relationships with Key Constituencies – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
29%
38%
33%
0%
0%
Coaches’ Perspective
33%
33%
33%
0%
0%
Self-Report
38%
38%
19%
4%
0%
Coaches’ Perspective
23%
46%
23%
4%
4%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Table 49: Superintendent Ability to Forge Collaborative Relationships with Key Constituencies – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
19%
33%
38%
10%
0%
Coaches’ Perspective
14%
29%
19%
33%
5%
Cohort 2 Superintendents (N=26)
Self-Report
8%
27%
38%
27%
0%
Coaches’ Perspective
12%
38%
31%
15%
4%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Superintendents and coaches were also asked to rate the nature of superintendents relationships with each of those
key constituencies. At least 80 percent reported that superintendents had positive (good or excellent) relationships
with each group. Positive relationships with central office leaders were most widespread, which seems logical
given the day-to-day working relationships that most superintendents have with their central office teams.
Overall, cohort 1 superintendent and coach responses were well aligned. However, there was more divergence for
cohort 2 with superintendents rating their relationships more positively than their coaches. Most notably, all
cohort 2 superintendents positively characterized their relationships with school building leaders compared with
only 81 percent of their coaches. In addition, 96 percent of cohort 2 superintendents reported positive
relationships with community and business leaders compared with 81 percent of their coaches.
UMass Donahue Institute
Research and Evaluation Group
46
Appendix B – Impact on Superintendents
NSIP Annual Evaluation Report: FY 2012
Table 50: Superintendent Relationship with Key Constituencies
Percent Responding “Excellent” or “Good”
Cohort 1
Cohort 2
Superintendents
Coaches
Superintendents
Coaches
School committee
81%
86%
96%
88%
Central office leaders
95%
95%
96%
92%
School building leaders
86%
81%
100%
81%
Teachers union
81%
86%
92%
83%
Community & business leaders
90%
90%
96%
81%
* Data reported for individuals with superintendent self-report and coach ratings
Supervision and Evaluation Systems
Superintendents were less positive about their initial ability to establish effective supervision and evaluation
systems. Most superintendents and coaches rated baseline ability as fair or good. Ratings of impact diverge
significantly for the two cohorts. Given NSIP’s focus on educator evaluation implementation for superintendents
in cohort 1, it is not surprising to see that 80 percent of superintendents and coaches reported modest to
substantial improvement in this area. In contrast, this was not yet a strong focus of the NSIP program as delivered
to cohort 2 in the 2011-2012 school year. As such, 61 percent of cohort 2 superintendents and 43 percent of their
coaches reported little or no impact in abilities to establish effective systems for supervision and evaluation.
Table 51: Superintendent Ability to Establish Effective Systems for Supervision and Evaluation – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
10%
45%
45%
0%
0%
Coaches’ Perspective
5%
30%
55%
5%
5%
Self-Report
15%
58%
23%
0%
4%
Coaches’ Perspective
4%
27%
46%
0%
23%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Table 52: Superintendent Ability to Establish Effective Systems for Supervision and Evaluation – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
25%
55%
20%
0%
0%
Coaches’ Perspective
30%
50%
15%
5%
0%
Cohort 2 Superintendents (N=26)
Self-Report
12%
23%
42%
19%
4%
Coaches’ Perspective
0%
38%
31%
12%
19%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
UMass Donahue Institute
Research and Evaluation Group
47
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
Appendix C – Impact on Districts
District-level Focus on Instructional Improvement
Cohort 1 superintendents and coaches agreed that initially few NSIP districts had excellent focus on instructional
improvement12. Half of the superintendents described their district’s baseline as fair and 30 percent rated it as
good. Their coaches’ perspectives were somewhat more split with 40 percent rating the districts as good and 35
percent rating them as fair. Nearly all coaches and most of the superintendents reported that their district-level
focus on instructional improvement improved at least modestly as a result of participation in NSIP. However, it is
notable that one-quarter of cohort 1 superintendents reported that NSIP had little or no impact in this area.
Cohort 2 superintendents offered a more favorable view of their districts’ initial focus on instructional
improvement with 69 percent rating them as good and 12 percent as excellent. Their coaches were less positive
with 57 percent indicating fair or poor. Similar to other findings, cohort 2 superintendents reported much less
impact with 19 percent indicating that there was slight improvement and 31 that there was no impact on their
district. Coaches offered a more favorable view of NSIP’s impact on the cohort 2 districts’ focus on instructional
improvement with 31 reporting substantial improvement and 46 percent reporting modest improvement.
Table 53: District-Level Focus on Instructional Improvement – Baseline**
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
5%
30%
50%
15%
0%
Coaches’ Perspective
5%
40%
20%
35%
0%
Self-Report
12%
69%
19%
0%
0%
Coaches’ Perspective
8%
31%
42%
15%
4%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
Table 54: District-Level Focus on Instructional Improvement – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
35%
40%
20%
5%
0%
Coaches’ Perspective
35%
60%
5%
0%
0%
Cohort 2 Superintendents (N=26)
Self-Report
19%
31%
19%
31%
0%
Coaches’ Perspective
31%
46%
23%
0%
0%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
One superintendent and one coach reported indicated responded “excellent” for a cohort 1 district. However, they were commenting on
different districts. For both of those districts the other respondent answered “good.” As might be expected, these are the same respondents
that responded “excellent” to the item about instructional improvement strategy.
12
UMass Donahue Institute
Research and Evaluation Group
48
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
District-level Strategy for Instructional Improvement
Comparing the responses about district focus to district strategy show that even districts that were perceived to
have had relatively good focus on instructional improvement had not effectively translated that focus into strong
instructional improvement strategies. Cohort 1 superintendents and coaches report that 75 percent of those
districts had fair or poor strategies. Cohort 2 superintendents were more generous in the evaluation of their district
strategies with 58 percent characterizing them as good or excellent. However, coaches’ ratings of strategies in
cohort 2 districts were far less favorable with 85 percent rating them as fair or poor.
Superintendent and coach responses for cohort 1 suggest that NSIP had a positive impact on district-level
strategies with 86 percent reporting modest or substantial improvement. Although not as strong, most cohort 2
respondents also reported at least modest improvement. However, about one-third of cohort 2 superintendents
reported that NSIP had little or no positive impact on district strategy and coaches reported that more than onequarter of cohort 2 districts improved their strategies as a result of their superintendents’ participation in NSIP.
This likely reflects differences in the cohorts’ stages of program implementation. In their second year of the
program, most cohort 1 superintendents had articulated their own strategies and were beginning to integrate them
into the district. In contrast, many cohort 2 superintendents had strengthened their own strategic thinking skills,
but were only beginning to translate their entry findings and subsequent analyses into strategy documents.
Table 55: District-Level Strategy for Instructional Improvement – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
5%
24%
29%
43%
0%
Coaches’ Perspective
5%
24%
24%
48%
0%
Self-Report
12%
46%
38%
4%
0%
Coaches’ Perspective
4%
8%
54%
31%
4%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
Table 56: District-Level Strategy for Instructional Improvement – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
48%
38%
14%
0%
0%
Coaches’ Perspective
38%
48%
14%
0%
0%
Cohort 2 Superintendents (N=26)
Self-Report
23%
38%
19%
15%
4%
Coaches’ Perspective
31%
42%
27%
0%
0%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
UMass Donahue Institute
Research and Evaluation Group
49
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
School-level Focus on Instructional Improvement
Nearly all cohort 1superintendents and most coaches described initial school-level focus on instructional
improvement as fair or good. Notably, about one-quarter of the coaches described it as poor. About 70 percent of
superintendents and their coaches reported that school-level focus on instructional improvement improved at least
modestly as a result of participation in NSIP.
Cohort 2 superintendents offered a more favorable view of schools’ initial focus on instructional improvement
with 58 percent rating it as good and an additional 12 percent as excellent. In contrast, their coaches were less
positive with 54 percent indicating fair or poor. Cohort 2 superintendents reported much less impact with 62
percent indicating that NSIP resulted in little or no improvement of school-level focus on instructional
improvement. Coaches offered a more favorable view of NSIP’s impact with 19 reporting substantial
improvement and 35 percent reporting modest improvement. However, 42 percent of coaches reported little or no
improvement in school-level focus within cohort 2 districts. Again, this is likely a reflection of the time it takes
for changes in superintendent abilities and practice to translate into changes in the districts and its schools.
Table 57: School-Level Focus on Instructional Improvement – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
0%
38%
57%
5%
0%
Coaches’ Perspective
0%
38%
33%
24%
5%
Self-Report
12%
58%
31%
0%
0%
Coaches’ Perspective
4%
38%
46%
8%
4%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
Table 58: School-Level Focus on Instructional Improvement – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
33%
38%
24%
5%
0%
Coaches’ Perspective
29%
43%
24%
5%
0%
Cohort 2 Superintendents (N=26)
Self-Report
12%
23%
35%
27%
4%
Coaches’ Perspective
19%
35%
27%
15%
4%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Principal Readiness to Lead Instructional Improvement
Most cohort 1 superintendents and coaches reported that their principals’ baseline readiness to lead instructional
improvement was only fair or poor. Most cohort 1 superintendents (85 percent) and coaches (60 percent) reported
at least modest improvement in principal readiness resulting from NSIP.
UMass Donahue Institute
Research and Evaluation Group
50
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
Similar to previous aspects of district performance, cohort 2 superintendents offered a more favorable assessment
of principals’ baseline readiness with nearly 70 percent rating them as excellent or good and 57 percent report
little or no improvement as a result of NSIP. Again, cohort 2 coaches’ perspectives of baseline readiness were less
positive with 54 percent responding fair or poor. However, they were more positive about NSIP impact reporting
at least a modest improvement in principal readiness within 58 percent of cohort 2 districts.
Table 59: Principal Readiness to Lead Instructional Improvement – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
0%
30%
45%
25%
0%
Coaches’ Perspective
15%
15%
40%
25%
5%
Self-Report
15%
54%
31%
0%
0%
Coaches’ Perspective
8%
35%
42%
12%
4%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
Table 60: Principal Readiness to Lead Instructional Improvement – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
20%
65%
15%
0%
0%
Coaches’ Perspective
15%
45%
30%
10%
0%
Cohort 2 Superintendents (N=26)
Self-Report
8%
31%
19%
38%
4%
Coaches’ Perspective
12%
46%
23%
19%
0%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Overall Quality of Instruction
Consistent with NSIP’s initial focus on districts in Level 3 or Level 4 accountability status, most cohort 1
superintendents and coaches described their districts’ baseline instructional quality as good or fair. About half
reported that instructional quality improved at least modestly as a result of NSIP participation. However,
reflective of the time needed to translate leadership improvements into better classroom practice, more than 40
percent of superintendents and coaches reported little or no improvement in instructional quality.
As seen in other areas and likely reflecting the involvement of Level 1 and 2 districts, cohort 2 superintendents
offered a more favorable view of initial instructional quality in their districts with 85 percent rating it as good or
excellent. Even so, coaches were less positive with 42 percent rating baseline instructional quality as only fair.
Predictably, given their higher baseline assessment and the fact that they had just completed their first year of the
program, cohort 2 superintendents perceive less impact with fewer than one-quarter reporting modest
improvement. In contrast, coach ratings of improvement were similar for both cohorts.
UMass Donahue Institute
Research and Evaluation Group
51
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
Table 61: Overall Quality of Instruction in the District – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
10%
38%
43%
10%
0%
Coaches’ Perspective
5%
24%
62%
10%
0%
Self-Report
8%
77%
15%
0%
0%
Coaches’ Perspective
4%
46%
42%
0%
8%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
Table 62: Overall Quality of Instruction in the District – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
5%
52%
38%
5%
0%
Coaches’ Perspective
5%
43%
38%
10%
5%
Cohort 2 Superintendents (N=26)
Self-Report
0%
23%
38%
35%
4%
Coaches’ Perspective
4%
42%
31%
15%
8%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Alignment of Budget to District Strategy
For both cohorts there was considerable initial variation in how well districts aligned their budget to their overall
improvement strategies, with superintendents and coach responses generally ranging from poor to good. More
than half of cohort 1 superintendents and their coaches reported that their districts’ budget and strategy alignment
improved at least modestly as a result of NSIP participation. As with other aspects of district practice, cohort 2
superintendents reported much less impact with 46 percent reporting no impact and 23 percent indicating that
there was only slight improvement. Again, coaches were somewhat more positive about the impact on cohort 2
districts with 55 percent attributing at least modest improvement to the superintendents’ participation in NSIP.
Table 63: Alignment of Budget to Overall District Improvement Strategy – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
0%
35%
30%
35%
0%
Coaches’ Perspective
0%
30%
30%
15%
25%
Self-Report
8%
35%
38%
19%
0%
Coaches’ Perspective
0%
15%
46%
23%
15%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
UMass Donahue Institute
Research and Evaluation Group
52
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
Table 64: Alignment of Budget to Overall District Improvement Strategy – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
5%
55%
20%
20%
0%
Coaches’ Perspective
15%
40%
10%
15%
20%
Cohort 2 Superintendents (N=26)
Self-Report
0%
27%
23%
46%
4%
Coaches’ Perspective
27%
38%
23%
4%
8%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Productivity of School Committee Meetings
There was also considerable initial variation in the perceived productivity of school committees among cohort 1
districts with most responses falling between poor and fair. About 57 percent of cohort 1 superintendents and 43
percent of their coaches reported that school committee productivity improved at least modestly as a result of
NSIP participation. However, it should be noted that about one-third of cohort 1 superintendents and coaches did
report that NSIP had no impact on school committee productivity. This further underscores the recognition that
building effective working relationships with school committees is a challenge that is not substantially addressed
through the content day curriculum.
Coaches’ initial perceptions of school committee productivity in cohort 2 districts were generally consistent with
reports from cohort 1 superintendents and their coaches. Yet again, cohort 2 superintendents offered more a
favorable baseline view of the productivity of school committee meetings with nearly two-thirds assessing them
as good or excellent. Cohort 2 superintendents reported much less impact with 58 percent reporting no impact.
Their coaches were more varied about the impact on cohort 2 districts with responses fairly well distributed across
the spectrum from no impact to substantial improvement.
Table 65: Productivity of School Committee Meetings – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
10%
29%
33%
29%
0%
Coaches’ Perspective
10%
5%
33%
33%
19%
Self-Report
15%
50%
19%
15%
0%
Coaches’ Perspective
0%
23%
38%
27%
12%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
UMass Donahue Institute
Research and Evaluation Group
53
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
Table 66: Productivity of School Committee Meetings – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
19%
38%
14%
29%
0%
Coaches’ Perspective
14%
29%
14%
33%
10%
Cohort 2 Superintendents (N=26)
Self-Report
8%
23%
12%
58%
0%
Coaches’ Perspective
19%
23%
23%
23%
12%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Productivity of District Leader Team Meetings
Most cohort 1superintendents and coaches described the initial productivity of district leadership team meetings
as fair or good. Notably, nearly 20 percent of coaches described it as poor. More than 70 percent of cohort 1
superintendents and their coaches reported district leadership team productivity improved at least modestly as a
result of participation in NSIP.
More than three-quarters of cohort 2 superintendents assessed their district leadership teams’ initial productivity
as good or excellent. In contrast, their coaches were less positive with 61 percent indicating fair or poor. In
contrast to other aspects of district functioning, cohort 2 superintendents did report that NSIP had a positive
impact in this area with nearly three-quarters reporting at least modest improvement. In fact, in this instance,
coaches offered a more varied view with slightly more than half reporting at least modest improvement and nearly
40 percent reporting little or no impact.
Table 67: Productivity of District Leader Team Meetings – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
5%
33%
48%
14%
0%
Coaches’ Perspective
5%
19%
38%
19%
19%
Self-Report
31%
46%
23%
0%
0%
Coaches’ Perspective
0%
15%
42%
19%
23%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
UMass Donahue Institute
Research and Evaluation Group
54
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
Table 68: Productivity of District Leader Team Meetings – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=21*)
Self-Report
29%
43%
19%
10%
0%
Coaches’ Perspective
29%
43%
14%
10%
5%
Cohort 2 Superintendents (N=26)
Self-Report
31%
42%
19%
8%
0%
Coaches’ Perspective
35%
19%
35%
4%
8%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Administrator and Teacher Evaluation Systems
Superintendents and coaches from both cohorts had relatively poor views of districts’ systems for evaluating
teachers and administrators with most characterizing them as fair or poor. As with the corresponding question
about superintendents’ own abilities to effectively implement supervision and evaluation, ratings of impact
diverged significantly for the two cohorts. Given NSIP’s deliberate focus on educator evaluation implementation
for superintendents in cohort 1, it is not surprising to see a higher degree of improvement for those districts.
However, even the responses for cohort 1 suggest that the program’s efforts around educator evaluation have not
yet had a widespread impact on those systems as implemented in participating districts.
Table 69: Administrator Evaluation Systems (as implemented) – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
5%
10%
40%
40%
5%
Coaches’ Perspective
0%
20%
25%
50%
5%
Self-Report
4%
12%
58%
23%
4%
Coaches’ Perspective
0%
12%
23%
23%
42%
Cohort 2 Superintendents (N=26)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
UMass Donahue Institute
Research and Evaluation Group
55
Appendix C – Impact on Districts
NSIP Annual Evaluation Report: FY 2012
Table 70: Administrator Evaluation Systems (as implemented) – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=20*)
Self-Report
20%
45%
20%
15%
0%
Coaches’ Perspective
15%
35%
20%
30%
0%
Cohort 2 Superintendents (N=26)
Self-Report
4%
15%
23%
54%
4%
Coaches’ Perspective
4%
19%
27%
19%
31%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
Table 71: Teacher Evaluation Systems (as implemented) – Baseline
Excellent
Good
Fair
Poor
Unsure
Cohort 1 Superintendents (N=18*)
Self-Report
6%
11%
61%
22%
0%
Coaches’ Perspective
0%
28%
22%
39%
11%
Self-Report
4%
20%
52%
20%
4%
Coaches’ Perspective
0%
12%
28%
15%
44%
Cohort 2 Superintendents (N=25*)
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
** A note about baseline data: Cohort 1 baseline data are taken from the spring 2011 superintendent survey. Cohort 2 baseline data come
from the fall 2011 superintendent survey. Coaches’ perspectives are from the spring 2012 survey.
Table 72: Teacher Evaluation Systems (as implemented) – NSIP Impact
Substantially
Improved
Modestly
Improved
Slightly
Improved
No Impact
Unsure
Cohort 1 Superintendents (N=18*)
Self-Report
11%
39%
17%
33%
0%
Coaches’ Perspective
0%
44%
28%
28%
0%
Cohort 2 Superintendents (N=25*)
Self-Report
0%
8%
28%
60%
4%
Coaches’ Perspective
0%
16%
24%
20%
40%
* Data reported for individuals with superintendent self-report and coach ratings for both baseline ability and impact
UMass Donahue Institute
Research and Evaluation Group
56