Massachusetts Race to the Top College and Career
Readiness Initiatives
Evaluation Annual Report, September 2012
Prepared for the Massachusetts Department of Elementary and Secondary
Education
RTTT C&CR Evaluation Annual Report, September 2012
Acknowledgements
The UMass Donahue Institute extends its sincere appreciation to the many people who supported and
collaborated with us on this evaluation. In particular, we want to thank personnel from the Massachusetts
Department of Elementary and Secondary Education and from the schools, districts, partners, and vendors
associated with the Massachusetts Pre-AP, STEM Early College High School, and MassCore initiatives.
Massachusetts Race to the Top College and Career Readiness Initiatives
Evaluation Annual Report, September 2012
Project Staff
Eliot Levine, Senior Research Manager, Project Manager
Jeremiah Johnson, Research Manager
Patricia Lee, Research Analyst
Steven Ellis, Director, Research and Evaluation
Report Information
This report was prepared by the UMass Donahue Institute, the project evaluator, under contract with the
Massachusetts Department of Elementary and Secondary Education.
About the Donahue Institute
The University of Massachusetts Donahue Institute is the public service, outreach, and economic
development unit of the University of Massachusetts President’s Office. Established in 1971, the Institute
strives to connect the Commonwealth with the resources of the University through services that combine
theory and innovation with public and private sector applications.
UMDI’s Applied Research and Program Evaluation group specializes in applied social science research,
including program evaluation, survey research, policy research, and needs assessment. The group has
designed and implemented research and evaluation projects for diverse programs and clients in the areas
of education, human services, economic development, and organizational development.
University of Massachusetts Donahue Institute
Applied Research and Program Evaluation Group
100 Venture Way Suite 5
Hadley, MA 01035-9462
UMass Donahue Institute
Research and Evaluation Group
413-587-2400 (phone)
413-587-2410 (fax)
www.donahue.umassp.edu
2
RTTT C&CR Evaluation Annual Report, September 2012
Table of Contents
Acknowledgements ..................................................................................................................................... 2
Executive Summary .................................................................................................................................... 4
Introduction ............................................................................................................................................... 10
Pre-AP Training and Alignment ............................................................................................................. 12
STEM-focused Early College High Schools............................................................................................ 51
MassCore Policy and Implementation .................................................................................................... 75
Appendices ................................................................................................................................................. 96
Appendix A1: Pre-AP Field Interview Protocol, Teacher or Administrator.................................. 97
Appendix A2: Pre-AP Teacher and Lead Teacher Survey .......................................................... 101
Appendix A3: Pre-AP Administrator Survey .............................................................................. 112
Appendix A4: Pre-AP Vendor Interview Protocol ...................................................................... 119
Appendix A5: Pre-AP Vertical Team Meeting Observation Protocol ......................................... 122
Appendix A6: Pre-AP Teacher Summer Training Observation Protocol .................................... 123
Appendix B1: ESE Interview Protocol ........................................................................................ 124
Appendix B2: STEM ECHS JFF Interview Protocol .................................................................. 126
Appendix B3: STEM ECHS Pre-Implementing Schools Interview Protocol .............................. 128
Appendix B4: STEM ECHS Implementing Schools Interview Protocol .................................... 131
Appendix B5: STEM ECHS IHE Partner Interview Protocol ..................................................... 134
Appendix B6: STEM ECHS School Personnel Focus Group Protocol ....................................... 136
Appendix B7: STEM ECHS Observation Protocol ..................................................................... 138
Appendix C1: MassCore Administrator Interview Protocol ........................................................ 139
Appendix C2: MassCore District Administrator Survey ............................................................. 142
UMass Donahue Institute
Research and Evaluation Group
3
RTTT C&CR Evaluation Annual Report, September 2012
Executive Summary
The Massachusetts Department of Elementary and Secondary Education (ESE) was awarded a federal
Race to the Top (RTTT) grant for the years 2010-2014. The college and career readiness (C&CR)
components of the grant were designed to help students develop "knowledge and skills necessary for
success in postsecondary education and economically viable career pathways." The RTTT initiatives are
intended to provide students with opportunities to participate in quality, upper-level high school
coursework and new approaches to assist them with high school completion and transition to higher
education and the workforce. These programs are part of a broader effort, as specified in the Delivery
Plan of ESE's College and Career Readiness group, to increase the Massachusetts 5-year high school
graduation rate to 88.3% and the number of students who complete the MassCore program of study to
85% statewide.
The UMass Donahue Institute is conducting an evaluation of three of the C&CR components of the
Massachusetts RTTT efforts – the Pre-AP Training and Alignment Program, the STEM-focused Early
College High Schools (ECHS), and the MassCore Policy and Implementation initiative.
For each of these three programs, this executive summary provides a brief program description, followed
by evaluation findings for the period ending September 30, 2012 (referred to as Year 2). Strategic
considerations are also presented for some programs. The full report that follows the executive summary
incorporates findings from all evaluation activities conducted to date, including a synthesis of the three
previous quarterly evaluation briefings. A companion document, the Technical Supplement – RTTT
C&CR 2012 Annual Evaluation Report, presents tables and figures that provide a more complete
summary of baseline graduation and dropout data.
Pre-AP Training and Alignment
The aims of the Pre-AP Training and Alignment program are to increase the number of low-income and
minority students prepared to participate and succeed in mathematics, science, and English Advanced
Placement courses and credit-bearing college-level coursework; to provide teachers in grades 6-12 with
high-quality professional development to assist them in developing curricula, instruction, and assessments
that prepare students for AP coursework; and to provide an opportunity for teachers to collaborate in
horizontal and vertical teams and to network with other teachers in their region for the purpose of
improving curriculum and instruction. Technical assistance for the project is based on the Laying the
Foundation (LTF) curriculum and is provided by the Massachusetts Math + Science Initiative (MMSI), a
project of Mass Insight Education in partnership with the Commonwealth of Massachusetts.
Districts are participating in Pre-AP training in multiple configurations. The configuration for RTTT Goal
4D is to send the same teachers for four-day trainings in three consecutive summers and to create
discipline-specific vertical teams, led by trained lead teachers, that meet quarterly to share and deepen
their Pre-AP practices. Districts that are using Goal 4A or non-RTTT funds have created several
variations of the basic training configuration, and the percentage of all teachers in a given school and
district who are being trained varies widely. Districts also selected different disciplines or combinations
of disciplines among the mathematics, science, and ELA trainings offered.
The Year 2 evaluation of the Pre-AP initiative included the following activities and data sources:
interviews of teachers and administrators; surveys of teachers, lead teachers, and administrators;
UMass Donahue Institute
Research and Evaluation Group
4
RTTT C&CR Evaluation Annual Report, September 2012
interviews of ESE Program Manager and Project Director; interviews of technical assistance vendor;
observations of vertical team meetings; observations of Pre-AP training sessions; training registration and
attendance database; vertical team meeting attendance database; supplemental school data requests; ESE
documents and databases; and, related project documents.
The Pre-AP Training and Alignment initiative is well underway. During summer 2011, MMSI provided
four-day summer trainings for 561 teachers from 37 districts (including 19 4D districts), and teachers
attended an average of 3.6 days (out of 4). Also during summer 2011, 30 teachers participated in lead
teacher training, and 25 administrators (from seven 4D districts) participated in administrator training. For
summer 2012, more than 900 teachers registered for training; final registration data were not available for
this report. The ESE program manager reported that, after summer 2012, Massachusetts will have met the
RTTT performance goal of having trained 1,000 teachers. In addition, the technical assistance and support
provided by MMSI and ESE were widely seen as helpful and satisfactory.
Lead teachers and administrators generally reported being satisfied with the Pre-AP training events.
Administrators and teachers largely agree that the LTF lessons and assessments are well-aligned with the
Common Core standards and are examples of high-quality pedagogical practices. Teachers were asked
about the impacts of their participation in Pre-AP training, and 70% or more of 4D teachers agreed that,
as a result of the Pre-AP training, they now teach more Pre-AP content, use more Pre-AP pedagogical
strategies in the classroom, and have greater awareness of the importance of using Pre-AP strategies.
Substantial challenges to Pre-AP implementation were reported by both administrators and teachers with
regard to curricular resources, classroom equipment, planning time, pacing guides, and student academic
readiness. Such concerns represented a substantial minority in some cases, and a clear majority in others,
particularly in relation to the adequacy of planning time and the academic readiness of students for the
Pre-AP curriculum. The number of Pre-AP lessons and assessments that teachers are expected to
implement varies widely across districts.
During the 2011-12 school year, 4D district teams reported attendance at an average of 3.5 vertical team
meetings, with a range of 2 to 4 meetings. The meetings were attended by an average of 63% of summer
2011 trainees, plus an average of four personnel who hadn’t completed Pre-AP training. Some lead
teachers were not satisfied with the agendas and activities provided by MMSI to structure vertical team
meetings. Some lead teachers made slight modifications to agendas, and others completely restructured
agendas to meet their district’s needs. The MMSI program director reported encouraging lead teachers to
make changes as needed to meet their school’s goals.
The LTF program philosophy, strongly shared by ESE, is that every student should be considered a PreAP student, but evaluation findings indicate diverging voices and practices on this topic in the field. Most
teachers and administrators who were interviewed agreed that every student should be considered a PreAP student, and multiple interviewees expressed mild to strong opposition to tracking. In contrast,
responses to the teacher survey provided more varied responses. The survey asked teachers if they used
more Pre-AP materials and strategies with some classes than with others. About half said yes, and most of
those indicated that they used more Pre-AP materials and strategies with their honors or advanced-level
students. This group of respondents generally agreed that Pre-AP materials were not suitable for all
students, such as those in remedial classes, those who had not mastered skills from a previous course, and
some English language learners.
Strategic considerations (explained in greater detail in the full report).
UMass Donahue Institute
Research and Evaluation Group
5
RTTT C&CR Evaluation Annual Report, September 2012

Expectations for minimum implementation levels should be clarified or reiterated.

Providing additional materials and suggestions for differentiation of LTF lessons may
increase classroom implementation.

Further assessments of MMSI trainer quality, and additional training of some trainers, may be
indicated.

The level of implementation of Pre-AP practices should be monitored and interpreted in
relation to student impacts.

Pre-AP schools may need to budget for additional equipment and professional development.

Clarification of the intended uses of vertical team meeting time might lead to increased PreAP implementation.

Sharing of modified vertical team meeting materials could save lead teacher time and
enhance meeting productivity.
STEM-focused Early College High Schools
In its Race to the Top proposal, ESE proposed to open six STEM early college high schools to prepare
students for productive STEM careers, to reduce achievement gaps, and to provide an accelerated
pathway to postsecondary education for underrepresented students by partnering with colleges and
providing opportunities to earn up to two years of college credit while still in high school. Six districts
were chosen in a competitive process and are currently receiving RTTT Goal 4E funds for this purpose.
Eight additional STEM ECHS sites in the state are being established, most with the support of
discretionary RTTT funds. These additional sites will be included in future evaluation activities, but ESE
has requested that UMDI wait until the work of those sites has progressed further.
Evaluation activities during Year 2 included interviews with the ESE Program Manager for STEM ECHS,
technical assistance vendor, administrators, and IHE partners; a focus group with school personnel from
one implementing site; two site visits, and analysis of relevant documents and databases. Baseline
graduation and dropout data are presented for STEM ECHS schools and districts.
The six Goal 4E STEM ECHS sites are in various phases of project development and implementation.
Two sites enrolled students in STEM ECHS activities during the 2011-12 school year, three sites plan to
start during the 2012-13 school year, and the last site’s STEM ECHS work has been incorporated into a
broader school redesign plan.
Interviewees from five districts described assembling a planning team as a successful aspect of their
planning process. Additional factors that contributed to the success of the planning teams were including
representatives from diverse stakeholder groups and consistent collaboration across members.
Respondents shared that holding planning meetings on a regular basis was difficult, because most team
members had many other substantial responsibilities and commitments. Interviewees from all districts
cited support from district leadership as essential to the planning process and establishing strong working
relationships with their IHE partner.
Informants reported many challenges that arose during the planning phase, including budget management,
instruction and curriculum development, staffing, engaging the community, and working with teachers’
UMass Donahue Institute
Research and Evaluation Group
6
RTTT C&CR Evaluation Annual Report, September 2012
unions. Several districts also reported challenges related to the quality, frequency, or clarity of
communication across stakeholders.
Marlborough is the only site that began full implementation during Year 2, with 189 students in grades 6
and 9. One Marlborough interviewee said that a primary success has been the staff’s ability to encourage
and support all students in their efforts to complete a curriculum of five honors-level courses. All teachers
said that the majority of STEM ECHS students were having a positive experience and “moving in the
right direction.” The interviewees expressed that the following factors have contributed to their successful
implementation to date: strong curriculum development, a collaborative partnership with the IHE, and the
district’s commitment to providing teachers with time for professional collaboration.
Many challenges were reported by Marlborough teachers. One high school teacher reported that some
colleagues working within the main high school resent the increased resources (e.g., structured planning
and collaboration time, professional development) and decreased class sizes enjoyed by the STEM ECHS
teachers. A middle school teacher reported that building administrators assigned personnel to the STEM
ECHS without giving them the opportunity to opt out of the program. STEM ECHS teachers also reported
that they had faced many challenges with technology, primarily related to the one-to-one laptop initiative,
which was not fully implemented until five months into the school year. Teachers said that it was difficult
and time-consuming to integrate the use of laptops into the curriculum at that point in the year.
All districts reported that ESE had consistently provided effective, professional, and timely assistance.
Districts noted that ESE personnel were aware of, and sensitive to, the contexts in which the districts were
working. Jobs For the Future, working in consultation with ESE, provided technical assistance to the Goal
4E STEM ECHS sites. The support provided to the sites has individual technical assistance, meetings
with all six schools, and “just-in-time” problem solving. The majority of districts said that JFF was
viewed as a valued partner that was helping to move their programs forward.
All interviewees (administrators, teachers, IHE partners, technical assistance vendors, and ESE) reported
that finding the financial resources to sustain these STEM ECHS sites will be a significant challenge. No
districts have established a clear plan for addressing this issue, although several said that they anticipated
committing increased levels of attention to sustainability issues over time. One interviewee also suggested
that, over time, the state could consider adopting policies that would benefit or expedite the development
of STEM ECHSs and other programs providing high school students with early college experiences.
Strategic considerations (Explained in greater detail in the full report).

Initial lessons from the early-implementing STEM ECHS schools could benefit the other
STEM ECHS schools and inform technical assistance from JFF and ESE.

Continuity of leadership and succession planning appear important to success of the STEM
ECHS schools.

Securing the financial resources to sustain STEM ECHS partnerships is a primary concern of
all STEM ECHS schools.

Changes to state and local policies may facilitate STEM ECHS development.

Increased connections with the broader STEM community could benefit the STEM ECHS
schools.
MassCore Policy and Implementation
UMass Donahue Institute
Research and Evaluation Group
7
RTTT C&CR Evaluation Annual Report, September 2012
The Massachusetts High School Program of Studies (MassCore) recommends a set of courses and other
learning opportunities which Massachusetts students should complete before graduating from high
school, in order to arrive at college or the workplace well-prepared and without need for remedial
coursework. The 155 districts that selected the RTTT college and career readiness goal committed to
implementing strategies to increase the percentage of their students who complete the MassCore
curriculum. The state’s RTTT goal is to increase the statewide MassCore completion rate from the 2010
baseline of 70% to 85% of graduates by 2014, and the state has created a formula-based goal for each
district. Each district was also expected to determine areas in which courses or supports needed to be
expanded in order to meet the 2014 targets, and to create and implement a plan to improve the accuracy of
their reporting of MassCore completion levels.
Evaluation activities during Year 2 included an interview with several RTTT district administrators, a
survey of RTTT district administrators, interviews with the ESE Program Manager for MassCore and
ESE Project Director for the RTTT C&CR evaluation, and analysis of ESE documents and databases.
Baseline graduation and dropout data are presented for RTTT MassCore districts.
Districts proposed many approaches to moving toward greater implementation of the MassCore program
of studies. These included structural changes (e.g., adding courses, shifting to block scheduling),
changing policies (e.g., permitting after-school sports to fulfill physical education requirements),
collaborating with external vendors (e.g., to introduce online credit recovery courses). Districts also
reported many challenges. The most common curricular challenge was the required fourth year of
mathematics, but lab sciences, physical education, world languages, and fine arts were also mentioned.
Additional challenges included inadequate funds and staffing in some curricular areas, course scheduling,
student academic readiness, severe student learning disabilities, and the fact that MassCore was not a
required curriculum. Districts suggested numerous supports that they believed could facilitate their efforts
at increasing MassCore completion rates.
ESE's intention was for MassCore to shift from a recommended to a de facto curriculum for the state
during the first year of RTTT. This is still the state's intention, but it has not yet been realized. As a result,
although many districts believe that increasing MassCore completion is a worthy goal, are committing
RTTT and other resources toward that end, and believe it is a realistic goal for the majority of their
students, districts have no clear accountability to the state for achieving progress and may be prioritizing
other initiatives that have greater accountability or a higher profile. In addition, substantial validity
limitations of the state’s MassCore completion indicator raise serious concerns about the ability to assess
changes in MassCore completion accurately at the state and district levels.
Strategic considerations (Explained in greater detail in the full report).

Most districts see substantial value in raising MassCore completion rates, but limited
accountability for reaching state MassCore targets may subordinate MassCore among district
priorities.

Meaningful assessment of changes in MassCore completion is challenged by serious validity
and reliability limitations of the MassCore element of the SIMS database.

A MassCore exemption process, or the promotion of alternative ways to fulfill certain
requirements, may reduce some districts’ concerns about conflicting mandates and priorities.
UMass Donahue Institute
Research and Evaluation Group
8
RTTT C&CR Evaluation Annual Report, September 2012

MassCore specifies the quantity of courses that must be completed, but questions remain
regarding expectations of course quality.

The most common challenges to increasing MassCore completion rates are: building capacity
to provide required courses, services, and facilities; inadequate funding; the gap between
student skill levels and MassCore expectations; course scheduling; and MassCore completion
not being required for graduation.

Additional supports and policy changes may increase district progress toward MassCore
completion goals.

More than a quarter of district administrator survey respondents did not agree that their
district is using RTTT funds to take actions that they believe will lead to increased rates of
MassCore completion.
UMass Donahue Institute
Research and Evaluation Group
9
RTTT C&CR Evaluation Annual Report, September 2012
Introduction
The Massachusetts Department of Elementary and Secondary Education (ESE) was awarded a federal
Race to the Top (RTTT) grant for the years 2010-2014. The college and career readiness (C&CR)
components of the grant were designed to help students develop "knowledge and skills necessary for
success in postsecondary education and economically viable career pathways." The RTTT initiatives are
intended to provide students with opportunities to participate in quality, upper-level high school
coursework and new approaches to assist them with high school completion and transition to higher
education and the workforce. These programs are part of a broader effort, as specified in the Delivery
Plan of ESE's College and Career Readiness group, to increase the Massachusetts 5-year high school
graduation rate to 88.3% and the number of students who complete the MassCore program of study to
85% statewide.
The UMass Donahue Institute is conducting an evaluation of three of the C&CR components of the
Massachusetts RTTT efforts – the Pre-AP Training and Alignment Program, the STEM-focused Early
College High Schools (ECHS), and the MassCore Policy and Implementation initiative – which include
activities at the school, district, and state levels. The evaluation began during the fourth quarter of the
RTTT grant’s first year, which coincided with the first major field activities. Year 2 was therefore the first
full year of substantial evaluation activity. For ease of explanation, “Year 2” in this report refers to the 15month period from 7/1/11 through 9/30/12, which includes the final quarter of Year 1 plus all of Year 2.
This report encompasses all RTTT C&CR evaluation activities that occurred during this 15-month period.
It synthesizes the three previous quarterly evaluation briefings (presented in December 2011, March
2012, and June 2012) and adds findings from evaluation activities that occurred since the most recent
quarterly briefing. This report also presents tables and figures of baseline graduation and dropout rates for
the Pre-AP, STEM ECHS, and MassCore programs. A companion document, Technical Supplement –
RTTT C&CR 2012 Evaluation Annual Report, presents tables and figures that provide a more complete
summary of these same baseline graduation and dropout data, including both average and individual
graduation and dropout rates for RTTT C&CR districts and individual schools.
Future evaluation reports will use these baseline data to describe trends in graduation and dropout rates in
targeted schools and districts. The data necessary to disaggregate baseline graduation and dropout rates by
subgroup (e.g., gender and race/ethnicity) were not available in time for this report, but will be included
in future analyses. Future reports will also describe trends in indicators of student engagement and
academic achievement for students in targeted schools and districts.
Evaluation Questions
This annual report contains separate sections for each of the three RTTT C&CR programs being
evaluated, including versions of the project-wide evaluation questions presented below tailored to each
program’s unique goals and characteristics.1 Each program’s section also contains the description of
methods used for the evaluation of that program.
Process questions.
1
Structuring the report in this way requires some repetition, but ESE and UMDI agreed that the repetition increased clarity.
UMass Donahue Institute
Research and Evaluation Group
10
RTTT C&CR Evaluation Annual Report, September 2012
P1. In what ways have grantees implemented the program components? What are the major
challenges to and facilitators of successful program implementation encountered by grantees?
What midcourse corrections and attempts to overcome challenges have been undertaken?
What additional steps are necessary?
P2. In what ways has ESE implemented the program components described in their grant
application? What are the major challenges to and facilitators of program support and
facilitation encountered by ESE? How have challenges been overcome and midcourse
corrections undertaken? What additional steps are necessary?
P3. How do key project stakeholders rate and explain the quality, relevance, and effectiveness of
major program components and services?
P4. What infrastructure, systems, and processes were put in place to aid program sustainability
during and beyond the grant period? What are the greatest challenges and barriers to creating
sustainability?
Outcome questions.
O1. What progress is being made toward the two top-priority goals of ESE's CCR Delivery Plan –
increasing the 5-year high school graduation rate to 88.3%, and increasing the number of
students who complete the MassCore program of study to 85%?
O2. To what extent are students in grantee programs achieving improved outcomes in college and
career readiness indicators including attendance, credit accumulation and recovery, measures
of academic achievement (e.g., MCAS, PSAT, SAT, and AP), enrollment in AP courses,
graduation, MassCore completion, dual enrollment accumulation of college credits, and
college enrollment and persistence?
O3. At the school and district levels, do observed changes differ across student characteristics
such as gender, race/ethnicity, free/reduced lunch status, ELL status, and special education
status? Is there evidence that gaps are narrowing? What strategies are associated with the
greatest changes for the most disenfranchised students? Are program services reaching
students who are at the greatest risk?
O4. To what extent are observed changes in student outcomes attributable to program activities
(including combinations of program activities) versus contextual variables or non-RTTT
interventions?
O5. What are the major differences in program features and implementation, as well as contextual
variables at the school and district levels, across programs whose levels of improvement
differ substantially?
O6. What is the relationship between level of program implementation and achievement of
targeted student outcomes?
UMass Donahue Institute
Research and Evaluation Group
11
RTTT C&CR Evaluation Annual Report, September 2012
Pre-AP Training and Alignment
Background
The aims of the Pre-AP Training and Alignment program are to increase the number of low-income and
minority students prepared to participate and succeed in mathematics, science, and English Advanced
Placement courses and credit-bearing college-level coursework; to provide teachers in grades 6-12 with
high-quality professional development to assist them in developing curricula, instruction, and assessments
that prepare students for AP coursework; and to provide an opportunity for teachers to collaborate in
horizontal and vertical teams and to network with other teachers in their region for the purpose of
improving curriculum and instruction.
Technical assistance for the Pre-AP project is based on the Laying the Foundation (LTF) curriculum and
is provided by the Massachusetts Math + Science Initiative (MMSI), a project of Mass Insight Education
in partnership with the Commonwealth of Massachusetts. Cohort 1 districts are those who completed their
Pre-AP training before the beginning of the 2011-12 school year. Goal 4D districts are those who paid for
their Pre-AP training and technical support using RTTT Goal 4D funds and who have agreed to send the
same teachers for four-day trainings in three consecutive summers, to create vertical teams of teachers, to
hold quarterly vertical team meetings, and to have “lead teachers” who organize and run the vertical team
meetings and attend an additional training day for lead teachers. Goal 4D districts from Cohort 1 are the
primary focus of this evaluation. Other Cohort 1 districts are using Goal 4A or non-RTTT funds to enroll
teachers (hereafter referred to as “non-4D teachers”) in the Pre-AP program, but those districts are not
required to commit to vertical teaming and sending teams for multiple years of training.
The state’s RTTT scope of work indicates that the performance goal for Pre-AP is 1,000 teachers trained
at the end of the 2011-12, 2012-13, and 2013-14 school years. UMDI has received different explanations
over time of what the number ‘1,000’ refers to; our understanding at present is that it refers to the total
number of summer training participants, whether 4D or other, and whether participating for multiple years
or just one. So, for example, the goal could be met by 200 teachers who each attend for three summers,
plus 400 teachers who only attend for a single summer. The evaluation questions specific to the Pre-AP
evaluation are listed below.
Process Evaluation Questions
P1. In what ways have grantees implemented Pre-AP program components? What are the major
challenges to and facilitators of successful program implementation encountered by grantees?
What midcourse corrections and attempts to overcome challenges have been undertaken?
What additional steps are necessary?
P2. In what ways has ESE implemented the Pre-AP program components described in their
RTTT grant application? What are the major challenges to and facilitators of program support
and facilitation encountered by ESE? How have challenges been overcome and midcourse
corrections undertaken? What additional steps are necessary?
P3. How do Pre-AP program teachers and administrators rate and explain the quality, relevance,
and effectiveness of major program components and services?
UMass Donahue Institute
Research and Evaluation Group
12
RTTT C&CR Evaluation Annual Report, September 2012
P4. What infrastructure, systems, and processes were put in place to aid Pre-AP program
sustainability during and beyond the grant period? What are the greatest challenges and
barriers to creating sustainability?
Outcome Evaluation Questions
O1. To what extent are students served by the Pre-AP program achieving improved outcomes in
measures of academic achievement (e.g., MCAS, PSAT, SAT, and AP), enrollment in AP
courses, graduation, and MassCore completion?
O2. At the school and district levels, do observed changes in students served by the Pre-AP
program differ across student characteristics such as gender, race/ethnicity, free/reduced
lunch status, ELL status, and special education status? Is there evidence that gaps are
narrowing? Are program services reaching students who are at the greatest risk?
O3. To what extent are observed changes in student outcomes attributable to Pre-AP program
activities versus other RTTT program activities and/or measurable contextual variables?
O4. What are the major differences in Pre-AP program implementation and contextual variables
across schools and districts whose levels of improvement on student outcomes differ
substantially?
O5. What is the relationship between level of program implementation and achievement of
targeted student outcomes?
Methods
The Year 2 evaluation of the Pre-AP initiative included the following activities and data sources:

Interviews of teachers and administrators. Three teachers and two administrators from 4D
districts and who had participated in Pre-AP training in summer 2011 were interviewed
during Year 2, either by phone or in-person (see interview protocol in Appendix A1 ). The
interviews were also used to inform development of the teacher and administrator surveys.

Surveys of teachers, lead teachers, and administrators. An online survey was sent in
February, 2012 to all Pre-AP teachers and lead teachers who had completed at least one day
of training during the summer of 2011 (see protocol in Appendix A2). An online survey was
sent in May, 2012 to administrators of all Goal 4D, Cohort 1 districts (see protocol in
Appendix A3).

Interviews of ESE program manager and project director. Three in-person interviews were
conducted during the year, as well as informal email and phone communications throughout
the year (see interview protocol in Appendix B1).

Interviews of technical assistance vendor. Two interviews were conducted with MMSI’s
director of the Pre-AP program, an in-person interview in November 2011 and a phone
interview in August 2012 (see interview protocol in Appendix A4). Additional
communications included informal email and phone communications throughout the year,
and a group meeting in August 2011with representatives of ESE, MMSI, LTF, and UMDI.

Observations of vertical team meetings. Two vertical team meetings were observed in
December 2011 and May 2012, to develop an understanding of this aspect of program
implementation (see observation protocol in Appendix A5).
UMass Donahue Institute
Research and Evaluation Group
13
RTTT C&CR Evaluation Annual Report, September 2012

Observations of Pre-AP training sessions. Four Pre-AP summer training days were observed
to develop an understanding of this aspect of program implementation (see observation
protocol in Appendix A6).

Training registration and attendance database. MMSI developed and populated a database
that tracked teachers, lead teacher, and administrator registration for and attendance of PreAP training. UMDI collaborated with MMSI on developing the database structure, and
utilized the database for analyses.

Vertical team meeting attendance database. In collaboration with MMSI and ESE, UMDI
developed and populated a database that tracked attendance at vertical team meetings. UMDI
created a form for districts to report attendance and sent them monthly reminders to submit
data updates as additional vertical team meetings were conducted.

Supplemental school data requests. MMSI is responsible for obtaining student PSAT scores
from schools participating in the Pre-AP program, to serve as an interim outcome indicator.
MMSI has not yet been able to obtain these scores, but UMDI hopes to incorporate them into
future reporting.

ESE documents and databases. The Pre-AP quantitative data presented below and in the
“Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report” were developed
using ESE’s 4-year and 5-year graduation cohort lists as well as SIMS data from October and
June of 2008-11. Findings were also informed by graduation and annual dropout rate
information reported by ESE.

Related project documents. Project documents including the Massachusetts RTTT proposal,
LTF curriculum materials, and MMSI quarterly reports were reviewed.
Findings
Findings are presented in relation to the process and outcome evaluation questions. Pre-AP questions P1
and P3 are addressed in the sections on district training configurations, training activities, implementation,
and factors facilitating and obstructing implementation. Questions P2 and P4 are primarily addressed in
the sections on training activities, support, and sustainability. The outcome questions are addressed with
the presentation of baseline graduation and dropout rates for Pre-AP Cohort 1 4D districts and a
discussion of how outcome data for subsequent project years will be utilized.
Survey Response Rates
The Pre-AP administrator survey was sent to administrators of all Goal 4D, Cohort 1 districts (N=25) in
May 2012, and had a 68% response rate (N=17). Respondents’ administrative roles included curriculum
coordinator (59%, N=12), program director (18%, N=3), assistant superintendent (6%, N=1), and
assistant principal (6%, N=1). Most respondents were district-level administrators (88%, N=15), and the
others were affiliated with specific middle or high schools. Some questions on this survey were parallel to
questions on the Pre-AP teacher and lead teacher survey, and comparisons between administrator and
teacher responses are noted throughout the findings section.
The teacher survey was sent to 347 4D teachers and had a 29% response rate (N=101); it was also sent to
198 non-4D teachers and had a 19% response rate (N=38). The survey included a section for lead
teachers, and 21 of the 4D respondents and 10 of the non-4D respondents indicated that they were lead
UMass Donahue Institute
Research and Evaluation Group
14
RTTT C&CR Evaluation Annual Report, September 2012
teachers. While a sample of 139 teachers is substantial, these low response rates raise the possibility that
the sample of respondents was not representative of the larger population of Pre-AP trained teachers.
Teacher survey respondents were asked their number of years as a certified teacher, and the average was
12 years (range 1 to 40). The academic discipline of respondents’ primary teaching assignment was
mathematics (35%), English language arts (35%), biology (8%), general science (6%), chemistry (5%),
physics (3%), social studies (2%), and one percent each computer science, life science, ESL,
environmental science, elementary, multiple disciplines, and administration. (This totals slightly more
than 100% because some teachers responded in multiple categories.) The majority of teachers reported
teaching at multiple grade levels, as shown in the table below, with the largest group of teachers being in
the high school grades.
Teacher Survey: The Academic Level of Your Primary
Teaching Assignment
N
(%)
5th Grade
5
4
6th Grade
15
11
7th Grade
25
18
8th Grade
27
19
9th Grade
39
28
10th Grade
42
30
11th Grade
39
28
12th Grade
38
27
Special Education
8
6
Classroom aide
1
1
ESL Specialist
2
1
Other
5
4
Survey Item
The responses of 4D and non-4D teachers are reported separately. The text will focus on 4D responses but
will indicate when non-4D responses differ by more than 15 percentage points from the reported 4D
responses.
UMass Donahue Institute
Research and Evaluation Group
15
RTTT C&CR Evaluation Annual Report, September 2012
District Participation Configurations
Districts are participating in Pre-AP training in multiple configurations. The configuration for RTTT Goal
4D is to send the same teachers for four-day trainings in three consecutive summers; to create vertical
teams across middle and high school grades of mathematics, science, or English language arts teachers; to
hold quarterly, two-hour vertical team meetings to deepen their pre-AP practices; and to have “lead
teachers” who organize and run the vertical team meetings and attend a training day for lead teachers.
Eighty-two districts initially designated their intention to be Goal 4D districts, but as of the FY12-FY14
proposals, 47 districts remain in that category. Currently known reasons for attrition include changes in
district leadership, difficulty accommodating the program’s structural requirements, and realizing that the
district couldn’t afford the training costs. At least one district substantially reduced the number of teachers
they intended to train.
Several variations of the basic configuration have occurred which will have implications for which
teachers and students are included in analyses and as which cohorts. For example, one large district
completed half of its training during summer 2011 and initiated vertical team meetings in December 2011
but did not finish Year 1 training until January 2012. Although they are likely to have begun substantial
change at the same time as Cohort 1 districts, ESE requested that they be included in Cohort 2 because of
when they completed their training.
The percentage of all teachers in a given school and district who are being trained varies widely. Districts
also selected different disciplines or combinations of disciplines among the mathematics, science, and
ELA trainings offered. In districts that trained only a portion of the teachers in a given discipline, those
who received training may have attributes that are not representative of their discipline as a whole. For
example, one administrator said, “We started with people who we knew who would dig into this. Picking
the first group carefully is important. It is incredibly powerful to have teachers implement it in their
classroom, and share their experiences with other teachers.” In districts where participating in Pre-AP
training was voluntary, those who chose to participate had to be willing to invest four days of their
summer. In some districts, collective bargaining agreements dictated that teachers could volunteer to
participate in the summer training but couldn’t be required to do so.
Training Activities
During summer 2011, MMSI provided four-day summer trainings for 561 teachers from 37 districts
(including 19 4D districts), and teachers attended an average of 3.6 days (out of 4). Also during summer
2011, 30 teachers participated in lead teacher training, and 25 administrators (from seven 4D districts)
participated in administrator training. For summer 2012, more than 900 teachers registered for training;
final registration data were not available for this report. The ESE program manager reported that, after
summer 2012, Massachusetts will have met the performance goal of having trained 1,000 teachers.
Teacher training. Results from the teacher survey indicated that prior to the summer 2011 PreAP training, just 4% of 4D teachers had already completed one or more days of the MMSI Pre-AP
program, and just 5% had participated in MMSI’s AP (Core) program. In contrast, a much higher
percentage of non-4D teachers had participated in the MMSI Pre-AP (21%) and AP (24%) training.
As shown in the table below, 73% of teachers reported participating in all four days of the MMSI Pre-AP
training. The 14% who had participated for more than four days of training were presumably among those
UMass Donahue Institute
Research and Evaluation Group
16
RTTT C&CR Evaluation Annual Report, September 2012
who had reported participating in previous years. Among the non-4D group, 33% had completed more
than four days, and 23% had completed exactly eight days, suggesting that this was their second full year
of Pre-AP training.
Teacher Survey: How many days of MMSI Pre-AP
training have you attended?
4D
# of Days
Non-4D
%
N
%
N
0
1
1
4
1
1
4
4
0
0
2
2
2
0
0
3
6
5
13
4
4
73
68
50
15
5
10
9
10
3
6
2
2
0
0
7
1
1
0
0
8
1
1
23
7
Almost all 4D teachers (91%) reported that they intended to complete all three years (12 total days) of
Pre-AP training, which is the commitment districts make when using 4D funds. A somewhat smaller
group of non-4D teachers (82%) made that commitment. The table below summarizes these findings.
UMDI’s observations of two summer training classrooms were striking, because the two classrooms were
so different. One science classroom was vibrant, with lively and engaged teachers working on an activity
related to distance, time, and motion sensors. A very affirming trainer repeatedly used phrases such as
“Love it!” and other teachers followed suit with “That’s cool!” and so on. There was much back and forth
about typical problems, such as what to do in the absence of needed technology, or the best ways to
describe certain scientific terms to enhance student understanding. The teacher prompted helpful
reflection on questions such as “How have the instructional strategies modeled by the trainer enhanced
the learning experience?” and offered a variety of ways that teachers could access support from her, LTF,
and each other after the training had ended.
The English classroom observed, in contrast, was much less lively, engaged, and reflective. It was also
much more facilitator-centered, with far less interplay among teachers. Teachers created poems and other
pieces of writing, and after a given poem was read out loud the facilitator in effect gave her own brief
assessment of the piece’s qualities without seeking input from others in the room. There was less talk
about what happens in actual classrooms and the unique challenges of the participants, and there was little
reflection on how the training activities connected back to the LTF model.
UMass Donahue Institute
Research and Evaluation Group
17
RTTT C&CR Evaluation Annual Report, September 2012
In short, while both facilitators were leading teachers through LTF activities, one facilitator seemed to be
generating more enthusiasm, creating more of the type of classroom that LTF promotes, and providing
high-level reflection and opportunities for mutual support. These findings suggest the importance of how
facilitators are selected, trained, evaluated, and given feedback, as well as whether they have the
opportunity to observe effective facilitators, either live or recorded.
Changes to MMSI’s teacher training. Teacher survey respondents were asked to suggest
changes to the four-day MMSI training they attended, and 38 responded. Thirteen wanted fewer lectures
and more group work time to discuss and develop lesson plans with other teachers. Six said that their
trainer did not exhibit mastery of the subject area, with one saying, “Having an instructor read a manual
aloud is not helpful, and completing the exercises as if we are students is a waste of time. The more
teachers talk with one another about lessons, units, philosophy, methods, the more valuable the time
becomes.”
Five respondents said that they would prefer training to be spread out over the school year, rather than a
four-day intensive training, because being presented with so much information at once was
overwhelming. These respondents also said that they would like to have time to digest the information
and materials at one training session, to practice them, and then to return to follow-up trainings in order to
share ideas. Four respondents stated that the training could be conducted in fewer than four days,
considering the slow pace and the amount of repetition, and four requested more training in using Pre-AP
assessments.
A total of five respondents commented on the activities and exercises conducted during the training.
Three said that actually conducting the activities during training was not an efficient use of time, and one
said “The experiments presented, except one, were very common. A welcome change would be new
experiments or new twists in the analysis of traditional experiments.”
Three respondents said that the materials presented were too difficult for their low-skilled or middle
school students. One said, “Some material was much too advanced and lost my attention, as it was
irrelevant to my teaching.”
One teacher suggested that, “instead of sending home giant books with all of the middle and high school
lessons combined, perhaps create grade-level guides/manuals for the teachers to use with the appropriate
lessons included.” Another suggested greater differentiation of trainings for more and less experienced
teachers.
With regard to the Pre-AP math training, the lead teacher would have liked more opportunities to interact
with teachers from other districts regarding their classroom procedures and curricular activities. He
recommended more “pair-share” types of activities, beyond the problems that LTF/MMSI provided for
the training session. For the lead teacher training, he felt that too much time was spent promoting
LTF/MMSI, and not enough on developing vertical team activities.
Lead teacher training. When asked about lead teacher training, as shown in the table below,
73% of lead teacher survey respondents agreed that the summer lead teacher training had been helpful in
supporting their work as a lead teacher, 64% agreed that the summer lead teacher training was engaging,
and 82% agreed that the lead teacher training was well-executed.
UMass Donahue Institute
Research and Evaluation Group
18
RTTT C&CR Evaluation Annual Report, September 2012
Teacher Survey: Summer Lead Teacher Training
Survey Item
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
The summer Lead Teacher training
session was helpful in supporting what I
need to do as a Lead Teacher.
11
27
46
9
18
0
The summer Lead Teacher training
session was engaging.
11
27
37
27
9
0
Overall, the Lead Teacher Training
session in September was well executed.
11
36
46
9
9
0
Note. Response options also included “Not Applicable”. The percentage of respondents who selected
those choices are listed below, but those respondents were not used to calculate the percentages
reported in the table. Q1. 4D: 42%, Non-4D: 0%, Q2. 4D: 42%, Non-4D: 0% , Q3. 4D: 39%, Non-4D:
0%.
Changes or improvements to lead teacher training sessions. Respondents were asked to
comment on what changes or improvements, if any, they would make to the lead teacher training
sessions. The five lead teachers who responded suggested focusing on collaboration to plan vertical team
meetings, having one additional day of lead teacher training in September, having all teachers bring
laptops to meetings, providing new information during lead teacher training compared to presenting the
same information from the first training, and clarifying goals of the meetings.
One lead teacher who was interviewed felt that the lead teacher training was less a practical session than a
pep talk about Pre-AP philosophy. She said, “At the lead teacher training it was kind of idealistic, like
very philosophical. Like you are going to get them all on board with LTF and ‘this works for kids’ and
the numbers are proven statistically, but then the rest was kind of dropped off into our hands as far as
what to do [at the vertical team meetings].” This topic is discussed further in the section on vertical teams.
Administrator training. Administrators appreciated the opportunity to observe teachers engaged
in LTF activities during the administrator training. One administrator noted that observations of LTF
activities “provide a great model of what administrators should be seeing when they walk through
schools.” He added, “I would suggest that more time be spent on observing and less time on nuts and
bolts. I had done some reading and knew about the approach, so I did not need that information. … I wish
we had spent more time in classes. They mentioned an evaluation instrument, but we never actually got to
see or use that tool during the training. I wish we had done that. To have practice with that type of tool
would have been beneficial.” Some administrators are constrained in using tools such as the UTOP
protocol to assess teacher practices related to LTF implementation, because instruments that will be used
for performance evaluation must be collectively bargained. However, such protocols may also be used to
generate discussion rather than to influence evaluation. Only five administrator survey respondents (29%)
reported having attended MMSI’s 2011 Pre-AP summer administrator training, but most responded
positively to the experience, as shown in the table below.
UMass Donahue Institute
Research and Evaluation Group
19
RTTT C&CR Evaluation Annual Report, September 2012
Administrator Survey: Pre-AP Training
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
As a result of participating in the
training, my awareness of the
importance of using Pre-AP strategies
has increased.
5
20
60
20
0
0
What I learned in the training has
helped me promote the Pre-AP program
with district-level personnel.
5
20
60
20
0
0
What I learned in the training has been
helpful with my district’s vertical
curriculum alignment efforts.
5
20
60
20
0
0
The training session was engaging.
5
20
80
0
0
0
Survey Item
The training session was well executed.
5
20
80
0
0
Note. Response options also included “Don’t Know,” but no respondents chose this option.
0
Administrator training recommendations. Administrators were asked to make suggestions to
improve the MMSI summer trainings. One commented that the long drive to the training location deterred
some teachers from attending, and also that teachers were more available to attend trainings at the end of
the school year, rather than later in the summer. One recommended that MMSI provide more vertical
team materials and resources. Finally, one administrator recommended that MMSI provide more help for
developing an implementation action plan and more models for integrating strategies.
Implementation
Tracking. The LTF program philosophy, strongly shared by ESE, is that every student should be
considered a Pre-AP student. Personnel from both ESE and MMSI have expressed concerns that some
middle schools are predicting that certain students are not bound for college and therefore tracking them
away from classes that contain Pre-AP content and pedagogy.
Most teachers and administrators who were interviewed agreed that every student should be considered a
Pre-AP student, and more than one expressed mild to strong opposition to tracking. Some teachers also
expressed support for anti-tracking practices such as including special education teachers in summer
training and vertical team meetings, as well as training all teachers in a given discipline so no student is
taught by a teacher who lacks Pre-AP training.
Responses to the teacher survey provided a different perspective on the issue of tracking, and on how PreAP materials are used more generally. The survey asked teachers if they used more Pre-AP materials and
strategies with some classes than with others, and 52% said yes. Of the 54 teachers who offered
explanations for their responses, 78% said that they used more Pre-AP materials and strategies with their
honors or advanced-level students. They generally agreed that Pre-AP materials were not suitable for all
students, such as those in remedial classes, those who had not mastered skills from a previous course, and
UMass Donahue Institute
Research and Evaluation Group
20
RTTT C&CR Evaluation Annual Report, September 2012
some English language learners. Several (N=8) also stated that planning, modifying, scaffolding, and
implementing Pre-AP lessons required more time than they had available, in part due to keeping up with
district pacing guides.
Teachers were asked to respond to the item, “Compared to my honors-level classes, implementing one
LTF activity with my non-honors classes requires [more, less, or the same amount of] time.” Twenty-two
percent reported that their school only has one level, and 17% that they only teach one level. Of the
remaining teachers, 79% reported that implementing an LTF lesson takes longer with their non-honors
classes, and just 1% (i.e., one teacher) reported that implementing an LTF lessons takes longer with their
honors-level classes.
Teacher Survey: Honor vs. Non-Honors
Survey Item
Compared to my honors-level classes,
implementing one LTF activity with my nonhonors classes requires [more, less, or the
same amount of] time.
4D
N
More
(%)
Less
(%)
The same
amount of
(%)
Y
61
79
1
20
N
28
86
0
14
Note. Response options also included “My school has only one level” and “My school has both
levels, but I only teach one level”. The percentage of respondents who selected those choices are
listed below, but those respondents were not used to calculate the percentages reported in the table
above. 4D: School only has 1 level – 22%; Only teach 1 level – 17%. Non-4D: School only has 1
level – 9%: Only teach 1 level – 9%.
Of the 48% of teacher survey respondents who said “no” and offered comments, 37% (N=14) stated that
they applied Pre-AP lessons and strategies evenly in all of their classes. Some respondents reported
simply modifying lessons and strategies to match class levels, while seven respondents said that, since all
of their classes were the same level and followed the same pacing guide, they were also provided with the
same Pre-AP materials and strategies. Five respondents said that they distributed the materials evenly
because their students all deserved to have access to, and benefit from, the same materials as other
students; four of these five respondents were science teachers.
In response to these findings, a MMSI representative emphasized that the Pre-AP activities can be broken
down into smaller pieces to make them more accessible for students with lower skill levels. She
acknowledged that this reduces the amount of content coverage, but she questioned the value of “getting
through” everything if the students can’t access the materials. However, to “cover” less content in
exchange for greater depth, as promoted by LTF, teachers need the approval of their administrators. Many
teachers are pressured to cover more content, and doing so is tied to their performance evaluation, so
without explicit approval they could be reluctant to make the changes that LTF implementation requires.
Expectations for implementation. Administrators were asked how many LTF lessons and
assessments each Pre-AP trained teacher in the school/district was expected to implement during the
2011-12 school year, and responses varied substantially. They included: at least one per quarter, at least
UMass Donahue Institute
Research and Evaluation Group
21
RTTT C&CR Evaluation Annual Report, September 2012
one per term, at least two, two lessons per quarter pairing them with assessments, two lessons and two
assessments, and four from their summer training. Two respondents stated that their teachers did not have
a specific number that they were expected to implement. One of these administrators stated:
There was no minimum expectation, but it is apparent through vertical team meetings and
conversations with staff that our trained Pre-AP teachers were trying out many of the lessons. We
still have one final Pre-AP vertical meeting where we will set the expectation for next year based
on our learnings from this year.
The other respondent said,
We are not using this as a standard for our teachers. Rather, we are looking for teachers to use
elements found in the LTF lessons and integrate them into learning experiences within the
classroom. We feel that it is better to help teachers understand what makes a quality lesson, rather
than give them a set of tasks to complete with their students.
In response to this quotation, the MMSI Pre-AP program director emphasized that 4D districts should be
implementing specific LTF activities and assessments, not just integrating LTF “elements” into their
classrooms, and that their letter of agreement specified this. She added, “I really can’t see how [district
representatives] can say that they aren’t putting in any activities. I haven’t seen a [letter of agreement] that
didn’t have some activities listed. To me, this would be unacceptable. I would strongly urge the
administrator to have the activities implemented, and if not I think they are wasting their money.” She
indicated that her “dream” would be to have teachers implement five LTF activities and/or assessments
each quarter.
All teacher survey respondents reported having implemented substantially more LTF lessons and
assessments than required by their districts (6.4 implemented vs. 2.4 required for 4D teachers; 10.0
implemented vs. 1.1 required for non-4D teachers). This difference – 10.0 vs. 1.1 – was particularly
notable for non-4D teachers. Last, teachers reported that, on average, implementing one LTF lesson
required 2.4 hours (for 4D) and 1.9 hours (for non-4D).
At present, it appears that the pace of change will be gradual. A strategic consideration is whether the
desired level of student impacts can be achieved at this pace, and whether that pace can or should be
increased. It appears that teachers are expected to implement only 2-3 LTF activities or assessments per
quarter, and that this is what MMSI training promotes as an expectable level of change for the first year,
believing that it will gradually gain momentum. Impact may also come through routes other than
implementation of specific LTF activities, such as if teachers’ deeper understanding of Pre-AP
pedagogical approaches also infuses aspects of their usual classroom activities.
Resources. Some LTF math and science activities require expensive equipment that some
schools lack, particularly at the middle school level. Some teachers require substantial professional
development and/or preparation time to be ready to use this equipment with students, and some schools
need guidance about which equipment to purchase. Several districts have asked MMSI to advise them
about what equipment to buy, and MMSI has been able to provide advice as well as referring them to
other consultants. Schools who wish to adopt these activities need to budget for the needed equipment and
professional development.
UMass Donahue Institute
Research and Evaluation Group
22
RTTT C&CR Evaluation Annual Report, September 2012
Several teacher survey items, shown in the table below, asked teachers about the adequacy of their
resources and supports, as well as the preparation of their students. With regard to their ability to
implement Pre-AP lessons and/or assessments, 66% agreed that they had adequate curricular resources,
53% that they had adequate classroom equipment and supplies, and 58% that they had long enough class
periods. Just 22% reported that they had adequate planning time and 21% that they had sufficient time to
share Pre-AP approaches with same-discipline colleagues. Seventy-one percent agreed that they had full
administrative support to integrate Pre-AP lessons into their teaching, and 26% agreed that performing
Pre-AP activities prevented them from adhering to district pacing guides. Teachers clearly felt strongly
that their students were not adequately prepared to participate in Pre-AP lessons and/or assessments
targeted to their grade level, as just 30% agreed with this statement.
UMass Donahue Institute
Research and Evaluation Group
23
RTTT C&CR Evaluation Annual Report, September 2012
Teacher Survey: Pre-AP Lessons and Assessments
Survey Item
I have access to adequate curricular
resources to implement Pre-AP
lessons and/or assessments.
I have adequate classroom
equipment and supplies to
implement Pre-AP lessons and/or
assessments.
I have adequate planning time to
prepare new Pre-AP lessons and/or
assessments.
My school provides adequate
planning time for same-discipline
teachers to share their Pre-AP
approaches with each other.
Class periods are long enough to
provide adequate time for my
students to perform Pre-AP
activities.
Performing Pre-AP activities with
my students prevents me from
adhering to my district’s pacing
guides.
My administrators fully support my
integration of Pre-AP lessons and
assessments into my teaching.
My students have sufficient
academic preparation to participate
in the Pre-AP lessons and/or
assessments that are targeted to their
grade level.
Strongly
Agree
Agree Neutral Disagree
(%)
(%)
(%)
(%)
Strongly
Disagree
(%)
4D
N
Y
99
16
50
18
12
4
N
38
13
53
18
16
0
Y
99
8
45
21
18
8
N
38
8
37
21
29
5
Y
97
2
20
31
36
11
N
37
3
22
19
48
8
Y
98
2
19
24
37
18
N
38
3
24
8
34
31
Y
96
5
53
20
19
3
N
37
11
46
5
30
8
Y
99
8
18
26
44
4
N
38
0
26
26
37
11
Y
97
21
50
20
7
2
N
38
31
50
16
0
3
Y
99
5
25
24
36
10
N
38
5
24
21
42
8
As shown in the next table, 81% of administrators (compared to 66% of respondents to the teacher
survey) agreed that teachers in their schools/districts have adequate curricular resources to implement
Pre-AP lessons and/or assessments, and 75% of administrators (compared to 53% of teachers) said that
UMass Donahue Institute
Research and Evaluation Group
24
RTTT C&CR Evaluation Annual Report, September 2012
they had adequate classroom equipment to do the same. When asked if teachers in schools/districts have
adequate time to prepare new Pre-AP lessons and assessments, 44% of administrators agreed, versus just
22% of teachers. Nineteen percent of administrators said that conducting Pre-AP activities prevents
teachers from adhering to their district’s pacing guidelines, as compared to 26% of teachers.
Administrator Survey: Pre-AP Resources and Supports
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
Teachers in my school/district have
adequate curricular resources to
implement Pre-AP lessons and/or
assessments.
16
31
50
6
13
0
Teachers in my school/district have
adequate classroom equipment to
implement Pre-AP lessons and/or
assessments.
16
19
56
13
6
6
Teachers in my school/district have
adequate planning time to prepare new
Pre-AP lessons and/or assessments.
16
6
38
31
25
0
Conducting Pre-AP activities prevents
teachers from adhering to my district’s
pacing guidelines.
16
6
13
13
37
31
I fully support the integration of Pre-AP
lessons and assessments into my
school/district’s curriculum.
16
75
19
6
0
0
Students in my school/district have
sufficient academic preparation to
participate in the Pre-AP lessons and/or
assessments that are targeted to their
grade level.
16
6
44
44
6
0
Survey Item
Scheduling vertical team meetings is a
16
13
31
25
25
challenge in my district.
Note. Response options also included “Don’t Know,” but no respondents selected that option.
6
Alignment of assessments and standards. Compared to the survey items above, responses to
questions about the LTF curriculum and its relation to the Common Core showed a greater concordance
between administrators and teachers, as shown in the following two tables. Eighty-two percent of teachers
(compared to 80% of administrators) said that the LTF curriculum is well-aligned with the Common Core
standards, 75% of teachers (compared to 72% of administrators) said that the connections between the
LTF curriculum and the Common Core are well specified, and 87% of teachers (versus 81% of
administrators) said that the LTF lessons and assessments are examples of high-quality pedagogical
practices.
UMass Donahue Institute
Research and Evaluation Group
25
RTTT C&CR Evaluation Annual Report, September 2012
Teacher Survey: LTF Lessons and Assessments
Survey Item
4D
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
The LTF lessons and assessments are
well-aligned with the Common Core
standards for my content area.
Y
89
22
60
14
3
1
N
31
29
58
7
6
0
Y
88
18
57
17
6
2
N
31
26
48
13
13
0
Y
97
29
58
11
2
0
N
35
29
57
11
3
0
Y
95
46
21
14
6
13
N
36
47
22
17
3
11
Y
97
27
55
15
3
0
N
37
16
57
19
5
3
The connections between the LTF
lessons and assessments and the
Common Core standards are well
specified for my content area.
The LTF lessons and assessments are
examples of high-quality pedagogical
practices in my content area.
LTF lessons and assessments provide
appropriate supports for
differentiation for students across a
wide range of skill levels.
In order to implement a typical LTF
lesson, I need to create supplemental
activities to introduce or otherwise
scaffold aspects of the LTF lesson.
Note. Response options also included “not applicable” and “don’t know”. The percentage of respondents
who selected those choices are listed below, but those respondents were not used to calculate the
percentages reported in the table. Q1. 4D: 1% NA, 9% DK, Non-4D: 3% NA, 16% DK. Q2. 4D: 1%
NA,10% DK, Non-4D: 3% NA, 16% DK. Q3. 4D: 0% NA, 1% DK, Non-4D: 3% NA, 5% DK. Q4. 4D:
0% NA, 2% DK, Non-4D: 0% NA, 5% DK. Q5. 4D: 1% NA, 0% DK, Non-4D: 0% NA, 3% DK.
UMass Donahue Institute
Research and Evaluation Group
26
RTTT C&CR Evaluation Annual Report, September 2012
Administrator Survey: LTF Curriculum and the Common Core
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
The LTF curriculum is well-aligned
with the Common Core standards.
15
40
40
13
7
0
The connections between the LTF
curriculum and the Common Core
standards are well specified.
15
29
43
21
7
0
Survey Item
The LTF lessons and assessments are
examples of high-quality pedagogical
16
62
19
19
0
0
practices.
Note. Response options also included “Don’t Know.” The percentage of respondents who selected this
choice was: Q1. 6%, Q2. 13%, Q3. 0%. These respondents were not included in the number of
respondents and percentages reported in the table.
Almost all administrators (94%) agreed that they fully support the integration of Pre-AP lessons and
assessments into their school/district’s curriculum, compared to 71% of teachers who said that their
administrators fully support them in integrating Pre-AP into their teaching. Both administrators and
teachers clearly felt that many of their students were not adequately prepared to participate in Pre-AP
lessons and/or assessments targeted to their grade level, as just 50% of administrators and 30% of
teachers agreed with this statement.
Changes in teaching practices. Teachers were asked about the impacts of their participation in
Pre-AP training, and responses are shown in the table below. In the 4D group, 70% or more of teachers
agreed that, as a result of the Pre-AP training, they now teach more Pre-AP content (70%), use more PreAP pedagogical strategies in the classroom (74%; 90% for non-4D), and have greater awareness of the
importance of using Pre-AP strategies (80%). A smaller but still substantial number agreed that they now
use more Pre-AP assessment strategies (47%), that they have changed their teaching philosophy to be
more consistent with the Pre-AP program (57%), and that implementing LTF lessons and assessments
represents a substantial change to their teaching practice (47%).
UMass Donahue Institute
Research and Evaluation Group
27
RTTT C&CR Evaluation Annual Report, September 2012
Teacher Survey: Impacts of MMSI Pre-AP Training
Survey Item
4D
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
As a result of participating in the
MMSI Pre-AP training, I now teach
more Pre-AP content.
Y
100
22
48
14
14
2
N
38
32
39
24
5
0
Y
99
18
56
15
9
2
N
38
32
58
8
2
0
Y
95
10
37
34
15
4
N
37
13
46
30
11
0
Y
98
12
45
31
11
1
N
38
18
24
48
5
5
Y
100
27
53
17
2
1
N
36
28
58
8
0
6
Y
99
11
36
33
16
4
N
38
13
40
29
13
5
As a result of participating in the
MMSI Pre-AP training, I now use
more Pre-AP pedagogical strategies
in the classroom.
As a result of participating in the
MMSI Pre-AP training, I now use
more Pre-AP assessment strategies.
As a result of participating in the
MMSI Pre-AP training, my teaching
philosophy has changed to be more
consistent with that of the MMSI
Pre-AP program.
As a result of participating in the
MMSI Pre-AP training, my
awareness of the importance of
using Pre-AP strategies has
increased.
Implementing LTF lessons and/or
assessments represents a substantial
change to my teaching practice.
As shown in the table below, teacher survey respondents reported that collaboration with colleagues to
develop Pre-AP lessons and/or assessments was most common with same-department colleagues, whether
the colleagues were Pre-AP trained (46% agree or strongly agree) or not (44%). Such collaboration was
lower with colleagues in other schools (31%).
UMass Donahue Institute
Research and Evaluation Group
28
RTTT C&CR Evaluation Annual Report, September 2012
Teacher Survey: Collaboration With Colleagues
Survey Item
I actively collaborate with Pre-AP
trained colleagues in my department
to develop Pre-AP lessons and/or
assessments.
I actively collaborate with colleagues
in my department who aren’t pre-AP
trained to develop Pre-AP lessons
and/or assessments
I actively collaborate with colleagues
in other schools in my district to
develop Pre-AP lessons and/or
assessments.
4D
N
Strongly
Agree
(%)
Y
98
14
32
18
26
10
N
38
16
26
18
16
24
Y
98
9
35
18
31
7
N
38
18
24
21
21
16
Y
98
5
26
23
32
14
N
38
0
13
16
42
29
Agree Neutral Disagree
(%)
(%)
(%)
Strongly
Disagree
(%)
The table below reports responses to several teacher survey items regarding Pre-AP implementation
during the 2011-12 school year, and some notable findings are highlighted next. First, non-4D
respondents, who are not required by the Pre-AP project to participate in vertical team meetings, reported
having participated in an average of 1.5 such meetings. More information about vertical team meetings is
provided in the next section of this report. Second, non-4D teachers reported accessing LTF online
materials, as well as meeting with same-discipline colleagues to jointly develop Pre-AP lessons and
assessments, more often than 4D teachers.
UMass Donahue Institute
Research and Evaluation Group
29
RTTT C&CR Evaluation Annual Report, September 2012
Teacher Survey: Pre-AP Implementation During the 2011-12 School Year
Survey Item
4D
Mean
Y
2.3
N
1.5
Y
3.2
N
4.1
Y
5.5
N
7.2
Y
0.8
N
0.6
Y
0.5
N
0.9
Y
2.4
N
1.1
Y
2.4
N
1.9
Y
4.8
N
7.5
Y
1.6
N
2.5
Number of times participated in vertical team meetings.
Number of times met with other teachers in your discipline
to jointly develop Pre-AP lessons and assessments
Number of times accessed online materials on LTF website
Number of times participated/reviewed discussions in
LTF/MMSI online forum
Number of times observed another teacher or team of
teachers presenting an LTF lesson in their classroom
How many LTF lessons and assessments has your school or
district told you that you must implement during this school
year?
On average, how many hours of class time have you needed
to spend in order to implement one LTF lesson?
Please think about the class (or classes) in which you have
implemented the most LTF lessons during the 2011–12
school year. How many LTF lessons have you implemented
in that class during the school year?
For that same class, how many LTF assessments have you
implemented this school year?
Vertical team meetings. One commitment of districts participating in RTTT Goal 4D is to
schedule four vertical team meetings per year to be attended by Pre-AP trained teachers and
administrators across the middle and high school grades in a common Pre-AP curricular area (i.e.,
English, math, and science). Other relevant school personnel are also encouraged to participate in these
UMass Donahue Institute
Research and Evaluation Group
30
RTTT C&CR Evaluation Annual Report, September 2012
meetings. The meetings are intended to be led by trained Pre-AP lead teachers with the purpose of
advancing implementation of the MMSI/LTF lessons and assessments. LTF provides materials to
facilitate these meetings, and MMSI provides technical assistance to the extent permitted by its staffing
capacity, and in relation to perceived district need, as well as in response to invitations from districts.
As summarized in the table below, district teams reported attendance for an average of 3.5 vertical team
meetings, with a range of 2 to 4 meetings. The meetings were attended by an average of 63% of summer
2011 trainees (range 9% to 100%) as well as an average of four personnel that hadn’t completed Pre-AP
training. Attendance data were also examined to identify attendance trends within districts. Two districts
showed a notable downward trend in attendance across meetings (Attleboro ELA – 13, 11, 4, 4; Waltham
Science – 7, 9, 4, 5), but other districts showed fairly even attendance, or had such low numbers of
trainees that substantial percentage changes might reflect changes of only one or two participants.
UMass Donahue Institute
Research and Evaluation Group
31
RTTT C&CR Evaluation Annual Report, September 2012
Vertical Team Meeting Attendance by District and Discipline
District
Discipline
VT
Meetings
Reported
(N)
Summer
2011
Trainees
(N)
Average
Attendance of
Trainees Per
Meeting
(%)
Average NonTrained Staff
Who Attended
Each Meeting
(N)
Attleboro
ELA
4
13
62
7
Auburn
ELA
4
6
100
1
Boston
Math
4
44
23
0
Boston
Science
3
22
30
1
Chelsea
ELA, Math, Science
2
7
71
13
Chicopee
Math
4
62
81
3
Everett
ELA
4
11
89
1
Everett
Math
4
12
73
1
Fall River
ELA
4
25
44
4
Malden
ELA
3
26
10
1
Malden
Science
4
20
9
7
North Adams
ELA, Math, Science
2
1
100
10
Saugus
Math
4
7
93
3
Somerset
Math
4
7
89
4
Swampscott
ELA
3
4
58
4
Swampscott
Math
4
4
63
4
Uxbridge
Science
4
2
100
3
Waltham
ELA
3
14
64
1
Waltham
Math
4
13
52
3
Waltham
Science
4
14
45
1
W. Springfield
ELA
3
9
70
8
W. Springfield
Math
2
4
63
13
3.5
14.9
63.1
4.2
Average
UMass Donahue Institute
Research and Evaluation Group
32
RTTT C&CR Evaluation Annual Report, September 2012
Scheduling four two-hour vertical team meetings per year presents districts with multiple challenges,
including the following, depending on the district:

Attendance is optional unless the meetings are held during contractual time. This can include
scheduled professional development days. The stipend was an important motivator for
teacher attendance, but still not sufficient for some teachers who preferred not to attend. If
schools want to prioritize attendance from all teachers, holding the meetings during
contractual time would be important. To accommodate this, at least one district has decided
to hold two vertical team meetings on each of two scheduled professional development days,
thereby meeting the requirement of four meetings. A strategic consideration is whether such
an arrangement achieves or shortchanges the purpose of holding four meetings per year.

Middle school and high school teachers must meet together. For after-school meetings, this
can present the challenge of the schools having different dismissal times

District Letters of Agreement, which specify the year’s four vertical team meeting dates and
times, were in some cases signed too late for the 2011-12 school year to be integrated into the
district schedule, and the first vertical team meetings for most districts were scheduled for
November or December. First-year startup issues were part of the problem. An implication
for future years is to secure LOAs early and set the first meeting earlier in the school year.
With regard to scheduling of Pre-AP vertical team meetings, four lead teacher survey respondents said
that their district does not hold vertical team meetings, perhaps because the survey was conducted before
that district’s first vertical team meeting. The remaining lead teachers said that vertical team meetings
were held after school (56%), during professional development time (31%), and during teacher planning
and/or grade-level or content meetings during the school day (38%). This sums to more than 100%
because some respondents indicated multiple times.
Administrators reported participating in an average of 2.9 Pre-AP vertical team meetings (range 1 to 8).
They also reported observing a teacher or teachers presenting an LTF lesson in their classroom an average
of 2.8 times (range 0 to 15), with four administrators reporting zero observations. Forty-seven percent of
administrators agreed that scheduling vertical team meetings was a challenge in their district. (A parallel
question was not asked of teachers.)
At one vertical team meeting that was observed, some of the teachers were already Pre-AP trained from
past years, but the agendas provided by LTF reportedly assume that teachers in attendance are new to the
model. It may be useful to provide differentiation of vertical team meetings under such conditions,
particularly as in subsequent years there may increasingly be a mix of new and already-trained teachers.
In one school there was an important divergence of understandings about the binder of materials that LTF
provides at lead teacher trainings which suggests activities for the four vertical team meetings.

The administrator was enthusiastic about the binder, felt that the lead teacher’s role was
straightforward and relatively undemanding, and believed that the lead teacher was mostly
following the provided materials.

The teacher, in contrast, didn’t feel that teachers would find the provided materials engaging,
saying “I didn’t want to bore everyone to death after a long day of school.” One activity for
the meeting was about writing multiple choice questions, and the lead teacher did substantial
advance preparation, finding additional articles on the topic and having teachers review and
discuss the articles before rating the quality of different items. She felt under considerable
UMass Donahue Institute
Research and Evaluation Group
33
RTTT C&CR Evaluation Annual Report, September 2012
pressure to provide a high-quality meeting because not only were her colleagues all invited,
but her curriculum coordinator and assistant superintendent would be present.

This suggests that the lead teacher role might be more demanding than some believe,
particularly when the role is taken on by a very conscientious teacher. It also suggests the
potential value of expanding the materials provided by LTF.

For this particular meeting, which was after school hours, only about 4 out of 20 invited
teachers attended. It would not be surprising if a lead teacher became less engaged over time
in response to such poor attendance after such concerted efforts. Finding ways to increase
attendance seem imperative for that reason, as well as for the more basic reason that low
attendance means that potential impacts are lost. Meetings held outside of contractual time
may have an extra level of burden to be engaging and useful to maximize attendance.
The MMSI program director has committed to attending many districts’ vertical team meetings, and has
forwarded emails to UMDI which indicate that she is collaborating with districts to help them plan their
meetings. She confirmed that many districts are not using the materials provided by LTF to guide the
vertical team meetings. She also reported that MMSI envisions the vertical team meetings including a
greater amount of sharing among colleagues, such as discussing strengths and weaknesses of LTF
activities and assessments that they have used, what they plan to try next and why, how they will work
together to facilitate the work, and how to improve over time. The MMSI program director also said that
vertical team time could be used to learn how to use new technology that is required by LTF activities,
such as probes or graphing calculators. She saw this as a shift from vertical team meeting activities that
LTF has suggested, such as discussing how to teach text annotation at appropriate levels across several
grades. She said,
I actually encourage districts to get rid of those [LTF vertical team meeting agendas] if that
wasn’t what they needed to do, and to do what they needed to do instead. It seems to have been
helpful advice. The overriding goal of most of the districts right now is alignment to the Common
Core in English and in math, and so that is where we are focusing right now… ‘Tell me what
your biggest needs are, and we will use Pre-AP to address those needs,’ as opposed to Pre-AP just
being another initiative on your plate.
She also said that one good strategy for developing vertical team meeting agendas would be to focus on
preparing teachers to implement one or two LTF lessons or assessments:
I would like to see districts…bring in a lesson or two for each of the grade levels [to] present at
the meeting, and then do those lessons together and discuss who is going to be responsible for
various portions of lesson development and implementation…By doing this through the vertical
team meetings, you can effortlessly get eight activities and assessments implemented during the
school year…We are talking about lessons that they have already done during summer training,
so this would be their second exposure.
Factors Facilitating and Obstructing Implementation
Factors facilitating implementation. The Pre-AP approach was seen by administrators as a
good fit for their district's need for greater rigor and higher student performance in mathematics, science,
and English language arts. Administrator survey respondents were asked to identify the factors and
strategies that were the most helpful in getting the Pre-AP program started, and 7 out of 15 reported that
UMass Donahue Institute
Research and Evaluation Group
34
RTTT C&CR Evaluation Annual Report, September 2012
the Pre-AP training was the most helpful. One said, “Having both the middle school and high school
teachers participating in the LTF Pre-AP trainings has provided the pedagogy and expectations for
teachers to identify and incorporate a greater level of rigor in their classes.” Seven also reported that the
support of staff, teachers, and other administrators was helpful, including two who reported that districtlevel curriculum coordinators have attended all of the district’s vertical team meetings. Six respondents
mentioned vertical team meetings, which they said have allowed teachers from different grade levels to
exchange ideas and strategies and to foster a greater connection with each other. One administrator said,
The relationships were made stronger; the respect from middle school to high school grew; we
broke down boundaries and had a really fun time with this. We never saw this as just a Pre-AP
initiative; we saw this as an effort that was in direct correlation with all of the other RTTT
initiatives.
Five administrators reported that having coordinators and lead teachers oversee Pre-AP program activities
and implementation was helpful, including one who said that this was “crucial” to implementation. Other
helpful factors mentioned were financial planning and compensation (N=2), good quality training and
resources (N=2), and having worked with MMSI before (N=1). One administrator felt that common
planning time is a key resource for helping same-discipline teachers deepen their use of Pre-AP
approaches, but that creating that within existing schedules will be a major challenge. He felt that vertical
team meetings are a good start but not sufficient for deeply embedding the Pre-AP activities.
Administrator obstacles to implementation. Administrator survey respondents were also asked,
“What have been the greatest challenges in getting your Pre-AP program started? Are there changes you
have made in response to these challenges?” The most common challenge reported by administrators was
time (N=5). Three noted that scheduling vertical team meetings after school is difficult, because middle
school and high school end at different times. One said that the district extended professional
development time in order for vertical teams to meet, but was still unable to hold the four required
vertical team meetings. Two respondents, both from the same district, said that their professional
development days had been set aside for category training and NEASC accreditation, so they had little
time for vertical team meetings. Finally, one said that time has been a challenge for Pre-AP trained
teachers who have attempted to integrate Pre-AP lessons into their curriculum maps.
Three administrators reported additional challenges in relation to vertical team meetings. One said that
vertical team meetings have not been very productive for middle school teachers. Two said that lead
teachers elicited resistance from their colleagues during vertical team meetings, by instructing teachers on
program content rather than presenting the LTF curriculum as a source of reference material from which
they could draw. The situation was improved by discussing the issue with lead teachers and receiving
guidance from MMSI.
Additional challenges included obtaining staff buy-in (N=2), launching the program when some teachers
hadn’t yet been trained (N=2), needing additional funding to purchase nonfiction reading materials and
hire additional personnel (N=2), and ensuring that teachers were actually embedding Pre-AP activities
into their lessons (N=2). One respondent said that implementation has not been visible, despite close
collaboration between lead teachers and department chairs, a challenge which one respondent said should
be addressed by encouraging administrators to monitor the classrooms of Pre-AP trained teachers.
One administrator reported that it has been difficult to have the vendor understand and integrate their new
curriculum demands together with the Common Core. This respondent said:
UMass Donahue Institute
Research and Evaluation Group
35
RTTT C&CR Evaluation Annual Report, September 2012
The vendor we must deal with, MMSI, offers PD that is seemingly lesson plans in a can, created
by LTF in Texas. Texas is not necessarily the go-to state on the Common Core.2 It has been a bit
of a battle to get the vendor to reference their work in terms of the new MA 2011 standards. This
is perplexing, given that they are the vendor for the state. It is challenging to support lesson
implementation when the lessons appear contrived and are not necessarily on point with the
language that transitions us to [the Common Core State Standards]. An internal change is that I
have reassigned more of my staff members' time to have oversight and push-back to some of the
plans within the vendor’s program.
MMSI noted that some efforts have been made to link LTF materials to the Common Core standards, and
to make these connections transparent for teachers using those materials in their classrooms.
Another respondent said that prior to stepping into her current administrator position, the district had
already decided to focus only on the English Language Arts strand of the LTF program. She expressed
interest in expanding the school’s Pre-AP programs to include science, math, and social studies.3
Teacher obstacles to implementation. The teacher survey asked respondents to describe any
other factors that significantly challenged their ability to implement Pre-AP lessons and/or assessments
(beyond the ones already asked about in the closed-ended items). Half of the respondents across all three
disciplines reported that the Pre-AP content was much too difficult for their low-level students. Seven
ELA teachers, five science teachers, and nine math teachers reported that Pre-AP materials were
extremely challenging for their students who lacked prior knowledge or still struggled with reading and
writing. For the math teachers, a typical sentiment was:
My students' skill level and content knowledge are so low that they are barely able to grasp basic
concepts, never mind participate in Pre-AP lessons. As an 8th-grade teacher, I am forced to spend
time teaching 5th–7th grade skills that my students have never mastered, such as adding fractions,
subtracting decimals, and comparing numbers.
When asked about the difficulty of the LTF lessons and assessments, the MMSI program director said,
We talk about the lessons and we talk about how to scaffold the lessons for the needs of their
students. We also talk about scaffolding during training. It is really easy to say that these
activities are too hard for students…These lessons are tied to the Common Core standards…[so]
we need to get the students to the point they need to be so that they can do these materials. We
need to work together to make the teachers comfortable presenting it and introducing [the
lessons] to the students. It is certainly not a reason to not do the activities.
Time was the second most common challenge mentioned (N=11). Some teachers also said that they do
not have any common planning time with other teachers in the same discipline at their schools, or with
teachers at other schools in their districts. Across all three subjects, respondents said that trying to balance
Pre-AP lessons and assessments with other competing demands has been difficult, given an already
stressful schedule of trying to keep up with pacing guides, implementing district curriculum, distributing
tests, and planning lessons. One math teacher said,
2Texas
is one of five states that has not adopted the Common Core State Standards Initiative. Reference: In the States. Retrieved
June 21, 2012, from http://www.corestandards.org/in-the-states.
3 Social studies is not currently an option for MMSI’s pre-AP program in Massachusetts.
UMass Donahue Institute
Research and Evaluation Group
36
RTTT C&CR Evaluation Annual Report, September 2012
Need to implement STEM, need to implement Pre-AP, need to implement stronger, broader, and
deeper common standards while students still come to class without mastering prior standards,
need to implement more summative evaluations, need to pull more time from class to implement
increased MCAS testing, and need to cut the homework ‘road to nowhere’. How can we continue
to pretend to add curriculum and increase rigor while at the same time increasing testing, cutting
total class time and eliminating stress by cutting homework? The LTF material is high quality but
extremely time consuming for the content presented and required within a course.
In response to this concern, the MMSI program director acknowledged that time can be a challenge. She
said, “We are asking [districts] to look at their curriculum and see that one activity covers multiple
standards. If you look at it that way, you are going to free yourself up with time to focus on the activity
and consolidate lessons.”
The third most common challenges for respondents were curriculum-related (N=9). They reported that a
poorly organized curriculum, a constantly changing curriculum, inconsistent pedagogical approaches used
by teachers, and a disconnection between district curriculum and Pre-AP materials significantly
challenged their ability to implement Pre-AP lessons and assessments. One respondent said that there was
“No explicit bridge or connection between prescribed curriculum and Pre-AP materials. With no flags or
alerts tied to the topics identified in pacing guides or standards, finding an appropriate Pre-AP activity
becomes a deliberate and tedious act of manual searching – not exactly inspirational for planning.”
Another common challenge was the lack of equipment to support Pre-AP lessons (N=7). Three
respondents were science teachers who reported lacking equipment or laboratory spaces. Other teachers
noted that they lacked needed calculators, reading materials, or computer technology.
One teacher expressed concern about having conducted a Pre-AP lesson that appeared successful on the
surface, because students carried out the intended activities, but that didn’t yield the intended learning.
She said, “I have very few students who possess the skills to do even the most basic activities. I used an
activity involving rate of change, which I thought was excellent. But after two days they had no idea what
they had done, and they did the activity quite well!” A final issue raised by one respondent was
contractual problems, saying “The fact that we are working without a contract leads to some reluctance to
go above and beyond the time and initiatives that have been agreed to.”
Implementing LTF lessons and assessments requires substantial change for some teachers. While some
welcomed the richness of the offerings, others who have well-established routines reported being
reluctant to invest the substantial time needed to change. The change is not just in the activities
themselves, but in the approach, such as being more student-driven than teacher-driven. Respondents
reported that this requires more complex management from teachers, because it’s not just telling students
what to do, but rather requires students to be more active, motivated, and generative. Student-driven
activities can also require more class periods to complete than a teacher-driven activity that addresses
comparable standards. However, a teacher reported that, when implemented well, many students find
these types of activities very engaging, and the activities can promote higher-order skills – such as
“thinking like a scientist” – that are needed for the AP exams and linked to college and career readiness.
UMass Donahue Institute
Research and Evaluation Group
37
RTTT C&CR Evaluation Annual Report, September 2012
Advice
Advice from teachers implementing LTF lessons and assessments. Teacher survey
respondents were asked to provide advice for others who are trying to implement the LTF lessons and
assessments in their classroom. Out of 61 respondents, 21 advised other teachers not to hesitate to try
using LTF materials in the classroom. Some suggested starting small and adding more lessons as their
comfort level increases. Others advised teachers to be patient and not to get discouraged. One teacher
said, “Just commit and practice as often as you can. We teachers have a tendency to stick to our methods,
so really embrace the strategies early in the year and continue to use them as often as possible.”
Respondents (N=21) also said that teachers should modify lessons to fit student levels, and to align
lessons with school curriculum. One teacher said, “Be ready for the kids to struggle with it initially, but
push them through, guide their attempts, fill in blanks where they don't have the knowledge, and it WILL
pay off in the long run. If necessary, scaffold the lessons even further so that kids will have an easier time
wrapping their heads around the new material.”
Six respondents advised teachers to plan ahead. Four advised them to complete the lessons and
assignments themselves first, before giving them to students. Other advice included using the LTF
website (N=3) and working with a partner to discuss lessons (N=3).
Advice from administrators implementing LTF curriculum. Administrator survey
respondents were asked to provide advice for others who were trying to implement the LTF curriculum in
their schools or district. Eleven respondents provided diverse advice, including to work hard to get
teacher buy-in (N=1), to be persistent (N=1), to start small and build trust among the team (N=1), and to
get teachers and administrative staff teams trained in implementation and support (N=2). Additional
comments included:

Be patient and really take the time to complete the trainings. The trainings are engaging and
challenging. The teachers I have worked with walk away feeling more confident themselves
about their content and skills.

Seek to collaborate with surrounding districts that are implementing the program. The LTF
program is outstanding and truly has impacted our district in a positive way. Collaboration
and communication on vertical team meetings and ways in which the program has been
implemented in other districts has been very useful.

Make it a priority. Make it content-focused. Make it fun. We had a great time. Our vertical
meetings flew by and people often stayed later than expected. Just don't view it (Pre-AP) in
isolation. View it as something that can help you reach many goals in your district. I got to
know the entire math staff 6-12, we had an opportunity to meet and talk and I was able to
listen to their needs and they felt supported.

Those who are trying to implement the curriculum need to make sure that they have the time
and capacity to work with the LTF curriculum to make it their own.

Slow and steady growth and implementation. Let word spread. Conduct comparative data
between teachers of the same content/grade level to talk about the strategies informally. This
will engender buy-in and build cohorts.

Connect to curriculum development. Use professional development strategically to help reach
scale. Think about who sits on the cohort team receiving the direct training from LTF and
UMass Donahue Institute
Research and Evaluation Group
38
RTTT C&CR Evaluation Annual Report, September 2012
how these folks can be leveraged for support of implementation in schools and at different
grade levels. How can these folks be leveraged for the curriculum development work and
connecting one to the other? Truly understand the MMSI goals...make them part of your own.
Communicate, communicate, communicate.
Support
The collaboration between ESE and MMSI seems to be productive and evolving. It appears that ESE sees
MMSI as expending tremendous efforts, and that MMSI has substantial abilities to facilitate work in the
field. ESE concerns with MMSI have included:

Insufficient attention to meeting reporting deadlines and tracking required data. (MMSI has
attempted to address this concern by hiring additional staff.)

Capacity concerns regarding the substantial increase in the number of teachers to be trained
and districts to be supported starting in summer 2012. (MMSI has attempted to address this
concern by hiring additional staff, although it was not clear at the time of this report whether
that change had been sufficient to address the capacity shortfall.)

Canceling some summer 2011 trainings, asserting that they needed at least 24 teachers in a
given subject area to offer a training, but not having stated that in their original agreement
with ESE. This left 20-30 teachers unable to receive training.

Providing different training for biology, chemistry, and physics teachers, which makes it
difficult to gather enough teachers to hold a training. ESE proposed that LTF might address
this challenge by combining the three science trainings into a single training that integrates
activities from all three disciplines.
With regard to technical assistance and support, most teacher and administrator survey respondents
agreed that the response from MMSI and ESE had been helpful and satisfactory when the school or
district had requested support.
Teacher Survey: MMSI Technical Assistance & Support
Survey Item
When my school or district has
requested technical assistance and
support from MMSI, the response
has been helpful and satisfactory.
4D
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
Y
44
20
34
46
0
0
N
16
19
44
37
0
0
Note. Response options also included “Not Applicable” and “Don’t Know”. The percentage of
respondents who selected those choices are listed below but, those respondents were not used to calculate
the percentages reported in the table above. 4D: 0% NA, 52% DK, Non-4D: 0% NA, 29% DK.
UMass Donahue Institute
Research and Evaluation Group
39
RTTT C&CR Evaluation Annual Report, September 2012
Administrator Survey: Technical Assistance and Support from MMSI and ESE
Survey Item
When my school or district has
requested technical assistance and
support from MMSI, the response has
been helpful and satisfactory.
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
16
77
15
8
0
0
When my school or district has
requested technical assistance and
support from the Department of
8
23
62
15
0
0
Elementary and Secondary Education,
the response has been helpful and
satisfactory.
Note. Response options also included “Don’t Know.” The percentage of respondents who selected this
choice is listed below, but these respondents were not used to calculate the percentages reported in the
table. Q1. 19%, Q2. 19%.
MMSI and ESE training and technical assistance. Administrators were asked to identify what
training or technical assistance from MMSI or ESE had been most helpful, and 12 administrators
provided comments. Half of those responding to this item (N=6) reported that support from MMSI and
ESE’s staff and their responsiveness to inquires had been the most helpful. Respondents stated that MMSI
quickly responded to calls, attended meetings, and made helpful suggestions regarding the purchase of
equipment. Five respondents reported that MMSI provided planning support. These administrators noted
that MMSI helped to plan and implement vertical team meetings, prepare for Year 2 cohort training, and
develop a Letter of Agreement. One district said, “They assisted in creating the agenda, topics and
activities for the teachers, facilitated and mentored the Pre-AP Team Leads, modeled the ideal format for
a vertical team meeting, remained flexible based on our needs... there is so much they did to support us
every step of the way.” Respondents also reported that digital resources and modules (N=1), follow-up
training on Vernier equipment (N=1), and LTF summer trainings (N=1) were helpful.
MMSI reported that some schools want to train extra lead teachers, both to develop leadership within
their departments and to provide equity across schools. Under current agreements, this would be
expensive for districts and a capacity challenge for MMSI, because MMSI is currently committed to
providing 12 hours of support per lead teacher. MMSI said that to accommodate districts that want to
train 5-10 lead teachers, MMSI needs to come up with a pricing structure and adequate staffing that
allows them to provided needed support.
Additional support from MMSI and ESE. Administrator survey respondents were asked to
identify what further training or support from MMSI or ESE would be must helpful, and 13 provided
comments. Five asked that MMSI and ESE continue to provide support for “the modules,” the coaching
of lead teachers, and the planning and implementing of vertical team meetings. One administrator was
unable to attend last year’s administrator training and expressed interest in attending this year. Another
administrator said,
UMass Donahue Institute
Research and Evaluation Group
40
RTTT C&CR Evaluation Annual Report, September 2012
It would be great if MMSI supported us by letting us make some of our own decisions and
followed communication protocol. In the past several months, they have sent out emails to our
teachers promising PD and possible stipends (without district permission). This has cost a great
deal of energy, as we respond to teachers who have misinformation. They have also seemed to
insist that we offer our teachers multiple PD dates when some of the dates are in direct conflict
with district PD, yet we have had to open the dates that MMSI wants.
Respondents reported that it would be helpful if MMSI or ESE provided additional materials and
resources such as textbooks and technology (N=1), provided modeling and in-district coaching (N=1),
followed up on implementation with new lead teachers (N=1), negotiated with ESE about the grant
process (N=1), and provided information about funding opportunities beyond the RTTT grant (N=1). One
administrator expressed interest in making better use of assessment materials, stating that she/he was less
certain of their alignment with the Common Core. Another administrator said, “I would also like to see
more about the Speaking and Listening and the Language strands of the CCSS in ELA/Literacy,” and to
know more about the “implications of Standards for Mathematical Practice and NGSS – practices and
cross-cutting concepts (in service to the content of math and science).”
The teacher survey asked what additional supports from MMSI would be most helpful, and 44 offered
comments. Across all three disciplines, 48% (N=21) said that access to more resources and materials
would be most helpful. These included reading texts and 9th-grade Pre-AP writing materials for ELA
teachers, more interactive lessons for math teachers, and more equipment for science teachers. Two
wanted monthly email blasts with new LTF lessons and resource links, and four wanted additional
differentiated lessons for their lower-level students. Three teachers requested having LTF materials in
Microsoft Word format, so that they could customize them, and two teachers wanted instructional or
demonstration videos.
Teacher survey respondents from all three disciplines (N=11) wanted help in learning how to modify their
LTF lessons for lower-level students and how to integrate Pre-AP curriculum with their district
curriculum. One teacher expressed, “It would be great to actually work with MMSI in fully integrating
different strategies into all classes, in targeting specific areas, and in creating Pre-AP lessons to coincide
with all curriculum.” Finally, six mentioned wanting more meetings and professional development,
including during the school year rather than just during the summer, as well as follow-up meetings with
their initial training cohort and other teachers to reflect on Pre-AP lessons they have conducted.
Additional requests included science equipment, graphic calculators, and the layered English text (N=5);
training in algebra for middle school students (N=1); another set of LTF manuals (N=1); English lessons
that incorporate technology such as Eno boards (N=1); a blog to communicate with fellow trainees (N=1).
Many teachers (N=10) also wanted all grade 6-10 teachers to complete Pre-AP training in order to
increase AP readiness.
Teacher survey respondents were asked to specify what training or technical assistance from MMSI had
been most helpful, and 72 offered comments. Forty-seven percent (N=34) said that access to resources
and materials such as lessons, assessments, manuals, and the LTF website had been most helpful. Fortynine percent (N=35) said that participating in training workshops and professional development activities
throughout the school year and during the summer provided them with useful strategies, including
opportunities to collaborate with peers and to work through the lessons themselves. Four math teachers
cited instruction in using graphing calculators as the most helpful form of assistance.
UMass Donahue Institute
Research and Evaluation Group
41
RTTT C&CR Evaluation Annual Report, September 2012
Lead teachers were asked about the support provided by MMSI. As shown in the table below, of those
whose districts had held vertical team meetings, 57% agreed that the materials and activities provided by
MMSI for structuring a Vertical Team meeting were effective in achieving the goals of the meeting. Of
those whose district had held a vertical team meeting, 50% agreed that the support provided by MMSI for
structuring and running Vertical Team meetings had helped them develop leadership skills.
Teachers were asked to respond to the item, “When my school or district has requested technical
assistance and support from MMSI, the response has been helpful and satisfactory.” Fifty-two percent
reported that they didn’t know, which is unsurprising, as teachers are less likely than administrators to be
involved in such communications. Of the remaining teachers, 54% agreed, and the rest were neutral; no
one disagreed or strongly disagreed.
Teacher Survey: MMSI Support for Vertical Team Meetings
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
The materials and activities provided
by MMSI for structuring a Vertical
Team Meeting are effective in
achieving the goals of the meeting.
14
7
50
36
7
0
The support provided by MMSI for
structuring and running Vertical Team
Meetings has helped me develop my
leadership skills.
14
7
43
43
7
0
Survey Item
Note. Response options also included “We have not held a vertical team meeting”. The percentage of
respondents who selected those choices are listed below, but those respondents were not used to calculate
the percentages reported in the table. Q1. 4D: 5%, Non-4D: 0% Q2. 4D: 26%, Non-4D: 0%.
When asked about the support provided by MMSI, 64% agreed that they were satisfied. When asked if
the role of lead teacher required more time than they had available, just 18% agreed.
UMass Donahue Institute
Research and Evaluation Group
42
RTTT C&CR Evaluation Annual Report, September 2012
Teacher Survey: MMSI Support Satisfaction and Lead Teacher Time Available
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
When I have requested support from
MMSI, I have been satisfied with the
support I received.
14
7
57
36
0
0
The role of the lead teacher requires
more time than I have available.
17
6
12
59
23
0
Survey Item
Note. Response options also included “Not Applicable”. The percentage of respondents who selected
that choice is listed below, but those respondents were not used to calculate the percentages reported
in the table. Q1. 4D: 26%, Non-4D: 50%, Q2. 4D: 11%, Non-4D: 0%.
The MMSI program director expressed concern that districts who are actively requesting assistance may
be receiving a disproportionate amount of MMSI’s time, while other districts who may also need support
are neither requesting nor receiving it. She said,
What I need to do is create a [process that I use to] reach out… if I haven’t heard from a given
district, I need to continually ping them until get some communication with them going. Not
everyone needs the same level of support from me…but I can’t assume that if someone hasn’t
reached out to me, or hasn’t responded to my offer of help, that they don’t need it.
Sustainability
The ESE program director for Pre-AP reported that there is currently no policy in place to promote the
sustainability of Pre-AP programming or practices beyond the RTTT funding period. Data collection in
Year 3 will investigate what strategies school or districts are employing to promote sustainability.
Baseline Graduation and Dropout Rates
The following three pages each contain a table paired with a figure that present the following baseline
graduation and dropout rate data for high schools in the 28 Cohort 1 districts.4
1. Average 4- and 5-year cohort graduation rates from 2007-08 to 2010-11.
2. Average annual dropout rates for 2007-08 to 2010-11.
3. Average 4- and 5-year cohort dropout rates 2007-08 to 2010-11.
The companion document “Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report”
presents tables and figures that provide a more complete summary of these same baseline graduation and
dropout data. The Technical Supplement contains tables and figures that present both average and
4
Current MMSI registration and attendance records indicate that 37 districts sent one or more teachers to training during the
summer of 2011. Of these 37 districts, 28 are currently assigned to Cohort 1. Questions remain regarding the cohort assignments
of the remaining 9 districts, which are not included in the analyses presented in this report.
UMass Donahue Institute
Research and Evaluation Group
43
RTTT C&CR Evaluation Annual Report, September 2012
individual graduation and dropout rates for RTTT Goal 4D districts and individual schools. Future
evaluation reports will use these baseline data to describe trends in graduation and dropout rates in PreAP schools and districts. The data necessary to disaggregate baseline graduation and dropout rates by
subgroup (e.g., gender, race/ethnicity) were not available in time for this report, but will be included in
future analyses. Future reports will also describe trends in indicators of student performance (e.g., MCAS,
attendance).
UMass Donahue Institute
Research and Evaluation Group
44
RTTT C&CR Evaluation Annual Report, September 2012
Pre-AP Cohort 1 4D & Non-4D: 4- & 5-Year Graduation Rates
4- & 5-Year Graduation Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11
Number of Students
4D
11,353
11,283
11,056
10,432
Year
2007-08
2008-09
2009-10
2010-11
Non-4D
3,012
2,988
3,060
2,831
4-Year Graduation
Rate
4D
67.5%
68.5%
69.5%
71.1%
5-Year Graduation
Rate
Non-4D
80.5%
81.0%
81.3%
82.5%
4D
72.3%
72.5%
73.5%
NA
Non-4D
83.0%
84.2%
84.4%
NA
Difference Between 4and 5-Year Rates
4D
4.8%
4.0%
4.0%
NA
Non-4D
2.5%
3.2%
3.1%
NA
4- & 5-Year Graduation Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11
100%
95%
4-Year Rate Pre-AP 4D
Graduation Rate
90%
85%
5-Year Rate Pre-AP 4D
80%
75%
4-Year Rate Pre-AP non-4D
70%
5-Year Rate Pre-AP non-4D
65%
60%
55%
50%
2007-08
UMass Donahue Institute
Research and Evaluation Group
2008-09
2009-10
School Year
2010-11
45
RTTT C&CR Evaluation Annual Report, September 2012
Pre-AP Cohort 1 4D & Non-4D: Annual Dropout Rates
Annual Dropout Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11
Total HS Enrollment
Number of Dropouts
Annual Dropout Rate
Year
4D
Non-4D
4D
Non-4D
4D
Non-4D
2007-08
41,195
10,971
2,692
426
6.5%
3.9%
2008-09
40,289
10,591
2,315
303
5.7%
2.9%
2009-10
39,569
10,398
2,075
316
5.2%
3.0%
2010-11
39,443
10,145
2,011
316
5.1%
3.1%
Annual Dropout Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11
10%
9%
Annual Dropout Rate
8%
7%
6%
5%
Pre-AP 4D
4%
Pre-AP non-4D
3%
2%
1%
0%
2007-08
UMass Donahue Institute
Research and Evaluation Group
2008-09
2009-10
School Year
2010-11
46
RTTT C&CR Evaluation Annual Report, September 2012
Pre-AP Cohort 1 4D & Non-4D: 4- & 5-Year Cohort Dropout Rates
4- & 5-Year Cohort Dropout Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11
Number of Students
4D
11,353
11,283
11,056
10,432
Year
2007-08
2008-09
2009-10
2010-11
Non-4D
3,012
2,988
3,060
2,831
4-Year Dropout Rate
4D
18.2%
16.7%
14.2%
13.2%
Non-4D
11.1%
9.4%
8.2%
7.7%
5-Year Graduation
Rate
4D
18.4%
17.0%
15.1%
NA
Difference Between 4and 5-Year Rates
Non-4D
10.5%
8.7%
8.3%
NA
4D
0.2%
0.3%
0.9%
NA
Non-4D
-0.6%
-0.7%
0.1%
NA
4- & 5-Year Cohort Dropout Rates for Pre-AP 4D & Non- 4D Districts, 2007-08 to 2010-11
30%
Cohort Dropout Rate
25%
20%
4-Year Rate Pre-AP 4D
15%
5-Year Rate Pre-AP 4D
10%
4-Year Rate Pre-AP non-4D
5-Year Rate Pre-AP non-4D
5%
0%
2007-08
UMass Donahue Institute
Research and Evaluation Group
2008-09
2009-10
School Year
2010-11
47
RTTT C&CR Evaluation Annual Report, September 2012
Conclusions
The Pre-AP Training and Alignment initiative is well underway, and districts are participating in Pre-AP
training in multiple configurations. During summer 2011, MMSI provided four-day summer trainings for
561 teachers from 37 districts (including 19 4D districts), and teachers attended an average of 3.6 days
(out of 4). Also during summer 2011, 30 teachers participated in lead teacher training, and 25
administrators (from seven 4D districts) participated in administrator training. For summer 2012, more
than 900 teachers registered for training; final registration data were not available for this report. The ESE
program manager reported that, after summer 2012, Massachusetts will have met the performance goal of
having trained 1,000 teachers. In addition, the technical assistance and support provided by MMSI and
ESE were widely seen as helpful and satisfactory.
Lead teachers and administrators generally reported being satisfied with the Pre-AP training events.
Administrators and teachers largely agree that the LTF lessons and assessments are well-aligned with the
Common Core standards and are examples of high-quality pedagogical practices. Teachers were asked
about the impacts of their participation in Pre-AP training, and 70% or more of 4D teachers agreed that,
as a result of the Pre-AP training, they now teach more Pre-AP content, use more Pre-AP pedagogical
strategies in the classroom, and have greater awareness of the importance of using Pre-AP strategies.
Substantial challenges to Pre-AP implementation were reported by both administrators and teachers with
regard to curricular resources, classroom equipment, planning time, pacing guides, and student academic
readiness. Such concerns represented a substantial minority in some cases, and a clear majority in others,
particularly in relation to the adequacy of planning time and the academic readiness of students for the
Pre-AP curriculum. The number of Pre-AP lessons and assessments that teachers are expected to
implement varies widely across districts.
During the 2011-12 school year, 4D district teams reported attendance at an average of 3.5 vertical team
meetings, with a range of 2 to 4 meetings. The meetings were attended by an average of 63% of summer
2011 trainees, plus an average of four personnel who hadn’t completed Pre-AP training. Some lead
teachers were not satisfied with the agendas and activities provided by MMSI to structure vertical team
meetings. Some lead teachers made slight modifications to agendas, and others completely restructured
agendas to meet their district’s needs. The MMSI program director reported encouraging lead teachers to
make changes as needed to meet their school’s goals.
The LTF program philosophy, strongly shared by ESE, is that every student should be considered a PreAP student, but evaluation findings indicate diverging voices and practices on this topic in the field. Most
teachers and administrators who were interviewed agreed that every student should be considered a PreAP student, and multiple interviewees expressed mild to strong opposition to tracking. In contrast,
responses to the teacher survey provided more varied responses. The survey asked teachers if they used
more Pre-AP materials and strategies with some classes than with others. About half said yes, and most of
those indicated that they used more Pre-AP materials and strategies with their honors or advanced-level
students. This group of respondents generally agreed that Pre-AP materials were not suitable for all
students, such as those in remedial classes, those who had not mastered skills from a previous course, and
some English language learners.
UMass Donahue Institute
Research and Evaluation Group
48
RTTT C&CR Evaluation Annual Report, September 2012
Strategic Considerations

Expectations for minimum implementation levels should be clarified or reiterated.
While some district administrators reported that their Pre-AP trained teachers were expected
to implement several Pre-AP lessons or assessments each quarter, others reported that they do
not require their teachers to implement any Pre-AP lessons or assessments. This suggests that
participating in the Pre-AP program does not imply participation in a specific set of practices
across schools or even classrooms within schools. However, MMSI indicated that 4D districts
have clearly committed to implementing Pre-AP lessons and assessments. To the extent that a
minimum expectation exists, it should be communicated clearly to program participants.

Providing additional materials and suggestions for differentiation of LTF lessons may
increase classroom implementation. Several teachers noted that the time required to modify
LTF lessons for differentiation purposes was an obstacle to implementation. Moreover, the
MMSI program director affirmed that such differentiation was strongly encouraged, and that
appropriate modifications could make LTF lessons more accessible for students with lower
skill levels. The more that the LTF lessons and assessments can incorporate materials and
suggestions for such differentiation, the more broadly the lessons and assessments may be
implemented by teachers.

Further assessments of MMSI trainer quality, and additional training of some trainers,
may be indicated. While a formal assessment was not undertaken, UMDI’s informal
observations (made by an experienced classroom teacher) suggested that trainers varied
widely in terms of the quality of their instruction and the extent to which their instruction
modeled LTF’s proposed pedagogy.

The level of implementation of Pre-AP practices should be monitored and interpreted in
relation to student impacts. Teachers and districts clearly have many factors in addition to
the Pre-AP training that affect their practices and policies, and implementation of Pre-AP
lessons and assessments appears so far to represent at most a moderate portion of classroom
activities. As the number of Pre-AP trained teachers and their level of experience with PreAP practices increases, the pace of change and the potential for student impacts may increase
accordingly. Over time it will be important to continue monitoring the extent of
implementation, to discuss whether that extent can or should be increased, and to interpret
findings regarding student impacts in relation to the extent of implementation.

Pre-AP schools may need to budget for additional equipment and professional
development. LTF math and science activities require expensive equipment that some
schools lack, particularly at the middle school level. Teachers may also require substantial
professional development and/or preparation time to be ready to use this equipment with
students, and schools may need guidance about which equipment to purchase. Several
districts have asked MMSI to advise them about what equipment to buy, and MMSI has been
able to provide advice as well as referrals to other consultants. Schools who wish to adopt
Pre-AP activities may need to budget for additional costs.

Clarification of the intended uses of vertical team meeting time might lead to increased
Pre-AP implementation. Vertical team meetings were well-launched, but clarifying intended
meeting content could be important. Some teachers and administrators have reported that they
have substantially modified the LTF curriculum provided for the vertical team meetings in
order to better meet their school’s needs, and the MMSI program director has approved of
this practice. However, it was unclear from UMDI’s lead teacher interviews and initial
observations of vertical team meetings, to what extent districts are using vertical team
UMass Donahue Institute
Research and Evaluation Group
49
RTTT C&CR Evaluation Annual Report, September 2012
meeting time to specifically advance Pre-AP implementation versus addressing more general
curricular goals. Clarification of expectations for 4D districts might lead to increased Pre-AP
implementation.

Sharing of modified vertical team meeting materials could save lead teacher time and
enhance meeting productivity. As already noted, Pre-AP lead teachers are modifying the
vertical team meeting materials provided by LTF. Collecting and sharing these materials
online might provide a resource that reduces the needed time investment and increases the
impact of lead teachers’ efforts. Lead teachers could receive reminders of this resource and
periodic updates regarding new or timely content.
UMass Donahue Institute
Research and Evaluation Group
50
RTTT C&CR Evaluation Annual Report, September 2012
STEM-focused Early College High Schools
Background
In its Race to the Top proposal, ESE proposed to open six STEM early college high schools to prepare
students for productive STEM careers, to reduce achievement gaps, and to provide an accelerated
pathway to postsecondary education for underrepresented students by partnering with colleges and
providing opportunities to earn up to two years of college credit while still in high school. Six districts
were chosen in a competitive process and are currently receiving RTTT Goal 4E funds for this purpose.
Eight additional STEM ECHS sites are being established with the support of discretionary RTTT funds,
and ESE has requested that UMDI postpone evaluation of those sites until their work has progressed
further. The evaluation questions specific to the STEM-focused Early College High Schools are listed
below.
Process Evaluation Questions
P1. To what extent have grantees developed and implemented STEM ECHS components? What
are the major challenges to and facilitators of successful program implementation
encountered by grantees? What midcourse corrections and attempts to overcome challenges
have been undertaken? What additional steps are necessary?
P2. To what extent has ESE implemented the STEM ECHS program components described in
their RTTT grant application? What are the major challenges to and facilitators of program
support and facilitation encountered by ESE? How have challenges been overcome and
midcourse corrections undertaken? What additional steps are necessary?
P3. How do STEM ECHS teachers and administrators rate and explain the quality, relevance, and
effectiveness of major program components and services?
P4. What infrastructure, systems, and processes were put in place to aid STEM ECHS
sustainability during and beyond the grant period? What are the greatest challenges and
barriers to creating sustainability?
Outcome Evaluation Questions
O1. To what extent are STEM ECHS students achieving improved outcomes in college and career
readiness indicators including attendance, credit accumulation and recovery, measures of
academic achievement (e.g., MCAS, PARCC, PSAT, SAT, and AP), enrollment in AP
courses, graduation, MassCore completion, accumulation of college credits, and college
enrollment and persistence (to the extent known during the grant period).
O2. Do observed changes in STEM ECHS students differ across student characteristics such as
gender, race/ethnicity, free/reduced lunch status, ELL status, and special education status? Is
there evidence that gaps are narrowing? Are program services reaching students who are at
the greatest risk?
O3. To what extent are observed changes in student outcomes attributable to STEM ECHS
activities versus other RTTT program activities and/or measurable contextual variables?
O4. What are the major differences in STEM ECHS implementation and contextual variables
across schools whose levels of improvement on student outcomes differ substantially?
UMass Donahue Institute
Research and Evaluation Group
51
RTTT C&CR Evaluation Annual Report, September 2012
O5. What is the relationship between level of program implementation and achievement of
targeted student outcomes?
Schools and Partners
The six chosen districts will each receive approximately $120,000 of RTTT funds to be spent over four
years for school planning, start-up expenses, and full implementation. Two schools opened with small
classes of students in fall 2011 (including one that suspended activities during the second semester); one
school opened during summer, 2012 (slightly ahead of schedule); and three additional schools are
scheduled to open in 2012. A list of funded school districts and their higher education partners is provided
in the table below.
STEM ECHS Sites and IHE Partners
District
Start
Institute of Higher Education
Date
Marlborough Public Schools
Fall 2011 Framingham State University
Quaboag Innovation STEM early college
Holyoke Community College,
Fall 2011
high school (Quaboag Public Schools)
Worcester Polytechnic Institute
Southeastern Regional Vocational Technical
Summer
(Massachusetts Association of Vocational
Northeastern University
2012
Administrators - MAVA)
Randolph High School
Fall 2012 Massasoit Community College
(Randolph Public Schools)
North High School
Quinsigamond Community
Fall 2012
(Worcester Public Schools)
College, College of Holy Cross
Dearborn Middle School
Fall 2012 Northeastern University
(Boston Public Schools)
Grades
Offered
6-12
9-12
9-12
9-12
7-12
6-12
Methods
The Year 2 evaluation included the following activities and data sources:

Interviews with ESE program managers, the technical assistance vendor, STEM ECHS
administrators, and IHE partners.
o
Three in-person interviews with ESE program managers in September 2011,
December 2011, and July 2012 (see interview protocol in Appendix B1).
o
Three phone interviews with the technical assistance vendor, Jobs for the Future
(JFF), in October 2011 and August 2012 (see interview protocol in Appendix B2).
o
Interviews with administrators from the six Goal 4E sites in February and March
2012. These included phone interviews with the four pre-implementing sites, and inperson interviews during site visits to the two implementing sites (see interview
protocols in Appendices B3 and B4).
UMass Donahue Institute
Research and Evaluation Group
52
RTTT C&CR Evaluation Annual Report, September 2012
o
A 1-hour phone interview in June, 2012 with an administrator from Framingham
State University regarding their partnership with Marlborough’s STEM ECHS (see
interview protocol in Appendix B5).

Focus group with school personnel. A focus group was conducted in June 2012 with two
middle school teachers and two high school teachers from the one STEM ECHS site that was
fully operational during the 2011-12 school year (see focus group protocol in Appendix B6).

Site visits. Two STEM ECHS sites were visited, one of which was implementing ECHS
activities at the time of the visit (see observation protocol in Appendix B7). Each site visit
included one or two interviews with key school personnel and brief observations of school
activities. In addition, UMDI attended two full-day technical assistance meetings hosted by
JFF and ESE.

Document review. UMDI reviewed two sets of reports from a JFF consultant's individual
meetings with the six schools, as well as several documents (e.g., proposals, continuation
reports, marketing materials) from several STEM ECHS sites. UMDI also monitored use of
“Base Camp” – an online resource maintained by JFF which serves as a professional learning
community and project collaboration tool.

ESE databases and documents. The STEM ECHS quantitative data presented below and in
the “Technical Supplement - RTTT C&CR 2012 Evaluation Annual Report” were developed
using ESE’s 4-year and 5-year graduation cohort lists as well SIMS data from October and
June of 2008-2011. Findings were also informed by ESE documents including the RTTT
application, and graduation and annual dropout rate information reported by ESE.

Supplemental student data request. Supplemental data requests were prepared for STEM
ECHS sites, asking them to indicate which students are receiving services, as well as
attendance and credit accumulation data that are not included in the state databases but are
needed for answering evaluation questions. The deadline for schools to respond is October
2012, so related findings will be incorporated into future reporting.
Findings
Findings are presented in relation to the process and outcome evaluation questions. Process questions P1
and P3 are addressed in the Project Status, Planning, and Implementation sections. Questions P2 and P4
are addressed in sections on Support and Sustainability. The outcome questions are then addressed with
the presentation of baseline graduation and dropout rate information for the STEM ECHS, a summary of
student participation to date, and a discussion of how outcome data that will be available for subsequent
project years will be utilized.
Project Status
The six STEM ECHS sites receiving RTTT support are in various phases of project development and
implementation. Two sites (Marlborough and Quaboag) enrolled students in STEM ECHS activities
during the 2011-12 school year. Three sites (MAVA, Worcester, and Randolph) are preparing to launch
with inaugural cohorts during the 2012-13 school year. Dearborn's STEM ECHS work has been
incorporated into a broader school redesign plan. Additional information for each site is provided below.
UMass Donahue Institute
Research and Evaluation Group
53
RTTT C&CR Evaluation Annual Report, September 2012
Marlborough. The Marlborough STEM ECHS began serving 6th- and 9th-grade students in fall
2011. District personnel reported that nearly 200 students (115 in grade 6 and 74 in grade 9) were served.
Marlborough intends to add two grades per year to the STEM ECHS until full enrollment (grades 6-12) is
reached in 2014-2015. The STEM ECHS programs are housed as schools-within-a-school in the middle
school and high school buildings.
The implementers for this site originally intended to place a higher priority on serving underachieving
students, many of whom were minorities. However, during the first year of program implementation all
applicants were accepted, approximately 50% of which were underrepresented minorities. The district
held several community forums in order to inform parents/guardians about the launch of the school, its
purpose, and the district’s plans for serving students.
Students entering 9th grade participated in an all honors curriculum (Geometry, Algebra, English, History,
Physics, and Engineering). District personnel reported that many students were unprepared for the
demands of this coursework. In order to accommodate student and teacher requests for increased support,
the district reduced teaching loads from five classes to four, and arranged a daily common planning
period at the high school. Similar accommodations were instituted for STEM ECHS teachers at the
middle school, but were subsequently cancelled because of perceived inequities in course loads. The
middle and high school both emphasized 21st-century skill development as well as inquiry- and projectbased learning experiences for students in all classes. The school issued laptops to all students to increase
technology use and enhance students’ educational experience. Marlborough is working with several
business partners to support the laptop initiative.
STEM ECHS administrators and Framingham State University, Marlborough’s IHE partner, are
developing a longitudinal study of the STEM ECHS. As a part of this work, a set of surveys and focus
groups were developed by FSU faculty and administered to STEM ECHS students and staff. To the extent
that these activities provide information relevant to UMDI's evaluation, they will be presented in future
reports. FSU has also designated a faculty fellow to provide professional development for high school
staff and students in the area of college readiness.
Marlborough has reported that most members of the current 6th and 9th grade STEM ECHS cohorts will be
promoted to 7th and 10th grade, respectively. A new group of approximately 135 6th-graders and 70 9thgraders have been selected for the 2012-13 school year. Marlborough’s STEM ECHS middle school
received 172 applications for the 135 available seats, so Marlborough selected students by lottery.
Marlborough anticipates that students will begin participating in college coursework for credit during
their junior year. There are no STEM ECHS students enrolled in college coursework during the 2011-12
school year, and none are scheduled to do so for the 2012-13 school year.
Quaboag. In the Fall of 2011, Quaboag enrolled 14 juniors and seniors in an electronics course
taught by a high school teacher. Students who passed the course and met Accuplacer testing score
requirements were eligible for non-transferable college credit at Quinsigamond Community College
(Quaboag’s IHE partner). This program was suspended during the spring semester to allow time for staff
to receive needed training and to further refine the articulation agreement between Quaboag and
Quinsigamond Community College. Quaboag is preparing to re-launch their STEM ECHS during the
2012-13 academic year.
UMass Donahue Institute
Research and Evaluation Group
54
RTTT C&CR Evaluation Annual Report, September 2012
MAVA. Representing nine vocational technical high schools, MAVA is preparing to launch a
STEM ECHS in the fall of 2012. The nine schools coordinate as one program through a planning
committee of school representatives that make all key decisions and appoint subgroups as needed.
Participating schools have agreed to make information technology the school's focus. The initial cohort of
students was selected at the end of their sophomore year through an application process that was
facilitated by school counselors and IT teachers. The selected students attended a June orientation
program at Northeastern University and completed their first college course during the 2012 summer
semester. Students in this STEM ECHS will participate in online courses and attend mandatory meetings
with their cohort group at Northeastern or another pre-determined site. Students may also take courses inperson at Northeastern.
Randolph. Randolph is preparing to launch a STEM ECHS in the fall of 2012. A partnership has
been established between Randolph High School and Massasoit Community College, Canton campus. A
total of 39 students have registered to participate in STEM ECHS activities during the 2012-13 school
year. Each of these students is participating in an Early College Experience and is scheduled to complete
the math Accuplacer. Twelve of these students are currently registered for a Heavy Truck Systems course,
in addition to their other courses. All courses are scheduled to be taught at Randolph High School.
Participating students are eligible to receive high school and transferrable college credit. The Early
College program is currently available to all high school students in grades 9-12.
Worcester. Worcester is preparing to launch a Health Sciences Academy at North High School
in the fall of 2012 with 50 sophomores and juniors. The program intends to target low-income,
underrepresented, and potential first-generation college students. Twenty-five rising juniors were
identified for participation in STEM ECHS activities during the spring of 2012 and attended a college
orientation course offered by Quinsigamond Community College (Worcester’s IHE partner). Worcester is
in the process of finalizing a Memorandum of Understanding with QCC.
Dearborn. Dearborn was awarded $1.2 million in School Improvement Grant funds to implement
an aggressive redesign plan to accelerate student achievement in the middle grades. Much of their focus is
on completing the activities related to that goal. The STEM ECHS project is part of that plan, but few
program specifics were available at the time of our evaluation activities. Developments at this site will be
monitored during Year 3.
Planning
During interviews and site visits, STEM ECHS program administrators described a wide range of
successes and challenges related to their STEM ECHS planning processes, as reported below.
Planning successes.
Assembling a planning team. Interviewees from five districts described assembling a planning
team as a successful aspect of their planning process. They said that a key to success was the intentionally
including representatives from diverse stakeholder groups. All districts included building administrators
and representatives from STEM ECHS faculty and partnering IHEs, and some districts also included
guidance counselors, union representatives, department heads, district-level administrators,
representatives from partnering middle schools, and other educational leaders. Interviewees said that the
inclusion of diverse stakeholders fostered support and buy-in for the project.
UMass Donahue Institute
Research and Evaluation Group
55
RTTT C&CR Evaluation Annual Report, September 2012
Several interviewees said that it was important to recruit volunteers for service on the planning committee
whenever possible. Several also said that they included people who understood the importance of college
readiness and who had experience in relevant content areas. Two interviewees stressed the importance of
including practitioners on the planning team, with one saying “In my view, the [people in the] trenches
every single day…the people who live it are the people that can tell you the most.”
The reported structure of planning team meetings was relatively uniform across sites. Teams and
subcommittees held regular weekly, biweekly, or monthly meetings that were mostly scheduled before or
after school hours to minimize time that teachers and administrators were removed from their regular
duties. Several interviewees said that supports for planning that were provided by JFF had been useful in
identifying important planning topics. Several interviewees said that consistent collaboration among team
members was a key to their success. Several also said that holding planning meetings on a regular basis
was difficult, because most team members had many other substantial responsibilities and commitments.
Interviewees agreed that they were pleased with the level of commitment, skill, and ability that members
of their planning teams had demonstrated.
Strong support from school and district leaders. Interviewees from five districts cited support
from district leadership as a success, and several said that this support was essential to the planning
process. They reported that their district and school leaders were generally responsive, positive,
supportive, and helpful. Some also said that it was important that district leadership had expressed a
strong commitment to the project and a willingness to participate in the planning process and assist with
challenges. One interviewee said, “We have the support right from the top, which is the superintendent.
That does not always happen. That is not the case in every district involved in this initiative, and it should
be. The district leadership has to be aware of the decisions that are being made and has to support the
work that is being done. That is the biggest hurdle.”
Establishing and formalizing relationships with IHE partners. Interviewees from five districts
said that they had established strong working relationships with their IHE partner. Four said that IHE
representatives regularly attended planning team meetings and were viewed as important team members.
Interviewees frequently noted that their IHE partners were willing to share expertise, contribute ideas, and
assist students in their transition into a college experience.
Supports for teachers. One district highlighted supports for teachers as a success in the planning
process. Supports for teachers at this site included the development of structured small learning
communities, common planning time, reduction of teacher duties, and professional development relevant
to English language learners and students with disabilities. Each of these supports was intended to help
teachers increase the quality of instruction provided to all students.
Other areas of planning success included selecting and securing a school site (two
districts),excitement in the school and/or community for the project (two districts), strong sense of
entrepreneurship and innovation (two districts), engaging the community (one district), being ahead of
schedule (one district), curriculum development (one district), and student recruitment (one district).
Planning challenges.
Budgeting. Three districts identified budget management as a challenge. One noted a lack of
clarity regarding the amount of funds available and how those funds could be used. Another wanted
greater clarity in the budgeting process, because funds received from the state for middle school and high
UMass Donahue Institute
Research and Evaluation Group
56
RTTT C&CR Evaluation Annual Report, September 2012
school activities had traditionally been lumped together and then reallocated at the district level. A third
interviewee reported not having the autonomy to make funding decisions for the program, resulting in
program delays while waiting for district administrators to make decisions.
Instructional design, curriculum development and assessment. Three sites said that they had not
made as much progress with instructional design and curriculum development as they had hoped, despite
having thought that they had established a clear process and appropriate supports. The first site explained
that their district had asked them to delay this work until a larger plan for the project was in place. The
second site said that delays had resulted from the time needed for staff members to receive training to
implement the target curriculum. The third site simply realized that their plans and supports had not been
as well-developed as they had believed.
Staffing. Three districts reported staff-related challenges. One interviewee from a relatively small
district said that they were constantly being asked to “do more with less” and assume multiple roles that
exceeded their capacity. Lack of capacity also limited the number and type of courses that this school
could offer to STEM ECHS students. Another interviewee said that staff turnover was a challenge,
because several members of their STEM ECHS planning team were replaced during the fall. A third
interviewee was concerned that staff might lack the skills to teach some of the courses that had been
envisioned for the STEM ECHS.
Engaging the community. Three districts said that engaging the community was a challenge.
Two said that parent/guardian and community interest and involvement in school-sponsored activities was
typically weak in their districts. One of these interviewees reported that their project had repeatedly made
efforts to contact parents and community members, with little success. The other site reported that they
had not made any effort to explain or promote their STEM ECHS project to the broader community. One
interviewee said, “We have not had the money or time to devote to telling our story, of how our STEM
ECHS is configured and [has] evolved.” One interviewee described the challenge and goal of engaging
the community by stating, “How do we engage our community in a fashion to show them that, through
the development of this new school, we were going to be doing the best for all kids in the city?”
A third site said that efforts to engage the community prompted negative feedback from local parent
groups and the media. This site reported that some parents were upset that the program had planned to
prioritize admitting underrepresented students:
We had an article written about us. It was an editorial article in one of the local newspapers that
talked about reverse discrimination, and the author of that editorial basically said that by…giving
preference to underrepresented kids, we were showing reverse discrimination towards those kids
who were high achievers, and whose parents were involved, and whose parents happen to be
either highly educated or educated…But we continued to present this as an opportunity for all
kids as we had established it. We really sort of just held our ground. We knew that a lot of the
clarity for the community would not occur until they actually saw this in action, saw it working
for students who hadn’t shown high levels of achievement in the same classrooms as students
who had shown high levels of achievement. And, more importantly, that even those students who
had exhibited high levels of achievement would still be challenged by the approach that we were
developing.
UMass Donahue Institute
Research and Evaluation Group
57
RTTT C&CR Evaluation Annual Report, September 2012
This district held three public forums to address the concerns of parents and guardians. Although the
district had originally intended to provide preferential admission to underrepresented students, in the end
the district decided to accept all students who applied.
Communication. Several districts reported challenges related to the quality, frequency, or clarity
of communication. Two said that it was difficult to communicate with parents/guardians, and another
reported difficulty communicating with their IHE partner. A third said that it was sometimes difficult to
communicate with district leadership. One interviewee said, “When you are recruiting students, informing
parents, talking to teachers…It is a tall order to make sure that people have a consistent conceptual
understanding of what is supposed to happen, how it is supposed to happen, when it is supposed to
happen, and what we hope to accomplish.”
Making connections between middle schools and high schools. Three sites are attempting to
integrate middle and high school learning experiences into a comprehensive STEM ECHS experience, but
none of these sites has clearly articulated a plan for bridging the two levels. One district reported that
there has been no significant contact between the middle and high schools related to the STEM ECHS
project.
Continuity. Two districts reported difficulty maintaining the continuity of their planning team.
One district reported that a change in school leadership had caused significant change to the planning
team, which disrupted work that was in progress. A second district had concerns about continuity,
because members were only assigned to the project for one year at a time. This interviewee said, “My big
question is whether people who are on the planning committee this year continue to be on the committee
next year. The degree to which that continuity is maintained or lost will determine, to a large degree, the
success of this project next year.”
District support. One interviewee reported challenges with securing district support, saying “We
need the collaboration and support of the whole district, and that has been difficult…The district asked
the school to put some structures in place before moving forward, and we are waiting for a final decision
to be made.”
Union concerns. Two districts reported challenges related to their teachers’ unions. The union in
one district said that the smaller class sizes in the STEM ECHS were not equitable, and the union in the
other district expressed concern that the agreement being negotiated with the IHE partner would reduce
employment opportunities for union members.
Student preparation. One interviewee said that students in their district had received little
exposure to science in elementary and middle school, which was raising concerns among school and
district personnel that students lacked ability and interest in STEM-related activities.
IHE partner. One district reported challenges in establishing an effective working relationship
with their IHE partner. The interviewee described a series of “stop and start” interactions, perceived
confusion by both parties, and a generally poor record of communication that resulted in what the school
perceives as a delay in establishing an articulation agreement in a timely fashion.
Suggestions for planning. Interviewees offered the following suggestions for others who are
planning to launch a STEM ECHS.
UMass Donahue Institute
Research and Evaluation Group
58
RTTT C&CR Evaluation Annual Report, September 2012

Schedule adequate planning time. Having a minimum of a 12-month planning period is a
realistic timeframe for planning the launch of a STEM ECHS. Schools and districts should be
aware that launching a STEM ECHS is difficult and that they should set aside sufficient time
and resources to address challenges as they arise.

Visit other STEM ECHS sites. Visit other sites and get information from people who have
already launched their programs. It is difficult to anticipate challenges if you do not talk to
people who have been through the process. These individuals can offer valuable insights.

Build interest and awareness of the program at the elementary and middle schools. Go to
elementary and middle schools to build interest in the STEM ECHS within your district. This
will help foster support for the program in the district and community.

Identify a point person for the project. There should be a “point person” for the STEM ECHS
project. Ideally, this person’s sole responsibility would be to coordinate the planning and
implementation of the program.

Build the program from the bottom up. Building a strong foundation of student success prior
to offering college courses is important. Start with a group of 9th graders, for example. Give
them the support they need. Have a variety of courses available, as well as a planned course
sequence.

Work closely with your IHE partner. Your IHE partners may have a lot to offer and play a
critical role in program development and implementation. IHEs may see the STEM ECHS as
a money-making proposition, and it is important that schools and IHE partners see the
collaboration as a “win–win” situation that is not confined to finances. The broader interests
of the school and of the IHE partner should be well-aligned. It may also be valuable to
explore your IHE partner’s capacity and interest in conducting research at your site related to
the development and success of the STEM ECHS project. Work with your IHE partner to
develop a clear articulation agreement. Plan a transitional experience for the students who
will be taking college courses with your IHE partner.

Identify concepts central to the work. Two concepts that everyone involved in STEM ECHS
should agree upon are that (a) the STEM ECHS should be available to and in the best interest
of all learners, not just high achievers; and (b) STEM comes first – the focus needs to be on
sustaining and strengthening the STEM pipeline.

Link the STEM ECHS to higher education and industry partners. Making meaningful
connections among school systems, higher education, and industry is key.

Combat a lack of support with clarity and honesty. Be transparent with your partners, and be
willing to answer questions and address challenges as they become apparent.
Implementation
For analytic purposes, only Marlborough is classified as an “implementing” site in this report. UMDI’s
understanding had been that two STEM ECHS sites (Marlborough and Quaboag) were operational and
conducted site visits to both of these sites. During our site visit, Quaboag representatives said that they
considered their site to be engaged in an “extended planning year” and did not consider themselves to be
an implementing site. One interviewee said, “We are still involved in the planning process. We had a soft
launch, a kind of pilot class running this fall, the electronics course…As we continue to develop the
program for a full launch this fall, we are still in the middle of program design and curriculum design.”
UMass Donahue Institute
Research and Evaluation Group
59
RTTT C&CR Evaluation Annual Report, September 2012
Therefore, the findings presented below are specifically about Marlborough, and are drawn from a site
visit to the school, interviews with program staff, a focus group with two middle school and two high
school teachers, and an interview with Marlborough’s IHE partner.
Implementation successes.
Continued support for students. One interviewee said that a primary success in launching their
STEM ECHS has been the staff’s ability to encourage and support all students in their efforts to complete
a curriculum of five honors-level courses. Because many students were included whose skill levels would
not typically lead to participating in an honors-level curriculum, teachers have had to structure their
curriculum to address gaps in students’ skills, content knowledge, and habits of mind. The interviewee
said that a key to the program’s success was the staff’s shared commitment to making every student
successful, including an intentional effort to address both academic and cultural components of the
student’s educational experience that can contribute to achievement.
All teachers said that the majority of STEM ECHS students were having a positive experience and were
“moving in the right direction.” Teachers also said that the school’s one-to-one laptop initiative had been
a success. One middle school teacher said, “Our 6th-grade students’ skills on computers far surpass 8thgrade students’ skills on the computers. Word processing, AutoSketch, research, the whole bit…I think it
has been wonderful.”
Curriculum development. An interviewee from the implementing site said that curriculum
development had been a significant strength, and that most students were enjoying their classes, engaging
with a challenging curriculum, and increasing their levels of achievement. The interviewee explained that
much of the STEM ECHS curriculum was based on the Massachusetts curriculum frameworks, saying
Massachusetts is well known for having a respected curriculum. Many people have done the
legwork to identify the concepts that need to be introduced to students at different points in time,
and we continue to improve upon that with things like the Common Core. So the conceptual work
– identifying the concepts – has been done for us in this state.
The interviewee said that, in order to build on the state curriculum frameworks, teachers continue to
research how they can integrate a “STEM philosophy” and “21st-century skills” into the students’ learning
experiences in every class. One successful strategy has been infusing activities that foster collaboration,
critical thinking, and creativity. These concepts have been integrated into the curriculum through projectbased learning experiences, such as a first-term project related to transportation safety.
Collaboration with IHE partner. One interviewee described FSU's partnership with Marlborough
as collaborative and said, “I really enjoy working with this group of people. It has been a really good
experience.” He noted that there have been regular communications and meetings throughout the year,
and that both FSU and Marlborough have hosted partnership meetings. At one such meeting, an FSU
faculty member met with representatives from Marlborough to discuss key aspects of college readiness.
The interviewee reported that this meeting enhanced FSU's understanding of the scope and structure of
the STEM ECHS and of the students being served. In addition, the FSU faculty member shared
information intended to help the Marlborough school personnel to prepare the STEM ECHS students for a
successful college experience.
UMass Donahue Institute
Research and Evaluation Group
60
RTTT C&CR Evaluation Annual Report, September 2012
FSU faculty members also worked with Marlborough personnel to develop online surveys that were
distributed to all STEM ECHS students and teachers in April. Students were asked why they applied to
the STEM ECHS, how the STEM ECHS was making a difference for them, what their expectations were,
and how their experiences differed from previous years of schooling. Teachers were asked how their
STEM ECHS experiences differed from previous years of teaching, and what impacts they felt the
program was having on student learning. These surveys were followed by focus groups led by FSU
faculty, and results were shared with Marlborough representatives in May. The interviewee said that
students most frequently cited the desire to earn college credit while in high school as a reason for
applying to the STEM ECHS. Both students and teachers reported that connectedness, collaboration,
project-based and hands-on learning, and high expectations were contributing to the success of the
program. FSU and Marlborough personnel are discussing how they could conduct longitudinal research to
track the progress of the STEM ECHS students for several years.
Factors facilitating success. Teachers said that one key to program success has been the district’s
commitment to providing teachers with time for professional collaboration. This time has been used to
clarify working relationships, develop curriculum, and discuss plans for supporting individual students.
One teacher said,
Any success we have had has been the direct result of us being able to collaborate as a team. At
the high school we have a period every single day that we meet for 48 minutes. Being able to talk
to the whole group each day, the six of us, has really helped so we all know what is going on. Just
the communication. We use email a lot, but just seeing each other in person has really helped out.
All high school teachers, not just those participating in the STEM ECHS, have one period per day to
collaborate. STEM ECHS teachers meet as a team each day during that time, rather than participating in
departmental meetings. In contrast, at the middle school only STEM ECHS teachers were given a
collaboration period, as a duty period, for a portion of the year. That period was later taken away after the
teachers union filed a grievance citing unfair distribution of labor, arguing that some teachers had been
assigned more traditional duties than others. Despite this challenge, a middle school teacher said, “At the
middle school, I think that coming together as a team has been a success. And I do think that we have
come together as a team…When we had that dedicated 45 minutes of collaboration, that was key in
helping us to continue to move towards the shared projects that we were working on.” Teachers at the
middle school now meet after school one day per week.
Interviewees reported that strong support from school and district administrators had been essential to the
success of the STEM ECHS. Teachers reported that administrators had made strong efforts to support the
work being done in their classrooms.
Implementation challenges.
Staff selection. The implementing site said that staff members at the middle school were assigned
to the STEM ECHS by their building administration and were not given the opportunity to opt out of the
program. One interviewee said that this had caused some tension with middle school teachers and that the
situation is now being addressed through clear communication with these teachers, who will be allowed to
opt out in the future. In addition, extensive efforts have been made to increase the enthusiasm of these
discontented middle school STEM ECHS teachers about the curriculum they are teaching.
UMass Donahue Institute
Research and Evaluation Group
61
RTTT C&CR Evaluation Annual Report, September 2012
Tensions between STEM ECHS staff and other staff. One interviewee said that some teachers at
the high school are resentful of teachers working with the STEM ECHS project, because the STEM
ECHS teachers benefit from increased resources (e.g., increased planning and collaboration time,
professional development) and decreased class sizes. The interviewee said that there is some sense in the
middle and high schools that STEM ECHS teachers are viewed as privileged. Although no formal
grievance has been filed by the local teachers union, the union is monitoring the situation. The
interviewee reported that, in response to this challenge, the next STEM ECHS cohort would increase to
100 students from its current level of 72 students.
Teachers said that they have been challenged by colleagues who doubt their ability to deliver an honorslevel curriculum to students with a broad range of academic abilities. One teacher said, “People don’t
understand what we do and who our kids are. We teach all honors classes to students who would
otherwise not be exposed to an honors curriculum.” Some teachers reported having to defend their work
when confronted by colleagues who perceive inequity in the STEM ECHS program's access to common
planning time and resources such as computers.
Technology. Teachers reported that they had faced many challenges with technology, primarily
related to the one-to-one laptop initiative. The initiative, which provided all STEM ECHS students with a
laptop computer for educational use, was not fully implemented until January. Teachers said that it was
difficult and time-consuming to integrate the use of laptops into the curriculum at that point in the year. It
took time for both teachers and students to become accustomed to having full access to the technology.
One teacher said, “It has been just as much of learning experience for the teachers as for the students.”
Teachers also said that there have been a significant number of reported issues with damaged computers
(e.g., cracked screens), and that the computer hardware was less durable than they had anticipated.
Despite these challenges, teachers agreed that they now view laptops as an important component of the
students’ classroom experience, and an essential tool for learning.
Teachers from both the middle school and high school reported that the availability of Wi-Fi internet
access had been somewhat sporadic, and that it was difficult to conduct technology-dependent lessons
with an unreliable internet connection. They said that retrofitting their older school buildings with Wi-Fi
technology proved to be challenging.
Teachers also said that some students lacked the maturity and sense of responsibility required to work in a
digital learning environment. Some students used their laptops for non-class-related activities (e.g., using
Skype during class), and these activities distracted other students and reduced the quality of their learning
experiences. Teachers also said that there were always one or two students who forgot to bring their
laptops to class, and that this also disrupted classroom activities.
Project-based learning. Teachers said that balancing the STEM program’s project-based learning
activities with more traditional approaches has been challenging. They said that they spent more time on
project-based learning activities at the beginning of the year, and less time on these activities toward the
end of the year. Teachers said that they enjoyed the project-based learning activities and that they were
still learning how best to leverage this approach to maximize student learning. Some teachers said that
they reduced the amount of classroom time devoted to project-based learning during the second half of
the year, so that they could cover more material during “crunch time.” Teachers said that it was a
challenge to cover all key content standards during the year, and one teacher said that it was “pretty much
impossible” to cover all of the relevant standards using a project-based curriculum, because there are “no
perfect projects.” Middle school teachers also noted that a lot of “core teaching and learning” still needed
UMass Donahue Institute
Research and Evaluation Group
62
RTTT C&CR Evaluation Annual Report, September 2012
to take place at the 6th-grade level, and that in some cases they preferred to use more traditional
pedagogical approaches.
Varied levels of student ability. Teachers said that it was challenging to implement the STEM
program’s honors-level curricula with students of a wide range of academic ability and skill levels. High
school teachers reported making continual efforts to support struggling students. Many students were
encouraged to, and often did, stay after school to receive extra support. English language learners, in
particular, had struggled with the STEM ECHS curriculum, and the system for supporting these students
was still being modified. A representative from FSU who expressed that preparing a heterogeneous cohort
of high school juniors and seniors for successful participation in college-level courses was going to be a
significant challenge said,
I think the challenge is in the philosophy of wanting as many kids as possible to already have
college credit when they enter college. To me, that is very different than being college ready.
That is going to be a challenge for some of these kids. We have lots of FSU students who are
freshmen and sophomores…and even they can’t succeed. With the STEM ECHS, we are talking
about an even younger group – juniors and seniors in high school. We have to make sure that they
are ready for this experience. Intellectually, psychologically, and socially ready. That is a huge
challenge. I don’t know if enough has gone into this kind of thing...I understand that the objective
is to get them all college credits by the time they finish high school, but there is a reality, too. I
think about that.
This FSU representative said that next year will be pivotal for sophomore STEM ECHS students, because
they will be preparing for participation in college courses as juniors. The representative said that these
students need to have a clear understanding of what it means to be prepared for college so that they can
have a positive experience at FSU.
Changes in teaching practices.
Teachers reported that they had made significant changes in their teaching practices, spending
less time lecturing and more time with project-based, lab-based, and hands-on activities. All students are
required to complete carefully planned multi-disciplinary group projects at the end of each quarter, and
one teacher said that team members were constantly thinking about ways to foster such cross-curricular
connections. All teachers agreed that teaching in the STEM ECHS felt like their first year of teaching in
the sense that there was a “huge growth curve.” Other instructional and support practices mentioned by
teachers included:

Collaborating with colleagues to schedule “project days” when students can work together
across multiple class periods to complete a group project.

Learning about best practices for facilitating, supporting, and grading group work (e.g.,
students working in a group may not receive the same grade).

Covering similar topics at the same time across multiple content areas so that students can
make connections across the curriculum.

Conducting regular “intervention team meetings” to discuss individual student progress and
strategies for supporting student achievement.
Suggestions for implementation.
UMass Donahue Institute
Research and Evaluation Group
63
RTTT C&CR Evaluation Annual Report, September 2012
Teachers made the following recommendations for others trying to implement a STEM ECHS:

Common planning time is essential for success. Structure the time with a leader, goals, and an
agenda, but be flexible.

Carefully consider the content and structure of public and parent communications. Invite
community members and parents to view student accomplishments.

Be willing to change (e.g., they revised grading rubrics to reflect individual student effort).

Solicit student feedback about the STEM ECHS.

Publicly recognize student achievements (e.g., annual STEMy awards, student speakers).

Allow adequate planning time for all components of the program (e.g., content, pedagogy,
administrative structures, collaboration).
Technical Assistance and Supports
All interviewees reported that ESE had consistently provided effective, professional, and timely
assistance. District representatives said that ESE staff were accessible, helpful, and accommodating. Five
of six districts said that ESE was viewed as a valued partner in their work. The one district that did not
make this assertion said that their interactions with ESE related to this project had been limited, but of an
appropriate level for their situation. Districts noted that ESE staff were aware of, and sensitive to, the
contexts in which the districts were working.
Jobs For the Future has provided technical assistance to the six Goal 4E sites, working in consultation
with ESE. The technical assistance has included network meetings, individual technical assistance, and
“just-in-time” problem solving. Network meetings have included professional development and technical
assistance activities that were intended to facilitate the development of the STEM Early College High
Schools. While technical assistance efforts have been tailored to meet each site’s needs, JFF has also
provided a planning framework with benchmarks and timelines intended to assist sites in determining
appropriate timelines and criteria for making key development and implementation decisions. JFF has
also provided each site with just-in-time phone and email access to technical support. JFF has provided
ESE with multiple written updates for each site, based on site visits and technical assistance
conversations. JFF also reports monitoring progress at each site to ascertain if key milestones are met, and
has hired an outside consultant who has overseen much of the technical assistance.
District representatives were asked to provide feedback on their interactions with JFF. Interviewees from
five of six districts said that JFF was viewed as a valued partner that was helping to move their project
forward. Districts said that periodic check-ins, combined with specific suggestions for next steps, have
been most helpful. The following comment was typical across these five sites:
Phone calls and conference calls with JFF and our planning committee have always been very
helpful. Our interactions with JFF at the recent technical assistance meeting helped us move
forward. They provided helpful suggestions for us, such as related to our communication
challenges. JFF has helped to pull the group together, including our IHE partners. I would like to
continue to work with JFF. They have really listened to where we are at and provided suggestions
for helping us move forward. Just working with someone who has gone through this process
before, and can anticipate our challenges before we can, is very helpful.
UMass Donahue Institute
Research and Evaluation Group
64
RTTT C&CR Evaluation Annual Report, September 2012
Two of these five sites said that additional support from JFF would be helpful but did not know what
additional services or supports JFF could offer. One interviewee suggested that it would be helpful if JFF
could identify, describe, and provide training on a suggested instructional approach or set of instructional
best practices for STEM ECHS sites, particularly those known to increase the success of at-risk students.
One of the six interviewees did not consider JFF to be a supportive partner, but sharing specific
comments here risks the interviewee’s confidentiality. After the interview, UMDI followed up to request
permission to share specific concerns with ESE, in accordance with the confidentiality agreement. The
interviewee responded that, because the topic and context were sensitive, he preferred and intended to
address concerns with ESE directly. He later confirmed that he had indeed spoken with ESE about it.
The teachers who participated in the focus group were not aware of technical assistance that had been
provided by JFF or ESE. They said that their only exposure to JFF had been when the vendor conducted
classroom observations. One teacher said, “I am sure there is stuff that we haven’t seen…but it would be
hard to say what benefit we got from these visits.” Teachers reported that they had made no requests for
assistance from JFF or ESE.
Similarly, a JFF representative expressed a desire to increase the scope of their work with the sites:
I just wish [the resources were available to] do more intensive work. This is our first venture of
this kind in Massachusetts along with the intensive work we have been doing [with other
clients]…if you don’t get into classrooms and really work with teachers it’s very hard to feel like
you are going to have a major impact and change the students’ outcomes…Given our resources,
we could really only do very limited professional development.
When asked how they would like to expand their work, a JFF representative stated that they would
increase the amount of coaching at each site and increase connections between the sites and the STEM
business community. She said,
I think that with these schools, if we were going to really fast track their progress, we would
really need much more coaching on a much more regular basis. With additional support, we
would do much more intensive professional development than we have been able to do in the
past. If we were doing this project in an intensive way, we would be meeting with small and large
employers consistently. We’d bring them into the schools and try to create some teacher
externships, because the first thing that needs to happen is that teachers need to understand the
contemporary STEM workplace, and most of them actually have no way of knowing much about
it. There are many things I would do to create urgency, visibility, and a sense of the importance of
the project.
The JFF representative also acknowledged current budget limitations and realities, and noted that these
limitations were well understood. She said,
Unfortunately it comes down to how much funding we have and how much time, and I think
everybody understood at the outset [what the constraints were]…I really do think that with some
exemplary professional development in some consistent way, we could get some classrooms to
really be clicking the way we would like them too.
UMass Donahue Institute
Research and Evaluation Group
65
RTTT C&CR Evaluation Annual Report, September 2012
Interviewees across all six sites made few specific requests for technical support. Two specific requests
included the following:
Members of this team have never done this sort of thing before. We are looking for expertise in
establishing a program so that we don’t spend our time going in wrong directions. There are
documents that provide timelines as to what tasks are supposed to be accomplished, but there are
limits to how helpful those documents are. Having a person to help guide or coach us through
discussions, not to take over, would be helpful.
Your reputation [as a program and as a school] is often defined by the local media, and they don’t
always put the message out there that you would prefer. [Media relations] is an issue that we
would like to have some assistance around, and hear what the other schools are doing to market
themselves and get their message out.
Several sites also expressed interest in additional structured time to meet with personnel from the other
STEM ECHS programs to collaborate on specific topics such as scheduling and sustainability. One
interviewee said,
Presentations and meeting times with other projects are very helpful. It is helpful to know what
others are doing, and to share ideas and best practices. It is important to keep those conversations
going. Perhaps not formal presentations, but sharing out about what we are doing and where we
see our programs going. That is important because this is new territory. Meeting quarterly is
good. That is how we learn.
One interviewee said that he would like JFF to provide additional technical assistance on how to better
prepare students, who have a broad range of skills and abilities, to participate in college courses, as well
as address the issues that high school students and administrators face when the students take courses on
college campuses. He said,
Kids learn at different rates, and this hasn’t really been discussed. I have no problem with
focusing on the administration of the program [at technical assistance meetings]. They have
presented some best practices, but I would really like to see some more best practices from
around the country in how these [STEM ECHS] partnerships are dealt with…I think one of the
things that JFF can do is talk about the different issues that students and administrators face when
these types of programs are at a college campus. I think it is really easy to do this at a high school
level. I think it is much harder to do this at a college campus.
Sustainability
All interviewees (administrators, teachers, IHE partners, technical assistance vendors, and ESE) indicated
that finding the financial resources to sustain the STEM ECHS programs is a significant challenge.
Districts acknowledged that, without additional funding, their STEM ECHS programs would be at risk.
Several districts said that the work of developing and launching a STEM ECHS was cost-intensive, and
that staff development, curriculum development, planning time, and supplies were all major expenses
related to the planning process. These districts also said that funding to support student participation in
college courses was tenuous, and that cost structures and articulation agreements with IHE partners were
still being negotiated. One interviewee said,
UMass Donahue Institute
Research and Evaluation Group
66
RTTT C&CR Evaluation Annual Report, September 2012
Sustainability is the huge, giant, 800-pound gorilla in the room that I think about all of the time.”
Another said, “Funding college courses will come down to the money that is available for this.
Ultimately, at the end of this there is going to be a need for a sustainability plan…The cost pieces
include transportation, supplies and books, and the actual [college] credits. That is going to be the
critical piece. Beyond the initial years of the grant, how do we make this sustainable?” A third
interviewee seemed pessimistic about long-term prospects, saying “After a few years the money
is going to go away. I think peoples’ ideas of how this program can be maintained over time will
be deflated.
Marlborough’s IHE partner predicted sustainability challenges, noting that the program's viability was
secure during, but not beyond, the RTTT grant period. He said,
Our tuition is basically $1,000 per course right now, and the grant funds that Marlborough has put
aside for college courses will not cover that entire cost ...We will provide the discount that is
necessary for this to happen. This is an important part of our mission in public education. I think
it is critical that there is cost sharing on this type of thing. But I don’t know how it is going to
play itself out after the Race to the Top grant is over. We are going to have to look for funding
from other sources for sure.
Although all districts identified sustainability as a significant concern, none have established a clear plan
for addressing this issue. Some said that they were beginning to discuss strategies for securing future
funding, and several said that they anticipated committing increased levels of attention to the issue of
sustainability over time. Ideas offered by district representatives to promote sustainability included:
altering program recruitment practices to attract students from other districts to participate; hosting fundraising events; exploring cost reduction strategies (e.g., grants, rate reductions, purchasing courses) with
IHE partners; securing support from business and industry partners, and other donors; and applying for
additional grants from public and private funding agencies.
A representative of Marlborough’s IHE partner reported that they plan to explore several funding options,
including recruiting community members to sponsor one or more students; contracting with the
Marlborough Public Schools to provide a fixed number of courses for a fixed number of students at a predetermined rate; and having students and their families pay part of the FSU tuition, with reduced rates
offered to students from low-income households. The interviewee reported that the STEM ECHS
partnership has been designated as a continuing education activity by FSU administrators, which provided
some financial flexibility because of unique aspects of the continuing education department's financial
structures.
An interviewee from Jobs For the Future said that although sustainability challenges were substantial,
securing and maintaining a strong commitment from stakeholders at each site was critical to their longterm viability. She said,
These schools have so little funding [from the state] to begin with, that if the small dollars go
away it’s not going to mean they are going to lose a lot of staff…What will be a problem is if
there are cuts across the board that impact the schools in general, since these STEM schools are
making additional demands on people without providing with much by way of additional
resources.
UMass Donahue Institute
Research and Evaluation Group
67
RTTT C&CR Evaluation Annual Report, September 2012
This interviewee also said that, over time, the state could consider adopting policies that would benefit or
expedite the development of STEM ECHSs and other programs providing high school students with early
college experiences. The ESE program manager for the STEM ECHS program provided similar thoughts
on sustainability, saying that sustainability “comes down to the commitment of the people at the
individual sites…We don’t have the needed state policies…there are inadequate connections between
K12 and [the Department of Higher Education].”
Baseline Graduation and Dropout Rates
The following three pages each contain a table paired with a figure that present the following baseline
graduation and dropout rate data for the schools in which the STEM ECHS sites are hosted5:
1. Average 4- and 5-year cohort graduation rates from 2007-08 to 2010-11.
2. Average annual dropout rates for 2007-08 to 2010-11.
3. Average 4- and 5-year cohort dropout rates 2007-08 to 2010-11.
The companion document, Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report,
presents tables and figures that provide a more complete summary of these same baseline graduation and
dropout data. The Technical Supplement contains tables and figures that present both average and
individual graduation and dropout rates for districts and schools implementing a RTTT-funded STEM
ECHS. Future evaluation reports will use these baseline data to describe trends in graduation and dropout
rates in STEM ECHSs and districts. The data necessary to disaggregate baseline graduation and dropout
rates by subgroup (e.g., gender, race/ethnicity) were not available in time for this report, but will be
included in future analyses. Future reports will also describe trends in indicators of student engagement
and academic achievement in STEM ECHS schools and districts.
5
One middle school site is not currently associated with a high school. For MAVA, Southeastern Regional Vocational-Technical
High School was included, due to its role as the applicant district.
UMass Donahue Institute
Research and Evaluation Group
68
RTTT C&CR Evaluation Annual Report, September 2012
STEM ECHS: 4- & 5-Year Graduation Rates
4- & 5-Year Graduation Rates for STEM ECHS Schools, 2007-08 to 2010-11
Year
Number of
Students
4-Year Graduation
Rate
5-Year Graduation
Rate
Difference
Between Rates
2007-08
1,243
77.6%
81.9%
4.3%
2008-09
1,236
75.3%
78.6%
3.3%
2009-10
1,253
76.1%
79.5%
3.4%
2010-11
1,179
75.4%
NA
NA
4- & 5-Year Graduation Rates for STEM ECHS Schools, 2007-08 to 2010-11
100%
95%
Graduation Rate
90%
85%
4 Year Rates
80%
5 Year Rates
75%
70%
65%
60%
55%
50%
2007-08
UMass Donahue Institute
Research and Evaluation Group
2008-09
2009-10
School Year
2010-11
69
RTTT C&CR Evaluation Annual Report, September 2012
STEM ECHS: Annual Dropout Rates
Annual Dropout Rates for STEM ECHS Schools, 2007-08 to 2010-11
Year
Total HS Enrollment
Number of Dropouts
Annual Dropout Rate
2007-08
4,806
167
2.4%
2008-09
4,675
183
3.9%
2009-10
4,659
168
3.6%
2010-11
4,621
182
3.9%
Annual Dropout Rates for STEM ECHS Schools, 2007-08 to 2010-11
10%
9%
Annual Dropout Rate
8%
7%
6%
5%
4%
3.9%
3%
2%
3.6%
3.9%
2.4%
1%
0%
2007-08
2008-09
2009-10
2010-11
School Year
UMass Donahue Institute
Research and Evaluation Group
70
RTTT C&CR Evaluation Annual Report, September 2012
STEM ECHS: 4- & 5-Year Cohort Dropout Rates
4- & 5-Year Cohort Dropout Rates for STEM ECHS Schools, 2007-08 to 2010-11
Number of
Students
Year
4-Year Cohort
Dropout Rate
5-Year Cohort
Dropout Rate
Difference
Between Rates
2007-08
1,243
10.9%
11.0%
0.1%
2008-09
1,236
11.9%
12.6%
0.7%
2009-10
1,253
10.5%
10.9%
0.4%
2010-11
1,179
10.0%
N/A
N/A
4- & 5-Year Cohort Dropout Rates for STEM ECHS Schools, 2007-08 to 2010-11
30%
Cohort Dropout Rate
25%
20%
4-Year Rate
5-Year Rate
15%
10%
5%
0%
2007-08
UMass Donahue Institute
Research and Evaluation Group
2008-09
2009-10
School Year
2010-11
71
RTTT C&CR Evaluation Annual Report, September 2012
Student Participation
The table below summarizes student participation at STEM ECHS sites during the 2011-12 school year.
These figures reflect the activities of each school as described in this report, with full implementation by
Marlborough and initial activities by Quaboag, MAVA, and Worcester.
STEM ECHS – Summary of Student Participation – 2011-12
District
Number of Students Impacted
Marlborough Public Schools
189
Quaboag Innovation STEM early college
14
high school (Quaboag Public Schools)
Southeastern Regional Vocational Technical
(Massachusetts Association of Vocational
28
Administrators - MAVA)
Randolph High School
0
(Randolph Public Schools)
North High School
25
(Worcester Public Schools)
Dearborn Middle School
0
(Boston Public Schools)
Total
256
Conclusions
The six Goal 4E STEM ECHS sites are in various phases of project development and implementation.
Two sites enrolled students in STEM ECHS activities during the 2011-12 school year, three sites plan to
start during the 2012-13 school year, and the last site’s STEM ECHS work has been incorporated into a
broader school redesign plan.
Interviewees from five districts described assembling a planning team as a successful aspect of their
planning process. Additional factors that contributed to the success of the planning teams were including
representatives from diverse stakeholder groups and consistent collaboration across members.
Respondents shared that holding planning meetings on a regular basis was difficult, because most team
members had many other substantial responsibilities and commitments. Interviewees from all districts
cited support from district leadership as essential to the planning process and establishing strong working
relationships with their IHE partner.
Informants reported many challenges that arose during the planning phase, including budget management,
curriculum development, staffing, engaging the community, and working with teachers’ unions. Several
districts also reported challenges related to the quality, frequency, or clarity of communication across
stakeholders.
Marlborough is the only site that began full implementation during Year 2, with 189 students in grades 6
and 9. One Marlborough interviewee said that a primary success has been the staff’s ability to encourage
and support all students in their efforts to complete a curriculum of five honors-level courses. All teachers
said that the majority of STEM ECHS students were having a positive experience and “moving in the
right direction.” The interviewees expressed that the following factors have contributed to their successful
UMass Donahue Institute
Research and Evaluation Group
72
RTTT C&CR Evaluation Annual Report, September 2012
implementation to date: strong curriculum development, a collaborative partnership with the IHE, and the
district’s commitment to providing teachers with time for professional collaboration.
Many challenges were reported by Marlborough teachers. One high school teacher reported that some
colleagues working within the main high school resent the increased resources (e.g., structured planning
and collaboration time, professional development) and decreased class sizes enjoyed by the STEM ECHS
teachers. A middle school teacher reported that building administrators assigned personnel to the STEM
ECHS without giving them the opportunity to opt out of the program. STEM ECHS teachers also reported
that they had faced many challenges with technology, primarily related to the one-to-one laptop initiative,
which was not fully implemented until five months into the school year. Teachers said that it was difficult
and time-consuming to integrate the use of laptops into the curriculum at that point in the year.
All districts reported that ESE had consistently provided effective, professional, and timely assistance.
Districts noted that ESE personnel were aware of, and sensitive to, the contexts in which the districts were
working. Jobs For the Future, working in consultation with ESE, provided technical assistance to the Goal
4E STEM ECHS sites. The support provided to the sites has individual technical assistance, meetings
with all six schools, and “just-in-time” problem solving. The majority of districts said that JFF was
viewed as a valued partner that was helping to move their programs forward.
All interviewees (administrators, teachers, IHE partners, technical assistance vendors, and ESE) reported
that finding the financial resources to sustain these STEM ECHS sites will be a significant challenge. No
districts have established a clear plan for addressing this issue, although several said that they anticipated
committing increased levels of attention to sustainability issues over time. One interviewee also suggested
that, over time, the state could consider adopting policies that would benefit or expedite the development
of STEM ECHSs and other programs providing high school students with early college experiences.
Strategic Considerations

Initial lessons from the early-implementing STEM ECHS schools could benefit the
other STEM ECHS schools and inform technical assistance from JFF and ESE. These
lessons include the importance of budgeting adequate time for planning and collaboration,
integrating needed technology, communicating with key constituencies, and preparing high
school students for the challenges of college-level courses.

Continuity of leadership and succession planning appear important to success of the
STEM sites. Multiple sites reported that lack of continuity in their planning team
membership, as well as turnover in school and district leadership positions, had disrupted
their planning and implementation processes. JFF also identified lack of continuity as a
significant threat to success and sustainability. Continuity should be encouraged whenever
possible, and when personnel transitions are anticipated, succession planning should be
undertaken to forestall problems.

Securing the financial resources to sustain STEM ECHS partnerships is a primary
concern of all STEM ECHS sites. Unsurprisingly, given the more pressing concerns of
launching a new school, none of the six schools have developed a plan for addressing this
challenge. JFF and ESE may therefore wish to make sustainability a focus for technical
assistance.

Changes to state and local policies may facilitate STEM ECHS development. Some
informants indicated that state or local policies exist that delay or impinge upon the
development of STEM ECHSs. While this was not formally evaluated in Year 2, it will be
specifically addressed in Year 3 evaluation activities.
UMass Donahue Institute
Research and Evaluation Group
73
RTTT C&CR Evaluation Annual Report, September 2012

Increased connections with the broader STEM community could benefit ECHS
programs. The technical assistance vendor noted that limited program funds have precluded
some of the STEM connections outside the school that are often beneficial to STEM ECHS
activities. To the extent possible with available resources, schools should increase efforts to
connect with businesses, universities, and other organizations with STEM expertise.
UMass Donahue Institute
Research and Evaluation Group
74
RTTT C&CR Evaluation Annual Report, September 2012
MassCore Policy and Implementation
Background
The Massachusetts High School Program of Studies (MassCore) recommends a set of courses and other
learning opportunities which Massachusetts students should complete before graduating from high
school, in order to arrive at college or the workplace well-prepared and without need for remedial
coursework. The 155 districts that selected the RTTT college and career readiness goal committed to
implementing strategies to increase the percentage of their students who complete the MassCore
curriculum.
The state’s RTTT goal is to increase the statewide MassCore completion rate from its baseline of 70% of
the Class of 2010 graduates to 85% of the Class of 2014 graduates. The state has created a goal for each
district, using a formula based on the district’s reported 2010 MassCore completion rate (calculated from
the MassCore element of the state’s SIMS database), the district’s number of 2010 graduates, and the total
number of graduates statewide needed to bridge the gap between the 70% baseline and the 85% goal.
Each district was also expected to determine areas in which courses or supports needed to be expanded in
order to meet the 2014 targets, and to create and implement a plan to improve the accuracy of their
reporting of MassCore completion levels. The evaluation questions specific to the MassCore evaluation
are listed below.
Process Evaluation Questions
P1. What steps have districts taken to facilitate MassCore completion by students? What are the
major challenges to and facilitators of district implementation and student completion of a
rigorous course of study in general, and the MassCore curriculum in particular? What
midcourse corrections and attempts to overcome challenges have been undertaken? What
additional steps are necessary?
P2. To what extent has ESE taken the steps to increase MassCore implementation described in
their RTTT grant application? What are the major challenges to and facilitators of district
support encountered by ESE? How have challenges been overcome and midcourse
corrections undertaken? What additional steps are necessary?
P3. How do district stakeholders explain the relevance of MassCore implementation and
completion to their efforts at promoting student college and career readiness, as well as the
effectiveness of their efforts toward fostering greater MassCore implementation and
completion?
P4. What infrastructure, systems, and processes were put in place to aid sustainability of
improvements in MassCore implementation and completion during and beyond the grant
period? What supports could ESE or other partners provide to districts to increase levels of
MassCore implementation and completion? What are the greatest challenges and barriers to
creating sustainability?
UMass Donahue Institute
Research and Evaluation Group
75
RTTT C&CR Evaluation Annual Report, September 2012
Outcome Evaluation Questions
O1. To what extent are districts increasing MassCore implementation, and to what extent are
students achieving increased MassCore completion? What are the gaps in student completion
of coursework required for MassCore completion?
O2. At the school and district levels, do observed changes in MassCore completion differ across
student characteristics such as gender, race/ethnicity, free/reduced lunch status, ELL status,
and special education status? Is there evidence that gaps are narrowing? Are district resources
related to increased MassCore program completion reaching students who are at the greatest
risk?
O3. To what extent are changes in student MassCore completion attributable to state and district
efforts in service of this goal versus other RTTT program activities and/or measurable
contextual variables?
O4. What are the major differences in MassCore implementation, completion, and contextual
variables across schools and districts whose levels of improvement on student outcomes
differ substantially?
O5. What is the relationship between level of MassCore implementation and student achievement
of MassCore completion and other targeted student outcomes such as increased graduation
rates and reduced dropout rates?
Methods
The Year 2 evaluation of the MassCore initiative including the following activities and data sources:

Interview with district administrators. Five district administrators were interviewed by phone
about their district's actions and plans for implementing the MassCore program of studies
(see interview protocol in Appendix C1). The interview’s purpose, in addition to providing an
in-depth understanding of MassCore implementation in a small number of districts, was to
help generate questions for an online survey that was subsequently sent to all MassCore
districts. The sample was drawn from a list of 20 potential interviewees from ESE's list of
Race to the Top contacts, provided by ESE’s MassCore program manager, who emphasized
urban districts because he said they have the greatest need for increased MassCore
completion rates. Nine districts were contacted, in some cases multiple times, and five
responded and were interviewed. Contacts were told that UMDI was seeking "a district
administrator who has in-depth knowledge of the district's positions, actions, and plans
related to MassCore," and they were invited to suggest an alternative interviewee as
appropriate, which they did in two cases.

Survey of district administrators. A survey was conducted of RTTT administrators from the
155 districts that selected RTTT Goal 4A (MassCore) (see protocol in Appendix C2). The
survey questions were informed by responses to the MassCore district administrator survey.
The survey sample was 142 of the 155 administrators, because 4 of the districts did not serve
high school students, 4 of the administrators had opted out of receiving surveys from the
SurveyMonkey website, and 5 of the administrators served two districts each.

Interviews with ESE program manager and project director. Three in-person interviews were
conducted, as well as frequent, informal email and phone communications throughout the
year.
UMass Donahue Institute
Research and Evaluation Group
76
RTTT C&CR Evaluation Annual Report, September 2012

ESE documents and databases. The MassCore quantitative data presented below and in the
Technical Supplement to RTTT C&CR 2012 Evaluation Annual Report were developed using
ESE’s 4-year and 5-year graduation cohort lists as well as SIMS data from October and June
of 2008-2011. ESE documents including the RTTT Goals Workbook, spreadsheet of RTTT
participating districts, and graduation and annual dropout rate information reported by ESE,
were also used.
Findings
Findings are presented in relation to the process and outcome evaluation questions. MassCore questions
P1 and P3 are addressed in the sections on commitment to MassCore, challenges to implementing
MassCore, and actions taken to implement MassCore. Questions P2 and P4 are primarily addressed in the
sections on support and sustainability. The outcome questions are then addressed with the presentation of
baseline graduation and dropout rates for MassCore schools, baseline MassCore completion rates, and a
discussion of how outcome data for subsequent project years will be utilized.
The discussion below draws primarily from the MassCore district administrator interviews and surveys,
interviews with ESE staff, and review and analysis of state documents and datasets. For the district
administrator interviews, the five respondents' districts ranged in enrollment from 5,000 to 14,000
students, with graduation rates ranging from 50% to 74% (mean = 64%) and with rates of free and
reduced lunch ranging from 55% to 83% (mean = 71%). Respondent roles within their districts included
assistant superintendent (2), director of grant-funded programs, director of student services, and RTTT
program specialist. For the district administrator survey, the response rate was 74% (N=106), and the
primary district roles of respondents were assistant superintendent (30%), superintendent (23%),
curriculum coordinator (11%), RTTT coordinator (10%), and "other" (27%). The "other" category
consisted of program, services, curriculum, and charter school directors (N=10); principals (N=2); and
grants coordinators (N=2).
Commitment to MassCore
In order to understand MassCore in relation to RTTT, it is essential to recognize that ESE's intention, as
expressed in their application for RTTT funding, was that "Massachusetts will make MassCore –
currently a recommended program of high school studies – the de facto curriculum for the
Commonwealth." As explained by ESE's MassCore program manager, this is still the state's intention, but
it has not yet been realized. As a result, although many districts believe that MassCore completion is a
worthy goal, and RTTT Goal 4 districts are spending funds on many college and career readiness
initiatives, districts have no clear accountability to the state for achieving progress toward greater
MassCore completion.
All five administrators who participated in the MassCore interview acknowledged that their district
recognized the advantages of MassCore in terms of college and career readiness, and all supported
MassCore to differing extents, mostly depending on the extent and types of challenges that they
anticipated encountering in attempting to implement it. These concerns are explored in depth in the
sections below.
Two districts portrayed themselves as strongly committed to and actively pursuing MassCore, although
possibly in modified versions that acknowledged the limitations of their school infrastructure and
capacity. One district said that the district administration strongly favors vigorous movement toward
UMass Donahue Institute
Research and Evaluation Group
77
RTTT C&CR Evaluation Annual Report, September 2012
increasing MassCore completion rates, but that the school committee is opposed and “doesn’t even want
to hear about it” because it would mean that far too many students would fail to graduate. Two additional
districts saw themselves as pursuing MassCore partially but anticipating major and potentially intractable
challenges. The strong statement of commitment was:
We see major benefits. Making a significant commitment of our Race to the Top dollars in
moving ourselves towards a state of readiness for MassCore and promoting that. We see it as
essential for college readiness, issues of rigor and readiness amongst students who begin college,
and we also see it as a standard that will increase our gradation rate.
Perhaps the least committed statement was “MassCore is kind of on the side burner right now because of
the other things that are going on in the system…I can’t even get [school] to take MassCore seriously at
this point.”
All districts reported that steps being taken toward higher rates of MassCore completion are being
overseen by high-level district administrators and school administrators (e.g., principals, guidance
directors, department heads, curriculum coordinators). Some also reported participation by high school
and middle school faculty as well as working groups that focus on college and career readiness.
Respondents to the administrator survey were asked if MassCore completion is a graduation requirement
in their district, and out of 84 respondents, 50% said yes, 46% said no, and 4% didn't know. As shown in
the table below, more than 80% of survey respondents agreed that increasing the rate of student MassCore
completion was a high priority and a realistic goal for almost all of their students. Respondents also
indicated that they were aware of RTTT Goals Workbook targets, felt a strong level of accountability for
achieving them, and regarded them as a realistic goal for their district. In addition, 72% agreed that their
district is using RTTT funds to take actions that they believe will lead to increased rates of MassCore
completion, and 80% agreed that their district is using non-RTTT funds for this purpose.
UMass Donahue Institute
Research and Evaluation Group
78
RTTT C&CR Evaluation Annual Report, September 2012
MassCore Goals, Knowledge, Attitudes, and Use of Funds
Please indicate your district's
level of agreement with the
following statements.
N
Strongly
Agree
(%)
Agree
(%)
Neutral
(%)
Disagree
(%)
Strongly
Disagree
(%)
Don’t
Know
(N)
Increasing the rate of student
MassCore completion is a high
priority for my district.
86
55
36
6
2
1
0
My district is using RTTT funds
to take actions that we believe
will lead to increased rates of
MassCore completion.
85
38
34
10
10
8
0
My district is devoting nonRTTT resources to taking
actions that we believe will lead
to increased rates of MassCore
completion.
85
43
37
11
7
2
0
My district is aware of the
MassCore targets for our district
in the RTTT Goals Workbook.
85
46
43
6
4
1
1
My district feels a strong level
of accountability for achieving
the MassCore targets for our
district in the RTTT Goals
Workbook.
85
38
45
13
2
2
1
Achieving the MassCore targets
in the RTTT Goals Workbook is 85
a realistic goal for my district.
40
43
12
5
0
1
MassCore completion is a
realistic goal for almost all
students in my district.
44
40
8
8
0
0
85
Note: Response options included “Don’t Know.” The number of respondents who selected this choice is
listed, but those respondents were not used to calculate the reported percentages.
One survey respondent said that MassCore has not been a district priority due to competing demands, but
that their commitment would increase if it became a state requirement. In addition, one respondent was in
favor of MassCore, but with exemptions as needed:
UMass Donahue Institute
Research and Evaluation Group
79
RTTT C&CR Evaluation Annual Report, September 2012
Maybe the state needs to define graduation requirements, rather than leaving it up to each district.
Not to say that EVERY student can achieve MassCore, but maybe there needs to be a process to
exempt specific kids for good reasons. MassCore completion should be the expectation for most.
Another respondent felt that MassCore needed to be a funded mandate:
Make MassCore requirements mandatory only IF you can provide the support that districts will
need to make it happen. Perhaps you could designate a specific grant toward this goal, if there is
not yet one already in place.
Finally, one respondent reported that their district had not received much information on MassCore, so it
has not been a priority for them.
Challenges to Implementing MassCore
Districts have taken many actions toward implementing MassCore, and those actions will be described in
the next section, as they are easier to understand in light of the challenges districts report. Those
challenges are described next.
Four years of mathematics. The most commonly mentioned challenge was related to
MassCore’s requirement of four years of mathematics, although the specific math-related challenges
varied. One district reported that the math department had recently split Algebra 1 into a two-year course,
against some strong opposition in the district, and that many students had been steered toward and
selected this course. The math department strongly favors keeping this course due to previously high rates
of failure in 9th grade mathematics. Because Algebra 1 only counts for one year of MassCore credit, these
students would not complete MassCore even if they did complete four years of math courses. This district
also offers a “technical math” course for the fourth year of math, but the interviewee didn’t feel that this
course would meet MassCore or Common Core standards and would therefore also not count toward
MassCore completion. The interviewee also said that many 9th graders are still failing even the slowerpaced algebra course, so reducing rigor is clearly not the solution to the failure problem.
One respondent summarized the rigor issue by saying
MassCore completion is simplistic in terms of really addressing the bigger picture issues…It is a
recommended course sequence, but it doesn’t speak to the quality of those courses. We all know
that there is quite a range within what is considered to be aligned to standards and what is
meeting the expectations of the standard. And so I think we could very easily get a higher
percentage of kids completing MassCore if we were willing to have courses that allowed for
fourth year math that [only] moves through algebra two, but the quality of the content of those
courses still isn’t meeting the needs of our students and getting them college or career ready.
One district reported an issue of low expectations of students, that struggling students were guided away
from a fourth year of math, and argued that that aspect of the district’s culture needed to change.
Similarly, another respondent argued:
Some school personnel don’t want to eliminate the below-college-prep level. They offer the
refrain of ‘You don’t understand; there are some kids who can’t do it,’ but I say ‘all kids can do
it.’ We have just got to figure out a way to get there…But at the high school there is still a
UMass Donahue Institute
Research and Evaluation Group
80
RTTT C&CR Evaluation Annual Report, September 2012
sentiment of ‘These are the kids who can do it, and these are the kids who can’t. The kids who
can are college bound. The kids who can’t are career bound.’ But we are now talking college and
career ready, not college or career ready. That is a big paradigm shift for traditional high schools.
So having that level of rigor, rather than the sorting of kids, is going to be something we are
going to struggle with. It is probably going to take us three years to get there with any level of
fidelity, but we will get there.
Another district administrator mentioned a similar situation, but reported a different current status: “The
math teachers are fighting us tooth and nail on this, but we are doing away with that standard level.”
Other curricular areas. Districts also reported challenges with implementing other aspects of
MassCore. Inadequate building infrastructure was cited by two districts for their inability to fulfill the
physical education requirement, and by one district in relation to inadequate science laboratory space.
Districts reported a shortage of teachers in MassCore-required areas including fine arts, lab science, and
world languages, as well as a shortage of ELL teachers in disciplines other than English Language Arts.
While some districts reported that their response to these shortfalls will be to work toward resolving them,
others expressed a preference for modifying MassCore so it focuses on core curriculum areas but not on
areas such as fine arts or physical education.
Some high-performing students may also encounter challenges with fulfilling MassCore. One district
reported having a strong arts program in which many students take Advanced Placement art courses and
devote substantial time to creating an arts portfolio. These students often lack openings in their schedule
for sufficient physical education classes. Another school reported similar conflicts with students having to
choose between a physical education course and an Advanced Placement lab science.
Other MassCore implementation challenges.

Two respondents said that 100% MassCore completion is a worthy goal, but that there will
always be a small number of students who, due to severe learning disabilities, won’t be able
to meet the MassCore requirements.

Three respondents noted constraints due to school schedules that lacked the flexibility to
schedule students for courses required by MassCore.

Two respondents said that they lacked adequate funds to implement changes required in order
to meet MassCore requirements. One cited future staff reductions, due to the pending
termination of federal ARRA funds, as a serious obstacle to reform.

One respondent said that rather than the 4th year of math, their major challenge was that 1520% of their 9th graders not being promoted to 10th grade, and concluded that preparedness
for high school is the district’s biggest challenge to meeting MassCore requirements. Multiple
respondents also cited student preparedness issues.
Survey respondents were asked the extent to which a series of challenges affected MassCore completion
in their district. Eight areas were considered major or moderate challenges to MassCore completion by
more than a third of respondents, including the need to build capacity to offer needed math courses
(38%), science courses (40%), world language courses (51%), special education services (45%), and ELL
services (50%); needing additional science laboratory space (41%); course scheduling (38%); and the gap
between MassCore expectations and student skill levels (40%). The remaining areas were also seen as
major or moderate challenges by a substantial share of respondents, including the need for additional
UMass Donahue Institute
Research and Evaluation Group
81
RTTT C&CR Evaluation Annual Report, September 2012
physical education facilities (23%) and art classrooms (20%), and that the MassCore curriculum requires
more time for academic curriculum than is available for vocational students (26%).
Challenges to Increasing MassCore Completion Rates
To what extent do the
Moderate
Not a
Don’t
following areas present a
Major
Minor
N
challenge to increasing
Challenge Challenge Challenge Challenge Know
MassCore completion rates
(%)
(%)
(%)
(N)
(%)
in your district?
We need to build capacity to
83
13
25
21
41
0
offer needed math courses.
We need to build capacity to
83
11
29
20
40
0
offer needed science courses.
We need to build capacity to
offer needed world language
81
26
25
15
34
2
courses.
We need to build capacity to
offer needed special
82
21
24
21
34
1
education services.
We need to build capacity to
79
23
27
20
30
3
offer needed ELL services.
We need additional science
80
23
18
13
46
3
laboratory space.
We need additional physical
81
12
11
14
63
2
education facilities.
We need additional art
78
6
14
17
63
3
classrooms.
Our master schedule makes it
difficult for students to take
81
17
21
25
37
2
the courses they need.
The gap between MassCore
expectations and student skill 80
13
28
26
33
3
levels.
For vocational students,
MassCore completion
requires more time for
53
17
9
19
55
22
academic curriculum than is
available.
Note: Response options included “Don’t Know.” The number of respondents who selected this choice
is listed in the table, but those respondents were not used to calculate the reported percentages.
Survey respondents were also asked about the influence of an additional set of challenges on MassCore
completion in their district, as shown in the table below. Those considered major or moderate challenges
UMass Donahue Institute
Research and Evaluation Group
82
RTTT C&CR Evaluation Annual Report, September 2012
by the most respondents included an inadequate budget to take needed actions (49%), that MassCore
completion is not a requirement in their district (36%), and that MassCore completion is not required by
ESE (23%). Only a small minority of respondents rated the remaining issues as major or moderate
challenges, including lacking a district needs assessment (11%) or action plan (11%), disagreement
between the district administration and the school board as to whether MassCore completion should be a
graduation requirement (3%), or that the district leadership (1%) or school board (1%) do not believe that
MassCore should be a graduation requirement.
Challenges to Increasing MassCore Completion Rates
To what extent do the
Moderate
Not a
Don’t
following areas present a
Major
Minor
Challenge
Know
N
challenge to increasing
Challenge Challenge Challenge
MassCore completion rates
(%)
(%)
(%)
(N)
(%)
in your district?
MassCore completion is not
a graduation requirement
81
9
27
10
54
2
within our district.
MassCore completion is not
82
7
16
15
62
1
required by ESE.
The district leadership does
not believe MassCore should 79
1
4
6
89
4
be a graduation requirement.
The school board does not
believe MassCore should be
73
1
0
9
90
10
a graduation requirement.
The district administration
and the school board disagree
as to whether MassCore
71
0
3
1
96
12
should be a graduation
requirement.
Our budget is inadequate to
82
32
17
19
32
1
take the needed actions.
We lack a district needs
assessment to identify
80
3
8
11
78
2
needed actions.
We lack a district action plan
for increasing MassCore
82
2
9
17
72
1
completion.
Other
9
33
11
0
56
4
Note: Response options included “Don’t Know.” The number of respondents who selected this choice
is listed in the table, but those respondents were not used to calculate the reported percentages.
Five administrators who selected the "Other" response provided further explanation. Three reported
budget challenges, with one saying, “Our budget was reduced by the town by more than $2 million from
UMass Donahue Institute
Research and Evaluation Group
83
RTTT C&CR Evaluation Annual Report, September 2012
our needs-based assessment. This results in major deficits to acquire faculty, equipment, and technology
necessary to complete MassCore for our students.” One respondent said their district struggled with
facilities and the allocation of time toward arts and physical education or academic courses. One reported
that their major challenge is that the district’s mathematics graduation requirement is only three years.
Actions Taken to Implement MassCore
Districts proposed many approaches to moving toward greater implementation of the MassCore program
of studies, including structural changes, ways of providing required coursework, needs assessment, and
action planning.
Structural changes. Two respondents reported their district’s reform efforts as proceeding in two
stages, starting with immediate work (such as online credit recovery) to target students at high current risk
of not graduating, followed by or simultaneously with structural changes. One respondent cautioned
against overemphasizing remediation, despite its importance, at the expense of larger structural changes,
because “there comes a point where we need to look at the existing standard structures of the middle and
high school experience and change those, and MassCore gives us the perfect opportunity and mandate
how to do that.”
Structural changes include revising the school’s course schedule to accommodate courses required for
MassCore completion. One school said that such changes will permit current 9th and 10th graders to take
needed courses, but that the changes will be too late for 11th and 12th graders to complete MassCore
requirements. Another school said that their recent conversion to a block schedule, although carried out
before the current emphasis on MassCore, has had two major benefits related to MassCore. First, the
block schedule creates more scheduling flexibility for electives. Second, and of substantial interest in
terms of MassCore’s four-year math requirement, the block schedule enables many 10th graders to receive
two years of math credit in one year by taking a math course both semesters. This means that many
students enter 11th grade having already completed three years of math credit, thereby making the fouryear MassCore graduation requirement more easily attainable.
Two districts also reported trying to move toward MassCore implementation at a practical level before
trying to do so at a policy level. One district reported that, despite the school committee’s opposition to
making MassCore a graduation requirement, “The principals and I and the guidance people have all been
told that they are now to schedule kids as though MassCore is the law of the school district, and that is
what they are doing.” They believe that establishing MassCore at a practical level will make it easier to
convince the school committee to embrace MassCore at the policy level. Another district reported a
similar strategy and said, “It is much better to have that internal debate and make these changes … so that
it happens more organically. So by the time we are going to a public forum, we can point to where all our
kids are, so changing our graduation requirements becomes more feasible.”
One district spoke about changes at the middle school level, explaining that younger students who have a
history of promotion despite course failure don’t understand that ways in which that changes in high
school and jeopardizes their ability to graduate in four years. As part of their implementation of the Mass
Model, this district intends to do some transition planning interventions with middle school students that
target this issue, and they believe that this could increase MassCore completion over the long term.
Survey respondents were asked about a series of actions that their district might have taken in service of
increasing MassCore completion rates, and their responses are in the table below. The most common
UMass Donahue Institute
Research and Evaluation Group
84
RTTT C&CR Evaluation Annual Report, September 2012
actions already taken were conducting a needs assessment to identify areas in which existing courses or
supports need to be expanded or new ones added (68%), adding new courses or expanding existing ones
(68%), proposing changes to district policies (63%), successfully changing district policies (66%), and
starting or expanding online credit recovery opportunities (58%). Only 47% reported that their district had
developed a written action plan to increase MassCore completion.
Actions Taken in Service of Increasing MassCore Completion Rates
N
Action
Already
Taken
(%)
Action
Planned But
Not Taken Yet
(%)
No Plan To
Take This
Action
(%)
Don’t
Know
(N)
Conducted a needs assessment to
identify areas in which existing courses
or supports need to be expanded or new
ones added.
81
68
22
10
2
Developed a written action plan to
increase MassCore completion.
80
47
37
16
2
Proposed changes to one or more district
policies.
78
63
22
15
5
Successfully changed one or more
district policies.
77
66
16
18
4
Added new courses or expanded existing
courses.
80
68
21
11
1
Hired additional personnel.
79
40
16
44
4
Added facilities (e.g., science labs, art
classrooms, physical education).
77
16
20
64
5
Started or expanded online credit
recovery opportunities.
77
58
26
16
5
Started or expanded teacher-led credit
recovery opportunities.
73
48
19
33
7
Other
15
53
7
40
2
My district has planned and/or taken
the following actions in service of
increasing MassCore completion
rates:
Note: Response options included “Don’t Know.” The number of respondents who selected this choice is
listed in the table, but those respondents were not used to calculate the reported percentages.
Administrators who selected the "Other" response option said that their students were already meeting
MassCore requirements, so their district’s focus was on making sure that this continued (N=3).
Respondents also reported partnering with colleges (N=2), providing summer credit recovery programs
(N=1), developing middle school programs to increase readiness for 7th and 8th graders (N=1), starting an
UMass Donahue Institute
Research and Evaluation Group
85
RTTT C&CR Evaluation Annual Report, September 2012
early college program for first-generation low-income students (N=1), providing dual enrollment (N=1),
developing a tiered system for academic and non-academic learning supports (N=1), and changing the
graduation requirements for 2016 (N=1).
Ways of providing required coursework. Districts described many strategies that directly
address student shortfalls in course completion and high school readiness, including the following:

Purchasing science lab equipment and creating new science lab space.

Hiring personnel to fill gaps in staffing for courses required for MassCore completion.

Hiring personnel to work directly with the highest-risk students, as identified by early
warning indicator data, in order to promote credit recovery and prevent students from falling
further behind.

Working with the APEX online credit recovery program.

Working with the Massachusetts Math and Science Initiative, both on their Advanced
Placement program and on their Pre-AP program, to establish vertical teams between middle
school and high school to improve vertical alignment and high school readiness.

Working with Agile Mind to provide a remedial summer algebra program for incoming 9th
graders, so that students can develop algebra skills and build relationships with teachers to
help them feel more engaged and welcomed when they arrive for the beginning of the school
year.

Piloting a strategy, in collaboration with Middlebury College, to use Plato online courses to
fill gaps in world language offerings, not as credit recovery, but as an approach for students
who can’t complete world language requirements due to schedule conflicts or personnel
shortfalls.

Developing a 4th-year math course that allows and challenges students to apply the algebra
and geometry that they learned in the first three years of high school in more exciting and
practical ways.

Allowing school-sponsored sports team participation to count toward PE requirements.

Using RTTT funds to finance these activities, and to pay stipends to school and district
personnel so that they can carry out MassCore-related administrative activities, curriculum
planning, and professional development.
Needs assessment and action planning. All interview districts reported having conducted a
review of course offerings and student performance data to identify gaps and problem areas, and all
districts were able to articulate strategies that they currently employ and/or intend to employ in order to
move toward greater MassCore completion. In addition, as noted in the tables above, a majority of survey
districts reported that they have either conducted a needs assessment already (68%) or have plans to
conduct one (22%), and nearly half reported that they have either written an action plan already (47%) or
have plans to write one (37%). In a discussion of these findings, ESE’s MassCore program manager
clarified that all Goal 4A districts had officially fulfilled ESE’s requirements for MassCore needs
assessment and action planning by writing relevant paragraphs in their RTTT applications.
Advice to others on increasing MassCore completion rates. Administrators were also asked to
provide advice to other districts about how to increase MassCore completion rates, based on their own
UMass Donahue Institute
Research and Evaluation Group
86
RTTT C&CR Evaluation Annual Report, September 2012
district’s successes and challenges. The most common advice was to align graduation policy to MassCore
requirements (N=8). Two of these respondents also said that districts should include MassCore as part of
their strategic planning process.
Six respondents commented on stakeholder buy-in. Two said that a high level of commitment from the
administration, school committee, and staff was vital to success. Four suggested that administrators
actively reach out, communicate, and campaign the public for support. One reported that their district was
planning to begin using the news media for this purpose. Another recommended that administrators begin
the campaign early and gather data to support the cause.
Four administrators recommended that districts incorporate intervention strategies and rigor at the
elementary and middle school levels so that students enter high school prepared to succeed under the
MassCore curriculum. Two respondents said that districts should be prepared to provide extra support and
guidance for students in order to help them meet requirements. One suggested that student schedules be
flexible enough to accommodate MassCore requirements, while another recommended a 4x4 AB
schedule to increase course offerings. One respondent advised districts to work closely with each student
and to place students on a trajectory for college acceptance.
Supports and Sustainability
For the most part, interview respondents did not feel that their districts had needed, requested, or received
MassCore-related technical assistance. Most mentioned having spoken with ESE’s MassCore program
manager when they had questions. One said “MassCore is common sense. I think this is just getting
people together on the same page and making sure that they do what they need to do.” Another said,
It was really a matter of diving into our own data and having some meaningful conversations
about looking at which students aren’t completing which requirements. What courses have they
taken? What does their day look like? What did their day look like as a freshman, as a
sophomore, as a junior, as a senior? Where did it fall apart for them? Was it right from the get go
in freshman year or was it something happened after MCAS and when they got into the upper
high school grades? All of that work we have been able to do ourselves, really.
When asked what their future needs or desires for technical assistance might be, interviewees offered
several responses:

One respondent said that maybe they in fact did need technical assistance but just didn’t
know what their needs were or what questions to ask.

Two respondents expressed interest in hearing about other districts’ successful practices,
including one who said, “I think we would love to all know who is doing it right, particularly
for districts that are similar to ours. If others have figured out creative solutions, we should be
learning from each other.”

One said that they could use technical assistance with some of the larger structural changes
needed related to MassCore completion.

One requested “tools or templates or exemplars that could help a district assess course-taking
at the middle and high school level…quick data tools for comparison of course-taking failure
rates and course sequencing…if there are tools that allow a district to do that or just some
expertise in doing that, that would be most helpful.”
UMass Donahue Institute
Research and Evaluation Group
87
RTTT C&CR Evaluation Annual Report, September 2012
One respondent expressed feeling that ESE had provided inconsistent answers to inquiries about
MassCore, and that there needs to be a clear repository of information that says either “Our clear policy is
as follows: ____,” or “We haven’t totally figured that out yet.” The respondent felt that different people at
ESE had provided different answers at different times, and that the answers had at times been confusing.
The administrator survey included three open-ended questions related to supports and technical
assistance, starting with “What useful supports could ESE or other partners provide to increase MassCore
completion in your district?” The most common responses were related to funding (N=14). One
administrator said, “If you truly want districts to implement the MassCore...require it and attached
funding to support it.” Respondents said they wanted additional funds for math, science, and technology
equipment and resources (N=1); flexible funding for partnerships with higher education (N=1); funding
for math and reading tutors to provide intensive intervention (N=1); and more funding for certified
science teachers (N=1). One respondent asked ESE to support funding allocations on a per pupil
expenditure basis. Another recommended that ESE “provide free Advanced Placement Summer Institute
training for districts in all major high school content areas. These courses are extremely expensive for
districts to find funds for staff to attend!” One respondent reported having budget and staffing challenges
for science courses because of their school’s small size.
Four respondents said they could use support for students with special needs who struggle to meet
MassCore requirements. Three of these were related to MassCore’s foreign language requirement, with
respondents asking if there were any alternative options or exceptions for students with disabilities. One
respondent asked ESE to “give practical suggestions for students with learning difficulties to be
successful in completing world languages and advanced math (Algebra II and more), as well as chemistry
and those kinds of programs.” One additional respondent asked for support regarding students with
interrupted formal education.
Four respondents asked for clarification of MassCore requirements, with two saying that the Board of
Higher Education and MassCore standards regarding Engineering and Technology are inconsistent. One
asked,
Why is engineering not considered a lab-based science course? Please consider vocationaltechnical courses taught by Chapter 74 instructors who, while not considered to be highly
qualified under NCLB, nevertheless teach rigorous courses with embedded science. Will an
advanced course in Electronics Engineering be able to be counted as a lab-based science to meet
college readiness requirements?
One additional respondent asked, “PARCC assessments – do we prepare for them? Science frameworks –
how do we change when the old standards are now determining AYP status?” One administrator
recommended that ESE provide criteria for determining course standards that meet MassCore
requirements.
Respondents also asked for more support with scheduling (N=4). Three said that vocational schools need
support with master schedule issues and special circumstances that can prevent students from taking
required MassCore courses. One of these respondents asked for guidance on how vocational students can
meet requirements for both MassCore and their vocational certificates in a seven-period day. One
respondent asked that ESE or their partners “hold some workshops where districts could discuss and
brainstorm solutions around master scheduling issues.”
UMass Donahue Institute
Research and Evaluation Group
88
RTTT C&CR Evaluation Annual Report, September 2012
Three respondents asked for more workshops and training. One expressed interest in attending workshops
on conducting needs assessment and strategic planning. Another asked for more professional
development for foreign language teachers who are delivering tiered instructions. Respondents also asked
that ESE and other partners make available more tools and information (N=3). Administrators requested
to have credit recovery tools at low cost, detailed information on how to develop an action plan, and an
electronic resource of tools and approaches. One additional respondent said that the Massachusetts
Teacher’s Association would be a useful resource to help with understanding teachers’ roles and
responsibilities in meeting the needs of all students.
Two respondents expressed their challenge with inadequate facilities, with one saying,
The best thing that you could do is eliminate the Physical Education requirement from MassCore.
We have neither the space nor the personnel to fulfill the “spirit” of the PE requirement, and we
will never be successful as long as it is a MassCore requirement – unless jumping jacks before
lunch counts for the “letter” of the law. I spend too much time on this one point, and it has
nothing to do with CCR (CPR maybe, but not CCR). As a graduation requirement, I would settle
for some version of MassCore, especially concerning the academics and arts.
One respondent asked ESE to “increase pressure on districts to achieve targets, or at least to show
focused, serious, coordinated effort. Link funding to achieving your outcomes, or at least to being able to
document that you're doing everything in your power to achieve them.” Finally, one respondent said their
district needed more time due to budget cuts and staff downsizing (N=1), and one requested support in
educating the community about MassCore to gain public buy-in (N=1).
Specific questions about sustainability will be a greater focus of data collection efforts in the next two
years, but the above comments clearly indicate that there are many supports that could aid districts’
implementation of a curriculum that fulfills the MassCore requirements. These include technical
assistance focused on implementing MassCore and related structural changes; sharing of best practices
from other districts; needs assessment tools; and an online repository of answers to MassCore policy
questions, as well as funding and policy support, including a funded state mandate related to higher rates
of MassCore completion and offering of the needed courses. Additional needs relate to staffing, facilities,
scheduling (particularly for vocational schools), and alternatives or exemptions for students with
disabilities. No technical assistance vendor has been enlisted to provide supports for and enhance
sustainability of the MassCore initiative, unlike the STEM ECHS and Pre-AP initiatives that are also part
of this evaluation of the RTTT college and career readiness initiatives.
Baseline Graduation and Dropout Rates
The following three pages each contain a table paired with a figure that present the following baseline
graduation and dropout rate data for the 155 schools that selected RTTT Goal 4A:
1. Average 4- and 5-year cohort graduation rates from 2007-08 to 2010-11.
2. Average annual dropout rates for 2007-08 to 2010-11.
3. Average 4- and 5-year cohort dropout rates 2007-08 to 2010-11.
The companion document, Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report,
presents tables and figures that provide a more complete summary of these same baseline graduation and
UMass Donahue Institute
Research and Evaluation Group
89
RTTT C&CR Evaluation Annual Report, September 2012
dropout data. The Technical Supplement contains tables and figures that present both average and
individual graduation and dropout rates for RTTT Goal 4A districts and individual schools. Future
evaluation reports will use these baseline data to describe trends in graduation and dropout rates in
MassCore schools and districts. The data necessary to disaggregate baseline graduation and dropout rates
by subgroup (e.g., gender, race/ethnicity) were not available in time for this report, but will be included in
future analyses. Future reports will also describe trends in indicators of student engagement and academic
achievement in MassCore schools and districts.
MassCore: 4- & 5-Year Graduation Rates
4- & 5-Year Graduation Rates for MassCore Schools, 2007-08 to 2010-11
Year
Number of
Students
4-Year Graduation
Rate
5-Year Graduation
Rate
Difference
Between Rates
2007-08
44,196
78.3%
81.7%
3.4%
2008-09
44,646
77.9%
81.0%
3.1%
2009-10
44,376
78.5%
81.4%
2.9%
2010-11
42,710
79.8%
NA
NA
4- & 5-Year Graduation Rates for MassCore Schools, 2007-08 to 2010-11
100%
95%
Graduation Rate
90%
85%
4 Year Rates
80%
5 Year Rates
75%
70%
65%
60%
55%
50%
2007-08
UMass Donahue Institute
Research and Evaluation Group
2008-09
2009-10
School Year
2010-11
90
RTTT C&CR Evaluation Annual Report, September 2012
MassCore: Annual Dropout Rates
Annual Dropout Rates for MassCore Schools, 2007-08 to 2010-11
Year
Total HS Enrollment
Number of Dropouts
Annual Dropout Rate
2007-08
170,164
6,999
4.1%
2008-09
169,525
6,245
3.7%
2009-10
167,975
5,892
3.5%
2010-11
166,543
5,730
3.4%
Annual Dropout Rates for MassCore Schools, 2007-08 to 2010-11
10%
9%
Annual Dropout Rate
8%
7%
6%
5%
4%
4.1%
3%
3.7%
3.5%
3.4%
2008-09
2009-10
2010-11
2%
1%
0%
2007-08
School Year
UMass Donahue Institute
Research and Evaluation Group
91
MassCore: 4- & 5-Year Cohort Dropout Rates
4- & 5-Year Cohort Dropout Rates for MassCore Schools, 2007-08 to 2010-11
Number of
Students
Year
4-Year Cohort
Dropout Rate
5-Year Cohort
Dropout Rate
Difference
Between Rates
2007-08
44,196
11.9%
11.7%
-0.2%
2008-09
44,646
11.5%
11.7%
0.2%
2009-10
44,376
10.6%
10.9%
0.3%
2010-11
42,710
9.2%
NA
NA
4- & 5-Year Cohort Dropout Rates for MassCore Schools, 2007-08 to 2010-11
30%
Cohort Dropout Rate
25%
20%
4-Year Rate
5-Year Rate
15%
10%
5%
0%
2007-08
UMass Donahue Institute
Research and Evaluation Group
2008-09
2009-10
School Year
2010-11
92
RTTT C&CR Evaluation Annual Report, September 2012
Baseline Rates of MassCore Completion
The table below shows the distribution of MassCore completion rates for the 2009-10 school year for
Massachusetts districts that include high school students. The data were provided by ESE and are based
on the MassCore completion element in the SIMS database, which is provided by districts based on their
internal determination of each graduate’s completion of MassCore requirements. The intervals in the table
include 0% and 100% as separate categories, which is unconventional but deliberate in this case, as will
be clear from the discussion below. The 2010 MassCore completion average was 70.3% (44,311 out of
63,072 graduates).
MassCore Completion by District for SY2009-10
Graduates
% Completion
# of Districts
% of Districts
0
18
6
1-20
14
5
21-40
20
7
41-60
19
7
61-80
27
10
81-99
41
14
100
146
51
Avg: 70.3%
Total: 285
Total: 100
As shown in the table, 18 districts (6%) indicated that none of their graduates completed MassCore, while
146 districts (51%) indicated that all of their graduates completed MassCore. Conversations with ESE’s
MassCore program manager, and with other ESE personnel, have made it clear that these numbers are
inaccurate. Specifically, ESE personnel have pointed out that some of the districts that report 0% are
high-performing districts by other academic measures, and some of the districts that report 100% are lowperforming districts by other academic measures. These are very improbable outcomes.
It is, of course, possible that some districts do not have the capacity to fulfill certain MassCore
requirements, such as if they lack adequate physical education facilities, which could explain a zero
percent completion rate. Also, more than 40 survey respondents to the MassCore survey indicated that
MassCore completion is a graduation requirement in their district, and additional districts could provide a
rigorous curriculum to the vast majority of their students, leading to a MassCore completion rate near
100%. However, with a statewide reported baseline (2010) rate of 70%, it is unlikely that more than half
of the districts have a 100% completion rate.
In short, many districts are reporting the SIMS MassCore element inaccurately. This means that baseline
estimates of MassCore completion for the state, and particularly for individual schools, are likely to be
very inaccurate. Moreover, estimates of post-baseline changes that are based on the SIMS MassCore
element are likely to reflect factors other than actual changes in MassCore completion rates. Such factors
could include improved reporting accuracy, which might actually lower the baseline level, since many
districts are apparently over-estimating their completion rates at present. In addition, the RTTT targets
that the state calculated for districts are, in many cases, different from the targets that would have been
calculated if accurate completion rates had been available.
UMass Donahue Institute
Research and Evaluation Group
93
RTTT C&CR Evaluation Annual Report, September 2012
A final element that challenges the validity of MassCore completion reporting is that districts determine
locally whether a given course meets MassCore requirements. There is no clear standard for districts to
follow in making this determination, and there is likely to be wide variability in the content and rigor of
courses that districts designate as meeting a particular requirement.
ESE reported that they currently do not know how most districts determine students’ MassCore
completion status, although ESE reports that they are taking steps to understand this process more clearly.
UMDI plans to interview some district data managers in Year 3 for the same purpose. However, given the
current data available, it will not be possible to evaluate changes in MassCore completion rates accurately
at the school, district, or state levels.
Strategic Considerations

Districts appear to see substantial value in raising MassCore completion rates, but
limited accountability for reaching state MassCore targets may subordinate MassCore
among district priorities. Districts have very limited accountability regarding
implementation of strategies designed to increase MassCore completion, or for raising
student MassCore completion rates. MassCore remains a recommended, rather than a default
curriculum, which reduces district accountability and the leverage of some MassCore
advocates in their own districts. Until greater accountability structures are in place, achieving
targeted rates of MassCore completion may be difficult.

Meaningful assessment of changes in MassCore completion is challenged by serious
validity and reliability limitations of the MassCore element of the SIMS database. This
SIMS element is the state’s primary way of assessing MassCore completion rates, yet
analysis of the SIMS element shows extensive inaccuracies in district reporting.
Improvements to implementation of the SIMS MassCore element, combined with increased
district accountability for accurate MassCore reporting, would contribute to the ability to
develop meaningful conclusions related to changes in MassCore completion. Implementation
of such changes would complicate interpretation of changes from the state’s RTTT baseline
MassCore completion rate of 70%, however, because the baseline rate was derived from the
flawed SIMS indicator. When the new Student Course Schedule database has been in use for
several years, and if implemented with high accuracy, it could also serve as a valuable source
of information about MassCore completion.

A MassCore exemption process, or the promotion of alternative ways to fulfill certain
requirements, may reduce some districts’ concerns about conflicting mandates and
priorities. Districts said that some high-performing students, who excel on typical measures
of academic achievement, may fall short of MassCore requirements due to taking Advanced
Placement courses, or developing advanced art portfolios that prevent them from taking some
required MassCore coursework. In addition, some students with severe learning disabilities
may be unable to complete MassCore requirements, and being required to do so may be
inconsistent with their IEPs. Some districts also lack costly infrastructure, such as adequate
physical education facilities, that is needed for students to complete MassCore. The state may
wish to consider whether there are cases in which conflicting priorities warrant exemption
from specific MassCore requirements. Permitting certain extended and out-of-school learning
opportunities to meet MassCore requirements may also address some of these concerns.

MassCore specifies the quantity of courses that must be completed, but questions
remain regarding expectations of course quality. Guidance from the state on issues of
expected course requirements, while perhaps unenforceable due to local control, may be
UMass Donahue Institute
Research and Evaluation Group
94
RTTT C&CR Evaluation Annual Report, September 2012
useful to and/or appreciated by districts who seek to understand what the state considers
adequate for MassCore completion.

The most common challenges to increasing MassCore completion rates are: building
capacity to provide required courses, services, and facilities; inadequate funding; the
gap between student skill levels and MassCore expectations; course scheduling; and
MassCore completion not being required for graduation.

Additional supports and policy changes may increase district progress toward
MassCore completion goals. Supports could include technical assistance with implementing
structural changes related to MassCore, sharing of best practices from other districts, needs
assessment tools, and an online repository of answers to MassCore policy questions.
Additional support may also include funding and policy changes, including a funded state
mandate related to higher rates of MassCore completion and the offering of required courses.
ESE may wish to consider enlisting a technical assistance vendor for the MassCore initiative
as they have done for the Pre-AP and STEM ECHS programs.

More than a quarter of district administrator survey respondents did not agree that
their district is using RTTT funds to take actions that they believe will lead to increased
rates of MassCore completion. While in some cases this could mean that districts are taking
actions but anticipate failure, but a more plausible interpretation is that they are using their
RTTT Goal 4 funds for purposes unrelated to MassCore. For these districts – some of whom
said that increasing MassCore completion is a high priority for their district – MassCore
completion may be a lower priority than other strategies for increasing college and career
readiness.
UMass Donahue Institute
Research and Evaluation Group
95
RTTT C&CR Evaluation Annual Report, September 2012
Appendices
UMass Donahue Institute
Research and Evaluation Group
96