Massachusetts Mathematics and Science Partnership Title IIB
Annual State-level Evaluation Report
Cohort 4 Reporting Period: September 1, 2008, through August 31, 2010
Cumulative Reporting Period: February 2, 2004, through August 31, 2010
Prepared for the Massachusetts Department of Elementary and
Secondary Education
June 2011
Annual Report of the MMSP State-level Evaluation
Contents
Contents
Program Description................................................................................................................................... 3
Report Organization .................................................................................................................................... 5
Evaluation Plan and Activities ................................................................................................................... 6
State-level Evaluation ............................................................................................ 6
Local Evaluation and Related Technical Assistance ............................................. 6
Cohort 4 Activity: September 1, 2008 through August 31, 2010 ............................................................ 7
State-level Participant Background Data ............................................................... 7
Partnership-level Participant Background Data ................................................... 17
Cumulative Summary: All Cohorts over All Funding Periods .............................................................. 26
Summary of Findings ............................................................................................................................... 41
Cumulative Findings ............................................................................................ 41
Cohort 4 Findings ................................................................................................. 43
UMass Donahue Institute
Research and Evaluation Group
I
Annual Report of the MMSP State-level Evaluation
Tables
Tables Index
Table 1:
Table 2:
Table 3:
Table 4:
Table 5:
Table 6:
Table 7:
Table 8:
Table 9a:
Table 9b:
Table 9c:
Table 10a:
Table 10b:
Table 10c:
Table 11:
Table 12:
Table 13:
Table 14:
Table 15:
Table 16:
Table 17:
Table 18:
Table 19:
Table 20:
Table 21:
Table 22:
Table 23.
Table 24:
Table 25:
Table 26:
Table 27:
Table 28:
Table 29a:
Table 29b:
Table 29c:
Table 30a:
Table 30b:
Table 30c:
Table 31:
Budgets: Cohort 4 Partnerships ......................................................................................................... 7
Teaching Areas: Cohort 4 Participants .............................................................................................. 8
Types of Schools of Unique Participants: Cohort 4 Participants .................................................... 9
High Need Status of Unique Participants from Public Schools: Cohort 4 Participants ............... 9
High Need District Participants by Partnership: Cohort 4 Participants ....................................... 10
Reasons for Participation: Cohort 4, All Seats................................................................................ 11
Repeat Participants: Cohort 4 Partnerships .................................................................................... 11
Total Enrollment and Attrition Information: Cohort 4 Partnerships .............................................. 12
Cohort 4 Science and Technology Teaching Areas – Regular Education ................................... 13
Cohort 4 Science and Technology Teaching Areas – Special Education .................................... 13
Cohort 4 Science and Technology Teaching Areas – ELL Education .......................................... 13
Cohort 4 Mathematics Teacher Levels – Regular Education ......................................................... 15
Cohort 4 Mathematics Teacher Levels – Special Education .......................................................... 15
Cohort 4 Mathematics Teacher Levels – ELL Education ............................................................... 15
Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores
for Cohort 4 ......................................................................................................................................... 17
Boston Public Schools Participant Background Information (M) ................................................. 18
Brockton Public Schools Participant Background Information (M) .............................................. 19
Gateway Regional School District Participant Background Information (S) ............................... 20
Lesley University C4 Participant Background Information (M) ..................................................... 21
Greater North Shore Participant Background Information (S) ...................................................... 22
Randolph Public Schools Participant Background Information (S).............................................. 23
Springfield College Participant Background Information (S) ........................................................ 24
Boston University Participant Background Information (M) .......................................................... 25
Overview of MMSP Partnership Participation ................................................................................. 26
Budgets: All Partnerships, All Funding Periods ............................................................................. 28
Teaching Areas: All Participants, All Funding Periods .................................................................. 29
Types of Schools: All Unique Participants, All Funding Periods .................................................. 30
High Need Status of All Unique Participants from Public Schools ............................................... 31
Reasons for Participation of Participants: All Seats, All Funding Periods .................................. 31
Repeat Participants: All Partnerships, All Funding Periods .......................................................... 32
Highly Qualified Status: All Unique Participants, All Funding Periods ........................................ 34
MTEL Tests Taken by All Participants – Total to Date ................................................................... 35
Science and Tech/Engineering Teaching Areas of All Participants – Regular Education ......... 36
Science and Tech/Engineering Teaching Areas of All Participants – Special Education .......... 36
Science and Tech/Engineering Teaching Areas of All Participants – ELL Education ................ 36
Mathematics Teacher Levels of All Participants – Regular Education ......................................... 37
Mathematics Teacher Levels of All Participants – Special Education .......................................... 37
Mathematics Teacher Levels of All Participants – ELL Education................................................ 37
Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores .. 39
UMass Donahue Institute
Research and Evaluation Group
II
Annual Report of the MMSP State-level Evaluation
Program Description
Program Description
The purpose of the Massachusetts Mathematics and Science Partnership Program (MMSP) is to improve student
achievement in mathematics, science, and technology/engineering through intensive, high-quality professional
development activities that focus on deepening teachers’ content knowledge. This multi-year project is funded by
the U.S. Department of Education as part of the No Child Left Behind (NCLB) Act Title IIB funding stream.
Funding to local partnerships is administered by state education agencies; in Massachusetts this is the
Massachusetts Department of Elementary and Secondary Education (ESE), which awards funding through a
competitive grant process.
Partnerships awarded these grants are required to include 1) a core partner that has been identified as a high need
school district in the subject matter on which the partnership is focusing and 2) a core partner that is a science,
technology and/or engineering, or mathematics (STEM) department from an institution of higher education. The
partnerships are composed of higher education institutions, school districts, and, in some cases, private
organizations involved in providing both pre-service and in-service training to teachers. Partnerships are required
to offer courses that equal at least 45 hours of direct instruction followed by at least 20 hours of follow-up contact
to support the implementation of course content in the classroom. Partnerships are encouraged to tailor the model
used to deliver the professional development and follow-up to best fit the objectives of their programs along with
their resources, expertise, and existing infrastructure. Partnership activities are guided by the following goals 1:
Goal I
Develop and implement an effective and sustained course of study for in-service teachers of STEM by
integrating the courses of study into schools of arts and sciences and/or education at institutions of
higher education.
Goal II
Increase the number of STEM teachers in the partner school districts who are licensed in the subject
area(s) and grade level(s) they teach.
Goal III Increase the number of STEM teachers in the partner school districts who participate in high quality
professional development and advance their content knowledge.
Goal IV Develop and implement a systemic approach to STEM education by integrating professional
development with district and school STEM improvement initiatives.
The program began in February 2004, and has had seven funding periods, defined as follows:
1

Year 1: February 2, 2004, through August 31, 2004

Year 2: September 1, 2004, through August 31, 2005

Year 3: September 1, 2005, through August 31, 2006

Year 4: September 1, 2006, through August 31, 2007

Year 5: September 1, 2007, through August 31, 2008
Program goals were modified slightly over the course of the program.
UMass Donahue Institute
Research and Evaluation Group
3
Annual Report of the MMSP State-level Evaluation

Year 6: September 1, 2008, through August 31, 2009

Year 7: September 1, 2009, through August 31, 2010.
Program Description
The partnerships who received initial funding in Year 1 are referred to as Cohort 1; those who received initial
funding in Year 2 are referred to as Cohort 2; those who received initial funding in Year 4 are referred to as
Cohort 3; and those who received initial funding in Year 6 are referred to as Cohort 4.
UMass Donahue Institute
Research and Evaluation Group
4
Annual Report of the MMSP State-level Evaluation
Report Organization
Report Organization
The purpose of this report is twofold: 1) to provide details regarding only Cohort 4 participation for the two years
of funding for it thus far and 2) to provide a cumulative summary regarding participation for all cohorts over all
funding periods.
Data supporting the first purpose address the period of September 1, 2008, through August 31, 2010; data
supporting the second purpose address the period of February 2, 2004, through August 31, 2010. Participant data
were collected through the Participant Background Survey, an instrument developed by the UMass Donahue
Institute (UMDI) and administered by partnerships to each participant on the last day of each course. See
Appendix A for the survey used during Year 7. The purpose of this survey is to gather data about participants’
professional backgrounds and qualifications. This information provides a picture of who the participants are, aids
in determining whether the courses are reaching the teachers who most need professional development, and aids
in tracking how teacher qualifications may change during the MMSP funding period. Data from the survey
regarding teacher licensure, possession of and progress towards earning degrees, and status in terms of
Massachusetts Tests for Educator Licensure (MTEL) exams allows determination of the number of teachers who
meet criteria defining highly qualified status. Unless noted, data from the survey are reported in terms of unique
individuals, regardless of the number of courses taken by each individual.
Data speaking to the strengthening of relationships between partnership members were collected through a section
of the local evaluation reports that partnerships were required to submit to the ESE. In this section, partnerships
were asked to describe the extent to which their courses had been integrated into activities of their higher
education partners.
UMass Donahue Institute
Research and Evaluation Group
5
Annual Report of the MMSP State-level Evaluation
Evaluation Plan and Activities
Evaluation Plan and Activities
State-level Evaluation
Although not required by the U.S. Department of Education, the ESE contracted with UMDI to conduct a statelevel evaluation of the MMSP. UMDI’s primary role as state-level evaluator is to coordinate program-wide
collection of outcome data on behalf of the ESE. Data collection for the state-level evaluation is organized around
a basic logic model for professional development initiatives shown below.
Local Evaluation and Related Technical Assistance
In addition to the state-level data collection, each partnership is required to conduct its own local evaluation. In an
effort to support strong local evaluations, ESE required that partnerships sub-contract with UMDI to provide
technical assistance on design and implementation of their local evaluations. The timeline listing the evaluation
activities is found in Appendix B.
UMass Donahue Institute
Research and Evaluation Group
6
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Cohort 4 Activity: September 1, 2008 through August 31, 2010
Cohort 4, which began in the 2008-2009 funding period, consisted of eight partnerships. Table 1 shows the
funding received by Cohort 4 partnerships for the two periods between September 2008 and August 2010.
Table 1: Budgets: Cohort 4 Partnerships
Partnership
Sep08-Aug09
Sep09-Aug10
Boston Public Schools (M)
$157,975
$405,747
TOTAL
$563,722
Brockton Public Schools (M)
$180,145
$255,758
$435,903
Gateway Regional School Dist. (S)
$186,609
$200,370
$386,979
Lesley Springfield (M)
$228,593
$324,820
$553,413
Greater North Shore (S)
$265,917
$306,690
$572,607
Randolph Public Schools (S)
$176,993
$183,150
$360,143
Springfield College (S)
$161,062
$148,896
$309,958
Boston University (M)
$241,586
$245,180
$486,766
$1,598,880
$2,070,611
$3,669,491
TOTAL
State-level Participant Background Data
Cohort 4 consisted of eight partnerships, with four of the eight partnerships offering mathematics professional
development and four offering science professional development. There were 74 Cohort 4 courses delivered. Of
these 74 courses, 46 were mathematics courses and 28 were science courses. Of those 74 courses, 33 (45%) were
unique, and 41 (55%) were repeat offerings. This section of the report summarizes data collected from
participants in these courses.
By the end of the 2009-2010 funding period for Cohort 4, 840 unique participants completed the Participant
Background Survey on one or more occasions. Of these 840 participants, 233 took two or more courses within
Cohort 4, which translated to 1,230 course seats. The term “unique participant” refers to each individual who
participated in the program, regardless of how many courses he or she took. Data for items from the survey that
help to convey participants’ professional backgrounds and motives for participation are discussed in the
remainder of this section. All survey data for the group completing the survey in 2009-2010 may be found in
Appendix C. The responses to the survey questions are presented as frequencies and percentages. Not all
percentages total 100% because many items allowed multiple responses and not all of the participants responded
to all of the items.
Position of Participants
At the time of their last completed survey from a Cohort 4 MMSP course, 96% of course participants identified
themselves as teachers. Of all respondents, 72% were regular education teachers; 19% were special education or
special education inclusion teachers; 5% were ELL teachers; <1% were long-term substitutes; <1% were
paraprofessionals; <1% were principals, assistant principals, or headmasters; and 3% indicated that they held
“other” positions.
UMass Donahue Institute
Research and Evaluation Group
7
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Content Taught
The distribution of teaching areas of respondents at the time of the survey is shown in Table 2. Because
respondents identified all teaching areas that applied to their positions at the time of the survey, some selected
multiple responses, so frequency totals exceed the number of respondents and percentages exceed 100%.
At the time of their last MMSP course, 38% of Cohort 4 participants were teaching mathematics (including those
teaching mathematics at the elementary level), 34% were teaching science, and 31% were teaching all subjects at
the elementary level.
Table 2: Teaching Areas: Cohort 4 Participants
Sep08-Aug09
Sep09-Aug10
TOTAL
N =341
N =630
N =840
Teaching Areas
(Multiple responses permitted)
Mathematics
Any science area
General Science
Biology
Earth Science
Chemistry
Physics
Technology/Engineering
Elementary (all subjects)
Elementary Mathematics
Other
Not Currently Teaching
n
102
138
98
27
22
17
17
14
87
21
11
7
%
30
41
29
8
7
5
5
4
26
6
3
2
n
184
213
123
52
25
31
34
13
197
55
26
12
%
29
34
20
8
4
5
5
2
31
9
4
2
n
250
286
175
67
39
43
45
22
260
67
37
17
%
30
34
21
8
5
5
5
3
31
8
4
2
Teaching Experience of Participants
At the time of their last completed survey from an MMSP course, the teaching experience of the 840 unique
Cohort 4 participants was as follows: 15% were in their first to third year of teaching, 41% had between four and
ten years experience in education, 27% had between 11 and 20 years of experience, 14% reported over 20 years of
experience, and 3% did not report.
Teaching Levels of Participants
For reporting purposes, schools in the participating districts were organized into categories of elementary schools
(grades K-5), K-8 schools, middle schools (grades 6-8), and high schools (grades 9-12). At the time of their last
completed survey from an MMSP course, 36% of Cohort 4 participants were teaching in an elementary or K-8
school, 44% were teaching in a middle school, 17% were teaching in a high school, and less than 1% were
teaching at the pre-K level. Those remaining either were not currently teaching or the level at which they taught
was unknown.
Types of Schools of Participants
As shown in Table 3, approximately 98% of unique Cohort 4 participants worked in a public school setting, and
approximately 3% worked in a non-public school setting.
UMass Donahue Institute
Research and Evaluation Group
8
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 3: Types of Schools of Unique Participants: Cohort 4 Participants
Sep08–Aug09
School Type
n
Public Schools (includes public charter schools)
Non-public School
Other or No Response
%
333
7
1
341
TOTAL
98
2
<1
100
Sep09–Aug10
TOTAL
n
%
n
%
614
16
0
98
3
0
819
21
0
98
3
0
630
100
840
100
High Need Status of Districts of Participants
MMSP partnerships were required to include at least one high need district. Appendix D identifies the criteria for
the high need designation.
The ESE expected that at least 50% of participants in each partnership would come from high need districts, and
further, they set an informal goal that at least 75% of participants for each partnership would come from high need
districts. Of all Cohort 4 participants, including those from non-public schools, 65% came from high need
districts. Table 4 shows that by the end of the 2009-2010 funding period, 67% of Cohort 4 participants from
public schools had come from high need districts.
Table 4: High Need Status of Unique Participants from Public Schools: Cohort 4 Participants
Sep08–Aug09
Sep09–Aug10
n
%
n
High Need District
Non-high Need District
213
120
64
36
412
202
TOTAL
333
100
614
TOTAL
School Type
%
n
%
67
33
545
274
67
34
100
819
101
Table 5 presents the number of participants from high need districts organized by each partnership. An
examination of high need district participation in individual partnerships reveals that over the course of each
partnership’s involvement in MMSP, three of the eight partnerships had at least 50% of their participants coming
from high need districts and those same three also exceeded the informal goal of having at least 75% of the
participants come from high need districts.
Reasons for Participation in MMSP Courses
For each course taken, respondents identified all of their reasons for participating. Unlike in the earlier portions of
this report where data were presented for unique participants, data for this topic are presented for all course seats,
since responses are uniquely relevant to each distinct course that a participant took. Table 6 presents findings for
all 1,230 seats for all courses taken by Cohort 4 participants through the end of the 2009-2010 funding period.
UMass Donahue Institute
Research and Evaluation Group
9
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 5: High Need District Participants by Partnership: Cohort 4 Participants
Sep08–Aug09
Partnership
Boston PS (M)
Brockton PS (M)
Gateway RSD (S)
Lesley Springfield (M)
Greater North Shore (S)
Randolph PS (S)
Springfield Coll. (S)
Boston University (M)
Sep09–Aug10
TOTAL
High Need Districts
Boston
Brockton
Fall River
Falmouth
Plymouth
Quincy
Seekonk
Swansea
Freetown/Lakeville
Subtotal
Easthampton
Gateway
Holyoke
Springfield
Subtotal
Chicopee
Holyoke
Springfield
Westfield
Subtotal
Boston
Lynn
Malden
Pioneer Charter School
of Science
Randolph
Revere
Somerville
Subtotal
Randolph
Springfield
Boston
Cambridge
Falmouth
Haverhill
Lawrence
Lowell
Salem
Somerville
Woburn
Worcester
Subtotal
n
%
n
40
21
9
0
0
0
0
0
0
30
1
6
2
8
17
1
9
35
4
49
26
3
0
100%
174
28
12
1
3
2
1
1
3
51
3
7
9
0
19
1
6
59
5
71
26
4
1
59%
39%
94%
%
100%
45%
38%
84%
n*
195
49
21
1
3
2
1
1
3
81
4
11
11
2
28
1
10
70
5
86
46
6
1
1
0
1
1
0
0
31
6
26
3
0
2
2
7
0
1
0
0
1
16
1
8
12
52
18
30
3
1
0
0
1
2
1
1
1
0
10
1
8
12
75
19
43
6
1
2
2
8
2
2
1
1
1
26
50%
25%
96%
39%
46%
41%
100%
29%
%
100%
49%
41%
85%
49%
34%
100%
38%
*Note that if this column is summed, the total will not correspond to data in Table 4 because 1) Table 4 presents data for unique participants in public schools
across all partnerships while Table 5 presents data for all unique participants within partnerships (and some participants took courses across partnerships) and 2)
The districts of some of those participants who crossed partnerships were not consistently classified as high need districts because of course content.
UMass Donahue Institute
Research and Evaluation Group
10
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 6: Reasons for Participation: Cohort 4, All Seats
Sep08-Aug09
Reasons for Participation
(Multiple responses permitted)
n
287
240
118
108
52
30
45
11
28
To increase knowledge in content
To obtain graduate credit
To pursue a personal interest
To earn PDPs for recertification
To get an additional license (certification)
To follow an administrator’s suggestion
To prepare for the Massachusetts Test for Educator Licensure (MTEL)
To obtain a first license (certification)
Other
Sep09-Aug10
%
of 391
course
seats
73%
61%
30%
28%
13%
8%
12%
3%
7%
TOTAL
n
%
of 839
course
seats
n
%
of 1,230
course
seats
603
565
239
217
123
105
74
29
37
72%
67%
29%
26%
15%
13%
9%
4%
4%
890
805
357
325
175
135
119
40
65
72%
65%
29%
26%
14%
11%
10%
3%
5%
Repeat Participation
Cohort 4 partnerships were successful at encouraging and retaining repeat participants. All eight partnerships
offered multiple courses, and all partnerships had participants who attended more than one course within Cohort
4. Of all unique Cohort 4 participants, 233 attended multiple courses within Cohort 4, which translated to 1,230
course seats filled for this cohort. Table 7 provides details regarding repeat participation, including information on
the 65 repeat participants who took courses from partnerships from previous MMSP cohorts. (Table 12 through
Table 19 also include data on repeat participation, as does Table 26.)
Table 7: Repeat Participants: Cohort 4 Partnerships
Total
Number of
Unique*
Participants
to Date
Number
Taking
Multiple
Courses in
Cohort 4
16
192
48
8
56
Brockton Public Schools (M)
9
163
3
22
25
Gateway Regional Sch Dis (S)
2
66
27
5
32
Lesley Springfield (S)
15
101
57
4
61
Greater North Shore (S)
19
150
60
15
75
Randolph Public Schools (S)
5
55
17
5
22
Springfield College (S)
4
44
14
3
17
Boston University (M)
4
69
7
3
10
74
840
233
65
298
Partnership
Boston Public Schools (M)
TOTAL
Number
of
Courses
Offered
to Date
Number Taking
One Course in
Cohort 4 & One or
More Courses in
Previous Cohorts
Number Taking
Multiple Courses in
Cohort 4 & Previous
Cohorts
* Participants who participated in multiple courses across partnerships were counted only once in the partnership of their most recent course.
UMass Donahue Institute
Research and Evaluation Group
11
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Attrition
For Cohort 4, the course attrition rate over all courses over all partnerships averaged 4%. Of the 74 Cohort 4
courses delivered, data on attrition rates were available for 73. Of the 73 courses for which data were available, 56
(77%) had an attrition rate of 0%, eight (11%) had an attrition rate ranging between 1% and 10%, and nine (12%)
had an attrition rate of greater than 10%. Table 8 provides a breakdown, by partnership, of enrollment and
attrition rates. Appendix E provides a breakdown, by course, of enrollment and attrition rates.
Table 8: Total Enrollment and Attrition Information: Cohort 4 Partnerships
Partnership
Number of Courses
Offered
Sep08Aug09
Sep09Aug10
Number of
Participants
Enrolled First Day
Number of
Participants
Completed Course
Sep08Aug09
Sep08Aug09
Sep09Aug10
Sep09Aug10
Attrition Rate
Sep08Aug09
Sep09Aug10
Boston PS (M)
Brockton PS (M)
Gateway Regional SD (S)
Lesley University C4 (M)
Greater North Shore (S)
Randolph PS (S)
Springfield College (S)
Boston University (M)
2
3
1
5
11
2
2
2
14
6
1
10
8
3
2
2
202
52
47
74
95
28
27
69
235
122
55
188
190
62
30
35
184
51
47
72
89
28
27
67
235
116
49
185
177
61
30
35
9%
2%
0%
3%
6%
0%
0%
3%
0%
5%
11%
2%
7%
2%
0%
0%
All Courses/Partnerships
28
46
594
917
565*
888
5%
3%
* This value exceeds the number of participants for 2008-2009 in Table 12 of this report because this partnership did not administer Partnership Background Surveys to 144 of
its participants.
Highly Qualified Status
For Cohort 4 by the end of the 2009-2010 funding period, 24 unique participants attained highly qualified status.
Of the 24 who attained highly qualified status, four did so by passing the appropriate MTEL, one did so by
earning a teaching license, six did so by changing their positions, and 13 did so through a combination of means.
Additionally, it is likely that MMSP participation contributed to the attainment of highly qualified status for more
participants after the 2009-2010 period had ended, but it was not possible to capture those gains during this
reporting period.
Of those who attained highly qualified status, eight were from the Greater North Shore Northeastern (S)
partnership, six were from the Boston Public Schools (M) partnership, five were from the Gateway (S)
partnership, three were from the Randolph (S) partnership, and one each were from the Lesley (M) and Boston
University (M) partnerships. An identification of the criteria by which teachers who gained highly qualified status
could demonstrate competency in their subject matter may be found in Appendix F.
Licensure and Degrees Held in Subjects Taught
By the end of the 2009-2010 funding period for Cohort 4, 230 regular education teachers, 36 special education
teachers, and 15 ELL teachers reported teaching subjects in science or technology/engineering. Tables 9a, 9b, and
9c show how many teachers taught each individual science or technology/engineering subject from Cohort 4
through the 2009-2010 funding period. The tables also show the percentages of teachers whose survey responses
indicated they were licensed in the subject they taught, and they show the percentages of teachers whose survey
responses indicated they held a degree in the subject they taught. Table 9a refers to regular education teachers,
Table 9b refers to special education teachers, and Table 9c refers to ELL teachers. (As some teachers taught more
than one subject, numbers in these tables exceed the number of teachers who reported teaching these subjects.)
UMass Donahue Institute
Research and Evaluation Group
12
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 9a: Cohort 4 Science and Technology Teaching Areas – Regular Education
Sep08-Aug09
Subject
n
Degree in
Subject
Taught
%*
81
25
17
16
20
14
91
72
65
44
45
36
15
60
35
6
10
14
Number of
Teachers
General Science
Biology
Chemistry
Physics
Earth Science
Technology/Engineering
Sep09-Aug10
License in
Subject
Taught
%*
n
License in
Subject
Taught
%*
98
42
28
28
16
7
71
74
68
46
13
14
Number of
Teachers
Total
Degree in
Subject
Taught
%*
13
67
57
21
13
14
Number of
Teachers
n
139
55
40
38
29
16
License in
Subject
Taught
%*
76
73
65
42
21
31
Degree in
Subject
Taught
%*
13
64
48
18
14
19
*Of the number of regular education participants teaching this subject for this period
Table 9b: Cohort 4 Science and Technology Teaching Areas – Special Education
Sep08-Aug09
n
General Science
Biology
Chemistry
Physics
Earth Science
12
1
0
1
2
0
58
100
0
100
50
0
0
0
0
0
0
18
5
1
4
5
78
40
0
25
20
6
0
0
0
0
Technology/Engineering
0
3
33
0
Number of
Teachers
n
License in
Subject
Taught
%*
Total
Degree in
Subject
Taught
%*
Number of
Teachers
Subject
Sep09-Aug10
License in
Subject
Taught
%*
Degree in
Subject
Taught
%*
License in
Subject
Taught
%*
Degree in
Subject
Taught
%*
25
6
1
5
6
76
50
0
40
33
4
0
0
0
0
3
33
0
Number of
Teachers
n
*Of the number of special education participants teaching this subject for this period
Table 9c: Cohort 4 Science and Technology Teaching Areas – ELL Education
Sep08-Aug09
n
Degree in
Subject
Taught
%*
5
1
0
0
0
0
80
0
0
0
0
0
0
0
0
0
0
0
Number of
Teachers
Subject
General Science
Biology
Chemistry
Physics
Earth Science
Technology/Engineering
Sep09-Aug10
License in
Subject
Taught
%*
n
License in
Subject
Taught
%*
4
3
2
2
1
1
50
33
0
0
0
0
Number of
Teachers
Total
Degree in
Subject
Taught
%*
50
33
0
0
0
0
n
License in
Subject
Taught
%*
8
4
2
2
1
1
75
25
0
0
0
0
Number of
Teachers
Degree in
Subject
Taught
%*
25
25
0
0
0
0
*Of the number of ELL education participants teaching this subject for this period
Across all science subject areas for regular education teachers, the licensing reported appeared to be appropriate
for approximately 63% of the science subjects taught, and the degrees held corresponded to the science subject
UMass Donahue Institute
Research and Evaluation Group
13
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
taught for approximately 27% of the science subjects taught. Across all science subject areas for special education
teachers, the licensing reported appeared to be appropriate for approximately 59% of the science subjects taught,
and one teacher held a degree that corresponded to the subject taught. Across all science subject areas for ELL
teachers, the licensing reported appeared to be appropriate for 39% of the subjects taught, and the degrees held
corresponded to the science subject taught for approximately 17% of the science subjects taught.
To see the extent to which licensure and degree were appropriate for a particular type of teacher for a particular
subject, look in the relevant table at data in the row corresponding to the subject of interest. For example, to see
the extent to which licensure and degree were appropriate for just regular education biology teachers in 20082009, look at Table 9a in the row for the subject area “Biology” to learn that 25 teachers taught regular education
biology and that of those 25, 72% were licensed in biology and 60% held degrees in biology.
By the end of the 2009-2010 funding period for Cohort 4, 211 regular education teachers, 85 special education
teachers, and 17 ELL teachers reported teaching mathematics. Tables 10a, 10b, and 10c show how many Cohort 4
teachers taught at each mathematics level through the 2009-2010 funding period. The tables also show the
percentages of teachers whose survey responses indicated they were licensed in the mathematics level at which
they taught, and they show the percentages of teachers whose survey responses indicated they held a degree in
mathematics. Table 10a provides information for regular education mathematics teachers, Table 10b provides
information for special education mathematics teachers, and Table 10c provides information for ELL mathematics
teachers.
From an “ease-of-reading perspective,” it would be ideal merely to report on the percentages of teachers who
were licensed to teach mathematics and the percentage of teachers who held mathematics degrees. Sometimes,
though, teachers held multiple positions and were only licensed to teach some of what they are teaching, and what
is really desired is to understand the extent to which what is being taught is done so by individuals licensed to
teach it and by individuals who held relevant degrees. As a result, the explanations of data that follow are done so
in terms of teaching positions.
For MMSP mathematics teaching positions in regular education, the licensing appeared to be appropriate for
approximately 81% of positions held, and 17% of the positions were held by individuals with mathematics
degrees. For special education mathematics teachers, the licensing appeared to be appropriate for approximately
46% of the positions held, and 7% of the positions were held by individuals with mathematics degrees. For ELL
mathematics teachers, the licensing appeared to be appropriate for 65% of the positions held, and 18% of the
positions were held by individuals with mathematics degrees.
To see the extent to which licensure and degree were appropriate for a particular type of teacher for a particular
mathematics level taught, look in the relevant table at data in the row corresponding to the level of interest and in
the column corresponding to the time period of interest. For example, to see the extent to which licensure and
degree were appropriate for just regular education middle school math teachers in 2008-2009, look at Table 10a in
the row for the level “Middle School” to learn that 60 teachers taught regular education middle school
mathematics and that of those 60, 87% were licensed to teach middle school mathematics and 8% held
mathematics degrees.
UMass Donahue Institute
Research and Evaluation Group
14
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 10a: Cohort 4 Mathematics Teacher Levels – Regular Education
Sep08-Aug09
Number
of
Teachers
Licensed
at Level
n
Elementary School
Middle School
High School
Middle & High School
TOTAL Math
Level
Sep09-Aug10
Degree in
Math
Number
of
Teachers
Licensed
at Level
%*
%*
n
%*
17
60
17
0
65
87
82
0
0
8
41
0
44
89
19
0
73
84
84
0
94
82
13
152
81
Total
Degree in
Math
Number
of
Teachers
Licensed
at Level
Degree in
Math
%*
n
%*
%*
0
21
42
0
53
126
32
0
72
84
84
0
0
18
38
0
18
211
81
17
*Of the number of regular education participants teaching mathematics at this level for this period
Table 10b: Cohort 4 Mathematics Teacher Levels – Special Education
Sep08-Aug09
Number
of
Teachers
Level
Sep09-Aug10
Licensed
at Level
Degree in
Math
%*
%*
n
Total
Number
of
Teachers
Licensed
at Level
Degree in
Math
n
%*
%*
Number
of
Teachers
n
Licensed
at Level
Degree in
Math
%*
%*
Elementary School
Middle School
High School
Middle & High School
3
17
5
0
100
53
40
0
0
12
0
0
3
53
11
2
33
42
46
50
0
0
27
50
5
65
13
2
60
45
46
50
0
3
23
50
TOTAL Math
25
56
8
69
42
6
85
46
7
*Of the number of special education participants teaching mathematics at this level for this period
Table 10c: Cohort 4 Mathematics Teacher Levels – ELL Education
Sep08-Aug09
Number
of
Teachers
Level
Sep09-Aug10
Licensed
at Level
Degree in
Math
%*
n
Total
Number
of
Teachers
Licensed
at Level
Degree in
Math
%*
%*
Number
of
Teachers
n
Licensed
at Level
Degree in
Math
%*
%*
%*
n
Elementary School
Middle School
High School
Middle & High School
1
4
0
0
100
50
0
0
0
0
0
0
1
7
4
1
0
43
100
100
0
0
75
0
2
10
4
1
50
50
100
100
0
0
75
0
TOTAL Math
5
60
0
13
73
27
17
65
18
*Of the number of ELL education participants teaching mathematics at this level for this period
UMass Donahue Institute
Research and Evaluation Group
15
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Degrees Currently Pursued
Information on degrees currently being pursued in science, technology/engineering, and mathematics was derived
from the most recently completed survey of each individual. Of the 230 regular education teachers, 36 special
education teachers, and 15 ELL teachers who reported teaching subjects in science or technology/engineering in
Cohort 4 through the 2009-2010 funding period, 24 reported that they were pursuing degrees in science and
technology/engineering. Of these 24 teachers, 16 were pursuing advanced degrees (master’s degrees and CAGS
degrees) in their current area of teaching: Five biology teachers each were pursuing master’s degrees in biology,
ten general science teachers each were pursuing master’s degrees in general science, and one general science
teacher was pursuing a CAGS in general science. Four participants were pursuing degrees that did not directly
correspond to their current areas of teaching: One biology teacher was pursuing a master’s degree in general
science, one general science teacher was pursuing a master’s degree in biology, another general science teacher
was pursuing a bachelor’s degree in technology/engineering, and a technology/engineering teacher was pursuing a
master’s degree in general science. Three participants who each taught multiple science and technology/
engineering subjects each were pursuing multiple degrees: One was pursuing a master’s degree in chemistry;
another was pursuing a master’s degree in general science plus bachelor’s degrees in biology and chemistry; and
the third participant who taught multiple science and technology/engineering subjects was pursuing CAGS
degrees in general science, physics, and technology/engineering. The final participant (of the 24 who were
pursuing degrees in the area of science and technology/engineering) was an elementary teacher who was pursuing
a master’s degree in general science.
Of the 211 regular education teachers 85 special education teachers, and 17 ELL teachers who reported teaching
mathematics in Cohort 4 through the 2009-2010 funding period, ten were pursuing mathematics degrees. The
levels at which these teachers taught mathematics were as follows: three taught at the elementary level, five taught
middle school, and two taught high school. Of all ten, eight were pursuing master’s degrees in mathematics and
two were pursuing bachelor’s degrees in mathematics.
Content Knowledge Gains
As a grant condition, partnerships were required to use a pre-course test and post-course test for each MMSP
course to assess participants’ knowledge growth of the content taught in the course. For each course, the same
instrument served as both the pre- and post-course test. While partnerships were permitted to locate and use an
instrument with established reliability and validity, it was most often the case that such instruments were not
available because a priority was placed on utilizing assessments that would reflect the precise content taught in
each course. As a result, the faculty members who had developed the courses usually developed the assessments,
and typically, neither validity nor reliability was determined for them because time and resource constraints
prohibited doing so.
Data were analyzed only for participants who completed both pre- and post-course assessments. To determine if
the scores showed statistically significantly changes between pre- and post-course test administrations, a paired
samples t-test was used for each course for which ten or more participants completed both pre- and post-course
assessments, and a Wilcoxon Matched Pairs test was used for each course for which fewer than ten participants
completed both pre- and post-course assessments.
Content assessments were administered for 70 of the 74 courses offered through Cohort 4. Gains in average
percentage of items correct between pre- and post-course test administrations occurred in 65 of the 70 courses
delivered across all Cohort 4 partnerships through the 2009-2010 funding period. Statistically significant
improvements in scores on content knowledge assessments occurred in 49 courses, which translates to 66% of all
74 courses delivered. Of the 21 courses not showing statistically significant improvement in scores, though,
UMass Donahue Institute
Research and Evaluation Group
16
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
fifteen had fewer than six participants complete content assessments2, and six is the smallest sample size at which
it is possible to detect statistical significance at the level used for these analyses. Table 11 provides an overview,
by subject matter of courses delivered, of the numbers of courses that showed statistically significant gains. For
detailed information on mean pre- and post-course content knowledge scores by partnership and by course, see
Appendix G.
Table 11: Number of Courses with Statistically Significant Gains in Mean Content
Knowledge Scores for Cohort 4
Sep08-Aug09
Content Area
Math
Science &
Technology/Engineering
TOTAL
Delivered,
with Preand Posttests
13
Sep09-Aug10
10
Delivered,
with Preand Posttests
29
14
7
27
17
Significant
Pre/Post
Gains
Total
18
Delivered,
with Preand Posttests
42
14
14
28
21
43
32
70
49
Significant
Pre/Post
Gains
Significant
Pre/Post
Gains
28
Partnership-level Participant Background Data
Presented in Table 12 through Table 19, this section offers an overview of selected participant survey data for
each Cohort 4 partnership. These data were collected at the end of each course that was offered by each
partnership. The responses to the survey questions are presented as frequencies and percentages of the total
number of participants in the partnership for the 2008-2009 and 2009-2010 periods and a cumulation of both. In
cases where not all participants responded to all of the items and in cases where multiple responses were
permitted, percentages presented may not total 100%.
The section of each table identified as “Participants Who Took Multiple Courses” contains two categories. Each is
defined as follows: “In Cohort 4” refers to participants who took two or more courses within the context of Cohort
4—they participated in no courses that had been offered during previous cohorts. “In previous cohorts” refers to
participants who took one course during Cohort 4 and at least one additional course that had been offered during a
previous cohort.
The “Highly Qualified” section of each table includes the response option “Private school/Not applicable.”
Participants categorized under this option refer to those who either were teaching at a private school or for whom
highly qualified status would be irrelevant, such as school administrators.
2
Some of these courses had fewer than six MMSP enrollees within the context of another course that included additional non-MMSP
enrollees; other courses were not able to administer content assessments to all participants because of logistical challenges.
UMass Donahue Institute
Research and Evaluation Group
17
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 12: Boston Public Schools Participant Background Information (M)
Number of
Participants
Sep08 – Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In previous cohorts
Teach Regular Education
Teach Special Education (sole and inclusion)
Teach ELL, ESL, or Sheltered English Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
40*
0
1
24
15
1
20
5
18
4
40
27
12
1
0
0
24
40
(0%)
(3%)
(60%)
(38%)
(3%)
(50%)
(13%)
(45%)
(10%)
(100%)
(68%)
(30%)
(3%)
(0%)
(0%)
(60%)
(100%)
Number of
Participants
Sep09 – Aug10
167
51
7
98
53
10
93
22
53
17
167
97
60
4
3
3
87
161
(31%)
(4%)
(59%)
(32%)
(6%)
(56%)
(13%)
(32%)
(10%)
(100%)
(58%)
(36%)
(2%)
(2%)
(2%)
(52%)
(96%)
Number of
Participants
TOTAL
195
51 (26%)
8
(4%)
115 (59%)
(59%)
63 (32%)
11
(6%)
106 (54%)
24 (12%)
67 (34%)
20 (10%)
195 (100%)
118 (61%)
67 (34%)
4
(2%)
3
(2%)
3
(2%)
107 (55%)
189 (97%)
* This value underreports by 144 the number of participants who completed courses for this partnership in 2008-2009. For this year, this partnership only administered
Partnership Background Surveys to 40 of its 184 participants, and data from those surveys are reported here.
UMass Donahue Institute
Research and Evaluation Group
18
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 13: Brockton Public Schools Participant Background Information (M)
Number of
Participants
Sep08 – Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In previous cohorts
Teach Regular Education
Teach Special Education (sole and inclusion)
Teach ELL, ESL, or Sheltered English Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
UMass Donahue Institute
Research and Evaluation Group
51
6
1
38
10
1
29
5
15
5
30
30
16
2
0
3
29
50
(12%)
(2%)
(75%)
(20%)
(2%)
(57%)
(10%)
(29%)
(10%)
(59%)
(59%)
(31%)
(4%)
(0%)
(6%)
(57%)
(98%)
Number of
Participants
Sep09 – Aug10
114
4
16
77
25
8
43
19
51
12
51
62
41
2
9
0
60
109
(4%)
(14%)
(68%)
(22%)
(7%)
(38%)
(17%)
(45%)
(11%)
(45%)
(54%)
(36%)
(2%)
(8%)
(0%)
(53%)
(96%)
Number of
Participants
TOTAL
165
4
23
115
35
9
72
24
66
17
81
94
59
2
9
1
89
159
(2%)
(14%)
(70%)
(21%)
(6%)
(44%)
(15%)
(40%)
(10%)
(49%)
(57%)
(36%)
(1%)
(6%)
(1%)
(54%)
(96%)
19
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 14: Gateway Regional School District Participant Background Information (S)
Number of
Participants
Sep08 – Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In previous cohorts
Teach Regular Education
Teach Special Education (sole and inclusion)
Teach ELL, ESL, or Sheltered English Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
UMass Donahue Institute
Research and Evaluation Group
46
2
6
43
2
1
4
0
6
39
17
19
19
6
2
0
20
45
(4%)
(13%)
94%
4%
2%
9%
0%
13%
(85%)
(37%)
(41%)
(41%)
(13%)
(4%)
(0%)
(44%)
(98%)
Number of
Participants
Sep09 – Aug10
50
28
3
47
1
1
10
0
2
40
19
21
18
10
1
0
26
48
(56%)
(6%)
(94%)
(2%)
(2%)
(20%)
(0%)
(4%)
(80%)
(38%)
(42%)
(36%)
(20%)
(2%)
(0%)
(52%)
(96%)
Number of
Participants
TOTAL
68
29
5
62
3
2
13
0
5
52
28
30
26
10
2
0
36
65
(43%)
(7%)
(91%)
(4%)
(3%)
(19%)
(0%)
(7%)
(77%)
(41%)
(44%)
(38%)
(15%)
(3%)
(0%)
(53%)
(96%)
20
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 15: Lesley University C4 Participant Background Information (M)
Number of
Participants
Sep08 – Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In previous cohorts
Teach Regular Education
Teach Special Education (sole and inclusion)
Teach ELL, ESL, or Sheltered English Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
UMass Donahue Institute
Research and Evaluation Group
52
22
1
40
10
2
15
7
28
4
49
28
17
2
4
1
29
49
(42%)
(2%)
(77%)
(19%)
(4%)
(29%)
(14%)
(54%)
(8%)
(94%)
(54%)
(33%)
(4%)
(8%)
(2%)
(56%)
(94%)
Number of
Participants
Sep09 – Aug10
85
57
2
51
24
4
23
10
47
3
71
47
28
1
4
5
46
74
(67%)
(2%)
(60%)
(28%)
(5%)
(27%)
(12%)
(55%)
(4%)
(84%)
(55%)
(33%)
(1%)
(5%)
(6%)
(54%)
(87%)
Number of
Participants
TOTAL
101
57
3
61
29
5
27
12
54
4
86
54
34
1
6
6
53
90
(56%)
(3%)
(60%)
(29%)
(5%)
(27%)
(12%)
(54%)
(4%)
(85%)
(54%)
(34%)
(1%)
(6%)
(6%)
(53%)
(89%)
21
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 16: Greater North Shore Participant Background Information (S)
Number of
Participants
Sep08 – Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In previous cohorts
Teach Regular Education
Teach Special Education (sole and inclusion)
Teach ELL, ESL, or Sheltered English Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
UMass Donahue Institute
Research and Evaluation Group
62
26
3
54
5
2
2
0
4
54
31
36
13
8
4
1
36
59
(42%)
(5%)
(87%)
(8%)
(3%)
(3%)
(0%)
(7%)
(87%)
(50%)
(58%)
(21%)
(13%)
(7%)
(2%)
(58%)
(95%)
Number of
Participants
Sep09 – Aug10
113
51
10
95
10
5
5
0
6
90
52
63
24
10
9
7
73
105
(45%)
(9%)
(84%)
(9%)
(4%)
(4%)
(0%)
(5%)
(80%)
(46%)
(56%)
(21%)
(9%)
(8%)
(6%)
(65%)
(93%)
Number of
Participants
TOTAL
152
61
13
130
12
7
7
0
9
124
75
85
36
10
12
9
95
141
(40%)
(9%)
(86%)
(8%)
(5%)
(5%)
(0%)
(6%)
(82%)
(49%)
(56%)
(24%)
(7%)
(8%)
(6%)
(63%)
(93%)
22
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 17: Randolph Public Schools Participant Background Information (S)
Number of
Participants
Sep08 – Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In previous cohorts
Teach Regular Education
Teach Special Education (sole and inclusion)
Teach ELL, ESL, or Sheltered English Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
UMass Donahue Institute
Research and Evaluation Group
24
5
0
20
3
0
10
2
2
10
6
14
5
3
1
1
15
24
(21%)
(0%)
(83%)
(13%)
(0%)
(42%)
(8%)
(8%)
(42%)
(25%)
(58%)
(21%)
(13%)
(4%)
(4%)
(63%)
(100%)
Number of
Participants
Sep09 – Aug10
44
17
4
38
4
0
17
1
0
23
18
27
13
2
0
2
29
43
(39%)
(9%)
(86%)
(9%)
(0%)
(39%)
(2%)
(0%)
(52%)
(41%)
(61%)
(30%)
(5%)
(0%)
(5%)
(66%)
(98%)
Number of
Participants
TOTAL
56
18
4
49
5
0
23
2
1
27
19
32
19
2
1
2
35
55
(32%)
(7%)
(88%)
(9%)
(0%)
(41%)
(4%)
(2%)
(48%)
(34%)
(57%)
(34%)
(4%)
(2%)
(4%)
(63%)
(98%)
23
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 18: Springfield College Participant Background Information (S)
Number of
Participants
Sep08 – Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In previous cohorts
Teach Regular Education
Teach Special Education (sole and inclusion)
Teach ELL, ESL, or Sheltered English Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
UMass Donahue Institute
Research and Evaluation Group
27
3
7
22
4
1
1
0
2
19
26
12
9
3
3
0
12
25
(11%)
(26%)
(82%)
(15%)
(4%)
(4%)
(0%)
(7%)
(70%)
(96%)
(44%)
(33%)
(11%)
(11%)
(0%)
(44%)
(93%)
Number of
Participants
Sep09 – Aug10
30
13
3
20
3
2
1
0
3
27
30
12
13
4
1
0
14
30
(43%)
(10%)
(67%)
(10%)
(7%)
(3%)
(0%)
(10%)
(90%)
(100%)
(40%)
(43%)
(13%)
(3%)
(0%)
(47%)
(100%)
Number of
Participants
TOTAL
44
14
3
30
6
3
2
0
5
38
43
19
19
4
2
0
21
43
(32%)
(7%)
(68%)
(14%)
(7%)
(5%)
(0%)
(11%)
(86%)
(98%)
(43%)
(43%)
(9%)
(5%)
(0%)
(48%)
(98%)
24
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 19: Boston University Participant Background Information (M)
Number of
Participants
Sep08 – Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In previous cohorts
Teach Regular Education
Teach Special Education (sole and inclusion)
Teach ELL, ESL, or Sheltered English Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
UMass Donahue Institute
Research and Evaluation Group
41
0
1
33
3
2
6
2
27
5
16
23
8
1
1
8
26
36
(0%)
(2%)
(81%)
(7%)
(5%)
(15%)
(5%)
(66%)
(12%)
(39%)
(56%)
(20%)
(2%)
(2%)
(20%)
(63%)
(88%)
Number of
Participants
Sep09 – Aug10
34
7
2
26
5
4
6
4
26
3
10
19
8
0
0
7
19
31
(21%)
(6%)
(77%)
(15%)
(12%)
(18%)
(12%)
(77%)
(9%)
(29%)
(56%)
(24%)
(0%)
(0%)
(21%)
(56%)
(91%)
Number of
Participants
TOTAL
69
7
3
53
8
6
12
6
47
8
26
39
16
0
1
13
42
62
(10%)
(4%)
(77%)
(12%)
(9%)
(17%)
(9%)
(68%)
(12%)
(38%)
(57%)
(23%)
(0%)
(1%)
(19%)
(61%)
(90%)
25
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Cumulative Summary: All Cohorts over All Funding Periods
Overview of Partnerships, Budgets, Courses, and Participants
Table 20 provides an overview of partnership projects since the inception of the program.
Table 20: Overview of MMSP Partnership Participation
Funding Period
MMSP Year 1
Feb04-Aug04
Partnership Grouping
Cohort 1
Cohort 2
Cohort 3
Cohort 4
UMass Donahue Institute
Research and Evaluation
Group
MMSP Year 2
Sep04-Aug05
MMSP Year 3
Sep05-Aug06
MMSP Year 4
Sep06-Aug07
Grant Year 1
Grant Year 2
Grant Year 3
EduTron/Fitchburg-M
Harvard-M
Lesley-M
MCLA-S
Salem-M
Springfield PS-S
Wareham PS-M
WPI-M
EduTron/Fitchburg-M
Harvard-M
Lesley-M
MCLA-S
Salem-M
Springfield PS-S
Wareham PS-M
WPI-M
EduTron/Fitchburg-M
Harvard-M
Lesley-M
MCLA-S
Salem-M
Springfield PS-S
Wareham PS-M
WPI-M
Grant Year 1
Grant Year 2
Grant Year 3
MCLA-M
PV STEMNET-M
MCLA-M
PV STEMNET-M
MCLA-M
PV STEMNET-M
MMSP Year 5
Sep07-Aug08
MMSP Year 6
Sep08-Aug09
MMSP Year 7
Sep09-Aug10
Grant
Year 3
Extension
Grant Year 1
Grant Year 2
Grant Year 3
EduTron/Lowell-M/S
EduTron/Fitchburg-M
Lesley-M
North Shore-S
PV STEMNET-M/S
Salem-M
SE/Cape-S
WPI-S
WPS-M
EduTron/Lowell-M/S
EduTron/Fitchburg-M
Lesley-M
North Shore-S
PV STEMNET-M/S
Salem-M
SE/Cape-S
WPI-S
EduTron/Lowell-M/S
EduTron/Fitchburg-M
Lesley-M
North Shore-S
PV STEMNET-M/S
Salem-M (no courses)
SE/Cape-S
WPI-S
Grant Year 1
Grant Year 2
Boston PS-M
Brockton PS-M
Gateway RSD-S
Lesley-M
Northeastern-S
Randolph PS-S
Springfield College-S
Boston U. Trustees-M
Boston PS-M
Brockton PS-M
Gateway RSD-S
Lesley-M
Northeastern-S
Randolph PS-S
Springfield College-S
Boston U. Trustees-M
26
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Cohorts 1, 2, 3, and 4 combined consisted of 27 partnerships, with 16 of the 27 partnerships offering mathematics
professional development, nine offering science professional development, and two offering professional
development in both mathematics and science content. Specifically, Cohort 1 consisted of eight partnerships, with
six of the eight partnerships offering mathematics professional development and two offering science professional
development. Cohort 2 consisted of two partnerships, both offering mathematics professional development.
Cohort 3 consisted of nine partnerships, with four of the nine partnerships offering mathematics professional
development, three offering science professional development, and two offering professional development in both
mathematics and science. Cohort 4 consisted of eight partnerships, with four of the eight partnerships offering
mathematics professional development and four offering science professional development.
Table 21 shows the funding received by each partnership since the inception of the program. While some
partnerships were awarded funding in more than one funding period, for evaluation purposes, a partnership was
identified as a “new” partnership each time it received funding that resulted from a different competition. Overall,
partnerships have been awarded a total of $13,351,031 since the inception of MMSP.
UMass Donahue Institute
Research and Evaluation Group
27
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 21: Budgets: All Partnerships, All Funding Periods
Feb04Aug06
Sep06Aug07
EduTron (M)
$770,000
$68,352
$838,352
Harvard Graduate School of Ed. (M)
$489,899
$87,425
$577,324
Lesley University (M)
$810,726
$43,838
$854,564
MCLA – Science (S)
$133,192
$38,247
$171,439
Salem State College (M)
$541,995
$43,648
$585,643
Springfield Public Schools (S)
$500,044
$74,737
$574,781
Wareham Public Schools (M)
$398,440
$43,962
$442,402
$601,778
$35,633
$637,411
Partnership
Sep07Aug08
Sep08Aug09
Sep09Aug10
TOTAL
COHORT 1
Initially funded February 2004
Worcester Polytechnic Institute (M)
COHORT 2
Initially funded September 2004
MCLA – Math (M)
$111,494
$51,874
$163,368
UMass Amherst (M)
$262,415
$181,581
$443,996
COHORT 3
Initially funded September 2006
EduTron Lowell (M/S)
$210,000
$220,000
$240,000
$670,000
EduTron Fitchburg (M)
$102,000
$110,000
$120,000
$332,000
Lesley University (M)
$347,911
$355,626
$355,357
North Shore (S)
$196,474
$194,729
$199,871
$1,058,89
4
$591,074
UMass Amherst (M/S)
$107,424
$216,281
$169,064
$492,769
Salem State College (M)
$120,882
$113,551
$36,604
$271,037
SE/Cape (S)
$129,438
$181,420
$169,246
$480,104
$99,586
$70,734
$94,852
$265,172
Worcester Polytechnic Institute (S)
Worcester Public Schools (M)
$231,210
$231,210
COHORT 4
Initially funded September 2008
Boston Public Schools (M)
$157,975
$405,747
$563,722
Brockton Public Schools (M)
$180,145
$255,758
$435,903
Gateway Regional School Dist (S)
$186,609
$200,370
$386,979
Lesley Springfield (M)
$228,593
$324,820
$553,413
Greater North Shore (S)
$265,917
$306,690
$572,607
Randolph Public Schools (S)
$176,993
$183,150
$360,143
Springfield College (S)
$161,062
$148,896
$309,958
Boston University (M)
$241,586
$245,180
$486,766
$2,983,874
$2,070,611
$13,351,031
TOTAL
$4,619,983
UMass Donahue Institute
Research and Evaluation Group
$2,214,222
$1,462,341
28
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Over the span of MMSP, Cohort 1, 2, 3, and 4 partnerships developed and implemented a total of 298 courses. Of
those 298 courses, 155 (52%) were unique, and 143 (48%) were repeat offerings. Of the 298 courses, 201 (67%)
of the courses offered mathematics content, 95 (32%) offered science and/or technology/engineering content, and
two (<1%) offered both mathematics and science/technology content. This section of the report summarizes data
collected from participants in these courses. In total, there were 2,673 unique participants, and 989 of them took
two or more courses, which translated to 5,172 course seats. The term “unique participant” refers to each
individual who participated in the program, regardless of how many courses he or she took. Data for items from
the Participant Background Survey that help to convey participants’ professional backgrounds and motives for
participation are discussed in the remainder of this section. In addition, this section discusses information obtained
from partnerships on the extent to which courses offered through MMSP became institutionalized.
Position of Participants
At the time of their last completed survey from an MMSP course, 93% of course participants identified
themselves as teachers. Of all respondents, 75% were regular education teachers; 15% were special education or
special education inclusion teachers; 3% were ELL, ESL, or Sheltered English Immersion teachers; 2% were
department heads or curriculum coordinators; 1% were principals, assistant principals, or headmasters; <1% were
support specialists; <1% were paraprofessionals; <1% were long-term substitutes; <1% were superintendents or
assistant superintendents; and 3% indicated that they held “other” positions.
Content Taught
The distribution of teaching areas of respondents at the time of the survey is shown in Table 22. Because
respondents identified all teaching areas that applied to their positions at the time of the survey, some selected
multiple responses, so frequency totals exceed the number of respondents and percentages exceed 100%.
At the time of their last MMSP course, 46% were teaching mathematics (including at the elementary level), 28%
were teaching science, and 27% were teaching all subjects at the elementary level.
Table 22: Teaching Areas: All Participants, All Funding Periods
Total
Teaching Areas
(Multiple responses permitted)
Mathematics
Any science area
General Science
Biology
Earth Science
Chemistry
Physics
Technology/Engineering
Elementary (all subjects)
Elementary Mathematics
Other
Not Currently Teaching
UMass Donahue Institute
Research and Evaluation Group
N =2,673
n
%
1049
745
467
163
96
108
111
61
718
197
60
198
39%
28%
18%
6%
4%
4%
4%
2%
27%
7%
2%
7%
29
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Types of Schools of Participants
For each funding period of the program, at least 96% of MMSP participants worked in a public school setting.
Over the course of the program to date, 97% of MMSP participants worked in a public school setting, and 2%
worked in a non-public school setting. Table 23 provides a breakdown, by funding period, of the types of schools
in which participants worked.
Table 23. Types of Schools: All Unique Participants, All Funding Periods
School Type
Public Schools
(includes public
charter schools)
Non-public School
Other or No
Response
TOTAL
Feb04Aug04
Sep04Aug05
Sep05Aug06
Sep06Aug07
Sep07Aug08
Sep08Aug09
Sep09Aug10
n
%
n
%
n
%
n
%
n
%
n
%
n
332
97
448
98
455
98
533
96
462
97
769
98
614
8
2
7
2
6
1
12
2
7
2
15
2
1
<1
1
<1
3
1
8
<1
8
2
4
341
100
456
100
464
100
553
100
477
100
788
Total
%
n
%
98
2594
97
16
3
56
2
<1
0
0
23
<1
100
630
100
2673
100
High Need Status of Districts of Participants
MMSP partnerships were required to include at least one high need district. Appendix D identifies the criteria for
the high need designation. In addition, the ESE expected that at least 50% of participants in each partnership
would come from high need districts, and further, they set an informal goal that at least 75% of participants for
each partnership would come from high need districts. The high need status of some school districts changed
across years. To classify participants from districts that were high need at one point in time but not high need at
another, a process was used that took into account the high need status of participants’ districts from the
beginning of each partnership’s MMSP involvement with the program. A district identified as high need in the
first year of a partnership’s funding continued to be classified as high need in subsequent years of the partnership,
even if the district's status changed. Additionally, any districts not on the high need list in the first year of a
partnership’s funding but subsequently added to the high need list in later years of the partnership were then
identified as qualifying for high need district designation.
As a whole, across all years of funding, 66% of participants were from high need districts. Table 24 shows that
across all years of funding, 67% of the public school participants in the program as a whole had come from high
need districts and that for each year of funding, over 50% of public school participants in the program had come
from high need districts. An examination of high need district participation in individual partnerships reveals that
over the course of each partnership’s involvement in MMSP since the beginning of MMSP, 16 of the 27
partnerships had at least 50% of their participants coming from high need districts across all years of their
involvement, and 12 of 27 partnerships exceeded the informal goal of having at least 75% of the participants
come from high need districts (again, across all years of their involvement). If individual years of participation are
examined, it is seen that 13 of the 27 partnerships had at least 50% of their participants coming from high need
districts for each and every year of funding, and 10 of the 27 partnerships exceeded the informal goal of having at
least 75% of the participants come from high need districts for each and every year of funding. The table in
Appendix H shows the number of participants from high need districts organized by partnership.
UMass Donahue Institute
Research and Evaluation Group
30
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
As of their last course in MMSP, of the 989 individuals who took multiple courses, approximately 68% were from
high need public school districts, approximately 29% were from other public school districts, and approximately
2% either were from private schools or did not provide information on their districts. In addition, less than 1%
were from high need districts for some of the courses they took but not for others3.
Table 24: High Need Status of All Unique Participants from Public Schools
School Type
High Need District
Non-high Need
District
Other*
TOTAL
Feb04Aug04
Sep04Aug05
Sep05Aug06
Sep06Aug07
Sep07Aug08
Sep08Aug09
Sep09Aug10
n
%
n
%
n
%
n
%
n
%
n
%
n
202
61
276
62
254
56
397
74
335
73
544
69
412
67
1742
67
129
39
161
36
196
43
126
24
122
26
240
31
202
33
829
32
1
<1
11
2
5
1
10
2
5
4
<1
0
0
23
<1
332
100
448
100
455
100
447
100
462
788
100
614
100
2594
100
1
100
Total
%
n
%
*Includes those who did not identify their districts and public school participants who took multiple courses whose districts were considered high need for only some of the
courses those participants took.
Reasons for Participation in MMSP Courses
For each course taken, respondents identified all of their reasons for participating. Unlike in most other portions
of this report where data were presented for unique participants, data for this issue are presented for all course
seats, since responses are uniquely relevant to each distinct course that a participant took. Table 25 presents
findings for all 5,172 seats for all courses taken by all participants across all funding periods.
Table 25: Reasons for Participation of Participants: All Seats, All
Funding Periods
Total
Reasons for Participation
(Multiple responses permitted)
%
n
of 5,172 course
seats
To increase knowledge in content
3729
72%
To obtain graduate credit
3513
68%
To earn PDPs for recertification
1617
31%
To pursue a personal interest
1605
31%
To get an additional license (certification)
923
18%
To prepare for the Massachusetts Test for Educator Licensure (MTEL)
805
16%
To follow an administrator’s suggestion
386
7%
To earn PDPs for HOUSSE plan requirement
331
6%
To obtain a first license (certification)
160
3%
Other
299
6%
3
Teachers who took MMSP courses from a math partnership when their districts were considered high need for only science were
identified as having come from a “non-high need” district, and teachers who took MMSP courses from a science partnership when their
districts were considered high need for only math were identified as having come from a “non-high need” district.
UMass Donahue Institute
Research and Evaluation Group
31
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Repeat Participation
Partnerships were very successful at encouraging and retaining repeat participants. Of the 27 partnerships, all
offered multiple courses, and all had participants who attended more than one course. In all, 989 participants
(37% of all participants) attended multiple courses, which translated to 5,172 course seats. Of those participating
in multiple courses, 141 took courses across partnerships. Table 26 provides details regarding repeat participation.
Table 26: Repeat Participants: All Partnerships, All Funding Periods
Partnership
Number of
Courses
Delivered
to Date
Total
Number of
Unique
Participants
to Date
Number Taking
Multiple
Courses within
Cohorts
1&2
EduTron (M)
7
128
36
N/A
N/A
54
Harvard (M)
8
166
24
N/A
N/A
26
19
107
83
N/A
N/A
85
3
23
13
N/A
N/A
13
26
162
59
N/A
N/A
81
Springfield PS (S)
7
96
41
N/A
N/A
43
Wareham PS (M)
3
43
12
N/A
N/A
12
WPI (M)
6
145
47
N/A
N/A
48
MCLA (M)
4
16
9
N/A
N/A
9
UMass Amherst (M)
11
76
39
N/A
N/A
52
EduTron Lowell (M/S)
10
166
N/A
34
N/A
34
EduTron Fitchburg (M)
7
125
N/A
46
N/A
56
Lesley Univ. C3 (M)
40
198
N/A
124
N/A
124
North Shore (S)
30
121
N/A
53
N/A
52
UMass Amherst C3 (M/S)
14
135
N/A
36
N/A
46
Lesley Univ. C1 (M)
MCLA (S)
Salem State College (M)
Salem State C3 (M)
Number Taking
Multiple
Courses within
Cohort 3
Number Taking
Multiple
Courses within
Cohort 4
Total Number
Taking
Multiple
Courses
8
78
N/A
26
N/A
40
15
178
N/A
69
N/A
70
WPI (S)
3
44
N/A
5
N/A
5
Worcester PS(M)
3
41
N/A
6
N/A
5
16
192
N/A
N/A
Brockton PS (M)
9
163
N/A
Gateway RSD (S)
2
66
N/A
Lesley Springfield (M)
15
101
Greater North Shore (S)
19
150
Randolph PS (S)
5
Springfield Coll. (S)
Boston University (M)
SE/Cape (S)
Boston PS (M)
Across All Partnerships
48
56
N/A
3
25
N/A
27
32
N/A
N/A
57
61
N/A
N/A
60
75
55
N/A
N/A
17
22
4
44
N/A
N/A
14
17
4
69
N/A
N/A
7
10
298
2673
386
365
233
989
UMass Donahue Institute
Research and Evaluation Group
32
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Highly Qualified Status
To comply with the federal No Child Left Behind legislation, public school teachers were required to meet the
federal definition of highly qualified by the end of the 2005-2006 school year. One of the expectations of the
MMSP is that providing high quality professional development would contribute to teachers’ attainment of
federal highly qualified status.
Information regarding the following areas was used to determine participants’ highly qualified status: licensure,
years in education, subject areas taught, High Objective Uniform State Standard of Evaluation (HOUSSE) plans
held, Professional Development Points (PDPs) held, MTELs passed, degrees held, undergraduate degree
equivalents, and advanced or national certifications.
To be considered highly qualified, a teacher must be licensed and demonstrate subject matter competency in the
areas of teaching. Demonstration of subject matter competency for elementary teachers was satisfied either by
passing the appropriate MTEL (general curriculum MTEL if teaching multiple core subjects or elementary math
MTEL if teaching mainly elementary math) or by having a HOUSSE plan and having earned a particular number
of PDPs. (The HOUSSE plan option was phased out in 2007.) Demonstration of subject matter competency for
middle and secondary school teachers was satisfied by one of the following means: passing the appropriate
MTEL, completing an appropriate undergraduate major or graduate degree, completing appropriate coursework
comparable to an undergraduate major, holding advanced or national certification in the appropriate subject area,
or by having a HOUSSE plan and having earned a particular number of PDPs. For all teachers with HOUSSE
plans, the minimum numbers of PDPs needed varied in relation to the date of June 30, 2006: Prior to June 30,
2006, the minimum was 48, and following June 30, 2006, the minimum was 96. Appendix F outlines options
available for demonstrating subject matter competency. A participant was identified as highly qualified if the
criteria for meeting highly qualified status were met for all subjects that a participant taught. If a participant taught
multiple subject areas and only met the highly qualified criteria for some of the subjects taught, he or she was
determined to be “highly qualified in some, but not all” content areas.
In the first MMSP funding period, the Participant Background Survey did not adequately capture information
about teachers that could be used to determine highly qualified status. The survey was re-designed for Year 2 to
capture this information. For the second and subsequent funding periods, though, survey responses indicated that
participants misunderstood the meaning of HOUSSE plans, and some who completed more than one survey
reported inconsistent data across surveys.
The Participant Background Survey permitted determination of the impact of MMSP courses on highly qualified
status only for 1) those participants who held HOUSSE plans and 2) those participants who took more than one
course (i.e., completed more than one survey). Because surveys were administered after participants had
completed MMSP courses, if a participant completed only one course and did not have a HOUSSE plan it was not
possible to determine whether that participant became highly qualified prior to or as a result of MMSP course
participation.
Over the span of the program for all Cohorts, at least 158 participants attained highly qualified status. Table 27
presents the highly qualified status of participants across all years of the program. This table provides an
unduplicated count of participants. Because the number of courses a participant took was relevant to the process
used to determine how many participants attained highly qualified status while participating in MMSP, the
findings are organized according to number of courses taken (only one vs. multiple). In Appendix I, a more
detailed version of the table is presented in which the data are further broken down by cohort of participation.
UMass Donahue Institute
Research and Evaluation Group
33
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 27: Highly Qualified Status: All Unique Participants, All Funding
Periods
Status
Took Only One Course Took Multiple Courses
n = 1684
n = 989
TOTAL
n = 2673
Became Highly Qualified
14
144
158
Became Highly Qualified
in only some content
areas
0
0
5
855
608
1463
50
34
84
Not Highly Qualified
501
221
722
Private school or
not teaching
128
74
202
Unknown
150
52
202
Highly Qualified
Highly Qualified in some
content areas but not all
MTEL Information
One method by which teachers may demonstrate subject-matter competency is to pass the Massachusetts Tests for
Educator Licensure (MTEL) in the content areas that they teach. Table 28 cumulatively identifies the tests taken
by participants across all years of the program along with passage rates. Of the 989 participants taking multiple
courses, 26% earned one or more new licenses by the end of the seventh year of MMSP, and 12% took and passed
an MTEL test. In addition, over 1% of these participants learned that they had passed an MTEL that they had
taken during a prior MMSP course.
As shown in Table 28, based on data from the last survey completed by each participant, of the 243 participants
who had taken the Mathematics MTEL, 189 (78%) reported passing the test, and 19 (8%) had not yet received
their scores at the time of survey completion. Of the 437 respondents who had taken the Middle School
Mathematics MTEL, 372 (85%) passed and 29 (7%) had not yet received their scores. Of the 88 participants who
completed the Middle School Mathematics/Science MTEL, 53 (60%) passed and 15 (17%) had not yet received
their scores. Of the 216 participants completing the General Science MTEL, 186 (86%) passed and 19 (9%) had
not yet received their scores. Ninety-seven respondents attempted the Biology MTEL, and 89 (92%) passed and
three (3%) had not yet received scores. Of the 61 participants who took the Chemistry MTEL, 53 (87%) passed
and three (5%) had not yet received scores. Of the 22 who took the Earth Science MTEL, 17 (77%) passed and
two (9%) had not yet received scores. Thirty-seven participants attempted the Physics MTEL, and 28 (76%)
passed while three (1%) still awaited scores. Eleven individuals completed the Technology/Engineering MTEL,
and nine (82%) passed. Of the 421 participants who reported taking General Curriculum (formerly elementary)
MTEL, 409 (97%) reported passing, and six (1%) had not yet received scores.
UMass Donahue Institute
Research and Evaluation Group
34
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 28: MTEL Tests Taken by All Participants – Total to Date
Based on each participant’s last survey
Taking Test
Passing Test
Failing Test
n
n
%
421
409
97
6
1
6
1
Early Childhood
37
32
87
2
5
3
1
Elementary Mathematics
96
83
87
5
5
8
8
Mathematics
243
189
78
35
14
19
8
Middle School Mathematics
437
372
85
36
8
29
7
88
53
60
20
23
15
17
216
186
86
11
5
19
9
Biology
97
89
92
5
5
3
3
Chemistry
61
53
87
5
8
3
5
Physics
37
28
76
6
16
3
1
Earth Science
22
17
77
3
14
2
9
Technology/Engineering
11
9
82
2
18
0
0
532
435
82
10
45
8
General Curriculum (formerly
Elementary)
Middle School
Mathematics/Science
General Science
TOTAL in STE Areas
n
52
%
Scores
Unknown
n
%
Licensure and Degrees Held in Content Area Taught
Over the course of the program, 627 regular education teachers 81 special education teachers, and 27 ELL
teachers reported teaching in science or technology/engineering areas. Tables 29a, 29b, and 29c show how many
teachers taught in each science and technology/engineering area over the course of the program. The tables also
show the percentages of teachers whose survey responses indicated they were licensed in the area in which they
taught, and they show the percentages of teachers whose survey responses indicated they held a degree in the area
in which they taught. Table 29a provides information for regular education teachers, Table 29b provides
information for special education teachers, and Table 29c provides information for ELL teachers. (The numbers
presented in Tables 29a, 29b, and 29c exceed the number of teachers who reported teaching in these areas because
some teachers taught in more than one area.) For regular education teachers across all subject areas in science, the
licensing reported by approximately 57% appeared to be appropriate for the content area taught, and the degrees
held by approximately 25% corresponded to content area taught. For special education teachers across all subject
areas in science, the licensing reported by approximately 42% appeared to be appropriate for the content area
taught, and the degrees held by two teachers (2%) corresponded to content area taught. For ELL teachers across
all subject areas in science, the licensing reported by approximately 40% appeared to be appropriate for the
content area taught, and the degrees held by 26% corresponded to content area taught.
UMass Donahue Institute
Research and Evaluation Group
35
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 29a: Science and Tech/Engineering Teaching Areas of All
Participants – Regular Education
Total
Content Area
General Science
Biology
Chemistry
Earth Science
Physics
Technology/Engineering
Teach in Area
License in Area
Taught
n
%*
Degree in Area
Taught
%*
385
64%
13%
131
71%
62%
98
58%
40%
74
23%
14%
98
44%
19%
53
42%
13%
*Of the number of regular education participants teaching in this area across all program years
Table 29b: Science and Tech/Engineering Teaching Areas of All
Participants – Special Education
Content Area
General Science
Biology
Chemistry
Earth Science
Physics
Technology/Engineering
Teach in Area
Total
License in Area
Taught
Degree in Area
Taught
n
%*
%*
58
48%
3%
21
29%
0%
6
50%
0%
16
31%
0%
8
50%
0%
3
33%
0%
*Of the number of special education participants teaching in this area across all program years
Table 29c: Science and Tech/Engineering Teaching Areas of All
Participants – ELL Education
Content Area
General Science
Biology
Chemistry
Earth Science
Physics
Technology/Engineering
Teach in Area
Total
License in Area
Taught
Degree in Area
Taught
n
%*
%*
18
56%
17%
7
29%
43%
4
25%
25%
3
0%
33%
3
0%
33%
3
33%
0%
*Of the number of ELL participants teaching in this area across all program years
UMass Donahue Institute
Research and Evaluation Group
36
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Over the course of the program and by the end of the 2009-2010 funding period, 825 regular education teachers,
187 special education teachers, and 28 ELL teachers reported teaching non-elementary level mathematics. Tables
30a, 30b, and 30c show how many teachers taught at each non-elementary mathematics level over the course of
the program. The tables also show the percentages of teachers whose survey responses indicated they were
licensed in the level at which they taught, and they show the percentages of teachers whose survey responses
indicated they held a degree in the area in which they taught. Table 30a refers to regular education teachers, Table
30b refers to special education teachers, and Table 30c refers to ELL teachers. For regular education mathematics
teachers, the licensing reported by approximately 77% appeared to be appropriate for the mathematics level
taught, and 20% held mathematics degrees. For special education mathematics teachers, the licensing reported by
approximately 33% appeared to be appropriate for the level taught, and 4% held mathematics degrees. For ELL
mathematics teachers, the licensing reported by 50% appeared to be appropriate for the mathematics level taught,
and 14% held mathematics degrees.
Table 30a: Mathematics Teacher Levels of All Participants – Regular
Education
Level
Teach in Area
Total
License in Area
Taught
Degree in Math
n
%*
%*
Middle School
High School
MS & HS grades
685
131
9
77%
82%
78%
16%
42%
22%
TOTAL Math
825
77%
20%
*Of the number of regular education participants teaching in this area for this year
Table 30b: Mathematics Teacher Levels of All Participants – Special
Education
Level
Teach in Area
Total
License in Area
Taught
Degree in Math
n
%*
%*
Middle School
High School
MS & HS grades
153
29
5
33%
31%
40%
3%
10%
20%
TOTAL Math
187
33%
4%
*Of the number of special education participants teaching in this area for this year
Table 30c: Mathematics Teacher Levels of All Participants – ELL Education
Level
Teach in Area
Total
License in Area
Taught
Degree in Math
%*
n
%*
Middle School
High School
MS & HS grades
21
6
1
38%
83%
100%
0%
67%
0%
TOTAL Math
28
50%
14%
*Of the number of ELL participants teaching in this area for this year
UMass Donahue Institute
Research and Evaluation Group
37
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Degrees Currently Pursued
Information on degrees currently being pursued in science, technology/engineering, and mathematics was derived
from the most recently completed survey of each individual. Of the 627 regular education teachers, 81 special
education teachers, and 27 ELL teachers reported teaching in science or technology/engineering areas, 55 science
and technology/engineering teachers reported that they were pursuing degrees in science and technology/
engineering. Of these 55 teachers, 35 were pursuing science degrees in their current areas of teaching, with 33 of
them pursuing advanced degrees (master’s degrees and CAGS degrees). Of the 35 teachers pursuing science
degrees in their current areas of teaching, 26 were in the subject of general science, four were in biology, three
were in physics, and two were in chemistry. Nine teachers taught multiple science and technology/engineering
subjects and of these nine, seven were pursuing degrees that were relevant to at least one area that they were
teaching (three were pursuing master’s degrees, two were pursuing CAGS degrees, one was pursuing two
bachelor’s degrees plus a master’s degree, and one was pursuing two bachelor’s degrees). Of the remaining two
teachers who taught multiple science and technology/engineering subjects, one was pursuing two master’s
degrees (one of which was in a science subject she taught), and the other was pursuing a master’s degree in a
science area in which he did not currently teach. Ten teachers were pursuing science and technology/engineering
degrees in areas that were not directly relevant to the subjects they were currently teaching: Nine were pursuing
master’s degrees and one was pursuing a bachelor’s degree. The final teacher (of the 55 who were pursuing
degrees in the area of science and technology/engineering) was an elementary teacher who was pursuing a
master’s degree in general science.
Of the 825 regular education teachers, 187 special education teachers, and 28 ELL teachers reported teaching
mathematics over the course of the program, 27 were pursuing degrees in mathematics. The levels at which these
teachers taught mathematics were as follows: four taught at the elementary level, 15 taught middle school, seven
taught high school, and one taught both middle and high school. Of all 27, 24 were pursuing advanced degrees
(master’s degree or CAGS degree) and three were pursuing bachelor’s degrees. Of the 24 mathematics teachers
pursuing advanced degrees in mathematics, 22 were pursuing master’s degrees, one was pursuing a master’s
degree and a CAGS, and one was pursuing a CAGS.
Content Knowledge Gains
As a grant condition, partnerships were required to use a pre-course test and post-course test for each MMSP
course to assess participants’ knowledge growth of the content taught in the course. For each course, the same
instrument served as both the pre- and post-course test. While partnerships were permitted to locate and use an
instrument with established reliability and validity, it was most often the case that such instruments were not
available because a priority was placed on utilizing assessments that would reflect the precise content taught in
each course. As a result, the faculty members who had developed the courses usually developed the assessments,
and typically neither validity nor reliability was determined for them because time and resource constraints
prohibited doing so.
Data were analyzed only for participants who completed both pre- and post-course assessments. To determine if
the scores showed statistically significantly changes between pre- and post-course test administrations, a paired
samples t-test was used for each course for which ten or more participants completed both pre- and post-course
assessments, and a Wilcoxon Matched Pairs test was used for each course for which fewer than ten participants
completed both pre- and post-course assessments.
Of the 298 courses that were delivered across all partnerships through the 2009-2010 funding period, content
assessments were administered for 292 courses. Gains in average percentage of items correct between pre- and
post-course test administrations occurred in 285 of the courses delivered across all partnerships through the 20092010 funding period. Statistically significant improvements in scores on content knowledge assessments occurred
in 245 courses, which translates to 82% of all 298 courses delivered. Of the 47 courses not showing statistically
UMass Donahue Institute
Research and Evaluation Group
38
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
significant improvement in scores, though, 30 had fewer than six participants complete content assessments4, and
six is the smallest sample size at which it is possible to detect statistical significance at the level used for these
analyses. Table 31 provides an overview, by subject matter of courses delivered, of the numbers of courses that
showed statistically significant gains.
Table 31: Number of Courses with Statistically Significant
Gains in Mean Content Knowledge Scores
Total
Content Area
Math
Science & Technology/Engineering
Math and Science
TOTAL
Delivered,
with Pre- and Posttests
Significant
Pre/Post
Gains
195
170
95
2
73
2
292
245
Course Institutionalization
For systemic change to occur at the higher education institutions, departments of arts and sciences and education
departments are encouraged to work together through MMSP to support stronger content courses in mathematics
and science for teacher preparation, undergraduate and graduate degree requirements, and for in-service teachers
pursuing graduate-level content courses for recertification. Integration of Title IIB courses into graduate programs
at Institutes of Higher Education ensures sustainability over time. The intent behind encouraging the partnerships
is that the faculty from the Arts and Sciences Departments bring strong content expertise to the partnership table.
This integration creates greater opportunities for participants to complete coursework leading to a content-area
degree and/or to licensure along with the highly qualified designation.
During the four most recent funding periods—2006-2007, 2007-2008, 2008-2009, and 2009-2010—partnerships
were asked to describe activities during those funding periods that were related to the institutionalization of their
courses. Many partnerships evinced integration, plans for future integration, or—in the case of partnerships with
previously established involvement with MMSP—work toward sustaining prior integration. As would be
expected in a program involving partnerships with diverse structures and styles, the extent and type of integration
varied across partnerships. To convey a sense of how integration occurred, following are significant activities,
grouped according to partnership:
EduTron Lowell Public Schools (M/S) and EduTron Fitchburg State College (M)
 Two developmental courses, based on the EduTron model for MMSP courses, will continue to be offered
at Fitchburg State College (FSC). EduTron partners supported FSC in designing three pre-service courses
that are optimized for education majors. EduTron has begun working with FSC to help FSC apply the
EduTron model used in MMSP math courses to science courses.
 FSC has partnered with Lowell Public Schools to offer a teacher certification/CAGS program.
 Six mathematics and four science courses were approved by FSC as offerings at the continuing education
level.
4
Some of these courses had fewer than six MMSP enrollees within the context of another course that included additional non-MMSP
enrollees; other courses were not able to administer content assessments to all participants because of logistical challenges.
UMass Donahue Institute
Research and Evaluation Group
39
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Lesley University C3 (M)
 Two courses created through Lesley University’s MMSP in 2007-2008 are now offered to Lesley’s oncampus pre-service teachers.
 Efforts through MMSP contributed to the development of an online Mathematics Education program at
Lesley leading to the Master of Arts degree for elementary and middle school teachers.
 Nine math content courses were developed through participation in the MMSP program in 2007-2008 and
in prior years. All of these courses are part of Lesley University’s mathematics major for undergraduates,
which would not have been possible without the MMSP program.
North Shore (S)
 As a result of their joint involvement in MMSP through the North Shore partnership and the National
Science Foundation MSP program, Northeastern University has institutionalized all MMSP courses. Ten
MMSP courses can be used to fulfill degree requirements toward a Master’s in Education for Middle
School Science. In addition, this degree was developed as a result of these courses.
UMass Amherst C3 (M/S)
 Four courses developed through the UMass Amherst partnership were approved for graduate level credit.
Salem State C3 (M)
 Salem State College offers courses developed through MMSP as part of a master’s level teaching
program in middle school mathematics. All courses developed by Salem State College through MMSP
can be applied towards earning a degree through that program.
Southeast/Cape (S)
 Participants of the three courses offered through the SE/Cape partnership may apply credit for the courses
towards the Master of Arts in Teaching in Physical Science program that is offered through Bridgewater
State College.
Worcester Polytechnic Institute (S)
 A Master of Science Education program was created through the physics department at Worcester
Polytechnic Institute, and the MMSP course that was offered through the WPI-Science partnership will
serve as the model for instruction of future courses that will be offered.
Worcester Public Schools (M)
 As a result of the experience of working with Worcester Public Schools on MMSP, Clark University has
expressed interest in exploring the institutionalization of courses that were offered through MMSP.
Springfield College (S)
 Springfield College has incorporated into its pre-service Best Practices of Teaching Science course
activities from an MMSP course that are designed to help teachers understand how to change
misconceptions that students have about life science.
Randolph Public Schools (S)
 Four courses developed through the Randolph Public Schools partnership were approved for graduate
level credit at Bridgewater State University.
UMass Donahue Institute
Research and Evaluation Group
40
Annual Report of the MMSP State-level
Evaluation
Appendix I
Summary of Findings
The MMSP partnership activities summarized in this section of the report occurred between February 2, 2004,
and August 31, 2010. This period spans the beginning of the MMSP program through the end of the 2009-2010
funding period. Since MMSP began in February 2004, progress has been made towards meeting the goals of the
program as evidenced by the following data for both the program as a whole, since its inception, and for Cohort 4.
Cumulative Findings
Overview of Partnerships

A total of 27 partnerships were funded across the Commonwealth. Of these, 16 were organized around
mathematical content, nine were organized around science content, and two were organized around both
mathematical and science content. Of the 27 MMSP partnerships, all delivered courses, all offered
multiple courses, and all had participants who attended more than one course.
 Cohort 1, which began in February 2004, consisted of eight partnerships, with six of the eight
partnerships offering mathematics professional development and two offering science professional
development.

Cohort 2, which began in September 2004, consisted of two partnerships, both offering mathematics
professional development.

Cohort 3, which began in September 2006, consisted of nine partnerships, with four of the nine
partnerships offering mathematics professional development, three offering science professional
development, and two offering professional development in both mathematics and science content.

Cohort 4, which began in September 2008, consisted of eight partnerships, with four of the eight
partnerships offering mathematics professional development and four offering science professional
development.
Overview of Courses

In total, 298 MMSP courses were delivered by the end of Year 7 of MMSP funding. Of these 298
courses, 201 were mathematics courses, 95 were science and/or technology/engineering courses, and two
were courses offering both mathematics and science content.
Overview of Participants

In total, 2,673 unique participants participated in MMSP courses by the end of Year 7.
 989 participants (37% of all participants) attended multiple courses. Of the 989 participating in
multiple courses, 141 took courses across partnerships.


5,172 course seats were filled by all participants across all funding periods.
Types of Schools of Participants
 Of all 2,673 unique participants, 97% came from public schools (including public charter schools),
2% came from non-public schools, and 1% did not indicate their school type.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation

Appendix I
High Need Status of Districts of Participants
 The partnerships exceeded the ESE target of having at least 50% of all participants come from high
need districts, with 66% of all participants in the program coming from high need districts.

Across all years of their involvement, 16 of the 27 partnerships had at least 50% of their participants
coming from high need districts.

For each and every individual year of funding, 13 of the 27 partnerships had at least 50% of their
participants coming from high need districts.

Across all years of their involvement, 12 of 27 partnerships had at least 75% of the participants come
from high need districts.

For each and every individual year of funding, ten of the 27 partnerships had at least 75% of the
participants come from high need districts.
As the following data reveal, not all MMSP participants were licensed in their teaching areas or held relevant
degrees, indicating that course participants were, in fact, those in need of the courses:
 Licensure in Mathematics and Science Content Areas for Middle and High School Teachers
 Of the regular education mathematics courses taught by MMSP teachers, 77% were taught by
teachers who were licensed in mathematics.


Of the special education mathematics courses taught by MMSP teachers, 33% were taught by
teachers who were licensed in mathematics.

Of the ELL mathematics courses taught by MMSP teachers, 50% were taught by teachers who were
licensed in mathematics.

Of regular education science or technology/engineering courses taught by MMSP teachers, 57% were
taught by teachers who were licensed in the subject of the course.

Of special education science or technology/engineering courses taught by MMSP teachers, 42% were
taught by teachers who were licensed in the subject of the course.

Of ELL science or technology/engineering courses taught to students of MMSP teachers, 40% were
taught by teachers who were licensed in the subject of the course.
Degrees Held in Content Area in which Teaching
 Of regular education mathematics courses taught by MMSP teachers, 20% were taught by teachers
who held mathematics degrees.

Of special education mathematics courses taught by MMSP teachers, 4% were taught by teachers
who held mathematics degrees.

Of ELL mathematics courses taught to students of MMSP teachers, 14% were taught by teachers who
held mathematics degrees.

Of regular education science or technology/engineering courses taught by MMSP teachers, 25% were
taught by teachers who held degrees that were relevant to the focus of the course.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation
Appendix I

Of special education science or technology/engineering courses taught by MMSP teachers, 2% were
taught by teachers who held degrees that were relevant to the focus of the course.

Of ELL science or technology/engineering courses taught by MMSP teachers, 26% were taught by
teachers who held degrees that were relevant to the focus of the course.
Highly Qualified Status
MMSP has contributed to increasing the numbers of highly qualified public school teachers.
 Attaining Highly Qualified Status
 By the end of Year 7, of the participants who had entered MMSP as not highly qualified, 158 had
attained highly qualified status.

New Licensures
 Of the 989 participants taking multiple courses, 26% earned one or more new licenses by the end of
the Year 7 of MMSP, and 12% took and passed an MTEL test.
Cohort 4 Findings
Overview of Partnerships

Eight Cohort 4 partnerships were funded during the 2008-2009 and 2009-2010 funding periods. Of these,
four were organized around mathematical content and four were organized around science content. Of the
eight Cohort 4 MMSP partnerships, all delivered courses. Of the eight partnerships, all offered multiple
courses, all had participants who took more than one course within Cohort 4, and all had participants who
had attended more than one course across the entire duration of MMSP.
Overview of Courses

In total, 74 Cohort 4 courses were delivered by the end of Year 7 of MMSP funding. Of these 74 courses,
46 were mathematics courses and 28 were science/technology/engineering courses.
Overview of Participants

By the end of Year 7, 840 unique Cohort 4 participants participated in MMSP courses.
 233 participants (28% of all Cohort 4 participants) attended multiple courses in Cohort 4 by the end
of 2009-2010.

65 participants (8% of all Cohort 4 participants) had attended additional courses outside of Cohort 4,
across all MMSP funding periods.

1,230 course seats were filled by Cohort 4 participants by the end of 2009-2010.

Course attrition rates were generally low and averaged 4% across all courses offered by Cohort 4
partnerships.
Reaching Targeted Participants

Types of Schools of Participants
 Of all 840 unique Cohort 4 participants, approximately 98% came from public schools (including
public charter schools), and approximately 3% came from non-public schools.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation

Appendix I
High Need Status of Districts of Participants
 The Cohort 4 partnerships exceeded the ESE target of having at least 50% of all participants come
from high need districts, with 65% of all Cohort 4 participants in the program coming from high need
districts.

Three of the eight partnerships had at least 50% of their participants coming from high need districts.

Three of the eight partnerships had at least 75% of the participants come from high need districts.
As the following data reveal, not all Cohort 4 participants were licensed in their teaching areas or held relevant
degrees, indicating that course participants were, in fact, those in need of the courses:
 Licensure in Mathematics and Science Content Areas for Middle and High School Teachers
 Of regular education mathematics courses taught by Cohort 4 teachers, 81% were taught by teachers
who were licensed in mathematics.


Of special education mathematics courses taught by Cohort 4 teachers, 46% were taught by teachers
who were licensed in mathematics.

Of ELL mathematics courses taught by Cohort 4 teachers, 65% were taught by teachers who were
licensed in mathematics.

Of regular education science or technology/engineering courses taught by Cohort 4 teachers, 63%
were taught by teachers who were licensed in the subject of the course.

Of special education science or technology/engineering courses taught by Cohort 4 teachers, 59%
were taught by teachers who were licensed in the subject of the course.

Of ELL science or technology/engineering courses taught by Cohort 4 teachers, 39% were taught by
teachers who were licensed in the subject of the course.
Degrees Held in Content Area in which Teaching
 Of regular education mathematics courses taught by Cohort 4 teachers, 17% were taught by teachers
who held mathematics degrees.

Of special education mathematics courses taught by Cohort 4 teachers, 7% were taught by teachers
who held mathematics degrees.

Of ELL mathematics courses taught by Cohort 4 teachers, 18% were taught by teachers who held
mathematics degrees.

Of regular education science or technology/engineering courses taught by Cohort 4 teachers, 27%
were taught by teachers who held degrees that were relevant to the focus of the course.

Of special education science or technology/engineering courses taught by Cohort 4 teachers, one was
taught by a teacher who held a degree that was relevant to the focus of the course.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation

Appendix I
Of ELL science or technology/engineering courses taught by Cohort 4 teachers, 17% were taught by
teachers who held degrees that were relevant to the focus of the course.
Highly Qualified Status
MMSP has contributed to increasing the numbers of highly qualified public school teachers.
 Attaining Highly Qualified Status
 By the end of the 2009-2010 funding period, of the Cohort 4 participants who had entered MMSP as
not highly qualified, 24 had attained highly qualified status.
Content Knowledge Gains
The content knowledge of Cohort 4 participants was increased:
 Statistically significant improvements in scores on content knowledge assessments occurred in 49 (66%)
of the 74 Cohort 4 courses.

Gains in average percentage of items correct between pre- and post-course test administrations occurred
in 65 of the 74 courses delivered through Cohort 4 partnerships.
Integrating Courses into Higher Education Institutions
Integration or plans for future integration of courses into institutions of higher education occurred within some
Cohort 4 partnerships.
UMass Donahue Institute
Research and Evaluation Group