Massachusetts Mathematics and Science Partnership Title IIB
Annual State-level Evaluation Report
Cohort 4 Reporting Period: September 1, 2008, through August 31, 2011
Cohort 5 Reporting Period: September 1, 2010, through August 31, 2011
Cumulative Reporting Period: February 2, 2004, through August 31, 2011
Prepared for the Massachusetts Department of Elementary and
Secondary Education
July 2012
Annual Report of the MMSP State-level Evaluation
Contents
Contents
Program Description................................................................................................................................... 4
Report Organization .................................................................................................................................... 6
Evaluation Plan and Activities ................................................................................................................... 7
State-level Evaluation ............................................................................................ 7
Local Evaluation and Related Technical Assistance ............................................. 7
Cohort 4 Activity: September 1, 2008 through August 31, 2011 ............................................................ 8
State-level Participant Background Data ............................................................... 8
Partnership-level Participant Background Data ................................................... 22
Cohort 5 Activity: September 1, 2010 through August 31, 2011 .......................................................... 31
State-level Participant Background Data ............................................................. 31
Partnership-level Participant Background Data ................................................... 40
Cumulative Summary: All Cohorts over All Funding Periods .............................................................. 43
Summary of Findings ............................................................................................................................... 60
Cumulative Findings ............................................................................................ 60
Cohort 4 Findings ................................................................................................. 62
Cohort 5 Findings ................................................................................................. 64
Appendix A: Participant Background Survey – September 2010–August 2011 .....Error! Bookmark not
defined.
Appendix B: Timeline for State-level Evaluation and TA Activities ........... Error! Bookmark not defined.
Appendix C: Results of the Participant Background Survey for 2008–2011 ..........Error! Bookmark not
defined.
Appendix D: High Need District Eligibility Criteria ...................................... Error! Bookmark not defined.
Appendix E: Enrollment and Attrition Rates by Course ............................. Error! Bookmark not defined.
Appendix F: Subject Matter Competency Demonstration Options ........... Error! Bookmark not defined.
Appendix G: Mean Percentage Scores for Pre- & Post-course Tests ....... Error! Bookmark not defined.
Appendix H: High Need Districts for All Funding Periods, by Partnership ............Error! Bookmark not
defined.
Appendix I: Criteria that Account for Gain in Highly Qualified Status ...... Error! Bookmark not defined.
UMass Donahue Institute
Research and Evaluation Group
I
Annual Report of the MMSP State-level Evaluation
Tables
Tables Index
Table 1: Budgets: Cohort 4 Partnerships ............................................................................................................. 8
Table 2: Teaching Areas: Cohort 4 Participants ................................................................................................... 9
Table 3: Types of Schools of Unique Participants: Cohort 4 Participants ...................................................... 10
Table 4: High Need Status of Unique Participants from Public Schools: Cohort 4 Participants .................. 10
Table 5: High Need District Participants by Partnership: Cohort 4 Participants ........................................... 11
Table 6: Reasons for Participation: Cohort 4, All Seats .................................................................................... 13
Table 7: Repeat Participants: Cohort 4 Partnerships ........................................................................................ 14
Table 8: Total Enrollment and Attrition Information: Cohort 4 Partnerships ................................................. 16
Table 9a: Cohort 4 Science and Technology Teaching Areas – Regular Education ..................................... 16
Table 9b: Cohort 4 Science and Technology Teaching Areas – Special Education ...................................... 17
Table 9c: Cohort 4 Science and Technology Teaching Areas – ELL Education ............................................ 17
Table 10a: Cohort 4 Mathematics Teacher Levels – Regular Education ......................................................... 19
Table 10b: Cohort 4 Mathematics Teacher Levels – Special Education ......................................................... 19
Table 10c: Cohort 4 Mathematics Teacher Levels – ELL Education ............................................................... 20
Table 11: Pursuit of Science Degrees: Cohort 4 ................................................................................................ 21
Table 12: Pursuit of Mathematics Degrees: Cohort 4 ....................................................................................... 21
Table 13: Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores:
Cohort 4................................................................................................................................................................... 22
Table 14: Boston Participant Background Information (M) .............................................................................. 23
Table 15: Brockton PS Participant Background Information (M) ..................................................................... 24
Table 16: Gateway Regional SD Participant Background Information (S) ...................................................... 25
Table 17: Lesley University Participant Background Information (M) ............................................................. 26
Table 18: Greater North Shore Participant Background Information (S) ........................................................ 27
Table 19: Randolph PS Participant Background Information (S)..................................................................... 28
Table 20: Springfield College Participant Background Information (S) .......................................................... 29
Table 21: Boston University Participant Background Information (M) ........................................................... 30
Table 22: Budgets: Cohort 5 Partnerships ......................................................................................................... 31
Table 23: Teaching Areas: Cohort 5 Participants .............................................................................................. 32
Table 24: Types of Schools of Unique Participants: Cohort 5 Participants ................................................... 32
Table 25: High Need Status of Unique Participants from Public Schools: Cohort 5 Participants ............... 33
Table 26: High Need District Participants by Partnership: Cohort 5 Participants ......................................... 33
Table 27: Reasons for Participation: Cohort 5, All Seats ................................................................................. 34
Table 28: Repeat Participants: Cohort 5 Partnerships ...................................................................................... 35
Table 29: Total Enrollment and Attrition Information: Cohort 5 Partnerships ............................................... 35
Table 30a: Cohort 5 Science and Technology Teaching Areas – Regular Education ................................... 36
Table 30b: Cohort 5 Science and Technology Teaching Areas – Special Education .................................... 36
Table 30c: Cohort 5 Science and Technology Teaching Areas – ELL Education .......................................... 36
Table 31a: Cohort 5 Mathematics Teacher Levels – Regular Education ......................................................... 38
Table 31b: Cohort 5 Mathematics Teacher Levels – Special Education ......................................................... 38
Table 31c: Cohort 5 Mathematics Teacher Levels – ELL Education ............................................................... 38
Table 32: Pursuit of Science Degrees: Cohort 5 ................................................................................................ 39
Table 33: Pursuit of Mathematics Degrees: Cohort 5 ....................................................................................... 39
Table 34: Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores for
Cohort 5................................................................................................................................................................... 40
Table 35: EduTron Participant Background Information (M)............................................................................ 41
UMass Donahue Institute
Research and Evaluation Group
II
Annual Report of the MMSP State-level Evaluation
Tables
Table 36: Everett-UMass Boston Participant Background Information (S) .................................................... 42
Table 37: Overview of MMSP Partnership Participation: Cohort 1–Cohort 3 ................................................. 43
Table 38: Overview of MMSP Partnership Participation: Cohort 4–Cohort 5 ................................................. 44
Table 39: Budgets: All Partnerships, All Funding Periods ............................................................................... 45
Table 40: Teaching Areas: All Participants, All Funding Periods .................................................................... 47
Table 41. Types of Schools: All Unique Participants, All Funding Periods .................................................... 48
Table 42: High Need Status of All Unique Participants from Public Schools ................................................. 48
Table 43: Reasons for Participation of Participants: All Seats, All Funding Periods .................................... 49
Table 44: Repeat Participants: All Partnerships, All Funding Periods ............................................................ 50
Table 45: Highly Qualified Status: All Unique Participants, All Funding Periods .......................................... 52
Table 46: MTEL Tests Taken by All Participants – Total to Date ..................................................................... 53
Table 47a: Science and Tech/Engineering Teaching Areas of All Participants – Regular Education ......... 54
Table 47b: Science and Tech/Engineering Teaching Areas of All Participants – Special Education .......... 54
Table 47c: Science and Tech/Engineering Teaching Areas of All Participants – ELL Education ................ 54
Table 48a: Mathematics Teacher Levels of All Participants – Regular Education ......................................... 55
Table 48b: Mathematics Teacher Levels of All Participants – Special Education ......................................... 55
Table 48c: Mathematics Teacher Levels of All Participants – ELL Education ............................................... 55
Table 49: Pursuit of Science Degrees: All Participants, All Funding Periods ................................................ 56
Table 50: Pursuit of Mathematics Degrees: All Participants, All Funding Periods ........................................ 57
Table 51: Number of Courses with Statistically Significant Gains in Mean Content Knowledge Scores.... 57
UMass Donahue Institute
Research and Evaluation Group
III
Annual Report of the MMSP State-level Evaluation
Program Description
Program Description
The purpose of the Massachusetts Mathematics and Science Partnership Program (MMSP) is to improve student
achievement in mathematics, science, and technology/engineering through intensive, high-quality professional
development activities that focus on deepening teachers’ content knowledge. This multi-year project is funded by
the U.S. Department of Education (USED) as part of the No Child Left Behind (NCLB) Act Title IIB funding
stream. Funding to local partnerships is administered by state education agencies; in Massachusetts this is the
Massachusetts Department of Elementary and Secondary Education (ESE), which awards funding through a
competitive grant process.
Partnerships awarded these grants are required to include 1) a core partner that has been identified as a high need
school district in the subject matter on which the partnership is focusing and 2) a core partner that is a science,
technology and/or engineering, or mathematics (STEM) department from an institution of higher education. The
partnerships are composed of higher education institutions, school districts, and, in some cases, private
organizations involved in providing both pre-service and in-service training to teachers. Partnerships are required
to offer courses that equal at least 45 hours of direct instruction followed by at least 20 hours of follow-up contact
to support the implementation of course content in the classroom. Partnerships are encouraged to tailor the model
used to deliver the professional development and follow-up to best fit the objectives of their programs along with
their resources, expertise, and existing infrastructure. Partnership activities are guided by the following goals 1:
Goal I
Develop and implement an effective and sustained course of study for in-service teachers of STEM by
integrating the courses of study into schools of arts and sciences and/or education at institutions of
higher education.
Goal II
Increase the number of STEM teachers in the partner school districts who are licensed in the subject
area(s) and grade level(s) they teach.
Goal III Increase the number of STEM teachers in the partner school districts who participate in high quality
professional development and advance their content knowledge.
Goal IV Develop and implement a systemic approach to STEM education by integrating professional
development with district and school STEM improvement initiatives.
The program began in February 2004, and has had eight funding periods, defined as follows:
1

Year 1: February 2, 2004 through August 31, 2004

Year 2: September 1, 2004 through August 31, 2005

Year 3: September 1, 2005 through August 31, 2006

Year 4: September 1, 2006 through August 31, 2007

Year 5: September 1, 2007 through August 31, 2008
Program goals were modified slightly between the beginning of the program and the beginning of the most recent funding period.
UMass Donahue Institute
Research and Evaluation Group
4
Annual Report of the MMSP State-level Evaluation

Year 6: September 1, 2008 through August 31, 2009

Year 7: September 1, 2009, through August 31, 2010

Year 8: September 1, 2010, through August 31, 2011.
Program Description
The partnerships who received initial funding in Year 1 are referred to as Cohort 1; those who received initial
funding in Year 2 are referred to as Cohort 2; those who received initial funding in Year 4 are referred to as
Cohort 3; those who received initial funding in Year 6 are referred to as Cohort 4; and those who received initial
funding in Year 8 are referred to as Cohort 5.
UMass Donahue Institute
Research and Evaluation Group
5
Annual Report of the MMSP State-level Evaluation
Report Organization
Report Organization
The purpose of this report is threefold: 1) to provide details regarding only Cohort 4 participation for all three
years of funding for it, 2) to provide details regarding only Cohort 5 participation for the only year of funding for
it thus far, 3) to provide a cumulative summary regarding participation for all cohorts over all funding periods.
Data supporting the first purpose address the period of September 1, 2008, through August 31, 2011; data
supporting the second purpose address the period of September 1, 2010, through August 31, 2011; and data
supporting the third purpose address the period of February 2, 2004, through August 31, 2011. Participant data
were collected through the Participant Background Survey, an instrument developed by the UMass Donahue
Institute (UMDI) and administered by partnerships to each participant on the last day of each course. See
Appendix A for the survey used during Year 8. The purpose of this survey is to gather data about participants’
professional backgrounds and qualifications. This information provides a picture of who the participants are, aids
in determining whether the courses are reaching the teachers who most need professional development, and aids
in tracking how teacher qualifications may change during the MMSP funding period. Data from the survey
regarding teacher licensure, possession of and progress towards earning degrees, and status in terms of
Massachusetts Tests for Educator Licensure (MTEL) exams allows determination of the number of teachers who
meet criteria defining highly qualified status. Unless noted, data from the survey are reported in terms of unique
individuals, regardless of the number of courses taken by each individual.
Data speaking to the strengthening of relationships between partnership members were collected through a section
of the local evaluation reports that partnerships were required to submit to the ESE. In this section, partnerships
were asked to describe the extent to which their courses had been integrated into activities of their higher
education partners.
UMass Donahue Institute
Research and Evaluation Group
6
Annual Report of the MMSP State-level Evaluation
Evaluation Plan and Activities
Evaluation Plan and Activities
State-level Evaluation
The ESE has contracted with UMDI to conduct a state-level evaluation of the MMSP. State-level evaluation
reports are submitted to the USED. UMDI’s primary role as state-level evaluator is to coordinate program-wide
collection of outcome data on behalf of the ESE. Data collection for the state-level evaluation is organized around
a basic logic model for professional development initiatives shown below.
Local Evaluation and Related Technical Assistance
In addition to the state-level data collection, each partnership is required to conduct its own local evaluation. In an
effort to support strong local evaluations, ESE required that partnerships sub-contract with UMDI to provide
technical assistance on design and implementation of their local evaluations. The timeline listing the evaluation
activities is found in Appendix B.
UMass Donahue Institute
Research and Evaluation Group
7
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Cohort 4 Activity: September 1, 2008 through August 31, 2011
Cohort 4, which began in the 2008–2009 funding period, consisted of eight partnerships. Table 1 shows the
funding received by Cohort 4 partnerships for the period beginning in September 2008 and ending in August
2009, for the period beginning in September 2009 and ending in August 2010, for the period beginning in
September 2010 and ending in August 2011, and for the cumulation of the three.
Table 1: Budgets: Cohort 4 Partnerships
Partnership
Sep08–Aug09
Sep09–Aug10
Sep10–Aug11
Boston Public Schools (M)
$157,975
$405,747
$218,986
$782,708
Brockton Public Schools (M)
$180,145
$255,758
$251,263
$687,166
Gateway Regional School Dist. (S)
$186,609
$200,370
$172,379
$559,358
Lesley Springfield (M)
$228,593
$324,820
$308,416
$861,829
Greater North Shore (S)
$265,917
$306,690
$266,480
$839,087
Randolph Public Schools (S)
$176,993
$183,150
$151,178
$511,321
Springfield College (S)
$161,062
$148,896
$156,832
$466,790
Boston University (M)
$241,586
$245,180
$244,394
$1,598,880
$2,070,611
$1,769,928
$731,160
$5,439,419
TOTAL
TOTAL
State-level Participant Background Data
Cohort 4 consisted of eight partnerships, with four of the eight partnerships offering mathematics professional
development and four offering science professional development.
Across all three years of funding, 115 Cohort 4 courses were delivered. Of these 115 courses, 73 were
mathematics courses, 42 were science courses. Of those 115 courses, 46 (40%) were unique, and 69 (60%) were
repeat offerings. This section of the report summarizes data collected from participants in these courses.
Across all three years of funding for Cohort 4, 1,169 unique participants completed the Participant Background
Survey on one or more occasions. Of these participants, 341 of them took two or more courses within Cohort 4.
The term “unique participant” refers to each individual who participated in the program, regardless of how many
courses he or she took. Data for items from the survey that help to convey participants’ professional backgrounds
and motives for participation are discussed in the remainder of this section. All survey data for this group may be
found in Appendix C. The responses to the survey questions are presented as frequencies and percentages. Not all
percentages total 100% because many items allowed multiple responses and not all of the participants responded
to all of the items.
Position of Participants
At the time of their last completed survey from a Cohort 4 MMSP course, 93% of course participants identified
themselves as teachers. Of all respondents, 65% were regular education teachers; 18% were special education or
special education inclusion teachers; 5% were ELL teachers; 2% were teaching mathematics or science coaches;
1% were non-teaching mathematics or science coaches; 1% were gifted and talented teachers; 1% were Title I
UMass Donahue Institute
Research and Evaluation Group
8
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
teachers; 1% were department heads or curriculum coordinators; 1% were paraprofessionals; <1% were
principals, assistant principals, or headmasters; <1% were support specialists; <1% were long-term substitutes;
and 4% indicated that they held “other” positions.
Content Taught
The distribution of teaching areas of respondents at the time of the survey is shown in Table 2. Because
respondents identified all teaching areas that applied to their positions at the time of the survey, some selected
multiple responses, so frequency totals exceed the number of respondents and percentages exceed 100%.
At the time of their last MMSP course, 39% of Cohort 4 participants were teaching mathematics (including
elementary mathematics), 35% were teaching science, and 29% were teaching all subjects at the elementary level.
Table 2: Teaching Areas: Cohort 4 Participants
Sep08–Aug09
Sep09–Aug10
Sep10–Aug11
TOTAL
N = 341
N = 630
N = 508
N = 1169
Teaching Areas
(Multiple responses
permitted)
n
%
Mathematics
Any science area
General Science
Biology
Earth Science
Chemistry
Physics
Technology/Engineering
Elementary (all subjects)
Elementary Mathematics
Other
Not Currently Teaching
102
138
98
27
22
17
17
14
87
21
11
7
30
41
29
8
7
5
5
4
26
6
3
2
n
184
213
123
52
25
31
34
13
197
55
26
12
%
29
34
20
8
4
5
5
2
31
9
4
2
n
159
207
89
58
26
38
39
4
135
30
18
12
%
31
41
18
11
5
8
8
1
27
6
4
2
n
371
409
210
102
54
69
70
21
343
80
46
12
%
32
35
18
9
5
6
6
2
29
7
4
1
Teaching Experience of Participants
At the time of their last completed survey from an MMSP course, the teaching experience of the 1,169 unique
Cohort 4 participants was as follows: 13% were in their first to third year of teaching, 42% had between four and
ten years experience in education, 28% had between 11 and 20 years of experience, 13% reported over 20 years of
experience, and 5% did not report.
Teaching Levels of Participants
For reporting purposes, schools in the participating districts were organized into categories of elementary schools
(grades K-5), K-8 schools, middle schools (grades 6-8), and high schools (grades 9-12). At the time of their last
completed survey from an MMSP course, 35% of Cohort 4 participants were teaching in an elementary or K-8
school, 42% were teaching in a middle school, 19% were teaching in a high school, 1% were teaching at both the
middle and high school levels, and less than 1% were teaching at the pre-K level. Those remaining either were not
currently teaching or the level at which they taught was unknown.
UMass Donahue Institute
Research and Evaluation Group
9
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Types of Schools of Participants
As shown in Table 3, 97% of unique Cohort 4 participants worked in a public school setting, and 3% worked in a
non-public school setting.
Table 3: Types of Schools of Unique Participants: Cohort 4 Participants
Sep08–Aug09
School Type
n
Public Schools (includes
public charter schools)
Non-public School
Other or No Response
TOTAL
Sep09–Aug10
%
n
%
Sep10–Aug11
n
TOTAL
%
n
%
333
98
614
98
484
95
1129
97
7
1
2
<1
16
0
3
0
24
0
5
0
40
0
3
0
341
100
630
100
508
100
1169
100
High Need Status of Districts of Participants
MMSP partnerships were required to include at least one high need district. Appendix D identifies the criteria for
the high need designation.
The ESE expected that at least 50% of participants in each partnership would come from high need districts, and
further, they set an informal goal that at least 75% of participants for each partnership would come from high need
districts. Of all Cohort 4 participants, 67% came from high need districts. Table 4 shows that by the end of the
2010–2011 funding period, 69% of Cohort 4 participants from public schools had come from high need districts.
Table 4: High Need Status of Unique Participants from Public Schools: Cohort 4 Participants
Sep08–Aug09
Sep09–Aug10
Sep10–Aug11
TOTAL
n
%
n
n
%
n
High Need District
Non-high Need District
213
120
64
36
412
202
67
33
344
140
71
29
774
355
69
31
TOTAL
333
100
614
100
484
100
1129
100
School Type
%
%
Table 5 presents the number of participants from high need districts organized by each partnership. An
examination of high need district participation in individual partnerships reveals that over the course of each
partnership’s involvement in MMSP since the beginning of MMSP, six of the eight partnerships had at least 50%
of their participants coming from high need districts across all years of their involvement and three of eight
partnerships exceeded the informal goal of having at least 75% of the participants come from high need districts.
If individual years of participation are examined, it is seen that three of the eight partnerships had at least 50% of
their participants coming from high need districts for each and every year of funding, and all three of those eight
partnerships also had at least 75% of the participants come from high need districts for each and every year of
funding.
UMass Donahue Institute
Research and Evaluation Group
10
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
As of their last Cohort 4 course, of the 341 individuals who took multiple courses within Cohort 4, approximately
73% were from high need public school districts, approximately 21% were from other public school districts, and
approximately 6% either were from private schools or did not provide information on their districts.
Table 5: High Need District Participants by Partnership: Cohort 4 Participants
Sep08–Aug09
Partnership
n
Boston PS (M)
Brockton PS (M)
Gateway RSD (S)
Lesley Springfield (M)
Greater North Shore (S)
Sep09–Aug10
Sep10–Aug11
TOTAL
High Need District
Boston
Medford
Subtotal
Bridgewater-Raynham
Brockton
Fall River
Falmouth
Freetown/Lakeville
New Bedford
Plymouth
Quincy
Randolph
Seekonk
South Shore CS
Swansea
Wareham
Weymouth
Subtotal
Agawam
Chicopee
Easthampton
Gateway
Hampshire
Holyoke
Springfield
Westfield
Subtotal
Agawam
Chicopee
Holyoke
Springfield
Westfield
Subtotal
Boston
Bridgewater-Raynham
Fitchburg
Lawrence
Lynn
Malden
UMass Donahue Institute
Research and Evaluation Group
40
0
40
0
21
9
0
0
0
0
0
0
0
0
0
0
0
30
0
0
1
6
0
2
8
0
17
0
1
9
35
4
49
26
0
0
0
3
0
%
n
%
174
100%
59%
39%
94%
174
0
28
12
1
3
0
3
2
0
1
0
1
0
0
51
0
0
3
7
0
9
0
0
19
0
1
6
59
5
71
26
0
0
0
4
1
100%
45%
38%
84%
n
112
1
113
1
24
13
0
0
1
0
0
2
0
1
0
3
4
49
4
2
2
6
4
2
2
12
34
1
0
5
29
5
40
35
1
2
1
0
5
%
99%
56%
72%
93%
n
270
1
271
1
72
33
1
3
1
3
2
2
1
1
1
3
4
128
4
2
4
11
4
11
4
12
52
1
1
11
77
7
97
76
1
2
1
7
6
%
100%
51%
59%
85%
11
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 5: High Need District Participants by Partnership: Cohort 4 Participants
Sep08–Aug09
Partnership
n
Greater North Shore (S)
(continued)
Randolph PS (S)
Springfield Coll. (S)
Boston University (M)
Sep09–Aug10
Sep10–Aug11
TOTAL
High Need District
Medford
Pioneer Charter School
of Science
Quincy
Randolph
Revere
Somerville
Waltham
Weymouth
Subtotal
Randolph
Weymouth
Subtotal
New Leadership CS
Springfield
Subtotal
Boston
Brockton
Cambridge
Chelsea
Chicopee
Falmouth
Haverhill
Lawrence
Lowell
Medford
Peabody
Randolph
Salem
Somerville
Waltham
Weymouth
Woburn
Worcester
Subtotal
%
n
%
n
%
n
%
0
0
3
3
1
0
0
1
0
1
0
0
0
0
31
6
0
6
0
26
26
3
0
0
0
0
2
2
7
0
0
0
0
1
0
0
0
0
1
16
0
1
8
12
0
0
52
18
0
18
0
30
30
3
0
1
0
0
0
0
1
2
0
0
0
1
1
0
0
1
0
10
1
0
2
9
3
1
63
7
1
8
1
20
21
1
2
3
1
1
0
0
5
2
1
1
1
0
1
1
3
0
0
23
1
1
10
18
3
1
131
20
1
21
1
50
51
7
2
3
1
1
2
2
13
2
1
1
1
2
1
1
3
1
1
45
50%
25%
96%
39%
46%
41%
100%
29%
50%
24%
100%
58%
54%
28%
98%
45%
Reasons for Participation in MMSP Courses
For each course taken, respondents identified all of their reasons for participating. Unlike in the earlier portions of
this report where data were presented for unique participants, data for this topic are presented for all course seats,
since responses are uniquely relevant to each distinct course that a participant took. Table 6 presents findings for
all 1,870 seats for all courses taken by Cohort 4 participants during the 2008–2009, 2009–2010, and 2010–2011
funding periods.
UMass Donahue Institute
Research and Evaluation Group
12
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 6: Reasons for Participation: Cohort 4, All Seats
Sep08–Aug09
%
of 391
course
seats
Reasons for Participation
(Multiple responses permitted)
To increase knowledge in content
To obtain graduate credit
To pursue a personal interest
To earn PDPs for recertification
To get an additional license
(certification)
To prepare for the Massachusetts
Test for Educator Licensure
(MTEL)
To follow an administrator’s
suggestion
To obtain a first license
(certification)
Other
n
Sep09–Aug10
%
of 839
course
seats
n
Sep10–Aug11
%
of 640
course
seats
n
TOTAL
n
%
of 1870
course
seats
287
240
118
108
73%
61%
30%
28%
603
565
239
217
72%
67%
29%
26%
491
447
217
150
77%
70%
34%
23%
1381
1252
574
475
74%
67%
31%
25%
52
13%
123
15%
112
18%
287
15%
45
12%
74
9%
74
12%
193
10%
30
8%
105
13%
24
4%
159
9%
11
3%
29
4%
13
2%
53
3%
28
7%
37
4%
35
6%
100
5%
Repeat Participation
Cohort 4 partnerships were successful at encouraging and retaining repeat participants. All of the eight
partnerships offered multiple courses and all had participants who attended more than one course within that
partnership. In all, 341 participants attended multiple courses within Cohort 4, which translated to 1,870 course
seats filled for this cohort. Table 7 provides details regarding repeat participation, including information on the
participants who took courses from partnerships from other MMSP cohorts. (Table 14 through Table 21 also
include data on repeat participation, as does Table 44.)
Highly Qualified Status
For Cohort 4 by the end of the 2010–2011 funding period, 39 unique participants attained highly qualified status.
Of the 39 who attained highly qualified status, six did so by passing the appropriate MTEL, three did so by
earning a degree in the content areas in which they taught, one obtained an undergraduate equivalent in the
content area in which he taught, one did so by earning a teaching license, nine did so by changing their positions,
and 19 did so through a combination of means. Additionally, it is likely that MMSP participation contributed to
the attainment of highly qualified status for more participants after the 2010–2011 period had ended, but it was
not possible to capture those gains during this reporting period.
Of those who attained highly qualified status, 11 were from the Greater North Shore Northeastern (S) partnership,
nine were from the Boston Public Schools (M) partnership, eight were from the Gateway (S) partnership, four
each were from the Randolph (S) partnership and the Lesley (M) partnership, and three were from the Boston
University (M) partnership. An identification of the criteria by which teachers who gained highly qualified status
could demonstrate competency in their subject matter may be found in Appendix F.
UMass Donahue Institute
Research and Evaluation Group
13
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 7: Repeat Participants: Cohort 4 Partnerships
Partnership
Number
of
Courses
Offered
to Date
Total
Number of
Unique*
Participants
to Date
Number
Taking
Multiple
Courses in
Cohort 4
Number Taking
One Course in
Cohort 4 & One or
More Courses in
Other Cohorts
Number Taking
Multiple Courses in
Cohort 4 & Other
Cohorts
Boston Public Schools (M)
29
270
76
10
5
Brockton Public Schools (M)
15
249
3
28
0
3
82
40
3
7
Lesley Springfield (M)
21
109
61
5
8
Greater North Shore (S)
28
235
95
25
24
Randolph Public Schools (S)
8
71
28
5
2
Springfield College (S)
5
52
21
5
8
Boston University (M)
6
101
17
2
2
115
1169
341
83
56
Gateway RSD (S)
TOTAL
*Participants who participated in multiple courses across partnerships were counted only once in the partnership of their most recent course.
Attrition
Course attrition rates were generally low and averaged 5% across all courses across all partnerships for Cohort 4.
Of the 115 Cohort 4 courses delivered across all three years, data on attrition rates were available for 114. Of the
114 courses for which data were available, 81 (71%) had an attrition rate of 0%, 14 (12%) had an attrition rate
ranging between 1% and 10%, and 19 (17%) had an attrition rate of greater than 10%. Table 8 provides a
breakdown, by partnership, of enrollment and attrition rates. Appendix E provides a breakdown, by course, of
enrollment and attrition rates.
Licensure and Degrees Held in Subjects Taught
By the end of the 2010–2011 funding period for Cohort 4, 324 regular education teachers, 49 special education
teachers, and 23 ELL teachers reported teaching subjects in science or technology/engineering. Tables 9a, 9b, and
9c show how many teachers taught each individual science or technology/engineering subject from Cohort 4
through the 2010–2011 funding period. The tables also show the percentages of teachers whose survey responses
indicated they were licensed in the subject they taught, and they show the percentages of teachers whose survey
responses indicated they held a degree in the subject they taught. Table 9a provides information for regular
education teachers, Table 9b provides information for special education teachers, and Table 9c provides
information for ELL teachers. (The numbers presented in these tables exceed the number of teachers who reported
teaching these subjects because some teachers taught more than one subject.) Across all science subject areas for
regular education teachers, the licensing reported appeared to be appropriate for approximately 66% of the science
subjects taught, and the degrees held corresponded to the science subject taught for approximately 37% of the
science subjects taught. Across all science subject areas for special education teachers, the licensing reported
appeared to be appropriate for approximately 69% of the science subjects taught, and the degree held by one
teacher translated to a 2% correspondence between degree and subject taught. Across all science subject areas for
ELL teachers, the licensing reported appeared to be appropriate for 48% of the subjects taught, and the degrees
held corresponded to the science subject taught for approximately 22% of the science subjects taught.
UMass Donahue Institute
Research and Evaluation Group
14
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
To see the extent to which licensure and degree were appropriate for any particular type of teacher for any
particular subject taught, look in the relevant table at data in the row corresponding to the subject of interest and
in the column corresponding to the time period of interest. For example, to see the extent to which licensure and
degree were appropriate for just regular education biology teachers in all of Cohort 4, look at Table 9a in the cells
of the “Total” column in the row for the subject area “Biology” to learn that 80 teachers taught regular education
biology and that of those 80, 78% were licensed in biology and 70% held degrees in biology.
UMass Donahue Institute
Research and Evaluation Group
15
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 8: Total Enrollment and Attrition Information: Cohort 4 Partnerships
Partnership
Number of Courses Offered
Number of Participants
Enrolled First Day
Sep08–
Aug09
Sep09–
Aug10
Sep10–
Aug11
Sep08–
Aug09
Boston Public Schools (M)
Brockton Public Schools (M)
Gateway RSD (S)
Lesley Springfield (M)
Greater North Shore (S)
Randolph Public Schools (S)
Springfield College (S)
Boston University (M)
2
3
1
5
11
2
2
2
14
6
1
10
8
3
2
2
13
6
1
6
9
3
1
2
202
52
47
74
95
28
27
69
All Courses/Partnerships
28
46
41
594
Sep09–
Aug10
Number of Participants
Completed Courses
Attrition Rate
Sep10–
Aug11
Sep08–
Aug09
Sep09–
Aug10
Sep10–
Aug11
Sep08–
Aug09
Sep09–
Aug10
Sep10–
Aug11
235
122
55
188
190
62
30
35
167
98
49
89
209
48
21
42
184
51
47
72
89
28
27
67
235
116
49
185
177
61
30
35
157
94
47
89
186
43
21
42
9%
2%
0%
3%
6%
0%
0%
3%
0%
5%
11%
2%
7%
2%
0%
0%
6%
4%
4%
0%
11%
10%
0%
0%
917
723
565*
888
679
5%
3%
6%
Table 9a: Cohort 4 Science and Technology Teaching Areas – Regular Education
Sep08–Aug09
Subject
General Science
Biology
Chemistry
Physics
Earth Science
Technology/Engineering
Number
of
Teachers
n
81
25
17
16
20
14
License in
Subject
Taught
%*
91
72
65
44
45
36
Degree in
Subject
Taught
%*
15
60
35
6
10
14
Sep09–Aug10
Number
of
Teachers
n
98
42
28
28
16
7
License in
Subject
Taught
%*
71
74
68
46
13
14
Sep10–Aug11
Degree in
Subject
Taught
%*
13
67
57
21
13
14
Number
of
Teachers
n
74
47
33
31
21
4
License in
Subject
Taught
%*
62
79
79
32
33
50
Total
Degree in
Subject
Taught
%*
15
79
52
18
5
25
Number
of
Teachers
n
91
80
61
57
40
17
License in
Subject
Taught
%*
77
78
69
39
28
35
Degree in
Subject
Taught
%*
13
70
51
23
10
24
*Of the number of regular education participants teaching this subject for this period
UMass Donahue Institute
Research and Evaluation Group
16
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 9b: Cohort 4 Science and Technology Teaching Areas – Special Education
Sep08–Aug09
Subject
General Science
Biology
Chemistry
Physics
Earth Science
Technology/Engineering
Number
of
Teachers
n
12
1
0
1
2
0
License in
Subject
Taught
%*
58
100
0
100
50
0
Sep09–Aug10
Degree in
Subject
Taught
%*
0
0
0
0
0
0
Number
of
Teachers
n
18
5
1
4
5
3
License in
Subject
Taught
%*
78
40
0
25
20
33
Sep10–Aug11
Degree in
Subject
Taught
%*
6
0
0
0
0
Number
of
Teachers
n
9
3
2
5
4
0
0
License in
Subject
Taught
%*
33
67
100
80
50
–
Total
Degree in
Subject
Taught
%*
0
0
0
0
0
–
Number
of
Teachers
n
21
8
3
8
9
3
License in
Subject
Taught
%*
81
63
67
63
44
Degree in
Subject
Taught
%*
5
0
0
0
0
33
0
*Of the number of special education participants teaching this subject for this period
Table 9c: Cohort 4 Science and Technology Teaching Areas – ELL Education
Sep08–Aug09
Subject
General Science
Biology
Chemistry
Physics
Earth Science
Technology/Engineering
Number
of
Teachers
n
5
1
0
0
0
0
License in
Subject
Taught
%*
80
0
0
0
0
0
Degree in
Subject
Taught
%*
0
0
0
0
0
0
Sep09–Aug10
Number
of
Teachers
n
4
3
2
2
1
1
License in
Subject
Taught
%*
50
33
0
0
0
0
Sep10–Aug11
Degree in
Subject
Taught
%*
50
33
0
0
0
0
Number
of
Teachers
n
5
6
0
2
1
0
License in
Subject
Taught
%*
60
67
–
100
100
–
Total
Degree in
Subject
Taught
%*
0
50
–
0
0
–
Number
of
Teachers
n
5
9
2
4
2
1
License in
Subject
Taught
%*
80
44
0
50
50
0
Degree in
Subject
Taught
%*
40
33
0
0
0
0
*Of the number of ELL participants teaching this subject for this period.
UMass Donahue Institute
Research and Evaluation Group
17
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
By the end of the 2010–2011 funding period for Cohort 4, 296 regular education teachers 112 special education
teachers, and 26 ELL teachers reported teaching mathematics. Tables 10a, 10b, and 10c show how many Cohort 4
teachers taught at each mathematics level through the 2010–2011 funding period. The tables also show the
percentages of teachers whose survey responses indicated they were licensed in the mathematics level at which
they taught, and they show the percentages of teachers whose survey responses indicated they held a degree in
mathematics. Table 10a provides information for regular education teachers, Table 10b provides information for
special education teachers, and Table 10c provides information for ELL teachers. (The sum of the numbers
presented in each of Tables 10a and 10b exceeds the number of teachers who reported teaching mathematics
because some teachers taught mathematics at both the elementary and middle school levels.)
From an “ease-of-reading perspective,” it would be ideal merely to report on the percentages of teachers who
were licensed to teach mathematics and the percentage of teachers who held mathematics degrees. Sometimes,
though, teachers hold multiple positions and may only be licensed to teach some of what they are teaching, and
what is really desired is to understand the extent to which what is being taught is done so by individuals licensed
to teach it and by individuals who hold relevant degrees. As a result, the explanations of data that follow are done
so in terms of teaching positions.
For MMSP mathematics teaching positions in regular education, the licensing appeared to be appropriate for
approximately 81% of positions held, and 18% of the positions were held by individuals with mathematics
degrees. For special education teachers, the licensing appeared to be appropriate for approximately 51% of the
positions held, and 6% of the positions were held by individuals with mathematics degrees. For ELL teachers, the
licensing appeared to be appropriate for approximately 62% of the positions held, and 19% of the positions were
held by individuals with mathematics degrees.
To see the extent to which licensure and degree were appropriate for a particular type of teacher for a particular
mathematics level taught, look in the relevant table at data in the row corresponding to the level of interest and in
the column corresponding to the time period of interest. For example, to see the extent to which licensure and
degree were appropriate for just regular education middle school math teachers in only 2010–2011, look at Table
10a in the cells of the “Sep10–Aug11” column in the row for the subject area “Middle School” to learn that 82
teachers taught regular education middle school mathematics and that of those 82, 81% were licensed to teach
middle school mathematics and 18% held mathematics degrees.
UMass Donahue Institute
Research and Evaluation Group
18
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 10a: Cohort 4 Mathematics Teacher Levels – Regular Education
Sep08–Aug09
Level
Sep09–Aug10
Sep10–Aug11
Number
of
Teachers
Licensed
at Level
Degree in
Math
Number
of
Teachers
Licensed
at Level
Degree in
Math
Total
Number
of
Teachers
Licensed
at Level
Degree in
Math
%*
Number
of
Teachers
Licensed
at Level
Degree in
Math
n
%*
%*
n
%*
%*
n
%*
n
%*
%*
Elementary School
Middle School
High School
Middle & High School
17
60
17
0
65
87
82
0
0
8
41
0
44
89
19
0
73
84
84
0
0
21
42
0
18
82
23
2
56
81
91
100
0
18
35
0
58
185
52
1
64
84
89
100
0
18
35
0
TOTAL Math
94
82
13
152
81
18
125
79
18
296
81
18
*Of the number of regular education participants teaching mathematics at this level for this period
Table 10b: Cohort 4 Mathematics Teacher Levels – Special Education
Sep08–Aug09
Number
of
Teachers
Level
n
Sep09–Aug10
Licensed
at Level
Degree in
Math
Number
of
Teachers
%*
Sep10–Aug11
Licensed
at Level
Degree in
Math
Number
of
Teachers
%*
Total
Licensed
at Level
Degree in
Math
Number
of
Teachers
Licensed
at Level
Degree in
Math
%*
%*
n
%*
%*
%*
n
%*
n
Elementary School
Middle School
High School
Middle & High School
3
17
5
0
100
53
40
0
0
12
0
0
3
53
11
2
33
42
46
50
0
0
27
50
2
27
7
2
100
56
43
50
0
4
14
0
6
84
18
4
83
50
44
50
0
2
22
25
TOTAL Math
25
56
8
69
42
6
38
55
5
112
51
6
*Of the number of special education participants teaching mathematics at this level for this period
UMass Donahue Institute
Research and Evaluation Group
19
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 10c: Cohort 4 Mathematics Teacher Levels – ELL Education
Sep08–Aug09
Number
of
Teachers
Level
n
Sep09–Aug10
Licensed
at Level
Degree in
Math
Number
of
Teachers
Licensed
at Level
%*
%*
Sep10–Aug11
Degree in
Math
Number
of
Teachers
%*
Total
Licensed
at Level
Degree in
Math
Number
of
Teachers
Licensed
at Level
Degree in
Math
%*
n
n
%*
%*
n
%*
%*
Elementary School
Middle School
High School
Middle & High School
1
4
0
0
100
50
–
–
0
0
–
–
1
7
4
1
0
43
100
100
0
0
75
0
1
10
1
0
100
50
100
–
0
20
0
–
3
17
5
1
67
47
100
100
0
12
60
0
TOTAL Math
5
60
0
13
73
27
12
58
17
26
62
19
*Of the number of ELL education participants teaching mathematics at this level for this period
Degrees Currently Pursued
Information on degrees currently being pursued in science, technology/engineering, and mathematics was derived from the most recently completed survey of each
individual. Through the 2010–2011 funding period, 132 Cohort 4 participants reported that they were pursuing degrees in either science and technology/
engineering or science education. Table 11 shows that these participants were pursuing a total of 139 undergraduate and graduate degrees. A degree was identified
as “In Area” if the content matter being studied corresponded to the area in which a teacher taught; a degree was identified as “Not in Area” if the content matter
being studied did not correspond to the area in which a teacher taught or if it was a science education degree. A science education degree was identified as “Not in
Area” because this degree is not linked to a specific science subject. Degrees pursued by teachers who taught multiple science subjects were included in the row
labeled “Multiple Science Areas” and those degrees were classified as “In Area” if the content matter being studied corresponded to any area in which a teacher
taught.
UMass Donahue Institute
Research and Evaluation Group
20
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 11: Pursuit of Science Degrees: Cohort 4
Type of Science Degree Pursued
MA/MS
CAGS
BA/BS
Teaching Area
In Area
Not in
Area
Not in
Area
In Area
In Area
Doctorate
Not in
Area
In Area
Not in
Area
General Science
1
1
12
28
1
1
0
1
Biology
0
0
3
19
0
0
0
0
Chemistry
0
0
2
4
0
0
0
0
Physics
0
0
0
9
0
0
0
0
Earth Science
0
0
0
1
0
0
0
0
Technology/Engineering
0
0
0
2
0
0
0
0
Multiple Science Areas
2
1
5
18
4
1
0
1
Elementary
0
Other
3
0
Total
3
2
0
17
22
1
0
101
5
1
2
0
4
Of all Cohort 4 participants through the 2010–2011 funding period, 138 were pursuing mathematics or
mathematics education degrees. Table 12 provides details about the types of degrees being pursued and the
teaching areas of those who were pursuing the degrees, and it shows that these participants were pursuing a total
of eight undergraduate and 133 graduate degrees.
Table 12: Pursuit of Mathematics Degrees: Cohort 4
Mathematics Degree Pursued
MA/MS
CAGS
Courses taught
BA/BS
Elementary Math
0
4
2
0
Middle School Math
0
54
9
1
High School Math
1
21
1
0
MS & HS Math
0
1
0
0
Other (non-Math)
7
32
6
2
Total
8
112
18
3
Doctorate
Content Knowledge Gains
As a grant condition, partnerships were required to use a pre-course test and post-course test for each MMSP
course to assess participants’ knowledge growth of the content taught in the course. For each course, the same
instrument served as both the pre- and post-course test. While partnerships were permitted to locate and use an
instrument with established reliability and validity, it was most often the case that such instruments were not
available because a priority was placed on utilizing assessments that would reflect the precise content taught in
each course. As a result, the faculty members who had developed the courses usually developed the assessments,
and typically neither validity nor reliability was determined for them because time and resource constraints
prohibited doing so.
UMass Donahue Institute
Research and Evaluation Group
21
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Data were analyzed only for participants who completed both pre- and post-course assessments. To determine if
the scores showed statistically significantly changes between pre- and post-course test administrations, a paired
samples t-test was used for each course for which ten or more participants completed both pre- and post-course
assessments, and a Wilcoxon Matched Pairs test was used for each course for which fewer than ten participants
completed both pre- and post-course assessments.
Of the 115 courses that were delivered across all Cohort 4 partnerships through the 2010–2011 funding period,
content assessments were administered for 110 of them. Of these 110 courses, 107 had gains in the average
percentage of items correct between pre- and post- test administrations. Statistically significant improvements in
scores on content knowledge assessments occurred in 77% of the 110 courses. Of the 25 courses not showing
statistically significant improvement in scores, though, 18 had fewer than six participants2, the smallest sample
size at which it is possible to detect statistical significance at the level used for these analyses. Table 13 provides
an overview, by subject matter of courses delivered, of the numbers of courses that showed statistically significant
gains. For detailed information on mean pre- and post-course content knowledge scores by partnership and by
course, see Appendix G.
Table 13: Number of Courses with Statistically Significant Gains in Mean Content
Knowledge Scores: Cohort 4
Sep08–Aug09
Delivered,
Content Area
with Preand Posttests
Math
Science & Technology/
Engineering
TOTAL
Significant
Pre/Post
Gains
Sep09–Aug10
Delivered,
with Preand Posttests
Significant
Pre/Post
Gains
Sep10–Aug11
Delivered,
with Preand Posttests
Significant
Pre/Post
Gains
Total
Delivered,
with Preand Posttests
Significant
Pre/Post
Gains
13
10
29
18
26
25
68
51
14
7
14
14
14
14
42
34
27
17
43
32
40
39
110
85
Partnership-level Participant Background Data
Presented in Table 14 through Table 21, this section offers an overview of selected participant survey data for
each Cohort 4 partnership. These data were collected at the end of each course that was offered by each
partnership. The responses to the survey questions are presented as frequencies and percentages of the total
number of participants in the partnership for the 2008–2009, 2009–2010, and 2010–2011 periods and a
cumulation of the three. In cases where not all participants responded to all of the items and in cases where
multiple responses were permitted, percentages presented may not total 100%.
The section of each table identified as “Participants Who Took Multiple Courses” contains two categories. Each is
defined as follows: “In Cohort 4” refers to participants who took two or more courses within the context of Cohort
4—they participated in no courses that had been offered through other cohorts. “In other cohorts” refers to
participants who took one course through Cohort 4 and at least one additional course that had been offered
through a different cohort.
2
Some of these courses had fewer than six MMSP enrollees within the context of another course that included additional non-MMSP
enrollees; other courses were not able to administer content assessments to all participants because of logistical challenges.
UMass Donahue Institute
Research and Evaluation Group
22
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
The “Highly Qualified” section of each table includes the response option “Private school/Not applicable.”
Participants categorized under this option refer to those who either were teaching at a private school or for whom
highly qualified status would be irrelevant, such as school administrators.
Table 14: Boston Participant Background Information (M)
Number of
Participants
Sep08–Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In other cohorts
Teach Regular Education
Teach Special Education (sole and
inclusion)
Teach ELL, ESL, or Sheltered English
Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or
Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
40^
Number of
Participants
Sep09–Aug10
167
Number of
Participants
Sep10–Aug11
Number of
Participants
TOTAL*
276#
114
0
1
24
(0%)
(3%)
(60%)
51
11
98
(31%)
(7%)
(59%)
44
7
62
(39%)
(6%)
(54%)
82
16
156
(30%)
(6%)
(57%)
15
(38%)
53
(32%)
33
(29%)
86
(31%)
1
(3%)
10
(6%)
13
(11%)
22
20
5
18
(50%)
(13%)
(45%)
93
22
53
(56%)
(13%)
(32%)
58
12
50
(51%)
(11%)
(44%)
143
29
108
(52%)
(11%)
(39%)
4
(10%)
17
(10%)
11
(10%)
31
(11%)
40
27
12
1
0
0
24
40
(100%)
(68%)
(30%)
(3%)
(0%)
(0%)
(60%)
(100%)
167
97
60
4
3
3
87
161
(100%)
(58%)
(36%)
(2%)
(2%)
(2%)
(52%)
(96%)
113
76
34
2
2
0
78
112
(99%)
(67%)
(30%)
(2%)
(2%)
(0%)
(68%)
(98%)
275
171
91
7
4
3
161
268
(100%)
(62%)
(33%)
(3%)
(1%)
(1%)
(58%)
(97%)
(8%)
*The total number of participants refers to unique participants across all years, combined. Because some participants took more than one course across
years, this total is not equivalent to the sum of participants over all three years.
^This value underreports by 144 the number of participants who completed courses for this partnership in 2008-2009. For this year, this partnership
only administered Participant background Surveys to 40 of its 184 participants, and data from those surveys are reported here.
# Because the value reported for Sep08–Aug09 was underreported by 144, this value is also underreported. The number by which it is underreported is
unknown because some of the participants who did not complete surveys in Sep08–Aug09 may have participated in a later year.
UMass Donahue Institute
Research and Evaluation Group
23
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 15: Brockton PS Participant Background Information (M)
Number of
Participants
Sep08–Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In other cohorts
Teach Regular Education
Teach Special Education (sole and
inclusion)
Teach ELL, ESL, or Sheltered English
Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or
Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
51
Number of
Participants
Sep09–Aug10
114
Number of
Participants
Sep10–Aug11
87
Number of
Participants
TOTAL*
252
6
7
38
(12%)
(14%)
(75%)
4
17
77
(4%)
(15%)
(68%)
2
7
49
(2%)
(8%)
(56%)
6
31
164
(2%)
(12%)
(65%)
10
(20%)
25
(22%)
22
(25%)
57
(23%)
1
(2%)
8
(7%)
4
(5%)
13
(5%)
29
5
15
(57%)
(10%)
(29%)
43
19
51
(38%)
(17%)
(45%)
40
8
31
(46%)
(9%)
(36%)
112
32
97
(44%)
(13%)
(38%)
5
(10%)
12
(11%)
9
(10%)
26
(10%)
30
30
16
2
0
3
29
50
(59%)
(59%)
(31%)
(4%)
(0%)
(6%)
(57%)
(98%)
51
63
40
2
9
0
60
109
(45%)
(55%)
(35%)
(2%)
(8%)
(0%)
(53%)
(96%)
49
48
35
0
2
2
47
80
(56%)
(55%)
(40%)
(0%)
(2%)
(2%)
(54%)
(92%)
130
141
91
4
11
5
136
239
(52%)
(56%)
(36%)
(2%)
(4%)
(2%)
(54%)
(95%)
*The total number of participants refers to unique participants across all years, combined. Because some participants took more than one course across
years, this total is not equivalent to the sum of participants over all three years.
UMass Donahue Institute
Research and Evaluation Group
24
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 16: Gateway Regional SD Participant Background Information (S)
Number of
Participants
Sep08–Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In other cohorts
Teach Regular Education
Teach Special Education (sole and
inclusion)
Teach ELL, ESL, or Sheltered English
Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or
Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
46
Number of
Participants
Sep09–Aug10
50
Number of
Participants
Sep10–Aug11
47
Number of
Participants
TOTAL*
85
2
6
43
(4%)
(13%)
94%
28
3
47
(56%)
(6%)
(94%)
30
7
44
(64%)
(15%)
(94%)
43
10
79
(51%)
(12%)
(93%)
2
4%
1
(2%)
1
(2%)
3
(4%)
1
2%
1
(2%)
1
(2%)
2
(2%)
4
0
6
9%
0%
13%
10
0
2
(20%)
(0%)
(4%)
7
0
5
(15%)
(0%)
(11%)
14
0
7
(17%)
(0%)
(8%)
39
(85%)
40
(80%)
40
(85%)
66
(78%)
17
19
19
6
2
0
20
45
(37%)
(41%)
(41%)
(13%)
(4%)
(0%)
(44%)
(98%)
19
21
18
10
1
0
26
48
(38%)
(42%)
(36%)
(20%)
(2%)
(0%)
(52%)
(96%)
34
18
23
3
3
0
21
44
(72%)
(38%)
(49%)
(6%)
(6%)
(0%)
(45%)
(94%)
52
37
35
8
5
0
41
79
(61%)
(44%)
(41%)
(9%)
(6%)
(0%)
(48%)
(93%)
*The total number of participants refers to unique participants across all years, combined. Because some participants took more than one course across
years, this total is not equivalent to the sum of participants over all three years.
UMass Donahue Institute
Research and Evaluation Group
25
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 17: Lesley University Participant Background Information (M)
Number of
Participants
Sep08–Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In other cohorts
Teach Regular Education
Teach Special Education (sole and
inclusion)
Teach ELL, ESL, or Sheltered English
Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or
Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
52
Number of
Participants
Sep09–Aug10
85
Number of
Participants
Sep10–Aug11
43
Number of
Participants
TOTAL*
109
22
2
40
(42%)
(4%)
(77%)
57
9
51
(67%)
(11%)
(60%)
34
2
33
(79%)
(5%)
(77%)
61
13
66
(56%)
(12%)
(61%)
10
(19%)
24
(28%)
4
(9%)
28
(26%)
2
(4%)
4
(5%)
0
(0%)
4
(4%)
15
7
28
(29%)
(14%)
(54%)
23
10
47
(27%)
(12%)
(55%)
10
7
21
(23%)
(16%)
(49%)
23
12
60
(21%)
(11%)
(55%)
4
(8%)
3
(4%)
3
(7%)
5
(5%)
49
28
17
2
4
1
29
49
(94%)
(54%)
(33%)
(4%)
(8%)
(2%)
(56%)
(94%)
71
47
28
1
4
5
46
74
(84%)
(55%)
(33%)
(1%)
(5%)
(6%)
(54%)
(87%)
40
31
9
1
2
0
29
40
(93%)
(72%)
(21%)
(2%)
(5%)
(0%)
(67%)
(93%)
93
60
35
3
7
4
59
97
(85%)
(55%)
(32%)
(3%)
(6%)
(4%)
(54%)
(89%)
*The total number of participants refers to unique participants across all years, combined. Because some participants took more than one course across
years, this total is not equivalent to the sum of participants over all three years.
UMass Donahue Institute
Research and Evaluation Group
26
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 18: Greater North Shore Participant Background Information (S)
Number of
Participants
Sep08–Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In other cohorts
Teach Regular Education
Teach Special Education (sole and
inclusion)
Teach ELL, ESL, or Sheltered English
Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or
Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
62
Number of
Participants
Sep09–Aug10
113
Number of
Participants
Sep10–Aug11
126
Number of
Participants
TOTAL*
237
26
4
54
(42%)
(6%)
(87%)
51
18
95
(45%)
(16%)
(84%)
58
27
97
(46%)
(21%)
(77%)
97
49
187
(41%)
(21%)
(79%)
5
(8%)
10
(9%)
9
(7%)
21
(9%)
2
(3%)
5
(4%)
11
(9%)
17
(7%)
2
0
4
(3%)
(0%)
(7%)
5
0
6
(4%)
(0%)
(5%)
4
1
16
(3%)
(1%)
(13%)
9
1
24
(4%)
(<1%)
(10%)
54
(87%)
90
(80%)
108
(86%)
200
(84%)
31
36
13
8
4
1
36
59
(50%)
(58%)
(21%)
(13%)
(7%)
(2%)
(58%)
(95%)
52
63
24
10
9
7
73
105
(46%)
(56%)
(21%)
(9%)
(8%)
(6%)
(65%)
(93%)
63
67
38
16
4
1
78
114
(50%)
(53%)
(30%)
(13%)
(3%)
(1%)
(62%)
(91%)
127
131
63
28
12
3
147
218
(54%)
(55%)
(27%)
(12%)
(5%)
(1%)
(62%)
(92%)
*The total number of participants refers to unique participants across all years, combined. Because some participants took more than one course across
years, this total is not equivalent to the sum of participants over all three years.
UMass Donahue Institute
Research and Evaluation Group
27
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 19: Randolph PS Participant Background Information (S)
Number of
Participants
Sep08–Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In other cohorts
Teach Regular Education
Teach Special Education (sole and
inclusion)
Teach ELL, ESL, or Sheltered English
Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or
Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
24
Number of
Participants
Sep09–Aug10
44
Number of
Participants
Sep10–Aug11
34
Number of
Participants
TOTAL*
73
5
0
20
(21%)
(0%)
(83%)
17
5
38
(39%)
(11%)
(86%)
18
2
29
(53%)
(6%)
(85%)
30
7
63
(41%)
(10%)
(86%)
3
(13%)
4
(9%)
4
(12%)
7
(10%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
10
2
2
(42%)
(8%)
(8%)
17
1
0
(39%)
(2%)
(0%)
12
1
2
(35%)
(3%)
(6%)
29
1
3
(40%)
(1%)
(4%)
10
(42%)
23
(52%)
18
(53%)
36
(49%)
6
14
5
3
1
1
15
24
(25%)
(58%)
(21%)
(13%)
(4%)
(4%)
(63%)
(100%)
18
27
13
2
0
2
29
43
(41%)
(61%)
(30%)
(5%)
(0%)
(5%)
(66%)
(98%)
8
28
4
2
0
0
27
34
(24%)
(82%)
(12%)
(6%)
(0%)
(0%)
(79%)
(100%)
19
50
14
6
1
2
53
72
(26%)
(69%)
(19%)
(8%)
(1%)
(3%)
(73%)
(99%)
*The total number of participants refers to unique participants across all years, combined. Because some participants took more than one course across
years, this total is not equivalent to the sum of participants over all three years.
UMass Donahue Institute
Research and Evaluation Group
28
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 20: Springfield College Participant Background Information (S)
Number of
Participants
Sep08–Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In other cohorts
Teach Regular Education
Teach Special Education (sole and
inclusion)
Teach ELL, ESL, or Sheltered English
Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or
Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
27
Number of
Participants
Sep09–Aug10
30
Number of
Participants
Sep10–Aug11
21
Number of
Participants
TOTAL*
52
3
7
22
(11%)
(26%)
(82%)
13
3
20
(43%)
(10%)
(67%)
13
5
15
(62%)
(24%)
(71%)
21
13
36
(40%)
(25%)
(69%)
4
(15%)
3
(10%)
4
(19%)
8
(15%)
1
(4%)
2
(7%)
1
(5%)
3
(6%)
1
0
2
(4%)
(0%)
(7%)
1
0
3
(3%)
(0%)
(10%)
0
0
3
(0%)
(0%)
(14%)
1
0
7
(2%)
(0%)
(14%)
19
(70%)
27
(90%)
19
(91%)
44
(85%)
26
12
9
3
3
0
12
25
(96%)
(44%)
(33%)
(11%)
(11%)
(0%)
(44%)
(93%)
30
12
13
4
1
0
14
30
(100%)
(40%)
(43%)
(13%)
(3%)
(0%)
(47%)
(100%)
21
9
11
0
1
0
10
20
(100%)
(43%)
(52%)
(0%)
(5%)
(0%)
(48%)
(95%)
51
21
23
5
3
0
22
50
(98%)
(40%)
(44%)
(10%)
(6%)
(0%)
(42%)
(96%)
*The total number of participants refers to unique participants across all years, combined. Because some participants took more than one course across
years, this total is not equivalent to the sum of participants over all three years.
UMass Donahue Institute
Research and Evaluation Group
29
Annual Report of the MMSP State-level Evaluation
Cohort 4 Activity
Table 21: Boston University Participant Background Information (M)
Number of
Participants
Sep08–Aug09
Total Number of Participants
Participants Who Took Multiple Courses
In Cohort 4
In other cohorts
Teach Regular Education
Teach Special Education (sole and
inclusion)
Teach ELL, ESL, or Sheltered English
Immersion
Teach Elementary (all content areas)
Teach Elementary Math
Teach Mathematics Above Elementary
Teach Science or
Technology/Engineering
Teaching in High Need District
Highly Qualified
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
Have Passed One or More MTEL Exams
Hold One or More Teaching Licenses
41
Number of
Participants
Sep09–Aug10
34
Number of
Participants
Sep10–Aug11
40
Number of
Participants
TOTAL*
101
0
1
33
(0%)
(2%)
(81%)
7
2
26
(21%)
(6%)
(77%)
10
2
34
(25%)
(5%)
(85%)
17
4
80
(17%)
(4%)
(79%)
3
(7%)
5
(15%)
3
(8%)
10
(10%)
2
(5%)
4
(12%)
1
(3%)
6
(6%)
6
2
27
(15%)
(5%)
(66%)
6
4
26
(18%)
(12%)
(77%)
5
1
32
(13%)
(3%)
(80%)
16
7
71
(16%)
(7%)
(70%)
5
(12%)
3
(9%)
1
(3%)
7
(7%)
16
23
8
1
1
8
26
36
(39%)
(56%)
(20%)
(2%)
(2%)
(20%)
(63%)
(88%)
10
19
8
0
0
7
19
31
(29%)
(56%)
(24%)
(0%)
(0%)
(21%)
(56%)
(91%)
23
29
10
0
1
0
30
35
(58%)
(73%)
(25%)
(0%)
(3%)
(0%)
(75%)
(88%)
45
67
23
1
2
8
69
91
(45%)
(66%)
(23%)
(1%)
(2%)
(8%)
(68%)
(90%)
*The total number of participants refers to unique participants across all years, combined. Because some participants took more than one course across
years, this total is not equivalent to the sum of participants over all three years.
UMass Donahue Institute
Research and Evaluation Group
30
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
Cohort 5 Activity: September 1, 2010 through August 31, 2011
Cohort 5, which began in the 2010–2011 funding period, consisted of two partnerships. Table 22 shows the
funding received by Cohort 5 partnerships for the period beginning in September 2010 and ending in August
2011.
Table 22: Budgets: Cohort 5
Partnerships
Partnership
Sep10-Aug11
EduTron (M)
$200,000
Everett (S)
$149,250
TOTAL
$349,250
State-level Participant Background Data
Cohort 5 consisted of two partnerships, with one partnership offering mathematics professional development and
one offering science professional development. There were six Cohort 5 courses delivered. Of these six courses,
four were mathematics courses and two were science courses. Of those six courses, all were unique, with no
repeat offerings. This section of the report summarizes data collected from participants in these courses.
During this first year of funding for Cohort 5, there were 127 participants, and 13 of them took two or more
courses within Cohort 5. By the end of the 2010–2011 funding period, 127 unique participants completed the
Participant Background Survey on one or more occasions. The term “unique participant” refers to each individual
who participated in the program, regardless of how many courses he or she took. Data for items from the survey
that help to convey participants’ professional backgrounds and motives for participation are discussed in the
remainder of this section. All survey data for this group may be found in Appendix C. The responses to the survey
questions are presented as frequencies and percentages. Not all percentages total 100% because many items
allowed multiple responses and not all of the participants responded to all of the items.
Position of Participants
At the time of their last completed survey from a Cohort 5 MMSP course, 93% of course participants identified
themselves as teachers. Of all respondents, 76% were regular education teachers; 13% were special education or
special education inclusion teachers; 3% were ELL teachers; 2% were department heads or curriculum
coordinators; 2% were math coaches; 2% were principals, assistant principals, or headmasters; and 2% indicated
that they held “other” positions.
Content Taught
The distribution of teaching areas of respondents at the time of the survey is shown in Table 23. Because
respondents identified all teaching areas that applied to their positions at the time of the survey, some selected
multiple responses, so frequency totals exceed the number of respondents and percentages exceed 100%.
At the time of their last MMSP course, 54% of Cohort 5 participants were teaching mathematics, 33% were
teaching science, and 13% were teaching all subjects at the elementary level.
UMass Donahue Institute
Research and Evaluation Group
31
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
Table 23: Teaching Areas: Cohort 5 Participants
Teaching Areas
Sep10–Aug11
(Multiple responses permitted)
n
%
Mathematics
Any science area
General Science
Biology
Earth Science
Chemistry
Physics
Technology/Engineering
Elementary (all subjects)
Elementary Mathematics
Other
Not Currently Teaching
68
42
17
19
5
7
8
4
16
3
4
8
54
33
13
15
4
6
6
3
13
2
3
6
Teaching Experience of Participants
At the time of their last completed survey from an MMSP course, the teaching experience of the 127 unique
Cohort 5 participants was as follows: 8% were in their first to third year of teaching, 43% had between four and
ten years experience in education, 32% had between 11 and 20 years of experience, 12% reported over 20 years of
experience, and 5% did not report.
Teaching Levels of Participants
For reporting purposes, schools in the participating districts were organized into categories of elementary schools
(grades K-5), K-8 schools, middle schools (grades 6-8), and high schools (grades 9-12). At the time of their last
completed survey from an MMSP course, 17% of Cohort 5 participants were teaching in an elementary or K-8
school, 30% were teaching in a middle school, 50% were teaching in a high school, and 2% were teaching at both
the middle and high school levels. Those remaining either were not currently teaching or the level at which they
taught was unknown.
Types of Schools of Participants
As shown in Table 24, 95% of unique Cohort 5 participants worked in a public school setting, and 5% worked in
a non-public school setting.
Table 24: Types of Schools of Unique Participants: Cohort 5 Participants
School Type
Public School (includes public charter schools)
Non-public School
TOTAL
UMass Donahue Institute
Research and Evaluation Group
Sep10–Aug11
N
121
6
127
%
95
5
100
32
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
High Need Status of Districts of Participants
MMSP partnerships were required to include at least one high need district. Appendix D identifies the criteria for
the high need designation.
The ESE expected that at least 50% of participants in each partnership would come from high need districts, and
further, they set an informal goal that at least 75% of participants for each partnership would come from high need
districts. Of all Cohort 5 participants, 73% came from high need districts. Table 25 shows that by the end of the
2010–2011 funding period, 77% of Cohort 5 participants from public schools had come from high need districts.
Table 25: High Need Status of Unique Participants from Public Schools: Cohort
5 Participants
Sep10–Aug11
School Type
High Need District
Non-high Need District
TOTAL
N
%
93
28
121
77
23
100
Table 26 presents the number of participants from high need districts organized by each partnership. An
examination of high need district participation in individual partnerships reveals that over the course of each
partnership’s involvement in MMSP, both partnerships had at least 50% of their participants coming from high
need districts and one of the two partnerships exceeded the informal goal of having at least 75% of the
participants come from high need districts.
Table 26: High Need District Participants by
Partnership: Cohort 5 Participants
Partnership
EduTron (M)
High Need
District
Boston
1
2
Lawrence
Lowell
3
28
Worcester
Greater Lawrence
Regional Voc-Tech
Subtotal
UMass Donahue Institute
Research and Evaluation Group
n
1
Brockton
Chelsea
Lynn
Revere
Everett (M)
Sep10-Aug11
%
2
1
27
1
66
Boston
Chelsea
2
2
Everett
Holbrook
6
1
Malden
1
76
33
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
Table 26: High Need District Participants by
Partnership: Cohort 5 Participants
Partnership
Everett (S)
(continued)
High Need
Sep10-Aug11
Medford
Rockland
11
1
Seekonk
Somerville
1
1
Waltham
Subtotal
1
27
68
Reasons for Participation in MMSP Courses
For each course taken, respondents identified all of their reasons for participating. Unlike in the earlier portions of
this report where data were presented for unique participants, data for this topic are presented for all course seats,
since responses are uniquely relevant to each distinct course that a participant took. Table 27 presents findings for
all 140 seats for all courses taken by Cohort 5 participants during the 2010–2011 funding period.
Table 27: Reasons for Participation: Cohort 5, All Seats
Sep10–Aug11
Reasons for Participation
(Multiple responses permitted)
To increase knowledge in content
To obtain graduate credit
To pursue a personal interest
To earn PDPs for recertification
To get an additional license (certification)
To prepare for the Massachusetts Test for Educator Licensure
(MTEL)
To follow an administrator’s suggestion
To obtain a first license (certification)
Other
%
n
of 140 course
seats
104
90
56
46
20
74
64
40
33
14
12
9
7
5
9
5
4
6
Repeat Participation
Cohort 5 partnerships were successful at encouraging and retaining repeat participants. Both partnerships offered
multiple courses. Both partnerships had participants who attended more than one course within Cohort 5. Of all
unique Cohort 5 participants, 13 attended multiple courses. Table 28 provides details regarding repeat
participation, including information on the 23 participants who took courses from partnerships from previous
MMSP cohorts. (Table 35 and Table 36 also include data on repeat participation, as does Table 44.)
UMass Donahue Institute
Research and Evaluation Group
34
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
Table 28: Repeat Participants: Cohort 5 Partnerships
Partnership
Number
of
Courses
Offered
to Date
Total
Number of
Unique*
Participants
to Date
Number
Taking
Multiple
Courses in
Cohort 5
Number Taking
One Course in
Cohort 5 & One or
More Courses in
Previous Cohorts
EduTron (M)
4
87
10
13
2
Everett (S)
2
40
3
8
0
TOTAL
6
127
13
21
2
Number Taking
Multiple Courses in
Cohort 5 & Previous
Cohorts
* Participants who participated in multiple courses across partnerships were counted only once in the partnership of their most recent course
Attrition
The course attrition rate over all courses over both partnerships averaged 5%. Of the six Cohort 5 courses
delivered, data on attrition rates were available for all of them. Of the six courses, three (50%) had an attrition rate
of 0%, two (33%) had an attrition rate ranging between 1% and 10%, and one (17%) had an attrition rate of
greater than 10%. Table 29 provides a breakdown, by partnership, of enrollment and attrition rates. Appendix E
provides a breakdown, by course, of enrollment and attrition rates.
Table 29: Total Enrollment and Attrition Information: Cohort 5 Partnerships
Partnership
Number of
Courses
Offered
Number of
Participants
Enrolled First
Day
Number of
Participants
Completed
Course
Attrition Rate
EduTron (M)
Everett (S)
4
2
102
45
97
43
5%
4%
All Courses/Partnerships
6
147
140
5%
Highly Qualified Status
For Cohort 5 during the 2010–2011 funding period, one participant from the EduTron partnership attained highly
qualified status by earning a teaching license. An identification of the criteria by which teachers who gained
highly qualified status could demonstrate competency in their subject matter may be found in Appendix F.
Licensure and Degrees Held in Subjects Taught
During the 2010–2011 funding period for Cohort 5, 33 regular education teachers, six special education teachers,
and three ELL teachers reported teaching subjects in science or technology/engineering. Tables 30a, 30b, and 30c
show how many teachers taught each individual science or technology/engineering subject from Cohort 5 during
the 2010–2011 funding period. The tables also show the percentages of teachers whose survey responses
indicated they were licensed in the subject they taught, and they show the percentages of teachers whose survey
responses indicated they held a degree in the subject they taught. Table 30a provides information for regular
education teachers, Table 30b provides information for special education teachers, and Table 30c provides
information for ELL teachers. (The numbers in these tables exceed the number of teachers who reported teaching
UMass Donahue Institute
Research and Evaluation Group
35
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
Table 30a: Cohort 5 Science and Technology Teaching Areas – Regular
Education
Sep10–Aug11
Subject
General Science
Biology
Chemistry
Physics
Earth Science
Technology/Engineering
Number of
Teachers
License in Subject
Taught
Degree in Subject
Taught
n
%*
%*
15
14
6
5
4
4
73
86
17
20
25
0
7
79
17
20
25
25
*Of the number of regular education participants teaching this subject for this period
Table 30b: Cohort 5 Science and Technology Teaching Areas – Special
Education
Sep10–Aug11
Subject
General Science
Biology
Chemistry
Physics
Earth Science
Technology/Engineering
Number of
Teachers
License in Subject
Taught
Degree in Subject
Taught
n
%*
%*
2
3
1
2
0
0
100
100
100
100
–
–
0
0
0
0
–
–
*Of the number of special education participants teaching this subject for this period
Table 30c: Cohort 5 Science and Technology Teaching Areas – ELL Education
Sep10–Aug11
Subject
General Science
Biology
Chemistry
Physics
Earth Science
Technology/Engineering
Number of
Teachers
License in Subject
Taught
Degree in Subject
Taught
n
%*
%*
0
2
0
1
1
0
–
50
–
100
0
–
–
50
–
100
0
–
*Of the number of ELL education participants teaching this subject for this period
these subjects because some teachers taught more than one subject.) Across all science subject areas for regular
education teachers, the licensing reported appeared to be appropriate for approximately 54% of the science
subjects taught, and the degrees held corresponded to the science subject taught for approximately 33% of the
UMass Donahue Institute
Research and Evaluation Group
36
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
science subjects taught. Across all science subject areas for special education teachers, the licensing reported
appeared to be appropriate for 100% of the science subjects taught, and no teachers held degrees that
corresponded to the subject taught. Across all science subject areas for ELL teachers, the licensing reported
appeared to be appropriate for 50% of the subjects taught, and the degrees held corresponded to the science
subject taught for 50% of the science subjects taught.
To see the extent to which licensure and degree were appropriate for a particular type of teacher for a particular
subject, look in the relevant table at data in the row corresponding to the subject of interest. For example, to see
the extent to which licensure and degree were appropriate for just regular education biology teachers, look at
Table 30a in the row for the subject area “Biology” to learn that 14 teachers taught regular education biology and
that of those 14, 86% were licensed in biology and 79% held degrees in biology.
During the 2010–2011 funding period for Cohort 5, 57 regular education teachers 11 special education teachers,
and one ELL teacher reported teaching mathematics. Tables 31a, 31b, and 31c show how many Cohort 5 teachers
taught at each mathematics level during the 2010–2011 funding period. The tables also show the percentages of
teachers whose survey responses indicated they were licensed in the mathematics level at which they taught, and
they show the percentages of teachers whose survey responses indicated they held a degree in mathematics. Table
30a provides information for regular education teachers, Table 30b provides information for special education
teachers, and Table 30c provides information for ELL teachers.
From an “ease-of-reading perspective,” it would be ideal merely to report on the percentages of teachers who
were licensed to teach mathematics and the percentage of teachers who held mathematics degrees. Sometimes,
though, teachers hold multiple positions and may only be licensed to teach some of what they are teaching, and
what is really desired is to understand the extent to which what is being taught is done so by individuals licensed
to teach it and by individuals who hold relevant degrees. As a result, the explanations of data that follow are done
so in terms of teaching positions.
For MMSP mathematics teaching positions in regular education, the licensing appeared to be appropriate for
approximately 95% of positions held, and 44% of the positions were held by individuals with mathematics
degrees. For special education teachers, the licensing appeared to be appropriate for approximately 27% of the
positions held, and none of the positions were held by individuals with mathematics degrees. The single ELL
teacher held neither a relevant license nor a mathematics degree.
To see the extent to which licensure and degree were appropriate for a particular type of teacher for a particular
mathematics level taught, look in the relevant table at data in the row corresponding to the level of interest and in
the column corresponding to the time period of interest. For example, to see the extent to which licensure and
degree were appropriate for just regular education middle school math teachers, look at Table 31a in the row for
the level “Middle School” to learn that 17 teachers taught regular education middle school mathematics and that
of those 17, 94% were licensed to teach middle school mathematics and 18% held mathematics degrees.
UMass Donahue Institute
Research and Evaluation Group
37
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
Table 31a: Cohort 5 Mathematics Teacher Levels – Regular Education
Sep10–Aug11
Number of
Teachers
Licensed at Level
Degree in Math
n
%*
%*
Elementary School
Middle School
High School
Middle & High School
3
17
35
2
67
94
97
100
0
18
60
50
TOTAL Math
57
95
44
Level
*Of the number of regular education participants teaching mathematics at this level for this period
Table 31b: Cohort 5 Mathematics Teacher Levels – Special Education
Sep10–Aug11
Level
Number of
Teachers
Licensed at Level
Degree in Math
n
%*
%*
Elementary School
Middle School
High School
Middle & High School
TOTAL Math
0
7
4
0
–
43
0
–
–
0
0
–
11
27
0
*Of the number of special education participants teaching mathematics at this level for this period
Table 31c: Cohort 5 Mathematics Teacher Levels – ELL Education
Sep10–Aug11
Number of
Teachers
Licensed at Level
Degree in Math
n
%*
%*
Elementary School
0
–
–
Middle School
High School
0
1
0
–
0
–
–
0
–
1
0
0
Level
Middle & High School
TOTAL Math
*Of the number of ELL education participants teaching mathematics at this level for this period
Degrees Currently Pursued
Information on degrees currently being pursued in science, technology/engineering, and mathematics was derived
from the most recently completed survey of each individual. Of all Cohort 5 participants in the 2010–2011
UMass Donahue Institute
Research and Evaluation Group
38
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
funding period, 17 were pursuing degrees in science and technology/engineering or science education. Table 32
provides details about the types of degrees being pursued and the teaching areas of those who were pursuing the
degrees, and it shows that these participants were pursuing a total of 17 undergraduate and graduate degrees. A
degree was identified as “In Area” if the content matter being studied corresponded to the area in which a teacher
taught; a degree was identified as “Not in Area” if the content matter being studied did not correspond to the area
in which a teacher taught or if it was a science education degree. A science education degree was identified as
“Not in Area” because this degree is not linked to a specific science subject. Degrees pursued by teachers who
taught multiple science subjects were included in the row labeled “Multiple Science Areas” and those degrees
were classified as “In Area” if the content matter being studied corresponded to any area in which a teacher
taught.
Table 32: Pursuit of Science Degrees: Cohort 5
Type of Science Degree Pursued
Teaching Area
BA/BS
Not in
In Area
Area
MA/MS
Not in
In Area
Area
CAGS
Not in
In Area
Area
Doctorate
Not in
In Area
Area
General Science
0
1
0
4
1
0
0
0
Biology
0
0
0
4
0
0
0
0
Chemistry
0
0
0
0
0
0
0
0
Physics
0
0
0
0
0
0
0
0
Earth Science
0
0
0
0
0
0
0
0
Technology/Engineering
0
0
0
0
0
0
0
0
Multiple Science Areas
0
0
1
3
0
0
0
0
Elementary
0
0
0
0
Other
0
3
0
0
Total
0
1
1
14
1
0
0
0
Of all Cohort 5 participants in the 2010–2011 funding period, 13 were pursuing mathematics or mathematics
education degrees. Table 33 provides details about the types of degrees being pursued and the teaching areas of
those who were pursuing the degrees, and it shows that these participants were pursuing a total of 16 graduate
degrees.
Table 33: Pursuit of Mathematics Degrees: Cohort 5
Mathematics Degree Pursued
Teaching Area
BA/BS
CAGS
Doctorate
Elementary Math
0
0
0
0
Middle School Math
0
2
0
0
High School Math
0
9
1
1
MS & HS Math
0
0
1
0
Other (non-Math)
0
1
1
0
Total
0
12
3
1
UMass Donahue Institute
Research and Evaluation Group
MA/MS
39
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
Content Knowledge Gains
As a grant condition, partnerships were required to use a pre-course test and post-course test for each MMSP
course to assess participants’ knowledge growth of the content taught in the course. For each course, the same
instrument served as both the pre- and post-course test. While partnerships were permitted to locate and use an
instrument with established reliability and validity, it was most often the case that such instruments were not
available because a priority was placed on utilizing assessments that would reflect the precise content taught in
each course. As a result, the faculty members who had developed the courses usually developed the assessments,
and typically, neither validity nor reliability were determined for them because time and resource constraints
prohibited doing so.
Data were analyzed only for participants who completed both pre- and post-course assessments. To determine if
the scores showed statistically significantly changes between pre- and post-course test administrations, a paired
samples t-test was used for each course for which ten or more participants completed both pre- and post-course
assessments, and a Wilcoxon Matched Pairs test was used for each course for which fewer than ten participants
completed both pre- and post-course assessments.
Gains in average percentage of items correct between pre- and post-course test administrations occurred in all 6 of
the courses delivered across all Cohort 5 partnerships for the 2010–2011 funding period, and statistically
significant improvements in scores on content knowledge assessments occurred in 100% of the courses. Table 34
provides an overview, by subject matter of courses delivered. For detailed information on mean pre- and postcourse content knowledge scores by partnership and by course, see Appendix G.
Table 34: Number of Courses with Statistically Significant
Gains in Mean Content Knowledge Scores for Cohort 5
Sep10–Aug11
Delivered
Significant
Pre/Post
Gains
Math
Science & Technology/Engineering
4
2
4
2
TOTAL
6
6
Content Area
Partnership-level Participant Background Data
Presented in Table 35 and Table 36, this section offers an overview of selected participant survey data for each
Cohort 5 partnership. These data were collected at the end of each course that was offered by each partnership.
The responses to the survey questions are presented as frequencies and percentages of the total number of
participants in the partnership for the 2010–2011 period. In cases where not all participants responded to all of the
items and in cases where multiple responses were permitted, percentages presented may not total 100%.
The section of each table identified as “Participants Who Took Multiple Courses” contains two categories. Each is
defined as follows: “In Cohort 5” refers to participants who took two or more courses within the context of Cohort
–they participated in no courses that had been offered during previous cohorts. “In previous cohorts” refers to
participants who took one course during Cohort 5 and at least one additional course that had been offered during a
previous cohort.
UMass Donahue Institute
Research and Evaluation Group
40
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
The “Highly Qualified” section of each table includes the response option “Private school/Not applicable.”
Participants categorized under this option refer to those who either were teaching at a private school or for whom
highly qualified status would be irrelevant, such as school administrators.
Table 35: EduTron Participant Background Information (M)
Number of Participants
Sep10-Aug11
Total Number of Participants
87
10
15
(12%)
(17%)
Teach Regular Education
63
(72%)
Teach Special Education (sole and inclusion)
13
(15%)
Teach ELL, ESL, or Sheltered English Immersion
1
(1%)
Teach Elementary (all content areas)
9
(10%)
Teach Elementary Math
3
(3%)
Teach Mathematics Above Elementary
65
(75%)
Teach Science or Technology/Engineering
10
(12%)
Teach in High Need District
66
(76%)
53
(61%)
27
(31%)
5
(6%)
2
(2%)
Participants Who Took Multiple Courses
Highly Qualified
In Cohort 5
In previous cohorts
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
0
(0%)
Have Passed One or More MTEL Exams
49
(56%)
Hold One or More Teaching Licenses
81
(93%)
UMass Donahue Institute
Research and Evaluation Group
41
Annual Report of the MMSP State-level Evaluation
Cohort 5 Activity
Table 36: Everett-UMass Boston Participant Background
Information (S)
Number of Participants
Sep10-Aug11
Total Number of Participants
40
Participants Who Took Multiple Courses
In Cohort 5
In previous cohorts
3
(8%)
8
(20%)
32
(80%)
Teach Special Education (sole and inclusion)
4
(10%)
Teach ELL, ESL, or Sheltered English Immersion
4
(10%)
Teach Elementary (all content areas)
7
(18%)
Teach Elementary Math
0
(0%)
Teach Mathematics Above Elementary
3
(8%)
Teach Science or Technology/Engineering
32
(80%)
Teach in High Need District
27
(68%)
20
(50%)
13
(33%)
7
(18%)
0
(0%)
0
(0%)
Have Passed One or More MTEL Exams
26
(65%)
Hold One or More Teaching Licenses
40
(100%)
Teach Regular Education
Highly Qualified
UMass Donahue Institute
Research and Evaluation Group
Yes
No
In some, but not all areas
Cannot be determined
Private school/Not applicable
42
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Cumulative Summary: All Cohorts over All Funding Periods
Overview of Partnerships, Budgets, Courses, and Participants
Tables 37 and 38 provide an overview of partnership projects since the inception of the program.
Table 37: Overview of MMSP Partnership Participation: Cohort 1–Cohort 3
Funding Period
Partnership Grouping
Cohort 1
Cohort 2
Cohort 3
UMass Donahue Institute
Research and Evaluation
Group
MMSP Year 1
MMSP Year 2
MMSP Year 3
MMSP Year 4
MMSP Year 5
MMSP Year 6
Feb04-Aug04
Sep04-Aug05
Sep05-Aug06
Sep06-Aug07
Sep07-Aug08
Sep08-Aug09
Grant Year 1
Grant Year 2
Grant Year 3
EduTron/Fitchburg-Math
Harvard-Math
Lesley-Math
MCLA-Science
Salem-Math
Springfield PS-Science
Wareham PS-Math
WPI-Math
EduTron/Fitchburg-Math
Harvard-Math
Lesley-Math
MCLA-Science
Salem-Math
Springfield PS-Science
Wareham PS-Math
WPI-Math
EduTron/Fitchburg-Math
Harvard-Math
Lesley-Math
MCLA-Science
Salem-Math
Springfield PS-Science
Wareham PS-Math
WPI-Math
Grant
Year 3
Extension
Grant Year 1
Grant Year 2
Grant Year 3
MCLA-Math
PV STEMNET-Math
MCLA-Math
PV STEMNET-Math
MCLA-Math
PV STEMNET-Math
Grant Year 1
Grant Year 2
Grant Year 3
EduTron/Lowell-Math/Sci
EduTron/Fitchburg-Math
Lesley-Math
North Shore-Science
PV STEMNET-Math/Sci
Salem-Math
SE/Cape-Science
WPI-Science
WPS-Math (discontinued)
EduTron/Lowell-Math/Sci
EduTron/Fitchburg-Math
Lesley-Math
North Shore-Science
PV STEMNET-Math/Sci
Salem-Math
SE/Cape-Science
WPI-Science
EduTron/Lowell-Math/Sci
EduTron/Fitchburg-Math
Lesley-Math
North Shore-Science
PV STEMNET-Math/Sci
Salem-Math
SE/Cape-Science
WPI-Science
43
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 38: Overview of MMSP Partnership Participation:
Cohort 4–Cohort 5
Partnership
Grouping
Funding Period
Cohort 4
Cohort 5
MMSP Year 6
MMSP Year 7
Sep08-Aug09
Sep09-Aug10
MMSP Year 8
Sep10-Aug11
Grant Year 1
Grant Year 2
Grant Year 3
Boston PS-Math
Brockton PS-Math
Gateway RSD-Science
Lesley-Math
Northeastern-Science
Randolph PS-Science
Springfield Coll.-Science
Boston U.-Math
Boston PS-Math
Brockton PS-Math
Gateway RSD-Science
Lesley-Math
Northeastern-Science
Randolph PS-Science
Springfield Coll.-Science
Boston U.-Math
Boston PS-Math
Brockton PS-Math
Gateway RSD-Science
Lesley-Math
Northeastern-Science
Randolph PS-Science
Springfield Coll.-Science
Boston U.-Math
Grant Year 1
EduTron-Worc-M EverettUMass Boston-S
Cohorts 1, 2, 3, 4, and 5 combined consisted of 29 partnerships, with 17 of the 29 partnerships offering
mathematics professional development, ten offering science professional development, and two offering
professional development in both mathematics and science content. Specifically, Cohort 1 consisted of eight
partnerships, with six of the eight partnerships offering mathematics professional development and two offering
science professional development. Cohort 2 consisted of two partnerships, both offering mathematics professional
development. Cohort 3 consisted of nine partnerships, with four of the nine partnerships offering mathematics
professional development, three offering science professional development, and two offering professional
development in both mathematics and science. Cohort 4 consisted of eight partnerships, with four of the eight
partnerships offering mathematics professional development and four offering science professional development.
Cohort 5 consisted of two partnerships, with one partnership offering mathematics professional development and
one offering science professional development.
Table 39 shows the funding received by each partnership since the inception of the program. While some
partnerships were awarded funding in more than one funding period, for evaluation purposes, a partnership was
identified as a “new” partnership each time it received funding that resulted from a different competition. Overall,
partnerships have been awarded a total of $15,470,209 since the inception of MMSP.
Over the span of MMSP, Cohort 1, 2, 3, 4, and 5 partnerships developed and implemented a total of 345 courses.
Of those 345 courses, 174 (50%) were unique, and 171 (50%) were repeat offerings. Of the 345 courses, 232
(67%) of the courses offered mathematics content, 111 (32%) offered science and/or technology/engineering
content, and two (1%) offered both mathematics and science/technology content. This section of the report
summarizes data collected from participants in these courses. In total, there were 3,079 unique participants, and
1,123 of them took two or more courses. The term “unique participant” refers to each individual who participated
in the program, regardless of how many courses he or she took. Data for items from the Participant Background
Survey that help to convey participants’ professional backgrounds and motives for participation are discussed in
the remainder of this section. In addition, this section discusses information obtained from partnerships on the
extent to which courses offered through MMSP became institutionalized.
UMass Donahue Institute
Research and Evaluation Group
44
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 39: Budgets: All Partnerships, All Funding Periods
Feb04Aug06
Sep06Aug07
EduTron (M)
$770,000
$68,352
$838,352
Harvard Graduate School of Ed. (M)
$489,899
$87,425
$577,324
Lesley University (M)
$810,726
$43,838
$854,564
MCLA – Science (S)
$133,192
$38,247
$171,439
Salem State College (M)
$541,995
$43,648
$585,643
Springfield Public Schools (S)
$500,044
$74,737
$574,781
Wareham Public Schools (M)
$398,440
$43,962
$442,402
Worcester Polytechnic Institute (M)
$601,778
$35,633
$637,411
Partnership
Sep07Aug08
Sep08Aug09
Sep09Aug10
Sep10Aug11
TOTAL
COHORT 1
Initially funded February 2004
COHORT 2
Initially funded September 2004
MCLA – Math (M)
$111,494
$51,874
$163,368
UMass Amherst (M)
$262,415
$181,581
$443,996
COHORT 3
Initially funded September 2006
EduTron Lowell (M/S)
$210,000
$220,000
$240,000
EduTron Fitchburg (M)
$102,000
$110,000
$120,000
$332,000
Lesley University (M)
$347,911
$355,626
$355,357
$1,058,894
North Shore (S)
$196,474
$194,729
$199,871
$591,074
UMass Amherst (M/S)
$107,424
$216,281
$169,064
$492,769
Salem State College (M)
$120,882
$113,551
$36,604
$271,037
SE/Cape (S)
$129,438
$181,420
$169,246
$480,104
$99,586
$70,734
$94,852
$265,172
Worcester Polytechnic Institute (S)
Worcester Public Schools (M)
UMass Donahue Institute
Research and Evaluation Group
$231,210
$670,000
$231,210
45
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 39: Budgets: All Partnerships, All Funding Periods
Sep08Aug09
Sep09Aug10
Sep10Aug11
Boston Public Schools (M)
$157,975
$405,747
$218,986
$782,708
Brockton Public Schools (M)
$180,145
$255,758
$251,263
$687,166
Gateway Regional School Dist (S)
$186,609
$200,370
$172,379
$559,358
Lesley Springfield (M)
$228,593
$324,820
$308,416
$861,829
Greater North Shore (S)
$265,917
$306,690
$266,480
$839,087
Randolph Public Schools (S)
$176,993
$183,150
$151,178
$511,321
Springfield College (S)
$161,062
$148,896
$156,832
$466,790
Boston University (M)
$241,586
$245,180
$244,394
$731,160
EduTron (M)
$200,000
$200,000
Everett (M)
$149,250
$149,250
$2,119,178
$15,470,209
Partnership
Feb04Aug06
Sep06Aug07
Sep07Aug08
TOTAL
COHORT 4
Initially funded September 2008
COHORT 5
Initially funded September 2010
TOTAL
$4,619,983
$2,214,222
$1,462,341
$2,983,874
$2,070,611
Position of Participants
At the time of their last completed survey from an MMSP course, 93% of course participants identified themselves as teachers. Of all respondents, 71% were
regular education teachers; 15% were special education or special education inclusion teachers; 3% were ELL, ESL, or Sheltered English Immersion teachers; 2%
were department heads or curriculum coordinators; 1% were non-teaching math or science coaches; 1% were principals, assistant principals, or headmasters; 1%
were support specialists; 1% were paraprofessionals; 1% were long-term substitutes; <1% were superintendents or assistant superintendents; and 3% indicated that
they held “other” positions.
Content Taught
The distribution of teaching areas of respondents at the time of the survey is shown in Table 40. Because respondents identified all teaching areas that applied to
their positions at the time of the survey, some selected multiple responses, so frequency totals exceed the number of respondents and percentages exceed 100%.
UMass Donahue Institute
Research and Evaluation Group
46
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
At the time of their last MMSP course, 39% were teaching mathematics, 29% were teaching science, and 27%
were teaching all subjects at the elementary level.
Table 40: Teaching Areas: All Participants, All Funding Periods
Total
Teaching Areas
(Multiple responses permitted)
Mathematics
Any science area
General Science
Biology
Earth Science
Chemistry
Physics
Technology/Engineering
Elementary (all subjects)
Elementary Mathematics
Other
Not Currently Teaching
N =3079
n
%
1214
885
507
213
115
139
137
62
829
182
101
211
39%
29%
17%
7%
4%
5%
4%
2%
27%
6%
3%
7%
Types of Schools of Participants
For each funding period of the program, at least 95% of MMSP participants worked in a public school setting.
Over the course of the program to date, 97% of MMSP participants worked in a public school setting, and 2%
worked in a non-public school setting. Table 41 provides a breakdown, by funding period, of the types of schools
in which participants worked.
High Need Status of Districts of Participants
MMSP partnerships were required to include at least one high need district. Appendix D identifies the criteria for
the high need designation. In addition, the ESE expected that at least 50% of participants in each partnership
would come from high need districts, and further, they set an informal goal that at least 75% of participants for
each partnership would come from high need districts. The high need status of some school districts changed
across years. To classify participants from districts that were high need at one point in time but not high need at
another, a process was used that took into account the high need status of participants’ districts from the
beginning of each partnership’s MMSP involvement with the program. A district identified as high need in the
first year of a partnership’s funding continued to be classified as high need in subsequent years of the partnership,
even if the district’s status changed. Additionally, any districts not on the high need list in the first year of a
partnership’s funding but subsequently added to the high need list in later years of the partnership were then
identified as qualifying for high need district designation.
As a whole, across all years of funding, 67% of participants were from high need districts. Table 42 shows that
across all years of funding, 68% of the public school participants in the program as a whole had come from high
need districts and that for each year of funding, over 50% of public school participants in the program had come
UMass Donahue Institute
Research and Evaluation Group
47
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 41. Types of Schools: All Unique Participants, All Funding Periods
Feb04Aug04
Sep04Aug05
Sep05Aug06
Sep06Aug07
Sep07Aug08
Sep08Aug09
Sep09Aug10
Sep10Aug11
n
%
n
%
n
%
n
%
n
%
n
%
n
n
%
n
%
332
97
448
98
455
98
533
96
462
97
769
98
614
98
600
95
2978
97
8
2
7
2
6
1
12
2
7
2
15
2
16
3
29
5
101
3
1
<1
1
<1
3
1
8
<1
8
2
4
<1
0
0
0
0
0
0
341
100
456
100
464
100
553
100
477
100
788
100
630
100
629
100
3079
100
School Type
Public Schools
(includes public
charter schools)
Non-public School
Other or No
Response
TOTAL
%
Total
from high need districts. An examination of high need district participation in individual partnerships reveals that over the course of each partnership’s
involvement in MMSP since the beginning of MMSP, 18 of the 29 partnerships had at least 50% of their participants coming from high need districts across all
years of their involvement, and (again, across all years of their involvement) 12 of 29 partnerships exceeded the informal goal of having at least 75% of the
participants come from high need districts. If individual years of participation are examined, it is seen that 14 of the 29 partnerships had at least 50% of their
Table 42: High Need Status of All Unique Participants from Public Schools
Feb04Aug04
Sep04Aug05
Sep05Aug06
Sep06Aug07
Sep07Aug08
Sep08Aug09
Sep09Aug10
Sep10Aug11
n
%
n
%
n
%
n
%
n
%
n
%
n
n
%
n
%
202
61
276
62
254
56
397
74
335
73
544
69
412
67
437
70
2037
68
129
39
161
36
196
43
126
24
122
26
240
31
202
33
191
30
927
31
1
<1
11
2
5
1
10
2
5
1
4
<1
0
0
1
<1
14
1
332
100
448
100
455
100
447
100
462
100
788
100
614
100
629
100
2978
100
School Type
High Need
District
Non-high Need
District
Unknown or No
Response
TOTAL
UMass Donahue Institute
Research and Evaluation Group
%
Total
48
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
participants coming from high need districts for each and every year of funding, and 12 of the 29 partnerships
exceeded the informal goal of having at least 75% of the participants come from high need districts for each and
every year of funding. The table in Appendix H shows the number of participants from high need districts
organized by partnership.
As of their last course in MMSP, of the 1,123 individuals who took multiple courses, approximately 71% were
from high need public school districts, approximately 26% were from other public school districts, and
approximately 3% either were from private schools or did not provide information on their districts. In addition,
less than 1% were from high need districts for some of the courses they took but not for others3.
Reasons for Participation in MMSP Courses
For each course taken, respondents identified all of their reasons for participating. Unlike in most other portions
of this report where data were presented for unique participants, data for this issue are presented for all course
seats, since responses are uniquely relevant to each distinct course that a participant took. Table 43 presents
findings for all 5,952 seats for all courses taken by all participants across all funding periods.
Table 43: Reasons for Participation of Participants: All Seats, All
Funding Periods
Total
Reasons for Participation
(Multiple responses permitted)
%
n
of 5,952 course
seats
To increase knowledge in content
4324
73%
To obtain graduate credit
4050
68%
To earn PDPs for recertification
1813
30%
To pursue a personal interest
1878
32%
To get an additional license (certification)
1055
18%
To prepare for the Massachusetts Test for Educator Licensure
(MTEL)
891
15%
To follow an administrator’s suggestion
417
7%
To earn PDPs for HOUSSE plan requirement
331
6%
To obtain a first license (certification)
178
3%
Other
343
6%
Repeat Participation
Partnerships were very successful at encouraging and retaining repeat participants. Of the 29 partnerships, all
offered multiple courses, and all had participants who attended more than one course. In all, 1,123 participants
(36% of all participants) attended multiple courses. Of those participating in multiple courses, 256 took courses
across partnerships. Table 44 provides details regarding repeat participation.
3
Teachers who took MMSP courses from a math partnership when their districts were considered high need for only science were
identified as having come from a “non-high need” district, and teachers who took MMSP courses from a science partnership when their
districts were considered high need for only math were identified as having come from a “non-high need” district.
UMass Donahue Institute
Research and Evaluation Group
49
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 44: Repeat Participants: All Partnerships, All Funding Periods
Partnership
Number of
Courses
Delivered
to Date
Total
Number of
Unique
Participants
to Date*
Number Taking
Multiple
Courses within
Own Cohort
EduTron (M)
7
128
36
54
Harvard (M)
8
166
24
26
19
107
83
85
Lesley Univ. C1 (M)
MCLA (S)
Total Number
Taking
Multiple
Courses
3
23
13
13
26
162
59
81
Springfield PS (S)
7
96
41
43
Wareham PS (M)
3
43
12
12
WPI (M)
6
145
47
48
Salem State College (M)
MCLA (M)
4
16
9
9
UMass Amherst (M)
11
76
39
52
EduTron Lowell (M/S)
10
166
34
34
EduTron Fitchburg (M)
7
125
46
56
Lesley Univ. C3 (M)
40
198
124
124
North Shore (S)
30
121
53
52
UMass Amherst C3 (M/S)
14
135
36
46
Salem State C3 (M)
SE/Cape (S)
WPI (S)
Worcester PS(M)
8
78
26
40
15
178
69
70
3
44
5
5
3
41
6
5
Boston PS (M)
29
270
76
84
Brockton PS (M)
15
249
3
30
Gateway RSD (S)
3
82
40
45
Lesley Springfield (M)
21
109
61
65
Greater North Shore (S)
28
235
95
114
Randolph PS (S)
8
71
28
34
Springfield Coll. (S)
6
52
21
24
Boston University (M)
6
101
17
19
EduTron (M)
4
87
10
24
Everett (S)
Across All Partnerships
2
40
3
8
345
3079
386
1123
*Within each cohort, participants who participated in multiple courses across partnerships were counted only once in the
partnership of their most recent course
Highly Qualified Status
To comply with the federal No Child Left Behind legislation, public school teachers were required to meet the
federal definition of highly qualified by the end of the 2005–2006 school year. One of the expectations of the
MMSP is that providing high quality professional development would contribute to teachers’ attainment of
federal highly qualified status.
UMass Donahue Institute
Research and Evaluation Group
50
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Information regarding the following areas was used to determine participants’ highly qualified status: licensure,
years in education, subject areas taught, High Objective Uniform State Standard of Evaluation (HOUSSE) plans
held, Professional Development Points (PDPs) held, MTELs passed, degrees held, undergraduate degree
equivalents, and advanced or national certifications.
To be considered highly qualified, a teacher must be licensed and demonstrate subject matter competency in the
areas of teaching. Demonstration of subject matter competency for elementary teachers was satisfied either by
passing the appropriate MTEL (general curriculum MTEL if teaching multiple core subjects or elementary math
MTEL if teaching mainly elementary math) or by having a HOUSSE plan and having earned a particular number
of PDPs. (The HOUSSE plan option was phased out in 2007.) Demonstration of subject matter competency for
middle and secondary school teachers was satisfied by one of the following means: passing the appropriate
MTEL, completing an appropriate undergraduate major or graduate degree, completing appropriate coursework
comparable to an undergraduate major, holding advanced or national certification in the appropriate subject area,
or by having a HOUSSE plan and having earned a particular number of PDPs. For all teachers with HOUSSE
plans, the minimum numbers of PDPs needed varied in relation to the date of June 30, 2006: Prior to June 30,
2006, the minimum was 48, and following June 30, 2006, the minimum was 96. Appendix F outlines options
available for demonstrating subject matter competency. A participant was identified as highly qualified if the
criteria for meeting highly qualified status were met for all subjects that a participant taught. If a participant taught
multiple subject areas and only met the highly qualified criteria for some of the subjects taught, he or she was
determined to be “highly qualified in some, but not all” content areas.
In the first MMSP funding period, the Participant Background Survey did not adequately capture information
about teachers that could be used to determine highly qualified status. The survey was re-designed for Year 2 to
capture this information. For the second and subsequent funding periods, though, survey responses indicated that
participants misunderstood the meaning of HOUSSE plans, and some who completed more than one survey
reported inconsistent data across surveys.
The Participant Background Survey permitted determination of the impact of MMSP courses on highly qualified
status only for 1) those participants who held HOUSSE plans and 2) those participants who took more than one
course (i.e., completed more than one survey). Because surveys were administered after participants had
completed MMSP courses, if a participant completed only one course and did not have a HOUSSE plan it was not
possible to determine whether that participant became highly qualified prior to or as a result of MMSP course
participation.
Over the span of the program for all Cohorts, at least 174 participants attained highly qualified status. Table 45
presents the highly qualified status of participants across all years of the program. This table provides an
unduplicated count of participants. Because the number of courses a participant took was relevant to the process
used to determine how many participants attained highly qualified status while participating in MMSP, the
findings are organized according to number of courses taken (only one vs. multiple). In Appendix I, a more
detailed version of the table is presented in which the data are further broken down by cohort of participation.
UMass Donahue Institute
Research and Evaluation Group
51
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 45: Highly Qualified Status: All Unique Participants, All Funding
Periods
Status
Became Highly Qualified
Took Only One Course Took Multiple Courses
n = 1956
n = 1123
TOTAL
n = 3079
14
160
174
0
0
5
1019
685
1704
64
41
105
Not Highly Qualified
594
270
864
Private school or
not teaching
123
72
195
Unknown
156
55
211
Became Highly Qualified
in only some content
areas
Highly Qualified (unable
to determine when
became HQ)
Highly Qualified in some
content areas but not all
(unable to determine
when became HQ)
MTEL Information
One method by which teachers may demonstrate subject-matter competency is to pass the Massachusetts Tests for
Educator Licensure (MTEL) in the content areas that they teach. Table 46 cumulatively identifies the tests taken
by participants across all years of the program along with passage rates. Of the 1,123 participants taking multiple
courses, 26% earned one or more new licenses by the end of the eighth year of MMSP, and 12% took and passed
an MTEL test. In addition, over 1% of these participants learned that they had passed an MTEL that they had
taken during a prior MMSP course.
As shown in Table 48, based on data from the last survey completed by each participant, of the 298 participants
who had taken the Mathematics MTEL, 242 (81%) reported passing the test, and 20 (7%) had not yet received
their scores at the time of survey completion. Of the 501 respondents who had taken the Middle School
Mathematics MTEL, 433 (86%) passed and 28 (6%) had not yet received their scores. Of the 96 participants who
completed the Middle School Mathematics/Science MTEL, 62 (65%) passed and 14 (15%) had not yet received
their scores. Of the 243 participants completing the General Science MTEL, 215 (89%) passed and 15 (6%) had
not yet received their scores. The Biology MTEL was attempted by 132 respondents, and 122 (92%) passed and
five (4%) had not yet received scores. Of the 79 participants who took the Chemistry MTEL, 67 (85%) passed and
three (4%) had not yet received scores. Of the 26 who took the Earth Science MTEL, 18 (69%) passed and three
(12%) had not yet received scores. Fifty participants attempted the Physics MTEL, and 35 (70%) passed while six
(12%) still awaited scores. Twelve individuals completed the Technology/Engineering MTEL, and ten (83%)
passed. Of the 491 participants who reported taking General Curriculum (formerly elementary) MTEL, 476 (97%)
reported passing, and seven (1%) had not yet received scores.
UMass Donahue Institute
Research and Evaluation Group
52
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 46: MTEL Tests Taken by All Participants – Total to Date
Based on each participant’s last survey
Taking Test
Passing Test
Failing Test
n
n
%
491
476
97
8
2
7
1
51
44
86
2
4
5
10
Elementary Mathematics
114
98
86
6
5
10
9
Mathematics
298
242
81
36
12
20
7
Middle School Mathematics
501
433
86
40
8
28
6
96
62
65
20
21
14
15
General Science
243
215
89
13
5
15
6
Biology
132
122
92
5
4
5
4
Chemistry
79
67
85
9
11
3
4
Physics
50
35
70
9
18
6
12
Earth Science
26
18
69
5
19
3
12
Technology/Engineering
12
10
83
2
17
0
0
638
529
83
10
46
7
General Curriculum (formerly
Elementary)
Early Childhood
Middle School
Mathematics/Science
TOTAL in STE Areas
n
63
%
Scores
Unknown
n
%
Licensure and Degrees Held in Content Area Taught
Over the course of the program, 737 regular education teachers 95 special education teachers, and 36 ELL
teachers reported teaching in science or technology/engineering areas. Tables 47a, 47b, and 47c show how many
teachers taught in each science and technology/engineering area over the course of the program. The tables also
show the percentages of teachers whose survey responses indicated they were licensed in the area in which they
taught, and they show the percentages of teachers whose survey responses indicated they held a degree in the area
in which they taught. Table 47a provides information for regular education teachers, Table 47b provides
information for special education teachers, and Table 47c provides information for ELL teachers. (The numbers
presented in Tables 47a, 47b, and 47c exceed the number of teachers who reported teaching in these areas because
some teachers taught in more than one area.) For regular education teachers, the licensing reported by
approximately 57% appeared to be appropriate for the content area taught, and the degrees held by approximately
27% corresponded to content area taught. For special education teachers, the licensing reported by approximately
48% appeared to be appropriate for the content area taught, and the degrees held by two teachers (2%)
corresponded to content area taught. For ELL teachers, the licensing reported by approximately 45% appeared to
be appropriate for the content area taught, and the degrees held by 24% corresponded to content area taught.
UMass Donahue Institute
Research and Evaluation Group
53
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Table 47a: Science and Tech/Engineering Teaching Areas of All
Participants – Regular Education
Total
Content Area
General Science
Biology
Chemistry
Earth Science
Physics
Technology/Engineering
Teach in Area
License in Area
Taught
Degree in Area
Taught
n
%*
%*
419
63
13
169
75
66
123
59
39
88
26
13
115
40
21
56
41
16
*Of the number of regular education participants teaching in this area across all program years
Table 47b: Science and Tech/Engineering Teaching Areas of All
Participants – Special Education
Content Area
General Science
Biology
Chemistry
Earth Science
Physics
Technology/Engineering
Teach in Area
Total
License in Area
Taught
Degree in Area
Taught
n
%*
%*
60
48
3
26
42
0
9
67
0
19
37
0
13
69
0
3
33
0
*Of the number of special education participants teaching in this area across all program years
Table 47c: Science and Tech/Engineering Teaching Areas of All
Participants – ELL Education
Content Area
General Science
Biology
Chemistry
Earth Science
Physics
Technology/Engineering
Teach in Area
Total
License in Area
Taught
Degree in Area
Taught
n
%*
%*
20
55
15
11
46
46
4
25
25
5
20
20
6
50
33
3
33
0
*Of the number of ELL participants teaching in this area across all program years
UMass Donahue Institute
Research and Evaluation Group
54
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Over the course of the program and by the end of the 2010–2011 funding period, 944 regular education teachers,
217 special education teachers, and 36 ELL teachers reported teaching non-elementary level mathematics. Tables
48a, 48b, and 48c show how many teachers taught at each non-elementary mathematics level over the course of
the program. The tables also show the percentages of teachers whose survey responses indicated they were
licensed in the level at which they taught, and they show the percentages of teachers whose survey responses
indicated they held a degree in the area in which they taught. Table 48a provides information for regular
education teachers, Table 48b provides information for special education teachers, and Table 48c provides
information for ELL teachers. For regular education teachers, the licensing reported by approximately 79%
appeared to be appropriate for the mathematics level taught, and 22% held mathematics degrees. For special
education teachers, the licensing reported by approximately 36% appeared to be appropriate for the level taught,
and 4% held mathematics degrees. For ELL teachers, the licensing reported by 47% appeared to be appropriate
for the level taught, and 14% held mathematics degrees.
Table 48a: Mathematics Teacher Levels of All Participants – Regular
Education
Level
Teach in Area
Total
License in Area
Taught
Degree in Math
n
%*
%*
Middle School
High School
MS & HS grades
751
181
12
77
87
83
16
44
25
TOTAL Math
944
79
22
*Of the number of regular education participants teaching in this area for this year
Table 48b: Mathematics Teacher Levels of All Participants – Special
Education
Level
Teach in Area
Total
License in Area
Taught
Degree in Math
n
%*
%*
Middle School
High School
MS & HS grades
172
38
7
38
29
43
2
11
14
TOTAL Math
217
36
4
*Of the number of special education participants teaching in this area for this year
Table 48c: Mathematics Teacher Levels of All Participants – ELL Education
Level
Teach in Area
Total
License in Area
Taught
Degree in Math
n
%*
%*
Middle School
High School
MS & HS grades
29
6
1
41
67
100
7
50
0
TOTAL Math
36
47
14
*Of the number of ELL participants teaching in this area for this year
UMass Donahue Institute
Research and Evaluation Group
55
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Degrees Currently Pursued
Information on degrees currently being pursued in science, technology/engineering, and mathematics was derived
from the most recently completed survey of each individual. Of all participants through the 2010–2011 funding
period, 260 reported that they were pursuing degrees in science and technology/engineering or science education.
Table 49 provides details about the types of degrees being pursued and the teaching areas of those who were
pursuing the degrees, and it shows that these participants were pursuing a total of 303 undergraduate and graduate
degrees. A degree was identified as “In Area” if the content matter being studied corresponded to the area in
which a teacher taught; a degree was identified as “Not in Area” if the content matter being studied did not
correspond to the area in which a teacher taught or if it was a science education degree. A science education
degree was identified as “Not in Area” because this degree is not linked to a specific science subject. Degrees
pursued by teachers who taught multiple science subjects were included in the row labeled “Multiple Science
Areas” and those degrees were classified as “In Area” if the content matter being studied corresponded to any area
in which a teacher taught.
Table 49: Pursuit of Science Degrees: All Participants, All Funding Periods
Type of Science Degree Pursued
MA/MS
CAGS
BA/BS
Teaching Area
In Area
Not in
Area
In Area
Not in
Area
In Area
Doctorate
Not in
Area
In Area
Not in
Area
General Science
2
3
23
80
3
2
0
1
Biology
0
0
2
25
0
2
0
0
Chemistry
0
0
4
11
0
0
0
0
Physics
0
0
4
12
0
0
0
0
Earth Science
0
0
0
6
0
0
0
0
Technology/Engineering
1
0
2
2
0
0
0
0
Multiple Science Areas
4
4
11
30
4
2
0
1
Elementary
0
23
1
1
Other
2
34
0
1
Total
7
9
46
223
7
7
0
4
Of all participants through the 2010–2011 funding period, 263 were pursuing degrees in mathematics or
mathematics education. Table 50 provides details about the types of degrees being pursued and the teaching areas
of those who were pursuing the degrees, and it shows that these participants were pursuing a total of 12
undergraduate and 260 graduate degrees.
Content Knowledge Gains
As a grant condition, partnerships were required to use a pre-course test and post-course test for each MMSP
course to assess participants’ knowledge growth of the content taught in the course. For each course, the same
instrument served as both the pre- and post-course test. While partnerships were permitted to locate and use an
instrument with established reliability and validity, it was most often the case that such instruments were not
available because a priority was placed on utilizing assessments that would reflect the precise content taught in
each course. As a result, the faculty members who had developed the courses usually developed the assessments,
UMass Donahue Institute
Research and Evaluation Group
56
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
and typically neither validity nor reliability was determined for them because time and resource constraints
prohibited doing so.
Table 50: Pursuit of Mathematics Degrees: All Participants, All
Funding Periods
Mathematics Degree Pursued
MA/MS
CAGS
Courses taught
BA/BS
Elementary Math
2
10
9
0
Middle School Math
2
91
16
2
High School Math
1
38
4
1
MS & HS Math
0
2
1
0
7
64
17
2
12
208
47
5
Other (non-Math)
Total
Doctorate
Data were analyzed only for participants who completed both pre- and post-course assessments. To determine if
the scores showed statistically significantly changes between pre- and post-course test administrations, a paired
samples t-test was used for each course for which ten or more participants completed both pre- and post-course
assessments, and a Wilcoxon Matched Pairs test was used for each course for which fewer than ten participants
completed both pre- and post-course assessments.
Of the 345 courses that were delivered across all partnerships through the 2010–2011 funding period, content
assessments were administered for 338 courses. Of these 338 courses, 334 had gains in the average percentage of
items correct between pre- and post- test administrations. Statistically significant improvements in scores on
content knowledge assessments occurred in 85% of the 338 courses. Of the 51 courses not showing statistically
significant improvement in scores, though, 34 had fewer than six participants4, the smallest sample size at which it
is possible to detect statistical significance at the level used for these analyses. Table 51 provides an overview, by
subject matter of courses delivered, of the numbers of courses that showed statistically significant gains. For
detailed information on mean pre- and post-course content knowledge scores by partnership and by course, see
Appendix G.
Table 51: Number of Courses with Statistically Significant
Gains in Mean Content Knowledge Scores
Total
Content Area
Delivered,
with Pre- and Posttests
Significant
Pre/Post
Gains
Math
225
197
Science & Technology/Engineering
Math and Science
111
2
88
2
TOTAL
338
287
4
Some of these courses had fewer than six MMSP enrollees within the context of another course that included additional non-MMSP
enrollees; other courses were not able to administer content assessments to all participants because of logistical challenges.
UMass Donahue Institute
Research and Evaluation Group
57
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Course Institutionalization
For systemic change to occur at the higher education institutions, departments of arts and sciences and education
departments are encouraged to work together through MMSP to support stronger content courses in mathematics
and science for teacher preparation, undergraduate and graduate degree requirements, and for in-service teachers
pursuing graduate-level content courses for recertification. Integration of Title IIB courses into graduate programs
at Institutes of Higher Education ensures sustainability over time. The intent behind encouraging the partnerships
is that the faculty from the Arts and Sciences Departments bring strong content expertise to the partnership table.
This integration creates greater opportunities for participants to complete coursework leading to a content-area
degree and/or to licensure along with the highly qualified designation.
Since the 2006–2007 funding period, partnerships have been asked to describe activities that were related to the
institutionalization of their courses. Many partnerships evinced integration, plans for future integration, or—in the
case of partnerships with previously established involvement with MMSP—work toward sustaining prior
integration. As would be expected in a program involving partnerships with diverse structures and styles, the
extent and type of integration varied across partnerships. To convey a sense of how integration occurred,
following are significant activities, grouped according to partnership:
EduTron Lowell Public Schools (M/S) and EduTron Fitchburg State College (M)
 Two developmental courses, based on the EduTron model for MMSP courses, will continue to be offered
at Fitchburg State College (FSC). EduTron partners supported FSC in designing three pre-service courses
that are optimized for education majors. EduTron has begun working with FSC to help FSC apply the
EduTron model used in MMSP math courses to science courses.
 FSC has partnered with Lowell Public Schools to offer a teacher certification/CAGS program.
 Six mathematics and four science courses were approved by FSC as offerings at the continuing education
level.
Lesley University C3 (M)
 Two courses created through Lesley University’s MMSP in 2007-2008 are now offered to Lesley’s oncampus pre-service teachers.
 Efforts through MMSP contributed to the development of an online Mathematics Education program at
Lesley leading to the Master of Arts degree for elementary and middle school teachers.
 Nine math content courses were developed through participation in the MMSP program in 2007-2008 and
in prior years. All of these courses are part of Lesley University’s mathematics major for undergraduates,
which would not have been possible without the MMSP program.
North Shore (S)
 As a result of their joint involvement in MMSP through the North Shore partnership and the National
Science Foundation MSP program, Northeastern University has institutionalized all MMSP courses. Ten
MMSP courses can be used to fulfill degree requirements toward a Master’s in Education for Middle
School Science. In addition, this degree was developed as a result of these courses.
UMass Amherst C3 (M/S)
 Four courses developed through the UMass Amherst partnership were approved for graduate level credit.
Salem State C3 (M)
 Salem State College offers courses developed through MMSP as part of a master’s level teaching
program in middle school mathematics. All courses developed by Salem State College through MMSP
can be applied towards earning a degree through that program.
UMass Donahue Institute
Research and Evaluation Group
58
Annual Report of the MMSP State-level Evaluation
Cumulative Summary
Southeast/Cape (S)
 Participants of the three courses offered through the SE/Cape partnership may apply credit for the courses
towards the Master of Arts in Teaching in Physical Science program that is offered through Bridgewater
State College.
Worcester Polytechnic Institute (S)
 A Master of Science Education program was created through the physics department at Worcester
Polytechnic Institute, and the MMSP course that was offered through the WPI-Science partnership will
serve as the model for instruction of future courses that will be offered.
Worcester Public Schools (M)
 As a result of the experience of working with Worcester Public Schools on MMSP, Clark University has
expressed interest in exploring the institutionalization of courses that were offered through MMSP.
Springfield College (S)
 Springfield College has incorporated into its pre-service Best Practices of Teaching Science course
activities from an MMSP course that are designed to help teachers understand how to change
misconceptions that students have about life science.
Randolph Public Schools (S)
 Four courses developed through the Randolph Public Schools partnership were approved for graduate
level credit at Bridgewater State University.
Scaling Up
While this evaluation did not set out to explore the reach of partnerships beyond documenting the numbers of
participants and their high need districts of origin, an exceptional instance of scaling up emerged through data
collection efforts. Since it speaks to the goals of MMSP and also is an indicator of project success, it is being
included here. As its professional development model, the Brockton Public Schools partnership used the 80-hour
Massachusetts Intel Math Initiative (MIMI) course and professional learning community follow-up. Through their
participation in MMSP, they—in effect—regionalized the program, expanding the model from a relatively small
partnership between school districts and higher education professors to include over 28 Southeastern
Massachusetts districts and three institutions of higher education.
UMass Donahue Institute
Research and Evaluation Group
59
Annual Report of the MMSP State-level
Evaluation
Appendix I
Summary of Findings
The MMSP partnership activities summarized in this section of the report occurred between February 2, 2004, and August
31, 2011. This period spans the beginning of the MMSP program through the end of the 2010–2011 funding period. Since
MMSP began in February 2004, progress has been made towards meeting the goals of the program as evidenced by the
following data for both the program as a whole, since its inception, and for both Cohort 4 and Cohort 5.
Cumulative Findings
Overview of Partnerships

A total of 29 partnerships were funded across the Commonwealth. Of these, 17 were organized around
mathematical content, ten were organized around science content, and two were organized around both
mathematical and science content. Of the 29 MMSP partnerships, all delivered courses, all but two offered
multiple courses, and all had participants who attended more than one course.
 Cohort 1, which began in February 2004, consisted of eight partnerships, with six of the eight partnerships
offering mathematics professional development and two offering science professional development.

Cohort 2, which began in September 2004, consisted of two partnerships, both offering mathematics
professional development.

Cohort 3, which began in September 2006, consisted of nine partnerships, with five of the nine partnerships
offering mathematics professional development, three offering science professional development, and one
offering professional development in both mathematics and science content.

Cohort 4, which began in September 2008, consisted of eight partnerships, with four of the eight partnerships
offering mathematics professional development and four offering science professional development.

Cohort 5, which began in September 2010, consisted of two partnerships, with one partnership offering
mathematics professional development and one offering science professional development.
Overview of Courses

In total, 345 MMSP courses were delivered by the end of Year 8 of MMSP funding. Of these 345 courses, 232
were mathematics courses, 111 were science and/or technology/engineering courses, and two were courses
offering both mathematics and science content.
Overview of Participants

In total, 3,079 unique participants participated in MMSP courses by the end of Year 8.
 1,123 participants (36% of all participants) attended multiple courses. Of the 1,123 participating in multiple
courses, 256 took courses across partnerships.


5,952 course seats were filled by all participants across all funding periods.
Types of Schools of Participants
 Of all 3,079 unique participants, 97% came from public schools (including public charter schools) and 3%
came from non-public schools, and <1% did not indicate their school type.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation

Appendix I
High Need Status of Districts of Participants
 The partnerships exceeded the ESE target of having at least 50% of all participants come from high need
districts, with 67% of all participants in the program coming from high need districts.

Across all years of their involvement, 18 of the 29 partnerships had at least 50% of their participants coming
from high need districts.

For each and every individual year of funding, 14 of the 29 partnerships had at least 50% of their participants
coming from high need districts.

Across all years of their involvement, 12 of 29 partnerships had at least 75% of the participants come from
high need districts.

For each and every individual year of funding, 12 of the 29 partnerships had at least 75% of the participants
come from high need districts.
As the following data reveal, not all MMSP participants were licensed in their teaching areas or held relevant degrees,
indicating that course participants were, in fact, those in need of the courses:
 Licensure in Mathematics and Science Content Areas for Middle and High School Teachers
 Of the regular education mathematics courses taught by MMSP teachers, 79% were taught by teachers who
were licensed in mathematics.


Of the special education mathematics courses taught by MMSP teachers, 36% were taught by teachers who
were licensed in mathematics.

Of the ELL mathematics courses taught by MMSP teachers, 47% were taught by teachers who were licensed
in mathematics.

Of regular education science or technology/engineering courses taught by MMSP teachers, 57% were taught
by teachers who were licensed in the subject of the course.

Of special education science or technology/engineering courses taught by MMSP teachers, 48% were taught
by teachers who were licensed in the subject of the course.

Of ELL science or technology/engineering courses taught to students of MMSP teachers, 45% were taught by
teachers who were licensed in the subject of the course.
Degrees Held in Content Area in which Teaching
 Of regular education mathematics courses taught by MMSP teachers, 22% were taught by teachers who held
mathematics degrees.

Of special education mathematics courses taught by MMSP teachers, 4% were taught by teachers who held
mathematics degrees.

Of ELL mathematics courses taught to students of MMSP teachers, 14% were taught by teachers who held
mathematics degrees.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation
Appendix I

Of regular education science or technology/engineering courses taught by MMSP teachers, 27% were taught
by teachers who held degrees that were relevant to the focus of the course.

Of special education science or technology/engineering courses taught by MMSP teachers, 2% were taught by
teachers who held degrees that were relevant to the focus of the course.

Of ELL science or technology/engineering courses taught by MMSP teachers, 24% were taught by teachers
who held degrees that were relevant to the focus of the course.
Highly Qualified Status
MMSP has contributed to increasing the numbers of highly qualified public school teachers.
 Attaining Highly Qualified Status
 By the end of Year 8, of the participants who had entered MMSP as not highly qualified, 174 had attained
highly qualified status.

New Licensures
 Of the 1,123 participants taking multiple courses, 26% earned one or more new licenses by the end of the
Year 8 of MMSP, and 12% took and passed an MTEL test.
Cohort 4 Findings
Overview of Partnerships

Eight Cohort 4 partnerships were funded through the 2010–2011 funding period. Of these, four were organized
around mathematical content, four were organized around science content. Of the eight Cohort 4 MMSP
partnerships, all delivered courses. Of the eight partnerships, all offered multiple courses, and all had participants
who attended more than one course.
Overview of Courses

In total, 115 Cohort 4 courses were delivered by the end of Year 8 of MMSP funding. Of these 115 courses, 73
were mathematics courses and 42 were science or technology/engineering courses.
Overview of Participants


By the end of Year 8, 1,169 unique Cohort 4 participants participated in MMSP courses.
 341 participants (29% of all Cohort 4 participants) attended multiple courses in Cohort 4 across the 2008–
2009, 2009–2010, and 2010–2011 years.

83 participants (7% of all Cohort 4 participants) had attended additional courses outside of Cohort 4, across
all MMSP funding periods.

1,870 course seats were filled by Cohort 4 participants across 2008–2009, 2009–2010, and 2010–2011.

Course attrition rates were generally low and averaged 5% across all Cohort 4 courses.
Types of Schools of Participants
 Of all 1,169 unique Cohort 4 participants, 97% came from public schools (including public charter schools),
and 3% came from non-public schools.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation

Appendix I
High Need Status of Districts of Participants
 The Cohort 4 partnerships exceeded the ESE target of having at least 50% of all participants come from high
need districts, with 67% of all Cohort 4 participants in the program coming from high need districts.

Six of the eight partnerships had at least 50% of their participants coming from high need districts.

Three of the eight partnerships had at least 75% of the participants come from high need districts.
As the following data reveal, not all Cohort 4 participants were licensed in their teaching areas or held relevant degrees,
indicating that course participants were, in fact, those in need of the courses:
 Licensure in Mathematics and Science Content Areas for Middle and High School Teachers
 Of regular education mathematics courses taught by Cohort 4 teachers, 81% were taught by teachers who
were licensed in mathematics.


Of special education mathematics courses taught by Cohort 4 teachers, 51% were taught by teachers who
were licensed in mathematics.

Of ELL mathematics courses taught by Cohort 4 teachers, 62% were taught by teachers who were licensed in
mathematics.

Of regular education science or technology/engineering courses taught by Cohort 4 teachers, 66% were taught
by teachers who were licensed in the subject of the course.

Of special education science or technology/engineering courses taught by Cohort 4 teachers, 69% were taught
by teachers who were licensed in the subject of the course.

Of ELL science or technology/engineering courses taught by Cohort 4 teachers, 48% were taught by teachers
who were licensed in the subject of the course.
Degrees Held in Content Area in which Teaching
 Of regular education mathematics courses taught by Cohort 4 teachers, 18% were taught by teachers who held
mathematics degrees.

Of special education mathematics courses taught by Cohort 4 teachers, 6% were taught by teachers who held
mathematics degrees.

Of ELL mathematics courses taught by Cohort 4 teachers, 19% were taught by teachers who held
mathematics degrees.

Of regular education science or technology/engineering courses taught by Cohort 4 teachers, 37% were taught
by teachers who held degrees that were relevant to the focus of the course.

Of special education science or technology/engineering courses taught by Cohort 4 teachers, 2% were taught
by teachers who held degrees that were relevant to the focus of the course.

Of ELL science or technology/engineering courses taught by Cohort 4 teachers, 22% were taught by teachers
who held degrees that were relevant to the focus of the course.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation
Appendix I
Highly Qualified Status
MMSP has contributed to increasing the numbers of highly qualified public school teachers.
 Attaining Highly Qualified Status
 By the end of the 2010–2011funding period, of the Cohort 4 participants who had entered MMSP as not
highly qualified, 39 had attained highly qualified status.
Content Knowledge Gains
The content knowledge of Cohort 4 participants was increased:
 Statistically significant improvements in scores on content knowledge assessments occurred in 85 of the 110
(96%) Cohort 4 courses for which content assessments were administered.

Gains in average percentage of items correct between pre- and post-course test administrations occurred in 107 of
the 110 (97%) courses delivered across all Cohort 4 partnerships for which tests were administered and scores
were obtained.
Integrating Courses into Higher Education Institutions
Integration or plans for future integration of courses into institutions of higher education occurred within some Cohort 4
partnerships.
Cohort 5 Findings
Overview of Partnerships

Two Cohort 5 partnerships were funded during the 2010–2011 funding period. Of these, one was organized
around mathematical content and one was organized around science content. Both of the Cohort 5 MMSP
partnerships delivered courses. Of the two partnerships, both offered multiple courses, both had participants who
took more than one course within Cohort 5, and both had participants who had attended more than one course
across the entire duration of MMSP.
Overview of Courses

In total, six Cohort 5 courses were delivered by the end of Year 8 of MMSP funding. Of these six courses, four
were mathematics courses and two were science/technology/engineering courses.
Overview of Participants

During Year 8, 127 unique Cohort 5 participants participated in MMSP courses.
 Thirteen participants (10% of all Cohort 5 participants) attended multiple courses in Cohort 5 during 2010–
2011.

21 participants (17% of all Cohort 5 participants) had attended additional courses outside of Cohort 5, across
all MMSP funding periods.

140 course seats were filled by Cohort 5 participants during 2010–2011.

Course attrition rates were low and averaged 5% across all courses offered by Cohort 5 partnerships.
Reaching Targeted Participants

Types of Schools of Participants
 Of all 127 unique Cohort 5 participants, 95% came from public schools (including public charter schools),
and 5% came from non-public schools.
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation

Appendix I
High Need Status of Districts of Participants
 The Cohort 5 partnerships exceeded the ESE target of having at least 50% of all participants come from high
need districts, with 73% of all Cohort 5 participants in the program coming from high need districts.

Both partnerships had at least 50% of their participants coming from high need districts.

One of the two partnerships had at least 75% of the participants come from high need districts.
As the following data reveal, not all Cohort 5 participants were licensed in their teaching areas or held relevant degrees,
indicating that course participants were, in fact, those in need of the courses:
 Licensure in Mathematics and Science Content Areas for Middle and High School Teachers
 Of regular education mathematics courses taught by Cohort 5 teachers, 95% were taught by teachers who
were licensed in mathematics.


Of special education mathematics courses taught by Cohort 5 teachers, 27% were taught by teachers who
were licensed in mathematics.

The one Cohort 5 teacher who taught ELL mathematics courses was not licensed in mathematics.

Of regular education science or technology/engineering courses taught by Cohort 5 teachers, 54% were taught
by teachers who were licensed in the subject of the course.

Of special education science or technology/engineering courses taught by Cohort 5 teachers, 100% were
taught by teachers who were licensed in the subject of the course.

Of ELL science or technology/engineering courses taught by Cohort 5 teachers, 50% were taught by teachers
who were licensed in the subject of the course.
Degrees Held in Content Area in which Teaching
 Of regular education mathematics courses taught by Cohort 5 teachers, 44% were taught by teachers who held
mathematics degrees.

Of special education mathematics courses taught by Cohort 5 teachers, none were taught by teachers who held
mathematics degrees.

The one Cohort 5 teacher who taught ELL mathematics courses did not have a degree in mathematics.

Of regular education science or technology/engineering courses taught by Cohort 5 teachers, 33% were taught
by teachers who held degrees that were relevant to the focus of the course.

Of special education science or technology/engineering courses taught by Cohort 5 teachers, none were taught
by teachers who held degrees that were relevant to the focus of the course.

Of ELL science or technology/engineering courses taught by Cohort 5 teachers, 50% were taught by teachers
who held degrees that were relevant to the focus of the course.
Highly Qualified Status
MMSP has contributed to increasing the numbers of highly qualified public school teachers.
 Attaining Highly Qualified Status
UMass Donahue Institute
Research and Evaluation Group
Annual Report of the MMSP State-level
Evaluation

Appendix I
By the end of the 2010–2011 funding period, of the Cohort 5 participants who had entered MMSP as not
highly qualified, one had attained highly qualified status.
Content Knowledge Gains
The content knowledge of Cohort 5 participants was increased:
 Statistically significant improvements in scores on content knowledge assessments occurred in all six of the
Cohort 5 courses.
Integrating Courses into Higher Education Institutions
Integration or plans for future integration of courses into institutions of higher education did not occur within Cohort 5
partnerships.
UMass Donahue Institute
Research and Evaluation Group