Milwaukee
Mathematics Partnership
Program Evaluation
MTL Meeting
November 6,7 2006
MMP Goals

Comprehensive Math Framework

Distributed Leadership

Teacher Learning Continuum

Student Learning Continuum
November 6-7, 2006
2
MMP Core Partners

University of Wisconsin—Milwaukee

Milwaukee Public Schools

Milwaukee Area Technical College
November 6-7, 2006
3
Evaluation Goals


Help the MMP better serve its
constituents and improve its
effectiveness
Serve the broader mathematics
education community through
documentation and
dissemination of MMP activities
November 6-7, 2006
4
Evaluation Logic Model
Student
Achievement
Distal
Outcomes
Proximal
Outcomes
Teacher Content
& Pedagogical
Knowledge
Classroom
Practice
Teacher
Involvement
Learning Team
Effort
MMP
Activities
New
Courses
Math Faculty
Involvement
MATC
Buy-In
School
Buy-in
District
Buy-in
UWM
Buy-In
MPA Ownership
November 6-7, 2006
5
2005-06 Evaluation Activities







MMP Online Survey
MTS Survey
Learning Team Observations
Classroom Observations
Assessment of Teacher MKT
Social Network Analysis
MPS Data Mining
November 6-7, 2006
6
Presentation Overview

Part I: District Wide Analysis

Part II: School Case Studies
November 6-7, 2006
7
Part I: District Wide Indicators
Student
Achievement
Classroom
Practice
Teacher
Involvement
Learning Team
Effort
November 6-7, 2006
School
Buy-in
8
Part I Activities

MMP Survey


MTS Survey


Designed to measure how well MTS perceived
school to be, in terms of meeting the goals of
the MMP
MKT Assessment


Designed to measure differences in the
quantity and quality of MMP related activities
Designed to assess teachers’ mathematical
knowledge for teaching
WKCE Tests

November 6-7, 2006
Designed to assess students’
mathematical content knowledge
9
Respondents
Number of MPS Respondents by Role in the MMP
Academic Year
2004-05
2005-06
Math Teacher Leader
124
140
Learning Team Member & Mathematics
Teacher
167
284
LT Member (Administrative)
80
171
Math Teacher Only
676
1340
Literacy Coach
47
94
1094
2029
Total
November 6-7, 2006
10
Research Questions
1.
2.
3.
4.
5.
6.
Validity of MMP Survey
Change in MMP Impact
Characteristics of math-focused schools
MMP Impact on student achievement
Characteristics of quality learning teams
Characteristics of quality MTLs
November 6-7, 2006
11
Research Question #2
Has the perceived impact or focus
of the MMP changed since last year?
Analytical Approach
 Dependent t-tests conducted at the
school level for all school level
variables obtained in both
administrations of the MMP Survey

November 6-7, 2006
12
Results – Trends in Impact of MMP

This year statistically significant increases



November 6-7, 2006
MTLs reported discussing mathematics with
others at their school (t(90) = 12.06, p <
.001)
Teachers reported engaging in activities
designed to align their curriculum to standards
(t(111) = 8.53, p < .001)
Teachers reported engaging in activities
designed around CABS or student work
samples (t(106) = 7.04, p < .001)
13
Research Question #3

What variables characterize schools
that are more focused on increasing
student achievement in
mathematics?
Analytical Approach
 Stepwise multiple regression
November 6-7, 2006
14
Results – Characteristics of
Schools with Greater Math Focus

68% of variability in schools’ overall selfreported focus on mathematics could be
explained by differences in:



November 6-7, 2006
Teachers reported working together to improve
content and pedagogical knowledge (b = .46, t
= 6.7, p < .001)
Teachers reported consistent instructional
practices used at their school (b = .14, t = 2.4,
p = .018)
Teachers perceived the Learning Team to be
supportive of efforts to improve math teaching
and learning (b = .38, t = 5.6, p <.001)
15
Research Question #4
What variables help to explain
differences in the percentage of
students classified as proficient in
mathematics?
Analytical Approach
 Stepwise multiple regressions
controlling for previous
achievement and SES

November 6-7, 2006
16
Results – Impact of MMP on Increasing
Student Achievement


Schools with a stronger focus on
increasing student achievement in
mathematics are have a higher
percentage of students proficient in
mathematics, after controlling for SES
and previous achievement (b = .26, t =
3.7, p =.001)
An additional 7% of variability in student
proficiency rates explained by the addition
of this predictor
November 6-7, 2006
17
Research Question #5
What variables characterize
Learning Teams that are perceived
to be more helpful in terms of
increasing student achievement in
mathematics?
Analytical Approach
 Stepwise multiple regression

November 6-7, 2006
18
Results – Characteristics of
Supportive Learning Teams

64% of variability in schools’ overall perception of
the level of support provided by the Learning
Team could be explained by differences in:



November 6-7, 2006
Teachers reported working together on
improvement activities designed around CABS or
student work samples (b = .41, t = 5.5, p < .001)
Teachers reported a greater alignment between
their school’s adopted curriculum and
standards/learning targets (b = .18, t = 2.4, p =
.021)
Teachers perceived the MTL to be supportive of
efforts to improve
mathematics teaching and learning
(b = .46, t = 5.9, p <.001)
19
Research Question #6
What variables characterize Math
Teacher Leaders that are perceived
to be more helpful, in terms of
increasing student achievement in
mathematics?
Analytical Approach
 Stepwise multiple regression

November 6-7, 2006
20
Results – Characteristics of
Supportive Math Teacher Leaders

42% of variability in schools’ overall perception of
the level of support provided by the MTL be
explained by differences in:



November 6-7, 2006
Teachers reported working together on
improvement activities designed around CABS or
student work samples (b = .38, t = 4.5, p < .001)
Teachers reported a greater alignment between
their school’s adopted curriculum and
standards/learning targets (b = .26, t = 3.0, p =
.004)
MTLs perceived themselves as being supported by
others at their school (b = .27, t = 3.2, p = .002)
21
Conclusions





MTSs in general have a strong sense of what is going
on with school leadership, but less awareness about
activity at the classroom level.
MMP efforts are being felt beyond the learning team
and MTL by school staff
MMP activities are helping schools become more
focused on increasing student achievement in
mathematics
Schools that are more focused have increased the
proportion of students proficient in mathematics
Adoption of MMP-related principles is reported to be
related to supportive learning teams and to
supportive Math Teacher Leaders
November 6-7, 2006
22
Part II: Case Study Schools
Student
Achievement
Teacher Content
& Pedagogical
Knowledge
Classroom
Practice
Teacher
Involvement
Learning Team
Effort
November 6-7, 2006
Collaboration
School
Buy-in
23
Eleven Case Study Schools

Schools were diverse in terms of



November 6-7, 2006
Type
Geography
Student demographics
24
Case Study Data Collection




22 learning team observations—2 in each
school
44 classroom observations—4 in each
school; 2 teachers observed 2 times each
MKT Assessment for math teachers
SNA Survey for math
education ‘stakeholders’
November 6-7, 2006
25
Results of Learning Team Observations
Team Functioning
MMP Issues
Leadership
Participation
Organization & Structure
Results
Overall Functioning
Math Vision
Integration
Math Leadership
MMP Work
Overall MMP
Overall, strongest areas were participation &
mathematics leadership
Biggest areas for improvement were math
vision & results
November 6-7, 2006
26
Characteristics of Hi-Lo Scoring
Learning Teams—Team Functioning
Hi







Lo
Distributed leadership
Positional authority is less
important
Multiple views are
represented and heard
Multiple segments of the
school are represented
Written agenda, note taker,
facilitator
Explicit action items
Participants have hi
knowledge and skill levels
November 6-7, 2006





Principal does all the
talking
A few individuals dominate
the discussion
No agenda or team is easily
distracted from the agenda
Little follow-through on
assignments
No clear action items
27
Characteristics of Hi-Lo Scoring
Learning Teams—MMP Issues
Hi





Lo
Consistent curriculum
Math is addressed
alongside and in
combination with other
subjects
Coherent within grades and
(at times) across grades
MTL clearly in charge with
respect to math
Attention to CABS;
reference to MMP courses;
reviewing student work
November 6-7, 2006






Variation in curriculum
Math not addressed at the
meeting
No clear math leader—i.e.,
hard to tell who the MTL is
Confusion about the MMP
and CMF
Too much non-academic
housekeeping
School climate is
the priority
28
Results of Classroom Observations
General Practice
Articulating Math Task
Formative Assessment
Overall
Comprehensive
Math Framework
Understanding
Computing
Application
Reasoning
Engagement
Overall, strongest areas were articulating the math
task & understanding
Biggest areas for improvement were use
of formative assessment & engagement
November 6-7, 2006
29
Characteristics of Hi-Lo Scoring
Classroom Performance—General
Hi





Lo
Math task within the lesson
was easy to identify
Math task was discrete and
level-appropriate
Encouraging selfassessment and peerassessment
Establish criteria for
proficiency
Promoting problem solving
and independent thinking
November 6-7, 2006




Math task was to complex
or obscure
Only feedback provided was
if answer was correct
Little teacher involvement
in the lesson
Feedback focuses on
student behavior
30
Characteristics of Hi-Lo Scoring
Classroom Performance—CMF
Hi





Lo
Student explanations
sought
Computation is presented
as a means to an end
Problem solving was
emphasized
Students had to justify
solutions
Lessons are made relevant
by using everyday things
like money or time and
seeking examples from
students’ lives
November 6-7, 2006





Close ended questions are
emphasized
Only one way to solve
problems presented
Minimal time allowed to
share solutions
Students not accountable
for responding to questions
Problems not presented
in context
31
Results of MKT Assessment
43 item assessment addressed 3 content areas:
Number &
Operations
&
Algebra
&
Geometry
Overall
Score
November 6-7, 2006
32
Results of MKT Assessment
Average IRT Scores
Number &
n
Operations
12
-0.78
-0.60
-0.65
-0.59
32
0.12
0.31
0.20
0.22
Mean
-0.36
-0.13
-0.32
-0.31
Median
-0.38
-0.09
-0.39
-0.44
0.70
0.77
0.79
0.75
Lo
HI
SD
November 6-7, 2006
Algebra
Geometry
Total
33
Social Network Analysis


Math stakeholders in each school
were asked to name individuals
with whom the communicated
about mathematics
Statistical analysis focused on
1.
2.
November 6-7, 2006
Network and in-school density
Importance of MTL and MTS
34
Overall SNA Results
Metric
n
Total Named
Network
density
Density in
school
MTL
Role--In
Degree
MTS
Role--In
Degree
Lo
13
31
3.9%
7.6%
3.33
1.14
Hi
32
91
11.4%
31.1%
28.24
18.52
21.1
54.0
6.7%
17.6%
13.81
5.31
SD
6.8
17.6
2.6%
9.6%
7.2
4.9
Median
19
48
6.2%
15.4%
13.07
3.75
Mean
Density—a perfect score is 100% where everyone
names everyone else
In-Degree scores are relative measures
November 6-7, 2006
35
Example Network
November 6-7, 2006
36
Example Network
November 6-7, 2006
37
Report Card Indicators

19 indicators in 7 domains based on inschool data collection, online surveys,
and MPS data
1.
2.
3.
4.
5.
6.
7.
November 6-7, 2006
MTS Assessment
Collaboration
Learning Teams
Classroom Practice
Professional Development
Teacher MKT
Student Achievement
38
Report Card Results
WKCE
Mean % Proficient = 44%
Teacher Content
& Pedagogical
Knowledge
Overall IRT = -0.34
Algebra IRT = -0.18
Student
Achievement
November 6-7, 2006
PD Hrs. = 17.8
Facilitation Hrs. = 1.0
PD Quality = 3.1
Classroom
Practice
Teacher
Involvement
Learning Team
Effort
Team Functioning = 3.5
MMP Principles = 3.6
LT Quality = 3.1
Overall rating = 3.5
Gap MTL v. other teacher = .2
Teacher Engagement = 3.2
MTS Assessment =
38.3 of 55
School
Buy-in
Network density = 6.7% / School density = 17.6%
MTL Role = 13.8 / MTS Role = 5.3
SR MTL Engagement = 4.4 / MTS Quality = 3.0
39
Student Achievement &
In-School Network Density
November 6-7, 2006
40
Student Achievement &
Learning Team MMP
November 6-7, 2006
41
Student Achievement &
Professional Development
November 6-7, 2006
42
Conclusions



MMP is advancing concepts, ideas, &
principles that can help schools improve
student achievement results in math.
Schools that score well with regards to
MMP-related metrics have higher student
achievement.
Learning team adoption of MMP ideas and
dense in-school communication
networks are predictors of high
student achievement
November 6-7, 2006
43
Conclusions

At the same time…



Some MPS schools are lagging behind
in terms of adopting MMP ideas.
These schools perform do not score as
well on MMP metrics, which is
consistent with student achievement
results.
We know that other factors—prior
year student achievement and
SES—are stronger predictors
November 6-7, 2006
44
Evaluation Next Steps
District Wide Analysis


Continue online survey & data mining
Improve ability to link student and teacher data working
with MPS
Case Study Schools
1.
2.
3.
4.
5.
6.
Recruit case study schools
Plan observations
Observations Round 1
Observations Round 2
MKT Assessment
SNA Survey
November 6-7, 2006
Nov
Nov-Dec
Jan-Feb
March-April
May
May
45