Research Indicators for Sustaining and
Institutionalizing Change
CaMSP Network Meeting
April 4 & 5, 2011
Sacramento, CA
Mikala L. Rahn, PhD
Public Works, Inc.
Session Overview
• CaMSP Evaluation Overview--What we can
learn moving forward
• Emerging themes from MSP meeting,
morning presentation, and panels on
Institutionalization and Sustainability
• Time for discussion and next steps after the
Network meeting
Statewide Evaluation
Research Questions
1.
2.
3.
How have the Partnerships ensured that all
students have access to, are prepared for, and
are encouraged to participate and succeed in
challenging and advanced mathematics and
science courses?
How have the Partnerships enhanced the
quality of the mathematics and science teacher
workforce?
What evidence-based outcomes from the
Partnerships contribute to our understanding of
how students effectively learn mathematics and
science?
Key Features of CaMSP
Features of the programs reviewed based on
the legislation:
•
•
•
•
•
Partnership driven
Teacher quality
Challenging courses and curricula
Evidence-based design and outcomes
Institutional change and sustainability
L. Desimone
Conceptual Framework
Critical
Attributes of
Professional
Development:





Content focus
Active learning
Coherence
Duration
Collective
participation
Increased
teacher
knowledge
and skills;
change in
attitudes and
beliefs
Change in
Instruction
FIGURE 1: Desimone Conceptual Framework for studying the effects of
professional development on teachers and students
Improved
student
learning
What’s the Match?
• Content Focus
IHE’s bring content lens
• Active Learning
Strengthened partnershipsstronger models
• Coherence
Standards/textbooks/
assessments embedded
84 hours/3 years per
teacher
• Duration
• Collective
Participation
Collaboration embedded in
classroom follow-up
CaMSP in California
• Nine cohorts funded so far; science grades 3 through
8 or mathematics grades 3 through Algebra I.
• 59 partnerships included in the 07-08 outcome study;
88 partnerships in the 08-09 and 09-10 outcome
study
• PD models currently incorporate significant hours of
training for three funding cycles--less attrition as
implementation requirements tightened in California
• Same cohort of teachers--we have data to measure
based on consistent dosage and rules for
implementation.
CaMSP Outcome Study
• 07-08 Outcome Study completed (Report
available)
• 08-09 and 09-10 Outcome Study will be
combined and reported Spring 2011
• Large studies:
– 07-08: 284,538 treatment and comparison
students/1,581 treatment teachers
– 08-09 and 09-10: 165,209 treatment and
comparison students/1,594 treatment
teachers
CaMSP Evaluation and
Outcome Study Results
• A positive and statistically significant—though
small—effect on overall mathematics CST scores
in 2008 and 2009. Also showed a significant, and
more dramatic, effect on Algebra I test scores in
2008 and 2009. However, not in 2010.
• Science partnerships appeared to have a slight,
though positive, impact on science learning in
California in 2008, 2009, and 2010. This is mostly
attributable to 8th grade performance. Limited to
the 5th and 8th grades where the science CST is
administered.
CaMSP Evaluation and
Outcome Study Results (cont.)
• A more in-depth analysis in the 2008-09 and
2009-10 study of partnerships by
dosage/funding cycles completed reveal the
diversity of mathematics and science
achievement among partnerships. Some
partnerships showed marked improvement,
while others were little different from the
comparison groups.
What are we learning from CaMSP?
• There is a lot of variation in implementation
despite rules--ability to be innovative
continues to exist yet there are challenges to
maintain cohort
• LEA’s are getting used to their role in building
partnerships with individuals from IHE’s and
building capacity for managing high quality
PD, especially in medium size districts;
however, rural districts are particularly
challenged by participation rules
What are we learning from CaMSP?
• Individuals from IHE’s are tailoring their
training and challenging teachers with
content knowledge in ways that meet teacher
needs often building on experience in Subject
matter projects
• Subject matter projects, CPEC grants and
other IHE-led professional development exist
in parallel with CaMSP but not necessarily
integrated
What are we learning from CaMSP?
• Structure and personnel in the classroom
follow-up piece is essential to successful
implementation and has improved; adaptation
to teacher needs is important but fidelity to
follow-up model supports institutionalization
• Teacher leaders are emerging in many
partnerships and being incorporated in
training as facilitators and in district
curriculum decision-making committees
What are we learning from CaMSP?
• Lack of involvement of site administrators in
planning and PD continues as an area of
concern
• Visibility of and continued focus on local
evaluation is beginning to lead to higher
quality instrumentation and documentation
but there is little sharing of what we are
learning
Emerging Themes
• General panel on Institutionalization to address
changes in policy and practice among the
partners and what lessons the partnership has
taken from the MSP project and infused into the
district's culture.
• General panel on Sustainability to address
funding the effort once the grant sunsets.
• ESEA Reauthorization and preparing to respond
to the direction for MSP program
Discussion & Wrap-up
• What has been most successful in moving our
partnership forward?
– Institutionalization of the model
– Integration in district or IHE policy and practice
– What change can we document as a result of
this effort?
• How do these efforts contribute to a competitive
position for California in mathematics and science
education?