Educator Evaluation:
Challenges and Opportunities
Massachusetts Association of School Superintendents
May 19, 2011
updated
1
Context
The three-year Superintendent Induction
Program
An ESE – M.A.S.S. Partnership
Four Focus Areas
1. Focused Instructional Leadership
2. Collaborative Relationships and Effective
Leadership Teams
3. Strategic Management of Human and other
Resources
4. Robust System of Supervision and Evaluation
2
Agenda
 Goals of the Proposed Regulations
 Key Features of the Proposed Regulations
 Components of the Model System
 Challenge: Assessing Educator Impact on Student
Growth using Multiple Measures
 Challenge: Self-Assessment and Goal Setting
 Challenge: Timeline for Implementation
3
Goals of Improved Educator Evaluation
• Promote leaders’ and teachers’ growth and
development
• Place student learning at the center using
multiple measures of student learning, growth
and achievement
• Recognize excellence in teaching and leading
• Set a high bar for professional status (tenure)
• Shorten timelines for improvement
4
Key Features
• 5-step Evaluation Cycle, starting with educator
self-reflection and goal setting
• 3 Categories of Evidence:
– Multiple measures of student learning, growth and
achievement
– Products of practice
– Educator’s collection of other evidence
5
Features, cont’d
• 4 Standards with “core” indicators for administrators
and teachers
• 4 Ratings on performance: exemplary, proficient,
needs improvement, unsatisfactory
• 3 Ratings of impact on student learning, with
focus on learning gains: high, moderate, low
• Different Paths & Plans depending on career stage
and performance
6
A 5-Step Evaluation Cycle
7
Rubrics for
4 Statewide Standards and Indicators
Administrators
Teachers
Curriculum, Instruction &
Assessment
Curriculum, Planning & Assessment
Management & Operations
Teaching All Students
Family & Community Partnerships
Family & Community Engagement
Professional Culture
Professional Culture
8
3 Categories of Evidence
• Multiple measures of student learning, growth and
achievement
• Products of practice, including observation of practice
(announced and unannounced)
• Educator’s collection of other evidence, including
analysis of feedback from:
– Students
– Parents
– Staff (for administrators)
9
The summative performance rating
(exemplary, proficient, needs improvement, unsatisfactory)
Based on:
• Performance against each of four standards
and
• Progress toward meeting student learning and
professional practice goals
 Educator’s impact on student learning, growth and
achievement “counts” in standards on curriculum,
instruction and assessment
10
Paths & Plans:
Differentiated by Career Stage and Performance
• Educators in their first three years:
Development Plan (one year)
• Performance rated as proficient or exemplary:
Self-directed Growth Plan (one or two years)
• Performance rated as in need of improvement:
Directed Growth Plan (one year)
• Performance rated as unsatisfactory:
Improvement Plan (up to one year)
11
Decision Flow for Experienced Educators
12
Rating of Educator Practice
Linking Student Learning and Educator Practice
Exemplary
Proficient
Needs
Improvement
Unsatisfactory
Low
Moderate
High
Impact on Student Learning
(multiple measures of student learning, including MCAS student growth
percentiles where available, with a focus on learning gains)
13
What Happens When There’s a
Discrepancy?
Low Rating of Educator Practice
BUT
Moderate or High
Impact on Student Learning
•Evaluator reviews
discrepancy with
educator.
•Evaluator’s
supervisor considers
discrepancy trends in
evaluating evaluator.
14
What Happens When There’s a
Discrepancy?
High Rating of Educator Practice
BUT
Low Impact on Student Learning
•Educator has 1-year
growth plan focused on
discrepancy.
•Evaluator’s supervisor
MUST review rating.
•Superintendent has final
authority to determine
summative rating.
15
Agenda
 Goals of the Proposed Regulations
 Key Features of the Proposed Regulations
 Components of the Model System
 Challenge: Assessing Educator Impact on Student
Growth using Multiple Measures
 Challenge: Self-Assessment and Goal Setting
 Challenge: Timeline for Implementation
16
Key components of the Model planned
• Contract language describing:
– Process
– Timelines
– Collection of evidence
• A rubric for each standard and indicator that:
– describes professional practice vividly and clearly at
four levels of performance
– Is differentiated for different roles, e.g., classroom
teacher, caseload teacher, counselor, principal
– Includes core and supplementary indicators
17
18
Other components of the Model
• Templates for:
– Self-assessments
– Goals
– Plans
•
•
•
•
Developing educator plan
Self-directed growth plan
Directed growth plan
Improvement plan
19
Other components of the Model,
con’t
• Guidelines for developing and using multiple measures
of student learning, growth and achievement
• Guidelines for determining low, moderate and high
impact on student learning
• Examples and Resources on:
– Multiple measures of student learning
– Determining educator impact
• Examples of ways to collect and use feedback from:
– Students
– Staff (for administrators)
– Parent feedback (initially for administrators)
20
Stakeholder Feedback
from Students, Staff and Parents
• Focus on school-wide feedback (initially)
• Students, starting in grade 6 (?)
• A possibility: ESE-supported on-line data
collection
21
Stakeholder Feedback Examples
New York City Learning Environment Survey
• Garners annual feedback from parents, students and teachers.
• Results factor into school progress report rating and help schools better
understand their own strengths and target areas for improvement.
• http://schools.nyc.gov/Accountability/tools/survey/default.htm
Massachusetts Teaching, Learning and Leading Survey (Mass TeLLS)
• Taken by 40,000 teachers and administrators in 2008.
• Educators provided views about teaching and learning conditions,
including leadership, empowerment, facilities and resources, PD, and time,
in their schools.
• http://www.masstells.org/index
Boston Student Advisory Council (BSAC) Student to Teacher Constructive
Feedback
• Students provide annual, anonymous feedback about individual teachers
• http://www.youtube.com/user/BSACbuzz#p/a/f/0/CMB_8DjAAgM
22
Supports planned for the Model
 Annual updates
 Orientation tools and resources for a variety of
audiences
 On-line and hybrid professional development on
observation, goal setting, etc.
 Networks of Practice
------------------ Web-based rubric “library”
23
Supports for the Model System
 Outreach to state associations, e.g.,
Principals (MESPA and MSSAA)
Department Heads and Supervisors (MASCD)
Counselors (MASCA)
ESL (MATSOL)
Art (MAEA)
 Training and support for regional collaboratives
to develop and share expertise and resources
among member districts
24
Agenda
 Goals of the Proposed Regulations
 Key Features of the Proposed Regulations
 Components of the Model System
 Challenge: Assessing Educator Impact on Student
Growth using Multiple Measures
 Challenge: Self-Assessment and Goal Setting
 Challenge: Timeline for Implementation
25
What are “multiple measures”?
 MCAS growth percentile data, when applicable
 MEPA growth scores, when applicable
 Other assessments comparable district-wide
across grade or subject, including approved
commercial assessments and district-developed
pre/post unit and course assessments
----------------------------------
 Teacher-developed assessments (individual and/or
team, school)
26
MCAS Growth to grade 7: Three students
Gina
MCAS ELA scaled score
280
260
Advanced
SGP
80 to 99
240
65%
Proficient
230
220
Needs Improvement
230
 230
35%
60 to 79
40 to 59
20 to 39
1 to 19
SGPs between 40 to 59 are typical
200
Warning/Failing
Grade 5
Grade 6
Grade 7
2006
2007
2008
source: www.doe.mass.edu/mcas/growth/
27
Growth to grade 7: Three students
Harry
MCAS ELA scaled score
280
260
240
Advanced
75%
Proficient
248
248
 244
25%
220
200
Needs Improvement
Warning/Failing
Grade 5
2006
Grade 6
2007
Grade 7
2008
source: www.doe.mass.edu/mcas/growth/
28
Growth to grade 7: Three students
Ivy
MCAS ELA scaled score
280
260
240
Advanced
Proficient
8%
 226
220
Needs Improvement
214
200
214
92%
Warning/Failing
Grade 5
2006
Grade 6
2007
Grade 7
2008
29
Growth to grade 7: Three students
Gina, Harry, and Ivy
MCAS ELA scaled score
280
260
Advanced
Harry
240
Proficient
248
248
Gina
230
220
Needs Improvement
214
200
230
Ivy
214
Warning/Failing
Grade 5
Grade 6
Grade 7
2006
2007
2008
source: www.doe.mass.edu/mcas/growth/
30
Growth to grade 7: Three students
English language arts
Grade 5
2006
Grade 6
2007
Grade 7
2008
SGP
2008
Gina
230
230
230
35
Harry
248
248
244
25
Ivy
214
214
226
92
source: www.doe.mass.edu/mcas/growth/
31
Median student growth percentile
Last name
SGP
Lennon
6
McCartney
12
Starr
21
Harrison
32
Jagger
34
Richards
47
Crosby
55
Stills
61
Nash
63
Young
74
Joplin
81
Hendrix
88
Jones
95
Imagine that the list of students to
the left are all the students in your
6th grade class. Note that they are
sorted from lowest to highest SGP.
The point where 50% of students
have a higher SGP and 50% have a
lower SGP is the median.
Median SGP for the 6th grade class
source: www.doe.mass.edu/mcas/growth/
32
Challenging a Level 4 School
Median Student Growth Percentile
English Language Arts
Murkland
Lincoln
Sokolovsky
2008
26.5
55.0
67.0
2009
22.5
66.5
71.0
2010
22.0
68.0
74.0
source: www.doe.mass.edu/sda/dart
33
A 5-Step Evaluation Cycle
34
Lowell Public Schools
District Mathematics • 2009 Growth
SOURCE: Massachusetts Department of Elementary and Secondary Education
STATE
20%
DISTRICT
20%
18%
Grade 6
13%
18%
18%
Grade 8
17%
20%
20%
14%
Grade 10
20%
20%
18%
24%
21%
20%
34%
18%
19%
20%
19%
20%
24%
20%
23%
22%
22%
A Case Study
Grade 7
15%
18%
22%
Grade 4
22%
19%
20%
18%
21%
Grade 5
23%
18%
20%
18%
21%
0%
20%
very low
40%
low
60%
moderate
80%
high
100%
very high
Research Testing and Assessment - Lowell Public Schools
Lowell Public Schools
35 35
Lowell Public Schools
Grade 6 Mathematics • 2009 Growth
SOURCE: Massachusetts Department of Elementary and Secondary Education
STATE
20%
DISTRICT
Pyne
Wang
20%
12%
2% 4%
3%
8%
Stoklosa
8%
Sullivan
18%
13%
53%
19%
11%
12%
47%
27%
44%
21%
22%
24%
26%
21%
26%
32%
20%
very low
40%
low
17%
15%
15%
14%
60%
moderate
18%
20%
21%
30%
0%
39%
16%
18%
22%
Bartlett
34%
21%
24%
Robinson
21%
20%
9%
20%
60%
16%
16%
Rogers
20%
21%
7%
7%
Butler
14%
7%
Daley
20%
16%
80%
high
8%
100%
very high
Research Testing and Assessment - Lowell Public Schools
Lowell Public Schools
36 36
Lowell Public Schools
Grade 6 Mathematics • 2008 Growth
SOURCE: Massachusetts Department of Elementary and Secondary Education
STATE
DISTRICT
Pyne
Daley
Stoklosa
Sullivan
Wang
Bartlett
Butler
Robinson
Rogers
20%
20%
14%
2%
18%
11%
17%
5%
14%
21%
41%
30%
32%
25%
21%
13%
26%
21%
22%
15%
33%
27%
28%
28%
22%
26%
33%
21%
19%
29%
27%
22%
30%
26%
22%
20%
very low
20%
74%
19%
24%
0%
21%
17%
13%
9%
4%
20%
13%
4%
7%
20%
40%
low
60%
moderate
10%
16%
13%
80%
high
6%
9%
100%
very high
Research Testing and Assessment - Lowell Public Schools
37
Lowell Public Schools
Grade 6 Mathematics • 2009 Growth
SOURCE: Massachusetts Department of Elementary and Secondary Education
Other LPS - ALL
12%
Stoklosa - ALL
8%
Other LPS - Reg. Ed
9%
Stoklosa - Reg. Ed.
5%
Other LPS - ELL
15%
9%
11%
14%
13%
9%
Stoklosa - ELL
19%
12%
0%
very low
21%
15%
20%
low
46%
21%
35%
26%
18%
12%
34%
25%
8%
6%
20%
47%
21%
38%
20%
40%
moderate
51%
60%
high
80%
100%
very high
Research Testing and Assessment - Lowell Public Schools
Lowell Public Schools
38 38
Lowell Public Schools
Stoklosa Middle School • 2009 Mathematics Growth
SOURCE: Massachusetts Department of Elementary and Secondary Education
Grade 5 - Regular
3%
Grade 6 - Regular
19%
5%
32%
13%
Grade 7 - Regular
8%
26%
16%
Grade 8 - Regular
6% 3%
Grade 6 - ELL
12%
Grade 7 - ELL
6%
30%
8%
22%
12%
6%
18%
12%
18%
very low
37%
20%
51%
24%
16%
20%
21%
16%
27%
40%
low
10%
59%
26%
0%
19%
21%
13%
Grade 8 - ELL
47%
24%
15%
Grade 5 - ELL
46%
moderate
33%
60%
80%
high
100%
very high
Research Testing and Assessment - Lowell Public Schools
Lowell Public Schools
39 39
Other measures that can help Stoklosa
staff assess impact on student learning
and growth
 Massachusetts English Proficiency Assessment
(MEPA)
 District math benchmark assessment
 District-approved commercial assessment tied
to district curriculum
 District-adopted curriculum-embedded
performance assessment
40
Agenda
 Goals of the Proposed Regulations
 Key Features of the Proposed Regulations
 Components of the Model System
 Challenge: Assessing Educator Impact on Student
Growth using Multiple Measures
 Challenge: Self-Assessment and Goal Setting
 Challenge: Timeline for Implementation
41
A 5-Step Evaluation Cycle
42
Self-assessment and Goal Setting
Based on:
 Standards & indicators (rubric) + district & school priorities
 An analysis of multiple measures of learning and growth of our
students in the past
 An analysis of the students we have now
At least:
 One goal for professional practice
 One goal for student learning, growth and achievement
Attributes of a Useful Goal
Specific
Relevant
Measurable
Attainable
Time-bound
43
SMART goals of Stoklosa’s 7th grade team
• Professional Practice goal: I/We will…
• Student Learning goal: My/Our students
will…
44
SMART goals of a middle school music
teacher
• Professional Practice goal: I will collaborate with
my colleagues in the music department to develop,
pilot, analyze, revise and share 2 performance-based
assessments
• Student Learning goal: My students will be able to
identify and apply music terms, symbols and definitions
in the curriculum guide for 6th, 7th and 8th grade. Using
a department-developed assessment, 75% of my
students will score 85% or above on the third quarter
assessment.
45
SMART Goals of an 8th grade social studies
teacher
• Professional practice goal: To strengthen
expository writing, I will study the “workshop process”
for writing, observe it in practice, and introduce it in at
least two of my classes by the start of second term.
• Student learning goal: At the end of the third
quarter unit on the constitution, students will
demonstrate proficiency by writing a pamphlet for new
citizens about their constitutional rights. Using a
department-developed rubric, a majority of my
students will have moved one level on the writing
component of the rubric since the start of the second
term
46
SMART Goals of a Middle School Principal
• Professional practice goal: I will complete 100% of
goal setting conferences with my fifth and sixth grade
teams by October 15th, seek anonymous feedback about
staff perceptions of their usefulness in improving their
practice, research effective goal setting with my
colleagues, and identify steps I will take in mid-year
formative assessment conferences to improve the
likelihood that their practice and student growth goals
will be achieved.
• Student learning goal: The proportion of fifth and
sixth grade students with high or very high SGP growth
will increase by 5% points in both ELA and Math.
47
SMART Goals of a 10th grade geometry
teacher
• Professional practice goal: To engage students
more, starting second term, we will incorporate at
least one real-world application of geometry into 2 of
every 5 homework assignments.
• Student learning goal: 85% of our students will score
80% or above on the district-developed third quarter
exam.
48
Agenda
 Goals of the Proposed Regulations
 Key Features of the Proposed Regulations
 Components of the Model System
 Challenge: Assessing Educator Impact on Student
Growth using Multiple Measures
 Challenge: Self-Assessment and Goal Setting
 Challenge: Timeline for Implementation
49
Anticipated timeline for Implementation
For 2011-2012
• Level 4 schools + volunteer “early adopters”
For 2012-13
• All Race to the Top Districts
For 2013-2014
• All other Districts
50
Educator Impact on Student Learning
603 CMR 35.10 (4)
By September 2013, each district shall adopt a districtwide set of student performance measures for each
grade and subject that permits a comparison of student
learning gains.
– MCAS Student Growth Percentile shall be employed
where it is available.
– At least two measures of student learning gains shall
be employed at each grade and subject in
determining impact on student learning.
51
Priorities for ESE action and support?
1.
2.
3.
4.
Model contract language and rubrics for teacher and principal
Strategies for “making time” to do evaluation well
Orientation materials for many audiences
Strategies for using the rubric to develop a shared, specific
picture of practice at four levels of proficiency
5. Access to low-cost PD for evaluators to use the rubric effectively
6. Guidelines and examples: Self-Assessment and Goals
7. Guidelines and examples: Educator Plans
8. Developing District-wide Measures of Student Learning*
9. Determining Educator Impact on Student Learning*
10. Student, Staff and Parent Feedback*
* Will take more than one year to develop, pilot and validate
52
If I were in your shoes….
This summer, I’d:
 Begin (or deepen) work on building educator’s capacity to analyze
data about student learning and set SMART goals.
 Work with my principals on their SMART goals and establish clear
expectations for what I want to be seeing when I do my first visits
to their schools this fall and observe practice with them.
 Begin to line up potential partners and supports; for example, I’d
ask my collaborative if it will work with my district and other
member districts on implementation.
 Introduce the first draft of ESE’s model rubric (available mid-July,
hopefully) to see how well it might match my district’s needs.
 Attend the M.A.S.S. Summer Institute to get the latest information
from ESE on the status of the regulations and the model; and to
confer with colleagues
53
If I were in your shoes….(con’t)
• I would not start bargaining now. There isn’t
enough information to go on yet.
• I would, however, let my committee and union
know that we will have to open the contract to
bargain this.
54
Questions? Suggestions?
Priorities?
Please complete the feedback sheet
Karla Brooks Baehr
Deputy Commissioner
kbaehr@doe.mass.edu
781-338-3101
55