Student & Staff Feedback in
Educator Evaluation
The ESE Model Feedback Surveys
Claire Johnson Abbott, MA
Shelagh Peoples, PhD
ESE Spring Convening, 2015
1
Pop Quiz
2
Massachusetts Department of Elementary and Secondary Education
Feedback from students or staff
informs:
A. The Summative
Performance Rating
B. The Student Impact
Rating
C. Both
D. Neither
3
Massachusetts Department of Elementary and Secondary Education
Surveys are required.
A. True
B. False
C. Not sure
4
Massachusetts Department of Elementary and Secondary Education
ESE collects student and staff
feedback.
A. True
B. False
C. Not sure
5
Massachusetts Department of Elementary and Secondary Education
Agenda
The Why
ESE Model Feedback Surveys
o Development
o Implementation & Use (examples from 25 current
districts)
o Implementation Going Forward
Other Approaches to Collecting Feedback
6
Massachusetts Department of Elementary and Secondary Education
WHY is feedback from students
and staff so important?
 Informs professional growth and development
 Contributes to a comprehensive picture of an
educator’s practice
 Gives students and staff a voice in
their experiences
ESE Model Feedback Surveys
 Development
 Implementation & Use
8
Massachusetts Department of Elementary and Secondary Education
OBJECTIVE
Identify a feedback instrument aligned to the MA
Standards and Indicators that would:
Yield concrete, actionable information about educator
practice to support educator growth and development
Differentiate between levels of practice such that an
educator could identify where they excelled and
where/how they could strengthen their practice.
9
10
Massachusetts Department of Elementary and Secondary Education
Item Response Option
Experiment
Test Specifications
Standard I:
Curriculum, Planning, and Assessment
Online Survey of Educators
Student Focus Groups
Standard II:
Teaching All Students
A. Curriculum & Planning Indicator (67%)
A. Instruction Indicator (30%)
B. Assessment Indicator (13%)
B. Learning Environment Indicator (30%)
C. Analysis Indicator (19%)
C. Cultural Proficiency Indicator (16%)
D. Expectations Indicator (23%)
Total: 254 Items
11
Massachusetts Department of Elementary and Secondary Education
Concurrent Calibration of
Four Forms of Items
Item Selection and Form
Building for Pilot 1
Expert Review Sessions
Item Review Criteria
Retain, Revise, Reject
Representative
Accessible
Meaningful
12
Massachusetts Department of Elementary and Secondary Education
Concurrent Calibration of
Four Forms of Items
Item Selection and Form
Building for Pilot 1
Expert Review Sessions
Proficiency Descriptor: Designs units of instruction with measurable outcomes and challenging tasks
requiring higher-order thinking skills that enable students to learn the knowledge and skills defined in
state standards/local curricula.
During our lessons, I am asked to apply what I know to new types of problems or
challenging
Item Item
Prompt tasks.
IA3.17 In this class, students are asked to apply what they already know to new types of
problems or tasks they have not seen before.
IA3.23 This class makes me a better thinker.1
Expert
Rating
Rasch
Delta
3
.05
4
.37
IA3.1
My teacher uses multi-step problems or questions that require me to think deeply.
4
.34
IA3.2
I combine materials from several sources (e.g., books, Internet, newspapers) when I 3
work on papers or projects in this class.
My teacher asks me to compare and contrast my ideas with those of my partner or
3
group.
We spend enough time on topics so we can get an in-depth understanding of them. 3
.42
IA3.8
IA3.5
My teacher uses open-ended questions that enable me to think of multiple possible
answers.
IA3.22 My teacher asks me to explain my thinking until he or she is sure I understand.
.13
.24
2
.37
2
.37
2
.10
IA3.16 In class discussions, students are encouraged to present alternative explanations or
ideas.
IA3.12 In this class, students listen and respond to explanations from other students.
2
-.03
1
-.12
IA3.19 I have to explain my thinking when I write, present my work, and answer questions.
1
-.55
IA3.25 My teacher wants me to explain my answers—why I think what I think.1
1
-.58
IA3.6
IA3.7
I am challenged to support my answers or reasoning in this class.
Massachusetts Department of Elementary and Secondary Education
1Item
was taken from or adapted from Tripod survey operated by Tripod Education Partners and designed by Ferguson (2010).
13
Post-Pilot Administration #1
Item Variable Maps
New Item Development and
Form Building for Pilot 2
Expert Review Sessions
Pilot District Site Visits
“In this teacher’s class, students use rubrics to judge for
themselves what they have learned.” (G612_IC3.42)
was considered an informative and actionable item for Indicator
I.C (Analysis) but teachers indicated that the item,
“I grade other students’ work using a rubric provided by the
teacher.” (G612_IC3.43)
would be improved if it included information on peer editing.
Using educator suggestions, this item was adapted to,
“Using rubrics given to us by my teacher, I suggest ways for my
classmates to improve their assignments.” (G612_IC3.43)
Massachusetts Department of Elementary and Secondary Education
14
New items Anchored onto
Pilot 1’s Scale Metric
Pilot 2 Administration
March/April 2014
(88 Items)
Post-Pilot Administration #2
Educator/Student Online
Surveys (rating 100 remaining items)
Student Cognitive Interviews & Focus
Groups
 Online Educator/Student Survey (rated
100 items)
500 educators (how actionable was the practice?)
40 students (how important was it to their student learning?)
 Cognitive Interviews and Focus Groups
15
Massachusetts Department of Elementary and Secondary Education
Rasch Construct Validity
Framework.
Item Selection for Long
Forms of Model Instruments
June 2014
Final Item Review Committee
Final item selection committee
 11 Educators (teachers and administrators) plus
10 ESE staff (former teachers/administrators)
Item Selection Criteria
 Easy  Difficult to Enact (pre-sorted)
Meaningful, Actionable, Accessible
Narrowed final items down from 100 to 56
16
Massachusetts Department of Elementary and Secondary Education
Rasch Construct Validity
Framework.
MEASURE
6
5
4
3
2
1
0
-1
-2
-3
-4
-5
EACH
PERSON - MAP - ITEM
<more>|<rare>
.## +
|
|
|
|
. +
. |
. |
. |
. |
. +
.# |
. |
.# |
. T|
.# +
.# |
.## |
.### |
.### |
.#### S+
.#### |
.##### | IC3.43
IIC2.14
.######### | IB1.41
IIA2.14
.######## |T
.########### + IIA2.8
IIB2.10
.####### M| IB1.42
IIA1.22
.############ |S IA4.6
IIA2.9
.####### | IIA3.8
IIB1.31
.########## | IA1.40
IA4.12
.###### +M IA3.6
IIA2.15
.####### | IA1.45
IA1.8
.#### S| IA1.41
IA3.17
.## |S IA1.1
IA3.19
.### | IID3.6
.# + IB1.2
IIB3.10
.# |T
.# |
. T|
. |
. +
. |
. |
. |
. |
. +
. |
. |
|
. |
+
. |
|
. |
|
. +
<less>|<frequent>
"#" IS 48: EACH "." IS 1 TO 47
Item Selection for Long
Forms of Model Instruments
June 2014
Final Item Review Committee
Psychometric Data
 Mean: 0.85; Standard Deviation: 1.21
 Person Separation Reliability: 0.86 – 0.88
 Person Strata: 3.6 – 4.0
 Variance Explained: 45.1%
 DIF: Functions Well across Grade, Gender,
Low-Income, Race, Special Needs
 DIF: Problematic for LEP
IIA3.3
IIB1.7
IID2.13
IA4.18
IIB2.11
IA3.7
IB1.12
IC3.13
IC3.5
IIC1.2
IIA1.21
IC3.1
IIA1.41
IID2.6
IIC2.2
IIA3.1
IC3.6
IIB3.12
IID3.7
IIB3.13
IIA2.10
IIC1.25
IIC1.42
IIB1.41
IID2.11
IID2.31
IIB3.16
IID1.41
IID1.9
IID3.43
IIC2.3
Stakeholder Feedback
“ This is very helpful, I am so pleased at how
the teachers have really reflected upon the
student responses. I have even seen some
positive changes in practice since the results
came this weekend. I think this is a very good
sign for the potential impact of these surveys.”
(Principal, Elementary School)
Massachusetts Department of Elementary and Secondary Education
17
The benefits of this development process
Surveys provide valid, actionable information that
can inform practice by MA educators
 Student Survey Technical Report
Guidance encourages the diagnostic and formative
use of feedback (Self-assessment and goal-setting)
 Using Student & Staff Feedback in the Evaluation
Process
18
Massachusetts Department of Elementary and Secondary Education
ESE Model Feedback Instruments
Student Feedback Surveys
o Grades 3-5 (Standard and Short Forms)
o Grades 6-12 (Standard and Long Forms)
o K-2 Discussion Prompts
Staff Feedback Surveys
o Standard and Short Forms
Upcoming External Validity Study: Results available
in July 2015
Massachusetts Department of Elementary and Secondary Education
19
ESE Model Feedback Surveys
 Development
 Implementation & Use
20
Massachusetts Department of Elementary and Secondary Education
Implementing Student Surveys in Your District
21
Massachusetts Department of Elementary and Secondary Education
Implementing Feedback Surveys
Communicate
Communicate
Communicate
22
Massachusetts Department of Elementary and Secondary Education
Implementing Feedback Surveys
Communicate: How will educators use
feedback? (ex: goal setting)
23
Massachusetts Department of Elementary and Secondary Education
Example: Shaping goals & communicating with students
24
Massachusetts Department of Elementary and Secondary Education
Implementing Feedback Surveys
Communicate: How will evaluators use
feedback?
25
Massachusetts Department of Elementary and Secondary Education



Your principal has access to your survey results.
He/she will not seek you out to discuss your results.
You should feel free to talk with your principal about
your results, if you want.
During the evaluation process when you and your
principal are discussing your self-assessment and
goal setting, your principal may ask you if you have
reviewed your survey results and if there was any
information from the survey that guided your
thinking about goals.
Principals may look at patterns and trends over time
on a school and individual basis to identify strengths
and areas of focus.
Lincoln Public Schools, 2015
Implementing Feedback Surveys
Communicate: How will students
and parents be informed of this
process?
27
Massachusetts Department of Elementary and Secondary Education
28
Massachusetts Department of Elementary and Secondary Education
Bridgewater-Raynham RSD
Communicate the Why
o Joint Labor-Management Committee
o School Committee
Communicate the How
o An Efficient Data Collection System (Google Forms)
Communicate the What
o Share w/evaluators, use for self-assessment & goalsetting
Special Thanks To:
• Bridgewater-Raynham’s Joint Labor-Management Committee
• Dr. Jacqueline B. Forbes, Superintendent of Schools
• Mr. Derek Swenson, Assistant Superintendent of Schools
• Paula Bartolo, Technology Curriculum Integration Specialist
29
Solicit Feedback from Everyone
Preview: MA Hospital School
o Adapted items
o Online implementation tool
Preview: Feedback Tools related to Inclusive
Classroom Environments
o Model Surveys aligned to Power Elements of Inclusive
Instruction
Resources coming in Summer 2015
Massachusetts Department of Elementary and Secondary Education
30
Alternative Feedback Mechanisms
Focus Groups
Meaningful
• Aligned to one or more
performance Standards
• Aligned to content
Actionable
• Information is useful
• Information is timely
Accessible
• All students can give feedback
• Students understand what is
being asked
Discussion Prompts
Exit Slips
Existing Surveys
31
Massachusetts Department of Elementary and Secondary Education
ESE Resources
Quick Reference Guide: Student & Staff Feedback
Guidance on Student and Staff Feedback
ESE Model Feedback Surveys & Protocols
Student Survey Technical Report
ESE/MASS Video Series and Training PPT
SSF Webpage: http://www.doe.mass.edu/edeval/feedback/
32
Massachusetts Department of Elementary and Secondary Education
Thank You
Claire Johnson Abbott (cabbott@doe.mass.edu)
Shelagh Peoples (speoples@doe.mass.edu)
http://www.doe.mass.edu/edeval/feedback/
33
Massachusetts Department of Elementary and Secondary Education