Washington State
Teacher and Principal
Evaluation Project
Preparing Educators for Rater Agreement and
Feedback: Planning for Professional Learning
TPEP Sessions for 2014-15

Face-to-Face Sessions




A Virtual Presentation

2
Student Growth 2.0
Rater Agreement Practices
TPEP/ Washington State Learning Standards
Connections
Sharing Electronic Resources
Group Norms
Pausing
Paraphrasing
Posing Questions
Putting Ideas on the Table
Providing Data
Paying Attention to Self and Others
Presuming Positive Intentions







3
TPEP Core Principles
Professional learning is the underpinning of the
evaluation system.
4
Learning Targets:
I can describe the OSPI working definition of rater
agreement and the stages for development.
I can identify best-practices and strategies to maximize
rater agreement.
I can explain what makes a high quality professional
learning process for evaluators and helps administrators
develop strong skills in providing feedback.
I can describe our district’s plan to maximize rater
agreement.




5
Activity 1: District Self-Assessment
Discuss: Using the survey tool, briefly discuss with your
district team to determine the current level of practice in
maximizing rater agreement.
Share: What is your districts identified area of focus?


6
Learning
I can describe the OSPI working definition of rater agreement
and the stages for development.
7
Rater Agreement
Turn and talk to a partner about the following topics:



8
I think rater agreement is important because…
My role in the rater agreement process is…
OSPI Definition of Rater Agreement
 The
extent to which the scores between
the raters have consistency and
accuracy against predetermined
standards. The predetermined standards
are the instructional and leadership
frameworks and rubrics that define the
basis for summative criterion-level scores.
9
Unwrapping the Definition…

Consistency: A measure
of observer data quality
indicating the extent to
which an observer is
assigning scores that agree
with scores assigned to
the same observation of
practice by another typical
observer.
10

Accuracy: A measure of
observer data quality
indicating the extent to
which an observer is
assigning scores that agree
with scores assigned to
the same observation by
an expert rater; the extent
to which rater’s scores
agree with the true or
“correct” score for the
performance.
Take “Five”

Discuss with your team:



What systems are in place for your district to focus on
accuracy?
Are these systems dependent on each other or could a district
start with agreement and then work towards accuracy?
Share: What systems have you identified?
11
Learning Content 1: OSPI Stages of
Rater Agreement
12
OSPI Stages of Rater Agreement


Read “Rater Agreement in Washington State’s Evaluation
System”
Discuss with your team:




How has your district communicated the vision of Rater
Agreement?
What structures are in place for staff to learn about the
Instructional Framework?
How do teachers and building principals use the language of
the frameworks to improve practice?
Share: One strategy that you have identified.
13
Continuous Improvement Process:




Once we reach Stage 3, we do not “graduate.”
Rater agreement is NOT ensured by a single training or
certification test.
We must revisit and review key learnings from Stage 1
and Stage 2 to avoid drift.
Provide ongoing professional development including
ongoing calibration conversations involving real-life or
video-based observation.
14
Rater Drift will naturally occur unless
evaluators have:



Ongoing opportunities to demonstrate agreement
Access to practice videos for difficult-to-score
domains/components
Expectations that their ratings will be monitored
15
How is Rater Agreement Assessed?

Rating through observations requires evaluators to do a lot of
things well. They must:





16
Be objective: Record evidence that is free of “bias, opinion, and
subjectivity” (Bell et al., 2013)
Understand the rubric: Thoroughly and deeply understand each
component and indicator on the district rubric instrument (Bell et
al., 2013)
Interpret evidence: Correctly align evidence to framework criteria
that reflect the context of the evidence (Bell et al., 2013)
Document evidence: Gather, sort, and record a preponderance of
evidence as it happens in the classroom or school (McClellan,
Atkinson, & Danielson, 2012).
Be reliable: Reliability requires that they be able to do all of these
things well over-time, in different classrooms and schools, and for
different educators.
Professional Learning Takeaways
Ensure that observers have opportunities to learn the
following:

Use the rubric language to explain their decisions.

Consistently take notes that gather useful evidence.

Avoid making scoring decisions during note-taking.

Resort back to the scoring criteria when uncertain.

Practice using videos and live classroom observations.
(Bell et al., 2013)
17
THINK – PAIR - SHARE


What did you learn that confirmed your understanding of
Rater Agreement in Washington State’s Evaluation
System?
What did you learn that challenged your understanding of
Rater Agreement in Washington State’s Evaluation
System?
18
Learning
Understand best-practices and strategies to maximize rater
agreement.
19
Why Rater Agreement?

An educator’s observation
scores should be the same
regardless of the observer.



20
Educators can trust the new
evaluation system.
The new system is legally
defensible for personnel
decisions.
Educators receive relevant,
useful feedback for
professional growth.
Two Research-based Rater Agreement
Activities
1.
2.
21
Address issues of rater bias,
interpretation, and objectivity (Jilliam,
Kosa,Tierney, & Tocci, 2014; MET, 2012).
Rater agreement activities should be
provided at least three times a year (Jilliam,
Kosa, Tierney, & Tocci, 2014), but monthly is
ideal (Jerald, 2011).
What Affects Agreement in Observation?
Observer Bias: What are
some of the various “lenses”
that might bias
 A teacher evaluator?
 A principal evaluator?
2222
Turnkey Activity: Common
Sources of Bias
Handout 3: Common Sources of Bias Match-Up
 Step 1. At your table, work as a group to match each
common rater error with a possible strategy evaluators can
use to avoid the error.
 Step 2. After each match, discuss other possible strategies
you have seen used or that you think might be effective.
2323
Debrief
Answer Key
1=D
2=G
3 =A
4=F
5=B
6=E
7=H
8=C
2424
Two Research-based Rater Agreement
Activities
1.
2.
25
Address issues of rater bias, interpretation,
and objectivity (Jilliam, Kosa, Tierney, & Tocci,
2014; MET, 2012).
Rater agreement activities should be
provided at least three times a year
(Jilliam, Kosa,Tierney, & Tocci, 2014),
but monthly is ideal (Jerald, 2011).
Rater Agreement Activities





Rater agreement activities can be as long as several days
of training and as short as 60-minute sessions. It’s
important to note that frequency is the key to conducting
the activities. Short and frequent could be just as
effective as lengthier activities done only a few times per
year. 60-minute rater agreement activities can be
embedded in activities that already exist:
Monthly PLCs
Walk-throughs
Instructional rounds
Professional development
26
Selecting Observation Focus for
Data Collection

Scaffolding Observations:




27
Group1: Management
Group 2: Environment
Group 3: Engagement
Count off table’ by “3’s” – be ready to gather evidence
on the video based on your number
Video 1: Collect Data



Ms. Warburton’s 8th grade math lesson: Sorting and
classifying equations
CCSS Math 8.EE.C.7a
Collect evidence statements
https://www.teachingchannel.org/videos/sortingclassifying-equations-overview
28
Video 1:Share Data
29
Data: Align with Framework

Let’s all look at evidence that aligns with:
Group
Danielson
Marzano CEL
1
2
3

At tables: Based on the evidence we have compiled, talk
about the level of performance that should be assigned.
30
Rater Agreement Activities

All opportunities should be grounded in common
protocols or structures that assist in developing and
extending the common language of the observation tools
and other measures of educator practice.
(Jilliam, Kosa, Tierney, & Tocci, 2014
31
Research-based Rater Agreement
Activities

Discuss with your team:



How has your district included ongoing discussions to address
bias in the evaluation?
What job-embedded strategies are in place for evaluators to
practice specific skills?
Share: One strategy that you have identified.
32
Learning
High quality professional learning process for evaluators in
providing feedback
33
“The post-conference cannot be treated
as a bureaucratic formality; it is one of
the most critical features of an effective
teacher evaluation system if the goal is
not just to measure the quality of
teaching, but also to improve it.”
~Jerald and Van Hook,
2011, p. 23
34
Avoid Dominating
the Conversation


•
A study of evaluation implementation in Chicago found that principals
generally dominated the conversation by speaking 75 percent of the
time in postobservation conferences (Sartain, Stoelinga, & Brown, 2011).
Encourage a balanced conversation (50/50) by asking reflective and follow-up
questions.
Ensure that teachers are prepared to participate and establish this as an
expectation through educator orientation for the new evaluation system.
35
Focus on Evidence
Reduces three big dangers in postobservation
conferences:
 Loose interpretation. Evidence-based feedback separates
observations and interpretations.
 Subjectivity. Drawing upon evidence during feedback
conversations can decrease subjectivity (Sartain et al., 2011).
 Emotion. Evidence-based feedback can also “remove some of
the emotion from the evaluation process” (Sartain et al., 2011,
p. 23).
36
Use Rubric Language and Descriptors
Incorporating rubric language when discussing evidence
helps in the following:
 To build and reinforce a shared understanding of
good instruction
 To ensure the rubric remains the objective point of
reference in the conversation
37
High-Level Questioning
In Chicago, researchers found that only 10 percent of questions asked by
evaluators during postobservation conferences were high level and
promoted discussions about instruction.
(Sartain et al., 2011)
38
Rubric
Example
Low
The evaluator’s question
• Requires limited teacher response—often a single word—rather than
discussion
• Is generally focused on simple affirmation of principal perception
“I think this was basic
because of the evidence I
collected. Do you agree?”
Medium
The evaluator’s question
• Requires a short teacher response
• Is generally focused on completion of tasks and requirements
“Which goals did you not
meet?”
High
High-Level Questioning
Rubric
The evaluator’s question
• Requires extensive teacher response
• Reflects high expectations and requires deep reflection about
instructional practice
• Often prompts the teacher and evaluator to push each other’s
interpretations
“How did student
engagement change in
your class after you that
strategy? Why do you
think that happened?”
(Modified from Sartain et al., 2011, p. 24)
39
Ends With Actions and Supports
Professional Growth Planning
 Connect feedback to professional
growth plan.
 Identify goals, timelines, and
benchmarks for areas for growth.
Action Strategies, Practice, and
Modeling
 Ensure the conversation culminates in
small, specific changes a teacher can
implement in the classroom immediately.
 Have the teacher practice or model the
practice.
 Suggest observing a colleague who strong
in the area
 Direct the teachers to additional
resources (online, print, or other
colleagues).
(Hill & Grossman, 2013)
40
Activity: A Good Conference or Not?
http://vimeo.com/89454466
41
Helpful Resources for
Coaching and Feedback




Learning-focused Supervision: Developing
Professional Expertise in Standards-Driven Systems
(Lipton & Wellman, 2013)
Principal Evaluator’s Toolkit for the Instructional
Feedback Observation (American Institutes for
Research, 2012)
Leveraging Leadership: A Practical Guide to Building
Exceptional Schools (Bambrick-Santoyo, 2012)
The Art of Coaching: Effective Strategies for School
Coaching (Aguilar, 2013)
42
Research-based Coaching & Feedback
Activities

Discuss with your team:



What strategies have you found successful in building a culture
that embraces feedback and reflective conversations about
teaching and learning?
What job-embedded strategies are in place for evaluators to
practice specific skills?
Share: One strategy that you have identified.
43
Resources
.
44
Framework Specific Tools
45
“Video” Protocol Example

North Mason School District



Viewed Video as group and collect Data/Evidence
Sorted and Placed Data/Evidence on framework
Determined Level of Performance



First Individually
With a Partner
Whole Group of Administrators (5-6)
North Mason utilized this process for two years and then in year
three they did classroom observations. They utilize “TeachScape”
Videos.
46
“Instructional Reviews”
Protocol Example

Central Kitsap School District






47
Observational tool created to focus the observation on the
framework: Purpose, Student Engagement and Academic Rigor
Teams entered one school and divided themselves
Observed in 15 minute time blocks
Met and talked about their ratings
Combined ratings into one common rating sheet
Checked for range and variations
eVal

A collection of 93 videos* for:



48
Exploring your instructional framework in the context of
classroom instruction…
Discussing the characteristics of quality instruction as
described by your framework…
Engaging groups in interpretation, analysis and discussion of
classroom practice with the use of tools and rubrics.
Swivel Video Camera

Tool to create internal video for use:
49
Additional Resources:

The URL for eVal Training Video: Debbie Tschirgi ESD112
http://vimeo.com/101444062

The URL for the statewide Ed Tech web site where you
can get the logins and URL for the eVAL Sandbox:
http://www.edtech.wednet.edu/eVALTraining/
50
Implementing
Develop long-range professional development based on the
district’s current level of practice.
51
Plan for Rater Agreement




How will new evaluators receive training in the
framework prior to evaluating staff?
How will new evaluators learn to apply the framework in
a formative process during their first year?
How can existing meetings or professional learning
communities be used to provide opportunities to
maximize rater agreement?
What opportunities will be provided to trained evaluators
to continuously hone their skills?
52
Rater Agreement: District Planning Tool



Initiating
Refining
Sustaining
53
District Planning Tool
Each stage of Rater Agreement Includes:




Assessment
Planning
Implementing
Reflection on Measuring
54
Learning Activity: Monitoring and
Maintaining Rater Agreement
 Read “High Fidelity:
Investing in Evaluation
Training” (15 minutes)


55
How are states
providing training for
new teacher and
principal evaluation
systems?
How is your district
providing support to
monitor and maintain
rater agreement?

Choose one strategy to
add to your District
Planning Tool.
Reflecting
56
Revisiting the Learning Targets

With a Partner:




57
Describe the OSPI working definition of rater agreement and
the stages
Identify a best practice or strategy that could be used to
maximize rater agreement.
Explain a strategy that would help administrators develop skills
in providing feedback.
Discuss how your district plan includes elements listed above.
What’s Next

Homework options:



58
I. Pilot with Administrative Team
II. Develop a Guidebook
III. Developing a Training Protocol
Thank you!
Presenter Name
XXX-XXX-XXXX
xxxxxxxxxxx@xxx.xxx
1234 Street Address
City, State 12345-1234
800-123-1234
59