Washington State
Teacher and Principal
Evaluation Project
Conducting High-Quality Observations
1
June 2013
Entry Task: Confidence Conversation
As you enter, please have a brief discussion with your district
team to decide your level of confidence in whether the
following statements are true for your district:
1. Our evaluators know how to minimize bias when collecting
observation data.
2. Our evaluators know how to capture observation evidence
using a running record approach to note-taking.
3. Our teachers trust our observers to capture accurate
evidence that will provide them with useful feedback for
professional growth.
Write your district name on three sticky notes and place them
on the confidence scales posted on [INSERT LOCATION] for
each statement.

2
Welcome!
Introductions
Logistics
Agenda




Agenda







3
Connecting
Learning I
Learning II
Learning III
Learning IV
Reflecting
Wrap-Up
Modules
Introduction to Educator Evaluation in Washington
Using Instructional and Leadership Frameworks in Educator
Evaluation
Preparing and Applying Formative Multiple Measures of
Performance: An Introduction to Self-Assessment, Goal Setting,
and Criterion Scoring
Including Student Growth in Educator Evaluation
Conducting High-Quality Observations and Maximizing Rater
Agreement
Providing High-Quality Feedback for Continuous Professional
Growth and Development
Combining Multiple Measures Into a Summative Rating







4
The Evaluation System Components
5
Session Norms

Pausing
Paraphrasing
Posing Questions
Putting Ideas on the Table
Providing Data
Paying Attention to Self and Others
Presuming Positive Intentions

What Else?






6
Connecting
Builds community, prepares the team for learning, and links to
prior knowledge, other modules, and current work
7
Module Overview: 2 Parts
Conducting High-Quality Observations
Maximizing Rater Agreement


Reminder!
 This module provides an orientation to the basic
concepts.
 This module does not go into great depth about evidence
relating to any of the specific instructional or leadership
frameworks and instead leaves it up to the districts to
seek additional training.

8
Overview of Intended Participant
Outcomes
Participants will know and be able to:
 Describe the role of observation as a methodology for
gathering evidence of educator performance and its role
in achieving rater agreement.
 Utilize the revised WACs to identify the number and
length of observations for different categories of staff.
 Identify common observation biases in their own and
others’ practice.
 Utilize appropriate strategies for minimizing bias in the
observation.
 Utilize best practices in collecting observation data.
9
Observation as One Piece of the Puzzle
Remember: Observation is one measure within a
comprehensive evaluation system that includes multiple
measures, including the following:
 Self-Assessment
 Goal-Setting
 Artifact Review
Observation is another
piece, helping us move
closer to enacting TPEP’s
Core Principles and the
new system
10
TPEP Core Principles






“We Can’t Fire Our Way to Finland”
The critical importance of teacher and leadership quality
The professional nature of teaching and leading a school
The complex relationship between the system for teacher
and principal evaluation and district systems and
negotiations
The belief in professional learning as an underpinning of
the new evaluation system
The understanding that the career continuum must be
addressed in the new evaluation system
The system must determine the balance of “inputs or
acts” and “outputs or results”
11
Rater Agreement Depends on HighQuality Evidence Collection Practices
Criterion 1
•
•
Criterion 2
Criterion 3
Criterion 4
Criterion 5
Criterion 6
Frameworks
+
Student
Growth
Rubrics
•
•
Observation
Student
growth data
Artifacts
Other
evidence
relevant to
the
frameworks
Districtdetermined
process
Statedetermined
process
Distinguished
Proficient
Basic
Unsatisfactory
Distinguished
Proficient
Basic
Unsatisfactory
Criterion 7
Student Growth
Measures
Criterion 8
(From 3 specific criteria)
12
Student
Growth Impact
Ratings:
Low, Average, High
Rater Agreement
The extent to which the scores between the raters exhibit
consistency and accuracy against predetermined
standards. The predetermined standards are the
instructional and leadership frameworks and rubrics that
define the basis for summative criterion-level scores.
13
What the Research Says…
Evaluators need multiple opportunities and settings to observe
and assess educator practice.
 Multiple observations paired with timely feedback are a key
part of a strong evaluation system.
 Using multiple observers and multiple observations improves
the reliability of scores more than having longer observation
periods.
 Using a mix of shorter and longer observations can decrease
costs while maintaining increased reliability of scoring.
 Using a mix of internal and external observers (school-based
and not) can help determine the validity of practice scores.
(Sources: Ho & Kane, 2013; Sartain et al., 2011)

14
Voices on Observation

Handout 1: Studies link classroom observations to student
achievement, by Stephen Sawchuck

Handout 2: Feedback for Better Teaching, by The Measures of
Effective Teaching (MET) Project. (pp. 1-2 and 6-7; skip pp. 3-5)
15
Voices on Observation Wrap-Up

What are the key messages and ideas in the two articles
with regard to observations?

Based on what we just read, what are some best practices
for observation?
16
Learning
Understand the new requirements for educator observation in
Washington, as well as best practices and strategies for
conducting high-quality observations
17
Learning Introduction: Meet the
New Rules

1. In the 2012-2013 school year, Mr. Davis was in his
eighth year of teaching at your school. He has continuing
contract status and was rated “satisfactory” in his last
four evaluations.
True or False: Mr. Davis is eligible for focused evaluation in
2013-2014.
Answer: True
RCW 28A.405.100 (12)(b)(iv)(c)(i)

18
Learning Introduction: Meet the
New Rules

2. Fill in the blank: Mr. Garvey has been a principal in
North Thurston for the past five years. He is starting his
first year as a principal in your district in September. Mr.
Garvey should have a ___________ evaluation [focused
or comprehensive] this year.
Answer: Comprehensive
RCW 28A.405.100 (12)(b)

19
Learning Introduction: Meet the
New Rules

3. Fill in the blank: Mr. Burke has taught for ten years in
your district and has continuing contract status. He is
currently on a comprehensive evaluation cycle. Mr. Burke
must be observed ____ times this year for no less than
_____ minutes total.
Answer: 2 and 60
WAC 392-191A-070

20
Learning Introduction: Meet the
New Rules

4. Fill in the blank: Mrs. Wright is a 20-year veteran
teacher but just started teaching at your school this year.
As a new employee, Mrs. Wright must be observed once
for ___minutes during the first ____calendar days of the
employee period.
Answer: 30 and 90
WAC 392-191A-070 (2)

21
Learning Introduction: Meet the
New Rules

5. Mr. Jones is starting his third year in teaching and is a
provisional teacher. This year, he must be observed:



A. At least two times for a total of 120 minutes
B. At least three times for a total of 90 minutes
C. At least four times for a total of 150 minutes
Answer: B
WAC 392-191A-070 (3)

22
Learning Introduction: Meet the
New Rules

6. Ms. Thomas has been on a focused evaluation for the
past two years. In both years, she was rated as proficient.
This year:



A. She must have a comprehensive evaluation.
B. She and her supervisor can decide whether to continue with
a focused evaluation or switch to a comprehensive evaluation.
C. She must have a focused evaluation.
Answer: B
WAC 392-191A-070(12)(c)(iv)

23
Learning Introduction: Meet the
New Rules

7. Mr. Tucker had a comprehensive evaluation last year
and was rated as “proficient”; however, he and his
evaluator identified Criteria 4 as an area that needed
improvement.

True or False: Mr. Tucker is required to use Criterion 4
for his focused evaluation next year.
Answer: False
WAC 392-191A-120(2)

24
Learning Introduction: Meet the
New Rules

8. Mr. Tanyu has continuing contract status and is on a focused
evaluation cycle. He selected Criteria 7: “Communicating and
collaborating with parents and school community.” You notice
that Criteria 7 does not need to be assessed through
observation; other methods, such as artifact review, may be
more appropriate.
Mr. Tanyu:



A. Does not need to be observed this year.
B. Must be observed this year.
C. Districts can decide whether or not to require observation in this
situation.
Answer: False
WAC 392-191A-110

25
Learning Introduction: Meet the
New Rules

9. Part I: Principal Smith completed an observation of Ms.
Williams on Tuesday and promptly wrote up the results of the
observation later that day. Principal Smith must give a copy of
the report to Ms. Williams within:




A. four days.
B. no more than one week.
C. three days.
9. Part II: Principal Smith and Ms. Williams must hold a
minimum of ____ confidential conferences during each year.
Answers: C and 2
WAC 392-191A-130

26
Learning Introduction: Meet the
New Rules

10.True or False? The purpose of observation is to:



(a) collect accurate, relevant evidence to evaluate against
instructional and/or leadership frameworks, and
(b) provide educators with quality feedback.
Answer: True!
27
Learning Content I: High-Quality
Observation

Frequent: research agrees that a single observation of
teacher or leader practice is insufficient for capturing evidence
to generate accurate and reliable ratings.

Comprehensive: captures evidence appropriate for assessing
the educator against the required criteria.

Accurate: minimizes bias and error to ensure that collection
and review of the data produces an authentic assessment of
the educator’s performance.

Useful & Timely Feedback: generates detailed, focused
feedback for improvement.
28
Learning Content I Activity: HighQuality Observation

What does it mean to be [INSERT DESCRIPTOR] in your
observations?

How does this principle of high-quality observations
better support the improvement of teaching and learning?
29
Learning Content I Activity: Debrief

Share out your two key ideas for your Guiding Principle

Frequent

Comprehensive

Accurate

Feedback
30
Learning Content II: Strategies for
Reducing Bias in Observation
A skilled observer:
 1. Understands each component and indicator on the
district rubric thoroughly and deeply.

2. Gathers and sorts sufficient evidence of practice as it
happens in the classroom or school.

3. Recognizes and puts aside preferences and biases.

4. Interprets the evidence appropriately to give an
accurate rating using the evaluation instrument.
31
(McClellan, Atkinson, & Danielson, 2012)
Common Sources of Bias in Observation

Bias enters the process in two places:

Observation bias: while recording data during an
observation, every observer has a “lens” altered by:





32
Specializations: content or grade level
Personal instructional preferences
Past observations or impressions of the educator
Individual bias or experiences.
Rater error and bias: observers rate performance too high
or too low based on considerations other than the
instructional or leadership framework.
Reducing Bias in Observation

Record evidence, not judgment or opinion:



Evidence describes a concrete behavior, practice, interaction, or
material that can be used to support a rating decision and provide
clear feedback.
Judgment or opinion comments on or interprets what is being
observed but does not provide concrete evidence.
Observation notes are written as evidence statements:

33
Evidence statement: A factual statement, free from opinion or bias,
that describes teacher practice.
Reducing Bias in Observation

EXAMPLE:
Opinion Statement: “Yes! Good questioning, engages the students.”
Evidence Statement:
T: “How do you know, or how did you get that answer?”
S: (Multiple hands shoot up)
T: Calls on Alissa
S: “I went back and used Pythagorean’s formula.” S writes out
process on board and explains each step.
34
Strategies for Recording Evidence

Record:





Direct quotes
Questions asked
Observed actions or movements
Literal descriptors
Avoid:


35
Jotting down extraneous information
Jotting down initial thoughts on ratings
Learning Activity II: Evidence Versus
Opinion

Examine Handout 4

Highlight factual statements

Underline judgment statements or statements not based
on evidence
36
Learning Content III: Best Practices for
Collecting Observation Data

Holding the Pre-observation Conference




37
Review the proposed lesson plan
Listen to the teacher’s perspective and learn about the
classroom context
Understand the teacher’s professional goals and expectations
Identify any specific areas the teacher would like feedback on
Best Practices for Collecting
Observation Data

Before Observing:

Focus: Review pre-conference notes and the educator’s
professional goals

Clean Your Slate: Purposefully identify and set aside previous
impressions from earlier observations
38
Best Practices for Collecting
Observation Data

While Observing:



39
Be “a fly on the wall”: Avoid interacting with the participants to
minimize your influence on what you are observing.
Record only information you actually observe—do not “fill in
the blanks” or assume other practices are occurring without
actually observing them.
Record the depth and frequency of key “look-fors” (e.g., types
of questioning, student behaviors, re-direction to task).
Best Practices for Collecting
Observation Data

Balance taking detailed notes with observing the action in
front of you.


40
Use abbreviations to increase efficiency (e.g., “T” for teacher,
“S” for student, or components by their first two letters—
“LD” for lesson delivery).
Paraphrase around short quotes to convey the spirit and
tone of the interaction.
Best Practices for Collecting
Observation Data

41
Example Observation Notes
41
Learning Activity III: Practicing
Collecting Observation Data

Video Background



“Ms. K” is teaching a sixth-grade science lesson.
25 students, ranging from 11 to 13 years old, are in her class.
Two students are English language learners and several
students have IEPs for accommodations in reading.
NBPTS Video #161: Probing Student Understanding
(http://198.101.195.20/video.php?p=15346487)
42
Learning Activity III: Practicing
Collecting Observation Data

Compare your notes for the following:

Step 1: Do the notes capture the same data?


Step 2: Do the notes record evidence, such as:





43
What are you missing?
quotes?
questions?
actions or movements?
Step 3: Do the notes include judgments or opinions?
Step 4: Did you identify any personal biases while you were taking
notes?
Learning Content IV: Overview of eVAL



44
How many of you know what eVAL is?
Is anyone currently using eVAL?
Is anyone considering using eVAL?
eVAL





A web-based tool designed to manage the evaluation
process and documentation
A free resource!
Personalized for each district for their instructional
framework, resources, and documents
Voluntary for all districts, who can use as many or as
few of eVAL’s features as they’d like (or none at all)
Extremely secure, with limited access physically and
virtually to its servers
45
eVAL
eVAL helps evaluators and educators in five specific ways:


Practice: Enter formative evaluation data regarding videos or
classroom observations and engage in a dialogue about shared
(or different) perceptions of quality instruction.

Conference: Plan for pre-conferences, observations, and
post-conferences—including sending questions in advance
of meetings and scheduling observations.
46
eVAL
Observe: Principals can…





47
Enter notes during observations (or upload them from other
software).
Align evidence from notes to instructional and leadership
frameworks and provide feedback to teachers.
Score and annotate the rubric and manage all aspects of
evidence gathered in the observation.
Create post-conference observation reports.
eVAL
Summary & Reports: Principals can…





48
Submit final summative scores for teachers to their district office.
Reflect on all the evidence gathered for the year.
Score final summative rubrics.
Print final summative reports, including schoolwide evaluation
summaries, individual evaluation trends, and discrepancies
between teacher self-assessments and evaluator scores.
eVAL

For more information, check out the eVAL website:


http://tpep-wa.org/resources/eval/
Watch eVAL video walkthroughs:

49
http://tpep-wa.org/resources/eval/eval-video-walkthroughs/
Reflecting
50
Whip Around and Plus/Delta Debrief
Select one of the Debrief Cards and share:
51

One key observation I made during this session

One key question I have as a result of this session

One way someone else helped me out during this session

Something new I have learned as a result of this session

One feeling I experienced during this session
Wrap-Up
Understand the options for homework
Preparation for the Maximizing Rater Agreement module
52
Preparation for the Maximizing Rater
Agreement Module

Bring a copy of:


Your observation notes from Learning Activity III.
Your district’s instructional framework.
You will use these to become familiar with best practices
in analyzing observation data and achieving rater
agreement.
53
Suggested Homework

Discuss and plan an annual schedule of observations:



54
Consider the number of evaluators, the number of teachers,
and the number of school days.
Consider how many teachers and principals are on focused
versus comprehensive evaluations.
Consider setting deadlines for completing the observation of
all teachers.
Thank you!
Presenter Name
xxxxxxxxxxx@xxx.xxx
55