Washington State
Teacher and Principal
Evaluation Project
Preparing and Applying Formative Multiple Measures of Performance
Gathering Evidence
1
June 2013
Entry Task
As you enter, jot down on a sticky note:

2

One disadvantage of having an observation only evaluation
system

One advantage of the current evaluation system
Welcome!
Introductions
Logistics
Agenda




Agenda





3
Connecting
Learning
Implementing
Reflecting
Wrap-Up
Modules
Introduction to Educator Evaluation in Washington
Using Instructional and Leadership Frameworks in
Educator Evaluation
Preparing and Applying Formative Multiple Measures of
Performance: An Introduction to Self-Assessment, Goal
Setting, and Criterion Scoring
Including Student Growth in Educator Evaluation
Conducting High-Quality Observations and Maximizing
Rater Agreement
Providing High-Quality Feedback for Continuous
Professional Growth and Development
Combining Multiple Measures Into a Summative Rating







4
Session Norms

Pausing
Paraphrasing
Posing Questions
Putting Ideas on the Table
Providing Data
Paying Attention to Self and Others
Presuming Positive Intentions

What Else?






5
Connecting
Builds community, prepares the team for learning, and links to
prior knowledge, other modules, and current work
6
Overview of Intended Participant
Outcomes for Part D
Participants will know and be able to:
 Determine specific measures to support implementation
of evaluation in their districts
 Understand the evaluation cycle across a school year and
how multiple measures fit in
 Articulate ways to integrate multiple forms of evidence
into educator evaluation to move beyond an observationonly evaluation system
 Determine how to create criterion scores after gathering
of evidence aligned with the Washington State Criteria
7
Guidance Icon Key
RCW 28A.405.100
8
A capital “G!” indicates that
the guidance represents
Washington state law.
RCW 28A.405.100
A lower-case “g” indicates that the
guidance represents research-based
best practice but is not mandated by law.
Connecting the Pieces of the Puzzle
Part A described how using multiple measures
strengthens a teacher evaluation system.
Part B discussed self-assessment.
Part C discussed goal setting.
The goal of criterion scoring is to begin the process of
putting the pieces of a comprehensive teacher evaluation
system together.




9
The Year-Long Evaluation Cycle
8 Criteria
Frameworks
+
Student Growth
Rubrics
• Observation
• Student
Growth
• Evidence
Districtdetermined
process
Statedetermined
process
Distinguished
Proficient
Basic
Unsatisfactory
Distinguished
Proficient
Basic
Unsatisfactory
Step 1
• Criteria aligned to
instructional /leadership and
student growth rubrics
• Professional goals (g)
• Instructional/leadership goals (g)
• Student growth goals (G!)
10
Step 2 & 3
Select and
collect evidence
• 2 observations (G!)
• Student growth (G!)
• Other evidence (g)
Step 4
Determine
8 Criteria Scores
(g)
Step 5
Summative Score
(G!)
Evidence Cover Page
11
Evidence Collection Form
12
Digging for Evidence: Evidence for John
Anderson

Let’s return to his self-assessment on Criterion 5




13
Student strengths: Last year’s data demonstrate that students
are well-behaved.
Student weaknesses: Students need to learn how to regulate
their behavior; students need to learn how to work
constructively with their peers.
Teacher strengths: Scored high on Criterion 5 last year;
Students felt comfortable asking for help from the teacher
last year
Teacher weaknesses: Did not participate in service-learning
projects last year; decreased engagement in classroom as
a result.
Digging for Evidence: Artifacts of John
Anderson

Included are six artifacts that can be used as evidence for
Criterion 5.

Evidence cover page has two things missing



Alignment to instructional framework dimensions in criterion 5
Evidence statement
With your partner, take a set of artifacts, align it to the
framework in Criterion 5, and develop evidence statements
14
Set
Artifacts
A
Student survey
Copies of student awards
B
Observation notes
Classroom layout
C
Student behavior plan
Rules and procedures document
Handout
Digging for Evidence Debrief

With your district team:



15
Do you think you need all six pieces of evidence to measure
Criterion 5? Why or why not?
How would you score each piece of evidence?
How would you put all of this evidence together to create a
score for Criterion 5?
Critical Decisions About Methods

Key messages for multiple measures




16
Need a variety of evidence
Not about collecting as much as
Criteria and
you can; this is about quality,
Instructional
Frameworks
not quantity
Evidence should be based on
naturally occurring documents/materials
Prioritize evidence collected based on
goals and criteria.
Measures
Alignment
Learning
Understand best practices in gathering evidence
17
OSPI’s: Guiding Principles for
Criterion Scoring

The primary goal of any system of teacher evaluation is to promote teacher
and student learning.

Accurate teacher evaluation requires trained observers using a researchbased instructional framework. Trained observers make accurate
assessments of practice based on evidence.

The value of accurate assessments of practice is to shape the conversations
that lead to improved practice.

Embedded in each instructional framework is a system for growth in
teaching practice.

Reliability and validity of the instructional framework relies on
implementation of the full framework rather than individual
components/indicators.

It is imperative to remain in the formative mindset until the final summative
rating
is determined.
18
The Criterion Scoring Process




Step 1. Collect, align, and draw conclusions about
evidence based on framework alignment to criteria
Step 2. Determine score for each indicator based on
preponderance of evidence
Step 3. Enter score in Criterion Scoring Matrix
Step 4. Determine criterion score
Criterion Scoring Process
1. Collect, align,
score evidence
3. Enter
indicator score
2. Determine
score,
preponderance
of evidence
19
4. Determine
criterion score
Learning Activity: Step through the
process
Step 1: Collect, align, and draw conclusions about evidence
based on framework alignment to criteria.


Organize the evidence that you have – both scores and more
formative information
Consideration:


How did you score the evidence? Is it on the same scale as the
criteria? If not how do you convert it?
If evidence aligns to multiple indicators, is the score the same on each
indicator?

20
For example, if you administer a student survey that relates to multiple
indicators, will each indicator get the same score, or a different score
based on the dimension in the survey?
Step 1: Sorting through the evidence
Criterion 5: Learning Environment (Danielson)
Obs
Student
Behavior
Plan
2a Respect
rapport
3
2c Class
procedure
2
2d Student
behavior
3
2e Physical
Space
2
21
Rules/
Procedure
Doc
Student
Survey
Student
Awards
Physical
Layout
3
Other
Notes
Her class is
polite to
others in
the hallway
3
2
1
Students
behave
great
throughout
school
3
Indicator
Score
Learning Activity: Step through the
process
Step 2: Determine score for each indicator based on
preponderance of evidence


Based on all of the evidence that you have, determine a
preponderance score for each indicator.
Consideration:


22
Does some evidence have more weight than other evidence?
Does the evidence provide enough information to be as accurate and
close to truth as possible?
Step 2: Creating indicator score
Criterion 5: Learning Environment (Danielson)
Obs
Student
Behavior
Plan
2a Respect
rapport
3
2c Class
procedure
2
2d Student
behavior
3
2e Physical
space
2
23
Rules/
Procedure
Doc
Student
Survey
Student
Awards
Physical
Layout
3
Other
Notes
Her class is
polite to
others in
the hallway
3
2
Indicator
Score
3
2
1
Students
behave
great
throughout
school
3
2
2
Learning Activity: Step through the
process
Step 3: Enter score in Criterion Scoring Matrix

Based on the indicator scores you determined in step 2, place
the scores in the Criterion Scoring Matrix
Criterion 5: Learning Environment (Danielson)
Indicator
Unsatisfactory
1
Basic
2
2a Respect
rapport
X
2c Class
procedure
X
2d Student
behavior
X
2e Physical
space
X
24
Proficient
3
Distinguished
4
Score
Learning Activity: Step through the
process
Step 4. Determine Criterion Score



Use the indicator scores in the Criterion Scoring Matrix to
determine criterion score.
Analyze the combination of scores by each indicator within a
criteria to determine the score.
Considerations:


25
Based on the preponderance scores for the instructional framework
indicators, is there a clear criterion score (e.g., are the majority of
indicators scored at one level)?
If there is not a clear criterion score based on the preponderance scores,
use the guiding questions in the scoring document (next slide).
Learning Activity: Step through the
process
Step 4. Determine Criterion Score Guiding Questions






26
What else do I need to see or consider to make a final decision –
what is available to me?
What is the distribution of evidence over time?
Has there been demonstrated and consistent improvement? If there
was growth, was the growth sustained?
What would be the tipping point? If I consistently saw “X,” I would
feel confident that the performance is Basic. If I consistently saw “Y,” I
would feel confident that performance is Proficient.
What is the essence of the indicator? The criterion? Go back and find
the key words in the framework/rubric. What does the evidence tell
you about the evaluatee’s performance and growth with regard to
this essential aspect of the indicator/criterion?
Is this evaluatee more Basic than s/he is Proficient, or more Proficient
than s/he is Basic in this indicator? What is the evidence based in the
framework/rubric to support your decision?
Learning Activity: Step through the
process

Step 4. Determine Criterion Score – clear score.
Criterion 5: Learning Environment (Danielson)
Indicator
Unsatisfactory
1
Basic
2
2a Respect
rapport
Distinguished
4
Score
X
2c Class
procedure
X
2d Student
behavior
X
2e Physical
space
X
27
Proficient
3
2
Learning Activity: Step through the
process

Step 4. Determine Criterion Score – unclear score

If, for example, the indicator scores were as follows, what
would you do? This is where the formative evidence is very
helpful.
Criterion 5: Learning Environment (Danielson)
Indicator
2a Respect
rapport
Basic
2
Proficient
3
Distinguished
4
Score
X
2c Class
procedure
X
2d Student
behavior
X
2e Physical
space
28
Unsatisfactory
1
?
X
Learning Activity 2: Austin’s Butterfly

We are going to watch a short video, Austin’s Butterfly


http://vimeo.com/38247060
After the video, discuss:



29
How did you feel after watching the video?
How does this video relate to the guiding principles of OSPI?
How does this relate to educator evaluations and the goal of
evaluations?
Learning Activity 2: Try it yourself.

Use the Criterion Five evidence from the beginning
activity to go through the 4-step criterion scoring process




30
Step 1. Collect, align, and draw conclusions about evidence
based on framework alignment to criteria.
Step 2. Determine score for each indicator based on
preponderance of evidence
Step 3. Enter score in Criterion Scoring Matrix
Step 4. Determine criterion score
Learning Debrief

With your district team:



31
What did you like about the criterion scoring process?
What did you find difficult about this process?
Share one outstanding question you have about the criterion
scoring process.
Implementing
Articulate the ways collecting evidence through artifacts can
operationalize the revised evaluation system to improve teaching
and learning
32
Increasing Clarity and Feasibility


Districts should think strategically about how to increase
clarity around the teacher evaluation process and ensure
the process is feasible for teachers and evaluators.
Some examples follow on the next few slides.
33
Increasing Clarity and Feasibility –
Step 1

RCW 28A.405.100
Create a table with the relevant evaluation measures
listed for all Washington State Criteria – so everyone
knows what measures should contribute to each rating


This also shows teachers where to focus their evidence
collection and decreases the amount of evidence evaluators
need to review
Use Handout 13A, B, or C depending on instructional
framework
Washington State
Criteria
Observation
Evidence Review
…
34
7
X
8
X
Increasing Clarity and Feasibility –
Step 2


RCW 28A.405.100
Describe the entire evaluation process in detail,
describing roles of teachers and evaluators.
Use Handout 14, which provides multiple questions to
consider.
35
Identifying Tools and Processes for
Gathering and Organizing Evidence


Create a plan for using Evidence Review in teacher
evaluation.
Use the “Plan for Implementing Evidence Review”
Handout 15 of your packet to structure your
conversation with your school team.
36
RCW 28A.405.100
Implementing Activities Debrief

Each team shares one decision that was made today to
increase the clarity and feasibility of the teacher
evaluation process.
37
Reflecting
38
Whip Around and Plus/Delta Debrief

Whip Around: One significant “ah-ha moment” today

Take a few minutes and create at least two sticky notes
for the Plus/Delta Chart on your way out.


39
Plus: What was a real “plus” of today’s session? What went well
and should be repeated?
Delta: Where is there room for improvement and change?
What’s Next?

Homework Options



40
District: Start a district teacher evaluation guidebook that
includes all of the nuts and bolts of the teacher evaluation
process. Use the information you recorded on the “Plan for
Implementing Evidence Review” handout as a starting place.
School or Teams: Identify times when teachers are already
asked to collect artifacts – whether that means submitting
lesson plans or turning in PLC meeting minutes.
Individual: Consider authentic evidence of practice that you
create. Choose a couple and identify which criteria they
provide evidence of.
Thank you!
INSERT PRESENTER’S NAME AND E-MAIL ADDRESS
41