Educator Evaluation Pilot
July 19-21, 2011
Stonewall Resort, Roanoke, West Virginia
Greetings
Framing the Day
SMART Goals &
Collaborative
Opportunities
SMART Goals and
Collaborative Opportunities
• Learning Target:
– Understand how to develop a SMART
goal
SMART Goals &
Collaborative Opportunities
Linda Bragg
Coordinator
Office of Title II, III & System
Support
West Virginia Department of
Education
This Session
Survey/Review of “Big Rock” Practices
that Focus on Student Learning
• Addressing Critical Questions of Learning
• Using Assessments Practices that Create
Reliable Data
• Analyzing Data
• Prioritizing Needs
• Creating SMART Goals
The Knowing-Doing Gap
Doing
Knowing
SCHOOL-WIDE FOCUS ON LEARNING
Addressing the Four Critical
Questions of Learning
1. What do we want students to learn?
(Essential learning in this unit of instruction)
2. How will we know if/when students have
learned?
3. How will we respond when some of our
students do not learn? What processes are put
in place to ensure students receive additional
time and support for learning in a timely,
systematic way?
4.
How do we enrich and extend the learning
for students who learned from our initial
instruction or are already proficient?
Keeping Abreast of our Current
Reality--Practices based on “Critical
Questions”
• Establish school, team SMART goals based
on assessment data
• Identify learning targets and plan
instruction
• Collect continuous evidence of learning
through formative assessment strategies
• Teams administer common formative
assessments to promote student learning
• Teams analyze common assessment results
& adjust instruction
A Focus on Student Learning
Question # 2 -- How will we
know if/when students have
learned?
A simple but powerful principle:
Small, measurable successes, achieved in
the short term (daily, weekly, interim,
quarterly) can release the energy and
enthusiasm required for students and
teachers to reach long-term learning
goals.
Types of Assessment and Users
Continuous (Formative)
Classroom Assessment
Classroom Level Users
Common Formative
Assessments
Content Level Users
Periodic Interim Assessments
Classroom Teachers
Job-Alike Assessment Partners,
Teacher Teams, Classroom
Teachers
Program Level Users
(Teachers, Teacher Teams and
School Leaders)
Annual Accountability Testing
(State Summative Test)
Institutional/Policy
Users
(School, District and State
Leadership)
Assessment Capacity
• The capacity of teachers to examine
student performance data and make
critical sense of them
• The capacity to develop goals & action
plans based on the understanding
gained from data analyses in order to
improve student achievement
Fullan, M. Leading in a culture of change, p. 117.
Transparency of Student Achievement
& Assessment Practices Illuminates
our Next Steps
1. Provides direction for next instruction
2. Helps educators maintain focus on goals
3. Identifies evidence-based strategies
(critical question # 3 How do we know
if students have learned?)
4. Reveals growth which leads to celebrations
**Team collaborative conversations about
student achievement makes a huge
difference in student learning.
Data is the Key to
Developing Goals
Write Goals in SMART
Terms
•
•
•
•
•
Specific
Measurable
Attainable
Results oriented & Relevant
Time-bound
**Sufficient number of subordinate goals to
encompass all aspects of the BROAD goal
they support
Goal Alignment is Essential
• District SMART Goal
• School SMART Goal
• Team SMART Goal
• Student /Class Goals
• Teacher Professional Goals for
Student Learning
Target Areas & Major
District Objectives
School-wide
Achievement Goals
Collaborative
Team Goals
Student Learning
Goals
Teacher
Professional
Goals
Written in SMART
Terms
Written in
SMART
Terms
Written in
SMART
Terms
Written
in
Written
in SMART
SMART
Terms
Terms
Organizing our
Assessment Data
into Information
Assessments that Produce
Effective Data for the Big
Picture
Exceeds
Student 3
Student 1
Does Not
Meet
Student 4
Student 5
Student 2
Student 6
Student 7
Student 10
Student 8
Student 11
Student 9
Student 12
Student 13
Student 15
Student 14
Meets
Assessments that Produce
Results We Can Use to Improve
Need help
Need help
with focus
keeping focus
throughout the in paragraph
essay
Need help
with thesis
statement
Student 3
Student 1
Student 4
Student 5
Student 2
Student 6
Student 7
Student 10
Student 8
Student 11
Student 9
Student 12
Student 13
Student 15
Student 14
Which Students Are You Concerned About?
Fifth Grade—Making Inferences
Student
Name
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Total
Class 1
Multiple
Choice
80
80
100
100
80
100
40
100
80
100
100
100
80
80
100
20
80
100
100
100
86
Class 2
Extended
Response
4
3
1
4
3
1
2
4
4
1
4
4
3
3
3
2
4
4
4
4
2.9
Multiple
Choice
80
80
100
100
100
100
80
80
100
100
100
100
80
80
100
100
100
100
80
100
93
Class 3
Extended
Response
4
3
3
3
1
4
4
3
4
4
4
3
4
3
4
4
3
3
4
4
3.45
Multiple
Choice
100
60
80
80
60
60
80
80
100
80
80
100
60
80
60
100
80
80
80
80
74
Class 4
Extended
Response
3
2
2
3
2
2
2
2
1
3
2
4
2
2
2
3
3
2
2
3
2.15
Multiple
Choice
100
80
40
100
100
80
100
80
100
100
60
100
100
80
60
80
60
100
80
20
81
Totals
Extended
Response
4
3
2
3
4
3
1
3
4
4
2
3
3
3
2
3
1
4
3
2
2.85
A Further Drill-down
What Do these Data Say about the “Highlighted”
Students?
Fifth Grade—Making Inferences
Student
Name
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Total
Class 1
Multiple
Choice
80
80
100
100
80
100
40
100
80
100
100
100
80
80
100
20
80
100
100
100
86
Class 2
Extended
Response
4
3
1
4
3
1
2
4
4
1
4
4
3
3
3
2
4
4
4
4
2.9
Multiple
Choice
80
80
100
100
100
100
80
80
100
100
100
100
80
80
100
100
100
100
80
100
93
Class 3
Extended
Response
4
3
3
3
1
4
4
3
4
4
4
3
4
3
4
4
3
3
4
4
3.45
Multiple
Choice
100
60
80
80
60
60
80
80
100
80
80
100
60
80
60
100
80
80
80
80
74
Class 4
Extended
Response
3
2
2
3
2
2
2
2
1
3
2
4
2
2
2
3
3
2
2
3
2.15
Multiple
Choice
100
80
40
100
100
80
100
80
100
100
60
100
100
80
60
80
60
100
80
20
81
Totals
Extended
Response
4
3
2
3
4
3
1
3
4
4
2
3
3
3
2
3
1
4
3
2
2.85
Interpret the Data
• What conclusions can you draw from
these data?
• What action steps can you take by
looking at these data?
• Can you create a SMART goal from
these data?
Reflecting upon Student
Learning
• How easy/hard was it to find the next
logical progression of learning from these
data? How might it be helpful to do this
work as a team?
• How might this process help us both
monitor and promote student learning?
Our own learning?
• Why might this be important to include
when doing the work of common
assessments?
Evidence of Student Learning
A Plan for Reliable Data
equals
A Plan for My Teacher
Evaluation!
• What evidence of student learning
do I/we have for each learning target
taught in each instructional cycle?
• What has my team determined as
evidence of student progress?
My Teacher EvaluationWhat Data?
Two Data Points in
Time
Evidence of student
performance collected through
assessment of taught curriculum
Important Record-keeping:
Measure between the Two Data
Points!
Where should we start?
Where did we end up?
**Monitor Along the
Way
SMART GOALS for Student
Learning
Reviewing SMART GOALS
•
•
•
•
•
Specific
Measurable
Attainable
Results oriented & Relevant
Time-bound
Sufficient in number to encompass all
aspects of the BROAD goal they support
SMART GOALS
• Must be connected to data
• Must be connected to students
• Should be shared & results
celebrated
• Should make users feel they are
attainable
Strategic and Specific
• Strategic means the SMART
goal is aligned to district and
school goals and allows schools
to focus on the “vital few”.
• Specific means there is enough
detail for everyone to understand
what it means and teams won’t
disagree on whether the goal has
been reached.
Measurable
• You can quantify it in some way
to know if you are making a
difference
• Seeing progress tends to
increase motivation to keep going
Attainable
• How large is the gap I’m trying to
close?
• Set your goal high enough to stretch
yourselves but not so high you give
up!
Results-Based & Relevant
•Results-based: Require a result
that will impact student
achievement
•SMART goals are more motivating
because the organization allows
you to see short-term successes
•Relevant: You can quickly answer
what students need to know or be
able to do these learning targets
Time-bound
• The time should be significant
enough that people remain
enthusiastic about working
toward it yet enough to
accomplish the goal.
• Time shouldn’t be so long that
the team loses interest.
Where should you start?
Current Reality
• Look at data you’ve collected: current
achievement data;
• Look at your team’s goals
• Consider your School’s Improvement
Goals?
• Other possible considerations: failure
rates; state test scores
Step One: Writing Current
Reality Statement(s) &
Expectations
Write down your “current
reality” and what you
expect to accomplish
Step Two: Make Sure that Your
“What I Expect to Accomplish
Statement” (Goal) is…
Focused on Student Achievement
• Linked to school goal
• Based on Data
• SMART (Check each part of
SMART)
•
Sample SMART Goal
(with no evidence of Current
Reality)
By the end of my next instructional
cycle, 90% of my fourth grade
students will write a nonfiction
report attaining a level of 5 (out
of 6) on the West Virginia writing
rubric.
Example: First Grade Reading Goal
Current Reality Statement
Last year 18% of first grade students were unable
to meet the grade-level proficiency standards in
reading fluency and comprehension as measured
by the assessment for early literacy
development.
My SMART Goal
By the end of my third instructional cycle,
at least 90 percent of my students will
demonstrate proficiency in reading fluency
and comprehension as measured by the
interim assessment for early literacy for
first graders.
Example Third Grade Writing
Goal
Current Reality Statement
Last year 85% of my students met or exceeded
the target score of 3 on the state writing prompt
in March.
My SMART Goal
This year, at least 90 percent of my students
will consistently meet or exceed the target
score of 3 on WV Writes practice tests with
the goal of obtaining at least a 3 on the state
writing test in March.
Eighth-Grade Math SMART Goal
Current Reality Statement
Last year, 24% of the eighth grade math
students failed one or more semesters of
math, and 31% were unable to meet the
proficiency standard in math.
My SMART Goal
By the end of the second instructional cycle, the
percentage of failing grades will reduce to 10% or
less, and the percentage of students unable to
meet state standards will reduce to no more than
15 percent as measured by an interim
assessment of taught objectives.
Let’s Practice!
Celebrate Your Successes!
Get Support!
Dr. Gregory M. Kuhns
Email: gkuhns@access.k12.wv.us
Setting Student Learning
Goals
Setting Student Learning
Goals
• Learning Target:
– Understand how to develop student
learning goals based on multiple criteria
Professional Conduct
Professional Conduct
• Learning Target:
– Understand how to the professional
conduct performance standard identifies
clear criteria for key competencies
related to: policies & procedures,
attendance, schedule, and respect
– Distinguish conduct that is below
standard from conduct that is
unsatisfactory
Table Tasks
• Read pg. 32 in The Guide
• Identify how this rubric is different
from the others; discuss why
different
• Brainstorm situations that could
result in a Below Standard Rating and
an Unsatisfactory Rating for each
competency
Professional Conduct
Takeaways
• Professional conduct is the expectation
– Three point rubric
– Not calculated into summative evaluation
• Select areas of concern may be
addressed without an improvement plan
• An Unsatisfactory rating should not be
given in the pilot
• A Below Standard rating only with
documentation
Research Considerations
Research Considerations
• Learning Target:
– Understand that the evaluation system
pilot involves research elements that are
not directly related to evaluating teacher
performance
– Understand that the pilot ensures
confidentiality of all information related
to personnel matters
Evaluation System Pilot: A
Validation Study
West Virginia Department of
Education (WVDE) Office of
Research
July 21, 2011
Introduction
• A variety of offices within the WVDE will be
working together throughout the course of the
pilot to conduct a validation study of the new
evaluation system.
– Office of Research
– Office of Professional Preparation
– Office of Title I
– Office of Assessment and Accountability
5/29/2016
Division of Curriculum and Instructional Services, Office of Research
56
Why Conduct a Validation Study?
• Ultimately to promote a rigorous, equitable, and
transparent evaluation system, by accomplishing
the following:
– determining the extent to which participants adhere to
the proposed evaluation model (implementation
fidelity)
– determining the extent to which the evaluation system
produces intended results
– identifying and minimizing the occurrence of any
unintended consequences
– using data to fix problems as they arise
5/29/2016
Division of Curriculum and Instructional Services, Office of Research
57
Evaluation Questions
• To what extent have participating schools successfully
implemented the evaluation system with fidelity?
• To what extent has the evaluation system resulted in
professional growth among educators and administrators?
• To what extent do the professional teaching standards relate to
student learning outcomes?
• What facilitators and barriers were encountered during the
implementation of the evaluation system?
• To what extent is the training and support provided by the
WVDE sufficient to support implementation of the system?
5/29/2016
Division of Curriculum and Instructional Services, Office of Research
58
Data Collection
• All forms will be electronic.
– Observation, Evidence, Goal-Setting,
Summative Evaluation
• Once you submit them, they could potentially be
sampled for review as part of the study.
• We will use either a random or purposeful
sample of documents for our study.
• All documents will be de-identified for reporting.
5/29/2016
Division of Curriculum and Instructional Services, Office of Research
59
Confidentiality
• Confidentiality is our number one priority in any
research study.
• The purpose of collecting identifiable information
for this work is to ensure we can track information
over time (e.g., observations 1 - 4 for Teacher A)
• The purpose IS NOT to make judgments about
districts, schools, principals, or teachers.
• Absolutely no names will be published in any
reports.
5/29/2016
Division of Curriculum and Instructional Services, Office of Research
60
Who to Contact about the
Validation Study
Andu Meharie
WVDE Office of Research
ameharie@access.k12.wv.us
(p) 304.558.5426
(f) 304.558.1613
5/29/2016
Division of Curriculum and Instructional Services, Office of Research
61
Enjoy lunch!
Be ready to rock ‘n roll at 1:00 p.m.
Developing a Summative
Rating
Developing a Summative
Rating
• Learning Target:
– Know how the summative evaluation is
calculated
– Complete a summative evaluation
A Revised
Evaluation Process:
A Review of the System and
the Parts
A Breakdown of the System
Components
• Overview of System Components
• How they fit together
• What are the “weights”
• How do the “weights” add up
An Overview of the System
The “7” Standards
Standard
Weight*
1. Curriculum and Planning
17.14%
2. The Learner and the Learning Environment
17.14%
3. Teaching
17.14%
4. Professional Responsibilities for Self-Renewal
11.44%
5. Professional Responsibilities for School and Community
17.14%
6. Student Learning
20%
7. Professional Conduct (required, but no weight)
n/a
*Totals may not equal to 100% due to
rounding
An Overview of the System
The 5 Professional Teaching Standards
• 80% of the total summative rating
• Each critical standard element (CSE) has been given equal
importance
• Goal is to provide as much feedback as possible.
– In addition to a single summative rating, there are overall Standard
ratings, as well.
– Standard ratings are based on the preponderance of ratings at the CSE
level (14 total ratings).
– Information is entered at the CSE level
– CSE ratings range from
•
•
•
•
Unsatisfactory
Emerging
Accomplished
Distinguished
An Overview of the System
A break down of the 5 professional teaching standards.
Standard
Rating
Total of System
Standard 1
Accomplished
17.14%
CSE 1.1
Emerging
5.714%
CSE 1.2
Accomplished
5.714%
CSE 1.3
Accomplished
5.714%
Standard 2
Emerging
17.14%
CSE 2.1
Accomplished
5.714%
CSE 2.2
Emerging
5.714%
CSE 2.3
Emerging
5.714%
An Overview of the System
A break down of the Critical Standard Elements.
Standard
Rating
Standard 1
Accomplished
CSE 1.1
Emerging
CSE 1.2
Accomplished
CSE 1.3
Accomplished
Standard 2
The
preponderance
of evidence
points to an
overall rating of
accomplished
Emerging
CSE 2.1
Accomplished
CSE 2.2
Emerging
CSE 2.3
Emerging
The intent of providing standard ratings at both the standard and CSE
level is to help one identify areas of best practice and need!
An Overview of the System
• The Standard of Student Learning
– A component to help triangulate data
around teaching and learning
– The outcome from the “process” of
teaching
– Multi-faceted using multiple measures
An Overview of the System
The Standard of Student Learning
Standard
Rating
Total of System
Student Learning (Std. 6)
Accomplished
20.00%
Student Learning Goals
Accomplished
15.00%
Standardized School
Growth Scores
Accomplished
5.00%
• This is a high-level overview.
• How does this break down even further?
An Overview of the System
The Standard of Student Learning
Standard
Rating
Total of System
Student Learning (Std. 6)
Accomplished
20.00%
Student Learning
Goals
Accomplished
15.00%
Goal 1
A
7.50%
Goal 2
E
7.50%
Standardized School
Growth Scores
Accomplished
5.00%
Math Growth
A
2.50%
RLA Growth
E
2.50%
An Overview of the System
The Standard of Student Learning
Standard
Rating
Student Learning (Std. 6)
Accomplished
Student Learning
Goals
Accomplished
Goal 1
A
Goal 2
E
Standardized School
Growth Scores
Accomplished
Math Growth
Accomplished
RLA Growth
Emerging
Remember!
Based on:
•Very Low Growth
•Lower Growth
•Typical Growth
•Higher Growth
An Overview of the System
The Standard of Student Learning
Standard
Rating
Total of System
Student Learning (Std. 6)
Accomplished
20.00%
Student Learning
Goals
Accomplished
Goal 1
A
Goal 2
E
Standardized School
Growth Scores
Accomplished
15.00%
20.00
%
10.00
5.00%
%
5.00%
10.00
%
5.00%
Math Growth
A
2.50%
RLA Growth
A
2.50%
An Overview of the System
• The Standard of Professionalism
– A required component of the system
– Does not contribute to the overall rating
– Is considered an important part of the
process
An Overview of the System
Critical Standard
Critical Standard
Elements
Critical1.1
Standard
Elements 1.1
Element 1.1
Critical Standard
Critical Standard
Elements
Critical2.1
Standard
Elements 2.1
Element 2.1
How the pieces add up
Standard 1
Math School
Math School
Growth Score
Growth Score
Standard 2
Critical Standard
Critical Standard
Elements
Critical3.1
Standard
Elements 3.1
Element 3.1
Standard 3
Critical Standard
Critical Standard
Elements 4.1
Element 4.1
Standard 4
Overall
Rating
Standard 7
Critical Standard
Critical Standard
Elements
Critical5.1
Standard
Elements 5.1
Element 5.1
Standard 5
Standard 6
Student
Student
Learning Goal
Student
Learning Goal
Learning Goal
Overview
An educator will receive feedback at 3
levels of varying granularity:
• On overall summative rating (n = 1)
• A rating for each Standard that
composes the overall rating (n = 7)
• A rating for each critical standard
element that composes each Standard
(n = 14 for PTSs; n ≥ 4 for Student Learning)
Next Steps:
Considerations for
August
Performance Assessment
Closing Remarks