1
Teacher Evaluation
Field Test
School Leader Development
January 22/23, 2013
BALTIMORE CITY
PUBLIC SCHOOLS
Agenda
2














8:00-8:30 – Registration and Breakfast
8:30-9:20 – Address from Dr. Sonja Brookins Santelises and Jennifer Bell-Ellwanger
9:20-9:40 – Q & A Session with Dr. Sonja Brookins Santelises and Jennifer Bell-Ellwanger
9:40-9:55 – Break
9:55-11:45 – Student Learning Objectives
11:45-12:00 – Q & A session on Student Learning Objectives
12:00-12:45 – Lunch
12:45-1:45 – Value-Added
1:45-2:00 – Q & A on Value-Added
2:00-2:15 – Break
2:15-2:45 – Professional Responsibilities
2:45-3:30 –Student Surveys
3:30-3:45 – Q & A on Student Surveys
3:45-4:00 – Close-Out
BALTIMORE CITY
PUBLIC SCHOOLS
The Problem…
3
“Current teacher evaluation systems are
not providing the information
needed to close the achievement
gap. Despite 40 years of research
pointing to huge differences in student
achievement gains across teachers,
most school districts…cannot
pinpoint what makes a teacher
effective, or identify their most
and least effective teachers”
Working with Teachers to Develop Fair and Reliable Measures of Effective Teaching
Bill & Melinda Gates Foundation
BALTIMORE CITY
PUBLIC SCHOOLS
The Problem…
4
“The Widget Effect describes the tendency of
school districts to assume classroom
effectiveness is the same from teacher to
teacher. This decades-old fallacy fosters an
environment in which teachers cease to
be understood as individual
professionals…In its denial of individual
strengths and weaknesses, it is deeply
disrespectful to teachers; in its
indifference to instructional effectiveness,
it gambles with the lives of students.”
The Widget Effect: Our National Failure to Acknowledge and Act on Differences in Teacher Effectiveness
The New Teacher Project
BALTIMORE CITY
PUBLIC SCHOOLS
The Road Behind:
Teacher Evaluation of the Past
5
Undifferentiated
BALTIMORE CITY
PUBLIC SCHOOLS
The Road Ahead:
An Evaluation System Grounded in Development
6
“Evaluations should provide all teachers with regular
feedback that helps them grow as professionals, no
matter how long they have been in the classroom. Evaluations
should give schools the information they need to build the
strongest possible instructional teams, and help districts hold
school leaders accountable for supporting each teacher’s
development. Most importantly, they should focus everyone
in a school system…on what matters most: keeping every
student on track to graduate from high school ready for
success in college or a career.”
Teacher Evaluation 2.0
The New Teacher Project
BALTIMORE CITY
PUBLIC SCHOOLS
The Road Ahead:
An Evaluation System Grounded in Student Achievement
7
Improved
Teacher
Evaluation
Based on
Multiple
Measures
Data-Driven
Professional
Development
Improved
Teaching
Practices
Strategic
Human Capital
Decisions
BALTIMORE CITY
PUBLIC SCHOOLS
Improved
Student
Outcomes
Implications for City Schools
8
 City Schools is committed to creating a new
evaluation process grounded in teacher development
that promotes student achievement
Clear, rigorous expectations
 Multiple measures
 Differentiated ratings
 Regular feedback
 Significance

BALTIMORE CITY
PUBLIC SCHOOLS
Implications for City Schools
9
Clear, Rigorous
Expectations
Multiple
Measures
Differentiated
Ratings
Teachers should be
evaluated against clear,
rigorous performance
expectations based
primarily on evidence of
student learning.
No single data point can
paint a complete picture
of a teacher’s
performance
Teachers should earn a
summative rating at the
end of each school year
that distinguishes
performance between
levels with meaningful
differentiation of
performance
•Expectations should reflect
excellence in the classroom,
not minimally acceptable
performance.
•Evaluations should
incorporate multiple
measures for assessing
professional practice and
student growth, with
greater weight afforded to
the most accurate measures
of student progress.
BALTIMORE CITY
•Teachers should have a
clear picture of their current
performance.
PUBLIC SCHOOLS
Implications for City Schools
10
Regular Feedback
Evaluations should not be
limited to a single end-of-year
rating, but instead are viewed as
an ongoing developmental
process.
•Cultivate a performance-focused
culture with frequent teacher
observations.
•Regular conversations with teachers to
discuss overall classroom performance
and student progress; professional goals
and developmental needs; and the
support school leaders will provide to
meet those needs.
BALTIMORE CITY
Significance
Evaluations should have
meaningful implications
•Teacher effectiveness should play an
important role in determining
strategic human capital decisions.
•Ability to identify, develop, and keep
talented teachers is one of the most
important priorities for schools leaders
PUBLIC SCHOOLS
Teacher Evaluation Field Test Measures
11
Teacher
Evaluation
Student Growth
Student
Learning
Objectives
(SLO)
School index
Professional
Responsibilities
Student Surveys
Observations/
Instructional
Framework
ValueAdded
Model
(VAM)
50%: Multiple measures of student growth
50%: Professional practice
• City Schools will propose weights after studying the results of the pilot and consulting with
our teachers and principals. No one measure can account for more than 35% of a teacher’s
evaluation.
BALTIMORE CITY
PUBLIC SCHOOLS
Why did we choose these measures?
12
 City Schools believes in the following:
 Measures should help inform and improve teacher practice
 Measures should demonstrate both validity and reliability
 Measures should align with teaching best practices and research
BALTIMORE CITY
PUBLIC SCHOOLS
Why did we choose these measures?
Research from the MET Project
13
What is the MET
Project?
The MET project, funded by
the Gates Foundation, is
testing five different measures
of teaching effectiveness:
1. Student achievement gains
on state standardized tests
and supplemental tests
2. Classroom observations
and teacher reflections
3. Teachers' pedagogical
content knowledge
4. Student perceptions of the
classroom instructional
environment
 MET stands for Measures of Effective
Teaching, and the project aims to help
teachers and schools understand what
great teaching looks like.
 The goal of the MET project is to
identify which multiple measures give
the best and most accurate information
about how well a teacher helps his or
her students learn – and how these
measures should be used together to see
the whole picture of a teacher's
effectiveness.
BALTIMORE CITY
PUBLIC SCHOOLS
The MET Study
14
•Two school years: 2009-10 and 2010-11
•>100,000 students
•Grades 4-8: ELA and Math
•High School: ELA I, Algebra I and Biology
BALTIMORE CITY
PUBLIC SCHOOLS
Three Key Take-Aways
15
High-quality classroom observations will require clear
standards, certified raters, and multiple observations
per teacher.
2. Combining three approaches (classroom observations,
student feedback, and value-added student achievement
gains) capitalizes on their strengths and offsets their
weaknesses.
3. Combining new approaches to measuring effective
teaching – while not perfect – significantly outperforms
traditional measures. Providing better evidence should
lead to better decisions.
1.
BALTIMORE CITY
PUBLIC SCHOOLS
Why did we choose these measures?
16
Component
Rationale: Why this measure?
Rationale: Why this tool?
Student growth:
VAM and SLOs
Under our theory of action, student
achievement is the ultimate end of our
evaluation work. Student growth is our
business as educators.
VAM provides longitudinal achievement data and controls
for factors outside the teacher’s control. VAM is costeffective because it can be used to measure student growth
for many teachers.
RTTT and the Ed Reform Act require measures
of student growth as a significant part of the
evaluation.
SLOs allow us to measure student growth in non-tested
content areas, and they support instructional best
practices.
Instructional
Framework
Professional development is critical to any
evaluation system. The Framework provides
teachers and school leaders with guidance on
how to improve professional practices.
The Framework defines highly effective teaching practices
in Baltimore City, and provides teachers with specific,
actionable feedback.
Professional
Responsibilities
Checklist
Professional responsibilities are included of
most teaching frameworks. Teachers should
model professionalism expectations for our
students.
The Checklist evaluates teachers on non-instructional
functions. Checklist reflects key district priorities, such as
teacher attendance and arrival, compliance with policies
and testing integrity.
Student Surveys
Students recognize effective teaching when
they experience it. Students are our customers,
and have more direct contact with teachers
than anyone else.
The Survey contains items that are aligned to Framework
indicators and developmentally appropriate for students.
Survey data reinforces feedback that teacher receive from
classroom observations.
School Index
Teaching is a team sport. All staff in a school
are responsible for certain student outcomes.
School indexes consider indicators such as attendance or a
positive school climate, which the entire school can
contribute to improving.
BALTIMORE CITY
PUBLIC SCHOOLS
Teacher Involvement
17
 How are teachers providing feedback on measures?
 Educator Support and Evaluation Committee (ESEC) –
4 principals, 16 teachers, and an ED
 How can teachers learn more about the evaluation field
test?




Emails from City Schools
Webinars at ILT and faculty meetings
City Schools’ website, under “About Us” and click on
“Effectiveness”
BTU/City Schools information sessions beginning February 5
BALTIMORE CITY
PUBLIC SCHOOLS
What Will Teachers Experience in 2012-13?
18
During SY 2012-13: Teachers will experience two formal observations using
PBES with Framework 2.0 as the guiding document.
Fall 2012
Launch Field Test
via email blast
from Dr. Alonso,
and joint message
with Marietta
English.
Winter 2013
SLOs will be rolled
out gradually to all
schools beginning
Jan 2013.
Professional
responsibilities
component will be
outlined for
teachers in Jan
2013.
BALTIMORE CITY
Spring 2013
All teachers
receive updates
on VAM as well
as other
measures in the
teacher
evaluation field
test
All teachers in grades
3-12 have at least one
grade/ course/section
complete a student
survey.
Teachers will validate
their course rosters.
PUBLIC SCHOOLS
Summer 2013
Summer
2013: All
teachers
receive a
composite
rating
Questions?
19
BALTIMORE CITY
PUBLIC SCHOOLS
Like Me?
20
 Type of Leader?
 Product of City Schools?
 Years in City Schools?
 Years in Leadership?
BALTIMORE CITY
PUBLIC SCHOOLS
Teacher Evaluation Field Test Components
21
Teacher
Evaluation
Student Growth
Student
Learning
Objectives
(SLO)
School index
Professional
Responsibilities
Student Surveys
Observations/
Instructional
Framework
ValueAdded
Model
(VAM)
50%: Multiple measures of student growth
50%: Professional practice
• City Schools will propose weights after studying the results of the pilot and consulting with
our teachers and principals. No one measure can account for more than 35% of a teacher’s
evaluation.
BALTIMORE CITY
PUBLIC SCHOOLS
Session Objectives
22
 Understand the major components of a City Schools




Student Learning Objective (SLO),
Develop an understanding of the rationale behind
using SLOs as part of the teacher evaluation system,
Discuss the potential challenges and learning
opportunities that SLOs provide,
Consider the impact that school leaders and teacherleaders may have on the successful roll-out of SLOs,
Begin to consider potential candidates as SLO
Ambassadors
BALTIMORE CITY
PUBLIC SCHOOLS
City Schools: the Numbers
23

Approximately ~35-45 % of teachers* fall into the “tested” category
 In other words, 35-45% of our teachers teach subjects assessed by the
MSA, HSA, or Stanford-10

The other 55-65% of our teachers fall into the “non-tested” category

These teachers do not teach subjects directly assessed by the MSA, HSA, or
Stanford-10
City Schools' Teachers
Tested
Teachers
Non-Tested
Teachers
* Estimates based on SMS data, Fall 2011
BALTIMORE CITY
PUBLIC SCHOOLS
Think-Pair-Share
24
 Silently consider your response to the following
question:
What tools do we currently have to analyze student
growth to help our teachers in the non-tested grades
and subjects (NTGS)?
 Turn to the person beside you and discuss your
response
 Share what you heard your discussion partner say
BALTIMORE CITY
PUBLIC SCHOOLS
Independent Reading
25
 Read the first four pages of the SLO Brief provided in
your folder (10 min)
 While reading consider the following questions:



Why are SLOs proposed to be a part of the teacher evaluation
system?
In what ways do SLOs connect to other district initiatives (ie
SPPs, etc)?
Where do you see yourself having the most impact on SLOs?
BALTIMORE CITY
PUBLIC SCHOOLS
The Research
26
 “In Denver, students whose teachers crafted high
quality SLOs showed more than a year’s worth of
gain… during each year of the study at all three school
levels.”
 “In Charlotte-Mecklenburg, during the first phase of
implementation we found another strong correlation
between the targeted objective and student
achievement….”
Source: Community Training and Assistance Center, 2012
BALTIMORE CITY
PUBLIC SCHOOLS
Why SLOs?
27
 Rationale: Of the measures of student growth we
studied for non-tested areas, SLOs had the most
potential for improving practice





SLOs reinforce good teaching practices
What all of our best teachers are already doing anyway
Are not compliance-based
Data-driven instruction
Provides an anchor and theme for collaborative planning
BALTIMORE CITY
PUBLIC SCHOOLS
“If properly implemented, student
learning objectives help teachers
bring more science to their art,
strengthen instructional support to
the classrooms, and improve the
quality of the outcome.”
William J. Slotnik
Founder and Executive Director
Community Training and Assistance Center
BALTIMORE CITY
PUBLIC SCHOOLS
28
SLO Cycle
29
1
REVIEW STUDENT DATA
Individual teachers or teams of
teachers review their student data
before the school year begins (or
pre-test data after).
(August-September)
4
MID-YEAR REVIEW
Teachers and/or evaluators may
review teacher progress on SLO
targets mid-year. Changes in
assignment or class composition
may warrant target adjustment.
(February)
2
3
SLO APPROVAL
Evaluators review and approve
proposed SLOs (or request
revisions).
(October-November)
DRAFT SLOS
Individual teachers or teams of
teachers draft SLOs.
(September-October)
5
6
SUMMATIVE SCORING
SLO results are included among
measures of student growth in
summative ratings of teachers’
performance.
(June)
SLO SCORING
Evaluator reviews and scores
teachers’/teams of teachers’
progress on SLOs.
(May-June)
7
BALTIMORE CITY
REFLECTION
Teachers/teams of teachers and
evaluator discuss progress and
implications for next year’s
SLO(s)
(June, Summer)
PUBLIC SCHOOLS
Exploring an SLO
30
 At your table you will be given a model SLO
 Review the document silently for 5 minutes
 Respond to the four questions on the Four Square
document provided
 Take 10 minutes to discuss your responses with your
table
 Share out your group’s responses
BALTIMORE CITY
PUBLIC SCHOOLS
What is an SLO?
31
Learning goal for students
Specific to a group of students for that year
Measurable
Collaborative: set in consultation with colleagues and
evaluator(s)
 Organic to good teaching practice
 Popular: Many states and districts are building systems
of educator effectiveness that include SLOs as one of
multiple measures in teacher and principal
evaluations, including:






Georgia, Indiana, Louisiana, New York, Rhode Island
Austin, Charlotte-Mecklenburg, Denver, Houston, New Haven
BALTIMORE CITY
PUBLIC SCHOOLS
Which Teacher Will Complete SLOs?
32
 All teachers except those with students that take
the following assessments:

Stanford 10, grades 2



Teachers with students that take Stanford 10 in grade 1 will
write SLOs
MSA Reading and Math, grades 3-8
Courses that trigger HSA Algebra, Biology, English, and
Government in grades 8-12
BALTIMORE CITY
PUBLIC SCHOOLS
What Has Already Happened with SLOs?
33
Fall 2012
Progress to date:
-September email/video
from Dr. Alonso and
Marietta English
announcing field test
-SLO 101 (2 sessions) for
central office personnel
-SLO Tools Development
Workshop (November 1516)
-On-going policy
decisions by City Schools
leadership team
Winter 2013
SLO WRITING
WORKSHOP –
Content Leaders and
non-tested subject
teachers
BALTIMORE CITY
SLO bank and
guidebook are
created by January
2013. Principals
receive training in
January, with
materials they will
take back to their
buildings so that
SLOs can be rolled
out to all schools for
SLO field test in
February 2013.
PUBLIC SCHOOLS
SLO Implementation for Field Test
34
Following Principals’
training on 1/22-1/23:
Principals select 2 SLO
ambassadors at their schools
Jan
Feb
April: Ambassadors draft SLOs in
work groups; principals trained on
SLO approval (April 2nd); principals
approve ambassadors’ SLOs
March
Feb-March: Ambassadors attend halfday PD on SLO 101; content-based
workgroups trained on SLO writing
(Dates: Feb. 12th, March 19th)
BALTIMORE CITY
April
May
June-July: OAA
collects and
analyzes SLO
data and reports
to MSDE
June
June: SLOs scored by district staff.
PUBLIC SCHOOLS
July
Aug
August/September:
All non-tested teachers
trained on SLO 101 and
SLO Writing
What role will you play in SLO Development?
35
 Nominate 2 SLO Ambassadors
 Send 2 nominations to educatorfieldtest@bcps.k12.md.us by
January 31st
 Assist with coverage for these teachers for 3-5 hours a month
 Attend SLO approval training on April 2nd
 Approve SLOs that have been written by
Ambassadors–ensuring “ambitious, yet attainable”
BALTIMORE CITY
PUBLIC SCHOOLS
SLO Ambassadors: Role Description
36
There will be 2 SLO Ambassadors at each school. SLO
Ambassadors will:



Attend monthly trainings
Submit draft SLOs to principals for approval
In SY13-14, become school-based SLO experts and provide SLO
support in their buildings
BALTIMORE CITY
PUBLIC SCHOOLS
SLO Ambassadors: Selection Criteria
37
SLO Ambassadors Should:





Teach in a non-tested content area; i.e. subjects/grades not
covered by MSA/HSA exams
Demonstrate strength in instructional practices during classroom
observations using the Framework
Be able to commit 3-5 hours per month to attend trainings,
workgroup meetings, and complete outside work, as needed
Plan to return to same school for SY13-14, and facilitate and lead
SLO trainings to colleagues
Candidates may be Model Teachers
BALTIMORE CITY
PUBLIC SCHOOLS
SLO EXAMPLE:
Basic Information
38
Teacher: Nick Greer
School: Alice Pinderhughes High
Subject: 7th Grade Life Science
BALTIMORE CITY
Grade Level: 7
# of Students 110
Addressed by SLO:
Interval of X Full Year
Instruction: ☐ Semester
☐ Other:
PUBLIC SCHOOLS
SLO Writing: Assessment
39
 Assessment should:
 Answer the question: “How will I know if students have learned
the key content?” – it provides evidence
 Be aligned to standard(s) targeted
 Meet quality criteria that will be established by City Schools
 For teacher-created SLOs, describes the assessment
 Assessments can be standardized, but they also can be
performance tasks or portfolios that are judged against
a rubric
BALTIMORE CITY
PUBLIC SCHOOLS
SLO Writing: Standards Addressed by SLO
40
 Lists specific standards that will have been
met/mastered if the SLO is achieved
 Address critical content, skills or knowledge
necessary for advancement to future coursework


That is, if students do not master these standards, they will not be
able to progress to the next level
No need to list every single standard
BALTIMORE CITY
PUBLIC SCHOOLS
SLO Writing: Baseline Data
41
 Baseline data can be collected from:
 Pre-assessments
 In-class assignments
 Observation
 Include any additional student data or background
information used in setting the target (e.g. ESOL
status, SWD incidence)
BALTIMORE CITY
PUBLIC SCHOOLS
SLO Writing: Student Learning Objective
42
 States the specific target for student achievement by
the end of the instructional period
 Target should be ambitious, yet achievable, given
the students’ baseline data
BALTIMORE CITY
PUBLIC SCHOOLS
SLO Hazards
43
These common errors make an SLO unscorable:
 THE DOUBLE TARGET SLO:
“My students will get to [this level] by January and [this level] by May”
Or
“My students’ average score will be 84 and no score will be lower than 70”
 THE VAGUE SLO:
“My students will all be able to write an excellent explanatory essay by
May.”
 THE NARROW SLO:
“My students will be able to describe three ways that the 1787
Constitutional Convention influences our country today.”
BALTIMORE CITY
PUBLIC SCHOOLS
SLO Writing:
Encouraging Powerful Teacher Practices
44
BALTIMORE CITY
PUBLIC SCHOOLS
Turn and Talk
45
 How can these sections impact collaboration in your
school and across the district?
 What practices will you plan to use here to encourage
teachers to collaborate?
BALTIMORE CITY
PUBLIC SCHOOLS
Scoring Plan
46
Scoring Plan:
Insufficient
Attainment of
Target
(1 point):
The teacher has
demonstrated an
insufficient impact on
student learning by
falling far short of the
target.
Achievement
Range:
Partial Attainment
of Target (2
points):
Full Attainment of
Target
(3 points):
The teacher has
demonstrated some
impact on student
learning, but did not meet
the target.
The teacher has
demonstrated a
considerable impact on
student learning by
meeting the target.
Achievement
Range:
BALTIMORE CITY
Achievement
Range:
PUBLIC SCHOOLS
Exceptional
Attainment of
Target
(4 points):
The teacher has
demonstrated an
outstanding impact on
student learning by
surpassing the target by a
meaningful margin.
Achievement
Range:
Debrief and Questions
47
 How would you explain what an SLO is to your
teachers? Why are we using them?
 What systems will you need in place to ensure that
your teachers are successful with SLOs? How can the
Ambassadors best support quality SLO work?
 What outstanding questions do you still have regarding
SLOs?
BALTIMORE CITY
PUBLIC SCHOOLS
Next Steps
48
 SLO 101 for School Leaders (Today)
 Select 2 SLO ambassadors (By Jan 31)
 Ambassadors attend half-day PD on SLO 101; content-based




workgroups trained on SLO writing (Dates: Feb. 12th, March
19th)
Ambassadors draft SLOs in work groups; principals trained
on SLO approval (April 2nd);
SLOs scored by district staff (June)
OAA collects and analyzes SLO data and reports to MSDE
(June-July)
All non-tested teachers trained on SLO 101 and SLO Writing
(August/September)
BALTIMORE CITY
PUBLIC SCHOOLS
Support and Resources
49
 For more information on Student Learning Objectives,
please contact us at educatorfieldtest@bcps.k12.md.us
BALTIMORE CITY
PUBLIC SCHOOLS
50
LUNCH!
School Leader Development
January 22/23, 2013
BALTIMORE CITY
PUBLIC SCHOOLS
Teacher Evaluation Field Test Components
51
Teacher
Evaluation
Student Growth
Student
Learning
Objectives
(SLO)
School index
Professional
Responsibilities
Student Surveys
Observations/
Instructional
Framework
ValueAdded
Model
(VAM)
50%: Multiple measures of student growth
50%: Professional practice
• City Schools will propose weights after studying the results of the pilot and consulting with
our teachers and principals. No one measure can account for more than 35% of a teacher’s
evaluation.
BALTIMORE CITY
PUBLIC SCHOOLS
Session Objectives
52
 Develop an understanding of the rationale behind
using value-added as part of the teacher evaluation
system
 Gain a deeper understanding of how value-added
models work in general and the details of City School’s
value-added model
 Discuss the potential challenges and learning
opportunities that value-added models (VAM) provide
BALTIMORE CITY
PUBLIC SCHOOLS
Why Use Value-Added Analysis?
53
 Teachers can and should contribute to student
learning from August to June.
 This is true whether the student comes in
above grade, at grade or below grade.
 Value added analysis shows how well teachers
do with all students, regardless of their initial
starting points – unlike simple proficiency
scores.
BALTIMORE CITY
PUBLIC SCHOOLS
VAM Research: Teachers Matter
54
Value-added measures of teacher effectiveness are
positively correlated with principals’ subjective assessments
of teachers. 1
 VAM measures are useful for differentiating between the
highest and lowest performing teachers.2
 Teachers with high value-added scores make significant
contributions to their students’ learning, regardless of the
student population.3

1 Jacob
and Lefgren (2005); 2 Rivkin,Hanushek, & Kain (2005); 3 Kane et. al (2008)
BALTIMORE CITY
PUBLIC SCHOOLS
VAM Research: Teachers Matter
55
Researchers have found evidence, using value-added
models, showing that:



Highly effective teachers can have an immediate impact on
student achievement.
Students assigned to highly effective teachers are more likely to
go to college, earn higher incomes and less likely to become
teenage parents.
Having a highly effective teacher for one year increases a child’s
cumulative lifetime income by $50K.
Source: Chetty, Raj et al. The Long Term Impacts of Teachers: Teacher Value Added
and Student Outcomes in Adulthood. December 2011. Available at
http://obs.rc.fas.harvard.edu/chetty/value_added.html.
BALTIMORE CITY
PUBLIC SCHOOLS
Understanding Value-Added
56
 The following video will walk us through the difference
between the following ways of looking at student test
scores



Achievement Model
Simple Growth or Gain Model
Value-Added Model
 If you would like to use this video with your teachers, it
can be found at
http://varc.wceruw.org/tutorials/Oak/index.htm
BALTIMORE CITY
PUBLIC SCHOOLS
How does City Schools’ VAM work?
57
 Value-Added is defined as the teacher’s contribution to




student growth.
The model calculates a predicted test score for a student
in a given grade and subject.
The predicted score is based on the student’s prior
academic achievement (and other factors).
Then, we compare the student’s predicted score to the
actual score in the grade and subject.
The difference between the predicted score and the actual
score is called the teacher’s value-added score.
BALTIMORE CITY
PUBLIC SCHOOLS
How does City Schools’ VAM work?
Graphic Representation
58
Current
Test
Score
Value
Added
Score
Student
Predicted
Gain
Baseline (Previous Year’s Score)
Student Student
Student
Value =
Predicted
Actual
Added
Score
Score
Score
-
BALTIMORE CITY
PUBLIC SCHOOLS
How does City Schools’ VAM work?
Aggregating Teacher VAM Scores
59
Current
Test
Score
Student A’s
VAM Score
Baseline (Previous Year’s Score)
BALTIMORE CITY
PUBLIC SCHOOLS
How does City Schools’ VAM work?
Aggregating Teacher VAM Scores
60
Current
Test
Score
Student B’s
VAM Score
Student A’s
VAM Score
Baseline (Previous Year’s Score)
BALTIMORE CITY
PUBLIC SCHOOLS
How does City Schools’ VAM work?
Aggregating Teacher VAM Scores
61
Current
Test
Score
Student B’s
VAM Score
Student A’s
VAM Score
Baseline (Previous Year’s Score)
Teacher
Value-Added
Score
=
Student A’s
VAM Score
BALTIMORE CITY
+
Student B’s
VAM Score
PUBLIC SCHOOLS
+…
Explaining Value-Added
62
 Take 2 minutes to think through the following
question:

How would you explain value-added to your teachers?
 Turn to a partner and try out this explanation (both
people try it!)
BALTIMORE CITY
PUBLIC SCHOOLS
Which Teachers Receive VAM Scores?
63
 Teachers with students that take the following
assessments:



Stanford 10, grades 2
MSA Reading and Math, grades 3-8
Courses that trigger HSA Algebra, Biology, English, and
Government in grades 8-12
BALTIMORE CITY
PUBLIC SCHOOLS
Current Control Variables
64
Student-Level Controls
School-Level Controls
1.
Prior Achievement (2 years of prior scores)
1.
Percent of students in school receiving free/reduced
price lunch
2.
Student’s prior year number of days absent
2.
Percent of students in the school classified as special
education students
3.
ELL Status
3.
Percent of students in school out of age level
4.
Mobility (number of transitions)
4.
Average number of special education service hours
per student
5.
Difference from modal age in grade
6.
Repeater status (i.e. did student repeat a
prior grade?)
7.
Skipper status (i.e. did student skip a prior
grade?)
BALTIMORE CITY
PUBLIC SCHOOLS
Roster Validation
65
 Roster validation allows us to assign the correct
students to teachers
 Without validation, it is possible for students that are
incorrectly on a teacher’s roster to count towards his or
her value-added or SLO score
 Teacher verification is planned for late April, with
principal review and approval scheduled for early May
BALTIMORE CITY
PUBLIC SCHOOLS
Support and Resources
66
 For more information on value-added, please contact
us at educatorfieldtest@bcps.k12.md.us
BALTIMORE CITY
PUBLIC SCHOOLS
Teacher Evaluation Field Test Components
67
Teacher
Evaluation
Student Growth
Student
Learning
Objectives
(SLO)
School index
Professional
Responsibilities
Student Surveys
Observations/
Instructional
Framework
ValueAdded
Model
(VAM)
50%: Multiple measures of student growth
50%: Professional practice
• City Schools will propose weights after studying the results of the pilot and consulting with
our teachers and principals. No one measure can account for more than 35% of a teacher’s
evaluation.
BALTIMORE CITY
PUBLIC SCHOOLS
Session Objectives
68
 Understand City Schools’ Professional Responsibilities
Checklist and how it fits in with the teacher evaluation
field test
 Discuss ways to capture and record Professional
Responsibilities data during the field test
BALTIMORE CITY
PUBLIC SCHOOLS
What are “professional responsibilities”?
69
 Definition: Professional Responsibilities are the core,
non-instructional job functions of teachers.


Examples: showing up to work, punctuality, turning in grades on
time, attending staff meetings, etc.
Professional responsibilities are not unique to teaching. These
expectations are consistent across all, or most, professions.
BALTIMORE CITY
PUBLIC SCHOOLS
Why measure professional responsibilities?
70
 Three-Pronged Rationale:
1.
Professional responsibilities are threshold skills to effective
teaching.
2.
Professional responsibilities are included in most other teaching
frameworks (see e.g. the Danielson Framework, Domain 4 of
PBES).
3.
City Schools’ staff should model the same expectations we have
of our students.
BALTIMORE CITY
PUBLIC SCHOOLS
What does the checklist look like?
71
 To create our Checklist, we reviewed professionalism
measures in other school districts, generated a list of
possible criteria (including PBES), and shared drafts
with teachers for feedback.
 The Checklist is comprised of four categories:
1.
Attendance
2.
Arrival
3.
Compliance with District and School Policies
4.
Testing Integrity
BALTIMORE CITY
PUBLIC SCHOOLS
Category 1: Attendance
72
Category
Meets Expectations
Does Not Meet
Expectations
Attendance
Teacher has zero or 1 unexcused absence.
Teacher has 2 or more
unexcused absences.
The term “unexcused absence” should be
interpreted in light of the BTU contract and City
Schools’ Attendance Reliability and Analysis
Program. In other words, an unexcused absence
is an absence that is not eligible for leave as
specified in Art. XV in the BTU contract OR an
absence (or occasion) in excess of seven previous
periods of absence.
BALTIMORE CITY
PUBLIC SCHOOLS
Category 2: Arrival
73
Category
Meets Expectations
Does Not Meet
Expectations
Arrival
Teacher has no more than 2 unexcused late
arrivals.
Teacher has more than 2
unexcused late arrivals, AND
fails to make improve
his/her punctuality over the
rest of the school year.
The term “unexcused late arrival” should be
interpreted in light of the BTU contact and City
Schools’ Attendance Reliability and Analysis
Program. In other words, an unexcused late
arrival occurs when a teacher is not at his/her
duty post prior to the start of the work day, prior
notification notwithstanding.
At the discretion of the supervisor, a teacher may
meet expectations if, after two unexcused late
arrivals and a discussion with his/her supervisor,
he/she improves his/her punctuality over the rest
of the school year.
BALTIMORE CITY
PUBLIC SCHOOLS
Category 3: Compliance with District and School
Policies
74
Category
Meets Expectations
Does Not Meet Expectations
Compliance
with District
and School
Policies
Teacher meets deadlines for submitting
progress reports, grades and attendance
data, AND
Teacher demonstrates a pattern of
missing deadlines for submitting
progress reports, grades and
attendance data, OR
Teacher always or usually follows all other
school and district policies and procedures
(e.g. administer benchmarks, take daily
attendance, etc.), AND
Teacher attends all mandatory meetings,
conferences and trainings during the
regular school day.
At the discretion of supervisor, a teacher
may meet expectations if, after failure to
follow district or school policies and a
discussion with his/her supervisor, he/she
improves his/her performance over the rest
of the school year.
BALTIMORE CITY
Teacher demonstrates a pattern of
failing to follow school and district
policies and procedures, OR
Teacher demonstrates a pattern of
missing mandatory meetings,
conferences or trainings during the
school day, AND
Does not improve his/her
compliance with school or district
polices over the rest of the school
year.
PUBLIC SCHOOLS
Category 4: Testing Integrity
75
Category
Meets Expectations
Does Not Meet Expectations
Testing
Integrity
Teacher strictly complies with all district
policies regarding testing security, AND
Teacher does not strictly comply
with all district policies regarding
testing security, OR
Teacher promptly discloses to school or
district leadership any incidents of cheating Teacher fails to disclose incidents
or testing impropriety.
of cheating or testing impropriety
to school or district leadership.
BALTIMORE CITY
PUBLIC SCHOOLS
Professional Responsibilities Checklist
Implementation
76
June: OAA will provide
school leaders with a
template for reporting
Checklist data. Data
templates must be returned
to OAA by June 21, 2012
Following Principals’ training
on 1/22-1/23: Share Checklist with
your staff at an upcoming faculty
meeting. Identify any school specific
priorities in the compliance category.
Jan
Feb
March
April
May
June
Now through June: Track data using the Checklist
Tracking Sheet. Update teachers on their performance,
especially if they are at risk of not meeting expectations
in a category.
BALTIMORE CITY
PUBLIC SCHOOLS
July
Aug
Discussion
77
 Are we setting the right targets for meeting
expectations?
 What best practices have you developed at your school
for tracking this kind of information?
 How can OAA and other central office staff support you
in tracking and reporting this data for the field test?
BALTIMORE CITY
PUBLIC SCHOOLS
Support and Resources
78
 For more information on the Professional
Responsibilities Checklist, please contact us at
educatorfieldtest@bcps.k12.md.us
BALTIMORE CITY
PUBLIC SCHOOLS
Teacher Evaluation Field Test Components
79
Teacher
Evaluation
Student Growth
Student
Learning
Objectives
(SLO)
School index
Professional
Responsibilities
Student Surveys
Observations/
Instructional
Framework
ValueAdded
Model
(VAM)
50%: Multiple measures of student growth
50%: Professional practice
• City Schools will propose weights after studying the results of the pilot and consulting with
our teachers and principals. No one measure can account for more than 35% of a teacher’s
evaluation.
BALTIMORE CITY
PUBLIC SCHOOLS
Session Objectives
80
 Develop an understanding of the rationale behind
using student surveys as part of the teacher evaluation
system
 Discuss the development of the survey and timeline for
implementation of student surveys in the field test
 Obtain feedback on the proposed student survey items
 Gain familiarity with interpretation of student survey
teacher reports
BALTIMORE CITY
PUBLIC SCHOOLS
Why Student Surveys?
81
 Students have the most contact with teachers and
are the direct consumers of a teacher’s service –
they are our ultimate customer
 Multiple measures provide more information and
a more robust definition of teacher effectiveness
 Student surveys provide data that can inform
teacher practice
BALTIMORE CITY
PUBLIC SCHOOLS
Review of Research on Student Surveys
82
• Teachers rated higher by students in instructional
effectiveness align with students achieving at higher levels in
that teacher’s class (MET Project, 2010a; Balch, 2012; Crow,
2011; Kyriakides, 2005).
• Student ratings are a valid and reliable data source (MET
Project, 2010a; Peterson et al., 2000; Worrell & Kuterbach,
2001).
• Elementary and secondary students are as reliable as older,
adult raters in rating teaching behaviors (Follman, 1992,
1995; Worrell & Kuterbach, 2001).
BALTIMORE CITY
PUBLIC SCHOOLS
MET Project findings
83
 When student surveys
are combined with
rubrics, the difference
between the bottom and
top teachers increases
from 2.6 month to 4.8
months of learning
BALTIMORE CITY
PUBLIC SCHOOLS
Aligning Survey Items
84
•How we aligned the survey items
•Start with appropriate Instructional Framework TEACH
behavior descriptions for Highly Effective and Effective
•Match to pre-validated items
•Write new items if necessary
BALTIMORE CITY
PUBLIC SCHOOLS
What Items Are on the Survey?
85
 In your folder is a copy of the student survey that will
be field tested
 Take 5-7 minutes to go through the items individually
and then discuss with others at your table



High School Principals – Items 1-17
Middle School Principals – Items 18-35
Elementary School Principals – items 36-51
 Guiding questions for review:
 What improvements could you make on any of the items?
 What items will your teachers be concerned with?
BALTIMORE CITY
PUBLIC SCHOOLS
Screening procedures
86
 Screening procedures were used to identify and
eliminate invalid survey responses
 First procedure identifies answers that have a
minimum difference from a teacher's overall
average for a question
 Student response to the question “I was being
honest when taking this survey”1
Surveys
with answers of either Disagree or
Strongly Disagree were eliminated.
1Reniscow
et al., 2001; GAO, 1993
BALTIMORE CITY
PUBLIC SCHOOLS
Student Survey Implementation
87
June-July: OAA
analyzes field test
data and reports
to MSDE
Following Principals’ training
on 1/22-1/23: Test coordinators will
receive information on survey
administration
Jan
Feb
March
March: Student surveys
administered in schools
from March 1st to March
22nd (Administered at the
same time as student
climate surveys)1
1P
April
May
June
July
Late Spring: Teacher will receive
individual student feedback reports
ARENT AND TEACHER CLIMATE SURVEYS WINDOW IS FEBRUARY 21
BALTIMORE CITY
Aug
PUBLIC SCHOOLS
ST
T O M A R C H 2 2 ND
Interpreting a Student Survey Report
88
 Take 2-3 minutes to think through the following
questions:



How do the category scores of this teacher compare to the school
and district average?
What are some strategies that you might tell this teacher for each
of the ‘Areas for Improvement’?
How else might you be able to use the information from this
report to assist in the professional development of a teacher?
 With a partner, discuss your answers to these questions
for 4-5 minutes
BALTIMORE CITY
PUBLIC SCHOOLS
Which Survey?
89
 School (Climate)
Surveys – Students
 Student Survey of
Teacher Practice

Asks about individual
teachers
BALTIMORE CITY

Asks about the climate of
the whole school
 School (Climate)
Surveys – Parents
 School (Climate)
Surveys – Teachers
PUBLIC SCHOOLS
What is the Role of the Principal?
90
 Provide feedback on student survey questions (today)
 Assist with survey administration (will occur at the
same time as school survey for students)




Administration should take place in the classes of the teacher
being evaluated
Teachers should switch classrooms
Need to survey the same class periods for high school/middle
school (i.e. period 2 and 3), only one survey per teacher in
elementary school
Further details provided to test coordinators Jan 29/30th
 Assist teachers with interpretation of reports
BALTIMORE CITY
PUBLIC SCHOOLS
Support and Resources
91
 For more information on student surveys, please
contact us at educatorfieldtest@bcps.k12.md.us
BALTIMORE CITY
PUBLIC SCHOOLS
Close Out Feedback
92
BALTIMORE CITY
PUBLIC SCHOOLS