Holistic Rater Training
February 14, 2013
Overview of the Rating
Process
• Each campus will identify ELLs on their campus and the
principal will designate raters to assess students
• All raters must complete the appropriate rater training
• Raters gather writing samples for ELLs in grades 2–12 to
include in each student’s writing collection
• The rater is responsible for assessing the English language
proficiency of students and assign them a rating
• The rater will record the students’ ratings and related info on
the TELPAS Rating Roster form
• Student’s ratings will be reviewed by the campus principal
Overview of the Rating
Process
• This year SMSD will complete a “Drop Everything and
Write Day”
• The CTC will ensure the writing collections are kept in the
students’ permanent record files for two years
• Campuses must maintain a file of the rating rosters
Holistically Rated Components of
TELPAS
Grade Levels
Language Domains
K–1
Observational assessments in listening, speaking,
reading, and writing
2–12
Observational assessments in listening and
speaking; writing collections for writing
Online multiple-choice reading tests
are used to assess the reading domain
for grades 2–12.
TEA Division
4
Holistic Rating Training
Requirements
Rater Credentials
Each teacher selected to rate an ELL must
1) have the student in class
2) be knowledgeable about the student’s ability
to use English in instructional and informal
settings
3) hold valid education credentials such as a
teacher certificate or permit
4) be appropriately trained, as required by TEA
New and Returning Rater
Definitions for K–1 and 2–12
•The training webpage informs raters that districts may sometimes require a
returning rater to complete new-rater training and to consult with their
testing coordinator if they need clarification.
TEA 7
Training Requirements for
K–1 and 2–12
• In the Fall ELPS-TELPAS professional development
for new teachers and new raters – foundational
(awareness) training
• In the Spring (required and in addition to
administration procedures training)
 New raters – complete online basic training course
and online calibration activities
 Returning raters – complete online calibration
activities
Training Flowchart
Fall ELPS-TELPAS Foundational (Awareness) Training
For teachers who will be trained as new TELPAS raters in the spring if they lack this
foundation
Spring TELPAS Administration Procedures Training
As a key part of this training, information from the TELPAS Manual for Raters and Test
Administrators is reviewed to prepare raters to proceed with online holistic rating training
New Raters
Returning Raters
Online Basic Training Course
Online Calibration
(Sets 1 and 2*)
Online Calibration
(Sets 1 and 2*)
If not calibrated: Supplemental Holistic Rating Training
*Set 2 required only if
not successful on Set 1
Final Online Calibration (Set 3)
T
Two Types of Training
•
Online basic training course
This course is for new raters. It provides instruction on using the
PLD rubrics and gives raters practice rating students in each
language domain. There are separate courses for K–1 and 2–12.
•
Online calibration
This is for all raters. Raters use the PLDs to rate students in each
language domain. Raters have three opportunities to calibrate on
assigned grade cluster.
Grade Clusters of Training and
Calibration
• Raters must know their assigned grade cluster to select the appropriate
online training.
• Raters should consult with their campus testing coordinator if they are
unsure of their assigned cluster.
Grade Clusters
Grades K–1
Grade 2
Grades 3–5
Grades 6–8
Grades 9–12
Raters with Students in Multiple Grade Clusters
within Grades 2–12
– Raters are required to train and calibrate in the cluster in which they have
the most ELLs.
– As a best practice, these raters should also review online basic training
course practice activities in the additional cluster(s).
Example: A grade 2–5 ESL teacher has most of her ELLs in grade 3. She must complete
training and calibration in grades 3–5. As a best practice, she should review the online
practice activities for grade 2 to check her readiness to apply the rubrics appropriately .
Raters with Students in Multiple Grade
Clusters that Include Grades K–1
– Because of differences in the instructional content and rating rubrics,
these raters must complete training and calibration for K–1 and at
least one other cluster in 2–12.
– Raters with more than one additional cluster should train in the
cluster in which they have the most ELLs.
– As a best practice, these raters should also review online basic
training course practice activities in the additional cluster(s).
Example: A grade 1–3 ESL teacher has most of his ELLs in grade 2.
He must complete training and calibration in grades K–1 and 2. As a
best practice, he should review the online practice activities for grades
3–5 to check his readiness to apply the rubrics appropriately for his
3rd grade students.
When and where do raters
take the online training?
• Raters may either complete training and calibration
during school hours, after school, or on weekends.
• The online training system allows raters to access the
training from any computer that meets the minimum
system requirements.
Why is calibration necessary?
• Calibration helps ensure that raters have adequate training,
including ample practice and feedback, before they assess
their students in the spring.
• Calibration ensures that raters clear their heads and
consider only the elements of student performance included
in the PLDs.
• Calibration supports assessment validity and reliability and
is an important part of holistically scored assessment
processes.
Preparing for Calibration Sets
• New raters must complete the online basic
training course before beginning calibration.
• Returning raters have the option to review the
online basic training course (which includes rating
activities) before beginning calibration.
Other Things to Know About
Calibration
• Raters affirm online that they will keep the contents of the calibration
sets secure and confidential.
• Calibration activities are taken from a bank and randomized. Trainees
will rate different sets of students.
• Raters can work at their own pace, go back and review students, and
change ratings as they work.
• Raters can exit and return later to finish. They click a “submit” button
when they are finished with a set.
• After completing a calibration set, raters immediately see results.
Results show the rating assigned by the rater and the correct rating.
• Raters see annotations explaining the ratings. Raters should use the
annotations to go back and review any incorrectly rated students.
Is Calibration a Test?
No, it is a training method that ensures
that raters have enough guidance,
practice, and support to assess
students consistently and accurately.
Will raters be able to refer to any
resources during calibration activities?
Yes, raters should use their rating rubrics (PLDs)
and refer to, as needed, information from the:
– online basic training course
– TELPAS Manual for Raters and Test Administrators
– Educator Guide to TELPAS
Supplemental Holistic
Training
– Raters not successful after sets 1 and 2 must receive
supplemental training.
– The rater will meet with a district/campus-appointed
supplemental support provider.
– After the rater has received supplemental training, he or she
will be able to access the third and final calibration set.
Recap of Calibration Process
• There are 3 sets of 10 students.
• Raters who calibrate on set 1 are done.
• Raters who don’t calibrate on set 1 go on to set
2. Raters who calibrate on set 2 are done.
• Raters who don’t calibrate on set 2 receive
supplemental training.
• Raters attempt third and final calibration set.
• Raters who calibrate on set 3 are done.
TEA 23
What happens if a rater is
unsuccessful in calibrating?
– If the individual is needed to serve as a rater, the district
must implement rater support procedures to ensure that
the rater’s students are evaluated consistent with the
rating rubrics.
Course and Calibration Certificates
• Raters who take the basic training course get a
certificate from the online TrainingCenter after
completing the course components.
• Raters will receive a certificate of successful
calibration when they calibrate.
• Raters must submit their calibration certificates to
their Campus Test Coordinator
Assembling and Verifying
Assemble and Verify Writing
Collections
• Raters of grades 2-12 students, must assemble student writing
collections and base their ratings on the writing in the
collections.
• Each writing collection must contain at least 5 writing samples
and must include at least one narrative writing sample about a
past event and at least 2 academic writing samples from math,
science, or social studies.
• Writing assigned on or after February 18, 2013, may be
considered for the writing collections. Writing prior to February
18, 2013 is not eligible.
• Writing samples may continue to be gathered until the time
raters are required to turn in students’ ratings to the CTC.
• All writing should reflect a student’s current proficiency level.
Assemble and Verify Writing
Collections cont.
• All writing assignments must include the student’s name and
date.
• The cover sheet and verification checklist located in Appendix
must be completed and stapled to each student’s writing
collection.
• Photocopies of classroom writing assignments may be included
in the writing collections as long as all copied pages are clear
and legible.
• Writing samples may be typed provided that spell check and
grammar check are disabled.
Types of Good Writing
Collections
• In all writing assignments, encourage
students to take their time and write as much
detail as they can.
• In each collection, include some writing tasks
that will allow students to show what they can
do.
• Also include some writing tasks that stretch
and push students’ limits so you can see the
full extent of their language abilities and any
second language acquisition they still need to
develop.
What Not to Include in a
Collection
• Papers containing language copied directly
from a textbook, lesson or written source
• Papers that require the student to rely on the
dictionary
• Papers that have teacher’s corrections on
them
• Papers that have been edited by peers,
parents or teachers
• Papers where the student writes primarily in
their native language
Four Language Domains
• Listening – The ability to understand spoken language,
comprehend and extract information, and follow social and
instructional discourse through which info is provided
• Speaking – The ability to use spoken language
appropriately and effectively in learning activities and social
interactions
• Reading – The ability to comprehend and interpret written
text at the grade-appropriate level
• Writing – The ability to produce written text with content and
format to fulfill grade-appropriate classroom assignments
Procedures for Rating
Students
• Review the holistic rating process before beginning
• Always refer to the PLDs when rating students; you may
photocopy the PLDs if necessary
• Begin with the students whose English Language proficiency
levels are clearest to you
• Your ratings should be based on observations of the student in
a variety of social and academic settings over time
• Evaluate the current level of English language proficiency
• The proficiency rating you designate should represent the level
at which a student performs most consistently
• Collaborate with other teachers and school personnel regarding
student’s ratings
• Complete all ratings for one student prior to proceeding to the
next student
Procedures for Rating
Students
• Compare each student to the PLDs and do not let the
proficiency levels of other ELLs you teach influence your ratings
• With the exception of grades 2-12 writing, you should be able to
determine a student’s rating by simply reflecting on the PLDs
and the student’s current ability to understand and use English
in social and academic settings.
• Remember – academic language proficiency is not the same as
academic achievement.
Review ELPS-TELPAS
Proficiency Level Descriptors
Creating and Managing
TELPAS Rating Entry Groups
Drop Everything and Write
(DEW Day)
Questions!!!