Florida Interim Assessment
Item Bank and Test Platform
(FL IBTP)
Scoring Overview
Agenda
• Overview of scoring process
– Identifying and using anchor sets
– Identifying and applying calibration sets
– Scoring to a rubric
– Working with exemplars
– Avoiding bias in scoring
• Panel discussion
2
Approach to Scoring
•
Score supportively.
•
Look for what is correct and done well in the
response, as opposed to what is incorrect or done
poorly.
•
View each response as a snapshot of the student’s
ability to respond to the item; look at the response
strictly for what it is.
3
The Scoring Process
4
Anchor Sets
• The anchor set is an item-specific guide to help the
scorer identify the correct score point for each
response.
• Determine which group of specialists in the district
will identify the anchor sets.
• Manage logistics of anchor set development: How
many responses will be in each set? How many
copies are required? How will security be
maintained? Who will store the anchor sets?
5
Anchor Sets Diagram
6
Determining Anchor Sets
• Establishing anchor sets is often called range-finding.
The goal is to
– Select from available student responses examples
that best match the description of each score point in
the rubric.
– Demonstrate a range of student performance within
each score point. Scorers will compare each live
paper to the corresponding anchor set responses to
determine the best-fit score for the live response.
– Maintain historical scoring standards for similar items.
7
Using Anchor Sets
• Use anchor sets to train scorers, explaining how
each anchor response illustrates that score point in
the rubric.
• Encourage scorers to compare each response to
the anchor responses.
8
Calibration Sets
• Like anchor sets, calibration sets are groups of
papers for which the scores have been agreed upon
previously by a select group of specialists.
• Responses in a calibration set are clear
representatives of one score point in the rubric.
• Calibration sets level-set teams of scorers and
ensure inter-rater reliability throughout the scoring
process.
9
Calibration Sets Diagram
10
Using Calibration Sets
• Ongoing training during scoring
• Distributed to scorers without being identified as
anything but additional responses to score
• Correct a drift in scoring
• Clarify or provide further training
11
Scoring to a Rubric
•
•
•
The rubric is a set of predefined criteria that
delineate the important characteristics on which
responses will be scored.
The rubric allows each scorer to remove subjectivity
from his or her scoring and reduces the variation in
scores created by individual biases and personal
preferences.
Traits not mentioned by the rubric must not affect the
score a response receives.
12
Applying a Rubric
• Assign a response to the score point where that
response best fits. It may not fit perfectly into any
one score point.
• Focus on the broad fit of each response to a score
point. Do not get caught up in the small details.
• Remember that many different types of responses
may be grouped into the same score point.
13
Using Exemplars
• The rubric allows for a general or generic description
of the level of understanding shown at each score
point. The exemplar defines those levels of
understanding in an item-specific manner.
• The exemplar is a representative response chosen
to clarify the rubric, similar to an anchor response.
• The exemplar will not show all possible correct
answers.
• The top score responses need not be perfect. Each
score point, even the top score, shows a range of
understanding.
14
Bias
• Guard against biases, assumptions, preferences,
and pre-conceived expectations when scoring.
• Look holistically for the parts of the response that
can receive credit.
15
Assumptions
• Student identity
•
•
•
•
•
Gender
Socioeconomic class
Regional location
Student personality
Sensitive or emotional content
16
Expectations
• Handwriting
• Response length
• Regionalism
• Five-paragraph essay
• Alternate approaches
• Subject-matter expertise
17
District Experiences with Scoring
18
Panel Discussion
• District sharing
• Project overview
– Managing scoring logistics
– Developing scoring policies
– Developing and maintaining protocols
• Questions from the audience
19
Thank You
NOTE: These materials were developed under a grant from the U.S. Department of Education.
However, the contents do not necessarily represent the policy of the U.S. Department of Education,
and you should not assume endorsement by the Federal Government.
Copyright © 2012 Florida Department of Education. All rights reserved.
20