Rangefinding with Committee
Prior to committee
1. OSPI and DRC work to determine an order and agenda for the items being reviewed.
Typically items are grouped by PE starting with those in the first area of emphasis of the lowest
grade level the committee is reviewing - start with 3.1 items in a grade 3-5 committee.
2. ETS provides OSPI with updated rubrics based on edits made during the most recent rounds
of review.
3. OSPI reviews the items and rubrics. OSPI may review and/or print rubrics from similar items
from previous years to compare and guide the rubric development for this year’s items.
4. OSPI support staff creates item-specific folders for all short-answer items with labels that
include:

the item number (usually the OSPI number but can be the ETS item number)

Performance Expectation(s)

item type (SA)

space to enter the date the item is range found

For pilot Rangefinding, space to enter the item name assigned at pilot Rangefinding.

For operational Rangefinding, the name of the item assigned at pilot Rangefinding.
5. For operational Rangefinding, OSPI pulls and reviews the pilot Rangefinding folders.
Information regarding the pilot year is in the change log or can be obtained through the OSPI
psychometrics associate.
6. OSPI determines invitees and works with DRC to extend invitations. OSPI sends invitees an
initial invitation and, when initial invitations are accepted, DRC sends official invitation which
includes travel, hotel, reimbursement, etc. information.
7. OSPI support staff creates a committee folder to hold all the signature pages and other
committee-related paperwork. The label for this folder should include the grade level(s), the
month, and year and should indicate whether it is pilot or operational Rangefinding.
Committee-related paperwork includes a list of teacher participants, clock hour forms,
evaluations, etc.
1
Rangefinding with Committee
Materials for committee work
DRC provides:

hard copies of training sets, one for each participant, ETS staff, DRC staff, and OSPI staff.
o For pilot Rangefinding, the set will be numbered and generally ordered from
highest score to lowest score.
o For operational Rangefinding, the set will include an anchor set, a practice set,
and two qualifying sets. The anchor and practice sets will have scores from pilot
Rangefinding.

name tents for each participant, ETS staff, and OSPI staff.

item-approval sign-off sheets.

one hard copy of the Test and Item Specifications for each participant.

pencils, sticky notes, pens, erasers, scratch paper, etc.

one projector per committee.

one set of score cards (0, 1, and 2) per participant.
ETS brings:

electronic versions of all the updated rubrics.

one laptop per committee.
OSPI brings:

for pilot Rangefinding, one folder per short-answer item with item-specific information
on the label.

for operational Rangefinding, the pilot Rangefinding folder that contains the pilot
training sets with scores and committee notes.

a folder for the forms that documents the committee’s work.

calculators for each participant.

own copy of Test and Item Specifications.

a thumb drive for the electronic versions of the final rubrics.
2
Rangefinding with Committee
Opening (Grade-Level Committees)
1. Ask everyone to turn off cell phones. Put your own on silent/vibrate.
2. Welcome and Introductions:

Thank everyone for taking time to participate and for the work they about to do.

Introductions:
o OSPI Content Specialists
o Teacher participants
o DRC Scoring Directors
o ETS Content Specialists

DRC goes over meeting paperwork, i.e., confidentiality agreement, travel, taxes,
substitute reimbursement, evaluation.

OSPI goes over contract information.
3. Purposes and roles each person has in Rangefinding.
Teacher participants serve as advisory committee to:

Assure all items align with areas of emphasis and performance expectations.

Set score ranges for all open-ended items.

Clarify task-specific rubrics that will be used to score all papers.

Generalize classroom experience to objectively score all papers; caution teachers
against connecting student work reviewed to a specific student or group of students in
their classroom.

Think about classroom applications for WA state teachers: common student
misconceptions, ways students can show clear evidence of understanding, etc.
DRC Scoring Director(s):

Present items, rubrics, and papers.

Lead discussions to clarify scoring.

Offer advice on interpretations of work presented in the training set and the wording of
rubrics to enhance scoring reliability.
ETS Content Specialist(s):

Keep electronic record of rubric changes and produce final rubrics.
3
Rangefinding with Committee

Offer advice on interpretations of work presented in the training set and the wording of
rubrics to enhance scoring reliability.
OSPI Content Specialist(s):

Participate in discussions and record advice given.

For pilot Rangefinding, record suggested edits in red on hard-copy of item and brings
them forward to Data Review; it is rare that items will need edits and rarer still that the
psychometricians will allow items to be edited at Data Review.

For operational Rangefinding, no edits can be made to the items. Edits may be made to
the rubric.

Capture committee feedback for updates to Test and Item Specifications, vocabulary
lists, “Lessons Learned”, “Updates…” documents, etc.

Make final decisions about items along with OSPI Mathematics Assessment Team, the
Assistant Superintendent, and Superintendent.
o At pilot Rangefinding, and on rare occasions, a pilot item can be marked as “Do
not score” if there are significant issues found in the item based on typical
papers, such as getting the right answer for the wrong reason or a pervasive
misinterpretation that prevents students from demonstrating the desired skill(s).
4. Give a general timeline for each day.

Begin at 8 a.m.

Two breaks for approximately 15 minutes, one morning and one afternoon.

Lunch (45 minutes or 1 hour, depending on pace and work load).

End at 5 p.m., or whenever an item provides a good stopping point. It may be necessary
to work late one night.

Ask for participant volunteer to track starting/ending times for day, breaks, and lunch
for clock hour purposes. This can be done on scratch paper. State that these will be
used to determine the number of clock hours for the committee work. Keep this
information with a hard copy of the clock hour form in the committee folder.

Encourage members to go to dinner together. Give restaurant recommendations.

Participants must sign in and out each day for clock hours and pay.
4
Rangefinding with Committee

Pictures taken for database on _______________________.
5. Group Norms

Be on time.

Be an active participant: ask questions, state opinions, listen and reflect on others’
input.

Keep conversations focused on and productive toward the task.

Respect others’ experiences and opinions.

Use the Parking Lot for general testing questions.
6. Begin Rangefinding Process.

DRC introduces item.

All committee members work the item and signal when done by turning their name
tents on end.

DRC introduces rubric. Apply any known edits. All participants verify information in the
rubric - PE, grade level, language from the standard, etc. - is correct. ETS makes edits as
needed.

DRC introduces papers in the training sets.
o For pilot Rangefinding, DRC walks the committee through papers in the training
set, starting from what is expected to be the clearest, highest scoring papers.
Participants can ask for clarification, open up discussion for questions about
scoring, language changes to the rubric, etc. Participants should avoid going
down the “What if a student does this or that?” road.
o For operational Rangefinding, DRC presents anchor papers that exemplify score
points, explaining salient points about why a paper received a particular score as
determined at pilot Rangefinding. The main purpose is to familiarize the
participants with how the item was scored so that participants can accurately
score the practice set and determine appropriate qualifying sets. Encourage
participants to use the sticky notes to capture questions about particular scoring
details for discussion at the end of the anchor set.
5
Rangefinding with Committee

Participants, with their understanding of the rubric and the papers presented by DRC,
score a number of the papers (5-7 depending on the size of the set), designated by the
scoring director, without discussion; group discussion will occur later, as needed. Ask
participants to signal when done by turning their name tents on end.

DRC calls for scores. Participants show score cards to indicate their individual scores.
OSPI records the initial scores on the hard copy. DRC/OSPI facilitate discussion to
determine a final score, as needed. OSPI captures salient points of discussion, rationales
for scoring, scoring bullets exemplified by the response, etc. on hard copy. It is
important to take good notes for future reference. ETS records rubric revisions as
needed.

For operational Rangefinding, participants score two qualifying sets, following the same
procedure described above. These papers should be clear examples of the rubric,
scoring notes, anchor and/or practice papers; nothing “new” should be introduced in
the qualifying sets.

When scores have been determined on all papers in a training set, the final scoring
rubric is reviewed once more and approved by all.
During the Rangefinding process:
-Papers can be identified as a good anchor, practice, or qualifying paper, or can be removed
from the training set.
-DRC specialist can restate why a paper received a score: “This paper received a (2, 1, or 0)
because…” to reinforce scoring criteria and look-fors, especially if a paper is unique or
exemplifies a scoring bullet or scoring rule.
-Papers can be flagged for later consideration. Sometimes scoring becomes clearer with more
papers.
-Final decisions can be made by OSPI/DRC/ETS team after discussion has yielded no new input
and consensus cannot be achieved.
6
Rangefinding with Committee
7. End-of-Day

Record edits for Test and Item Specifications, vocabulary list, “Lessons Learned”,
“Updates…” documents, etc.

Call for questions about process, ways to improve process, etc.

Tidy up. Determine what needs to be “secure”, etc. Check to make sure participants do
not leave with secure materials.
o Check bags, notebooks, etc. for secure materials.
o Peruse any notes that committee members have for information specific to
secured items or rubrics.

Thank participants for hard work.

Let participants know “start time” in morning.
o On second-to-last day, remind participants about checking out from the hotel in
the morning (should be done before start of the day if working offsite from the
hotel or, if onsite and ahead of schedule, during the morning break) and that
they can bring luggage from their hotel room and leave in the meeting space,
away from secure materials.

Dinner questions?

Make sure participants sign out as they leave.

Debrief with DRC and ETS and prep for next day’s work.
o Verify OSPI, DRC, and ETS have the same scores and scoring rationales (specific
bullets, notes, etc.) for all items; this is called reconciling.
o Make any necessary decisions about item scoring and rubric language.
o Determine adjustments to pace/flow of work as needed.
o Compare rubrics for similar items for consistency. ETS prints approved rubrics
(may be done at the end of the week or grade level). . ETS, DRC, and OSPI initial
every rubric.
o Coordinate with DRC to securely store secure materials.

As needed, debrief with other OSPI staff regarding decisions still pending.
7
Rangefinding with Committee
Closing (Last Day of Committee)
With teacher participants:
1. Tidy up.
2. Document any other suggestions for Test and Item Specifications, vocabulary lists, “Lessons
Learned”, “Updates…” documents, etc. Advertise End-of-Course and MSP Supports Moodle.
3. Thank everyone for work they have done (DRC scoring director, ETS content specialist, and
especially the teachers).
4. Determine the total number of clock hours using the times recorded by one of the
participants, and tell all the participants. Document the number of hours on one clock hour
form and place it, with the scratch paper used to track the hours, in the OSPI committee folder.
5. Distribute evaluation forms and collect them in exchange for a clock hour form. Encourage
participants to fill out the clock hour form prior to leaving (we will mail it to them if we find it
with teacher-specific information), to make a copy for their records, and submit them to the
person in their district/ESD who tracks clock hour information. Participants should not send the
form to OSPI. OSPI only provides one copy of clock hours.
6. Complete a security check as participants leave. Participants may take their copy of the Test
and Item Specifications.
OSPI, DRC and ETS:
1. Make sure ETS, DRC, and OSPI have the same scores and scoring rationales for all papers in
all the training sets; this is called reconciling. This can be done intermittently during the week.
2. Make any necessary scoring decisions and/or rubric edits.
3. For pilot Rangefinding, make sure all suggested item edits for Data Review are clearly
marked in red on the item.
4. Make sure all rubrics have been finalized and initialed.

OSPI staff takes 2 copies of each initialed rubric. One copy stays with the training set for
the item and the other goes in a folder with all the rubrics from that year’s pilot
Rangefinding. DRC and ETS each take 1 copy of the initialed rubric.
8
Rangefinding with Committee

Either obtain an electronic copy of each rubric on a thumb drive or ask ETS to post the
rubrics on the sftp. Store the electronic version on the S: drive.

Complete and copy sign-off sheet for all items reviewed at Rangefinding.
5. Coordinate with DRC staff to shred or store for shredding the participants’ secure materials.
9
Rangefinding with Committee
After Rangefinding
1. OSPI files all documents from Rangefinding with this year’s test development.
2. DRC mails a hard copy or sends an electronic copy of all Rangefinding training sets to OSPI.
OSPI stores hard copies in the vault with the set with OSPI notes or stores electronic copy on
the S: drive.
3. After operational Rangefinding, DRC presents validity papers, either in hard copy form if
meeting face-to-face or by posting them to the sftp. OSPI reviews and approves these papers.
4. OSPI reviews the training sets and works with DRC and ETS on any edits needed, either prior
to scoring or as a result of papers that appear during scoring.
5. After pilot Rangefinding, DRC sends OSPI a scoring summary for each item. OSPI reviews the
scoring summary and works with ETS and DRC to make any changes to the item, training
materials, rubric, etc. OSPI files the scoring summary in the relevant item-specific pilot
Rangefinding folder.
6. Committee feedback is included in the Test and Item Specifications, vocabulary lists,
“Lessons Learned”, “Updates…” document, etc. Updates for the Test and Item Specifications
and vocabulary lists should be determined by all OSPI mathematics assessment specialists and
given to support staff to apply and update the online documents.
7. OSPI updates the item bank and item status file.

After pilot Rangefinding, any edits to items and/or rubrics should be captured in the
pilot item bank prior to Data Review. Update item status file with pilot Rangefinding
year, item status, and comments.

After operational rangefinding, clean items and rubrics should be filed in the operational
item bank. Update item status file with pilot Rangefinding year, item status, and
comments.
10