Constructed Response Scoring
for the NC Final Exams (NCFE)
Collaborative Conference for Student Achievement
March 2015
1
NCDPI
Test Development Section
Hope Lung – Section Chief
NCSU
Technical Outreach for Public
Schools (TOPS)
Randy Craven
Christopher Warren
Akia Beverly-Worsley
2
Constructed Response Items
• Benefits
– Allows measurement of complex skills
– Forces students to have a deeper knowledge of
the concept (Must know the correct answer and be able to
explain it)
– Ensures writing will be taught
• Disadvantages
–
–
–
–
Expensive
Scores are not immediate
Students do not score as well
Often performance tasks are administered months
prior to a multiple-choice assessment
3
North Carolina State University
Technical Outreach for Public
Schools (TOPS)
4
Technical Outreach for Public
Schools (TOPS)
• Public university partner (non-profit)
• Provides test development resources such
as NC teacher recruitment, content
expertise, and production
• Created the online test development system
• Manages shipping and distribution
• Created and manages NC Education and
NCTest
• Scans and scores NCFEs with CR items
5
Brief History
NC Final Exams
6
Brief History–Assessment
2012-13 Measures of
Student Learning (MSL):
NC Common Exams
• Purpose (EVAAS)
• Designed as classroom
exams
• Paper format only
• Items written by teachers with
no additional vetting
• Items not field tested
• Local administration policy
• Printed and locally scored
2013-14 NC Final Exams
• Purposes (EVAAS, student final
grade)
• Redesigned as state-developed
• Paper format only
• Items written by teachers
• EOG/EOC test development
process implemented
• Field test items embedded
• Administration standardized
• Printed, administered, and
scored by NCDPI
7
Brief History–Items
2012-13 Measures of Student
Learning (MSL):
NC Common Exams
• Multiple-choice
• Constructed response items
– Students wrote responses
on separate paper
– Response length was not
limited
– Across each content area
(ELA, math, science,
social studies)
– Variety of rubrics 0-5
2013-14 NC Final Exams
• Multiple-choice
• Short-answer constructed
response items
– Defined as short-answer
– State-mandated answer
sheets standardized
response length
– Across each content area but
reduced number of exams
due to cost
– Rubrics 0-1 (math), 0-2
8
Overview of 2014-15
Constructed Response (CR)
Items in the NC Final Exams
9
Overview of 2014-15
Constructed Response (CR) Items
• Number of CR items reduced
– Concerns regarding test administration time
– Score turn-around time for student final grades
– Cost
• Included items field tested in 2013-14
• Test administration time increased from 90 to 120
minutes
– Adding multiple-choice items improved reliability
• Online format available
– Ten NC Final Exams added fall 2014
– All available spring 2015
10
Overview of 2014-15
Constructed Response (CR) Items
Exams with Short-Answer CR items
•
•
•
•
•
•
Grade 8 Social Studies
Civics and Economics
American History I
American History II
World History
English III
11
Overview of 2014-15
Constructed Response (CR) Items
NC Final Exams offered Approximate Number Tested
Online (NCTest) Fall 2014
Online (NCTest)
English III
23,696
Math II
15,015
Math III
14,013
Grade 8 Social Studies
975
Civics and Economics
19,854
World History
23,777
US History
887
Grade 7 Science
846
Earth/Environmental
17,180
Physical Science
8,963
12
Overview of 2014-15
Constructed Response
Items
NC Final Exam
Answer Sheet
for Paper-andPencil
13
Overview of 2014-15
Constructed
Response Items
World History
Short Answer
Constructed
Response
Released Item
14
Overview of
2014-15
Constructed
Response Items
Rubric–
World
History Short
Answer
Constructed
Response
Item
15
Scorer Selection & Training
16
Scorer Selection & Training
Scorer Employment Criteria
• Human-scored (not machine-scored or artificial
intelligence)
• 4-year college degree required
• Current and retired NC teachers targeted (preference
given to content experience in tested subject)
• Agree to the Test Security Agreement terms
• Available 20 hours per week (minimum of 2 hours per
weeknight)
• Have personal computer to use in private setting (no
scoring at school)
17
Scorer Selection & Training
Scorer Recruitment
• Notification is sent to school districts
• Interested parties are directed to the online
course “Constructed Response Scoring 201415”
https://center.ncsu.edu/ncpd/course/view.p
hp?id=15
• Interested parties submit the “Interest Form” at
the end of the course
– Database is created
18
Scorer Selection & Training
Training Scorers
• Training consists of a combination of
on-site session and online modules
• Scoring Tools used in training:
Constructed Response Item
Constructed Response Rubric
Reference Materials (Stimulus, Guide Papers,
etc.)
19
Scorer Selection & Training
Training Procedures for Scorers
• Review the information in the scoring tools
• Discuss any questions about the scoring
tools
• Take practice quizzes independently
Receive feedback
• Take qualification quizzes independently
Receive score to determine employment
20
Scorer
Selection &
Training
NC Education
online courses
are subjectspecific to
provide
training for
each content
area
21
Scorer Selection & Training
22
Scorer
Selection &
Training
Scorers are notified
when they pass the
qualification
quizzes and they
are given access to
begin scoring
student responses
23
NC State University
Technical Outreach for Public
Schools (TOPS)
Rangefinding
24
Rangefinding
• Conducted for each new set of constructed
response items
• TOPS staff pre-scores field test responses
• NC teachers discuss student responses to
field tested items and score them
• Defines cuts between scores
• Validates scoring
• Generates training materials
25
NC State University
Technical Outreach for Public
Schools (TOPS)
Material Processing
26
TOPS
Material
Processing
27
TOPS Material Processing
Online Testing
• Student registration and
accommodations completed
on NC Education before test
day
• Test sessions scheduled by
LEA Test Coordinator
• Student completes NCTest
online assessment
• Responses are automatically
loaded into CROSS
immediately for scoring
Paper-and-Pencil Testing
• LEA precodes answer sheets &
distributes test materials to schools
Answer sheets are shipped to TOPS
using return kits
• TOPS conducts initial check for
missing QR codes and form numbers
(follow-up with LEA)
• Answer sheets are image scanned
and paper documents archived
• Remark processing (verify scan
results and create jpg of each CR)
• Final check for incorrect/missing info
on documents (error report
generated with LEA follow-up)
• Cleared items loaded into CROSS
28
TOPS Material Processing
Material Check-In
•
•
•
All answer sheets are
unboxed and reviewed for
completion errors
Types of errors
– No form indicated
– Multiple forms marked
– Mismatch between
bubble and written form
indicator
– Incorrect or damaged
QR Code
– Write in book not
transcribed
Why manually check?
Won’t the computer catch it
anyway?
29
TOPS Material Processing
Image Scanning
Documents
• Canon optical
Scanners
• 130,000 answer
sheets scanned for
Fall 2014
• Scans 100
documents in a
minute (front and
back)
• Creates pdf file
30
TOPS Material Processing
Document Archive
• Need to be able to locate any
one answer sheet at any time
• After scanning all answer
sheets are returned to
original box
• Each scanner station labels
each box with color coded
numeric sequence
• LEA or charter school
including pdf(s) documented
31
TOPS Material Processing
Image Scan
Processing
• All image scans
are reviewed
• Every document
reviewed
• Scan quality
checked
• Blanks & Multi
• CR sections are
snipped and
jpegs created for
CROSS
32
TOPS Material Processing
Input Management
• Data files output image scan processing
reviewed and data sent to Constructed
Response Online Scoring System
(CROSS)
• Any concern (missing form number etc…)
generated in error report for resolution
• Constructed response images are moved
to CROSS
33
Scoring
Constructed Response Items
34
Scoring Constructed Response Items
Remote Scoring using CROSS (Constructed
Response Online Scoring System)
• Scorers review student responses on the
screen (paper-and-pencil and online exams)
• System prioritizes seniors
• Scores assigned (0-1-2)
Scorers have an “Alert” button to report
responses (threats, danger to self, vulgar
language, etc.)
35
Scoring Constructed Response Items
• Users can access anywhere
• Only accounts added to CROSS have access and those
accounts can only access assigned scoring queues
• Users login using NC Education credentials
36
37
Scoring Constructed Response Items
38
Scoring Constructed Response Items
Quality Control Processes
• All alerted responses are reviewed and reported for
action
• Scorers are monitored throughout the window
• TOPS Lead scorers review disagreement
• Double-scoring process
• Reliability statistics are monitored throughout the
window
• Processes are reviewed following each window
39
Test Record Data Transfer
40
Data Transfer
Received Records
• Received test records that are scanned
(answer sheets) or uploaded (NCTest) into
CROSS are reported to LEA test coordinators
overnight
• LEAs can check student rosters in WinScan
to verify scanned student answer sheets and
received online test records
41
Data Transfer
Scored Records
• Multiple-choice and constructed response items are
scored
• Multiple-choice and constructed response item scores
are merged back into the student’s test record
• Scored exams are reported electronically to LEA test
coordinators each business day
• Online test records are scored within 5 business days
from test finalization
• Shipped paper-and-pencil records are scored within 8
business days beginning with the first business day
following receipt at TOPS facility
42
Resources
43
NC Final Exams Resources
• Assessment Specifications (with number of items per
standard/objective)
http://www.ncpublicschools.org/accountability/commonexams/specifications/
• Released Items Fall 2014 (with item statistics)
http://www.ncpublicschools.org/accountability/commonexams/released-items/
• Social Studies Documents and Assessment Guides
http://www.ncpublicschools.org/accountability/commonexams/addresources/
44
NC Final Exams Resources
• State Testing Results (Green Book) 2013-14
http://www.ncpublicschools.org/docs/accountability/report
s/green/1314/ncfescorefreq1314.pdf
• Test Development Training (TOPS)
https://center.ncsu.edu/nc/x_courseNav/index.php?id=21
• To participate in item writing or review (TOPS)
http://goo.gl/forms/wXv4Imh0ko
• To participate in constructed response scoring(TOPS)
https://center.ncsu.edu/ncpd/course/view.php?id=15
45
Questions?
46
Thank you!
47