Plans for Reporting Results and
Performance Level Setting
The Second Annual New Jersey Leadership Summit
January 21, 2015
Jeffrey Nellhaus
Chief of Assessment, PARCC, Inc.
Objectives for this morning …
1. Priorities
Provide a brief review of the priorities that drive PARCC
2. Reports of Results
You will become knowledgeable about the kinds of metrics and
reports that will be used to report the results of the PARCC
assessments next fall
3. Performance Level Setting (aka standard setting)
You will be knowledgeable about the process PARCC intends to
conduct this summer to identify the threshold scores for each of its
five performance levels
2
PARCC Consortium in 2015
• Administering
the test in 12
members
NY
• 5 million
students
OH
IL
CO
• Nearly 15% of
students in
grades 3-11
nationally
• About 3 in 4 will
take the tests on
computer
MA
RI
NJ
MD
DC
TN
NM
AR
MS
LA
Governing Board States
3
Goals …
Develop tests
worth taking
Texts worth reading
Essays worth writing
Problems worth solving
Accessible to all students
Report results
worth using
Valid
Reliable
Timely
Actionable
4
Priority Purposes
 Determine whether students are
college- and career-ready or “on track”
 Provide tools to assess student
learning and support instruction
during the school year
 Report comparable results across
schools, districts and member states
 Generate data, including growth
metrics, for accountability uses
 Use technology to
increase student access
and engagement, and
create efficiencies
5
PARCC Assessment System
6
Draft Score Report Designs
“Results Worth Using”
7
PARCC Score Report Design Study
• PARCC conducted a 5-month, multi-phase Score Report
Design Study to establish priorities for the kinds of data the
reports should include and how they should be presented.
• Stakeholders providing input included:
Teachers and Administrators (Central Office and Building)
Parents
Higher Education Representatives
PARCC state education leadership
PARCC Technical Advisory Committee (TAC)
8
Data Management & Reporting System
• PARCC score reports will be available to local educators online
through its interactive web-based Data Management and Reporting
System, and to parents via a printed student reports
• PARCC’ web-based system will provide authorized users to customize
reports using the following “filters”
Student
School
District
State
PARCC
Year
Subject
Type of Test
Grade
Student Group
Overall Scores
Status Scores
Growth Scores
Sub-Scores
Item Level Scores
9
Metrics for Reporting Results
Performance
Levels
•Math
•ELA/L
Scale Scores
• Mathematics
• ELA/L
• Reading
• Writing
Sub-Scores
•Whether the
student’s
performance in
the subcategory was
At/Above,
Near, or Below
the
performance of
students who
scored “just
Level 4” overall
in the content
area
Item Scores
• At the level,
student the
number of
points earned
• At the school
level, the
average
number of
points earned
by all the
students in
the school
who took the
item.
Draft of Individual Student Report
11
Draft of Student Roster - Sub-Scores
12
Draft of Student Roster – Item Scores
13
Draft School by Grade Summary Report
14
Draft District by School Summary Report
15
Draft State by District Summary Report
16
Draft PARCC by State Summary Report
17
Performance Level Setting
18
PARCC Performance Levels
• PARCC will report results using five performance levels
(names under discussion)
• Students performing at Levels 4 or 5 on Algebra II,
Integrated Math 3, or ELA/L grade 11 assessments will earn
a College- and Career-Ready Determination
• Public colleges and universities in PARCC states have
agreed to allow students who have been accepted and
have earned CCR Determinations to enter directly into
credit-bearing courses without remediation
19
Performance Level Setting: What is it?
Use Performance Level
Descriptors & Data
from Research
Identify Threshold
Scores
Five PARCC
Performance Levels
20
Performance Level Setting
Who will participate?
K-12 educators
Postsecondary faculty
Grade-Span Panels
Business people
•
•
•
•
High School Panels (6)
Grade 3-4 Panel (2)
Grade 5-6 Panel (2)
Grade 7-8 Panel (2)
Community members
Each Panel will
Include 20
Participants
21
Schedule of Performance Level Setting Events
PARCC Performance Level Setting Schedule
Subject/Grade
Panel
Start Date End Date
Mathematics Panels
Dry Run
ELA/Literacy
Panels
5/21/2015 5/28/2015 Algebra II and Integrated 3 Grade 11
Algebra I and Integrated 1
Grade 9
High School 7/27/2015 7/31/2015 Geometry and Integrated 2 Grade 10
Algebra II and Integrated 3 Grade 11
Grades 7–8 8/17/2015 8/21/2015 Grades 7 and 8
Grades 7 and 8
Grades 5 and 6
Grades 5 and 6
Grades 3 and 4
Grades 3 and 4
Grades 3–6 8/24/2015 8/28/2015
22
Performance Level Setting:
What Process Will be Used?
PARCC will Use an Evidence-Based Process
• The process combines expert judgment with data from
research
• The process will generate threshold scores that
– Reflect PARCC’s performance level descriptors (PLDs)
– Are consistent with the results reported by other well-regarded
state, national and international assessment programs
• Because comparability is a priority, every state will adopt
the same threshold scores
23
Performance Level Setting: Training
Panelists will:
• Practice with online tutorial
• Take the computer-based assessment
• Review the performance level descriptors (PLDs)
• Write “borderline” performance level descriptors
(PLDs)
• Practice with the process
24
Proposed Performance Level Setting Method
Yes/No Method (proposed)
Review test items
• Answer the
question:
• Based on the
PLDs, would a
student
performing at
Level X be
likely to
answer the
item correctly?
Impact data
• Impact data
includes:
• Empirical data
from research
• Judgments of
other panelists
• % of students
performing at
each level
• % who got
item correct
Multiple Rounds
• After an initial round
of review, panelists
discuss their
judgments, are given
certain impact data
and conduct a
second round of
review,
• After the second
round, panelists
discuss their
judgments again, are
given additional data
and conduct a third
and final round.
25
Grade 3 Sample Item and PLDs
Patricia needs to read for 120 minutes each week.
• She read for 26 minutes on Monday.
• She read for 39 minutes on Tuesday.
• She read for 38 minutes on Thursday.
How many more minutes does Patricia need to read this week?
minutes
Click on the answer box and type in a number. Backspace to erase.
Grade 3 Math : Sub-Claim A
The student solves problems involving the Major Content for the grade/course with connections to the Standards for
Mathematical Practice.
Level 5: Distinguished
Level 3: Moderate
Level 4: Strong Command
Level 2: Partial Command
Command
Command
Time
3.MD.1-1
3.MD.1-2
Tells, writes and measures
time to the nearest minute.
Tells, writes and measures time
to the nearest minute.
Tells, writes and measures
time to the nearest minute.
Tells, writes and measures
time to the nearest minute.
Creates two-step real world
problems involving addition
and subtraction of time
intervals in minutes.
Solves two–step word problems
involving addition and
subtraction of time intervals in
minutes.
Solves one-step word
problems involving addition or
subtraction of time intervals in
minutes.
Solves one-step word
problems involving addition
or subtraction of time
intervals in minutes, with
scaffolding, such as a
number line diagram.
26
Research to Inform the Performance Level
Setting Process
• Research Questions
– What percentage of students are likely to be college- and career-ready,
according to PARCC’s definition?
– What percentage of students are likely to be on track to college and career
readiness at earlier grades?
• Two studies to inform performance level setting
Postsecondary
Educators
Judgment Study
Benchmark Study
SAT
ACT
NAEP
TIMSS
PISA
27
Performance Level Setting Timeline
28
Updates and more information
• E-mail me
jnellhaus@parcconline.org
•
•
Follow me on Twitter
@jmnellhaus
• Follow us on Twitter
@parccplace
• Visit our website
www.parcconline.org
• http://parcc.pearson.com/
• Sign up for the PARCC Updates
newsletter on our website
29