Office of Assessment
October 22, 2014
Smarter
ELA/Literacy
DCAS Science
DCAS
Social Studies
PSAT
SAT
End of Course
US History
ACCESS (ELL)
DCAS-Alt1
Smarter
Mathematics
Grade 9/10
Option
Smarter Interim
Assessment Blocks
NAEP
End of Course
Alg II and IM III
Smarter Digital
Library
Smarter Interim Comp
Assessments
1
Summative Assessments for
Accountability
2
Smarter ELA/Literacy and Mathematics
Assessments
 Replaces DCAS reading and mathematics in the
2014–2015 school year
 These assessments will be available for one (1)
opportunity and
 Administered in grades 3–8 and 11 in spring of 2015
 Aligned with the Common Core State Standards
 Comprised of 2 components
 Computer Adaptive Test (CAT) – contains multiple
choice, grid, and interactive items
 Performance Tasks (PT) – requires students to
complete extended projects with real-world skills
(research, writing, application)
3
Summative Assessment
(Computer Adaptive)
 Administered within the last 12 weeks (grades 3–8)
and 7 weeks (grade 11) of the instructional year
 Includes a variety of questions types
 Selected response
 Short constructed response
 Extended constructed response
 Technology enhanced
 A small number of items will require hand-scoring by
professionally trained scorers
4
Summative Assessment
(Computer Adaptive)
 Assesses the full range of Common Core in ELA
and mathematics for students in grades 3–8 and
11—interim comprehensive assessments can be
used in grades 9 and 10
 Measures current student achievement and growth
across time, showing progress toward college and
career readiness
5
Summative Assessment
(Performance Task)
 A PT is made up of a Classroom Activity and an
individually administered, computer-generated task
 The Classroom Activity takes place before the
student engages in the performance task, is not
scored, and does not have special technology
requirements
 The Classroom Activity is a teacher-led group
activity designed to provide needed context for a
particular PT topic
6
Summative Assessment
(Performance Task)
 Each student will complete one PT in each content
area
 There are multiple performance tasks associated
with a particular Classroom Activity and PT topic
 PT items require hand-scoring—for summative by
professionally trained scorers
7
Assignment of Performance Task
Topics and Tasks
States will be responsible for ensuring that:
 No performance task topic is overassigned within the
state, and
 No performance task is overassigned within the state.
1. States should randomly assign performance task topics to
schools by grade.
2. The test engine should randomly assign performance tasks
to students within a topic, except for students requiring a
fully accessible performance task.
3. States should periodically monitor assignments to verify
that the process is functioning correctly and neither task
topics nor tasks are over-assigned.
8
Flexible Item Pool
Design Principles
 The item selection algorithm will be designed to
administer out-of-grade-level items.
 All tests will adhere to on-grade blueprint content
requirements.
 Testing out of grade level will not increase test
length.
10
Conditions for Students Receiving Items
from Adjacent Grade Levels
On-grade content
criteria minimum
met
Performance is
clearly in Level 4
or Level 1.
Off grade level item
better meets
measurement and
content
requirements than
on-grade items.
11
1. On-Grade Content Criteria
Minimum Met
 A decision point is reached when a substantial
proportion of content has been covered.
 ELA-CAT (no human-scored) about 60%
 Mathematics-CAT (no human-scored) about 60%
12
2. Performance is clearly above/below Level 3
 Decision – Is there a high degree of certainty that
the student current score is at Level 1 or Level 4?
 Action:
If yes, add appropriate items from adjacent grades to
the pool. (Content experts have screened items for
appropriateness.)
If no, continue with the on-grade-level pool.
13
Off-grade level item better meets measurement
and content requirements than on-grade items
 For all students, continue with the on-grade
blueprint.
 Item selection:
For students with expanded item pools, an offgrade level item can be chosen if it satisfies content
and information constraints better than an on-grade
item.
14
Final Scores and
Achievement Categories
 The final score is based on responses to items from
both the PT and CAT components.
 All PTs are delivered on grade level.
 Smarter Balanced will report final overall scores,
achievement levels, and claim information from the
whole test.
 Score reports will display student performance
relative to enrolled grade; there will not be off-grade
score reports.
15
What Is a Test Blueprint?
 Test blueprints (specifications) should clearly
describe
 Assessment purpose(s)
 What assessment is designed to measure
 Sufficient details
 Blueprint specifies intended test construct and
desired psychometric properties
 Sample blueprints presented today focus on content
specifications for summative assessments
17
Content Weight at Claim Level – ELA
ELA/Literacy Content Weighting
Grades
3–5
40%
Grades
6–8
41%
Grade 11
40%
Claim 2. Writing
26%
25%
26%
Claim 3. Listening
18%
18%
18%
Claim 4. Research
16%
16%
16%
Claims
Claim 1. Reading
18
ELA Preliminary Blueprint – Grade 3
19
ELA Blueprint Notes – Grade 3
 The number of items is not necessarily equal to the
weight by claim
 Each student will receive an overall ELA/Literacy
score and 4 claim-level scores
 The CAT component includes machine-scored and
short-text items. The 2 short-text items in Reading
(associated with the long reading stimuli) and 1
short-text item in Writing will be hand-scored.
20
ELA Blueprint Notes – Grade 3
 Each student will receive 1 PT, which includes a set
of stimuli on a given topic. A classroom activity will
be conducted for each PT to help orient students to
the PT context.
 The same set of stimuli is used for the Writing and
Research items.
 The short-text items and the essay are hand-scored.
 The Research PT includes 2 short-text items and 1
machine-scored item.
21
ELA Blueprint Notes – Grade 3
 Total items in Claim 2 – Writing includes 10 CAT
items and 3 items from the PT
 Writing PTs are noted as 3 separate “items”;
however, the Writing PT score is derived from a
single student response scored on 3 distinct traits
 For more information on content categories, see the
Content Specifications document at
http://www.smarterbalanced.org/smarter-balancedassessments/
22
Content Weight at Claim Level –
Mathematics
Mathematics Content Weighting
Grades
3–5
54%
Grades
6–8
54%
Grade 11
56%
Claim 2. Problem Solving
Claim 4. Modeling and Data Analysis
24%
24%
23%
Claim 3. Communicating Reasoning
22%
22%
21%
Claims
Claim 1. Concepts and Procedures
23
Mathematics Preliminary Blueprint –
Grade 6
24
Mathematics Blueprint Notes – Grade 6
 The number of items is not necessarily equal to the
weight by claim
 Claim 2 (Problem Solving) and Claim 4 (Modeling
and Data Analysis) are combined because of
content similarity for the flexibility in item
development. Although there are 4 claims, only 3
claim-level scores will be reported.
25
Mathematics Blueprint Notes – Grade 6
 In the CAT component, each student at grades 6–8
will receive one item from either Claim 3 Target B or
Claim 4 Target B that needs Hand-scoring
 For more information on content categories, see the
Content Specifications document at
http://www.smarterbalanced.org/smarter-balancedassessments/
26
Additional Information About Blueprints
 The blueprints for CAT component are specified at
the target level to
 Control item type and task specifications
 Determine item proportion in the pool
 Control of the proportion of item DOK (depth of
knowledge) at the claim level
 In ELA, there are also genre considerations
 In mathematics, the content specifications further
delineates detail
27
Estimated Testing Times
28
Three Phases in Process
 Four achievement levels are reported for student
performance
 Over 10,000 people registered to virtually participate
in the online setting of 1 cut score for level 3. 142
participants were from Delaware (exceeded
Smarter’s expectations for Delaware)
 In-person workshops with panels of educators and
other stakeholders set cut scores of the 4
achievement levels. 25 Delawareans participated.
 Vertical articulation committee, a subset of the inperson workshops, examined recommended cut
scores across grades for adjustment
30
Reviewing Items in the OIB for Bookmarking
31
Level 3 Bookmark Placement
 When you place your Level 3 bookmark, think about a
student at the threshold of Level 3 based on the
achievement level descriptors
 Place your Level 3 bookmark at the point in the OIB such that
 A threshold Level 3 student would be have at least a 50%
likelihood of success on each of the items before the bookmark
(and about a 50% chance of success on the item just before the
bookmark).
 A threshold Level 3 student would have less than a 50%
likelihood of success on the items from the bookmark on.
 It may be easiest for you to first identify a range of items that
begins where threshold Level 3 students begin to be challenged
and end where they would definitely be challenged. Then pick
the best page in that range.
32
Vertical Articulation
33
Follow-Up Activities
 Technical Advisory Committee review
 Smarter Balanced review
 State Chiefs review and approval
 Documentation – technical report
34
http://www.smarterbalanced.org/wordpress/
wp-content/uploads/2014/06/ReportingSystem-OverviewV2.pdf
36
Smarter Score Reporting
 Things we know:
 Scores will not be available until after testing is
completed
 Score reports to families a little later than for DCAS
 Score reports to schools – date to be determined in
summer
 Full score reporting presentation at December DTC
meeting
37
Smarter Accessibility
38
Smarter Assessments …
 … provide more accommodations/supports than
any one state could develop individually
 … are some of the most accessible tests available
 American Sign Language (ASL) (video)
 Refreshable Braille
 Text to speech (TTS)
 Various language glossaries available
 External devices/software
 Supports available to general education students
39
Universal Tools, Designated Supports,
and Accommodations
40
Smarter
ELA/Literacy
DCAS Science
DCAS
Social Studies
PSAT
SAT
End of Course
US History
ACCESS (ELL)
DCAS-Alt1
Smarter
Mathematics
Grade 9/10
Option
Smarter Interim
Assessment Blocks
NAEP
End of Course
Alg II and IM III
Smarter Digital
Library
Smarter Interim Comp
Assessments
41
42