December 2014

• Achievement level descriptors (ALDs) are a means of describing performance on a standardized test in terms of levels or categories of performance.
• For the Smarter Balanced assessments, outcomes will be reported in terms of four levels of achievement: Level 1, Level 2, Level 3, and Level 4.
• The ALDs are text descriptions of the knowledge, skills, and processes demonstrated by students in each category of performance.

“…the ALDs reflect the depth and rigor of the
CCSS as well as the way in which Smarter Balanced intends to assess the CCSS.”

• Smarter Balanced developed multiple levels
• Four interrelated levels
• Used to
• Guide development of the assessment
• Set cut scores
• Interpret results

• Labeled as “initial” because they will be refined and finally adopted by Smarter Balanced after student performance data are collected through field test and after standard setting.
• Will be augmented to include the reporting ALDs.

• Target D: Apply and extend previous understandings of numbers to the system of rational numbers.

• Level 1 students should be able to place all integers on a number line and integer pairs on a coordinate plane with one-unit increments on both axes.
• The Level 4 student demonstrates thorough ability to produce compelling, well-supported writing for a diverse range of purposes and audiences.

• The student who just enters level 3 should be able to:
• Generate measurement data by measuring length using rulers marked with quarter-inch intervals and represent the data on a line plot marked with quarter-inch intervals.
• Solve word problems involving perimeters of polygons.

• Target 12: Interpret, explain, or connect information presented within or across texts (e.g. compare/contrast, cause/effect, integrate information).

• Level 3 students should be able to adequately interpret, explain, or connect information presented within or across texts of moderate-to-high complexity.
• Level 4 students should be able to thoroughly interpret, explain, or connect information presented within or across texts of unusually high complexity.

• The student who just enters level 3 should be able to:
• Use details and information from texts of moderate complexity to support answers and inferences.
• Identify or summarize central ideas/key events in texts of moderate complexity.
• Use supporting evidence to justify/explain own inferences in texts of moderate complexity.

• Labeled as “initial” because they will be refined and finally adopted by Smarter Balanced after student performance data are collected through field test and after standard setting.
• Will be augmented to include the reporting ALDs.

• Edits/refinements to draft ALDs based on field test data and standard setting
• Development of Reporting ALDs

Joe Willhoft, Ph.D., Executive Director
Smarter Balanced Assessment Consortium
California State Board of Education
November 13, 2014

•
Online Panel
– Over 10,000 registered
•
In-Person Panel
– 504 panelists confirmed
40+ alternates
482 participants
– Demographics
Representation from each state in each panel
In-Person panel composition matched SB teacher workforce
•
Cross-Grade Panel
– 64 panelists from In-Person panels

Demographics:
Representation from each state in each panel
In-Person panel matched SB teacher workforce by gender and ethnicity

•
Opening of the Window
– October 6-17
– Individual windows and total window extended
•
Level of Participation
– 10,099 registered
– 5,840 logged in
– 2,660 submitted
•
Support Provided
•
Results Shared with In-Person Panel

•
Training Activities
– Software
– Common Core and ALDs
– Ordered Item Booklet
•
Panel Activities
– Bookmark placement
– Discussion
Table Leader training before Day 1 begins

Training
As you study each item in the OIB, discuss two questions with your fellow panelists:
22
21
20
19
1. What do you know about a student who 18
17 responds successfully to this item; that is, what
16 skills must a student have in order to know the 15
14 correct answer?
13
12
11
2. What makes this item 10
9 more difficult than preceding
8 items?
6
7
5
4
3
2
1
Ordered
Item
Book

1.
What do you know about a student who responds successfully to this item; that is, what skills must a student have in order to know the correct answer?

41

Targets
1–7:
Reading
Literary
Text
The student who just enters Level 3 should be able to:
 Use details and information from texts of moderate complexity to support answers and inferences.
 Identify or summarize central ideas/key events in texts of moderate complexity.
 Begin to determine the intended meanings of words, including words with multiple meanings, based on context, word relationships, word structure, and use of resources in texts of moderate complexity.
 Use supporting evidence to justify/explain own inferences in texts of moderate complexity.
 Interpret, specify, or compare how information is presented across texts of moderate complexity.
 Begin to relate knowledge of text structures, genre-specific features, or formats to obtain, interpret, explain, or connect information within texts of moderate complexity.
 Determine or interpret figurative language, literary devices, or connotative meanings of words and phrases used in context and partially explain the impact of those word choices on meaning and tone in texts of moderate complexity.

Items 1-17:
At least 50% chance of success
Items 18 – 22: less than a 50% chance of success.
9
10
11
12
13
14
15
16
L3
17
18
19
20
21
22
8
7
6
5
Ordered
Item
Book
1
2
3
4 Ask yourself: Would a student at the threshold of Level 3 have at least a 50% chance of earning this point?
Yes: Move on to the next item.
No: Place your bookmark here.
24

•
Each grade-bycontent panel recommended 3 “cut scores” to define 4 achievement levels
– Group median
– 10 th , 25 th , 75 th , 90 th percentile of panel also recorded

How confident are you about the three bookmarks you just entered? (At end of Round 3)
Bookmark
Very
Confident Confident Uncertain
Very
Uncertain Total
Level 2 222 (47%) 237 (51%) 10 (2%) 0 (0%) 469
Level 3
Level 4
234 (50%) 220 (47%) 15 (3%)
245 (52%) 217 (46%) 7 (1%)
0 (0%) 469
0 (0%) 469
Overwhelming endorsement of process (92-99% positive) on 14 separate measures.

•
Consider recommendations through policymaker lens
•
Two principles to balance:
Honor the work/advice of the Ach. Level Setting panels
Consider external information about student readiness for college-level coursework
•
Conversations with states are continuing

•
After Smarter Balanced states approve Consortiumlevel cut scores…
•
Each state follows its own processes for adoption of scores for its purposes/uses

• As you review the Threshold scale scores, what do you notice?
• As you review the percentage of students scoring at each achievement level?
• What are the similarities and differences between the ELA and mathematics scale scores and achievement levels?

• Scale scores from the Smarter Balanced assessments cannot and should not be compared with scale scores from the previous STAR/CST assessments.
• Different standards are being assessed
• Different assessment methods are being used
• Different levels of cognitive rigor involved

• http://www.smarterbalanced.org/achievementlevels/
• http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2012/11/Smarter-Balanced-ELA-
Literacy-ALDs.pdf
• http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2012/11/Smarter-Balanced-Math-
ALDs.pdf