Examining Data From the Smarter
Balanced Summative Assessment
Presented to the
Leadership Convening – October 1, 2015
Deb Sigman,
Deputy Director,
Assessment & Standards Development
Services
WestEd.org
WestEd.org
Topics for Discussion
1. Making use of Summative Assessments
2. Tools available for analysis
3. Moving beyond the scores
a) Templates and Guiding questions
b) Practice and reflections
WestEd.org
Learning Outcomes
• Gain knowledge of types of scores
– new vocabulary
• Make use of the summative
assessment data
• Knowledge of and practice with
analysis tools
• Draft your own data template
WestEd.org
Making the Most of the
Smarter Balanced
Summative Assessments
Statewide Summative Assessments are Like
Icebergs
It Pays to Pay Attention
On the Surface…
• Summative tests provide information not
to be ignored
• May offer clues about what to do next
•
•
•
•
Slow Down
Go Back
Change Direction
Full Speed Ahead
How Summative Tests Can Add Value
• Rarely provide definitive answers, but raise
•
•
•
•
•
many questions allowing reflection on
practice
Provide a general direction
Provide an entry point into a collaborative,
honest conversation
Provide a necessary story
Must dig deeper to determine cause
Focus on groups, programs and
disaggregation (not individuals)
Types of Scores – New
Terms, New Vocabulary
Smarter Balanced Hierarchy of
Scores
•
.
Overall Scaled Score
Claim
Target
Types of Information
• Overall Scaled Scores
• ELA/Literacy
• Mathematics
•
•
•
•
Claims
Targets
Performance levels
Percentages in each level
Overall Scores
• Scaled scores in English language
arts/literacy and mathematics
• 4-digit number on scale from 2000 to
3000
• Group means within a grade level
• Each score corresponds to a
performance level on the scale
Performance/Achievement
Levels
• Scaled scores divided into 4
performance/achievement levels by the
standard setters
• Smarter Levels
California Levels
–
–
–
–
Level 1 Standard Not Met
Level 2 Standard Nearly Met
Level 3 Standard Met
Level 4 Standard Exceeded
• Within each level can divide into high,
medium, low performance
Assessment Claims
• Assessment claims are broad evidence-based
statements about what students know and
can do as demonstrated by their performance
on the assessment.
• Grades 3–8 and Grade 11 each have one
overall claim encompassing the entire content
area for ELA/literacy and one for mathematics.
• For ELA, there are four specific content claims.
• For math, there are three specific claims
Claim Descriptors
 Claim #1 – Concepts and Procedures
“Students can explain and apply mathematical concepts and
interpret and carry out mathematical procedures with precision and
fluency.”
 Claims #2 and #4 – Problem Solving and Modeling and Data
Analysis
“Students can solve a range of complex well posed problems in
pure and applied mathematics, making productive use of
knowledge and problem solving strategies. Students can analyze
complex, real world scenarios and can construct and use
mathematical models to interpret and solve problems.”
 Claim #3– Communicating Reasoning
“Students can clearly and precisely construct viable arguments to
support their own reasoning and to critique the reasoning of others.”
Claim Score Performance
Descriptors
• Below Standard
• At or Near Standard
• Above Standard
Assessment Targets
• Assessment targets connect the content
standards to evidence that will be collected
from the assessment.
• Targets map the common core state standards
(CCSS) onto assessment evidence that is
required to support the content categories
and claims.
• Targets are used to guide the development of
items and tasks that will measure the CCSS.
Documents to Aid in the
Analysis of the Smarter
Summative Results
Mathematics – Reporting Level Descriptors
Grade Level
Standard Exceeded
Standard Met
Standard Nearly Met
Standard Not Met
Grades 3
–5
The student has exceeded the
achievement standard and
demonstrates advanced
progress toward mastery of the
knowledge and skills in
mathematics needed for likely
success in future coursework.
The student has met the
achievement standard and
demonstrates progress toward
mastery of the knowledge and skills
in mathematics needed for likely
success in future coursework/
The student has nearly met the
achievement standard and may
require further development to
demonstrate the knowledge and skills
in mathematics needed for likely
success in future coursework.
The student has not met the achievement
standard and needs substantial
improvement to demonstrate the
knowledge and skills in mathematics
needed for likely success in future
coursework.
Grades 6
–8
The student has exceeded the
achievement standard and
demonstrates advanced
progress toward mastery of the
knowledge and skills in
mathematics needed for likely
success in entry-level, creditbearing college coursework
after high school.
The student has met the
achievement standard and
demonstrates progress toward
mastery of the knowledge and skills
in mathematics needed for likely
success in entry-level, creditbearing college coursework after
high school.
The student has nearly met the
achievement standard and may
require further development to
demonstrate the knowledge and skills
in mathematics needed for likely
success in entry- level, credit-bearing
college coursework after high school.
The student has not met the achievement
standard and needs substantial
improvement to demonstrate the
knowledge and skills in mathematics
needed for likely success in entry-level,
credit-bearing college coursework after
high school.
Grade 11
The student has exceeded
the achievement standard and
demonstrates the knowledge
and skills in mathematics
needed for likely success in
entry- level, credit-bearing
college coursework after high
school.
The student has met the
achievement standard and
demonstrates progress toward
mastery of the knowledge and skills
in mathematics needed for likely
success in entry-level, creditbearing college coursework after
high school.
The student has nearly met the
achievement standard and may
require further development to
demonstrate the knowledge and skills
in mathematics needed for likely
success in entry- level, credit-bearing
college coursework after high school.
The student has not met the achievement
standard and needs substantial
improvement to demonstrate the
knowledge and skills in mathematics
needed for likely success in entry-level,
credit-bearing college coursework after
high school.
Early Assessment Program Status
 EAP status will provide an indicator of a student’s
predicted readiness to take college-level English and
mathematics courses when student begins college.
Standard Exceeded: Ready
for English and/or
mathematics college-level
coursework.
Standard Met: Conditionally
Ready for English and/or
mathematics college-level
coursework.
Standard Nearly Met: Not yet
demonstrating readiness for
English and/or mathematics
college-level coursework.
Standard Not Met: Not
demonstrating readiness for
English and/or mathematics
college-level coursework.
Math Claim Achievement Level Descriptors
Area (Claim) Descriptors
Above Standard
Concepts and
Procedures
Applying mathematical
concepts and procedures.
The student demonstrates a thorough
ability to consistently explain and apply
mathematical concepts and interpret and
carry out mathematical procedures
Problem
Solving/Modeling and
Data Analysis
The student demonstrates the thorough
ability to consistently solve a range of
complex, well-posed problems in pure and
applied mathematics, making productive use
Using appropriate tools and of knowledge and problem-solving
strategies to solve real
strategies. The student demonstrates the
world and mathematical
ability to consistently analyze complex,
problems
real-world scenarios and can construct and
use mathematical models to interpret and
solve problems.
Communicating
Reasoning
Demonstrating ability to
support mathematical
conclusions
The student demonstrates the thorough
ability to clearly and precisely construct
viable arguments to support their own
reasoning and to critique the reasoning of
others.
At or Near Standard
The student demonstrates some ability to
explain and apply mathematical concepts and
interpret and carry out mathematical procedures
with precision and fluency.
The student demonstrates some ability to
explain and apply mathematical concepts and
interpret and carry out mathematical procedures
with precision and fluency.
Below Standard
The student does not
demonstrate the ability to explain and apply
mathematical concepts and interpret and
carry out mathematical procedures with
The student does not demonstrate the
ability to solve a range of complex, wellposed problems in pure and applied
mathematics, making productive use of
knowledge and problem-solving
strategies. The student does not
demonstrate the ability to analyze complex,
real-world scenarios and construct and using
mathematical models to interpret and solve
problems.
The student demonstrates some ability to
The student does not demonstrate the
explain and apply mathematical concepts and
ability to clearly and precisely construct viable
interpret and carry out mathematical procedures
arguments to support their own reasoning and
with precision and fluency.
to critique the reasoning of others.
Summative Assessment Blueprints
Target Sampling Mathematics Grade 5
Claim
Content
Category
Problem
Solving
(drawn across
content
domains)
2. Problem Solving
4. Modeling and
Data Analysis
3. Communicating
Reasoning
Assessment Targets
CAT
A. Apply mathematics to solve well-posed problems arising in everyday life, society, and
the workplace.
2, 3
2
B. Select and use appropriate tools strategically.
C. Interpret results in the context of a situation.
D. Identify important quantities in a practical situation and map their relationships (e.g.,
using diagrams, two-way tables, graphs, flow charts, or formulas).
1, 2, 3
1
2, 3
1
A. Apply mathematics to solve problems arising in everyday life, society, and the
workplace.
D. Interpret results in the context of a situation.
Modeling and
Data Analysis
(drawn across
content
domains)
Communicating
Reasoning
(drawn across
content
domains)
Items
DOK
1–2
8-10
B. Construct, autonomously, chains of reasoning to justify mathematical models used,
interpretations made, and solutions proposed for a complex problem.
E. Analyze the adequacy of and make improvements to an existing model or develop a
mathematical model of a real phenomenon.
2, 3, 4
C. State logical assumptions being used.
F. Identify important quantities in a practical situation and map their relationships (e.g.,
using diagrams, two-way tables, graphs, flow charts, or formulas).
1, 2, 3
1
G. Identify, analyze, and synthesize relevant external resources to pose or solve
problems.
3, 4
0
A. Test propositions or conjectures with specific examples.
D. Use the technique of breaking an argument into cases.
2, 3
3
2, 3, 4
3
2, 3
2
B. Construct, autonomously, chains of reasoning that will justify or refute propositions or
conjectures.
E. Distinguish correct logic or reasoning from that which is flawed, and—if there is a flaw
in the argument—explain what it is.
C. State logical assumptions being used.
F. Base arguments on concrete referents such as objects, drawings, diagrams, and
actions.
PT
Total
Items
1
1–3
0-2
8-10
Range Achievement Level
Descriptors
• Grade and content-specific descriptors
• Used by test developers to guide item
writing
• Describe the cognitive and content rigor
that is encompassed within particular
each of the achievement levels
• The knowledge, skills, and processes
described in the range ALDs are ones
that are expected of students
Range Achievement Level Descriptors
(Grade 8 Example)
OVERALL CLAIM: Students
can demonstrate progress
toward college and career
readiness in mathematics.
CLAIM 1: Students can
explain and apply
mathematical concepts and
carry out mathematical
procedures with precision
and fluency.
POLICY ALD: The Level 1 student
demonstrates minimal understanding
of and ability to apply the
mathematics knowledge and skills
needed for success in college and
careers, as specified in the Common
Core State Standards.
POLICY ALD: The Level 2 student
demonstrates partial understanding of
and ability to apply the mathematics
knowledge and skills needed for
success in college and careers, as
specified in the Common Core State
Standards.
POLICY ALD: The Level 3 student demonstrates
adequate understanding of and ability to apply the
mathematics knowledge and skills needed for success in
college and careers, as specified in the Common Core
State Standards.
CONTENT ALD: The Level 1 student
can minimally explain and in a
minimal way apply mathematical
concepts. The Level 1 student
interprets and carries out
mathematical procedures with
minimal precision and fluency.
CONTENT ALD: The Level 2 student can
partially explain and partially apply
mathematical concepts. The Level 2
student interprets and carries out
mathematical procedures with partial
precision and fluency.
CONTENT ALD: The Level 3 student can adequately
explain and adequately apply mathematical concepts.
The Level 3 student interprets and carries out
mathematical procedures with adequate precision and
fluency.
POLICY ALD: The Level 4 student demonstrates
thorough understanding of and ability to apply the
mathematics knowledge and skills needed for success
in college and careers, as specified in the Common
Core State Standards.
CONTENT ALD: The Level 4 student can thoroughly
explain and accurately apply mathematical concepts.
The Level 4 student interprets and carries out
mathematical procedures with high precision and
fluency.
Concepts and Procedures: Domain #1
Expressions and Equations
RANGE ALD
Target B: Work with
radicals and integer
exponents.
Level 1 students should be
able to identify and calculate
square roots of familiar
perfect squares and calculate
the square of integers. They
should be able to translate
between standard form and
Level 2 students should be able to identify and
calculate the cube root of familiar perfect cubes and
calculate the cube of integers. They should be able
to use appropriate tools (e.g., calculator, pencil and
paper) to translate large or small numbers from
scientific to standard notation. They should be able
to work with and apply the properties of integer
exponents of degree 2 or less in order to produce or
identify equivalent numerical expressions.
Level 3 students should be able to identify that the
square root of 2 is irrational, calculate or approximate to an
appropriate degree of precision the square or cube of a rational
number, solve quadratic and cubic monomial equations, and
represent the solution as a square or cube root, respectively. They
should be able to work with and perform operations with scientific
notation and work with and apply the properties of integer exponents
in order to produce or identify equivalent numerical expressions.
Level 4 students should be able to
use scientific notation and choose units of
appropriate size for realistic
measurements, solve binomial quadratic
and cubic equations, and represent the
solution as a square or cube root,
respectively.
Level 2 students should be able to compare two
different proportional relationships represented in
different ways. They should also be able to calculate
the slope of a line and identify the y-intercept of a
line.
Level 3 students should understand that slope is a unit
rate of change in a proportional relationship and convert proportional
relationships to linear equations in slope-intercept form while also
understanding when and why the y-intercept is zero. They should also
be able to use repeated reasoning to observe that they can use any
right triangle to find the slope of a line.
Level 4 students should be able to
use similar triangles to explain why the
slope is the same between any two
distinct points on a non-vertical line in a
coordinate plane.
scientific notation.
RANGE ALD
Target C: Understand
the connections
between proportional
relationships, lines, and
linear equations.
Level 1 students should be
RANGE ALD
Target D: Analyze and
solve linear equations
and pairs of
simultaneous linear
equations.
Level 1 students should be
able to graph a proportional
relationship on a coordinate
plane.
Level 2 students should be able to analyze and solve Level 3 students should be able to classify systems of
linear equations as intersecting, collinear, or parallel; solve linear
systems of linear equations graphically by
systems algebraically and estimate solutions using a variety of
understanding that the solution of a system of linear
approaches; and show that a particular linear equation has one
equations in two variables corresponds to the point
solution, no solution, or infinitely many solutions by successively
transforming the given equation into simpler forms until an
of intersection on a plane. They should be able to
equivalent equation of the form x = a, a = a, or a = b results (where a
solve and produce examples of linear equations in
and b are different numbers).
one variable with rational coefficients with one
They should be able to solve and produce examples of linear
equations in one variable, including equations whose solutions
solution, infinitely many solutions, or no solution.
WestEd.org
require expanding expressions using the distributive property and
able to
solve linear equations in one
variable with integer
coefficients.
Level 4 students should be able to
analyze and solve problems leading to two
linear equations in two variables in
multiple representations.
Threshold Achievement Level
Descriptors
• Threshold ALDs are used to guide standard setting.
• Define the minimum performance required for
meeting a particular achievement-level
expectation.
• Reflect the knowledge, skills, and processes that
are expected of students.
• The knowledge, skills, and processes in ALDs are
cumulative.
• The student who has achieved the threshold Level
3 is assumed to have the knowledge, skills, and
processes of the range Levels 1 and 2 ALDs.
Threshold Achievement Level
Descriptors (Grade 8 Example)
THRESHOLD ALD
Expressions and
Equations Targets
B, C, and D
WestEd.org
The student who just enters
Level 2 should be able to:
Find the cube of one-digit
numbers and the cube root of
perfect cubes (less than
1,000).
Use appropriate tools (e.g.,
calculator, pencil and paper)
to translate large numbers
from scientific to standard
notation.
Identify the y-intercept and
calculate the slope of a line
from an equation or graph.
Graph a system of linear
equations and identify the
solution as the point of
intersection.
The student who just enters Level
3 should be able to:
Solve simple quadratic monomial
equations and represent the
solution as a square root.
Work with and perform
operations with scientific notation
of large numbers.
Identify unit rate of change in
linear relationships (i.e., slope is
the rate of change).
Solve linear equations with
rational number coefficients,
including equations whose
solutions require expanding
expressions using the distributive
property and collecting like terms
and equations with infinitely many
solutions or no solution.
Solve a system of linear
equations with integer coefficients
using an algebraic strategy.
The student who
just enters Level 4
should be able to:
Write a system of
two linear
equations with two
variables to
represent a
context.
Making Sense of the Data
Data Examination
• Comprehensive, complex, difficult
process
– Not a check list
– Not meant to be completed at a single staff
meeting or a single PD event
• Focus on improving learning
– Not solely about increasing scores
• Collaborative process that requires
–
–
–
–
Honesty
Willingness to commit the time
Ability to handle ambiguity
Patience
Data Examination
• There are multiple ways to look at data
– Maximize data pictures and perspectives
– Different displays can elicit different responses
• Every district/site has its own
culture/climate/context –
– Examine data within the particular context of
district/site to emphasize particular point or
encourage a particular conversation
• Determine what you want audience to
walk away with
– What’s the next conversation?
Data Examination
 Use the tools available to manage the data.
 Use tools effectively to reflect on instructional
practice and standards implementation.
 Use data to inform practice and improve
teaching and learning.
 Develop a template to meet the needs of
your school context
Smarter Balanced Mathematics Threshold
Scale Scores
Grade
Level 2
Level 3
Level 4
3
2381
2436
2501
4
2411
2485
2549
5
2455
2528
2579
6
2473
2552
2610
7
2484
2567
2635
8
2504
2586
2653
11
2543
2628
2718
31
Achievement Level Ranges
Mathematics
Level 1
Level 2
Level 3
Level 4
Grade
Equal to or Below
From
To
From
To
Equal to or
Above
3
2380
2381
2435
2436
2500
2501
4
2410
2411
2484
2485
2548
2549
5
2454
2455
2527
2528
2578
2579
6
2472
2473
2551
2552
2609
2610
7
2483
2484
2566
2567
2634
2635
8
2503
2504
2585
2586
2652
2653
11
2542
2543
2627
2628
2717
2718
Managing the Numbers
Comparative Template Using Threshold Scores
and Claim Scores
Spring 2015
Average
Smarter
Scale
Balanced
Score
Grade
Content
or
Area
Group
Perf.
Level
Threshold
Score
Or Range
Below Above
Expect
ed
Strongest
Claim Area
Weakest
Claim
Area
6
Math
2475
2 (Low)
2473
(2473-2551)
Below
Concepts
Modeling
7
Math
2565
2 (High)
2484
(2484-2566)
8
Mathematics Data Template – Looking for Patterns
Grade
Level
Group
Average Scale Score and
Performance Level for Score
Percentage in Each Claim Performance Category
Perf. Level
Claims
2414
Nearly_Met
Concepts and Procedures
39
41
20
Problem Solving and Modeling & Data Analysis
39
44
18
Communicating Reasoning
28
53
19
Concepts and Procedures
51
38
11
Problem Solving and Modeling & Data Analysis
52
40
8
Communicating Reasoning
37
53
10
Concepts and Procedures
29
42
29
Problem Solving and Modeling & Data Analysis
33
40
27
Communicating Reasoning
20
50
29
Concepts and Procedures
45
41
14
Problem Solving and Modeling & Data Analysis
44
44
12
Communicating Reasoning
32
54
13
Concepts and Procedures
16
39
45
Problem Solving and Modeling & Data Analysis
14
45
41
Communicating Reasoning
11
48
41
Concepts and Procedures
56
37
8
Problem Solving and Modeling & Data Analysis
56
39
5
Communicating Reasoning
42
52
6
Concepts and Procedures
45
41
14
Problem Solving and Modeling & Data Analysis
45
43
12
Communicating Reasoning
33
54
13
2388
Nearly_Met
African American
2433
Nearly_Met
Asian
2402
Nearly_Met
Hispanic
2463
Met
White
2384
Nearly_Met
English Learners
2401
Socio-economically
Disadvantaged
% Below Standard
% At/Near Standard
% Above Standard
Average Scale Score
All Students
3rd
Percentage of Students in Each Claim
Performance Category
Nearly_Met
Mathematics Data Template –
Looking for Patterns
Percent of students at each performance level on 2015 CAASPP Math, by claim area and grade
County:
District:
Claim Area 1:CONCEPTS & PROCEDURES
11
35
11
19
38
40
41
54
AA
38
32
Hispanic
22
24
White
Asian
AA
38
39
37
35
25
26
White
Asian
34
38
57
41
17
49
Hispanic
3rd
10
25
65
AA
12
36
53
Hispanic
4th
Below Standard
32
31
38
41
30
28
White
Asian
26
25
49
44
26
31
White
Asian
5th
At or Near Standard
Above Standard
A=African American
Information from some subgroups are suppressed and not shown on the chart due to a small sample size (<=10)
Claim Area 2: PROBLEM SOLVING & MODELING/DATA ANALYSIS
8
38
54
AA
13
32
30
48
45
20
25
White
Asian
48
39
Hispanic
6
43
51
AA
3rd
11
26
27
48
50
48
23
25
White
Asian
6
32
62
41
Hispanic
AA
10
38
52
Hispanic
4th
Below Standard
At or Near Standard
5th
Above Standard
A=African American
Information from some subgroups are suppressed and not shown on the chart due to a small sample size (<=10)
Guiding questions to Help Reflect on the Data
- Observations Only
– What do you notice about the overall scores?
– What surprises you? What is consistent with your expectations or
predictions?
– Did you expect these scores; if not what did you expect?
– How do the grade level scores compare with the Smarter threshold
scores?
– Which scores look most noticeably different from the population
being studied?
– What differences in scores exist with your sub-groups all student
group?
– Which claims within the content area are higher performing?
– Which claims within the content area are weaker performing?
– Are there any patterns that emerge, by grade level, by subgroup, by
content area, by claim area?
– Describe data patterns that you observe.
Practice With Data
Protocols and
Displays
 Review your data and tools for
analysis
 Use the example protocols and
guiding questions to make
observations about the data
 Draft a data template to be
used with your district/site staff
Moving Beyond the Scores
– Reflection
Improving Teaching and Learning– Moving
Beyond the Numbers
• Data (scores) are necessary, but not sufficient
•
•
Data rarely provide ready answers
Be aware of data stall
Always consider the culture, climate and context
of your school/district.
• Reflect on what you can control to move beyond
the scores:
•
•
•
•
•
Practice
Policies and Procedures
People
Programs
Guiding questions to Help Reflect on Practice and
Implementation – Getting to Cause and Possible
Next Steps
–
–
–
–
–
–
–
–
–
–
–
–
What curriculum and materials do we have to address these areas of strength and
areas of need for the coming year?
What might the implications be for instructional practice?
What might the implications be for student learning?
How do I find examples of student work that address the target area?
What evidence do I need during classroom instruction to know that my students are
making progress toward meeting the targets for each claim?
Where can I find examples of evidence to meet the targets for each claim?
How might I use the performance tasks to illustrate student performance; to guide
the direction of intervention given the data observed?
How do these results affirm areas where instruction was provided?
How do these results point to gaps in instruction?
What would you consider is the single-most important factor contributing to the
apparent successes/needs as indicated by the scores?
Looking at claim area where large percentage of students are below standard,
what instructional strategies might we change?
How could instructional time be adjusted to meet the needs of students and close
the gaps observed in the data?
Data Examination – Beyond the Numbers
Example Reflection Template
Claim Area
Concepts
and
Procedures
Group/Grade/Program
Comparison
Data Observations
Guiding Questions
Grade 3 to Grade 4
How do these results
affirm areas where
instruction was
provided?
White to Asian
What might the
implications be for
instructional practice?
All to English learners
How do I find
examples of student
work that address the
claim and/or target
area?
Reflections
Possible
Action/Next Step
Table
Talks
How do you propose to have your
site leaders interact with the data?
Think about how you might use
these documents, templates and
displays to advance the
conversation about the data
patterns you have observed.