California Department of Education
Tom Torlakson, State Superintendent of Public Instruction
Assessment
Literacy Module
Unit 6: Identifying, Developing,
and Implementing Sound,
Quality Assessments
Welcome to Unit 6
The purpose of this unit is to help educators identify, select,
and develop high quality classroom assessments, implement
them at multiple levels to continuously monitor student
progress, and increase the accuracy of student performance
information.
2
Learning Objectives for Unit 6
By the end of this unit, participants will be able to:
Evaluate the quality of assessments and their
items
Recognize when linguistic complexity makes assessment
unfair
Describe a variety of checking for understanding strategies
3
Identifying, Selecting, and Developing
Appropriate Classroom Assessments
Refer again to the teaching-assessment cycle and the assessment
literacy attribute to be covered first in this unit:
4
Identifying, Selecting, and Developing
Appropriate Classroom Assessments
Key Concepts in Accurately
Measuring Student Performance
Educators need accurate and relevant student performance
information from high-quality assessments to make informed
instructional decisions. The quality of assessments rests on
three key concepts: validity, reliability, and fairness.
5
Key Concepts in Accurate
Measurement: Validity
Validity
Validity is the extent to which inferences and actions made on
the basis of assessment results are appropriate and backed by
evidence (Messick, 1989). It is the use and/or interpretation of
assessment data that is key to the concept of validity.
The validity of any assessment is the extent to which test
scores accurately reflect the relevant knowledge and skills
of test takers.
Discuss:
What are the actions and decisions you make based on
students’ assessment results?
Do any of the actions/decisions call for more valid
assessments than others?
6
Key Concepts in Accurate
Measurement: Validity
Validity
For the Smarter Balanced assessments, the relevant knowledge and
skills are defined by the CCSS. These assessments are developed
using the principles of Evidence-Centered Assessment Design
(ECD). The three basic elements of ECD are:
(1) stating the claims to be made about test takers,
(2) deciding what evidence is required to support the claims, and
(3) administering test items that provide the required evidence
(Mislevy, Steinberg, & Almond 1999).
Additional information about ECD is available at
http://padi.sri.com/downloads/TR9_ECD.pdf
7
Key Concepts in Accurate
Measurement: Reliability
Reliability
Reliability is the degree of confidence that both scores
and student performance are repeatable over time and
across different circumstances.
Repeatability of scores means that different scorers or the same
scorer at different times should assign the same score(s) to the
same piece of student work (O’Neill & Stansbury 2000).
Discuss:
How can teachers ensure that their classroom
assessments are reliable?
8
Key Concepts in Accurate
Measurement: Fairness
Fairness
“Test scores are fair when they yield score interpretations
that are valid and reliable for all students taking the test.
Regardless of race, national origin, gender, or disability,
academic tests must measure the same knowledge of
content for all students who take the test. Test scores must
not systematically underestimate or overestimate the
knowledge of students of a particular group.”
‒California Education Code Section 60208
9
Key Concepts in Accurate
Measurement: Fairness
Fairness
For the Smarter Balanced assessments, the Bias and
Sensitivity Guidelines helps to ensure that the evidence
provided by the items means the same thing for various
groups of test takers and allows the ECD to work as intended
(Educational Testing Service 2010).
For additional information, download the Smarter Balanced
Assessment Consortium: Bias and Sensitivity Guidelines at
http://www.smarterbalanced.org/wordpress/wp-content/
uploads/2012/05/TaskItemSpecifications/Guidelines/Biasan
dSensitivity/BiasandSensitivityGuidelines.pdf
10
Assessment Fairness and Bias
Educators have the greatest opportunity to improve
their assessments and directly impact students by
increasing the fairness of their classroom assessments.
“Assessment bias occurs whenever test items offend or
unfairly penalize students for reasons related to students’
personal characteristics, such as their race, gender,
ethnicity, religion, or socioeconomic status” (Popham 2003).
11
Assessment Fairness and Bias
Fairness includes:
1. Instructional Learning Targets: Communicating clear, specific learning
targets to students ensures they know what will and will not be assessed,
what method will be used, and how the assessment will be scored.
2. Opportunity to Learn: Providing all students equally with adequate time
and appropriate instruction will enable them to obtain mastery.
3. Prerequisite Skills and Knowledge: Ensuring students have the
prerequisite skills and knowledge is necessary to complete a task, but are
not the learning target itself. If learning targets are NOT related to writing or
reading comprehension, this source of unfairness must be reduced by:
a. Identifying the prerequisite skills and knowledge of the learning
target
b. Pre-assessing students’ abilities in the prerequisite skills and
knowledge, either formally or informally through assignments,
observations, or questioning
Source: McMillan 2000
12
Assessment Fairness and Bias
Respond to the following questions:
1. How does bias, opportunity to learn, and prerequisite skills
and knowledge impact students and the way they are assessed?
2. What groups of students are most impacted when
assessments are not fair?
3. What action can you take to ensure that all students have
adequate time and appropriate instruction to enable them to
achieve mastery?
4. What steps can you take to minimize bias in your
assessment?
13
Assessment Accessibility
To ensure that assessments are fair and accessible, it is
essential to allow all students to show what they know and
can do, particularly students with special learning needs such
as students with disabilities and English learners.
In California classrooms, a Universal
Design approach is being used to
modify instruction according to IEPs,
504 plans, and language needs.
Applying some of these processes
and approaches to assessment
takes time and practice.
14
Assessment Accessibility
Smarter Balanced uses an “Access by Design” approach,
which includes an array of universal digital tools and features
available to all students.
Examples include: Text-to-speech administration of the
mathematics test and ELA items, screen magnification, guided
line readers, and color options designed to increase contrast
and readability.
IEPs or 504 plans may specify tools and features such as
keyword translation glossaries, auditory calming, audio
captioning, administration in American Sign Language, or braille
administration and response formats.
Additional information about Universal Design is
available on The National Center on Universal Design for
Learning Web site at http://www.udlcenter.org/aboutudl.
15
Linguistic Accommodation
for English Learners
“Linguistic accommodation” is a research-based approach in
which the language in test items, directions, and/or selected
response choices is changed in ways that clarify and simplify
the text without simplifying or significantly altering the learning
target.
It is especially useful with assessments in content areas such
as math, science, social science, and technical subjects when
language may be irrelevant to the learning target.
To facilitate comprehension, the complexity of text is reduced
by shortening sentence length and complexity, using common
or familiar words and concrete language.
Source: Abedi, Courtney, Mirocha, Leon, & Goldberg 2005;
Abedi, Lord, & Plummer 1997; Rivera & Stansfield 2001
16
Linguistic Accommodation
for English Learners
The Linguistic Accommodation approach is aligned with the
Universal Design used in Smarter Balanced assessments.
Considering that 37 percent of California’s students come from
families that speak a language other than English, it is
important for educators to know the impact of linguistics on
classroom assessment (CalEdFacts 2011).
17
Linguistic Accommodation
for English Learners
Watch Kenji Hakuta of Stanford University discuss
how the needs of English learners are being considered
in the CCSS and the creation of new assessments:
“Kenji Hakuta on ELLs and the Common Core Standards”
http://www.youtube.com/watch?
eature=player_embedded&v=PY2fhP8Q0OY
18
Limited English Proficiency
Affects Learning and Assessment
“While ELL students are struggling to learn English, learning
content-based knowledge cannot occur at the same rate as for a
native speaker of English when that instruction is offered only in
English. Limited English proficiency may also make it difficult for
ELL students to benefit fully from the teacher's instructions and to
understand assessment questions. Therefore, limited English
proficiency affects both learning and assessment. To help close the
performance gap between ELL and non-ELL students both learning
and assessment conditions must be addressed."
‒Abedi 2007
Key question in deciding when to use linguistic accommodations in
creating tests:
Is the learning target in ELA or another content area?
19
Limited English Proficiency
Affects Learning and Assessment
If the learning target is not in language, assessmentliterate educators use strategies to reduce the
interference of unnecessarily complex language in test
directions and test items.
Even when language is the learning target,
assessment-literate educators make test directions
clearer and simpler.
20
Limited English Proficiency
Affects Learning and Assessment
Refer to the table of interferences and
accommodation strategies handout and complete the
steps on the next slide:
Handout
Analyzing Linguistic Interferences and Accommodations
21
Limited English Proficiency
Affects Learning and Assessment
1. Look at the Examples of Interferences on the next slide. If an
example matches an “Interference” in Column 1 of your handout,
write the letter of the example into the adjacent cell in Column 2.
Some examples will be used more than once.
2. In Column 4, use the corresponding “Accommodation Strategy”
from Column 3 and write in your idea to fix the Interference.
3. Compare your answers to the Completed Table handout.
22
Limited English Proficiency
Affects Learning and Assessment
Examples of Interferences:
A. Alba needed to know about how much the sum of 19.6, 23.8,
and 38.4 is. She correctly rounded each of these numbers to
the nearest whole number. What three numbers did she use?
B. Ted can no longer drive over 40 mph in his truck.
C. The weights of two objects were measured.
D. As long as you bring your own bedding, you can stay with us.
E. According to the article, what role did some Navajo speakers
play during World War II?
23
Limited English Proficiency
Affects Learning and Assessment
Compare your responses to the Completed Table
Discuss findings and comparisons within your group.
Handout
Completed Table: Analyzing Linguistic Interferences and Accommodations
24
Limited English Proficiency
Affects Learning and Assessment
Important: Linguistic accommodation is not in conflict with the
CCSS’ emphasis on text complexity and academic language
because it is NOT about “dumbing down” texts. It IS about access to
the CCSS for ALL students. The challenges for assessment-literate
educators are:
1. To be clear about when language is the learning target and
when it is not;
2. To recognize and eliminate unnecessarily complex language.
By using linguistic accommodation strategies to develop and
improve assessments, assessment-literate educators allow
students to dedicate their attention to show what they know
instead of translating text (Abedi, Lord, & Plummer 1997).
25
Limited English Proficiency
Affects Learning and Assessment
To explore accommodations options for your own
classroom, use a new copy of the Interferences and
Accommodations handout, and try the following:
1. Choose an assessment you are currently using. Look at the
directions, questions, and answer choices (if applicable). Using
the list of Interferences, how many can you spot? Use the list
of strategies to edit the assessment. Be clear about what is
being assessed — language or another content area?
2. For students with communication disorders and/or reading
disabilities, which linguistic accommodations are appropriate
for making assessments more accessible? Discuss how to
address linguistic accommodations in IEPs or 504 Plans. This
is especially critical for English learners with disabilities.
Handout
Interferences and Accommodation Strategies Template
26
Variations, Accommodations,
or Modifications
Becoming familiar with the principles of UDL, the diverse
needs of students, and the various accessibility options
available is the first step that educators can take to choose
fair and accessible assessments — so that ALL students have
the opportunity to show what they know and understand.
The choice of variations, accommodations, or modifications in
classroom assessments must be made thoughtfully, based on
the students’ individual needs.
If students with disabilities are entitled to variations in state
testing, these variations should be used routinely in the
classroom.
Assessment-literate educators understand the test variations
indicated for their students AND are able to carry out these
variations in their classroom instruction and assessment.
27
Selecting and Developing Appropriate
Classroom Assessments: Assessment Methods
Now that you understand the key concepts behind measuring
student learning, you are ready to develop sound, quality
assessments.
In Unit 2, you learned about two basic types of
assessment methods that might be used in your
classroom to gather information about student learning:
selected-response and constructed-response.
28
Selecting and Developing Appropriate
Classroom Assessments: Assessment Methods
Selected-Response Methods
In this short video, W. James Popham explains
relevant features of selected-response assessment
methods:
“Selected-Response Assessment”
http://www.k-state.edu/ksde/alp/module6/
Selected-response (SR) methods prompt students to select
one or more correct responses from a set of choices. Carefully
constructed SR items allow students to demonstrate complex
thinking skills such as developing comparisons or contrasts;
identifying cause and effect; identifying patterns or conflicting
points of view; categorizing, summarizing, or interpreting
information.
29
Selecting and Developing Appropriate
Classroom Assessments: Assessment Methods
Constructed-Response Methods
In this short video, W. James Popham explains
relevant features of constructed-response assessment
methods:
“Constructed-Response Assessment”
http://www.k-state.edu/ksde/alp/module8/
Constructed-response (CR) methods prompt students to
generate a text or numerical response in order to collect
evidence about their knowledge or understanding of a given
assessment target. CR items can be both short and extended.
Short items may require test takers to enter a single word,
phrase, sentence, number, or set of numbers, whereas
extended items require more elaborated answers and
explanations of reasoning. These kinds of CR items allow
students to demonstrate their use of complex thinking skills.
30
Selected-Response vs. ConstructedResponse Assessment Methods
Meet Mr. Bumble.
Mr. Bumble is a hard-working “every
teacher.” He struggles with how to
assess his students and often shows you
what not to do. You will learn about Mr.
Bumble’s exploits throughout this unit.
Mr. Bumble realizes that he needs to improve his classroom
assessments. He thinks the way to do this is to choose between
selected- or constructed-response methods. So, he compares the
two to find out which one is better for classroom assessments.
31
Selected-Response vs. ConstructedResponse Assessment Methods
Selected-response
Constructed-response
(e.g., multiple choice, true/false, matching)
(e.g., short answer, essay)
Advantages
• Easier to score
• Can be answered quickly
• Covers a broader range of
curriculum in a shorter time
• Allows students to demonstrate
complex, in-depth understanding
• Less likelihood of guessing correct
answer
• Motivates students to learn in a way
that stresses the organization of
information, principles, and
application
Disadvantages
• Constrains students to single
appropriate answer
• Encourages students to learn by
recognition
• Subject to guessing correct answer
• More time-consuming to score
• More time-consuming to answer
EUREKA! Mr. Bumble realizes there are advantages and disadvantages to
both methods. Now he’s confused about when he should use which one...
32
Selected-Response vs. ConstructedResponse Assessment Methods
If Mr. Bumble asked you which method to use, how
would you answer him?
Refer to the Target-Assessment Method Match table handout from
Unit 4. Recall that the key determination in selecting the type of
assessment method is how well it measures the type of learning
target.
Handout
Target-Assessment Method Match (from Unit 4)
33
Developing Better Assessments
Assessment-literate educators develop assessments that are
appropriate and yield accurate information on which they can
make sound instructional decisions.
Popham (2003) alerts educators to:
These roadblocks apply to any type of item.
34
Developing Better Assessments
Now let’s look at the directions and some of the items from
one of Mr. Bumble’s science tests. As you review them,
identify how many roadblocks you find.
Read sections of Mr. Bumble's exam on the following
slides and then learn about the roadblocks they contain, along
with tips for overcoming them.
To review the roadblocks and tips after the workshop, refer to
the online module at
http://www.myboe.org/portal/default/Content/Viewer/Content?ac
tion=2&scId=505706&sciId=15350
35
Mr. Bumble’s Science Exam:
African Elephant Unit
Directions:
"This exam consists of four parts. Each part contains different
types of items (for example: true/false, multiple-choice, and
essay). Each of the exam’s four parts covers one of the major
units you have studied. Work your way through the test
efficiently because there is a time limit. Don’t write on the test.
Good luck!"
Discuss the roadblocks that you encounter.
36
Mr. Bumble’s Science Exam:
African Elephant Unit
True/False:
T F "Several research studies show that adult elephants
always become domineering toward younger elephants
because of their inherited characteristics."
Multiple Choice:
The commonly recognized example of a pachyderm is an
a. elephant
b. turtle
c. lion
d. pigeon
37
Mr. Bumble’s Science Exam:
African Elephant Unit
Essay Prompt:
“Discuss elephants in Africa."
Apply the item-writing rules you have learned so far to fix
Mr. Bumble’s essay question. Then, compare your response
to how the expert would fix it.
38
Mr. Bumble’s Science Exam:
African Elephant Unit
To develop high quality assessments, assessmentliterate educators review all items and directions from the
perspective of students. Answering essay questions
mentally or in writing can help to uncover problems in
advance.
For more information on developing assessments,
including matching learning objectives with test items and
writing test questions aligned with Bloom’s Taxonomy, see
“Is This a Trick Question? A Short Guide to Writing
Effective Test Questions” at http://www.k-state.edu/ksde/
alp/resources/Handout-Module6.pdf.
39
Instructionally Embedded Formative Assessment:
Checking for Understanding Strategies
Here is where we are in the teaching-assessment cycle and
the assessment literacy attribute that will be covered.
40
Sources of Assessment Data
The graphic below shows the varying frequency and duration of
teaching-assessment cycles. This section addresses an effective
assessment strategy for shorter-cycle, daily, and minute-by-minute
formative assessment. Checking for understanding provides useful
assessment information for adjusting instruction in real time.
Sources of Assessment Data
Source: Heritage 2010
41
The Benefits of
Checking for Understanding
Checking for understanding is a formative assessment
practice where educators ask questions or give short
assignments and students select or construct their responses.
Assessment-literate educators use “end of class”
formative assessment results to guide planning for the
next day’s lesson, or embed the assessment seamlessly
into instruction to make adjustments “in the moment.”
42
The Benefits of
Checking for Understanding
The benefits of checking for understanding when responses
are obtained from all students:
Gives educators information quickly about the need to
differentiate instruction
Helps educators understand students’ background knowledge,
skills, and misconceptions
Models for students how to monitor their own understanding
The biggest payoff is the positive effect on student
achievement. Research findings support the boost in student
achievement when educators pause to check for understanding and
use the information to adjust their teaching (Fisher and Frey 2007).
43
The Benefits of
Checking for Understanding
Watch this scene in Mr. Bumble's math class right after
he completes an explanation of the statistical concept of the
mean. Is he really checking for understanding?
“Mr. Bumble Checks for Understanding”
http://goanimate.com/videos/08KkNc8Zt5_E?utm_source=linkshare
Later, when Mr. Bumble reviews the results of the unit test, he is
surprised to see that only one or two students understood the
concept of the statistical mean. Where did Mr. Bumble go wrong?
What assumptions did he make?
44
The Benefits of
Checking for Understanding
Nonverbal checks for understanding in the formative
assessment process include observing facial expressions and
body language, as well as establish classroom routines such
as hand signals, colored cards, or answers to selectedresponse questions.
What nonverbal cues can students give spontaneously that
indicate whether they understand?
45
The Benefits of
Checking for Understanding
Watch how an assessment-literate educator
checks for understanding nonverbally on a minute-byminute basis.
“Show your Cards”
http://successatthecore.com/teacher-development/featuredvideo.aspx?v=41
How is the data gathered from colored cards used to adjust
instruction?
46
Checking for
Understanding on a Daily Basis
The next four videos show how assessment-literate educators
can check for understanding on a daily basis where students
respond verbally or in writing.
As you watch, think about:
• How does the teacher ensure that checking for
understanding is a low-stress formative assessment?
• How can the data gathered from the assessment be used
to adjust instruction?
• How are students benefiting from the strategy? What
feedback are they getting?
• How would you use the strategy?
47
Checking for
Understanding on a Daily Basis
1. Stoplight
Watch an ELA teacher use exit slips to assess learning and
how students self-assess their learning:
“The Stoplight Method: An End-of-Lesson Assessment”
https://www.teachingchannel.org/videos/daily-lesson-assessment
2. Exit Cards
How does this strategy help teachers assess learning and
plan future math lessons?
“Assess and Plan with Exit Tickets”
https://www.teachingchannel.org/videos/teacher-assessmentstrategy?fd=1
48
Checking for
Understanding on a Daily Basis
3. Tiered Exit Cards
Watch the video to see how a secondary math teacher uses
differentiated exit cards to group students for the next day.
“Daily Assessment with Tiered Exit Cards”
https://www.teachingchannel.org/videos/student-daily-assessment
4. Text What You Learned
A Secondary English teacher uses texting to check for
understanding.
“Text What You Learned: Using Technology to Assess”
https://www.teachingchannel.org/videos/texting-to-assess-learning
49
Using Think-Pair-Share to
Check for Understanding
“Think-Pair-Share” is a peer-to-peer discussion strategy where students
discuss their constructed responses to a prompt or question before
sharing with the whole class. It is a strategy that engages every student
so that checking the understanding of the whole class can be done
efficiently. The following steps outline the strategy (Fisher & Frey 2007).
1. Pose a question, prompt, reading, visual, or observation.
2. Allow a few minutes to individually THINK about the question.
3. PAIR up students with designated partners to discuss their
respective responses, comparing thoughts and agreeing on the
best responses.
4. After an appropriate length of time, ask the pairs to SHARE
their thinking with the whole class. Students need specific
feedback on speaking and listening especially since these skills
do not always have products for reviewing.
50
Using Think-Pair-Share to
Check for Understanding
View an example of a student rubric that elementary students
can use to assess and improve their Think-Pair-Share skills at
http://www.rcampus.com/rubricshowc.cfm?code=M3X599&sp=yes&
View an example of a rubric that teachers can use to assess
and improve Think-Pair-Share activities in their classrooms at
http://www.rcampus.com/rubricshowc.cfm?code=H46482&sp=true
More information about rubrics will be covered in Unit 7.
51
Set the Stage for Student SelfAssessment of Understanding
Many of these strategies ask students to self-assess their
level of understanding. When students overestimate their
understanding, assessment-literate educators:
1. Set a comfortable climate where honest self-assessment has
no consequences
2. Clearly communicate learning targets and criteria for mastery
3. Make learning public and welcome mistakes as opportunities
for everyone to learn
52
Reflect on Checking for
Understanding
Respond to the questions below and include other
thoughts you have about checking for understanding.
1. How often do you check for understanding? What strategy
do you use most often?
2. Are your students comfortable with publicly self-reporting
when they don’t understand? If so, how did you create the
appropriate classroom climate? If not, what questions do
you have for improving your classroom climate?
3. What ideas for your classroom did you get from watching
the videos?
53
Apply What You Learned
Using a learning target you have developed for
your grade level and/or content area, develop multiple
methods for assessing the target.
For assessments you can use formatively, develop a
selected-response item, a constructed-response item
or prompt (written or performance as appropriate), and
one check-for-understanding strategy you will try for
monitoring student progress in real time during instruction.
Then, develop a summative assessment you can use to
assess mastery of the learning target.
54
Summary of Unit 6
Let’s review the learning targets for Unit 6. Can you:
• Evaluate the quality of assessments and assessment
items?
There are three key concepts that distinguish quality
assessments -- reliability, validity, and fairness.
Assessment-literate educators are aware of these concepts and
select or develop classroom assessments of the highest
possible quality. Their assessments are fair and accessible to a
wide range of students while yielding the most accurate picture
of student achievement.
55
Summary of Unit 6
Can you:
• Recognize when linguistic complexity makes assessment
unfair?
When developing assessments it is important to identify
unnecessary linguistic features that slow down the reader, add
to their cognitive load, and interfere with the intended
assessment task.
There are strategies that can be used to mitigate these
interferences and improve the quality of assessments. Use of
these strategies will improve the fairness and validity of
classroom assessments.
56
Summary of Unit 6
Can you:
• Describe a variety of check-for-understanding strategies for
classroom formative use?
Checking for understanding is a formative assessment process
that provides information that educators can use to make
instructional adjustments on a minute-by-minute basis or on a
daily basis.
Checking for understanding can be done using a wide variety of
methods – verbal, nonverbal, constructed response, selected
response, etc.
57
Summary of Unit 6
Review the assessment item Mr. Bumble
developed to measure CCSS Math standard 1.0A.1
(using subtraction within 20 to solve word problems) and
respond to the questions:
Three alpacas were sold at auction by Stephan, and he went home
with five of the beasts. Therefore, with how many alpacas did
Stephan start?
1. Is this item fair? To what groups of students might it be biased
against? To what groups of students might it be biased in favor?
2. How would you improve the quality of this item?
3. Describe one strategy to check for understanding nonverbally.
4. Describe one strategy to check for understanding verbally.
58