Special thanks to Cori Wixson, Tanya Talapatra,
and Tamika LaSalle for their assistance in coding
the think-aloud videos.


Purpose: To evaluate the influence of test
item modifications on students’ problemsolving and test-taking behaviors.
Our study involved three components:
1. Students completed a series of 16 assessment
items (8 reading; 8 mathematics).
2. Students were asked to think aloud as they
completed or solved these items.
3. We also asked follow-up questions about
students’ perceptions of the assessment items.
(Johnstone, Bottsford-Miller, & Thompson, 2006; Branch,
2000).
1
Test A
Test B
2
3
4
5
6
7
8
X
X
X
X
X X X X
X = Item modifications used.
Test A
group
Test B
group
Total
Students without
disabilities
2
1
3
Students with
disabilities (not
eligible for AA-MAS)
1
2
3
Students with
disabilities (eligible
for AA-MAS)
1
2
3




We explained the think-aloud procedures,
had the students restate their understanding
of the process, and modeled thinking aloud
on a practice item.
We used a script adapted from a study
conducted by Johnstone, Bottsford-Miller,
and Thompson (2006).
Students were prompted only when they were
silent for 10 consecutive seconds.
If students verbalized infrequently, we
reminded them to “keep thinking aloud” or
“keep talking.” Otherwise we generally did
not give encouragement or support.




Most SWDs (67%) saw the visuals as being helpful
and providing support on reading questions and
passages.
100% of the students without disabilities indicated
the pictures made no difference in understanding
the reading questions or passages.
Students with (50%) and without disabilities (67%)
generally saw the visuals and graphs as being
helpful and providing support on math items.
…However, 33% of SWDs indicated that the
visuals/graphs were distracting or made it harder
to answer the questions.


Student with disability (eligible for AA-MAS): "The one
talking about the $100 bills…well it showed me, and I
was understanding how it goes with what it was talking
about, and I looked at it and it helped me even more.“
Student with disability (not eligible for AA-MAS): “When
people do math, they're working on a sheet and what's
the point of looking at a picture. It doesn't really help
you. For example, on (questions) #1 and #2, those two
pictures were really messing me up.”



Students without disabilities (67%) and
SWDs who were eligible for AA-MAS (100%)
generally preferred test directions that were
explicit, such as “Read…then answer….”
Some students indicated that the less
explicit directions (i.e., “Use the passage…”)
might encourage test takers to skim rather
than read closely.
The majority of students from all groups
(78% of the total) felt the use of bold type to
identify key terms was helpful in answering
the reading items.



SWDs (with one exception) perceived no
difference in difficulty between items having 3
or 4 possible answers on reading items.
Conversely, 67% of the students without
disabilities identified the 3-answer
modification as making the reading items
easier.
The results suggest that this modification did
not affect either groups’ performance on most
reading items (e.g., only one item [“Pesticides”]
had a discernable difference in student
accuracy between modified and unmodified
versions).
Pesticides
In the late 1980s, farmers began to use a pesticide
to control insects that harmed their cotton crops.
This problem was solved. However, an insect group
that pollinated the corn crops was also injured.
Without pollination the corn kernels did not fully
develop. This affected the corn harvest on which
the farm families had come to depend. What is not
mentioned as one effect of pesticide usage?
A.
B.
C.
D.
soil contamination
destruction of pests
destruction of friendly insects
crop losses
Passage revised
Pesticides
In the late 1980s, farmers began to use a chemical pesticide. It was used to
control insects that harmed their cotton crops. This solved one problem, but
caused another. An insect group that pollinated the corn crops was also
harmed by the pesticide. Without pollination the corn kernels did not fully
Question
separated
develop. This decreased the corn harvest.
What is NOT mentioned as one effect of using chemical pesticides?
A.destruction of the soil
B.destruction of pests
3 answer choices;
all use word
“destruction”
C.destruction of friendly insects
Picture added
from text;
“NOT” bolded
and
capitalized;
and the word
“chemical”
added



Students without disabilities (67%) and noneligible SWDs (67%) generally indicated 3
answer choices made the math items easier.
Some students in these groups appeared to use
the possible answer choices to help solve math
items, but it was not clear that they used this
same strategy in reading.
For the SWDs eligible for AA-MAS, the 3answer choice modification was less likely to be
identified as helpful, but it did seem to make a
difference in performance on one particular
item (i.e., “scientific notation”).


Student with disability (not eligible): “The
bold type made (the answer) easier to find,
but it didn’t help to understand the passage.”
Student without disability: "If you didn't get
the answer right the first time, you know you
only had 3 choices to go back and look at,
instead of 4"


Most students (including 2/3 of SWDs)
found the traditional format for the analogy
easier (i.e.,“meteor:space::dolphin:_______”).
Some students indicated they had been
taught analogies using this format and it
was familiar to them.
This was supported by the results as SWDs
correctly answered all the traditional
analogy items. SWDs missed items with a
modified analogy format (i.e., “meteor is to
space as dolphin is to ___”) 40% of the time.
Group
% of
Items
Correct
Time
spent
per item
(mean)
Miscues
on
passages
(mean)
Fluency on
passages
(mean)
Researcher
prompts per
item
(mean)
Students
without
disabilities
Original
Items
83.3%
79.6 s
2.7
153.3 wpm
.49
Modified
Items
83.3%
51.0 s
1.5
163.3 wpm
.29
SWDs
(not
eligible)
Original
Items
83.3%
123.8 s
9.8
92.6 wpm
.65
Modified
Items
75.0%
100.5 s
9.0
78.7 wpm
.28
SWDs
(eligible
for AAMAS)
Original
Items
66.7%
149.4 s
12.3
86.9 wpm
.81
Modified
Items
75.0%
98.5 s
9.5
85.8 wpm
.28
Group
% of
Items
Correct
Time
spent per
item
(mean)
Researcher
prompts per
item (mean)
Problem Solving Strategies
Used on Items
Correct
strategy
used
Incorrect
strategy
used
Appeared
to guess*
Students
without
disabilities
Original
Items
66.7%
65.8 s
.33
66.7%
(8)
25.0%
(3)
16.7%
(2)
Modified
Items
50.0%
54.1 s
.08
50.0%
(6)
33.3%
(4)
41.7%
(5)
SWDs
(not
eligible)
Original
Items
50.0%
125.2 s
.33
41.7%
(5)
50.0%
(6)
16.7%
(3)
Modified
Items
75.0%
126.2 s
.08
41.7%
(5)
50.0%
(6)
41.7%
(5)
SWDs
(eligible
for AAMAS)
Original
Items
33.0%
102.5 s
.58
25.0%
(3)
58.3%
(7)
50.0%
(6)
Modified
Items
50.0%
72.8 s
.08
8.3%
(1)
58.3%
(7)
83.3%
(10)



“Conservative” modifications were used in this study
and the effects (on student performance) generally
were modest. More “aggressive” modifications might
result in more robust effects.
SWDs often appeared unfamiliar with some concepts
(e.g., percentages) or incorrectly applied problem
solving strategies on math items. In these cases,
item modifications are unlikely to provide support
or facilitate access.
Reading fluency may be an issue for SWDs. In some
cases, SWD’s slower rates of reading resulted in
testing sessions that were almost twice as long as
their peers. How could (or should) technology be
used to address this barrier?



Students appeared more likely to guess on the
modified items. Could this be attributed to
having fewer answer choices?
In most cases, students in each group spent
less time and required fewer prompts on the
modified items. The difference was most
pronounced for the SWDs eligible for AA-MAS.
Conversely, oral reading fluency did not appear
to be influenced by the modifications made to
reading passages.

Johnstone, C. J., Bottsford-Miller, N. A., & Thompson, S.
J. (2006). Using the think aloud method (cognitive labs)
to evaulate test design for students with disabilities and
English language learners (Technical Report 44).
Minneapolis, MN: University of Minnesota, National
Center on Educational Outcomes. Retrieved from the
World Wide Web:
http://education.umn.edu/NCEO/OnlinePubs/Tech44/.

Branch, J. L. (2000). Investigating the informationseeking processes of adolescents: The value of using
think-alouds and think-afters. Library and Information
Science Research, 22(4), 371–392.