Exploring Assistive
Technology Issues in Reading
Assessment
ATIA 2010 Orlando:
Jason Altman, NCEO
Technology Assisted Reading Assessment
Statement of Problem
• Little evidence that technology improves
validity of tests for students w/ VI
• One reason is that students’ experience
and exposure to AT varies
• TARA seeks to develop a test of student
proficiency with AT
• For use prior to using AT on large-scale
reading assessments
Technology Assisted Reading Assessment
Preliminary Studies: The
State of AT Use with VI
Students
• Study #1: TVI survey (presented at
ATIA ’09)
• Study #2: TVI follow-up interview
• Study #3: Student Observational
Interview
Technology Assisted Reading Assessment
Revisiting Teacher Survey
• AT use is a large part of instruction for
students with VI
• TVIs take a “blended” approach to teaching
reading (e.g., using a variety of modalities)
• Standardization of assessments may be a
challenge because students use different
technology for different purposes
Technology Assisted Reading Assessment
More on Teacher Survey
• Reading and technology are intertwined
activities for students with VI
• The larger the caseload, the less likely
students will have individualized AT
experiences
• Teachers who focus on foundational
reading skills tend to provide fewer
technology opportunities for students.
Technology Assisted Reading Assessment
Overview: Results of Teacher
Interview Study (#2)
• 1st study showed “accommodations” or
“modifications” are common for students w/
VI
• Both high- and low-tech approaches to
teaching reading are used by TVIs
– Most likely for different purposes and at different
times
– Such flexible approaches to reading and
situational use of technologies create
challenges for standardization of assessments
Technology Assisted Reading Assessment
Method
• Interviewed 27 TVIs
• At least two TVIs were interviewed in the
following categories:
–
–
–
–
TVIs who work in Schools for the Blind
TVIs who are itinerant teachers
TVIs who teach students with low-vision
TVIs who teach students who are completely
blind
– TVIs who do not teach braille
– TVIs who teach braille
Technology Assisted Reading Assessment
Instrument and Procedures
• All interviews were conducted by telephone
• In most cases, there were two (and up to
four) researchers present during telephone
interviews
• Research participants were asked to answer
a series of structured interview questions
• Interviews generally lasted 30 minutes,
although some interviews lasted as long as
one hour
Technology Assisted Reading Assessment
Results
• All 27 TVIs indicated that they would use a
technology-based assessment if one were
readily available
• All TVIs who taught students with low-vision
used at least some form of magnification
equipment
• TVIs who taught students who use braille
readers also reported using a variety of
technologies such as audio devices and
braille note takers
Technology Assisted Reading Assessment
Specific Products
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Alpha Smart
Bookport
Braille and Speak
DAISY Readers
Extreme Reader
Freedom Box
IPod
JAWS
Kurzweil
MAGIC
OpenBook
OutSpoken
PacMate
Zoomtext
Technology Assisted Reading Assessment
Technology and reading
• Some TVIs used technology fluidly in instruction
• Those who had wide-ranging knowledge of the
tools available (and had access to such tools)
match instructional tasks with technology tools.
• 5 TVIs spent most of their time teaching students
how to use CCTVs and were hesitant to engage
in teaching students using computer-based tools
(especially those that added voicing to files)
• In all cases, however, TVIs used some form of
technology to assist teaching
Technology Assisted Reading Assessment
Formal AT assessment
protocols
• About 1 of 4 teachers were using formal assessment
protocols/tools to assess students’ assistive technology
use
• Among those most frequently used are the SETT
Protocol and the Texas School for the Blind Assistive
Technology Assessment Summary for Students with
Visual Impairments
• One teacher who used the Texas assessment also
mentioned using the Georgia Project for Assistive
Technology’s (n.d.) Assistive Technology Evaluation
and the Wisconsin Assistive Technology Initiatives
(2004) WATI Assistive Technology Checklist
Technology Assisted Reading Assessment
Assessment: Proficient
Reader Who Uses AT
• Teachers recognized that “advanced proficiency” in
technology-assisted reading meant the ability to move fluidly
between technologies in order to optimize the potential of
various technologies
• A proficient user of technology-assisted reading would be a
reader who read and understood grade level material
– Most TVIs were concerned that their students be
assessed on grade-level, but with a fair opportunity to
demonstrate grade-level skills
– TVIs were concerned about the validity of tests
Technology Assisted Reading Assessment
Discussion
• There is a desire for a formal
assessment or evaluation of student
skills
• Should discriminate between various
levels of proficiency
• There are a small number of
assessments/frameworks in use
Technology Assisted Reading Assessment
Activity
TVIs in this room: Had you been
participants would have findings
changed
Technology Assisted Reading Assessment
Student Observational
Interview: Study #3
Technology Assisted Reading Assessment
Method: Sample
• Students with visual impairments in grades 610
• In total, we interviewed 18 students for this
study from 5 states across the US
• We sampled students from both general
education school systems (n=9) and state
schools (n=5) for the blind
• Four additional students were educated at
state schools through general education
classes in a nearby public school
Technology Assisted Reading Assessment
Method: Instrument and
Procedures
• Students participated in “observational interviews”
facilitated by three researchers on the project
– Observational interviews were a hybrid between
verbal interviews, where respondents describe
phenomenon (Bogdan & Biklen, 1998) and
cognitive interviews, where interviewees
participate in an activity and describe their
thoughts and actions (Ericsson & Simon, 1993)
Technology Assisted Reading Assessment
Procedures (cont.)
• During these interviews, students were
asked to complete reading tasks using AT
– Afterward, students were asked how they
use AT in the reading process, including
how to download files, retrieve
information from printed material, and to
explain preferences
• Each interview lasted between 30 and 60
minutes
Technology Assisted Reading Assessment
Results of Student Study
• Demands of a rigorous high school
curriculum at times left students without
enough time to explore and become
proficient in new and possibly more efficient
technology strategies
• For students with visual impairments reading
large print or Braille, the time it takes to
complete a task is often far greater than that
for their peers with full functioning vision
Technology Assisted Reading Assessment
Results
• 11 of the 18 students could read large
print – 3 of them regular print as well,
and many were audio book users.
• The remaining 7 students all read braille
– and 3 of the large print readers also
read braille.
• 13 students used audio books to access
print no matter what their primary
method of print reading was
Technology Assisted Reading Assessment
Results
• 8 students used technology for reading
Braille – 5 used Braille Note
• For magnification, students used a variety of
technologies, from simple handheld
magnifiers to computer-based products
• Zoomtext (most often also with speech),
CCTV
• Technology use for 7 depended on the
reading situation
Technology Assisted Reading Assessment
Results
• Technologies are sometimes
unreliable (e.g., not all formats work
for internet-based text), take time to
learn, and do not always accompany
the student beyond the school walls
(e.g., many students cannot bring AT
devices home to assist with
homework)
Technology Assisted Reading Assessment
Student characteristics and
independence
• Students varied from completely relying on
help from sighted teachers, para-professionals,
and peers to being assertively independent
• Most students described their dependence on
others as situational
• One student mentioned that he is not afraid to
ask his friends or family to read things for him
• Another student, said that help is not needed
Technology Assisted Reading Assessment
Technology Choices
• Depended on availability of technology in student
location
• For example, students at a state school typically
would have more access to technology than a
student in a rural public school that did not serve
a large population of students with visual
impairments
• Access at home may differ from access at school
– One student, who was on the academic honor roll at
school, often stayed after school and came in before
school every morning to finish homework
Technology Assisted Reading Assessment
Choice of specific technology
• The student’s TVI or para-professional
generally was very involved in the decision
• Some students attended technology
conferences
• It is important to note that another person
(teacher, parent, or para-professional)
needed to learn the technology along with the
student
• TVIs had to be careful not to over-extend
themselves with multiple new technologies
Technology Assisted Reading Assessment
Role of expected change in
situation
• Decisions were impacted by the nature
of the visual impairments
• Changes in student vision and forecasts
for future change often called for a
change in products
– For example, one student used a CCTV up
until 6th grade but stopped using it because
there was a decrease in his field of vision
and it was no longer practical
Technology Assisted Reading Assessment
Difficulty with AT
• The amount of time that must be dedicated to
training and practice in using the equipment
sometimes outweighed the potential benefit of the
technology.
• Another complication was consistent access to
technology
• Students were sometimes forced to share
technology with several other students, including
CCTVs
Technology Assisted Reading Assessment
Other Hangups
• There was also difficulty in transporting, setting
up, and stowing equipment
• Upon learning a particular technology platform,
students were hesitant to move to other
platforms
• Equipment malfunction, or general hang-ups and
glitches also were sometimes troublesome for
students, teachers, and paraprofessionals alike.
• One student stated that the Web sites that
contained many graphics can be most
problematic
Technology Assisted Reading Assessment
Testing and Large-Scale
Assessment
• Multiple students reported using enlarged text
to access the items and then having the
paraprofessional fill in the bubble answer
sheet
• At least one student was able to take a math
test using a computer; however, he was
required to take the reading test using a
paper and pencil version
• Other students used standard question and
answer forms, accessing them via CCTVs
Technology Assisted Reading Assessment
Discussion
• What do these findings mean for
TVIs?
• Does the situation play out similar in
your classroom?
• Would you agree that your student’s
participation in large-scale
assessment could be more
meaningful?
Technology Assisted Reading Assessment
Conclusions
• As statewide assessment policy is still
in flux (Thurlow et al., 2006), it is
important for research to continue in
the area of technology-assisted
reading to provide stakeholders with a
clearer picture of the avenues that
students with visual impairments take
to access reading and
demonstrate their reading skills
Technology Assisted Reading Assessment
Also at this Conference
• Also see today from 2:45-3:45
Bonaire 8 EDU-15
Development of an Accountability Test
to Measure a Student’s Ability to
Access Text with AT
Presenting: Laitusis, Hall
Technology Assisted Reading Assessment
Next Steps
• Field Testing the new Technology
Based Reading Assessment in school
classrooms
• We need your help!
– Administrators of the field test
– Students who can participate
Technology Assisted Reading Assessment
Thank you! For more information…
National Center on Educational Outcomes
University of Minnesota
612-626-1530
http://www.nceo.info
Jason Altman
Technology Assisted Reading Assessment
altma014@umn.edu
35