DEVELOPING MOBILE APPS FOR
THE VISUALLY IMPAIRED
HEARTLAND COMMUNITY COLLEGE
Nicolaas tenBroek
Todd Simeone
WHO ARE YOU?
•
And why are you here?
HCC’S MOBILE CURRICULUM
Computer
Science I
Apps
Windows
Phone
Android
iOS
Games
Comm
Apps for
BLV
HOW WE BECAME INVOLVED IN
DEVELOPING APPS FOR THE BLIND
•
Independence Science
•
Desire for classroom apps on iOS
•
Chemistry
•
Basic Mathematics
•
Graphing
•
Advanced Mathematics
•
Other Educational Apps
ISSUES CONVERTING VISUAL INTERFACES
TO SUPPORT VOICEOVER
•
The amount of information conveyed by a visual interface is tremendous. Graphics and the
relationship between controls instantly convey functional information that would take
paragraphs of text to explain.
•
Animation and graphical gloss are often used to explain what is happening within an app.
•
Many common idioms simply do not transfer. For instance:
•
What does a “Back” button mean to someone who did not see the screen move “forward”?
•
“Pull down to refresh” is only meaningful because you saw the hidden setting when the
table bounced after scrolling to the top.
•
Items partially obscured by a translucent control instantly tell a sighted person that more
things are available, but overlapping controls will confuse a screen reader
ISSUES CONVERTING VISUAL INTERFACES
TO SUPPORT VOICEOVER
•
Most apps are designed to be used quickly and in non-ideal
circumstances, but screen readers take a long time to read, and can
interfere with a blind user’s interaction with the app, increasing
frustration and confusion.
•
Gestures that make sense when tied to the visual display (i.e. those
that mimic real world interactions) have to change when the screen
reader is active, and often lose all real world meaning. For instance,
if you used a swipe gesture to push a ball around a screen, that
would mimic a real-world interaction. With the screen reader turned
on, that same gesture will cycle between elements on the screen
(from top to bottom and left to right).
HOW PEOPLE WITH BLV INTERACT
WITH MOBILE DEVICES
•
Often listen to screen reader
•
Explore app by moving finger around
•
Use “List Mode” to cycle between items on screen
•
Tap off the selected control to use it
COMMON DESIGN APPROACHES AND
THE PROBLEMS WE FOUND
•
Just add screen reader support to existing design
•
Can make it very difficult to reliably hit controls
•
Lack of spacing can cause incorrect controls to be selected
•
List Mode often rearranges the controls which can quickly remove
all meaning as positioning of labels and images are often used to
inform the user of the meaning of other controls
•
Constant screen reading slows down the user’s interaction with
the app, making it even more difficult to use
COMMON DESIGN APPROACHES AND
THE PROBLEMS WE FOUND
•
Add menus
•
Complicates the interaction of the app
•
Tends to lead to desktop style designs which do not work in
the mobile world
•
Nested menus can cause users to become lost in the
structure, and makes finding features even more difficult
•
Contextual menus require users to see when and where
they appear
COMMON DESIGN APPROACHES AND
THE PROBLEMS WE FOUND
•
•
Add vibration
•
Used too often it will lose its meaning
•
Causes user’s fingers to go numb
Speak commands
•
Not an ideal computer interface in a classroom, nor frankly, any
public location
•
Computer comprehension of speech is not good enough to work on
its own. When it fails we fall back to on-screen controls, which a blind
person might easily miss.
BETTER DESIGN CHOICES
•
Create two completely separate interfaces, one for use with the screen reader,
and another for use without. This can drastically reduce the need to
compromise on the design. It also simplifies coding by reducing the need to
constantly test the environment and select the appropriate display or action
based on the state of the screen reader.
•
Always test with screen off!
•
Larger controls and much more space between controls than used with designs
for the sighted
•
Reduce colour, increase contrast
•
Reduce the amount of information read by the screen reader. If possible, add
beginner and pro modes that change the amount of information read.
BETTER DESIGN CHOICES
•
Do everything you can to speed the user’s interaction with the app. Remove
steps whenever possible.
•
Add sound, but carefully
•
•
Best used with headphones
•
Map sounds to 3D space (we use OpenAL)
•
Use sparingly
•
Don’t make user learn meanings of random sounds. Sounds should
convey meaning and match the physical world as much as is practical.
Add vibration, but sparingly
BETTER DESIGN CHOICES
•
Invert the position of important items. Layouts for the sighted often
place important items on the top of the view, but they should be at the
bottom for the blind.
•
Contextual menu display must be optimised to reduce the distance
from the control that initiated it. This can cause issues when scrolling
leaves a control near the edge of the screen.
•
Consider adding unique actions to the interface for the blind user. For
instance, you might relocate the currently selected item so that it is
always where the user taps if they tap an empty part of the screen.
•
Sleeves which cover the bezel are very useful.
INTERESTING APPROACHES THAT
SURPRISED US BY WORKING
•
Drawing - Used within a constrained environment,
‘drawing’ elements onto the screen works well, but
introduces issues if the drawn item does not exactly
match the finger movements.
•
Scrolling - When paired with other elements like
sound, user-driven scrolling can be effective.
However, you must ensure that scrolling ends with
complete elements still on the screen.
ISSUES WE ENCOUNTERED
•
There are some serious issues with VoiceOver
•
An item always selected, there is no ability to have nothing selected,
which means that something will always respond. This is not the
behaviour when VO is off
•
You cannot locally disable accessibility. Often things we needed to
do within our apps would have been much easier to do ourselves
rather than try to work within the constraints of VO
•
VO intercepts every user interaction before passing it to the app.
This makes overriding default gestures impossible, even when they
don’t make sense.
BENEFITS OF THE COURSES
•
Educational Community
•
Heartland Community College
•
Students
•
Employers