Passive Haptic Learning of Typing
Skills Facilitated by Wearable
Computers
Caitlyn Seim
College of Computing,
85 5th St., TSRB 338
Georgia Institute of Technology
Atlanta, GA 30332-0760
ceseim@gatech.edu
David Quigley
College of Computing,
85 5th St., TSRB 338
Georgia Institute of Technology
Atlanta, GA 30332-0760
quigs@gatech.edu
Thad Starner
College of Computing,
85 5th St., TSRB 338
Georgia Institute of Technology
Atlanta, GA 30332-0760
thad@gatech.edu
Permission to make digital or hard copies of part or all of this work for personal
or classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this notice
Abstract
Passive Haptic Learning (PHL) allows people to learn
“muscle memory” through vibration stimuli without
devoting attention to the stimulus. PHL can be
facilitated by wearable computers such as gloves with
an embedded tactile interface. Previous work on PHL
taught users rote patterns of finger movements
corresponding to piano melodies. Expanding on this
research, we are currently exploring the capabilities
and limits of Passive Haptic Learning as we investigate
whether more complex skills and meaning can be
taught through wearable, tactile interfaces. We are
creating and studying a system for passively teaching
typing skills, with the ultimate goal of passively
teaching Braille typing. Our initial studies in perception
and learning provide key information for system
development including the importance of visual
feedback in learning to type; while our pilot study using
the current system for Passive Haptic Learning of
typing on an unfamiliar keyboard shows passive
learning in all participants.
Author Keywords
Haptic; tactile; typing; wearable; passive training;
learning; PHL.
and the full citation on the first page. Copyrights for third-party components of
this work must be honored. For all other uses, contact the owner/author(s).
Copyright is held by the author/owner(s).
CHI 2014, Apr 26 - May 01 2014, Toronto, ON, Canada
ACM 978-1-4503-2474-8/14/04.
http://dx.doi.org/10.1145/2559206.2581329
ACM Classification Keywords
H.5.2 Information Interfaces and Presentation:
Miscellaneous.
Introduction
Haptic systems can help users learn manual tasks [1,
2, 3]. One convenient method of creating a tactile
interface is to insert tactors into a glove. The vibration
motors are placed at the bases of the fingers and a
computing system controls which fingers receive
vibration stimulation. Markow et al. [7] demonstrated
that such a wearable, tactile interface system can
facilitate Passive Haptic Learning (PHL) and
Passive Haptic Rehabilitation (PHR). Passive Haptic
Learning is a phenomenon where users can learn
through tactile stimulation without devoting active
attention to the stimulus. Passive learning is caught,
rather than taught, and is typically effortless [4].
Gloves for PHL have been found to be an effective
learning tool for manual dexterity skills of the fingers.
The previous work on Passive Haptic Learning, the
Mobile Music Touch (MMT) project, focused on teaching
rote order tasks, such as the order of notes for a piano
melody [7]. Now, with the interest of exploring the
capabilities and limits of PHL, we are investigating,
creating, and studying a system to facilitate Passive
Haptic Learning of typing skills. This work has the
ultimate goal of lowering barriers to learning Braille
typing and will better define the abilities of a wearable,
tactile interface in passively teaching users. We are
currently researching human perception of vibration
stimuli, the importance of visual feedback in active
haptic learning, and Passive Haptic Learning of typing
on an unfamiliar keyboard. Work conducted thus far
defines some important needs of the system and shows
successful Passive Haptic Learning in three subjects.
Background and Motivation
Previous work has established Passive Haptic Learning
for rote muscle movement patterns of the fingers on
one hand. The Mobile Music Touch (MMT) project
focused on passively teaching a pattern of keys for a
piano melody using Gloves for PHL. The piano student
wore the glove while doing other tasks, such as reading
email, taking a test, or watching a video. The MMT
glove played the song to be learned and stimulated the
appropriate finger for each note. The student could
ignore the glove, even while performing distracting
tasks, and yet learning still occurs. Studies showed
that participants could learn the first 45 notes of simple
melodies, such as Amazing Grace, in 30 minutes using
this method [7].
We are now expanding research in Passive Haptic
Learning to investigate whether more complex systems
and tasks can be taught through PHL. Our group is
exploring the use of gloves with embedded vibration
motors in the fingers for use in Passive Haptic Learning
of typing skills. Though this research has significance
in the areas of interface design and passive learning as we discover what, and how, information can be
conveyed through passive haptics - this work is also
motivated by a specific problem.
Over six million people in the United States alone are
blind. We are investigating the use of PHL for learning
Braille typing, a chorded text entry system. Learning
to type the Braille system is time consuming and a
major component of rehabilitation and independence
training for individuals who are blind or visually
impaired. Braille is especially difficult to learn for those
who lose their sight later in life, such as the aging
population, wounded veterans, and the increasing
number of diabetics. Passive learning of Braille typing
would reduce the seven or more months typically spent
learning at special Blind Rehabilitation Centers, if the
patient is fortunate enough to even have access to one
of these instructional facilities.
Creating a system that passively teaches Braille typing
through vibrations stimuli (PHL) would facilitate a
reduction in rehab training time by allowing patients to
learn while doing other tasks such as cane training,
orientation and mobility or even tasks in their daily life
or at home. With knowledge of the dot system that
comprises Braille and the Braille alphabet, the system
for PHL of typing skills may better help individuals to
learn to read Braille as well. The lightweight and userconscious system that we are developing also would
provide access to Braille instruction for those with
financial or geographic constraints that prevent them
from getting rehabilitation instruction. This research
aims not only to explore the subject of Passive Haptic
Learning, but to also create this system for Braille
instruction.
Figure 1. A Right and Left BAT keyboard comprise our unique keyboard that is unfamiliar to users. Red and blue keys are not used and serial output is decoded to represent letters A‐H, space and enter. Initial Work and Findings
Expanding upon previous research, we now work to
show that PHL can be used to teach typing systems,
not just key patterns for piano melodies. The next
sections discuss our efforts in determining how best to
provide PHL for typing and how to create experiments
that are suitably sensitive for detecting the effect of
PHL on learning a text entry keyboard.
The Current System
Figure 2. Current system in use (Active Practice). A. is the microcontroller that controls the attached gloves’ vibrations motors (C.). B. are the keyboards pictured in Figure 4. D. shows the feedback display (letters condition). Figure 3. Phrases in the current system. The current system consists of fingerless gloves with
embedded vibration motors (one per finger) controlled
by a microcontroller, our typing training software that
records user data, and two BAT keyboards from
InfoGrip (which form a unique keyboard consisting of
only letters A-H, space and enter). Figures 1 and 2
illustrate the system and use. Users learn one phrase
at a time and may encounter several different kinds of
sessions:
Active Practice sessions (used in our feedback study)
consist of audio of a phrase (e.g. “he had”), followed by
a carefully devised pattern of vibrations that stimulate
the pattern of finger presses that type that phrase on
our unique keyboard. Timing of vibrations in the
sequence was determined to maximize vibration
strength and clear perception (400 ms vibrations on
average, with 100 ms pauses). The audio is then
played again, and the user attempts to type the pattern
into the training software which displays typed letters
(see feedback study) and records user statistics.
Passive Haptic Learning (PHL) and control sessions
consist of users playing a distracting memory card
game while they have audio of the phrase they are
learning played for them. Those not in the control
group also wear the PHL gloves and feel the phrase’s
vibration pattern after each audio clip. Users focus on
the distraction task and pay no attention to stimuli.
During Testing sessions, participants are played the
audio of the phrase that they were to learn (without
tactile stimulation). They then attempt to type that
phrase. The testing software displays an asterisk for
each letter typed so as to minimize any additional
learning (see feedback study). Following three tries at
typing the phrase, various words from that phrase are
also presented via audio in a random order to be typed.
Phrases in the current system were chosen to be 15-17
characters in length, the same as the number of notes
presented in our piano work [6]. In order to be typed
on our unfamiliar keyboard, phrases consist of only the
first 8 letters of the alphabet, and were chosen to have
meaning for easy recall. Each phrase consists of 4
words, with an average of 3 letters each, and they
contain almost similar frequency distributions of letters
between A-H (i.e. only missing one of these). They also
contain simply-spelled words with minimal homophones
(i.e. ‘be’ vs. ‘bee’) as found to be ideal in other audioprompted typing research [8]. System phrases are
depicted in Figure 3.
Visual Feedback during Active Typing Practice
Some studies that we will be conducting on PHL will
contain a session of Active Practice, guided by vibration
stimuli. To optimize our system for this portion of the
research, some aspects must be studied.
During any portions of our studies where subjects must
type, our specially designed typing tutor software
prompts the subjects with what to type (via audio),
records what text they enter, and calculates statistics
like average error rate. The software also displays a
Figure 4. Full “letter” feedback (above): what the user types is displayed to them; versus Asterisks “stars” only feedback (below): ‘*’s are displayed when the user enters a key or Space Figure 5. Randomized keyboard mappings for our feedback study. Top is the standard mapping for our system, which is used in PHL work,
Figure 6. Errors in typing the first phrase presented, averaged over 10 attempts. Users presented with informative feedback (the letters they typed) show lower errors on average and better final accuracy. blank screen with our choice of visual feedback. We
wondered if the type of visual feedback affected the
user’s ability to learn to type. Previous research
showed that expert typists perform slightly faster and
more accurately with limited visual feedback.
only asterisks presented on a computer screen,
depending on the visual feedback condition they are
experiencing. Each user starts in one condition,
assigned randomly (users switch to the opposite
condition for the 2nd and 3rd phrase).
We hypothesized that hiding the letters typed during
Active Practice might aid in learning as the users would
be forced to focus on the audio and the vibration
pattern as opposed to the screen. This is a desirable
effect when we are allowing users to actively practice
typing, a time where learning is encouraged. During
Testing sessions, though, we do not want any further
learning to occur. In order to better understand
whether visual feedback during typing practice does
affect active learning, we are studying two conditions of
feedback in our system. The feedback conditions that
we chose to test are full “letter” feedback and asterisks
“stars” only feedback.
Users repeat the audio-vibration-audio-type process for
ten attempts for the first phrase presented (this is the
main test - to test learning under that condition).
Presumably, after learning to type a phrase in this
feedback condition, the user may have learned the
finger-to-key mapping. A 2nd phrase is then presented
in the same manner (with only three attempts given).
In this extra test, the feedback condition is opposite to
what was used for the first phrase; this allows us to see
how well participants learn a phrase when they have
some knowledge of the keyboard, and how well they
learned the keyboard under the previous condition.
In the letter feedback condition, users see each
character as they enter it. This informative condition
allows users to see where they make errors (Figure 4,
top). We also test an uninformative “stars” condition
where users are shown only an asterisk for each
character typed (Figure 4, bottom). Our choice of
asterisks as the alternative form of (uninformative)
visual feedback comes from the idea that users need
some type of visual signal that indicates they are
interacting with the keyboard correctly to produce input
(i.e. pressing keys sufficiently hard to register, yet not
producing repeated keys unintentionally: ‘hhhad’ vs.
‘had).
In a study, containing 7 participants thus far, four
phrases (from Figure 3, in random order) are presented
to be typed on an unfamiliar 8-key keyboard. Audio of
a phrase is presented, followed by a pattern of
vibrations corresponding to the fingers that type that
phrase. The training software then plays the audio
again, and participants try to type the phrase. When
typing, participants either see the letters they enter or
Users then experience a second pair of phrases in the
same way (main test, extra test), but under the
opposite feedback conditions (e.g. 1st and 2nd were
stars then letters, so 3rd – letters, 4th - stars). The
keyboard mapping (what finger types what letter) is
changed for the second pair of phrases (so participants
begin learning a totally new keyboard). An image of
the randomized keyboard mapping is presented in
Figure 5. The second mapping is mirrored to avoid
transfer of learning between the two keyboards while
still maintaining the same letter distribution on each
hand. Orders of what phrase is presented and
feedback formats are counterbalanced.
Findings for this study thus far indicate that users who
are provided feedback on the letters that they typed,
being totally unfamiliar with the keyboard and
otherwise only guided by vibration stimuli, actually
perform better. These users show better
improvements in accuracy from beginning to end,
better average (as can be seen in Figure 6 for user
performance on the first phrase presented) and finaltry accuracy, but lower WPM. Users also become more
comfortable (with fewer errors overall) from the
beginning to the end of the study.
Results from this study inform us of the effect of visual
feedback on data entry and allow us to better create a
system of training for PHL.
Passive Haptic Learning of Typing
After iterative design modifications and study of our
PHL system, we have arrived upon our current system
which shows evidence of Passive Haptic Learning.
Figure 7. Typing errors during test after PHL only (solid lines). Contrast to average errors of participants during first three tries of Active Practice (no previous learning) sessions from feedback study, plotted here for visualization purposes (dashed line). All users were typing the phrase for the first time (and 2ed and 3rd), on the same keyboard mapping, and were all in the stars feedback condition
Figure 8. Typing errors during test on totally new phrase (solid lines). Contrast to average errors of participants during first three tries of Active Practice (no previous learning) sessions from feedback study, plotted for visualization purposes (dashed line). All users were typing the phrase for the first time (and 2ed and 3rd) and were all in the stars feedback condition
In our previous exploration with PHL for typing, we
discovered:

Visual prompts may not work for PHL typing –
users may need audio prompts like that with
which they learn

Teaching letters and words, randomly presented
only a couple times, showed no effect – learning
time and information “chunk” size may be pivotal

Teaching chords did not work – our users reported
difficulty in distinguishing which fingers were
being stimulated

Speed is the wrong metric – accuracy is much
more indicative of PHL
After consideration of these previous experimentations,
we have made typing prompts audio-based to maintain
consistency with learning conditions, and altered the
system to teach one phrase at a time. This phrase
structure and use of a simple keyboard devotes ample
time to learning (an entire 30 min. PHL session),
“chunks” information for better learning, and improves
on our previous research in that it investigates twohanded learning (as opposed to our efforts on onehanded piano melodies). We are now exploring the
effectiveness of our improved system.
In a three subject experiment, we presented
participants with no Active Practice before placing them
into a PHL session (while playing a distracting memory
card game) for a different phrase each. During this
learning session, users heard each word in the phrase
through headphones, then felt the word’s vibration
sequence before the following word’s audio was played.
After 30 minutes of Passive Haptic Learning of their
phrase, users were tested on the phrase they passively
learned, as well as a totally new phrase using the same
unfamiliar keyboard.
All users were able to type the passively learned phrase
with less than 20% errors and maintained the same
errors throughout (i.e. “ecc” for “egg”), if any at all.
This is a significantly lower error rate than participants
typically exhibit during their first 3 tries at typing a
phrase (typically ~70%), during our feedback study for
example (see Figure 7). Having never practiced typing
the phrase before, this level of accuracy seems to
indicate that participants successfully passively learned
typing across both hands during the PHL session.
Users were also consistently near 0% errors when
tested on individual words in the phrase. In addition,
users were able to figure out how to type the new
phrase with less than 20% errors, but at a slightly
reduced speed (Figure 8). These findings lead us to
infer that these users learned not only the PHL phrase’s
rote pattern, but also the mapping of our simple
keyboard. These results are a promising beginning to
our exploration into Passive Haptic Learning for typing
skills.
Future Work
Moving forward, we will expand the current studies to
confirm our findings. Studies in human perception of
haptic stimuli at multiple points on the fingers and
other relevant research will also be conducted in order
to best refine our system and inform future research.
We will then implement a formal study of our system
for PHL of typing on a unique keyboard. Following
successful demonstration of robust passive learning of
typing skills, we will design and test efficient mappings
between haptic stimuli and key combinations on the
Braille keyboard.
A subsequent goal of this research is to develop a
system that aids in learning Stenography, a phonemebased, chorded text entry technique used for real-time
transcription. Learning Stenography is currently an
obstacle to those wanting to enter the field; schools
report 85%-95% drop out rates, and even experts must
practice for hours a week in order to maintain industry
completive speeds [5]. Passive Haptic Learning of
stenography would lower the barriers to entry into this
industry. With these similarities to Braille typing,
alteration of our system to passively teach Stenography,
once proven feasible via PHL, will be straightforward.
Conclusion
With the goal of lowering the barriers to learning Braille
typing, we are exploring Passive Haptic Learning of
typing skills. While previous research in our group,
which established the existence of PHL, focused on
teaching rote patterns to one hand, our research
explores teaching a system with meaning across both
hands with the addition of speech stimuli. Research on
the need for visual feedback in learning aids us in
system design and informs other researchers working in
the areas of haptics, text entry and interfaces. Our
initial work on using PHL (facilitated by wearable
computers) to teach typing suggests positive results in
each subject and transferred learning of the unfamiliar
keyboard’s mapping.
Acknowledgements
This material is based upon work supported, in part, by
the National Science Foundation under grant No.
1217473.
References
[1] Bluteau, J., Coquillart, S., Payan, Y., and Gentaz, E.
Haptic guidance improves the visuo-manual tracking
of trajectories. In PLoS ONE (2008), Vol. 3 Issue 3,
p1.
[2] Chellali, A. M., Dumas, C., and Milleville-Pennel, I.
Wyfiwif: A haptic communication paradigm for
collaborative motor skills learning. In Web Virtual
Reality and Three-Dimensional Worlds (2010).
[3] Eid, M. A., Mansour, M., El Saddik, A. H., and
Iglesias, R. A haptic multimedia handwriting learning
system. In Proc. the international workshop on
Educational multimedia and multimedia education
((EMME), ACM (2007), 103–108.
[4] Krugman, H. E., and Hartley., E. L. Passive Learning
From Television. The Public Opinion Quarterly 34, 2
(1970), 184–190.
[5] Kuhlin, T. Considering a Career Change to Become a
Court Reporter? Suite101.
http://suite101.com/article/considering-acareerchange-to--become-a-court-reportera370134#.UWj_c0nft0x.
[6] Markow, T. T. (2012). Mobile music touch: using
haptic stimulation for passive rehabilitation and
learning (Doctoral dissertation, Georgia Institute of
Technology).
[7] Markow, T., Ramakrishnan, N., Huang, K., Starner,
T., Eicholtz, M., Garrett, S., Profita, H., Scarlata, A.,
Schooler, C., Tarun, A., and Backus, D. Mobile music
touch: Vibration stimulus in hand rehabilitation. In
4th International Conference on Pervasive Computing
Technologies for Healthcare (PervasiveHealth)
(2010).
[8] Southern, C., Clawson, J., Frey, B., Abowd, G. D.,
Romero, M., "An Evaluation of BrailleTouch: Mobile
Touchscreen Text Entry for the Visually Impaired."
Mobile HCI 2012. San Francisco, September 2012.