PianoTouch: A Wearable Haptic Piano Instruction System For Passive
Learning of Piano Skills
Kevin Huang
College of Computing,
Georgia Institute of
Technology
kevinhh@gatech.edu
Ellen Yi-Luen Do
College of Architecture and
College of Computing, Georgia
Institute of Technology
ellendo@gatech.edu
Abstract
We present PianoTouch, a wearable, wireless
haptic piano instruction system, composed of (1) five
small vibration motors, one for each finger, fitted
inside a glove, (2) Bluetooth module mounted on the
glove, and (3) piano music output from a laptop. With
this system, users hear the piano music, and feel the
vibrations indicating which finger is used to play the
note. In this paper, we investigate the system’s
potential for passive learning, i.e. learning piano
playing automatically, while engaged in everyday
activities. In a preliminary study, subjects learned two
songs initially, and then wore the PianoTouch glove
for 30 minutes while listening to the songs repeated.
One of the songs included tactile sensations and the
other did not. The study found that after 30 minutes,
the PianoTouch subjects were able to play the song
accompanied by tactile sensations better than the nontactile song.
1. Introduction
While playing music can provide great relaxation
and enjoyment, the process of learning an instrument,
such as the piano, can be time-consuming and often
beyond the time constraints of a busy working adult.
Also, time consumption is not simply confined to the
initial learning of songs, but is compounded by the
difficulty in retaining the knowledge. As soon as a new
song is learned, forgetting begins immediately [2].
Thus, time-consuming and repetitious practice is
needed to retain the new skills.
However, learning does not always occur as an
active process. Much research has been conducted on
the phenomenon of passive learning. Passive learning
Thad Starner
College of Computing,
Georgia Institute of
Technology
thad.starner@cc.gatech.edu
is described as learning that is “caught, rather than
taught”, and is characterized as “typically effortless,
responsive to animated stimuli, amenable to artificial
aid to relaxation, and characterized by an absence of
resistance to what is learned” [5]. Studies have shown
that passive learning of information can occur when
subjects are exposed to media rich environments. In a
study by Zukin and Snyder, it was found that subjects
who lived in a media rich environment and were
passively exposed to political information were 40%
more likely to have acquired the information than
subjects living in a media poor environment [6]. Both
subject groups had no interest in the political
information.
In this paper, we investigate the potential for passive
learning of physical skills. If a user can be exposed to
the practice and repetition in a passive manner while
engaged in their daily routines (e.g. working at a desk,
commuting on the subway, etc), then perhaps they can
learn and reinforce the skills automatically. For this
purpose, we have designed and implemented a system
to allow for the learner to practice piano at any time,
any place, and while engaged in other tasks. The
system’s main component is a haptic glove, fitted with
small vibration motors. Fingertips on the glove were
cut to improve manual dexterity. With this glove, users
can not only hear the music, but feel the notes on their
fingers as well. Research has also shown that a multimodal combination of audio and haptic cues gives the
user a richer understanding of musical structure and
improves performance of the musical piece [3]. A
small Bluetooth module is integrated into the glove for
wireless communication. When music is played from
an associated computing device (a laptop computer in
this case), the glove sends vibrations to each finger to
indicate which fingers play which notes. Although a
ISWC 2008, 12th IEEE International Symposium on Wearable Computers, ISWC, pp 41-44, Sep 28 - Oct 1, 2008, Pittsburgh, Pennsylavania
laptop is used in this set up, the PianoTouch device can
be easily integrated with other mobile devices such as
cell phones or MP3 players.
2. System Components
The PianoTouch system is composed of 3 distinct
parts: (1) actuators – small vibration motors, one for
each finger, (2) a Bluetooth-to-serial module coupled
with an ATMega8 chip to provide wireless usage, and
(3) a laptop computer which produces music output and
controls the glove via Bluetooth to send synced
vibrations.
Figure 3. Fingertip-less Golf glove with
components attached, including battery
Figure 3. shows the golf glove used to house the
motors and the wireless components. The rectangular
black box is a battery pack with 2 AAA batteries.
Figure 1. Vibration motor
Figure 1. shows the vibration motor used. Though
small, they are able to produce strong tactile sensations.
Figure 4. Wearing the glove
Figure 2. Bluetooth module and ATMega8 Chip
Figure 2. shows the components for wireless
communication. The Bluetooth module is a Bluetoothto-serial DIP module from Sparkfun Electronics. The
laptop sends Bluetooth information to the module,
which is then pushed out of its serial pin. The serial
output is then fed into the RX pin of the ATMega8
Chip, which is produced by ATMel Corp. The
ATMega8 chip was programmed to read the serial
information and output current on its pins to drive the
motors accordingly.
Figure 5. User initially learning a song
ISWC 2008, 12th IEEE International Symposium on Wearable Computers, ISWC, pp 41-44, Sep 28 - Oct 1, 2008, Pittsburgh, Pennsylavania
Figure 4 shows the glove being worn. The battery,
Bluetooth module, wires and vibration motors are on
the back side of the hand and fingers so that the user’s
grip is not affected.
Figure 5 shows a user wearing the glove and
learning a song initially. The laptop showed a virtual
keyboard which indicated which keys to press and
provided the piano audio. The glove provided synced
tactile sensations.
4. Results
Table 1. Performance of both songs after 30
minutes
Subject
Number
Tactile
Song
Non
Tactile
Song
1
Amazing
Grace
Jingle
Bells
Amazing
Grace
Jingle
Bells
Jingle
Bells
Jingle
Bells
Amazing
Grace
Amazing
Grace
3. User Study
In a preliminary study, we tested 4 subjects with
virtually no piano background. The 4 subjects were
male graduate students. The subjects rated themselves
as novices in piano and reported 0 years of piano
playing in the last 15 years. Each subject learned two
songs that were rated as Level B in a Song Book
published by Casio [1]. Level B songs were chosen for
the study because Level A songs are entry level
selections and of trivial difficulty. The songs chosen for
this pilot study were “Jingle Bells” (verse version,
without chorus), and “Amazing Grace”, both in the key
of C, spanned the same octave, and contained roughly
the same notes. Jingle Bells contained 45 notes, and
Amazing Grace contained 44 notes. Both songs were
one handed (using the right hand) for this pilot study.
Each subject first started in an active learning
session by learning the two songs. During this active
learning session, they wore the glove with tactile
sensations for both songs, watched a virtual piano on
the laptop playing the songs, and practiced until they
could play both songs without mistakes. This took
approximately 10 to 15 minutes for each subject. The
tactile sensation was a vibration of 350 milliseconds for
each note, did not vary in magnitude or duration, and
was synced exactly with the onset of each note as it
played. After both songs were learned, they were
asked to go back to their daily tasks for a period of
time. Due to experimental constraints, a period of 30
minutes was chosen for this pilot study. Subjects 1 and
2 chose to read and type at their desks. Subjects 3 and
4 chose to read a book and paper respectively. During
that time, they kept the glove on and wore headphones
to hear the two songs alternating in a loop. For one of
the songs, they received tactile indications (hereon
referred to as the tactile song), and for the other song,
they did not. Two subjects were randomly assigned to
receive “Jingle Bells” as the tactile song, and the other
two received “Amazing Grace” as the tactile song.
After 30 minutes, each participant was asked to play
the two songs again, and their performance and
impressions were noted.
2
3
4
# of
mistakes
for tactile
song
0
# of
mistakes
for nontactile
song
7+ (could
not
continue)
0
5
0
4
3
7
Table 1. shows that the subjects were able to play
their tactile songs better than the non-tactile songs.
When the subjects attempted to play the non-tactile
songs, there appeared to be more hesitation, confusion
and delay in recall. During the playing of the tactile
songs, the subjects played through them more smoothly
and with less hesitation. Subjects 1, 2 and 3 were able
to play the tactile song without any mistakes. Mistakes
were defined by unexpected note substitutions,
insertions and deletions. Timing (i.e. playing the
correct note but on the wrong beat) was not considered.
Table 2. How likely would you use this device
as a student learning piano? (choices: “would
definitely not use”, “would likely not use”,
“neutral”, “would likely use”, “would definitely
use”)
would likely use
Subject 1
would
likely use
Subject 2
Subject 3
would definitely use
Subject 4
would likely use
As shown in Table 2, the users in general stated that
they would use the PianoTouch system for learning
piano.
5. Comments from users
In general, the users stated that the device was
helpful. Users 3 and 4 stated that they could notice the
tactile sensations reinforcing their muscle memory
during the 30 minutes. User 4 stated that it was
particularly helpful to know the finger ordering and
ISWC 2008, 12th IEEE International Symposium on Wearable Computers, ISWC, pp 41-44, Sep 28 - Oct 1, 2008, Pittsburgh, Pennsylavania
patterns when returning to play the song. The user
stated that in such a case, the remaining cognitive load
was only to remember a key position for a finger, and
the other fingers “fell into place.” The user stated that
for the tactile song, there was much less information
that had to be consciously retrieved from memory.
When asked if their productivity was affected,
subjects 2, 3 and 4 stated that there was little effect.
Subject 2 stated that he only paid attention to the
device during moments of boredom. Subject 3
remarked that he was completely engrossed in a book
and did not notice the device at all. Subject 1 however
reported being significantly distracted. The subject
stated that a few times, he instinctively tried to “play
along” with the vibrations when the tactile song played.
With regards to typing, subjects 1 and 2 reported that
there was no noticeable interference from the
vibrations. The glove, however, did interfere slightly
with typing as the material was slightly stiff. Subjects
3 and 4 did not type. Further research should be done
to determine the effects of the tactile sensations during
daily activities such as typing and walking. We plan to
use the NASA Task Load Index to measure the level of
distraction for these activities.
When asked about possible areas for improvement
and general feedback, most users stated that the golf
glove was uncomfortable. The discomfort would be
their greatest barrier in adopting this device in a real
setting. They believe that any product that would gain
wide adoption would have to use an extremely thin,
breathable, “barely-notice-it” glove, or preferably not a
glove at all. A gloveless system using a wristband and
freely placed vibration nodes is under design for future
evaluations.
finger coordination, and facilitated overall recall. A
more detailed study on a larger scale is needed to
obtain more quantitative measures on its effects, such
as how does time of exposure (wearing the device for
an hour, a day, 2 days, etc) correlate with learning.
Also, what is the difference in effectiveness between
different types or difficulty of music? Would it be less
effective on an advanced song with fast, multi-octave
passages such as Beethoven’s Fur Elise?
6. Analysis and Future Work
[2] D. Coon, Psychology: A Modular Approach to Mind and
Behavior, Thomson Wadsworth, 2005.
6.1 Comments on piano
The piano is somewhat unique in that there are 10
fingers and 88 keys (on average). The users needed to
learn the songs in an active setting first (with the virtual
piano providing spatial information) to see which keys
to press. However, two users reported that during
learning, the difficulty came from not only knowing
which keys to press, but also which fingers to use.
Correct finger usage is very important in proper piano
playing. Fingering inconsistently will lead to “running
out of fingers”, unpleasant phrasing, and inconsistent
rehearsals which compound the difficulty in
remembering [4]. The users reported that having the
device reinforce the fingers’ ordering, patterns, and
rhythm made the fingers ready to respond, improved
6.2 Applications for other instruments
Because the piano has many more keys than we
have fingers, the active session is needed to initially
learn the songs, and the passive sessions can then
reinforce the learning. However, we also plan to test
the system’s effectiveness for instruments with roughly
1 to 1 finger to key mapping, such as the saxophone,
flute, recorder, etc. Given that there are enough fingers
for all the keys, perhaps the entire song-learning
process can be done in the passive phase. One may be
able to engage in normal tasks while learning entirely
new songs throughout the day.
7. Acknowledgements
We would like to extend a special thanks to Steve
Cuzzort and Scott Gilliland of College of Computing,
Georgia Institute of Technology, for their assistance
during the development of the system’s hardware.
8. References
[1] Song Book, Casio Computer Co., LTD.
[3] G. C. Grindlay, The Impact of Haptic Guidance on
Musical Motor Learning, Masters Thesis, Massachusetts
Institute of Technology, 2007.
[4] J. Hofmann, Piano Playing, with Piano Questions
Answered, Courier Dover Publications, 1976.
[5] H. E. Krugman, and E. L. Hartley, Passive Learning
From Television, The Public Opinion Quarterly, Oxford
University Press, 1970, Vol. 34, No. 2, pp. 184-190.
[6] C. Zukin, and R. Snyder, Passive Learning: When the
Media Environment Is the Message, The Public Opinion
Quarterly, Oxford University Press, 1984, Vol. 48, No. 3 pp.
629-638.
ISWC 2008, 12th IEEE International Symposium on Wearable Computers, ISWC, pp 41-44, Sep 28 - Oct 1, 2008, Pittsburgh, Pennsylavania