Tactile Teacher: Toward Sensing Taps When Playing Piano
First Author Name (Blank for Blind Review)
Affiliation (Blank for Blind Review)
Address (Blank for Blind Review)
e-mail address (Blank for Blind Review)
ABSTRACT
In a piano lesson, a student often imitates how her teacher is
playing in terms of speed, dynamics, and fingering. We
seek to convey the touch sensations in piano lessons
between a teacher and a student by sensing the teacher's
keystrokes and then tapping the student's fingers
appropriately. Here, we propose an implementation to
sense finger taps on hard surfaces by using an instrumented
glove. Since finger taps generate acoustic signals and cause
vibrations on the palm, we embed two vibration sensors on
the glove and use machine learning to analyze the data from
the sensors. After a brief training procedure, this prototype
can accurately identify which fingers are being tapped in
real time.
Second Author Name (Blank for Blind Review)
Affiliation (Blank for Blind Review)
Address (Blank for Blind Review)
e-mail address (Blank for Blind Review)
The design of the sensing glove is another challenging task.
Pianists usually do not like additional sensors on their
fingers since it would affect dexterity. Thus, we will need to
avoid additional materials on fingers and deploy as few
sensors as possible on the palm portion of the glove. To
detect fingering, we employ sensors mounted on the back of
the hand and a machine-learning approach to distinguish
fingers from the input data. Since fingers are not necessary
in the same position while playing piano, we also create a
training procedure that captures these factors.
Author Keywords
Wearable Computing, Sensing, Finger Input.
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous.
General Terms
Human Factors, Experimentation.
INTRODUCTION
Transferring tactile sensations from one’s fingers to
another’s fingers is a challenging task, especially for those
applications that require immediate responses. The task
includes the sensing of speed, dynamics, and fingering on
the originator hand, and the rendering of sensation on the
receiving hand. Mobile Music Touch [3] and ConcertHands
[6] employ vibration motors to convey the tactile sensations
for piano teaching purposes. However, the data used for
output is preset in the program and lacks spontaneity as
well as artistic aspects in terms of timing and strength of
hitting the keys. In a piano lesson, a teacher often asks
students to imitate the timing of hitting the keys and the
volume of each note. Except the sound and vision, we
propose an implementation that adds an additional medium,
touch sensations, for assisting students in piano lessons.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
UbiComp ‘13, Sep 8–Sep 12, 2013, Zurich, Switzerland.
Copyright 2013 ACM 978-1-4503-1770-2/13/09...$10.00.
Figure 1. Left: The prototype consists of two sensors. The
orange region is the left sensor while the blue region is the right
sensor. Right: The glove that senses taps with two vibration
sensors, and a Maple Mini Board for initial data processing.
PREVIOUS WORK
The primary objective of this project is to provide an
always-available input device for capturing finger taps that
can be applied to regular pianos. There are various
approaches that can help achieve the goal. For instance,
using computer vision could help us detect the finger
movements and the location of the fingers [1]. However,
this approach usually requires intensive computation and is
error prone. An alternative method is to wear sensors on
hands or arms for detecting motion. We have seen many
different approaches in this domain. Sensing muscle
movements by using EMG is one feasible method for
detecting finger movements as an always-available input
device [7]. However, EMG sensors cannot help recognize
accurate timing when a finger is tapping on hard surfaces.
The other approach proposed by Chris Harrison is to wear
an array of vibration sensors on arms for detecting bioacoustic signals on the skin [2]. In his implementations,
users need to train the system every time before use since
the locations of the sensors are varied when wearing the
armband. We propose a lightweight sensing glove with a
fun way for training the system to distinguish fingers and
recognize the strength of the finger taps.
SENSING AND PROCESSING
As shown in Figure 1, we employ a Maple Mini board
(from LeafLabs) [4] to sample two ADC channels at
6.5KHz from piezo vibration sensors and send the data to a
computer through USB for further data processing before
running a machine learning program for classification. This
data processing includes exponential smoothing and
filtering. When the smoothed data of either channel exceeds
a threshold, the samples around the threshold from both
channels are considered as a tap event. As shown in Figure
2, each tap comes with the tallest spike and one or two
shorter spikes, each of which lasts for about 12-17ms (5883Hz). To ensure that the samples influenced by a tap event
are filtered, the key parameters, such as the threshold,
smoothing factor, and number of samples around the
exceeding time, are tuned carefully.
Classified as
Actual
T
I
M
R
P
Actual finger taps
in a training
session
Thumb
16
3
1
0
0
28
Index
0
20
0
0
0
32
Middle
0
0
19
0
1
36
Ring
0
0
0
20
0
18
Pinky
1
0
0
1
18
10
Table 1. The classification results after a user plays “Ode
to Joy” twice to train the system. In a test session, each
finger taps 20 times on the piano keys.
results of the training data and independent test data. In a
testing session, the tester taps 20 times per finger for getting
the results. Since each person’s hands are different, the
trained model cannot be re-used for different people.
However, a user would not need to train the system again if
she wears the glove in the same position on the same hand.
FUTURE WORK
Figure 2. The waves when a user taps twice on their index
finger. The orange line represents the left sensor while the
blue lines represents the right sensor.
After smoothing and filtering, sixty features can be
extracted from each input event. These features include six
averages and square averages of each channel and the ratios
between them. The system also computes a 256-point FFT
for transforming input from the time domain to frequency
domain. Because the frequency of the tap waveform is
roughly between 50 to 100 Hz, only the lower nine bins (23
Hz each) are used for computing the normalized decibels
(36 features). Finally, we extract 18 features from
calculating the ratios of respective values of the two
channels. The 60 features are then classified using the
Support Vector Machine (SVM) package from Weka [5].
The latency of computing the data for each tap event is
about 1ms to 2ms.
MACHINE LEARNING PROCEDURE AND THE RESULT
To train the system for classifying the fingering, we ask
users to play “Ode to Joy” twice for capturing the real
tapping position when the fingers are moving on the
keyboards. Since the system knows the order of the notes
and the fingering of each note that the user would use, it
can map the samples that exceed the threshold to specific
fingers in the entire song for training. The entire process is
automated so that the user only needs to play the song when
the system says start. We tested the system internally with
two participants and achieve about 94% accuracy from the
We have implemented an easy way to recognize the finger
taps with a couple of sensors. However, there are still few
tasks we would like to achieve in the near future. For
instance, when a pianist plays a piece of music, she often
uses multiple fingers to play chords. Our current
implementation only recognizes single finger taps when the
user plays piano. Thus, sensing multiple simultaneous taps
is our next major tasks. In addition, we are planning to test
the prototype in a piano lesson to understand how the
additional sensation influences the student to learn how to
play piano.
REFERENCES
1. Erol, A., Bebis, G., Nicolescu, M., Boyle, R.D., and
Twombly, X. A Review on Vision-Based Full DOF
Hand Motion Estimation. In Proc. Computer Vision and
Pattern Recognition - Workshops 2005. IEEE, 75–82.
2. Harrison, C., Tan, D., and Morris, D. Skinput:
appropriating the body as an input surface. In Proc. CHI
2010, ACM, 453–462.
3. Huang, K., Starner, T., Do, E., et al. Mobile Music
Touch: mobile tactile stimulation for passive learning.
In Proc. CHI 2010, ACM, 791–800.
4. LeafLabs. The Maple Mini.
[Accessed: 30-March-2013]
http://leaflabs.com/.
5. Machine Learing Group at the University of Waikato.
Weka.
http://www.cs.waikato.ac.nz/ml/weka/.
[Accessed: 30-March-2013]
6. Rubato
Productions.
ConcertHnads
System.
http://concerthands.com/. [Accessed: 30-March-2013]
7. Saponas, T.S., Tan, D.S., Morris, D., Balakrishnan, R.,
Turner, J., and Landay, J.A. Enabling always-available
input with muscle-computer interfaces. In Proc. UIST
2009, ACM, 167–176.