The Vibrobyte: A Haptic Interface for Co-Located Performance
Kyle McDonald
Dane Kouttron
Curtis Bahn
Jonas Braasch
Pauline Oliveros
Rensselaer Polytechnic Institute
{ mcdonk, kouttd, crb, braasj, olivep }@rpi.edu
Abstract
The Vibrobyte is a wireless haptic interface specialized for
co-located musical performance. The hardware is designed
around the open source Arduino platform, with haptic control data encapsulated in OSC messages, and OSC/hardware
communications handled by Processing. The Vibrobyte was
featured at the International Computer Music Conference
2008 (ICMC) in a telematic performance between ensembles in Belfast, Palo Alto (California, USA), and Troy (New
York, USA). This paper will discuss the background and
motivation for developing the Vibrobyte, technical details
of the hardware and software, provide an overview of the
first artistic applications during the ICMC performance, and
describe future directions for research.
Keywords: haptics, interface, telematic, performance.
1. Introduction
Telematic performances regularly rely on audio and video
transmissions, but haptic communication is generally neglected. The reason for the latter is partly the high cost
for haptic displays such as motion platforms for larger audiences. The Vibrobyte project began as an exploration of
haptic actuators for augmenting co-located performances,
and evolved into a general interface supporting a variety
of basic actuators. Cost efficiency was one of the main
design goals of the interface. Early experiments focused
on developing novel haptic display devices, including work
with shape memory alloy (SMA wire), magnetorheological
fluid (MR-fluid), thermoelectric devices (Peltier modules),
vibrating motors and solenoids (or speakers as solenoids).
Instead of developing a novel display device, we decided to
develop a reconfigurable wireless interface and protocol for
controlling haptic actuators. 1
The Vibrobyte operates as a haptic connection between
spaces, allowing a composer to send haptic signals to performers, for performers to haptically affect each other, or
various other modes of interaction. Each location has a single wireless transmitter and multiple Vibrobytes. Control
1 The primary actuator used for testing this interface was the vibrating
motor – hence the name “Vibrobyte”.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page.
c 2009 Carnegie Mellon University
Figure 1. Current revision Vibrobyte prototype.
data is sent from any location to any other location in the
form of OSC messages.
This paper will discuss the background and motivation
for developing the Vibrobyte, technical details of the hardware and software, provide an overview of the first artistic
applications during the ICMC performance, and describe future directions for research.
2. Related Work
There are numerous examples of haptic displays for musical
performance. A good overview on haptic displays can be
found in the work of Altinsoy[1]. Gillespie summarizes the
role of haptic perception for music applications[2]. Here we
will give a few specific examples similar to the Vibrobyte
in that they are music-related or focus on similar actuators
(vibrating motors).
2.1. Haptics in Music
Some haptic displays are more focused on teaching technique, like Graham Grindlay’s Magnetic Musical Training[3]
project and Haptic Guidance System[4]. Both these systems
augment percussive training with drum sticks – using electromagnets to direct the stick in the case of Magnetic Musical Training, and using a servo and encoder to record and
guide gestures in the case of the Haptic Guidance System.
Other haptic displays focus on providing a haptic component to otherwise purely sonic systems. StickMusic[5]
uses a joystick and mouse with haptic feedback, providing
four degrees of freedom and simple feedback (pulses for
octaves, force-guidance for a sense of directionality). The
Phase Project[6] installed a haptic arm in a public, mapped
to a system that allowed participants to arrange/improvise
a prerecorded composition using a vinyl record metaphor.
OROBORO[7] allows two musicians to face each other across
a table (i.e., networked, but nearby), each using two hand
orientation controllers to negotiate/improvise with a virtual
instrument.
Other work focuses on replicating specific musical haptic sensations, and understanding the meaning of those mappings. Charles Nichols’ V-bow[8] uses servomotors to simulate the haptic feedback of bowing a violin. Haptic Music Exercises[9] builds on the PLANK[10], experimenting
with interfaces that mirror the basic tacticle characteristics
of traditional acoustic instruments. Tactile composition research[11] surveys techniques for mapping musical and tactile events.
2.2. Vibrotactile Displays
Two examples of other haptic interfaces that have made heavy
use of vibrating motors are the Shoulder Pad Insert Vibrotactile Display[12] and Feelspace[13] project. The Shoulder Pad uses a small array of pancake motors (coin motor)
to mimic social conventions like shoulder-tapping for capturing attention, or to provide guidance. Feelspace uses a
linear array of pancake motors in the form of a belt, connected to a compass, allowing for directions or headings to
be communicated.
Figure 2. Current revision Vibrobytes during the fabrication
process.
3. Hardware
The creation of a lightweight, wirelessly addressable hardware platform that would be unobtrusively small, yet rugged
and functional, required many design revisions and hardware tradeoffs. After arduous brainstorming and testing,
each revision grew more functional and usable than its predecessor. The current revision houses an efficient DC-DC
converter, ICSP header, microcontroller, high power LEDs,
RJ-11 jack, MOSFET power amplifiers, and a breakout for
extra analog and digital inputs and outputs.
3.1. Vibrobyte
The Vibrobyte interface is functionally similar to an Arduino Mini 2 , in that uses the Atmel ATmega 168 microcontroller, running off an external crystal oscillator at 16 MHz.
It can be programmed using the Arduino IDE 3 and an AVR
ISP MKII programmer (or a serial connection and the Arduino bootloader). After careful design of the electronics
and PCB, the dimensions of the current revision are similar
2 The Arduino Mini is a small microcontroller board based on the
ATmega168 microcontroller. See http://arduino.cc/en/Main/
ArduinoBoardMini for more.
3 The Arduino hardware is complemented by a free, open-source IDE
that can compile and upload C code to the board using AVR-GCC and
other open-source software.
Figure 3. Current revision Vibrobytes PCB: both sides, then
each side individually.
to a stick of chewing gum. The DC-DC converter 4 allows
the interface to be powered from a range of power sources,
including AAs, AAAs, and smaller rechargeable lithium ion
packs. The four ultra-bright LEDs 5 – red, green, blue and
IR – are hardware addressable. RGB can be dimmed with
hardware PWM, while IR can be flashed. To power external devices two dual MOSFETs 6 are used, capable of driving outputs ranging from vibrating motors and solenoids to
Peltier devices. Each of these outputs are also tied to hardware PWM channels of the microcontroller. Actuators are
connected via an RJ-11 jack. The connector also has a direct path to the on board battery, for recharging. Finally, the
wireless receiver 7 operates on the 915 MHz band with a
trace antenna on the PCB. Testing indoors and line-of-sight,
we observed a maximum reliable unamplified data rate as
115200 baud @ 1 meter, 600 baud @ 10 meters. With 5
Watts of amplification, indoors and no longer line-of-sight,
we achieved 19200 baud @ 100 meters.
was chosen instead 9 as its performance was more favorable. Feedback issues between the transmitter module and
amplifier were addressed by simply shielding the transmitter. With such high amplification (5 Watts), undocumented
automatic gain control features in the receiver units cause
unreliable operation and due to signal saturation. Research
into using the amplifiers effectively is ongoing.
4. Software
The Vibrobyte software consists of protocols and implementations of those protocols. Open Sound Countrol (OSC 1.0) 10
is the only protocol used at a high level to address the Vibrobytes or send data across the network. A Processing 11
application running locally at each location translates the
OSC messages to serial packets and load-balances them before sending them to the wireless receiver. Firmware on the
Vibrobyte listens to incoming packets.
3.3.2. RF Amplifiers
In the early development stages we designed a high power
RF amplifier for the 915 MHz transmitter, but had feedback and drift issues. A commercially available amplifier
4.1. OSC Protocol
OSC was chosen for its ease of implementation and increasing commonality. Because OSC only requires a UDP connection, it is a natural addition to the other IP-based technologies driving the audio and video communications in previous telematic performances. The best way to describe Vibrobyte OSC protocol is to give a quick description of the
Vibrobyte’s current functionality.
Vibrobytes each have a unique ID, and may be addressed
individually or in groups. If a group is addressed, the characteristics of its members are overridden. If an individual in
a group is addressed, its unique characteristics take precedence.
The two types of displays on the Vibrobyte are the RGB+IR
LEDs and the three amplified PWM output channels. RGB
LEDs may be controlled directly, or used as indicator lights
to give visual feedback of the Vibrobyte’s status: reporting
battery life, wireless signal quality, visualizing serial data,
or the RGB LEDs can be asked to mirror (visualize) the amplified output signals.
The three amplified outputs can output one-hit impulses
or regular pulses. Both impulses and pulses can be shaped
with basic envelopes: square, ascending saw, descending
saw, and absolute sine waves. The amplitude, frequency,
and “sustain” (duty cycle) may also be specified.
The OSC messages controlling these functions are fairly
straightforward. All messages have at least two arguments:
a destination type (single or group), and the ID or group
number. Messages are in three categories: /group, /led and
/output. Grouping (/vibrobyte/group) is controlled using ../add,
4 Maxim’s MAX756 CMOS step-up DC-DC switching regulator, which
has a 3.3 V to 5 V operating range at 87% efficiency.
5 LEDs are brightness matched, ∼1500 mcd with ∼ 45◦ viewing angle.
6 Microchip TC4427 dual 4.5-18 V, 1.5 A MOSFET drivers.
7 Radios, Inc. MRX-005 (now discontinued) rated for 1200 baud and
available custom-order up to 115200 baud.
8 The MTX-105, a paired transmitter from Radios, Inc.
RF Bay, Inc. MPA-0915, 5 Watts @ 915 MHz.
Open Sound Control is an open-ended and simple protocol for communicating between multimedia devices, optimized for modern networks.
See http://opensoundcontrol.org/ for more.
11 Processing is an open-source programming language and IDE for
prototyping interactive media art based on Java.
See http://
processing.org/ for more.
3.2. Transmitters
The transmitter operates on the same 915 MHz band, and
was designed to function as a shield for an Arduino. The
shield contains the necessary connections for the transmitter module 8 and an SMA connector for either directly connecting a 1/4 wave whip-style antenna, or for connecting an
amplifier.
3.3. Experiments
Many features were experimented with early in development but not included in the current revisions of the Vibrobyte. Two we intend to pursue further are the possibility
of inductively charging the Vibrobytes, and amplifying the
wireless signal.
3.3.1. Inductive Charging
Design of a wireless communication interface that would
scale to large ensemble sizes suggested that an alternative
charging method should be explored. A short range inductive charging system was investigated. A completely wireless system would reduce the preparation time prior to a performance, and reduce the hardware necessary for maintaining an active battery state. Numerous designs were tested,
but we have not yet developed a compact solution with efficient power transfer characteristics.
9
10
cation as it allows for rapid prototyping and deployment of
code, and is cross-platform.
4.3. Serial Protocol
Figure 4. Mock up of Processing application visualizing
incoming OSC messages and balancing outgoing wireless
serial data.
../remove and ../reset with arguments specifying the respective groups. LEDs are controlled directly using /vibrobyte/led
with floats specifying the LED intensities, or the mode is
given via ../mode. Outputs are controlled using /vibrobyte/output
with arguments specifying envelope type, amplitude, frequency, sustain, and applicable outputs, and one-shot/repeat
is specified with ../mode.
This message structure was chosen to provide flexibility and expressiveness simultaneously. The OSC messages
were one of the first things to be agreed on, and guided a lot
of the other protocol choices and even some minor hardware
decisions.
4.2. Wireless Optimization
An application was developed using Processing and OSCP5 12
that received the control OSC messages, updated a virtual
representation of the Vibrobytes, and sent out serial data to
the wireless transmitter. To optimize the wireless serial messages, we decided to continually send out the entire state of
all the Vibrobytes. If we were to simply repeat the states in
order, there would be a massive latency between OSC messages being received and finally being sent out – especially
if one Vibrobyte was steadily having a value changed.
To overcome this potential for high latency, we developed a serial protocol that allowed partial states to be sent
(similar to the OSC messages), and then used a load-balancing
algorithm to send out the most recently updated states more
often while still regularly sending all the other states. To
make sure groups were handled correctly, every Vibrobyte is
modeled on the transmitting computer and each of its properties carries lastChange and lastTransmit variables, allowing for proper balancing.
A variety of visualizations and a few GUI elements allow
real time monitoring of the outgoing wireless data, incoming OSC messages, and Vibrobyte states 13 (see Figure 4).
Processing was an essential tool for developing this appli12 OSCP5 is a library for handling OSC messages in Processing. See
http://www.sojamo.de/libraries/oscP5/ for more.
13 A short clip of an early version of the application, with a naive load
balancing algorithm, is available at http://vimeo.com/1560346.
The serial protocol was envisioned before the OSC protocol, simply to get an idea of the potential refresh rate on a
large ensemble of wireless Vibrobytes. We started with 3byte packets describing, with low resolution, the entire state
of each Vibrobyte. From there we moved to higher resolution data in 4-byte packets, describing partial states. We discovered some things didn’t need that much resolution (e.g.:
vibrating motor amplitude), so we returned to 3-byte packets, still with partial states. When we discovered issues with
the receivers syncing to the packet stream, we introduced a
“null packet” that was sent whenever the wireless signal was
not in use.
Finally deciding that we wanted to constantly broadcast
as much state information as possible, we got dropped the
“null packet” and used the first bit of every byte to define
the start of a packet. The first byte of every packet also
contained 7 bits indicating what kind of data would follow –
allowing the packets to be anywhere from 3 to 8 bytes long.
This was a huge speed optimization and allowed the packet
stream to be easily synced with while remaining simple to
decode for the receiver.
The number of bits representing various parameters were
chosen empirically within the constraints of n 7-bit words.
For example, the output state is encoded with two bits describing wave shape, three for amplitude, two for sustain,
four for frequency, and three for applicable outputs (14 bits
total).
Serial data was sent at 28800 baud, allowing for anywhere from 360-960 packets a second, or a minimum of 60
Hz refresh rate for an ensemble of 6 players.
4.4. Vibrobyte Firmware
Firmware for the Vibrobytes was originally written in BASIC, but after a handful of revisions the firmware was prototyped on an Arduino and eventually on the Vibrobytes directly using the Arduino IDE and an AVR ISP MKII programmer connected to the ICSP header.
The first prototype firmware supported only the three outputs, with grouping and LED support added later. Once all
the functionality was in place and tested, the majority of the
firmware writing and debugging time went into implementing the serial protocol. The wireless “signal quality” level
was a very helpful feature for determining effectiveness of
the protocol as it was developed. Because the “header”
byte in each packet describes the content of the packet, it
serves as a checksum and allowed us to keep track of wellformatted and ill-formatted packets.
Figure 5. Max patch for Tele-morphosis.
5. Tele-morphosis: Performance and
Composition
Tintinnabulate is an improvisation ensemble of diverse instrumentation formed in 2005 during an “Experimental Telepresence” seminar under the direction of Pauline Oliveros.
Tintinnabulate has engaged in numerous network performances
beginning with Arizona State University, Wesleyan University, Brown University, SoundWire at Stanford University,
and more recently with the Roots Ensemble at Queens University in Belfast Ireland.
During the ICMC 2008 Tintinnabulate proposed a colocated performance between the Roots Ensemble in Belfast,
SoundWire in Palo Alto, California and Tintinnabulate Troy,
New York. There was almost no rehearsal time available.
Vibrobytes were employed for the first time in this performance to help coordinate the ensembles. Each performer in
the three groups was provided with a Vibrobyte. A Max/MSP
composition was written by Curtis Bahn which sent messages to each Vibrobyte cueing the performers with dynamically varying intensity, rhythm and instrumentation combinations. The performers (with the exception of Pauline
Oliveros who was wearing a prototype of a vibrating haptic
device) could observe bright multicolored LEDs bringing
them in and out of the improvisation. players entered when
their LEDs were on and exited when off accordingly. Furthermore, players could interpret the rhythms and intensities
of their LEDs freely when on.
This first test of the Vibrobyte technology was successful
for the musicians and helped to shape the music in a new
way that had not been possible before. Many more strategies
could be developed with rehearsal time to work more with
the technology.
Figure 6. Pauline Oliveros, Curtis Bahn and Jonas Braasch
playing in the Tintinnabulate ensemble in Troy, New York.
6. Discussion
Starting with a goal of developing a wireless haptic interface
for co-located performance, we regularly ran into the issue
of over-thinking relatively simple aspects of the project due
to the unconstrained nature of the problem. In the testing
stages we etched our own PCBs and designed our own wireless amplifiers, and later used prepackaged products for their
reliability, ease of use, and documentation. To reach the
same goal with only prepackaged products is not currently
possible, but future development would be aided by prototyping on Arduino Minis (which fulfill the size requirement
while remaining accessible and inexpensive), and creating
a mini-shield that implements the extra functionality (output amplification, LED indicators, and wireless communication).
7. Future Work
Development is currently focused on artistic explorations
with these devices. These devices will be useful in situations where complex multi-tempo compositions are desired,
or where a haptic connection is feasible but visual connections are not. We will explore a variety of compositions for
these devices: the intimacy of touch suggests the possibility
of more intimate co-located performances, like heartbeatdriven interaction and haptically guided improvisation between dancers and musicians.
Technically, we have a number of additional features planned.
Recent developments in resonant inductive charging[14] will
be explored as a more efficient way of powering many devices in large ensembles. The IR LEDs will be used to
uniquely track Vibrobytes in space over time with a webcam. This might used, for example, in a composition where
the strength of the haptic feedback is modulated by a performer’s distance from a remotely located performer. Other
wireless modules will be explored. 14 Finally, we will implement a variety of protocol improvements. For example,
the maximum and minimum frequency for actuating outputs
is fixed, but it would be significantly easier to specify this
at the beginning of a performance (or throughout a performance) rather than reprogramming all the modules.
8. Acknowledgments
Thanks to the entire Spring 2008 “Experimental Telepresence” seminar at Rensselaer, which provided a space for
this project to approach its full potential; and to all the performers and technicians involved in the ICMC performance.
The development of the Vibrobyte has been funded by an
RPI/EMPAC seed grant.
References
[1] E. Altinsoy, “Audio-tactile interaction in virtual environments,” Doct. Diss. Ruhr-University Bochum, Germany,
2006.
[2] H. Gillespie. “Haptics,” in P. R. Cook (Ed.), Music, Cognition, and Computerized Sound: An Introduction to Psychoacoustics, MIT-Press, 1999, pp. 229–245.
[3] “Magnetic Musical Training,” [Web site] 2007, [2009
Jan 29], Available: http://web.media.mit.edu/
˜grindlay/FielDrum.html
[4] G. Grindlay, “Haptic Guidance Benefits Musical Motor
Learning,” Symposium on Haptic Interfaces, IEEE Virtual
Reality, 2008.
[5] H. Steiner. “StickMusic: Using haptic feedback with a phase
vocoder,” in Proc. of the Conf. on New Instruments for Musical Expression (NIME), 2004, pp. 203-204.
14 For example, XBee, Radiotronix, and the Nordic nRF2401A modules
are all readily available for prototyping from distributors like Sparkfun
(http://www.sparkfun.com/commerce/categories.php?
c=16) in the US for ∼$20-$40 USD – more expensive than the current
wireless modules, but better supported and documented.
[6] X. Rodet, J. Lambert, R. Cahen, T. Gaudy, F. Guedy, F. Gosselin and P. Mobuchon. “Study of haptic and visual interaction for sound and music control in the Phase project,” in
Proc. of the Conf. on New Instruments for Musical Expression (NIME), 2005, pp. 109-114.
[7] J. Carlile and B. Hartmann. “OROBORO: A Collaborative
Controller with Interpersonal Haptic Feedback,” in Proc.
of the Conf. on New Instruments for Musical Expression
(NIME), 2005, pp. 250-251.
[8] C. Nichols. “The vBow: Development of a Virtual Violin
Bow Haptic Human-Computer Interface,” in Proc. of the
Conf. on New Instruments for Musical Expression (NIME),
2002.
[9] W. Verplank. “Haptic Music Exercises,” in Proc. of the Conf.
on New Instruments for Musical Expression (NIME), 2005,
pp. 256-257.
[10] E. Gunther, G. Davenport and S. OModhrain. “Cutaneous
Grooves: Composing for the Sense of Touch,” in Proc. of the
Conf. on New Instruments for Musical Expression (NIME),
2002.
[11] B. Verplank, M. Gurevich and M. Mathews. “THE PLANK:
Designing a simple haptic controller,” in Proc. of the Conf.
on New Instruments for Musical Expression (NIME), 2002.
[12] A. Toney, L. Dunne, B. H. Thomas and S. P. Ashdown, “A
Shoulder Pad Insert Vibrotactile Display,” Proc. of the 7th
IEEE Int. Symposium on Wearable Computers (ISWC), 2003,
pp. 35-44.
[13] S. K. Nagel, C. Carl, T. Kringe, R. Märtin and Peter König,
“Beyond sensory substitution – learning the sixth sense,” J.
Neural Eng. 2, 2005, R13-R26
[14] A. Kurs, A. Karalis, R. Moffatt, J. D. Joannopoulos, P.
Fisher, M. Soljačić, “Wireless Power Transfer via Strongly
Coupled Magnetic Resonances,” Science, Vol. 317, 2007, pp.
83-86.