Music Coloring
for Biofeedback
Presented By
Eyal Balla
ballaeya@tau.ac.il
The Demo System
Original Music
EKG Monitor
PC with A/D
and music coloring software
Patient
Colored Music
Music Coloring What is it?
Music coloring is changing of music in
different aspects (such as pitch, rhythm,
timber, volume, reverberation and other
aspects of the music) in order to embed
data into the music.
Music Perception in Humans
 Recent research shows that several distinct
neural functions exist in music perception:
Volume
Pitch
Rhythm
Timber
Direction of source of music
 This introduces the possibility of
multidimensional display with audio.
 Moreover music has been shown to have effect
on human breathing and heart rate
Music Perception in Humans
Visual Display
Auditory Display
Number of dimensions is Able to percept many
limited to 3D
dimensions such as
pitch, rhythm, timber etc.
Uses full attention of
human using it.
Uses partial attention.
Other tasks can be done
simultaneously.
Previous Works in Auditory Display
Sonification Mappings:
sMax: A Multimodel Toolkit for Stock Market Data
Sonification. Maps stock market data into multistreams of audio. User has control of mapping.
Requires training. Mapping is through pitch, pulse
speed, brightness and attack time of different
instruments
MarketBuzz. A sonification of real-time financial
data. 4 behaviors were identified and sonified in
beeps and pitch sonifications. Requires training.
Previous Works in Auditory Display
Polyrhythm in the Human Brain. Sonification of
EEG brain waves in four different streams
according to EEG wave types.
Listening To the Mind listening. Sonification of
ECG heart rate and two streams of EEG with
difference between the two temporal sections
(above ears) were used as stereo positioning.
Previous Works in Auditory Display
Feedback Systems:
Auditory Feedback of Human EEG for Direct
Brain Computer Communication. A spelling
machine for the handicapped was built. User
would use SCP EEG brain waves to control it.
Learning was through auditory instruction. They
also found other EEG waves controlable using
multi-stream audio teaching.
Previous Works in Auditory Display
Previous Works in Auditory Display
Data Analysis through Auditory Display:
Applications in Heart Rate Variability. The work
deals with creating a auditory system for heart
rate variability analysis. The auditory display
created is made of several streams each
representing a different aspect of heart
condition. The work mentions the importance of
using pitch and volume.
Conclusion of Current Works
Several orthogonal features of music have
been proved as distinguishable by humans
in experiments
Sonification is a good way of displaying
multidimensional data.
Works usually create new music according
to data streams extracted from the raw
data.
Current Directions
Last year’s workshop created an interface
which inserts samples of EKG into
MAX/MSP environment using a A/D
sampler.
The workshop showed that online realtime analysis of this data is possible using
the MAX/MSP development environment.
Current Directions
We are looking for auditory “Axis” which
can be precept orthogonally.
We are using already build components
which color music to learn the reaction
graph of humans to change in different
music features.
References
The structural Components of music
perception. Herve´ Platel, Cathy Price, Jean-Claude Baron,
Richard Wise, Jany Lambert, Richard S. J. Frackowiak, Bernard
Lechevalier and Francis Eustache
Max/Msp 4.5 Documentation.
http://www.cycling74.com/products/dldoc.html
sMax: A Multimodel Toolkit for Stock
Market Data Sonification. Fabio Cifariello Ciardi
MARKETBUZZ: Sonification of real-time
financial data. Petr Janata and Edward Childs
References
 Polyrhythm in the Human Brain. Thomas Hermann,
Gerold Baier, Markus Muller.
 Listening To the Mind listening. Hans van Raaij
 Listening to the mind listening concert.
http://www.icad.org/websiteV2.0/Conferences/ICAD2004/concert.htm
 Auditory Feedback of Human EEG for Direct
Brain Computer Communication. Thilo Hinterberger,
Gerold Baier, Jürgen Mellinger, Niels Birbaumer
 Data Analysis through Auditory Display:
Applications in Heart Rate Variability. Mark Ballora
MAX/MSP Introduction
MAX/MSP is an object oriented
development environment for music
synthesis and creation.
The MAX part of the environment handles
MIDI format manipulation.
The MSP part of the environment handles
signal processing and manipulation.
MAX/MSP Example
The following example outputs via audio
several 2 dimension data items using 2
streams of data for pitch and volume.