Listening to the Blink of an Eye: Using the Sonification of EEGs to
Recognize and Classify Types of Brain Activity
by
Stacy Philip
AN HONORS THESIS
for the
HONORS COLLEGE
Submitted to the
Honors College
at Texas Tech University in
partial fulfillment of the
requirement for
the degree designation of
HIGHEST HONORS
MAY 2015
Approved by:
__________________________________________________
Carl Seaquist, Ph.D., Thesis Director
Associate Professor, Department of Mathematics & Statistics
______________
Date
__________________________________________________
Brock Williams, Ph.D., Second Reader
Professor, Department of Mathematics & Statistics
______________
Date
__________________________________________________
Keira V. Williams, Ph.D., Thesis Instructor
Assistant Professor, Honors College
______________
Date
__________________________________________________
Michael San Francisco, Ph.D.
Dean, Honors College
______________
Date
The author approves the photocopying of this document for educational purposes.
ABSTRACT
A series of clinical tests were conducted at a Lubbock hospital, from which ten patient
electroencephalogram (EEG) recordings were selected for further analysis. These recordings
consisted of two normal and eight abnormal EEGs, which were stripped of all personal
identifying information. Recent publications indicate that sonification (converting data to sound)
allows the human ear to analyze series data and detect irregularities that might otherwise go
unnoticed. Since brain rhythms are typically lower than the human hearing range, signalprocessing techniques, including modulation, Fourier transforms, wavelet analysis, and digital
filtering, can be applied in an attempt to convert EEG signals to sound in the audible human
hearing range. In particular, this exploratory study strives to sonify blink artifacts and alpha
rhythms in an excerpt of an EEG recording from a ‘normal’ patient using modulation to produce
an A-C-E chord. The objective is to demonstrate that in addition to traditional visual analysis,
auditory acuity may be useful in the analysis of EEGs and in the identification of important EEG
features, without the loss of information.
ii
ACKNOWLEDGEMENTS
This author would like to extend gratitude to the Proactive Recruitment in Introductory
Science and Mathematics (PRISM) undergraduate research program for funding this research
and for all of the doors it has opened through research in regards to providing a wholesome and
enriching college experience. This author is also very thankful for Dr. Carl Seaquist for allowing
her to work on such an intriguing and fun project, while being a wonderful mentor throughout
this entire endeavor. This author is also grateful for fellow researcher Kate Ehnis, for the
partnership, friendship, and teamwork without which this venture would not be a success. Also,
this author appreciates Dr. Renato Gonik and Frederick Ramirez for their contributions and
assistance to make this project efficacious. In addition, special thanks to Dr. Ram Iyer for his
succors and support of this research. Also, this author thanks Dean Michael San Francisco, Dr.
Keira Williams, and Dr. Brock Williams for their helpfulness, mentorship, and guidance
throughout the Honors Thesis writing process. This author would also like to extend thanks to
the Texas Tech University Mathematics and Statistics Department and the Honors College.
iii
TABLE OF CONTENTS
TITLE PAGE.……………………...………………………………………………………………i
ABSTRACT………………………………………………………………………………………ii
ACKNOWLEDGEMENTS………………………………………………………………………iii
TABLE OF CONTENTS ………………………………………………………………………...iv
Chapter 1…………………………………………………………………………………………..1
Introduction………………………………………...………………………………...……1
Background………………………………………………………….…………….………2
Electroencephalograms…………………………………………………….……...3
Eye Blink Artifacts…………………………………………………….………….3
Listening to Data…………………………………………………….…………….5
Chapter 2…………………………………………………………………………………………..7
Previous Sonification Work…………………………………………………………….....7
Chapter 3…………………………………………………………………………………………15
Methodology……………………………………………………………………………..15
Data Selection...………………………………………………………………….15
MATLAB and EEGLAB………………………………………………………...17
Channel Selection………………………………………………………………..17
Montages…………………………………………………………………………18
Filtering………………………………………………………………………..…20
Resampling, Modulation, and Chord Selection………………………………… 21
Chapter 4…………………………………………………………………………………………25
Results……………………………………………………………………………………25
Conclusion……………………………………………………………………………….26
Chapter 5…………………………………………………………………………………………27
Future Work……………………………………………………………………………...27
REFERENCES…………………………………………………………………………………..29
APPENDIX A (MATLAB CODE)...……………………………………………………………32
iv
Chapter 1
This chapter discusses introductory information that will be further discussed throughout this
paper for basic knowledge and reference.
Introduction
Electroencephalograms (EEGs) monitor the brain’s electrical functions by comparing
differences in surface voltages. The analysis and display of EEGs for visual analysis is
ubiquitous. When one is able to understand a normal EEG, it is much easier to identify
abnormalities and better understand how the brain works. While a normal EEG does not exclude
a clinical diagnosis, an abnormal finding on EEGs supports a diagnosis, indicates cerebral
dysfunction, or may have nothing to do with the reason that the study was performed (Tatum,
2008). Therefore, EEGs are quite utilitarian and can be an advantageous diagnostic tool in the
identification of abnormal brain functions.
Visually learning how to classify various types of brain activity is a complex, rigorous,
and time-consuming process. Recent work indicates that auditory analysis may be useful in
detecting irregularities in EEGs and assessing EEG data apart from traditional visual analysis
(Khamis, Mohamed, Simpson, & McEwan, 2012). Since brain waves consist of frequencies
inaudible to the human ear, it is necessary to use digital signal processing techniques to convert
the signals to sound without losing valuable information. The auditory analysis of EEG signals
will aid in the early detection of abnormal EEG signals to better understand the human brain, and
hopefully improve diagnosis and treatment of neurological abnormalities in the future. Thus,
training non-experts how to recognize various types of brain activity through simple auditory
means will allow those with neural irregularities to gather information about their condition to
determine appropriate medical regimes as treatment in a timely and efficient manner.
1
Background
Research on the sonification of electroencephalograms has been attempted in the past,
though the work is still novel in this developing field. Dale Purves’s Neuroscience (2008)
textbook was used over the course of this exploration because this text addresses complex
neuroscience topics in an approachable manner, using the analytical tools of genetics, molecular
and cell biology, systems anatomy and physiology, behavioral biology, and psychology.
Therefore, it was helpful for the study of neuroanatomical and neurophysiological concepts that
were necessary during the research process to better understand brain structure and function.
Electroencephalograms
To begin work in this area, Tatum’s Handbook of EEG Interpretation was referenced in
order to learn about key features of electroencephalograms and their functional uses throughout
the course of this research (Tatum, 2008). Tatum explains that the basis of cerebral potentials
relies upon the nervous system’s intrinsic electrophysiological properties. Being able to identify
the generator source(s) and electrical field(s) of propagation provides the foundation for
recognizing electrographic patterns that denote the expression of the “brain waves” as normal or
abnormal. In general, the majority of regular EEGs recorded at the surface of the scalp represent
pooled electrical activity produced by large numbers of neurons. Electrical signals are created
when electrical charges move within the central nervous system. Neural function is normally
maintained by ionic gradients established by neuronal membranes. Sufficient duration and length
of small amounts (measured in μV) of electrical currents of cerebral activity are required to be
amplified and displayed for interpretation (Tatum, 2008).
From the patient’s scalp, electrodes conduct electrical potentials to an electrode box, also
known as a jackbox. Subsequently, a montage selector allows EEG signals to pass through
2
amplifiers before filtering and auxiliary controls regulate the signal output. Data display follows
acquisition and processing and has a wide variety of data presentations for EEG interpretation.
Electrode placement has been standardized by an international 10–20 system that uses
anatomical landmarks on the skull. These sites are then subdivided by intervals of 10% to 20%
and to designate the site where an electrode will be placed. The designations for where on the
scalp electrodes are assigned in this 10-20 system are: Fp (frontopolar), F (frontal), T (temporal),
O (occipital), C (central), and P (parietal). Afterward, numbers combined following the letters
for location designate either the left (odd numbers) or right (even numbers) hemisphere of
electrode placement, where the “z” designation reflects midline position (i.e., Cz = central
midline) as seen in Figure 1 from the Handbook of EEG Interpretation below.
Figure 1. The black circles indicate the 10-20 electrode placement system.
3
Eye Blink Artifacts
The Handbook of EEG Interpretation aided in the identification of artifacts in the EEG
signals and then interpreting them to distinguish between normal and abnormal EEGs. Due to the
fact that recording the brain’s electrical activity may come with noncerebral interference, diverse
generators of nonphysiological and physiological artifacts may deceive the interpreter to believe
that the apparent sources are abnormal or epileptiform. When in doubt, an EEG interpreter must
assume that the source is an artifact until proven otherwise (Tatum, 2008).
Specifically, the Handbook of EEG Interpretation text was exceptionally useful to
identify eye blink artifacts, which are generated by the electrical potential produced by vertical
movement of the eye. Normally, the eye functions as an electrical dipole with a relative
positivity of the cornea compared to the retina. The potential created is a DC potential of higher
amplitude (mV) than the amplitude produced by the brain (μV). The artifact is produced in the
electrodes around the eye (FP1/2) during vertical eye movements. With an eye blink, the cornea
rolls up with resultant positivity in the FP1/2 electrodes relative to the F3/4 electrodes and
creates a downward deflection during the normal Bell’s phenomenon (Tatum, 2008). Bell’s
phenomenon is a medical sign that enables an upward and outward movement of the eye to be
observed when an attempt is made to forcefully close the eyelids (Francis & Loughhead, 1984).
Tatum (2008) states that by having electrodes record above and below the eye helps distinguish
the brain as the “generator,” where every channel is the same polarity, as opposed to it being
distinguished as an artifact, where there is opposite polarity in electrode sites above and below
the eye (Tatum, 2008).
Having a vertical eye blink artifact present helps determine the state of the patient as
being awake. In the same way, during drowsiness, slow rolling (lateral) eye movements are
4
beneficial. Lateral eye movements are usually easily recognized because they create phase
reversals in the anterior temporal derivations that are of opposite polarity on opposite sides of the
scalp EEG. When the eyes move to the left yielding a positive phase reversal in F7 due to the
cornea polarity, the homologous F8 electrode site demonstrates a negative phase reversal from
the retina. The positive phase reversals noted at the F8 derivation is due to the proximity of the
cornea. The homologous F7 electrode site is negative due to the conjugate effect from the retina
(Tatum, 2008).
Eye movements can be detected by using a single channel connecting the right upper
lateral eyebrow and the left lower lateral eyebrow. However, due to the fact that vertical eye
movements are usually the source of confusion, bilateral infraorbital electrodes referred to the
ipsilateral ear as a reference may better represent the eye as a dipole and demonstrate phase
reversals that are out-of-phase with cerebral activity when due to eye movements (Tatum, 2008).
Listening to Data
Transforming EEG data signals that are traditionally only analyzed visually by a trained
expert into the auditory spectrum requires interpreting the data sequence, which is usually a time
series, through signal processing as an audio waveform. Listening to data provides a means by
which large datasets can be explored by mapping properties of the EEG data points into music.
Then, by hearing this continuous, dynamic, and broad-spectrum data, a new platform is provided
in which to interpret the signals. Mills (2012), explains that this idea of “Electroacoustics has
been at the forefront of signal engineering and signal processing since ‘the transducing 1870s,’
when the development of the telephone marked the first successful conversion of a sensuous
phenomenon (sound) into electrical form and back again.” Thus, hearing is a simple technique to
perceive data, because there is already a strong relationship between hearing and electronic
5
signal processing.
Signal Processing for Communications (2008) was referred to in order to learn more
about how to understand and modulate digital signals. The authors Martin Vetterli and Paolo
Prandoni, produced a text from course notes developed over a period of a dozen years teaching
undergraduates the basics of signal processing or communications at École Polytchnique
Fédérale de Lausanne. This textbook has valuable information on the basis of various digital
signal-processing techniques, which were consulted over the course of the research project.
6
Chapter 2
This chapter discusses former work conducted by individuals in the field of sonification and how
this specific research team sought to conduct its own sonification study using EEG signals.
Previous Sonification Work
In 1999, Barrass and Kramer introduced the many benefits that sonification, also referred
to as audification, provides for a plethora of disciplines, including practical uses in the medical
field, in assistive technologies to help blind individuals be more independent, in assisting
engineers analyze computer models and simulations of complex systems, in scientific
visualizations, in emergency services, and in aircraft cockpits, among many other examples. The
authors also survey and outline the many methods to designing and realizing a sonification.
Kramer classifies methods for designing an auditory display along a spectrum between analogic
and symbolic. An analogic representation has an immediate and intrinsic correspondence
between the sort of structure being represented and the representation medium. There is a one-toone mapping between points in the data space and points in representation space. On the
contrary, a symbolic representation categorically denotes the thing being represented and
relations within the representation do not necessarily reflect intrinsic relationships in what is
being represented. The authors also discuss the semiotic distinctions (syntactic, semantic and
lexical) that have also been used to classify the best-known methods of auditory display
(earcons, auditory icons and parameter mapping).
Barrass and Kramer (1999) also present the issues that have arisen from this range of
approaches and applications of converting data into sound. For example, listeners need to
correctly deduce data relations from the sounds. They say that “one of the reasons that
audification fails for arbitrary data, such as stock market figures or daily temperatures, is even a
7
slow playback rate of 8000 samples/s requires many data points to make a sound of any duration.
The resultant short sound does not reveal valuable information to the listener. Even for long
sequences, the sounds do not resemble natural or familiar noises, because arbitrary data points
seldom obey the physics of natural sounds transmitted through air.” In addition, some
classifications of sound events tend to be categorical. They say that relevant changes in data
should insure a change in what is perceived and signify meaningful changes in the data. In
addition, the authors relate sonification to other areas where similar issues have arose including
human-computer interaction, sound design for commercial products, and computer music.
In 2002, Hermann, Meinicke, Bekel, Ritter, Müller, and Weiss conducted a study that
presented different techniques in order to produce acoustic representations for EEG data. Their
data was received from “psycholinguistic experiments where subjects are exposed to three
different conditions based on different auditory stimuli.” The stimuli included Spoken Language
(story) with an Austrian-German speaker (mean duration of two minutes); Pseudo Speech, which
consisted of auditory patterns generated by amplitude and frequency modulation using a base
frequency of 200 Hz and an amplitude envelope that resembles the real spoken sentences; and
EEGr, in which the EEG was recorded for two minutes during rest with open eyes. They utilized
nineteen scalp electrodes that were placed according to the 10-20 positioning system and
measured against the averaged signal of both earlobes. Before beginning their analysis, the
signals were band-pass filtered between 0.3 Hz to 35 Hz and digitized using a 16-bit quantization
and a sampling rate of VSR = 256 Hz. They hoped to unearth elements of neural processes that
correlate with high-level cognitive activity. The three types of sonification described in their
paper include spectral mapping sonification, which provides a direct assessment of the recorded
data; distance matrix sonification, which enables nonlinear long range correlations to be detected
8
at high time resolution; and differential sonification, which summarizes the comparison of EEG
measurements where each subject experiences different conditions. Since EEG data consists of
multiple time series, it contains a lot of noise, which complicates automatic pattern detection that
requires high-developed human auditory skills in signal/noise separation. The authors believe the
trained listener can conclude coarsely which electrodes and what frequency bands were affected
by the relevant condition.
In 2004, Hinterberger and Bair introduced a device known as POSER, which stands for
the Parametric Orchestral Sonification of EEG in Real-Time, for the self-regulation of brain
states. This device is implemented in the Thought-Translation-Device TTD, which is a braincomputer interface designed for the communication with completely paralyzed patients. The
device enables auditory feedback of multiple EEG characteristics in real time. Specifically, each
of six frequency bands is assigned an instrument of a MIDI device. The time-dependent
parameters extracted from each frequency band modulate the timing, pitch, and volume of the
instrument.
In a pilot study, Hinterberger and Bair tested the ability of subjects to perform a simple
discrimination task using this parametric sonification in real time. The EEGs in the study were
sampled at a rate of 256 per second in an amplitude range of +/-1mV and a resolution of 16 bits.
The pilot study was carried out with the ProComp+ amplifier system. The raw EEG signal is
stored to disk and complemented with status information such as a code for the feedback task
and other events during a trial. Their data processing was performed at 256 samples/s, while
sonification was paced at a rate of 127 per second, thus offering a sufficient update rate and
timing resolution. For excellent real-time behavior, a 1.6 GHz Pentium IV machine running
under MS Windows XP was utilized. In regards to processing, after calibration of the EEG-
9
signal into μV, the spatial filter offers the user to arrange the EEG-channels by a linear
combination of incoming channels. In the standard setting, three channels of EEG were used,
including the Cz-mastoids, C3-mastoids, and C4-mastoids. They roughly corrected eye
movement artifacts by subtraction of a fixed factorized amount of the vertical Electrooculogram.
Afterwards, they then band-passed the EEG signal into several different frequency bands and
used an FIR (finite impulse response) filter with either 127 or 63 coefficients used. In order to
sonify EEG signals that were below 12 Hz, they triggered the touches of a note at the maxima of
a wave. In addition, the potential differences between subsequent extrema (maxima minus
previous minima or minima minus previous maxima) were calculated. The authors used the
inverse time difference between consecutive maxima of a band pass filtered signal to serve as an
estimate of the “instantaneous” frequency of a signal. In order to lower the update rate of the
sounds to half the sampling rate, they down-sampled all output signals.
For the study, ten healthy participants, five male and five female, between the ages of 20
and 44, were selected to receive full orchestral feedback of their brain activity from one channel
of EEG (Cz versus C3). The subjects all appreciated the characteristic timbres of instruments and
found the stereo presentation helpful for distinction, e.g. between theta and alpha rhythms. The
participants were instructed to focus their attention alternately on two different sounds. The
sequence of the tasks was defined by the computer in a balanced order and symbolized on a
screen by two vertically arranged rectangles. A red colored upper rectangle asked participants to
focus attention on the rhythmic vibraphone sounds (alpha, theta, and gamma). A red colored
lower rectangle asked them to focus attention on the smooth sounds (SCP, delta, and beta). T-test
revealed significant differentiation between the tasks in at least one parameter for all participants.
The use of the POSER device in a closed-loop feedback application leads to a new level of
10
interaction with one’s state of consciousness.
In 2006, Baier, Hermann, Sahle, and Stephani, described a technique used to sonify
rhythmic activity of epileptic seizures as measured by human EEG, particularly using eventbased mapping of parameters, which relates information in regards to auto- and crosscorrelations of the multivariate data. They used a group of patients with childhood absence
seizures (because they only show one epileptic EEG symptom), and they found consistent intrapatient conservation of the rhythmic pattern as well as inter-patient variations, especially in terms
of cross-correlations. Therefore, they were able to determine that sonification in clinical
monitoring is now possible. However, for real-time application, one fixed set of sonification
parameters might not be sufficient, so one would need to adjust the sonification interactively
according to features such as the state of the patient, the pathologic features being detected, and
the amount of artifacts. This could be solved by providing a graphical user interface for real-time
interaction with the main parameters involved in the sound rendering.
The following year, in 2007, Baier, Hermann, and Stephani introduced a multivariate
event-based sonification that, in addition to displaying salient rhythms, uses pitch and spatial
location to provide such cues in a multi-speaker environment. Using clinical recordings with
epileptic seizures, they have demonstrated how the spatio-temporal characteristics of EEG
rhythms can be perceived in such sonifications. They support the argument that one of the
advantages of auditory displays stems from how the human sense of listening deals with multiple
events that arrive in one sound stream, which they say can be particularly suited for data mining
and data exploration of multivariate data sets. They also agree that there have not been any
explicit medical applications of the auditory representation of space in the case of the human
EEG. The data was used in a source density montage, which creates an individual reference for
11
each electrode defined as the weighted sum of the potentials from its nearest neighbors. This
reference acts as a spatial high-pass filter that improves localization and reduces spurious crosscorrelations compared to common references, particularly in the case of generalized rhythms.
They were successful in obtaining an EEG sample in real time. This work is significant in that it
shows that the sonification of EEGs has been successfully executed; however, it only deals with
the spacio-temporal aspect of the human brain rather than an all-encompassing approach.
Also in 2007, Campo, Hoeldrich, Eckel, & Wallisch described two software
implementations for EEG data screening and realtime monitoring via sonification. Their study
explains how the alpha range is between 8 and 13 Hz, and it is associated with “general state of
relaxedness, and non-activity of the brain region for visual tasks; it is most prominent with eyes
closed.” For each of the two software designs, the EEG signals were split into frequency ranges
which closely correspond to the traditional EEG bands constant relative bandwidths were used of
one octave each to attain a more equal energy balance between the bands. The ranges are:
deltaL(ow) from 1-2 Hz, deltaH(igh) from 2-4 Hz, theta from 4-8 Hz, alpha from 8-16 Hz,
which is wider than the traditional value of 13, and includes the mu-rhythm band, which is a bit
wider than the traditional value of 13, beta from 16-32 Hz, and gamma from 32-64 Hz.
Trevisan and Jones’s (2010) study provides a simple and portable system that is able to
generate MIDI output based data collected through an EEG collecting device. This device is
beneficial because the therapeutic effects of listening to the music created by the brain waves
have been seen in different documented cases to successfully treat health problems. This paper
seeks to make the most accurate representation of brainwaves, which had yet to be performed.
Several neuroscientists have adopted MIDI protocol in order to monitor brain waves in real time.
Therefore, the authors alternated several variants of positioning, including power spectrum,
12
event-related potential, and spectral centroid. This study contributes to the field of neurofeedback
by providing criteria tools for assessment.
Finally, “A Review of Real-Time EEG Sonification Research” was also used in the
research process in order to obtain a thorough and extensive overview on the work that has been
conducted on the sonification of EEGs thus far. The authors present an overview and critical
review of the principal research to date in EEG data sonification. First, the authors identify
several sub-domains of real-time EEG sonification and discuss their diverse approaches and
goals. Then, they describe the search and inclusion criteria, and present a synoptic summary
table spanning over fifty different research projects or published research findings. Thirdly, they
analyze sonification approaches to the various EEG data dimensions, such as time-frequency
filtering, signal level, and location, before going on to consider higher order EEG features.
Lastly, they discuss future application domains, which may benefit from new capabilities in the
real-time sonification of EEG data. They hope that their critical review may help to reduce
research fragmentation and may aid future collaboration in this emerging multidisciplinary area,
which was definitely made useful throughout the course of this study.
The limitations of this study are reflected in the fact that there is a lack of common
terminology due to the high level of multidisciplinary collaborations in the field of EEG
sonifications. Also, many papers lack technical details on the sonifications used, the EEG
recording equipment, and the processing techniques. Future papers addressing the topic of EEG
sonification, in any of the application domains, would benefit from providing more
documentation on the EEG methodology and sonifications used. EEG recording methodology
description should include: equipment used (amplifier models, electrodes), electrode placement,
channels used and referencing technique, EEG sampling rate, applied low and high pass pre-
13
filtering, signal processing details (e.g., FFT block size), artifacts reduction. In addition, there
has not been proper validation done on sonification methods in order to determine which is the
most efficient for representation of EEG data for particular purposes with proper control
methodologies.
14
Chapter 3
This chapter discusses the methodology used to conduct this study as a test of proof to determine
if EEG signals could successfully be transformed into sound.
Methodology
Ten patient Electroencephalogram (EEG) recordings were selected from a group of
clinical EEGs conducted at a Lubbock hospital. These recordings include two normal and eight
abnormal EEGs, which were stripped of all personal identifying information. The patients that
these recordings were taken from include individuals who were ‘normal,’ meaning that they did
not have any apparent medical conditions, to those who were comatose, to some who were
epileptic.
Data Selection
For the initial study, to test if it was feasible to convert the electroencephalogram signals
into the auditory spectrum, a normal patient’s recordings were selected for analysis. In particular,
a seventeen-second excerpt (from 79 seconds to 96 seconds) from the EEG recording was chosen
for further exploration. This selection contained blink artifacts, followed by the eyes closing,
leading to alpha rhythms, as seen in Figure 2. The alpha rhythms are characterized by their
frequency between 7.5 and 13 Hz, and usually appear when the eyes are closed and the subject is
relaxed.
15
16
Figure 2. Raw data of blink artifacts, eyes closing, and alpha rhythms viewed in EDF Browser
MATLAB and EEGLAB
These raw signals were read into a numerical computing software program known as
MATLAB, specifically using a compatible EEGLAB toolbox, which can process data from
electroencephalography, among other signals through the aid of coding. While using EEGLAB,
the normal patient’s EEG data, saved file in EDF format, is uploaded in MATLAB.
Channel Selection
From the EEG recording, electrodes were positioned on the patient’s scalp using the 1020 placement system to obtain signals from the brain’s frontal, occipital, and parietal regions.
The odd numbered electrodes correspond with the left side of the brain, while the even numbered
ones correspond to the right side of the brain. Utilizing a multi-platform viewing application
known as EDF Browser, the data received from the electrodes placed on these regions can be
visualizes as displayed in Figure 2. Though several electrodes were placed on the patient’s head
during the EEG recording to obtain data, the specific channels used for this study’s analysis were
those from the Fp1, Fp2, F3, F4, A1, A2, O1, and O2 electrodes (although data from all
electrodes is initially all loaded into MATLAB). The researchers chose to use strictly these
channels for further analysis, because according to the Handbook of EEG Interpretation, these
key channels are those that are primarily associated with the blinking of an eye. Therefore, this
would eliminate having to exclude extraneous data retrieved from EEG signals that gather data
from other parts of the brain later on in the analysis stage.
Based on suggestions from Niedermeyer’s Electroencephalography: Basic Principles,
Clinical Applications, and Related Fields, the optimal way to detect alpha rhythm amplitude is
by using a referential montage to the ipsilateral ear, taking into careful consideration the
17
interelectrode distances. The authors of this text, Schomer and Lopes da Silva, say, “the
maximum alpha voltage is usually over the occipital region as such, but a bipolar montage with a
parasagittal array may obscure rather than reveal the true alpha maximum.” They have found that
“alpha amplitude may be quite small in the channels displaying P3-O1 and P4-O2 because of
massive homophasic activity, which results in canceling out” (Schomer and Lopes da Silva,
2010).
Montages
Representations of EEG channels can be exemplified through various montages, also
known as arrays, which are combinations of multiple derivations or electrode recording
combinations. Most commonly, the two different types of montages that are used are the
Monopolar (Referential) and the Bipolar. Referential montages include using the common
average reference, the Cz reference, or an ear reference. Bipolar montages consist of an adjacent
pair of electrodes of the 10-20 system of electrode placement and potentials are localized based
on phase reversal; the types of bipolar montages include Longitudinal (anterior to posterior) and
Transverse (left to right).
For this study, a referential montage was used to display the EEG signal data, because it
provides a perspective in which there is a common reference electrode for all of the channels.
The reference is placed in an electrically quiet spot on the patient in order not to contaminate the
other leads. A specific type of referential montage uses ear reference (contralateral and
ipsilateral), which provides an advantage in that there is little to no cancellation because there are
long interelectrode distances, which allow potentials to appear with maximal amplitude waves,
making it easier to read. Both bipolar and ipsilateral ear referential montages were used during
18
this study, as seen in Figure 3 below, with the Fp1, Fp2, F3, F4, A1, A2, O1, and O2 electrodes
highlighted.
Figure 3. Bipolar and referential montages with primary electrodes chosen for sound file
highlighted
Specifically, a bipolar (double banana) montage is particularly useful at blink detection
so it takes readings of Fp1-F3 and Fp2-F4 and a referential montage (ipsilateral ear) of channels
aids in alpha rhythm detection using readings of O1-A1 and O2-A2. Therefore, these eight
channels were picked off for further analysis. When the raw signals from EDF Browser using the
bipolar and referential montage were read as code into MATLAB, raw, unfiltered data for each
of the eight electrodes was obtained, as seen in Figure 4.
19
Figure 4. Raw data using bipolar and referential (ipsilateral) montages in MATLAB
Filtering
The montages were used for further exploration and filtering to remove extraneous noise.
In particular, a Butterworth filter, also known as a maximally flat magnitude filter, was used
because it is a signal processing filter that has the flattest frequency response possible in the
passband, as seen in Figure 5. Therefore, undesired frequencies that are too high and too low are
removed, leaving a uniform sensitivity for the desired frequencies.
20
Figure 5. Butterworth (BandPass) Filter applied to the bipolar and referential ipsilateral
ear montage in MATLAB
Resampling, Modulation, and Chord Selection
The bandpass filtered signals were then resampled at a higher frequency (sped up) of
44,100 Hz, as seen in Figure 6.
21
Figure 6. Bipolar and referential ipsilateral ear montage filtered and resampled at 44,100
Hz
Then, a wave file was written out for the resampled signal. Since this frequency is too high to be
heard by the human ear (humans can only hear between 20 – 20,000 Hz), these EEG signals
were modulated into the auditory range, using the tones of A3 (frequency of 220 Hz), A4
(frequency of 440 Hz), C5 (frequency of 523.25 Hz), and E5 (frequency of 659.25 Hz), thus
creating an A-C-E chord. These specific frequencies were chosen, as they seemed to produce a
harmonious sound to the ear.
22
Figure 7. Bipolar and referential ipsilateral ear montage filtered and modulated at 44,100
Hz with 16 bit
New files were written out, as seen in Figure 7, and a digital audio workstation known as Adobe
Audition was used to mix the sound files. To do so, the newly written out and modulated EEG
signal files were read into the software. A sampling rate of 44,100 with a bit depth of 16 bits,
was selected under the Multitrack options. Then, the .wav files were dragged into tracks 1
through 4. The volume for each track was adjusted to where all tracks were equal. Then, all
tracks except the Master were deleted. Ultimately, this produces a sound file that can be listened
to for further analysis and playback, as seen in Figure 8.
23
Figure 8. Modulated EEG signals read into Adobe Audition
24
Chapter 4
This section discusses the results and conclusions obtained from this study.
Results
By using the Adobe Audition digital audio software, it was relatively simple to place the
code that was written to take the EEG signals from EDF Browser and place it through an
interface where waveforms could easily be entered and edited to produce the desired outcome.
This was particularly beneficial because having the capacity to work with multi-tracks enabled
the data from multiple electrode channels to be read and played simultaneously. The particular
workstation software made available through Adobe Audition has enabled real-time and nondestructive processing and also supplied plugins for noise reduction and FFT equalization. Thus,
as seen in Figure 7, the modulated signals were indeed able to translate the original blink
artifacts and alpha rhythms seen in the EDF Browser into the auditory spectrum through Adobe
Audition. It does not appear that any data was lost throughout this conversion process and excess
noise does not interfere with being able to identify when the blinks are occurring or when the
alpha rhythms are detected.
Therefore, it now possible to detect blinks and determine whether the eye is open or
closed simply by listening to appropriate EEG channels. This then sufficiently supports the test
of proof conducted to determine if sonification of visual data was indeed possible. Through these
methods, it can be seen that it is feasible to determine when the blink of an eye occurs through
listening to the sound produced when the appropriate channels are first converted into auditory
range frequencies through code and then transformed into an acoustic arena. By demonstrating to
the general public what this noise sounds like and training them to be able to identify this sound,
it would be possible for non-specialists to classify this type of brain activity when hearing the
25
correct pattern of noise. This then revolutionizes what trained physicians have traditionally been
able to classify only through visual techniques into something in audible form that can be offered
to non-specialist individuals to classify brain patterns.
Conclusions
In this study, the Adobe Audition Software was in fact able to keep the original blink
artifacts and alpha rhythms intact when the signals were read through as a sound file, as
evidenced by a pattern very similar to the one seen with the raw data from the EEG excerpt in
EDF Browser. Although the blinking of an eye is visually an easily detectable artifact in an EEG,
it is now possible to detect blinks and to determine whether the eye is open or closed simply by
listening to the correct EEG channels.
26
Chapter 5
This chapter discusses the potential prospective projects that will be further explored from the
basis of the results of this study to make additional discoveries in this area of sonification.
Future Work
By sonifying EEG signals, it appears that new insight into EEG analysis is now
achievable. Since the sonification of EEG signals was made possible by successfully being able
to “listen to the blink of an eye,” further exploration seeks to go beyond listening to and
assessing blinking towards the analysis of other situations. For example, it would be very
beneficial to be able to detect and classify various types of seizures through auditory methods.
Hopefully, this insight about brain signals will be helpful in the early detection of abnormalities
and thus the ability to take better care of patients. The ultimate goal is to enable specialists and
non-specialists to be able to recognize important features of an EEG, such as certain kinds of
seizures, acoustically.
Although significant progress has been made on the sonification of EEGs using clinical
data, the addition of a Cadwell Easy III EEG with 32 channels in the laboratory has enabled
further information to be gathered in order to record data to test the algorithms for sonification
that are currently being developed. Due to the fact that initially the algorithms to convert EEG
data to sound will be issue-specific, the acquisition of real EEG data is crucial. It is critical to
find the means by which information from the EEG machine can be obtained, analyzed, and
converted into sound in real-time for convenient real-world applications. For this reason, a
particular topic of interest to this study would be electrooculography, the monitoring of the
subset of the electrodes of the EEG that show eye and eyelid position (Seaquist, 2014). For this
work, a volunteer human subject would be required to have an EEG performed in the laboratory
27
while cues are given requesting that the subject perform various actions. These requests include
opening and closing his or her eyes, singularly and together, as well as blinking several times.
Therefore, more relevant data can be attained from the sonification of these signals.
The results of this study also introduce the question of using blink detection in work
related to the use of contact lenses. This would be beneficial because when rigid contact lenses
are placed on the surface of the eye, the eye blinking causes the rigid contact lens to move out of
place, which can cause irritation and the inability to see. Being able to detect the specific
movements of the rigid contact lenses when the eyes blink, and determining how to combat or
account for this movement, can allow for a more comfortable and the ideal fit for the wearer.
Therefore, there are several broad implications of this sonification of EEGs research, which can
be applied to different fields of study, furthermore than the current clinical diagnoses of brain
functions.
28
References
Baier, G., Hermann, T., & Stephani, U. (2007). Multichannel Sonification of Human EEG.
Proceedings of the 13th International Conference on Auditory Display, 491 – 496.
Retrieved from http://pub.unibielefeld.de/luur/download?func=downloadFile&recordOId=2017124&fileOId=2456079
Baier, G., Hermann, T., Sahle, S., & Stephani, U. (2006). Sonified Epileptic Rhythms.
Proceedings of the 12th International Conference on Auditory Display, 148 – 151.
Retrieved from
http://www.sonification.de/publications/media/BaierHermannSahleStephani2006SER.pdf
Barrass, S. & Kramer, G. (1999). Using Sonification. Multimedia Systems 7, 23-31. Retrieved
from https://ccrma.stanford.edu/courses/120-fall-2005/using_sonification.pdf
Benitez, R., Holland, S., Jordà, S., Marimon, X., Mealla, S., Oliveira, A., Steffert, T., &
Váljamäe, A. (2013). A Review of Real-Time EEG Sonification Research. International
Conference on Auditory Display. 85 – 93. Retrieved from
http://oro.open.ac.uk/38478/1/Väljamäe-Steffert-2013-ICADA%20REVIEW%20OF%20REALTIME%20EEG%20SONIFICATION%20RESEARCH.pdf
Campo, A., Hoeldrich, R., Eckel, G. & Wallisch, A. (2007). New sonification tools for EEG data
screening and monitoring. Proceedings of the 13th International Conference on Auditory
Display, 1-7. Retrieved from
http://iem.kug.ac.at/fileadmin/media/iem/altdaten/projekte/publications/paper/new/new.p
df
29
Francis & Loughhead. (1984). Bell’s Phenomenon: A Study of 508 Patients. Australian Journal
of Ophthalmology, 15-21. Retrieved from
http://onlinelibrary.wiley.com/store/10.1111/j.1442-9071.1984.tb01119.x/asset/j.14429071.1984.tb01119.x.pdf;jsessionid=3F85E3D3BB77324A61CD828C5144E2A5.f04t01
?v=1&t=i5qigoe9&s=bd30facece6bb8cc99cecc147a41c0a4ab40bab5
Hermann, T., Meinicke, P., Bekel, H., Ritter, H., Muller, H.M., Weiss, S. (2002). Sonifications
for EEG Data Analysis. Proceedings of the 2002 International Conference on Auditory
Display, 1-5. Retrieved from
http://www.icad.org/websiteV2.0/Conferences/ICAD2002/proceedings/22_Thomas_Her
mann_EEG.pdf
Hinterberger, T. & Baier, G. (2004). POSER: Parametric Orchestral Sonification of EEG in Real
Time for the Self-Regulation of Brain States. Proceedings of the International Workshop
on Interactive Sonification, 1-6.
Khamis, H., Mohamed, A., Simpson, S. & McEwan, A. (2012). Detection of temporal lobe
seizures and identification of lateralisation from audified EEG. Clinical Neurophysiology,
123 (9). 1714-1720. doi: 10.1016/j.clinph.2012.02.073.
Mills, M. (2012). Do Signals Have Politics? Inscribing Abilities in Cochlear Implants. The
Oxford Handbook of Sound Studies. New York, New York: Oxford University Press.
Prandoni, P., & Vetterli, M. (2008). Signal Processing for Communications. Lausanne,
Switzerland: EPFL Press. Retrieved from http://www.sp4comm.org/docs/sp4comm.pdf
Purves, D. (2008). Neuroscience. Sunderland, MA: Sinauer Associates.
Schomer, D.L. & Lopes da Silva, F. H. (2010). Niedermeyer’s Electroencephalography: Basic
30
Principles, Clinical Applications, and Related Fields. Philadelphia, PA: Lippincott
Williams & Wilkins.
Seaquist, Carl. (2014). Sonification of Electroencephalograms IRB. Lubbock, TX.
Tatum, W.O., Husain, A.M., Benbadis, S., & Kaplan, P.W. (2008). Handbook of EEG
Interpretation. New York, New York: Demos Medical Publishing.
Trevisan, A. A., & Jones, L. (2010). A Low-end Device to Convert EEG Waves to Music. 128th
Convention of the Audio Engineering Society. 1-7. Retrieved from
http://menteheadband.com/wp-content/uploads/2014/02/Low-end-device-that-convertsEeg-to-Music.pdf
31
Appendix A
MATLAB Code
% How to get started:
% load EEGLAB
% >eeglab;
% Now go to EEGLAB Menu and pull down from the
% File Menu --->Import Data-->Using EEG functions and Plugins-->
%
From EDF/EDF+/GDF files (BIOSIG Toolbox)
%
When prompted for file click on subject.edf...A box will
%
appear...hit return to allow defaults to be in force. A new box
%
will appear...name what you will.
% Pull down Plot-->Channel data (scroll) to look at data.
%
%%
eegdata=EEG.data; % Save all channels from subject.
save('subject1eegdata.mat','eegdata','-mat'); % From now on loaddata from
this file
%% If data is saved start here.
clear;
load('subject1eegdata.mat')
%%
% Channel
% Event :
% Fp1
:
% Fp2
:
% F7
:
% F3
:
% Fz
:
% F4
:
% F8
:
% T3
:
% C3
:
% Cz
:
% C4
:
% T4
:
% T5
:
% P3
:
% Pz
:
% T4
:
% T6
:
% O1
:
% O2
:
% A1
:
% A2
:
% others:
% read in all eegdata(channel #, time frames)
Numbers
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23-43 (Pg1, Pg2, Oz, X1-X8, DC1-DC8, OSAT, PR)
32
% Recall that odds are left side and evens are right side.
%
%
%
%
%
%
%
For this study Looking at blinks and alpha rhythmes we are interested in
bipolar montage (double banana) of channels Fp1-F3 and Fp2-F4 for blink
detection and
in referential montage (ipsi ear) of channels O1-A1 and O2-A2 for alpha
detection. (See page 185 of Niedermeyer's Electroencephalography, 2011.)
Thus we want to pick off
Fp1:2, F3:5, Fp2:3, F4:7, O1:19, A2:22, O2:20, and A1:21
% For this study we are interested in the blinks and alpha
% in the 17 seconds from 79secs to 96 seconds.
%% Pick off data
%
%
channels=[2 5 3 7 19 21 20 22];
% Channels of interest and names
channelnames={'Fp1';'F3';'Fp2';'F4';'O1';'A2';'O2';'A1'};
montagenames={'Fp1-F3';'Fp2-F4';'O1-A1';'O2-A2'}; % Montages of interest
freq=512;
% frequency of sampling of original data
starttime=60+19;
% start time of interest: 1 minute 19 seconds
endtime=60+36;
% end time of interest:
1 minute 36 seconds
istrt=freq*starttime;
iendt=freq*endtime;
newdata=eegdata(channels,istrt:iendt);
%% plot the raw data to compare with the EDF file to makesure correct data is
being used.
figure('Name','Original data e unfiltered: 1:19-1:36; i.e., 17secs');
ticks={'0';'1';'2';'3';'4';'5';'6';'7';'8';'9';'10';'11';'12';'13';'14';'15';
'16';'17'};
for i=1:18
ta(i)=(i-1)*freq+1;
end
for i=1:8
subplot(4,2,i); plot(newdata(i,:),'black');
set(gca,'XTick',ta)
set(gca,'XTickLabel',ticks);
xlabel('Secs');
ylabel('uV');
title(channelnames(i));
end
%% Create the actual signals to be studied
Fp1mF3=newdata(1,:)-newdata(2,:);
Fp2mF4=newdata(3,:)-newdata(4,:);
O1mA1=newdata(5,:)-newdata(6,:);
33
O2mA2=newdata(7,:)-newdata(8,:);
%% Plot the data with correct montage...should again compare with original
EDF
figure('Name','Montage (Double Banana + Ipsi Ear) Unfiltered')
subplot(2,2,1); plot(Fp1mF3,'black')
set(gca,'XTick',ta)
set(gca,'XTickLabel',ticks);
xlabel('Secs');
ylabel('uV');
title(montagenames(1));
subplot(2,2,2); plot(Fp2mF4,'black')
set(gca,'XTick',ta)
set(gca,'XTickLabel',ticks);
xlabel('Secs');
ylabel('uV');
title(montagenames(2));
subplot(2,2,3); plot(O1mA1,'black')
set(gca,'XTick',ta)
set(gca,'XTickLabel',ticks);
xlabel('Secs');
ylabel('uV');
title(montagenames(3));
subplot(2,2,4); plot(O2mA2,'black')
set(gca,'XTick',ta)
set(gca,'XTickLabel',ticks);
xlabel('Secs');
ylabel('uV');
title(montagenames(4));
%% At this point we might wonder how to filter the data.
%
%
%
%
%
%
One approach might be to use the EEGLAB menu and use (Under the Edit Pull
down) Select Data. This will prompt for Time Range, e.g. [79 96] and
Channel range, e.g., 2 3 5 7 15 17 19 20. The resulting file/dataset can
be written out to file...I used fselectnewdata.
I then thought I would try to run an order 5 highpass butterworth filter
at 1Hz.
x(1,:)=Fp1mF3;
x(2,:)=Fp2mF4;
x(3,:)=O1mA1;
x(4,:)=O2mA2;
%% High Pass Filter over 1Hz
34
[zh,ph,kh]=butter(5,1/(0.5*freq),'high');
[sosh,gh]=zp2sos(zh,ph,kh);
Hd=dfilt.df2tsos(sosh,gh);
% hh=fvtool(Hd);
% set(hh,'Analysis','freq');
for i=1:4
y(i,:)=filter(Hd,x(i,:));
end
figure('Name','Montage (Double Banana + Ipsi Ear) High Pass Filter (1Hz)');
for i=1:4
subplot(2,2,i); plot(y(i,:),'black');
set(gca,'XTick',ta);
set(gca,'XTickLabel',ticks);
xlabel('Secs');
ylabel('uV');
title(montagenames(i));
end
%% Low Pass Filter under 70Hz
[zl,pl,kl]=butter(7,70.0/(0.5*freq),'low');
[sosl,gl]=zp2sos(zl,pl,kl);
Ld=dfilt.df2tsos(sosl,gl);
% h=fvtool(Ld);
% set(h,'Analysis','freq');
for i=1:4
z(i,:)=filter(Ld,y(i,:));
end
figure('Name','Montage (Double Banana + Ipsi Ear) Band Pass Filter
(1Hz/70Hz)');
for i=1:4
subplot(2,2,i); plot(z(i,:));
set(gca,'XTick',ta);
set(gca,'XTickLabel',ticks);
xlabel('Secs');
ylabel('uV');
title(montagenames(i));
end
35
%% Resample at 44100
fsample=44100;
for i=1:4
v(i,:)=resample(z(i,:),fsample,freq);
end
lth=(endtime-starttime)*fsample;
ibg=22;
for i=1:4
w(i,:)=v(i,ibg:ibg+lth-1)/200.0;
end
%% Plot Resampled
figure('Name','Montage (Double Banana + Ipsi Ear) Filtered and Resampled at
44,100Hz');
for i=1:18
ta(i)=(i-1)*fsample+1;
end
for i=1:4
subplot(2,2,i); plot(w(i,:));
set(gca,'XTick',ta)
set(gca,'XTickLabel',ticks);
title(montagenames(i));
end
%% Write out wave file of modulating signal
for i=1:4
a=['xFnewa',char(montagenames(i)),'mod.wav'];
audiowrite(a,w(i,:),44100)
end
%% Set up cords and x
clear x
clear y
clear z
sample=44100;
interval=2^(1/12);
36
A3=220;
A4=440;
C5=440*interval^3;
E5=440*interval^7;
freq(1)=A3;
freq(2)=A4;
freq(3)=C5;
freq(4)=E5;
l=length(w(1,:))/sample;
x=0:(1/sample):l-(1/sample);
%% Build tones
for i=1:4
m(i,:)=sin(2*pi*x*freq(i));
end
%% Modulate tones with EEG signals
for i=1:4
z(i,:)=m(i,:).*w(i,:);
end
%% Plot Modulated
figure('Name','Montage (Double Banana + Ipsi Ear) Filtered and Modulated at
44,100Hz');
for i=1:18
ta(i)=(i-1)*fsample+1;
end
for i=1:4
subplot(2,2,i); plot(z(i,:));
set(gca,'XTick',ta)
set(gca,'XTickLabel',ticks);
title(montagenames(i));
end
%% Write files
for i=1:4
a=['yFnewa',char(montagenames(i)),'tones.wav'];
audiowrite(a,z(i,:),44100);
37
end
%% Now use Audition to mix the sound files
%
% Fire up Audition and read in the yFXXXXX files. These are the ones that
% are the original EEGs that have been modulated.
% You can listen to individual ones by selecting and playing.
%
% Select Multitrack (Upper left corner) and fill in box with name, file to
% same in, and select a sample rate of 44100 and bit depth of 16 bits.
%
% Drag the .wav files into tracks 1 to 4.
%
% You can adjust volume of each track separately by using 'volume' knob on
the
% far left of each track. Try to keep all tracks in the Green.
%
% Delete other tracks except Master
%
% Adjust balance on each track and volume on master. This is easier if you
% loop the sound and hit mixer tab
38