Bringing Hearing to the Deaf:
A technical and personal account
1
TALK OUTLINE
How we hear: The physiology of natural hearing
Causes of Deafness (27 million Americans cannot hear well)
Solutions for hearing loss
The cochlear implant: How to get one. What is it like? How well
do they work? Why do they work?
Political & social issues
What cochlear implants are telling us about the functioning of
the brain
The future
2
ca. 550 B.C.
Sound
Pythagoras reasons that
sound is a vibration of air.
3
Physical and perceptual
characteristics of sound
Physical
• Amplitude
• Frequency
• Complexity ,
and phase
relationship of
constituent
frequencies
Perceptual
• Loudness
• Pitch
• Timbre
4
Acoustic Pressure is measured in
decibels (dB)
• 1 atm = 100,000 pascals = 1011 micropascals
• Threshold: the softest sound detectable is 20
micropascals (at 1000 Hz). 2 parts in 10 billion of an
atmosphere
• We hear sounds 1-10 million times more intense than
threshold
• dB are logarithmic units with 0 dB at threshold
• adding 20 dB = factor of 10 increase in pressure
– 6 dB approximately doubles the pressure
• 40 dB SPL = 20 x 100 = 2,000 micropascals
5
loud
Hearing
threshold
of a
severely
deaf
person
Hearing threshold
of a profoundly
deaf person
(ex: the speaker)
soft
6
The Ear Has Three Distinct Regions
ca. 175 A.D. Galen
ca. 550 B.C.
Pythagoras &
successors
Nerve transmits
sound to the brain
It has taken until the present
to unravel the rest
7
The Outer Ear
The videos shown in this talk are based on Auditory Transduction by
Brandon Pletcsh which was awarded 1st place in the NSF/AAAS Science and
Engineering Visualization Challenge 2003
8
Tympanic Vibrations
9
The tympanic membrane & ossicles
1543
Anatomist
Andreas Vesalius
describes the
structure of the
middle ear.
10
The tympanic membrane & ossicles
11
Bony Labyrinth stapes and round window
12
Why is our “sound
sensor” not on the
outside of our head?
Hermann Ludwig von
Helmholtz first to
understand the role of the
ossicles
Impedance mismatch
overcome by ratio of
areas and lever action
13
The cochlea and it chambers
The cochlea is about the size of a pea
1561 Gabriello Fallopio
discovers the snail-shaped
cochlea of the inner ear.
14
The Cochlea houses the Organ of Corti
Auditory
Nerve
15
Organ of Corti
Hair Cells are a mechano-electric
transducer
1st detailed study of
Organ of Corti
by Alfonso Corti
Original figures (scanned) from:
16
Zeitschrift für wissenschaftliche Zoologie (1851)
The Basilar Membrane is a Frequency Analyzer
17
Tonotopic Organization
18
End of Early History
The Middle Ages
Georg von Békésy
(Nobel 1961)
Hermann Ludwig von
Helmholtz first theory of the
role of BM as a spectrum
analyzer providing a
frequency-position map of
sound Fourier components.
Experimentally measured
traveling wave profiles
published by von Békésy
in Experiment in Hearing,
McGraw-Hill Inc., 1960.
base
apex
19
The Copernican Revolution
Von Békésy's findings stimulated the production of
numerous cochlear models that reproduced the observed
wave shapes, but were in contrast with psychophysical data
on the frequency selectivity of the cochlea.
displacement
Davies (1983): a revolutionary new hypothesis
there exists an active process within the
organ of Corti that increases the vibration
of the basilar membrane.
20
Active amplification
Careful measurements on living animal cochlea
Same animal post
mortem
Johnstone et al (1986)
What causes the
amplification?
21
Rows of Hair Cells in the healthy cochlea
Inner hair cells 10,000 afferent
(signals go the brain)
Outer Hair Cells 30,000 Sparsely
innervated
Hair
5m
30mHair
cell
22
Hair cells are mechano-electrical transducers
1980’s
500 nm
Both inner and outer hair cells work this way
2nm diameter
23
The inner hair cells send signals to the
brain that are interpreted as sound. What
do the outer hair cells do?
Outer hair cells exhibit electro motility
they are also electro-mechanical
1987-2003
transducers
24
The Five Main Causes of Hearing Loss
1.
2.
3.
4.
5.
Heredity. At least 100 hereditary syndromes can result in hearing loss.
Infections, such as bacterial meningitis and rubella (German measles).
Acoustic trauma produced by acute or chronic exposure to loud sounds.
Prescription drugs, such as ototoxic antibiotics (streptomycin and
tobramycin) and chemotherapeutic agents, such as cisplatin.
Presbycusis, the hearing loss of old age,
All of us
Me in 1990
27 million Americans cannot hear well
25
The main types of hearing loss and treatments
1) Conductive (the ossicles no longer function)  BAHA
2) Sensorineural (loss of hair cells)
(a) (70%) loss of some hair cells (mild, moderate hearing loss)
 hearing aids(1940’s-present) / MEIHD (in clinical trials)
(b) (4%) Loss of large numbers of hair cells
(severe (3%) profound (1%))
Hearing aids do not help: no matter how loud the amplified
sound there is little electricity reaching the brain
 Cochlea Implant (CI)
3) Loss of the auditory nerve Auditory Brain Stem Implant (ABI)
(a cochlea implant at the cochlea nucleus) (Experimental)
26
Action of ototoxic antibiotics on hair cells
Loud noise also destroys hair cells
27
Don’t lose your hair…. cells
Many of the differences in perception between
natural hearing and hearing in people with cochlear
Normal
hearing loss can be accounted for in terms of
Hearing
a loss or reduction in basilar compression.
* Loss of gain (can’t hear softer sounds)
* Reduced dynamic range
* Loss of frequency sensitivity
* Preferential loss of high frequency
sensitivity. (Since hair cells at the base of
the cochlea are more prone to damage.)
28
The first cochlea implant (1800)….
Volta placed two metallic probes in both ears
and connected the end of two probes to a 50volt battery, and observed that:
"... at the moment when the circuit was
completed, I received a shock in the head,
and some moments after I began to hear a
sound, or rather noise in the ears, which I
cannot well define: it was a kind of
crackling with shocks, as if some paste or
tenacious matter had been boiling... The
disagreeable sensation, which I believe
might be dangerous because of the shock
in the brain, prevented me from repeating
this experiment..."
Alessandro Volta, Philosophical
Transactions, Vol. 90 (1800), Part 2, pp.
403-431.
29
Cochlea Implant
1.
2.
3.
4.
5.
6.
Sounds are picked up by a microphone &
turned into an electrical signal.
The signal passes to a speech processor
(ASIC) where the spectrum is analyzed
and “coded” (turned into a special digital
pattern of electrical pulses).
These pulses are sent to a coil antenna,
then transmitted across the intact skin (by
radio waves) to a receiver in the implant.
The implant (ASIC) reads the program
(data) and follows the instructions sending
a pattern of analog electrical pulses to
multiple electrodes in the cochlea.
The auditory nerve picks up the electrical
pulses and sends them to the brain.
The brain recognizes the signals as sound.
Unlike hearing aids, which make
sounds louder, a Cochlear Implant
bypasses the non-functional hair
cells of the ear and delivers weak
electrical signals directly to the
auditory nerve.
30
Multiple electrodes
at separate locations to
exploit the tonotopic
arrangement of the
cochlea
amplitude
time
AmplitudeCurrent
31
Channel Interaction
The cochlea is filled with conductive salt solutions which allow current to
spread. Current spread is detrimental to speech recognition when currents from
different electrodes interact.
Interleaved pulsing of the electrode array minimizes
channel interaction
18,800
pulses
per second
32
Sentence Recognition (% correct)
100
90
Cochlea implants have improved dramatically
in twenty years
SPEAK
80
CA/CIS
70
60
50
CIS
CIS
speech
coding
strategies
Multipeak
40
F0F1F2
30
CA
F0F2
20
10
SingleChannel
0
3M Nucleus Nucleus Nucleus Nucleus Ineraid Ineraid Clarion
House WSP WSP II MSP Spectra 22 MIT
RTI
ABC
1980 1982
1985
1989
1994
1992
1993 1996
1 electrode
multi- electrodes
MedEL
Combi
1996
Indication for Cochlear Implant
• Adults (I’ll discuss children separately)
– 18 years old and older (no limitation by age)
– Bilateral severe-to-profound sensori-neural hearing
loss (70 dB hearing loss or greater with little or no
benefit from hearing aids for 6 months) ~1 million
citizens now qualify but only ~20,000 CI’s in U.S.)
– Psychologically suitable
– No anatomic contraindications
– Medically not contraindicated
34
General Workup
• Audiological exam with binaural amplification
• CT scan/MRI of temporal bones (cochlea structure)
• Extended trial with state of the art high-powered
hearing aids
• Psychological evaluation
• Medical evaluation
• Any necessary tests to discover etiology of hearing loss
• Patient chooses device: 3 major manufacturers of state
of the art multi channel implants: Cochlear (Australia),
MEDEL (Austria), Clarion (U.S.). All devices have
similar performance the patient is the largest variable in
the outcome
35
• Wait for surgery (can be many months….)
Surgical Technique
Surgery 2-4 hrs under
general anesthesia
36
Postoperative Management
•
•
•
•
•
•
•
Complication rate <5%
Wound infection/breakdown
Facial nerve injury
Vertigo
Device failure—re-implantation usually successful
Avoid MRI
Wait ~8 weeks for wound to heal before activation day
Porter & Gadre (Galveston, TX)
37
The cost of a CI: Insurance Issues
A CI costs $50,000 including evaluation, surgery post operative hospital care, extensive
audiological (re)habilitation.
Medicare/Medicaid pays <$20,000. Some private insurers refuse to cover
the devices, others provide excellent coverage. In general coverage is probably
easier under a traditional plan than a HMO
“The reimbursement levels have forced eight hospital to close CI programs due
to the cost of subsidizing the implants.” (B. March President Cochlear America)
Other hospitals ration services by putting children on waiting lists
Currently 45,000 US children are CI eligible but only 9,000 have a CI
And yet the cost of CI is small compared to the cost in government aid for education and
training estimated at $1 million over the course of a lifetime (not to mention the massive
human cost).
“Ultimately this is about the way society views hearing..Being deaf is not going to kill
you and so the insurance companies do not view this as necessary.”
D. Sorkin, VP Consumer Affairs, Cochlear Corp. (A manufacturer).“
In my case, the cost of the device was fully covered by insurance
38
How well does it work? My experience
0
Normal
dB
Now
BEFORE
Pre-op
100
125 Hz
Frequency 
8000 Hz
Speech
tests
pre-op now
39
Comparison of CI’s from the three main manufacturers
Comparative Test Scores
Percent Words/Sentences
Correctly Understood
100%
90%
Now
Clarion HiFocus
80%
Nucleus 24 Contour
70%
Med-El Combi 40+
60%
50%
40%
30%
20%
10%
0%
CNC (1
month)
CNC (3
mo.)
CNC (6
mo.)
HINT N (3
mo.)
HINT Q (3
mo.)
Test
HINT Q (6 CUNY Q (3 CUNY Q (6
mo.)
mo.)
mo.)
HSM (3
mo.)
Types of word & sentence recognition
Pre-op
My test scores are no longer exceptional.
75% of recent patients with state of the art devices can use the phone.
Why does the CI work so well 10,000 inner hair cells  10 electrodes ?40
Hearing doesn’t end
at the cochlea
Perception (visual or auditory) is a Dynamic
Combination of Top-down and Bottom-Up Processing
• The need for sensory detail depends on
the distinctiveness of the object and the
level of familiarity
• “If you see a huge gray animal in the
distance you don’t need much detail to
know that it is an elephant”
Visual examples…
41
42
43
44
45
46
47
48
49
50
51
52
Speech pattern recognition problem
Vowel perception by
normal hearing listeners.
F1 and F2 values of
English vowels
(Peterson and Barney,
1952)
Vowels are quite distinct
•What features of the pattern of neural output from the cochlea
•are most critical? Amplitude? Temporal? Place?
53
“CHOICE”
SPECTROGRAPH
TIME
ELECTRODOGRAPH
(SPEAK STRATEGY))
TIME
54
Input Dynamic Range
Amplitude Study
Pain
50 dB gives best result
Cf normal hearing: 120 dB
(20dB)
(but speech falls in a ~50 dB
Just
audible range)
1
0
0
A
.
C
o
c
h
l
e
a
r
I
m
p
l
a
n
t
L
i
s
t
e
n
e
r
s
9
0
Implants
Normal Hearing
B
:
N
o
r
m
a
l
H
e
a
r
i
n
g
L
i
s
t
e
n
e
r
s
8
0
Compression
7
0
%
6
0
1
0
0
OutpAmplitude(DynamicRange%)
8
0
4
0
PercntCorect(%)
O
5
0
p
=
0
.
1
0
p
=
0
.
2
0
p
=
0
.
3
0
L
O
G
3
0
p
=
0
.
5
6
0
2
0
p
=
0
.
8
4
0
V
O
W
E
L
S
C
O
N
S
O
N
A
N
T
S
1
0
p
=
1
.
0
p
=
1
.
5p
=
2
.
0p
=
3
.
0
p
2
0
Output = (Input)
0
A
m
i
n
0
2
0
0
4
0
0
6
0
0
I
n
p
u
t
A
m
p
l
i
t
u
d
e
(
U
n
i
t
)
I
8
0
0
A
m
a
x
1
0
0
0
0
0
.
0
5 0
.
1 0
.
2
P
0
.
5
P
0
.
30
.
50
.
8
1
23
P
o
w
e
r
f
u
n
c
t
i
o
n
e
x
p
o
n
e
n
t(
P
)
•Speech recognition is only mildly
affected by large distortions in amplitude55
100
Temporal Study
DAIP Consonants (20)
Quiet
n=7
Information Transferred
80
60
%
Correct
16 Electrode CIS
40
12 Electrode CIS
8 Electrode CIS
4 Electrode CIS
16 Electrode QPS
20
12 Electrode QPS
8 Electrode QPS
Types of implant
with variable
numbers of channels
& speech
coding strategies
4 Electrode QPS
+10 dB
SNR
High stimulation pulse
rates
•High rates should better
represent temporal
features in speech.
•No improved use of
temporal cues in speech
at higher rates observed
•A significant effect for
number of channels was
seen for vowels (not
shown) but not for
consonants,monosyllabic
words or sentences
0
500/s
1000
1000 /s
10000
10000
Stimulation
Rate/s(ppse)
11 S/phase
1000
10000
56
Spectral Resolution (Number of Channels) Study
1-channel
2-channel
4-channel
8-channel
16-channel
Original
57
Spectral Resolution (Number of Channels) Study
%
Correct
•Most important
factor is the number
of spectral channels
of information
–number of
distinct channels
•Number of effective
channels is
proportional to, but
not the same as, the
number of electrodes
(due to interactions
between channels)
58
Place study
Partial insertion
Typical insertion
22
Apex
20
18
16 14
12
10 9 8 7 6 5 4 3 2 1
0
5
10
15
20
25
20
184
513
1168
2476
5085
Base
30
35 mm
10290
20677 Hz
Frequency is logarithmically distributed along cochlea
Typically pitch from 500- 5000 Hz is covered (speech is 250- 6800 Hz)
Partial insertion can lead to more dramatic shift
Frequencies of speech may not go to the correct tonotopic place for that sound
 tonotopic shift
59
60
61
62
63
64
65
Place study
It takes time to adjust to the tonotopic shift i.e.to learn how
to understand speech with an implant
N = 67
100
%
Percent Correct
90
80
70
60
50
40
30
20
10
0
PREOP
2
1
WEEKS
MONTH
3
MONTHS
6
Time 
MONTHS
Trials indicate that it may be possible to reduce the learning time by
gradually shifting the frequencies Talavage (Purdue)/Svirsky (Indiana)
The adult brain is quite plastic
All of the adults in this study were post lingually deaf, What about
prelingually deaf children?
66
The Deaf Community and Cochlear Implants
• Until recently, strong opposition to pediatric implants while
generally neutral towards adult implantation.
• An implant will delay a deaf child’s acquisition of sign
language (a deaf child’s “natural language”) and
assimilation into the deaf community.
• 1991 position statement National Association of the Deaf:
“deplores the FDA decision to approve pediatric implantation as
being unsound scientifically, procedurally, and ethically.”
• 2000 position statement (www.nad.org):
– Emphasizes taking advantage of technological advancements
that have the potential to improve the quality of life for deaf
and hard of hearing persons, and “strongly supports the
development of the whole child and of language and literacy.”
67
Language Development in Profoundly Deaf Children With
Cochlear Implants (Svirsky, Miyamoto et al. Indiana )
N=23
HEARING
60
LANGUAGE AGE (months)
LANGUAGE AGE (months)
96
72
48
24
DEAF
0
0
24
48
72
96
Without CI (predicted)
With CI
48
36
24
12
0
0
12
24
36
48
60
CHRONOLOGICAL AGE (months)
AGE (months)
Before & at 3 intervals after implant
To be implanted
Figure 1
Figure 2
“Despite a large amount of individual variability, the best performers
in the implanted group seem to be developing an oral linguistic system
based largely on auditory input from a cochlear implant”
68
Alien tries to figure out how
a car works from the outside;
By observation develops model:
pressing gas pedal causes car
to accelerate
But alien would not know from this
how internal combustion engine
works. To do that they would need
to take a look under the hood
“Cochlear implants
Enable us to study the role
of the cochlea in perception
(i.e. to take a look
under the hood ”)
69
Physical stimulus
Cochlea
Normal
Cochlea
Auditory
Nerve
CI
Auditory
Nerve
Neural coding
ABI
Brain
“c
a
Perception
t”
Cochlear
Nucleus
Brain
Using hearing impaired listeners
to probe normal auditory functions
Pitch estimate
100
90
80
70
60
50
40
30
20
10
0
100
Ineraid implant:
DC
10
basal electrode
apical electrode
Monopolar
Bipolar
2 4 6 8 10 12 14 16 18 20 22
1
10
Electrode Position (base to apex)
100 300 1000 5000
Pulse rate (Hz)
Place pitch resolution is very poor it can be even be reversed for adjacent
electrodes. Temporal coding for pitch = 300 Hz But in steps of 20 Hz
Original
(normal hearing (NH) discriminates in steps of 1-2Hz at 100 Hz
NH uses tonotopic code to obtain frequency resolution at low frequencies CI
Melody recognition is extremely difficult for CI users (lyrics help)
Improving Cochlear Implants
1) Combined electric and acoustic stimulation
Targets patients with reasonable low frequency hearing
(usually with hearing aid) add shallow CI electrode for high
frequency stimulation
72
2) Bilateral cochlear implants
are 2 implants better than one?
With one CI there is no directionality
Localization
NH 10 Bilateral CI 160
(Helms & Muller)
50%
correct
Bilateral
73
Bilateral cochlear implants Benefit #2
Better speech recognition in noise
100%
Hearing subjects score 100% in all three tests
For patients who do poorly with one implant the effects are much more dramatic
74
Plans to improve cochlear implants
*
*
*
*
Combining acoustic and electric hearing
Bilateral Cochlear implants
Increasing the number of channels/greater cochlea coverage
Reducing power fully implantable device
(Use latest CMOS technology for ASIC or
stay analog as long as possible. Microphone
under skin or transducer attached to ossicles )
* Regeneration of neurons: molecular scaffold and
electrical stimulation (nanotechnology)
75
Summary: Implants, Neuroscience & Bio-engineering
Implants enable the postlingually deaf to hear & provide sufficient
information to support language development in children
Implants are a probe of speech recognition
–Amplitude and temporal cues are not critical
–Spectral/tonotopic cues are the key
•number of effective channels (not electrodes)
•frequency assignments to electrodes
learning curve adult brain is plastic
Music/speech quality (male/female & accents)
Fine spectral information
“The ear is
is critical (not there yet)
made for music”
Implants are teaching us how the brain hears
Example: pitch
Implants, as the first prosthesis to successfully restore neural
function, are a benchmark for biomedical engineering.
76
Final Thoughts
A Cochlear Implant is a wonderful example of electrical engineering,
computer science, mechanical engineering, physics, biology
all working together in a tiny package inside a human being
A wonderful example of why it is important to invest in basic
research.
There are 60,000 implantees worldwide. With the latest devices
¾ of post lingually deaf adults can use a telephone, and small
children can hear their parents voices and learn to understand them
At a personal level 357 days ago I had my hearing restored. It has
enabled me to more easily conduct research & teach, and hear
my wife’s voice for the first time in 12 years and my 11 year
old daughter’s voice for the first time.
77
Acknowledgements
This talk could not have been put together without the
essential help of the following:
At Purdue:
Kirk Arndt & Steve Lichti (Physics)
Donna Fekete (Biology) Beth Strickland (Audiology)
Tom Talavage (ECE)
At MedEl:
Peter Knopp (Vienna) Jason Edwards (US), Amy Barco (US)
Elsewhere:
David Ashmore (London), Bill Brownell (Baylor),
Phil Louzoi (UT Dallas), Richard Miyamoto (Indiana),
Brandon Pletsch (IowaMed), Bob Shannon (House Ear Institute),
Mario Svirsky (Indiana), Fan-Gang Zeng (UC Irvine)
78
Cochlea Implants are a hot topic
PubMed Search: "Cochlear AND Implant” (Total=2086)
250
150
100
50
0
1972
73
74
75
76
77
78
79
1980
81
82
83
84
85
86
87
88
89
1990
91
92
93
94
95
96
97
98
99
2000
N umbe r of pape rs
200
1970s = 32 : 0
1980s = 453 : 1
1990s = 1557 : 5
Y e ar
Pitch encoding
Place code
Timing code
Vowel perception by cochlear implant users.
20 filters associated with 20 stimulation channels
Stimulation channels
20 19 18 17 16 15* 14 13 12 11 10 9 8
7 6
5
4 3
2
1
Filters
Amplitude
200 Hz
10 kHz
Frequency
*Stimulation mode:
BP+1
81
Bilateral cochlear implants Benefit #2
Better speech
recognition in noise:
Bilateral implant vs.
Monaural implant
with noise on opposite
side. (Modest gain)
Bilateral implant vs.
Monaural implant
with noise on same
side. (Enormous gain)
82
Loudness encoding
L=f(A)
E
L=f[g(E)]
A
Loudness growth and balance functions
L=exp(E)
100
A
600
Electric level (uA)
Loudness estimate
700
B
500
400
10
300
200
0.29 0.29
L=3.65[(I+Io) -Io ]]
r=0.99
1
E(uA)=-72.98+8.02 A(dB)
r=0.98
100
0
20
40
60
80
100
Acoustic level (dB HL)
20
40
60
80
100
Acoustic level (dB HL)
Auditory
Brain
Cochlear
nerve
expansion
compression
transmission
20-dB transmission line
log ( )
exp ( )
Cochlear compression followed by central expansion
80-dB internal world
100-dB external world
Loudness coding scheme
86
87
Standard Electrode
3) Better electrodes positions
– closer to the target cells
Comparison: a significant
improvement: easier to
hit the target when closer.
Standard Electrode
closer to target
88
The Copernican Revolution
Von Békésy's findings stimulated the production of
numerous cochlear models that reproduced the observed
wave shapes, but were in contrast with psychophysical data
on the frequency selectivity of the cochlea.
Davies (1983): a revolutionary new hypothesis
there exists an active process within the
organ of Corti that increases the vibration
of the basilar membrane.
89
Mountain
November 2003
90
At near-threshold stimulus levels,
the frequency tuning of auditory
nerve fibers closely parallel that of
basilar membrane displacement
Only minor signal transformations
intervene between mechanical
vibration and auditory nerve
excitation.
In mammals, cochlear frequency
selectivity is fully expressed in the
vibrations of the basilar membrane.
91
Hydrodynanic Model of the Basilar Membrane
f 0 =frequency of stimulus
Z  (1/ A2 ) (M 0  K / 0 )2  D2 
M  mass
0 =frequency of stimulus
D = damping
K  stiffness
A = area
0 = k 2 / m low Z resonance
92