1
Searching for phonological processing in the human brain using
natural and synthesized vowels
Stefan Uppenkamp1, Roy Patterson1,
Ingrid Johnsrude2, Dennis Norris2, William Marslen-Wilson2
1
Centre for the Neural Basis of Hearing, Department of Physiology,
University of Cambridge, Downing Street, Cambridge, CB2 3EG, United Kingdom.
2
MRC Cognition and Brain Sciences Unit, , Downing Street, Cambridge, CB2 3EG, United
Kingdom.
2
Abstract
It is commonly assumed that, in the cochlea and the brainstem, the auditory system processes
speech sounds without differentiating them from any other sounds. At some stage however, it
must treat speech and non-speech sounds differently. The purpose of this study is to try and
delimit the location of this stage in the auditory pathway by means of functional MRI. We
assume that this is the point where the sound is found to match a specific phonological
category, well before lexical or semantic processing begins. Results suggest that phonological
processing of vowels may begin just outside auditory cortex in Superior temporal gyrus.
1. Introduction
Brain imaging studies in speech research are typically more concerned with locating
the centres involved in lexical or semantic processing rather than phonological processing;
that is, more concerned with the later, rather than the earlier, stages of speech processing.
Accordingly, they use continuous speech and contrast it with, for example, rotated or reversed
speech (Scott et al., 2001) to preserve and balance the spectral and temporal complexity of
the sounds in their imaging contrasts. Transients, tones and noises are not transmuted from
one category to another by reversal or rotation; they remain transients, tones or noises, albeit
slightly different instantiations of these sound types. So any centre involved in processing
transients, tones and noises as such will be just as active for reversed and rotated speech as for
normal speech, and not surprisingly, all of these speech and non-speech stimuli activate the
larger part of the temporal lobe when contrasted with silence. To locate where phonological
processing begins, a more specific contrast is needed, and preferably one that does not involve
lexical, syntactic or semantic processing. Accordingly, we have developed a new class of
synthetic vowel sounds to try and delimit the point in the auditory system where phonological
processing begins.
3
Vowel production can be described as a two-stage process: air flows from the lungs
through the glottis to produce the initial sound wave, a sequence of pulses at a rate
corresponding to the pitch of the voice. This wave then passes through the vocal tract, which
impresses the characteristic formant spectrum by its multi-resonance filter properties.
Artificial vowels can be synthesised using exactly these principles (Klatt, 1980). A different
approach would be to put sine waves at formant frequencies, but despite having all the
formant information, these waves don't sound like speech.
Using damped sinusoids (Patterson, 1994) at formant frequencies with an envelope
period in the range of the vocal pitch period and with half lives that produce proper formant
bandwidth creates artificial vowels that automatically activate the phonological system. These
sounds produce a clear speech perception provided the sound is syllable length. This approach
enables us to produce non-speech control sounds with similar long-term distributions of
energy over frequency and time, by randomising the envelope onsets and the carrier
frequencies within each of four formant tracks.
2. Stimulus generation
The basic building block of the synthetic vowels is a ‘damped’ sinusoid (Patterson et
al., 1994) constructed by applying an exponentially decaying envelope with 4 ms half-life to
a short segment of a sinusoid (Figure 1). A single damped sinusoid is like one cycle of a
formant in a vowel. The upper left panel shows four damped sinusoids with the same 16-ms
envelopes. The carrier frequencies are fixed at the formant frequencies of /a/, and the sound
produced by the sum of the damped sinusoids (bottom row) automatically activates the
phonological system and produces a speech perception provided the sound is syllable length
(about 300-400 ms). The other three panels show how non-speech control sounds with very
similar distributions of energy over frequency and time can be produced, some of which
activate the phonological system and some of which do not. In the upper right panel, the start
4
points of the damped sinusoids within each 16-ms ‘cycle’ were randomised while keeping the
carrier frequencies fixed at formant frequencies. The resulting sound is still recognisable as an
/a/ but from a vocal tract with a pathological degree of jitter. In the lower left panel, the
carrier frequencies within their formant bands were randomised keeping the start points fixed,
and in the lower right panel both the start points and carrier frequencies have been
randomised. These sounds do not activate the phonological system at all; indeed, they sound
like two strange forms of ‘musical rain’, one with a continuous low pitch due to the
synchronous envelope onset points across the four formant tracks. They do, however, have the
same average statistics as those in the upper panels, and so will serve as the appropriate
controls in the search for the location of phonology and pitch centres in the auditory system.
3. Stimulus evaluation
[[[THE FOLLOWING IS A SHORT VERSION OF PSYCHOPHYSICS, FULL VERSIN
FURTHER DOWN]]]
The speech-like sound quality of the synthetic vowels was quantified relative to nonvowel sounds using the method of paired comparisons. Listeners were asked to choose the
interval that sounded most vowel like. Eighteen sound conditions (see Table 2) were included
in the experiment which gave 306 trials per listener to cover all comparisons. A relative scale
of preference was constructed from the judgements using the Bradley-Terry-Luce method [3].
The data show that proper formant frequencies are essential for producing a vowel
percept. Adding the damped envelope, either with periodic or with random onsets, results in a
substantial increase on the scale of preference when compared to flat-envelope sinusoids.
Even regular, single-formant damped sounds are rated as more speech-like than four-formant
sinusoidal vowels. Vocal pitch, produced by regularising the onsets, is the next most
important property for producing a vowel percept. Two-formant, regular, damped vowels are
preferred over "pathological" four-formant vowels with no pitch. On the other end of the
5
scale, randomisation of both the carrier frequency and the onset time within each track
produces waves which sound very different from human vowels. They are more like some kind
of “musical rain”.
A paired comparison experiment was performed to quantify the perceptual quality of
the synthetic vowels relative to the non-vowel sounds. Eighteen sound conditions were
presented in this experiment, including the four sounds shown in Figure 1. Table I
summarises the sound conditions; the sounds from Figure 1 are marked "*". The sounds
differed in the amount of randomisation of the position of the damped sinusoids both in the
time and frequency domains. The procedure was a two-interval, two-alternative forced choice
task. Each stimulus interval contained a sequence of three randomly chosen stimuli (vowels or
non-vowels) from one particular sound condition. The stimuli had a duration of 400 ms and
they were separated by 200 ms of silence. The two stimulus intervals within one trial were
separated by 500 ms and their onsets were marked by lights. The listeners were asked to
choose the interval that sounded most vowel-like. In the case of two completely non-vowellike sounds, they were asked to choose the one that was more like speech. They could repeat a
trial once before they were forced to give a response. All stimuli were scaled to have the same
RMS level. They were played by a TDT system II at a sampling frequency of 20 kHz into a
lowpass filter with cutoff at 8 kHz, and presented diotically via headphones (AKG 240D) at
50 dB HL.
Nine normal hearing subjects participated in the experiment. Before the main
experiment with all sound conditions, several examples were played to illustrate the full range
of the stimuli. During the experiment, no stimulus was compared to itself, and each
comparison was carried out twice, once with the order A-B, and once with the order B-A, and
so there were 1817 = 306 trials. The order of the trials was randomised, and the experiment
6
was divided into 17 runs with 18 trials. The subjects were asked to take a short break about
every 4 runs. A complete session for one subject lasted for approximately one hour.
A relative psychophysical scale of preference, reflecting the speech-like quality of the
sounds, was constructed from the judgements using the Bradley-Terry-Luce method [6]. This
method is based on a linear model which assumes that on the dimension of interest, the
stimuli can be ordered according to a linear scale. Pooling the judgements from all nine
listeners gives a total of 2754 trials, or 18 observations for each pair of stimuli (9 A-B, 9 B-A)
and 153 observations for each single stimulus. Figure 2 shows the resulting hierarchy of
preference. Arrows indicate the positions of the sounds from Figure 1. The whole scale covers
a range of approximately 5 points (-2.5 to 2.5). This scale should be read as a relative scale;
that is, only differences have meaning. The zero-line is intrinsic to the analysis and the value
has no particular meaning with regard to perceptual quality.
With regard to the two dimensions of randomisation, carrier frequency and onset time,
it is obvious that formant frequency is essential for producing a vowel perception. The five
sounds at the top of the scale have fixed, proper, formant frequencies, while four of the five
sounds at the bottom end of the scale have randomised carrier frequencies. Speech pitch,
produced by regularising the onsets, is the next most important property for producing a
vowel percept. Two-formant, regular, damped vowels are preferred over "pathological" fourformant vowels (no pitch), and they are rated as much more speech-like than four-formant
sinusoidal vowels without damped envelopes (sin_vow). Adding the damped envelope, either
with periodic or with random timing, results in an increase of nearly 1.5 points on the
preference scale. Even regular, single-formant, damped sounds are rated as more speech-like
than any sound made out of flat-envelope sinusoids.
7
4. fMRI data acquisition
Sparse imaging with a TR of 10 s was used to separate scanner noise and acoustic stimuli in
time. Whole-brain volumes were acquired on a 3T MRI system (Bruker). In addition to the
four sounds from Figure 1, there were two more conditions included as controls; these were
silence and vowels recorded from a real speaker. The sounds were presented as sequences of
16 stimuli (vowels or non-vowels) of 400 ms duration each with inter-stimulus intervals
between 70 and 900 ms. There was no speech-specific task, but occasionally two single
sounds in the sequence were presented at an attenuated level. The subjects were asked to press
a button when this occurred to maintain their attention. Nine right-handed listeners
participated in this experiment; each of the six conditions was repeated 32 times for each
listener, giving 1728 scans in total.
5. Activation in response to sound
Figure 3 shows a glass brain view in axial orientation of the group analysis for all listeners of
the activation in response to sound. Each of the five sound conditions was contrasted with
silence. All of these sounds produce very similar patterns of activation, which are largely
confined to the temporal lobes bilaterally, with the most prominent peak towards the lateral
end of HG. The main peaks are very consistent across conditions. Nevertheless, the vowel
sounds lead to additional peaks in an area posterior to the line connecting the main peaks of
activation which is assumed to be Heschl’s gyrus.
6. Activation specific to vowel and non-vowel sounds
Contrasts between vowel sounds
There was no significant activation specific to natural vowels, when contrasted with damped
vowels. There was also no significant activation specific to the fixed pitch in damped vowels
when contrasted with pathological vowels. There was, however, one area in the left
8
hemisphere with slightly more activation for natural vowels than for “pathological” vowels
This might indicate an area sensitive to the voice-like sound quality of spoken language,
rather than to the speech-like fundamental pitch.
Contrasts between vowels and non-vowels
Similar to synthesized vowels, there was no significant effect of the presence of a
fundamental pitch in the musical rain conditions. There was, however, significantly more
activation in all speech-like conditions when contrasted with non-speech conditions,
bilaterally, in an area below and posterior to Heschl’s gyrus on the superior temporal gyrus.
7. Location of the activation specific to speech sounds
The left column of Figure 4 shows three sections of the mean structural scan for the nine
listeners in this study. Each individual anatomical scan was inspected for the positions of
Heschl’s gyri (HG) by four independent judges, and the results were combined into a map of
the most likely location in the group (painted in white). Activation maps of the main contrasts
rendered on this brain are given in the figure in different colours. Activation specific to vowel
sounds is clearly outside HG, posterior and inferior to the main area of activity in response to
sound. Whereas this activation appears bilateral and symmetric in the group, single-subject
analyses revealed that some listeners show more activation on the right while others show
more on the left.
8. Discussion
The stimuli used in the current study were designed to cover a wide range of perceived sound
qualities from speech-like to completely non-speech-like, while their long-term acoustical
properties were closely matched. The non-speech sounds emerge from the synthesized vowels
by means of simple manipulations in the time and frequency domains. The design includes
variation along two perceptual dimensions: speech-like pitch, and speech-like formant
9
frequencies. It appears that phonological category, as determined by the presence or absence
of formants, was
much more influential than pitch in producing activation. The areas
posterior to Heschl’s gyrus that were specifically sensitive to speech sounds can be
considered as candidate regions for the beginning of phonological processing. This is,
however, not proof of a phonology-specific area in the brain, since similar areas are activated
when contrasting sequences of musical notes with fixed pitch and moving pitch (Patterson at
al., 2002), as is shown in the right column of Figure 4. The activation may simply reflect the
process of feature extraction from sounds that meet certain (unspecified) criteria.
9. Conclusions
The auditory images of tones, noises and transients suggest a method for producing
matched sets of sounds that either do, or do not, produce vowel perceptions. The perceptual
distinction between these damped vowels and musical rain was established psychophysically
in a paired comparison experiment. Subsequently, an fMRI study with a high-fidelity,
magnet-friendly sound system and a sparse imaging technique revealed a candidate location
for phonological processing in the Brodmann area 21, just below and behind auditory cortex
in superior temporal gyrus. The activation is bilateral and symmetric in the group, although a
detailed analysis of individual listeners revealed a lateralisation of the activation pattern
sometimes to the left, sometimes to the right, and sometimes bilateral activation was found.
10. Acknowledgment
This work was supported by the Medical Research Council (G9901257). Functional MRI was
carried out at the Wolfson Brain Imaging Centre in Cambridge and the data were analysed
using SPM 99 (http://www.fil.ion.ucl.ac.uk/spm).
10
11. References
Uppenkamp S, Kothari A, Bailes, J, Patterson RD (2001) When is a vowel not a vowel? Brit J
Audiology 35, 159 (A).
Patterson RD (1994) The sound of a sinusoid: time-interval models. J Acoust Soc Am 96,
1419-1428.
Hall DA, Haggard MP, Akeroyd MA, Palmer AR, Summerfield AQ, Elliott MR, Gurney EM,
Bowtell RW (1999) “Sparse” temporal sampling in auditory fMRI. Hum. Brain Mapping
7, 213-223.
Griffiths TD, Uppenkamp S, Johnsrude I, Josephs O, Patterson RD (2001) Encoding of the
temporal regularity of sound in the human brainstem. Nature Neuroscience 4, 633-637
Figure captions
Figure 1
Four classes of stimuli constructed from sets of isolated formants (damped
sinusoids). Top left: damped vowel, regular envelope onsets and fixed formant frequencies.
Top right: “pathological” vowel, fixed formants but randomised envelope onsets. Bottom left:
regular envelopes (produces a pitch), but randomised carrier frequencies from cycle to cycle.
Bottom right: both carriers and envelope onsets randomised (“musical rain”).
Figure 2 Relative scale of preference for synthetic vowels and non-vowel sounds from a
paired-comparison experiment. On the other end of the scale, randomisation of both the
carrier frequency and the onset time within each track produces sounds which sound very
different from human vowels; they are more like some kind of "musical rain". Synthesising
the vowels as sets of single damped sinusoids centred at formant frequencies, and with a
realistic repetition rate, enables the creation of sounds covering a wide range of perceptual
quality from speech-like to completely non-speech like by means of simple manipulations in
the time and frequency domains. Damped vowels and "musical rain" have similar long-term
distributions of energy in time and frequency, so they should activate primary auditory areas
in a very similar way. Any area in the brain that extracts phonological information from
incoming sound should show a strong contrast between these two conditions.
11
dmp_vow
dmp_two
dmp_fst
dmp_snd
flt_vow
jit_vow
pth_vow
noi_vow
sin_vow
sin_two
sin_fst
sin_snd
noi_pit
noi_ran
fxr_pit
fxr_ran
mus_pit
mus_ran
* damped vowels, four tracks of damped sinusoids at formant
frequencies
as dmp_vow, but only first and second formant
as dmp_vow, but first formant only
as dmp_vow, but second formant only
as dmp_vow, but no lowpass slope in spectrum
as dmp_vow, 10% jitter in envelope timing
as dmp_vow, irregular envelopes (100% jitter in timing), i.e. no
pitch
* as dmp_vow, but narrow bands of noise as carriers of triangles
four sinusoids at formant frequencies, no damped envelope
as sin_vow, but only first and second formant
as sin_vow, but first formant only
as sin_vow, but second formant only
four tracks of damped noise bands, one octave wide
as above, but irregular envelope (no pitch)
four tracks of damped sinusoids, random change of carrier
frequencies within limited bandwidth, regular timing
as above, but random timing (no pitch)
* complete randomisation of carrier frequencies for each track,
regular timing
* randomisation of carrier frequencies and timing
Table 1
Summary of sound conditions used in Experiments 2 and 3. The sounds from Fig. 1 are
marked “*”.