ASA 06
Yue Wang
Dawn Behne
Simon Fraser University
Vancouver, Canada
40
20
A
audio
/sa/
la r
eo
en
a lv
e rd
i nt
en
io d
la b
t al
.
ta l
la r
t al
eo
a lv
ta l
e rd
en
io d
la b
en
la r
t al
eo
a lv
ta l
e rd
i nt
en
io d
la b
en
la r
eo
en
a lv
e rd
i nt
en
io d
la b
tal
.
.
ta l
la r
eo
en
a lv
e rd
int
en
la b
tal
.
ta l
la r
io d
e rd
% co rre ct re spo n se s
at c
e
i at
ed
e rm
I nt
Vm
t ch
ma
A
h
.
h
at c
Vm
e
i at
ed
e rm
I nt
A
ma
t ch
h
at c
Vm
e
i at
ed
e rm
I nt
Summary
Expected percept
θa
video
δa
FINDINGS
/fa/
Acknowledgements
We thank Nicole Carter, Angela Feehan, Elaine Pang, Lindsey Shaw, Daisy Shen, Kristy Stefanucci, & Allana Yang at Simon Fraser University (SFU) for their assistance. This project was
supported by research grant #410-2006-1034 from the Social Sciences & Humanities Research Council of Canada (SSHRC Standard Research Grant) & an SFU Institutional SSHRC grant.
References
[1] Jongman, A., Wang, Y., & Kim, B. (2003). Contributions of semantic and facial information to perception of nonsibilant fricatives. JSLHR 46, 1367-1377.
[2] Sumby, W. & I. Pollack (1954) Visual contribution to speech intelligibility in noise. JASA 26, 212-215.
[3] Erber N. P. (1969). Interaction of audition and vision in the recognition of oral speech stimuli. JSHR 12, 423-425.
[4] Hazan, V., Sennema, A., Faulkner, A. & Ortega-Llebaria, M. (2006). The use of visual cues in the perception of non-native consonant contrasts. JASA 119, 1740-1751.
[5] Hazan, A., A. Sennema, Iba, M., & Faulkner, A. (2005). Effect of audiovisual perceptual training on the perception and production of consonants by Japanese learners of English. Speech Commun. 47, 360-378.
• All groups
- A & AV better than V
- quiet better than noise.
• Compare the 3 groups
- early arrivals’ scores: intermediate to
English Ss & late arrivals’
• Compared to English Ss, Chinese showed
- lower % correct with L2 inter-dental AV congruent.
- higher % inter-dental responses with incongruent.
- greater degree of reliance on V.
Conclusions
• Similarities among the 3 groups suggest perceptual
universals (e.g., A better than V).
• Differences between Chinese early & late arrivals
suggest effect of linguistic experience.
• Nonnatives: attended to V info to perceive L2 sounds,
but failed to adopt the correct V cues containing
linguistically contrastive info, suggesting language
specific processing.
Together, results point to an integrated crossmodal network in speech processing.
Contact: Yue Wang, yuew@sfu.ca; Dawn Behne, dawn.behne@svt.ntnu.no
fa - sa
sa – fa
va - za
za – va
Fig. 2. Percent correct for incongruent AV stimuli. “A match” or ”V match”: correct responses matching A or V component.
Intermediate: inter-dental responses matching neither A nor V.
AV-incongruent
Incongruent AV stimuli
t ch
English
Chinese
ma
English
A
English
Chinese
0
.
Occurrence
40
0
h
More back
English N
Early Chinese N
Late Chinese N
60
20
at c
Intermediate
A alveolar + V labiodental
80
20
Vm
More front
f)
40
e
Place of
articulation
( e.g. s +
English Q
Early Chinese Q
Late Chinese Q
ed
si, sa, su
zi, za, zu
θa
s)
60
e rm
θi, θa, θu
δI, δa, δu
AVc
A labiodental + V alveolar
A alveolar + V labiodental
I nt
fi, fa, fu
vi, va, vu
AV-congruent
( e.g. f +
80
ma
Stimuli
V
100
A labiodental + V alveolar
A
s/z
A
Noise
100
i at
Visual-only
Congruent AV stimuli
θ/δ
20
Quiet
• Identification task
f/v
English N
Late Chinese N
Early Chinese N
40
Fig. 1. Percent correct for A, V, and AV stimuli in quiet (Q) and noise (N) by English Ss, & Chinese early and late arrivals.
• Modalities: A, V, AV-congruent, AV-incongruent.
Fricatives
int
en
la b
io d
• CV in quiet background, & café-noise (S/N=0dB).
60
0
t ch
Stimuli & Procedure
AVc
80
0
% correct responses
20 Mandarin Chinese adults;
Arrival in Canada: > 18, Residence: < 5 yrs
θa
Audio-only
Incongruent stimuli
15 Mandarin Chinese adults;
Early arrivals
Arrival in Canada: 7-13 yrs old
4 AV modalities
Native English 15 Canadian English adults
V
% c o rre c t re s p o n s e s
English Q
Late Chinese Q
Early Chinese Q
ta l
Participants
60
tal
METHOD
80
eo
• Tested both L1 & L2 perceivers: English, &
Chinese natives (no inter-dental in L1), with
different L2 experience (early vs. late arrivals ).
100
a lv
• Nonnative (L2) perceivers may be impeded
in correct use of L2 visual cues non-existent
in their L1 [4,5].
Noise
100
en
• Examined audio-visual (AV) perception of
English inter-dental fricatives in quiet and cafénoise backgrounds.
% c o rre c t re s p o n s e s
• Native (L1) speech perception is enhanced
by visual information, especially in a poor
listening environment [1,2,3].
Congruent stimuli
Quiet
Current study
A - V
Simon Fraser University
Vancouver, Canada
RESULTS
Background
Late arrivals
Chad Danyluck
Simon Fraser University
Vancouver, Canada
i nt
INTRODUCTION
Haisheng Jiang
Norwegian University of Science &
Technology, Trondheim, Norway