Temporal integration windows in sign language processing
Language for presentation: English
Temporal integration windows are durations within which the input is “chunked” in
cognition. There have been several studies on integration windows of auditory
processing, visual processing, and speech perception (Viemeister & Wakefield, 1991;
Poizner & Tallal, 1987; Poeppel, et al., 2008), but none on sign language processing.
These durations are of theoretical interest for sign language research because they can
reveal the temporal dynamics for processing representational units.
In this study, the duration of windows for integrating American Sign Language was
tested by measuring the intelligibility of locally-reversed sentences and making
comparisons with findings from speech (Greenberg & Arai, 2001; Figueroa et al., 2009;
Stilp et al, 2010). Local reversals are created by dividing the sentences in various
increments and time-reversing the segments. Although sentences that are reversed at
small increments are almost perfectly intelligible, intelligibility falls dramatically at
larger increments, revealing limits for integrating the input and mapping it to meaningful
representations. In Experiment 1, native signers (age-of-acquisition < 1 year, n=14)
viewed 40 sentences of ASL under 8 different conditions ranging from zero (control
condition) to 934 ms of local reversal. Intelligibility was measured by the participants’
accuracy at signing back the original, unaltered sentence. Beginning at 267 ms, we found
a significant decrease in intelligibility, in contrast to 50 ms reported for speech. This
demonstrates a modality-effect for the temporal integration of language. In Experiment 2,
native signers (n=14) viewed 40 sentences that were both compressed at 2x rate and
locally reversed in the same increments used in Experiment 1. Previous work has shown
that ASL sentences played at 2x rate is similar in intelligibility to control conditions
(Fischer, et al., 1999), which we replicated. By compressing and locally reversing the
stimuli, we manipulated the amount of linguistic information in the reversed increments.
We found that the temporal integration window was proportionally reduced by half,
suggesting that it is dependent on the duration of linguistic units, as found in speech.
Finally, in Experiment 3, we used the same materials from Experiment 1 and tested late
learners of ASL (age-of-acquisition > 10 years, n=8). Intelligibility fell significantly at
the first reversal condition (133 ms). The ability to cognitively restore distorted stimuli
and integrate the input according to the size of the representational units requires
perceptual flexibility that late-learners do not have to the same extent as early learners.
In summary, our findings demonstrate 1) the impact of modality (auditory versus visual
processing) on the duration of temporal integration windows, where visually based ASL
is dramatically more resistant to this temporal distortion than spoken English and
involves longer time-windows for integration, 2) modality independent properties of
temporal integration where duration is directly linked with the rate of linguistic
information in both signed and spoken languages, and 3) illustrates the importance of
early language acquisition on temporal processing of language. These findings have
implications for the neurocognitive underpinnings of integration in perception and rates
in production and the role of input in early development for these aspects of language
processing.
Figure 1: Demonstration of how locally time-reversed stimuli were created for sentences
of ASL. This specific example shows reversals 133 ms in duration (reversals by 4 frames).
Figure 2: Results from Experiment 1.
References
Figueroa, V., Howard, M., Idsardi, W., & Poeppel, D. (2009). Rate and local reversal
effects on speech comprehension. Abstract in Neurobiology of Language
Conference, October 2009, Chicago, IL.
Fischer, S. D., Delhorne, L. A., & Reed, C. M. (1999). Effects of rate of presentation on
the reception of American Sign Language. Journal of Speech, Language, and
Hearing Research, 42(3), 568-582.
Greenberg, S., & Arai, T.(2001). The relation between speech intelligibility and the
complex modulation spectrum. In the 7th International Conference on Speech
Communication and Technology, Scandinavia, 473– 476.
Poeppel, D., Idsardi, W. J., van Wassenhove, V. (2008). Speech perception at the
interface of neurobiology and linguistics. Philosophical Transactions of the Royal
Society London B, 363,1071-86.
Poizner, H., & Tallal, P. (1987). Temporal processing in deaf signers. Brain and
Language, 30(1), 52–62.
Stilp, C. E., Kiefte, M., Alexander, J. M., & Kluender, K. R. (2010). Cochlea-scaled
spectral entropy predicts rate-invariant intelligibility of temporally distorted
sentences. Journal of the Acoustical Society of America, 128(4), 2112-2126.
Viemeister, N. F., & Wakefield, G. H. (1991). Temporal integration and multiple looks.
Journal of the Acoustical Society of America, 90, 858-865.