DEP 3115
Dr. Bahrick
Intersensory Perception
I.
Definitions and questions
A. Intersensory/intermodal perception - perception of an object or event that makes
information available to two senses simultaneously
B. How does the infant learn what sights and sounds belong together?
C. Capability is fundamental to most human abilities
II.
Two theoretical positions
A. Association-integration view (traditional learning/empiricist)
1.
Infants learn to put together (associate) stimulation from different senses
2.
Before integration - perceive unrelated streams of light and sound
3.
Piaget has similar integration view, not based on association
B.
Gibson's invariant detection view
1.
Senses unified at birth - no integration necessary
2.
Infants differentiate (separate) objects and events by detecting invariants
3.
Similar views: Bower; Intensity matching
III.
Amodal invariants - the same information available in 2 or more senses
A. E.g, rhythm, synchrony, tempo- can be seen and heard)
B. Some invariants differentiated at birth, others later
C. Perceive amodal invariant = attend to the event as a whole - e.g. ventriloquism
effect (temporal synchrony)
IV.
Infant research promoted by invariant detection view
A. Wertheimer (1961) - coordination of audiovisual space at birth
B. Facilitates development of intermodal perception
C. Amodal invariants tell us which object is producing a sound
V.
Which amodal invariant relations can infants detect?
A. Method - Spelke - intermodal preference method
B. Infants detect temporal synchrony (Spelke, 1979)
1.
2 objects bouncing at same rate; one synchronous
C. Tempo of action (Spelke, 1979)
1.
Two objects bouncing at different rates; soundtrack out of synchrony with
both but tempo matched to one; infants search to matching tempo
D. Temporal microstructure specifying object substance (rigid vs elastic) (Bahrick,
1983)
1.
Wooden blocks & elastic sponges; infants look more to film that was
previously in sound
E.
Rhythm/temporal pattern (Allen, Walker, Symonds & Marcell)
1.
VI.
Habituate to a rhythm visually or acoustically; infants dishabituate to a new
rhythm regardless of modality
Amodal versus arbitrary intermodal relations-- most events make both available
A. Amodal (synchrony, rhythm)- detected without much specific experience
B. Arbitrary (specific voice-appearance of face; color-pitch) must be learned through
specific experience
VII. People: Faces and voices (provide amodal and arbitrary relations)
A. Spelke & Owsley 1979: 4-mo-olds match mother's and father's faces and voices
B. Gender matching- amodal (Walker-Andrews, Bahrick, Raglioni, & Diaz, 1991)
C. Age matching: amodal- Infants match faces and voices of adults and children (Bahrick,
Netto, Hernandez-Reif,1998)
D. Arbitrary face-voice matching (Bahrick, Hernandez-Reif & Flom, 2005)
1.
Infants learned which face and voice of two women or two men belonged
together
E.
Infants (5-6 mos) match own face and voice: arbitrary relations
F.
Affect matching -amodal
1.
Walker-Andrews- 5-7 mos match happy, sad, angry faces and voices in unfamiliar
adults
2.
3-mo-olds match affect in mother- familiar adult
VIII. Blind infants- sonic guide (Bower & Aitken) use amodal invariants to “see” with sound
IX.
Perception of amodal visual-tactile relations
A. 3 amodal invariants -- texture, substance, and shape
B. Meltzoff & Borton (1979) - cross modal transfer of shape
1.
1-month-olds recognize shape visually on basis of tactile familiarization
C. Streri- newborns show visual-tactile transfer of shape
X.
Conclusions- Invariant Detection Research (“What” questions)
A. Consistent with Gibson's view
B. Invariant detection 2-7 mos doesn't refute, but weakens integration view
C. “What” questions vs “how” questions
XI.
“How” questions: How does development proceed?
A. Intermodal learning (Bahrick 1988)
1.
Training with film-soundtracks: 1) appropriate/synchronous,
2) approp nonsynch 3) inappropriate/synch, 4) inapprop/synch 5) no training
2.
Test learning- two films and one soundtrack sync with both films
3.
Results: Learning only with appropriate/synchronous training
4.
“How” questions: Learning based on two amodal relations, synchrony and object
composition, and not on association-- supports Gibson’s view
B. Increasing specificity and intermodal perception (Bahrick 2001)
1.
Infants detect global amodal relations (synchrony) prior to more specific
amodal (microstructure specifying composition)
C.
D.
E.
Increasing specificity: Infants detect amodal (synchrony & microstructure) prior to
arbitrary intermodal (pitch-color/shape) relations (Bahrick 1992)
Newborns detect arbitrary relations if they are linked by amodal contingency (Slater et al
1999)
Arbitrary relations learned only in context of amodal relations (gatekeeper)
1.
Gogate & Barhick (1998) speech sound-object relations learned if synchronous
2.
Pitch-color/shape (Bahrick, 1992) learned in presence of synchrony
3.
Arbitrary face-voice relations (Bahrick et al 2005) learned if synchronous
4.
Arbitrary visual-tactile relations (Hernandez Reif & Bahrick, 2001) learned in
presence of amodal shape information
XII. “How” questions: The intersensory redundancy hypothesis (IRH) Bahrick & Lickliter, 2000
A. A theory of selective attention to amodal vs modality specific information
B. Amodal = synchrony, rhythm, tempo, temporal microstructure, intensity
C. Modality specific information - can be detected in only one modality (color,
pattern, pitch, timbre, temperature)
D. Intersensory redundancy: same information presented in synchrony to more than one sense
(only amodal)
E.
Predictions of the IRH: Early development
1.
Amodal properties detected in multimodal, redundant stimulation
2.
Modality specific properties detected in unimodal, nonredundant stim.
3.
Later in development infants detect both amodal and modality specific
properties in unimodal and multimodal stimulation
F.
Research supporting IRH- Prediction 1: Detecting amodal properties (A>B)
1.
Discrimination of rhythm at 5 mos only in bimodal and not unimodal stimulation
2.
Discrimination of tempo at 3 mos only in bimodal and not unimodal stimulation
3.
Discrimination of affect (happy, sad, angry): first only bimodal (4 mos), then
auditory (5-mos), then visual (7-mos).
4.
Discrimination of prosody bimodal--> unimodal
G. Animal studies also support IRH Prediction 1
1.
discrimination of amodal information (timing of maternal call in bobwhite quail
chicks) is facilitated in bimodal redundant stimulation
H. Summary Prediction 1 of the IRH: For young humans and animals, learning of amodal,
redundant information is facilitated in bimodal (audiovisual) stimulation and attenuated in
unimodal stimulation
I.
Research supporting IRH- Prediction 2: Detecting modality specific properties (D>C)
(Faces, voices, orientation of hammer)
1.
Face discrimination: 2 mo olds discriminate faces in unimodal and asynchronous
stimulation, but not bimodal synchronous (audiovisual speech); 3 mo-olds discriminate
faces in bimodal stimulation
2.
Voices and orientation similar patterns
J.
Summary Prediction 2 of IRH: For young infants, learning modality specific information
is facilitated in unimodal stimulation and attenuated in bimodal stimulation -because
redundancy competes for attention
XIII. Summary and conclusions: Development of intersensory perception
A
B.
C.
D.
E.
F.
Temporal synchrony binds stimulation across the senses
Since most stimulation is multimodal, this favors detection of amodal, redundantly
specified properties over modality specific properties in early development.
Perception of amodal properties emerges first in multimodal stimulation and is
later extended to unimodal stimulation.
Perception of modality specific properties in a given domain emerges later in
development than perception of amodal properties.
Across development, attention becomes more efficient and flexible.
The development of perception is characterized by increasing specificity- (Gibson).