Integrating Research on Unimodal and
Multimodal Perceptual Development:
The Intersensory Redundancy Hypothesis
Robert Lickliter and Lorraine E. Bahrick
Infant Development Research Center
Department of Psychology
Florida International University
IDRC
Infant Development Research Center
Our World is Multimodal and Dynamic
• Objects and events can be concurrently
seen, heard, and felt
• Visual information typically occurs at the
locus of a sound
• Objects and events move and perceivers
move in relation to them
IDRC
Infant Development Research Center
Specific Questions to Consider
• How and on what basis do infants begin to
parse and derive meaning from the flux of
multimodal and unimodal stimulation?
• What rules govern selective attention so that
some aspects of events are perceived and
others ignored?
• How do these capabilities change and
develop with experience?
IDRC
Infant Development Research Center
Dichotomy Between Unimodal and
Multimodal Research Hinders
Integrative Theories
• Research on the development of unimodal
perception has rarely been integrated with research
on intersensory perception.
• Theories of unimodal and multimodal functioning
remain separate yet address similar content areas
(e.g. faces, voices, speech, memory)
• All events provide both amodal and modality
specific information: Despite this fact, little
research has addressed how perception of this
information is intercoordinated
IDRC
Infant Development Research Center
Amodal Information
• Amodal information is information that is
invariant across the senses. It is not specific to a
particular sense modality, but is completely
redundant across more than one modality (e.g.
tempo, rhythm, synchrony, intensity, colocation)
• Amodal information consists of temporal, spatial,
or intensity information.
• All events provide amodal information because
they occur over time and space.
IDRC
Infant Development Research Center
Modality Specific Information
• All events also provide modality specific
information.
• Modality specific information is information that
can only be perceived through a particular sense
modality (e.g., color, pattern, pitch, timbre,
temperature)
• All events provide both amodal and modality
specific information
IDRC
Infant Development Research Center
Integration of Unimodal and Multimodal
Research:
The Intersensory Redundancy Hypothesis (IRH)
• The IRH is a framework that describes how selective
attention is allocated to different properties of events in the
context of multimodal vs. unimodal stimulation during
early development.
• The IRH describes how detection of amodal and modality
specific properties of events is intercoordinated.
• For example, when attending to a person talking, one could
detect redundant amodal information (synchrony, rhythm,
tempo of audiovisual speech) or modality specific
information (appearance of the face, clothes or
pitch/timbre of the voice).
• According to the IRH, redundant stimulation is highly
salient and recruits attention to the redundantly specified
properties at the expense of other stimulus properties.
IDRC
Infant Development Research Center
Predictions of the Intersensory Redundancy
Hypothesis
(Bahrick & Lickliter, 2000, 2002)
1.
2.
3.
Amodal properties are most salient in multimodal,
redundant stimulation (e.g. rhythm & tempo of
audiovisual speech)
Modality specific properties are most salient in unimodal
stimulation (e.g. facial configuration; pitch and timbre of
voice)
With development, attention becomes more flexible and
infants can detect both amodal and modality specific
properties in unimodal and multimodal stimulation
IDRC
Infant Development Research Center
Predictions of the Intersensory Redundancy
Hypothesis
(Bahrick & Lickliter, 2000, 2002 )
A>B
and
D>C
Stimulus Property
Stimulus Available
for Exploration
Amodal
Modality-Specific
Multimodal
A
C
Unimodal
B
D
IDRC
Infant Development Research Center
Research Supporting the Salience of Amodal
Properties in Multimodal Stimulation
•
•
•
•
discrimination of rhythm (Bahrick & Lickliter, 2000, 2004)
discrimination of tempo (Bahrick, Flom, & Lickliter, 2000)
discrimination of affect (Flom & Bahrick, in prep.)
perceptual learning in bobwhite quail
embryos and chicks (Lickliter, Bahrick, & Honeycutt, 2002, 2004)
• discrimination of prosody in speech (Bahrick,
Castellanos, & Shuman, in prep.)
IDRC
Infant Development Research Center
Discriminating Prosody of Speech
Bahrick, Castellanos, & Shuman (in prep.)
Prosody Change
No Prosody Change
IDRC
Infant Development Research Center
Discriminating Prosody of Speech
Prosody Change
No Prosody Change
IDRC
Infant Development Research Center
Visual Recovery (seconds)
Visual Recovery to a Novel Passage with a Change in
Prosody versus No Change in Prosody for 4-monthOld Infants
8
7
6
5
4
3
2
1
0
-1
-2
-3
7.05*
(8.35)
3.36
(7.60)
2.55
(5.72)
.48
(7.13)
-2.41
(4.93)
-.45
(4.48)
Prosody Change
No Prosody Change
*p < .05
Bimodal
Synchronous
IDRC
Infant Development Research Center
Unimodal
Auditory
Bimodal
Asynchronous
Summary: Findings on Perception of
Amodal Properties
• For both human and animal infants, perception,
learning, and memory for redundant amodal
stimulation such as rhythm, tempo, and prosody is
facilitated in bimodal (audiovisual) contexts and
attenuated in unimodal stimulation.
• With experience, attention becomes more flexible
(i.e., infants can detect both amodal and modality
specific properties in multimodal stimulation).
IDRC
Infant Development Research Center
Predictions: Discrimination of Modality
Specific Properties
• Detection of modality specific properties
(faces, voices, orientation) is facilitated in
unimodal stimulation and attenuated in
bimodal stimulation in early infancy
• Again, later in development attention
becomes more flexible and infants can
detect modality specific properties in both
unimodal and bimodal stimulation
IDRC
Infant Development Research Center
Research Supporting the Salience of
Modality Specific Properties in
Unimodal Stimulation
• Discrimination of Faces (Bahrick, Lickliter, Vaillant, Shuman, &
Castellanos, 2004)
• Discrimination of Voices (Bahrick, Lickliter, Shuman, Batista, &
Grandez, 2003)
• Discrimination of Orientation (Bahrick, Lickliter, & Flom,
submitted)
IDRC
Infant Development Research Center
Discrimination of Faces
Bahrick, Lickliter, Vaillant, Shuman, & Castellanos (2004)
• Infants of 2- and 3-months were habituated
to films of a woman speaking under
bimodal or unimodal visual conditions
• Infants were then tested to determine if they
could discriminate between the familiar and
a novel woman’s face under their respective
conditions
IDRC
Infant Development Research Center
Face Change Task:
Unimodal Visual
IDRC
Infant Development Research Center
Face Change Task:
Bimodal
IDRC
Infant Development Research Center
Face Discrimination: 2-month-olds
Results supported our predictions: 2-month-olds discriminated between the two faces
when they were presented unimodally (speaking silently), but not bimodally (speaking
audibly). Further, they also discriminated when the faces and voices were
asynchronous, indicating that it was the redundancy of the bimodal presentation that
hindered face perception.
16.58**
(10.00)
18
Visual Recovery (s)
16
14
10.01**
(10.89)
12
10
8
6
4
3.63
(9.05)
2
0
Bimodal
Synchronous
Face and Voice
IDRC
Infant Development Research Center
Unimodal
Speaking Face
Bimodal
Asynchronous
Face and Voice
***p < .001
**p < .01
*p < .05
Face Discrimination: 3-month-olds
However, one month later, when infants were 3 months old, they were able
to discriminate the faces under both bimodal and unimodal conditions.
Visual Recovery (s)
12
10
8.65*
(13.51)
9.26*
(14.78)
8
6
4
2
0
Bimodal
IDRC
Infant Development Research Center
Unimodal
***p < .001
**p < .01
*p < .05
Summary: Face and Voice
Discrimination
• Young infants are better at discriminating between two
voices and between two faces when they are presented
unimodally than bimodally
• Older infants have more flexible attentional skills and can
discriminate between faces in both unimodal and bimodal
stimulation
• Detection of modality specific information (such as that
distinguishing faces or voices) emerges first
developmentally under conditions of unimodal stimulation
and is later extended to bimodal stimulation
IDRC
Infant Development Research Center
Summary of Findings Across Studies:
Early Attentional Biases (IRH)
Attentional biases are evident in early development and they promote
perceptual processing of some properties of events at the expense of others.
Later in development attention becomes more flexible.
Stimulus Property
Stimulus Available
for Exploration
Amodal
Modality-Specific
A
Multimodal
Unimodal
IDRC
Infant Development Research Center
C
Rhythm Tempo
Prosody Affect
Timing of Call
B
D
Faces
Voices
Orientation
A Path to Flexibility:
Educating Attention
• Detection of amodal properties that “pop out” in
bimodal stimulation (rhythm, tempo) can educate
attention to those same properties in subsequent
unimodal stimulation of the same event.
• For example, if infants selectively attend to the
rhythm of a hammer tapping in bimodal redundant
stimulation, then they might be able to continue to
perceive the rhythm in unimodal auditory
stimulation,when they turn their head away.
• We have recently tested this hypothesis with both
human and animal subjects.
IDRC
Infant Development Research Center
Educating Attention: Animal Research
(Lickliter, Bahrick, & Markham submitted)
• Quail embryos were exposed to an individual maternal call
for 10 min/hr during the 24 hr prior to hatching under
several conditions: (a) bimodal (redundant call and light)
unimodal, (b) unimodalbimodal, (c) unimodal only,
and (d) a no stimulation control condition.
• Following hatching, chicks were tested at 2 days of age for
their preference for the familiar maternal call over a novel
maternal call.
IDRC
Infant Development Research Center
Results: Educating Attention
• Results supported our predictions and indicated that only
the chicks that received the bimodal unimodal
stimulation as embryos showed a postnatal preference for
the familiar call.
• Even though embryos in the unimodal bimodal
condition received the same amount and kind of
stimulation, the order mattered: embryos only learned the
call when the bimodal preceded the unimodal presentation,
creating the opportunity for educating attention.
• Attention was educated because the redundancy between
the light and the call highlighted the timing of the notes of
the call and this then generalized to the subsequent
auditory exposure to the call.
IDRC
Infant Development Research Center
Development of Attentional
Flexibility
• So, education of attention provides one account
for the development of attentional flexibility in
early development
• That is, it provides one avenue for infants’
sensitivity to amodal properties to generalize from
bimodal to unimodal stimulation
• There is likely a similar educating attention effect
for modality specific properties, where attention is
educated from unimodal to bimodal stimulation
(e.g. face and voice perception).
IDRC
Infant Development Research Center
To conclude: What rules govern selective
attention to amodal and modality-specific
properties of multimodal events in early
development?
• The IRH provides a description of what properties
are attended to first: redundant amodal properties
in multimodal stimulation and non-redundant
modality-specific properties in unimodal
stimulation. These processing precedents can have
a powerful organizing effect on early perceptual
development, due in large part to the fact that
initial conditions serve to guide and constrain
what follows.
IDRC
Infant Development Research Center
Integrating Unimodal and Multimodal Research
• To develop unified and more ecologically valid
theories of perceptual development, we need to
address both unimodal and multimodal
functioning.
• Accomplishing this will require studies that
examine both unimodal and multimodal
conditions in single research designs.
• The IRH provides one example of this integrative
approach.
• Such approaches can help bridge the currently
separate literatures on the development of early
unimodal and multimodal perception.
IDRC
Infant Development Research Center