Multisensory Interplay
Jon Driver and Toemme Noesselt
Sensory Research
Sensory Research
Behavioral Consequences of
Multimodality
• Joint estimates of single property
o Spatial Ventriloquism
o Auditory Driving
o McGurk Effect
• Can modalities affect each other
without creating a single unified
percept?
o Touch at a location can help
perception of color
o Sound-induced Illusory Flash
o Orientation discrimination
improves with multiple beeps
Convergence Zones
• Superior Colliculus
o inputs from somatosensory, auditory, and visual areas
o Super or Sub-additive responses found for combination
Additivity in SC
• most likely with weak
unisensory inputs
o Ceiling effect?
o Neural limitations?
• Late onset in
development
• Depends on
multisensory cortex
Convergence Zones
Testing for Convergence
• Anatomical studies:
o Direct connections between different sensory areas
• Single-cell studies:
o Response to stimulation from different modalities
• Neuroimaging:
o Large-scale responses based on BOLD signal
Influences on 'Sensory-Specific' Areas
• Growing body of research shows that sensory-specific areas
might be an artifact of the studies done with them
• Examine studies that use:
o fMRI
o EEG
o Invasive recording in animal models
fMRI
• Caveat:
o fMRI has been shown to respond to attention and imagery
• For example, speech may be imagined when viewing lip
movements
fMRI analysis
• Inspired by Stein (SC), some look for sub-, super-additive
responses
• Maybe linearity is normal, though, so some use max or
mean criteria
• Difficult because of spatial resolution
Convergence in V1?
Amedi, Jacobsen, Hendler, Malach, and Zohary,
2002
ERP results example
• Tactile stimulation
• Visual cue
• ERP extracted
ERP studies
• visual N1 enhanced when tactile stimulation occurred at
same location as a visual event
• visual P1 modified by taskirrelevant sound
• P1 modified by attend-visual
relative to attend-tactile
conditions
ERP studies
• ERPs show early multi-sensory effects (~30 ms)
• Poor localization
• Potential methodological confounds
Invasive Studies
• Current-source densities
(CSD) reflect local PSPs
• Region of auditory
association cortex
• Location and timing of
stimulation consistent with
auditory feed-forward,
visual feed-back
Invasive Studies
• Posture may affect responses to auditory signals in A1
• Tactile stimuli modulate initial response to auditory signals in
A1
Multisensory Interplay
• Examples of converging zones of multi-sensory input
• Examples of interplay: one modality affects another
• What frameworks does this evidence suggest?
Possible Frameworks
• A) All Multisensory
• B) Bimodal Brain Areas
• C) Critical Feedback Circuitry
All Multisensory
• Unlikely to be completely
undifferentiated
• Even primary sensory
areas responsive to
multiple modalities
All Multisensory
• Thalamus might be
source of multisensory
interplay
• Tactile stimulation can
affect first neural
response in A1,
hypothesized from
thalamus
• Found in gerbils, hard to
study in humans
All Multisensory
• Direct coritco-cortical
influences
• Anatomical evidence:
single synapse from ACVC and AC-SSC, AC-OC
• However, not as many as
to conventional Multisensory areas
• Role still unclear
All Multisensory
• Still overwhelmingly
"sensory-specific"
Bimodal Brain Areas
• Less extreme version of
account A
• Similar to current account,
with more multi-sensory
regions
• Parallel multi- and singlemodality processing could
explain early EEG
modulation
Bimodal Brain Areas
• Different areas in auditory
areas may be connected
to distinct visual areas
• Bimodal interplay would
be affected by
transduction time,
explaining BOLD
response time
differences
Feedback Circuitry
• Effects in primary areas
might be feedback from
convergence zones
• Evidence from effective
connectivity in fMRI,
tactile stimulation
increases visual response
• Evidence from EEG
source-localization: STS VC
Feedback Circuitry
• Evidence from invasive
recordings: late A1
stimulation from vision
(speculation?)
• Feedback can be tested
directly, but very little has
been done
Remarks
• Perhaps the rival frameworks are all valid for certain
situations
• Perhaps primary cortex responses in the blind and deaf can
help tease out what the multisensory roles are
• New techniques will allow testing causal interplay