Multimodal Virtual Environments:
Response Times, Attention,
and Presence
B93902033陳柏叡
Introduction




Efficiently combine sensory information from
two or three channels
Three channels: vision, audio, haptic
Differentiates from older technologies
- communicating only via a single channel
The underlying cognitive mechanisms are still
elusive and unknown
Environments Richness Results in a
Complete and Coherent Experience


Single channel conveys limited and
insufficient information to the senses
- Engenders a lower sense of presence
Multimodal VE provide a greater extent of
sensory information to the observer
- Sense of being present is felt stronger
Multimodal VE are Mimicking
Reality Better




Many of daily experiences in the real world
are fundamentally multimodal
Grasping an object – visual, haptic
Gastronomic pleasure – taste, smell, vision
Multimodal VE have a clear advantage, in
mimicking a multimodal phenomenon
Better Integration and Filling in of
Missing Information



Synthetic VEs provide fewer sensory cues
than physical environments
User needs to interpolate sensory stimuli to
create a functional mental modal
Result in an enhanced sense of presence
Faster Mental Processing Enables
Deeper and Richer Experience



Using Reaction Time to compare the brain
processing speed of unimodal, bimodal, and
trimodal signals
Faster processing at the perceptual stage
allow users more time in consequent
cognitive stages
Contributing to the great sense of presence
Experimental Design



Materials – touch-enabled computer interface,
haptic device(force-feedback)
Participants – 16 students(unaware of the
purpose of the experiments)
Under the guidelines of the ethical committee
and with its approval
Stimuli





Holding the stylus in their hand as if they
were writing
On each trail the computer generated a
sensory stimulation – uni, bi, trimodal
Visual – changing color
Auditory – compound sound
haptic – resisting force
Procedure




Pressing a button on the stylus as soon as
they detected the stimuli
Computer generated the stimulation randomly
Consisted of six blocks of trails (three with
dominant hand, three with the other hand)
Each of blocks consisted of 105 single trail, in
which each of the seven conditions
Result-Unimodal



V – dominant hand 430ms (SD=94)
not dominant 436ms (SD=92)
A – dominant hand 330ms (SD=103)
not dominant 326ms (SD=76)
H – dominant hand 318ms (SD=99)
not dominant 334ms (SD=91)
Result-Bimodal



VA – dominant hand 302ms (SD=78)
not dominant 304ms (SD=70)
HV – dominant hand 294ms (SD=75)
not dominant 306ms (SD=77)
HA – dominant hand 272ms (SD=81)
not dominant 280ms (SD=69)
Result-Trimodal


dominant hand
not dominant
430ms (SD=94)
436ms (SD=92)
Between-Hand comparisons
- Revealed insignificant differences in the
VA and HA conditions, and only marginal
difference in HV conition
Multimodal Stimulation and Attention


Multisensory enhancement is modulated by
attention
Hypothesizing that the brain allocates a
greater amount of attention to multimodal
and employing a larger neural network
Attention and Presence

The link between attention and presence is
already documented in literature
Conclusion


At the initial perceptual stage –
 Multimodal information is processed faster
 Activating larger neural network, increasing user’s
attention
At the consequent cognitive stages, more time to




Acquire a wider range of details and subtle for the display
Fill in missing information
Integrate all these informative cues
Contributing to a greater sense of presence