Effects of Handling Real Objects and
Avatar Fidelity on Cognitive Task
Performance in Virtual Environments
Benjamin Lok
University of North Carolina at Charlotte
Samir Naik
Disney VR Studios
Mary Whitton, Frederick P. Brooks Jr.
University of North Carolina at Chapel Hill
March 25th, 2003
Current Immersive
VE Approaches
• System has limited shape
and motion information of
real objects.
• Most objects in the VE are
purely virtual
– User
– Tools
– Parts
• Most virtual objects are
not registered with a
corresponding real object.
Ideally
• Would like:
– Real objects in the VE, including accurate
virtual representations, or avatars
– Virtual objects responding to real objects
– Haptic feedback
– Correct affordances
– Constrained motion
• Example: Unscrewing a virtual oil filter from a
car engine model
Dynamic Real Objects
• By tracking and modeling dynamic objects
(change shape and appearance) would
– Improve interactivity
– Enable visually faithful
virtual representations
Previous Work: Incorporating
Real Objects into VEs
• Non-Real Time
– Virtualized Reality (Kanade, et al.)
• Real Time
– Image Based Visual Hulls [Matusik00, 01]
– 3D Tele-Immersion [Daniilidis00]
• Augment specific objects for interaction
– Doll’s head [Hinkley94]
– Plate [Hoffman98]
• How important is to get real objects into a
virtual environment?
Previous Work: Avatars
• Self - Avatars in VEs
– What makes avatars believable? [Thalmann98]
– What avatars components are necessary? [Slater93, 94,
Garau01]
• VEs currently have:
– Choices from a library
– Generic avatars
– No avatars
• Generic avatars > no
avatars [Slater93]
• Are visually faithful avatars
better than generic avatars?
Object Reconstruction System
• System takes n live
camera images.
• At each frame, generates
a view of the visual hull
of the real objects from
the user.
• It is combined with the
virtual environment
• 15-18 fps
• Allows incorporation of
dynamic real objects into
a virtual environment
Spatial Cognitive Task Study
Study Motivation
• Effects of
– Interacting with real objects
– Visual fidelity of self-avatars
• On
– Task Performance
• For spatial cognitive manual tasks
Spatial Cognitive
Manual Tasks
• Spatial Ability
– Visualizing a manipulation in 3-space
• Cognition
– Psychological processes involved in the
acquisition, organization, and use of knowledge
Task Performance Hypotheses
Real Objects: Participants will complete a spatial
cognitive manual task faster when manipulating
real objects, as opposed to virtual objects only.
Avatar Fidelity: Participants will complete a spatial
cognitive manual task faster with visually faithful
self-avatars, as opposed to generic avatars.
Task
• Manipulated identical
painted blocks to
match target patterns
• Each block had six
distinct patterns.
• Target patterns:
– 2x2 blocks (small)
– 3x3 blocks (large)
Measures
• Task performance
– Time to complete the patterns correctly
– If errors were made:
• Participants were notified
• Participants continued until all errors were corrected
• Other factors
– (After experience) Steed-Usoh-Slater Sense of
Presence Questionnaire (SUS) (larger study)
– (Before experience) spatial ability
– (Before and after experience) simulator sickness
Conditions
Purely Virtual
• All participants
did the task in a
real space
environment.
• Each
participant did
the task in one
of three VEs.
Real Space
Hybrid
Vis. Faithful Hybrid
Conditions
Task performance
Avatar Fidelity
Visually
Generic
faithful
Task performance
Interact
with
Real
objects
HE
Virtual
objects
PVE
VFHE
Real Space Environment
• Task was conducted
within a draped
enclosure
• Participant watched
monitor while
performing task
• RSE performance
was a baseline to
compare against VE
performance
Purely Virtual Environment
• Participant manipulated virtual objects
• Participant was presented with a generic avatar
Hybrid Environment
• Participant manipulated real objects
• Participant was presented with a generic avatar
Visually-Faithful Hybrid Env.
• Participant manipulated real objects
• Participant was presented with a visually faithful
avatar
Task Performance Results
120.00
Time (seconds)
100.00
80.00
Real Space
Purely Virtual
Hybrid
Visually Faithful Hybrid
60.00
40.00
20.00
0.00
Small
Large
Small Pattern Time (seconds)
Large Pattern Time (seconds)
Mean
S.D.
Mean
S.D.
Real Space (n=41)
16.8
6.3
37.2
9.0
Purely Virtual (n=13)
47.2
10.4
117.0
32.3
Hybrid (n=13)
31.7
5.7
86.8
26.8
Visually Faithful Hybrid (n=14)
28.9
7.6
72.3
16.4
Task Performance Results
120.00
Time (seconds)
100.00
80.00
Real Space
Purely Virtual
Hybrid
Visually Faithful Hybrid
60.00
40.00
20.00
0.00
Small
Large
Small Pattern Time
Large Pattern Time
T-test
p
T-test
p
Purely Virtual vs. Vis. Faithful
3.32
0.0026**
4.39
0.00016***
Purely Virtual vs. Hybrid
2.81
0.0094**
2.45
0.021*
Hybrid vs. Vis. Faithful Hybrid
1.02
0.32
2.01
0.055
* - significant at the =0.05 level
** - =0.01 level
*** - =0.001 level
Debriefing Responses
• They felt almost completely immersed while performing the task.
• They felt the virtual objects in the virtual room (such as the painting,
plant, and lamp) improved their sense of presence, even though they
had no direct interaction with these objects.
• VFHE and HE participants felt tactile feedback of working with real
objects improved their performance and sense of presence.
• They felt that seeing an avatar added to their sense of presence.
• PVE and HE participants commented on the fidelity of motion,
whereas VFHE participants commented on the fidelity of appearance.
• VFHE participants reported getting used to manipulating and
interacting in the VE significantly faster than PVE participants.
Conclusions
Study Conclusions
• Interacting with real objects provided a quite
substantial performance improvement over
interacting with virtual objects for cognitive
manual tasks.
• Avatar fidelity does not appear to affect cognitive
task performance.
Handling real objects makes task
performance and interaction in the VE
more like the actual task.
Future Work
• Improved Object Reconstruction System
– model fidelity
– lag and latency (could improve results)
• Further studies to illuminate:
– Effect of real objects and avatar fidelity on:
• Sense of Presence
– Expand on the relationship between avatar
kinematic fidelity and visual fidelity
Thanks
Collaborators
Dr. Larry F. Hodges
UNC-CH Effective Virtual Environments
Object Reconstruction System
I3D2001 and I3D2003
For more information:
http://www.cs.uncc.edu/~bclok/research/vr2003
Funding Agencies
The LINK Foundation
NIH (Grant P41 RR02170)
National Science
Foundation
Office of Naval Research