Human Activity Behavior and
Gestures Generation
in VR Systems
Conference on Intelligent Motion and Interaction
within Virtual Environments
The Gustav Tuck Theatre, UCL, London
15-17 September 2003
Maarten Sierhuis, Ph.D. (NASA ARC/RIACS)
William J. Clancey, Ph.D. (NASA ARC/IHMC)
Bruce Damer (Digitalspace, inc.)
Research Objective
• Ultimate Objective:
Develop just-in-time
uploadable VR training
applications for crews on
Mars or Space Stations.
• Technical challenges:
– Develop a generic agent
language for modeling and
“driving” bot-behavior in a
Virtual World Environment.
– Create 3-D models of
human work practices.
Overview
•
•
•
•
•
Virtual Worlds
Brahms
Work Practice Analysis
Brahms VE
The FMARS Model
Virtual Worlds
• Multi-user web-based VR
environments
– Allow people to interact
– Develop virtual communities
• Functionality
Users enter as Avatars
Bots (i.e. agents)
Virtual spaces
Virtual artifact and behavioral
objects
– Links to web-sites
– Chat
Active Worlds
–
–
–
–
Adobe Atmosphere
Benefits for VR Training Systems
• Multi-user
– Trainer and trainee
– Chat capability
•
“Light weight” VR application
– Web-based
– Applications are downloadable
– Training at a distance
• Multi-media functionality
– Different media as training material
• Commercial Virtual World
Builders
– User community
– Consultants for building Worlds
– Development community
Issue with Virtual Worlds
• Limited Avatar  World Interaction
– Strongly scripted behavior
– Limited variations
• Limited behavioral programming environment
– VW development support
– Limited Bot language support
What is needed is a way to model Avatar &
Bot behavior in the Virtual World.
Brahms
The Composer
The AgentViewer
Agent-Oriented Language
Java-based
Java API
Language Compiler
Interactive Development Environment
Discrete-event Virtual Machine
XML
End-user Simulation Displays
Runs on PC’s, Simulation History Data Base
Mac, Unix,
Linux ...
© Maarten Sierhuis
Symbolic Discrete
Event System
Situated-action Rules
Production Rules represent Qualitative Relationships!!
Multi-Agent Behavior
BRAHMS
Virtual
Machine
World State
fact F1
.
fact Fn
Java
fact Y
class Activity2 extends
AbstractExternalActiv
{ …. }
Scheduler
T:0 belief X
T:1 fact Y
….
….
T:N belief Z
World
environment
Java Agent 5
class Agent5 extends
AbstractExternalAgent
{ …. }
Brahms Engine
World State
Scheduler
Agent
Belief Set
Fact Set
Discrete
Event
Queue
Event
Distributor
Engine
Fact
Belief
RSN
RSN
Discrete Event
Queue
Work Selector
Work Executor
Reasoning with
Beliefs of Agents and World Facts
World
Fact: (It is 10:40AM)
Agent
Pete Conrad
belief: (the door of the SEQBay is closed)
open
belief: (PeteConrad is ready to offload the ALSEP)
belief: (AlBean is located in the SEQBayArea)
Fact: (the door of the SEQBay is closed)
open
detect
detect
Send(PeteConrad is ready to offload the ALSEP)
belief: (PeteConrad is located in the SEQBayArea
belief: (the door of the SEQBay is closed)
open
Fact: (It is Thursday Nov. 19, 1969)
Agent
Al Bean
belief: (AlBean is located in the SEQBayArea)
belief: (PeteConrad is located in the SEQBayArea)
belief: (PeteConrad is ready to offload the ALSEP)
Fact: (AlBean is located in the SEQBayArea)
Fact: (PeteConrad is located in the SEQBayArea)
Activity Scheduling and
Subsumption in Brahms
Priority 5
ACTIVITY 1
(other
activities)
Priority 0
ACTIVITY 3
Priority 10
ACTIVITY 4
WF 4
(impasse)
WF 1
ACTIVITY 2
(other
activities)
WF 2
ACTION X
WF 5
ACTIVITY 6
(other
workframes)
(other
activities)
Analyze the Work Practices of
Astronauts and Field Scientists
Research Method: Developing Models and
Simulations of existing experiences of
human collaboration.
Deploying Scientific Instruments
“Very sensitive instruments cannot tolerate the rough
handling of robotic deployment.” (Spudis, 1999)
• 1969 – 1972 Apollo Project - ALSEP Deployment
on the Lunar Surface
Scientific Field Studies
“The process that scientists use to conduct
exploration in the field is poorly understood, and
one cannot simulate what is not understood.”
(Spudis, 1999)
• 1969 – 1972 Apollo’s Geo-Science Field Studies
• 1998 – 2003 Haughton Mars Project (FMARS)
• 2001 – 2003 Mobile Agents Project
FMARS
ALSEP
Understand Work Practice
•
•
•
•
•
•
Individuals and teams (CoP)
Individual activities
Collaboration between individuals
Communication
Tools and artifacts
Environment/geography
Brahms VE
• Research Goal:
3
Brahms Integrated Development Environment (IDE)
– 3-D Virtual Reality Display
for Brahms Simulations
– Seamless integration between
Brahms VM and VW
Graphics Engine
– Bot language for Virtual
Worlds
• Inspired by The Sims and
Virtual World Communities
(Active Worlds)
• Collaboration with
DigitalSpace, Inc.
• Supported by Adobe
Atmosphere Development
Team
Brahms VM
Brahms Model Development
Source
Code
Compiler
Brahms Virtual Machine
Simulate
1
World
State
Dynamic
Model
Static
Model
VM
Controller
Compiled
Concepts
Config
Agent /
Engine
Scheduler
Logger
Event
Notifier
JAPI
Logs
Model
DB
Scenario
development
Analysis
Tools
Run
Virtual World Model Development
2
Load
4
Oworld / Adobe Atmosphere
Coupling Behavioral Model
with VW Model
Brahms => VW
• Agents => Bots
• Agent activities =>
Bot gestures
• Locations => 3D
model
• Agent belief-state =>
Bot display
VW => Brahms
•
•
•
•
Movement
Collision detection
Line of sight
Avatar Interaction
The FMARS Model
Flashline Mars Arctic/MDRS
Research Station
FMARS 2001 Phase 2
• Mars Society:
Research & Promotion
• “Holistic Sims”: Geology,
biology, computing,
medicine,… + Reporting
+ Mission support ops
• Investigate how we will
live & work on Mars
• Six-person crew
• Crew rotations
Dec-Aug 2001-2003
• Extensive publicity
Time Lapse Video Analysis
Activity Model
Brahms Model
Brahms Sim Output
Walk Upladder
CheckWaterLevel
Walk
Downladder
Brahms VM Output Format for Brahms VE:
<Event Name>|<Activity Type>|<Start Time>|<End Time>|<Actor>|<Activity Name>|[|<Origin Area>|<Destination Area
activity|move|156|164|projects.fmarsvre.RZ|Walk||projects.fmarsvre.WardroomTableArea|projects.fmarsvre.GalleyLadderArea
activity|move|164|169|projects.fmarsvre.RZ|Upladder||projects.fmarsvre.GalleyLadderArea|projects.fmarsvre.WaterTankArea
activity|primitive|169|256|projects.fmarsvre.RZ|CheckWaterLevel|
activity|move|256|261|projects.fmarsvre.RZ|Downladder||projects.fmarsvre.WaterTankArea|projects.fmarsvre.GalleyLadderAre
activity|move|261|269|projects.fmarsvre.RZ|Walk||projects.fmarsvre.GalleyLadderArea|projects.fmarsvre.WardroomTableArea
Gestures
• All scenarios, all agents
–
–
–
–
–
–
–
MovingToArea
GettingUp
SittingDown
OpeningCabinetDoor
FetchingFood
ClosingCabinetDoor
EatingSnack
• Planning Meeting
– All agents
• SittingThroughPlanning
– SB
• ReportingGeneratorStatusToCommander
• ReportingRadioRepairStatusToCommander
– RZ
Gestures are programmed as
• AnnouncingStartOfPlanning
Java Script
• RequestingGeneratorStatus
• RequestingWeatherReport
attached to Bots in the VW
• RequestingRadioRepairStatus
VR: Data, Sim, Depiction
• Depicted three scenarios in Atmosphere action scripts
• Not coupled real-time: Hard-coded timing and waypoints, so agents don't
run through objects and each other
• Next: Step into, inspect, & change running simulation; send personal agents
to remote locations (e.g., from RST to Mars EVA) get back data & QT
Future:
From “video” to “interactive” sims –
issues and solutions
• Making things dynamic
– Dynamic interface
– Synching Brahms & VE
clocks
• Interaction with
simulation
–
–
–
–
–
Movement
Collision detection
Line of sight
Displaying mental states
Avatar Interaction
Summary
• Research: “Just in Time” VR Training on Mars
• Combining Human Behavior Modeling with VR
• Brahms Agent Activity Language as Bot
Language
• Brahms VE: Brahms + Adobe Atmosphere
• Example: FMARS Model
• Future: Dynamic Interaction