Networked Virtual Environments Helmuth Trefftz htrefftz@eafit.edu.co Eafit University - Medellín, Colombia Introduction • Why are you here? Helmuth Trefftz • Associate Professor at Eafit University, Medellín, Colombia. • Director of Virtual Reality lab at Eafit. • Involved in NVEs since 1996. • Ph.D. in Computer Engineering in 2001 (Rutgers University, NJ). • Ph.D. thesis: Heterogeneity in Virtual Environments. Agenda • • • • • • • • • Introduction (15´) History (10´) Networking Concepts (25) Architectures (20´) Shared State (45´) Scalability and Performance (45´) Research Topics (30´) Toy application (30´) Summary and Conclusions (10´) Introduction • Why Networked Virtual Environments? • 1. Technology is here: – Powerful commodity 3D graphics accelerators on every PC. – Internet is almost everywhere (wired or wire-less). • 2. Need for applications is here: – – – – Multiplayer games Military combat training Design teams across continents … Introduction • What is a Networked Virtual Environment? – “A Networked Virtual Environment is a software system in which multiple users interact with each other in real-time, even though those users may be located around the world” [Singhal/Zyda 1999] http://www.crg.cs.nott.ac.uk/research/systems Introduction • “Collaborative Virtual Environments (CVEs) are online digital places and spaces where we can be in touch, play together and work together, even when we are, geographically speaking, worlds apart… In CVEs we can share the experience of worlds beyond the physical” [Churchill/Snowdon/Munro 2001] Introduction • Key elements: – A shared sense of space (common virtual space) – A shared sense of presence (avatars) – A shared sense of time (real-time) – A way to communicate (text, voice,…) – A way to interact (among users and with the virtual world) – TELEPRESENCE Introduction • Key components – Graphic engines and displays – Control devices (keyboard…trackers) – Processing Systems – Data Network Introduction Common Virtual World Current challenges • • • • Network Bandwidth/Latency Heterogeneity Distributed Interaction (real-time) Resource Management - Scalability History - Military • SIMNET (Simulator networking)[Miller/Thorpe95] – Distributed military environment developed for DARPA between 1983 and 1990. – Aimed at: • Building high-quality, low cost simulators • How to network the simulators together. – Created 11 sites with 50 - 100 simulator each. History - SIMNET History - Military • SIMNET Architecture – Object-event architecture – Autonomous nodes (each entity is responsible for informing its state to others) – Dead Reckoning • Average messages: – Slow moving vehicles: 1mess/sec. – Fast moving vehicles: 3mess/sec. History - Military • SIMNET Modules – Network Interface – Other-vehicle state table – Controls – Own-vehicle dynamics – Sound generation History - Military • SIMNET – Multicast groups: each exercise is assigned a multicast address. – Multiple exercises can co-exist • Scalability – Exercise in March 1990: • 850 objects • 5 sites • Network: T-1 History - Military • DIS (Distributed Interactive Simulation) [IEEE93] – Started in 1989 as an effort to document the comm. protocol of SIMNET. – Became an IEEE Standard 1278. • Definition of 27 PDUs (Protocol Data Unit). Examples: – – – – Entity State: position, orientation, velocity changes. Fire Detonation Collision • Each 5 seconds: a “heart-beat” state PDU. History - Military • DIS - Traffic: – 96% Entity State PDUs – Rest: • • • • Fire Detonation Collision Others… History - Military • DIS - Scalability – Designed for 300-500 users History - Military • DIS - Fully distributed and Heterogeneous – Each machine that can read/write DIS PDUS and manages the state, can participate. – Heterogeneity can pose problems. • Slow machines can fall behind processing messages from fast machines. • Inconsistencies! History - Games • SGI Flight – Flight simulator demo program used from 1984 to 1992. – Created in 1983. – Networked version shown in SIGGRAPH 1984. History - SGI Flight Download source code from: http://www.berkelium.com/OpenGL/readme.html History - Games • Doom [idSoftware] – Released in December 1993. – Initial version: no dead-reckoning, messages at frame-rate. – 10s of millions of downloads. – Source code was available. History - Doom History - Academia • NPSNET [NPSNET] – “The longest continuing academic research effort in networked virtual environments” [Singhal/Zyda 1999] – Focus on software technology for implementing large-scale virtual environments (LSVE). History - Academia Courtesy of Naval Postgraduate School. History - Academia • NPSNET – Contracted initially to handle SIMNET terrain. – Afterwards, improvements on protocols: • • • • Protocol for Lans only Bridges for Lans and Wans Implementation of IP Multicast for Wans vrtp proposal. History • DIVE (Distributed Interactive Virtual Environment) [Hagsand96] [DIVE] – Built at the Swedish institute of Computer Science. – World Database: • Distributed • Fully Replicated • New objects can be added/modified in a RELIABLE fashion. • Distributed lock mechanism. History - DIVE History - DIVE History - DIVE History - DIVE • Videos History • DIVE – Because of the reliable multicast implementation they use, DIVE does not scale well beyond 32 participants. – DIVE 3 uses a basic communications library based on IP multicast and Scalable Reliable Multicast (SRM). History • MR - TPP Minimal Reality Toolkit Peer Package [Shaw/Green93] – MR-TTP is based on UDP (unreliable). Lost packages are ignored. – Instead of heartbeat: sends packages at the rate of the input device. – Topology: Complete Graph Connection. – Number of users: 4 or less. History • MASSIVE-1 [MASSIVE][Greenhalgh/Benford9 5] • Developed at Nottinham University’s CRG (Computer Research Group), led by Steve Bendford and Chris Greenhalgh. History • MASSIVE-1 – Multi-user distributed V.R. system – Runs on Sun and SGI platforms – Textual, graphical and audio client programs – Interaction controlled by aura, focus and nimbus – Connection oriented networking – Scalability: “MASSIVE usually works with up to about 10 users ” (from Massive-1 home page). History - MASSIVE Video History • MASSIVE 2 and 3 – Scalability and Distribution based on Locales – Current networking via TCP client-server style (to be combined with multicast) – The agent that creates an environment acts as distribution server for that environment – Persistency – Management of audio links. History • Summary – A number of successful Networked VEs has been created: • Military • Gaming • Research – Differ greatly: • What can the users do. • Technological decisions. Networking Concepts • Latency – Amount of time to transfer a bit of data from one point to another. – Latency has a direct impact on interaction inside the virtual world. – The designer cannot really reduce latency. It is possible to hide it or reduce its impact. Networking Concepts • Latency - causes: – Physical limitations: speed of electromagnetic waves in the transmission material (aprox. 8.25 msec per time zone). – Delays introduced by the endpoint computers. – Delays introduced by the network itself (routers). Networking Concepts • Bandwidth – Rate at which the network can deliver data to the destination host. – Examples: • Modem: 56Kbits per second. • Ethernet: 10 or 100 Mbits per second. • Fiber-optic: 10 Gbps or more. Networking Concepts • Reliability – How much data is lost by the network and how that loss is handled. – Causes: • Routers discard some of the messages (up to 50% in some cases). • Data gets lost in the communication media. Networking Concepts • Reliability - How to handle data that is lost? – Detect/Correct: Error-correcting codes. – Detect: ACKS, NACKS. Networking Concepts • TCP: Point-to-point connection to an application running in another machine. • Data Checksum for integrity. • Flow-control: to avoid flooding of messages, including acknowledgements. • Keeps strict ordering and consistency of packages. Is this necessary in NVEs? Networking Concepts • UDP: Lightweight protocol. • Differences with TCP: – Connectionless transmission. Information about the state of the connection is not kept. – Best-effort delivery. No guarantee of delivery or order of messages. – Each packet is auto-contained. Networking Concepts • UPD Advantages: – No overhead for connection oriented semantics. – Packages can be transmitted/received immediately, no need for queues. – It is possible to send information to groups of users (multicasting). Networking Concepts • Unicast – One computer sends information to only another one computer. Networking Concepts • Broadcast – One computer sends information to every computer in a subnet. Networking Concepts • Multicast – Only computers listening to a specific multicast group receive the message. Networking Concepts • Multicast: Similar to distribution of newspapers: – Subscribers explicitly request the newspaper. – Duplicate copies are not sent down the same distribution tree. – The publisher does not need to know the subscribers. Networking Concepts • Mulsticast is emerging as the recommended way to build large-scale NVEs. • BUT many routers do not handle multicast messages still. Networking Concepts TCP UDP IP Broadcasting IP Multicasting Small number of users Limited data requirements Typically client-server configuration Higher data requirements Used both in client-server and peer-to-peer configurations. Small peer-to-peer Net VEs with high data requirements and time sensitive delivery. Large peer-to-peer NetVEs, be careful with routers. Networking Concepts • Summary – Available networking infrastructure is a very important factor when designing a NetVE. – Networking-related decisions have a big impact on: • Complexity of implementation • Overall performance • Scalability Architectures • Client-Server Systems – logical architecture Server Client 1 Client 2 … Client n Architectures • Client-Server Systems – physical architecture with phone lines Server Phone Line Client 1 Phone Line Phone Line Client 2 … Client n Architectures • Client-Server Systems – physical architecture on a LAN Client 1 Client 2 … Server Client n Architectures • Client-Server Systems – The Server can become a bottleneck. – What are the advantages? The server can decide:: • • • • • Which clients should receive a message. What protocol to use with different clients. Sub-sample messages to slow users. Keep statistics. ... Architectures • Multiple-Server Architectures Client 1 Client 2 Server 1 Server 2 Client 1 Client 2 … Client n … Client n Architectures • Multiple-Server Architectures – Several servers have the following advantages: • System scales better. • Communication between clients attached to different servers takes longer. • Key issue: how to assign clients to servers? Architectures • Peer-to-peer NETWORK Client n Client 1 Client 2 Architectures • Peer-to-peer – “Network” will be: • Broadcast • One or multiple multicast groups. – In the case of multicast groups: • Area of Interest Management: assign different users to different multicast groups, based on some criteria. Architectures • How many participants can co-exist in a virtual world? • From a Network infrastructure point of view: – Infinite compute power at each node – Network does not saturate – Packet size: 144 bytes – PDUs per second: 5 - 30 Architectures [Dawson98] Technology Modem Speed (Kbps) Min # players Max # players 56 1 6 DSL 1500 39 163 T-1 1500 39 163 10BT 10,000 263 1085 100BT 100,000 2630 10851 Shared State • Main idea in a NVE: • Provide users with the illusion that every participant is seeing the same things and interacting with each other. • Things in the NVE change. • Without dynamic shared state, the illusion of sharing space and time breaks. Shared State • Definition of the problem - The consistency-throughput tradeoff – It is impossible to allow dynamic shared state to change frequently and guarantee that all host simultaneously access identical versions of that state. [Singha/Zyda 1999] Shared State • Absolute consistency: – Wait until everybody acknowledges before moving to a new position. – Considering latency and available bandwidth, this may imply slowing down too much. • Frequent updates: – Users send the updates and move on. – But: some users will receive the updates quickly and others wont. – Inconsistent views! Shared State • Centralized Repositories - File Repository – For pedagogical purpose: – One file per entity – Each update: • Open File • Read/Updat • Close File User 1 User 2 User n Shared State • Centralized Repositories - File Repository – Advantage: • absolute consistency. • Avoids concurrency problems. • Locks on objects easily implemented (when are locks necessary?) – Problems: • SLOW! Shared State • Centralized Repositories - Repository in Server Memory – Entry in memory per entity. – Users need to acquire a “lock”. User 1 User 2 User n Shared State • Centralized Repositories - Repository in Server Memory • Advantages – Fast! • Disadvantages – No persistency. – What happens if server crashes? Shared State • Distributed Repositories - Virtual Repository – Different clients manage different parts of the world. – Clients have a local cache of used objects. Shared State • Distributed Repositories - Virtual Repository • Advantages: – No bottleneck • Disadvantages: – Complex to program. Shared State • Frequent State Regeneration • Is absolute consistency always necessary? • When is it NOT necessary? Shared State • Frequent State Regeneration – For instance in a group flight simulator program: – A temporal inaccuracy in the position of another plane is not too serious. – Example: if updates are sent every 40 milliseconds, one lost package is almost imperceptible. Shared State • Frequent State Regeneration – Producer • “Broadcast” location of local objects • Either when they move or at a fixed rate. – Consumers (other participants) • Draw the scene using the locations in the local cache. • Update the local cache with the remote events. Shared State • Frequent State Regeneration – Producers and consumers are decoupled. • All interactions can be decoupled? – Independent movements: YES. – Tight-collaborative tasks: NO. Fast cycles Slow cycles Shared State • Situation 1: – A slow computer controls a plane moving in a straight line. – A participant in a fast computer perceives a “jumpy” movement. • Situation 2: – A fast computer floods a slow computer with messages of a car moving in a straight line. Shared State • Issue 1 (slow updates - Fast computer sees jumpiness) • Issue 2 (fast updates - Slow computer overwhelmed) Shared State • • • • Solution to alleviate both issues: DEAD RECKONING. Instead of sending updates at frame rate, Send parameters that describe the trajectory of the object (example: initial position and velocity) • Each participant displays the trajectory at its own rate. Shared State • Advantages: – Accommodates heterogeneity. – Bandwidth usage is reduced. • Cost: – More cycles to compute trajectory at each node. – Need to re-synchronize. Shared State • Dead Reckoning - Two parts: – Prediction • Predict where the object is based on the received parameters. – Convergence • Once an actual position is received: how to move the object from the predicted to the actual position. Shared State • Prediction Algorithms – Derivative Polynomials: – Order one: pos1 pos0 vel * t – Order two: 1 pos1 pos0 vel * t acc * t 2 2 – Part of DIS protocol. Shared State • Other prediction algorithms: – Circles – Spirals –… • Planes seem to follow many such curves. Shared State • Other strategy: – Send an update when the actual position is very different from the predicted position. – Advantage: the error in the system is kept under certain threshold. – See paper by C. Faistnauer in IEEE VR 2000. Shared State • Convergence Algorithms Predicted Position Convergence Position Current Position Last actual position Shared State • Convergence Algorithms • Snap: Just move the object to the most recent “actual” position. • Linear: Linearly interpolate to a point in the new predicted path (convergence point). • Cubic Spline: Create a path in the form of a cubic spline from the current position to the convergence point. Scalability and Performance • Improve the size and performance of a Net-VE by reducing network bandwidth and processor resources. • Less bandwidth requirements means more users in the network. • Less processor requirements means free cycles for other purposes (better graphics,…) and more heterogeneous participants. Scalability and Performance • Required resources are a function of: – Number of transmitted messages – Average number of destination hosts per messsage – Bandwidth required per message – Timeliness (minimal delays) – Processor cycles needed to process each message. Scalability and Performance • A reduction in any of these items is a gain. • BUT usually a reduction in one means a penalty in another. • Example: – Dead Reckoning reduces required bandwidth but more processor cycles are required. Scalability and Performance • Compression & Aggregation • Compression: – Aims at reducing bandwidth utilization by reducing the size of the messages. – Internal: encode in a more efficient manner. – External: Avoid retransmitting information that is identical to previous packages. Scalability and Performance • Compression – Example: – Send “Snapshots” at periodic rates. – Send updates to the snapshots. – Snapshots sent over reliable protocol. – Updates sent over unreliable protocol. Scalability and Performance • Packet Aggregation – Aims at reducing the number of packets by merging several packets together. – The saving comes from the headers (25 bytes for UDP and 40 bytes for TCP). – Which packets can be merged? • Packages from all entities managed by the node (at the client). • Packages from several clients (at the host). Scalability and Performance • Which packages to merge? – Based on a timeout (send a message every timeout units of time). – Based on quorum (send a message only when the quorum has been reached). – Combination of the previous two. Scalability and Performance • Aggregation Servers – Define servers that aggregate messages for sets of entities having common characteristics: • Virtual World Location. • Entity type. Scalability and Performance • General model: – Each node has an Aura, or area of influence. – Each node has a Nimbus, or a type of entities from which to receive data. Dest2 Aura Nimbus 2 Source Dest1 Nimbus 1 Scalability and Performance • An event is only send from the source to the destination if the source’s aura intersects the destination’s nimbus. • In Massive-1 unicast, peer-to-peer, unreliable protocols are used to send events from the sender to the destination. • This does not scale very well. Scalability and Performance • Multicast – Key question: how to map groups of users to multicast groups. Approaches: • Group-per-Entity • Group-per-Region Scalability and Performance • Group-per-Entity [Abrams/Watsen/Zyda98]. • There is a multicast group per entity. • Each node subscribes to multicast groups of entities in its nimbus. • A directory server is necessary to inform nodes about the position and multicast groups of other entities. • Problem: The number of multicast groups grows rapidly. Scalability and Performance • Group-per-region – The virtual world is partitioned into cells. Each cell is assigned a multicast group. – Users subscribe/unsubscribe from multicast groups as they travel. – As a user approaches a region, it needs to subscribe to the neighbor cells. Hexagonal cells are an advantage. Scalability and Performance • Level of Detail perception • Objects that are very far away will: – Appear smaller – Probably not be the focus of attention. • Therefore: – Can be updated less frequently. Scalability and Performance • Need for multiple channels. • Different channels have different “resolution” (update frequencies, spacial granularity, etc…) • Who handles the channels? – A server – Each client Research Topics • Multimedia – Incorporating audio/video live streams to NVEs – Questions: • What to display • Where to display it • A single rate or multiple rates? Research Topics - Multimedia • FreeWalk [Nakanishi96]. – Mapped a picture of the user in the front face of a pyramid (avatar). – Users can move freely and establish voice conversations. – The volume depends on the distance and mutual orientation. – A community server informs users about positions and orientation of other users. – Actual voice messages and pictures are exchanged among users. Research Topics • FreeWalk Research Topics - Multimedia • Awareness-Driven Video QOS [Reynard 98] – Three different QOS levels are established: • Porthole: 1 update every 5 minutes. At the top of the virtual office. • Glance: 1 frame/sec. At the front of the office. • Communications: 20 frames/sec. On top of user’s avatar. Research Topics - Multimedia • Awareness-Driven Video QOS Research Topics - Architectures • Hierarchy of Servers [Funkhouser95] – Compared Static Allocation of clients to servers vs. Dynamic Allocation (a server per room). – The number of server-to-server messages was reduced. Research Topics - Architectures Courtesy of T. Funkhouser http://www.cs.princeton.edu/funk/ring.html Research Topics - Architectures • Dynamic partitioning of space [Restrepo03] • Partition the space dynamically as the users move in the world. • Assign one partition to each server. • If a space becomes to crowded, split in a quadtree fashion. • Load among servers was better balanced. Research Topics - Architectures Courtesy if the Eafit University. Research Topics Heterogeneity • How to accommodate heterogeneous nodes in a NetVE [Trefftz03] • The system handles multiple variables: – Display Rate – Frequency of remote updates – Frequency of “video” updates • The user can specify her preferences. • The System Administrator can define minimum levels • The system “optimizes” the function and defines fidelity levels for each client. Research Topics - Heterogeneity • Proof-of-concept application: Research Topics - Heterogeneity • The “Switchboard” architecture Research Topics - Collaborative AR • Two or more users collaborating in a reality augmented with virtual objects. • See Studio3D by Vienna University. • What happens if the users are in different physical locations? Research Topics • What are the “big players” doing? – Mike Zyda (NPS): • Very successful army game. • Agents in VEs. • Defense-Entertainment cooperation for Training. – Greenhalgh/Benford (Nottingham) • Heterogeneity • Wireless – Steve Feiner • IEEE VR 2003: Merge of VR and wireless technologies. Research Topics • America´s Army From America´s Army´s home page. Commercial Systems • ActiveWorlds – Commercial system. – Free sessions for guests. – Used for: • Education • Social interaction of groups – Provide avatars and connected virtual locales – Communication: chat tool. Toy Application (cubes) • Demonstration • Will be downloadable from: • http://arcadia.eafit.edu.co/CGIMtutorial/ Toy Application • Packages - Client Application Toy Application • Packages - Server Application – Jus the Server Package. Toy Application - server Toy Application - send Toy Application - receive Toy Application - replicate Toy Application - Documentation • JavaDoc – Documentation of classes, methods and atributes. Created by Java. • UML – Poseidon UML • SourceCode – JBuilder Personal Edition project. Toy Application • Changes you can do: – Use UDP instead of TCP - Compare performance. – Add audio-video streaming (JMF) – Make it peer-to-peer - Compare performance. – Try the techniques for scalability defined in the tutorial. – Use a VRML loader to import prettier geometry. Toy Application – Test your own ideas! Conferences • IEEE Virtual Reality (formerly IEEE VRAIS) • ACM VRST (Virtual Reality Software and Technology). • ACM CVE (Collaborative Virtual Environments). (every two years). • ACM CSCW (Computer Supported Collaborative Work). • IASTED CGIM. Journals • PRESENCE. MIT Press. • In General networking or HumanComputer Interaction Journals have more and more articles on NVEs. Summary and Conclusions • • • • • “New” field. Technology is here. The need & interest are growing. Many areas to be explored. Any idea for your own research? I will be happy to discuss. • Enjoy! Thank you! Bibliography - BOOKS • [Singhal/Zyda 1999] Singhal, S. and Zyda, M. Networked Virtual Environments: Design and Implementation. AddisonWesley.1999. • [Churchill/Snowdon/Munro 2001]Churchill, E., Snowdon, D. and Munro, A. Collaborative Virtual Enironments. Digital Places and Spaces for Interaction. Springer Verlag. 2001. Bibliography - Papers and Articles • [Abrams/Wasten/Zyda98] Abrams, H., Waksen, K. and Zyda M. Three tiered interest management for largescale virtual environments. In Proceedings of VRST 1998. Taipei, Taiwan, November 1998. • [Dawson98] Dawson, F. XDSL market blooming. Interactive Week 5(39):28-29. October, 1998. • [Funkhouser96] Funkhouser, T. Network Topologies for Scalable Multi-User Environments. In Proceedings of IEEE VRAIS. 1996. Bibliography - Papers and Articles • [Greenhalgh/Benford95] Greenhalgh, C., Benford, S. Massive: a Virtual Reality System for Teleconferencing. ACM Transaction on Computer Human Interfaces. 2(3):239-261. • [Hagsand96] Hagsand, O. Interactive multiuser VEs in the DIVE system. IEEE Multimedia 3(1):30-39, 1996. • [IEEE93] IEEE standard for information technology / protocols for distributed simulation applications. IEEE Standard 1278-1993. IEEE Computer Society, 1993. • [Macedonia95] Macedonia, M., Zyda, M., Pratt, D. and Barham, P. Exploiting reality with multicast groups: A network architecture for large-scale virtual environments. In Proceedings of IEEE VRAIS. 1995. Bibliography - Papers and Articles • • [Miller/Thorpe95] Miler, D., Thorpe J.A. SIMNET: The advent of Simulator Networking. In Proceedings of the IEEE 83(8):1114-1123, August 1995. [Nakanishi96] Nakanishi, H., Yoshida, C., Nishimura, T. and Ishida, T. FreeWalk: Supporting Casual Meetings in a Network. In Proceedings of ACM CSCW. 1996. • [Restrepo03] Restrepo A., Montoya, A. and Trefftz H. “Dynamic Server Allocation in Virtual Environments using QuadTrees for Space Partitioning”. In Proceedings of IASTED Computer Science and Technology, Cancun, Mexico, 2003. • [Reynard98] Reynard, G., Benford, S., Greenhalgh, C. & Heath, C. Awareness Driven Video Quality of Service in Collaborative Virtual Environments. In Proceedings of ACM CHI 98. Bibliography - Papers and Articles • [Shaw/Green93] Shaw, C. Green, M. The MR toolkit peers package and experiment. In Proceedings of the 1993 IEEE Virtual RealityAnnual International Symposium, 463/469. Seattle, September 1993. • [Trefftz03] Trefftz, H., Marsic, I. and Zyda M. Handling Heterogeneity in Networked Virtual Environments. In Presence 12(1) 37. February 2003. Bibliography - Web Pages • [DIVE] DIVE home page http://www.sics.se/dce/dive/ • [idSoftware] idSoftware home page http://www.idsoftware.com/ • [MASSIVE-3] MASSIVE home page http://www.crg.cs.nott.ac.uk/research/systems/MASSIVE -3/ • [NPSNET] NPSNET home page http://www.npsnet.org/~npsnet/ • Toy application: • http://arcadia.eafit.edu.co/CGIMtutorial/