Scalable
Peer-to-Peer Virtual Environments
Shun-Yun Hu (胡舜元)
(syhu@csie.ncu.edu.tw)
CSIE, National Central University, Taiwan
2008/05/08
1
2
4
Massively Multiplayer Online Games

MMOGs are growing quickly




Multi-billion dollar industry
10 million subscribers for World of Warcraft
600,000 concurrent users, but 3,000 per world
Can we scale to millions in the same world?
Imagine you start with a globe
Zoom in…
To Phoenix..
and Hyatt..
Right now it’s flat…
But in the near future…
Virtual Environments (VEs):
A Shared Space
Issues for Creating VEs






Consistency
Interactivity
Security
multiplayer
Scalability
Persistency
Reliability
massively multiplayer
Model for virtual worlds



Many nodes on a 2D plane
Message exchange with those within Area of Interest (AOI)
How does each node receive the relevant messages?
Area of Interest
15
A simple solution (point-to-point)
Source: [Funkhouser95]
But…too many irrelevant messages
N * (N-1) connections ≈ O(N2)  Not scalable!
16
A better solution (client-server)
Source: [Funkhouser95]
Message filtering at server to reduce traffic
N connections = O(N)  server is bottleneck
17
Current solution (server-cluster)
Source: [Funkhouser95]
Still limited by servers.
Expensive to deploy & maintain.
18
The Problem

Client-server: resources limited by provisioning
Resource limit
[Funkhouser95]
The Solution

Peer-to-Peer: resources grow with demand
Resource limit
[Keller & Simon 2003]
Voronoi-based Overlay Network
(VON)
Design Goals

Observation:



for virtual environment applications, the contents we want
are messages from AOI neighbors
Content discovery is a neighbor discovery problem
Specific goals:


Scalable
 Limit per-node message traffic
Responsive  Direct connection with AOI neighbors
If you talk with your AOI Neighbors directly…
But how to discover new neighbors?
23
Voronoi Diagram


2D Plane partitioned into regions by sites, each
region contains all the points closest to its site
Can be used to find k-nearest neighbor easily
Neighbors
Region
Site
24
Design Concepts
Use Voronoi to solve the neighbor discovery problem




Identify enclosing and boundary neighbors
Each node constructs a Voronoi of its neighbors
Enclosing neighbors are minimally maintained
Mutual collaboration in neighbor discovery
Circle
Area of Interest (AOI)
White
self
Yellow
enclosing neighbor (E.N.)
L. Blue
boundary neighbor (B.N.)
Pink
E.N. & B.N.
Green
normal AOI neighbor
L. Green unknown neighbor
25
Procedure (JOIN)
1) Joining node sends coordinates to any existing node
Join request is forwarded to acceptor
2) Acceptor sends back its own neighbor list
joining node connects with other nodes on the list
Acceptor’s region
Joining node
26
Procedure (MOVE)
1) Positions sent to all neighbors, mark messages to B.N.
B.N. checks for overlaps between mover’s AOI and its E.N.
2) Connect to new nodes upon notification by B.N.
Disconnect any non-overlapped neighbor
Non-overlapped
neighbors
Boundary
neighbors
New
neighbors
27
Procedure (LEAVE)
1) Simply disconnect
2) Others then update their Voronoi
new B.N. is discovered via existing B.N.
Leaving node
(also a B.N.)
New boundary neighbor
28
Demonstration
Simulation demo



Random movements (100 nodes, 1200x700 world)
Local vs. global view
Dynamic AOI adjustment
29
Simulation Method

C++ implementation of VON (open source VAST library)

World size:
Trials from
Connection limit:
3000 time-steps



1200 x 1200
200 – 2000 nodes
20
(AOI: 100)
(~ 300 simulated seconds, assuming 10 updates/seconds)

Behavior model



Random movement:
Constant velocity:
Movement duration:
random destination
5 units/step
random (until destination is reached)
Scalability: Avg. Transmission / sec
30
basic
dAOI
basic (fixed density after 1000 nodes)
dAOI (fixed density after 1000 nodes)
Size (kb /
25
20
15
10
5
0
0
400
800
1200
Number of Nodes
1600
2000
Scalability: Max. Transmission / sec
70
basic
dAOI
basic (fixed density after 1000 nodes)
dAOI (fixed density after 1000 nodes)
Size (kb /
60
50
40
30
20
10
0
0
400
800
1200
Number of Nodes
1600
2000
Scalability: Avg. Neighbor Size
90
connected neighbors (basic)
80
AOI neighbors (basic)
connected neighbors (dAOI)
Neighbor Size
70
AOI neighbors (dAOI)
60
50
40
30
20
10
0
0
400
800
1200
Number of Nodes
1600
2000
Reliability: Effects of Packet Loss
100
90
Units
80
70
Topology Consistency (%)
60
Recovery Steps
50
40
30
20
10
0
0%
20%
40%
60%
Loss Rate
80%
100%
Voronoi State Management
(VSM)
Voronoi State Management

VON deals only with positions, but we want to
manage states stored in spatial objects (with x, y).




Let game states be managed by all clients
Each client has two roles: peers & arbitrators
i.e. Voronoi partitioning
Three problems:



O(n2) connections at hotspots
Some cells have large sizes
Constant ownership transfer
36
VSM: solution ideas



Connection overload
Large cell-size
Constant transfers
→ Aggregators clustering
→ Virtual peers incremental transfer
→ Explicit ownership transfer
37
VSM: Consistency control

Managing arbitrator
receives and
processes events

Events are forwarded
if necessary

Updates sent to
affected arbitrators,
then peers
38
VSM: Load balancing


Traditional:
VSM:


Overload:
Underload:
high-capacity nodes first, then adjust
low-capacity nodes first, then cluster
ask for aggregator, submit control
disintegrate, release control
39
VSM: Load balancing (2)


Sphere of control adjustable
More than one aggregator → choose nearest
40
VSM: Fault tolerance

Regular arbitrator:



Pick backup arbitrator, backup states
Backup transfers ownership to enclosing arbitrators
Aggregators:



Pick backup aggregators
Take over original if failed
Choose new backup
41
Peer-to-Peer 3D Streaming
Background

MMOGs today need CD installation
(too slow!)

But 3D data is huge

Content streaming is needed

80% - 90% content is 3D (e.g., 3D streaming)
43
3D streaming

Object streaming



base
refinements
Scene streaming


[Hoppe 1996]
[Teler & Lischinski 2001]
multiple objects
object selection & prioritization
44
3D streaming vs. media streaming

Video / audio media streaming is very matured

User access patterns are different for 3D content



Highly interactive
Behaviour-dependent


Latency-sensitive
Non-sequential
Analogy

Constant & frequent switching of multiple channels
45
Benefits of peer-to-peer

Scalable


Affordable


Growing amount of total resources
Commodity PCs
Feasible


Better client hardware (CPU, broadband networks)
Availability of user-hosted machines
46
Challenges for P2P 3D streaming

Appropriate peer grouping



Dynamic group management



Matching interests / needs
Matching capabilities
Interest groups are dynamic
Real-time constraints
(non-sequential)
(latency-sensitive)
Minimal server involvement


Visibility determination
Request prioritization
(object selection)
(piece selection)
Observation


Limited & predictable area of interest (AOI)
Overlapped visibility = shared content
48
overlapped visibility = shared content
49
Download content from AOI neighbors
star:
circle:
self
AOI
triangles:
neighbors
rectangles: objects
50
Neighbor discovery via VON
Voronoi diagrams identify boundary neighbors for
neighbor discovery
Non-overlapped
neighbors
Boundary
neighbors
New
neighbors
[Hu et al. 06]
51
Prototype experiment



Progressive models in a scene
(by NTU)
Peer-to-peer AOI neighbor requests
(by NCU)
Found matching client upload / download
52
Simulation setup

Environment



Objects



1000x1000 world, 100ms / step, 3000 steps
client: 1 Mbps / 256 Kbps, server: 10 Mbps (both)
Random object placement (500 objects)
Object size based on prototype
User behavior

Random & clustering movement (1.5 * ln(n) hotspots)
53
Server bandwidth usage
54
Client bandwidth usage
55
Fill ratio
56
Base latency
57
Impacts of P2P VEs…




No server as bottleneck
Commodity hardware
2D web  3D web
Earth-scale virtual worlds
(millions/billions of people)
 scalable
 affordable
Q&A
VON: A Scalable Peer-to-Peer Network for Virtual Environments
IEEE Network, vol. 20, no. 4, Jul./Aug. 2006
FLoD: A Framework for Peer-to-Peer 3D Streaming
IEEE INFOCOM, Apr. 2008
Thank you!
http://vast.sourceforge.net
http://ascend.sourceforge.net
59
Prototype experiment

Data


Setup



3D scene converted from a game demo
100 Mbps LAN
10 participants, 48 logins captured in 40 min.
Results


Found matching client upload & download
Avg. server request ratio (SRR): 36%
60
The flow of FLoD



Prefetching:
not considered
Prioritization:
visual importance & view-dependency
Peer & piece selection: query-response, random peer,
sequential piece
61
Client bandwidth usage
(cluster)
63
Effect of user density
64
Effect of upload bandwidth
65
Cache utilization
66
Collaborations

With Publications






Guan-Ming Liao
Dr. Jui-Fa Chen, Tsu-Han Chen
Dr. Bing-Yu Chen, Ting-Hao Huang
Dr. Jehn-Ruey Jiang
3 graduated & 3 current masters
(Actainment Co., Taiwan)
(TKU, Taiwan)
(NTU, Taiwan)
(NCU, Taiwan)
(NCU, Taiwan)
On-going




Dr. Pedro Morillo Tena
Dr. Wei Tsang Ooi
Dr. Maha Abdallah
Dr. Gregor Schiele
VON
FLoD
Heaven
MMVE
(Universitat de València, Spain)
(NUS, Sinagpore)
(Université Paris 6, France)
(Univ. of Mannheim, Germany)
67
Voronoi-based Overlay Network
Voronoi diagrams identify boundary neighbors for
neighbor discovery
Non-overlapped
neighbors
Boundary
neighbors
New
neighbors
VON: A Scalable Peer-to-Peer Network for Virtual Environments
IEEE Network, vol. 20, no. 4, Jul./Aug. 2006
68
FLoD (with CM Lab, CSIE, NTU)
Drastic server bandwidth reduction
First P2P 3D streaming prototype
FLoD: A Framework for Peer-to-Peer 3D Streaming
IEEE INFOCOM, Apr. 2008
69