Abstract
The Research Proposal is a part of the project: Middleware Services for Management of Shared State in Large-Scale Distributed Interactive
Applications (MiSMoSS). MiSMoSS is funded by the Research Council of Norway and is Project No. 15992/431. The project is expected to lead to three PhD theses supervised by faculty members Carsten
Griwodz, Paal Halvorsen and Ellen Munthe-Kaas in the Networks and
Distributed Systems Group. The Research Proposal recognizes the problem of object location management and will search for fault tolerance mechanisms along with Quality of Service management to enhance the performance in Large-Scale Networked Virtual Environments. The
Research Proposal is meant to lead to the formal PhD Thesis Proposal by the candidate.
Networked Virtual Environments (NVE) allow multiple users to interact in real-time even though users may be located around the world.
This environment is characterized by nice 3D graphics and sounds all there to make the experience closer to a real life scenario. We have seen in the last few decades an increasing interest and popularity among professional researchers at universities and laboratories, commercial companies and groups to develop NVEs. The military is interested in
NVEs as combat training systems and research is vital such that they can grow in size, scope and complexity.

NVEs have been applied succesfully in the gaming industry and have led to the development of Interactive Applications spread out in large-scale Networks. A person who is moving in the real world, which is the space that all of us share, is constantly experiencing his perceptional limits. He can not look around corners, through doors, into pockets, listen in on conversations across a busy hall, and so on. Many cooperative applications try to recreate this reality in the digital domain, for example in distributed virtual environments, virtual meeting rooms and museums, or in games. In all meeting situations, options for private conversations in smaller groups are appreciated, as they are in reality. Thus, the core of these distributed applications is that all participants share one space, but not every participant receives all details of the shared space. Instead, he may receive only a subset, a quality-degraded version of the information, or a combination of these limits. The amount of information that is offered and the way in which it is offered are application-specific.
We observe that the problem of managing a shared space occurs frequently. Our project is thus pursuing the goal of extracting this facet of distributed applications, and to offer it as a generic service.
We start the project with the following hypothesis:
• Management of shared state information can be factored out of applications and provided as a middleware service that supports an application specific presentation of the state information to the users.
• Thus, we can simplify application development by masking out issues such as the heterogeneity of networks and resource availability when handling shared space.
• By combining state management in middleware with appropriate resource management, monitoring and reservation, we can increase the application’s scalability and increase its performance.
By this, we want to support developers in making their distributed interactive applications reliable, reactive and scalable. We have identified several basic mechanisms that can be employed by our middleware, which have already been employed separately in dedicated applications.
We will consider the combined use of these mechanisms, and develop additional ones during the course of the project. During the development of our middleware, we will analyze application requirements for several distributed application classes, and draft the selection of appropriate basic mechanisms and the policies for their use. For one large distributed application, our work will be evaluated in field tests.

The goal for this project is to create a middleware for managing shared space, which will provide developers of distributed interactive applications with means to make their applications scalable, increase their performance, and decrease their resource consumption. The use of shared space is central to our approach, i.e. we apply a programming model in which distributed applications share one distributed state.
We have divided the area of research into three parts: data reduction, latency hiding and replication [6]. We will combine them into a middleware in such a way that they become usable for application developers.
It is not immediately clear how this can be achieved if the appropriate use depends on application semantics. We have decided to approach this in a manner that is analogous to classical middleware logic:
• Application programmers concentrate on application development
• Tools are used in the development process that remind of an IDL compiler and that add functionality which is part of and interacts with the middleware.
At development time we consider two independent tasks, the development of data types and the development of the application itself. In data type development, data types are specified that appear to application development in a fashion that is similar to standard data types.
They can be instantiated as distributed objects that are controlled by the middleware, which applies mechanisms such as replication, data reduction and latency hiding in compliance with the developer-selected policies. In application development, applications are written in a manner similar to non-distributed applications. However, transparent state updates must be taken into account, i.e., displays as well as derived variables of native data types must be either avoided entirely or refreshed in an appropriate manner.
The run-time application is generated from these specifications, and the specific data types are translated into distribution code that integrates with the distribution middleware. The middleware takes care of the transparent communication needs and update notification.
Based on the data type specifications, applicable policies and current resource conditions of hosts and networks, it will manage replication, dereplication, frequency of updates, and preference of updating some objects over others.
The fundamental goal of an NVE is to give users the illusion that they are all seeing the same things and interacting with each other within

that virtual space [14]. In other words, NVEs want to present the virtual world such that every player is experiencing the same environment at all times. In theory this is close to if not impossible, but in practice the human eye and brain can only process information at a certain speed making it possible to minimize the latency such that it is hidden from the human perspective.
This section introduces some mechanisms and concepts for this research proposal. The mechanisms are replication, area of interest, and multicasting. The concepts are fault tolerance, Quality of Service, latency minimazation, and interest management in NVEs. These are all areas of research that will be addressed further in the formal PhD
Thesis Proposal.
The overall goals of the project is to achieve a degree of fault tolerance and Quality of Service. By doing this we also want to minimize the latency experienced by the clients. In the next subsections these terms are introduced.
A characteristic feature of distributed systems that distinguishes them from single-machine systems is the notion of partial failure [17]. This failure may affect the proper operation of other components, while at the same time leaving yet other components totally unaffected. A fault tolerant system is strongly related to what are called dependable systems. Dependability is a term that covers a number of useful requirements for distributed systems including the following [17]:
• Availability
• Reliability
• Safety
• Maintainability
There are many fault tolerance techniques that are applicable to distributed systems, and it will be a challenge finding techniques and mechanisms suitable for large-scale NVEs. Initially, replication is considered as a method of achieving better fault tolerance.
In computer networks we generally have applications with different requirements to the performance in terms of end-to-end latency, available bandwidth, jitter etc [5]. Quality of Service is a collective term

that is focused on giving applications certain degrees of network service guarantees.
The Internet does not and cannot give end-to-end QoS guarantees as of October 2004. This does not mean that having QoS mechanisms in a distributed system using the Internet is useless. On the contrary, it has been shown that having soft QoS guarantees can improve the service experienced by the end user [18]. NVEs can be spread over a geographically large area having considerable amount of clients and servers involved. The communication between the clients and servers must be efficient to enhance the performance [10].
Whenever there is need to send a message between two points there is a certain delay or latency. In computer networks latency is described as the time it takes from sending a packet until it is received. In NVEs the latency is one of the biggest challenges a developer has to face, because latency directly impacts the realism of the NVE experience
[12].
In computer terminology bandwidth is referred to as the amount of data a given transport medium can deliver per time unit. In the last decade the average bandwidth available has increased considerably, therefore lessening the importance of bandwidth as a source of performance problems. The focus is shifting towards latency minimization as the main research area within NVEs.
NVEs often consists of users spread out over a geographically large area only bounded thus far within the vicinity of the earth. It is agreed that the speed of light is as fast as anything can move, hence setting an upper bound on the speed of data transfers. In addition to this limitation, there is also a considerable latency in the physical transport medium. The sources of latency in an NVE can typically be divided into three categories [16]:
• Latency in the physical transport medium
• Latency in the network intersections (routers)
• Latency in the endpoint computers
To illuminate the readers with an example let us say we have the entire virtual environment in a centralized server which means we can only respond to the clients placed geographically in the outer regions of the network at a speed bounded by the three latency categories just mentioned. It is therefore important to distribute functions and data of distributed applications geographically, such that all the participants of the network have close to the same experience of latency.

Interest management in NVEs consists of resource optimization techniques that aims to reduce bandwidth consumption by reducing the average number of hosts that receive each message [14]. When resource optimization techniques are applied to an NVE, it is assumed that the virtual environment contains such an amount of information that it simply will not scale otherwise. A developer should strive towards intelligent information dissemination throughout the NVE. In broad terms this means leaving out the participants that will not gain from receiving specific data messages. Interest management in NVEs is generally divided into three categories [15]:
• Area-of-Interest Management
• Multicast Routing
• Subscription-Based Aggregation
Multicast Routing and area-of-interst management are within the scope of the proposed work, and is presented in the next section.
The mechanisms thus far conceived to take part in the project will be replication, area-of-interest management and multicast routing. They are all introduced in this section. It is certainly possible that further mechanisms will be found and added later in the project.
Replication is generally used to improve the reliability and the performance of a system [17]. The reliability is increased since the system avoids a single point of failure by having replicas distributed, plus it is also possible to avoid corrupted data using this technique. The performance is likewise improved when on average clients are closer to a replica server than the main server. For example, by placing a copy of some data in the proximity of the processes using it, the time to access the data decreases. Although the advantages are nice it comes with a price, which most importantly is inconsistency. Whenever a copy of the data is modified the rest of the copies need to be updated. Thus it follows that an inconsistency measurement function must include the number of replicas at a given time in the system. However, there is typically a relaxed consistency model applied to NVEs. By having replication it is also possible that the global load on the network increases keeping all the replicas up-to-date.

Some applications require that widely-separated processes work together in groups, for example a group of processes implementing a distributed database system [16]. Multicast routing is ideally used when groups are large but small compared to the entire network. By applying multicasting in these scenarios the problems associated with broadcasting and unicasting messages in large groups are avoided. Broadcasting sends out messages to all the nodes in the networks regardless to whether they are interested or not. Unicasting requires the source to send one message per receiver and that is expensive when the groups grow large.
Multicast is a hybrid between unicast and broadcast in that it allows senders to pass single copies of data which are then replicated and routed to hosts that signal that they want to receive it. This eliminates the need to send redundant traffic over the network and also tends to eliminate CPU load on systems that are not using multicast, yielding significant enhancements to the efficiency for both sender and network.
Multicasting requires group management [17], such that there must be functionality to create and destroy groups, and allowing processes to join and leave groups.
Area of interest management are mechanisms that filter data such that clients ideally do not receive and process data they do not need in order to correctly proceed in the virtual environment.
Area of interest can be categorized into aura, focus and nimbus for each entity. Aura is the sphere of influence, nimbus is the sphere of interest of the observed object, and focus is the sphere of interest of the observer. In an area-of-interest filtering scheme the nodes transmit their state changes to subscription managers [15]. These managers also receive subscriptions that express nodes’ information interests (foci).
The managers will only transmit interesting data to the subscriber nodes.
By disseminating the information about the different players in an intelligent manner using an area of interest approach the number of updates required is reduced.
Latency minimization, Fault Tolerance and Quality of Service are areas of research tightly linked with the research for the candidate. A big part of the project will consist of thoroughly testing mechanisms to see how they work together and in different scenarios.

As previously mentioned the project will be divided into three parts: latency hiding, data aggregation and replication. It is expected that the three project parts will interoperate in varying degree throughout the project period, in particular data reduction and replication.
The research proposal presented here introduces the proposed goals that shall lead to the PhD Thesis Proposal for the candidate.
Replication in large-scale NVEs is vital, but using replication will add complications in handling the inconsistensies that occur in the replicated data. Avoiding temporary inconsistencies is hard in NVEs and may cause performance degradation, it is therefore important to relax the consistency demands and rather focus on how the human experience is [11]. The goal of any multimedia application should always be to satisfy the users first.
Our middleware will replicate objects to satisfy application-specific constraints. All nodes in the system may be holders of replica, and the number and placement of replica will vary dynamically. Objects in distributed applications will typically be related to each other semantically, and these relations will impose constraints on the sensibility of replicating them separately. Such relations may change dynamically while the distributed application runs. We will develop a means of specifying them at application development time in a formal manner, so that the middleware can perform replication of groups of objects transparently.
We will consider the resources of heterogeneous end systems (such as PCs, PDAs, or sensors), the regional load and the communication overhead to decide the number of update events that can be handled by the nodes that hold reference copies, and restrict the number of replicas to this. Ideas from research approaches in P2P and grid systems will be considered to determine the number and placement of the replica. A concentration of accesses to sets of objects must be expected. This may be due to denial-of-service (DoS) attacks, flash crowd (slashdot) effects, or simply due to application semantics that encourage collaboration.
Since the shared data space should be accessible and durable, we will consider backup copies that are located outside hot areas of the system.
Persistence of objects is not planned as a native feature, but rather application function.
Area-of-interest filters provide NVEs with intelligent information dissemination through the network. In the project we plan to implement and test existing approaches [9] in different scenarios. The results we

get will function as guidelines for developers when we add the algorithms to our middleware.
In addition to existing ideas we also want to design and implement our own algorithms and see how they compare to existing ones. One of the ideas we are considering for our area of interest algorithms is a mobility prediction of players where we record client data throughout its session and from that perform a prediction that aids an area of interest filter in its decision making.
Multicast Routing is a network protocol technique that realizes the area of interest approach explained previously. Multicast routing is an efficient network dissemination protocol compared to unicast and broadcast. Our goal is to implement and test different multicast routing algorithms and compare them to each other. We assume that
IP-Multicast is not available but that we will apply application layer multicast. The multicast algorithms currently being considered are the center based [3] and the Steiner-tree [4] multicast.
The center based multicast algorithm bases the root selection on which node is closest to the center of the tree in terms of latency. This is a valuable property in large-scale NVEs where we potentially can have huge differences in latency based on which node functions as the root.
Steiner-tree multicast is optimal with respect to cost, which is latency in our case. Constructing Steiner-trees is an NP-complete problem but there are algorithms that approaches the problem by optimizing greedy minimum spanning tree algorithms [8].
A multicast routing scheme needs group management handling, therefore one of the most important goals is to create a protocol that can handle various types of failures that might occur in diverse largescale NVEs. The creation of the multicast trees should also be QoS aware not only in terms of latency [13] but also stability measures of the clients connected to the NVEs. This stability measure can be as easy as timing how long the clients are logged on at a time, whether they disconnect frequently and based on the current time predict if it is likely that a client will disconnect soon.
To evaluate and demonstrate our ideas, we intend to apply our results to a massive multiplayer role playing game based on existing open source development, such as Vegastrike [7], Parsec [2] or BzFlag [1].
Other applications that can exploit all of these abilities are remote control, control systems, distributed simulation, and financial and stock

market. Applications that can exploit dynamic replication but only a subset of the other functions are distributed file systems, code repositories, surveillance applications, active sensor networks, web services, rescue operations, distributed business applications, and catastrophe resistant systems.
A distributed, interactive computer game has a single environment that is concurrently manipulated by all participants. Our approach allows us to focus on the distribution system middleware, while the goals of the application, its storyline and its user interface are developed externally. We can expect a large number of users without much solicitation, and deployment is cheap. We can also expect to receive a large amount of user feedback and debugging support from such a non-critical application. Users of such an application use widely different network connections, hardware, and access it from world-wide distributed locations.
The research will add replication, area-of-interest filters and multicast routing schemes to our middleware. We should give the developer recommendations to what schemes to use together and in which scenarios.
The recommendations must be based on simulation results. The presentation of the results should be in a format that makes it possible for a game developer to make informed design decisions fairly quickly.
Transparency should be a goal but generic transparency is not a goal for all applications. For NVEs having transparency is a goal but only to a certain degree [19]. Therefore we must give a game developer design time choises through our middleware that enables a degree of control without unnecessary micromanagement.
A PhD Project spans several years where the reading gradually turns into testing and writing, more details are however required to achieve a successful project. The proposed timeline presents an overview of how the PhD Project for the candidate will proceed, it will function as a reference point for both the candidate and the advisors. In Table
1 different work categories are described and Figure 1 illustrates the workload for each work category.

Workload (Category)
Work Category 1 − Literature Survey
Work Category 2 − Experiments analysis and Design
Work Category 3 − Experimentation
Work Category 4 − Data Analysis
Work Category 5 − Writeup
1 2 3 TIME (years)

The project will implement and test existing schemes within replication, fault tolerance and Quality of Service and use these in varying scenarios to make it possible for an application designer to choose the mechanisms from a middleware most fitting to the interactive application. The middleware will also consist of algorithms designed by the candidate in collaboration with the teammembers that are specially designed for NVEs and add them to the proposed middleware.
The PhD Research Proposal gives an outline to the work that the candidate will do until the formal PhD Thesis Proposal is written. The algorithms that will be the result of the PhD Thesis will be included in the middleware which is the final product from the MiSMoSS project.
[1] Bzflag.
[2] Parsec, June 2003.
[3] M. Donahoo, K. Calvert, and E. Zegura. Center selection and migration for wide-area multicast routing. Technical report, 1997.
[4] Jr. F.C. Harris.
Steiner Minimal Trees: An Introduction, Parallel
Computation, and Future Work . Kluwer Academic Publishers,
2002.
[5] S. Froelund and J. Koistinen. Quality of service specification in distributed object systems.
Distributed Systems Engineering Journal , 5:179–202, 1998.
[6] C. Griwodz, P. Halvorsen, and Ellen Munthe-Kaas. (mismoss), middleware services for management of shared state in large-scale distributed interactive applications, nfr project no. 159992/431.
Technical report, January 2004.
[7] D. Horn. Vegastrike, June 2003.
[8] Hougardy and Promel. A 1.598 approximation algorithm for the steiner problem in graphs. In SODA: ACM-SIAM Symposium on Discrete Algorithms (A Conference on Theoretical and Experimental Analysis of Discrete Algorithms) , 1999.
[9] Katherine L. Morse.
Interest management in large-scale distributed simulations. Technical Report ICS-TR-96-27, 1996.
[10] K. Nahrstecht and R. Steinmetz. Resource management in multimedia networked systems.
IEEE Computer , 28:52–63, 1995.
[11] Zink On and Griwodz. Replication for a distributed multimedia systems.
International Conference on Parallel and Distributed
Systems (ICPADS) , June 2001.

[12] L. Pantel and L. Wolf. On the impact of delay on real-time multiplayer games.
Proceedings of the International Workshop on Network and Operating System Support for Digital Audio and Video
(NOSSDAV) , 2002.
[13] Mehrdad Parsa, Qing Zhu, and J. J. Garcia-Luna-Aceves. An iterative algorithm for delay-constrained minimum-cost multicasting.
IEEE/ACM Transactions on Networking , 6(4):461–474, 1998.
[14] S. Singhal and M. Zyda.
Networked Virtual Environments, Design and Implementation . ACM Press, SIGGRAPH Series - New York,
1999.
[15] J. Smed, T. Kaukorenta, and H. Hakonen. A review on networking and multiplayer computer games. Technical Report 454, April
2002.
[16] A. S. Tanenbaum.
Computer Networks 4th Edition . Prentice Hall
PTR - Upper Saddle River, New Jersey, 2003.
[17] A. S. Tanenbaum and M. van Steen.
Distributed Systems, Principles and Paradigmes . Prentice Hall, Inc - Upper Saddle River,
New Jersey, 2002.
[18] K-H Vik and S. Medidi. QoS aware Source Initiated Ad-hoc Routing QuaSAR.
1st IEEE Communications Society Conference on
Sensor and Ad Hoc Communications and Networks , October 2004.
[19] W. Vogels, R. van Renesse, and K. Birman. Six misconceptions about reliable distributed computing. Technical report, 1999.

Paal Halvorsen
Main Advisor
Carsten Griwodz
Advisor
Knut-Helge Vik
Candidate