Seamless Code and Content Update
in Connected Videogames
Thomas Bousquet
Georgia Tech
tbousquet@gatech.edu
Edouard Kieffer
Georgia Tech
ekieffer@gatech.edu
April 30, 2008
Abstract
With the advent of high speed Internet in the late 1990s, the videogame
industry on the PC platform has begun relying strongly on patches delivered after the official release of the game to fix bugs and make more
content available to their customers. Nowadays, with cable or ADSL
connections available virtually everywhere, and the release of “next-gen”
consoles such as the XBox 360, the Playstation 3 or the Wii, all boasting
Internet connectivity, the distribution of those patches have been largely
eased. However, to install the new update, a player must quit the game
and wait for the patch to be applied. In this paper, we present a solution to this problem, the “Dynacode” application. Built on a back-end, a
front-end, and a flexible client API, our solution enables seamless update
of both code and content in an application.
Contents
1 Introduction
2
2 Related Work
2
3 System Overview
3.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
3
5
4 Back-End
4.1 Technologies and Architecture . . . . . . . . . . . . . . . . . . . .
4.2 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
5
5
5 Front-End
5.1 Technologies and Architecture . . . . . . . . . . . . . . . . . . . .
5.2 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
6
8
1
6 Client API
6.1 Technologies and Architecture . . . .
6.2 Dynamic Classes . . . . . . . . . . .
6.2.1 Explicit Dynamic Classes . .
6.2.2 Implicit Dynamic Classes . .
6.2.3 Dynamic Classes Migration .
6.2.4 Dynamic Classes Evaluation .
6.3 Dynamic Resources . . . . . . . . . .
6.3.1 Architecture . . . . . . . . .
6.3.2 Process Overview . . . . . . .
6.4 Challenges . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
9
10
10
11
11
12
12
12
13
14
7 Sample Implementation
14
8 Limitations
14
9 Conclusion
16
1
Introduction
Videogames have become a prominent media over the years, becoming in 2007
the second major source of entertainment. The last evolution of the media as
a whole are multiplayer online games and solo connected games. Both involve
the presence of a server to deliver content and code updates. In the first case,
this server is required to simply play, while in the second case, being able to
connect to the server will ensure a player has the latest updates installed when
he begins playing. In terms of development cycle, this shift in use created new
opportunities: the developers could now react to users’ feedback in matters of
hours or days. but they also had the responsibility of handling hundreds of
thousand users. The main challenge then lies in the distribution of the updates
in code and content.
In this paper, we will first present related work from the videogame industry
in section 2, before giving a brief overview of our solution in section 3. In secions
4 to 6, we will go into more details about several components of our system,
namely the back-end, the front-end and the client API. We will then present in
section 7 our sample integration with aircarrier, a 3D videogame before exposing
the limitations of our solution in section 8. We will finally conclude in section
9.
2
Related Work
Three major applications appeared in this domain in the past two years:
• World of Warcraft by Blizzard is the most popular massively online role
playing game, it gathers millions of players 24/7, most of them shaping
the universe of the game on a daily basis
2
• Steam, an online distribution service by Valve which automatically delivers
updates or additional content for a large library of online and offline games.
It is considered the best of its kind as it also integrates a social network
and advanced system such as matchmaking and achievements.
• Sam & Max, a digital distributed game series which witnesses a new
episode every month over a period of six months.
These various applications highlight three advantages of connected games:
uninterrupted service, user impacted worlds and shortest, highly dynamic development cycles of incremental game. On the whole it helps create new kind
of games, brought to the end user in new fashions.
The main challenge behind these opportunities is to deliver new content
(both in terms of software and of assets) in a way that is both tolerable to the
user but also noticeable, as new content usually means new gameplay possibilities. The current paradigm is to check for new content at startup (or in the
case of Steam, at regular interval of times), retrieve it on the client machine and
apply it when the game restarts. When it comes to art assets integration, some
game such as “Guild Wars” adopts a more elegant approach: the game world
being separated in zones, it downloads and updates the content when entering
a new zone. The other tour de force of Guild Wars is to be able to maintain all
the art content in one 1.5GB file that is never decompressed. However, updating
the software is still a stop-and-go operation. The main improvement we want to
bring with our solution is transparent software update delivery and applications
through dynamic coding.
3
System Overview
3.1
Architecture
Our Dynacode application is designed to bring dynamic code and content update
to an application with limited costs during implementation and use. To achieve
this, we divided our project in three units:
• Back-End: The back-end is connected to all clients and will respond to
client requests to get up-to-date code and content on connection, as well
as pro-actively distribute new updates as soon as they are made available.
• Front-End: A front-end is provided to enable easy administration of the
back-end.
• Client API: A flexible and easy to use API to enable easy integration of
dynamic code and content in new or existing applications.
We propose a representation of our architecture in figure 1. The whole
application is based on technologies close to the Java world.
We also present in figure 2 the data model we used to represent our objects.
An arrow should be read as “contains”.
3
Figure 1: Dynacode Overview
Figure 2: Dynacode Data Model
4
3.2
Use Case
We will quickly describe how our system is expected to be used. Developers
should first include the client API in their project and use our instantiation
of dynamic classes instead of standard instantiation (more on this topic in the
client API section). Once the development of the application is complete, the
“static” part will be packaged in a jar file.
The front-end is then used to create a new “dynamic application”, by providing the said jar file and a name. Dynamic classes must then be added to
the application using the front-end. If any content is dynamic (eg: textures,
models, etc.), it should also be uploaded through the front-end.
Once the process is over, users can download the jar file to their computer
and launch the game. At the first launch, the client will proceed to retrieve all
dynamic classes and content while the game starts running. At each subsequent
launch, only updated classes and cotent will be retrieved.
In the following sections, we will go over more details about each of our unit.
4
4.1
Back-End
Technologies and Architecture
Our back-end is a pure Java EE dynamic web application tailored for the JBoss
application server. Its architecture is shown in figure 3. The architecture is a
rather traditional Hibernate / Spring implemenation up to the DAO layer. From
there we built our services as standard POJO’s. We proceeded to expose them
as web services through XFire for all administration-related features. Spring
controllers are used in place of web services when file upload is required. Lastly,
client-specific services are exposed as RMI services to enable more efficient and
flexible communication. The latter services are used when the client requests
data from the back-end. We also need data-push-like capabilities to be able to
dispatch newer revisions of code and contents as soon as they become available.
This functionality is provided by two JMS topics, a producer – consumer queue.
We use one queue to dispatch code updates and another to dispatch content
updates.
4.2
Challenges
Although the back-end development was rather straightforward, we encountered
a few problems. First, we were planning to use AMF as a communication
protocol between our back-end and our front-end. This was supposed to be
achieved through the use Granite Data Services[2]. However the front-end didn’t
seem to manage deserializing collections from Hibernate entities.
Another problem we had due to the use of web services, is that those do
not support circular references. In our persistence model, we were planning to
use bidirectional relationships, which we had to abandon. On the same topic,
we used XFire to handle serialization between xml and Java objects. Through
5
Figure 3: Back-End Architecture
the process however, XFire was creating proxy objects instead of objects of the
persistence class. Hibernate however uses the class of the object and instrospection to determine how to handle the save. Both of these operations would fail
on a proxy object. We had to design a casting system which would enable to
recreate an object of the proper class from the proxy object, which was much
more difficult than it seems due to object nesting and nested lists of persistence
entities.
5
5.1
Front-End
Technologies and Architecture
The front-end is built using the RIA-oriented Adobe Flex 3 development language. We’re also using the Cairngorm Framework, an MVC implementation
for Flex. The figure 4 sums up important features of our front-end. Starting
from a view, which is basically the presentation layer of the code, data will be
read from the model (eg: the list of available applications, etc.). When the user
interacts with the administration interface, for example by asking to refresh
the list of available applications, the view will send an event to the controller,
which will in turn invoke a command. This command will use a delegate to
contact the back-end through a web service or a controller. The delegate then
forwards the result he received to the command, or notifies the command of an
error if applicable. If no error has occurred, the command will then update the
model with the information he received from the delegate. the view will then be
automatically updated with the new data from the model thanks to Flex data
binding.
Our front-end exposes several modules. The first and most complete one is
the dynamic classes interface, which enables creation / deletion of applications,
6
Figure 4: Front-End Architecture
7
adding or deleting dynamic classes for a given application and management of
revisions and dependencies of a dynamic class. It can also be used to enable a
batch of classes or revisions, so that all classes sent to a client are also sent with
their dependencies. An important feature of this class management interface
is its robustness. Indeed, we heavily use reflection on the server-side when
performing class modifications. For example, when uploading a new revision,
we make sure the uploaded file is a valid java bytecode file, and that the class
name of the new revision matches that of the previous one.
The second module is a simpler resource management interface. It’s also
possible to add or delete a resource, as well as adding a revision for a given
resource. Lastly, a user management module is also provided. It currently
only enables creation of administrators for the administration interface, however
in the future it will also be used to manage users’ access rights to a given
application or resource package. We give a screenshot of our interface in figure
5.
Figure 5: Front-End Interface Sample
5.2
Challenges
We had a rather smooth development process for this front-end, save for the
AMF communication peoblem we already stated previously. We did not encounter any particular problems. We have to highlight that the release of Flex
3 a few months ago was a big help to us. Indeed, Flex 3 introduces the concept
of “Type registry”. This enabled us to use a DTO layer to communicate with
the back-end very easily. In Flex 2, we would have had to convert manually
each DTO received or sent from the back, or just use basic types in our communication routine. The type registry however handles automatic instantiation
of a DTO object upon receiving data from a web service through the use of the
WSDL data. Also, thanks to the Cairngorm framework, we were able to quickly
8
adapt when we had to undergo some modification in the business part or the
interface part of the code.
6
6.1
Client API
Technologies and Architecture
The last part of our Dynacode application is the client API, which can be used
to quickly port an existing application to a dynamic code paradigm or even
more easily to build one from scratch. The figure 6 describes the architecture
of the client API. Our dynamic instantiation process is derived from Li Yang’s
excellent article[4].
Figure 6: Client Architecture
At the heart of the code update process is the DynacodeManager, which is
actually the only part of our code an application developer has to worry about
to get dynamic code in his application. This critical object is implemented using
the singleton design pattern. When created, it will try to establish a connection
to the back-end to enable dynamic update. To do this, it will connect to the JMS
services offered on the server and also prepare our RMI client. It will also set
up our custom classloader for instantiation of dynamic classes. All this process
is transparent to the developer and of course the end user. When requesting a
9
dynamic instance from the dynacode manager, the returned object will actually
be a proxy object. That’s why we need an invocation handler, which will forward
all method calls to the real object. When called, the invocation handler will
also check if the dynamic object it is hosting is up to date. If it’s not, it will ask
the classloader to reload the class and will transfer all fields from the previous
version to the new one before proceeding to the call.
The other main component of our API is the DynaContent manager. It
has a similar role when compared to the Dynacode manager, except that it
is obviously responsible for content, or resource update. It will also be using
the JMS services provided by the back-end to be notified of any change in its
resource pool. When such a change happens, it will undertake necessary actions
required for the resource to be available to the player.
Lastly, another important feature of our client API is that it is fully multithreaded. Both code and content updates are handled in separate threads so as
to minimize interference with the game engine.
6.2
Dynamic Classes
Our API offers two types of dynamic classes, that we refer to as “Explicit
Dynamic Classes” and “Implicit Dynamic Classes”. Both offer advantages and
drawbacks that we will detail in the following section. We also show in figure
7 how one would instantite an object of class Example from package net.test
conforming to the IExample interface, supposing dynMan points to the instance
of DynacodeManager.
a) Example = new Example();
b) Example = dynMan.buildNewExplicitDynamicClass(IExample.class,
‘‘net.test.Example’’);
c) Example = dynMan.buildNewImplicitDynamicClass(‘‘net.test.Example’’);
Figure 7: Class Instantiation
a)standard class – b)explicit dynamic class – c)implicit dynamic class
6.2.1
Explicit Dynamic Classes
The only prerequisite for a class to be used as an explicit dynamic class, as can
be seen in figure 7.b is that such a class should always conform to an interface:
this means that all public functions should be known and more importantly
that no public method can be added when compared to the initial revision. We
deem this to be a considerable, but sensible constraint considering that a class
is defined to complete a certain task, and that other classes should not have to
worry about how the task gets done. We also highlight the fact that interfaces
only define public methods, so the developer is free to add, delete or modify
private and protected methods as he sees fit between the revisions.
10
On the other hand, a developer can use an explicit dynamic object pretty
much as he would use a standard object. The whole dynamic process is completely hidden, and the only actual difference is the instantiation process as
stated before. For the time being, before we polish the implicit dynamic classes
implementation, we strongly advocate using explicit dynamic classes.
6.2.2
Implicit Dynamic Classes
Aware that our explicit dynamic classes might still impose too much of a limitation to developers for specific classes, we also provide a powerful tool in the
form of implicit dynamic classes. In our current implementation however, we
sacrifice usability to obtain a truly dynamic class. Indeed, those classes get rid
of the two limitations of explicit dynamic classes: they don’t need to conform
to an interface and consequently enable a developer to add a public function
when changing revision.
The cost for this flexibility is currently high however. First, an implicit
dynamic class must extend ImplicitDynamicClassImpl. In other words, it may
not inherit from another class. Moreover, to invoke a method on an implicit
dynamic object, we use a rather complicated process, as exposed in figure 8.
ReturnType result = (ReturnType)implicitInstance.invoke(
‘‘methodName’’,parameter1,parameter2,parameter3\ldots)
Figure 8: Invoking a Method on an Implicit Dynamic Class
The biggest downside to this method is the lack of autocompletion, and
even the number and types of parameters can’t be known without refering to
the source code of the class. This is however the only way to get rid of the
interface limitation. Indeed, the concept of proxy used in Java requires that
a proxy be always linked to an interface, so our implementation of implicit
dynamic classes is actually a trick: we provide the proxy with an interface only
containing a public “invoke” method, the back-end object in the proxy will be
the implicit dynamic object itself. This invoke method will then call the method
on the back-end object with the parameters as specified in the argument list.
6.2.3
Dynamic Classes Migration
While the game is running, if a new revision for a class is received, the invocation
handler will proceed to its migration. We provide two complimentary ways of
handling migration. The first is an automatic migration process which should
suit the needs of most classes. We simply proceed by recovering all public,
protected and private fields of the current revision using introspection. We then
create an instance of the new revision, and we transfer all the fields we recovered
from the old version to this new instance. If a given field has been deleted from
a revision to another, we will simply ignore it.
11
The second way the migration process can occur is by using custom migration. As implied by the name, this mode leaves the developer responsible for
the actual object migration. To enable this, the dynamic class must implement
the ICustomMigration interface. The latter includes two public methods, the
“enableAutomaticMigration”, which is simply supposed to return a boolean indicating if the developer wants to proceed to migration from a “blank” object
(returns false), or if he desires to continue the migration process where the
automatic migration left off (returns true). The second one is the “doCustomMigration” method. The parameter it receives will be the old revision of the
object. From there, the developer is free to implement the migration any way
he desires.
6.2.4
Dynamic Classes Evaluation
In figure 9, we sum up the advantages and drawbacks of our two types of dynamic
classes. As stated before, we think the constraints of the explicit dynamic classes
are worth conforming to, as they give a very clean result. It is also considered
to be a good Java development practice to have classes implement an interface
when they are subject to change (eg: a DAO usually implements an interface
in case the current Hibernate ORM is replaced later).
Figure 9: Evaluation of Explicit and Implicit Dynamic Classes
6.3
6.3.1
Dynamic Resources
Architecture
Alongside our dynamic code update, we also provide dynamic resource update
capability. We go into a bit more details about the “DynaContent manager” in
figure 10. The cornerstone of our updating system is JNotify[3]. This Linux-
12
and Windows-compatible library enables capturing OS messages related to file
access. We are thus notified of any change in our resource directory and can
then handle the updating process as soon as a new resource revision is received.
Figure 10: DynaContent Architecture
6.3.2
Process Overview
However, we could not factor as much of the functionality for resources as we did
for the code. Indeed, although a piece of code is a determined concept that we
perfectly know how to handle, and that will be consistent from one application
to another, a resource is a much broader concept that can’t be guaranteed
to be the same even within a single application: you wouldn’t have the same
updating mechanism for a texture or for a model. Even more so, the process of
updating a texture might differ from object to object, the most obvious example
being texture management for a terrain and a character model. That’s why the
framework provided is highly customisable, leaving to the developer’s discretion
how the updatable resources have to be handled.
Our framework is composed of an interface, IUpdatableResource, and two
abstract classes: ResourceListener and UpdatableResourceManager. For each
resource concept, a developer must create a new interface extending our generic
IUpdatableResource. More importantly, he must also provide an implementation of ResourceListener. The abstract class provides detection of a modified
file, handling of a new revision must then be provided. Lastly, for each application, an implementation of UpdatableResourceManager must be provided. A
manager is a singleton class used to register listeners.
13
6.4
Challenges
The client API development was a constant challenge as we were working with
technologies relatively unknown to us (JMS, RMI, etc.) and that we had to
tackle problems that have not been covered extensively. In the end however, this
proved to be more a good opportunity for learning than a frustrating experience.
7
Sample Implementation
Aircarrier[1] is a 3D dogfighting game in which a player controls a plane and
must defeat a number of AI planes. As one of the rare Java game in 3D, and
considering its fast-paced nature, we thought it would be the perfect platform for
us to demonstrate the capacity of our Dynacode application. We thus proceeded
to two modifications to showcase our work. We modified tha targetting AI using
the explicit dynamic class model, and we also enabled texture updating on one
of the planes.
The AI modification process actually took us about 10 minutes. Although
the class was not implementing any interface to begin with, we used Eclipse to
extract an interface from this class. In the main class, we added a DynacodeManager instance as a private field, and modified the instantiation process as
described in figure 7.b. It was then a simple matter of adding the new application using the front-end, and uploading our dynamic class.
The texture update process however took a bit longer, as we needed to implement the classes described in previous section. Moreover, we were modifying
an application which was not our own, and was to us somewhat cryptic at times.
Factoring the JNotify library tests, implementation process and debugging, it
took us about one day to make it workable. We deem that it should not take
more than half that time in the general case (we were still discovering JNotify
and aircarrier), especially if the developer has taken part in the development of
the application to modify. The same time requirements would hold true for an
application designed from scratch.
The most intereting fact we got out of this experiment is that the framerate
did not suffer from our dynamic class and content mechanism. Although aircarrier does not include a benchmark utility, basing our judgement on the FPS
counter displayed in game, we did not notice any noticable performance issues,
if at all. This is very encouraging for an AI routine called once per frame!
8
Limitations
Although we think our Dynacode application is pretty powerful and efficient,
we still have to note a few limitations in our current system.
First, when migrating a class from one revision to another, we need the class
to define a default constructor. For example, the AI class we had to modify
in aircarrier only defined a constructor taking several parameters. We had
to change this to a default constructor, and move the part of the constuctor
14
using the parameters in a separate function. This has more a cosmetic than a
functional impact though. The developer must however keep in mind that all
final fields of a class must be initialized in the body of the class and not in the
constructor. This limitation might be lifted in the future, as we think that the
Java reflection API is poweful enough to provide a solution.
Another limitation is the fact than an explicit dynamic class needs to implement an interface. As we explained earlier though, we think that this is more
a matter of application architecture, as we don’t think a class’s goal should
change from one revision to another. It would however be possible that following a refactoring, a method name has to change, this is not currently supported
by our system. Also, the instantiation of such a dynamic class as presented in
7.b is a bit clumsy. In the future we want to see if we could hide a bit of the
complexity by only passing the interface as a parameter to the buildNewExplicitDynamicInstance method. We’re still wondering if this can be done in a
straightforward manner though.
Linked to the previous limitation is also the fact that if a developer was to
actually change a method name and dispatch this revision, an exception would
be thrown on the client side. This could be solved by simply selecting the
previous revision as the revision to be used through the front-end, however it
would definitely be good to have a bit more error handling on the client API, and
maybe even a “rollback” feature that would cancel an update if something went
wrong so that we can guaranty the application on the client-side will always be
in a working state.
Another important limitation is our current implementation for implicit dynamic classes, which is definitely not robust enough and difficult to read. We’ve
yet to find an elegant solution to correct this problem. For now, the best solution we’ve thought of is to develop an Eclipse plugin which would be able
to help the developer know what parameters are needed through the use of
auto-completion.
For any commercial use of our solution, we would definitely need to implement some security into our application. The communication can easily be
hidden from eavesdroppers by running JBoss on https, however we would also
need to secure our services. We’ve had a quick look at Acegi Security for our
web services, and didn’t have time to check RMI security. Securing individual
applications / resource packages so that only a given group of users can access
them however seems easier, as we would only need to add a security layer at the
service level.
The dynamic resource delegates a lot a of the work to the developer when
compared to how dynamic code is handled. This is however something we can’t
really help due to the numerous types of resources that could be used in a game.
We could however design a more standardized “resource description language”,
based for example on XML that could help integrate part of the updating process
in the API.
Finally, we offer little guaranty in terms of update synchronization. Although
all connected client are likely to all be updated at a few seconds interval, there
is no mechanism that would ensure this. It is a rather minor limitation for con15
nected single player videogames, however this makes our current implementation
unsuitable for online multiplayer games.
9
Conclusion
In this paper, we’ve described an innovative distribution system for connected
videogames. It is the first system (we are aware of) that enables seamless update
of both code and content in an application without requiring a restart of the
application or any action from the player that would hinder his immersion. Our
front-end enables easy modification of any application or resource that have to
be dispatched, while the powerful and flexible client API enables developers to
quickly integrate dynamic components in their applications. We deem that with
a few more features, especially those related to security, our system could be
considered suitable for a connected single player game. The next step would
then be to ensure the scalability of our application and add some synchronization
guaranty so as to also support online multi player games.
References
[1] ”shingoki”. aircarrier (https://aircarrier.dev.java.net), 2006.
[2] Franck Wolff. Granite data services.
[3] Omry Yadan. Jnotify (http://jnotify.sourceforge.net).
[4] Li Yang.
Add dynamic java
(http://www.javaworld.com), 2006.
16
code
to
your
application