Risk identification taxonomy
questionnaire for game development
projects
Raúl A. Véjar Llugany
Contents
Introduction ...................................................................................................................... 2
About game development projects ................................................................................... 3
The taxonomy based risk identification questionnaire ..................................................... 4
The requirements category ........................................................................................... 4
The management process and methods category ......................................................... 6
Conclusions and further steps ........................................................................................... 8
References ........................................................................................................................ 9
1
Introduction
Game development has always been a risky business. In recent studies it was revealed
that only 4% of projects undertaken reach commercial success (either dying incomplete
or failing to sale enough as to regain the investment of producing it). Many factors
made game development risky, some inherited from the software development aspect of
game production and some from unique characteristics of this industry such as being
exclusively a retail market, highly multidisciplinary teams developing both art assets as
well as code, and a very small lifetime for most companies that makes maturity almost
impossible.
Risk management is one of those disciplines that is seldom performed in a consistent
and effective manner, especially in an industry were much of the effort goes into
activities that contribute directly to the final product since the deadline is always lurking
in the horizon. If risk management is to be effective, it needs to be done in a continuous
way, involving constant cycles of identification, analysis, planning, tracking and
controlling them; all facilitated by a strong culture of communicating potential problems
instead of hiding or denying them. Of all these activities, identification is probably the
most important one because is the first step that enables the others and by itself can
trigger changes to our behavior once we are aware of the potential issues.
Unfortunately, identification is also the trickiest discipline in the list since it depends on
factors such as:
 Having the right domain knowledge to be able to recognize the symptoms
 Having a culture that rewards raising awareness on potential problems
 Prioritizing risk management enough to have all the relevant stakeholders take
the time to participate in these activities
 Being able to discriminate those issues that are relevant from those that do not
affect the goals of the project significantly
Luckily this gets easier with resources that different organizations contribute to the
general practitioners, one of such the taxonomy based questionnaire from the SEI. The
questionnaire is meant to help software development teams identify risks by going
through a list of questions organized in categories including such as design, integration
and test, and resources.
Unfortunately, the questionnaire was developed with DoD systems development
programs in mind, and as such, it contains several categories and questions that are not
relevant (and should be skipped/adapted) for game development such as those regarding
the program structure. It also uses a vocabulary typical of these types of projects that if
exposed to a game development team will generate immediate rejection and distrust.
Besides that, other aspects particular to game development such as the fitness of the
game inside the genre for which is developed or the relationship with the publisher of
the game are not covered by the questionnaire directly. These have to be derived from
questions regarding requirements or contract management with a very long jump of
abstraction on the participants or improvisation to change the question by the facilitator.
Both of these options are not great since they distract the participants from the real
purpose of an identification session which is to find as many “good” risks as possible.
Given the scope of this assignment, the purpose of this paper is to review selected
sections of the current taxonomy based questionnaire for risk identification from the
SEI under the light of a game development project, identify those instances in which a
change is recommendable and suggest changes. The ultimate goal is to facilitate the task
of people interested in doing risk identification using these guidelines but adapting them
to a game development project.
2
About game development projects
Game development has a number of particular traits that make it different from the
typical software development project, just to enumerate a few:
 Games have a large artistic component, not only because of the game aspect but
because they include a large number of art and music assets as part of the
experience. This involves a problem coordinating different activities not only on
terms of schedule but also in terms of content because the overall result must be
consistent with the design vision for the product.
 Games are in most cases retail products that require retail partners to sell and
market them. They are also highly perishable products since much of their value
comes from the hype of their release and the fast pace of technology makes them
obsolete in a short time so retailers will be reluctant to keep them in the shelf for
a long period. They also are depreciated rapidly, with a new game costing close
to 50$ on the release date but between 25$ to 5$ one year later.
 Most every game involves highly complex technical disciplines such as artificial
intelligence, real time systems, complex user interfaces, graphic programming,
and networking.
 Game development is a young discipline, with very few programs in universities
or specialized centers training people specifically for it. As a result, a lot of the
people involved in this industry have diverse backgrounds. There is also not a
common body of knowledge or much specialized research being done about
them which makes most of their methods borrowed from industries ranging from
traditional software development to movie production.
 Game development companies usually depend on having a Publisher company
pay for the development of the game and later take care of selling it. Even
though the concept for the game and the initial prototypes are developed by the
development company (sometimes called “studio”), the publisher will have its
own ideas of what market segments should be targeted at, what platforms it
should support and when it should be released. They also provide common
resources such as testing, marketing, legal and sometimes even project
management.
 Game sales are seasonal, depending on special occasions such as Christmas or
summer break start.
 Given the high costs of production of AAA titles (close to 3 years with hundreds
of people involved), the industry is geared towards a hit oriented model which
has many bad practices associated such as the reluctance to try proven-to-work
models or formats going for the innovation point of sales.
 Game development companies are usually short lived, were one bad project
(either on implementation or sales) will ruin their reputation making it extremely
hard to get publishers for their next game essentially cutting their funding. This
results in low maturity since organizations have little chance of improve their
performance over the course of different projects.
Although it is as impossible to talk about a single development process in game
development as it is in traditional software development since every company and
project has their own adaptations, over the years a common vocabulary has been
developed which is used to described the different activities and roles involved in
creating a video game. It would be impossible to describe it in this paper because of
space reasons, but this vocabulary will be used when discussing the adaptations to the
3
questionnaire. Any reader interested in it can consult the reference section for ones of
the books dealing with game production that describe some of these models.
The taxonomy based risk identification questionnaire
The taxonomy based risk identification questionnaire was developed by the SEI
originally to help DoD identify risks in their settings that matched the problems that
have been identified in industry in general over the years the SEI’s staff had performed
as consultants and evaluators. As such, it was developed using the vocabulary common
to that domain and the categories in which questions are organized represent the aspects
of projects that were the top concerns for these projects. The questionnaire was
developed as a tool to guide risk identification sessions and its effectiveness depends on
the moderator conducting the assessment to identify the relevant questions and be able
to dig into the concerns answering them might arise.
Game development projects use a different vocabulary and have slightly different
concerns. Of course basic things such as scheduling and team collaboration are still
main issues, but others such as the relationship with the publisher, the stability of the
game design, and the interdisciplinary nature of game projects are not covered in any of
the categories. These differences go beyond what a moderator could be asked to adapt
“on the fly” during a risk identification session and thus it presents an opportunity to
adapt them in advance which is the focus of this paper.
As the scope of this paper is limited, we will focus on only a few of the categories and
the problems that might be relevant to the equivalent space in game development
projects. We have chosen to focus on the “Requirements” and “Management Process
and Methods” categories given that a great deal of the issues rise on these grounds
which we could interpret as the design of the game and the production management of
it.
The requirements category
The requirements group tries to capture risks regarding stability, completeness, clarity,
validity, feasibility, precedent and scope. In game development projects requirements
do not exist as such, there is though a design document that contains a description of the
features the game is targeting to support which is the main measurement of scope. The
design also directs the style that the game is going to have and thus this becomes a
requirement for the art, sound and writing production. It can also have a description of
the settings and characters the game is going to have, this is also another measure of
scope since it relates directly to the amount of assets that have to be produced. We can
call all these factors “design elements”, thus producing the first recommended change:
1. Talk about design elements instead of requirements
This contributes to leveling the vocabulary of the questionnaire, an essential issue if we
want to generate buy-in from the people using it.
One issue with taking this approach is that the questionnaire also has a “design”
category which deals with those quality attributes such as performance, reliability and
usability. These are also usually addressed in the design document for the game since
they directly affect the gameplay. It is common to find performance targets in number
of frames per second (FPS), or the target platforms that the game is targeted at. Game
4
developers understand that these aspects make an integral part of the resulting
experience for users and thus make them an integral part of the overall design of the
product. It would be difficult to draw the line of where the requirements stop and the
design start in this domain, so our second suggestion is:
2. Do not try to separate design from requirements categories as this distinction is
much blurrier in developing games
Design stability is usually a tricky thing since most designers plan on altering the design
of the game as the product development progress and they can asses in a better way how
fun the final game is going to be. This is usually is potentiated by the philosophy of
getting the hit game of the season which tends to foster wild designs in search of new
attractive formulas for gameplay. Many development studios are willing to go through
several iterations of zero-increment while trying different designs so as to get the one
that appeals more to their potential customers. Nevertheless, it is still important to
question what design elements instability is going to do to the schedule, integration and
other aspects of the development.
3. Keep the questions regarding the instability of the design elements’ impact on
the other aspects of production, but also add questions about what is being done
to make the design stable in terms of prototypes, user testing, etc.
When talking about completeness of requirements, an important aspect of it is the
publishers’ requirements regarding changes to the design that would improve the sales
potential of the game. This is a very important aspect of the game, but you want to be
able to evaluate such changes as early as possible and not as a result of the publisher’s
new policy regarding their target markets. Question 5 of the completeness section is a
good candidate for this.
4. Ask questions regarding other stakeholders such as the publisher introducing
new changes to the design and the effect this could have on the current plans for
production (including scheduling, design, feasibility, etc).
Clarity and validity are important issues since it can lead to costly rework or a final
product that is uneven in its different parts. The main issue with the questionnaire at this
point is identifying who all the relevant stakeholders are instead of just the “client” as it
is prompted in the original questions. In general the design is developed and maintained
by the lead designer while the developers, other designers, artists, and the rest of the
team work to implement it. We must also remember the publisher who might have
inputted requirements and must have the same vision regarding the product being
developed.
5. Consider both the lead designer and the publishers are customers for the clarity
and validity questions, the rest of the people involved (including programmers,
designers, artists, and producers) should be considered the target of these
questions.
Feasibility, precedent and scale questions do not need major changes, but their
importance is bigger now. Since most companies are short-lived and the features
targeted is constantly increasing/changing from project to project, there is a very good
chance that there will be a great many things that are completely new to the
organization. Just remember that prototyping is a usual tool in trying to determine
feasibility and usually replaces more traditional feasibility studies.
There are additional issues that belong to this category and should be addressed as they
are usually a source of problems. Among them we recommend considering:
6. Theme conventions. For example, if we are developing a role playing game
(RPG), are we sticking to the rules defined in Western or Japanese games? What
would this mean to our players? Will they be more enthusiastic? Will the game
5
be harder to learn? Will the players be driven away because they expected the
game to follow the theme codes?
7. Localization. Are we planning to localized the game to other places? Does our
plan consider sufficient time for this? Are the asset producers aware of this? Are
we planning to release multiple versions simultaneously or localization will take
place afterwards? Does our game architecture support easily changing the
content to match this? Will we be doing the localization or someone else
unfamiliar with our tools?
8. Ratings. What rating classification are we targeting? Is this consistent with our
design? Are there guidelines in place for asset producers that are consistent with
the targeted rating? Are all members of the team aware of this? Is there an
internal review process of the content to validate the expected rating?
9. Reviews. Will our game appeal to reviewers?
10. Engine/Middleware choice. Is the choice of engine or middleware consistent
with the design?
The management process and methods category
The management aspect of projects is divided into two parts: process and methods.
Process deals with things such as planning, project organization, management
experience and upper level interfaces (called “program interfaces”). Methods, on the
other hand, deals with monitoring (or tracking), personnel management, quality
assurance and configuration management.
One important thing to remember is that since the questionnaire was developed for the
DoD, there are many references to the “program”, which is the way projects are
structured in that domain (a program has a “mission” and is composed of several
projects that seek to accomplish different aspects of the mission). In the game
development context, we can thing of the equivalent of the program as the upper
management of the company and the outside stakeholders such as the publisher.
1. Consider the “program” references as the sum of those stakeholders outside of
the team that have significant decision power over its fate such as upper
management and the publisher.
We must also remember that in game development, the project management aspect is
referred to as “producing” and the manager is the “producer”. It is also common to have
multiple producers (or associated producers) for the different aspects of the game, so
each question should target all producers in their different groups and not just the main
producer.
2. Consider the “project manager” references as the producer. Take into account
that many producer may exist for the different groups inside the team so the
questions should address their individual areas of responsibilities in separate.
All of the questions in these sections talk about good management practices that all
healthy organizations should have such as Quality Assurance and Configuration
Management, independent if they are producing games or IT software. It is very
important to have a working understanding of what these disciplines mean as most
people in game development will have a wrong concept of, for example, what “quality
assurance” means. Nonetheless, given the backgrounds of people in game development
environment, these disciplines are rare in game development organizations so the main
concern here should be having a clear understanding of the potential negative effects of
not performing these practices as to make people aware of potential issues. If added to
the questionnaire it must be done with a sufficient level of abstraction since it would
6
defeat the purpose of the instrument if the participants would just take the risks
“suggested” in the questionnaire instead of thinking and discussing about these
questions in their setting (not to mention the poor buy-in these risks would have)
3. Consider adding to the questions potential problems not having these disciplines
in place could have regarding schedule, rework, quality, etc. Be careful not to
be explicitly suggesting risks to the participants.
The questionnaire includes some questions targeted towards projects that involve
developing a system for a specific customer, for example if there are special
requirements of deployment. These should be scrubbed beforehand since the end result
of a game development project is a product that will be installed by the consumer.
Another option is reinterpreting these as issues when targeting multiple platforms and
the impact it could have on planning, configuration management and so forth.
4. Consider scrubbing questions related with customized development of systems
instead of product-focused, or shifting those questions towards issues that might
arise when targeting multiple platforms.
Finally, there are a number of issues not covered in these sections of the questionnaire
that belong in them and are particular to the game development domain, such as:
5. Coordination between producers. Given the multi-disciplinary nature of projects
and the fact that there might exist multiple producers in charge of different areas
of development, additional questions should be asked regarding the mechanisms
in place to ensure coordination.
6. Special schedule requirements. Are we targeting a special date such as the
week before Christmas? Do we have enough buffer to avoid overrunning
important milestones? Are there any other high profile games coming out at the
same time as ours?
7. Responsibilities of the publisher. Is the publisher providing resources? Is the
publishing schedule coordinated with our plan?
8. Special platform requirements. Is there a certification requirement for any of the
platforms we are targeting? Does the plan have sufficient time to support a
certification rejection?
7
Conclusions and further steps
On this paper we have discussed the importance of performing risk management
effectively in game development. We have also identified a gap of domain-specific
guidance and support. To help practitioners trying to fill this gap we have identified
several adaptations that can be made to the SEI’s risk identification taxonomy based
questionnaire which was developed originally for DoD software development projects.
We have looked at changes in the X,Y,Z categories as we have considered them the
most critically in need of changes given the characteristics of game development also
identified in this paper.
If you or your company are concerned about the way risks are being performed in your
settings, you should start by looking at the guidelines for continuous risk management
that are available through the SEI. Although most of these were developed with more
traditional software development projects in mind, the concepts are generic enough that
they can be applied to any project-based structure, including game development.
The work done on this paper needs to be refined further as they provide only
suggestions of possible changes and examples of them. Before being able to deploy it
needs to be adapted fully which will include deciding to implement the adaptations
suggested in this paper or others as well as tackling the other sections of the
questionnaire. Once deployed it needs to be continuously improved as applying it will
reveal further refinement needed to make it truly effective. There are many important
issues to be decided in this effort such as the scope of projects targeted, the level of
detail required and the changes needed to the supporting processes that will use this
asset.
One final word of warning, identifying risks has little value on its own. To stop being
surprised by problems you need to be able to properly manage the risks you are aware
of and decide on which you are going to spend resources to mitigate them. The focus
taken on the identification discipline in this paper is a reflection of the importance of
this activity but does not pretend to support the idea that it can stand on its own as an
effective measure to manage risks.
8
References
[Marvin 1993]
Carr, Marvin J., Konda, Suresh L., Monarch, Ira, Ulrich, F. Carol,
Walker, Clay F. Taxonomy-Based Risk Identification. Software
Engineering Institute, June 1993.
[Chandler 2008]
Chandler, Heather Maxwell. The Game Production Handbook,
Second Edition. Jones & Bartlett Publishers, August 2008. (ISBN10: 1934015407), (ISBN-13: 978-1934015407).
[Hight 2007]
Hight, John, Novak, Jeannie. Game Development Essentials: Game
Project Management. Delmar Cengage Learning, March 2007.
(ISBN-10: 1418015415), (ISBN-13: 978-1418015411)
[Dunniway 2009]
Dunniway, Troy. Small developers: minimizing risks in large
productions Part I & II.
http://www.gamasutra.com/view/feature/4180/small_developers_mi
nimizing_risks_.php?print=1 (November 2009)
http://www.gamasutra.com/view/feature/4195/small_developers_mi
nimizing_risks_.php?print=1 (November 2009)
[Ryan 2003]
Ryan, Tim. Risk Management With Development Schedules.
http://www.gamasutra.com/view/feature/2900/risk_management_w
ith_development_.php (February 2003)
9