Serious Games for Improved Defense Contracting
by
Roi John Guinto
B.S. Mechanical Engineering and Physics
Rutgers University, 2012
Submitted to the Engineering Systems Division
in Partial Fulfillment of the Requirements for the Degree of
Master of Science in Technology and Policy
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
June 2014
@ 2014 Massachusetts Institute of Technology.
ARCHIVU
All Rights Reserved.
IMASSACHUSETTS INST
OF TECHNOLOGY
MAY 2172014
BRARIES
Signature redacted
Signature of Author
Technology and Policy Program
Engineering Systems Division
May 23, 2014
Signature redacted
Certified by
Olivier de Weck
Professor of Aeronautics and Astronautics and Engineering Systems
Thesis Supervisor
Sig nature redacted
Accepted by
Dava
ewman
Professor of Aeronautics and Astronfutics and Engineering Systems
Director, Technology and Policy Program
I
E
-2-
Serious Games for Improved Defense Contracting
by
Roi Guinto
Submitted to the Engineering Systems Division on May 23, 2014
in Partial Fulfillment of the Requirements for the Degree of
Master of Science in Technology and Policy
Abstract
Gaming has been explored as a research technique in complex systems to explore human
interactions within technical domains. These so-called "serious games" are designed for purposes
outside of pure entertainment and are intended to help researchers engage with qualitative data
that focuses on human behavior. This thesis focuses on a research game called ColumBID which
was designed for the Production in the Innovation Economy shipbuilding study. This game seeks
to understand the negotiation process in defense contracting and is used to facilitate and record
the participants' movement through the tradespace. From a game theory perspective, we
analyzed the non-Pareto dominant movements through the tradespace and found that humans
explore the space by sliding along fronts until they reach an optimal and then jumping to another
front to negotiate another optimal point. The players then compare these optimal points to
simplify the multi-variable optimization. We found that an 8% contract price improvement in the
competitive case compared to the monopolistic case. This thesis recommends that serious
research game design focus on questions about the process rather than the endgame optimization.
Thesis Supervisor: Olivier L. de Weck
Title: Professor of Aeronautics and Astronautics and Engineering Systems
-3-
-4-
Acknowledgments
My parents have always encouraged me to avoid letting school get in the way of my
education. From a very young age, they were always willing to let me learn through games,
whether or not they were explicitly educational. They never questioned why it was so important
for me to reach the next save point, yell at that blue shell, slay the dragon, and keep going for
one more turn. None of this work would be possible without my parents trusting that the
countless hours I've spent offline and online playing with friends near and far were worth
something. Thank you for always supporting me. I also thank those faraway friends for helping
out whenever I needed a break from MIT life. Extra special thanks to Joe, Chris, and Tony for
remote playtesting, and to Jamie for bringing my game concepts to life with her art.
MIT has given me two of the happiest years of my life. It starts with my advisor, Olivier
de Weck, and the Strategic Engineering Research Group. From day one, they've given me a
home base in the warmest sense of the word. Oli has given me incredible freedom to pursue
serious games from the start and trust to make my own decisions about the direction that I've
taken this research. Everyone in SERG has always been willing to help playtest ideas and has
been invaluable for feedback throughout this research process. Thanks to Ross, Sydney, Paul,
Koki, Ioana, Chaiwoo, Andrew, Sung, Margaret, Narek, Patricia, Amanda, and Kaushik. Special
thanks to Paul Grogan for being a fantastic mentor on everything from game design, to bike
shopping, to class selection and settling into Cambridge. Whatever university is lucky enough to
land you is going to get not only a great researcher, but a great human being as well.
This research was made possible through the support of Mr. Sean Stackley and his team
in Research, Development and Acquisition. This extension to the Production in the Innovation
Economy study is funded by U.S. government contract No. N00014-09-D-0584, subcontract
number 2013-432. Mr. Stackley's team has provided great resources in leaming about the
shipbuilding process through visits at the Pentagon and the Navy Yard, as well as through
continued remote support with the assistance of Capt. Jim Weiser and Capt. Casey Moton. There
was also support from Mr. Kevin Carpentier and his group at SCRA in South Carolina, as well as
from Capt. Mark Thomas and the 2N program here at MIT. They were instrumental in reaching
out to contacts in shipyards around the country as well as providing their own team support in
data collection. I received tremendous help from our program manager Dr. Eric Rebentisch, who
provided great insight into the research process and his experience in defense contracting.
-5-
It is a privilege to support brave men and women in the U.S. military like my grandfather,
Col. Joven Villanos, who has sparked my interest in science for as long as I can remember.
There are countless other research groups at MIT and the surrounding Boston area that
have welcomed me with open arms and have been invaluable in learning about research games. I
thank Philip Tan and the MIT Game Lab for being an excellent sounding board on game design
as well as introducing me to many people around the institute and beyond. I had the privilege of
designing and teaching a course on research games and they were a tremendous resource for its
creation and execution. I thank both Konstantin Mitgutsch of the MIT Game Lab and Leah
Stokes of the MIT Science Impact Collaborative who both allowed me to study their design
documents for their own research games and gave me frameworks for creating my own.
I was fortunate to have the opportunity to learn from Eric Klopfer, Scot Osterweil, and
Jason Haas from the Education Arcade about educational game development and deployment. It
was a dream come true to meet the designers of some of my favorite childhood games. I could
never have imagined that they would be willing to help me design my own course and develop
my own games. Thanks as well to Todd Schenk of the Science Impact Collaborative, Amy
Robinson of the EyeWire team, Ari Epstein from the Terrascope project, Adam Ross from the
Systems Engineering Advancement Research Institute, T.L. Taylor from the Comparative Media
Studies department, Eric Gordon from the Emerson Media Arts department, and Casper
Harteveld from the Northeastern Game Design department for participating in the research game
course. It was a joy to listen to game researchers around the area working in so many different
fields.
Above all, the people who have shaped my MIT experience the most are the gals and
guys of the Technology and Policy Program. You can always find difficult courses and
interesting research projects wherever you may choose to study, but I think what sets TPP apart
is the wonderful cast of characters you'll meet. Many hours have been spent getting help from
Ed, Barb, Frank, Joel, and Dava and the rest of the core faculty. The cohorts of 2013 and 2015
have been fantastic as well, but there's no group quite like my friends known as TPP '14. We
like to say that your acceptance into TPP comes with a support system of 40 friends built right
in. They've helped me countless times with economics, offered to join my playtesting, taught me
about energy policy, abused my curveball in kickball, and always supported me mentally and
emotionally. In short, you're the best group of friends I could ask for.
-6-
Table of Contents
Acknowledgments
5
List of Figures
9
List of Tables
10
Chapter 1 - Introduction
11
11
1.1 Research Framework
1.1.1 Research Questions
11
1.1.2 Research Methods
12
12
1.2 Thesis Overview
Chapter 2 - Background and Motivation
15
2.1 Commercial Games
15
2.2 Serious Games
17
2.2.1 War Games
18
2.2.2 Urban Design Games
20
2.2.3 Educational Games
21
2.2.4 Management Games
24
2.2.5 Policy Games
26
2.3 Naval Acquisitions
27
2.4 Research Motivation
29
Chapter 3 - Model Design
31
3.1 Reality
34
3.1.1 Flexibility
35
3.1.2 Fidelity
35
3.1.3 Validity
36
37
3.2 Meaning
3.2.1 Purpose
38
3.2.2 Strategy
38
42
3.3 Play
3.3.1 Systems of Information
43
3.3.2 Game Theory Systems
44
3.3.3 Systems of Conflict
46
-7-
3.3.4 Play of Simulation
47
3.4 Final Design
47
Chapter 4 - Analysis
53
4.1 Monopolistic Case
54
4.2 Competitive Case
61
4.3 Differences Between Cases
62
Chapter 5 - Implications & Future Work
65
5.1 Case Study Implications
65
5.2 Game Design Implications
66
5.3 Future Work
67
5.3.1 Supply Chain Expansion
68
5.3.2 Congress as a Player
69
5.3.3. Cost-Plus Contracting
70
5.3.4 Multiple Contracts
70
5.3.5 Further Qualitative Analysis
71
Appendix A - Interview personnel records
72
Appendix B - Aggregate sample demographics
74
Appendix C - COUHES Materials (Survey, Consent Form)
78
References
85
Gameography
89
-8-
List of Figures
Figure 1 - Thesis roadmap
13
Figure 2 - Evolution of serious games
18
Figure 3 - Wargaming team positions
19
Figure 4 - Screenshot of SimCity
21
Figure 5 - Screenshot of Civilization 111
22
Figure 6 - List of management games
25
Figure 7 - Annual shipbuilding rates
28
Figure 8 - Key stakeholder drivers
32
Figure 9 - Triadic game design framework
33
Figure 10 - Functional decomposition of games for supporting policy
38
development
Figure 11 - Twine developer interface
41
Figure 12 - Player briefing interface
48
Figure 13 - Spreadsheet tool interface
50
Figure 14 - Monopolistic price results
54
Figure 15 - Communications vs. price
56
Figure 16 - Tradespace quadrants in negotiation
57
Figure 17 - Representative negotiation
58
Figure 18 - Construction of overall Pareto front
59
Figure 19 - Evolution in block cost space
60
Figure 20 - Competitive price results
61
Figure 21 - Case price comparisons
63
Figure 22 - Currently existing game variants
67
Figure 23 - Supply chain expansion variant
68
Figure 24 - Congress as a player variants
69
-9-
List of Tables
Table 1 - Chicken & hawk-dove game payoff matrix
44
Table 2 - Resulting final contracts for the 37 negotiations
53
Table 2 - Representative data set
56
-10-
Chapter 1 - Introduction
1.1 Research Framework
"Through play, we encounter challenges with delight, we brave overwhelming odds with
hope, and we conquer our world with imagination. Play, as expressed in games, is the most
positive response of the human spirit to a universe of uncertainty." - MIT Game Lab
This research framework combines the best research game techniques developed by
GAMBIT and its successor, the MIT Game Lab, with policy analysis in technical systems as
described by the Technology and Policy thesis manual [1][2]. The inspiration for a research
game for modeling the contracting process draws from earlier work by the RAND Corporation
who helped pioneer the field of wargaming [3]. This research takes these historical developments
and infuses it with the modem pedagogies of the MIT Education Arcade as well as the technical
analyses of the MIT Engineering Systems Division to create a new tool for analysis [1][2].
1.1.1 Research Questions
The primary questions asked by the Production in the Innovation Economy shipbuilding
study are:
- Can the government be doing more to put more pressure on the contractors and their
suppliers to get better cost performance and innovation?
- Why is the escalation in shipbuilding costs greater than general inflation?
This thesis focuses on a subdomain of the shipbuilding process: the contract negotiation
between shipyards and the government. Specifically, through the use of a research game, the
questions about negotiation are:
- What are the differences between a sole-source and a competitive environment?
-
11
-
- Can we identify the most important tradeoffs that move a negotiation process and
influence tradespace exploration?
1.1.2 Research Methods
This research uses an original research game to simulate the negotiation process with
human actors. This captures their values when dealing with the social aspects of both negotiation
and the nested politics that drive these contracts. Data is captured that elucidates the tactics used
in negotiation as well as how the proposed contract moves through the tradespace as offers are
resolved by the players. This movement exposes players to areas within the Zone of Possible
Agreements (ZOPA).
1.2 Thesis Overview
This thesis combines two primary fields of knowledge. The first studies games as a
research tool, drawing heavily on success in the fields of education, urban planning, war gaming,
and management as serious applications. The second field focuses on the current state of the U.S.
shipbuilding sector and how policies have impacted the development of this industry into a
monopoly or duopoly in many cases.
Chapter 2 deals with the history of both fields, specifically on the development of serious
games as an answer to social complexities in technical systems and on the legacy of the Jones
Act as a market distortion. Policy gaming takes roots from developments in policy making
theories as well as successes in commercial games that led to adoption of games in classrooms
and offices. It shares these roots with other tools such as system dynamics and operations
research that attempt to incorporate qualitative aspects into quantitative systems.
Chapter 3 discusses the framework of Triadic Game Design as it pertains to the creation
of a suitable research game. By balancing the three main components of reality, meaning, and
play, a game can achieve many of the research goals that seem to be at-odds with each other.
Through this process, the underlying system model is created. A suitable scenario is then
designed alongside game mechanics to bring the game, known as ColumBID, to life. Finally, the
-
12
-
game is sent through iterative playtesting to polish components such as interface, game engine
selection, and number balancing.
Chapter 4 analyzes the data collected through the research game trials. The scenarios are
first studied individually to highlight particular points in the case that provide unique insight.
They are then compared against each other to learn about differences between the two cases.
Chapter 5 offers implications for policy making as well as for research game design
based off of the results. Proposals for future work based off of the current game structure are
suggested which may expand on questions about strategic behavior and model accuracy.
Figure 1: Thesis roadmap
-
13
-
-
14-
Chapter 2 - Background and Motivation
This section covers the development of serious games by looking back at the evolution of
commercial games into uses for military, management, and educational purposes. The chapter
then covers the history of the United States shipbuilding industry and ends with the research
motivation.
2.1 Commercial Games
What is a game? Games are often defined by a rule set with goals that must be reached
through some voluntary action. Bernard Suits writes in The Grasshopper:Games, Life, and
Utopia that a game is "the voluntary attempt to overcome unnecessary obstacles." While no
definition is perfect, this captures the heart of what makes games interesting: players willingly
take on a set of constraints to discover the best way to reach an objective [4][5]. This definition
covers both physical games and digital games.
Games have a history that is both very old and very new. Tabletop and board games such
as Go can be dated by thousands of years through the first millennium BC. Herodotus, writing in
The Histories in 5th century BC, describes a scheme used by the Lydian empire to survive
through a famine by organizing his subjects into teams to play games on certain days as part of
their fasting [6]. They would invent common game pieces such as dice and the ball. Ultimately,
they resolved their famine problem by playing a grand game of chance to split into two groups.
They would send away half of their population to find new, fertile land and settle their own
colony [5]. This is one of the earliest written records of games. In that time, games also took the
form of physical competition, such as the Olympic Games. Today, games are enjoyed in many
forms. Tabletop and board games continue to be played both in ancient forms such as
backgammon and checkers and in modem versions such as Monopoly and Dungeons & Dragons.
Sports of all kinds are played around the world. Games are no longer limited to these physical
forms; many are played at least partially in an electronic form.
Publically available digital games began with Spacewar! in 1961. The game was
designed by a fictional research group known as the Hingham Institute Group on Space Warfare
at MIT, led by designer Stephen Russell [7]. The game involved two players maneuvering
- 15 -
starships displayed on a Programmed Data Processor-1 (PDP-1). The computer and program
were available for student and public use in MIT computer labs. In 1966, the Magnavox Odyssey
became the first commercially available video game console. This moved electronic games from
the public space to people's own homes. Pong became the first video game sensation around the
world when it was released in 1972 in arcades. It appealed to a broad audience because it
simulated tennis, a game well understood by the public [8]. Even with limited graphics,
audiences and designers such as Mr. Russell were drawn to video games because "one can
simulate a reasonably complicated physical system and actually see what is going on." [7]
Today, digital games have matured into a $76 billion dollar industry internationally. The
widespread adoptions of smartphones and tablets have contributed to games' accessibility to
more people. Growing markets include the 10 million people in Mexico, the 15 million players
in South America, and the 17 million in South Korea. There are over 100 million people playing
games in Europe, another 100 million in India, and 200 million in China [5]. In the United
States, 59% of people are regularly playing video games on a tablet, smartphone, console or PC.
Regularly is defined as playing an average of thirteen hours per week, according to the
Entertainment Software Association [9]. The average game player is 31 years old and has been
playing for at least 14 years. The gender split is almost even; 48% of players are women. These
people are not playing games by themselves. 77% play with another person for at least an hour
every week, with 32% of the overall population playing with their family. Games are becoming
increasingly tied into social networks and are useful ways for people to communicate with
people they know. This is especially true for families: 51% of households have a video game
console. Out of the group of parents with children that play games, they play together 58% of the
time on at least a monthly basis. 56% of parents believe video games are a positive force in their
child's life [10].
While the mental image of a video game player has been a kid, alone in his room, the
reality is that gaming has become ubiquitous in the United States and in many other places
around the world. Men and women of all ages are playing with friends and family. Massively
multiplayer online role-playing games (MMORPG) such as World of Warcraft and EVE Online
bring millions of players into virtual worlds together. These online players create groups called
guilds and coordinate in events known as raids. Players are tasked with working together with as
many as 72 players at a time. As T.L. Taylor reports about her adventures in EverQuest [11], she
-
16 -
finds "the main mechanisms at work in all guilds, to varying degrees, are reputation, trust, and
responsibility." Players aren't just socializing; they're actively engaging in building a community
with connections that bleed over into the real world. Together, humankind is spending three
billion hours per week playing digital games [10].
These games span genres including first-person shooter, role-playing, puzzle, racing,
adventure, and simulation. Commercial games can be played on a personal computer, specialized
game console, a tablet, or a smartphone. While video games evoke images of blockbuster hits
such as Call ofDutv, FinalFantasy, World of Warcraft, not all of them are focused exclusively
on player fun as the sole outcome. Papers,Please is a game which places players in the role of
an immigration inspector in a fictitious Eastern Bloc inspired country. The game is designed to
stress the player by simulating the tensions at the border of a dystopian country. Depression
Quest is a text adventure which aims to create empathy for people with depression. Players are
meant to feel frustrated that they cannot take the common options that would make them happy
and resolve problems in life. Brands such as Burger King have produced games such as Sneak
King in order to promote their product while political campaigns such as John McCain's in 2008
have designed games such as Tax Invaders to convey their campaign promises to players [12].
Games in the commercial space are being used not only for fun, but also for important messages.
Games created for these specific purposes are known as serious games.
2.2 Serious Games
In his 1970 book, Serious Games, Clark Abt describes games which were designed for
particular educational or academic purposes and "are not intended to be played primarily for
amusement [13]." This term has become a catch-all for any game, digital or physical, which is
used as a tool to achieve a goal outside of fun. This does not mean that these games are not fun;
serious games utilize play to either teach or understand more about a process. Play as defined as
George Santayana as "whatever is done spontaneously and for its own sake." He relates work
and play to "servitude and freedom" to convey the idea that play is voluntary. Jesse Schell
elaborates further and describes play as "manipulation that indulges curiosity [14]." This
definition shows how serious games are an effective tool: they invite players to explore systems
without hesitation, allowing them to identify how the underlying game model works. These
-
17 -
mechanics and dynamics teach players about an idea that may be transferred to the real world.
When people are forced to learn a lesson or play a game, they lose what Bernard calls the "lusory
attitude" and are no longer accepting of the game's rules [4]. They reject the reality of the
game's rules and underlying model and will not treat the system as a suitable analog for the real
world. This prevents the transfer of lessons learned through play [15]. Serious games must
balance engagement to draw players into the simulated world and learning by both the players
and the game designers [16].
This section provides an overview of the branches of serious games: war games, urban
games, educational games, management games, and policy games.
Policy/War
Gaming
(c. 1810s)
Management
---
Educational
Gaming
(c. 1980s)
Gaming
(c. 1950s)
Operations
Systems Analysis
Research (c. 1945)
(c. 1960s)
Urban Design
(c. 1970s)
Game Theory
(c. 1945)
Figure 2. The evolution of serious games over time. Included are other tools used to
quantify social components of technical systems (operations research, systems analysis, and
game theory) [3].
2.2.1 War Games
The earliest serious games were used for military purposes. Games such as Chaturanga
and Wei Chi were derived from the philosophies of warfare in India and China [17][18]. These
games have lasted through nearly three millennia and now are called Chess and Go, respectively
[15]. While these are abstractions of warfare were used primarily to teach tactics and strategy to
officers, modern war games resemble either the roleplaying games of the
-
18 -
1 9 th
century or the
high-fidelity simulators designed to bring soldiers into virtual training fields. We focus on the
roleplaying games as simulators are often extensions of the commercially available versions,
designed to place players in a cockpit of a vehicle such as a submarine or fighter jet [19]. The
simulators are focused on technical details of execution rather than on sociotechnical systems.
As Mood writes in his paper War Gaming as a Technique ofAnalysis, "War gaming is
the traditional final step after the preparation of a war plan; it is universally regarded as the best
peacetime test of a plan." These games investigate situations that cannot be readily simulated due
to time, money, or scope [20]. As an example, Mood discusses the creation of a war strategy:
"Destruction of certain targets may have a profound effect on ground
warfare in the main theater of operations, for example, while destruction of others
are vital for neutralization of enemy air power. It is possible to assign relative
priorities to these kinds of targets in a limited context; they must be assigned in
consideration of the war as a whole."
The roleplaying war games popularized in the 1940s and 1950s conventionally included
three teams: red, blue, and the judges. Blue adopts the strategy that wants to be tested while red
must identify countermeasures to highlight potential flaws. The judges then debate on the
resolution. This is summarized below in Figure 3. Because there is no technical support tool, the
outcome of the games are qualitative and serve to highlight potential issues rather than hard
estimates on losses of life or quantified damage costs.
Figure 3: Wargaming team positions
-19-
These games allow experts in each part of a complex system to come together and
interpret the system as a whole. By sharing knowledge this way, the game allows for new
insights into problematic areas where a single expert may decide that there are no issues under a
proposed plan. Repeated play also allows for strategy and tactics to be studied together; war
games were popular to test efficacy, feasibility, flexibility, and sensitivity of a given plan.
In his book, The Signal and the Noise, Nate Silver recalls a proposed war game by the
North American Aerospace Defense Command prior to the events of September 11, 2001. The
game would simulate a hijacked airliner crashing into the Pentagon [21]. The game was never
run due to concerns that the scenario was "too unrealistic." The officers protested, believing that
the plane would never come from a domestic airport and that a terrorist would rather hold
hostages for negotiation instead of destroying the plane. This is an example of an event that
would be outside of typical system boundaries. This would have been a case of a war game that
explored a highly unlikely situation and possibly helped prepare a national plan in case of
emergency.
2.2.2 Urban Design Games
In the 1970s, RAND used their insight with war games to develop games for urban
planners. These games were developed to tackle social issues that often came up in planning
such as public participation and community development. Because these were tangled with
technical systems such as transportation and land use, games became a useful tool to attempt to
quantify these societal values [3]. Urban design games in the 1960s focused on using large-scale
urban planning models (LSUM) to simulate different policies. The urban planning community
did not find great results in this analysis, highlighted by Douglas Lee's Requiemfor Large-Scale
Models. He argued that computer modeling had innate limitations and that modelers could not
possibly use rational behavior to quantify all social issues within systems [22].
-
20
-
Figure 4: Screenshot of SimCity
Urban design games responded with Will Wright's SimCity. As one of the most beloved
franchises in all of gaming, SimCity brought urban planning to the masses using the principles
developed by Jay Forrester to create the field of system dynamics. These types of models and
simulation games inspired by SimCity and system dynamics were used to predict issues with
energy and ecology. SimCity and its successors continue to bring urban planning to schools and
homes around the world, bringing with them a complex, technical system combined with the
ideals of players about what types of places they would like to build and live in [3].
2.2.3 Educational Games
Educational games are arguably the most well-known type of serious games. In the 1980s
and 1990s, educational games were produced as personal computers began to infiltrate
classrooms around the country. Also known as "edutainment", educators aimed to draw in their
students by introducing game elements into the learning process. While earlier software focused
on automated drills and instructions, these games pulled from arcade culture to "sugar coat" rote
learning tasks. Game play is presented as an external reward to hide the learning; however, the
main gameplay comprises the same drills and instructions that the classroom traditionally uses.
Successful games in this genre include Number Munchers, Oregon Trail, Reader Rabbit, and
Where in the World is Carmen San Diego [23][24]. As these games drew criticism for simply
repackaging old education, educational scholars such as Kurt Squire have begun to champion
games such as Civilization for including educational content as part of the core gameplay
systems [25].
-21-
,
Li
,6.)
ortuiN
Figure 5: Screenshot of Sid Meier's Civilization III
In Civilization, players are exposed to ideas about what factors go into a successful
civilization. For example, cities on rivers enjoy benefits to their trade capabilities in-game, which
reflects how real life cities grew around important tributaries such as the Nile, the Seine, and the
Thames. These principles have ushered in a new set of educational games that draw on the
strengths of gameplay, based off of recent literature that determines why games are effective in
teaching players new skill sets.
James Paul Gee's What Video Games Have To Teach Us About Learning and Literacy
introduces learning principles gleaned from games and how they can be used to improve
classroom plans. These focus on semiotic domains, which is a set of signs such as symbols and
images which describe a complex system. As an example, the classic video game Super Mario
Bros. includes items such as coins and flowers. These do not represent coins and flowers as they
are in real life; players learn to recognize coins as score-increasing collectibles and flowers as a
-
22
-
tool to grant Mario the ability to shoot fireballs. This learning is facilitated by various principles,
some of which are excerpted below from Gee [26]:
- "Psychosocial Moratorium" Principle
Learners can take risks in a space where real-world consequences are lowered.
- Identity Principle
Learning involves taking on and playing with identities in such a way that the learner has
real choices (in developing the virtual identity) and ample opportunity to meditate on the
relationship between new identities and old one. There is a tripartite play of identities as learners
relate, and reflect on, their multiple real-world identities, a virtual identity, and a projected
identity.
- "Regime of Competence" Principle
The learner gets ample opportunity to operate within, but at the outer edge of, his or her
resources, so that at those points things are felt as challenging but not "undoable."
- Probing Principle
Learning is a cycle of probing the world (doing something); reflecting in and on this
action and, on this basis, forming a hypothesis; reprobing the world to test this hypothesis; and
then accepting or rethinking the hypothesis.
- Discovery Principle
Overt telling is kept to a well-thought-out minimum, allowing ample opportunity for the
learner to experiment and make discoveries.
These principles are useful when considering not only how to design educational games,
but other genres of serious games where the goal is to teach players about the system in order to
transfer ideas or to explore the underlying model's behavior to novel system inputs.
-
23
-
2.2.4 Management Games
The Beer DistributionGame was developed at MIT's Sloan School of Management in
the 1960s to help teach managers about their flaws in how they perceive systems such as the
supply chain [27]. This game highlights the problem of the bullwhip effect, nested in a more
general lesson about system thinking. The bullwhip effect refers to small fluctuations in
upstream demand that are amplified downstream. This concept is not easily understood because
it challenges superficial ideas about how supply chains work. Games excel in revealing system
concepts and relationships that are not obvious at first glance. Companies have used management
games such as the Beer Game to combine the best parts of war games and educational games;
corporate training incorporates the lessons and strategy and tactics when considering business
decisions over time while also taking principles from educational games to teach new skills to
employees. Serious games are especially useful in employee development as they offer some
quantitative assessment. Businesses are concerned that the time their employees spend in
corporate training should be fruitful. They aim to develop their employees' people,
communication, and strategic skills. These sets of skills are not always quantifiable, but the game
does provide feedback mechanisms for players to track their own growth in abilities [28][29].
Below is a table of modern business games and what skills they aim to teach:
-
24
-
AMdiadnmatSkRa
GameNaw
COSIGA
X
x
x
GLOTRAIN
LOGTRAIN
ONE
PRAE
SHARE
X
x
x
x
SPKO
X
X
X
X
x
x
x
EIS
Snilb
X
X
GloVEd
Simbu
Bewrgame
x
KUS,
x
MINT
anin
TAC-SCM 07
X
x
TAC-SCM 07
(Caoins)
x
MARGAhndiUy
TOP Simg
X
Top singlobal
Top uuprc
Wet
Top scinsistir
FROST
Mupg Saice
Debta desig Sme
E
E
E
x
x
E
E
x
x
x
x
x
x
x
E
E
E
x
x
E
x
x
E
x
x
X
x
x
x
x
x
x
x
x
x
x
x
x
E
x
x
x
x
x
x
X
x
x
x
E
X X
D
X X
D
X
D
X X
D
X
E
E
D
E
x
X X
X
X
X
x
--
puisti
E
E
E
x
x
x
x
E
x
x
x
X
X
x
M
x
x
Pcess
mgt
x
X
x
GAMdE
p
-waX
x
x
x
SUPPLYNET
Citycar
REFQUEST
Binumz
E
E
W.e
Figure 6: List of serious management games and what skills they aim to teach [28].
-
25
-
2.2.5 Policy Games
Policy gaming focuses on the "chaotic and messy" theory of policy making. This theory
draws from models such as those from Graham Allison's 1971 book Essence of Decision:
Explaining the Cuban Missile Crisis. Allison details the bureaucratic model of politics where the
values of individuals lead to a negotiated decision where consensus is drawn from a group of
leaders [30]. Other important models include the garbage can model and the policy stream
model. These models focus on separate tracks such as the problem, policy-making, and politics
coalescing at a given time to form a window of opportunity. Because policy making occurs in a
dynamic arena, games help reflect this complex system where several values have to be
negotiated along with interpretation of technical facts. While policy gaming grew alongside the
roleplay negotiations in war games, electronic games have become more useful as policy making
tackles technological systems with layers of social complexity [3].
Mayer puts forth three primary lines of reasoning for why electronic games are
becoming favored for policy analysis. First, he argues that there is a cultural coming of age of the
"digital natives" that grew up with computer and video games. This generation of students has
interacted with games such as SimCity and will be familiar with these complex system models.
Modification has become a common activity in these games. These students have created their
own technical models to test theories and accept the idea that games can be used as a digital
analog of real world systems [3].
Second, this level of technology has become readily available. Common tools such as
smartphones are powerful enough to create virtual models. Game companies are developing
partnerships with research institutes and private organizations to help them explore systems and
teach the public about important issues.
Finally, video games have reached a level of public acceptance as an expressive medium.
People now recognize that games can be loaded with message. The rhetoric carried depends on
the user experience; because it is experiential, players are more likely to believe the message
than if someone had simply told them the lesson.
-
26
-
2.3 Naval Acquisition
American shipbuilding is shaped by the Merchant Marine Act of 1920, commonly known
as the Jones Act. This requires that all goods transported by water between U.S. ports must be
carried by ships constructed in the United States and manned by U.S. crews. The act has helped
American shipyards sustain growth throughout the years. Growth was also led by a sudden
proliferation of U.S. shipyard activity during World War II as part of the liberty ship campaign.
These liberty ships were cargo ships that were produced in high quantities at fast rates. After the
war ended, there was a sudden excess in capacity and inventory. The shipbuilding market has
slowly dwindled since as domestic shipbuilding has slowed outside of government contracts
[31][32].
Several shipbuilding studies have highlighted the performance of American shipyards
lagging behind those in other countries in terms of productivity. Because of the Jones Act,
remaining shipyards enjoy a protective policy that keeps a set amount of work nationally. This
limits their motivation to improve their infrastructure and adopt certain best practices overseas.
Specifically, shipyard contracting with the government has changed in the 1970s and 1980s.
Prior to this age, shipyards brought their own designs to government bidding and put an
emphasis on finishing a design before submitting their proposals. This stabilized the shipbuilding
process and kept prices low. As the technological pace has accelerated, government contracting
has adopted a policy of partially finished designs when signing contracts [33]. Because the
lifetime of a ship program spans well over a decade, the ship design is unstable and can change
from ship to ship. This leads to high cost overruns when dealing with unproven technology and
change propagation in complex, technical systems [34]. At present, the rate of inflation in
shipbuilding is outpacing the national rate of inflation, as shown below.
-
27 -
Continued
Box 2.
intlat 1 1
gl
hipb
din
AFuma s of
atm
Mocn
ete
infion
6
4
0
19U
]LM
INS
199
1M
1997
2MS
2M
236
9B
2M
201S
Figure 7: The annual rates of shipbuilding inflation is outpacing the general rate of
economic inflation [31].
The current shipbuilding environment relies on two companies that own all five of the
first-tier shipyards, General Dynamics and Huntington Ingalls Industries. General Dynamics
owns the National Steel and Shipbuilding Company (NASSCO), Electric Boat, and Bath Iron
Works. They receive 70% of their revenue from government contracts. Huntington Ingalls
Industries comprises Newport News Shipbuilding and Ingalls Shipbuilding in Pascagoula. They
are responsible for nuclear aircraft carriers, submarines, and surface combatants; their revenue
comes virtually only from government contracts because they were spun out of Northrop
Grumman to handle government shipbuilding contracts.
Because foreign shipyards have competitive advantages, there are high barriers to entry
for new competitors in this market. The country cannot afford to have the industries suffer in
case ships need to be developed and produced to meet a strategic effort. These two factors have
led to a relatively secure position for the current first-tier shipyards. The question is what else
can be done to encourage improved defense contracting?
-28-
233
2.4 Research Motivation
The research was motivated by two related gaps in the literature. The shipbuilding
industry is interested in revealing how cost pressure may be passed on to the shipbuilders and
suppliers. We pursue the possibility of reducing overall cost pressure by optimizing the
negotiation process or by creating a tradespace with the potential for more fruitful contracts for
both sides. The policy game literature is still generating ideas on how to use games as an
accepted methodology for policy research and analysis [16] [35]. Games are being increasingly
recognized for their ability to open up the inner workings of technical models, but decision
makers are still unsure how well they can trust the results from repeated play sessions [3]. We
interpret the data from our game sessions to highlight the use of games as a descriptive,
exploratory tool, rather than a prescriptive tool that provides specific, technical details.
The research goals are to identify trends in the negotiation process that could provide
insight on either the specific issues in the contracting process or on how the contract moves
through the tradespace. We explore two scenarios:
"
A sole-source case (one buyer, one seller)
*
A competitive case (one buyer, two sellers)
This allows us to visualize how the process changes between environments. We consider the
competitive case as an analog to the hypothetical scenario where a second-tier shipyard receives
investments from the government. By becoming a first-tier supplier, they may be able to provide
lower costs over the long run to offset the costs of investment. Additionally, by creating more
shipyards, the government expands their base of knowledgeable, experienced builders. The
expanded shipbuilding base will be capable of responding more effectively to any required ramp
up in production. In the current sole-source environment, some of the major shipyards are
operating under maximum capacity in order to guarantee work over a sustained period of time
and keep their workforce skilled. The research aims to identify the effects of removing this
practice by producing a competitive base.
-29
-
-30-
Chapter 3 - Model Design
In this section, we identify the three primary components of the game: the underlying
model of contract negotiation, the game mechanics, and the game scenarios. The model was
created by identifying key drivers for the stakeholders and mapping their relationships and
tensions. We conducted interviews with members of both sides of the negotiation table. To
understand the shipyard's viewpoint, we consulted contracting officers and directors at shipyards
around the country. These members represented the Pascagoula shipyard owned by Huntington-
Ingalls, the NASSCO shipyard in San Diego owned by General Dynamics, and Vigor Shipyards.
This set of interviews gave us the perspectives of major shipyards in sole-source and competitive
cases as well as the perspective from a lower tier shipyard who has participated in naval
contracting in the past. The Navy's perspective on the negotiation process was elucidated
through conversations with program managers in the Navy Yard in Washington, D.C., as well as
from the members of the naval acquisitions office in the Pentagon. These groups offered a host
of experiences from both civilian and military backgrounds and covered a range of programs
from the VIRGINIA class submarines to the Littoral Combat Ships.
We asked both sides the following questions:
- What are your primary drivers and goals in negotiation?
- What issues do you believe the other side is not addressing?
- What information do you believe the other side is hiding?
From this, we identified the key stakeholder positions as seen below as part of our
underlying model of negotiation.
-31 -
Stakeholder
Congress
Navy
Motivation/Concerns
*
National Security
-
Optimal resource allocation
-
Constituency
*
National Security
o Technological Innovation
o Fleet Size
o Industrial Capacity
* Full Funding for Projects
* Efficient use of available funds including:
o Level of Quality
o Vessels delivered on Schedule
- Maintaining differentiated value from other services
Shipbuilders
- Stable workload
Differentiated capabilities from competitors
-
Maximize value to shareholders by maximizing
sharevalue
o Profit = Revenue - Cost
a
-
To tProfit, either tRevenue or 4Cost
Reputation
o Leads to increased Revenue by winning bids
Figure 8: Key stakeholder drivers and positions in the negotiation process.
We then laid out a basic set of targets for the data collection to occur as smoothly as
possible. The target audience would be both experts at shipyards and in the Navy. This meant
that the game would have to cater to a wide range of expertise about the shipbuilding process.
The game would have to be playable within a time frame of ninety minutes in order to fit into the
schedules of experts and students alike. This meant that the scope of the model would have to be
scaled back for players without experience to comprehend the game scenario in a short time. The
game should be easily distributable in order to minimize time commitments.
-
32
-
To overlay game mechanics on this model, we used the triadic game design framework
created by Casper Harteveld [19]. This balances the reality, meaning, and playability of the game
to create an experience which is engaging, but produces meaningful results because the game
mimics the actual system effectively enough.
a
PLAY
DESIGN SPACE
POOM
;0000
RFALTY
MEANING
(van""
mscwL#msf~N
L /
-
Figure 9: Triadic game design framework
These allowed us to establish an appropriate scope for the model. The game mechanics
and dynamics were then built to simulate the negotiation process. Case scenarios were then
overlaid on top of this game system to focus on the research questions. The creation of the
research game ColumBID is detailed below, as seen through the different trade-offs considered
between the goals of the triadic game design framework.
-33-
3.1 Reality
Every game is based off of a model of reality which allows players to act intuitively with
the game system. The players' ability to understand the game's mechanics and dynamics is
affected by how well they can accept the relationships between components of the game's
system. The theory of constructivist learning explains that this understanding occurs because of
mental representations being as meaningful as their physical representations. Jean Piaget is
widely regarded as an important figure in constructivism and believes that the learning occurs
through interaction when actions on mental objects produce results in a way that can be
understood by the learner. His theory of learning is based off of the key principles that
knowledge is built by the learner, not passively from others, and that this learning is experiential
[36]. Ernst von Glasersfeld takes these ideas and moves it to radical constructivism. He argues
that this internal system understanding is fundamentally different from the repetition of
behaviors, which he calls "training". Because students learn through experiential methods, it is
more interesting to view their errors [37]. Games provide an interesting avenue for experiential
learning as they often give players the ability to fail and challenge mental constructs.
According to Hewson and Thorley [38], these challenges to knowledge must satisfy three
conditions. The new ideas must be intelligible, plausible, and fruitful. The learner must be able to
comprehend what this new idea means in order to repeat it in the future. The verisimilitude of the
idea allows the learner to believe that the interaction can be applied to reality. This idea must
provide a novel interaction for the learner to develop the new piece of knowledge.
The game's depiction of reality has effects on all three components of constructivist
learning theory. For players to understand the system, the player must be convinced that the
virtual world is sufficiently comparable to the real world to understand what their actions are in
the game. The scope of the game world should allow players to assume that the actual system
could behave in this manner. Finally, the insights developed should have some ability to be
tested in the real world.
The "reality" of the game is a model, much like one used in any other scientific test. The
model can be analyzed using three criteria: flexibility, fidelity, and validity. Flexibility
determines how well the model holds up to changes in the system parameters. Fidelity analyzes
the visualization of the system in-game. Validity discusses the underlying system principles and
-
34
-
how well the virtual model interactions simulate those of the actual system.
3.1.1 Flexibility
The flexibility of the system refers to how well the system can adapt to new parameters.
The system should be able to behave in a believable manner in the current scenario. However,
the model may be versatile enough to react to changes in the game's assumptions and still
behave in a realistic fashion. The functional decomposition of games includes the usage of
games as a laboratory to explore unknown situations [39]. This is one of the advantages of using
games as a research tool; they may be used to gain insight into scenarios that are unreasonable to
explore through normal testing. The hypothetical case may be too large in scale of either time or
money.
From parallel research in the study, we found that contracts could be assigned as either
fixed-price or cost-plus type in both sole-source and competitive cases. The game should then be
designed to handle both of these scenarios as well as adopting other levels of model detail. We
decided to focus on fixed-price contracts as they would be easier to explain in a short period of
time compared to the cost-plus contracts [40]. The key stakeholder drivers may not be present in
the game in all cases, but there is enough flexibility to incorporate these features as desired. The
particular drivers included in the game are discussed below in the Validity section.
3.1.2 Fidelity
Fidelity in a game system typically refers to the graphical fidelity. A game usually should
have enough fidelity to prevent players from being distracted by how different the simulation is
from reality. However, there was a key constraint in the game development which focused on the
ability to conduct long-distance gameplay sessions with experts. Because the target audience
included contracting officers at both shipyards and in NAVSEA, the game would have to be built
as simply as possible. Due to the nature of government computers and the time limit imposed on
the game, the graphical fidelity was scaled back. Rather than utilizing a game engine to produce
graphics for fidelity, ColumBID relies on a written scenario to convince the player that the
situation is realistic enough.
-
35 -
The scenarios are based on the LCS program [31]. The final costs in the game were tuned
from several play test sessions throughout the summer and fall. The build rate and total years
available for negotiation were designed to mimic the length of a generic ship program; a class of
ships is typically built for more than a decade. Ten years in ten ships allows for players to
experience some of the learning effects in shipbuilding and uncertainty over long stretches of
time. These ships are built over two blocks, which are commonly used in defense contracting to
split a program up into different contracts to allow for re-negotiation as the program is evaluated
every few years.
3.1.3 Validity
Validation is the most important component of the reality framework when dealing
specifically with research games. This ensures that the data provides some meaning to both the
research team as well as to the players. Without a valid model, the research team would be
unable to transfer lessons learned from the gameplay sessions to the real world model [19]. From
the players' standpoint, they must believe the model behaves with reasonable precision and
accuracy in order to immerse themselves in the game world. This allows them to act without
concern for breakdowns in the model and accept that discrepancies in their mental models of
how the system should work are flawed, rather than blaming the model for being incorrect. This
validation requires a certain amount of detail; the particular scope desired changes depending on
the composition for the testing group. We aimed to reach a level where players with contracting
experience would recognize the system and be able to provide insights from their past
experiences, without being distracted by key details left out in the scenario.
To validate the model, stakeholders were consulted during model creation, scenario
design, and data collection. Assistance outside of the MIT study came from the shipyards,
NAVSEA and SCRA. They prioritized the model's ability to capture the cooperative aspects of
negotiation as well as the issues with uncertainty in future material costs. Program managers in
NAVSEA provided historical ship and program schedules as baselines for the scenario. Pilot
testing with players who were unfamiliar with the situation showed that they were not distracted
by gaps left in the model. The model does not currently incorporate technology changes as years
pass because of the complexity in the system. Learning effects are simplified to an initial lead
-
36 -
ship costing more time and money. The budget is assumed to decrease exponentially according
to an internal rate of return.
3.2 Meaning
When constructing the game's mechanics and dynamics, the overarching research
questions must be kept in mind. The game's actions should allow for a set of data from which
meaningful results may be interpreted. In ColumBID, the game's systems are designed to allow
the negotiation process to be studied. By creating two primary scenarios, sole-source and
competitive environments could be compared. The scenarios were built from historical data of
the LCS program. This section highlights how the goals of the study affected the game design
and how we incorporated the additional constraints in the research process as part of the game
design.
3.2.1 Purpose
ColumBID was selected as our research tool of choice because it could incorporate the
human elements into a technical analysis. Bots and van Daalen offer a functional decomposition
which reveals how games can support this analysis [39]:
-
37 -
Game as a laboratory:
Research and analyze
Game as a design studio:
Design and recommend
Game as a parliament:
Game as a practice ring:
Clarify values and arguments
Provide strategic advice
Game as a consultative forum:
Democratize
Game as a negotiation table:
Mediate
Figure 10: Schematic representation of functions of games for supporting policy
development [39].
ColumBID focuses on three of these functions. As a laboratory, the game is designed to
research theoretical scenarios. The boundaries of the game are not restricted and will still
respond to input values that are outside of the initial scenario. This makes the game useful in
exploring hypothetical cases such as a competitive market in current sole-source environments.
The game is also useful in analyzing patterns in the contract negotiation. Functioning as a design
studio, the game collects time-stamped data on the proposed contract prices and build orders.
This data can trace the negotiation process as it evolves and can be used to see how the players
move through the trade space.
Finally, the game is useful as a parliament. Because players are not initially aware of
each other's goals, the process can be used to clarify how each player interprets the system.
People placed in a similar role may not value all goals equally. With a game, their relative values
can be captured in some sense as key tradeoffs are identified.
3.2.2 Strategy
For ColumBID to be useful as a research tool, we looked towards established game
designs with similar goals. Through the efforts of groups such as the GAMBIT and its successor,
-
38
-
the MIT Game Lab, we were able to draw inspiration from local serious game efforts. We looked
at two efforts in particular: The Mercury Game from the MIT Science Impact Collaborative and
Movers and Shakers from the MIT Game Lab.
The Mercury Game was designed "to teach people about the role of science in
international environmental policy making." It was made to be playable in 3-4 hours for students
and decision makers in groups of 9-11 players. To create the scenario, the design team for the
Mercury Game created a set of roles which approximated real world stance on scientific
problems. Four primary issues form the heart of the negotiation scenario, with each role being
given a set of finite positions to take on each issue. The game's dynamics follow a moderator as
they move from opening statements to negotiation to final agreement [41].
The form of ColumBID was built in a similar manner. With the stakeholder matrix, the
roles naturally took shape as the national government and commercial shipyards. Congress was
moved to a set of parameters that would shape the negotiation process, specifically the national
budget and goals for the government player. The researcher functioning as moderator and game
master would be responsible for interpreting the role of Congress to the players. A similar set of
issues were designed as two variables: the build order of the ships and a contract price. Rather
than giving players a finite set of options, the variables were left as open choices with any range
acceptable if both players could agree upon it. This makes no statement about if the budget can
be violated or if the shipyard is able to operate at a loss; this freedom was important to see if the
value system imparted by the role briefings could be flexible.
These role briefings were crafted to represent the key drivers for each player.
Government players were ordered to value flexibility while keeping the ship prices as low as
possible. Historical profit values from the two major shipyard companies, General Dynamics
[42] and Huntington-Ingalls [43], were used to generate idealized profit margins in the shipyard
briefings. In the sole-source case, both sides were told to value preserving the capabilities of the
work force. Finally, the briefings gave information about various expected costs to both parties.
To assist in the negotiation process, two types of spreadsheet tools were designed. One is a
private tool which contains the cost information in the briefings. This sheet is designed to ease
the level of arithmetic needed during the game. The other sheet is a shared spreadsheet between
each government-shipyard pair. This allows players to share proposals without giving away what
their private tools hold.
-39-
Movers and Shakers was designed by the MIT Game Lab [44]. The design group was
headed by Dr. Konstantin Mitgutsch, who calls Movers and Shakers an example of a subversive
game; this is a type of serious game that focuses on challenging players' expectations. Movers
and Shakers focuses on leadership dynamics by placing two players in charge of workers in a
fantastic office setting. In the game's fiction, the planet rotates because of workers operating a
machine deep underground. Two players are assigned the role of office managers and must
position their workforce to properly motivate their workers. One player sees how well their
workers communicate and must organize them so that they are seated with favorable
relationships to neighboring workers. The other player views the workers' productivity and
positions them to obtain an optimal work rate. However, both players are viewing the same
workers and must position them simultaneously. The game transitions from a competitive game
to a cooperative game as players realize they must work together to reach their common goal
[45].
Similarly, ColumBID highlights the fact that negotiation is ultimately cooperative
between both sides to reach their goals, particularly in the sole-source case. Just as Movers and
Shakers uses asymmetric information to teach players that a mutual goal is seen through two
different lenses, ColumBID offers different interpretations of the scenario depending on their
role. The government sees a singular, best-guess for ship costs, while the shipyard is given a
different baseline estimates including best-guess, best-case, and worst-case. This draws players
to notice discrepancies in their proposed costs; they are unable to reconcile the gaps if they do
not understand the overhead costs and risk judgments present on one side. The risk also leads to
a value judgment by the shipyard player. Because of this, repeated play sessions with different
players leads to new asymmetries of information.
The presentation of the game uses a game engine called Twine. While normally used to
create interactive fiction such as text-based adventures games, Twine possesses the ability to host
the game on a single web space for easy distribution. The Twine interface links pages of text and
links to each other in a format similar to a choose-your-own-adventure [46]. The Twine interface
for ColumBID is displayed below:
-
40 -
#0W3p1
-Tv
Figure 10: Twine interface for the game designer.
In conjunction with Twine, the spreadsheet tools were designed on Google Drive.
Because of the auto-save features in Google Drive, the negotiation's evolution would
automatically be recorded as the players proposed to each other on the sheets. This occurs for
every event when a cell is not edited for a period of around ten seconds. This allows the tool to
capture time stamped data about the current proposal's prices and build orders. The shared
functionality works because Google Drive updates in real time; spreadsheet sharing would
otherwise be impractical without the use of a federated system to create an online, multiplayer
environment. This also simplifies the game requirements down to only an internet connection;
they would not need to open these tools in a separate program.
Pre-questionnaires and post-questionnaires finish the game's basic design. These
questions reveal the players' school and work experiences in similar situations. They are used to
help identify any effects of prior knowledge on the players' experiences with the game. The
-41-
testing sessions concludes with a debriefing with the players about the goals of the research and
their experience with the tool. The debriefing is important to find out about player strategies and
which components of the model they understood. As an experiential method, game research
requires post-game discussion with the players to identify drivers for their behavior throughout
the play session [16]. While playing, people may not have time to properly reflect about why
they chose to do a certain action. As explained by Schell [14], "How can we observe our own
experiences without tainting them, since the act of observation itself is an experience?" This
problem is referred to as the Heisenberg principle, in reference to the quantum theory principle
that poses a related issue about observations and measurement. Current methodologies use
debriefing sessions to allow players to dive into their memories. Combined with silent
observation by the researcher, this analysis provides the best currently known way to reveal a
player's motivations at a given point of time in the game.
3.3 Play
This section details how each game system ties to the research purpose through a type of
"play". This draws heavily from the frameworks of Richard Bartle and Roger Callois on
different types of players and the experiences that they seek.
Richard Bartle created the first multi-user dungeon (MUD), an interactive, multiplayer
role-playing game. Bartle identified four types of players who enjoyed these text-based games:
killers, achievers, explorers, and socializers. Killers enjoy using the game's mechanics to disrupt
the experiences of other players. Achievers are players who pursue maximum attainment of their
goals in the game system. Explorers are interested in analyzing the game's systems to figure out
how the mechanics and dynamics work. Socializers look to communicate with other players
within the game's particular context, taking on a roleplay aspect [47]. As a game rooted in textbased briefings with simple mathematical aids, ColumBID appeals to these different player types
in ways similar to a MUD. Through iterative playtesting, a variety of players provided insight
into how the design team could make the game engaging to a general audience.
In Man, Play and Games, Roger Callois classifies play into four fundamental categories,
known as ag6n, alea, mimicry and ilinx [48]. His terms correspond to competition, chance,
simulation, and physical play, respectively. ColumBID focuses primarily on competition and
-
42
-
simulation with elements of chance. Salen and Zimmerman further discuss these elements as
different schemas of play [15]. The aspects of competition and simulation are further discussed
in the schema for analyzing a game as a system of conflict and as the play of simulation.
ColumBID also engages players as a system of information and as a game theory system. This
section identifies how the gameplay may be analyzed through these different frameworks.
3.3.1 Systems of Information
Games allow for imperfect information to be explored. Similar to how imperfect
information is explored in Movers and Shakers, each player is given a different set of statistics
about the current scenario [45]. They are also not told specific information about what each other
player holds. The naval side is given a best estimate on the costs per ship, while the shipyards are
given a best-case and worst-case scenario in addition to the most likely scenario presented by the
Navy. Both sides are given approximate annual budgets allocated by the Navy. Players are given
freedom to reveal as much of their private information as they feel is useful.
This creates an economy of information within the game, where players are assign values
to key pieces of knowledge. The negotiation process slowly reveals information to all players,
but this can be obscured if not every player is truthful. Information in the game is also
manipulated by false knowledge created by one side. This gameplay system is known as
objective information against perceived information. Objective information refers to the true
structure of information asymmetries in the game, while perceived information is what the
players observe as they play. Players then have a choice between trading value for their game
objectives or more information towards their goals. In the negotiation, this means that they can
make a trade-off to improve their position on the contract or reveal part of their private
information.
Celia Pearce breaks down information by who knows it. Her four cases are information
known to all players, information known to only one player, information known to the game
only, and randomly generated information [15]. In ColumBID, there is no randomly generated
information. The hypothetical risk resolves to the same answer so that the scenario is consistent.
With the block system, the players are able to move information from private to public if they
choose to agree on a partial order fulfillment. After one block, the players learn about the actual
-
43
-
shipbuilding costs as well as the real profit margins made by the shipyard. This reduces
uncertainty by revealing the specific scenario. In gameplay, this presents an interesting tension
because a player may not want to resolve the first block if they believe they have made a great
deal. Imperfect information provides gameplay tensions which reflect the reality of negotiation
sessions; information has value, both in learning new information and in denying information to
the other players.
In playtesting, these information asymmetries are especially important to Bartle's killers
and socializers. They are more likely to use information as a leveraging tool to reveal how the
game system works. Because the information is shared primarily through social interaction, these
roles are likely to enjoy conversing with other players as part of their game strategy. While
killers are likely to engage in manipulation of information through feints and bluffs, socializers
may value information more as part of the negotiation process instead of becoming engrossed in
the fight over numbers [49].
3.3.2 Game Theory Systems
Game theory revolves around rational players who decide strategies simultaneously to
maximize their utility functions. In the monopolistic case, ColumBID resembles the theoretical
Chicken or Hawk-Dove game. This is envisioned a game where two players are facing a
catastrophic outcome if they stay on their current course. However, this course is also the best
way to reach their goals. One or both of them must decide to divert to a lower strategy to avoid
the worst-case scenario. In the monopolistic case, the shipyard and government may both take
strategies maximize their utility functions by suggesting contracts that heavily favor them. The
difference between their offers may be too high to negotiate and they may be unable to come up
with a contract, causing the government to not have any ships and for the shipyard to not have
any work. This is only resolved if one side acquiesces and makes major concessions to the other
[50]. In formal game terms, three of the four outcomes result in a zero-sum:
Aggressive Strategy
Concession Strategy
Aggressive Strategy
-x,-x
y,-y
Concession Strategy
-y,y
0,0
Table 1: Strategies for the chicken or hawk-dove game
-
44 -
In ColumBID, there are important differences from the game theoretic model. The players
are not assumed to be rational. While game theory allows for players to bluff, it does not assume
that players can be "vindictive, forgetful, self-destructive, or lazy" and deviate from a utilitymaximizing function. Players are not necessarily making simultaneous decisions; one player may
choose for an initial offer to come from the other side. This reveals player strategy to the other
side, allowing them to make an informed counter-offer. Utility-maximization differs among
players; each person may rank the various goals differently and may bring outside knowledge to
create an entirely different set of goals.
The game theory model does allow for two important parts of ColumBID's design. As an
economic model, it allows for researchers to imagine negotiation as a decision tree and analyze
the final contract as an outcome of successive, iterative deals. As a game design model, the
designers may avoid including a degenerative strategy in the game system. A degenerative
strategy is always the best choice and results in an unsatisfying game; once this strategy is
discovered, there is no uncertainty left in the gameplay. In game theory, the best strategy is
known as the Nash equilibrium. Formally, there are multiple Nash equilibriums in the Chicken
game, not to be confused with a mixed strategy. This means that each game will have a single
Nash equilibrium as a non-coordination game; one player should make a concession and the
other should remain in a strong position. There is no formal way to resolve which player should
take which position. This avoids a degenerative strategy and creates interesting gameplay [15].
A research objective is to map the negotiation space and study how the negotiation moves
over time. Game theory says that players should pursue Pareto dominant deals to current
proposals. It is commonly held that players will only accept moves that improve both of their
positions until they reach the Pareto frontier. At this point, they will then slide along the frontier
until they reach an agreeable outcome. By creating the game model with this in mind, the study
will be able to examine how well game theory holds up with respect to Pareto dominant moves
to the frontier. Bartle's achievers are the most important playtesters with respect to this gameplay
schema. Since these players naturally aim to maximize their outcomes, they will agree with game
theory's predictions if they accurately perceive the situation.
-
45
-
3.3.3 Systems of Conflict
In any multi-player game, conflict is at the heart of gameplay. ColumBID is both a
cooperative and competitive game. While each player cannot achieve their goals without the help
of the others, they also cannot reach all of their goals without others' concessions. Just as in
Movers and Shakers, the research game appears to be primarily competitive from the outset.
Through iterative playtesting, the design team included several subtle components to help guide
the players to this idea. Players are assigned teams and are seated facing each other. Their
briefings create an external pressure in the form of corporate stockholders or Congressional
members to increase the illusion of having a side to take. The goals of the game are easiest to
reach when players decide to cooperate and reveal information truthfully. The game shifts to a
cooperative phase when the players realize that it is possible for everyone to succeed. Then, the
players work together to maximize their goals in a single system instead of separate systems that
share common, scarce resources.
Callois considers competitive play to be fundamental to game engagement. By providing
an equal field, games allow players to engage with one another under similar rules. This creates
the illusion that the winner won because of their skill instead of any external factors outside the
"magic circle" of the game [15] [48]. This opportunity at a fair contest is what draws players into
competitive play. Although ColumBID does not provide a level playing field for all roles in the
game, it does provide a structure where participants believe they can achieve their goals. By
providing a set of criteria to evaluate players' performance, the game provides necessary
feedback to the players on how well they are doing in the game. These reference points are vital
to guiding the players towards a satisfactory negotiation. The alternative would be to have
players unsure of what they are doing initially, with no feedback to place them on the right track.
The competitive aspect of ColumBID engages players and keeps them moving forward in the
negotiation process.
-
46 -
3.3.4 Play of Simulation
Salen and Zimmermann define play as "free movement within a more rigid structure" to
frame games through the experience of engaging with them. In a computer model, this section
focuses on the interactions of players with a model that simulates a system they are
knowledgeable about [15].
Transformative play refers to the structure being changed due to playful interactions. In
the model, the boundaries are governed by a set of rules described in the briefing. These include
the number of ships, the usage of blocks, the number of years, and the allowable budget. In the
game, players are allowed to violate any of these rules whether through the introduction of
outside information, the exclusion of these rules with justification, or as part of the negotiation
process itself.
Roger Callois classifies this simulation play as "mimicry" where players take on a
different role and behave appropriately. Players are drawn in by the opportunity to pretend to be
someone else. These players would fall under Bartle's socializers and explorers. They are
interested in the roles given to them by the game and want to explore how the system behaves
from their perspective [48][49]. Combining mimicry and transformative play allows for the game
to explore novel scenarios and consequences of changes in the system parameters. Players
change the rules of the game if they either feel it will provide an advantage as they play or if
their sense of reality motivates them to justify a rule change. This transformative play allows for
the exploration of "unknown unknowns", as described by Donald Rumsfeld. The system may be
changed to a situation that has no real-world analog [21]. Through this simulation, players are
drawn in by the chance to immerse themselves in a new role and partake in novel situations.
3.4 Final Design
ColumBID focuses on a single negotiation, which splits a contract into one or two blocks.
After a pre-questionnaire, players are given a briefing which details one of two scenarios. The
game features either a two-player game that simulates a monopoly or a three-player game that
simulates a duopoly. A screenshot from this briefing is shown below:
-
47 -
Figure 12: A screenshot of the briefing page of ColumBID.
Players are given a set of spreadsheet tools to assist them in the negotiation process.
These tools take one of two forms: they are either a private tool designed to provide useful
information and ease the level of math required, or a shared tool to propose offers to the
opposing side. These are displayed below:
-48
-
Block I
Internal Rate
of Return:
Proposed build
order:
Year from
start of block
1l
1
(All prices are in millions)
Estimated budget
for block 1:
$450
(A blank ship count means no production; A zero means ships are
being built but none are delivered.)
Estimated Cos t
Estimated Cost
Estimated Cost
(Worst-case
(Naval-guess
(Best-case
#
scenario)
scenario)
Ships scenario)
$100
$100
$100
0
$727
$6821
$636
1$744
$5791
$496
$75
$75
$751
$68
$68
$68
$62
$62'
$62
$56
$561
$56
7
$51
$511
$51
8C
$47
$47
$47
IC
S.4
$421
$42
10%!
#Total
# Years
_
Expected
Ships Block 1 Cost
$1,6$35
3
3
$1,763
$588
$588-
$545'
$545]
$1,973 Total
$658 Iear
$658, /ship
First Block: $1800 million for 3 ships over 3 years
Proposed
$1,800
contract price:
$600 /year
$600 /ship
$371
$165'
2.12%
10.12%1
Expected Block
-$173 Profits
Expected Block
-8.79% Profit Margin
Both Blocks
Combined:
-$539
-$4101
-23.03%
-18.57%'
-49
-
-$749 Expected Profits
Expected Profit
-29.39% Margins
SHIPYARD
PROPOSAL
NAVY
PROPOSAL
Block 1 cost:
Block 2
cost:
Block 1 cost:
Block 2
cost:
Block 1 build
order:
Block 2
build
order:
Block 1 build
order:
Block 2
build
order:
Ships
delivered
Year #
Ships
delivered
Year #
Ships
delivered
Year #
Ships
delivered
Year #
1
1
1
1
2
2
2
2
3
4
3
4
3
4
3
4
5
6
7
5
6
7
55
6
7
6
7
8
8
8
9
9
9
8
9
10
10
10
10
Figure 13: Spreadsheet tool interface seen by the players
The sheets accept two inputs: a build order and a proposed price. The shipyards are
granted three different scenarios which represent a certain risk profile, while the government sees
a single, best-guess case to work with. This reflects conversations with the experts about how
each side manages risk and uncertainty. The shipyard is given overhead costs for each year and
an internal rate of return that outpaces the expected rate of inflation. Because of this, the
spreadsheet calculates net present values which will differ for both sides even if the same inputs
are calculated. The sheets also provide error messages for common input errors such as
simultaneous block execution and budget overruns.
All players' spreadsheets calculate the government's budget as beginning at $450 million
per year and decreasing at the internal rate of return. When players enter their proposed ship
counts, the sheet counts the initial ship for$600 million, $650 million, or $700 million,
depending on the scenario. Subsequent ships are priced either at $250 million, $300 million, or
$350 million, again depending on scenario. In the shipyard player's case, $100 million is added
for each year. This value is then transformed according to rate of return to produce a final value
which is revealed each year to the player. The sheet then adds up all years to create expected
-
50 -
costs. The tool automatically counts how many years were involved and calculates annual rates
and per ship rates. The proposed price is then compared against these outputs to generate useful
values such as expected profit margin and budget overruns, depending on the player's role.
If the players choose to agree on block I separately, the game master calculates the actual
contract price according an initial cost of $650 million for the lead ship and $325 million for the
following ships. These values are fixed in all versions of the game. Players may also decide to
use only block 1 or to agree on both blocks 1 and 2 simultaneously without resolving the
outcome of block 1. The game ends with a post-questionnaire and debrief so that players have a
chance to reflect on their strategy and tactics.
-51 -
-52-
Chapter 4 - Analysis
The research game produced 37 different negotiations with 88 participants:
Price
type
notes
trial #
Year B
Ship B
Year A
Ship A
Repeat
Steps
involved?
3650
3650
4000
3150
3250
3000
3600
3150
4131
3400
3300
2
2
2
2
2
2
2
2
2
2
2
1
2
3
4
5
6
7
8
9
10
11
2
12
give all cash
give all cash
4
5
3
5
5
8
10
3
1
6
6
3
4
2
4
5
10
10
2
0
5
5
7
6
8
6
5
0
8
10
5
5
6
5
7
5
5
N
N
N
N
N
0 N
N
6 N
9 N
4N
4 N
N
no block 2
13
14 no agreement
15
16
17
18
19
20
7
8
3
2
5
5
4
2
4
4
6
6
5
4
5
5
5
5
6
8
6
6
4
4
5
6
5
5
N
N
N
Y
Y
N
N
N
2700
3350
3640
3000
3760
3400
2
2
2
2
2
2
2
2
3280
2940
3
3
1
2
Y
Y
3050
3100
3030
3100
3350
3550
2750
2700
3850
3400
3
3
3
3
3
3
3
3
3
3
3
4
5
6
7
8
9
10
11
12
Y
N
N
N
N
N
N
N
N
N
3640
3160
3
3
13
14
2741
3000
3000
3
3
3
15
16
17
3150
10
7
11
15
17
9
11
13
9
6
12
bribery
N____
_______N
Y
Y
Y
Table 2: Resulting final contracts for the 37 negotiations
-
53
-
11
16
13
5
5
8
8
9
Some of these participants were invited to play both variants of the game and were noted
as having repeat experience. The sample drew from a convenient pool of MIT students as well as
remote experts representing NAVSEA and SCRA. The player pool was 75% male, 25% female
with a median age between 25 and 29. The average player had 5-6 years of college experience in
a scientific or technical background and 1-2 years of work experience in a technical field. The
aggregated sample details are presented in Appendix A. We found that gender, work experience,
and school experience did not have any produce any significant differences in the results.
4.1 Monopolistic Case
The resulting final agreement prices in the monopoly games are presented below.
Distributions type=2
Price
4000
3500
3000
2500
- - - - -
-
Summary Statistics
Mean
Std Dev
Std Err Mean
3404.5
366.11606
86.294383
Figure 14: Monopolistic case price results
-
54-
The average shipyard costs in each round were around $2.981 billion. This resulted in a
mean profit margin of 14.2% for the shipyards. While this number is slightly higher than the
expected outcomes of 8-10%, it does not seem unreasonable when considering the ranges of
uncertainty included profit margins ranging from 0% to 30%.
The survey included questions on how truthful players were during the game, and on how
much they had to compromise to meet their goals. The survey data showed significant results
with a 95% confidence rating for both questions using t-tests on the two player roles of shipyard
and Navy. Survey ratings went from I (strongly disagree) to 7 (strongly agree) and can be found
in the appendix. Shipyard players averaged a 4.7 on compromising and a 5.1 on truthfulness. The
means for Navy players were 3.4 and 6.2 on compromise and truth, respectively. The shipyards
were able to command higher prices by bluffing in a position of power during the negotiation.
Even though the players in the Navy role were not receiving great returns on their costs, they
were still reaching their goals of ensuring defense industry stability.
The final price agreements are graphed against the number of negotiation offers that were
presented in the game session below:
Bivariate Fit of Price By Steps
3800
3600
0
*
3400
3200
3000
*
*
2800
2600
----
4
6
8
12
10
16
14
Steps
I-
Polynomial Fit Degree=2
Polynomial Fit Degree=2
Price = 3671.5521 - 37.420247*Steps + 1.9191307*(Steps-10.2941)A2
-
55
-
18
Summary of Fit
RSquare
0.197842
RSquare Adj
0.074433
Root Mean Square Error
Mean of Response
281.809
3321.875
Figure 15: Number of proposed contract drafts ("steps") fit against final contract price
There was no statistically significant correlation between the number of steps and the
final contract price. While this seems to point that better communication does not necessarily
lead to better deals, it should be explained that the tradespace may not have been fully explored
within the time constraints of the research sessions. We present a representative case from the
two-player game below to show how players explore the tradespace:
Cost 1 Ships 1 Years Cost 2 Ships 2 Years Time Proposed by
1400
4
5
6:30 N
1650
5
5
6:40 N
2250
5
6
1200
5
4 6:49 S
2250
5
6
1150
5
4 6:51 S
2000
5
6
1150
5
4 6:57 N
3400
10
10
6:58 N
3300
10
10:
7:00 N
2140
5.
600
6. 1150
5
4 7:10 S
2. 2700
9
8 7:19 N
3350
10:
10
7:23, N
3600.
10:
10.
7:27. N
Table 3: Data set of a representative case from the monopolistic game
-56-
For a given number of ships and years, the Pareto frontier may be constructed where the Navy's
annual budget and the breakeven costs for the shipyard create bounds of a feasible tradespace:
Profit
(Shipyard
utility)
0
0
Pareto Frontier lines
Price (Navy utility)
Figure 16: Tradespace quadrants in the negotiation game
The green represents the feasible space where both sides are happy, while the red represents the
area where no one reaches their goals. In the blue spaces, one side is happy and a negotiation
may be possible. The Pareto Frontier slides and rotates when the ship and year counts change. It
is possible for the current terms to be in a suboptimal position, as shown by the Pareto front on
the right that does not intersect the green area. This came up during the gameplay sessions,
especially in the second block when the national budget decreases significantly. Players who
locked into block 1 faced unwinnable decisions when deciding block two terms and had to
decide who would take the failure state position. Note that with the rotation and shifting, it is
possible for both players to be fully in the lose-lose situation.
The representative case is graphed below, for goals of 8% profit and a national budget
requirement of $450 million per year.
-57-
40.00%
30.00%
/1
20.00%
20.00%
-
10.00%
/e
i
0.00%
50
2
100
-
-
150
200
/
250
300
I
Il
400
-10.00%
6 -20.00%
-30.00%
-40.00%
-50.00%
-60.00%
Annual Contract Price
Figure 17: Representative negotiation graphed along with different Pareto frontiers
We find that players shift along a frontier after locking in the number of ships and years
desired. They explore a front over a few offers and then jump to another frontier if the deal is not
agreeable to both sides. This makes the relationships between the three variables easier to
understand since cost can be modified independently of the other factors in general. The human
players do not explore the tradespace by finding the best point in a multivariate optimization;
they take educated guesses to set a frontier and then negotiate over where the optimal resting
point on that line should be. The slope of the line is governed by the available annual budget
while the intercept point is determined by the number of ships in the deal. Given sufficient time
and motivation to explore the entire tradespace, the Pareto frontier could be imagined as a unique
set of equilibrium points depending on the negotiators' personalities, as imagined below by the
blue line:
-
58
-
Profit
Annual Budget
Figure 18: Consruction of overall Pareto front as a set of unique optimizations of case
specific Pareto frontiers
Note that this graph considers only one block of the overall negotiation. It is possible to
construct a space using the proposed prices of both blocks to find an overall ZOPA where one
block may not be in the win-win quadrant as proposed by Figure 14. We take a different case
from pilot testing and present this notion below in Figure 15:
-
59 -
1 - Initial offer
We are able to collect data on the state of the contract (years, # ships,
price) through the negotiation process with timestamps. This is data from
a session during a January class with two players.
is over budget
(user error)
2 - Navy offers
contract at
expected cost
3 - Shipyard
counteroffers
to increase
odds of profit
in block A
4 - Navy
counteroffers
in block A
5 - Shipyard
takes tradeoff
in exchange for
guaranteed
profit in Block
B (the light
blue line)
Figure 19: The evolution of the sample through time in a block cost-block cost space
This figure presents the evolution through time of the contract in the ZOPA, highlighted
by the yellow area. The red areas are considered the lose-lose segments while the orange presents
the possible win-lose area; because of the uncertainty in the game, the players are unsure about
exactly which area they are in. The area bounded by the cyan line represents a case where the
shipyard has almost certaintly assured themselves of a winning outcome.
An interesting notion that occurred in the representative sample featured in Figure 13 as
well as three others in the monopolistic case is that the Navy player decided to give as much as
they could to the shipyard player. They recognized that since the budget did not carry over if
unused, they could place the money towards their goal of ensuring the shipyard's well-being.
Although it went against the written goal of reducing cost per ship, they felt they understood the
system well enough to say that the cost minimization was a useless ship. Some of these players
had experience with defense contracting and cited the tactic as one that validated the model. The
two most expensive cases in this monpolistic mode involved using this tactic.
-
60 -
4.2 Competitive Case
We present the final result prices in the competitive case below:
Distributions type=3
Price
4000 3800
3600
3400
3200
3000
2800
1
2600
Summary Statistics
Mean
Std Dev
Std Err Mean
3155.3529
321.84953
78.059976
Figure 20: Competitive case price results
The average shipyard costs in this round were $3.155 billion. The mean profit margin
was approximately 6% for the shipyard that landed the contract. In all of these cases, the
from the
government did not choose the dual-award option. With a 95% statistical validity
student's t-test, the players again differed on how truthful they were. The shipyard players
a 4.9. With
averaged a 5.7 on their truthfulness, while the Navy players' average response was
bluff about
the ability to play both shipyards against each other, the Navy player was free to
offers and counter offers since they could not be publicly revealed to all players.
a worse
One particular case involved the Navy selecting a shipyard which was offering
-61-
deal than their competitor. In this game session, two of the players would step outside and have
private conversations, leaving the other player unable to share counter offers. They engaged in
bribery and thought of different ways to minimize costs together, such as lying about ship
delivery times. While we do not believe the government engages in behavior such as bribery, we
have talked with NAVSEA members about tactics where one side refuses to talk until a given
time, putting pressure on the negotiation to create a favorable contract so that the situation can be
resolved. Here, players walked away from the table to achieve the same effect.
4.3 Differences Between Cases
The two price sample sets were compared using t-tests:
Oneway Analysis of Price By type
4000
5
S
I
0
S
3500
S
0
S
0
S
S
... X
3000
0
0
S
0
2500
0
2
3
type
Each Pair
Student's
0.05
Means Comparisons
Comparisons for each pair using Student's t
Confidence Quantile
t
2.03452
Alpha
0.05
LSD Threshold Matrix
Abs(Dif)-LSD
2
3
-
62
-
Abs(Dif)-LSD
2
3
2
-234.22
3
11.51
11.51
-241.01
Positive values show pairs of means that are significantly different.
Figure 21: Comparison of prices by type
The two data sets were found to have a 95% significant difference from each other. The
overall costs dropped by 8% by moving from the sole-source environment to the competitive
case. We do not interpret this as suggesting an 8% price drop if the U.S. government were to
follow suit. Instead, we focused on the debriefings to understand why the price drops occurred
even though the initial values and constraints in both games were similar.
In conversations during and after the game, players cited confidence in the competitive
case as a reason to take risks and rely on the Naval-guess scenarios rather than safeguarding
against the worst-case scenarios. They assumed that in the game world, they must be reaching
their best-guess targets if they are remaining in a competitive market. The monopolistic sellers
were free to focus only on the lowest possible profit and ensure that they satisfied their
stakeholders.
The successful shipyards in the competitive case also leveraged their learning curve for
the second block. They created a brief window where the other player was almost entirely
removed from the game because they could not compete with the learning effects. They were
then free to ask for higher prices from the government even as the prices of their ship production
went down. This helped transfer risk; they were able to take larger chances in the first block
knowing that they would be in a strong position to force the Navy to internalize the risk in
subsequent contracts. While this is not the way that we view long-term stability in the current
sole-source environment, it provides more evidence that support over long periods of time allows
for a shipyard to take its more optimistic forecasts and take risks.
-
63 -
-64-
Chapter 5 - Implications & Future Work
5.1 Case Study Implications
By creating a competitive market through investment or other means, the U.S. would be
able to stimulate the market and drive down prices. This price drop depends on the willingness of
shipyards to accept more of the risk and uncertainty. With another American shipyard, the
industry would have to adapt to higher standards and expect to be closer to their optimistic
targets in order to land contracts and survive.
The current reality discourages financial prudence because of the interests in national
security. The game provides insight into the policy of spending all of the allocated funds. While
this occurred in a quarter of the monopolistic games, it never occurred in the competitive
environment. Moving to a competitive case would not only drive down prices through shipyard
production, but also by removing disincentives from the government's side.
The players' actions reveal that they placed more importance on keeping the shipyards
healthy in the monopolistic case and that the yards played for longer contracts. While the
requirement for work for seven of the ten years was never broken, players did ignore the best
cost goals to compensate. We find that the most significant driver in the negotiation is the
promise of continued work through uncertainty.
From a game theory perspective, we draw from a previous thesis in the shipbuilding
study by Dominic Alvaraan [40]. He explores an extension of principle agent theory where the
Navy may choose to continue working with the shipbuilder if it trusts it, or it may terminate the
contract and end the game. The shipyard may execute the contract as written or "betray" the
shipyard and receive a greater benefit through contract manipulation. While this research game
has assumed the shipyard will always execute the contract as written, it provides insight into the
power structure that drives this theory. In the sole-source environment, the shipyard players were
more likely to lie as they had the ability to gain leverage by creating a false baseline for their
performance. In the competitive case, the Navy players were more likely to lie since they could
distort the baseline used to compare shipyards to one another. This survey data strengthens the
principle agent theory analysis and provides a basis to use it in a competitive environment where
the power roles are reversed.
-
65 -
The game theory analysis from Chapter 3 focused on the consequences of game theory
where players are theorized to move to the Pareto front and then slide along it, only accepting
Pareto dominant improvements. We see from the tradespace analysis that players are more likely
to select boundary conditions of a front, slide along it until they come to a point where both
conditions may be satisfied or they find that no such point exists. At this point, they will then
suggest another set of Pareto front conditions and repeat the process, comparing equilibrium
points until they are satisfied.
5.2 Game Design Implications
We have shown how a research game is useful in analyzing the negotiation process. It
does not make specific recommendations about numbers associated with policy changes, but
brings up system changes that are not inherently obvious such as the removal of full government
allocation strategies. It provides insight into processes such as negotiation by creating a space
where qualitative and quantitative information may be tracked in tandem to explain moves that
do not make sense according to game theory because they are not Pareto dominant.
We observed that repeat players of the game were not significantly different than firsttime players in the amount of compromises they had to make, or in their ability to reach their
goals. Where they stood out was in their ability to explain the system to the researchers and how
their negotiation strategy evolved. They were able to provide additional insights into the power
dynamics of the game that supported the survey data and into theoretical cases that could be
developed into future work. The one trend where repeat players differed from first-time players
was that they understood the interface much better. As people explore the game environment and
become accustomed to the tools, they are free to focus more on the mechanics and dynamics and
provide meaningful feedback in the debriefing sessions.
Research games in this field should focus on the process by which goals are achieved. By
definition, games are never the most efficient way to reach a goal. We would not recommend a
game to create the best contract; we would recommend a game to study how contracts are made.
A standalone simulation is more effective in optimizing values than human players will ever be
able to, as seen in how humans explore the contract tradespace. The insights gleaned from this
game are important because they can highlight the flaws in the human dynamics and provide
-
66 -
training to work around these errors.
5.3 Future Work
ColumBID is currently presented as a game with a monopolistic and competitive case,
designed to explore the negotiation process. It is currently limited in scope to the table between
shipbuilders and the government. This section offers future work aimed at improving the
shipbuilding process by identifying the system responses to human-influenced drivers. Because
of the flexibility included in the system design, the game is capable of expanding the amount of
players involved to include Congress and the supply chain members as part of the negotiation
and resolution process. Scenarios could also be designed with cost-plus contracts instead of
70
fixed-price contracts through careful management by the game master. With expanded time
constraints, the game could be expanded in scope to include multiple contracts.
Figure 22: Currently existing game variants
-
67
-
5.3.1 Supply Chain Expansion
Shipyard
N avy
L(Congress) Ai
Shipyard
ae~a
Figure 23: Supply Chain Expansion Variant
The supply chain is responsible for much of the uncertainty in projected shipbuilding
costs. Because of the specialized nature of shipbuilding, government-furnished materials are
often bought by the government for the shipyards to use. Similar to the number of shipyards over
time, the number of suppliers has also been reduced and they may be in monopolistic situations
for specific parts. This may be responsible for the shipyard's ability to reduce cost pressure by
passing it up the supply chain. By including extra players in the role of material suppliers, the
game may be expanded to a five-player scenario with a supply chain which allows for multiple
levels of negotiation. While the time required to explain and execute the game scenario is
infeasible in the current study's time restrictions, future work could include these extra players in
a similar scenario. The uncertainty in the current scenarios is currently not assigned a given
distribution, nor is it assigned to anything else aside from technological maturity and material
costs.
-
68
-
5.3.2 Congress as a Player
Congress
Shnipyard
Congress
Shipyardi
Figure 24: Congress as a player variants
The current scenarios include Congress as an external player, who affects the gameplay
by setting the budget constraints on the government player. A feasible deal may be constructed
by finding a position where the set of possible deals intersects with a Pareto front of optimal
outcomes for both players. When plotting out the trade space, it is possible that the negotiation
process may be taking place in an environment where no feasible deal can be made. Because the
budget determines the shape of this trade space, we may choose to create a scenario with no win-
69
-
win outcome. In the terms of game mechanics, the budget will be set to a price where it is
impossible for both the government to stay under the budget and for the shipyard to make any
profit. This would provide evidence that the government may be placed in this position
occasionally if the scenario behavior mimics real world cost overruns.
Congress could also be included as a player within the system. The simplified system
assumes that Congress talks directly to the government player, who then converses with the
shipyards and their stockholders. In reality, the shipyards exert pressure by talking to the
representatives in their district. As key industry players, they wield significant political power
and can ask for more money to be allocated in the national budget. Similarly, Congress has
decisions to make on whether or not they accept the deal proposed by the negotiators. The
current goals include limits on ships per block and idealized building rates. Because of the term
limits in the U.S. government, senators may be hesitant to commit to deals which extend past
their effective time in office. New faces in government can lead to reworked future plans and
changes in prior commitments. By including Congress as another player, we can explore the
feedback loops outside of the formal command chain.
5.3.3 Cost-Plus Contracting
ColumBID currently uses fixed-cost price contracts. In discussions with the program
managers in the Navy Yard, talks revealed that contracts seem to vary from fixed-cost and costplus structures through cycles; each contract type appears to be favored during different periods
of time. While the gameplay in ColumBID does not disallow the creation of cost-plus contracts,
it would be possible to create a similar scenario in terms of costs while forcing the players to
create cost-plus contracts. The game master would have to calculate the final contract price after
resolving the uncertainty in the ship cost. This detail is already included in the game design as
part of the block feature, so it would require little adjustment in the underlying model. This
scenario would be useful as a test bed for the Navy to see if there are certain cost regimes where
one contract type outperforms the other.
5.3.4 Multiple Contracts
-
70 -
The current scenarios include options by both sides to invest in a shipyard's
infrastructure. However, infrastructure development is typically a decision for the long run, when
the improved capacity is realized in subsequent contracts. Likewise, the notion of trust as an
important factor in negotiation reveals itself over many contracts. A useful variant of the game
would be to either run a long session where several contracts are negotiated in succession or with
the same group of players multiple times. This would put much more of a focus on the strategy
of the negotiators rather than the individual tactics that can win a contract in the short term.
While we see some evidence of players willing to take worse positions in order to secure the
entire contract, the results of each block provides new pieces of information for future
negotiations. Players would develop a sense of how the negotiation process plays out between
the particular set of people present and what their common tactics would be. In war, strategy is
the overarching plan using all available resources. Tactics is defined as how to reach the
particular objectives set out by the strategy. Strategy comprises different sets of tactics executed
over time. Similarly, this variant expands on the idea of negotiation strategy versus negotiation
tactics. This plays to the strengths of using a game's tool by exploring a space which would be
impractical through repeated real-world trials. Furthermore, it could include the other variants
suggested in this section to see how flexible a particular strategy is. ColumBID currently presents
an amount of space for strategic consideration, but does focus primarily on the tactics of a single
negotiation. By expanding scope in both model detail and game length, effective strategic
behavior can be identified.
5.3.5 Further Qualitative Analysis
Future sessions could include video and audio recording to facilitate qualitative analysis.
The researchers recorded key pieces of private information that were shared during the
negotiation. There was evidence of relationships, but not statistically significant. For example,
sharing information about annual budgets in multiple cases led to the government player
realizing that they did not receive a budget rollover. They were incentivized to fund the shipyard
with the entire budget. A wider qualitative analysis could provide more insight on the personal
dynamics during the negotiation process. For example, although we showed no relationship
between the number of trade offers and the resulting contract, recording the conversation lengths
-71-
could provide evidence because the trade offers vary with times between offers. These
recordings could supplement the human interactions found in the tradespace tracings.
-
72
-
Appendix A - Interview personnel records
Interviews with Program Managers at the Navy Yard/Pentagon on February 21/22, 2013:
PMS 450 (VCS); Attendees: Lisa Bonacic, Danielle Barton, CDR Terry Nawara, Lt Dom
Alvarran, and Roi Guinto.
PMS 500 (DDG 1000); Attendees: Edward McGill, Matt Evans, Joe Wineke, Chris
Covalt, CDR Terry Nawara, Lt Dom Alvarran, Roi Guinto, and Eric Rebentisch.
PMS 501 (LCS); Attendees: Ed Foster, CDR Cara Lapointe, Greg, CDR Terry Nawara,
Lt Dom Alvarran, Roi Guinto, and Eric Rebentisch.
PMS 317 (LPD); Attendees: Sharon Linsenmeyer, Cary Knapp, LT CDR Andy Gold,
CDR Terry Nawara, Lt Dom Alvarran, Roi Guinto, and Eric Rebentisch.
DASN(AP) Elliott Branch; Attendees: DASN (AP) Elliott Branch, Lt Dom Alvarran, Roi
Guinto, and Eric Rebentisch.
Interviews were also conducted remotely with:
Carl Perry - Contracting Officer, Huntington-Ingalls (Ingalls Shipbuilding)
C.L. Rector - Director of Contracts, Huntington-Ingalls (Ingalls Shipbuilding)
Charles Zigleman - Contracting Officer, NASSCO
Matt von Ruden - Contracting Officer, Vigor Shipyards
-
73 -
Appendix B - Aggregate sample demographics
Distributions:
Gender - Male or female players?
M
F
Frequencies
Level
Count
Prob
F
M
Total
21
65
86
0.24419
0.75581
1.00000
Age - How old were our players?
50+
30-34
25-29
18-24
Frequencies
Level
Count
Prob
-74-
Level
18-24
25-29
30-34
50+
Total
Count
Prob
34
44
7
1
86
0.39535
0.51163
0.08140
0.01163
1.00000
Years of Education in a Technical/Scientific field - How educated were our players?
9+
7-8
5-6
3-4
Frequencies
Level
0
1-2
3-4
5-6
7-8
9+
Total
Count
5
2
4
62
9
4
86
Prob
0.05814
0.02326
0.04651
0.72093
0.10465
0.04651
1.00000
-75-
Years of Work in a Technical/Scientific Field - Did our players have work experience?
9+
5-6
3-4
1-2
0
Frequencies
Level
0
1-2
3-4
5-6
9+
Total
Cour t
34
29
15
6
2
86
Prob
0.39535
0.33721
0.17442
0.06977
0.02326
1.00000
Years of Shipyard or Negotiating Experience - Were our players already familiar with the
context?
9+
7-8
5-6
3-4
1-2
0
Frequencies
Level
Count
Prob
-76-
Level
Count
Prob
0
1-2
3-4
5-6
7-8
9+
Total
76
2
1
3
1
3
86
0.88372
0.02326
0.01163
0.03488
0.01163
0.03488
1.00000
Repeat player - Did this player already run a different version of the game?
Y
N
Frequencies
Level
N
Y
Total
Count
69
17
86
Prob
0.80233
0.19767
1.00000
-77-
Appendix B - COUHES Materials (Surveys, Consent Form)
CONSENT TO PARTICIPATE IN
NON-BIOMEDICAL RESEARCH
Production in the Innovation Economy: How to Create Excellence Through Competition and
Benchmarking in the U.S. Shipbuilding and Defense Industry
You are asked to participate in a research study conducted by Roi Guinto, a master's student from the
Engineering Systems Division at the Massachusetts Institute of Technology (M.I.T.). Results from this
study will contribute to a conference and/or a master's thesis. You were selected as a possible
participant in this study because you have a background in a technical field such as engineering or
mathematics. You should read the information below, and ask questions about anything you do not
understand, before deciding whether or not to participate.
*
PARTICIPATION AND WITHDRAWAL
Your participation in this study is completely voluntary and you are free to choose whether to be in it or
not. If you choose to be in this study, you may subsequently withdraw from it at any time without
penalty or consequences of any kind. The investigator may withdraw you from this research if
circumstances arise which warrant doing so.
*
PURPOSE OF THE STUDY
Negotiations involving acquisitions for the U.S. military attempt to mitigate future risk through various
incentives and contract structures. The economic situation of the U.S. shipbuilding industry has led to
market cases of a sole buyer and seller. In these cases, both sides of the negotiation are dependent on
each other to fulfill their respective goals. Although their goals are at odds with each other, they must
cooperate in order to be successful.
-
78 -
This study investigates strategic behavior in these contract negotiations, as well as possible effects of
changes in the supply chain structure. Through a series of simulations, this study hopes to improve the
understanding of the shipbuilding process. This new knowledge will be used to provide mutual benefits
for both buyers and sellers in the defense industry.
*
PROCEDURES
Participants will be randomly assigned to a negotiation position and seated at a computer station. The
examiner will present the negotiation simulation to the participants and give a short tutorial on its uses.
During the test session, participants will discuss a series of values to simulate the negotiation process
and attempt to arrive at a mutually acceptable contract position. Participants may not look at each other
screens, but may discuss freely otherwise.
The negotiation involves the creation of a complete set of ships through the use of multiple contracts.
Each contract represents a "block" and may be created individually or all at once. During this negotiation
process, screen capture software will record the state of the contract as well as verbal discussions via a
microphone. The research may record written observations. Participants have the option to exit the
study at any point during the testing session.
*
POTENTIAL RISKS AND DISCOMFORTS
This experiment involves interaction with a computer and communication with other experimental
subjects. It is unlikely that you would encounter any risks or discomforts resulting from the study.
*
POTENTIAL BENEFITS
You will receive no benefits from participation. The results may contribute to improved contract design
and negotiations.
-
79 -
*
PAYMENT FOR PARTICIPATION
You will not receive any payment for participation.
*
CONFIDENTIALITY
Any information that is obtained in connection with this study and that can be identified with you will
remain confidential and will be disclosed only with your permission or as required by law. Results from
the experiments will be shared with members of the shipbuilding study at MIT, including personnel of
the U.S. Navy.
All personally identifiable information will be protected using codes assigned during screening. Codes
will be used on all testing materials (consent forms will be collected separately from results) and the
map between personally-identifiable information and codes will be saved in an encrypted computer file
for use by authorized personnel only. Following the completion of this study all personally-identifiable
information will be destroyed.
*
IDENTIFICATION OF INVESTIGATORS
If you have any questions or concerns about the research, please feel free to contact lead researcher Roi
Guinto at (201) 887-3326 or faculty sponsor Professor Olivier de Weck at (617) 816-4956.
*
EMERGENCY CARE AND COMPENSATION FOR INJURY
If you feel you have suffered an injury, which may include emotional trauma, as a result of participating
in this study, please contact the person in charge of the study as soon as possible.
-
80 -
In the event you suffer such an injury, M.I.T. may provide itself, or arrange for the provision of,
emergency transport or medical treatment, including emergency treatment and follow-up care, as
needed, or reimbursement for such medical services. M.I.T. does not provide any other form of
compensation for injury. In any case, neither the offer to provide medical assistance, nor the actual
provision of medical services shall be considered an admission of fault or acceptance of liability.
Questions regarding this policy may be directed to MIT's Insurance Office, (617) 253-2823. Your
insurance carrier may be billed for the cost of emergency transport or medical treatment, if such
services are determined not to be directly related to your participation in this study.
0
RIGHTS OF RESEARCH SUBJECTS
You are not waiving any legal claims, rights or remedies because of your participation in this research
study. If you feel you have been treated unfairly, or you have questions regarding your rights as a
research subject, you may contact the Chairman of the Committee on the Use of Humans as
Experimental Subjects, M.I.T., Room E25-143B, 77 Massachusetts Ave, Cambridge, MA 02139, phone 1617-253 6787.
-
81
-
SIGNATURE OF RESEARCH SUBJECT OR LEGAL REPRESENTATIVE
I understand the procedures described above. My questions have been answered to my satisfaction,
and I agree to participate in this study. I have been given a copy of this form.
Name of Subject
Name of Legal Representative (if applicable)
Signature of Subject or Legal Representative
Date
SIGNATURE OF INVESTIGATOR
In my judgment the subject is voluntarily and knowingly giving informed consent and possesses the legal
capacity to give informed consent to participate in this research study.
Signature of Investigator
Date
-
82 -
FOR ADMINISTRATOR USE ONLY
Participant ID
Session ID
Production in the Innovation Economy Pre-Questionnaire
Below are some questions about your background before participating in the study. For the
following questions please circle, mark or write the one best response. Your participation in
this study is voluntary, and you may refuse to answer any questions or end your participation in
the questionnaire at any time.
I
I
I
Gender
2
Age
Female
Male
18-24
25-29
30-34
35-39
40-49
50+
0
1-2
3-4
5-6
7-8
9+
0
1-2
3-4
5-6
7-8
9+
0
1-2
3-4
5-6
7-8
9+
Years of college-level
3
(undergraduate + graduate)
education in a technical field
(sciences/engineering).
Years of work experience
4
(professional) in a technical
field (sciences/engineering).
Years of work experience
5
(professional) in
shipbuilding/contracting.
-
83
-
FOR ADMINISTRATOR USE ONLY
Participant ID
Session ID
Production in the Innovation Economy Post-Questionnaire
Below are some questions about the experiences you had while completing the design tasks. For
the following questions please circle, mark or write the one best response. Your participation
in this study is voluntary, and you may refuse to answer any questions or end your participation
in the questionnaire at any time.
Sbungly +- Disagree <-- Neutral -+ Agree -disag=e
agree
Don't
know
6
I made significant
compromises to achieve
my goals.
1
2
3
4
5
6
7
9
7
I was able to achieve
1
2
3
4
5
6
7
9
1
2
3
4
5
6
7
9
1
2
3
4
5
6
7
my goals.
I was able to navigate
8
and understand the
interface.
I was usually truthful in
negotiation talks.
2
Other comments or feedback for the researchers:
-
84 -
References
[1] MIT Game Lab IMIT Game Lab. (n.d.). . Retrieved May 21, 2014, from
http://gamelab.mit.edu/
[2] Richard de Neufville, Frank Field, "Thesis Definition and Preparation: Some General
Guidelines." MIT Technology and Policy Program, Internal Release, 10-Sep-20 10.
[3] Mayer, I. S. The Gaming of Policy and the Politics of Gaming: A Review. Simulation &
Gaming, 825-862.
[4] Suits, B. (1978). The grasshopper:games, life, and Utopia. Toronto: University of Toronto
Press.
[5] McGonigal, J. (2011). Reality is broken: why games make us better and how they can change
the world. New York: Penguin Press.
[6] Herodotus (440 B.C.E.) The History of Herodotus: Book VII.
[7] MITI 50 Exhibition Nomination. (n.d.). MIT150 Exhibition Nomination RSS. Retrieved May
21, 2014, from http://museum.mit.edu/noml 50/entries/1437
[8] The History Of Pong: Avoid Missing Game to Start Industry. (2009, January 9). . Retrieved
May 21, 2014, from
http://www.gamasutra.com/view/feature/3900/the history of pong avoid missing .php
[9] The Entertainment Software Association. (n.d.). . Retrieved May 21, 2014, from
http://www.theesa.com/facts/gameplayer.asp
[10] Essential Facts about the Computer and Video Game Industry. (n.d.). . Retrieved May 21,
2014, from http://www.theesa.com/facts/pdfs/ESA
EF 2014.pdf
[ 11] Taylor, T. L. (2006). Play between worlds exploring online game culture. Cambridge,
Mass.: MIT Press.
[12] Bogost, I. (2011). How to do things with videogames. Minneapolis: University of Minnesota
Press.
[13] Abt, C. C. (1970). Serious games. New York: Viking Press.
[14] Schell, J. (2008). The art ofgame design: a book of lenses. Amsterdam: Elsevier/Morgan
Kaufmann.
[15] Zimmerman, E., & Salen, K. (2003). Rules ofplay: game designfundamentals. Cambridge,
Mass.: MIT Press.
[16] Crookall, D. Serious Games, Debriefing, and Simulation/Gaming as a Discipline.
-
85 -
Simulation & Gaming, 898-920.
[17] Related Historical Readings. (n.d.). The United States Chess Federation.Retrieved May 21,
2014, from http://www.uschess.org/content/view/7326/28/
[18] Flanagan, M. (2009). Criticalplay radicalgame design. Cambridge, Mass.: MIT Press.
[19] Harteveld, C. (2010). Triadicgame design. London: Springer.
[20] Mood, A. War Gaming as a Technique of Analysis.
[21] Silver, N. (2012). The signal and the noise: why so many predictionsfail--but some don't.
New York: Penguin Press.
[22] Lee, D. B. Requiem for Large-Scale Models. Journalof the American Institute ofPlanners,
163-178.
[23] Klopfer, E. (2008). Augmented learningresearch and design of mobile educationalgames.
Cambridge, Mass.: MIT Press.
[24] Ito, M. (2009). Engineeringplay a cultural history of children'ssoftware. Cambridge,
Mass.: MIT Press.
[25] Squire, K., & Jenkins, H. (2011). Video games and learning: teaching andparticipatory
culture in the digital age. New York: Teachers College Press.
[26] Gee, J. P. (2003). What video games have to teach us about learning and literacy.New
York: Palgrave Macmillan.
[27] Beer Game. (n.d.). MIT Forumfor Supply Chain Innovation. Retrieved May 21, 2014, from
http://supplychain.mit.edu/games/beer-game
[28] Riedel, J., & Hauge, J. State of the Art of Serious Games for Business and Industry.
[29] Roceanu, I. Measuring the Effectiveness of Learning with Serious Games in Corporate
Training. Procedia ComputerScience, 221-232.
[30] Allison, G. T. (1971). Essence of decision; explainingthe Cuban missile crisis. Boston:
Little, Brown.
[31] An analysis of the Navy's Fiscal year 2013 Shipbuilding Plan. (n.d.). . Retrieved May 21,
2014, from http://www.cbo.gov/
[32] Performance of Major US Shipyards in 20th/21 st Century. Journalof Ship Production,24,
202-213.
[33] Arena, M. V. (2006). Why has the cost of Navy ships risen?: a macroscopic examination of
the trends in US. Naval ship costs over the past several decades. Santa Monica, CA: RAND.
-
86 -
[34] BEST PRACTICES: High levels of knowledge at key points differentiate commercial
shipbuilding from Navy shipbuilding. (n.d.). . Retrieved May 21, 2014, from
http://www.cbo.gov/
[35] Raybourne, E., & Bos, N. Design and Evaluation Challenges of Serious Games.
[36] T. Husen & T. N. Postlethwaite, (eds.) (1989) The International
Encyclopedia of Education, Supplement Vol.1. Oxford/New York:
Pergamon Press, 162-163. [37] Glasersfeld, E. v. (2002). Radical constructivism in mathematics
education. New York: Kluwer Academic.
[38] Thorley, N. R. The conditions of conceptual change in the classroom. InternationalJournal
of Science Education, 541-553.
[39] Daalen, E. V. Functional design of games to support natural resource management policy
development. Simulation & Gaming, 512-532.
[40] Thesis (Nav. E.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering,
2013."June 2013." Cataloged from PDF version of thesis. Includes bibliographical references (p.
79-81). http://hdl.handle.net/1721.1/82300
[41] The Mercury Game. (n.d.). .Retrieved May 21, 2014, from
http://mercurygame.scripts.mit.edu/game/
[42] Retrieved May 21, 2014, from
http://investorrelations.gd.com/phoenix.zhtml?c=85778&p=irol-reportsannual
[43] Retrieved May 21, 2014, from
http://ir.huntingtoningalls.com/phoenix.zhtml?c=243052&p=irol-reportsannual
[44] Movers and Shakers. (n.d.). Singapore-MIT GAMBIT Game Lab. Retrieved May 21, 2014,
from http://gambit.mit.edu/loadgame/moversandshakers.php
[45] Mitgutsch, K., & Schirra, S. (n.d.). Subversive Game Design and Meaningful Conflict. MIT
Game Lab. Retrieved May 21, 2014, from http://gamelab.mit.edu/research/subversive/
[46] Anthropy, A. (2012). Rise of the videogame zinesters: howfreaks, normals, amateurs,
artists,dreamers, dropouts, queers, housewives, andpeople like you are taking back an art form
(Seven Stories Press 1st ed.). New York: Seven Stories Press.
[47] Bartle, R. HEARTS, CLUBS, DIAMONDS, SPADES: PLAYERS WHO SUIT MUDS.
http://mud.co.uk/richard/hcds.htm
[48] Caillois, R. (1961). Man, play, and games. New York: Free Press of Glencoe.
-
87
-
[49] Personality and Play Styles. Retrieved May 21, 2014, from
http://www.gamasutra.com/view/feature/6474/personality
and play styles a .php?print--l
[50] Alexander, J. (2002, January 14). Evolutionary Game Theory. Stanford University.
Retrieved May 21, 2014, from http://plato.stanford.edu/entries/game-evolutionary/
-
88 -
Gameography
All games mentioned in this thesis are cited here.
Call ofDuly 4: Modern Warfare. PC, Mac, PlayStation 3, Xbox 360, Wii. Developed by Infinity
Ward. Activision, 2007.
Depression Quest. Browser. Developed and published by Zoe Quinn. 2013.
EVE Online. PC. Developed and published by CCP Games, 2003.
EverQuest. PC. Developed and published by Sony Online Entertainment, 1999.
FinalFantasy VII PlayStation. Developed and published by Square, 1997.
FinalFantasyX PlayStation 2. Developed and published by Square, 2001.
Movers and Shakers. Android. Developed and published by the MIT Game Lab, 2012.
Number Munchers. PC. Developed by the Minnesota Educational Computing Corporation.
Broderbund, 1986.
Papers,Please. PC. Developed and published by Lucas Pope. 2013.
Pong. Coin-op, Magnavox Odyssey. Developed and published by Atari, 1972.
Reader Rabbit. PC. Developed and published by The Learning Company, 1986.
Sid Meier's CivilizationIT PC. Developed and published by MicroProse, 1996.
Sid Meier's Civilization III. PC. Developed by Firaxis Games. Published by Atari, 2001.
Sid Meier 's Civilization IV. PC. Developed by Firaxis Games. Published by 2K Games, 2005.
SimCity. Amiga, Macintosh, IBM PC, Commodore 64, SBES, Windows, Browser, Unix, Acorn
Archimedes, Amstrad CPC, Sinclair ZX Spectrum, BBC Micro, Acorn Electron, EPOC32, FM-
Towns, OLPC XO-1. Developed by Will Wright. Maxis, 1989.
Sneak King. Xbox, Xbox 360. Developed by Blitz Games. 2006.
Spacewar!PDP-1. Developed by Steve, Russell, Martin Graetz, and Wayne Witaenem, 1962.
Super Mario Bros.. NES/Famicom. Developed and published by Nintendo, 1985.
Tax Invaders. Browser. Developed and published by the Republican National Committee, 2004.
No longer online.
The Oregon Trail. PC. Developed by the Minnesota Educational Computing Corporation.
Broderbund, 1974.
World of Warcraft. PC. Developed and published by Blizzard Entertainment. 2004.
Where in the World is Carmen Sandiego? PC, Amiga, Sega Genesis, NES, SNES, TurboGrafx-
CD. Developed and published by Broderbund, 1985.
-
89 -