The Turing Test for Game AI
Bing Gordon
Electronic Arts
June 2005
“I wish someone would…”
…develop a 'metric' for computer npc ai.
Kind of an eliza scorecard.”
1. Context
Powers of 10…
1981 Atari VCS
1986 NES
1989 Sega Genesis
1994 Sony Playstation
2001 Playstation 2
2006 PSX3/Xbox 2
2012? PS4/Xbox 3?
128 bytes
2K + 2K
64K+64K
2 MB + 512K
32MB +4MB
512MB
5G?
Context
Powers of polish …
75% of 90+ rated games are hits.
25% of 90+ rated movies are hits.
Source: Metacritic.com
Powered by Feature Innovation
Aspirations
Tiger swing
Gameface
Franchise mode
Dynamic sports news
Assymmetrical sides
Badges
Anti-grav gun
Auction House
Living city
Crash mode
Freestyle
Console online voice
and custom maps
Quality bars of the near-future
Procedurally generated content
“Emergent” behaviors, collisions
Believable characters
100x physics
Portable avatars, persistent assets
Communities
Economies and money
Camera POV and LOD drives gameplay
Collaborative and dynamic intelligences
AI could be a “killer app” feature of
next gen
Characters:
–
–
–
–
Awareness
Memory
Complex motives, simple commands
100x RAM allocation
Procedural responses
– Parsers
– Stitching
– Pathfinding and cover
Must be co-developed with animators!
– Game AI must be acted out and seen
– Expressions & gestures> pathing
2.
EA Learning
We are Influenced by the
success of our “Killer Apps”
Pinball Construction Set -- floppy disk
Deluxe Paint -- graphics co-processor
EA Sports – 16-bit processor
Interactive movies – cd-rom
MMO’s – servers
Sims – object-oriented programming
Spore – procedural animation
Etc.
2. EA Learning
EA Sports is a tech story
16-bit (1990) Sports simulations
– Individual ratings
– Dynamic simulations
Playstation (1994) 3D Sports
– Z-axis passing, tackling
Playstation 2 (2000) Articulation
– Skeletal modeling and animation
What EA learned from John Madden
The Oakland Raiders playbook
– Matchup strategy
– 5 zones of field-position
One Knee Equals Two Feet
– Player ratings
– All-Madden team
The Madden Test (of game AI)
1985: “That’s not football!”
1990: “I’d fire the coach!”
1995: “What are those guys doing?”
2000: “Rookie, you’re cut!”
2005: “That’s the way I designed it!”
Madden Football Genesis
23%
Madden 97 Playstation
54%
Madden 2001 Playstation 2
63%
Madden 2005 PS2
83%
3. Reviewers
What EA learned from
www.metacritic.com
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11) 94 -- Madden NFL 2002
97 -- Grand Theft Auto III
12) 94 -- Madden NFL 2004
97 -- Tony Hawk's Pro Skater 3
13) 94 -- Burnout 3: Takedown
96 -- Metal Gear Solid 2: Sons of Liberty 14) 94 -- NCAA Football 2004
95 -- Grand Theft Auto: Vice City
15) 94 -- Virtua Fighter 4
95 -- Grand Theft Auto: San Andreas
16) 93 -- Pro Evolution Soccer
95 -- Gran Turismo 3 A-Spec
17) 93 -- SSX
95 -- Madden NFL 2003
18) 93 -- SSX 3
95 -- Tony Hawk's Pro Skater 4
19) 93 -- World Soccer Winning
95 -- Devil May Cry
Eleven 6 International
94 -- God of War
20) 93 -- NFL2K
3. Reviewers
AI is not critical…yet
AI cited for 6/20 top PS2 games.
– Metal Gear, NFL x 4, Soccer
AI cited for 3/10 top PC games.
– Half Life x 2, Civilization
When AI is applauded…
Appropriate npc behaviors
– Dynamic adjustments
– Satisfying mistakes
– Coordinated attacks/retreats
Challenging opponent
When AI is punished…
Too easy
– No cover
– “Money plays”
Too dumb
– Low awareness
– “Deer in headlights”
"Do NOT duck inside those doors, men!
Honorable warriors need no cover!"
Pointless Waste of Time.com
4. An experiment
A Turing Test for Game AI
(version 1.0)
Clips from 5 games
– Madden, FIFA, Fight Night, Mortal Kombat
– Many others could not be tested
“Player or AI controlled?”
App by Karim Mahrous, UCD phd
candidate
Format of test
Watch a clip, multiple replays
Player or AI control for specific character.
Rationale:
–
–
–
–
–
–
Pathing
Reaction
Timing
Effectiveness
Orientation
Awareness
80 respondents
App
Madden 2001
Madden 2001
Results
Madden 2005 – Clip 1
Madden 2005 – Clip 1
Results
Madden 2005 – Clip 2
Madden 2005 – Clip 2
Results
Results are discouraging (so far)
60% accurate
Clip > game or technology
0 correlation between rationale, correctness
Possible improvement in “orientation” AI
between Madden 01 and 05.
5. For example
Feature metacritic planning for
Competitive quality bar
Enemy Combat AI
Metal Gear Solid 2
–
–
–
–
–
–
95 metacritic overall
Cited by 4/5 top reviews
#2-4 feature mentioned
“communication chain”
“perceptive reinforcements”
“teams of soldiers will find
you”
– “frighteningly intelligent”
– “Outclassing every game
on PS2”
Half Life 2
–
–
–
–
–
–
95 metacritic overall
Cited by 2/3 top reviews
#5 feature mentioned
“use cover effectively”
“run away from grenades”
“actively hunt players down
and flank you ”
– “if you hurt them, they set
up traps”
– “Nothing less than
incredible”
Hall of fame feature
Opponent command AI
Civilization 2
–
–
–
–
–
94 overall
Cited by 3/3 top reviews
#2-5 feature mentioned
“you pay for your mistakes”
“clever and resourceful in
negotiation”
– “respects your power in
battles”
– “may act like a bonehead
at times”
Madden Football
–
–
–
–
–
–
–
–
94 overall
Cited by 3/6 top reviews
#2-6 feature mentioned
“Geek out the AI to the
viciousness you like”
“Players make plays all
over the field”
“Tough half-time
adjustments”
“Gone are the massive
scores…”
“AI is advanced to another
level”
Hall of Fame Feature
Pathfinding and formations
Rome: Total War
– 92 overall
– Cited by 2/4 top reviews
– #5 feature mentioned
– “enjoyable and convincing”
– “centered around troop morale”
– “moments of divine inspiration”
– “on occasion, cretinous”
Recommended Spore AI goals
Half Life 2: enemy hunting AI
Metal Gear Solid 2: communication chains
Civilization 2: city/star combat and diplomacy
Madden Football: “half time adjustments”
Rome Total War: pathfinding and formations
Plus:
– Procedural combat
– Collaborative filtering
10 predictions
Areas of discovery for next-gen features
My bookshelf
Confessions of an Advertising Man, Ogilvy
Positioning, Ries and Trout
The Fifth Discipline, Senge
How Hits Happen, Farrell
Net Gain, Hagel & Armstrong
The Innovator’s Dilemma, Christenson
The Medici Effect, Frans Johansson
Thanks
bgordon@ea.com