Human-Level AI
for Computer Games
John E. Laird
University of Michigan
laird@umich.edu
http://ai.eecs.umich.edu/people/laird
Why I’m Interested in Video Games
• AI is going to be more important in games
• A great environment for developing human-level AI
• Academic AI has something to offer the game industry
Applications of AI to Games
• Synthetic Characters
• Sports games, FPS, RTS, …
• Dynamic Game Control
• Modify play calling based on experience (coaching)
• Directing characters for plot and story
• Providing play-by-play, camera control, …
• Development of Games
• Automatic testing
Human-Level AI & Games
• FPS Enemies: Quake, Unreal, Half-Life, ...
• FPS Partners: Spec Ops, Rainbow Six
• Adventure Characters: Blade Runner
• RPG Characters: Baldur’s Gate
• Massively Multiplayer Games: Asheron’s Call, Everquest
Human Behavior Modeling
Time Scales
Seconds
104
1 hour
Game
103
10 min
Level
102
1 min
Engagement
101
10 sec
Unit task
100
1 sec
Operations
10-1
100 msec Deliberative Act
10-2
10 msec
Neural Circuit
10-3
1 msec
Neuron
10-4
100 msec Organelle
Rational Band
Cognitive Band
Neural Band
Human Behavior As A
Sequence Of Deliberative Actions
• Consensus:
• Human task-level behavior (> 100 milliseconds) can be
model as a sequence of deliberate actions, each taking
approximately 50 milliseconds.
• Basic Perception takes > 100 milliseconds.
• Simplest action (key press) takes 70 milliseconds.
• “Press a key when the light goes on.”
Perception
100 msec.
+
Thinking
Action
50 msec. + 70 msec.
=
Total
220 msec.
We found reaction time (shooting or sidestepping) to enemy in Quake II to be
.5-2.5 seconds
Processing is Pipelined
• Perception/Cognition/Motor can run in parallel
Perception
Cognition
Motor
Dual Task: Tracking/Choice
50
45
RMS Tracking Error
40
35
30
25
20
15
Observed
EPIC
Lockout (novice)
Lockout + prepare
10
5
0
0.0
3.2
6.4
9.6
12.8
16.0
19.2
22.4
25.6
28.8
32.0
35.2
Vertical separation between tracking and CRT tasks (degrees)
50
45
RMS Tracking Error
40
35
30
25
20
Observed
EPIC
Lockout + Promotions + Interleaving (expert)
Interleaving + Promotions + Retrained (expert)
15
10
5
0
0.0
3.2
6.4
9.6
12.8
16.0
19.2
22.4
25.6
28.8
32.0
35.2
Vertical separation between tracking and CRT tasks (degrees)
Cognitive Architectures
• ACT-R (John Anderson, Carnegie Mellon)
• Detailed models of behavior and learning
• 10 milliseconds-10 seconds.
• Used by cognitive psychologists
• Issues on performance and scalabiltiy
• EPIC (David Kieras, University of Michigan)
• Detailed models of human-computer interaction (no learning)
• Emphasizes models of perception and action
• 50 milliseconds-10 seconds
• Used by cognitive psychologist
• Soar (John Laird, University of Michigan)
• Models of human behavior and learning
• 50 milliseconds-1 hour
• Emphasis more on complex behavior and learning
• Used by AI researchers and cognitive psychologist
Conflicting Goals for
Human-level AI in Games
Goal
Driven
Knowledge
Reactive
Human
Characteristics
Intensive
Low CPU &
Memory
Usage
Fast & Easy
Development
Soar/Games Project
• Build an AI Engine around the Soar AI architecture
• Soar/Quake II
• Soar/Descent 3
• Soar/Half-life
Quake II
Interface Sensor Data
DLL
Actions
• Build autonomous, unscripted AIs
AI Engine
(Soar)
Knowledge
Files
Soar
• AI engine to support multi-method problem solving
• Applied to wide variety of tasks and methods
• Proposed unified theory for modeling human cognition
• Models a wide variety of human behavior: language, HCI, ...
• Combines reactive and goal-directed symbolic processing
• Supports very large bodies of knowledge (>100,000 rules)
• Optimized implementation in ANSI C
• In the public domain
Soar Quakebot
• Goal: Human-like behavior
• Fun and challenging to play against
• Not super-human
• Currently plays as well as good player
• Completely implemented in Soar
• > 750 rules
• Spread across operators for exploring, wandering, chasing, …
• Sensing similar to human
• Runs on separate PC from game
• Uses ~5-10% of 400 MHz Win-95/98/NT machine
Execution Flow of an AI Engine
Sense
Simulated Perception
Self: orientation, position, health
Senses: vision, hearing, smell, taste, touch
Augmented Senses: radar, night-vision, …
> 80 sensors in Quakebot
G
Field of Vision
A
M
Think
Obstacle
E
Act
Simulated Actions
Movement, weapons’ use, communication
~20 actions in Quakebot
Execution Flow of an AI Engine
Sense
Finite-state machines
G
Subsumption
A
M
Think
Neural nets
?
Rule-based systems
E
C code
Planning systems
Act
Rule-based Systems
• Program = set of if-then rules, not sequential code.
• Use whatever knowledge that is relevant to current situation
Rule Memory
Tactics, Movement Knowledge, ...
[
[
[
] [ ][[
] [ ][
] [ ][[
]
]
]
[
[
[
]
]
]
Match
Changes
Working Memory
sensing, elaborations, persistent memories, goals, actions …
Rules
• Conditions contain variables and other first-order tests
• If sense a weapon and do not have that weapon,
pickup the weapon.
(p rule1
(sense weapon ?weapon1)
(self weapon <> ?weapon1)

(action pickup ?weapon1))
• A single rule can have multiple instantiations
• A single condition can match multiple working memory
elements
• Compilers convert rules into discrimination networks
• Only test changes to working memory.
• Share tests across similar rules
• Efficient matching
Which Rule Should Fire?
• Matching finds which rules can fire in current situation.
Rule Instantiations
Rule
Memory
[][]
Match
Working
Memory
[][][][
]
[][][][
]
[][]
Conflict resolution
?
[][]
[][]
• Conflict resolution picks which rules actually fire.
• Based only on syntactic features of rules and data.
• Must engineer rules so correct rule is selected each time.
Weaknesses in Rules
• Conflict resolution based on syntactic features
• Not knowledge of the task
• Rules combine three types of knowledge about actions
• What is possible
• What should be done (based on conflict resolution)
• How to do it
Leads to duplication if have multiplicity in one of the types
• No hierarchical organization of knowledge and goals
Soar’s Approach
• Organize knowledge as operators:
• Primitive & abstract actions
• Turn, move, climb, goto-door, get-item, wander, attack, chase
• Provides trace of behavior comparable to human decisions
• Use rules to propose, select, and apply operators
• Rules fire in parallel - act as an associative memory
• Provides fine-grain, situation-dependent behavior
• Conditional selection and execution determined dynamically
Elaboration
(apply operator)
Elaboration
(terminate operator & propose)
Decide
Decide
(select operator)
Decide
Elaboration
(propose operators)
Execution Flow of Soar Engine
Sense
Soar
G
Elaboration
Propose Operators
A
M
Think
Evaluate Proposed Operators
E
Select One Operator
Perform Operator Actions
Act
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Soar Quakebot Top Operators
Attack
Retreat
Wander
Explore
Record
Enemy
Remove
Enemy
Chase
Soar Quakebot Top Operators
Attack
Retreat
Wander
Explore
Record
Enemy
Remove
Enemy
If enemy visible and
my health is > very-low-health-value (20%) and
his weapon is not much better than mine
THEN propose attack
If enemy visible and
my health is < very-low-health-value (20%) or
his weapon is much better than mine
THEN propose retreat
If no enemy visible or recorded and explored level
THEN propose wander
If no enemy visible or recorded and not explored level
THEN propose explore
Chase
Soar Quakebot Top Operators
Attack
Retreat
Wander
Explore
Record
Enemy
Remove
Enemy
Chase
If enemy just became not visible
and no enemy visible or recorded
THEN propose record-enemy
If hear enemy and no enemy visible or recorded
THEN propose record-enemy
If record-enemy selected
THEN record enemy’s last position and time + 20 seconds.
If time > recorded enemy’s saved time
THEN propose remove-enemy
If remove-enemy is selected
THEN remove recorded enemy structure
If there is a recorded enemy structure
THEN propose chase
Example Rules
Attack
Get-item
Face-enemy
Side-step
Approach
Shoot
IF I see a weapon that is much better than any I have
THEN I need that weapon
IF my health is less than low-health-value (40%)
THEN I need any health object
IF attacking an enemy and
there is an object I need and that
object is closer than attack-get-item-range
THEN propose get-item for that object
IF two get-item operators are proposed
THEN prefer selecting the one for the closer object
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Get item
Go through
door
Exit
room
Face
door
Align
with door
Get item in room
Go to
door
Record
at door
Move
to door
Face item
Shoot
Stop
moving
to item
Slide to
door
Move
to item
Stop move
to door
Detect item
missing
Stop slide
to door
Example Tactics
• Collect-Powerups
•
•
•
•
•
Pick up best weapons from spawn locations
Remember when missing items will respawn
Use shortest paths to get objects
Get health/armor if low on health/armor
Pickup up other good weapons/ammo if close by to deny enemy
• Attack
• Use circle-strafe
• Move to best distance for current weapon
• Chase enemy based on sound of running
• Ambush in corner that can’t be seen by enemy
• Hunt at nearest spawn room after killing enemy
Mapping
• Need information on location of walls, doors, etc.
• Many tactics require this information
• Built up through exploration of a level
• Similar to a robot exploring with range sensors
• Saved and reused when return to level
• Represented internally as a graph structure
• Currently restricted to 2D rectangles
Anticipation?
Dennis (Thresh) Fong:
“Say my opponent walks into a room. I'm visualizing him walking
in, picking up the weapon. On his way out, I'm waiting at the
doorway and I fire a rocket two seconds before he even rounds the
corner. A lot of people rely strictly on aim, but everybody has their
bad aim days. So even if I'm having a bad day, I can still pull out a
win. That's why I've never lost a tournament.”
Newsweek, 11/21/99
Anticipation
• Predicts enemy’s actions
• Creates internal model of enemy’s state
• Uses own knowledge to predict what enemy would do
• Can be modulated by knowledge of enemy’s tactics
• Stops planning when it knows what to do or
uncertainty in actions
• Uses prediction to select actions:
• Ambush, Deny-Powerups
?
His Distance: 1
My Distance: 1
His Distance: 2
My Distance: 2
His Distance: 2
My Distance: 2
His Distance: 3
My Distance: 1 (but hall)
His Distance: 4
My Distance: 0
Ambush!
References
• The Psychology of Human-Computer Interaction,
Card, Moran, and Newell, LEA, 1983.
• Unified Theories of Cognition, Newell, Harvard Press,
1990.
• Modeling Human and Organizational Behavior:
Applications to Military Simulations, National
Research Council, National Academy Press, 1998.
• Soar Homepage, http://ai.eecs.umich.edu/soar