Agent Architecture Considerations for
Real-Time Planning in Games
Jeff Orkin
Monolith Productions
F.E.A.R.
Agenda
 Motivation
 Problems
 Solutions
 Was
it worth it?
Agenda
– Why plan?
 Problems – Performance!
 Solutions – Agent architecture
 Was it worth it?
 Motivation
Why plan in real-time?
Goal-Oriented Behavior
Goal-Oriented Behavior
Problems:
 Managing dependencies
 Sharing behaviors
Problem: Dependencies
AttackFromCover
Gotot
Attack
Problem: Dependencies
AttackFromCover
Drawt
Gotot
Attack
Problem: Dependencies
AttackFromCover
Drawt
Gotot
Gotot
Flipt
Attack
Problem: Dependencies
AttackFromCover
Gotot
Drawt
Attack
Lightt
Reloadt
Gotot
Lightt
Flipt
Problem: Dependencies
t
t
t
t
t
t
t
t
t
t
t
t
t
t
t
t
Problem: Sharing
No One Lives Forever 2
TRON 2.0
Problem: Sharing
No One Lives Forever 2
TRON 2.0
Problem: Sharing
No One Lives Forever 2
TRON 2.0
P.D.D.L.


Planning Domain Definition Language
Goals
–

Desired state
Actions
–
–
Preconditions
Effects
P.D.D.L.
Goal:
(define (problem get-paid)
(:domain briefcase-world)
(:init (place home) (place office)
(object p) (object d) (object b)
(at B home) (at P home) (at D home) (in P))
(:goal (and (at B office) (at D office) (at P home))))
P.D.D.L.
Action:
(:action put-in
:parameters (?x - physob ?l - location)
:precondition (not (= ?x B))
:effect (when (and (at ?x ?l) (at B ?l))
(in ?x)) )

Other actions: take-out, move
P.D.D.L.

Modular
–
–

Decoupled Modules Related by symbols
–
–
–

Goals
Actions
World State
Preconditions
Effects
Applied PDDL’s structure to C++ toolkit in game
code.
Monolith Productions
Management
How to Plan in Real-Time and
Keep Your Job
Jeff Orkin
Monolith Productions
The Plan
Goal: KillEnemy
Plan:
Goto (couch)
UseObject (couch)
Goto (coverNode)
AttackFromCover
AI Performance Guideline: 1.0ms /
frame
1.0ms
AI Performance: Off the Chart!
300ms
1.0ms
AI Performance: Off the Chart!
300ms
Preconditions:

1.0ms
Visibility
AI Performance: Off the Chart!
300ms
Preconditions:


1.0ms
Visibility
Pathfinding
AI Performance: Off the Chart!
300ms
Preconditions:



1.0ms
Visibility
Pathfinding
Tactical Position
Validity
Solution
Re-consider Agent Architecture
 Distributed processing
 Caching
Solution
Re-consider Agent Architecture
 Distributed processing
 Caching
Inspiration:
 MIT Media Lab’s C4
 Robotics
Solution
World
Sensors
Working
Memory
Targeting
Planner
Navigation
Animation /
Movement
Weapons
World
Blackboard
What is a soldier?
What is a soldier?
What is a soldier?
What is a soldier?
Sensors:

See
What is a soldier?
Sensors:


See
Hear
What is a soldier?
Sensors:



See
Hear
Feel pain
What is a soldier?
Subsystems:

Navigate / Move
What is a soldier?
Subsystems:


Navigate / Move
Attention Selection
(Targeting)
What is a soldier?
Subsystems:



Navigate / Move
Attention Selection
(Targeting)
Weapons
Distributed Processing with Sensors



Pseudo-parallel
Amortize precondition processing across
many frames
Allow incremental processing
Distributed Processing with Sensors



More than Vision, Hearing, Touch
Tactical analysis
Internal desires
Distributed Processing with Sensors

Update:
–
–
–

Every frame
Periodic Polling
Event-driven
Limit total number of expensive sensor
updates per frame.
Sensor Example: CoverNode Sensor

Update 3 times / second
E
A
Sensor Example: Cover Node Sensor

Update 3 times / second
E
A
Sensor Example: PassTarget Sensor

Incremental update
E
A
Sensor Example: PassTarget Sensor

Incremental update
E
A
Sensor Example: PassTarget Sensor

Incremental update
E
A
Sensor Example: PassTarget Sensor

Incremental update
E
A
Sensor Example: PassTarget Sensor

Incremental update
E
A
Sensor Example: PassTarget Sensor

Incremental update
E
A
Sensor Example: PassTarget Sensor

Incremental update
E
A
Sensor Example: PassTarget Sensor

Incremental update
A
E
Caching
Sensor
Sensor
Sensor
Working
Memory
Targeting
Planning
Caching: WorkingMemoryFacts


Facts are really beliefs
Common representation of all knowledge:
–
–
–
–
–
–
–
Character
Object
Node
Disturbance
Task
PathInfo
Desire
Caching: WorkingMemoryFacts
WorkingMemoryFact
{
Attribute<Vector3D>
Attribute<Vector3D>
Attribute<StimulusType>
Attribute<HANDLE>
Attribute<DesireType>
// 16 attributes total...
}
Position
Direction
Stimulus
Object
Desire
Caching: WorkingMemoryFacts
Attribute<Type>
{
Type
Value
float
fConfidence
}
Caching: WorkingMemoryFacts
SeeEnemy Sensor: Stimulus Confidence
C = 0.0
C = 0.0
Caching: WorkingMemoryFacts
SeeEnemy Sensor: Stimulus Confidence
C = 1.0
C = 1.0
Caching: WorkingMemoryFacts
SeeEnemy Sensor: Stimulus Confidence
C = 1.0
C = 0.63
Caching: WorkingMemoryFacts
SeeEnemy Sensor: Stimulus Confidence
C = 1.0
C = 0.0
Caching: WorkingMemoryFacts
CoverNode Sensor: Position Confidence
E
C=0.2
C=0.4
C=0.1
C=0.8
C=0.9
A
The Planning Process
The Planning Process
P
The Planning Process
P
The Planning Process
P
The Plan
Goal: KillEnemy
Plan:
Goto (couch)
UseObject (couch)
Goto (coverNode)
AttackFromCover
The Plan
Plan:
Goto (couch)
UseObject (couch)
Goto (coverNode)
AttackFromCover
Goal: KillEnemy
The Planning Process
Goal: KillEnemy
Satisfaction: ???
Planning Process: Symbols
Simple representation of World State:
 Array of symbols
 Fixed number of symbols
 Symbol = Key, Value pair
 Union of 4 byte values (int, float, HANDLE, etc)
Planning Process: Symbols
Simple representation of agent’s World State:
 Array of symbols
 Fixed number of symbols
 Symbol = Key, Value pair
 Union of 4 byte values (int, float, HANDLE, etc)
Examples:
 kWeaponArmed = TRUE
 kUsedObject = hSomeObject
Planning Process: Symbols
Goal: KillEnemy
Satisfaction: kTargetIsDead = TRUE
Planning Process: Symbols

Agent-centric representation
Working Memory
Character X
Stim Conf = 0.6
Character Y
Stim Conf = 1.0
Character Z
Stim Conf = 0.0
Targeting
Target =
Character Y
Blackboard
Planning Process: Symbols

Agent-centric representation
–
–
–
–
kTargetIsDead
Subsystem handles target selection.
Select from Character Facts in WorkingMemory.
Less symbols to evaluate during plan formulation.
The Plan
Plan:
Goto (couch)
UseObject (couch)
Goto (coverNode)
AttackFromCover
Goal: KillEnemy
Planning Process: Actions
Attack
Goto
Use
Object
Idle
Draw
Weapon
Reload
Look At
Planning Process: Actions
Action Preconditions & Effects:
 Represented with same array of symbols
 Value may be constant or variable
Planning Process: Actions
Attack
from
Cover
Attack
Attack
from
Vehicle
E: kTargetIsDead = TRUE
Planning Process: Actions
Cost = 5
Attack
from
Cover
Attack
Attack
from
Vehicle
Cost = 1
E: kTargetIsDead = TRUE
Cost = 3
Planning Process: Context
Preconditions
Context preconditions
 Filter function of arbitrary code
–
–
–
–
Am I riding a vehicle?
Does a path exist?
Is object visible?
Is valid cover available?
Symbolic preconditions are only used when we
want the planner to chain a preceding action.
Planning Process: Context
Preconditions
A: AttackFromCover
bool ValidateContextPreconditions() Working Memory
{
Character X
// Is target visible?
Character Y
...
CoverNode29
// Does a valid node
CoverNode34
// exist?
Character Z
}
CoverNode576
Planning Process: Context
Preconditions
Query Q
Q.Type = kFact_Node
Q.Node = kNode_Cover
Fact* pFact = FindFactMax( Q, kAtt_Position )


Finds the WMFact with highest confidence in some
attribute.
If returns NULL, no valid cover exists.
The Plan
Plan:
Goto (couch)
UseObject (couch)
Goto (coverNode)
AttackFromCover
Goal: KillEnemy
Planning Process: Variables
G: KillEnemy
S: kTargetIsDead = TRUE
A: AttackFromCover
E: kTargetIsDead = TRUE
Planning Process: Variables
G: KillEnemy
S: kTargetIsDead = TRUE
A: AttackFromCover
E: kTargetIsDead = TRUE
P: kWeaponArmed = TRUE
kWeaponLoaded = TRUE
kAtObject = ???
Planning Process: Variables
A: AttackFromCover
E: kTargetIsDead = TRUE
P: kWeaponArmed = TRUE
kWeaponLoaded = TRUE
FindFactMax( Q, kAtt_Position )
kAtObject =
Planning Process: Variables
A: Goto
E: kAtObject =
Parent.kAtObject
Planning Process: Variables
A: AttackFromCover
P: kAtObject =
FindFactMax( Q, kAtt_Position )
A: Goto
E: kAtObject =
Parent.kAtObject
Planning Process: Variables
Goto
Attack
From
Cover
KillEnemy
The Plan
Plan:
Goto (couch)
UseObject (couch)
Goto (coverNode)
AttackFromCover
Goal: KillEnemy
Planning Process: Variables
A: Goto
E: kAtObject =
Parent.kAtObject
Planning Process: Variables
A: Goto
E: kAtObject =
P: kUsedObject =
Parent.kAtObject
GetDependency( Parent.kAtObject )
The Plan
Plan:
Goto (couch)
UseObject (couch)
Goto (coverNode)
AttackFromCover
Goal: KillEnemy
Was It Worth It?
Was It Worth It?
Did we handle?
 Managing dependencies 
 Sharing behaviors 
Additional benefit:
 Replanning
Replanning
P
A
Replanning
P
A
Replanning
P
A
Replanning
P
A
Replanning
P
A
Parting Thoughts
Warcraft (1994)
Parting Thoughts
Star Wars (1987)
Parting Thoughts
F.E.A.R. (2005)
Parting Thoughts
2D
Pre-rendered
graphics
3D
Real-time rendered
graphics
Parting Thoughts
Scripts, FSMs
Pre-rendered
reasoning & behavior
Planning
Real-time simulated
reasoning & behavior
What new types of gameplay are possible?!
Questions?
Jeff Orkin
Monolith Productions
http://www.jorkin.com