Hierarchical Plan Representations
for Encoding Strategic Game AI
Hai Hoang
Stephen Lee-Urban
Héctor Muñoz-Avila
Lehigh University
http://www.cse.lehigh.edu/~munoz/projects/AIGames/
Outline

Motivation
Related Work

Background






Unreal Tournament© (UT)
Hierarchical task Network (HTN) Planning
Encoding Strategic Game AI in HTNs
 Coordinating teams of UT Bots using HTNs
 Video
Using Processes-Based Languages to Encode HTNs
Final Remarks
 Future work
Planning Operators
Motivation: FSM vs. Planning
FSM:
Monster In Sight
Patrol
Fight
No Monster
•Patrol
Preconditions:
No Monster
Effects:
patrolled
•Fight
Preconditions:
Monster in sight
Effects:
No Monster
A resulting plan:
Monster in sight
Fight
No Monster
patrolled
Patrol
Neither is
more
powerful
than the
other one
But Planning Gives More
Flexibility

“Separates implementation from data” --- Orkin
knowledge
Planning Operators
inference
•Patrol
Preconditions:
No Monster
Effects:
patrolled
•Fight
Preconditions:
Monster in sight
Effects:
No Monster
…
Many potential plans:
Fight
Fight
Fight
Fight
Fight
…
Patrol
Patrol
Patrol
Patrol
Patrol
Motivation: HTNs vs. GOAP



Goal-Oriented Action Planning (Orkin, 2004, 2005)
 characters can find alternate solutions to situations
encountered in the game
Goals &
 characters can handle dependencies that may not have
operators
been thought of at development time
HTNs provide distinct advantages over “STRIPS” planning,
on which GOAP is based
Key questions:
 What kinds of Game AI knowledge to represent in
Game developers should
HTNs?
Learn LISP?
 How to reason in dynamic game environments?
 How to encode the HTNs
Related Work: FPS and AI

Games have the potential to become testbed for different
AI algorithms (Laird & van Lent, 1999)

FPS-based system developed and used for actual
training of military personnel (Laird & Duchim, 2000)

Soar Bots for UT using the Soar Architecture (Laird &
Duchim, 2000)

GameBots: A group at USC-ISI and CMU jointly
developed the Gamebot project to create an API for
other clients to connect to the UT server.
Related Work: HTNs and
Game AI

Hierarchical planning can be useful in games (Wallace,
2003)

Bridge Baron game uses HTN ideas (Smith et al., 1998).

Hierarchical FSMs are an extension of FSMs in which
states can expand into their own sub-FSMs (Houlette &
Fu, 2003)

HTNs used for dialog generation (Cavazza & Charles,
2005)
Unreal Tournament©

Online FPS developed by Epic Games Inc.

Various gameplay modes including team deathmatch and
capture the flag

Provides a client-server
architecture for controlling
bots

High-level script
language: UnrealScript
(Sweeney, 2004)
Client-Server Architecture for
UT Bots

The UT server provides sensory information about
events in the UT world and controls all the gameplay
events.

Event-driven: The client program uses this information
to decide commands controlling the behavior of a Bot
and passes them to the server.

Two types of messages:
 Asynchronous: special events
 Synchronous: regular intervals

Variants: Enhanced server (USC/ISI), Java Bots (CMU),
Soar Bots (U. of Michigan).
Hierarchical Task Network (HTN)
Planning

Complex tasks are decomposed into simpler tasks.
task t
task t


task t
Seeks to decompose compound tasks into primitive
tasks, which define actions changing the world
The knowledge artifacts indicating how to decompose
task into subtasks are called methods
Why HTN Planning (1)?

HTN planning is provable more expressive than STRIPS
planning (Erol et al, 1994)

Even for those domains in which HTN representations can be
translated into STRIPS representations, a much larger number
of STRIPS operators is required to represent the HTN methods
(Lotem & Nau, 2000)

Fast planning through domain-configurable HTN planners
(SHOP system)
Why HTN Planning (2)?

Hierarchical planning is natural in many domains
 Military planning
Strategic
National
Strategic
Theater
Operational
Tactical
JCS / NCA
CINC
JTF
Application: UT Domination
Games




A number of fixed domination
locations.
When a team member steps into
one of these locations, the status
of the location changes to be
under the control of his/her team.
The team gets a point for every
five seconds that each
domination location remains
under the control of that team.
The game is won by the first team
that gets a pre-specified amount
of points.
Idea


Use of hierarchical task network (HTN) planning
techniques to laid out a grand strategy to accomplish
gaming tasks
Use standard event-driven programming allowing the
bots to react in highly dynamic environments while
contributing to the grand task
UT task: Domination
Strategy:
secure most
locations
UT action: move Bot1 to
location B
Technical Difficulties of using an
HTN Planner (1)
UT task: Domination
Strategy:
secure most
locations
UT action: move Bot1 to
location B
2. Dealing with
contingencies while
Bots are executing
actions
1. Declarative language
for expressing
conditions
Solution to 1: use builtin predicates to
represent conditions
Solution to 2: use Java
Bots to represent
actions (event-driven)
Data Flow to generate HTNs
(i.e., HTN Planning)
Selected method’s
subtasks
task
compound?
Yes
Select applicable
method
Evaluate method’s
preconditions
No
Execute action
Method
library
UT Bot
UT Bot
…
UT Server
Video
Empirical Results
60
50
40
30
HTN
Improved
20
10
0
1
2
3
4
5
avg
It is possible to encode strategies that coordinate
teams of bots in FPS games using HTNs
Using Processes to Encode
Game AI
Think-Act Cycle
Shoot Something
Go Somewhere Else
Call Shoot
Operator
Ask Decision System:
Where Do I Go?
Ask Decision System:
Where Do I Go?
Pick up a Healthpack
Call Pickup
Operator
TIELT Project: Middleware between
Games and AI Decision Systems
TIELT’s User Interface
Game
Engine
Library
Evaluation
Interface
Selected
Game
Engine
Processed
State
Action
TIELT’s
Internal
Communication
Modules
Decision
Learned
Knowledge
(inspectable)
Game
Player(s)
TIELT’s
KB
Editors
Game
Model
Game
Interface
Model
Decision
System
Interface
Model
Selected
Decision
System
Agent
Description
Experiment
Methodology
Selected/Developed Knowledge Bases
Knowledge
Base
Libraries
GM
GM
GM
GIM
GIM
GIM
DSIM
DSIM
DSIM
AD
AD
AD
Decision
Reasoning
System
Reasoning
System
System
Learning
Learning
Module
Learning
Module
Module
...
...
...
TIELT
User
Coordination
Interface
Raw State
...
Full
Spectrum
Command
Prediction
Interface
TIELT
User
Stratagus
EE2
Advice
Interface
Decision
System
Library
EM
EM
EM
Learning
Learning
Module
Learning
Module
Module
Result: Processes can Be Translated
into HTNs
TIELT
HTN
Iterations (while)
Recursion
Iterations (for)
Convert to while, recursion
Tasks: preconditions and effects
Methods + operators
If-elseIf-else
Method
Functions
Parameters of task (or call to
external code, or Horn clause)
Set-statement
= command
Priorities
If-then-else
Final Remarks



HTNs can be used to model effective team strategies for
Unreal Tournament (UT) bots.
A grand strategy is laid out by the HTNs and eventdriven programming allows the bots to react in this highly
dynamic environment while contributing to the grand
task.
Processes languages (e.g., TMKLs) can be translated
into HTNs
Future Work



Reinforcement learning to learn adequate task
decomposition
Case-based reasoning techniques to capture and reuse
HTNs
Experiment to measure performance of translated
processes