Hierarchical Plan Representations
for Encoding Strategic Game AI
Hai Hoang
Stephen Lee-Urban
Héctor Muñoz-Avila
Lehigh University
http://www.cse.lehigh.edu/~munoz/projects/AIGames/
Motivation for using HTN planning
versus Standard (STRIPS) Planning



Goal-Oriented Action Planning (Orkin, 2004, 2005)
 characters can find alternate solutions to situations
encountered in the game
Goals &
 characters can handle dependencies that may not have
operators
been thought of at development time
HTNs provide distinct advantages over “STRIPS” planning,
on which GOAP is based
Key questions:
 What kinds of Game AI knowledge to represent in
Game developers should
HTNs?
Learn LISP?
 How to reason in dynamic game environments?
 How to encode the HTNs
Related Work: FPS and AI

Games have the potential to become testbed for different
AI algorithms (Laird & van Lent, 1999)

FPS-based system developed and used for actual
training of military personnel (Laird & Duchim, 2000)

Soar Bots for UT using the Soar Architecture (Laird &
Duchim, 2000)

GameBots: A group at USC-ISI and CMU jointly
developed the Gamebot project to create an API for
other clients to connect to the UT server.
Related Work: HTNs and
Game AI

Hierarchical planning can be useful in games (Wallace,
2003)

Bridge Baron game uses HTN ideas (Smith et al., 1998)
 We will expand on this latter.

Hierarchical FSMs are an extension of FSMs in which
states can expand into their own sub-FSMs (Houlette &
Fu, 2003)

HTNs used for dialog generation (Cavazza & Charles,
2005)
Unreal Tournament©

Online FPS developed by Epic Games Inc.

Various gameplay modes including team deathmatch and
capture the flag

Provides a client-server
architecture for controlling
bots

High-level script
language: UnrealScript
(Sweeney, 2004)
Client-Server Architecture for
UT Bots

The UT server provides sensory information about
events in the UT world and controls all the gameplay
events.

Event-driven: The client program uses this information
to decide commands controlling the behavior of a Bot
and passes them to the server.

Two types of messages:
 Asynchronous: special events
 Synchronous: regular intervals

Variants: Enhanced server (USC/ISI), Java Bots (CMU),
Soar Bots (U. of Michigan).
Hierarchical Task Network (HTN)
Planning

Complex tasks are decomposed into simpler tasks.
task t
task t


task t
Seeks to decompose compound tasks into primitive
tasks, which define actions changing the world
The knowledge artifacts indicating how to decompose
task into subtasks are called methods
Hierarchical Task Network (HTN) Planning
Principle: complex tasks are decomposed into simpler tasks. The goal is to
decompose compound tasks into primitive tasks, which define actions
changing the world.
Travel(Lehigh,NRL)
Travel from Lehigh to NRL
alternative
methods Travel by car
Travel by plane
Enough money for air
fare available
Enough money
for gasoline
Seats available
Roads are passable
Travel(Lehigh,Newark Airport)
Drive(Lehigh, Newark Park)
Bus(Newark Park,airport)
Fly(Newark, BWI)
Travel(BWI, NRL)
Taxi(BWI, NRL)
Task:
travel(x,y)
Method: taxi-travel(x,y)
get-taxi
ride(x,y)
pay-driver
Method: air-travel(x,y)
get-ticket(a(x),a(y))
fly(a(x),a(y))
travel(a(y),y)
travel(x,a(x))
HTN Planning

travel(UMD, LAAS)
get-ticket(BWI, TLS)
go-to-travel-web-site
find-flights(BWI,TLS)
Problem reduction
BACKTRACK
 Tasks (activities) rather than goals
 Methods to decompose tasks into
subtasks
 Enforce constraints


E.g., taxi not good for long distances
Backtrack if necessary
(slide from Dana S. Nau – University of Maryland)
get-ticket(IAD, TLS)
go-to-travel-web-site
find-flights(IAD,TLS)
buy-ticket(IAD,TLS)
travel(UMD, IAD)
get-taxi
ride(UMD, IAD)
pay-driver
fly(BWI, Toulouse)
travel(TLS, LAAS)
get-taxi
ride(TLS,Toulouse)
pay-driver
HTN Planning


HTN planners may be domain-specific
 e.g., Bridge Baron
Or they may be domain-configurable
 Domain-independent planning engine
 Domain description that defines not only the
operators, but also the methods
 Problem description

domain description, initial state, initial task network
Task:
travel(x,y)
Method: taxi-travel(x,y)
get-taxi
ride(x,y)
pay-driver
Method: air-travel(x,y)
get-ticket(a(x),a(y))
fly(a(x),a(y))
travel(x,a(x))
(slide from Dana S. Nau – University of Maryland)
travel(a(y),y)
Why HTN Planning (1)?

HTN planning is provable more expressive than STRIPS
planning (Erol et al, 1994)

Even for those domains in which HTN representations can be
translated into STRIPS representations, a much larger number
of STRIPS operators is required to represent the HTN methods
(Lotem & Nau, 2000)

Fast planning through domain-configurable HTN planners
(SHOP system)
Why HTN Planning (2)?

Potential for efficiency gains: Rather than searching
through possible states and actions a top-down task
decomposition process is performed that may prune the
search space substantially
Achieve 3-tower
Pile 3rd block
C
B
A
A
C
B
Achieve 2-tower
Method:
• Task: Achieve 3-tower(?A, ?B, ?C)
• Subtasks: Achieve-2-tower(?B,?C),
Piler-3rd-block(?A,?B)
• Preconditions: on(?B,?C?), on(?C,?A)
A
B
C
A
C
A
B
A
A B C
B
A
B
C
A
B
C
B
C
B
C
A
A
B
C
B
C
A
C
B
A
C
Why HTN Planning (3)?

Hierarchical planning is natural in many domains
 Military planning
Strategic
National
Strategic
Theater
Operational
Tactical
JCS / NCA
CINC
JTF
Application to Computer Bridge
• Chess: better than all but the best humans
• Bridge: worse than many good players
• Why bridge is difficult for computers
– It is an imperfect information game
– Don’t know what cards the others have (except the dummy)
– Many possible card distributions, so many possible moves
• If we encode the additional moves as additional branches
in the game tree, this increases
the number of nodes exponentially
– worst case: about 6x1044 leaf nodes
24 leaf nodes
– Not
average
case:
about
10
enough time to search the game tree
(Dana S. Nau)
How to Reduce the Size
of the Game Tree?
• Bridge is a game of planning
– Declarer plans how to play the hand
by combining various strategies (ruffing, finessing, etc.)
– If a move doesn’t fit into a sensible strategy,
then it probably doesn’t need to be considered
• HTN approach for declarer play
– Use HTN planning to generate a game tree in which each move
corresponds to a different strategy, not a different card
• Reduces average game-tree size to about 26,000 leaf
nodes
• Bridge Baron: implements HTN planning
– Won the 1997 World Bridge Computer Challenge
– All commercial versions of Bridge Baron since 1997 have
include an HTN planner (has sold many thousands of copies)
(Dana S. Nau)
Another Application: UT Domination
Games




A number of fixed domination
locations.
When a team member steps into
one of these locations, the status
of the location changes to be
under the control of his/her team.
The team gets a point for every
five seconds that each
domination location remains
under the control of that team.
The game is won by the first team
that gets a pre-specified amount
of points.
Idea


Use of hierarchical task network (HTN) planning
techniques to laid out a grand strategy to accomplish
gaming tasks
Use standard event-driven programming allowing the
bots to react in highly dynamic environments while
contributing to the grand task
UT task: Domination
Strategy:
secure most
locations
UT action: move Bot1 to
location B
Technical Difficulties of using an
HTN Planner (1)
UT task: Domination
Strategy:
secure most
locations
UT action: move Bot1 to
location B
2. Dealing with
contingencies while
Bots are executing
actions
1. Declarative language
for expressing
conditions
Solution to 1: use builtin predicates to
represent conditions
Solution to 2: use Java
Bots to represent actions
(the usual FSMs!)
HTNBots: 3-layer Decision
Making



HTN planning layer: Which strategy to follow?
 HTN planning generates the strategy
 Input: task to perform (e.g., win a domination game), game
events
 Output to next layer: specific tasks to individual bots (e.g., Bot
“A” goes to domination location 2)
FSM layer: how does each bot performs its task?
 FSM code the bots behavior
 Input: specific task to perform, game events
 Output to next layer: bots commands (e.g., attack)
Game-engine layer:
 In charge of animation, passing events, pathfinding
 Input: bot command
 Output to previous layers: executed in game engine, game
events (e.g., see an enemy)
Data Flow in the HTN
Planning Layer
Selected method’s
subtasks
task
compound?
Yes
Select applicable
method
Evaluate method’s
preconditions
No
Execute action
Method
library
UT Bot
UT Bot
…
UT Server
Video
Empirical Results
60
50
40
30
HTN
Improved
20
10
0
1
2
3
4
5
avg
It is possible to encode strategies that coordinate
teams of bots in FPS games using HTNs
Final Remarks



HTNs can be used to model effective team strategies for
Unreal Tournament (UT) bots.
A grand strategy is laid out by the HTNs and eventdriven programming allows the bots to react in this highly
dynamic environment while contributing to the grand
task.
Three layer decision making that encompass:
 Strategy generation with HTN planning
 Individual bot behavior generation with FSMs
 Execution in game engine