Subbarao Kambhampati
Arizona State University rakaposhi.eas.asu.edu/yochan.html

Reinforcing “small chunk at a time” behavior
--Theory of inverted reinforcement

• A quick overview of the ideas of abstraction, hierarchy, reuse and leastcommitment in planning
– With special emphasis on new-age planners
• Share some (hopefully portable) lessons...

• Planning -- Then and Now
• Abstraction/hierarchy in planning
– Detail Abstraction
• Least Commitment
– Task decomposition
– Experiential abstraction (reuse/replay)
• Lessons

 States are modeled in terms of (binary) state-variables (factored rep.)
-- Complete initial state, partial goal state
 Actions are modeled as state transformation functions
-- Syntax: ADL language (Pednault)
 Plans are sequences of actions
At(A,M),At(B,M)
¬In(A), ¬In(B)
At(A,E), At(B,E),At(R,E)
Effects
In(o
1
)
Load(o
1
)
Preconditions
At(o
1
,l
1
), At(R,l
1
)
¬In(o
1
)
Unload(o
1
)

At(R,M), ¬At(R,E) xIn ( x )

At ( x, M)
& ¬At(x, E)
Fly()
In(o
1
)
At(R,E)

All action sequences
P
All Sol
P’
P
P’
All Seq.
All Solutions
Refine
Refinements
Partial plans
Narrowing sets of action sequences to progress towards sets of solutions
Remove non-solutions
[AIMAG-97]

0
At(A,E)
At(B,E)
At(R,E)
1: Unload(A)
 progression
0
2: Load(A)
0
2: Load(B)
0
2: Fly()
1: Unload(A)
1: Unload(A)
1: Unload(A)



0
0
PSR
At(A,M)@

1:Unload(A) 
At(A,M)@

2:Fly()
In(A)@2
At(A,M)
¬At(A,M)
3:Unload(A)

0
At(A,E)
At(A,M)
1: Transport(A)
 1: Fly()
0
Regression
0
0
1: Fly()
1: Fly()
•
At(A,M)
At(B,M)
¬In(A)
¬In(B)
2: Unload(A) •
2: Unload(B)
•
0
HTN
At(A,E)
1: Load(A)
At(A,M)

2: Fly()
3: Unload(A)

Conjunctive planners Disjunctive planners
• Search in the space of conjunctive partial plans
– Disjunction split into the search space
– Solution extraction is trivial
• Examples:
– STRIPS & Prodigy
– SNLP & UCPOP
– NONLIN & SIPE
• Search in the space of disjunctive partial plans
– Disjunction handled explicitly
– Solution extraction is nontrivial
• CSP/SAT methods
• Examples:
– Graphplan
– SATPLAN
[AIMag-97;IJCAI-97]

0
1: Load(A)
0
1: Load(B)
0
1: Fly(R)
0
1: Load(A) or
2 : Load(B) or
3 : Fly(R)
•
•
•
Refining conjunctive plans
0
1: Load(A) 2: Fly(R)
0
0
1: Load(B) 2: Fly(R)
1: Load(A) 2: Unload(A,E)
0
0
1: Load(B) 2: Unload(B,E)
1: Load(A) 2: Load(B)
0
0
1: Load(B) 2: Load(A)
1: Fly(R)

Refining disjunctive plans

0
1: Load(A) or
2 : Load(B) or
3 : Fly(R)
1: Load(A) or
2 : Load(B) or
3 : Fly(R) or
4 : Unload(A,E) or
5 : Unload(B,E)






•

• Idea
– Abstract some details of the problem or actions.
– Solve the abstracted version.
– Extend the solution to the detailed version
• Precondition Abstraction
– Work on satisfying important preconditions first
• Importance judged by:
– Length of plans for subgoals [ABSTRIPS, PABLO]
– Inter-goal relations [ALPINE]
– Distribution-based [HighPoint]
– Strong abstractions (with downward refinement property) are rare
– Effectiveness is planner-dependent
• Clashes with other heuristics such as “most constrained first”

• Most planners thrash by addressing planning and scheduling considerations together
– Eg. Blocks world, with multiple robot hands
• Idea: Abstract resources away during planning
– Plan assuming infinite resources
– Do a post-planning resource allocation phase
– Re-plan if needed
(with Biplav Srivastava)

0
• Postpone commitments unless forced
PSR
– Big idea in conjunctive refinement planning
• Partial-order planners: UCPOP, SNLP
– Interacts with precondition abstraction
0
– Becomes a non-issue in disjunctive planning
• There is very little commitment to begin with
• Encodings based on partial order planning can actually be worse off [Mali, 98]
– Exception: Variablized (“lifted”) representations
2:Fly()
In(A)@2
At(A,M)
¬At(A,M)
3:Unload(A)


• Domain model contains non-primitive actions , and schemas for reducing them
• Reduction schemas are given by the designer
– Can be seen as encoding user-intent
• Two notions of completeness:
– Schema completeness
• (Partial Hierarchicalization)
– Planner completeness
GobyBus(S,D)
Getin(B,S)
Travel(S,D)
GobyTrain(S,D)
BuyTickt(T)
BuyTickt(B)
Getout(B,D)
Getin(T,S)
Getout(T,D)
Hitchhike(S,D)

GobyBus(S,D) t1: Getin(B,S)
Hv-Tkt
In(B) t2: BuyTickt(B)
Hv-Money t3: Getout(B,D)
At(D)
Get(Money) GobyBus(Phx,Msn)
Hv-Money
At(Msn)
Buy(WiscCheese)
Get(Money) t1: Getin(B,Phx)
Hv-Tkt
In(B) t2: BuyTickt(B)
Hv-Money t3: Getout(B,Msn)
At(D)
Buy(WiscCheese)

• Capturing hierarchical structure of the domain
– Motivates top-down planning
• Start with abstract plans, and reduce them
• Many technical headaches
– Respecting user-intent, maintaining systematicity and minimality [AAAI-98]
• Phantomization, filters, promiscuity, downwardunlinearizability ..
• Capturing expert advice about desirable solutions
– Motivates bottom-up planning
• Ensure that each partial plan being considered is
“legal” with respect to the reduction schemas
• Connection to efficiency is not obvious
Relative advantages are still unclear...
[Barrett, 97]

• The ideas of top-down and bottom-up HTN planning can be ported to disjunctive planners [AIPS-98]
– Abstract actions can be seen as disjunctive constraints
• Add constraints to the SAT/CSP encodings of the planning problem to ensure that:
– Abstract actions are related to primitive actions through the reduction schemas [ Top-down version ] OR
– Each primitive actions must be part of some task reduction schema
[ Bottom-up version ]
• Puzzle : How can increasing encoding sizes lead to efficient planning?
– New constraints support simplification of the original constraints
[with Amol Mali]

40x speedup
[with Mali, AIPS-98]

• Structures being reused
– Opaque vs.
Modifiable
– Solution vs. Solving process ( derivation )
• Acquisition of structures to be reused
– Human given vs.
Automatically acquired
• Mechanics of reuse
– Phased vs. simultaneous
• Costs
– Storage & Retrieval costs; Solution quality

• Modifiable derivational traces were reused
• Traces were automatically acquired during problem solving
– Analyze the interactions among the parts of a plan, and store plans for non-interacting subgoals separately
• Reduces retrieval cost
– Use of EBL failure analysis to detect interactions
• All relevant trace fragments were retrieved and replayed before the control is given to from-scratch planner
– Extension failures are traced to individual replayed traces, and their storage indices are modified appropriately
• Improves retrieval accuracy
(with Ihrig, JAIR 97)

45000
40000
35000
30000
25000
20000
15000
10000
5000
0
1
Performance with increased Training
2 3
0
4 5
600
500
400
300
200
100
0
20
6
40
60
120
Library Size
5
3
1
100
90
80
70
60
50
40
30
20
10
0
% Solvability with increased traning
(JAIR, 97)

• Harder to make a disjunctive planner commit to extending a specific plan first
• Options:
– Support opaque macros along with primitive actions
– Modify the problem/domain specification so the old plan’s constraints will be respected in any solution
– MAX-SAT formulations of reuse problem
[with Amol Mali]

• Reachability analysis
– Analyze which actions cannot be executed together and which propositions cannot be made together at particular time steps
• Graphplan mutual exclusions
• Domain invariants
• Relevance analysis
– Analyze which actions are relevant and must occur together
• Greedy Regression (RIFO)
– Operator Graphs
• Inseperability constraints
Explicate which parts of a disjunctive structure cannot be part of a solution
(focusing)
0
1: Load(A) or
2 : Load(B) or
3 : Fly(R)
1: Load(A) or
2 : Load(B) or
3 : Fly(R) or
4 : Unload(A,E) or
5 : Unload(B,E)
•

• Dual views:
Detail reduction vs. Expert advice
– Detail reduction => hierarchical solving with promise of improved efficiency
– Expert advice implies further constraints on the solutions
• Strong abstractions are rare
– Must take abstractions as advice that can be overridden
• The interaction between abstraction and search mechanism
• Emphasis on automatic generation of abstractions
– Need to consider utility issues
• Emphasis on satisficing solutions
– Few quantitative guarantees on solution quality