RMPL Model-based Program
Reactive Planning of Hidden States
in Large State Spaces Through
Decomposition and Serialization
Control Program
Executes concurrently
� Preempts
� non -deterministic choice
� A[l,u ] timing
� A at l
location
�
Brian C. Williams
Environment Model
Joint with
Seung H. Chung
T
O
W
N
E
O
HOME
Landing Site: ABC
COLLECTION POINT
Enroute
Selects consistent
threads of activity
from redundant
methods
Control
Sequencer
Schedules and Dispatches
Activities Dynamically
location estimates
location goals
Tracks
Mode
RENDEZVOUS
Diverge
Landing Site: XYZ
Kirk Model-based Executive
SCIENCE AREA 1’
SCIENCE AREA 1
Finds
least
Mode
Deductive
Controller
Estimation
Reconfiguration
location
cost paths
SCIENCE AREA 3
Executive
• pre-plans activities
Observations
• pre-plans paths
• dynamically schedules [Tsmardinos et al.]
16.412J/6.834J
March 16 th, 2005
Outline
Commands
Plant
DS 1 Attitude Control System
• The need for model-based reactive planning
• The Burton model-based reactive planner
Flight
Computer
PDE
Data
Commands
SRU
He
N2H4
PDU
z facing thrusters
BC
x facing thrusters
GDE
PASM
DSEU
1553 bus
CSAIL
Massachusetts Institute of Technology
CSAIL
Massachusetts Institute of Technology
How do we reconfigure a valve?
Remote
Terminal
Computer
Mode Reconfiguration
Driver
Bus
Control
Remote
Terminal
PEPE
Valve
Configuration
Goal
Valve
Goal
Interpreter
Driver
• Device modes are changed through indirect commanding .
• Communication paths are established by reconfiguring other devices.
Goal State
• The task of reconfiguring devices in the proper order generalizes statespace planning to handle indirect effects.
State Estimate
(Current)
• To achieve reactivity all possible plans for all possible goal states should be
pre -compiled (a generalization of universal plans).
� To achieve compactness we decompose these universal plans according
to a goal/sub-goal hierarchy.
CSAIL
Livingstone reconfigured modes using one step
commands. But how does the flight computer really
open a valve?
• Requires turning on device drivers
• Requires repairing bus controllers
• Sending commands
• Powering down devices . . .
Massachusetts Institute of Technology
Reactive
Planner
Command
Model -based Programming of Intelligent Embedded Systems and Robotic Explorers
[Williams et al., IEEE’03]
Reactive Planner for a Model -based Executive
[Williams & Nayak, IJCAI 97]
MERS
M o d e -l based Embedded & Robotic Systems Group
Massachusetts Institute of Technology
1
Example: Driver Valve Command Sequence
Goal: No thrust
Valve vlv
Valve Driver dr
vcmdin
dcmdin
Commands
ME:
GI:
MRP
ME:
MRP
ME:
MRP
ME:
MRP
ME:
Driver State
dr = off,
dr = off,
Valve State
vlv = open
vlv = closed
dr = on,
vlv = open
To achieve reactivity we eliminate all forms of search.
dcmdin = on
dcmdin = close
dr = reset failure,
vlv = open
dr = on,
vlv = open
dr = off,
vlv = open
dcmdin = reset
dcmdin = off
CSAIL
Massachusetts Institute of Technology
CSAIL
Model-based Reactive Planning
Massachusetts Institute of Technology
Model-based Reactive Planning
Achieved by:
Achieved by:
1. Eliminate Indirect Control
1. Eliminate Indirect Control
. . . through Compilation
. . . through Compilation
2. Eliminate Search for Goal Ordering
2. Eliminate Search for Goal Ordering
. . . through Reversibility and Serialization
. . . through Reversibility and Serialization
3. Eliminate Search to find Suitable Transitions
3. Eliminate Search to find Suitable Transitions
. . . by Constructing Hierarchical Polices
CSAIL
. . . by Constructing Hierarchical Polices
Massachusetts Institute of Technology
CSAIL
To Handle Indirect Control . . .
Massachusetts Institute of Technology
. . . Compile Out Constraints
dcmdout = vcmdin
dcmdout = vcmdin
dcmdin = dcmdo u t
on
dcmdin = reset
dcmdin = dcmdo u t
inflow = outflow
resettable
stuck
open
open
dcmdin = on dcmdin = off
inflow = outflow
resettable
dcmdin = on dcmdin = off
vcmdin = open
dcmdin = off
off
on
dcmdin = reset
failed
vcmdin = close
dcmdin = off
stuck
closed
closed
off
failed
stuck
open
open
driver = on
vcmdin = open
driver = on
vcmdin = close
dcmdin = open
dcmdin = close
stuck
closed
closed
flowin
dcmdin
vcmdin
flowin
dcmdin
flowout
flowout
CSAIL
Massachusetts Institute of Technology
vcmdin
CSAIL
Massachusetts Institute of Technology
2
. . . Compile Out Constraints
To Compile Out Constraints
• Eliminate intermediate variables.
� Transitions are conditioned on mode and control variables
dcmdin = reset
on
stuck
open
open
resettable
• Generate transitions as prime implicates:
dcmdin = on dcmdin = off
driver = on
dcmdin = off
off
dcmdin = open
driver = on
F i � next(yi = ei)
dcmdin = close
stuck
closed
closed
failed
where F i is a conjunction of mode and control variable
assignments.
� Prime implicates for transitions enumerated using OpSAT
dcmdin
– 40 seconds on SPARC 20 for 12,000 clause spacecraft model.
CSAIL
Massachusetts Institute of Technology
CSAIL
Massachusetts Institute of Technology
Why Search is Needed
Model-based Reactive Planning
1) An achieved goal can be clobbered by a subsequent goal.
Achieved by:
1. Eliminate Indirect Effects
Driver
Valve
command
. . . through Compilation
2. Eliminate Search for Goal Ordering
• Example
. . . through Reversibility and Serialization
– Current State: driver = on, valve = closed
– Goal State: driver = off, valve = open
– Achieving (driver = off) and then (valve = open) clobbers (driver = off)
3. Eliminate Search to find Suitable Transitions
�
. . . by Constructing Hierarchical Polices
CSAIL
Massachusetts Institute of Technology
Achieve Valve goal before Driver goal
CSAIL
Massachusetts Institute of Technology
Observation: Engineered systems tend not to have loops
Remote
Terminal
Computer
Bus
Control
Driver
Valve
Solution
• The only variables used to set some variable (y7) is its
ancestors ,
� y7 can be changed without affecting its descendants .
Remote
Terminal
Driver
Valve
Affected 13
dcmdin
Valve
– Define: Causal Graph G of compiled transition system S
• vertices are state variables.
• edge from v i to v j if v j’s transition is conditioned on v i.
8
2
3
4
1
� Safe to achieve goals in an upstream order.
• Simple check:
– Requirement: The causal graph is acyclic.
CSAIL
Unaffected
5
6
7
12
9
�
10
11
Driver
– Number causal graph depth first
– achieve goals in order of increasing depth first number.
Work conjunctive goals upstream from outputs to inputs
Massachusetts Institute of Technology
CSAIL
Massachusetts Institute of Technology
3
Why Search is Needed
• Sibling goals (7,4) may both need shared ancestors.
2) Two goals can compete for the same variable in their
subgoals .
Not Shared 13
10
11
Switch
8
1
Shared
Latch1
data
2
5
Unaffected
6
7
12
2
3
9
4
1
• But ancestors no longer needed once goal (7) is satisfied.
Latch2
13
10
11
• Example
– Latch1 and Latch2 compete for the position of Switch if latch goals
achieved concurrently.
5
Unaffected
6
7
12
8
2
3
4
Not Shared 9
1
• Solution: Solve one goal before starting next sibling (Serialization).
• Feature: Generates first control action of plan first!
CSAIL
Massachusetts Institute of Technology
Why Search is Needed
CSAIL
Massachusetts Institute of Technology
Solution: Mark Allowed Transitions/Assignments
3) A state transition of a subgoal variable has irreversible effect.
dcmdin
3
Driver
on
Switch
dcmdin = on
2
resettable
dcmdin = off
1
stuck
open
open
driver = on
dcmdin = off
Latch2
• Example
off
– Assume Switch can be used once,
– Then Latch1 must be latched before Latch2.
•
dcmdin = reset
Valve
1
Latch1
data
2
dcmdin = open
failed
driver = on
dcmdin = close
stuck
closed
closed
• Mark all control variable assignments allowed:
But irreversible effects aren’t desirable for reactive planners
� Don’t allow irreversible actions
�. . . Except to repair failure modes
CSAIL
Massachusetts Institute of Technology
Solution: Mark Allowed Transitions/Assignments
dcmdin
3
Driver
on
dcmdin = on
2
dcmdin = reset
Valve
resettable
dcmdin = off
off
dcmdin = open
failed
Massachusetts Institute of Technology
Solution: Mark Allowed Transitions/Assignments
1
dcmdin
stuck
open
open
driver = on
dcmdin = off
CSAIL
3
Driver
on
dcmdin = on
driver = on
dcmdin = reset
Valve
resettable
dcmdin = off
stuck
closed
off
dcmdin = open
failed
stuck
open
driver = on
dcmdin = close
stuck
closed
closed
• Mark all control variable assignments allowed:
• Mark all control variable assignments allowed:
• For each mode variable v, in decreasing order of DF number:
• For each mode variable v, in decreasing order of DF number:
• Select each transition of v, whose guard has only allowed assignments.
1
open
driver = on
dcmdin = off
dcmdin = close
closed
2
• Select each transition of v, whose guard has only allowed assignments.
• Given current assignment v = I for v:
• Find strongly connected component of selected transitions that contains I.
• Mark assignments and transitions in SCC allowed.
CSAIL
Massachusetts Institute of Technology
CSAIL
Massachusetts Institute of Technology
4
Solution: Mark Allowed Transitions/Assignments
dcmdin
3
Driver
2
Valve
on
dcmdin = on
dcmdin
stuck
open
open
dcmdin = off
driver = on
dcmdin = open
3
2
Driver
dcmdin = off
driver = on
driver = on
dcmdin = open
dcmdin = close
stuck
closed
closed
off
• Mark all control variable assignments allowed:
• Mark all control variable assignments allowed:
• For each mode variable v, in decreasing order of DF number:
• For each mode variable v, in decreasing order of DF number:
• Select each transition of v, whose guard has only allowed assignments.
• Select each transition of v, whose guard has only allowed assignments.
• Given current assignment v = I for v:
• Given current assignment v = I for v:
• Find strongly connected component of selected transitions that contains I.
• Find strongly connected component of selected transitions that contains I.
• Mark assignments and transitions in SCC allowed.
• Mark assignments and transitions in SCC allowed.
CSAIL
Massachusetts Institute of Technology
CSAIL
3
Driver
2
Valve
on
1
Massachusetts Institute of Technology
Model-based Reactive Planning
Solution: Mark Allowed Transitions/Assignments
dcmdin
stuck
open
open
dcmdin = close
stuck
closed
1
Valve
on
dcmdin = on
driver = on
closed
off
Solution: Mark Allowed Transitions/Assignments
1
Achieved by:
open
1. Eliminate Indirect Effects
dcmdin = on
dcmdin = off
driver = on
driver = on
. . . through Compilation
dcmdin = open
dcmdin = close
closed
off
2. Eliminate Search for Goal Ordering
• Mark all control variable assignments allowed:
. . . through Reversibility and Serialization
• For each mode variable v, in decreasing order of DF number:
• Select each transition of v, whose guard has only allowed assignments.
3. Eliminate Search to find Suitable Transitions
• Given current assignment v = I for v:
• Find strongly connected component of selected transitions that contains I.
. . . by Constructing Hierarchical Polices
• Mark assignments and transitions in SCC allowed.
CSAIL
Massachusetts Institute of Technology
Solution
CSAIL
Massachusetts Institute of Technology
Plan by passing sub-goals up causal graph
Goal: Driver = off, Valve = closed
Goal
Current
open
driver = on
cmd = open
closed
Open
driver = on
cmd = close
Closed
Stuck
Open
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
idle
fail
fail
Current: Driver = off, Valve = open
Driver
Goal
Current
• Convert automata into hierarchical policies, one per automaton
– Policy selects first transition towards achieving each automata goal
state, given current state.
– Policy maps goals to subgoals and commands, in proper order
– Ensures only reversible transitions are taken,
by only using transitions marked allowed.
CSAIL
Massachusetts Institute of Technology
On
Off
On
idle
cmd = off
Off
cmd = on
idle
cmd = reset
cmd = off
Resettable
CSAIL
Valve
2
Goal
Current
Open
Closed
Stuck
1
Open
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
idle
fail
fail
Massachusetts Institute of Technology
5
Plan by passing sub-goals up causal graph
Goal: Driver = off, Valve = closed
Goal: Driver = off, Valve = closed
Current: Driver = off, Valve = open
Driver
Goal
Current
Valve
Goal
Current
Off
On
idle
cmd = off
Off
cmd = on
idle
cmd = reset
cmd = off
Resettable
Current: Driver = off, Valve = open
2
On
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
fail
Open
Closed
1
Open
Stuck
CSAIL
idle
cmd = off
idle
Off
cmd = on
idle
fail
Resettable
cmd = reset
cmd = off
cmd = off
Off
cmd = on
idle
cmd = reset
cmd = off
Resettable
Valve
Goal
Current
idle
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
fail
Open
Closed
Stuck
Driver
fail
fail
Stuck
Goal
Current
Valve
2
Goal
Current
On
idle
cmd = off
idle
Off
cmd = on
idle
fail
Resettable
cmd = reset
cmd = off
Goal: Driver = off, Valve = closed
Open
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
idle
fail
fail
Open
Closed
1
Stuck
CSAIL
Massachusetts Institute of Technology
Plan by passing sub-goals up causal graph
Goal: Driver = off, Valve = closed
Current: Driver = on, Valve = open
Current: Driver = resettable, Valve = open
Valve
2
1
Send
cmd = reset
CSAIL
idle
Plan by passing sub-goals up causal graph
Off
Plan by passing sub-goals up causal graph
Driver
Valve
2
1
Send
cmd = close
On
Off
On
idle
cmd = off
Off
cmd = on
idle
cmd = reset
cmd = off
Resettable
driver = on
cmd = open
Massachusetts Institute of Technology
On
Massachusetts Institute of Technology
Goal
Current
driver = on
cmd = close
CSAIL
1
Open
CSAIL
Driver
Closed
idle
Current: Driver = resettable, Valve = open
Failed
Resettable
2
On
Open
Open
Closed
1
Goal: Driver = off, Valve = closed
Current: Driver = resettable, Valve = open
Off
Goal
Current
On
Goal: Driver = off, Valve = closed
On
Goal
Current
Valve
2
Off
Plan by passing sub-goals up causal graph
Goal
Current
Driver
Send:
cmd = on
On
Massachusetts Institute of Technology
Driver
Plan by passing sub-goals up causal graph
Goal
Current
Open
Closed
Stuck
Goal
Current
Open
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
idle
fail
fail
Resettable
Massachusetts Institute of Technology
CSAIL
On
Off
On
idle
cmd = off
Off
cmd = on
idle
cmd = reset
cmd = off
Goal
Current
Open
Closed
Stuck
Open
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
idle
fail
fail
Massachusetts Institute of Technology
6
Plan by passing sub-goals up causal graph
Plan by passing sub-goals up causal graph
Goal: Driver = off, Valve = closed
Goal: Driver = off, Valve = closed
Current: Driver = on, Valve = closed
Current: Driver = off, Valve = closed
Driver
Send
cmd = off
Goal
Current
2
On
Off
On
idle
cmd = off
Off
cmd = on
idle
cmd = reset
cmd = off
Resettable
1
Valve
Goal
Current
Open
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
fail
Open
Closed
Stuck
CSAIL
Driver
Goal
Current
Off
On
idle
cmd = off
idle
Off
cmd = on
idle
fail
Resettable
cmd = reset
cmd = off
1
Valve
2
On
Massachusetts Institute of Technology
Success
Goal
Current
Open
Closed
idle
driver = on
cmd = close
driver = on
cmd = open
idle
fail
fail
Open
Closed
Stuck
CSAIL
Massachusetts Institute of Technology
Complexity of Reactive Planning
Hierarchical, Model-based Reactive Planning
• Compile-time Analysis:
– Compile-out interactions
– Confirm schematics are loop free.
– Depth first number variables.
• Worst Case per action: Depth * Sub-goal branch factor
• Average Cost per action: Sub-goal branch factor
• Periodic, Run-time Analysis:
– Given initial state
Valve1 = open
• Identify allowed transitions and assignments
Valve2 = open
Driver1 = off
Driver2 = off
CU = on
CU = on
– Given autonomous jump to failure state
• Identify allowed transitions and assignments
Driver1 = on
• Run-time Plan Execution:
–
–
–
–
Work conjunctive goals from outputs to inputs.
Achieve goals serially.
Only perform reversible transitions.
Lookup control actions and sub -goals in policies
CSAIL
Driver2 = on
CU = on
CU = on
Massachusetts Institute of Technology
CU = on
CU = on
CSAIL
What If Plan is Not Serializable?
Massachusetts Institute of Technology
Composing Cyclic Components
Transmitter
Amplifier
Antenna
Computer
Bus
Control
K-band
Transmitter
Amplifier
on
Antenna
K-band
Transmitter
Amplifier
A = off
cmdT = on
off
• What if causal graph G contains cycles?
• Solution:
onT
onA
on
A = off
cmdT = off
T = on
cmdA = on
off
�
cmdA = off
cmdA = on
cmdT = on
cmdA = off
off T
onA
cmdA = off
onT
off A
cmdT = off
off T
off A
– Isolate the cyclic components (compute SCCs)
– compose each cycle into a single component.
• New causal graph G’ is acyclic,
• Goals of G’ are serializable
CSAIL
Massachusetts Institute of Technology
CSAIL
Massachusetts Institute of Technology
7
Policy for Composed Components
Model-based Reactive Planning
• Problem: Composition grows
exponential in space usage.
onT
onA
• Solution: Use BDD encoding
(Seung Chung SM).
cmdA = off
cmdA = on
cmdT = on
OnT, OnA
idle
OnT, OffA
cmd A = on
OffT, OffA cmd T = on
OffT, OnA
fail
OnT, OffA OffT, OffA OffT, OnA
cmd A = off
cmd A = off
fail
fail
idle
cmd T = off
cmd T = on
idle
fail
fail
cmd A = off
idle
off T
onA
cmdA = off
cmdT = off
off T
off A
�
Goal
Current
OnT, OnA
onT
off A
CSAIL
Massachusetts Institute of Technology
1.
2.
3.
4.
Compile away constraints from the model
Compile away cyclic components
Plan serially pursuing causal graph upstream
Generate actions using hierarchical policies
Only performs reversible actions
Responds to failure at each step
Average cost per step = subgoal branching factor
CSAIL
Massachusetts Institute of Technology
Model-based Programming
of Embedded Systems
Next Challenge: Mars Smart Lander (2009)
• To survive decades embedded systems orchestrate
complex regulatory and immune systems.
• Future systems will be programmed with models,
describing themselves and their environments.
• Runtime kernels will be agile, deducing and planning by
solving optimization problems with propositional
Mission Duration: 1000 days
Total Traverse: 3000-69000 meters
Meters/Day: 230-450
Science Mission: 7 instruments, sub-surface science
package (drill, radar), in-situ sample “lab”
Technology Demonstration:
(2005).
CSAIL
constraints.
• Model-based reactive planners respond quickly to failure,
while using compile-time analysis of structure to respond
quickly and concisely to indirect effects.
Massachusetts Institute of Technology
Telecommunication Subsystem Example
Appendix 1
Handling Cycles In
The Causal Graph
Slides by Seung Chung
Computer
•
Computer
•
Bus Controller
•
Transmitter
•
Amplifier
Transmitter
Amplifier
Antenna
Transmitter
Amplifier
Antenna
Bus
Controller
– Controls the devices and sends data to the devices.
– Routes the commands and the data to the appropriate devices.
– Generates a signal that corresponds to the data to be transmitte d.
– Amplifies the signal and transmits it to an antenna.
MERS
M o d e -l based Embedded & Robotic Systems Group
Massachusetts Institute of Technology
8
Interdependent Components
Composing Strongly Connected CA
• Compose all automata into a single automaton
• Turning the transmitter on or off can
generate a noise (i.e. transient signal).
Transmitter
on
B = on
A = off
cmdT = on
• The transient signal may damage the
amplifier.
B = on
A = off
cmdT = off
RSCC =
Transmitter
�R
i
Amplifier
off
• The amplified transient signal may
damage other devices down stream of
the amplifier.
Amplifier
onT
onA
on
on
on
• Constraint on the system:
B = on
T = on
cmdA = on
– The amplifier must be turned off before
the transmitter can be turned on or off.
– The transmitter must be turned on
before the amplifier can be turned on.
B = on
cmdA = off
off
MERS
onT
onA
on
B = on
A = off
cmdT = on
B = on
A = off
cmdT = off
B = on
T = on
cmdA = on
B = on
cmdA = off
?
off T
onA
off
off
B = on
cmdA = off
B = on
cmdT = off
off T
off A
B = on
cmdA = off
One Transition Missing!
B = on
A = off
cmdT = off
T = on
A = off
B = on
T = on
cmdA = on
MERS
T = off
T = on
A = off
A = on
A = off
on
cmdB = on
cmdB = off
B = on
cmdT = off
cmdA = on
T = off
A = on
off
Amplifier
on
on
B = on
A = off
cmdT = off
off
Massachusetts Institute of Technology
B = on
A = off
cmdT = off
B = on
T = on
cmdA = on
d!
r
a
z
Ha
T = off
T = on
A = off
A = on
B = on
cmdT = off
cmdA = on
T = off
A = on
• Unless system is perfectly synchronous, simultaneous commanding
cannot be guaranteed.
• If the amplifier were commanded on first, and then the transmitter is
commanded off, the amplifier can be damaged.
MERS
Massachusetts Institute of Technology
M o d e -l based Embedded & Robotic Systems Group
Assuring Proper Execution of
Interdependent Transitions
Transmitter
B = on
A = off
cmdT = on
off T
off A
• Both the transmitter and the amplifier depend on one another for the
transition.
Concurrent Automata (CA)
Bus Controller
B = on
cmdT = off
• The transmitter must be commanded “off” and the amplifier must be
commanded “on” precisely at the same time.
Massachusetts Institute of Technology
M o d e -l based Embedded & Robotic Systems Group
B = on
cmdA = off
onT
off A
Simultaneous Commanding
T = on
B = on
cmdT = on
off T
onA
off
M o d e -l based Embedded & Robotic Systems Group
onT
off A
B = on
cmdA = on
B = on
cmdA = on
B = on
cmdT = on
B = on
cmdA = off
MERS
Interdependent Concurrent Transitions
on
B = on
B = on
A = off
T = on
cmdT = off cmdA = on
off
Massachusetts Institute of Technology
M o d e -l based Embedded & Robotic Systems Group
B = on
A = off
cmdT = on
B = on
cmdA = off
B = on
T = on
cmdA = on
B = on
cmdA = off
• Enforce concurrency as interleaving:
– For a given transition, the interdependent state constraints
become the pre- and post-conditions.
– No change to all other automata that are not independent.
Transmitter
T = on
B = on
A = off
cmdT = off
A = off
B = on
T = on
cmdA = on
off
T = off
B = on
T = on
A = off
cmdT = off
T = off
A = off
T = on
A = off
B = on
cmdA = on
T = on
A = on
• Synchronous
– Assume that each automaton performs a single state transition at each time
step.
Amplifier
• Interleaved execution within a time step
– A single main processor executes synchronous activities by interleaving.
– Devices are not synchronized.
MERS
M o d e -l based Embedded & Robotic Systems Group
Massachusetts Institute of Technology
MERS
M o d e -l based Embedded & Robotic Systems Group
A = on
Massachusetts Institute of Technology
9