Distributed Control in Multi-agent
Systems: Design and Analysis
Kristina Lerman
Aram Galstyan
Information Sciences Institute
University of Southern California
Design of Multi-Agent Systems
Multi-agent systems must function in
Dynamic environments
Unreliable communication channels
Large systems
Solution
Simple agents
No reasoning, planning, negotiation
Distributed control
No central authority
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Advantages of Distributed Control
• Robust
• tolerant of agent error and failure
• Reliable
• good performance in dynamic environments with
unreliable communication channels
• Scalable
• performance does not depend on the number of
agents or task size
• Analyzable
• amenable to quantitative analysis
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Analysis of Multi-Agent Systems
Tools to study behavior of multi-agent systems
• Experiments
• Costly, time consuming to set up and run
• Grounded simulations: e.g., sensor-based simulations of robots
• Time consuming for large systems
• Numerical approaches
• Microscopic models, numeric simulations
• Analytical approaches
• Macroscopic mathematical models
• Predict dynamics and long term behavior
• Get insight into system design
• Parameters to optimize system performance
• Prevent instability, etc.
USC Information Sciences Institute
ISI
01/23/2002
K. Lerman
Distributed Control in MAS
DC: Two Approaches and Analyses
• Biologically-inspired approach
• Local interactions among many simple agents leads
to desirable collective behavior
• Mathematical models describe collective dynamics of
the system
• Markov-based systems
• Application: collaboration, foraging in robots
• Market-based approach
• Adaptation via iterative games
• Numeric simulations
• Application: dynamic resource allocation
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Biologically-Inspired Control
Analysis of Collective Behavior
Bio control modeled on social insects
• complex collective behavior arises in simple, locally
interacting agents
Individual agent behavior is unpredictable
•
•
•
•
external forces – may not be anticipated
noise – fluctuations and random events
other agents – with complex trajectories
probabilistic controllers – e.g. avoidance
Collective behavior described probabilistically
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Some Terms Defined
• State - labels a set of agent behaviors
• e.g., for robots Search State = {Wander, Detect
Objects, Avoid Obstacles}
• finite number of states
• each agent is in exactly one of the states
• Probability distribution

• P ( n , t ) = probability system is in configuration n at
time t

• n  ( N1,, N L ) where Ni is number of agents in the
i’ th of L states
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Markov Systems
• Markov property: configuration at time t+Dt
depends only on configuration at time t
P(n, t  Dt )   P(n, t  Dt | n, t ) P(n, t ).
n
• also,  P(n, t  Dt | n, t )  1
n
• change in probability density:
P(n, t  Dt )  P(n, t )   P(n, t  Dt | n, t ) P(n, t )
n
  P(n, t  Dt | n, t ) P(n, t )
n
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Stochastic Master Equation
In the continuum limit, Dt  0
dP(n, t )
 W (n | n; t ) P (n, t )  W (n | n; t ) P (n, t )
dt
n
n
with transition rates
P(n, t  Dt | n, t )
W (n | n; t )  lim
Dt 0
Dt
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Rate Equation
Derive the Rate Equation from the Master Eqn
• describes how the average number of agents in state
k changes in time
• Macroscopic dynamical model
d N k (t )
  W (k | k ; t ) N k  (t )   W (k  | k ; t ) N k (t )
dt
k
k
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Collaboration in Robots
Stick-Pulling Experiments
(Ijspeert, Martinoli & Billard, 2001)
A. Ijspeert et al.
• Collaboration in a group of reactive robots
• Task completed only through collaboration
• Experiments with 2 – 6 Khepera robots
• Minimalist robot controller
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Experimental Results
Key observations
• Different dynamics
for different ratio
of robots to sticks
• Optimal gripping
time parameter
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Flowchart of robot’s controller
Ijspeert et al.
State diagram for a
multi-robot system
look for sticks
start
object
detected?
Y
N
Y
obstacle?
N
Y
search
obstacle
avoidance
grip
gripped?
N
success
grip & wait
Y
time out?
N
N
teammate
help?
u
s
Y
release
Model Variables
• Macroscopic dynamic variables
Ns(t) = number of robots in search state at time t
Ng(t) = number of robots gripping state at time t
M(t) = number of uncollected sticks at time t
• Parameters
• connect the model to the real system
a = rate of encountering a stick
aRG = rate of encountering a gripping robot
t = gripping time
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Mathematical Model of Collaboration
find & grip sticks successful collaboration
dNs (t )
 aN s (t )M (t )  N g (t )   aRG N s (t ) N g (t )
dt
 aN s (t  t )M (t  t )  N g (t  t )  (t ;t )
N s  N g  N0
M (t )  const
unsuccessful collaboration
for static environment
Initial conditions: N s (0)  N0 , N g (0)  0, M (0)  M 0
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Dimensional Analysis
• Rewrite equations in dimensionless form by
making the following transformations:
n(t )  N s (t ) / N 0 , t  aM 0t ,t  aM 0t
~
b  N 0 / M 0 , b  RG b
• only the parameters b and t appear in the eqns and
determine the behavior of solutions
• Collaboration rate
• rate at which robots pull sticks out
~
R( b ,t , t )  bb n(t )1  n(t )  N 0 f ( N 0 )
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Searching Robots vs Time
t=5
b=0.5
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Collaboration Rate vs t
b=1.5
Key observations
• critical b
• optimal gripping
time parameter
b=1.0
b=0.5
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Comparison to Experimental Results
b=1.5
b=1.0
b=0.5
Ijspeert et al.
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Summary of Results
• Analyzed the system mathematically
• importance of b
• analytic expression for bc and topt
• superlinear performance
• Agreement with experimental data and
simulations
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Foraging in Robots
Robot Foraging
• Collect objects scattered in the
arena and assemble them at a
“home” location
• Single vs group of robots
• no collaboration
• benefits of a group
• robust to individual failure
• group can speed up collection
• But, increased interference
Goldberg & Matarić
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Interference & Collision Avoidance
• Collision avoidance
• Interference effects
• robot working alone is more efficient
• larger groups experience more interference
• optimal group size: beyond some group size,
interference outweighs the benefits of the group’s
increased robustness and parallelism
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
State Diagram
start
look for pucks
object
detected?
obstacle?
searching
homing
avoiding
avoiding
avoid
obstacle
grab puck
go home
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Model Variables
• Macroscopic dynamic variables
Ns(t) = number of robots in search state at time t
Nh(t) = number of robots in homing state at time t
Nsav(t), Nhav(t) = number of avoiding robots at time t
M(t) = number of undelivered pucks at time t
• Parameters
ar = rate of encountering a robot
ap = rate of encountering a puck
t = avoiding time
th0 = homing time in the absence of interference
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Mathematical Model of Foraging


dN s
1
1 s
h
 a p N s M  N h  N av
 a r N S [ N s  N 0 ]  N h  N av
dt
th
t
dN h
1 h
1
h
 a p N s [ M  N h  N av
]  a r N h [ N h  N 0 ]  N av
 Nh
dt
t
th
dM
1
  Nh
dt
th
Average homing time:
t h  t h0 1  a rtN0 
Initial conditions: N s (0)  N 0 , M (0)  M 0
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Searching Robots and Pucks vs Time
robots
pucks
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Group Efficiency vs Group Size
t=1
t=5
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Sensor-Based Simulations
Player/Stage simulator
number of robots = 1 - 10
number of pucks = 20
arena radius = 3 m
home radius = 0.75 m
robot radius = 0.2 m
robot speed = 30 cm/s
puck radius = 0.05 m
rev. hom. time = 10 s
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Simulations Results
1600
avoid time = 3s
1400
avoid time = 1s
model (3 s)
1200
model (1 s)
time (s)
1000
800
600
400
200
0
0
2
4
6
8
10
number of robots
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Simulations Results
0.006
t=3 s
model 3 s
t=1 s
model 1 s
efficiency
0.004
0.002
0
0
2
4
6
8
10
number of robots
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Summary
• Biologically inspired mechanisms are feasible for
distributed control in multi-agent systems
• Methodology for creating mathematical models
of collective behavior of MAS
• Rate equations
• Model and analysis of robotic systems
• Collaboration, foraging
• Future directions
• Generalized Markov systems – integrating learning,
memory, decision making
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Market-Based Control
Distributed Resource Allocation
• N agents use a set of M common resources with
limited, time dependent capacity LM(t)
• At each time step the agents decide whether to
use the resource m or not
• Objective is to minimize the waste
w   ( Am (t )  Lm (t ))2
t m
where Am(t) is the number of agents utilizing
resource m
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Minority Games
• N agents repeatedly choose between two alternatives (labeled
0 and 1), and those in the minority group are rewarded
• Each agent has a set of S strategies that prescribe a certain
action given the last m outcomes of the game (memory)
strategy with m=3
input
action
000 001 010 011 100 101 110 111
0
1
1
0
0
1
0
1
• Reinforce strategies that predicted the winning group
• Play the strategy that has predicted the winning side
most often
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
MG as a Complex System
• Let A(t ) be the size of the group that chooses ”1” at time t
• The “waste” of the resource is measured by the standard deviation
2
  A2 (t )  A(t ) ,    - average over time
• In the default Random Choice Game (agents take either action with
probability ½) , the standard deviation is N / 2
1.5
For some memory size the waste is
smaller than in the random choice
game
standard deviation
1.25
Coordinated phase
1
0.75
0.5
0.25
0
2
4
6
8
10
12
14
memory length
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Variations of MG
• MG with local information
Instead of global history agents may use local interactions (e.g.,
cellular automata)
• MG with arbitrary capacities
The winning choice is “1” if A(t )  L where L is the capacity, A(t )
is the number of agents that chose “1”
To what degree agents (and the system as a whole) can
coordinate in externally changing environment?
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
MG on Kauffman Networks
Set of N Boolean agents: si  {0,1}, i  1..N
Each agent has
A set of K neighbors {k j }, j  1..K
A set of S randomly chosen Boolean functions of K variables Fi j , j  1..S
Dynamics is given by si (t  1)  Fi j ( sk (t ),...sk (t ))
MAX
1
K
N
The winning choice is “1” if A(t )  L(t ) where A(t )   si (t ), L(t )  L0  L(t )
i 1
1T
2
Global measure for optimality:    [ A(t )  L(t )]
T t 1
2
For the RChG (each agent chooses “1” with probability  t   L(t ) / N )
1T
  N  dt t [1   t ]
T0
2
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Simulation Results
K=2 networks show a tendency towards selforganization into a coordinated phase characterized by
small fluctuations and effective resource utilization
Traditional MG
m=6
K=2
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Results (continued)
Coordination occurs even in
the presence of vastly
different time scales in the
environmental dynamics
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Scalability
For K=2 the “variance”
per agent is almost
independent on the
group size,   N  const
In the absence of
coordination   N
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Phase Transitions in Kauffman Nets
Kauffman Nets: phase transition at K=2 separating ordered (K<2)
and chaotic (K>2) phases
For K>2 one can arrive at the
phase transition by tuning the
homogeneity parameter P (the
fraction of 0’s or 1’s in the
output of the Boolean functions)
K=3
Pc  0.78
The coordinated phase might be
related to the phase transition in
Kauffman Nets.
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Summary of Results
• Generalized Minority Games on K=2 Kauffman Nets are
highly adaptive and can serve as a mechanism for
distributed resource allocation
• In the coordinated phase the system is highly scalable
• The adaptation occurs even in the presence of different
time scales, and without the agents explicitly
coordinating or knowing the resource capacity
• For K>2 similar coordination emerges in the vicinity of
the ordered/chaotic phase transitions in the
corresponding Kauffman Nets
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS
Conclusion
• Biologically-inspired and market-based
mechanisms are feasible models for distributed
control in multi-agent systems
• Collaboration and foraging in robots
• Resource allocation in a dynamic environment
• Studied both mechanisms quantitatively
• Analytical model of collective dynamics
• Numeric simulations of adaptive behavior
USC Information Sciences Institute
01/23/2002
ISI
K. Lerman
Distributed Control in MAS