AAAI Technical Report FS-12-04
Human Control of Bioinspired Swarms
Controlling Swarms of Unmanned Vehicles through User-Centered Commands
Gilles Coppin
François Legras
Télécom Bretagne – Institut Mines-Télécom
CNRS UMR 6285, Lab-STICC
Brest, France
Deev Interaction, SAS
Brest, France
Abstract
managing simultaneously macro and micro level of control
for different entities.
Intrinsically, a member of a bio-inspired swarm will have
local and environment-driven rules that will be directly impacting on its autonomous navigation function. The level of
randomness and the impossibility of controlling the environment itself lead to trajectories that may seem erratic for the
human operator, or at least impossible to correlate to highlevel objectives that the operator could be likely to reach and
patrol. When controlling “down” at the micro-level of each
agent / UAV, the operator is rapidly overwhelmed and cannot
provide an adequate amount of attention for managing more
than a few UAVs. Moreover, this would mean going back
to deterministic plan-driven control, that is out of scope by
nature for swarms.
The main results issued from our first experiments (Legras
et al. 2008; Coppin and Legras 2012) were that the swarm
approach seemed to be robust and adapted for simple mission of surveillance, but that the operators in charge of
such a system were not ready to understand and dialog with
this new kind of system, so that the global performance of
the system was potentially spoiled by human intervention.
These conclusion is shared by recent work by Kolling et al.
who observe that “human operator are generally poor at
solving foraging tasks with swarms, not beating the simplest
form of autonomy –” (Kolling, Nunnally, and Lewis 2012).
Consequently and along with other researchers in the domain (Miner and desJardins 2009), we propose to think
about interaction at the swarm level (or at least sub-part of
the swarm) rather than at the individual agent’s level within
the swarm. Therefore it was proposed to define some intermediary (or transactional) level of interaction that could fill
the gap between human and agents representation.
This paper presents innovative and intuitive principles
of controlling swarms of unmanned vehicles through
mission oriented interactions handled on a tabletop.
Embedded bio-inspired algorithms are based on the use
of artificial pheromones and adapted to the high-level
command language mostly based on gestures. Users’
feedback present a synthetic real-time indicator of the
current quality of the situation assessment. Preliminary
tests results of experiments handled on a scenario of
multi-area patrolling are presented.
1
Swarms and UAVs
Unmanned Vehicle Systems (UVSs) will considerably
evolve within the next two decades. In the current generation of UV Systems, several ground operators operate a single vehicle with limited autonomous capabilities, whereas,
in the next generation of UV Systems, a ground operator
will have to supervise a system of several cooperating vehicles performing a joint mission, i.e. a Multi-Agent System
(MAS) (Johnson 2003; Coppin and Legras 2012). In order
to enable mission control, the autonomy of the vehicle and
of the system will increase and will require new and richer
forms of Human-system interaction.
The SMAART project (Legras et al. 2008) demonstrated
that interaction between a Human operator and a swarm intelligence is far from being trivial to implement. As this
technology shows great promises for empowering large distributed robotic systems, the SUSIE project (SUpervision de
Système d’Intelligence en Essaim, supervising swarm intelligence systems) was initiated in order to propose new modes
of interaction beyond the simple “mouse clicks – waypoints”
paradigm: in a large system the individual robot is probably
not the right level of action for an operator, and it may even
be superfluous to display each robot’s position.
2
Topology-Based Commands
Our approach relies on a topological and geometrical description of objects and actions to be effectuated on these
objects. On the algorithm side, these topological features
(or constraints) are interpreted in terms of pheromone distribution (masks) or evolution. This has naturally potential
indirect effects on the behaviors of the swarm itself. On the
operator side, this allows to map operational concepts on traditional geometry defined intuitively through the interaction,
and therefore to support a natural and easy way of sending
The Need for an Adequate Command
Architecture
Interacting with swarms gathering a large number of agents
(in our case UAVs) strongly emphasizes the dilemma of
c 2012, Association for the Advancement of Artificial
Copyright Intelligence (www.aaai.org). All rights reserved.
21
orders to the swarms.
Table 1 was defined as an elementary taxonomy of
topology-based tasks to be supported by a generic swarmbased surveillance or detection system. A set of actions is
systematically crossed with a set of topological features in
order to explore the space of operational possibilities. The
actions that we identified are:
The human operator has to express is intent within the
available combinations of topological features and actions
(e.g patrol a border, avoid a zone), this command is then
converted in an input in a form suitable for the corresponding swarm algorithms and send to the swarm for execution.
Watch implies for the agents to cover the associated topological feature with their sensors’ field of perception. This
coverage should be as thorough (cover the feature in its
entirety) and as permanent (cover the feature at every moment) as possible. This action should result in detections
i.e. the agents’ sensors registering a certain phenomenon.
Project Susie was an effort to test these principles by producing a prototype of a simulated swarm system that would
demonstrate very intuitive and innovative interaction principles. The principles of this demonstrator are:
3
Application: Project Susie
• use a multitouch tabletop system as a way for the operator
to easily express topological commands;
Avoid implies to maneuver in order to put as much distance
as is reasonable between the agents and the topological
feature. This action should be parameterized with a maximum and/or minimum distance.
• do not give control of individual agents, but let the operator act at the swarm level (no micro-management).
Swarm Characteristics and Missions
Find is the action of moving oneself and one’s sensors in order to detect an object in the environment, some of whose
characteristics are known (e.g. a red car). This is slightly
different from Watch in the sense that an unique or small
number of such objects are known to exist and the agents
are not waiting for the object to appear, but are actively
searching for it.
The system selected for Susie consists in a few tens of small,
simulated fixed-wing uninhabited aircrafts (15-30 in most of
our tests) that are able to perform detection missions under
the supervision of a human operator from a tabletop multitouch system. The demonstrator is able to run two kinds of
scenarios:
• detection and identification of suspicious vehicle convoys
on roads or in the open (via EOIR sensors);
Follow makes agents align their trajectories with a topological feature.
• detection of forest fire (chemical sensors).
Intercept makes the agents maneuver so that their trajectories intersect a topological feature as fast as possible.
In a typical scenario, the UAVs start in a loitering zone
that also acts as a simplified refueling zone. The operator
activates or creates operational areas to which some of the
UAVs are affected in order to perform detections. These
decisions are made by analyzing various sources of intelligence that come from inside and outside the swarm system.
When the UAVs are on-site, the system reports to the operator the detections made by the UAVs. IF these detections
are confirmed by the operator, he or she his able to position
an exclusion corridor that the UAV swarm leave in order to
allow another aircraft to intervene safely (e.g. fire-fighting
plane).
We wanted the operator to be able to deploy the swarm
on several zones of interest at the same time, therefore the
surveillance of multiple topological features (tasks) can be
conducted at the same time. In order to facilitate this — and
to further free the operator from the micromanagement of
individual vehicles — upon activation or creation of a task,
the system computes a nominal subset of the swarm from the
loiter zone (if available) and affects these UAVs to said task.
The human operator is able to manage the system by requesting transfers of resources (UAVs) between tasks (loiter
zone and tasks). UAVs are also able to refuel autonomously
by going back to the loiter zone.
The topological features that were selected are: points,
lines (open or closed), areas and mobile objects (actually a
mobile point, one could consider a spreading phenomenon
like a cloud). This process allowed us to obtain a set of
high-level, behaviors that maps to a broad range of operational concepts like e.g. search & rescue, demining, pollution cleaning, border patrolling, tracking.
Swarm Algorithms
In order to implement in an actual system some of the behaviors identified in the preceding section, one has to select
or develop the corresponding algorithm that will enable the
swarm (or sub-swarm) to answer to the human operator’s
commands. Figure 1 illustrates the architecture of the command channel (one-way).
Swarm
Algorithms
Swarm Behaviors
Topological
x actions
features
In order to fulfill these missions, we selected the combinations [1], [2], [3] and [4] as labelled in Table 1. The Watch
action was selected rather than the variations of Find ([5]
and [6]) because the objective of the Human operator is to
Figure 1: High-level view of the swarm system control architecture.
22
Table 1: Preliminary taxonomy of topology-based tasks for a generic (swarm-based) surveillance system. A brief description
of all combinations of actions and topological features is given when applicable. Some combinations sport a numeric label in
order to be referred to in the text.
Intercept
Follow
Find
Avoid
Watch
Mobile
Point
Open line
Bear regularly
Bear regularly one's
Bear regularly one's
one's sensors on a
sensors on every points of sensors on every points of
fixed point in
a given line.
a closed line.
space.
1
2
See Follow Mobile.
E.g. collision avoidance,
fleeing a threat, or not
disturbing another agent.
Often associated with a
minimum security
distance.
Do not cross a given line
in the environment. One
can circumvent the line.
See Closed line,
with line reduced
to a single point.
The position of the line
can be known in advance
or detected by the agents.
One has to have access to
the position of the mobile
via sensors or external
intelligence.
Find one or several
mobile targets in the
environment using one's
sensors.
Intel about the targets'
whereabouts or behavior
(e.g. direction of travel,
speed) can be used.
Closed line
Reduced case of
Mobile. Simpler
strategies can be
used e.g.
systematic search
of the
environment.
Can be associated with a
minimum security
distance.
Maneuver so as to cross a
mobile's trajectory in
order to carry an action
upon it.
Implies some knowledge
about the mobile's
whereabouts (sensors or
intelligence).
Intel about the geometry
of the line can be useful
(e.g. length, orientation).
The position of the line
can be known in advance
or detected by the agents.
Similar to Open line with
the additional dimension
of outside/inside.
Bear regularly
one's sensors
on every points
of an area.
3
See Closed
Line.
If the agent
happens to be
in the area, it
has to leave as
soon as
possible.
4
See Closed
line.
6
5
Maneuver so as to keep
one's sensors on a mobile
target.
Find at least one point of
a line known to exist in
the environment.
Do not cross a given
closed line (border) in the
environment. Equiv. to
not leaving/entering an
area.
Area
N/A
N/A
Maneuver so as to align
one's trajectory with a
line (oriented or not).
Position of the line is
known in advance. No
sensing activity.
Maneuver so as to meet
an open line as fast as
possible i.e. find closest
point.
23
See Open line.
Leads to repetitive
trajectories e.g. holding
pattern.
N/A
See Open line.
N/A
deploy its system on different parts of the environment in order to detect possible targets rather than to look for a particular target that is known to exist somewhere. The behaviors
that the swarm system is able to exhibit are the following:
D
• watch lines in the environment i.e. roads that may be used
by suspicious vehicle convoys [2];
C
• watch pre-defined zones or user-defined zones [3];
B
E
• watch user-designated points within these two kinds of
features i.e. the human operator needs a fly-by of a particular point of interest, for example to confirm a detection;
A
• avoid a user-positioned air corridor so as to let another
aircraft intervene safely [4].
Figure 3: Schematics of the main user interface components:
A is the loiter zone where the UAVs are kept in standby and
refuel; B is the air corridor that allows other aircrafts to intervene; C is a user-defined active patrol zone; D is a predefined patrol zone that is not yet activated; E is a linear
patrol zone.
Human Interactions
Our whole approach is dedicated to natural and intuitive use
of swarms for missions such as patrolling of tracking targets within an area. As explained here below, we propose
to found this intuitive operations on topological features and
actions that can influence the swarm algorithm and adapt
swarm behavior consequently. But these topological feature and objects have also to be determined, defined and
pointed at by the operator. Supporting this action by a natural gesture-based interaction seems therefore completely intuitive.
We developed a set of simple gestures allowing to draw areas, contours, lines directly on a geographical map displayed
on an interactive tabletop system (see Figure 3). While giving a minimal thickness to a line or a contour drawn through
a one-finger gesture (see Figure 2), we allow UAVs to consider these objects as special areas and we can manage them
with the same pheromone mechanisms.
The actual demonstrator was deployed an interactive
tabletop DT107 from CircleTwelve with an SXGA+ projector, illustrated on Figure 4.
Figure 4: Interactive tabletop setup.
Figure 2: Linear zone (bottom) created via a gesture with
one finger (top).
Algorithms
For points of interest, we propose to the operator to tap on
the desired location, the more tap the more UAVs to be sent
towards the point. In this case, we use the hybrid principle
again, so that the point becomes a way-point for the attached
UAVs until they reach it, then switching back to pheromonedriven patrol mode.
In addition, the user can use simple transfer gestures between zones in order to manage its UAVs assets according
to his or her mission-driven constraints and priorities.
The agents are driven by several prioritized behaviors:
1. avoid the safety corridor;
2. return to loiter zone if low on fuel;
3. visit interest point if tasked to do so by the system;
4. rejoin patrol zone if affected to one and is outside of it;
5. either:
24
(a) patrol the zone or;
(b) stay in the loiter zone.
Behavior 1 is achieved by calculating a simple escape
heading when the UAV is too close to the corridor. Behaviors 2, 3 and 4 are achieved by computing an appropriate
waypoint and assigning to the UAV. Behavior 5b is a custom
circling behavior developed for demonstration purposes.
Behavior 5a is achieved via a pheromone grid-based algorithm. Figure 5 illustrates how real-world coordinates are
mapped to the pheromone grid that supports the algorithm.
Given a cell-size, the smallest enclosing grid for the region
of interest is computed, as well as a mask to constrain the
behavior to the arbitrary shape input by the operator. Maximum levels of pheromone (255, white on the figure) are
deposited within the region to patrol.
The algorithm follows the two principles of consumption
and deposition:
• the agents consume (set to zero) pheromone along their
movement with respect to their perception radius;
• the agents locally determine their trajectory so as to consume the highest levels of pheromone;
• fine layers of pheromone (value increments) are regularly
deposited so that agents come back to places visited earlier as the pheromone that was consumed is replaced.
One can note a difference with other bio-inspired
pheromone algorithms that employ deposition, diffusion and
evaporation principles.
4
Discussion and Perspectives
We have only discussed in this paper the broad functionalities that allow the operator intuitively to issue commands to
the swarm. These commands include: designation, creation
and suppression of arbitrary-shaped patrol zones (linear and
polygonal), designation of interest point for the swarm, positioning of an exclusion corridor and transfer of resources
(UAVs) between patrol zones. We have developed a synthetic indicator that allows the system to express the performance status of patrol tasks. This indicator is not detailed in
this article, but can be seen on Figures 2 and 5, it consists
in a panel that indicates the optimal and current numbers
of UAVs for a zone, as well as a composite representation
of instantaneous patrol performance, optimality and upper
bound.
Future work on this topic include working with heterogeneous robots (different UAVs, or mixing ground and air
vehicles), developing new tasks that require more complex
interactions between the swarm and its tactical environment
(e.g. rendezvous) and moving beyond simulation by setting
up a live demonstrator.
Figure 5: An arbitrary-shaped zone (top) has been created
and two UAVs are patrolling. One can see a representation
of the rectangular pheromone grid that supports the corresponding swarm algorithm (bottom).
References
Coppin, G., and Legras, F. 2012. Autonomy spectrum and
performance perception issues in swarm supervisory control. Proceedings of the IEEE 100(3):590–603.
Johnson, C. 2003. Inverting the control ratio: Human control of large, autonomous teams. In Proceedings of AAMAS’03 Workshop on Humans and Multi-Agent Systems.
Kolling, A.; Nunnally, S.; and Lewis, M. 2012. Towards
human control of robot swarms. In Yanco, H. A.; Steinfeld,
A.; Evers, V.; and Jenkins, O. C., eds., HRI, 89–96. ACM.
Legras, F.; Glad, A.; Simonin, O.; and Charpillet, F. 2008.
Authority sharing in a swarm of UAVs: Simulation and experiments with operators. In Proceedings of the International Conference on Simulation, Modeling and Programming for Autonomous Robots (SIMPAR 2008).
Miner, D., and desJardins, M. 2009. Predicting and controlling system-level parameters of multi-agent systems. In
AAAI Fall Symposium on Complex Adaptive Systems and the
Threshold Effect.
Acknowledgements
SUSIE is an Exploratory Research and Innovation contract
(Recherche Exploratoire et Innovation: REI, 2009.34.0003)
of the French Defense Procurement Agency (Délégation
Générale pour l’Armement: DGA, Mission pour la
Recherche et l’Innovation Scientifique: MRIS) conducted by
Télécom Bretagne and INRIA Loria.
25