J Intell Robot Syst (2014) 74:221–232
DOI 10.1007/s10846-013-9970-z
Persistent UAV Service: An Improved Scheduling
Formulation and Prototypes of System Components
Byung Duk Song · Jonghoe Kim ·
Jeongwoon Kim · Hyorin Park ·
James R. Morrison · David Hyunchul Shim
Received: 11 September 2013 / Accepted: 17 September 2013 / Published online: 12 October 2013
© Springer Science+Business Media Dordrecht 2013
Abstract The flight duration of unmanned aerial
vehicles (UAVs) is limited by their battery or
fuel capacity. As a consequence, the duration of
missions that can be pursued by UAVs without
supporting logistics is restricted. However, a system of UAVs that is supported by automated
logistics structures, such as fuel service stations
and orchestration algorithms, may pursue missions of conceivably indefinite duration. This may
be accomplished by handing off the mission tasks
to fully fueled replacement UAVs when the current fleet grows weary. The drained UAVs then
seek replenishment from nearby logistics support
Parts of this paper are reprinted from the paper of the
same title in the 2013 Proceedings of the International
Conference on Unmanned Aircraft Systems
(ICUAS’13), May 2013.
The work reported here has been supported in part by
KAIST HRHRP grant N10120008 and by the National
Research Foundation of Korea (NRF) Basic Science
Research Program grant 2013012292.
B. D. Song · J. Kim · H. Park · J. R. Morrison (B) ·
Department of Industrial and Systems Engineering,
KAIST, 373-1 Guseong-dong, Yuseong-gu,
Daejeon 305-701, Korea
e-mail: james.morrison@kaist.edu
URL: http://xS3D.kaist.edu
J. Kim · D. H. Shim
Department of Aerospace Engineering, KAIST, 373-1
Guseong-dong, Yuseong-gu, Daejeon 305-701, Korea
facilities. To support the vision of a persistent
fleet of UAVs pursuing missions across a field of
operations, we develop an improved mixed integer linear programming (MILP) model that can
serve to support the system’s efforts to orchestrate
the operations of numerous UAVs, missions and
logistics facilities. Further, we look toward the
future implementation of such a persistent fleet
outdoors and develop prototype components required for such a system. In particular, we develop and demonstrate the concerted operation of
a scheduling model, UAV onboard vision-based
guidance system and replenishment stations.
Keywords Persistent UAV service · Scheduling
for persistence · Vision-based guidance systems ·
Replenishment stations
1 Introduction
Unmanned aerial vehicles (UAVs) can be employed as a perimeter patrol, security escort, communication relay, environmental monitor and fire
monitor. All of these application missions would
be significantly enhanced by UAVs that exhibit
persistence—the capability to be available for
long or indefinite durations of time. While highcost military UAVs such as the Predator can operate for as long as 24 hours, commercial and
practical UAVs are not as capable. A system
222
J Intell Robot Syst (2014) 74:221–232
of UAVs, a planning module, automated UAV
guidance systems and replenishment stations
can enable persistent operation. A vision-based
(possibly coupled with GPS) guidance system may
enable outdoor operation.
We envision a system that operates as follows;
refer to Fig. 1. Customers submit requests for
UAV service to the central planning module and
provide their intended space-time trajectory. We
assume that at least one UAV must be present
on the space-time trajectory at all times (thus, it
may be several UAVs that execute the mission,
each of which serves a portion of the path). During
service, a weary UAV may hand-off the mission
to another UAV and seek succor at a nearby
service station. A planning algorithm such as a
mixed integer linear program (MILP) determines
the UAV schedules and service station activities.
The automated service stations provide automatic
recharge or replacement service to enable a persistent UAV workforce. The guidance system
directs the UAVs on their paths based on the
directives received from the planning module. For
outdoor operation, vision-based UAV localization is used. Wireless communications are employed by the system to direct all activities.
Toward such a persistent service system, we:
•
Develop an improved MILP formulation of
the persistent UAV scheduling problem. It
Customer
information
UAV
schedules
Central
planning
by MILP
Web or
smart phones
Automatic
control
UAV
guidance
system
Feedback
Tracking
Customers
UAVs
Automatic
replenishment
Replenishment
stations
Fig. 1 UAV service system
•
•
•
requires less computation than that of [1] and
will serve as the planning algorithm.
Develop a prototype vision-based guidance
system that directs the UAVs to perform their
tasks and guide the UAVs to the service stations according to the schedule generated by
our MILP.
Develop automated service stations to
recharge a UAV upon landing.
Integrate these components into a functioning
system and develop a demonstration of the
system in operation inside a building on the
KAIST campus.
Note that as our ultimate goal is to deploy such
a system outdoors, the vision-based guidance is
an essential component (the typical IR camera
systems used for UAV localization methods in
lab environments are difficult to deploy outside).
While the systems we deploy are far from robust
and require development in several directions,
this represents the first effort to deploy a persistent system of UAVs using onboard vision-based
control and guidance supported by service stations
spread across a field of operations. It represents a
step toward the use of low cost UAVs to provide
persistent service outdoor.
The paper is organized as follows. In Section 2,
we review related UAV scheduling and implementation literature. Section 3 develops the
new MILP formulation. The prototype automated
UAV guidance system and service stations are
introduced in Section 4. Experiments to demonstrate the efficiency of the improved MILP are
given in Section 5. There, we also discuss the integration of the system components and demonstration of the concerted operation. Section 6 provides
concluding remarks.
2 Literature Review
Scheduling research on UAVs focuses on three
classes of models: uncapacitated, capacitated and
persistent. We discuss each of these next, followed
by work related to implementation (such as service stations and computer vision methods for
UAV and target localization).
J Intell Robot Syst (2014) 74:221–232
Scheduling research on uncapacitated UAVs
does not consider a maximum traveling time or
maximum traveling distance. Such models have
been used in the context of UAVs for the battlefield [2]. In [3], a nonlinear mathematical model
was suggested to assign multiple UAVs to simultaneously perform cooperative tasks. There,
a genetic algorithm was employed to obtain solutions. In [4], Alighanbari and How introduced
a decentralized task assignment concept for the
scheduling of UAVs. They argue that centralized
task assignment for a fleet of UAVs is often not
practical due to communication limits, robustness
issues and scalability.
Scheduling research on capacitated UAVs allows for limited flight duration or distance. In
this case, the UAVs should finish their tasks before their fuel source (e.g., a battery) is depleted.
Some of this research employs extended versions
of the vehicle routing problem. In [5], the UAV
scheduling problem was formulated as a vehicle
routing problem with time windows. There is a
single take-off and landing depot and the UAVs
may have different travel time capabilities. Tasks
are defined by their location, required service time
and time window. The objective was to minimize
the total time for all UAVs to cover all jobs.
Multiple home bases with different locations have
been studied in [6]. The authors developed a
MILP scheduling model for a system of capacitated UAVs. Each UAV is forced to return to
its home base. There are n tasks and m UAVs,
each with a capacity limit q. A MILP to minimize
the operation time and risk is developed. Alidaee
et al. [7] improved on the model in [6] to reduce
the number of variables and constraints and increase the efficiency of the MILP model.
The scheduling of UAVs for persistent operations can be accomplished with the support of
logistics service stations. In [8–10], the concept of
using a multiport replenishment station at a single
location was studied and mathematical formulations developed. Valenti et al. [11] extended the
Recent Horizon Task Assignment (RHTA) algorithm to reflect the fuel state of vehicles for persistent operation with a single multiport service
station. In addition, they demonstrated persistent
operation of coordinated multiple UAVs in their
lab. Recently, Kim et al. [1] considered schedul-
223
ing models for persistent operation of UAVs of
diverse capabilities with multiple and shareable
replenishment stations dispersed across the field
of operations. A MILP model was developed
to direct the UAVs to follow space-time trajectories using the concept of split job. To overcome the computational issues associated with
the MILP formulation, a genetic algorithm was
developed.
For persistent operation, an automated guidance system is essential. In [12], the authors
developed a real time monitoring and control system consisting of an embedded computer onboard
the UAVs and a ground station served by a laptop. The onboard computer performs numerous
tasks including data acquisition and measurement,
servo control, automatic flight control and communications. The ground station laptops provide
a flexible graphical interface monitoring the realtime status of the UAVs. In [13], an image-based
visual servo scheme is suggested for the visionbased guidance of a UAV. They developed a feedback control strategy for the UAV using images
from an airborne camera.
Automated service stations also play a key role
in achieving persistence. In [14, 15], the authors
suggested various design concepts of service stations and studied the coverage capabilities of a
persistent system of UAVs. In [16], a prototype
battery replacement system was introduced and
tested with UAVs (along with system control algorithms). In [17], Godzdanker et al. discussed
the development of an intelligent self-leveling and
nodal docking system (ISLANDS) for the mobile
recharge/refueling of UAVs. They provide the
design of the individual ISLANDS and study the
placement of them via an optimization problem.
In this paper, we develop a MILP formulation
for the persistent UAV scheduling problem that
improves upon the formulation proposed in Kim
et al. [1]. An automated vision-based UAV guidance system is introduced to direct the movement
of the UAVs according to the schedule generated by the MILP. A prototype battery recharge
service station is also developed. Though these
systems are not robust and of limited capability,
these efforts represent a step toward the possibility of outdoor persistent operation of visionbased UAVs across a field of operations. We
224
demonstrate the operation of the system in the
basement corridors of a facility at KAIST.
3 Mathematical Formulation
Kim et al. [1] discussed the idea that service platforms and a collection of UAVs can be employed
to follow given time-space trajectories for long
durations across a large field of operations. To
accomplish this, they divided the trajectories of
the moving target into segments called split jobs
that may each be served by a different UAV.
Each split job must be served by one UAV in
their formulation. Each split job i is defined by
a start location (xis ,yis ), start time Ei, finish location (xie ,yie ) and end time Li. (We assume the
speed during that split job is constant.) The travel
distance Dij from split job i’s finish point or station i to split job j’s start point or station j is
calculated using the Euclidean distance. Note that
Dij is not equal to D ji (the start and end points
change). They are not decision variables and must
be determined in advance. This approach serves to
discretize each mission. Using more split jobs may
enable an improved objective function value but
will increase computational complexity. Figure 2
depicts the split job concept.
Kim et al. [1] developed a MILP to schedule a fleet of UAVs and service platforms seeking to provide uninterrupted long term missions.
The missions and split jobs are described above.
Service stations located about the field provide
restorative services for the UAVs.
J Intell Robot Syst (2014) 74:221–232
redundant decision variables, Yikr . Here we eliminate them.
In [1], a dummy job was assigned to a UAV
that does not process any split jobs (a segment of
a customer’s space-time trajectory). This dummy
job concept causes an increase in the number of
decision variables and constraints. By allowing a
UAV to standby at a station (see Eqs. 3 and 4
below), we eliminate the need for the dummy job;
the number of decision variables and constraints
are reduced.
The decision variable Yikr in [1] is used to indicate that split job i is assigned to UAV k during
that UAV’s r th trip from a station. However, this
information is already embedded in their decision
variable Cikr . Cikr gives the start time of the work
if UAV k is assigned to split job i on its r th trip
from a service station. It is set to zero if there
is no assignment relationship between split job i
and UAV k on trip r. We can remove decision
variables Yikr and reduce the number of decision
variables and constraints. Our MILP formulation
follows. It is less computationally complex than
that of [1].
3.2 Notation
i, j
s
k
r
NJ
NU AV
N ST A
NR
3.1 A More Efficient Mathematical Formulation
The MILP formulation in [1] suffers from two
inefficiencies: 1) unnecessary dummy jobs and 2)
M
Dij
Ei
Li
Pi
qk
Sok
TSk
J
SS
Fig. 2 A time-space trajectory is divided into split jobs
Indices for jobs
Index for stations
Index for UAVs
Index of a UAV’s r th flight
Number of split jobs
Number of UAVs in the system
Number of recharge stations
Maximum number of flights per UAV
during the time horizon
Large positive number
Distance from the finish point of split job
i to the start point of split job j
Start time of split job i
End time of split job i
Processing time of split job i(Li – Ei) or
replenishment time in station i
Maximum traveling time of UAV k
Initial location (station) of UAV k
Maximum travel speed of UAV k
= {1, . . . , N J }, Set of split jobs
= {N J + 1, N J + 3, . . . , N J + 2 · N ST A −
1}, set of UAV flight start stations
J Intell Robot Syst (2014) 74:221–232
SE
225
= {N J + 2, N J + 4, . . . , N J + 2 · N ST A },
set of UAV flight end stations
= ( J U SS U SE ) = {1,. . . ,
N J +2·N ST A }, set of all jobs and recharge
stations
Binary decision variable, 1 if UAV k
processes split job j or recharges at station j after processing split job i or
recharging at station i during the r th
flight; 0, otherwise.
Binary decision variable, split job i’s start
time by UAV k during its r th flight or
UAV k’s recharge start time at station i;
otherwise its value is 0.
A
Xijkr
Cikr
Note that each station has two indices which
are used to distinguish when a UAV starts a trip
(flight) from that station or ends a trip (flight)
at that station. This is helpful when we allow a
UAV to remain at a station for some duration and
enables the removal of the dummy jobs used in [1].
Also note that the replenishment time at a service
station (Pi ) is constant. We are thus assuming that
the refresh process for a UAV is independent of
its fuel level when it arrives to the station.
3.3 Mixed integer linear program
Minimize
Dij Xijkr
(1)
i∈ A j∈ A k∈K r∈R
Subject to
X Sok , jk1 = 1 (k ∈ K)
(2)
i∈ J ∪SE
s∈SS i∈ J ∪SE
s∈SE i∈ J ∪SS
Xsjkr = 1 (k ∈ K, r ∈ R)
(3)
Xiskr = 1 (k ∈ K, r ∈ R)
(4)
Xiskr =
i∈ J ∪SS
Xs−1,ikr+1
i∈ J ∪SE
(k ∈ K, r = 1, . . . , N R − 1, s ∈ SE ) (5)
Cskr = Cs−1,kr+1
(k ∈ K, r = 1, . . . , N R − 1, s ∈ SE )
k∈K r∈R i∈ A
Xijkr = 1 ( j ∈ J )
(6)
(7)
Xijkr −
j∈ A
X jikr = 0
j∈ A
(i ∈ J , k ∈ K, r ∈ R)
(8)
Xiskr = 0 (k ∈ K, r ∈ R, s ∈ SS )
(9)
i∈ J ∪SS
Cikr + Pi + Dij T Sk − C jkr ≤ M 1 − Xijkr
(10)
(i ∈ J ∪ SS , j ∈ J ∪ SE , k ∈ K, r ∈ R)
M·
Xijkr ≥ Cikr
j∈ J ∪SE
(i ∈ J ∪ SS , k ∈ K, r ∈ R)
Cikr = Ei (i ∈ J )
(12)
k∈K r∈R
i∈ A j∈ A
Dij
T Sk
· Xijkr +
≤ qk
(11)
Pi · Xijkr
i∈ J j∈ A
(k ∈ K, r ∈ R)
(13)
Cikr ≥ 0(k ∈ K, r ∈ R, i ∈ A )
(14)
Xijkr ∈ {0, 1} (i ∈ A , j ∈ A , k ∈ K, r ∈ R)
(15)
The objective function (1) is to minimize the total travel distance of the UAVs. Constraint (2)
ensures that every UAV should start its first
flight from its initial service station. Constraint (3)
guarantees that UAV k starts its flight from the
starting service station and proceeds to a split job
or ending service station. Constraint (4) indicates
that UAV k must finish each flight at an ending
service station. This helps to remove the dummy
jobs of [1]. Constraint (5) requires that if UAV k
finishes its r th flight at service station s, then its
r+1th flight starts at service station s. Constraint
(6) states that the finish time of a UAV’s r th
flight is equal to the start time of UAV’s r+1th
flight. Split job assignment constraint (7) forces
every split job in J to be processed by a UAV.
Constraint (8) is a flow balance constraint that
ensures a UAV does not finish its flight at a job.
Constraint (9) ensures that a UAV cannot finish
its flight at the starting station which comes from
the notational issue related to each station having
two indices.
226
J Intell Robot Syst (2014) 74:221–232
Constraints (10) and (11) are used to link the
Cikr and Xijkr decision variables. The start time
constraint (10) gives the relationship between the
split job start time or service station visit start
time (which is the time that the fuel replacement
process begins) and that of its successor for the
same UAV during its r th flight. Constraint (11)
implies that the value of Cikr is set to zero if
split job i is not assigned to UAV k’s r th flight.
Constraint (12) ensures that every split job in J
starts its process at the pre-determined start time
of the split job. The fuel/battery constraint (13)
states that the total flight time, including travel
time between split jobs and process time for split
jobs, cannot exceed UAV k’s fuel/battery capacity. Constraints (14) and (15) define the ranges of
the decision variables.
The proposed mathematical formulation requires N R > 1 due to constraints (5) and (6). N R
is just the maximum number of flights allowed
for each UAV; this is not in any way a restriction on the modeling capability. Table 1 provides
the number of decision variables and constraints
of Kim et al. [1] and the proposed MILP; there
are fewer in the later. Also note that there are
trade-off relationships among the number of split
jobs, computational complexity and feasibility.
Too few split jobs can result in an infeasible problem due to the fuel limitation of the UAVs. On
the other hand, too many split jobs can increase
the computational complexity because of the increase in the number of decision variables and
constraints.
4 Implementation Environment
We next discuss the development of prototype
system components that we have integrated into
a functioning proof-of-concept system. The MILP
scheduling model is used to determine UAV actions in response to given customer space-time
trajectories. A UAV guidance system serves to
implement the schedule provided by the MILP
model, track the customer and land on the stations. Computer vision-based feedback control,
employing video from the UAV nose or belly
cameras, is used by the UAV to pursue the customer, arrive at the designated customer pick-up
locations and return to the service stations.
4.1 UAV Guidance System
Kim and Shim [18] proposed a vision-based target
tracking control system for a quadrotor via tablet
computer. We employ their system to control our
UAVs and track the customer. The UAV guidance system performs these tasks:
1. Receive and implement the schedule from the
MILP. This includes directing the UAVs to
depart from a service station to intercept the
customer (and prosecute a particular split job)
and directing the UAV to return to a service
station. It must ensure that a UAV is available
for the customer at the start of a split job.
2. Convert video from the UAV cameras into
usable information for directing UAV motion.
Table 1 Number of variables and constraints in the mathematical formulations
Kim et al. [1]
Total # of binary
decision variable
Total # of continuous
decision variable
Total # of decision
variable
Total # of
constraints
NU AV · N R · {(N J + 2 · N ST A
Proposed MILP
+ 1)2
+ (N J + 2 · N ST A + 1)}
NU AV · N R · (N J + 2 · N ST A )2
NU AV · N R · (N J + 2 · N ST A + 1)
NU AV · N R · (N J + 2 · N ST A )
NU AV · N R · {(N J + 2 · N ST A + 1)2 + 2(N J + 2 · N ST A + 1)}
NU AV · N R · {(N j + 2 · N ST A )2
+N J + 2 · N ST A }
NU AV {2(Nr − 1) · N ST A
+N R · N ST A + 1}
+N J (NU AV · N R + 2)
+NU AV · N R {(N J + N ST A )2
+2N J + 3N ST A + 3}
NU AV {2(N R − 1) · N ST A + 1} + N J (3NU AV · N R + 2)
+NU AV · N R {(N J + N ST A + 1)2 + 2N J + 4N ST A + 5}
J Intell Robot Syst (2014) 74:221–232
This information is used to track customers,
locate a hand-off or land on a station.
3. Enable a human overseer to monitor the
UAV progress via video and adjust feedback
control gain values for various situations. It
also allows for a human overseer to initiate
emergency actions such as immediate landing.
These functions are used in the testing and
setup phases.
The UAV guidance system is responsible for directing the UAV to find the customer at the location and time of the split job. This is done with
computer vision using the UAV’s onboard cameras. The system must also ensure that the UAV
follows the customer during its time of service.
Feedback control based on vision processing of
the UAV’s cameras is employed to accomplish
these tasks. These same methods are employed to
ensure that the UAV can locate and land on the
service stations.
4.1.1 Target Specif ication
The guidance algorithm uses a color based tracking strategy. The customer wears a large red circle
on his jacket. The station and handoff locations
are also flagged with large colored polygons. To
identify our customer, we identified the largest
polygon in the UAV field of vision (nose camera)
filled with the color red. Algorithm 1 shows the
pseudo algorithm of target specification logic.
227
Algorithm 1 Pseudo-algorithm of extracting binary decision matrix whether each image pixel is
seen as red or not
Input:, (I R , I G , I B ) where each I R , I G and I B are
8-bit image matrices or red, blue and green
channels, respectively.
Output: A filtered binary image frame matrix, I Rb
Get RedBinary Image (I R , I G , I B )
For every pixel value I R,if of i-th row and j-th
column in the matrix I R
If (I R,if ≥ K R (I G,if + I B,i f ) + C R then
I Rb ,if ← 1
Else
I Rb ,if ← 0
endIf
endFor
return I Rb
our target to be the largest red polygon in the
frame. This is done by extracting contours and
comparing their areas with each other in the binary image. The center of the largest contour is
regarded as the target position. Figure 3 shows
4.1.2 Color Filter
Within the target identification algorithm, one
must detect image pixels with significant red color.
Color filtering to detect red pixels compares the
intensity of the red channel I R to a linear combination of the other channels’ intensity I G +I B .
The detailed algorithm is described in Algorithm
1. The gain K R and constant C R are set and tuned
manually considering camera properties and light
conditions. Typical values used were K R = 0.5
and C R = 40.
After filtering out other colors from the image
frame, the red pixels are the only remaining ones
in the binary image I Rb . This frame includes not
only our target, but other red objects. We declared
Fig. 3 The original image (top) and the filtered image
(bottom)
228
Fig. 4 Target in the image frame. Control commands are
indicated
the result of the color filter algorithm. The red
contour outline depicts the boundary of the red
polygon. The target and its center is visualized as
a green square and a yellow circle, respectively.
J Intell Robot Syst (2014) 74:221–232
a given distance from the target. Figure 4 shows
the commands issues to the UAV based on the
distance from the target and orientation of the
target to the center of the frame.
First, we calculate the difference between the
target position and the image center in the image coordinates. The differences in horizontal and
vertical direction are the controller inputs that
generate control commands to the quadrotor. The
horizontal error generates the yaw rate of our
UAV, to rotate its heading toward the target. The
vertical difference is used to calculate the vertical
velocity of the quadrotor to maintain the same
altitude as the target.
Second, the radius of the target in the image
frame is computed via Eq. 16:
r=
S
π
(16)
4.1.3 Visual Servoing
We employ a visual servoing control scheme using
the target point in the image frame. The quadrotor
UAV platform used in our efforts has a camera
facing forward and the control algorithm is established based on this setting. The UAV has a
responsibility to pursue the customer and observe
the surroundings. Thus the main objective of the
visual servoing is to maintain the detected target
at the center of the image frame while maintaining
Fig. 5 Controller block diagram
where S indicates the area of the target in square
pixels and r is the radius of the target. We set
a desired reference value of r and calculate the
difference between the observed radius and the
reference value. This difference is directly is used
as the error signal for our controller so that the
UAV maintains the distance from the customer.
We use a PD (Proportional—Differential) controller for the visual servoing; the controller block
diagram is provided in Fig. 5.
J Intell Robot Syst (2014) 74:221–232
229
4.2 Replenishment Station
Descriptions
: Front (Bottom) Camera Video
: Start Procedure Interface
: Color Filtered Video
: Control Gain Adjustment Sliders
: Emergency Landing Button
Fig. 6 Tablet PC application layout with descriptions
4.1.4 Hardware and Software System Description
Our UAV platform is the AR.Drone 2.0 of Parrot
Inc. The AR.Drone can communicate with a PC
or other devices via Wi-Fi. It simultaneously sends
its video and receives control commands via this
WiFi. We developed the control application to run
on our tablet PC (iPad 3 of Apple Inc.).
The application window provided to the user
on the iPad is depicted in Fig. 6. The application uses multiple threads that receive the video
data from the drone, process each image frame to
generate control inputs and sends the commands
to the drone. Visualization of the original video
feed from the drone and the processed video are
provided. There are several buttons enabling the
user to manually control the UAV in the event of
an emergency situation.
To support such a persistent system, service
stations must be incorporated. Various battery
recharge and battery swapping service stations
have been developed and studied in the literature,
c.f., [7, 8, 12–14]. To serve our system needs we
also developed a service station; it is a battery
recharging station (not a battery replacement station). Refer to Fig. 7.
Each AR Drone 2.0 uses a three cell lithium
polymer battery. Four copper leads (three for the
terminals and one for ground) were threaded from
the battery inside the UAV to the four feet of the
drone. The service station consists of four pads;
one for each foot of the drone. Each such pad
connects to the UAV battery via the leads on the
drone feet and will provide a different voltage
relative to the ground pad.
The AR Drone 2.0 should not be recharged
while it is on. Thus, a large circular button switch
is located in the center of the station. When the
UAV touches down, its weight activates the switch
and a circuit controlling the drone on/off is activated. After a 5 s delay, the drone power is turned
off and recharging is initiated. Once the battery
is fully charged by the on-station OEM charger,
an LED signal is sent by the charger. The station
detects this condition, disconnects the charger and
activates the drone. It is ready to return to service.
5 Demonstration
We conduct numerical experiments to assess the
performance of the MILP. We demonstrate the
operation of the integrated system for a simple
case in the basement of a facility at KAIST.
5.1 Comparison of MILP Formulations
Fig. 7 Recharge of UAV on the service station
We test our MILP formulation against that of
[1] using CPLEX 12.4 on an Intel(R) Core(TM)2
Quad CPU Q8400, 2.66 GHz and 4.00 GB RAM
personal computer. We compare computation
time, number of decision variables and number
of constraints. Table 2 provides the results. While
both formulations give the same optimal value,
there are significant improvements associated
230
J Intell Robot Syst (2014) 74:221–232
Table 2 Comparison of the efficiency of the mathematical formulations
NJ
N ST A
8
14
15
20
2
3
3
4
NU AV
2
6
6
8
Kim et al. [1]
Proposed MILP
CPU time
# of dec.
vars.
# of
constraints
CPU
time
Obj.
value
# of dec.
vars.
# of
constraints
CPU
time
Obj.
value
change (%)
780
5796
6336
14384
722
5002
5508
12048
3.00
15.84
220.3
N/A
2048
1846
724
N/A
624
5040
5544
12992
566
4222
4680
10592
1.84
4.36
35.78
348.97
2048
1846
724
2894
38.67
72.47
83.75
–
with the new MILP. In particular, for (N J , N ST A ,
NU AV ) = (20, 4, 8), the MILP of [1] could not
be solved by CPLEX. The new formulation can
solve this case. By reducing the complexity, the
new formulation increases the size of problems
that can be addressed.
5.2 System Demonstration
We integrated these components to provide a
demonstration UAV security escort service. The
service was provided in the basement of a facility
at KAIST, but we use vision-based tracking and
control of the UAVs so that the system may be
extendable to the outdoors. The objective is to
follow a customer without interruption of service.
The situation follows. A single customer requests the UAV security escort service. Two
UAVs and 3 replenishment stations serve the
customer. The mission is divided into eight split
jobs. Due to our vision-based tracking algorithm,
the customer attaches a red marker on his back.
Station 3
UAV 2
Neither GPS nor fixed IR cameras are used. Color
markers are used to identify the stations, customer
pick up point and UAV handoff sites. Refer to
Fig. 8.
The relevant locations are detailed next. The
three battery recharge service stations 1, 2 and
3 are located at positions (4, −1), (0, −1) and
(1, 4), respectively. Each unit on the grid in Fig. 8
represents 5 m.
UAVs 1 and 2 initially reside at stations 1 and 2,
respectively. The customer path follows the x-axis
from (4, 0) to (0, 0), then travels along the y-axis
from (0, 0) to (0, 4). The travel speed is constant.
The path is split into eight split jobs (of one unit
of distance each) for the MILP.
UAV 1 hands off the customer to UAV 2 at
the point (0, 0). Table 3 provides the assignment
provided by the MILP.
The detailed action proceeded as follows.
1. UAV 1 takes off from station 1 in advance to
meet the customer at the planned arrival time.
It meets the customer at the start point of
split job 1 and searches for the marker on the
customer’s back. Once identified, the UAV
follows the customer as he proceeds along his
path.
2. For uninterrupted service, UAV 2 moves to
the start point of split job 4 in advance of the
customer’s arrival.
3. UAV 1 disengages from the customer at the
end of split job 4, searches for service station
Hand-off
Table 3 UAV assignment schedule
UAV
Start
Assigned
station job
End
Service
station start time
Service
end time
1
2
1
2
2
3
10
18
UAV 1
Station 2
Station 1
Fig. 8 Top down layout of the security escort demonstration
1, 2, 3, 4
5, 6, 7, 8
2
10
J Intell Robot Syst (2014) 74:221–232
231
6 Concluding Remarks
vice stations and system orchestration algorithms,
a fleet of UAVs and service stations may be
able to provide persistent service across a field
of operations. Looking toward an outdoor persistent UAV service system, we developed several components: an improved MILP scheduling
formulation, a UAV guidance system and battery recharge service stations. The MILP model is
used to develop a schedule that the system should
follow to meet customer space-time trajectory
demands. It significantly improves upon the computational complexity available in the literature
for such a problem [1]. The UAV guidance
system—based on the iOS—uses a given schedule
(e.g., as provided by the MILP) and directs the
UAVs about their tasks.
Onboard cameras are used for vision-based
feedback control of the UAV actions. The system
of multiple UAVs and service stations distributed
across the field is controlled via wireless communication from iPads. The battery recharge service
stations are used to provide the logistics support
necessary for persistence.
We demonstrated the integration of these components. UAVs provided a security escort to a
moving customer without interruption via the
wireless automated guidance system. While the
system deployed is not particularly robust, it
demonstrates that outdoor operation may be possible for a persistent system of low cost UAVs.
There are many directions for future work.
Despite the MILP’s improved computational performance, there is a need for real-time capable
models. The guidance system should be fully integrated with the higher level customer order and
planning layer. A key challenge for deployment of
such a system outdoors is the creation of reliable
tracking algorithms and UAV control. Indoors,
there is no wind and the environment is well
controlled. Outdoors, UAV and customer localization with precision in the centimeter range (as
is required for landing and handoffs) will be a
challenge.
The flight duration of commercially available
UAV’s is significantly constrained by their reliance on a battery or fuel supply. As such, the
duration of missions that can be prosecuted without support is limited. However, by including ser-
Acknowledgements The authors wish to acknowledge
the technical support of Jungho Park, Kyukwang Kim
and Khurshidjon Khudoyberdiyev. We also thank the reviewers for their thoughtful comments. The paper has
significantly improved based on their suggestions.
(a)
(b)
Fig. 9 UAV tracking service a and handoff operation b
2, proceeds to service station 2 and lands to
receive replenishment.
4. UAV 2 searches for the customer at the start
of split job 5 and follows the customer to the
end point of split job 8.
5. Once reaching the location of split job 8,
UAV2 searches for and then proceeds to service station 3. UAV 2 lands on the replenishment station and receives replenishment.
Figure 9a and b depict UAV 1 tracking the customer and the UAV handoff at the end of split job
4, respectively. Throughout, based on the schedule dictated by the MILP, the UAVs were controlled by the automated guidance system using
wireless communication.
232
J Intell Robot Syst (2014) 74:221–232
References
1. Kim, J., Song, B.D., Morrison, J.R.: On the scheduling
of systems of UAVs and fuel service stations for longterm mission fulfillment. J. Intell. Robot. Syst. 70(1),
347–359 (2013)
2. Zeng, J., Yang, X., Yang, L., Shen, G.: Modeling for
UAV resource scheduling under mission synchronization. J. Syst. Eng. Electron. 21(5), 821–826 (2010)
3. Shima, T., Schumacher, C.: Assignment of cooperating
UAVs to simultaneous tasks using genetic algorithm.
In: Proc. AIAA Guidance, Navigation, and Control
Conference and Exhibit. San Francisco (2005)
4. Alighanbari, M., How, J.P.: Decentralized task assignment for unmanned aerial vehicles. In: Proceedings
of the 44th IEEE Conference on Decision and Control, and the European Control Conference, pp. 12–15.
Seville, Spain (2005)
5. Weinstein, A.L., Schumacher, C.: UAV scheduling
via the vehicle routing problem with time windows.
In: Proc. AIAA Infotech@Aerospace 2007 Conference
and Exhibit. Rohnert Park, California (2007)
6. Kim, Y.S., Gu, D.W., Postlethwaite, I.: Real-time
optimal mission scheduling and flight path selection. IEEE Trans. Autom. Control. 52(6), 1119–1123
(2007)
7. Alidaee, B., Wang, H., Landram, F.: A note on integer programming formulations of the real-time optimal
scheduling and flight selection of UAVS. IEEE Trans.
Control. Syst. Technol. 17(4), 839–843 (2009)
8. Bethke, B.: Persistent vision-based search and track
using multiple UAVs. Master Thesis, Massachusetts
Institute of Technology (2005)
9. Valenti, M.: Approximate dynamic programming with
applications in multi-agent systems. Ph.D. thesis,
Massachusetts Institute of Technology (2007)
10. Nigam, N., Bieniawski, S., Kroo, I., Vian, J.: Control
of multiple UAVs for persistent surveillance: algorithm
11.
12.
13.
14.
15.
16.
17.
18.
and flight test results. IEEE Trans. Control Syst. Technol. 20(5), 1236–1251 (2012)
Valenti, M., Bethke, B., How, J.P., de Farias, D.P.,
Vian, J.: Embedding health management into mission
tasking for UAV teams. In: American Control Conference (ACC), pp. 5777–5783. New York City, NY (2007)
Dong, M., Chen, B.M., Cai, G., Peng, K.: Development
of a real-time onboard and ground station software
system for a UAV helicopter. J. Aerosp. Comput. Inf.
Commun. 4, 933–955 (2007)
Azinheira, J.R., Rives, P.: Image-based visual servoing
for vanishing features and ground lines tracking: application to a UAV automatic landing. Int. J. Optomech.
2(3), 275–295 (2008)
Kemper Filho, P., Suzuki, K.A.O., Morrison, J.R.:
UAV consumable replenishment: design concepts for
automated service stations. J. Intell. Robot. Syst. 61(1),
369–397 (2011)
Suzuki, K.A.O., Kemper Filho, P., Morrison, J.R.: Automatic battery replacement system for UAVs: analysis
and design. J. Intell. Robot. Syst. 65(1), 563–586 (2012)
Swieringa, K.A., Hanson, C.B., Richardson, J.R.,
White, J.D., Hasan, Z., Qian, E., Girard, A.: Autonomous battery swapping systems for small-scale
helicopters. In: Proceedings of the 2010 IEEE International Conference on Robotics and Automation
(ICRA), pp. 3335–3340 (2010)
Godzdanker, R., Rutherford, M.J., Valavanis, K.P.:
Islands a self-leeling landing platform for autonomous
miniature UAVs. In: IEEE/ASME International
Conference on Advanced Intelligent Mechatronics.
Budapest, Hungary, 3–7 July 2011
Kim, J., Shim, D.H.: A vision-based target tracking control system of a quadrotor by using a tablet
computer. In: 2013 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 1165–1172.
Atlanta, GA, 28–31 May 2013. doi:10.1109/ICUAS.
2013.6564808