Proceedings of ICSC’15:
The Canadian Society for Civil Engineering
5th International/11th Construction Specialty Conference
The University of British Columbia,
Vancouver, Canada
June 7-10, 2015
Editors:
Thomas M. Froese
Linda Newton
Farnaz Sadeghpour
Dana J. Vanier
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
425
MAINTAINING VERTICAL GARDENS USING QUADROTOR AERIAL
INSPECTION
Alexey Bulgakov1,5, Sergey Emelianov1, Rainer Schach2, Daher Sayfeddine3, Vladimir
Erofeev4
1
South West State University, Russian Federation
Technical University Dresden, Germany
3
South Russian State Polytechnic University, Russian Federation
4
State University Mordovia, Russian Federation
5
a.bulgakow@gmx.de
2
Abstract: The upkeep of the building requires continuous monitoring to maintain the safety aspects as
well as aesthetic elements of the building including facades, marble constructions and gardens. With the
decrease of the amount of green areas, human race will face climatic catastrophes, which will affect the
quality, type and life longitude of the Earth habitants. To tackle this issue, vertical gardens were designed
to fit in the overwhelming world of concrete and to adapt in order to do photosynthesis. The growth of the
plants is supported by spraying insecticide periodically. In this paper, we offer to use an unmanned aerial
vehicle, to recognize the targeted green areas using HSV algorithm, track the generated trajectory and
perform the spraying process all along the recognized area. We believe this will automate the
maintenance of the gardens, improve the effectiveness of the treatment and assist to the ecologic factor,
by applying the necessary amount of chemicals on the targeted place only.
1
INRODUCTION
Recently unmanned aerial vehicles are used to perform several civil tasks. Particularly, in construction
fields, the idea of aerial robot is directly related to photogrammetry, site scanning and buildings status. In
this regard, many literatures supported the cost efficiency, safety and quality control for such
technological implementation (Bulgakov A. et al. 2014; Emelianov S. et al. 2014; Rango, et al. 2009.
Chao, H et al. Shang et al. 2010)
Similarly, aerial robots can be used to scan vertical gardens and improve the irrigation system by
controlling the quantity of water supplied to a certain site. This can be of great importance when talking
about large agricultural areas, mega-polis with limited greenery and quantifying forest fire impact.
The technology being used to sense is based on satellite inspection. The limitation of such technology
consists of the low resolution and the update ratio of the images provided. On the other hand, UAVs are
more cost effective, can reach difficult scanning corners and provide more accurate visual information, as
they can be equipped with high definition cameras. In this paper, we offer to use a vertical take off and
landing aerial robot to perform visual inspection for vertical gardens.
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
2
426
RELATED WORKS
UAV’s successful results in improving military tasks have pushed the developers of such technology to try
it in the civil market. The verdict: we got a miniature class of rotorcraft assisting in fire fighting,
cinematography, television broadcasting and so on. The technology is being used to improve irrigation
systems (Chao H, et al.), to address the climatic changes in the north and south poles (Lucieer A. et al.
2012), inspection of bridges and facades (Bulgakov A. et al. 2014; Emelianov S. et al. 2014).
As a sensing technology, band -reconfigurable fixed wing UAV were used more frequently. This is due to
the optimized flight range, lifting capability and improvement in Micro (MEMS) and Nano
electromechanical systems (NEMS). The sensors used are based on thermal infrared and machine
vision. But these sensing technologies have limitations, especially the thermal infrared: the cost involved
and the reliability of the acquired data especially in a humid atmosphere (Lucieer A. et al. 2012). The
other limitation is the UAV itself: stability problem leads to false reading.
Having analyzed the pros and cons of aerial inspection, the optimization problem can be divided into two
subtasks:
1. Coverage control problem consisting of path planning and control of the UAV;
2. Geo-reference problem consists of registration of each acquired pixel from the aerial images including
temporal and spatial information (time and location).
Both of the subtasks can be addressed in either of two ways: global path planning consisting of trajectory
planning before the flight and local planning that generates the trajectory during flight.
3
AIM OF THE PAPER
The aim of the paper is to address the aforementioned subtasks in order to inspect vertical gardens in the
cities using quadrotors. As it is noticeable in most of the mega polis in the world, where green areas are
lacking. The uncontrollable population growth is destroying the remaining of “mother nature”. Engineers
and social workers have invented the concept of green wall or vertical gardens. The idea, as shown is
figure 1, consists of planting wall climbing trees and plants on vertical walls. Due to the ergonomics
involved the idea is widely spread.
Figure 1: Vertical garden concept
The role of the quadrotor is to acquire visual information such as images and pictures and spray the
insecticide on the target area.
3.1
Coverage Control Problem
This subtask is divided into two points: path planning and control. Since, for the geo-reference problem,
we will be using GPS data to acquire the temporal and spatial information, we will limit ourselves with the
control part of it. As it is known, a quadrotor is a take off and landing rotorcraft, having six degrees of
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
427
freedom. But this miniature aircraft can perform only four flight regimes: roll, pitch, yaw and hover. This
mechanical limitation makes the quadrotor nonlinear system. The dynamics of the quadrotor can be
described as follows:
;
[1]
[2]
;
[3]
[4]
[5]
[6]
.
Where, , and are the projection of the linear acceleration of the quadrotor in the Earth fixed axis, ,
and are projection rotational acceleration of the quadrotor in the body fixed axis,
the gravitational
the torque generated from the rotors,
the mass of the quadrotor,
,
and
acceleration,
is the rotational speed of the propellers, ,
and
are the projection of the Inertia of the quadrotor,
and
are the aforementioned flight regimes.
are the roll, pitch and yaw angle consequently, , ,
3.2
Fuzzy Logic Position Controllers
As it is important to know the value of the error, it is similarly critical to understand how it is changing over
time. The error and its derivative in time are one of the possibilities to configure a Fuzzy logic controller. It
is an artificial intelligence approach that computes mathematical operations based on degree of truth
rather than the conventional True-False Boolean logic. The Fuzzy logic allows having more adaptable
controller specially when dealing with nonlinearities (i.e. nonlinear aerodynamic model of the quadrotor as
described in equations 1-6) and uncertainties (i.e. permanent changes in the flight circumstances, wind,
temperature, obstacles positions etc.).
Although in many cases, heuristic algorithms were ruled out, due to the time consumption, we saw a need
to include this algorithm in our survey to cover most of the techniques used. We would like to clarify that
the term heuristic is used to describe the estimation ability in artificial intelligence.
Moreover, the
linguistic power of the fuzzy logic may be extremely useful while navigating in totally unknown areas. The
trick in such controller is how to set up the linguistic rules for input and output.
We fine tuned the rules by using the resulting graph (figure 2) for the function de(t)=f(e(t)), where e(t) is
the deviation in position of the quadrotor with reference to the desired trajectory and de(t) is the variation
of the deviation.
Figure 2: Resulting Graph de(t)= f(e(t))
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
428
Using the graph in figure 2, we assign the following labels for the numeric values: Negative big (NB) =-1,
Negative Small (NS) = -0.5, Zero (Z)=0, Positive Small (PS)= 0.5 and Positive Big (PB) =1. As a result we
obtain the linguistic rules listed in table (1). A linguistic rule defines the output of the fuzzy controller
based on discrete logic (i.e. if e (t)= NB and d(e(t)= PB, then the output is Z). the graphical representation
of the linguistic rules is illustrated in figure 3.
Table 1: Fuzzy rules
NB
NS
Z
PS
PB
NB
NB
NS
NB
NB
Z
NS
NS
NS
NS
Z
NS
Z
NB
NS
Z
PS
PB
PS
NS
Z
PS
PS
PS
PB
Z
PB
PB
PB
PB
de
e
Figure 3: Graphical Representation of the linguistic rules
The inputs of the fuzzy controller de(t) and e(t) are represented using triangular membership functions as
shown in figure 4 a and 4b. The output of the fuzzy control is also 5 triangular membership functions
(figure 4c).
Figure 4 a: membership function for input “e(t)”
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
429
Figure 4 b: membership function for input “de(t)”
Figure 4 c : membership function for fuzzy regulator output
The results of controlling the quadrotor using fuzzy logic regulators are shown in figure 5a and b for
position and rotation control consequently.
Figure 5a: Quadrotor position control with reference to GPS waypoints. Horizontal axis – time [s], vertical
axis- position [m]
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
430
Figure 5b: Quadrotor rotation control with reference to obtained position coordinates in 3a. Horizontal axis
– time [s], vertical axis- orientation [degree/s]
The results shown in figure 5a and 5b give a fair idea about the reliability of the fuzzy controller. For
instance, figure 5a illustrates the fact of stabilizing the linear movement of the quadrotor (described in
equations 1-3, X (red curve), Y (blue curve), Z (green curve)) without overshooting. This allows the
quadrotor to move along Earth-axis OXYZ smoothly. The other advantage of avoiding overshooting is
contributing to better energy-efficient control systems (i.e. overshooting in altitude control means flying
higher and consuming more power).
Taking into consideration the nonlinearity of the aerodynamic model of the quadrotor, linear movement
cannot be achieved unless stabilizing Euler angles (roll (red curve), pitch (blue curve) and yaw angles
(green curve)) as shown is figure 5b. Control signals are generated in order to increase or decrease Euler
angles (equation 4-6) with reference to linear position control task (equations 1-3).
3.3
Geo-reference Identification Problem
Identification of the vertical garden can be done using the green pattern. Many machine vision algorithms
are dedicated to recognize colors. We will be using the HSV algorithm due to its mathematical model,
which is more reliable than the other methods. The HSV consists of giving a certain color 3 coordinates:
Hue, saturation and value as shown in picture 4. HSV is used is many literatures in order to detect static
and dynamic agents/objects based on histogram criteria, such as recognition of traffic lights (Yixin Chen
et Al. 2013), gesture identification (Mokhtar M. Mohsen et al. 2010), image and structure retrieval
(Durgesh N. et al. 2013) and face detection (Naresh Kumar R. et al. 2014,Sayfeddine D. 2013).
Figure 6: HSV shade representation
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
431
From figure 6, we can see clearly that the green color is varying between 60 and 180 degrees. Hence, the
role of the algorithm is to identify the green patches on the wall and apply the insecticide on the
recognized area. This can be done by dividing the frame of the camera mounted on the quadrotor into 9
identical cells. Hereafter the registered pixel coordinates will be modified to metric using the following
equations:
[7]
[8]
[9]
[10]
[11]
[12]
and
the projection of the altitude on the pixel coordinates,
is the flying altitude,
Where,
and
are the horizontal and vertical pixel coordinates of the recognized area,
and
are the
and
are the rotation of the quadrotor in the pixel axis, and are the rotation of the
scale factors,
diaphragm of the camera from the body axis.
After identification of the desired pixels based on color criteria, Cartesian subtraction is computed to
identify the distance between the center of the fixed camera and the centroid of the detected colored
surface. The subtraction result represents the distance that the quadrotor has to fly in pixels. The
following approach is known as visual odometry. Depending on the position of the centroid, metric control
task is generated for flight altitude stabilization (equation 7 and 8), linear movement (equations 9 and 10)
and rotational movement (equation 11 and 12).
Using the concept of reactive control and based on the results obtained in equations (8,9 and 11,12),
control signals are generated to initialize one of the four available flight regimes described in equations
(3-6).
4
REAL-TIME RESULTS
HSV and visual odometry algorithms were tested in real-time in order to trace the possibility of their
implementation on high-rise structures covered with vertical gardens. As figure 7 shows, the identification
process was successful and the generation of the pixel coordinates was obtained.
Figure 7 can be interpreted as follows; the green surface is detected by algorithm HSV (i.e. position
= 69,
). The Cartesian
Middle-Middle in the 9-cells frame). The green pixels are generated (i.e.
subtraction is obtained and metric control task is formed. The graph is figure 7 shows how the quadrotor
(blue
changes its position (X (green curve), Y (purple curve)) with the changes of the pixel coordinates
curve) and (red curve).
A comparison between real time two dimensional flight results (figure 7) and simulation results (figures
5a/5b) shows that the deviation in sensors readings is as follows: roll reading (0.045°/s); yaw reading
(0.125°/s); pitch reading (0.15°/s). The reasons of such deviations can be caused by the Euler angles
MEMS gyroscopes, asymmetric structure of the quadrotor, influence of shifted center of gravity on the
control systems, and propellers non-uniformity. The obtained deviation values lay in the tolerance interval
and allowed Metrologic norms (deviation of real-time results ±5% from the simulation results).
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
432
Figure 7: Trajectory tracking
5
CONCLUSIONS AND FUTURE WORK
The paper focused on how to use the quadrotor to maintain vertical gardens on high-rise structures. The
adopted concept was to divide the autonomous flight problem into control of the quadrotor over an
identified area using HSV algorithm. The control was performed using fuzzy logic position regulators
based on reactive control concept interconnecting the rotation speed of the propellers with the positioning
of the quadrotor. No overshooting was registered in the simulation results. Regarding the identification of
the green area, HSV algorithm was used. The pixel coordinates were modified to metric in order to create
flight mission. The identification and control algorithms were tested in real time using a commercial
quadrotor (AR DRONE) linked to a stationary computer using datagram protocol UDP through a Wi-Fi
connection.
Although the results were outstanding and the computational power was enough to drive the quadrotor to
the end of the experience, we have noticed the influence of the light nature and temperature on
identifying the green color. Having just the color as tracking criteria, the stationary computer could run into
local minimum problem (between green and blue). This issue was addressed earlier as extrapolation of
motion function. A solution was suggested based on nonlinear neural network model with exogenous
input (NARX). The results can be accessed in the following literatures (Sayfeddine D. 2014, Sayfeddine
D. et al. 2014).
References
Bulgakov, A., Emelianov, S., Bock, T. and Sayfeddine, D. 2014. Control of hovering altitude of a
quadrotor with shifted center of gravity for inspection of high-rise structures. ISARC, Sydney, Australia:
762-768.
Chao, H. and al. Band reconfigurable multi- UAV based cooperative remote sensing for real time water
management and disturbed irrigation control. 9 pages.
Durgesh Nirapure, Udaypal Reddy. 2013. Fast retrieval of images using filtered HSV color level detection.
International journal of Emerging Technologies and Advanced Engineering. Volume 3, issue 8: 414419.
Emelianov, S., Bulgakov A. and Sayfeddine, D. 2014. Aerial laser inspection of building facades using
quadrotor. Proceedia of Creative construction. CC2014. Prague, Czech Republic: 24-30.
Lucieer, A. and al. 2012. Using a micro UAV for ultra high resolution multi sensor observations of
Antarctic moss beds. International Archives of the photogrammetry, remote sensing and spatial
information science. Volume XXXIX-B1, 2012: 429-433.
Mokhtar M. Mohsen, Pramod K. Misra. 2011. HSV brightness factor matching for gesture recognition
system. International Journal of Image Processing (IJIP), Volume (4): issue 5: 456-467.
Naresh Kumar R., P. Varum Varma, P. Surya Abhilash. 2014. Face detection using HSV color space and
minimal distance maximum variation approach. International Journal of Electrical, Electronics and Data
Communication, Volume 2, issue 4: 8-12.
ICSC’15 - The CSCE International Construction Specialty Conference, Vancouver, June 7 - 10, 2015
433
Rango, A. and al. 2009. Unmanned aerial vehicle based remote sensing for rangeland assessment,
monitoring and management. Journal of Applied Remote sensing, 4: 033452.
Sayfeddine D. 2013. Face identification and tracking using quadrotor. Translated from Russian: Control
problems, processing and data transmission (ATM 2013) : 178-180.
Sayfeddine, D. 2014. Nonlinear autoregressive neural network with exogenous inputs based solution for
local minimum problem of agent tracking using quadrotor. Electronic scientific journal. Engineering
Journal of Don. 2: http://www.ivdon.ru/en/magazine/archive/n2y2014/2355
Sayfeddine, D. and al. 2014. Position prediction of a dynamic agent tracked by quadrotor using neural
networks. Electronic scientific journal. Engineering Journal of Don. 1:
http://www.ivdon.ru/en/magazine/archive/n1y2014/2293
Shang, H. and al. 2010. Low cost UAV based thermal infrared remote sensing: platform, calibration and
application. IEEE: 38-43.
Yixin Chen, Yi Xie and Yulin Wang. 2013. Detection and recognition of traffic signs based on HSV vision
model and shape features. Journal of computers, Vol.8., No.5: 1366-1370.