A Project Report on
“Design and Simulation of Last Mile
Delivery Robot”
Submitted in partial fulfilment of the requirements
for the award of degree of
BACHELOR OF ENGINEERING
In
MECHANICAL ENGINEERING
Submitted by
Mr. Ghure Mohd. Muneer (7117015)
Mr. Hamza Shabbir Husein (7117020)
Mr. Ammaar Shahid Solkar (7218075)
Mr. Khatib Salman Mohammed Rafi (7114027)
Under the guidance of
Er. Parvez Waghoo
Department of Mechanical Engineering
M. H. SABOO SIDDIK COLLEGE OF ENGINEERING
8, SABOO SIDDIK POLYTECHNIC ROAD,
MUMBAI - 400 008
(A.Y 2020-21)
A Project Report on
“Design and Simulation of Last Mile
Delivery Robot”
Submitted in partial fulfilment of the requirements
for the award of degree of
BACHELOR OF ENGINEERING
In
MECHANICAL ENGINEERING
Submitted by
Mr. Ghure Mohd. Muneer (7117015)
Mr. Hamza Shabbir Husein (7117020)
Mr. Ammaar Shahid Solkar (7218075)
Mr. Khatib Salman Mohammed Rafi (7114027)
Under the guidance of
Er. Parvez Waghoo
Department of Mechanical Engineering
M. H. SABOO SIDDIK COLLEGE OF ENGINEERING
8, SABOO SIDDIK POLYTECHNIC ROAD,
MUMBAI - 400 008
(A.Y 2020-21)
i
CERTIFICATE
This is to certify that the project entitled “Design and Simulation of Last
Mile Delivery Robot” is a bonafide work of Mr. Ghure Mohd. Muneer
(7117015), Mr. Hamza Shabbir Husein (7117020), Mr. Ammaar Shahid
Solkar (7218075), Mr. Khatib Salman Mohammed Rafi (7114027) submitted
to the University of Mumbai in partial fulfillment of the requirement for the award
of the degree of Undergraduate in Mechanical Engineering.
_________________
Prof. Parvez Waghoo
(Guide)
_______________________
_______________________
Dr. Javed Shaikh
(HOD - Mechanical)
Dr. Ganesh Kame
(I/C Principal)
_______________________
_______________________
Internal Examiner
External Examiner
i
Project Report Approval for B. E.
This project report entitled “Design and Simulation of Last Mile Delivery
Robot” by Mr. Ghure Mohd. Muneer, Mr. Hamza Shabbir Husein,
Mr. Ammaar Shahid Solkar, Mr. Khatib Salman Mohammed Rafi is
approved for the degree of Mechanical Engineering.
Examiners
1.---------------------------------------------
2.---------------------------------------------
Date:
Place: Mumbai
ii
DECLARATION
We declare that this written submission represents our ideas in our own words
and where others' ideas or words have been included, we have adequately cited
and referenced the original sources. We also declare that we have adhered to all
principles of academic honesty and integrity and have not misrepresented or
fabricated or falsified any idea/data/fact/source in my submission. We understand
that any violation of the above will be cause for disciplinary action by the Institute
and can also evoke penal action from the sources which have thus not been
properly cited or from whom proper permission has not been taken when needed.
Name
Roll No.
Ghure Mohd. Muneer
7117015
Hamza Shabbir Husein
7117020
Ammaar Shahid Solkar
7218075
Khatib Salman Mohammed Rafi
7114027
Date:
iii
Signature
ABSTRACT
E-commerce has become an indispensable part of our life today. An important aspect of ecommerce is the fast, economical and safe delivery of the product to the customer right at their
doorstep i.e. the last mile delivery. The last mile delivery contributes to a major proportion of
the overall shipping costs and logistics companies have been looking to reduce this cost by
using robots and drones. Several companies have built delivery robots which can deliver
packages to the customers in open areas. The limitation of these robots however is that they
cannot deliver the package at the door of the customer in an apartment building, at the most
delivering it at the lobby. This leaves out a major proportion of the customers using ecommerce to benefit from this technology. Here we have proposed a method, using which, a
robot can deliver packages not only at the lobby of an apartment building but also right at the
doorstep of the customer at any floor accessible by an elevator. We have developed a method
which uses grid maps to map multiple floors and perform deliveries. The developed method
has been successfully implemented in ROS, a popular robotics framework and simulated using
Gazebo, a multi-robot physics simulator. The implementation can be used as a starting point
for more robust and generalized methods for indoor delivery robots
The following report contains an explanation of the work done. The first three chapters discuss
the need for this project, existing literature and designs on delivery robots. In chapter 4 we lay
out the parameters we considered during the design process. Chapter 5 contains details on the
planning of the project and chapter 6 contains the architecture of the designed system for
mapping and delivery. Chapter 7 aims to give a brief overview of the major tools used during
the development process. The chapters 8 through 10 contain details of the implementation and
work done. Chapter 11 lays out the result and in chapter 12 we discuss the conclusions and the
areas to which can be improved. Finally chapter 13 contains the references used and is followed
by Appendix 1 which contains some important parts of the code base.
Keywords - Last Mile Delivery, Mobile Robotics, Autonomous Navigation, Multifloor Navigation, Multi-floor Mapping.
iv
CONTENTS
Certificate
i
Approval Sheet
ii
Declaration
iii
Abstract
iv
Contents
v
List of Figures
viii
List of Tables
ix
Chapter 1 Introduction
1.1 Problem Statement
1
1.2 Project Goals
2
1.3 Project Objectives
2
1.4 Project Overview
3
Chapter 2 Literature Review
2.1 Introduction
4
2.2 Navigation methods
4
2.3 Image Processing
5
Chapter 3 Existing Designs
3.1 Introduction
7
3.2 Savioke
7
3.3 Starship Technologies
8
3.4 Kiwibot
9
Chapter 4 Design Parameters
4.1 Objective
10
4.2 Mechanical Design Parameters
10
4.3 Software Design Parameters
12
Chapter 5 Methodology
5.1 Work Breakdown Structure
13
5.2 PERT Analysis
14
i
Chapter 6 Architecture
6.1 Introduction
18
6.2 Mapping
19
6.3 Delivery
20
6.4 Discussion on Map Security
21
Chapter 7 Technologies
7.1 Introduction
23
7.2 ROS (Robot Operating System)
23
7.3 Gazebo
24
7.4 Computer Vision
26
Chapter 8 Implementation
8.1 Introduction
28
8.2 ROS Packages
29
8.3 Robot Description
30
8.4 Perception
32
8.5 Elevator
34
8.6 Simulation Environments
35
8.7 Mapping
38
8.8 Navigation
41
8.9 Vision
47
8.10 Delivery
48
Chapter 9 Mechanical Design
9.1 Introduction
51
9.2 Motor Torque & Power Calculations
51
9.3 Components Suggestion
55
9.4 Concept Design Sketches
58
Chapter 10 Image Processing
10.1 Introduction
59
10.2 Necessity
60
10.3 Working
61
ii
Chapter 11 Results
64
Chapter 12 Conclusion
12.1 Conclusion
68
12.2 Future Work
69
Chapter 13 References
70
Appendix I
72
iii
LIST OF FIGURES
Fig. No.
Title
Page No.
Fig 2.1
Image processing of a sign to obtain text
6
Fig 2.2
Objects as seen from camera (right) and objects detected after image
processing (left)
6
Fig 3.1
Savioke Relay Robot
8
Fig 3.2
Starship Technologies Robot
8
Fig 3.3
Kiwibot Robot
9
Fig 5.1
Work Breakdown Structure
14
Fig 5.2
Critical Path
16
Fig 6.1
Mapping Architecture
19
Fig 6.2
Delivery Architecture
20
Fig 7.1
ROS Logo
23
Fig 7.2
Gazebo logo
24
Fig 7.3
Gazebo features
25
Fig 7.4
Object detection and labelling
26
Fig 8.1
ROS package distribution of project
28
Fig 8.2
DeliveryBot simulation model
30
Fig 8.3
GUI for DeliveryBot control
31
Fig 8.4
DeliveryBot dropping package
31
Fig 8.5
rqt_robot_steering plugin for control of DeliveryBot’s motion
31
Fig 8.6
Visualization of PointCloud2 laser data from simulation environment
(left) in RViz (right)
33
Fig 8.7
Visualization of depth cloud data from simulation environment (left) in
RViz (right)
33
Fig 8.8
Visualization of camera data from left, right, and front cameras
respectively
34
Fig 8.9
GUI for Elevator control
34
Fig 8.10
CAD model of elevator (left), transparent visualization of elevator
model in gazebo (center), visualization of elevator model with door
open in gazebo (right)
35
Fig 8.11
Simulation environment of building in Gazebo
36
iv
Fig 8.12
Close up of simulation environment of building in Gazebo
36
Fig 8.13
CAD model of another simulation environment with ramp and stairs
based on real building
37
Fig 8.14
A grid map
38
Fig 8.15
Multi layered map composed of grid maps for multi-floor map
39
Fig 8.16
Mapping GUI
39
Fig 8.17
Created map with marked points
41
Fig 8.18
Figure showing different types of costmaps and plans
42
Fig 8.19
Door detection and number plate recognition
47
Fig 8.20
Delivery GUI
48
Fig 8.21
Multi-floor navigation illustration
49
Fig 8.22
Flow of delivery process
50
Fig 9.1
MATLAB Script for Torque Requirement Estimation
52
Fig 9.2
Torque Estimation Results
53
Fig 9.3
MATLAB Script for motor power estimation
54
Fig 9.4
Motor Power Estimation Results
54
Fig 9.5
Components Packaging Concept Sketch
57
Fig 9.6
Shape Study
57
Fig 9.7
Concept Exploration
58
Fig 9.8
Packaging Concept
58
Fig 10.1
Door and apartment number detection in Python
61
Fig 10.2
Door and apartment number detection implementation in ROS
61
Fig 11.1
Mapping a building
65
Fig 11.2
DeliveryBot dropped at building for delivery
65
Fig 11.3
DeliveryBot waiting for elevator
66
Fig 11.4
DeliveryBot dropping package at door
66
Fig 11.5
DeliveryBot waiting to be picked up after delivering package
67
v
LIST OF TABLES
Table No.
Title
Page No.
Table 5.1
Duration of work
15
Table 5.2
Slack table
17
Table 9.1
Processing Board Options
55
Table 9.2
Depth Camera Options
55
Table 9.3
Laser Scanner Options
56
vi
Chapter 1
Introduction
1.1 Problem Statement
The last mile delivery market accounted to $ 1.99 B in 2018 and is expected to grow at a CAGR
of 16.7% during the forecast period 2019 – 2027, to account to $ 7.69 B by 2027 [1]. The
requirement for better and optimized last mile delivery methods is increasing rapidly as more
and more customers are taking their shopping online for food, grocery, medicines, electronics,
furniture, etc. Consumers want faster deliveries and want them more frequently. The timeliness
of deliveries is getting more important as the online shopping culture becomes mainstream and
customer satisfaction is very important even in the delivery aspect. Last mile delivery costs
account for 53% of the total cost of shipping [2] and in order to stay profitable while providing
deliveries at lower costs, newer methods and technologies have to be developed and adopted.
To do this several companies and startups are exploring ways to use drones, automated ground
vehicles, autonomous vehicles droids and others to fulfill this requirement. Companies like
StarShip, Eliport, Delivers AI and OTSAW have started delivery services using ground
vehicles mainly working using GPS and delivering in open areas. Companies like Amazon,
UPS, Wing are working with drones to execute deliveries where they drop the packages at the
customer's house. One important limitation of these drones and AGVs is that they currently
only work for open areas and places with suburban housing. They cannot deliver to the
customers' doors in buildings and high-rises either requiring them to collect the packages from
the lobby or outright leaving them out of the target customer base. Closing the last mile delivery
1
gap is an important topic to be able to completely automate the delivery processes. Exploring
the limitations that current products and methods used for package delivery face in delivering
to buildings and developing a proof of concept to overcome it is the goal of this project.
1.2 Project Goals
A last mile delivery robot for delivering to apartments in buildings will require certain critical
functionalities. These functionalities are listed below:
1. Ability to navigate autonomously in crowded indoor areas as well as outdoor areas.
2. Ability to communicate with elevators in the building where the robot is executing the
delivery.
3. Ability to autonomously operate and ride elevators to go from one floor to another while
carrying packages.
4. Ability to navigate in areas with little to none prior mapping.
5. Ability to safely deliver packages to the customer’s house or office by either delivering
the package in a safe storage box outside the customer’s house or by going into the
customers house by using IoT, dropping the package in the house and leaving.
6. Ability to communicate with the central or decentralized server to get delivery details
and packages.
7. Keep all invested parties up to date on the status of the package by providing on time
updates.
8. Ability to work as a swarm of robots in collaboration with delivery vans which can
transport these robots from the delivery sites to the warehouses.
9. Have a likeable appearance and not cause any discomfort or harm to the humans it
encounters.
These goals are some of the qualities and functionalities a delivery robot for buildings should
have.
1.3 Project Objectives
The objectives of this project are to implement some of the goals in some form or other to
develop a proof of concept for the delivery robot. Some goals stated previously will require
technologies which are considered state of the art at the time of writing and a major time
commitment so they are reduced to a simpler and currently implementable level. Taking these
factors and time constraint into consideration, the following are the objectives of this project:
2
1. Develop a simulation environment for the delivery robot that resembles the
environment the robots will face in the real world.
2. Develop the simulation version of the delivery robot which can be simulated in the
simulation environment and can be directly deployed on hardware with little to no
changes.
3. Implement navigation techniques in the delivery robot to allow the robot to navigate
autonomously while avoiding obstacles.
4. Develop software to allow robots to map buildings which it can later use during
navigation.
5. Develop a method for multi-floor navigation of the delivery robot.
6. Explore concepts for storage and delivery mechanisms.
7. Explore concept designs for the delivery robots.
8. Select hardware components which will be required to build a physical prototype of the
robot.
1.4 Project Overview
In this project we develop and simulate a delivery robot which can be dropped outside the
building by an autonomous delivery van from where it can go to the required apartment door
by navigating the hallways and operating the elevators. Important functionality which are
focused upon are the ability to plan the path from the dropped location to the delivery apartment
door, perform multi-floor navigation, navigate the area while avoiding obstacles, operating
elevators to move between floors, verify that the package is being delivered to the correct
location and then dropping the package. This is developed in the industry standard for robot
middleware Robot Operating System (ROS) [3] and simulated in a multi-robot physics
simulation software Gazebo [4]. We have developed a method to create multi-floor maps using
grid maps and perform multi-floor navigation using these maps using the ROS navigation stack
[5].
3
Chapter 2
Literature Review
2.1 Introduction
Before we discuss our approach to solving the problem statement, we first have to look
into existing literature in order to familiarize ourselves with the current progress or solutions
found within certain domains. By conducting a literature review, we are able to identify the
methods used to gain a conclusion, and also understand the strength and weaknesses or the
opinions of researchers on a topic. We are also able to justify our choice of design, as well as
how we aim to improve on existing research.
2.2 Navigation methods
2.2.1 Navigation tuning guide
Kaiyu Zheng [6] discusses the tuning process of the ROS navigation stack which is responsible
for producing safe paths for the robot to navigate in the environment. The author first discusses
how to obtain and set the maximum velocity and maximum acceleration of the robot in the
navigation stack. After that the author discusses the different types of global planners available
and their differences. The author then explains the global planner parameters and their effects
on the produced paths and also suggests values for the parameters which the author found to
be acceptable after tuning. The author selected Dynamic Window Approach (DWA) [7] local
4
planner for the navigation stack. We used the DWA planner in the initial version of the robot
then switched to Timed Elastic Band (TEB) [8] local planner due to the latter having some
advantages which we required. The author also discusses costmap parameters, recovery
behaviours, localization and problems. We found this paper to be extremely valuable in our
initial versions and saved us a lot of time.
2.2.2 Multi-floor mapping with uncertainty
James Ellingson et al. [9] outline their research on autonomous robots where they developed a
robot capable of multi-floor navigation. They used a turtlebot [10] as their robot base with a
RGB-Depth (RGB-D) camera for perception. They described their method of tackling elevator
banks at their campus for navigation. As they were sending the commands to the robot through
a base computer over Wi-Fi, they reported problems with the robot losing Wi-Fi connectivity
which they solved by putting the robot in autonomous mode while it is in the elevator. The
authors suggest the need of a method to change map after changing the floor and using a
robotics manipulator to operate the elevator buttons in the future work section.
2.3 Image Processing
Here we will look into the various research papers which have been selected for our project
based on the findings of the papers.
2.3.1 Text Detection & Recognition
Muhammad Sami et al. discuss methods to extract text from indoor sign boards to aid in
navigation of a robot [11]. The authors discuss the two steps of information extraction from an
image, first by understanding the type of objects in the image captured, and next interpreting
the letters and symbols present (if any). The focus of the paper is to discuss current ongoing
research related to text recognition and the proposed algorithms which can yield high accuracy
and maximum results. The authors also discuss in depth the process they have selected for
detecting text. While we are not primarily concerned with the color of the signage as it does
not aid in our solution and can free up processing power, we do learn about shape and text
extraction. Based on the paper’s findings, we have decided to move forward with our approach
for object detection, image processing. We have also chosen to implement Optical Character
Recognition for text detection as per the findings of the paper.
5
Fig 2.1 Image processing of a sign to obtain text
2.3.2 Object Recognition
Chae, Hee-Won, et al. discuss the implementation of object recognition by analyzing their
surface appearance using depth sensors [12]. Currently, there is a vast amount of research being
done in the field of object detection using depth sensors, primarily for indoor navigation. The
authors discuss how to recognize objects which are intended for Simultaneous Localization
and Mapping (SLAM). Since most robots travel on the ground, this method aims to also reject
the floor surface, allowing for faster processing and detection. This process is also applied for
other surfaces, such as walls. Further, the authors also discuss the algorithm they have used in
order to detect the object and its depth, i.e. distance from the robot. While we will not be
implementing such an algorithm in our project, the purpose of studying this paper is to
understand how object detection can be achieved and to understand how the robot should
perceive and interact with objects, as well as how to fine tune processing so that high accuracy
is achieved to separate objects from other surfaces.
Fig 2.2 Objects as seen from camera (right) and objects detected after image processing (left)
6
Chapter 3
Existing Designs
3.1 Introduction
In this chapter we discuss the autonomous ground robots currently being used for deliveries
in gated areas like universities and suburban neighborhoods.
3.2 Savioke
Founded in San Jose, California in 2012, Savioke robots have gained popularity as friendly and
sophisticated robots that are primarily used as indoor delivery bots in hotels and residences
[13]. These robots are designed such that they are excellent at navigating winding corridors
and highways, however, outside that environment, they are very vulnerable. Their most popular
model is ‘Relay’. The robot is primarily used in hotels to deliver small and lightweight items
to rooms on different floors of the hotel. The robot is designed such that it performs superbly
to deliver items, however as a parcel delivery bot it is not good enough. The cargo hold of the
robot, which is designed for small items, cannot fit a majority of package sizes that are ordered
by consumers from various online E-tailers. Another design flaw is the overall shape of the
robot. Being a tall and slim robot, such a design is easily vulnerable to toppling over due to a
high center of gravity. To overcome this the footprint of the robot has to be substantially larger,
which increases its overall width and thus faces difficulty in avoiding obstacles or humans. It’s
7
very minimal ground clearance also disables it on uneven terrain. The robot operates on ROS
which is what we will be implementing in our delivery robot.
Fig 3.1 Savioke Relay robot
3.3 Starship Technologies
Founded in San Francisco in 2014, Starship technologies[14] is the largest autonomous
delivery robot maker to-date. The robot has found success as a food and grocery delivery robot.
Even with large wheels and clearance, it is unable to drive on uneven terrain and cannot
overcome obstacles higher than a sidewalk. Current iteration of robot is well suited to navigate
large sidewalks and suburban neighborhoods, but is unable to navigate in an indoor
environment due to it being dependent on GPS location to move about. Its design is such that
it can carry a substantial variety of package sizes and weights, while still maintaining a decent
footprint such that it doesn't take up a large amount of space. The implementation of cameras
on both the front and back allow for a 360-degree view of its surroundings, thus allowing it to
easily navigate and avoid obstacles.
Fig 3.2 Starship Technologies robot
8
3.4 Kiwibot
Founded in Berkeley in 2017, it is one of the more recent entries into the robot delivery
space[15]. The company operates primarily on college campuses, delivering food to students
either at college or hostels. The goal of the company is to minimize the number of humans
delivering and rather allow a person to oversee five to six KiwiBots. The main advantage the
company has are the cheap food options that it provides to its customers, thereby allowing it to
grow its user base. One of the perks of the robot is its smaller footprint than the competition,
allowing it to easily avoid and go around obstacles while taking up less space on the floor as it
moves. Currently, the robot can only deliver up to the main floor of a residence, requiring the
customer to leave their homes to retrieve their order. The small size of the robot also means it
can't always handle uneven surfaces, and require input from operators from time to time.
Fig 3.3 Kiwibot robot
9
Chapter 4
Design Parameters
4.1 Objective
To create a prototype design of a robot that can navigate from the main entrance of a residential
complex to the door of the delivery target utilizing the ramps and elevators present in the
buildings. The robot should be able delivery packages to the door without any person having
to interact with it.
4.2 Mechanical Design Parameters
These are some of the parameters taken into account before starting the design process. The
influence of the parameters and the assumptions made to exclude them will depend on the time
constraints of the project.
1. Gradients: The robot should be able to climb gradients it encounters in its path without
toppling. These gradients will generally be the wheelchair ramps at the entrance of the
lobby for the building. Standards for these have to be found and the robot should be
designed accordingly.
2. Elevators: To be able to deliver products in buildings, the robot will have to use the
elevator to reach floors other than the ground floor. For this the robot should have some
10
means to interact with the elevator. This consideration will also affect the height of the
robot to make it easier to reach the elevator control panel.
3. Footprint: The robot footprint should not be very large so as to not inconvenience
people coming in its way. This consideration goes hand in hand with elevator and
gradient considerations since to make the footprint smaller, the height will have to be
increased which will in turn increase the toppling tendency. A suitable compromise will
have to be reached.
4. Package Size: The robot has to be designed to accommodate several packages. The
capacity of each package compartment has to be decided based on package sizes of
frequently ordered goods. To get this data, product dimensions of high and frequently
sold goods can be web scraped from e-commerce websites. Variations of the robot can
be made to carry fewer packages of larger goods.
5. Coexistence: To ensure that the robot does not pose danger to any person or animal it
encounters in its way it should have no sharp corners or exposed moving parts as much
as possible along with sensing capability to stop when it comes in contact with
someone. The aesthetics of the robots also have to be designed in such a way that they
are accepted by a vast majority of people and not feel uncomfortable in the presence of
a robot.
6. Range and Battery: Increasing operating time of the robot will require an increase of
the battery size which in turn will increase the weight of the robot and decrease space
available for cargo. A suitable compromise has to be reached between operating time,
battery size and operating costs. The vans in which the robots will arrive also need to
have charging docks available for the robots.
7. Privacy and Security: There will be concerns regarding privacy and security with the
robots having some information of the delivery target and the fear of being hacked.
These have to be addressed and the architecture has to be designed in such a way that
only the minimal amount of information required is given to the robot and major data
such as the map of the housing complex does not leave the complex.
8. Delivery Mechanism: The delivery mechanism should be such that it does not harm
the package during delivery and be able to cycle through packages in the cargo.
11
4.3 Software Design Parameters
Software plays a very important role in the robots and has to be designed carefully for efficient
functioning. Some important parameters taken into account are listed below.
1. Map Creation: Creating a map is an important part of the navigation. SLAM
(Simultaneous localization and mapping) technology has not yet advanced so much as
to allow quick and reliable navigation without maps. The map creation process has to
be easy and as much automated as possible. Places where operator input is required
have to be carefully designed to minimize human errors and time wastage.
2. Navigation: The navigation module of the robot is one of the most important modules
in the robot. This module has to be able to plan and navigate efficiently taking the
shortest safest path to the destination to minimize the delivery times. It has to be reliable
and self-correcting. Care has to be taken that the module also consumes as less
computation resources as possible to allow use of lower end and cheaper hardware to
be used in the robots.
3. Obstacle Avoidance: Obstacle avoidance is another module which has to be designed
carefully. This module is responsible for avoiding any static and dynamic obstacles
which come in the way of the robot. This module prevents accidents from happening
and makes the robots feel safe to the humans around the robots.
4. Image Recognition: This module provides the robot with a way to perceive and
recognize the objects it encounters in the world. This module works and provides
information to both navigation and obstacle avoidance modules and also helps in
delivery verification. This is a resource intensive module and tradeoffs will have to be
made between functionality and resource requirements.
5. Delivery Module: This module is supposed to take care of the application layer of the
delivery robot. The module has to receive delivery details, create a plan from these
details and provide information to the navigation module to create the path, perform
verification before delivering the package, etc.
6. Hardware Control: This module provides an interface between the hardware and the
software. The navigation instructions like heading and orientation and operation of
joints are converted into motor actuation and signals by this module.
12
Chapter 5
Methodology
5.1 Work Breakdown Structure
In order to realize our project, we have to list down every task that needs to be completed. The
below chart shows the 4 primary domains of our project and also sub-divides it into different
categories:
1. Mechanical Design
2. Environmental Design
3. ROS
4. Imaging
13
Fig 5.1 Work Breakdown Structure
5.2 PERT Analysis
To ensure that each task is completed on time and any delay does not affect the overall output
of the project, we have used Program Evaluation and Review Technique (PERT) to understand
the interdependencies of the work components and find the minimum time within which the
project can be completed. The table below lists out each task and its duration of work (in days).
14
Task
Description
Depends Optimistic
on
Time (Days)
Most
Likely
Time
(Days)
A
Design
Requirements
-
5
7
8
7
B
Site Survey
-
3
7
8
7
C
[ROS] Sensors
-
4
7
8
7
D
Apartment
Identification
-
10
14
18
14
E
Components
Selection
A
3
4
7
5
F
Floor Env
B
3
5
7
5
G
[ROS] Mapping
C, F
4
5
6
5
H
Elevator
Identification
D
10
14
16
14
I
Interaction
Mechanism
E
7
10
12
10
J
Storage
Mechanism
I
7
10
12
10
K
Robot Design
J
15
20
25
20
L
Lobby Env
F
3
5
6
5
M
Complex Env
L
3
5
6
5
N
Elevators Env
M
3
5
6
5
O
Env Files
Launch
N
1
2
3
2
P
[ROS]
Navigation
M, G
3
5
7
5
Q
[ROS]
Controller
P
4
5
7
6
R
Elevator
Identification
H
10
14
16
14
S
Actors
Identification
R
7
10
12
10
T
[ROS]
Interaction
K, O, H
10
14
16
14
U
Integration
T, S, Q
10
20
25
20
Table 5.1 Duration of work
15
Pessimistic
Expected
Time (Days) Time (Days)
Below diagram illustrates the various paths that can be taken to complete the project. Based on
the priority of tasks that need to be completed, the critical path for the given project is: A - E I - J - K -Y - U
Fig 5.2 Critical path
To ensure that each task is started at the earliest, we need to know the earliest start, the number
of days to the latest start such that further tasks are least affected due to the slack. The below
table illustrates that, where black indicates slack days are available, and red indicates that no
slack days are available.
16
Task
Earliest Start
Latest Start
Slack (Days)
A
0
0
0
B
0
23
23
C
0
23
23
D
0
14
14
E
7
7
0
F
7
30
23
G
7
51
44
H
14
28
14
I
12
12
0
J
22
22
0
K
32
32
0
L
12
35
23
M
17
40
23
N
22
45
23
O
27
50
23
P
17
56
39
Q
22
61
39
R
28
48
20
S
42
56
14
T
52
52
0
U
66
66
0
Table 5.2 Slack table
Based on the PERT discussed above, we have come to a final expected duration of the project,
from start to end, which is 84 days. However, the duration is subjected to change due to factors
that are not in our control.
17
Chapter 6
Architecture
6.1 Introduction
The working of the delivery robot is divided in two different phases. The first phase is mapping
and the other part is the actual delivering phase. Before a robot can deliver to a building, the
building first has to be mapped out by the robot. This mapping includes creating a floor plan
and marking important locations on the created map. This map helps the robot to navigate when
it comes back later to deliver a package. The mapping has to be only done once and updated
only if there is a significant change in the landscape or structure of the building.
18
6.2 Mapping
Fig 6.1 Mapping architecture
The mapping phase is the first phase in the process. When a building or complex agrees to
allow delivery robots to perform deliveries, the vendor will then need to map out the building
or complex and add it to its database. The vendor can send one robot with an operator to the
site to carry out the mapping process. The operator has to control the robot using a remote
control device and wait for the robot to create a complete map of the area. The mapping
(gmapping)[16][17][18] package utilizes the sensor information from sensors like Light
Detection and Ranging sensor (LiDAR), Depth Cameras and odometry to create an area map.
During the mapping the operator can mark important locations like entry points, elevators,
pickup and drop points and room locations on the map. The robot only needs to map out the
lobby area and only one floor area if all floors are identical. If the layout of the floors change,
Different maps will have to be created for different floors. Once the map is complete, the map
is tagged with a unique id which will be used to identify the site and the map during the delivery
phase. This map is then sent to the mapping server of the vendor and stored in the database for
later use.
19
6.3 Delivery
Once the mapping phase is complete the building can be delivered to by any robot in the fleet.
The overview of the working and dependencies are shown in the figure below.
Fig 6.2 Delivery architecture
When a delivery is requested, the package is put into the robot and the delivery van drops the
robot at the set drop zone for the building. From here the robot's delivery client communicates
with the vendor's delivery server from where it receives the delivery details such as the floor
number, room number, building map, etc. The delivery details are then sent to the path planner
which uses the map to create a path which the robot has to follow to reach the desired
apartment. Once the path planning is complete, the plan is sent to the move base which is the
module which executes the plan. The move base receives the sensor data from the world which
it utilizes to avoid obstacles. When the robot has to interact with an elevator the Elevator
controller module kicks in which communicates with the elevator API for the building. In this
way the robot can request the elevator to open or close doors and go to specific floors. Once
the robot reaches the apartment, it runs a verification check to see if it is in front of the correct
location. This is done by running an object detection and Optical Character Recognition (OCR)
20
program to get the room number from the number plate of the door. Once the location is verified
the robot drops the package in front of the door and sends a verification message to the
customer as well as the vendor. This completes the delivery of the package. The robot then
takes a similar route back to its drop location from where it will be picked up by the delivery
van.
6.4 Discussion on map security
Since the map created by the robot contains important information about the site, security
becomes a big concern when saving them for later use and deciding who can access these maps.
There are several alternatives as to where maps can be stored depending on the level of control
required. Some are discussed in the following subsections.
6.4.1 Map with Robot
This is the easiest way to store maps after being created. The maps are created on the robot and
annotated by a user using an external device. When the map creation is complete the maps can
be stored on the storage system on the robot. This makes it so that only the robot which was
used during the map creation can be used for delivery in that location. This provides good
security as the map is not shared on a central database and no vendor has access to the map.
This poses a serious limitation to the generality of the delivery robots. Storing maps on the
robot makes it so that only that specific robot can be used for delivery. This method can be
used if the business model is such that the building or housing complex purchases robots for
internal delivery use and can be accessed by vendors by allowing the vendors to drop packages
in the robot along with the delivery information. This also reduces the chances of monopoly
being created by a single vendor. This requires building complexes to make a capital
investment of purchasing robots and then bear the cost of maintaining them. This method does
not require a significant amount of digital storage for saving the maps since the robot will only
have a store a handful of maps.
6.4.2 Map with Building
An alternative to storing maps on the robot is to store the maps within the IoT infrastructure of
the building. This is assuming the building already has an IoT infrastructure which provides
communication lines between the site elevators, doors and other IoT devices. This
infrastructure can be leveraged to store maps of the building. In this alternative method, any
vendor can send their delivery robots to the building or complex with the delivery details and
21
the robot can receive the map for the building IoT Access Point. This eliminates the need for
the buildings to have their own robots and reduces the capital investment and maintenance cost
while at the same time not giving up the map to any specific vendor. With this method the map
can still potentially be stored by the vendor permanently when the robot receives the map but
then it comes under the umbrella of regulations. This method is a compromise between security
and convenience. This method does not require storage space on the robot for storing maps
since the map is received during the beginning of delivery and then forgotten once the delivery
is complete.
6.4.3 Map with Vendor
Storing maps on robots posed limitations of investment in purchasing and maintaining a robot
while providing convenience and preventing vendor monopoly in the robotic delivery segment.
Storing maps with the IoT infrastructure on the other hand did not face the limitation of having
to purchase robots as the map data can be accessed by any vendor executing a delivery. This
poses a technical challenge as the API of the IoT infrastructure and the system in place to
communicate with these robots will have to be standardized across buildings and vendors. This
is a difficult task and will take careful planning and collaborative effort from Delivery vendors,
IoT vendors and Builders to execute successfully. A third alternative to the two aforementioned
methods is to have the map available with the vendor. Since a vendor will be investing in
getting the maps created for the building they can argue that they have the right to not share
the map with other vendors. This will then require that all vendors separately create their own
maps and store them individually on their own central databases. The maps can be stored on
the vendor’s map database from where the map can be sent to the robot during delivery along
with the delivery details. This however exposes the map with several vendors a breach of data
of any of which will leave the building vulnerable.
22
Chapter 7
Technologies
7.1 Introduction
The three major technologies that we are using are ROS, Gazebo and Computer Vision. A brief
description of their capabilities are provided in the following pages.
7. 2 ROS (Robot Operating System)
Fig 7.1 ROS Logo
The Robot Operating System (ROS) is a flexible framework for writing robot software. It is a
collection of tools, libraries, and conventions that aim to simplify the task of creating complex
and robust robot behavior across a wide variety of robotic platforms. Why? Because creating
truly robust, general-purpose robot software is hard. From the robot's perspective, problems
that seem trivial to humans often vary wildly between instances of tasks and environments.
Dealing with these variations is so hard that no single individual, laboratory, or institution can
hope to do it on their own. As a result, ROS was built from the ground up to encourage
collaborative robotics software development. For example, one laboratory might have experts
23
in mapping indoor environments, and could contribute a world-class system for producing
maps. Another group might have experts at using maps to navigate, and yet another group
might have discovered a computer vision approach that works well for recognizing small
objects in clutter. ROS was designed specifically for groups like these to collaborate and build
upon each other's work, as is described throughout this site.
7.2.1 Reason for using ROS:
ROS makes it easier to build robotic applications faster without having to reinvent the wheel
and build core robotic functionality from scratch saving precious time. ROS is also used in a
large number of robots such as the PR2, Turtlebots and such.
7.3 Gazebo
Fig 7.2 Gazebo Logo
Robot simulation is an essential tool in every roboticist's toolbox. A well-designed simulator
makes it possible to rapidly test algorithms, design robots, perform regression testing, and train
AI systems using realistic scenarios. Gazebo offers the ability to accurately and efficiently
simulate populations of robots in complex indoor and outdoor environments. At your fingertips
is a robust physics engine, high-quality graphics, and convenient programmatic and graphical
interfaces. Best of all, Gazebo is free with a vibrant community.
24
Fig 7.3 Gazebo features
7.3.1 Reason for using Gazebo:
Creating physical prototypes is expensive. Gazebo provides us with a realistic simulation
environment where main functionality can be tested. Gazebo integrates with ROS to provide
simulation which can be directly deployed on hardware with minimal to no changes as the
simulator can simulate actuators available on the market.
25
7.4 Computer Vision
Fig 7.4 Object detection and labelling
Computer vision is an interdisciplinary scientific field that deals with how computers can gain
high-level understanding from digital images or videos. From the perspective of engineering,
it seeks to understand and automate tasks that the human visual system can do.
Computer vision tasks include methods for acquiring, processing, analyzing and understanding
digital images, and extraction of high-dimensional data from the real world in order to produce
numerical or symbolic information, e.g. in the forms of decisions. Understanding in this context
means the transformation of visual images (the input of the retina) into descriptions of the
world that make sense to thought processes and can elicit appropriate action. This image
understanding can be seen as the disentangling of symbolic information from image data using
models constructed with the aid of geometry, physics, statistics, and learning theory.
The scientific discipline of computer vision is concerned with the theory behind artificial
systems that extract information from images. The image data can take many forms, such as
video sequences, views from multiple cameras, multidimensional data from a 3D scanner, or
medical scanning device. The technological discipline of computer vision seeks to apply its
theories and models to the construction of computer vision systems. [19]
7.4.1 Reason for using Computer Vision:
Cameras provide a high amount of information as opposed to sensors based on light or
ultrasound. However, to be able to utilize this rich information, a high amount of processing is
required which is done by computer vision techniques. Depth cameras provide even higher
amounts of information with point cloud data and imaging which can be used to recreate a 3d
26
world from the data. Computer vision has evolved over the years and has extensive use in
robotics applications and will help us reduce the number and types of sensors required.
27
Chapter 8
Implementation
8.1 Introduction
The firmware and simulation of the delivery bot is done in ROS and Gazebo. In ROS the
functionality is divided into packages where each package handles one functionality of the
robot. As such for Delivery Bot the whole project is divided into three categories and then
further divided into packages.
Fig 8.1 ROS package distribution of the project
28
8.2 ROS Packages
8.2.1 Delivery Bot
This meta package contains all the packages required to give the desired functionality to the
delivery robot.
1. deliverybot_control - Control contains the code to provide control of the hardware
actuators. Also it provides the ability to simulate and control these actuators in the
Gazebo simulation.
2. deliverybot_delivery - Delivery provides the application level functionality for
planning and executing deliveries.
3. deliverybot_description - Description contains the Universal Robot Description
Format (urdf) description of the robot. It also contains the meshes for the visual and the
collision geometry of the robot.
4. deliverybot_gui - GUI contains the graphical user interfaces for different functionality
of the robot.
5. deliverybot_mapping - Mapping functionality to create and load maps for navigation
using map_server.
6. deliverybot_navigation - Navigation provides basic navigation ability.
7. deliverybot_vision - Vision provides image processing and recognition abilities for
various tasks.
8.2.2 Elevator
This meta package contains packages to simulate an elevator in gazebo.
1. elevator_controls - Controls package provides the code to control the elevator in the
simulation.
2. elevator_description - Description provides the urdf description of the elevator and
contains the meshes for visual and collision geometry.
3. elevator_gui - GUI provides graphical user interfaces for controlling the elevator
during simulation.
8.2.3 Simulations
This meta package contains the package with the simulations for the project.
29
1. deliverybot_simulations - Simulations package contains the code to simulate the
robot and environment in Gazebo.
8.3 Robot Description
Fig 8.2 DeliveryBot simulation model
A simple robot is used for simulations. It contains a LIDAR sensor, a depth camera, 3 side
cameras, a door and a pusher for pushing the package out of the robot. The door and pusher are
actuated. The door has a revolute joint which can be controlled to open and close the door. The
pusher has a prismatic joint to move the package out of the robot. The wheels are actuated and
a differential drive plugin is used for their control.
To control the delivery bot joints, these services are provided.
/dbot/open_bot_door
/dbot/close_bot_door
/dbot/set_pusher_in
/dbot/set_pusher_out
/dbot/deliver_package
These services can either directly be called programmatically or by using a GUI for easier use.
30
Fig 8.3 GUI for DeliveryBot control
Fig 8.4 DeliveryBot dropping package
To move the robot, we have to publish Twist messages to the cmd_vel topic. The Twist message
is composed of two vectors one each for Linear and Angular. The differential drive plugin
converts this Twist message into individual speeds for left and right wheels. The Twist message
can be published manually, by a GUI or by a navigation node. To publish manually, we have
to publish a message to /dbot/cmd_vel. To use a GUI, we can use rqt_robot_steering.
Fig 8.5 rqt_robot_steering plugin for control of DeliveryBot’s motion
31
The GUI will mostly be used when creating a map or when the robot gets stuck and has to be
manually controlled. The rest of the time it will be published to by the navigation node based
to the planned path.
8.4 Perception
Robot perception is a very important aspect which is crucial for the robot to understand its
environment. We have provided three main different types of sensors to the robot which are
2D LIDAR, 3D depth camera and RGB cameras. The 2D LIDAR is an important sensor used
for mapping and navigation. The depth camera helps to perceive objects and obstacles which
are missed by the LIDAR and identify them. The RGB cameras are used to keep a tab on the
environment when the robot is out for delivery.
The specifications of the sensors are listed below.
1. 2D LIDAR
i) Max Distance - 10.0 m
ii) Min Distance - 0.1 m
iii) Samples - 360
iv) Angle - 180°
2. Depth Camera
i) Resolution - 640 x 480
ii) Format - R8G8B8
iii) Horizontal FOV - 60°
3. RGB Camera
i) Resolution - 800 x 800
ii) Format -R8G8B8
iii) Horizontal FOV - 80°
The output from these sensors is visualized using RViz (ROS Visualization) a visualization
tool for sensory data.
32
Fig 8.6 Visualization of PointCloud2 laser data from simulation environment (left) in
RViz (right)
Fig 8.7 Visualization of depth cloud data from simulation environment (left) in RViz
(right)
33
Fig 8.8 Visualization of camera data from left, right, and front cameras respectively
8.5 Elevator
Simulating an elevator is an essential part of the project since the robot has to ride in the
elevators to move to different floors during delivery. Initially, we used an elevator plugin which
is provided by Gazebo. This plugin had the downside of having very low amount of control
available to the user. To overcome this, we created our own elevator model and added controls
to it such that it can be controlled using a GUI or by calling service which allows to change
floors, open doors and close doors. These services act as the API assumed at the start of the
project. To use the elevator in the simulation, we went for a multi-robot approach to allow the
elevator to be controlled using ROS. In this approach both the actual robot and the elevator are
considered as robots and we can control them using ROS.
Fig 8.9 GUI for elevator control
The services which are advertised for control are /elevator/open_elevator_doors
/elevator/close_elevator_doors
/elevator/elevator_goto_floor
34
These services provide the functionality for the elevator. The GUI uses these services in the
background and calls them on button press.
Fig 8.10 CAD model of elevator (left), transparent visualization of elevator model in
gazebo (center), visualization of elevator model with door open in gazebo (right)
8.6 Simulation Environment
To simulate the robot, we need an environment which resembles the actual environment the
robot will face during the delivery but the detail on the environment has to be reduced to be
able to simulate it at a sufficient speed. We created several different environments of buildings
with elevators to give the robot an environment to execute deliveries. The 3D models for the
environment can be created in any CAD modelling software. We used Onshape[20] and
Blender[21] for modelling the environments. Blender can be used for modelling as well as to
apply textures to the environment so that the cameras can get realistic looking images instead
of solid colors. These models can then be composed into a ‘.world’ file which can be launched
in the Gazebo simulator. The ‘.world’ files have a 3d model for the environment, some other
object models for obstacles and the elevator robot to act as a part of the environment.
35
Fig 8.11 Simulation environment of building in Gazebo
Fig 8.12 Close up of simulation environment of building in Gazebo
36
Fig 8.13 CAD model of another simulation environment with ramp and stairs based on
real building
37
8.7 Mapping
8.7.1 Mapping Mechanism
We use occupancy grid maps created from laser scan data as the basic method of storing maps.
There are several map creation packages available for ROS which require different types of
data input and different assumptions. We use gmapping for our map building. Gmapping
requires odometry and laser scan data as input for creating maps. Occupancy grid maps
represent an area by having different pixel values depending on the type of the area on the map.
The ones used here are obstacle, free and unknown. How much distance a pixel represents
depends on the resolution when creating the map (ex: - 1px = 5cm).
Fig 8.14 A grid map (White - Free, Black - Obstacle, Grey - Unknown)
These occupancy grid maps are good enough for navigation when the robot is already in the
area of the map but we run into problems when we have to change maps like when a robot
changes the floor using an elevator. To be able to create multi floor maps we add another layer
to the map and stack 2d grid maps with important points marked on the map.
To allow multi floor navigation we created a node which uses the map_server node to load
different layers of the map. When the robot changes a floor for example, from lobby to floor
we use the map loading service to load our floor map. In this way by using the robot’s odometry
we can localize the robot in the new map without any issues and have a multi floor map server.
38
Fig 8.15 multi layered map composed of grid maps for multi-floor map
8.7.2 Mapping Implementation
Now that we have the underlying mechanism for creating maps, we developed a method to
mark important points on the map which will be useful during the navigation. To create a map
of a building the user has to use the GUI which provides the required interface to create a map.
Fig 8.16 Mapping GUI
Steps to create a map 1. To create a map first, we input the building name in the building and hit create. This
will create a new map if a map of the same name does not exist. If a map of the same
name exists, it will load the files from that map to edit.
2. Once the map is created or loaded we can start the map server by pressing the start
button next to the server label. This will run the gmapping node and start creating the
39
map. The operator can drive the robot around the location till they are satisfied with the
occupancy grid map. Once the operator is satisfied they can press the lobby or floor
button depending on the location and it will create the .yaml and .pgm files for the map.
3. The GUI has buttons to mark the location of the elevator, lobby and rooms which can
be pressed during the mapping process to mark the locations. For the room marking on
the room number has to be marked along with the scheme so that the room and floor
number can be extracted from the room number during the delivery process.
4. The server has to be cleared before starting a new map section such as the lobby or
floor.
The building.yaml file for each building/map is created which contains the details of the
marked locations. This file is used when creating the navigation plan during the delivery phase.
Building Name - test_building
building: {floors: 3, name: test_building, scheme: F0R}
elevator:
in: {a: -1.61853604817792, x: 0.6934827549218775, y: 0.08420044952974128}
out: {a: 1.595608575754123, x: 0.3409216603784452, y: 2.2235174763106094}
lobby:
drop: {a: 1.574721731693996, x: 8.999215828701779, y: 17.49701041263829}
pickup: {a: -1.5457419755519313, x: 9.178838972561948, y: 16.942039005619698}
rooms:
1: {a: -0.06342197332648702, x: 3.3880842476089565, y: 3.0718819998791416}
2: {a: 0.060238990106272636, x: 3.606012970188086, y: 7.5487842424071205}
3: {a: 3.1305571781020154, x: -3.116973458806227, y: 7.5779168592677575}
4: {a: -3.1382769460531232, x: -3.194904404283088, y: 3.002927451759802}
40
Fig 8.17 Created map with marked points
8.7.3 Implementation Limitations
The current implementation is sufficient as a proof of concept however this implementation is
restrictive to a certain type of buildings which have identical floors and only two different
categories of locations, floor and lobby. This implementation also is not very user friendly for
end use so the GUI will have to be redesigned to be used with touch screens for easier and
intuitive usage where the occupancy grid map can be directly displayed on the screen and the
locations can be marked directly using touch. Provision to handle more elevators and different
categories of marked locations can be added to make mapping and navigation easier.
8.8 Navigation
For basic navigation we used the ROS navigation stack. It provides the basic navigation
functionality required for a robot. The navigation stack takes odometry and sensor information
and outputs velocity commands to the mobile base. The navigation stack has to be configured
to work for each robot. These configurations are done using a series of config files. The config
files are a set of costmap files, the global planner and the local planner.
41
8.8.1 Costmaps
A costmap is a grid map where each cell is assigned a specific value or cost: higher cost
indicates a smaller distance between the robot and an obstacle. Path-finding is done by a
planner which uses a series of different algorithms to find the shortest path while avoiding
obstacles. Optimization of autonomous driving at close proximity is done by the local costmap
and local planer whereas the full path is optimized by the global costmap and global planner.
Together these components find the most optimized path given a navigation goal []. Correctly
configuring costmaps is essential to ensure that the navigation is smooth and the robot
maintains an appropriate distance from obstacles while also not deviating too much from the
path and taking a longer detour.
The local costmap is the costmap for a short distance around the robot created using sensor
information and updated as obstacles come in the way. The global costmap is created using the
map. The global costmap is used when creating the global path that is the main path that the
robot has to follow to get to the destination. It however does not account for the obstacles that
come in the way. For this the local costmap comes into the picture where the local costmap
takes into account the obstacles and allows for creation of a local path which can avoid the
obstacles.
Fig 8.18 Figure showing different types of costmaps and plans
42
The common costmap parameters set for our robot are as follows.
costmap_common_params.yaml
robot_base_frame: dbot/base_link
max_obstacle_height: 20.0
robot_radius: 0.2
static_layer:
map_topic: /map
subscribe_to_updates: true
obstacle_layer:
observation_sources: laser #depth
laser:
topic: /dbot/scan
sensor_frame: dbot/sensor_laser
observation_persistence: 0.0
expected_update_rate: 5.0
data_type: LaserScan
clearing: true
marking: true
max_obstacle_height: 20.0
min_obstacle_height: -20.0
obstacle_range: 4.0
raytrace_range: 6.0
voxel_layer:
observation_sources: depth
depth:
topic: /dbot/kinect/depth/points
sensor_frame: dbot/kinect
observation_persistence: 2.0
expected_update_rate: 5.0
data_type: PointCloud2
clearing: true
43
marking: false
max_obstacle_height: 20.0
min_obstacle_height: -0.02
obstacle_range: 3.0
raytrace_range: 6.0
inflated_layer:
inflation_radius: 4.0
cost_scaling_factor: 3.0
This configuration file contains the details of the behaviour of the costmap, the details of the
sensor information sources and layers of the costmap.
8.8.2 Global Planner
For global planning, we use the global_planner package. The global_planner offers various
algorithms for path planning which can be configured using the global planner config params.
The lethat_cost, neutral_cost and cost_factor are important params which affect the final plan
and have to be tuned correctly.
global_planner_params.yaml
GlobalPlanner:
allow_unknown: true
default_tolerance: 0.2
visualize_potential: true
use_dijkstra: true
use_quadratic: true
use_grid_path: false
old_navfn_behavior: false
lethal_cost: 253
neutral_cost: 66
cost_factor: 0.55
publish_potential: true
orientation_mode: 1
outline_map: true
orientation_window_size: 1
44
8.8.3 Local Planner
The local planner produces velocity commands to send to a mobile base using a global plan to
follow and a costmap. There are several local planner packages available like
dwa_local_planner,
dwb_local_planner,
teb_local_planner,
etc
all
having
different
requirements and tradeoffs. Initially we used the dwa_local_planner which was fast and
required lower computation but struggled when the goal was directly opposite to the starting
orientation and in spot rotation. Later we switched to the teb_local_planner. The underlying
method called Timed Elastic Band locally optimizes the robot's trajectory with respect to
trajectory execution time, separation from obstacles and compliance with kinodynamic
constraints at runtime.
teb_base_local_planner_param.yaml
TebLocalPlannerROS:
odom_topic: /dbot/odom
map_frame: /map
# Trajectory
teb_autosize: True
dt_ref: 0.3
dt_hysteresis: 0.1
global_plan_overwrite_orientation: True
max_global_plan_lookahead_dist: 3.0
feasibility_check_no_poses: 5
# Robot
max_vel_x: 0.4
max_vel_x_backwards: 0.2
max_vel_theta: 0.3
acc_lim_x: 0.5
acc_lim_theta: 0.5
min_turning_radius: 0.0
footprint_model: # types: "point", "circular", "two_circles", "line",
"polygon"
type: "radius"
radius: 0.15 # for type "circular"
line_start: [-0.3, 0.0] # for type "line"
line_end: [0.3, 0.0] # for type "line"
front_offset: 0.2 # for type "two_circles"
front_radius: 0.2 # for type "two_circles"
rear_offset: 0.2 # for type "two_circles"
rear_radius: 0.2 # for type "two_circles"
45
vertices: [ [0.25, -0.05], [0.18, -0.05], [0.18, -0.18], [-0.19, 0.18], [-0.25, 0], [-0.19, 0.18], [0.18, 0.18], [0.18, 0.05], [0.25,
0.05] ] # for type "polygon"
# GoalTolerance
xy_goal_tolerance: 0.2
yaw_goal_tolerance: 0.1
free_goal_vel: False
# Obstacles
min_obstacle_dist: 0.3
include_costmap_obstacles: True
costmap_obstacles_behind_robot_dist: 1.0
obstacle_poses_affected: 30
costmap_converter_plugin: ""
costmap_converter_spin_thread: True
costmap_converter_rate: 5
# Optimization
no_inner_iterations: 5
no_outer_iterations: 4
optimization_activate: True
optimization_verbose: False
penalty_epsilon: 0.1
weight_max_vel_x: 2
weight_max_vel_theta: 1
weight_acc_lim_x: 1
weight_acc_lim_theta: 1
weight_kinematics_nh: 1000
weight_kinematics_forward_drive: 1
weight_kinematics_turning_radius: 1
weight_optimaltime: 1
weight_obstacle: 50
weight_dynamic_obstacle: 10 # not in use yet
alternative_time_cost: False # not in use yet
# Homotopy Class Planner
enable_homotopy_class_planning: True
enable_multithreading: True
simple_exploration: False
max_number_classes: 4
roadmap_graph_no_samples: 15
roadmap_graph_area_width: 5
h_signature_prescaler: 0.5
h_signature_threshold: 0.1
obstacle_keypoint_offset: 0.1
obstacle_heading_threshold: 0.45
46
visualize_hc_graph: False
8.9 Vision
The vision module uses object detection and optical character recognition to identify doors and
read number plates to verify before completing the delivery process. When the robot reaches
the marked point for the room on the floor, before delivering the package considering that there
is no human to receive the package, the robot will use the vision package to photograph from
the front camera and run number plate detection and door detection. If the robot correctly
detects the door and the number plate to be correct, it will execute the package delivery and
drop the package at the door. If any of the conditions are not met, the robot will go into
corrective routine which may have an exploration algorithm if the robot is on the correct floor,
a rerouting algorithm if the robot is on the incorrect floor and if all fails the robot can be put
into manual mode and be controlled by a remote human operator. Details of the implementation
of the vision package are given in the chapter 10.
Fig 8.19 Door detection and number plate recognition
47
8.10 Delivery
8.10.1 Local Planner
The delivery process is handled by the deliverybot_delivery package. This package uses the
maps created during the mapping process, the navigation and path planning functionality and
the delivery details to plan to deliver a package from the lobby of the building to the door of
the target apartment.
Fig 8.20 Delivery GUI
In this simulation the delivery GUI is used to input details of the delivery such as the building
name, room number and floor number. These details are sent to the delivery server where the
node creates the plan which is to be followed by the delivery robot. The planner gets the map
description file and uses all the marked locations along with the grid maps to create a global
navigation path.
48
Fig 8.21 Multi-floor navigation illustration
The planner also has to take into account the waiting for the elevators and verification process
during the delivery. Once the path is created the robot starts executing the plan. The plan is
essentially an array composed of actions and navigation goals. The execution of the plan is the
executing of all actions one by one. The different types of actions are:
1. ELEVATOR_DOOR_OPEN
2. ELEVATOR_DOOR_CLOSE
3. DELIVER_PACKAGE
4. CHANGE_FLOOR
5. CLEAR_COSTMAP
6. WAIT
7. MOVE
8. MAP
The elevator actions are used to communicate with the elevator and request the doors to be
opened or closed. The CHANGE_FLOOR action is also related to the elevator and requests
the elevator to go to a certain floor. The CLEAR_COSTMAP is an implementation requirement
49
for the simulation which clears the costmap on changing the floor if they are not automatically
cleared. The MOVE action sends navigation goals to different locations. The MAP action
changes the maps when the robot has to move from one mapped area to another. In the
DELIVER_PACKAGE action, the robot runs the verification process and executes a
predetermined sequence of actions to drop the package out of the robot and in front of the
apartment door.
Fig 8.22 Flow of delivery process
50
Chapter 9
Mechanical Design
9.1 Introduction
This chapter looks into the calculations for drive motors, component suggestion for the physical
prototype of the robot, concept sketches. Before starting the mechanical design some
assumptions and considerations which area taken are listed below:
1.
Robot is Differential Drive
2.
Maximum gradient is 6 degrees
3.
Gross Vehicle Weight with packages is 30 kg
4.
Maximum speed is 2 m/s and normal speed is 1.5 m/s (Human walking speed)
9.2 Motor Torque and Power Calculations
Motor torque calculations were done using a MATLAB script to allow tweaking of parameters.
The procedure is taken from the EML2322L Laboratory Document [22].
9.2.1 MATLAB script for torque calculations
51
Fig 9.1 MATLAB Script for Torque Requirement Estimation
52
9.2.2 Result for torque calculation
Fig 9.2 Torque Estimation Results
53
9.2.3 MATLAB script for motor power calculation
Fig 9.3 Matlab Script for motor power estimation
9.2.4 Result for motor power calculation
Fig 9.4 Motor Power Estimation Results
54
9.3 Components Suggestion
The processing board and sensors are components on which the performance of the robot
depends. Some components suggested for creating the physical prototype are listed below.
9.3.1 Processing Board
Name
Price
Description
Comparison Docs
Raspberry Pi 4
(8GB)
₹ 7,666.00
https://www.raspberrypi.org
/products/raspberry-pi-4model-b/specifications/
https://build5nines.com/rasp
berry-pi-4-vs-nvidia-jetsonnano-developer-kit/
Nvidia Jetson
Nano
₹
10,549.00
https://developer.nvidia.com
/embedded/jetson-nanodeveloper-kit
Table 9.1 Processing Board Options
9.3.2 Depth Camera
Name
Price
Image specifications
Intel RealSense
LiDAR Depth
Camera L515
₹
41,399.00
1. 30FPS Depth at
1024×768 (XGA)
2. 30FPS Color at
1920×1080 (FHD)
Intel RealSense
Depth Camera
D435i with IMU
₹
25,969.00
1. Depth Output
Approx. 10m
Resolution: up to 1280
x 720 active stereo
depth resolution
2. Frame Rate: Up to 90
fps
Intel D455
₹
1280 × 720 @ 30fps
RealSense Depth 29,649.00 848 × 480 @ 90fps
Camera
Intel RealSense
₹
FOV: Two Fisheye lenses
Tracking Camera 25,999.00 with combined 163±5°
T265
(Non Depth,
Feature Camera)
Table 9.2 Depth Camera Options
55
Depth Range
Type
0.25 to 9m
Lidar
Active
IR
Stereo
Recommended Active
Range:
IR
From 0.4 to 6 m Stereo
9.3.3 Laser Scanner
Name
Price
Range
Specifications
YDLIDAR X2 360 Degree
ROS Scanner for
Navigation, Collision
Avoidance
₹
7,999.00
0.1 8m
1. Scanning Frequency:7Hz
2. Ranging Frequency: 3000Hz
3. 360-degree two-dimensional
rangefinder
4. Adjustable motor speed(7Hz
recommend)
5. UART Communication at 3.3v
6. Laser power meets Class I
laser safety standards
YDLIDAR X4 360 Degree
ROS Scanner for
Navigation, Collision
Avoidance 10M
₹
10,999.00
0.1210 m
1. Small distance error, stable
performance, and high
accuracy
2. 10m Ranging distance
3. Strong resistance to ambient
light interference
4. Low power consumption,
small size, and long life
5. Laser power meets Class I
laser safety standards
6. Adjustable motor speed
frequency differ from
6Hz~12Hz
7. Ranging frequency up to 5K
Hz
RP LIDAR A2M8 360
₹
degrees Laser Range Finder- 29,499.00
12 Meter Range
18 m
1. 4cm Ultra-thin
2. Ideal for all kinds of the
service robot.
3. 8000 times sample rate, the
highest in the current
economical LIDAR industry
4. 360-degree laser range
scanning
5. Low Noise, Brushless Motor
New Non-contact Drive
6. 12m Range Radius
7. OPTMAG Original Design,
prolong the life-span
8. Class 1 Laser Safety Standard
9. High-speed RPVision Range
Engine: RPVision 2.0
10. Five years ultra-long life.
Table 9.3 Laser Scanner Options
56
9.4 Concept Design Sketches
Some concept design sketches are shown here
Fig 9.5 Components Packaging Concept Sketch
Fig 9.6 Shape Study
57
Fig 9.7 Concept Exploration
Fig 9.8 Packaging Concept
58
Chapter 10
Image Processing
10.1 Introduction
In order for the robot to be able to understand the tasks it is performing, it needs a real world
view. Since we aim to let the robot make its own decisions, such as avoiding obstacles,
navigating its environment, it needs to be able to visualize everything around it. If we were to
analyze the way the human eyes work, we can see how they are able to adapt to dynamic
changes in an instant, and aid the human brain in understanding its environment thus guiding
us in what we see and do. However, a robot is unable to do this on its own. Thus, one of the
main challenges in the robotics domain is to extend the visual ability to robots so that they can
also visualize their environment. The primary way to do this is with cameras.
A camera is a device which contains an image sensor, that captures pictures, and converts into
a digital format that can be interpreted by a computer. The use of cameras allows the robot to
collect visual data about its environment and by using mathematical processes/ algorithms, it
is able to generate an image which it understands, along with other relevant information such
as colors and edges. This is necessary, since what a human sees is interpreted from light
entering the eyes from different objects, a camera sees its environment as a collection of pixels,
each having 3 values, ranging from 0 to 255, pertaining to each of the color spectrum (Red,
Green and Blue).
59
In this chapter, we will go over the necessity of image detection and processing, then compare
various papers to decide the ideal algorithm to implement, then we discuss the working of
image processing in the delivery bot.
10.2 Necessity
Since our robot should be able to navigate itself from the buildings entrance, along with
overcoming various hurdles (elevators, ramps), avoid obstacles (both human and non-living)
and detect its environment in order to successfully deliver its package, we need to develop an
application that allows the robot to analyze the problem it is facing and find the optimum
solution in real time. Let's discuss the problems the robot will face in its operation.
10.2.1 Overcoming Objects/Obstacles
As the robot is not a sentient being, it is not able to distinguish various objects in its
environment. If we are to implement autonomous delivery robots, they should be able to
overcome challenges that it will face. In order for the robot to be able to distinguish between,
say, an elevator and a hallway plant, we have to train the robots to understand what they see.
This is the fundamental part of image processing, where we train models to be able to
distinguish various objects. In our application, we propose to implement models that allow the
robot to identify objects in its surroundings, and be able to react accordingly.
10.2.2 Door Detection
This is a subset of object detection, but it is an important part of the delivery process. The main
focus of this chapter is on image processing for door detection. While the robot is navigating
along its chosen route, it should be able to read the various door/apartment numbers and check
whether it matches that listed on the delivery order. Using its camera, the robot obtains the
digital image, and using image processing, it both checks whether the object visible is a door,
and if any alphanumeric characters are visible. By further fine tuning the model, we are able to
optimize the character recognition by the process of elimination. The process we have
implemented in our robot is similar to that used in Automatic Number Plate Recognition
(ANPR), which uses trained models and image processing to capture a vehicle’s number plate
for further action.
60
Fig 10.1 Door and apartment number detection in Python
Fig 10.2 Door and apartment number detection implementation in ROS
10.3 Working
As covered in previous sections, the robot can successfully enter and navigate the building.
However, in order to deliver the product to the recipient, robot should be able to know where
61
the recipient's location is. Primary locations where image capture and processing is required
are: Ramps and stairs, Elevator and its buttons, building signage, and door/apartment numbers.
We first begin with image capture. We propose installation of the camera on the front of the
robot, which gives it the required field of view. As a summary, we have chosen to use Intel
RealSense Depth Camera D435i, which has a 3um x 3um image sensor, and is able to output a
resolution of 1280x720, while also being able to use Inertial Measurement Unit to map out its
surroundings and know the relative position of objects with respect to the location of the robot.
The camera also allows depth perception, allowing the robot to detect objects as far as 3 meters.
The camera is able to output images continuously, up to 90fps, thus allowing for smoother
output for image recognition and processing.
Once the image is captured by the camera, it is outputted to the computer, where it undergoes
image processing. From the literature review, we have chosen to implement the YOLOv3[23]
algorithm, which is a real-time object detection algorithm, particularly for constant video or
images. We have also chosen to use predefined classes and weights which had been trained to
detect doors by MiguelARD [24][25]. We have gone this route as creating custom classes and
weights requires a lot of compute power and is out of the scope of this project. The algorithm
is able to convert the video into still images, which undergoes scaling, resizing and cropping
to obtain images with high confidence scores. It then undergoes post-processing, where it uses
the vectors obtained to draw bounding boxes around the objects detected, and further label
them as given in the class.
The output obtained are still images which have every object present bounded by a label box.
This gives the computer information as to what objects it sees and is interacting with in its
environment.
Next we move on to character recognition. While in some cases, the robot may only require to
know what objects are in its environment, it also needs to be able to read any alphanumeric
text, either as signage or door numbers. Using the OpenCV library, we implement Optical
Character Recognition(OCR), which takes the image obtained with labelled objects for further
processing. Here, the program first filters the image for noise. Using edge detection, it
generates each edge present in the object, which will aid in contour detection. The program
also applies a mask to the image to further ease detection in the forthcoming steps. The images
generated are passed to the OCR library, using the set default language of the text, detects any
alphanumeric text is present in the image. The final output is the original image along with a
62
bounding box with labels specifying the type of object detected and the text on the object, if
applicable. The source code for door and door number detection is available on github[26].
63
Chapter 11
Results
We successfully simulated the delivery robot system for delivering packages to the doorstep of
customers who live in apartment buildings in Gazebo and a video of this demo can be viewed
on YouTube [27]. The robot was able to navigate around obstacles, ride in elevators, and
deliver the package to the correct room. The mapping of the simulated worlds was done using
the mapping GUI we created before running deliveries. Once the map was created the robot
was ready to perform deliveries. The deliveries were executed using the Delivery GUI. The
delivery process used the maps created during mapping for path planning and navigation. The
source code of the project is openly available on Github[28]. Screenshots of the simulation at
various stages can be seen in the series of images below.
64
Fig 11.1 Mapping a building
Fig 11.2 DeliveryBot dropped at building for delivery
65
Fig 11.3 DeliveryBot waiting for elevator
Fig 11.4 DeliveryBot dropping package at door
66
Fig 11.5 DeliveryBot waiting to be picked up after delivering package
67
Chapter 12
Conclusion
12.1 Conclusion
In this project we tackled the limitation of delivery robots which cannot deliver to the
customer’s door in residential buildings with an important assumption that the robot has some
way of communicating with the elevator in the building which can be either hardware or
software. We created and simulated a robot capable of multi-floor navigation and package
delivery. To do so we implemented a multi-floor mapping and navigation method using 2D
grid maps in ROS and simulated the whole system in Gazebo.
Delivery robots capable of delivering packages to the door of customers living in residential
buildings will help to bridge the final gap in the last mile delivery process and help in achieving
automation of the complete process from warehouse to the customer doorstep. Along with ecommerce site deliveries, these robots can also be used by local businesses like restaurants,
medical stores, grocery stores, etc. Automation of this process can help reduce the delivery
costs to both the vendors and the customers. It will also help to make the process more efficient
and reduce delivery times allowing faster deliveries. These robots can even help reduce traffic
on the streets by completing deliveries during the night time.
68
12.2 Future Work
There are several areas of this project that can be improved either mainly to make the methods
more general or improve performance.
1. The current planning process has a fixed structure for the plan created that is the drop
point, elevator and the apartment door. In the real world the robot may have to go
through more waypoints to reach the elevator or door. The mapping and planning
process can be generalized to provide more flexibility in the choice of waypoints.
2. We assume there is some means for the robot to communicate with the elevator. This
can be hardware based i.e. by using a manipulator to push the elevator buttons or
software based as demonstrated in the project where the robot can wirelessly
communicate with the elevator. More work can be done in this area to build these
systems. The software method is preferred as it will allow simpler designs for the robots
and better communication.
3. We suggest robots containing the packages will be dropped off at the building by
autonomous vans for the delivery. These robots can be operated in fleets to maximize
their impact and methods have to be developed to optimize the paths and assignment of
the autonomous vans. This is a suggested area for future work.
69
Chapter 13
References
Weblink,
[1] Last Mile Delivery Market to 2027 - Global Analysis and Forecasts by Technology
[Online]. Available: https://www.theinsightpartners.com/reports/last-mile-delivery-market
(April 05 2021)
[2] Report on Last Mile delivery challenge [Online]. Available: https://www.capgemini.com/
research/the-last-mile-delivery-challenge. (April 05, 2021)
[3] Robot Operating System (ROS) [Online]. Available: https://www.ros.org/about-ros/
[4] Gazebo Sim [Online]. Available: http://gazebosim.org/
[5] ROS Navigation Stack [Online]. Available: http://wiki.ros.org/navigation
[7] DWA Local Planner. [Online]. Available: http://wiki.ros.org/dwa_local_planner
[8] TEB Local Planner [Online]. Available: http://wiki.ros.org/teb_local_planner
[10] TurtleBot Product Line [Online]. Available: http://wiki.ros.org/Robots/TurtleBot
[13] Savioke Website [Online]. Available: https://www.savioke.com/
[14] Starship Website [Online]. Available: https://www.starship.xyz/
[15] Kiwibot Website [Online]. Available: https:// www.kiwibot.com/
[18] Gmapping implementation for ROS. Available: http://wiki.ros.org/gmapping
[19] Wikipedia entry for Computer Vision [Online] . Available: https://en.wikipedia.org/
wiki/Computer_vision
Book,
[6] Kaiyu Zheng, “ROS Navigation Tuning Guide” in Robot Operating System (ROS) The
Complete Reference, Volume 6, 2016/09, Springer International Publishing
[16] Giorgio Grisetti, Cyrill Stachniss, and Wolfram Burgard: Improved Techniques for Grid
Mapping with Rao-Blackwellized Particle Filters, IEEE Transactions on Robotics, Volume 23,
pages 34-46, 2007
[17] Giorgio Grisetti, Cyrill Stachniss, and Wolfram Burgard: Improving Grid-based SLAM
with Rao-Blackwellized Particle Filters by Adaptive Proposals and Selective Resampling, In
Proc. of the IEEE International Conference on Robotics and Automation (ICRA), 2005
70
Conferences and Conference Proceedings,
[9] Ellingson, J., & Nepal, K., & McGill, M. R., & Hoffmann, M. J. (2013, June), Multi-floor
Mapping and Navigation with Uncertainty Paper presented at 2013 ASEE Annual Conference
& Exposition, Atlanta, Georgia. 10.18260/1-2--22302
[11] Sami, Muhammad, et al. "Text detection and recognition for semantic mapping in indoor
navigation." 2015 5th International Conference on IT Convergence and Security (ICITCS).
IEEE, 2015.
[12] Chae, Hee-Won, et al. "Object recognition for SLAM in floor environments using a depth
sensor." 2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence
(URAI). IEEE, 2016.
[23] Joseph Redmon & Ali Farhadi, “YOLOv3: An Incremental Improvement” arXiv:
1804.02767 [cs.CV], 2018.
[24] Miguel Arduengo, & Carme Torras, & Luis Sentis, “Robust and Adaptive Door Operation
with a Mobile Robot” arXiv: 1902.09051[cs.RO], 2020.
Lecture Notes,
[22] EML2322L – MAE Design and Manufacturing LaboratoryDrive Wheel Motor Torque
Calculations [PDF]. Available: https://mae.ufl.edu/designlab/motors/
EML2322L%20Drive%20Wheel%20Motor%20Torque%20Calculations.pdf
Software
[20] OnShape, PTC Inc. Available: https://www.onshape.com/en/
[21] Blender 2.9, The Blender Foundation. Available: https://www.blender.org/
[24] Door Detection Dataset [Online]. Available: https://github.com/MiguelARD/DoorDetectDataset
[26] Door_Number_Detector [Online]. Available: https://github.com/Hamdrive/
Door_Number_Detector
[28] DeliveryBot, A delivery robot with multi-floor navigation capabilities [Online]. Commit:
2862f904d97e2f724ae5b7cc773dbe84bcb0ff86. Available: https://github.com/ammaar8/
DeliveryBot
Online Video,
[27] Ammaar Solkar, DeliveryBot Demo Video | ROS (Apr. 26, 2021). Accessed: May 2, 2021
[Online Video]. Available: https://youtu.be/9i83VW3t_Xk
Patents,
1. Adrian Canoso et al., inventor; Jan. 03, 2017, Mobile delivery robot with interior cargo
space, US patent no. 9535421B1.
2. Ahti Heinla et al., inventor; Aug. 16, 2018, Method and system for autonomous or semiautonomous delivery, US patent no. US20180232839A1.
71
Appendix - I
This appendix provides the code files for some important functions in the ROS
packages used in simulating this project. The source code is available on Github[27]. The
codebase is divided into three parts namely,
1. Deliverybot
2. Elevator
3. Simulation
The deliverybot metapackage contains the packages required for the working of the delivery
robot. It contains the packages for control, delivery, description, gui, mapping and navigation.
The elevator metapackage contains the packages required for the working of the elevator
created for running simulations. It contains the description, control and gui packages.
The simulations metapackage contains the delvierybot_simulations package which is used for
running all the gazebo simulations.
Only important files which contain core functionality are shown here to maintain brevity.
1. Deliverybot
1.1 deliverybot_control
scripts/deliverybot_hw_controller.py
#!/usr/bin/env python3
import rospy
import sys
from std_msgs.msg import Float64
from sensor_msgs.msg import JointState
from std_srvs.srv import Empty, EmptyResponse
global ns
ns = rospy.get_namespace()
print("Controller started with ns " + ns)
DOOR_OPEN = 0.51
DOOR_CLOSED = -1.57075
PUSHER_OUT = 0.24
72
PUSHER_IN = 0.0
TOLERANCE_DOOR = 0.05
TOLERANCE_PUSHER = 0.01
door_pub = rospy.Publisher(
rospy.names.ns_join(ns, "door_position_controller/command"),
Float64,
queue_size=10
)
pusher_pub = rospy.Publisher(
rospy.names.ns_join(ns, "pusher_position_controller/command"),
Float64,
queue_size=10
)
def open_bot_door(req):
rospy.loginfo("Opening Bot Door")
door_pub.publish(DOOR_OPEN)
return EmptyResponse()
def close_bot_door(req):
rospy.loginfo("Closing Bot Door")
door_pub.publish(DOOR_CLOSED)
return EmptyResponse()
def pusher_out(req):
rospy.loginfo("Pushing Package Out")
pusher_pub.publish(PUSHER_OUT)
return EmptyResponse()
def pusher_in(req):
rospy.loginfo("Retracting Pusher")
pusher_pub.publish(PUSHER_IN)
return EmptyResponse()
def deliver_package(req):
global DOOR_CLOSED, DOOR_OPEN, PUSHER_IN, PUSHER_IN,
TOLERANCE_DOOR,TOLERANCE_PUSHER
DOOR_STATE = None
PUSHER_STATE = None
def check_door():
msg = rospy.wait_for_message(rospy.names.ns_join(ns,
'joint_states'), JointState)
DOOR_STATE = msg.position[0]
return DOOR_STATE
73
def check_pusher():
msg = rospy.wait_for_message(rospy.names.ns_join(ns,
'joint_states'), JointState)
PUSHER_STATE = msg.position[1]
return PUSHER_STATE
DOOR_STATE = check_door()
PUSHER_STATE = check_pusher()
while DOOR_STATE < DOOR_OPEN - TOLERANCE_DOOR:
DOOR_STATE = check_door()
open_bot_door(req)
while PUSHER_STATE < PUSHER_OUT - TOLERANCE_PUSHER:
PUSHER_STATE = check_pusher()
pusher_out(req)
while PUSHER_STATE > PUSHER_IN + TOLERANCE_PUSHER:
PUSHER_STATE = check_pusher()
pusher_in(req)
while DOOR_STATE > DOOR_CLOSED + TOLERANCE_DOOR:
DOOR_STATE = check_door()
close_bot_door(req)
rospy.loginfo("Package Drop Complete")
return EmptyResponse()
if __name__ == "__main__":
rospy.init_node("deliverybot_hw_controller")
rospy.Service('close_bot_door', Empty, close_bot_door)
rospy.Service('open_bot_door', Empty, open_bot_door)
rospy.Service('set_pusher_out', Empty, pusher_out)
rospy.Service('set_pusher_in', Empty, pusher_in)
rospy.Service('deliver_package', Empty, deliver_package)
rospy.spin()
1.2 deliverybot_delivery
scripts/delivery_client.py
#!/usr/bin/env python3
import actionlib
import deliverybot_navigation.msg
import tkinter as tk
import rospy
class Application(tk.Frame):
def __init__(self, master=None):
74
tk.Frame.__init__(self, master)
self.client = actionlib.SimpleActionClient(
'delivery_server',
deliverybot_navigation.msg.DeliverAction
)
self.client.wait_for_server()
self.master = master
self.pack()
self.create_widgets()
def create_widgets(self):
widget_frame = tk.LabelFrame(self, text="Delivery GUI")
widget_frame.grid_columnconfigure(0, weight=1)
widget_frame.grid_rowconfigure(0, weight=1)
widget_frame.pack(fill='x', pady=5)
building_name_var = tk.StringVar()
floor_var = tk.StringVar()
room_var = tk.StringVar()
self.status_label = tk.Label(widget_frame, text="Idle")
self.status_label.grid(row=0, column=0, columnspan=2,
sticky="ew")
building_name_label = tk.Label(widget_frame, text="Building
Name")
building_name_label.grid(row=1, column=0, sticky="ew")
building_name_entry = tk.Entry(widget_frame,
textvariable=building_name_var)
building_name_entry.grid(row=1, column=1, sticky="ew")
floor_label = tk.Label(widget_frame, text="Floor")
floor_label.grid(row=2, column=0, sticky="ew")
floor_entry = tk.Entry(widget_frame, textvariable=floor_var)
floor_entry.grid(row=2, column=1, sticky="ew")
room_label = tk.Label(widget_frame, text="Room")
room_label.grid(row=3, column=0, sticky="ew")
room_entry = tk.Entry(widget_frame, textvariable=room_var)
room_entry.grid(row=3, column=1, sticky="ew")
self.send_goal_button = tk.Button(
widget_frame,
text="Deliver",
command=lambda: self.send_delivery_goal(
building_name_var.get(),
int(floor_var.get()),
int(room_var.get())
)
)
self.send_goal_button.grid(row=4, column=0, sticky="ew")
def reset():
building_name_entry.delete(0, tk.END)
floor_entry.delete(0, tk.END)
75
room_entry.delete(0, tk.END)
reset_btn = tk.Button(widget_frame, text="Reset", command =
reset)
reset_btn.grid(row=4, column=1, sticky="ew")
def send_delivery_goal(self, building_name, floor, room):
self.client.wait_for_server()
goal = deliverybot_navigation.msg.DeliverGoal(
building_name,
floor,
room)
self.client.send_goal(goal)
self.status_label.config(text="Running Delivery")
self.send_goal_button["state"] = "disabled"
self.master.update()
self.client.wait_for_result()
self.send_goal_button["state"] = "normal"
self.status_label.config(text="Idle")
if __name__ == "__main__":
rospy.init_node("delivery_client_gui")
root = tk.Tk()
root.title("Delivery GUI")
root.resizable(False, False)
app = Application(master=root)
app.mainloop()
scripts/delivery_server.py
#!/usr/bin/env python
import rospy
import sys
import os
import yaml
import tf_conversions
import actionlib
import rosparam
from std_srvs.srv import Empty
from elevator_controls.srv import ElevatorFloorGoal
from move_base_msgs.msg import MoveBaseAction, MoveBaseGoal
from deliverybot_navigation.srv import MapFilePath
import deliverybot_navigation.msg
class DeliveryServer(object):
76
_feedback = deliverybot_navigation.msg.DeliverFeedback()
_result = deliverybot_navigation.msg.DeliverResult()
def __init__(self, name):
self._action_name = name
self._as = actionlib.SimpleActionServer(
self._action_name,
deliverybot_navigation.msg.DeliverAction,
execute_cb = self.execute_cb,
auto_start = False
)
self.dbot_deliver =
rospy.ServiceProxy('/dbot/deliver_package', Empty)
self.change_map = rospy.ServiceProxy('/load_map', MapFilePath)
self.el_open_door =
rospy.ServiceProxy('/elevator/open_elevator_doors', Empty)
self.el_close_door =
rospy.ServiceProxy('/elevator/close_elevator_doors', Empty)
self.el_change_floor =
rospy.ServiceProxy('/elevator/elevator_goto_floor',
ElevatorFloorGoal)
self.clear_costmap =
rospy.ServiceProxy('/move_base/clear_costmaps', Empty)
self._as.start()
rospy.loginfo("Delivery server started")
def move(self, x, y, a):
client = actionlib.SimpleActionClient('move_base',
MoveBaseAction)
client.wait_for_server()
goal = MoveBaseGoal()
goal.target_pose.header.frame_id = "map"
goal.target_pose.header.stamp = rospy.Time.now()
goal.target_pose.pose.position.x = x
goal.target_pose.pose.position.y = y
goal.target_pose.pose.position.z = 0.0
quat = tf_conversions.transformations.quaternion_from_euler(
0.0,
0.0,
a
)
goal.target_pose.pose.orientation.x
goal.target_pose.pose.orientation.y
goal.target_pose.pose.orientation.z
goal.target_pose.pose.orientation.w
77
=
=
=
=
quat[0]
quat[1]
quat[2]
quat[3]
client.send_goal(goal)
wait = client.wait_for_result()
rospy.loginfo("Sent Goal")
if not wait:
rospy.logerr("Action server not available!")
rospy.signal_shutdown("Action server not available!")
else:
return client.get_result()
def load_map(self, map_path):
rospy.loginfo("Waiting for service /load_map")
self.change_map.wait_for_service()
self.change_map.call(map_path)
rospy.loginfo("Map Changed.")
def execute_action(self, action):
# action = (type: string, value: object)
if action[0] == "ELEVATOR_DOOR_OPEN":
result = self.el_open_door.call()
elif action[0] == "ELEVATOR_DOOR_CLOSE":
result = self.el_close_door.call()
elif action[0] == "DELIVER_PACKAGE":
result = self.dbot_deliver.call()
elif action[0] == "CHANGE_FLOOR":
result = self.el_change_floor.call(action[1])
elif action[0] == "CLEAR_COSTMAP":
result = self.clear_costmap.call()
elif action[0] == "WAIT":
pass
else:
rospy.logwarn("Action type " + str(action[0]) + " not
recognized.")
return result
def load_building_description(self, building_name):
# TODO - replace map folder with DB
map_folder =
os.path.abspath(os.environ.get('MAPS_FOLDER_PATH'))
with open(os.path.join(map_folder, building_name,
'building.yaml')) as f:
yaml_desc = yaml.safe_load(f)
return yaml_desc
def generate_plan(self, goal):
path = []
rospy.loginfo("Generating Plan")
building_desc =
self.load_building_description(goal.building_name)
# load lobby map
path.append((
78
"MAP",
(
os.path.abspath(os.path.join(os.environ.get('MAPS_FOLD
ER_PATH'), goal.building_name, 'lobby.yaml'))
)
))
# elevator sequence
path.append((
"MOVE",
(
building_desc["elevator"]["out"]["x"],
building_desc["elevator"]["out"]["y"],
building_desc["elevator"]["out"]["a"]
)
))
path.append((
"ACTION",
(
"CHANGE_FLOOR",
0
)
))
path.append((
"ACTION",
(
"ELEVATOR_DOOR_OPEN",
)
))
path.append((
"MOVE",
(
building_desc["elevator"]["in"]["x"],
building_desc["elevator"]["in"]["y"],
building_desc["elevator"]["in"]["a"]
)
))
path.append((
"ACTION",
(
"ELEVATOR_DOOR_CLOSE",
)
))
path.append((
"ACTION",
(
"CHANGE_FLOOR",
goal.floor
)
))
path.append((
"ACTION",
(
79
"ELEVATOR_DOOR_OPEN",
)
))
path.append((
"MAP",
(
os.path.abspath(os.path.join(os.environ.get('MAPS_FOLD
ER_PATH'), goal.building_name, 'floor.yaml'))
)
))
path.append((
"MOVE",
(
building_desc["elevator"]["out"]["x"],
building_desc["elevator"]["out"]["y"],
-building_desc["elevator"]["out"]["a"]
)
))
path.append((
"ACTION",
(
"ELEVATOR_DOOR_CLOSE",
)
))
path.append((
"MOVE",
(
building_desc["rooms"][goal.room]["x"],
building_desc["rooms"][goal.room]["y"],
building_desc["rooms"][goal.room]["a"]
)
))
path.append((
"ACTION",
(
"DELIVER_PACKAGE",
)
))
path.append((
"MOVE",
(
building_desc["elevator"]["out"]["x"],
building_desc["elevator"]["out"]["y"],
building_desc["elevator"]["out"]["a"]
)
))
path.append((
"ACTION",
(
"CHANGE_FLOOR",
goal.floor
)
80
))
path.append((
"ACTION",
(
"ELEVATOR_DOOR_OPEN",
)
))
path.append((
"ACTION",
(
"CLEAR_COSTMAP",
)
))
path.append((
"MOVE",
(
building_desc["elevator"]["in"]["x"],
building_desc["elevator"]["in"]["y"],
building_desc["elevator"]["in"]["a"]
)
))
path.append((
"ACTION",
(
"ELEVATOR_DOOR_CLOSE",
)
))
path.append((
"ACTION",
(
"CHANGE_FLOOR",
0
)
))
path.append((
"ACTION",
(
"ELEVATOR_DOOR_OPEN",
)
))
path.append((
"MAP",
(
os.path.abspath(os.path.join(os.environ.get('MAPS_FOLD
ER_PATH'), goal.building_name, 'lobby.yaml'))
)
))
path.append((
"MOVE",
(
building_desc["elevator"]["out"]["x"],
building_desc["elevator"]["out"]["y"],
81
-building_desc["elevator"]["out"]["a"]
)
))
path.append((
"ACTION",
(
"ELEVATOR_DOOR_CLOSE",
)
))
path.append((
"MOVE",
(
building_desc["lobby"]["pickup"]["x"],
building_desc["lobby"]["pickup"]["y"],
building_desc["lobby"]["pickup"]["a"]
)
))
rospy.loginfo("Completed Generating Plan")
return path
def execute_cb(self, goal):
rospy.loginfo("Goal Received.")
success = True
plan = self.generate_plan(goal)
for action_type, action_value in plan:
if action_type == "MAP":
self.load_map(action_value)
elif action_type == "ACTION":
self.execute_action(action_value)
elif action_type == "MOVE":
self.move(*action_value)
else:
rospy.logerr("Unknown action type found in plan.")
if success:
rospy.loginfo("Successfully delivered.")
self._as.set_succeeded(self._result)
if __name__ == "__main__":
rospy.init_node("delivery_server")
server = DeliveryServer(rospy.get_name())
rospy.spin()
1.3 deliverybot_mapping
scripts/mapping_gui.py
#!/usr/bin/env python
82
import
import
import
import
import
import
import
import
import
import
Tkinter as tk
rospkg
roslaunch
rospy
os
geometry_msgs.msg
tf2_ros
tf_conversions
yaml
subprocess
class MappingGUI(tk.Frame):
def __init__(self, master=None):
tk.Frame.__init__(self, master=master)
self.master = master
self.BUILDING_NAME = None
self.map_server_node = None
self.rospack = rospkg.RosPack()
self.tfBuffer = tf2_ros.Buffer()
self.tfListener = tf2_ros.TransformListener(self.tfBuffer)
self.PKG_DIR = self.rospack.get_path('deliverybot_mapping')
self.MAPS_DIR = os.path.join(self.PKG_DIR, 'maps')
self.LAUNCH_DIR = os.path.join(self.PKG_DIR, 'launch')
self.map = {
"building": {
"name": None,
"scheme": None,
"floors": None,
},
"lobby":{
"pickup": None,
"drop": None,
},
"elevator":{
"in": None,
"out": None,
},
"rooms":{
}
}
self.pack(expand=True, fill="both")
self.create_widgets()
def save_yaml(self):
83
with open(os.path.join(self.MAPS_DIR, self.BUILDING_NAME,
'building.yaml'), 'w') as f:
yaml.dump(self.map, f)
def connect_map(self, building_name):
self.BUILDING_NAME = building_name
self.map["building"]["name"] = building_name
if os.path.isdir(os.path.join(self.MAPS_DIR, building_name)):
with open(os.path.join(self.MAPS_DIR, building_name,
'building.yaml'), 'r') as f:
self.map = yaml.safe_load(f)
rospy.loginfo("Building " + building_name + "
loaded.")
else:
os.mkdir(os.path.join(self.MAPS_DIR, building_name))
with open(os.path.join(self.MAPS_DIR, building_name,
'building.yaml'), 'w') as f:
yaml.dump(self.map, f)
rospy.loginfo("Building " + building_name + "
created.")
def start_map_server(self):
uuid = roslaunch.rlutil.get_or_generate_uuid(None, False)
roslaunch.configure_logging(uuid)
self.map_server_node = roslaunch.parent.ROSLaunchParent(
uuid,
[
os.path.join(self.LAUNCH_DIR, "map_server.launch")
]
)
self.map_server_node.start()
rospy.loginfo("Map Server started")
def kill_map_server(self):
self.map_server_node.shutdown()
self.map_server_node = None
rospy.loginfo("Map Server shutdown")
def save_map_lobby(self):
uuid = roslaunch.rlutil.get_or_generate_uuid(None, False)
roslaunch.configure_logging(uuid)
cli_args = [
os.path.join(self.LAUNCH_DIR, "map_saver_lobby.launch"),
str("location:=" + os.path.join(self.MAPS_DIR,
self.BUILDING_NAME))
]
roslaunch_args = cli_args[1:]
84
roslaunch_file = [
(
roslaunch.rlutil.resolve_launch_arguments(cli_args)[0],
roslaunch_args
)
]
map_saver_lobby = roslaunch.parent.ROSLaunchParent(
uuid,
roslaunch_file
)
map_saver_lobby.start()
rospy.loginfo("Lobby map saved.")
def save_map_floor(self):
uuid = roslaunch.rlutil.get_or_generate_uuid(None, False)
roslaunch.configure_logging(uuid)
cli_args = [
os.path.join(self.LAUNCH_DIR, "map_saver_floor.launch"),
str("location:=" + os.path.join(self.MAPS_DIR,
self.BUILDING_NAME))
]
roslaunch_args = cli_args[1:]
roslaunch_file = [
(
roslaunch.rlutil.resolve_launch_arguments(cli_args)[0],
roslaunch_args
)
]
map_saver_lobby = roslaunch.parent.ROSLaunchParent(
uuid,
roslaunch_file
)
map_saver_lobby.start()
rospy.loginfo("Floor map saved.")
def mark_pickup(self):
print("Pickup Marked")
trans = self.tfBuffer.lookup_transform(
'map',
'dbot/base_link',
rospy.Time()
)
self.map["lobby"]["pickup"] = {
"x": trans.transform.translation.x,
"y": trans.transform.translation.y,
"a": tf_conversions.transformations.euler_from_quaternion(
[
trans.transform.rotation.x,
trans.transform.rotation.y,
85
trans.transform.rotation.z,
trans.transform.rotation.w,
]
)[2]
}
self.save_yaml()
def mark_drop(self):
print("Drop Marked")
trans = self.tfBuffer.lookup_transform(
'map',
'dbot/base_link',
rospy.Time()
)
self.map["lobby"]["drop"] = {
"x": trans.transform.translation.x,
"y": trans.transform.translation.y,
"a": tf_conversions.transformations.euler_from_quaternion(
[
trans.transform.rotation.x,
trans.transform.rotation.y,
trans.transform.rotation.z,
trans.transform.rotation.w,
]
)[2]
}
self.save_yaml()
def mark_elevator_in(self):
print("Elevator In Marked")
trans = self.tfBuffer.lookup_transform(
'map',
'dbot/base_link',
rospy.Time()
)
self.map["elevator"]["in"] = {
"x": trans.transform.translation.x,
"y": trans.transform.translation.y,
"a": tf_conversions.transformations.euler_from_quaternion(
[
trans.transform.rotation.x,
trans.transform.rotation.y,
trans.transform.rotation.z,
trans.transform.rotation.w,
]
)[2]
}
self.save_yaml()
86
def mark_elevator_out(self):
print("Elevator Out Marked")
trans = self.tfBuffer.lookup_transform(
'map',
'dbot/base_link',
rospy.Time()
)
self.map["elevator"]["out"] = {
"x": trans.transform.translation.x,
"y": trans.transform.translation.y,
"a": tf_conversions.transformations.euler_from_quaternion(
[
trans.transform.rotation.x,
trans.transform.rotation.y,
trans.transform.rotation.z,
trans.transform.rotation.w,
]
)[2]
}
self.save_yaml()
def add_room_coods(self, room_number):
print("Marked Room", room_number)
trans = self.tfBuffer.lookup_transform(
'map',
'dbot/base_link',
rospy.Time()
)
self.map["rooms"][int(room_number)] = {
"x": trans.transform.translation.x,
"y": trans.transform.translation.y,
"a": tf_conversions.transformations.euler_from_quaternion(
[
trans.transform.rotation.x,
trans.transform.rotation.y,
trans.transform.rotation.z,
trans.transform.rotation.w,
]
)[2]
}
self.save_yaml()
def add_floor_count(self, count):
print("Floor Count Added", count)
trans = self.tfBuffer.lookup_transform(
'map',
'dbot/base_link',
rospy.Time()
87
)
self.map["building"]["floors"] = int(count)
self.save_yaml()
def add_naming_scheme(self, scheme):
print("Floor Count Added", scheme)
trans = self.tfBuffer.lookup_transform(
'map',
'dbot/base_link',
rospy.Time()
)
self.map["building"]["scheme"] = scheme
self.save_yaml()
def create_widgets(self):
self.var_room_number = tk.StringVar()
self.var_building_name = tk.StringVar()
self.var_wing_name = tk.StringVar()
self.var_floors = tk.StringVar()
self.var_scheme = tk.StringVar()
def add_room_coods_helper():
if (self.var_room_number.get() != ""):
self.add_room_coods(self.var_room_number.get())
room_entry.delete(0, tk.END)
mapping_frame = tk.LabelFrame(self, text="Mapping")
mapping_frame.grid_columnconfigure(0, weight=1,
uniform="something")
mapping_frame.grid_rowconfigure(0, weight=1,
uniform="something")
mapping_frame.pack(side="top", expand=True, fill="both")
#1
building_name_label = tk.Label(mapping_frame, text="Bldg Name")
building_name_label.grid(column=0, row=0, sticky="ew")
#2
building_name_entry = tk.Entry(mapping_frame, bd=2,
textvariable=self.var_building_name)
building_name_entry.grid(column=1, row=0, sticky="ew")
#3
building_name_btn = tk.Button(mapping_frame, text="Create",
command=lambda: self.connect_map(self.var_building_name.get()))
building_name_btn.grid(column=2, row=0, sticky="EW")
#4
label_server = tk.Label(mapping_frame, text="Server")
label_server.grid(column=0, row=1, sticky="EW")
#5
88
server_start_btn = tk.Button(mapping_frame, text="Start",
command=self.start_map_server)
server_start_btn.grid(column=1, row=1, sticky="EW")
#6
server_kill_btn = tk.Button(mapping_frame, text="Clear",
command=self.kill_map_server)
server_kill_btn.grid(column=2, row=1, sticky="EW")
#7
label_area = tk.Label(mapping_frame, text="Area")
label_area.grid(column=0, row=2, sticky="EW")
#8
lobby_btn = tk.Button(mapping_frame, text="Lobby",
command=self.save_map_lobby)
lobby_btn.grid(column=1, row=2, sticky="EW")
#9
floor_btn = tk.Button(mapping_frame, text="Floor",
command=self.save_map_floor)
floor_btn.grid(column=2, row=2, sticky="EW")
#10
label_elevator = tk.Label(mapping_frame, text="Elevator")
label_elevator.grid(column=0, row=3, sticky="EW")
#11
elevator_out_btn = tk.Button(mapping_frame, text="OUT",
command=self.mark_elevator_out)
elevator_out_btn.grid(column=1, row=3, sticky="EW")
#12
elevator_in_btn = tk.Button(mapping_frame, text="IN",
command=self.mark_elevator_in)
elevator_in_btn.grid(column=2, row=3, sticky="EW")
#13
label_pickup = tk.Label(mapping_frame, text="Lobby")
label_pickup.grid(column=0, row=4, sticky="EW")
#14
drop_btn = tk.Button(mapping_frame, text="Mark Drop",
command=self.mark_drop)
drop_btn.grid(column=2, row=4, sticky="EW")
#15
pickup_btn = tk.Button(mapping_frame, text="Mark Pickup",
command=self.mark_pickup)
pickup_btn.grid(column=1, row=4, sticky="EW")
#16
label_room = tk.Label(mapping_frame, text="Room")
label_room.grid(column=0, row=5, sticky="EW")
#17
room_entry = tk.Entry(mapping_frame, bd=2,
textvariable=self.var_room_number, width=4)
room_entry.grid(column=1, row=5, sticky="EW")
#18
mark_room_btn = tk.Button(mapping_frame, text="Mark",
command=add_room_coods_helper)
mark_room_btn.grid(column=2, row=5, sticky="EW")
#19
89
building_floors_label = tk.Label(mapping_frame, text="Floors")
building_floors_label.grid(column=0, row=6, sticky="ew")
#20
building_floors_entry = tk.Entry(mapping_frame, bd=2,
textvariable=self.var_floors)
building_floors_entry.grid(column=1, row=6, sticky="ew")
#21
building_floors_btn = tk.Button(mapping_frame, text="Save",
command=lambda: self.add_floor_count(self.var_floors.get()))
building_floors_btn.grid(column=2, row=6, sticky="EW")
#22
building_scheme_label = tk.Label(mapping_frame, text="Scheme")
building_scheme_label.grid(column=0, row=7, sticky="ew")
#23
building_scheme_entry = tk.Entry(mapping_frame, bd=2,
textvariable=self.var_scheme)
building_scheme_entry.grid(column=1, row=7, sticky="ew")
#24
building_scheme_btn = tk.Button(mapping_frame, text="Save",
command=lambda: self.add_naming_scheme(self.var_scheme.get()))
building_scheme_btn.grid(column=2, row=7, sticky="ew")
if __name__ == "__main__":
rospy.init_node("dbot_mapping", anonymous=True)
root = tk.Tk()
root.resizable(False, False)
app = MappingGUI(master=root)
tk.mainloop()
1.4 deliverybot_naivgation
launch/amcl.launch
<launch>
<!-- Arguments -->
<arg name="scan_topic"
<arg name="initial_pose_x"
<arg name="initial_pose_y"
<arg name="initial_pose_a"
default="/dbot/scan"/>
default="8.998051126081153"/>
default="-17.496187924233098"/>
default="1.5891208177771623"/>
<!-- AMCL -->
<node pkg="amcl" type="amcl" name="amcl">
<param
<param
<param
<param
<param
name="min_particles"
name="max_particles"
name="kld_err"
name="update_min_d"
name="update_min_a"
value="500"/>
value="3000"/>
value="0.02"/>
value="0.20"/>
value="0.20"/>
90
<param name="resample_interval"
<param name="transform_tolerance"
<param name="recovery_alpha_slow"
<param name="recovery_alpha_fast"
<param name="initial_pose_x"
initial_pose_x)"/>
<param name="initial_pose_y"
initial_pose_y)"/>
<param name="initial_pose_a"
initial_pose_a)"/>
<param name="gui_publish_rate"
value="1"/>
value="0.5"/>
value="0.00"/>
value="0.00"/>
value="$(arg
value="$(arg
value="$(arg
value="50.0"/>
<remap
<param
<param
<param
<param
<param
<param
<param
<param
<param
<param
from="scan"
name="laser_max_range"
name="laser_max_beams"
name="laser_z_hit"
name="laser_z_short"
name="laser_z_max"
name="laser_z_rand"
name="laser_sigma_hit"
name="laser_lambda_short"
name="laser_likelihood_max_dist"
name="laser_model_type"
to="$(arg scan_topic)"/>
value="6"/>
value="180"/>
value="0.5"/>
value="0.05"/>
value="0.05"/>
value="0.5"/>
value="0.2"/>
value="0.1"/>
value="2.0"/>
value="likelihood_field"/>
<param
<param
<param
<param
<param
<param
<param
name="odom_model_type"
name="odom_alpha1"
name="odom_alpha2"
name="odom_alpha3"
name="odom_alpha4"
name="odom_frame_id"
name="base_frame_id"
value="diff"/>
value="0.02"/>
value="0.02"/>
value="0.02"/>
value="0.02"/>
value="dbot/odom"/>
value="dbot/base_link"/>
</node>
</launch>
1.5 deliverybot_vision
scripts/vision_testing.py
#!/usr/bin/env python3
from __future__ import print_function
import roslib
roslib.load_manifest('deliverybot_vision')
import sys
import os
91
import rospy
import easyocr
from cv2 import cv2
import numpy as np
from std_msgs.msg import String
from sensor_msgs.msg import Image
from cv_bridge import CvBridge, CvBridgeError
class image_converter:
def __init__(self):
self.image_pub = rospy.Publisher('door_number_image', Image,
queue_size=10)
self.bridge = CvBridge()
self.ocr_reader = easyocr.Reader(['en'], gpu=True)
self.image_sub =
rospy.Subscriber('/dbot/upper_camera/upper_camera_image', Image,
self.callback)
dir_path = os.path.dirname(os.path.realpath(__file__))
self.yolo_weights = os.path.join(dir_path,
"weights_configs/yolo-obj.weights")
self.yolo_config = os.path.join(dir_path,
"weights_configs/yolo-obj.cfg")
self.yolo = cv2.dnn.readNet(self.yolo_weights,
self.yolo_config)
with open(os.path.join(dir_path, "weights_configs/obj.names"),
"rt") as f:
self.classes = [line.strip() for line in f.readlines()]
self.layer_names = self.yolo.getLayerNames()
self.output_layers = [self.layer_names[i[0] - 1] for i in
self.yolo.getUnconnectedOutLayers()]
def callback(self, data):
try:
cv_image = self.bridge.imgmsg_to_cv2(data, 'bgr8')
except CvBridgeError as e:
print(e)
res = self.run_ocr(cv_image)
res = self.run_detection(res)
cv2.waitKey(10)
try:
self.image_pub.publish(self.bridge.cv2_to_imgmsg(res,
'bgr8'))
except CvBridgeError as e:
print(e)
def run_detection(self, image):
try:
92
height, width, _ = image.shape
blob = cv2.dnn.blobFromImage(image, 0.00392, (416, 416),
(0, 0, 0), True, crop=False)
self.yolo.setInput(blob)
outputs = self.yolo.forward(self.output_layers)
class_ids = []
confidences = []
boxes = []
for output in outputs:
for detection in output:
scores = detection[5:]
class_id = np.argmax(scores)
confidence = scores[class_id]
if confidence > 0.5: #Ideally should work with
higher tolerance
center_x = int(detection[0] * width)
center_y = int(detection[1] * height)
w = int(detection[2] * width)
h = int(detection[3] * height)
x = int(center_x - w / 2)
y = int(center_y - h / 2)
boxes.append([x, y, w, h])
confidences.append(float(confidence))
class_ids.append(class_id)
indexes = cv2.dnn.NMSBoxes(boxes, confidences,
0.5, 0.4)
img = image.copy()
for i in range(len(boxes)):
if i in indexes:
x, y, w, h = boxes[i]
label = str(self.classes[class_ids[i]])
cv2.rectangle(img, (x, y), (x + w, y + h),
(0,255,0), 3)
cv2.putText(img, label, (x, y + 100),
cv2.FONT_HERSHEY_PLAIN, 2, (0,0,255), 3)
return img
except:
return image
def run_ocr(self, image):
try:
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
result: list = self.ocr_reader.readtext(gray)
door_number = result[0][1]
location = result[0][0]
contours = np.array(location).reshape((-1, 1,
2)).astype(np.int32)
font = cv2.FONT_HERSHEY_COMPLEX
93
res = cv2.putText(
image,
text=door_number,
org=(contours[0][0][0], contours[1][0][1]+50),
fontFace=font,
fontScale=1,
color=(0,255,0),
thickness=2,
lineType=cv2.LINE_AA
)
res = cv2.rectangle(
image,
tuple(contours[0][0]),
tuple([2][0]),
(0,255,0),
3
)
return res
except:
return image
def main(args):
ic = image_converter()
rospy.init_node('image_converter', anonymous=True)
try:
rospy.spin()
except KeyboardInterrupt:
print('Shutting Down')
cv2.destroyAllWindows()
if __name__ == "__main__":
main(sys.argv)
94