Seminar Report On
ROBOT VISION
Submitted in partial fulfilment of the requirement for the degree of
Bachelor Of Engineering
Department of Electronics and Communication Engineering
In
Deccan College of Engineering and Technology
Affiliated to Osmania University
Presented by
Syed Abrar Ur Rahman
160319735061
I
DECLARATION
I hereby declare that the work, which is being presented in the Dissertation, entitled “Robot
Vision” in partial fulfillment for the award of Degree of “Bachelor of Engineering” in
Department of Electronics and Communication Engineering and submitted to the Department
of Electronics and Communication Engineering, Deccan College of Engineering and
Technology, Osmania University is a record of my own investigations carried under the
Guidance of Dr. M. A. Nayeem, Department of Electronics and Communication Engineering,
Deccan College of Engineering and Technology.
I have not submitted the matter presented in this report anywhere for the reward of any other
Degree.
SYED ABRAR UR RAHMAN
1603-1973-5061
ECE VII SEM
Deccan College of Engineering and Technology
Name of Supervisor
Dr. M. A. NAYEEM
HOD(ECE)
Name of Principal
Dr. Syeda Gauhar Fatima
Principal DCET
II
ACKNOWLEDGEMENT
The satisfaction that accompanies that the successful completion of any task would be
incomplete without the mention of people whose ceaseless cooperation made it possible,
without constant guidance and encouragement crown all effort with success.
We are grateful to our project guide Dr. M. A. Nayeem sir, Head of the Department,
Department of Electronics and Communication Engineering for the guidance, inspiration
and constructive suggestions that helpful me in the preparation of this project.
We wish to express our sincere thanks to all staff members of Department of ECE, DCET for
their valuable suggestion and assistance rendering throughout the period of project
development.
SYED ABRAR UR RAHMAN
ECE VII SEM
Roll Number: 1603-1973-5061
III
ABSTRACT
Robot vision refers to the capability of a robot to visually perceive the environment and use
this information for execution of various tasks. Visual feedback has been used extensively for
robot navigation and obstacle avoidance. In the recent years, there are also examples that
include interaction with people and manipulation of objects. In this paper, we review some of
the work that goes beyond of using artificial landmarks and fiducial markers for the purpose
of implementing vision-based control in robots. We will discuss about the advantages and
limitations of robot vision, and the programming languages used to program the robot vision
using different microcontrollers. We discuss different application areas, both from the
systems perspective and individual problems such as object tracking and recognition.
IV
TABLE OF CONTENTS
Contents
Page number
DECLARATION..................................................................................................................... II
ACKNOWLEDGEMENT ..................................................................................................... III
ABSTRACT ............................................................................................................................IV
LIST OF TABLES AND FIGURES .....................................................................................VI
1.
INTRODUCTION ............................................................................................................ 7
2.
WHAT IS ROBOT VISION? .......................................................................................... 2
3.
WORKING OF ROBOT VISION .................................................................................. 3
4.
OBJECT DETECTION AND RECOGNITION ........................................................... 4
5.
CAMERA MODULES USED IN ROBOT VISION ..................................................... 5
6.
PROGRAMMING LANGUAGES USED IN ROBOT VISION .................................. 6
7.
ADVANTAGES AND LIMITATIONS OF ROBOT VISION ..................................... 7
8.
APPLICATIONS OF ROBOT VISION ......................................................................... 9
9.
CONCLUSION ............................................................................................................... 10
10.
REFERENCES............................................................................................................ 11
V
LIST OF TABLES AND FIGURES
Figure 1: Robot Vision................................................................................................................... 2
Figure 2: Working of Robot Vision ................................................................................................ 3
Figure 3: Object Detection .............................................................................................................. 4
Figure 4: VC0706 camera module .................................................................................................. 5
Figure 5: Raspberry pi Camera module .......................................................................................... 5
Figure 6: ESP32 Cam module......................................................................................................... 5
Figure 7: C with Arduino ................................................................................................................ 6
Figure 8: Python with Raspberry Pi ................................................................................................ 6
VI
1. INTRODUCTION
For many living species, not least in the case of humans, visual perception plays a key role in
their behaviour. Hand–eye coordination ability gives us flexibility, dexterity and robustness
of movement that no machine can match yet. One of the important factors is our ability to
track objects, that is, to maintain an object in the field of view for a period of time using our
vision system as well as head and body motions. Humans are able to do this quickly and
reliably without much effort.
Robot vision refers to the capability of a robot to visually perceive the environment and
interacts with it. Robot vision extends methods of computer vision to fulfil the tasks given to
robots and robotic systems. Typical tasks are to navigate towards a given target location
while avoiding obstacles, to find a person and react to the person’s commands, or to detect,
recognise, grasp and
deliver objects.
Thus, the goal of robot vision is to exploit the power of visual sensing to observe and
perceive the environment and react to it. Computer vision attempts to achieve this function of
understanding the scene and the objects of the environment. With the increasing speed of
processing power and progress in computer vision methods, making robots see became a
main trend in robotics.
VII
2. WHAT IS ROBOT VISION?
Vision for robots requires the ability to identify and accurately determine the positions of all
relevant three dimensional(3d) objects within the robot work place.
Robot Vision may be defined as the process of extracting, characterizing, and interpreting
information from images of a three dimensional world.
The method of processing, characterizing, and decoding data from photographs leads to
vision- based robot arm guidance, dynamic inspection, and enhanced identification and
component position capability, called Robot Vision or Robotic Vision. The robot is
programmed through an algorithm, and a camera, either fixed on the robot or in a fixed
location, captures picture of each workpiece with which it can communicate.
In basic terms, Robot Vision involves using a combination of camera hardware and computer
algorithms to allow robots to process visual data from the world. For example, your system
could have a 2D camera which detects an object for the robot to pick up. A more complex
example might be to use a 3D stereo camera to guide a robot to mount wheels onto a moving
vehicle.
Figure 1: Robot Vision
Without Robot Vision, the robot is essentially blind. This is not a problem for many robotic
tasks, but for some applications Robot Vision is useful or even essential.
2
3. WORKING OF ROBOT VISION
Robots “see” via one or more integrated cameras. At least one robot vision camera will be
mounted on the robotic arm itself, literally serving as the eye of the machine. In some cases,
additional cameras are installed in strategic locations in the robot’s working cell. This set-up
allows the camera to have a wider visual angle and capture as much visual data as it needs to
perform its function in collaboration with human workers.
There are three segments in a robot visual system:
3.1.
Image Capture:
The camera/s will start capturing visual data from a calculated distance. Afterward, the
machine will analyse the images or footage and enhance it to produce a clear picture.
3.2.
Image Processing:
The picture will go through further processing and analysed by pixel. The system will
compare the colours and apparent shape of the object with the image programmed in its
database.
3.3.
Connectivity and Response:
Once the machine recognizes that the object in the picture matches the pre-programmed
image, it will perform a corresponding action onto the object before it.
This entire process happens in quick succession within seconds.
Image
Image
Acquisition
processing
Response
Figure 2: Working of Robot Vision
database
3
4. OBJECT DETECTION AND RECOGNITION
4.1.
Object Detection:
Object detection is computer vision technique for locating instances of objects in images or
videos. Object detection algorithms typically leverage machine learning or deep learning to
produce meaningful results. When humans look at images or video, we can recognize and
locate objects of interest within a matter of moments. The goal of object detection is to
replicate this intelligence using a robot.
Figure 3: Object Detection
4.2.
Object Recognition:
Object recognition allows robots and AI programs to pick out and identify objects from
inputs like video and still camera images. Methods used for object identification
include 3D models, component identification, edge detection and analysis of appearances
from different angles.
4.3.
Difference between Object Detection and Object Recognit ion:
Object Detection
Object Recognition
Object detection solves the question –
Where is the object in a image?
Object recognition answers – What is the
object in the image.
Table 1: Difference between Object detection and Recognition
4
5.1.
5. CAMERA MODULES USED IN ROBOT VISION
VC0706 Camera module:
The VC0706 vision sensor is designed to work in unfriendly working
conditions where the humidity and temperature can affect other
sensors.
Figure 4: VC0706 camera module
5.2.
Raspberry pi Camera module:
The Raspberry Pi Camera Module is one of the most interesting camera vision
sensors designed to be attaches to the Raspberry Pi’s Camera Serial Interface
(CSI). It has a 5 mega pixel sensor capable of 2592 x 1944 pixels for static images
or to capture video images at a high definition resolution of 1080p.
Figure 5: Raspberry pi Camera module
5.3.
ESP32 Cam module:
ESP32-CAM is a low-cost ESP32-based development board with onboard camera,
small in size. The board integrates WiFi, traditional Bluetooth and low power
BLE , with 2 high performance 32-bit LX6 CPUs. It adopts 7-stage pipeline
architecture, on-chip sensor, Hall sensor, temperature sensor.
Figure 6: ESP32 Cam module
5
6. PROGRAMMING LANGUAGES USED IN ROBOT VISION
6.1.
C programming with Arduino microcontroller :
The C/C++ language is one of the most widely used programming languages
in robot vision. The Arduino microcontroller uses a programming language
based on C.
Figure 7: C with Arduino
6.2.
Python programming with Raspberry pi:
The raspberry pi foundation specifically selected python as the main language
because of its power, versatility, and ease of use. Python comes preinstalled on
raspberry pi. We have many different options for writing python on raspberry
pi.
Figure 8: Python with Raspberry Pi
6
7. ADVANTAGES AND LIMITATIONS OF ROBOT VISION
7.1.
•
Advantages of Robot Vision:
Reliability:
Computers and cameras don’t have the human factor of tiredness, which is
eliminated in them. The efficiency is usually the same, it doesn’t depend on
external factors such as illness or sentimental status.
•
Accuracy:
The precision of Computer Imagining, and Robot Vision will ensure a better
accuracy on the final product.
•
Reduction of cost:
Time and error rate are reduced in the process of Computer Imagining. It
reduces the cost of hire and train special staff to do the activities that
computers will do as hundreds of workers.
•
Wide range of use:
We can see the same robotic system in several different fields and activities.
Also, in factories with warehouse tracking and shipping of supplies, and in the
medical industry through scanned images, among other multiple options.
•
Eliminates human error:
While vision is ideal for qualitative interpretation, robot vision can
successfully measure product quantities. When paired with high-resolution
cameras, machine vision systems can also inspect object details too small for
discernment by the human eye.
7
7.2.
•
Limitations of robot vision:
Need for regular monitoring:
If a computer vision system faces a technical glitch or breaks down, this can
cause immense loss to companies. Hence, companies need to have a dedicated
team on board to monitor and evaluate these systems.
•
Spoiling:
Eliminate the human factor may be good in some cases. But when the machine
or device fails, it doesn’t announce or anticipate that problem. Whereas a
human person can tell in advance when the person won’t come.
•
Failing in image processing:
When the device fails because of a virus or other software issues, it is highly
probable that Robot Vision and image processing will fail. But if we do not
solve the problem, the functions of the device can disappear. It can freeze the
entire production in the case of warehouses.
8
8. APPLICATIONS OF ROBOT VISION
•
•
•
•
Automotive:
-
Driver assistance systems.
-
Reading automobile license plates, and traffic management.
-
Lane departure warning system.
Movie and video:
-
Multiple cameras to precisely track tennis and cricket balls.
-
Human expression recognition.
-
Tracking consistent regions in video and insert virtual advertising.
General purpose:
-
Image retrieval based on content.
-
Image inspection, people counting, and security etc.
Industrial purpose:
- Defect detection.
- Sorting.
- Bar code reading.
9
9. CONCLUSION
Robots have become a core element of Industry and flexibility can be incorporated to them
by vision systems and other sensor technologies in order to achieve the requirements and
functionalities of the new applications. New tasks are becoming more or more complex and it
is necessary to improve the accuracy and to work collaborative with humans, which means
making decisions in real-time and triggering actions. For these goals, visual feedback is the
key issue, and this is in fact what vision systems provide to robots. Thus, 3D machine vision
is the future for robotics.
Robots need flexibility and accuracy to carry out more complex and diverse tasks, such as
collision avoidance with static and moving objects during navigation, collaborative work with
humans, fine positioning, inspection, etc. Each vision system has its single purpose. There
has not been found one single vision system able to perform several tasks. Multiple vision
systems are used (one for each individual purpose) instead of one single multi-tasking vision
system. This is because requirements of each task are quite different and each technique has
its scope and is more adequate than others.
Each application and each type of robot need a specific vision solution. There is no universal
vision technique to perform several tasks. Future woks may focus on multi-tasking or multipurpose vision systems and their integration with other sensor types and systems.
10
10. REFERENCES
1. Y. Shirai and H. Inoue, “Guiding a robot by visual feedback in assembling tasks,”
Pattern Recognition, vol. 5, pp. 99–108, 1973.
2. M. Vincze and G. Hager, Robust Vision for Vision-based Control of Motion. IEEE
Press, 2000.
3. M. Goodale and A. Milner, “Separate visual pathways for perception and action,”
Trends Neuroscience, vol. 15, no. 1, p. 205, 1992.
4. M. J. Tarr and H. H. Blthoff, “Image-based recognition in man, monkey, and
machine,” Cognition, vol. 67, pp. 1–20, 1998.
5. J. Hill and W. Park, “Real time control of a robot with a mobile camera,” in Proc. 9th
ISIR, pp. 233–246, 1979.
6. G. Metta and P. Fitzpatrick, “Early integration of vision and manipulation,” Adaptive
Behavior, vol. 11, no. 2, pp. 109–128, 2003.
7. D. Kraft, E. Baseski, M. Popovic, N. Kruger, N. Pugeault, D. Kragic, S. Kalkan, and ¨
F. Worg ¨ otter, “Birth of the Object: Detection of Objectness and Extraction of Ob- ¨
ject Shape through Object Action Complexes,” International Journal of Humanoid
Robotics, vol. 5, pp. 247–265, 2008.
8. G. Hager and P. Belhumeur, “Real-time tracking of image regions with changes in
geometry and illumination,” in Proceedings of the Computer society Conference on
Computer Vision and Pattern Recognition, CVPR’96, pp. 403–410, 1996.
9. W. Grimson, Object Recognition by Computer: The Role of Geometric Constraints.
MIT Press, 1990.
10. I. Weiss and M. Ray, “Model-based recognition of 3d objects from single images,”
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp.
116 –128, 2001.
11. F. Jurie, “Robust hypothesis verification: application to model-based object
recognition,” Pattern Recognition, vol. 32, no. 6, 1999.
11
12
13
14
15
16
17
18
19
20
21
22