ilovepdf merged (8)

Development of intelligent systems (RInS) Introductory information Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Academic year: 2021/22 Development of intelligent systems About the course  Development of Intelligent Systems  University study program Computer and Information Science, 3rd year  3 hours of lectures and 2 hours of tutorials (practice classes) weekly  6 ECTS credits  Lectures on Fridays 8:15 – 11:00 (in P22)  (Online using MS Teams)  Tutorials, 4 groups:  Mondays 8:15 - 10:00 and 11:15-13:00 (in R2.38)  Tuesdays 10:15 – 12:00 and 12:15 – 14:00 in (R2.38)  Course home page: https://ucilnica.fri.uni-lj.si/course/view.php?id=69 Development of intelligent systems, Introductory information 2 About the course  Module Artificial intelligence Development of intelligent systems, Introductory information 3 Lecturer  Prof. dr. Danijel Skočaj       Visual Cognitive Systems Laboratory e-mail: danijel.skocaj@fri.uni-lj.si url: http://www.vicos.si/danijels tel: 01 479 8225 room: second floor, room R2.57 office hours: Thursday, 13:00-14:00  Teaching assistants: Matej Dobrevski  e-mail: matej.dobrevski@fri.uni-lj.si  tel: 01 479 8245  room: second floor, room R2.37 Assist. prof. Luka Čehovin Zajc  e-mail: luka.cehovin@fri.uni-lj.si  tel: 01 479 8252  room: second floor, room R2.35 Development of intelligent systems, Introductory information 4 Goal of the course The course aims at teaching the students to develop an intelligent system by integrating techniques from artificial intelligence and machine perception. Students will learn how to design an intelligent system, how to select which tools and methods to use, and how to implement new components and integrate them into a functional system. Development of intelligent systems, Introductory information 5 Course goals  To learn about intelligent robot systems     requirements methodologies applications middleware  Development of an intelligent robot system        design architecture use of appropriate components development of new components integration robot programming testing, debugging  Extremely practically oriented course! Development of intelligent systems, Introductory information 6 Development platform  Robot platform: iRobot Roomba 531 + TurtleBot + Kinect  Software platform: ROS, Robot Operating System Development of intelligent systems, Introductory information 7 TurtleBot++ Development of intelligent systems, Introductory information 8 Herd Development of intelligent systems, Introductory information 9 Robot platform Development of intelligent systems, Introductory information 10 Diploma theses G. Pušnik D. Tratnik A. Rezelj Development of intelligent systems, Introductory information J. Bizjak 11 RInS 2012  Slalom Development of intelligent systems, Introductory information 12 RInS 2013  Object search Development of intelligent systems, Introductory information 13 RInS 2014  Mini Cluedo Development of intelligent systems, Introductory information 14 RInS 2015  DeliveryBot Development of intelligent systems, Introductory information 15 RInS 2016  TaxiBot Development of intelligent systems, Introductory information 16 RInS 2017  Robot of the rings Development of intelligent systems, Introductory information 17 RInS 2018  CryptoBot Development of intelligent systems, Introductory information 18 RInS 2019  TreasureHunt Development of intelligent systems, Introductory information 19 RInS 2020 Development of intelligent systems, Introductory information 20 2021 StopCorona Development of intelligent systems, Introductory information 21 2022 Final task FooDeRo Food Delivery Robot Development of intelligent systems, Introductory information 22 2021 Final task  Setup:     „Small city“ scene (fenced area). Several persons (faces) in the scene. Four „restaurants“ (cylinders) of different colours. Four parking slots marked with rings of different sizes and colours.  Goal:  Deliver the (virtual) food to the persons.  Task:         Find all persons in the city. Find all the restaurants and look at them which food do they serve. Park in the starting parking slot. Accept orders from the web application (who orders what). Collect the food and (virtually) bring it to the corresponding persons. Talk to them if they need anything else. Fulfil their requests. Park in the end parking slot. Development of intelligent systems, Introductory information 23 2022 Final task Development of intelligent systems, Introductory information 24 2022 Final task Development of intelligent systems, Introductory information 25 Intermediate tasks  Task 1: Autonomous navigation and human search  The robot should autonomously navigate around the competition area  It should search and detect faces  It should approach every detected face  It should be completed in simulator AND with a real robot in real world  Task 2: Parking      Detect rings Recognize and say the colour of the rings Approach the green ring Detect the marked parking place below the ring Park in the marked parking place Development of intelligent systems, Introductory information 26 Competencies to be developed  System setup  Running ROS  Tele-operating TurtleBot  Autonomous navigation        Autonomous control of the mobile platform Acquiring images and 3D information Simultaneous mapping and localization (SLAM) Path planning, obstacle avoidance Advanced fine manoeuvring Basic mobile manipulation Intelligent navigation and exploration of space  Advanced perception and cognitive capabilities     Detection of faces, circles, 3D rings, 3D cylinders Recognition of faces, food, digits, colour Speech synthesis, speech recognition, dialogue processing (reading QR codes) Belief maintenance, reasoning, planning Development of intelligent systems, Introductory information 27 Autonomous navigation  Autonomous control of the mobile platform  components for controlling the robot  Acquiring images and 3D information  using Kinect  OpenCV for processing images  Point Cloud Library for processing 3D information  Simultaneous mapping and localization (SLAM)  building the map of the environment, navigation using the map  transformations between coordinate frames  Path planning, obstacle avoidance  setting the goals, approaching to the specific local goals  detect and avoid the obstacles  Advanced maneuvering  precise maneuvering  Intelligent navigation and exploration of space  autonomus exploration, setting the goals Development of intelligent systems, Introductory information 28 Perception and recognition  Face detection and recognition  Circle detection  Detection of restaurants  detection of 3D cylinders  Detection of rings  localization of rings in 3D space  Food recognition  Colour learning and recognition  Circles, cylinders, rings  (QR code reading)  Dialogue processing  Speech synthesis  Speech recognition  Speech understanding Development of intelligent systems, Introductory information 29 Advanced perception and cognition  Belief maintenance, reasoning, planning      anchoring the detected objects to the map creating and updating the beliefs reasoning using the beliefs planning for information gathering What to do next?  Intelligent navigation     considering the map optimize the exploration of space optimize the distance travelled needed Where to go next?  Visual servoing  move the mobile camera to optimise perception  visual servoing while parking Development of intelligent systems, Introductory information 30 Challenges  Robot control (ROS)  „engineering“ issues  robot system (actuators, sensors,…), real world  Selection of appropriate components for solving subproblems  many of them are given, many of them are available in ROS  Development of new components  implementing algorithms for solving new problems  Integration  integrating very different components  „Estimate the time needed for integration, multiply it by 3, but you have still probably underestimated the time actually needed.“  Difficult debbuging; visualize, log!  Very heterogeneous distributed system  mobile robotics, navigation, manipulation  computer vision, machine learning  reasoning, planning Development of intelligent systems, Introductory information 31 Competencies to be developed  System setup  Running ROS  Tele-operating TurtleBot  Autonomous navigation       Autonomous control of the mobile platform Acquiring images and 3D information Simultaneous mapping and localization (SLAM) Path planning, obstacle avoidance, approaching Advanced fine manoeuvring and parking Intelligent navigation and exploration of space  Advanced perception and cognitive capabilities      Detection of faces, circles, 3D rings, 3D cylinders Recognition of faces, food, digits, colour Basic manipuation and visual servoing Speech synthesis, speech recognition, dialogue processing (reading QR codes) Belief maintenance, reasoning, planning Development of intelligent systems, Introductory information 32 Types of challenges  System setup  Running ROS  Tele-operating TurtleBot  Autonomous navigation       Autonomous control of the mobile platform Acquiring images and 3D information Simultaneous mapping and localization (SLAM) Path planning, obstacle avoidance, approaching Advanced fine manoeuvring and parking Intelligent navigation and exploration of space engineering issues integration of components development of components  Advanced perception and cognitive capabilities      Detection of faces, circles, 3D rings, 3D cylinders Recognition of faces, food, digits, colour Basic manipuation and visual servoing Speech synthesis, speech recognition, dialogue processing (reading QR codes) Belief maintenance, reasoning, planning Development of intelligent systems, Introductory information 33 Research areas  System setup  Running ROS  Tele-operating TurtleBot  Autonomous navigation       Mobile robotics Computer vision, ML Dialogue processing, AI Autonomous control of the mobile platform Acquiring images and 3D information Simultaneous mapping and localization (SLAM) Path planning, obstacle avoidance, approaching Advanced fine manoeuvring and parking Intelligent navigation and exploration of space  Advanced perception and cognitive capabilities      Detection of faces, circles, 3D rings, 3D cylinders Recognition of faces, food, digits, colour Basic manipuation and visual servoing Speech synthesis, speech recognition, dialogue processing (reading QR codes) Belief maintenance, reasoning, planning Development of intelligent systems, Introductory information 34 Requirements  System setup  Running ROS  Tele-operating TurtleBot  Autonomous navigation       For 6 For + max. 2 For + max. 2 Autonomous control of the mobile platform Acquiring images and 3D information Simultaneous mapping and localization (SLAM) Path planning, obstacle avoidance, approaching Advanced fine manoeuvring and parking Intelligent navigation and exploration of space  Advanced perception and cognitive capabilities      Detection of faces, circles, 3D rings, 3D cylinders Recognition of faces, food, digits, colour Basic manipuation and visual servoing Speech synthesis, speech recognition, dialogue processing (reading QR codes) Belief maintenance, reasoning, planning Development of intelligent systems, Introductory information 35 Tasks  System setup  Running ROS  Tele-operating TurtleBot  Autonomous navigation       Task 1 Task 2 Task 3 Autonomous control of the mobile platform Acquiring images and 3D information Simultaneous mapping and localization (SLAM) Path planning, obstacle avoidance, approaching Advanced fine manoeuvring and parking Intelligent navigation and exploration of space  Advanced perception and cognitive capabilities      Detection of faces, circles, 3D rings, 3D cylinders Recognition of faces, food, digits, colour Basic manipuation and visual servoing Speech synthesis, speech recognition, dialogue processing (reading QR codes) Belief maintenance, reasoning, planning Development of intelligent systems, Introductory information 36 Simulation vs. real-world robot Majority of the tasks to be implemented in simulator At least part to be implemented on a real robot Also depending on the pandemic situation Development of intelligent systems, Introductory information 37 Simulation vs. real-world robot  System setup  Running ROS  Tele-operating TurtleBot  Autonomous navigation       Autonomous control of the mobile platform Acquiring images and 3D information Simultaneous mapping and localization (SLAM) Path planning, obstacle avoidance, approaching Advanced fine manoeuvring and parking Intelligent navigation and exploration of space Sim.+real Sim. only Task 1 Task 2 Task 3 Optional: also parts of Task 2 and Task 3  Advanced perception and cognitive capabilities      Detection of faces, circles, 3D rings, 3D cylinders Recognition of faces, food, digits, colour Basic manipuation and visual servoing Speech synthesis, speech recognition, dialogue processing (reading QR codes) Belief maintenance, reasoning, planning Development of intelligent systems, Introductory information 38 Homeworks  In case of only the work in simulation is possible  Will be decided during the semester  A couple of assignments on computer vision tasks on real data:     Face detection observation model Colour recognition Digit recognition Food recognition  To be submitted at učilnica  Assessed at major milestones Development of intelligent systems, Introductory information 39 Lectures  Additional knowledge needed for understanding and implementation of the individual components of the system:         introduction to intelligent systems ROS sensors transformations between the coordinate frames mobile robotics computer vision and machine learning robot manipulation artificial cognitive systems Development of intelligent systems, Introductory information 40 Practical implementation  Five robots are available  Teams of three students  Each team there is at least one good C++/Python programmer there is at least one member who can work with Linux and robots/hardware there is at least one member good in computer vision and machine learning all the members are equivalent – the work should be fairly distributed – no piggybacking!  all the members of the groups attend the same tutorial group  preferably also have their own laptop /powerful desktop      sufficiently powerful  native Linux (Ubuntu 20.04 Focal Fossa,…; ROS Noetic Ninjemys)  USB port  Mobile platforms are available  during the practice classes (tutorials)  at other times in RoboRoom R2.38 (booking required) Development of intelligent systems, Introductory information 41 Continuous integration  It is essential to work during the entire semester  Time during the official classes does not suffice  Book the robot and work at other times in RoboRoom R2.38 Development of intelligent systems, Introductory information 42 Milestones  Milestone 1: Autonomous navigation and human search  Autonomous navigation around the competition area  Find and approach the faces 25.3. Milestone 1  Milestone 2: Parking  Detect the 3D rings  Basic visual servoing  Fine manoeuvring and parking  Milestone 3: FooDeRo     Deliver the food Computer vision, machine learning Dialogue, mobile manipulation Belief maintenance, reasoning, planning 29.4. Milestone 2 27.5. Milestone 3 + Autonomous navigation and human search on a real robot Development of intelligent systems, Introductory information 43 Evaluation      Huge emphasis on practical work Continuing work and assessment during the semester Different levels of requirements available There is no written exam! Oral exam  Grading: 10 points: M1 in simulator (system operation) 10 points: M1 on real robot (system operation) 15 points: M2 in simulator (system operation) 25 points: M3 in simulator (concepts used, system operation, system performance) 10 points: Final report (description of the methods used, experimental results, implementation, innovation)  10 points: Active participation (participation at the practice classes, active work in the lab, active problem solving, homeworks)  20 points: Oral exam (concepts presented during the lectures, discussion about theoretical and implementation details of the developed system, experimental results)      Development of intelligent systems, Introductory information 44 Requirements  Requirements:  at least 30 points (50%) earned at milestones  at least 5 points (50%) for the final report  at least 50 points (50%) altogether  If the student fails to carry out the work in time (fails to successfully demonstrate the robot at the milestones), he/she can not pass the course in the current academic year.  If the student does not pass the oral exam, he/she is allowed to take it again (this year).  If it is determined that the student has not contributed enough to the development of the system in his/her team, he can not pass the course in the current academic year.  The students have to participate in the continuous assessment of their work (at milestones M1, M2, M3).  Attendance at the practice classes is compulsory.  Completed requirements are only valid in the current academic year. Development of intelligent systems, Introductory information 45 Conclusion         Very „hands-on“ course Gaining practical experience on a real robot Real-world problems Collaboration Creative thinking Problem solving Innovativeness Practical skills Development of intelligent systems, Introductory information 46 Development of intelligent systems (RInS) Introduction Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Academic year: 2021/22 Development of intelligent systems Intelligent systems  Software intelligent systems  Passive situated robot systems  Active embodied robot systems Development of intelligent systems, Introduction 2 Robotics ro·bot noun \ˈrō-ˌbät, -bət\: a real or imaginary machine that is controlled by a computer and is often made to look like a human or animal : a machine that can do the work of a person and that works automatically or is controlled by a computer Merriam – Webster dictionary  Robot  Karel Čapek: R.U.R. (Rossum's Universal Robots), 1921  „robota“ – work; forced, hard labour Development of intelligent systems, Introduction 3 Intelligent autonomous robot systems Drive Development of intelligent systems, Introduction Walk 4 Intelligent autonomous robot systems Float Development of intelligent systems, Introduction Dive 5 Intelligent autonomous robot systems Fly Development of intelligent systems, Introduction Surround us 6 Types of robots       Industrial robots Robot manipulators Mobile robots Humanoid robots Cognitive systems Unmanned aerial vehicles, ... Development of intelligent systems, Introduction 7 Industrial robots Development of intelligent systems, Introduction 8 Domestic robots Development of intelligent systems, Introduction 9 Autonomous car navigation  Autonomous navigation  Self-driving cars  Navigation assistants   Pedestrian detection Several cameras + other sensors http://www.mobileye.com Development of intelligent systems, Introduction Bloomberg, Uber self-driving car 10 Autonomous boat navigation (USV) UNI-LJ, FE, LSI FRI, LUVSS Harhpa Sea Development of intelligent systems, Introduction 11 Autonomous drones (UAV) UNI-LJ, FRI, LUVSS Development of intelligent systems, Introduction 12 Cognitive robotics  Wikipedia: Cognitive robotics is concerned with endowing robots with mammalian and human-like cognitive capabilities to enable the achievement of complex goals in complex environments. Robotic cognitive capabilities include perception processing, attention allocation, anticipation, planning, reasoning about other agents, and perhaps reasoning about their own mental states. Robotic cognition embodies the behaviour of intelligent agents in the physical world.  A cognitive robot should exhibit:       knowledge beliefs preferences goals informational attitudes motivational attitudes (observing, communicating, revising beliefs, planning) Development of intelligent systems, Introduction 13 Cognitive systems  Cognitive assistant  Explores the environment and builds a map of it  Learns to recognize and identify objects  Understand object affordances  Can verbally and non-verbally communicate with people in its vicinity  Detects new situations and reacts accordingly • Built-in basic functionalities, which are then further developed, adapted and extended by learning Development of intelligent systems, Introduction Morpha Univ. Karlsruhe 14 Cognitive systems EURON video Development of intelligent systems, Introduction 15 Intelligent robot systems UL FRI Development of intelligent systems, Introduction 16 Mobile robots EURON video Development of intelligent systems, Introduction 17 Mobile robots IRobot Roomba TurtleBot Development of intelligent systems, Introduction Ubiquity robotics Magni 18 Mobile robots UL FRI LUVSS Development of intelligent systems, Introduction 19 Robotics  Routine industrial robotic sensor system EURON video EURON video  Intelligent artificial visual cognitive systems Development of intelligent systems, Introduction 20 Sensor-robot system  Perception – action cycle Development of intelligent systems, Introduction 21 Simulation of robot perception and control Development of intelligent systems, Introduction 22 Sensors  Range sensors  Object recognition  Bumper – collision detector  Odometer Development of intelligent systems, Introduction 23 Planning and control  Planning  Control Development of intelligent systems, Introduction 24 CS 545 Robotics Introduction to Slides adapted from Sachin Chitta and Radu Rusu (Willow Garage) Overview ? my new application web browser email client window manager memory management scheduler process management device drivers CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab file system OS Overview Standards Hardware: PCI bus, USB port, FireWire, ... Software: HTML, JPG, TCP/IP, POSIX, ... my new application web browser email client window manager memory management scheduler process management device drivers CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab file system OS Overview ...but what about robots ? my new application web browser email client window manager memory management scheduler process management device drivers CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab file system OS Lack of standards for robotics CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab Typical scenario robot world 1 perceiving 2 processing 3 manipulating 1 Many sensors require device drivers and calibration procedures For example cameras: stereo processing, point cloud generation... Common to many sensors: filtering, estimation, coordinate transformation, representations, voxel grid/point cloud processing, sensor fusion,... 2 Algorithms for object detection/recognition, localization, navigation, path/motion planning, decision making, ... 3 Motor control: inverse kinematics/dynamics, PID control, force control, ... CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab Control loops robot world 1 perceiving 2 processing 3 manipulating Many control loop on different time scales Outer most control loop may run once every second (1Hz) or slower Inner most may run at 1000Hz or even higher rates Software requirements: Distributed processing with loose coupling. Sensor data comes in at various time scales. Real time capabilities for tight motor control loops. CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab Debugging tools robot world 1 perceiving 2 processing 3 manipulating visualization simulation Simulation: No risk of breaking real robots, reduce debugging cycles, test in super realtime, controlled physics, perfect model is available... Visualization: Facilitates debugging, ...looking at the world from the robot’s perspective. Data trace inspections allow debugging on small time scales. CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab Overview ROS task executive navigation simulation planning visualization perception data logging control message passing real-time capabilities device drivers web browser email client window manager memory management scheduler process management device drivers CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab file system OS Overview 1 2 3 4 5 6 7 8 Orocos: <http://www.orocos.org> OpenRTM: <http://www.is.aist.go.jp> ROS: <http://www.ros.org> OPRoS: <http://opros.or.kr> JOSER: <http://www.joser.org> InterModalics: <http://intermodalics.eu> Denx: <http://denx.de> GearBox: <http://gearbox.sourceforge.net/gbx_doc_overview.html> Why should we agree on one standard ? Code reuse, code sharing: stop inventing the wheel again and again... instead build on top of each other’s code. Ability to run the same code across multiple robots: portability facilitates collaborations and allows for comparison of similar approaches which is very important especially in science. CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab What is ROS ? ROS is an open-source, meta-operating system and stands for Robot Operating System. It provides the services you would expect from an operating system, including hardware abstraction, low-level device control, implementation of commonly-used functionality, message-passing between processes, and package management. http://www.ros.org (documentation) https://lists.sourceforge.net/lists/listinfo/ros-users (mailing list) http://www.ros.org/wiki/ROS/Installation (it’s open, it’s free !!) Mainly supported for Ubuntu linux, experimental for Mac OS X and other unix systems. http://www.ros.org/wiki/ROS/StartGuide (tutorials) CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab Robots using ROS http://www.ros.org/wiki/Robots many more.... ...and many more to come... CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS package system How to facilitate code sharing and code reuse ? A package is a building block and implements a reusable capability Complex enough to be useful Simple enough to be reused by other packages A package contains one or more executable processes (nodes) and provides a ROS interface: Messages describe the data format of the in/output of the nodes. For example, a door handle detection node gets camera images as input and spits out coordinates of detected door handles. Service and topics provide the standardized ROS interface to the rest of the system. CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS package system CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS packa ge system Collection of packages and stacks, hosted on line Many repositories (>50): Stanford, CMU, TUM , Leuven, USC, Bosch, ... http://www.ros.org/wiki/Repositories (check it out...) CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS package system ROS packages tend to follow a common structure. Here are some of the directories and files you may notice. • bin/: compiled binaries (C++ nodes) • include/package_name: C++ include headers • msg/: Message (msg) types • src/package_name/: Source files • srv/: Service (srv) types • scripts/: executable scripts (Python nodes) • launch/: launch files • CMakeLists.txt: CMake build file (see CMakeLists) • manifest.xml: Package Manifest • mainpage.dox: Doxygen mainpage documentation CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS package system manifest.xml The manifest is a minimal specification about a package and supports a wide variety of ROS tools. <package> <description brief="one line of text"> long description goes here, <em>XHTML is allowed</em> </description> <author>Alice/alice@somewhere.bar</author> <license>BSD</license> <depend package="roscpp"/> <depend package="my_package"/> <rosdep name="libreadline5-dev"/> <export> <cpp cflags="-I${prefix}/include" lflags="-L${prefix}/lib -lmy_lib"/> </export> </package> CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS core The roscore is a collection of nodes and programs that are pre-requisites for a ROS-based system. master roscore It provides naming and registration services to the rest of the nodes in the ROS system. It tracks publishers and subscribers to topics as well as services. The role of the master is to enable individual ROS nodes to locate one another. Once these nodes have located each other they communicate with each other peer-to-peer. ROS uses socket communication to facilitate networking. The roscore starts on http://my_computer:11311 CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS package system node1.launch <launch> param server my_param=42 master set <node pkg="my_package" type="node1" name="node1" args="--test"> <param name="my_param" value="42" /> roscore </node> get </launch> node 1 CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: message passing Problem: Synchronization and message passing across multiple processes, maybe even across multiple computer and/or robots. master roscore node 3 ? node 1 ? ? node 2 CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: message passing #include "ros/ros.h" #include "std_msgs/String.h" #include <sstream> Problem: int main(int argc, char **argv) across multiple processes, maybe even { Synchronization and message p assing ros::init(argc, argv, "node1"); across multiple computer and/or robots. ros::NodeHandle n; ros::Publisher chatter_pub = n.advertise<std_msgs::String>("info", 1000); master ros::Rate loop_rate(10); int count = 0; roscore while (ros::ok()) { std_msgs::String msg; std::stringstream ss; node 3 ss << "hello world " << count; msg.data = ss.str(); init ROS_INFO("%s", msg.data.c_str()); chatter_pub.publish(msg); ros::spinOnce(); node 1 loop_rate.sleep(); ++count; node 2 } return 0; } CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: message passing #include "ros/ros.h" #include "std_msgs/String.h" #include <sstream> Problem: int main(int argc, char **argv) across multiple processes, maybe even { Synchronization and message p assing ros::init(argc, argv, "node1"); across multiple computer and/or robots. ros::NodeHandle n; ros::Publisher chatter_pub = n.advertise<std_msgs::String>("info", 1000); master ros::Rate loop_rate(10); int count = 0; roscore while (ros::ok()) { std_msgs::String msg; std::stringstream ss; node 3 ss << "hello world " << count; msg.data = ss.str(); advertise ROS_INFO("%s", msg.data.c_str()); chatter_pub.publish(msg); ros::spinOnce(); node 1 loop_rate.sleep(); ++count; node 2 } return 0; } CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: message passing #include "ros/ros.h" #include "std_msgs/String.h" #include <sstream> Problem: int main(int argc, char **argv) across multiple processes, maybe even { Synchronization and message p assing ros::init(argc, argv, "node1"); across multiple computer and/or robots. ros::NodeHandle n; ros::Publisher chatter_pub = n.advertise<std_msgs::String>("info", 1000); master ros::Rate loop_rate(10); topic int count = 0; roscore while (ros::ok()) /node1/info { std_msgs::String msg; std::stringstream ss; node 3 ss << "hello world " << count; msg.data = ss.str(); advertise ROS_INFO("%s", msg.data.c_str()); chatter_pub.publish(msg); ros::spinOnce(); node 1 loop_rate.sleep(); ++count; node 2 } return 0; } CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: message passing #include "ros/ros.h" #include "std_msgs/String.h" #include <sstream> Problem: int main(int argc, char **argv) across multiple processes, maybe even { Synchronization and message p assing ros::init(argc, argv, "node1"); across multiple computer and/or robots. ros::NodeHandle n; ros::Publisher chatter_pub = n.advertise<std_msgs::String>("info", 1000); master ros::Rate loop_rate(10); topic int count = 0; roscore while (ros::ok()) /node1/info { std_msgs::String msg; std::stringstream ss; node 3 ss << "hello world " << count; msg.data = ss.str(); ROS_INFO("%s", msg.data.c_str()); publish chatter_pub.publish(msg); ros::spinOnce(); node 1 loop_rate.sleep(); ++count; node 2 } return 0; } CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: message passing Problem: Synchronization and message passing across multiple processes, maybe even across multiple computer and/or robots. master topic roscore /node1/info subscribe node 3 publish node 1 subscribe node 2 CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: message passing Problem: Synchronization and message passing across multiple processes, maybe even across multiple computer and/or robots. master topic roscore /node1/info subscribe advertise publish node 1 subscribe node 2 CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab node 3 ROS: message passing Problem: Synchronization and message passing across multiple processes, maybe even across multiple computer and/or robots. master topic roscore /node1/info /node3/info subscribe publish subscribe publish node 1 subscribe node 2 CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab node 3 ROS: logging Problem: Synchronization and message passing across multiple processes, maybe even across multiple computer and/or robots. logging subscribe master topic roscore /node1/info /node3/info subscribe publish subscribe publish node 1 subscribe node 2 CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab node 3 ROS: logging rosbag: This is a set of tools for recording from and playing back to ROS topics. It can be used to mimic real sensor streams for offline debugging. http://www.ros.org/wiki/rosbag CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: device drivers Problem: Many sensors do not come with standardized interfaces. Often the manufacturer only provides support for a single operating system (e.g. Microsoft Windows). Thus, everybody that wants to use a particular sensor is required to write their own device driver, which is time consuming and tedious. Instead, a few people did the work and the rest of the world (re-)uses their code and builds on top of it. CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: robot descriptions urdf: This package contains a C++ parser for the Unified Robot Description Format (URDF), which is an XML format for representing a robot model. <robot name="test_robot"> <link name="link1" name="link1" /> /> <link name="link2" name="link2" /> /> <link name="link3" name="link3" /> /> <link name="link4" name="link4" /> /> calibration required !! <joint name="joint1" type="continuous"> <parent link="link1"/> <child link="link2"/> </joint> <joint name="joint2" type="continuous"> <parent link="link1"/> <child link="link3"/> </joint> <joint name="joint3" type="continuous"> <parent link="link3"/> <child link="kinect_link"/> link="link4"/> </joint> </robot> http://www.ros.org/wiki/urdf CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: calibration Provides a toolchain running through the robot calibration process. This involves capturing pr2 calibration data, estimating pr2 parameters, and then updating the PR2 URDF. http://www.ros.org/wiki/pr2_calibration CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: visualization rviz: This is a 3D visualization environment for robots. It allows you to see the world through the eyes of the robot. http://www.ros.org/wiki/rviz CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: 2D/3D perception OpenCV: (Open Source Computer Vision) is a library of programming functions for real time computer vision. http://opencv.willowgarage.com/wiki/ Check out CS 574 (Prof. Ram Nevatia) !! PCL - Point Cloud Library: a comprehensive open source library for n-D Point Clouds and 3D geometry processing. The library contains numerous state-of-the art algorithms for: filtering, feature estimation, surface reconstruction, registration, model fitting and segmentation, etc. http://www.ros.org/wiki/pcl CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: planning The motion_planners stack contains different motion planners including probabilistic motion planners, search-based planners, and motion planner based on trajectory optimization. http://www.ros.org/wiki/motion_planners CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: navigation navigation: A 2D navigation stack that takes in information from odometry, sensor streams, and a goal pose and outputs safe velocity commands that are sent to a mobile base. http://www.ros.org/wiki/navigation CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab ROS: task executive SMACH, which stands for 'state machine', is a task-level architecture for rapidly creating complex robot behavior. http://www.ros.org/wiki/smach CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab Example application CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab Overview ROS task executive navigation simulation planning visualization perception data logging control message passing real-time capabilities device drivers web browser email client window manager memory management scheduler process management device drivers CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab file system OS Why should one use ROS ? Build on top of existing software, make use of existing tools, and focus on your own research. Provide the community your own work such that people can reproduce your experiments and build on top of it. More information about ROS Stanford Course: Robot Perception http://pr.willowgarage.com/wiki/Stanford_CS324_PerceptionForManipulation PR2 workshop (Good tutorial videos) http://www.ros.org/wiki/Events/PR2BetaTraining/Videos CS 545 Robotics Introduction to ROS Peter Pastor Computational Learning and Motor Control Lab Development of intelligent systems (RInS) Robot sensors and TurtleBot Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Academic year: 2021/22 Development of intelligent systems Robotic sensors Sensors Robot platforms http://ias.cs.tum.edu Development of intelligent systems, Robot sensors 2 Sensors  Equivalent to human senses  Acquire information from the environment  Electronic/mechanic/chemical device that maps the attributes of the environment into a quantitative measurement  Robot can differentiate only between the states in the environment, which can be sensed differently see action AGENT ENVIRONMENT Development of intelligent systems, Robot sensors 3 Perception action cycle  Significant abstraction of the real world sense perception modelling planning task execution motor control Development of intelligent systems, Robot sensors act 4 Senses  Human senses:  The list of robot senses is much longer!       Beyond human capabilities Vision beyond visual spectrum (IR cameras, etc.) Active vision (radar, LIDAR) Hearing beyond the range 20 Hz-20 kHz (ultrasound) Chemical analysis for better taste and smell Measurement of temperature, humidity, illumination, radiation, pressure, volume, position, direction, acceleration, velocity, etc. Development of intelligent systems, Robot sensors 5 Classification of sensors  Proprioceptive and exteroceptive sensors  Proprioceptive: measure internal states of the robot (batter status, position of wheels, angle between the segments in the robot arm)  Exteroceptive: measure the state of the environment (majority of the sensors)  Passive and active sensors  Passive: only receive the energy from the environment (e.g., camera)  Active: also emit the energy in the environment (e.g., radar)  Noninvasive and invasive sensors  Noninvasive (contactless): no contact with the object  Invasive: measurement with contact  Visual, non-visual Development of intelligent systems, Robot sensors 6 Classification of sensors Development of intelligent systems, Robot sensors 7 Classification of sensors Development of intelligent systems, Robot sensors 8 Sensors in robots Gas Sensor Piezo Bend Sensor Metal Detector Pendulum Resistive Tilt Sensors Gieger-Muller Radiation Sensor UV Detector Pyroelectric Detector Resistive Bend Sensors Digital Infrared Ranging CDS Cell Resistive Light Sensor Pressure Switch Miniature Polaroid Sensor Limit Switch Touch Switch Mechanical Tilt Sensors Gyro IR Pin Diode IR Sensor w/lens Thyristor Magnetic Sensor Polaroid Sensor Board IR Reflection Sensor Magnetic Reed Switch IR Amplifier Sensor IRDA Transceiver Hall Effect Magnetic Field Sensors Accelerometer Lite-On IR Remote Receiver Radio Shack Remote Receiver IR Modulator Receiver Development of intelligent systems, Robot sensors Solar Cell Compass Compass Piezo Ultrasonic Transducers 9 Cameras Electromagnetic spectrum Visual “light” Near infrared “light” (NIR) Long-wavelength infrared “light” (FLIR) Terahertz “light” (T-ray) Development of intelligent systems, Robot sensors 10 Sensing EM radiation Development of intelligent systems, Robot sensors 11 Resistive sensors  Band sensor  The resistance changes by bending the sensor  Potentiometer  Position sensor in sliding or rotating mechanisms  Photoresistor  Small resistance at high illumination  Light detection Sensor Development of intelligent systems, Robot sensors Sensor Sensors 12 Infrared sensors  Intensity IR sensors  Emit an receive IR light  Photo-transistor  Sensitive on daylight, reflections, distance  Robust, cheap  Application: object detection, optical encoder  Modulated IR sensors     Modulation in demodulation Pulse detection More robust IR remotes, itn. Development of intelligent systems, Robot sensors 13 Infrared sensors  Range sensors  Measuring angle between the emitted and received light -> triangulation  Non-sensitive on ambient light Development of intelligent systems, Robot sensors 14 Measuring rotation  Incremental Optical Encoders  Relative rotation light sensor light emitter decode circuitry  Incremental Optical Encoders  Absolute position  Gray code Development of intelligent systems, Robot sensors 15 Inertial sensors  Gyroscope  Measuring change of orientation  based on the principles of angular momentum  Accelerometer  Measures acceleration, also orientation  Uniaxial, triaxial  Vibration sensor, vibration analysis, detection of orientation  Nintendo Wii, smart phones Development of intelligent systems, Robot sensors 16 Compass  Electronic compass  Absolute orientation of the robot  N, S, E, W Development of intelligent systems, Robot sensors 17 GPS  Global Positioning System  24 satellites at the height of 20200 km  Atomic clock  Satellite emit the time and position data  At least 4 satellites should be visible  Differential GPS – additional (terrestrial) signals are considered Development of intelligent systems, Robot sensors 18 Tactile sensors      Haptic technology Buttons, switches Bumpers (collision sensors) Touch sensors on the robot arm Different types:     Piezoresistive Piezoelectric Capacitive Elastoresistive  Artificial skin Development of intelligent systems, Robot sensors 19 Acustic sensors  Perception of sound  Sonar  Microphone  Array of microphones  Detection the sound direction Development of intelligent systems, Robot sensors 20 Range sensors  Stereo vision  Shape from X  Coded light range sensor  IR range sensor  Time Of Flight sensors      Emit the signal, wait until it is back, measure the time RADAR SONAR LIDAR ToF cameras Development of intelligent systems, Robot sensors 21 Sonar Emits ultrasound Measure the time Bat, dolphin From a couple of cm to 30 m  30 degrees angular accuracy  Quite slow: 200ms for 30m     Development of intelligent systems, Robot sensors 22 Sonar  Usage: Mapping of space Robot chair Length of Echo Doorway chair Scan moving from left to right  Problem: noise, interference Development of intelligent systems, Robot sensors 23 Laser range sensors         LIDAR (Light Detection And Ranging) Emits laser pulses Rotating mirror – different angles (up to 180 degrees) Vertical movement – the entire hemisphere Better angular accuracy (0.25 degrees) Faster Different ranges, indoor, outdoor Robust Development of intelligent systems, Robot sensors 24 TOF cameras  Time-of-flight cameras  Time of pulse travel Development of intelligent systems, Robot sensors 25 Coded light range sensor  Camera and stripe projector v s u Color coding: Development of intelligent systems, Robot sensors 26 Stereo cameras Development of intelligent systems, Robot sensors 27 Structure from motion Development of intelligent systems, Robot sensors 28 Other sensors  Exteroceptive sensors  Wind speed  Temperature  Humidity  Proprioceptive sensors  Baterry level  Temperature of CPU, motors, sensors, etc. Development of intelligent systems, Robot sensors 29 Multimodal perception Setup Thermal Development of intelligent systems, Robot sensors Polarized cam Zed – stereo LIDAR Radar Wide-baseline stereo UL FE, FRI, Janez Perš, Matej Kristan 30 Sensor fusion  One sensor often does not suffice     Noise Limited accuracy Non-reliabilty Limited sensing range =>Fuse the results of several sensors  Sensor fusion: fusion on the level of sensors  Combine signals in one data structure on a low level  Sensor integration: Fusion on the level of representations  Process data from every sensor independently and merge the obtained information on a higher level  Fusion of data from multiple sources:  Measurement from different sensors  Measurement from different times  Measurement from different locations Development of intelligent systems, Robot sensors 31 TurtleBot++ Development of intelligent systems, Robot sensors 32 iRobot Roomba  Actuators and sensors Development of intelligent systems, Robot sensors 33 Motors  Changeable speed of the wheels  pulse-width modulation (PWM) wheels brushes vacuum cleaner  On/off motors for brushes and vacuum cleaner Development of intelligent systems, Robot sensors 34 Wheels  Differential control system  Two independently controlled wheels  Electric motor  high speed  25:1 reduction  large torque Development of intelligent systems, Robot sensors 35 Sensors bumper base buttons Development of intelligent systems, Robot sensors wall cliff dirt odometry wheels 36 IR sensors  IR sensors base bumper wall  Micro switches:  Capacitive sensor: Development of intelligent systems, Robot sensors buttons odometry cliff wheels dirt 37 Power supply  Measuring power supply     capacitance of the accumulator [mAh] voltage [V] current [A] connectors temperature Development of intelligent systems, Robot sensors 38 Indicators  Led lights  Status (green, red)  Dirt detection (blue) status dirt  Speaker  piezoelectric beeper speaker Development of intelligent systems, Robot sensors 39 RGBD sensor Kinect  PrimeSense sensor Development of intelligent systems, Robot sensors 40 Components IR projector IR camera RGB camera Development of intelligent systems, Robot sensors 41 Scheme Development of intelligent systems, Robot sensors 42 Projected pattern Development of intelligent systems, Robot sensors 43 Projected pattern Development of intelligent systems, Robot sensors 44 Patent Development of intelligent systems, Robot sensors 45 Patent Development of intelligent systems, Robot sensors 46 Kinect performance  Specifications:        Horizontal field of view: 57 degrees Vertical field of view: 43 degrees Physical tilt range: ± 27 degrees Depth sensor range: 1.2m - 3.5m 320x240 16-bit depth @ 30 frames/sec 640x480 32-bit colour@ 30 frames/sec 16-bit audio @ 16 kHz Kotnik, 2018 Khoshelham, 2011 Development of intelligent systems, Robot sensors 47 RGBD information Development of intelligent systems, Robot sensors 48 TurtleBot in simulation Development of intelligent systems, Robot sensors 49 Gazebo and RViz Development of intelligent systems, Robot sensors 50 Literature  Dr. John (Jizhong) Xiao, City College of New York, Robot Sensing and Sensors  Tod E. Kurt, Hacking Roomba: ExtremeTech, Wiley, 2006  http://www.ifixit.com/Teardown/Microsoft-Kinect-Teardown/4066/3  Futurepicture, http://www.futurepicture.org/?p=116  United States Patent, Garcia et. al, Patent No. 7,433,024 B2  Peter Corke, Robotics,Vision and Control, 2017  other Development of intelligent systems, Robot sensors 51 Development of intelligent systems (RInS) Object detection Matej Kristan, Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Academic year: 2021/22 Development of intelligent systems Computer vision Visual information Computer vision tasks Face detection! Development of intelligent systems, Object detection 2 Visual information Images Video 3D Development of intelligent systems, Object detection 3 Classification What is depicted in the image? Categorisation Localisation Recognition/identification of instances Development of intelligent systems, Object detection 4 Detection Where in the image? Detection Development of intelligent systems, Object detection Instance segmentation 5 Segmentation What does every pixel represent? Semantic segmentation Development of intelligent systems, Object detection Panoptic segmentation 6 Two stage object detection and recognition very fast efficient Face detection HOG+SVM AdaBoost SSD „Scarlet“ Face recognition CNN PCA/LDA could be slower computationally more complex Development of intelligent systems, Object detection 7 Observation model  Three face detectors given     HOG+SVM AdaBoost SSD Any other?  Not perfect  Which one is better?  More true positives  Less false positives  Test set  Images, videos  Different angles, illumination  Motion blur, etc.  Observation model  Performance  at different distances and angles  at different illuminations Development of intelligent systems, Object detection 8 Robustification of detection  Use and robustify the better detector  Take into account temporal dimension  Repetitive detections more robust  Filter out false positives  Take into account spatial dimension  Non-maximum suppression  Observation model     Map the image from 2D image to 3D world Anchor the image into the map Non-maximum suppression in the map Redetection of faces from different directions Development of intelligent systems, Object detection 9 Two stage object detection and recognition very fast efficient Face detection HOG+SVM AdaBoost SSD „Scarlet“ Face recognition CNN PCA/LDA could be slower computationally more complex Development of intelligent systems, Object detection 10 Machine Perception Category detection Matej Kristan Laboratorij za Umetne Vizualne Spoznavne Sisteme, Fakulteta za računalništvo in informatiko, Univerza v Ljubljani Machine Perception Category detection Matej Kristan Laboratorij za Umetne Vizualne Spoznavne Sisteme, Fakulteta za računalništvo in informatiko, Univerza v Ljubljani Object categorization • How to detect/recognize any car? • How to detect/recognize any cow? slide credit: B. Leibe Challenge: Robustness Illumination Occlusion Object pose Within-class variability Clutter Aspect Slide credit: Kristen Grauman Detection by classification: Standard approach • Apply machine learning to learn “features” that differ between object in interest and background. • Basic component: a binary classifier Classifier: car / non-car No, nota acar. car. Yes, Slide credit: Kristen Grauman Detection by classification: Standard approach • Apply a sliding window to cope with clutter and localize the object. Classifier Car / non-Car • This is essentially a greedy search using a (very) large number of local decisions. Slide credit: Kristen Grauman Detection by classification: Standard approach A bit more detailed description: 1. Get training data 2. Determine features (semi or fully automatic) 3. Train a classifier Examples of cars Examples of non-cars Training examples Classifier Car / non-car Feature space Not a car Images taken from: Kristen Grauman Is recognition really that difficult? Find this chair in the image Output of a normalized cross correlation. A chair Not really?! Slide credit: A. Torralba Is recognition really that difficult? Analyze this! Completely useless – does not work at all. Main issue: Feature space! Images from: A. Torralba Straight-forward global features Feature space A simple holistic description of image content, e.g., • Intensity/color histograms • Intensity vector • … Slide credit: Kristen Grauman Learned global features (e.g., PCA) The PCA learns features that maximize dataset reconstruction Calculate a lowdimensional representation by a linear subspace. Average image Eigenimages calculated from the covariance matrix Train images ...  + average + ++ Project new image into the subspace. Recognize by using a nearest-neighbor search in the subspace. [Turk & Pentland, 1991] Slide credit: Kristen Grauman Hand-crafting global features • Problem 1: Pixel-based representations, are sensitive to small shifts: • Problem2: Color or gray-level representation is sensitive to illumination changes or within-class color variations. Slide credit: Kristen Grauman Hand-crafting global features • Solution: Edges, contours and oriented intensity gradients Change intensity features into gradient-based features… Slide credit: Kristen Grauman Gradient-based representation • Edges, contours and oriented intensity gradients • Encode local gradient distributions using histograms • Locally unordered: invariant to small shifts and rotations • Contrast normalization: addresses non-uniform illumination and varying intensity. Slide credit: Kristen Grauman Gradient-based representation: HOG Histogram of Oriented Gradients: HOG For each cell visualize the strongest gradient orientations. Code available at: http://pascal.inrialpes.fr/soft/olt/ [Dalal & Triggs, CVPR 2005] Slide credit: Kristen Grauman Let’s build a classifier • We hand-crafted a feature descriptor that is invariant to illumination changes and small deformation. • How do we calculate a decision in each sub-window? Classifier Car / non-car Feature space Feature calculated from the image Lots of choices for a classifier Neural networks Nearest neighbor 106 examples Shakhnarovich, Viola, Darrell 2003 Berg, Berg, Malik 2005... Support Vector Machines Guyon, Vapnik Heisele, Serre, Poggio, 2001,… LeCun, Bottou, Bengio, Haffner 1998 Rowley, Baluja, Kanade 1998 … Boosting Conditional Random Fields Viola, Jones 2001, Torralba et al. 2004, Opelt et al. 2006,… McCallum, Freitag, Pereira 2000; Kumar, Hebert 2003, … Adapted from Antonio Torralba Consider a linear classifier A decision boundary, in general, a hyper-plane: 𝑥2 ax1  cx2  b  0 ? 𝒙𝑖 w Define: Class id: y = 1 a  w  c  𝑥1  x1  x   x2  A general hyper-plane eq: Class id: y = −1 w xb  0 T Classification of x = sign checking: Learning = Choosing 𝒘 and 𝑏! f (x)  sign (w T xi  b) Best separation hyper-plane? A general hyper-plane eq: 𝑥2 w xb  0 T Classification of x = sign checking: f (x)  sign (w T x  b) 𝑥1 Choosing 𝒘 and 𝑏? Best separation hyper-plane? A general hyper-plane eq: 𝑥2 w xb  0 T Classification of x = sign checking: f (x)  sign (w T x  b) margin 𝑥1 Support vectors 𝒙𝑖 , 𝑦𝑖 𝑖=1:𝑁 • The hyper-plane that maximizes the margin between positive (𝑦𝑖 = 1) and negative (𝑦𝑖 = −1) training examples. w   i 1 i yi xi N Have to select SVs and learn 𝛼𝑖 s! Classification == similarity to SVs A general hyper-plane eq: 𝑥2 w xb  0 T Classification of x = sign checking: f (x)  sign (w T x  b) w   i 1 i yi xi N 𝑥1 f ( x)  sign (w T x  b)  sign T  y x i i i i x  b  sign    y  ( x , x)  b  i i i i A “kernel” Non-linear kernels A non-linear kernel, e.g.,: 𝑥2  ( x i , x)  e 1 (  (x i  x)2 / 2 ) 2 Classification funct. unchanged: 𝑥1 f (x)  sign    y  ( x , x)  b  i i i i Non-linear kernels lift the dimensionality of the data such and apply a linear hyper-plane in this high-dimensional space. The hyper-plane becomes nonlinear in the original space. Application: Pedestrian detection • Sliding window: 1. extract HOG at each displacement 2. classify by a linear SVM HOG 𝒙 f ( x)  sign (w T x  b) HOG cells weighted by the positive support vectors input HOG cells weighted by the negative support vectors HOG Dalal and Triggs, Histograms of oriented gradients for human detection, CVPR2005 Pedestrian detection HoG+SVM Navneet Dalal, Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005 Slide credit: Kristen Grauman Objects (non)rigidly deform • Nonrigid/deformable objects poorly detected using a fixed structure. Slide credit: Kristen Grauman Deformable parts models (DPM) • Each part is a HOG descriptor. • Learn a classifier for HOGs and geometric constraints simultaneously by structured SVM. Root (global) part Overlaid parts Geometric constraints P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010 A Great tutorial on DPMs at ICCV2013: http://www.cs.berkeley.edu/~rbg/ICCV2013/ Time/computation criticality • A lot of applications are time- and resources-critical • Require efficient feature construction • Require efficient classification • A case study: • Face detection Face detection Application specifics: • Frontal faces are a good example, where the global appearance model + sliding window works well: • Regular 2D structure • Central part of the face is well approximated by rectangle. Images from: Kristen Grauman Faces: Terminology • Detection: Given an image, where are the faces? Anna Image credit: H. Rowley • Recognition: Whose face is it? • Classification: Gender, Age? Fast face detection • To apply in real-time applications 1. Feature extraction should be fast 2. Classifier application should be fast • These points addressed next Slide credit: Kristen Grauman Choice of classifiers Neural networks Nearest neighbor 106 examples Shakhnarovich, Viola, Darrell 2003 Berg, Berg, Malik 2005... Support Vector Machines Guyon, Vapnik Heisele, Serre, Poggio, 2001,… LeCun, Bottou, Bengio, Haffner 1998 Rowley, Baluja, Kanade 1998 … Boosting Conditional Random Fields Viola, Jones 2001, Torralba et al. 2004, Opelt et al. 2006,… McCallum, Freitag, Pereira 2000; Kumar, Hebert 2003, … Adapted from Antonio Torralba Boosting • Build a strong classifier from a combination of many “weak classifiers” – weak learners (each at least better than random) • Flexible choice of weak learners • This includes fast but inaccurate classifiers! • We’ll have a look at the AdaBoost (Freund & Schapire) • Simple to implement. • Basis for the popular Viola-Jones face detector. Y. Freund and R. Schapire, A short introduction to boosting, Journal of Japanese Society for Artificial Intelligence, 14(5):771-780, 1999. Adaboost: Intuition weight weak classifier 𝑓2 (𝑥) • Task: Build a classifier which is a weighted sum of many classifiers ℎ(𝑥) 𝑓1 (𝑥) 𝑓2 (𝑥) Example of a weak classifier: 𝜃𝑡 ℎ (𝑥) 𝑡 𝑓1 (𝑥) ℎ2 (𝑥) 𝑓2 (𝑥) 𝑓2 (𝑥) 𝑓2 (𝑥) AdaBoost: Intuition ℎ3 (𝑥) ℎ1 (𝑥) 𝑓1 (𝑥) 𝑓1 (𝑥) 𝑓1 (𝑥) • Train a sequence of weak classifiers. • Each weak classifier splits train examples with at least 50% accuracy. • Those examples that are incorrectly classified by the weak classifier, get more weight in training the next weak classifier. Final classifier is a combination of many weak classifiers! AdaBoost algorithm Start with uniform weights of training samples. {x1,…xn} Repeat T-times: (add a weak classifier) Select a feature that minimizes weighted classification error and build a weak classifier with that feature. Reweight the examples: Incorrectly classified  higher weights Correctly classified  lower weights Final classifier is a combination of weak classifiers, which are weighted according to their error. [Freund & Schapire 1995] Face detection • To apply in real-time applications 1. Feature extraction should be fast (? How to calculate fast/strong features ?) 2. Classifier application should be fast (weak classifiers = fast evaluation) Computing features Simple rectangular filters as feature extractors +1 -1 𝑓1 𝑥 -1 +1 -1 +1 -1 𝑓2 𝑥 𝑓3 𝑥 … 𝑓4 𝑥 The output of each feature is the difference between the intensity in „black“ and „white“ regions. Black is weighted as -1, white as +1. Sum=1500 + Sum=2000 - 𝑓1 𝑥 = 500 Computing features Simple rectangular filters as feature extractors +1 -1 𝑓1 𝑥 +1 -1 Require evaluation at many displacements and multiple scales! Possible to evaluate such a simple filter efficiently! Efficient computation – Integral images • Our filters are based on sums of intensities within rectangular regions. • This can be done in constant time for arbitrary large region! • Require precomputing integral image. Integral image Value at (x,y) is the sum of pixel intensities above and left from (x,y) +1 -1 Efficient computation – Integral images • Consider a more complex filter * =? Integral image -1 +2 (x,y) -1 +1 -2 +1 Large collection of filters Account for all possible parameters: position, scale, type More than 180,000 different features in a 24x24 window. etc... Apply Adaboost for (i) selecting most informative features and (ii) composing a classifier. [Viola & Jones, CVPR 2001] Slide credit: Kristen Grauman Training Adaboost for face detection Train a cascade of classifiers using the Adaboost faces non-faces Selected features, thresholds and weights. AdaBoost algorithm Start with uniform weights of training samples. Repeat T-times {x1,…xn} Build a test classifier along each feature. Calculate weighted classification error for each feature and choose a feature with smallest error (and its classifier). … [Freund & Schapire 1995] Feature selection and classification: Adaboost • In each round select a single feature and threshold that best separate positive (faces) and negative (non-faces) examples given the weighted error. Obtained weak classifier: … Outputs of rectangular filters (features) for faces and non-faces. For next round of training reweight the training examples using the errors and select the next feature-threshold pair. [Viola & Jones, CVPR 2001] Slide credit: Kristen Grauman AdaBoost algorithm Start with uniform weights of training samples. Repeat T-times {x1,…xn} Build a test classifier along each feature. Calculate weighted classification error for each feature and choose a feature with smallest error (and its classifier). Reweight the examples: Incorrectly classified  higher weights Correctly classified  lower weights Final classifier is a combination of weak classifiers, which are weighted according to their error. [Freund & Schapire 1995] Adaboost and feature selection (summary) • Image features = weak classifiers • In each round of the Adaboost: 1. Evaluate each rectangular filter on each training example 2. Sort examples w.r.t. filter responses 3. Select the threshold for each filter (with minimum error) • Determine the optimal threshold in sorted list 4. Select the best combination of filter and threshold 5. The weight of the features is the classification error 6. Reweight examples P. Viola, M. Jones, Robust Real-Time Face Detection, IJCV, Vol. 57(2), 2004. (first version appeared at CVPR 2001) Slide credit: Kristen Grauman Efficiency issues Extract features at each bounding box and apply Adaboost classifier. • Filter responses can be evaluated fast. • But each image contains a lot of windows, that we need to classify – potentially great amount of computation! • How to make detection efficient? Cascade of classifiers • Efficient: Apply less accurate but fast classifiers first , to reject the windows that obviously do not contain the particular category! … Cascade of classifiers • Chain classifiers from least complex with low true-positive rejection rate to most complex ones: The ROC curve % False Pos 0 50 50 % Detection 100 vs false neg determined by Window cropped from the image T Classifier 1 F non-face T Classifier 2 F non-face T Classifier 3 Face F non-face Slide credit: Svetlana Lazebnik Viola-Jones face detector Train a cascade of classifiers using the Adaboost New image faces non-faces Selected features, thresholds and weights. Postprocess detections by non-maxima suppression. • Train using 5k positives and 350M negatives • Real-time detector using 38 layers in cascade • 6061 features in the final layer (classifier) • [OpenCV implementation: http://sourceforge.net/projects/opencvlibrary/] Slide credit: Kristen Grauman Viola-Jones: results Guess what these correspond to! Interesting: First two selected features. • Performance   384x288 images, detection 15 fps on 700 MHz Intel Pentium III desktop (2001). Training time = weeks! Slide adapted from Kristen Grauman Detection in progress • The video visualizes all the “features”, i.e., filter responses checked in a cascade. • Observe the increase of cascade once close to face. http://cvdazzle.com/ Make your face invisible • Know how it works? Brake it! http://cvdazzle.com/ Viola-Jones: results Slide credit: Kristen Grauman Viola-Jones: results Slide credit: Kristen Grauman Viola-Jones: results Note the missing profiles! Detector trained only on frontal faces. Slide credit: Kristen Grauman Profile detection Profile detection requires learning a separate detector using profile faces. Slide credit: Kristen Grauman Profile detection Paul Viola, ICCV tutorial Try it at home! • Viola & Jones detector was a great success • First (!) real-time face detector • Lots of improvements since initial publication • C++ implementation OpenCV [Lienhart, 2002] • http://sourceforge.net/projects/opencvlibrary/ • Matlab wrappers for C++: • OpenCV version: Mex OpenCV • Without OpenCV: http://www.mathworks.com/matlabcentral/fileexchange/20976-fdlibmexfast-and-simple-face-detection Slide credit: Kristen Grauman Application example Frontal faces detected and tracked. Names inferred from subtitles and scripts. Everingham, M., Sivic, J. and Zisserman, A. "Hello! My name is... Buffy" - Automatic naming of characters in TV video, BMVC 2006. http://www.robots.ox.ac.uk/~vgg/research/nface/index.html Slide credit: Kristen Grauman Boosting by context D. Hoiem, A. Efros, and M. Herbert. Putting Objects in Perspective. CVPR 2006. slide credit: Rob Fergus Drawbacks remain… • Some objects poorly described by a single box • Occlusion not accounted for at all ? Choice of classifiers Neural networks Nearest neighbor % 106 examples Shakhnarovich, Viola, Darrell 2003 Berg, Berg, Malik 2005... Support Vector Machines Guyon, Vapnik Heisele, Serre, Poggio, 2001,… LeCun, Bottou, Bengio, Haffner 1998 Rowley, Baluja, Kanade 1998 … Boosting Conditional Random Fields Viola, Jones 2001, Torralba et al. 2004, Opelt et al. 2006,… McCallum, Freitag, Pereira 2000; Kumar, Hebert 2003, … Adapted from Antonio Torralba Recent advances in feature learning • Learning features by convolutional neural networks Yan LeCun, http://yann.lecun.com/ • First successful application to large-scale object detection by Krizhevsky et al. 1 • Huge number of parameters to learn (millions). • Impressive performance. • Currently a hot research topic (Microsoft, Facebook, Google, etc.) 1Krizhevsky NIPS2012 et al., ImageNet Classification with Deep Convolutional Neural Networks, Recent advances in feature learning • Some examples on COCO challenge • CNN trained for region proposals + CNN trained for classification. Zagoruyko et al., “FAIR” team at MS COCO & ILSVRC Object Detection and Segmentation Challenge, ICCV2015 Recent advances in feature learning • Some examples on COCO challenge • CNN trained for region proposals + CNN trained for classification. Zagoruyko et al., “FAIR” team at MS COCO & ILSVRC Object Detection and Segmentation Challenge, ICCV2015 References • David A. Forsyth, Jean Ponce, Computer Vision: A Modern Approach (2nd Edition), (prva izdaja dostopna na spletu) • R. Szeliski,Computer Vision: Algorithms and Applications, Springer, 2011 • Viola, M. Jones, Robust Real-Time Face Detection, IJCV, Vol. 57(2), 2004. • Viola-Jones Face Detector • C++ implementation in OpenCV [Lienhart, 2002] • http://sourceforge.net/projects/opencvlibrary/ • Matlab wrappers: • http://www.mathworks.com/matlabcentral/fileexchange/19912 • Convolutional neural networks • Yan LeCun, http://yann.lecun.com/ • Caffe, Torch, Tensor flow, etc. Development of intelligent systems (RInS) Transformations between coordinate frames Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Literature: Tadej Bajd (2006). Osnove robotike, chapter 2 Academic year: 2021/22 Development of intelligent systems Coordinate frames Development of intelligent systems, Transformations between coordinate frames 2 3D environment Development of intelligent systems, Transformations between coordinate frames 3 2D navigation Development of intelligent systems, Transformations between coordinate frames 4 Degrees of freedom  DOF  6 DOF for full description of the pose of an object in space  3 translations (position)  3 rotations (orientation) Development of intelligent systems, Transformations between coordinate frames 5 Degrees of freedom Development of intelligent systems, Transformations between coordinate frames 6 Degrees of freedom 3 translations 2 rotations 1 rotation ORIENTATION POSITION POSE Development of intelligent systems, Transformations between coordinate frames 7 Position and orientation of the robot Development of intelligent systems, Transformations between coordinate frames 8 Pose of the object in 3D space Development of intelligent systems, Transformations between coordinate frames 9 Robot manipulator  ViCoS LCLWOS robot manipulator  5DOF  6DOF needed for general grasping Development of intelligent systems, Transformations between coordinate frames 10 Chains of coordinate frames  Transformations between coordinate frames Development of intelligent systems, Transformations between coordinate frames 11 Position and orientation  Pose=Position+Orientation      Position(P2)=Position (P3) Position(P1)~=Position (P2) Orientation(P1)=Orientation (P3) Orientation(P2)~=Orientation (P3) Pose(P1)~=Pose(P2)~=Pose(P3) Development of intelligent systems, Transformations between coordinate frames 12 Translation and rotation  Moving objects:  P1 v P3: Translation (T)  P2 v P3: Rotation (R)  P1 v P2: Translation in rotation Development of intelligent systems, Transformations between coordinate frames 13 Position  Position: vector from the origin of the coordinate frame to the point  Position of the object P1: Development of intelligent systems, Transformations between coordinate frames 14 Orientation  Right-handed coordinate frame  Rotation around x0 axis:  Rotation matrix:  Orientation of c.f. with respect to c.f.  Transformation of the vector expressed in the c.f. in the c.f. : Development of intelligent systems, Transformations between coordinate frames into the coordinates expressed 15 Rotation matrices  Rotation around x axis:  Rotation around y axis :  Rotation around z axis : Development of intelligent systems, Transformations between coordinate frames 16 Properties of rotation matrix  Rotation is an orthogonal transformation matrix  Inverse transformation:  In the right-handed coordinate frame the determinant equals to 1  Addition of angles:  Backward rotation: Development of intelligent systems, Transformations between coordinate frames 17 Consecutive rotations  Postmultiplicate the vector with the rotation matrix  Consecutive rotations:  Rotation matrices are postmultiplicated:  In general:  Postmultiplicate matrices for all rotations  Rotations always refer to the respective relative current coordinate frame Development of intelligent systems, Transformations between coordinate frames 18 Transformations  Transformation from one c.f. to another:  If c.f. are parallel:  Only translation  If c.f. are not parallel:  Rotation and translation  General pose description Development of intelligent systems, Transformations between coordinate frames 19 Matrix notation  Three coordinate frames:  Combine the transformations:  We can add the translation vectors if they are expressed in the same coordinate frame  The two equations in the matrix form: Development of intelligent systems, Transformations between coordinate frames 20 Homogeneous transformations  General pose can be expressed in the matrix form:  Homogeneous transformation - homogenises (combines) rotation and translation in one matrix  Very concise and convenient format  Homogeneous matrix of size 4x4 (for 3D space)  One row is added, also 1 in the position vector Development of intelligent systems, Transformations between coordinate frames 21 Homogenous matrix  Rotation R and translation d:  Only rotation: Only translation: Development of intelligent systems, Transformations between coordinate frames 22 Properties of homogeneous transformation  Inverse of homogeneous transformation:  Consecutive poses:  Postmultiplication of homogeneous transformations:  An element can be transformed arbitrary number of times – by multiplying homogeneous matrices Development of intelligent systems, Transformations between coordinate frames 23 Example  Two rotations  Vector first rotate for 90o around z axis and then for 90o around y axis Development of intelligent systems, Transformations between coordinate frames 24 Example– two rotations Development of intelligent systems, Transformations between coordinate frames 25 Example - translation  After two rotations also translate the vector for (4,-3,7)  Merge  Translation with rotations  Transformation of the point (7,3,2): Development of intelligent systems, Transformations between coordinate frames 26 Transformation of the coordinate frame  Homogeneous transformation matrix transforms the base coordinate frame  Vector of origin of c.f.:  Unit vectors: Development of intelligent systems, Transformations between coordinate frames 27 Pose of the coordinate frame  Unit vectors of the new coordinate frame:  Transformaction matrix descibes the coordinate frame! Development of intelligent systems, Transformations between coordinate frames 28 Movement of the coordinate frame  Premultiplication or postmultiplication (of an object or c.f.) with transformation  Example:  Coordinate frame:  Transformation: Development of intelligent systems, Transformations between coordinate frames 29 Premultiplication  The pose of the object is transformed with respect to the fixed reference coordinate frame in which the object coordinates were given.  Order of transformations: 2. 1. Development of intelligent systems, Transformations between coordinate frames 30 Postmultiplication  The pose of the object is transformed with respect to its own relative current coordinate frame  Order of transformations: 2. 1. Development of intelligent systems, Transformations between coordinate frames 31 Movement of the reference c.f.  Example: Trans(2,1,0)Rot(z,90) Development of intelligent systems, Transformations between coordinate frames 32 Movement of the reference c.f.  Example: Trans(2,1,0)Rot(z,90) With respect to the reference coordinate frame: Rot(z,90) Trans(2,1,0) With respect to the relative coordinate frame: Trans(2,1,0) Development of intelligent systems, Transformations between coordinate frames Rot(z,90) 33 Package TF in ROS  Maintenance of the coordinate frames through time Development of intelligent systems, Transformations between coordinate frames 34 Conventions  Right-handed coordinate frame  Orientation of the robot or object axes  x: forward  y: left  z: up x  Orientation of the camera axes  z: forward  x: right  y: down  Rotation representations     z z y y x quaternions rotation matrix rotations around X, Y and Z axes Euler angles Development of intelligent systems, Transformations between coordinate frames 35 Coordinate frames on mobile plaforms  map (global map)     world coordinate frame does not change (or very rarely) long-term reference useless in short-term  odom (odometry)     world coordinate frame changes with respect to odometry useless in long-term uselful in short-term  base_link (robot)  attached to the robot  robot coordinate frame Development of intelligent systems, Transformations between coordinate frames 36 Tree of coordinate frames  ROS TF2  tree of coordinate frames and their relative poses  distributed representation  dynamic representation y  changes through time  accessible representation  querying relations between arbitrary coordinate frames x W W A W A B B Development of intelligent systems, Transformations between coordinate frames 37 Development of intelligent systems (RInS) Mobile robotics Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Slides: Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras, UNI Freiburg, Introduction to Mobile Robotics Academic year: 2021/22 Razvoj inteligentnih sistemov Introduction  Slides credit to: Autonomous Intelligent Systems Wolfram Burgard Autonomous Intelligent Systems Cyrill Stachniss Humanoid Robots Maren Bennewitz Social Robotics Kai Arras Albert-Ludwigs-Universität, Freiburg, Germany Razvoj inteligentnih sistemov, Mobilna robotika 2 Introduction  Wolfram Burgard, Albert-Ludwigs-Universität Freiburg  Sebastian Thrun, Wolfram Burgard and Dieter Fox,Probabilistic Robotics, The MIT Press, 2005 Razvoj inteligentnih sistemov, Mobilna robotika 3 Introduction  Course Introduction to Mobile Robotics – Autonomous Mobile Systems at the Albert-Ludwigs-Universität Freiburg Razvoj inteligentnih sistemov, Mobilna robotika 4 Introduction  Course Introduction to Mobile Robotics – Autonomous Mobile Systems at the Albert-Ludwigs-Universität Freiburg  This course: Razvoj inteligentnih sistemov, Mobilna robotika 5 Goal of this course  Provide an overview of problems / approaches in mobile robotics  Probabilistic reasoning: Dealing with noisy data  Hands-on experience AI View on Mobile Robotics Sensor data Control system World model Actions Components of Typical Robots cameras sensors sonars laser base actuators 8 Architecture of a Typical Control System Introduction to Mobile Robotics Robot Control Paradigms Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras 10 Classical / Hierarchical Paradigm Sense Plan Act  70’s  Focus on automated reasoning and knowledge representation  STRIPS (Stanford Research Institute Problem Solver): Perfect world model, closed world assumption  Find boxes and move them to designated position 11 Classical Paradigm Stanford Cart 1. Take nine images of the environment, identify interesting points in one image, and use other images to obtain depth estimates. 2. Integrate information into global world model. 3. Correlate images with previous image set to estimate robot motion. 4. On basis of desired motion, estimated motion, and current estimate of environment, determine direction in which to move. 5. Execute the motion. 12 Stanford Cart 13 Sensing Motor Control Plan Execute Plan Sense Model Perception Classical Paradigm as Horizontal/Functional Decomposition Act Action Environment 14 Reactive / Behavior-based Paradigm Sense Act  No models: The world is its own, best model  Easy successes, but also limitations  Investigate biological systems 15 Reactive Paradigm as Vertical Decomposition … Explore Wander Avoid obstacles Sensing Action Environment 16 Characteristics of Reactive Paradigm  Situated agent, robot is integral part of the world.  No memory, controlled by what is happening in the world.  Tight coupling between perception and action via behaviors.  Only local, behavior-specific sensing is permitted (ego-centric representation). 17 Behaviors  … are a direct mapping of sensory inputs to a pattern of motor actions that are then used to achieve a task.  … serve as the basic building block for robotics actions, and the overall behavior of the robot is emergent.  … support good software design principles due to modularity. 18 Subsumption Architecture  Introduced by Rodney Brooks ’86.  Behaviors are networks of sensing and acting modules (augmented finite state machines AFSM).  Modules are grouped into layers of competence.  Layers can subsume lower layers.  No internal state! 19 Level 0: Avoid Polar plot of sonars Feel force Sonar force Run away heading encoders polar plot Collide heading Turn halt Forward 20 Level 1: Wander heading Wander Feel force Sonar force force Avoid Run away s heading polar plot Collide modified heading halt Turn heading encoders Forward 21 Level 2: Follow Corridor Stay in middle Look corridor Wander Feel force Sonar distance, direction traveled heading to middle s force force Avoid Run away Collide modified heading s heading polar plot halt Integrate Turn heading encoders Forward 22 Reactive Paradigm  Representations?  Good software engineering principles?  Easy to program?  Robustness?  Scalability? 23 Hybrid Deliberative/reactive Paradigm Plan Sense Act  Combines advantages of previous paradigms   World model used for planning Closed loop, reactive control 24 Introduction to Mobile Robotics Probabilistic Motion Models Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras 25 Robot Motion  Robot motion is inherently uncertain.  How can we model this uncertainty? 26 Dynamic Bayesian Network for Controls, States, and Sensations 27 Probabilistic Motion Models  To implement the Bayes Filter, we need the transition model p(x | x’, u).  The term p(x | x’, u) specifies a posterior probability, that action u carries the robot from x’ to x.  In this section we will specify, how p(x | x’, u) can be modeled based on the motion equations. 28 Coordinate Systems  In general the configuration of a robot can be described by six parameters.  Three-dimensional Cartesian coordinates plus three Euler angles pitch, roll, and tilt.  Throughout this section, we consider robots operating on a planar surface.  The state space of such systems is threedimensional (x,y,). 29 Typical Motion Models  In practice, one often finds two types of motion models:  Odometry-based  Velocity-based (dead reckoning)  Odometry-based models are used when systems are equipped with wheel encoders.  Velocity-based models have to be applied when no wheel encoders are given.  They calculate the new pose based on the velocities and the time elapsed. 30 Example Wheel Encoders These modules require +5V and GND to power them, and provide a 0 to 5V output. They provide +5V output when they "see" white, and a 0V output when they "see" black. These disks are manufactured out of high quality laminated color plastic to offer a very crisp black to white transition. This enables a wheel encoder sensor to easily see the transitions. Source: http://www.active-robots.com/ 31 Dead Reckoning  Derived from “deduced reckoning.”  Mathematical procedure for determining the present location of a vehicle.  Achieved by calculating the current pose of the vehicle based on its velocities and the time elapsed. 32 Reasons for Motion Errors ideal case bump different wheel diameters carpet and many more … 33 Odometry Model • Robot moves from x , y , to x ' , y ' , ' . • Odometry information u   rot1 ,  rot 2 ,  trans .  trans  ( x ' x )  ( y ' y ) 2 2  rot1  atan2( y ' y , x ' x )    rot 2   '   rot1 x , y ,  rot1  rot 2  trans x ' , y ' , ' Noise Model for Odometry  The measured motion is given by the true motion corrupted with noise. ˆrot1   rot1   1 | rot1 | 2 ˆtrans   trans   ˆrot 2   rot 2   3 | trans | | trans | 4 | rot1  rot 2 | 1 | rot 2 | 2 | trans | Typical Distributions for Probabilistic Motion Models Normal distribution   ( x)  2 1 2 2 e 1 x2  2 2 Triangular distribution 0 if | x | 6 2    2 ( x)   6 2  | x |  6 2  Application  Repeated application of the sensor model for short movements.  Typical banana-shaped distributions obtained for 2d-projection of 3d posterior. p(x|u,x’) x’ x’ u u Sample Odometry Motion Model 1. Algorithm sample_motion_model(u, x): u   rot1 ,  rot 2 ,  trans , x  x, y, 1. ˆrot1   rot1  sample(1 |  rot1 |  2  trans ) 2. ˆtrans   trans  sample( 3  trans   4 (|  rot1 |  |  rot 2 |)) 3. ˆ    sample( |  |   ) rot 2 rot 2 1 6. x'  x  ˆtrans cos(  ˆrot1 ) y '  y  ˆtrans sin(  ˆrot1 )  '    ˆ  ˆ 7. Return 4. 5. rot 1 rot 2 x ' , y ' , ' rot 2 2 trans sample_normal_distribution Sampling from Our Motion Model Start Examples (Odometry-Based) Introduction to Mobile Robotics Probabilistic Sensor Models Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Giorgio Grisetti, Kai Arras 41 Sensors for Mobile Robots  Contact sensors: Bumpers  Internal sensors  Accelerometers (spring-mounted masses)  Gyroscopes (spinning mass, laser light)  Compasses, inclinometers (earth magnetic field, gravity)  Proximity sensors     Sonar (time of flight) Radar (phase and frequency) Laser range-finders (triangulation, tof, phase) Infrared (intensity)  Visual sensors: Cameras  Satellite-based sensors: GPS 42 Proximity Sensors  The central task is to determine P(z|x), i.e., the probability of a measurement z given that the robot is at position x.  Question: Where do the probabilities come from?  Approach: Let’s try to explain a measurement. 43 Beam-based Sensor Model  Scan z consists of K measurements. z  {z1 , z2 ,..., z K }  Individual measurements are independent given the robot position. K P ( z | x , m )   P ( z k | x, m ) k 1 44 Beam-based Sensor Model K P ( z | x , m )   P ( z k | x, m ) k 1 45 Typical Measurement Errors of an Range Measurements 1. Beams reflected by obstacles 2. Beams reflected by persons / caused by crosstalk 3. Random measurements 4. Maximum range measurements 46 Proximity Measurement  Measurement can be caused by …     a known obstacle. cross-talk. an unexpected obstacle (people, furniture, …). missing all obstacles (total reflection, glass, …).  Noise is due to uncertainty …     in measuring distance to known obstacle. in position of known obstacles. in position of additional obstacles. whether obstacle is missed. 47 Beam-based Proximity Model Measurement noise 0 zexp Phit ( z | x, m)   Unexpected obstacles zmax 1 e 2b 1 ( z  zexp )  2 b 2 0 zexp   e  z Punexp ( z | x, m)    0 zmax z  zexp   otherwise 48 Beam-based Proximity Model Random measurement 0 zexp Prand ( z | x, m)   zmax 1 z max Max range 0 zexp Pmax ( z | x, m)   zmax 1 z small 49 Resulting Mixture Density   hit     unexp  P ( z | x, m)    max       rand  T  Phit ( z | x, m)     Punexp ( z | x, m)   Pmax ( z | x, m)     P ( z | x, m)   rand  How can we determine the model parameters? 50 Raw Sensor Data Measured distances for expected distance of 300 cm. Sonar Laser 51 Approximation  Maximize log likelihood of the data P ( z | zexp )  Search space of n-1 parameters.     Hill climbing Gradient descent Genetic algorithms …  Deterministically compute the n-th parameter to satisfy normalization constraint. 52 Approximation Results Laser Sonar 300cm 400cm 53 Example z P(z|x,m) 54 Scan-based Model  Probability is a mixture of …  a Gaussian distribution with mean at distance to closest obstacle,  a uniform distribution for random measurements, and  a small uniform distribution for max range measurements.  Again, independence between different components is assumed. 55 Example Likelihood field Map m P(z|x,m) 56 San Jose Tech Museum Occupancy grid map Likelihood field 57 Scan Matching  Extract likelihood field from scan and use it to match different scan. 58 Scan Matching  Extract likelihood field from first scan and use it to match second scan. ~0.01 sec 59 Properties of Scan-based Model  Highly efficient, uses 2D tables only.  Smooth w.r.t. to small changes in robot position.  Allows gradient descent, scan matching.  Ignores physical properties of beams.  Will it work for ultrasound sensors? 60 Additional Models of Proximity Sensors  Map matching (sonar, laser): generate small, local maps from sensor data and match local maps against global model.  Scan matching (laser): map is represented by scan endpoints, match scan into this map.  Features (sonar, laser, vision): Extract features such as doors, hallways from sensor data. 61 Landmarks  Active beacons (e.g., radio, GPS)  Passive (e.g., visual, retro-reflective)  Standard approach is triangulation  Sensor provides  distance, or  bearing, or  distance and bearing. 62 Distance and Bearing 63 Summary of Sensor Models  Explicitly modeling uncertainty in sensing is key to robustness.  In many cases, good models can be found by the following approach: 1. Determine parametric model of noise free measurement. 2. Analyze sources of noise. 3. Add adequate noise to parameters (eventually mix in densities for noise). 4. Learn (and verify) parameters by fitting model to data. 5. Likelihood of measurement is given by “probabilistically comparing” the actual with the expected measurement.  This holds for motion models as well.  It is extremely important to be aware of the underlying assumptions! 64 Introduction to Mobile Robotics Mapping with Known Poses Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras 65 Why Mapping?  Learning maps is one of the fundamental problems in mobile robotics  Maps allow robots to efficiently carry out their tasks, allow localization …  Successful robot systems rely on maps for localization, path planning, activity planning etc. 66 The General Problem of Mapping What does the environment look like? 67 The General Problem of Mapping  Formally, mapping involves, given the sensor data, d  {u1 , z1 , u2 , z2 ,, un , zn } to calculate the most likely map m  arg max P(m | d ) * m 68 Mapping as a Chicken and Egg Problem  So far we learned how to estimate the pose of the vehicle given the data and the map.  Mapping, however, involves to simultaneously estimate the pose of the vehicle and the map.  The general problem is therefore denoted as the simultaneous localization and mapping problem (SLAM).  Throughout this section we will describe how to calculate a map given we know the pose of the vehicle. 69 Types of SLAM-Problems  Grid maps or scans [Lu & Milios, 97; Gutmann, 98: Thrun 98; Burgard, 99; Konolige & Gutmann, 00; Thrun, 00; Arras, 99; Haehnel, 01;…]  Landmark-based [Leonard et al., 98; Castelanos et al., 99: Dissanayake et al., 2001; Montemerlo et al., 2002;… 70 Problems in Mapping  Sensor interpretation  How do we extract relevant information from raw sensor data?  How do we represent and integrate this information over time?  Robot locations have to be estimated  How can we identify that we are at a previously visited place?  This problem is the so-called data association problem. 71 Occupancy Grid Maps  Introduced by Moravec and Elfes in 1985  Represent environment by a grid.  Estimate the probability that a location is occupied by an obstacle.  Key assumptions  Occupancy of individual cells (m[xy]) is independent Bel (mt )  P(mt | u1 , z 2 , ut 1 , zt )   Bel (mt[ xy ] ) x, y  Robot positions are known! 72 Incremental Updating of Occupancy Grids (Example) 73 Resulting Map Obtained with Ultrasound Sensors 74 75 Occupancy Grids: From scans to maps 76 Tech Museum, San Jose CAD map occupancy grid map 77 Introduction to Mobile Robotics Bayes Filter – Discrete Filters Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras 78 Probabilistic Localization 79 Grid-based Localization 80 Sonars and Occupancy Grid Map 81 Introduction to Mobile Robotics Bayes Filter – Particle Filter and Monte Carlo Localization Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras 82 Motivation  Recall: Discrete filter  Discretize the continuous state space  High memory complexity  Fixed resolution (does not adapt to the belief)  Particle filters are a way to efficiently represent non-Gaussian distribution  Basic principle  Set of state hypotheses (“particles”)  Survival-of-the-fittest 83 Sample-based Localization (sonar) Mathematical Description  Set of weighted samples State hypothesis Importance weight  The samples represent the posterior 85 Function Approximation  Particle sets can be used to approximate functions  The more particles fall into an interval, the higher the probability of that interval  How to draw samples form a function/distribution? 86 Rejection Sampling  Let us assume that f(x)<1 for all x  Sample x from a uniform distribution  Sample c from [0,1]  if f(x) > c otherwise keep the sample reject the sampe f(x’) c’ c OK f(x) x x’ 87 Particle Filters Sensor Information: Importance Sampling Bel ( x)   p( z | x) Bel  ( x)  p( z | x) Bel  ( x) w    p ( z | x)  Bel ( x) Robot Motion Bel  ( x)   p( x | u x' ) Bel ( x' ) , d x' Sensor Information: Importance Sampling Bel ( x)   p( z | x) Bel  ( x)  p( z | x) Bel  ( x) w    p ( z | x)  Bel ( x) Robot Motion Bel  ( x)   p( x | u x' ) Bel ( x' ) , d x' Particle Filter Algorithm  Sample the next generation for particles using the proposal distribution  Compute the importance weights : weight = target distribution / proposal distribution  Resampling: “Replace unlikely samples by more likely ones” 93 Particle Filter Algorithm 1. Algorithm particle_filter( St-1, ut-1 zt): 2. St  , 3.  0 For i  1 n Generate new samples 4. Sample index j(i) from the discrete distribution given by wt-1 5. Sample xti from p( xt | xt 1 , ut 1 ) using xtj(1i ) and ut 1 6. wti  p ( zt | xti ) Compute importance weight 7.     wti Update normalization factor 8. St  St  { xti , wti } Insert 9. For i  1 n 10. wti  wti /  Normalize weights 94 Mobile Robot Localization  Each particle is a potential pose of the robot  Proposal distribution is the motion model of the robot (prediction step)  The observation model is used to compute the importance weight (correction step) [For details, see PDF file on the lecture web page] 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 Initial Distribution 114 After Incorporating Ten Ultrasound Scans 115 After Incorporating 65 Ultrasound Scans 116 Estimated Path 117 Summary – Particle Filters  Particle filters are an implementation of recursive Bayesian filtering  They represent the posterior by a set of weighted samples  They can model non-Gaussian distributions  Proposal to draw new samples  Weight to account for the differences between the proposal and the target  Monte Carlo filter, Survival of the fittest, Condensation, Bootstrap filter 118 Summary – PF Localization  In the context of localization, the particles are propagated according to the motion model.  They are then weighted according to the likelihood of the observations.  In a re-sampling step, new particles are drawn with a probability proportional to the likelihood of the observation. 119 Introduction to Mobile Robotics SLAM: Simultaneous Localization and Mapping Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras Slides by Kai Arras and Wolfram Burgard Last update: June 2010 The SLAM Problem SLAM is the process by which a robot builds a map of the environment and, at the same time, uses this map to compute its location • Localization: inferring location given a map • Mapping: inferring a map given a location • SLAM: learning a map and locating the robot simultaneously 121 The SLAM Problem • SLAM is a chicken-or-egg problem: → A map is needed for localizing a robot → A pose estimate is needed to build a map • Thus, SLAM is (regarded as) a hard problem in robotics 122 The SLAM Problem • SLAM is considered one of the most fundamental problems for robots to become truly autonomous • A variety of different approaches to address the SLAM problem have been presented • Probabilistic methods rule • History of SLAM dates back to the mid-eighties (stone-age of mobile robotics) 123 The SLAM Problem Given: • The robot’s controls • Relative observations Wanted: • Map of features • Path of the robot 124 The SLAM Problem • Absolute robot pose • Absolute landmark positions • But only relative measurements of landmarks 125 SLAM Applications SLAM is central to a range of indoor, outdoor, in-air and underwater applications for both manned and autonomous vehicles. Examples: •At home: vacuum cleaner, lawn mower •Air: surveillance with unmanned air vehicles •Underwater: reef monitoring •Underground: exploration of abandoned mines •Space: terrain mapping for localization 126 SLAM Applications Indoors Undersea Space Underground 127 Map Representations Examples: Subway map, city map, landmark-based map Maps are topological and/or metric models of the environment 128 Map Representations • Grid maps or scans, 2d, 3d [Lu & Milios, 97; Gutmann, 98: Thrun 98; Burgard, 99; Konolige & Gutmann, 00; Thrun, 00; Arras, 99; Haehnel, 01;…] • Landmark-based [Leonard et al., 98; Castelanos et al., 99: Dissanayake et al., 2001; Montemerlo et al., 2002;… 129 Why is SLAM a hard problem? 1. Robot path and map are both unknown 2. Errors in map and pose estimates correlated 130 Why is SLAM a hard problem? Robot pose uncertainty • In the real world, the mapping between observations and landmarks is unknown (origin uncertainty of measurements) • Data Association: picking wrong data associations can have catastrophic consequences (divergence) 131 SLAM: Simultaneous Localization And Mapping • Full SLAM: p(x 0:t ,m | z1:t ,u1:t ) Estimates entire path and map! • Online SLAM: p(x t ,m | z1:t ,u1:t )    K  p(x 1:t ,m | z1:t ,u1:t ) dx1dx2 ...dxt1 Integrations (marginalization) typically done recursively, one at a time Estimates most recent pose and map! 132 Graphical Model of Full SLAM p( x1:t , m | z1:t , u1:t ) 133 Graphical Model of Online SLAM p( xt , m | z1:t , u1:t )      p( x1:t , m | z1:t , u1:t ) dx1 dx2 ...dxt 1 134 Graphical Model: Models "Motion model" "Observation model" 135 EKF SLAM: State representation • Localization 3x1 pose vector 3x3 cov. matrix • SLAM Landmarks are simply added to the state. Growing state vector and covariance matrix! 136 EKF SLAM: Building the Map Filter Cycle, Overview: 1. State prediction (odometry) 2. Measurement prediction 3. Observation 4. Data Association 5. Update 6. Integration of new landmarks 137 EKF SLAM: Building the Map • State Prediction Odometry: Robot-landmark crosscovariance prediction: (skipping time index k) 138 EKF SLAM: Building the Map • Measurement Prediction Global-to-local frame transform h 139 EKF SLAM: Building the Map • Observation (x,y)-point landmarks 140 EKF SLAM: Building the Map • Data Association Associates predicted measurements with observation ? (Gating) 141 EKF SLAM: Building the Map • Filter Update The usual Kalman filter expressions 142 EKF SLAM: Building the Map • Integrating New Landmarks State augmented by Cross-covariances: 143 EKF SLAM Map Correlation matrix 144 EKF SLAM Map Correlation matrix 145 EKF SLAM Map Correlation matrix 146 SLAM: Loop Closure • Loop closure is the problem of recognizing an already mapped area, typically after a long exploration path (the robot "closes a loop") • Structually identical to data association, but • high levels of ambiguity • possibly useless validation gates • environment symmetries • Uncertainties collapse after a loop closure (whether the closure was correct or not) 147 SLAM: Loop Closure • Before loop closure 148 SLAM: Loop Closure • After loop closure 149 SLAM: Loop Closure • By revisiting already mapped areas, uncertainties in robot and landmark estimates can be reduced • This can be exploited to "optimally" explore an environment for the sake of better (e.g. more accurate) maps • Exploration: the problem of where to acquire new information (e.g. depth-first vs. breadth first) → See separate chapter on exploration 150 KF-SLAM Properties (Linear Case) • The determinant of any sub-matrix of the map covariance matrix decreases monotonically as successive observations are made • When a new landmark is initialized, its uncertainty is maximal • Landmark uncertainty decreases monotonically with each new observation [Dissanayake et al., 2001] 151 KF-SLAM Properties (Linear Case) • In the limit, the landmark estimates become fully correlated [Dissanayake et al., 2001] 152 KF-SLAM Properties (Linear Case) • In the limit, the covariance associated with any single landmark location estimate is determined only by the initial covariance in the vehicle location estimate. [Dissanayake et al., 2001] 153 EKF SLAM Example: Victoria Park Syndey, Australia 154 Victoria Park: Landmarks [courtesy by E. Nebot] 155 156 Victoria Park: Estimated Trajectory [courtesy by E. Nebot] 157 Victoria Park: Landmarks [courtesy by E. Nebot] 158 EKF SLAM Example: Tennis Court [courtesy by J. Leonard] 159 EKF SLAM Example: Tennis Court odometry estimated trajectory [courtesy by John Leonard] 160 EKF SLAM Example: Line Features • KTH Bakery Data Set [Wulf et al., ICRA 04] 161 EKF-SLAM: Complexity • Cost per step: quadratic in n, the number of • • landmarks: O(n2) Total cost to build a map with n landmarks: O(n3) Memory: O(n2) Problem: becomes computationally intractable for large maps!  Approaches exist that make EKF-SLAM amortized O(n) / O(n2) / O(n2) D&C SLAM [Paz et al., 2006] 162 SLAM Techniques • EKF SLAM • FastSLAM • Graphical SLAM • Topological SLAM (mainly place recognition) • Scan Matching / Visual Odometry (only locally consistent maps) • Approximations for SLAM: Local submaps, Sparse extended information filters, Sparse links, Thin junction tree filters, etc. 163 Introduction to Mobile Robotics SLAM – Grid-based FastSLAM Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras 164 Grid-based SLAM  Can we solve the SLAM problem if no pre-defined landmarks are available?  Can we use the ideas of FastSLAM to build grid maps?  As with landmarks, the map depends on the poses of the robot during data acquisition  If the poses are known, grid-based mapping is easy (“mapping with known poses”) 165 Mapping with Known Poses  Mapping with known poses using laser range data 166 Rao-Blackwellization poses map observations & movements Factorization first introduced by Murphy in 1999 167 Rao-Blackwellization poses map observations & movements SLAM posterior Robot path posterior Mapping with known poses Factorization first introduced by Murphy in 1999 168 Rao-Blackwellization This is localization, use MCL Use the pose estimate from the MCL and apply mapping with known poses 169 Rao-Blackwellized Mapping  Each particle represents a possible trajectory of the robot  Each particle  maintains its own map and  updates it upon “mapping with known poses”  Each particle survives with a probability proportional to the likelihood of the observations relative to its own map 170 Particle Filter Example 3 particles map of particle 1 map of particle 2 map of particle 3 171 Problem  Each map is quite big in case of grid maps  Since each particle maintains its own map  Therefore, one needs to keep the number of particles small  Solution: Compute better proposal distributions!  Idea: Improve the pose estimate before applying the particle filter 172 Pose Correction Using Scan Matching Maximize the likelihood of the i-th pose and map relative to the (i-1)-th pose and map xˆt  arg max p( zt | xt , mˆ t 1 )  p( xt | ut 1 , xˆt 1 ) xt current measurement robot motion map constructed so far 173 Motion Model for Scan Matching Raw Odometry Scan Matching 174 FastSLAM with Improved Odometry  Scan-matching provides a locally consistent pose correction  Pre-correct short odometry sequences using scan-matching and use them as input to FastSLAM  Fewer particles are needed, since the error in the input in smaller [Haehnel et al., 2003] 175 176 177 FastSLAM with Scan-Matching 178 Conclusion (so far…)  The presented approach is a highly efficient algorithm for SLAM combining ideas of scan matching and FastSLAM  Scan matching is used to transform sequences of laser measurements into odometry measurements  This version of grid-based FastSLAM can handle larger environments than before in “real time” 179 What’s Next?  Further reduce the number of particles  Improved proposals will lead to more accurate maps  Use the properties of our sensor when drawing the next generation of particles 180 Intel Lab  15 particles  four times faster than real-time P4, 2.8GHz  5cm resolution during scan matching  1cm resolution in final map 181 Outdoor Campus Map  30 particles  250x250m2  1.75 km (odometry)  20cm resolution during scan matching  30cm resolution in final map 182 MIT Killian Court  The “infinite-corridor-dataset” at MIT 183 MIT Killian Court 184 MIT Killian Court - Video 185 Conclusion  The ideas of FastSLAM can also be applied in the context of grid maps  Utilizing accurate sensor observation leads to good proposals and highly efficient filters  It is similar to scan-matching on a per-particle base  The number of necessary particles and re-sampling steps can seriously be reduced  Improved versions of grid-based FastSLAM can handle larger environments than naïve implementations in “real time” since they need one order of magnitude fewer samples 186 More Details on FastSLAM  M. Montemerlo, S. Thrun, D. Koller, and B. Wegbreit. FastSLAM: A factored solution to simultaneous localization and mapping, AAAI02 (The classic FastSLAM paper with landmarks)  D. Haehnel, W. Burgard, D. Fox, and S. Thrun. An efcient FastSLAM algorithm for generating maps of large-scale cyclic environments from raw laser range measurements, IROS03 (FastSLAM on grid-maps using scan-matched input)  G. Grisetti, C. Stachniss, and W. Burgard. Improving grid-based slam with rao-blackwellized particle filters by adaptive proposals and selective resampling, ICRA05 (Proposal using laser observation, adaptive resampling)  A. Eliazar and R. Parr. DP-SLAM: Fast, robust simultainous localization and mapping without predetermined landmarks, IJCAI03 (A representation to handle big particle sets) 187 188 189 190 191 192 Introduction to Mobile Robotics Robot Motion Planning Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Kai Arras Slides by Kai Arras Last update July 2011 With material from S. LaValle, JC. Latombe, H. Choset et al., W. Burgard Robot Motion Planning J.-C. Latombe (1991): “…eminently necessary since, by definition, a robot accomplishes tasks by moving in the real world.” Goals  Collision-free trajectories  Robot should reach the goal location as fast as possible 194 Problem Formulation  The problem of motion planning can be stated as follows. Given:     A A A A start pose of the robot desired goal pose geometric description of the robot geometric description of the world  Find a path that moves the robot gradually from start to goal while never touching any obstacle 195 Problem Formulation Motion planning is sometimes also called piano mover's problem 196 Piano mover's problem 197 Configuration Space  Although the motion planning problem is defined in the regular world, it lives in another space: the configuration space  A robot configuration q is a specification of the positions of all robot points relative to a fixed coordinate system  Usually a configuration is expressed as a vector of positions and orientations 198 Configuration Space Rigid-body robot example Robot Reference direction y  Reference point x  3-parameter representation: q = (x,y,)  In 3D, q would be of the form (x,y,z,,b,g) 199 Configuration Space  Example: circular robot  C-space is obtained by sliding the robot along the edge of the obstacle regions "blowing them up" by the robot radius 200 Configuration Space  Example: polygonal robot, translation only  C-space is obtained by sliding the robot along the edge of the obstacle regions 201 Configuration Space  Example: polygonal robot, translation only Work space Configuration space Reference point  C-space is obtained by sliding the robot along the edge of the obstacle regions 202 Configuration Space  Example: polygonal robot, trans+rotation  C-space is obtained by sliding the robot along the edge of the obstacle regions in all orientations 203 Configuration Space Free space and obstacle region  With being the work space, the set of obstacles, the robot in configuration  We further define  : start configuration  : goal configuration 204 Configuration Space Then, motion planning amounts to  Finding a continuous path with  Given this setting, we can do planning with the robot being a point in C-space! 205 C-Space Discretizations  Continuous terrain needs to be discretized for path planning  There are two general approaches to discretize C-spaces:  Combinatorial planning Characterizes Cfree explicitely by capturing the connectivity of Cfree into a graph and finds solutions using search  Sampling-based planning Uses collision-detection to probe and incrementally search the C-space for solution 206 Combinatorial Planning  We will look at four combinatorial planning techniques     Visibility graphs Voronoi diagrams Exact cell decomposition Approximate cell decomposition  They all produce a road map  A road map is a graph in Cfree in which each vertex is a configuration in Cfree and each edge is a collision-free path through Cfree 207 Combinatorial Planning  Without loss of generality, we will consider a problem in with a point robot that cannot rotate. In this case:  We further assume a polygonal world qG qI 208 Visibility Graphs  Idea: construct a path as a polygonal line connecting qI and qG through vertices of Cobs  Existence proof for such paths, optimality  One of the earliest path planning methods qG qG qI qI  Best algorithm: O(n2 log n) 209 Generalized Voronoi Diagram  Defined to be the set of points q whose cardinality of the set of boundary points of Cobs with the same distance to q is greater than 1  Let us decipher this definition...  Informally: the place with the same maximal clearance from all nearest obstacles qI' qI qG' qG 210 Generalized Voronoi Diagram  Geometrically: two closest points p p one closest point q q clearance(q) q p p p  For a polygonal Cobs, the Voronoi diagram consists of (n) lines and parabolic segments  Naive algorithm: O(n4), best: O(n log n) 211 Voronoi Diagram  Voronoi diagrams have been well studied for (reactive) mobile robot path planning  Fast methods exist to compute and update the diagram in real-time for lowdim. C's  Pros: maximize clearance is a good idea for an uncertain robot  Cons: unnatural attraction to open space, suboptimal paths  Needs extensions 212 Exact Cell Decomposition  Idea: decompose Cfree into non-overlapping cells, construct connectivity graph to represent adjacencies, then search  A popular implementation of this idea: 1. Decompose Cfree into trapezoids with vertical side segments by shooting rays upward and downward from each polygon vertex 2. Place one vertex in the interior of every trapezoid, pick e.g. the centroid 3. Place one vertex in every vertical segment 4. Connect the vertices 213 Exact Cell Decomposition  Trapezoidal decomposition ( (a) (c) (b) (d) max)  Best known algorithm: O(n log n) where n is the number of vertices of Cobs 214 Approximate Cell Decomposition  Exact decomposition methods can be involved and inefficient for complex problems  Approximate decomposition uses cells with the same simple predefined shape qI qG qG qI Quadtree decomposition 215 Approximate Cell Decomposition  Exact decomposition methods can be involved and inefficient for complex problems  Approximate decomposition uses cells with the same simple predefined shape  Pros:     Iterating the same simple computations Numerically more stable Simpler to implement Can be made complete 216 Combinatorial Planning Wrap Up  Combinatorial planning techniques are elegant and complete (they find a solution if it exists, report failure otherwise)  But: become quickly intractable when C-space dimensionality increases (or n resp.)  Combinatorial explosion in terms of facets to represent , , and , especially when rotations bring in nonlinearities and make C a nontrivial manifold ➡ Use sampling-based planning Weaker guarantees but more efficient 217 Sampling-Based Planning  Abandon the concept of explicitly characterizing Cfree and Cobs and leave the algorithm in the dark when exploring Cfree  The only light is provided by a collisiondetection algorithm, that probes C to see whether some configuration lies in Cfree  We will have a look at  Probabilistic road maps (PRM) [Kavraki et al., 92]  Rapidly exploring random trees (RRT) [Lavalle and Kuffner, 99] 218 Probabilistic Road Maps  Idea: Take random samples from C, declare them as vertices if in Cfree, try to connect nearby vertices with local planner  The local planner checks if line-of-sight is collision-free (powerful or simple methods)  Options for nearby: k-nearest neighbors or all neighbors within specified radius  Configurations and connections are added to graph until roadmap is dense enough 219 Probabilistic Road Maps  Example specified radius Example local planner What means "nearby" on a manifold? Defining a good metric on C is crucial 220 Probabilistic Road Maps Good and bad news:  Pros:     Probabilistically complete Do not construct C-space Apply easily to high-dim. C's PRMs have solved previously unsolved problems qG Cobs Cobs Cobs qI qG  Cons:  Do not work well for some problems, narrow passages  Not optimal, not complete Cobs Cobs Cobs Cobs qI 221 Rapidly Exploring Random Trees  Idea: aggressively probe and explore the C-space by expanding incrementally from an initial configuration q0  The explored territory is marked by a tree rooted at q0 45 iterations 2345 iterations 222 From Road Maps to Paths  All methods discussed so far construct a road map (without considering the query pair qI and qG)  Once the investment is made, the same road map can be reused for all queries (provided world and robot do not change) 1. Find the cell/vertex that contain/is close to qI and qG (not needed for visibility graphs) 2. Connect qI and qG to the road map 3. Search the road map for a path from qI to qG 223 Sampling-Based Planning Wrap Up  Sampling-based planners are more efficient in most practical problems but offer weaker guarantees  They are probabilistically complete: the probability tends to 1 that a solution is found if one exists (otherwise it may still run forever)  Performance degrades in problems with narrow passages. Subject of active research  Widely used. Problems with high-dimensional and complex C-spaces are still computationally hard 224 Potential Field Methods  All techniques discussed so far aim at capturing the connectivity of Cfree into a graph  Potential Field methods follow a different idea: The robot, represented as a point in C, is modeled as a particle under the influence of a artificial potential field U U superimposes  Repulsive forces from obstacles  Attractive force from goal 225 Potential Field Methods  Potential function + =  Simply perform gradient descent  C-pace typically discretized in a grid 226 Potential Field Methods  Main problems: robot gets stuck in local minima  Way out: Construct local-minima-free navigation function ("NF1"), then do gradient descent (e.g. bushfire from goal)  The gradient of the potential function defines a vector field (similar to a policy) that can be used as feedback control strategy, relevant for an uncertain robot  However, potential fields need to represent Cfree explicitely. This can be too costly. 227 Robot Motion Planning  Given a road map, let's do search! 228 A* Search  A* is one of the most widely-known informed search algorithms with many applications in robotics  Where are we? A* is an instance of an informed algorithm for the general problem of search  In robotics: planning on a 2D occupancy grid map is a common approach 229 Search The problem of search: finding a sequence of actions (a path) that leads to desirable states (a goal) Uninformed search: besides the problem definition, no further information about the domain ("blind search") The only thing one can do is to expand nodes differently Example algorithms: breadth-first, uniformcost, depth-first, bidirectional, etc. 230 Search The problem of search: finding a sequence of actions (a path) that leads to desirable states (a goal) Informed search: further information about the domain through heuristics Capability to say that a node is "more promising" than another node Example algorithms: greedy best-first search, A*, many variants of A*, D*, etc. 231 Any-Angle A* Examples  A* vs. Theta* (len: path length, nhead = # heading changes) len: 30.0 nhead: 11 len: 24.1 nhead: 9 len: 28.9 nhead: 5 len: 22.9 nhead: 2 232 D* Search  Problem: In unknown, partially known or dynamic environments, the planned path may be blocked and we need to replan  Can this be done efficiently, avoiding to replan the entire path?  Idea: Incrementally repair path keeping its modifications local around robot pose  Several approaches implement this idea:  D* (Dynamic A*) [Stentz, ICRA'94, IJCAI'95]  D* Lite [Koenig and Likhachev, AAAI'02]  Field D* [Ferguson and Stentz, JFR'06] 233 D* Family  D* Lite produces the same paths than D* but is simpler and more efficient  D*/D* Lite are widely used  Field D* was running on Mars rovers Spirit and Opportunity (retrofitted in yr 3) Tracks left by a drive executed with Field D* 234 Still in Dynamic Environments...  Do we really need to replan the entire path for each obstacle on the way?  What if the robot has to react quickly to unforeseen, fast moving obstacles?  Even D* Lite can be too slow in such a situation  Accounting for the robot shape (it's not a point)  Accounting for kinematic and dynamic vehicle constraints, e.g.  Decceleration limits,  Steering angle limits, etc. 235 Collision Avoidance  This can be handled by techniques called collision avoidance (obstacle avoidance)  A well researched subject, different approaches exist:  Dynamic Window Approaches [Simmons, 96], [Fox et al., 97], [Brock & Khatib, 99]  Nearness Diagram Navigation [Minguez et al., 2001, 2002]  Vector-Field-Histogram+ [Ulrich & Borenstein, 98]  Extended Potential Fields [Khatib & Chatila, 95] 236 Collision Avoidance  Integration into general motion planning?  It is common to subdivide the problem into a global and local planning task:  An approximate global planner computes paths ignoring the kinematic and dynamic vehicle constraints  An accurate local planner accounts for the constraints and generates (sets of) feasible local trajectories ("collision avoidance")  What do we loose? What do we win? 237 Two-layered Architecture low frequency Planning map sub-goal Collision Avoidance sensor data high frequency motion command robot 238 239 240 241 Development of intelligent systems (RInS) Object detection in 3D Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Academic year: 2021/22 Development of intelligent systems Detection of obstacles and objects Development of intelligent systems, Object detection in 3D 2 3D perception Development of intelligent systems, Object detection in 3D 3 Development of intelligent systems, Object detection in 3D Radu Bogdan Rusu, Nico Blodow, Willow Garage, PCL 4 Detection of planes Development of intelligent systems, Object detection in 3D 5 RANSAC  Random Sampling Consensus [Fischler, Bolles ’81] Development of intelligent systems, Object detection in 3D 6 RANSAC Development of intelligent systems, Object detection in 3D 7 RANSAC Development of intelligent systems, Object detection in 3D 8 RANSAC in PCL Development of intelligent systems, Object detection in 3D 9 RANSAC in PCL Development of intelligent systems, Object detection in 3D 10 Detection of cylinders http://pointclouds.org/documentation/tutorials/cylinder_segmentation.php Development of intelligent systems, Object detection in 3D 11 Detection of cylinders  Noisy data Development of intelligent systems, Object detection in 3D 12 Detection of objects Development of intelligent systems, Object detection in 3D 13 Collision map Development of intelligent systems, Object detection in 3D 14 Cylinder detection Development of intelligent systems, Object detection in 3D 15 Ring detection Development of intelligent systems, Object detection in 3D 16 Development of intelligent systems (RInS) Colours Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Literature: W. Burger, M. J. Burge (2008). Digital Image Processing, chapter 12 Academic year: 2021/22 Development of intelligent systems Colour images Development of intelligent systems, Colours 2 Colour images  Sometimes colours include meaningful information! Development of intelligent systems, Colours 3 RGB colour images  RGB colour scheme encodes colours as combinations of three basics colours: red, green and blue  Very frequently used  Additive colour system Development of intelligent systems, Colours 4 RGB colour space  Every colour is a point in the 3D RGB space Development of intelligent systems, Colours 5 RGB channels Development of intelligent systems, Colours 6 Conversion to grayscale images  Simple conversion:  Human eye perceives red and green as brighter than blue, hence we can use the weighted average:  Grayscale RGB images have all three components equal: Development of intelligent systems, Colours 7 HSV colour space  Hue, Saturation, Value Development of intelligent systems, Colours 8 HSV channels Development of intelligent systems, Colours 9 Conversion from RGB to HSV 255 Development of intelligent systems, Colours 10 Algorithm Development of intelligent systems, Colours 11 Conversion from HSV to RGB 256 Development of intelligent systems, Colours 12 Examples Development of intelligent systems, Colours 13 Other colour spaces  HLS  TV colour spaces  YUV  YIQ  YCbCr  Colour spaces for print  CMY  CMYK  Colorimetric colour spaces     CIE XYZ CIE YUV, YU‘V‘, L*u*v, YCbCr CIE L*a*b* sRGB Development of intelligent systems, Colours 14 3D colour histograms  3 components -> 3D histogram  High space complexity, „sparse“ Development of intelligent systems, Colours 15 1D colour histograms  1 D histograms of the individual components  Do not model correlations between individual colour components Development of intelligent systems, Colours 16 2D colour histograms  Calculate pairs of 2D histograms  Encompass at least a partial correlation between the individual components Development of intelligent systems, Colours 17 Algorithm Development of intelligent systems, Colours 18 Examples Development of intelligent systems, Colours 19 Object colours  Rings of different colours  Cylinders of different colours Development of intelligent systems, Colours 20 Colour recognition  Detect and segment the object  in 2D or 3D  Modelling colours  Probability distribution  Gaussian, mixture of Gaussians  Train a classifier  SVM, ANN, kNN,…  In 1D, 2D or 3D space  RGB, HSV and other colour spaces  Working with the individual pixels or histograms  Working with images Development of intelligent systems, Colours 21 Development of intelligent systems (RInS) Robot manipulation Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Literature: Tadej Bajd (2006) Osnove robotike, poglavje 4 Anže Rezelj (2017) Razvoj nizkocenovnega lahkega robotskega manipulatorja Academic year: 2021/22 Development of intelligent systems Robot manipulator  Industrial robot as defined by the standard ISO 8373: An automatically controlled, reprogrammable, multipurpose manipulator programmable in three or more axes, which can be either fixed in place or mobile for use in industrial automation applications. Development of intelligent systems, Robot manipulation 2 Characteristics  Closed-loop control  Electrical or hydraulic motors  Sensors  Proprioceptive: rotation encoders, measurement od distance, speed  Exteroceptive: tactile sensors, robot vision  Reprogrammable:  designed so that the programmed motions or auxiliary functions can be changed without physical alteration  Multipurpose:  capable of being adapted to a different application with physical alteration  Physical alteration:  alteration of the mechanical system  fixed or mobile robots  Axis: direction used to specify the robot motion in a linear or rotary mode  3 or more DOF Development of intelligent systems, Robot manipulation 3 Robot manipulator  Arm+wrist+end effector/gripper  6DOF – can put an object in an arbitrary pose  arm positions the object into the desired position  wrist rotates it into the desired orientation  gripper grasps the object Development of intelligent systems, Robot manipulation 4 Robot arm  Serial chain of three links  connected by joints  Revolute/rotational joint  Prismatic/translational joint Development of intelligent systems, Robot manipulation 5 Robot arm types  Joints  Revolute/rotational  Prismatic/translational  Axis of two neigbouring links  Parallel  Perpendicular  3DOF  In practice typically five different arms:      Anthropomorphic Spherical SCARA Cylindrical Cartesian Development of intelligent systems, Robot manipulation 6 Anthropomorphic robot arm  Three rotational joints (RRR)  Workspace: sphere-like  Resembles a human arm Development of intelligent systems, Robot manipulation 7 Spherical robot arm  Two rotational, one translational joint (RRT)  Workspace: sphere-like Development of intelligent systems, Robot manipulation 8 SCARA robot arm  Selective Articulated Robot for Assembly  Two rotational, one translational joint (RRT)  Workspace: cylinder-like Development of intelligent systems, Robot manipulation 9 Cylindrical robot arm  One rotational, two translational joints (RTT)  Workspace: cylinder Development of intelligent systems, Robot manipulation 10 Cartesian robot arm  Three translational joints (TTT)  Workspace: cuboid Development of intelligent systems, Robot manipulation 11 Robot wrist  Rotates the object in an arbitrary orientation  Three rotational joints (RRR)  Sometimes also one or two suffice  Links should be as short as possible Development of intelligent systems, Robot manipulation 12 Robot end-effector  The final link of the robot manipulator  Grippers with fingers  With two fingers  With more than two fingers  Other type of grippers  Vacuum  Magnetic  Perforation  Other tools as end-effectors  Welding gun  Spray painting gun Development of intelligent systems, Robot manipulation 13 Robot workspace  Reachable workspace  The end-effector can reach every point in this space  Dexterous workspace  The end-effector can reach every point in tis space from the arbitrary orientation Development of intelligent systems, Robot manipulation 14 Kinematics  Base coordinate frame [X1,Y1,Z1]  Usually also world coordinate frame  Used for defining of the robotic task  End-effector reference frame [Xm,Ym,Zm]  End-effector position  Vector between the origins of the coordinate frames  Object orientation  Three angles  Internal robot coordinates / joint variables  Joint states (angles, translations)  Uniquely describe the pose of the robot  Direct kinematics  Determine the external robot coordinates from the internal coordinates  Inverse kinematics  Determine the internal robot coordinates from the external coordinates Development of intelligent systems, Robot manipulation 15 Geometrical robot model  Robot manipulator = a serial chain of segments connected by joints  Every joint can be either rotational or translational  1DOF – 1 internal coordinate  Geometrical robot model describes  the pose of the last segment of the robot (end-effector) expressed in the reference (base) frame  depending on the current internal coordinates Development of intelligent systems, Robot manipulation 16 Geometrical robot model  Geometrical model can be expressed by a homogenuous transformation:  p : position of the end effector in the reference coordinate frame  n, s, a : unit vectors of the end-effector coordinate frame:  a: approach  s: sliding  n: normal  q : vector of internal coordinates Development of intelligent systems, Robot manipulation 17 Poses of segments  Every joint connects two neighbouring segments/links  Determine the transformation between them  Recursively build the full model for the entire robot  Coordinate frames can be arbitrarily attached to the individual segments  Denavit – Hartenberg rules simplify computation of the geometrical robot model  Determine the pose of the i-th c.f. with respect to the pose of the (i-1)-th c.f.  Axis i connects segments (i-1) and i Development of intelligent systems, Robot manipulation 18 Denavit – Hartenberg rules  Describe the coordinate frame of the i-th segment (having the joint i+1): 1. Define the axis zi through the axis of the joint i+1 2. Find the common normal, perpendicular to the axes zi-1 and zi    Position the origin of Oi into the intersection of the axis zi with the common normal Position the origin of Oi` into the intersection of the axis zi-1 with the common normal If the axes are parallel, position the origin anywhere 3. Position the axis xi on a common normal in a way, that it is oriented from the joint i towards the joint i+1  If the axis zi-1 and zi intersect, orient the axis xi perpendicular to the plane defined by the axes zi-1 in zi 4. Determine the axis yi in a way that gives the rigt-handed c.f.  Similarly we also describe (have already described) the coordinate frame of the segment (i-1)  The origin Oi-1 is determined by the intersection of the common normal of the axes i-1 and i  The axis zi-1 is oriented along the i-th axis  xi-1 is oriented along the common normal and directed from the joint i-1 towards the joint i Development of intelligent systems, Robot manipulation 19 Graphical illustration of DH parameters Development of intelligent systems, Robot manipulation 20 Denavit – Hartenberg parameters  The pose of i-th c.f. with respect to (i-1)-th c.f. is determined by 4 parameters: 1. 2. 3. 4. ai di αi θi – – – – distance between Oi and Oi` along xi distance between Oi-1 and Oi` along zi-1 angle between zi-1 and zi around xi angle between xi-1 and xi around zi-1 Development of intelligent systems, Robot manipulation 21 Denavit – Hartenberg parameters Development of intelligent systems, Robot manipulation 22 Denavit – Hartenberg parameters  ai and αi are always constant  They depend on the geometry of the robot, the links between the joints, etc.  They do not change during the operation of the robot  One of the two remaining parameters is a variable  Θi, if the i-th joint is rotational  di, if the i-th joint is translational Development of intelligent systems, Robot manipulation 23 Denavit – Hartenberg parameters  Illustration  r=a  n=i Wikipedia Development of intelligent systems, Robot manipulation 24 Denavit – Hartenberg parameters  Video: http://en.wikipedia.org/wiki/Denavit-Hartenberg_Parameters Development of intelligent systems, Robot manipulation 25 Exceptions  Some exceptions in certain situations can be used to simplify the process:  Axis zi and zi-1 are parallel -> di=0  Axis zi and zi-1 intersect -> Oi is in the intersection  In case of the base (0-th) segment: only the axis z0 is defined -> put the origin of O0 in the first joint -> align x0 and x1  In case of end-effector (n-th c.f.): Only axis xn is defined; it is perpendicular to zn-1 -> zn should be parallel to zn-1  In case of translational joint: -> orient the axis zi-1 in the direction of translation -> position Oi-1 at the beginning of translation Development of intelligent systems, Robot manipulation 26 Denavit – Hartenberg transformation  Transformation between the i-th and (i-1)-th c.f.: 1. Take the c.f. Oi-1 attached to the segment (i-1) 2. Translate it for di and rotate it for Θi along and around zi-1, to align it with the c.f. Oi` 3. Translate c.f. Oi‘ for ai in rotate it for αi along and around xi`, to align it with the c.f. Oi 4. DH transformation is obtained by postmultiplication of both transformation matrices  Function of a single variable:   Θi for rotational joint di for translational joint Development of intelligent systems, Robot manipulation 27 Calculation of the geometrical robot model 1. 2. 3. 4. Set the coordinate frames for all segments Define the table of DH parameters Calculate DH transformations for Calculate the geometrical model: Development of intelligent systems, Robot manipulation for all segments 28 Using geometrical model  Geometrical robot model gives the pose of the last segment of the robot (endeffector) expressed in the reference (base) frame  Geometrical robot model defines the pose (position and orientation) of the endeffector depending on the current internal coordinates q Development of intelligent systems, Robot manipulation 29 Anthropomorphic robot manipulator Development of intelligent systems, Robot manipulation 30 Anthropomorphic robot manipulator Development of intelligent systems, Robot manipulation 31 Stanford robot manipulator Development of intelligent systems, Robot manipulation 32 Stanford robot manipulator Development of intelligent systems, Robot manipulation 33 Spherical robot wrist  Usually attached to the end of the robot arm  All three axes of the rotational joints intersect in the same point Development of intelligent systems, Robot manipulation 34 Spherical robot wrist Development of intelligent systems, Robot manipulation 35 Stanford manipulator with the wrist Development of intelligent systems, Robot manipulation 36 Stanford manipulator with the wrist Development of intelligent systems, Robot manipulation 37 Inverse kinematics model  Direct kinematics defines the pose of the end-effector depending on the internal coordinates   Where will the end-effector move T(q)  The pose of the end-effector is uniquely determined Inverse kinematics defines the internal coordinates that would bring the robot endeffector in the desired pose  How to move the end-effector to reach the desired pose  Challenging problem: q(T)  Nonlinear equations  The solution is not uniquely defined  Several solutions  Sometimes even infinite number of solutions  Sometimes the solution does not exist  Take into account several criteria that determine which solution is optimal  Sometimes we can get analytical solution, sometimes only numerical are possible Development of intelligent systems, Robot manipulation 38 ViCoS LCLWOS robot manipulator Development of intelligent systems, Robot manipulation 39 Requirements Development of intelligent systems, Robot manipulation 40 Related work Development of intelligent systems, Robot manipulation 41 Robot manipulator Development of intelligent systems, Robot manipulation 42 Forward model DH parameters: Development of intelligent systems, Robot manipulation 43 Transformation Development of intelligent systems, Robot manipulation 44 Frame  3D printed Development of intelligent systems, Robot manipulation 45 Frame parts Development of intelligent systems, Robot manipulation 46 Motors  Servo motors Development of intelligent systems, Robot manipulation 47 Motors Development of intelligent systems, Robot manipulation 48 Upgrading servomotors     New control circuit Potentiometer OpenServo Current protection Development of intelligent systems, Robot manipulation 49 PID controller  Proportional, integral and derivative part Development of intelligent systems, Robot manipulation 50 AD converter     Increasing resolution Multiple sampling and decimation Resolution increased from 10 to 12 bits Sampling frequency 256 Hz Development of intelligent systems, Robot manipulation 51 Communication  I2C interface  I2C bus  OpenServo  Communication with microcontroller  OpenServoRobot  Robot model (DH parameters)  Communication with application Development of intelligent systems, Robot manipulation 52 Workspace Development of intelligent systems, Robot manipulation 53 Theoretical accuracy  Expected deviation of the estimated position from the reference position of the end effector  Only considering motor errors  Theoretical upper limit of precision Development of intelligent systems, Robot manipulation 54 Theoretical accuracy Development of intelligent systems, Robot manipulation 55 Theoretical accuracy Development of intelligent systems, Robot manipulation 56 Empiciral repetabilty Development of intelligent systems, Robot manipulation 57 Calibration Grasping a cube (100 times): Mean error: Development of intelligent systems, Robot manipulation 58 Characteristics Development of intelligent systems, Robot manipulation 59 Integration in ROS Development of intelligent systems, Robot manipulation 60 Integration in Manus Development of intelligent systems, Robot manipulation 61 Integration with programming languages  Matlab  Python  Blockly Development of intelligent systems, Robot manipulation 62 Communication using VGA and USB cable  VGA cable and I2C protocol  USB port Development of intelligent systems, Robot manipulation 63 Registration with camera Development of intelligent systems, Robot manipulation 64 Augmented reality Development of intelligent systems, Robot manipulation 65 Multi-level teaching approach Development of intelligent systems, Robot manipulation 66 Video Development of intelligent systems, Robot manipulation 67 Mobile manipulation Development of intelligent systems, Robot manipulation 68 Vision-based control Moving the end effector to the desired pose determined by the pose of an object  Camera on the robot  Fixed camera Development of intelligent systems, Robot manipulation 69 Vision-based control  Position-based servoing  Explicit control  In world coordinate frame  Image-based servoing  Implicit control  In image coordinate frame Development of intelligent systems, Robot manipulation 70 Vision-based control Development of intelligent systems, Robot manipulation 71 Vision-based control  Mobile manipulation  Joint control of mobile robot and robot manipulator Development of intelligent systems, Robot manipulation 72 Development of intelligent systems (RInS) Object recognition with Convolutional Neural Networks Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Academic year: 2021/22 Media hype Development of intelligent systems, Object recognition with CNNs 2 Superior perfomance 1k categories 1,3M images Top5 classification ILSVRC results 30 28,2 25,8 25 20 16,4 15 Deep learning era 11,7 10 6,7 5 3,6 3,1 2015 2016 2,2 0 2010 Development of intelligent systems, Object recognition with CNNs 2011 2012 2013 2014 2017 3 New deep learning era  More data!  More computing power - GPU!  Better learning algorithms! Development of intelligent systems, Object recognition with CNNs 4 New deep learning era ICCV 2019, Seoul, Korea, 27. 10. - 2. 11. 2019 Development of intelligent systems, Object recognition with CNNs 5 Machine learning in computer vision  Conventional approach feature extraction PCA, LDA, CCA, HOG, SIFT, SURF, ORB, … Development of intelligent systems, Object recognition with CNNs features classification learning model class kNN, SVM, ANN, AdaBoost, … 6 Deep learning in computer vision  Conventional machine learning approach in computer vision feature extraction features classification learning model class  Deep learing approach classification deep learning Development of intelligent systems, Object recognition with CNNs class Deep model 7 Deep learning – the main concept Oseba Person Kolo Bike Car Avto Development of intelligent systems, Object recognition with CNNs 8 End to end learning  Representations as well as classifier are being learned Development of intelligent systems, Object recognition with CNNs 9 Perceptron       Rosenblatt, 1957 Binary inputs and output Weights Threshold Bias Very simple! Development of intelligent systems, Object recognition with CNNs 10 Sigmoid neurons  Real inputs and outputs from interval [0,1]  Activation function: sgimoid function  output = Development of intelligent systems, Object recognition with CNNs 11 Sigmoid neurons  Small changes in weights and biases causes small change in output  Enables learning! Development of intelligent systems, Object recognition with CNNs 12 Feedfoward neural networks  Network architecture: Development of intelligent systems, Object recognition with CNNs 13 Example: recognizing digits  MNIST database of handwritten digits      28x28 pixes (=784 input neurons) 10 digits 50.000 training images 10.000 validation images 10.000 test images Development of intelligent systems, Object recognition with CNNs 14 Example code: Feedforward  Code from https://github.com/mnielsen/neural-networks-and-deep-learning/archive/master.zip or https://github.com/mnielsen/neural-networks-and-deep-learning git clone https://github.com/mnielsen/neural-networks-and-deep-learning.git  or https://github.com/chengfx/neural-networks-and-deep-learning-for-python3 (for Python 3) Development of intelligent systems, Object recognition with CNNs 15 Loss function  Given: for all training images  Loss function:  (mean sqare error – quadratic loss function)  Find weigths w and biases b that for given input x produce output a that minimizes Loss function C Development of intelligent systems, Object recognition with CNNs 16 Gradient descend  Find minimum of  Change of C:  Gradient of C:  Change v in the opposite direction of the gradient:  Algorithm: Learning rate  Initialize v  Until stopping criterium riched  Apply udate rule Development of intelligent systems, Object recognition with CNNs 17 Gradient descend in neural networks  Loss function  Update rules:  Consider all training samples  Very many parameters => computationaly very expensive  Use Stochastic gradient descend instead Development of intelligent systems, Object recognition with CNNs 18 Stochastic gradient descend  Compute gradient only for a subset of m training samples:  Mini-batch:  Approximate gradient:  Update rules:  Training: 1. Initialize w and b 2. In one epoch of training keep randomly selecting one mini-batch of m samples at a time (and train) until all training images are used 3. Repeat for several epochs Development of intelligent systems, Object recognition with CNNs 19 Example code: SGD Development of intelligent systems, Object recognition with CNNs 20 Backpropagation  All we need is gradient of loss function  Rate of change of C wrt. to change in any weigt  Rate of change of C wrt. to change in any biase  How to compute gradient?  Numericaly  Simple, approximate, extremely slow   Analyticaly for entire C  Fast, exact, nontractable   Chain individual parts of netwok  Fast, exact, doable  Backpropagation! Development of intelligent systems, Object recognition with CNNs 21 Main principle  We need the gradient of the Loss function  Two phases:  Forward pass; propagation: the input sample is propagated through the network and the error at the final layer is obtained  Backward pass; weight update: the error is backpropagated to the individual levels, the contribution of the individual neuron to the error is calculated and the weights are updated accordingly Development of intelligent systems, Object recognition with CNNs 22 Learning strategy  To obtain the gradient of the Loss function :  For every neuron in the network calculate error of this neuron  This error propagates through the netwok causing the final error  Backpropagate the final error to get all  Obtain all and from Development of intelligent systems, Object recognition with CNNs 23 Equations of backpropagation  BP1: Error in the output layer:  BP2: Error in terms of the error in the next layer:  BP3: Rate of change of the cost wrt. to any bias:  BP4: Rate of change of the cost wrt. to any weight: Development of intelligent systems, Object recognition with CNNs 24 Backpropagation algorithm  Input x: Set the corresponding activation for the input layer  Feedforward: For each compute  Output error : Compute the output error  Backpropagate the error: For each compute  Output the gradient: Development of intelligent systems, Object recognition with CNNs 25 Backpropagation and SGD For a number of epochs Until all training images are used Select a mini-batch of training samples For each training sample in the mini-batch Input: set the corresponding activation Feedforward: for each compute and Output error: compute Backpropagation: for each compute Gradient descend: for each Development of intelligent systems, Object recognition with CNNs and update: 26 Example code: Backpropagation Development of intelligent systems, Object recognition with CNNs 27 Local computation values gradient Development of intelligent systems, Object recognition with CNNs 28 Activation and loss functions Activation function Loss function Linear Quadratic Sigmoid Binary cross-entropy Softmax Categorical Cross-entropy Other Custom Development of intelligent systems, Object recognition with CNNs 29 Activation functions Sigmoid Leaky ReLU tanh ELU ReLU tanh(x) max(0.1x, x) max(0,x) Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 30 Overfitting  Huge number of parameters -> danger of overfitting  Use validation set to determine overfitting and early stopping  Hold out method overfitting overfitting early stopping 1,000 MNIST training images Development of intelligent systems, Object recognition with CNNs 50,000 MNIST training images 31 Regularization  How to avoid overfitting:  Increase the number of training images   Decrease the number of parameters   Regularization   Regularization:     L2 regularization L1 regularization Dropout Data augmentation Development of intelligent systems, Object recognition with CNNs 32 Regularisation  How to avoid overfitting:  Increase the number of training images   Decrease the number of parameters   Regularization            Data Augmentation L1 regularisation L2 regularisation Dropout Batch Normalization DropConnect Fractional Max Pooling Stochastic Depth Cutout / Random Crop Mixup Development of intelligent systems, Object recognition with CNNs [Wan et al. 2013] [Huang et al. 2016] [Graham, 2014] 33 Data augmentation  Use more data! Development of intelligent systems, Object recognition with CNNs  Synthetically generate new data  Apply different kinds of transformations: translations, rotations, elastic distortions, appearance modifications (intensity, blur)  Operations should reflect real-world variation 34 Parameter updates  Different schemes for updating gradient       Gradient descend Momentum update Nesterov momentum AdaGrad update RMSProp update Adam update  Learning rate decay Image credits: Alec Radford Development of intelligent systems, Object recognition with CNNs 35 Setting up the network  Set up the network  Coarse-fine cross-validation in stages  Only a few epochs to get a rough idea  Even on a smaller problem to speed up the process  Longer running time, finer search,…  Cross-validation strategy  Check various parameter settings  Always sample parameters  Check the results, adjust the range  Hyperparameters to play with:  network architecture  learning rate, its decay schedule, update type  regularization (L2/Dropout strength)…  Run multiple validations simultaneously  Actively observe the learning progress Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 36 Convolutional neural networks  From feedforward fully-connected neural networks  To convolutional neural networks Development of intelligent systems, Object recognition with CNNs 37 example Convolution  Convolution operation:  Discrete convolution:  Two-dimensional convolution:  Convolution is commutative:  Cross-correlation: Development of intelligent systems, Object recognition with CNNs flipped kernel 38 Convolutional neural networks  Data in vectors, matrices, tensors  Neigbourhood, spatial arrangement  2D: Images,time-fequency representations  1D: sequential signals, text, audio, speech, time series,…  3D: volumetric images, video, 3D grids Development of intelligent systems, Object recognition with CNNs 39 Convolution layer * Development of intelligent systems, Object recognition with CNNs 40 Convolution layer * * σ σ 4 3 * σ 8 8× 4× 3 Development of intelligent systems, Object recognition with CNNs 4 41 Sparse connectivity  Local connectivity – neurons are only locally connected (receptive field)  Reduces memory requirements  Improves statistical efficiency  Requires fewer operations The receptive field of the units in the deeper layers is large => Indirect connections! from below Development of intelligent systems, Object recognition with CNNs from above 42 Parameter sharing  Neurons share weights!  Tied weights  Every element of the kernel is used at every position of the input  All the neurons at the same level detect the same feature (everywhere in the input)  Greatly reduces the number of parameters!  Equivariance to translation  Shift, convolution = convolution, shift  Object moves => representation moves  Fully connected network with an infinitively strong prior over its weights  Tied weights  Weights are zero outside the kernel region => learns only local interactions and is equivariant to translations Development of intelligent systems, Object recognition with CNNs 43 Convolutional neural network [From recent Yann LeCun slides] Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 44 Convolutional neural network one filter => one activation map example 5x5 filters (32 total) input image: Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 45 Pooling layer  makes the representations smaller and more manageable  operates over each activation map independently  downsampling Example: Max pooling Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 46 Pooling  Max pooling introduces translation invariance Development of intelligent systems, Object recognition with CNNs  Pooling with downsampling  Reduces the representation size  Reduces computational cost  Increases statistical efficiency 47 CNN layers  Layers used to build ConvNets:  INPUT: raw pixel values  CONV: convolutional layer  (BN: batch nornalisation)  (ReLU:) introducing nonlinearity  POOL: downsampling  FC: for computing class scores  SoftMax Development of intelligent systems, Object recognition with CNNs 48 CNN architecture  Stack the layers in an appropriate order Babenko et. al. Hu et. al. Development of intelligent systems, Object recognition with CNNs 49 CNN architecture Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 50 Typical solution Development of intelligent systems, Object recognition with CNNs 51 Network architecture  Training the model  Inference Development of intelligent systems, Object recognition with CNNs 52 Example implementation in TensorFlow Segmentation network Classification network Development of intelligent systems, Object recognition with CNNs 53 Case study – LeNet-5 [LeCun et al., 1998] Conv filters were 5x5, applied at stride 1 Subsampling (Pooling) layers were 2x2 applied at stride 2 i.e. architecture is [CONV-POOL-CONV-POOL-CONV-FC] Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 54 Case study - AlexNet [Krizhevsky et al. 2012] http://fromdata.org/2015/10/01/imagenet-cnn-architecture-image/ CONV3 INPUT CONV1 CONV2 POOL1 POOL2 NORM1 NORM2 Development of intelligent systems, Object recognition with CNNs CONV4 FC6 CONV5 POOL3 FC7 FC8 Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 55 Case studay - VGGNet [Simonyan and Zisserman, 2014] best model Only 3x3 CONV stride 1, pad 1 and 2x2 MAX POOL stride 2 11.2% top 5 error in ILSVRC 2013 -> 7.3% top 5 error Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 56 Case study - GoogLeNet [Szegedy et al., 2014] Inception module ILSVRC 2014 winner (6.7% top 5 error) Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 57 Case study - ResNet spatial dim. only 56x56! - Development of intelligent systems, Object recognition with CNNs Batch Normalization after every CONV layer Xavier/2 initialization from He et al. SGD + Momentum (0.9) Learning rate: 0.1, divided by 10 when validation error plateaus Mini-batch size 256 Weight decay of 1e-5 ILSVRC 2015 winner No dropout used (3.6% top 5 error) Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 58 DenseNet  Densely Connected Convolutional Networks Huang et al. 2017  Every layer connected to every other layer in a feed-forward fashion  Dense connectivity  Model compactness  Strong gradient flow  Implicit deep supervision  Feature reuse Development of intelligent systems, Object recognition with CNNs 59 MobileNets  Efficient Convolutional Neural Networks for Mobile Applications Howard et al. 2017  Efficient models for mobile and embedded vision applications  Depthwise separable convolution:  Depthwise convolution  Pointwise (1x1) convolution  MobileNetV2: Inverted Residuals and Linear Bottlenecks Sandler et al. 2018  MobileNetV3: NAS+ NetAdapt Development of intelligent systems, Object recognition with CNNs Howard et al. 2019 60 NASNet  Neural Architecure Search [Zoph et al. 2018]  Search the space of architetures to find the optimal one given available resources  500 GPUs across 4 days resulting in 2,000 GPU-hours on NVidia P100 Available operations to select from: Best convolutional cells (NASNet-A) for CIFAR-10  Other architecture search methods:  AmoebaNet, Real et al., 2018  MoreMNAS, Chu et. al, 2019, … Development of intelligent systems, Object recognition with CNNs 61 EfficientNet  Scaling the network in [Tan and Le, 2019] Development of intelligent systems, Object recognition with CNNs 62 Architectures overview  Date of publication, main type [Hoeser and Kuenzer, 2020] Development of intelligent systems, Object recognition with CNNs 63 Analysis of DNN models [Canziani et al., 2017] Development of intelligent systems, Object recognition with CNNs 64 Pretrained models Development of intelligent systems, Object recognition with CNNs 65 Transformers [Vaswani et.al, NIPS 2017] Development of intelligent systems, Object recognition with CNNs [Khan et.al, 2021] 66 ViT - Vision Transformer  AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE [Dosovitskiy et.al, Google, 2020, ICLR 2021] Development of intelligent systems, Object recognition with CNNs 67 Transfer learning  If you don‘t have enough data use pretrained models! 1. Train on Imagenet 2. Small dataset: feature extractor 3. Medium dataset: finetuning more data = retrain more of the network (or all of it) Freeze these Freeze these tip: use only ~1/10th of the original learning rate in finetuning top layer, and ~1/100th on intermediate layers Train this Train this Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 68 Two stage object detection and recognition very fast efficient Face detection HOG+SVM AdaBoost SSD „Scarlet“ Face recognition CNN PCA/LDA could be slower computationally more complex Development of intelligent systems, Object recognition with CNNs 69 Object detection and recognition  Two stage approach:  Detection of region proposals  Recognition of the individual region proposals Development of intelligent systems, Object recognition with CNNs 70 Object detection in RInS  Information in circles  -> detecting circles as region proposals (Region Of Interests)  Rectify ROIs  Recognize the content of ROIs 2 6 ROIs  Rectification using homography Development of intelligent systems, Object recognition with CNNs 71 Homography  Two views on the same (planar) object:  Homography: plane to plane mapping Slide credit: Matej Kristan Development of intelligent systems, Object recognition with CNNs 72 Computing homography  Four corresponding points: 𝒙 𝒙′ 𝑤𝒙′ = 𝑯𝒙  H11  x'   w  y'    H 21  H 31  1   H12 H 22 H 32 H13   x   H 23   y  H 33   1   The elements of the matrix 𝑯 can be computed using Direct Linear Transform (DLT)! Slide credit: Matej Kristan Development of intelligent systems, Object recognition with CNNs 73 Application of homography Homography mapping between part of the image and a sqare Flagellation of Christ (Piero della Francesca) Slide credit: Antonio Criminisi Development of intelligent systems, Object recognition with CNNs 74 Two-stage detectors First stage: Run once per image - Backbone network - Region proposal network Second stage: Run once per region -Fei-Fei Crop features: RoI pool / align Li, Ranjay Krishna, Danfei Xu - Predict object class - Prediction bbox offset 75 Ren et al, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, NIPS 2015 Figure copyright 2015, Ross Girshick; reproduced with permission Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Ranjay Krishna, Danfei Xu 75 Single-Stage Object Detectors Input image 3 xH xW Redmon et al, “You Only Look Once: Unified, Real-Time Object Detection”, CVPR 2016 Liu et al, “SSD: Single-Shot MultiBox Detector”, ECCV 2016 Lin et al, “Focal Loss for Dense Object Detection”, ICCV 2017 Divide image into grid 7x7 Image a set of base boxes centered at each grid cell Here B = 3 Development of intelligent systems, Object recognition with CNNs Within each grid cell: - Regress from each of the B base boxes to a final box with 5 numbers: (dx, dy, dh, dw, confidence) - Predict scores for each of C classes (including background as a class) - Looks a lot like RPN, but category-specific! Output: 7 x 7 x (5 * B + C) Slide credit: Fei-Fei Li, Ranjay Krishna, Danfei Xu 76 SSD: Single Shot MultiBox Detector  Multi-scale feature maps for detection  Convolutional predictors for detection  Default boxes and aspect ratios  Real time operation [Liu et al., ECCV 2016] Development of intelligent systems, Object recognition with CNNs 77 Wide usabilty of ConvNets Detection Instance segmentation [Redmon, Yolo, 2018] [Liu, SSD, 2015] [He, Mask R-CNN, 2012] Development of intelligent systems, Object recognition with CNNs 78 Wide usabilty of ConvNets Semantic segmentation [Farabet et al., 2012] [Chen, DeepLab 2017] Development of intelligent systems, Object recognition with CNNs 79 Wide usabilty of ConvNets  Image segmentation [Jonson, 2016] [Caicedo, 2018] [Marmanis, 2016] Development of intelligent systems, Object recognition with CNNs 80 Wide usabilty of ConvNets  Action recognition [Donahue et al. 2016] [Luvizon et al. 2016] [Simonyan et al. 2014] Development of intelligent systems, Object recognition with CNNs 81 Wide usabilty of ConvNets  Biometry [Taigman et al. 2014] [Najibi, SSH, 2017] [Emeršič, 2017] Development of intelligent systems, Object recognition with CNNs 82 Wide usabilty of ConvNets  Person/pose detection [Güler, DensePose, 2018] [Cao 2017] Development of intelligent systems, Object recognition with CNNs 83 Wide usabilty of ConvNets  Reinforcement learning for game playing [Google DeepMind] Development of intelligent systems, Object recognition with CNNs 84 Wide usabilty of ConvNets Image Captioning [Vinyals et al., 2015] Development of intelligent systems, Object recognition with CNNs Slide credit: Fei-Fei Li, Andrej Karpathy, Justin Johnson 85 Surface-defect detection Segmantation-based data-driven surface-defect detection Development of intelligent systems, Object recognition with CNNs 86 Surface-defect detection Development of intelligent systems, Object recognition with CNNs 87 Surface-defect detection Development of intelligent systems, Object recognition with CNNs 88 Polyp counting Development of intelligent systems, Object recognition with CNNs 89 Polyp counting Development of intelligent systems, Object recognition with CNNs 90 Ship detection Development of intelligent systems, Object recognition with CNNs 91 Face detection Development of intelligent systems, Object recognition with CNNs 92 Mask-wearing detection Development of intelligent systems, Object recognition with CNNs 93 Obstacle detection on autonomous boat USV equipped with different sensors: • stereo camera • IMU • GPS • compass Segmentation based on RGB + IMU Development of intelligent systems, Object recognition with CNNs 94 Semantic edge detection Development of intelligent systems, Object recognition with CNNs 95 Object (traffic sign) detection Development of intelligent systems, Object recognition with CNNs 96 Object (traffic sign) detection Development of intelligent systems, Object recognition with CNNs 97 Image anonymisation  Detection and anonimysation of car plates and faces Development of intelligent systems, Object recognition with CNNs 98 Visual tracking Development of intelligent systems, Object recognition with CNNs 99 Plank classification Development of intelligent systems, Object recognition with CNNs 100 Place recognition Development of intelligent systems, Object recognition with CNNs 101 Semantic segmentation Development of intelligent systems, Object recognition with CNNs 102 Image enhancement  Deblurring, super-resolution Development of intelligent systems, Object recognition with CNNs 103 Development of intelligent systems, Object recognition with CNNs 104 Development of intelligent systems, Object recognition with CNNs 105 Deep reinforcement learning  Automatic generation of learning examples  Goal-driven map-less mobile robot navigation Development of intelligent systems, Object recognition with CNNs 106 Innate and learned  Goal-driven map-less mobile robot navigation  Constraining the problem using a priory knowledge Engineering approach Engineering approach + deep learning Development of intelligent systems, Object recognition with CNNs Pure learning 107 Problem solving Complexity  Different problem complexities Simple, well defined problems Complex, vaguely defined problems Rule-based decision making Data-driven decision making Programming Machine learning Development of intelligent systems, Object recognition with CNNs 108 Complexity Problem solving Routine solutions Rule-based solutions Development of intelligent systems, Object recognition with CNNs Data-driven solutions General intelligence 109 Adequate tools Routine solutions Rule-based solutions Development of intelligent systems, Object recognition with CNNs Data-driven solutions General intelligence 110 Development, deployement and maintainance • Data, data, data! • Enough data, representative data • Correctly annotated data • Appropriate deep architecture design • Proper backbone, architecture, loss function, … • Learning, parameter optimisation • Efficient implementation • Execution speed • Integration • Maintenance • Incremental improvement of the learned model • Reflecting to changes in the environment Development of intelligent systems, Object recognition with CNNs 111 Development of deep learning solutions 20% % of solution 80% 80% 80:20? 90:10? 99:1? 60:40? 20% time Development of intelligent systems, Object recognition with CNNs 112 Knowledge and experience count Development of intelligent systems, Object recognition with CNNs 113 Software  Neural networks in Python  Convolutional neural networks using PyTorch or TensorFlow  or other deep learning frameworks  Optionally use Google Colab Development of intelligent systems, Object recognition with CNNs 114 Literature  Michael A. Nielsen, Neural Networks and Deep learning, Determination Press, 2015 http://neuralnetworksanddeeplearning.com/index.html  Ian Goodfellow and Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016 http://www.deeplearningbook.org/  Fei-Fei Li, Andrej Karpathy, Justin Johnson, CS231n: Convolutional Neural Networks for Visual Recognition, Stanford University, 2016 http://cs231n.stanford.edu/  Papers Development of intelligent systems, Object recognition with CNNs 115 Development of intelligent systems (RInS) Cognitive robot systems Danijel Skočaj University of Ljubljana Faculty of Computer and Information Science Academic year: 2021/2022 Development of intelligent systems, Cognitive robot systems Robotics  Routine industrial robotic system EURON video EURON video  Intelligent artificial visual cognitive system Development of intelligent systems, Cognitive robot systems 2 Cognitive robot systems cognitive robots industrial robots SF human Development of intelligent systems, Cognitive robot systems perception action attention goals planning reasoning communication learning 3 Cognitive robotics  Wikipedia: Cognitive robotics is concerned with endowing robots with mammalian and human-like cognitive capabilities to enable the achievement of complex goals in complex environments. Robotic cognitive capabilities include perception processing, attention allocation, anticipation, planning, reasoning about other agents, and perhaps reasoning about their own mental states. Robotic cognition embodies the behaviour of intelligent agents in the physical world.  A cognitive robot should exhibit:       knowledge beliefs preferences goals informational attitudes motivational attitudes (observing, communicating, revising beliefs, planning) Development of intelligent systems, Cognitive robot systems 4 Researchers‘ definitions            Cognition is the ability to relate perception and action in a meaningful way determined by experience, learning and memory. Mike Denham A cognitive system possesses the ability of self-reflection (or at least selfawareness). Horst Bischof Cognition is gaining knowledge through the senses. Majid Mermehdi Cognition is the ability to ground perceptions in concepts together with the ability to manipulate concepts in order to proceed toward goals. Christian Bauckhage An artificial cognitive system is a system that is able to perceive its surrounding environment with multiple sensors, merge this information, reason about it, learn from it and interact with the outside world. Barbara Caputo Cognition is self-aware processing of information. Cecilio Angulo Cognitive Systems are ones that are able to extract and (most importantly) represent useful aspects of largely redundant, possibly irrelevant sensory information in a form that is most conducive to achieving a particular high level goal. Sethu Vijayakumar A cognitive system is a system that can change its behaviour based on reasoning, using observed evidence and domain knowledge. Bob Fisher Cognition is when I know what I am doing, when I can judge how good or bad it is, and explain why I am doing it. Markus Vincze Cognition is the ability to plan, reason, adapt and act according to high level motivations or goals and using a range of senses, typically including vision, and may be communicate. Patrick Courtney A cognitive system is an autonomous anti-entropy engine. David Vernon Development of intelligent systems, Cognitive robot systems 5 Researchers‘ definitions            Cognition is the ability to relate perception and action in a meaningful way determined by experience, learning and memory. Mike Denham A cognitive system possesses the ability of self-reflection (or at least selfawareness). Horst Bischof Cognition is gaining knowledge through the senses. Majid Mermehdi Cognition is the ability to ground perceptions in concepts together with the ability to manipulate concepts in order to proceed toward goals. Christian Bauckhage An artificial cognitive system is a system that is able to perceive its surrounding environment with multiple sensors, merge this information, reason about it, learn from it and interact with the outside world. Barbara Caputo Cognition is self-aware processing of information. Cecilio Angulo Cognitive Systems are ones that are able to extract and (most importantly) represent useful aspects of largely redundant, possibly irrelevant sensory information in a form that is most conducive to achieving a particular high level goal. Sethu Vijayakumar A cognitive system is a system that can change its behaviour based on reasoning, using observed evidence and domain knowledge. Bob Fisher Cognition is when I know what I am doing, when I can judge how good or bad it is, and explain why I am doing it. Markus Vincze Cognition is the ability to plan, reason, adapt and act according to high level motivations or goals and using a range of senses, typically including vision, and may be communicate. Patrick Courtney A cognitive system is an autonomous anti-entropy engine. David Vernon Development of intelligent systems, Cognitive robot systems 6 Main emphasis       Perception Action Reasoning, planning Goals Autonomy, self-awareness Environment AGENT see next action state ENVIRONMENT Development of intelligent systems, Cognitive robot systems 7 An example of a cognitive system  Household robot Robi  My command: “Fetch me a beer”. Development of intelligent systems, Cognitive robot systems 8 Example  Sequence of actions:  The robot has to be attentive and has to listen for my command. [attention, motivation]  It has to hear me and understand my command. [perception, speech recognition, communication]  It has to set the corresponding goal and aiming at fulfilling it. [goal, proactive behaviour]  It has to know where the beer is located, it had to previously learn that. [learning]  He has to plan how to fetch the beer. [planning]  He has to plan the most appropriate path to the refrigerator, based on the map, which had to be previously built. [navigation, map building]  He has to move along the planned path. [action – moving]  On the way, it has to continuously monitor its path. [perception, action]  It has to avoid obstacles. [perception, replanning, reactive behaviour] Development of intelligent systems, Cognitive robot systems 9 Example  When arrives in front of the refrigerator, it has to position itself appropriately. [embodiment, situatidness]  It has to know how to open the refrigerator. [recognition of object affordances]  It has to search for the beer in the refrigerator (it has to learn in advance the corresponding appearance). [perception, categorisation, learning]  It has to plan how to grasp the beer. [planning]  It has to grasp the bottle suitably. [action, visual servoing, haptic control]  It will take the reverse path and return to me. [planning, navigation, action, perception, recognition]  Robi: “Here is your beer”. [communication] Development of intelligent systems, Cognitive robot systems 10 Cognitive systems  Cognitive assistant Explores the environment and builds the map Learns to recognize objects Understands object affordances Knows to interpret verbal and nonverbal communication with persons  Detects new situations and reacts correspondingly  Operates robustly in real time in unconstrained domestic environment     Willow Garage • Basic functionalities are built in; they are further developed and extended by learning Development of intelligent systems, Cognitive robot systems 11 An example of a cognitive system  Autonomous car  City drive  Competencies            Perception (image, 3D, collision) Planning Reasoning Learning Navigation Obstacle avoidance Action Flexibility Robustness Efficiency … Development of intelligent systems, Cognitive robot systems Google self-driving car 12 Requirements of cognitive systems          Perception Representations Recognition Learning Reasoning Planning Communication Action Architecture Development of intelligent systems, Cognitive robot systems 13 Perception  Perception      Visual information (image, video; RGB, BW, IR,…) Sound (speech, music, noise, …) Haptic information (haptic sensors, collision detectors, ect.) Range/depth/space information (range images, 3D models, 3D maps, …) Many different modalities – very multimodal system  Attention  Selective attention  Handling complexity of input signals Development of intelligent systems, Cognitive robot systems 14 Representation of visual information = Development of intelligent systems, Cognitive robot systems + a1 + a2 + a3 +… 15 Representation of space  Metric information  Topological map  Hierarchical representation Development of intelligent systems, Cognitive robot systems 16 Representation of audio Development of intelligent systems, Cognitive robot systems 17 Representation of linguistic information Development of intelligent systems, Cognitive robot systems 18 Representation of knowledge 1. Natural language   understanding the meaning of the individual words Spot is a brown dog and, like any dog, has four legs and a tail. 2. Formal language    Formal logic “Spot is a brown dog” : dog(Spot) AND brown(Spot) “Every dog has four legs“: (∀x) dog(x) -> four-legged(x) 3. Graphical representation   Knowledge is represented with nodes and edges Semantic nets 4. Ect.  appropriateness, efficiency, scalability, suitability Development of intelligent systems, Cognitive robot systems 19 Recognition  Recognition of          objects properties faces rooms affordances actions speech relations intentions,…  Categorisation  Multimodal recognition Development of intelligent systems, Cognitive robot systems 20 Learning        Buildnig representations Continuous learning Different learning modes Multimodal learning Forgetting, unlearning Robustness Nature:nurture training samples test samples … representation learning updated representation Development of intelligent systems, Cognitive robot systems recognition learning result new training sample 21 Multimodal learning Self-supervised learning Weakly-supervised cross-modal learning Cross-modal learning Development of intelligent systems, Cognitive robot systems 22 Reasoning  Reasoning In unpredictable environment With incomplete information With robot limitations In dynamic environment Considering different modalities Self-awareness, introspetion, knowledge gap detection and communication  Expert systems       Development of intelligent systems, Cognitive robot systems 23 Planning  Planning     In unpredictable environment With incomplete information With robot limitations In dynamic environment Development of intelligent systems, Cognitive robot systems 24 Communication  Communication  With human  With other (different) agents  In time and space      Transfer of knowledge Clarification Coordination Taking initiative in the dialogue Verbal and nonverbal communication  Symbol grounding  Semantic description  Learning language  syntax  ontology building  Learning using language Development of intelligent systems, Cognitive robot systems 25 Action  Object manipulation (manipulator)  Moving around in space (mobile robot)  Other: sound, light signals, other grippers, ect.  Embodiment  Situatidness Development of intelligent systems, Cognitive robot systems 26 Perception – action cycle  Large abstraction of the real world Development of intelligent systems, Cognitive robot systems 27 Architecture Autonomy, self-awareness Goals Perception Action Proactive Reactive Reasoning, planning Environment Development of intelligent systems, Cognitive robot systems CogAff architecture 28 Examples – PR2 U Tokyo, TUM Willow Garage UC Berkley Development of intelligent systems, Cognitive robot systems 29 Examples - iCub IIT Development of intelligent systems, Cognitive robot systems 30 Examples - Asimo Development of intelligent systems, Cognitive robot systems 31 Examples – Boston Dynamics Development of intelligent systems, Cognitive robot systems 32 Examples - Yaskawa Development of intelligent systems, Cognitive robot systems 33 Examples - Nao Aldebaran Robotics Expero, FRI LUI Development of intelligent systems, Cognitive robot systems 34 George FRI, LUVSS CogX, http://cogx.eu/results/george/ Development of intelligent systems, Cognitive robot systems 35 Curious robot George  Incremental learning in a dialogue with a human  Curiosity driven learning  Learning categorical knowledge Development of intelligent systems, Cognitive robot systems 36 Self-understanding for self-extension Development of intelligent systems, Cognitive robot systems 37 Learning mechanisms Situated tutor driven learning Situated tutor assisted learning Non-situated tutor assisted learning Situated autonomous learning Development of intelligent systems, Cognitive robot systems 38 Video http://cogx.eu/results/george Development of intelligent systems, Cognitive robot systems 39 System Development of intelligent systems, Cognitive robot systems 40 Conclusion  Cognitive systems are          intelligent very heterogeneous and asynchronous coherent multimodal They continuous upgrade their knowledge by learning They communicate with a human They interact with the environment They move around the environment They are able of autonomous reasoning and decision making  Literature: SKOČAJ, D., VREČKO, A., MAHNIČ, M., JANÍČEK, M., KRUIJFF, GJ, HANHEIDE, M., HAWES, N., WYATT, J., KELLER, T., ZHOU, K., ZILLICH, M., KRISTAN, M. An integrated system for interactive continuous learning of categorical knowledge. Journal of experimental & theoretical artificial intelligence, ISSN 0952-813X. [Print ed.], 2016, vol. , no. , str. 1-26 Development of intelligent systems, Cognitive robot systems 41 Conclusion T-60 Development of intelligent systems, Cognitive robot systems T-30 T T+30 42

ilovepdf merged (8)

Related documents

Products

Support

ilovepdf merged (8)

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib