The 2010 IEEElRSJ International Conference on Intelligent Robots and Systems October 18-22, 2010, Taipei, Taiwan Semantic Map Partitioning in Indoor Environments using Regional Analysis Carlos N ieto-Granda, John G. Rogers III, Alexander J. B. Trevor, and Henrik I. Christen se n Abstract- Classification of spatial regions based on semantic information in an indoor environment enables robot tasks such as navigation or mobile manipulation to be spatially aware. The availability of contextual information can significantly simplify operation of a mobile platform. We present methods for 3utomHted recognition and classification of SI)aceS into separHte semantic regions and use of sueh information for generHtion of a topological map of an environment. The association of semantic labels with spatial regions is based on Humal/ Augmented Mappil/g. T he methods presented in this paper are evalu ated both in simu lation and on real data acquired from an office environment. I. INTRODUC TIO N Humans are constantly trying to make their lives easier. Se rvice robots capable of operating in human environments have the pote nti a l to improve daily li fe by assisting humans in a variety of tasks. Endowi ng these robots with the abil ity to understand and reason about spatial regions such as indi vidual room s, as well as understanding the semantic labels of sllch spaces cou ld faci litate tasks such as navigation and mob ile manipulation in human envi ronments. Human env ironme nts are typically partitioned into di sc rete spaces, such as offices, corridors, li ving roo ms. etc. Such a partitioning allows human s to organize and enable their everyday acti vities, and these spaces typicall y have specirk: purposes and labels. Service robot s that understands the partitioni ng of human envi ronments can uti lize thi s information to better assist hum ans in everyday tasks. For example, if a robot is given the command "fetch the red mug from the kitchen" , having an understanding of the location and exte nt of the region considered "kitchen" is beneficial. In thi s paper, we prese nt a method for buildi ng a se manti c map partition in cooperation with a human guide. Given a metric map that the robot can loca li ze in. the system creates a se manti c map partition . The resulting semantic map part ition provides a probabili sti c classifi cati on of the metric map into a set of labe ls provided by the hum an guide. The robot can then use thi s semantic map part ition to navigate to a region with a specific label, and can determine the maxi mum like lihood label for any point in th e partition. Addi ti ona lly. if the robot is not confident that it knows a likel y semantic label for its current pose in the map, it wi ll prompt the guide to provide one. C. Nieto-Granda. J. Rogers Itl and 1\. Trevor and :Ire Ph .D. students at Georgia Tech College of Computing {car l os . nieto , atrevor , j g roge r s }@gate ch . e d u H. Christensen is the Kuka Chair of Robotics al Georgia Tech College of Computing hic@cc . gatech . edu 978-1-4244-6676-4/10/$25.00 © 201 0 IEEE The paper is organi zed as follows. We briefl y describe related work in Section II , followed by the motivation of our researc h in Section III. We then present our Gauss ian probabilistic regions approach in Section rv. In Section V. we present so me experiments and resu lts both in simulation and on a real robot. Finally, conclu sions and future work are given in Section VI. II . RELATED WORK In rece nt years, we have seen important develo pments in service and assist ive robots for domestic app li cations and tasks. These works focu s o n the understanding of the environment using semantic information in order to create a sy nergisti c interaction between human s and robots. Dellaert and Brue mmerl 3 ) proposed ex tending FastSLAM to add se mantic infonnat ion of the environment (Q each particle's map. Several approaches have been presented fo r map partitioning, using topological and geometri c represe ntation s of the environment. For example. OberHinder Ill] proposed a SLAM algorithm based on FastSLAM 2.0 [91 that maps features represe nting region s with a se manti c type, topo logical properties. and an approximate geometric extent. The res ulting maps e nable spatial reasoning on a semanti c leve l and provide abstract in fo rmati on allowing effic ien t se mantic planning and a co nve nient interface for humanmachine interaction. Thrun 1"16] integrated grid-based maps to learn the environment using artificialncu ral networks and narvc Bayesian in tegra ti on to generate a lOpulugic~ll map by partitioni ng the latter into co herent regions. Another body of work focuses on extracting se manti c spatial properties of th e environment from 2D and 3D data. Donsu ng and Nevatia [71 introduced a new spatial represe ntation. s-map, for an indoor navigation robot. This map represents the locations of visible 3D-surfaces of obstacles in a 20 space. O'Callaghan [ 121 developed a new statistical modeling technique for bui lding occupancy maps by providing both a continuous representat ion of the robot's surroundin g and an associated predictive variance e mploying a Gaussian process and Bayesian learning. Ek va ll [41 applied an automatic st rategy for map partitio ning based on detecting borders between rooms and narrow open in g to denote doors or gateways usi ng different types of features (lines, points. SlIT). 'R hino' [21 is an example of a service robot which integrates localization. mapping. collision avoidance. planning. and va ri ous modules concerned with user interaction telepresence giving lo urs on a mu seum. BIRON, a mobile Home Tour Robot 115] . uses integrated vision based localization a modular architecture and extending a spoken dialog system for o n-line labeling 1451 and interaction about different location s in a real, full y furni shed home environment where it was able to learn the names of different rooms. Th e approach presented by Topp and Christensen ll 8J and [1 7J, provides a separation of regions that relare to a users view on the environment and detection of transitions between them . They assumed an interac ti ve sctup ror the spedfkation of regions and showed the applicability of their method in terms of di stin ct ive ness for space segmentation and in term s of locali sation purposes. IV. A t' PROAC H III. SEMA NTt C SLAM As robots have to cooperative with hum ans it is advantageous that they have a shared representation of the space, preferab ly a model that is simple for the human to use as part of commanding the robot and understanding feedback. Semantic mapping li terature has focused on developing robotic mapping techniques capable of functionally supporting these types of interaction s. To perform the se tasks, o ne of the strategies that is used is to portray the relationship between a place and the knowledge that is associated with it e.g.(functionality. obj ective location), is semantic mapping. Kuipers l8J proposed the Spalial Semantic Hierarchy (55 H), which is a qua litative and quantitative model of knowledge of large-scale space consisting of multiple interacting representati ons. This map also informs the robot of the control strategy that should be used to traverse between locations in the map. This represent atio n is based on the relationship of objects. actions and the dependencies from the environment. More recently. Beeson et al. f II provided a more specific framework representation of spatial know ledge in small scale space. This framework is focused on the robot 's sensory hori zon e.g.(global and local sy mbo lic, and metri ca l reasoning o f the space), but also human interaction. Existi ng approaches for robot indoor navigation bu i Id an occ upancy grid map using range dara from its se nsors. These maps, however, on ly provide geometric inform at ion such as obstacl es and open areas in the env ironment without a se mantic understa ndi ng of it. Martfnez-Mozus and Kottmann 1101 l l4J introduce a scmantic understand ing of the environment creati ng a conceptual representation refe rring to functiona l properties of typical indoor environments. Providin g se mantic info rmation enab les a mobile robot to more effici ently accomplish a variety of tasks sllch as hum anrobot interactio n, path-planning, and localization. Ekvall [41 integrated an augme nted SLAM map with information based on object recognition , providing a richer representation of the environment in a servi ce robot scenario. In this work , we focus on provi ding a semantic partition of a metric map using semantic labels provided by a human . We believe thi s representation cou ld be used to support semantic reasoning for a variety of mobil e robot tasks in indoor environments. As an example. we present navigation to the nearest puint in a metric map that has a spccific semantic label. Our goa l was to des ign a syste m capable of reasoning about spaces. In contrast to work such as flO] . whi ch builds a topologi ca l map o n top o f a metric map. we instead provide a co ntinuous c lass ifi cati on o f th e met ric map into scmanti ca ll y labeled regions. The semantic map layer o f our system is a multi va riate probability di stribution on the coordinates of our metric map to a set of se manti c labels. This multivariate di stribution is modeled as a Gau ss ian mode l. Each of the Gaussians in the model is based on the robot's sensor data when it was provided a label by a human g uide. Eac h spatial reg ion is represent ed usin g one or more Gallss ians in our met ric map's coordinate fram e. So, a reg ion with label L and 11 Gaussians. each with mean I-t and covariance E , is rep rese nted as; Region = {L. {litl , E l }, {1'.2, E z }, .... {IL" , E,, }}} A semantic map is then just a co ll ecti on of such regions. so a se manti c map with m region s wou ld be represented as; Our system builds these maps part itio ns of our met ri c maps through human guidance. The human takes the robot on a tour of th e space (either by driving the robot manuall y, or lI sing a perso n fo ll owing behavior), and teaches the robot typing the appropri ate label for the space that it is currently in. The regi onal analysis technique is to take a laser scan mcasurcmt:nt , fi t a Ga uss ian tu the resulting puints. and store the mean and covariance in th e map along with th e labe l provided by the human. Using thi s se manti c map partition , the robot can be asked for its belief of the name of the region it is currently occupying. This is done by evaluati ng the Maha lanobis di stance of the robot' s current pose :r close by labels coded as Gau ss ian region mode ls (Equation I), and choosing the region that is cl osest using thi s metric. DA/(.c) = J(,. - ltrrE- l(:< - IL) ( I) Thi s map represe ntati on all ows for prohahili sti c class ifi cation of the map by region label. Additionally, while navigat ing throu gh the envi ron ment , the robot continuously checks its posi ti on with re spec t to the semantic Illap partition. If it is not sufficiently confident (more than a ce rtain thresho ld ) that it is in a region with a known labe l, it prompts the user to inp ut the name of the current region . Once the robot has a se mantic map partition, use rs can requ est that the robot navi gate to onc of the regions. such as " livi ng roo m" . The robo t can then find the region in the map with label " living room", and calculate the Mahalanobi s di stance from its current position 10 the mean of each Gaussian in the region. The robot selects the closest of these as the goal, and se nd s thi s to its path planner in order to autonomou sly navigate to that region. While traveling. the robot continuou sly calculates its confidence or which region 1452 it is. and stops when it is con fidem that it is mo re likel y to be in the goa l region than any other region as follows: in orde r to test the system's effectiveness ,)( <lccepting thi s type o f navigat ion comm and . The robot is able to successfully nav igate with the calc ulated trajectory, avo id ing obstacles. One of the tests was to move from a room labeled " roo m 9" to "room 4" in the map as appea rs in Fi g. I. 1 . Distance to goal < - * Nearest ("stance to the lIon ·goal 'I This results in the robot ente ring a regio n, but not attemp ting to move to the reg ion's ce nter. Additionally. if the robot enters the reg ion with the desired label at any po int while navigating to its goa l poin!. perhaps because the path to the closest poi nt was blocked, it will recog ni ze this and stop once it is suffi cientl y in the goa l region. In thi s work. we o nly use the se manti c Illap parti ti on for navigat ion tas ks; however. we be lieve this map representation has a num ber of other applications. such as searchi ng for objects. For exa mple, if we have a mobile manipul ation platform and as k the robot to "get the mug fro m the kitchen". our map representation ca n be used to give a spat ial reg ion of our metric map wh ich shoul d be search. by findin g th e area thai is labeled as kitch en with at least a certain confidence leve l. Fig. 2. The robot's path is obstructed with :L sinmlated block. The ro bot is the smaller blue o bjeci and the o bstacles are the large red cubes. V . EXPER I ME NTS Our approach has been implemented and eval uated in several ex pe ri ments. We designed two simulated environments in Stage[S ] l in which the robot can be taught locations and navi gate between thetTI. Preli minary ex peri ments we re also performed using our Segway RMP-200 mobile platform to verify our tec hn ique on a rea l robot platform. Show n in Fig. 2. the shortest path was o bstructed with a simulated bl ock and the robot rep lan ned <l new trajectory to reach the goal as can be seen in Fig 3. The laser hi ts on the obstacle that bloc ks the sho rtest path ca n be see n nca r the robot in Fig. 3. A. Sill/Illa tion Env;,vnmem In thi s simulated ex periment, we tested the effect iveness of our method by providin g labels for each of the roo ms and hallways. Fi g. 3. Robot path replanned 10 navigate from room '9' to central hallway and arri\'ed in room ' 4'. The blue circle re presents the robot. the green line the robot path and in red the o bstacles detected by the laser scan . Fig. 1. A visualiz:ltion of the Gaussian regions representi ng the rooms and hallways. All the Gaus~ians are colored in different colors for idell1ification . T hi ~ figure is hc~ t viewed in l'Olor. The first experi ment , shown in Fig. I, consisted of labe li ng each room and each hallway and navigating betwee n them I Stage is a 20 multi ple-robot si mulalOr from the Pl ayer project. http://playerstage.sourceforge.net For the second ex peri ment. we labelled twe nty seve n rooms and a hallway in the map. and left o ne unkn own are a unlabeled. as can be secn in the upper ri ght corner of Fi g. 4. The main purpose W<lS to sim ulate an office enviro nment. where the tran sition region between rooms is a hallway. The robot is <lblc to continuously prov ide the curre nt locat ion and naviga te from Oll e region to anothe r. Based on thi s map c1<lssifi cati on, we creatcd several diffe rent scenarios to test our system. One o f the tes t scenari os, show n in Fig 5, involved the robot nav iga ting between two reg ions wit h different labels, a room and a hall way. 1453 • • Fig. 4 . The second ).imui;llcd offi ce cnvironmcnl us(.'(\ for our cxpt.!rimcnl .~. Colored ellipses re present the Gaussians in our model. and different colors rcprcscl\! d io-ere nl spaces. T his fi gure i~ hcst viewed in Clllnr. represented by Ga uss ians with means very close to each other. The robot was requested to navigate from the room to the hallway and bac k. OUf system calculat ed the Mahalanobis distance to find the closest Gau ss ian region. but the reg ions arc ve ry close to eac h other that the robot on ly turn and move onl y a short di stance. This demonstrates that our sys tem wi II cause the robot to move into a region until it has a certain co nfidence level that it is in the region. rather than slOpping on the equiprobablc dec isio n boundary betwee n the regions as show n in Fig 6. Fig. 6. A vi sualization of the deci sion boundarics of the regions representing the roo ms and hallways o n the map shown in 4. All the Gaussians are col ored in different col o~ fo r ide ntification . Th is fig ure is he:.! viewed in color. Localizing the robot betwee n roo ms and hallways worked bette r whe n the use r taught new locatio ns to the robot in the middle o f the room as opposed to in the doorways because the laser hits were more rep rese ntat ive of the room 's ex tent. Finall y, we tested the robot by starting it with no semanti c map informati on, so that it would prompt th e user im medi ately for a label. whi ch the human guide provided. The robot then navigated arou nd the environment. and would stop wheneve r it was not confident that it knew th e labe l for the current pose. Upon being provided with a l ~lbcl for the current location by th e human guide. the tour resumed until it again needed a new label. This resulting map is shown in Fig. S. This ex perim ent demonstra tes our method 's effect iveness at determining when a new labe l is required. The robot began in the left middle roo m. and ended ill the upper ri ght. Fig. 5. Visualiz:ltion of the robot navigating between to regions in the environ ment. Another scenario. shown in Fig. 7 is to dri ve the robot through a hallway to an unknown area whi ch has never been assigned any label. When the robot reac hed a location that was not likely to be part o f any prev iously labeled region, it di splayed " I do II 0 f kllow whe re I alii." and requ ested that the use r provide th e current loca tion 's label display ing " Please tell me where I am ". Several more scenarios consisted of teaching th e robot with different orientations and locat ions in side of the rooms. f ig. 7. The robot is driven in an unknown area whic h has not bee n assigned any label. 1454 mnKr~ :-----..----------.., 1, Offie 1 Fig. 10. Fig. 8. Resul t rrom starting the robot with no known locations in the !'cmallt ic map. Hnd prompting the user when thc robot was 110t confident of the appropriate label. B. Real En vironment Our approach has been implemented on a Segway RMP200 mobile platform (Fig 9). It is equip ped with a S IC K LMS-29 1 laser scan ner. which is used for localizalion, mapping and obs(ac le avo idance, and is co n(rolled by an on-boa rd Mac mini compu ter (2.26GH z/Co re 2 Duo). We cunducted an expcrimelll in u ur rea l u ni ce environ ment (Fi g. 10). teach ing Ih e robot with 5 different rooms and several different points in the hal l. Robotics and Intell igent Machines Lilboratory Map. Ih e location. During the tour. Ih e robot coul d be qu cried for its cu rrent location. then the robot wou ld calc ulate the Mahala nobi s di sta nce to the Gau ss ians reg ions and report the label of the neares( one. If (he robot pose wa s not near to a previously labe lled location, the robot repon the locati on as "u nknow n". Also, the robot can be as ked to move to a specific known locati on. ror ex amp le move from "C-3" to " LAB /". Then, th e robot ca lculates a safe lrajectory to the room using the g lobal map and conti nu ously runs a local pla nner to avoid obstacles throughout the ellvironmenl. When the robot success full y co mpl eted the task, it reported that it arrived at the current location's la be l. \ ,.:.\-, \ \ \ Fig. 9, - \, --~ Fig. t I. Generated Occupancy Grid Map for locatization used in our expe riments. Our robot platform used in our experi ments. The first step WHS 1u dri ve our robot th roug h the environment whi le COll ect in g laser data and odometry. Th e SLAM gmappin g module incl uded with ROS[,131 . which is based on the Rao- Blaekwe lli zed partic le filter tec hnique by Gri selli ef. al l6j, was then used to build a metric grid map of the environment. The map shown in Fig. II was then used fo r loca li zation in our ex periments. Beginnin g with the map. the hum an tour startcd from the hall way to one of the th ree cu bicles. Wh en the human guide stopped. the robot was provided with the name of V I. CONC L US tON AND F UTU RE W ORK We presented a technique for parti tioning metri c maps into labeled spatial regions using a Ga ussian mod el. We th en used (his represe ntati on to perform navigation lasks in the map, as demo nstrated by our pre liminary ex periment s both in simul ation and on a real robot platform . In fu ture work, we pl an to investi ga te techniques fo r aU lomaticall y labeling regions. One ex ampl e of how to do thi s could be by read in g signs Ihat arc presenl in office 1455 environments. Objects detected by the robot co uld also provide informati on as to an appropri ate label for a space, for exam ple, if a microwave and a toaster are detected, then "kitchen" mi ght be an appropriate labe l. Al so, we wou ld li ke to investigate add it ional uses for this map representat ion such as constraining the search space when process ing a req uest such as "fetch the red mug from the kitchen". V II . AC K NOWLEDG M EN TS Thi s work was made poss ible through the KORUS project, the Boe ing corporati on, and the ARL MAST CTA project 104953. R EFER ENCES II J P:ltric k Becson. Mall MacMahon. Joseph Modayi l. An ikct Murarka. Benjami n Kui pers. and Brian Stankiewicz. Intcgrating multiple re prcse ntations of spatial knowledge for mapping. navigation. and COIll mUnication. In SY"'I)().n·/I111 on IlIIeraction Clwllt' lIge.f fur Il1tt'lIigcm A .f.l·i.Wllllf.l', AAA I Spri ng Symposiu m Series. Swnford. CA. March 2007. 12] Wo lfram Burgard. Armin B. Cremers, Dietcr Fox. Dirk H;ihnc l. Gcrhard Lakcrneyc r. Dirk Schu lz. Walter Stci ner. and Scbastian Thru n. Experiences wi th a n interactive museum tour-guide ro bot. In Artificial Iml'iligl'llce , volume 114, p:lgcs 1- 2. 1998. 131 Fr•.IIl k De ll:lert and David Bruemmer. Semantic slam for co llaborative cognitivc workspaces. In I" AAAI FlIlI S)'I"I)()~'i/l1ll Seri es. 2004. 14 J Stamm Ekval l. Danica Kragic. and Patric Je nsfe lt . Object detection and mapping for service robOi tasks. RobO/ic(l.' IlIIematiollal JO/lI'IIlll of Illformm inil. I~'dlll'mi()/l alld Re~'eardl ill Robotic.\' alld Artificial Illtelligence. 25(2): 175- 187, March/Apri l 2007. 15] B. Gcrkey. R. Vaughan. and A. Howard. The player/stage project: Tools for mu lti-robot and distributed sensor systems. In 11th IlIIer Iwtiu/UlI COllfer('llce 011 Ar/l'{lfIcn/ Robotics (ICAR 2(03). Coimbra, Port ugal. June 2003. [6} Giorgio Griscll i, Cyrill St3chn iss, and Wol fra m Burgard . Improved techniq ue.~ for grid nwpping with rao-blackwcllized particle f]lte r.~. In IEEE 7hlll.m ctiom 01/ Robotic.f. volu me 23. 2007. [71 Dongsung Kim and Ramakant Nevatia. A method for recognition and loc;lliza(ion of generic objects for indoor navigmion. I IIUlge and Vi~'io" COII/lmtillg . 16:729-743. 1994. 1456 18] Benjamin Kuipers. Rob Brown ing. Bill Gribble. Mikc Hewett , and Emilio Renlolina. The !>patial seman tic hierarchy. In Artificial ''''''/ligellee. volu me 119. p;lges 19 1- 233.2000. 19 J Michael Montelllerio. Seba!>lian Thrun. Daphne Koller. and Ben Wegbreil. Fa.'i t.'ilarn 2.0: An improved particle filtering algorithm for s imu ltaneous locali zation and mapping that provably convergcs. In III I'mc. of the 11/1. Cm!{ 1111 li.rti/ic:ial1lIfl' lIiRl'lIce ( IJCAI, pages 1151 1156, 2003. IIOJ Oscar Man inez Mows. Rudolph Triebcl. Pmric Je nsfdt, Axel Roltm:lnn. and Wolfr..Ull Bu rgard. Supervised semantic labeling of pbces using infonnal ion eXlmcted from sensor data. Roboti('.f (lnd AWOI/OIIIOlIS Sys,em.l· ( HAS). 55(5):391--402, May 2007. J II] Jan Oberliinder. Kl aus Uhl. Johann Marhls Zollner, a nd Rudi ger Dill mann. A region- based slam algorithm capturing metric. tOI>ological. and semantic propeni es. In In Proc. of tile IEEE Inti. COli! OIl Rob()lic.~ lIlId AI/tomlllio" ( ICRA). pages 1886-1 89 1. 2008. 112 / Simon O'Ca liaghan, Fabio T. R;unos, and Hugh F. Durrant-Whyte. Contextual occupancy 1l1 ;l p.~ using gaussian procc.~ses. In III Pm!'. of Ihe IEEE Inti. COli! Oil Robotic.~ alld A/ltOlllllfioll (lCRA ). pages 1054- 1060. IEEE. 2009. [ 13! Morgan Quigley, Ke n Conley. Brian Gcrkey. Josh Faust, Tully B. Foote. Jeremy Lcibs. Rob Wheeler. and Andrew Y. Ng . ROS ;:1I1 open-souree robot operating system . In 11/fl'l'IllIliollal COllfi' I'e IlCl' Oil Robotic.f fllld Autoll/lltiol/, Open-Source Software workshop. 2009. 114 J Axe l Rott mann . OSCar Maninez Mozo.~. Cyrill Stach nis.~. and Wo lfram Burgard. Semantic pl:lcc cl as.. if]I;,lIinn or indoor en\' i ron m ent~ with mobi le ro bots using boosting. In A AA I '05: Procel!c/illgs of the 20,,, /latiullal {'(mferelln! IJ/I A rtific'ial illle/IiXl! IIC'e, pages 1306--13 11. AAA I Press. 2005 . [15] Thorstell Spexard. Shu yi n Li. Brilla Wrede. Jannik Fri tsch. Gerhard Sagcrcr, Olaf Booij. loran Zivkovic, Bas Te rwijn, and Ben KrUse. Biron, where are you'! - enabling a robot to leam new places in a real home environment by integrati ng spoken dia log and visual local ization. In Proc. IEEE/RSJ 1111. COli! Oil IlIIelligC'1II Robot.l· (/11(1 Systt:lll.l· . IEEE. October 2006. 1161 S. Thrun. Lcaming me tric-topological maps for indoor mobi le robot navigati on. In Arlijit'ial lmd/iKt!II('e. volume 99. pages 21 - 71. 1998. [ 17J E. I\. Topp and H. I. Christensc n. Detecting regio n transi tions for human -augmemed mapping. Rohotic.~. IEEE TmllslIct;OIl.\· 0/1 . pages 1- 5. 20 10. [1 8 / Eli n A. Topp and Henrik J. Christe nsen. Topological modc ll ing for human augmen ted mappi ng. In Imel/igel1 l Robols alld Sy.I"ff'III.I'. 2()()6 IEEE/RSJ IlIIel'llatioll(l f COllfereflCl! Oil. pages 2257- 2263, Oct . 2006.