Ross Creed
Temple University
ICAR Workshop June 20, 2011
Talk Outline
• 10 min – Background and Motivation
• 10 min – Discussion of Temple Map Evaluation
Toolkit
• 10 min – Demo and Questions
Project Overview
 Project seeks to address the following questions:
• Is it possible to develop a world modeling framework that can enable
autonomous navigation of AGVs with particular emphasis on requirements
and constraints imposed by manufacturing environments such as factory
floors where humans and machines work side by side?
–Such a framework should not only provide an accurate representation of the
environment but also should be smart and flexible in terms of task-dependent goals
and sensor modalities. Furthermore, can the framework be sufficiently generic so
that it can be extended to other domains?
• How can we develop scientifically sound methodologies to evaluate world
modeling schemes for use in manufacturing environments?
–It is our view that such methodologies are better developed by taking into
account requirements imposed by end-users and domain specific constraints that
are grounded in practicality
• How can we design experiments and test methods to enable performance
evaluation and benchmarking towards characterizing constituent
components of navigation and world modeling systems that provide
statistically significant results and quantifiable performance data?
Map Evaluation: Motivation &
Background
• (Quantitative) Map Quality is a performance measure of how well a robot
or team of robots can explore, understand and interpret the operational
domain; subsequently, indicative of the utility of the robot-generated map
Evaluation Philosophy
•
•
•
To design and develop capable, dependable, and affordable robotic systems, their
performance must be measurable (quantitative)
Repeatable and reproducible test artifacts and measurement methodologies to
capture performance data  focus research efforts, provide direction, and
accelerate the advancement of mobile robot capabilities (objective)
Only by involving users, developers and integrators in a coupled fashion, can
meaningful solutions be produced that can stand the ever-varying requirements
imposed by:
•
•
•
1) tasks that are either application or environment dependent,
2) hardware and software advancements/restrictions that affect the development cycle,
and
3) budgetary constraints that interrupt and hamper sustained progress
Motivation & Background
• Qualitative comparison of resulting maps is used to assess performance,
e.g. visual inspection
• Common practice in the literature to compare newly developed mapping
algorithms with former methods by presenting images of generated maps
• suboptimal, particularly when applied to large–scale maps
• clearly not a good choice of evaluation
• hard to inter-compare results
• Prevalent problem spanning multiple domains: rescue, manufacturing,
military, service robotics, …
• No accepted standard nor consensus on objective evaluation procedures
exist today for quantitatively measuring the performance of robotic
mapping systems against user–defined requirements
Which is better?
OR
J. Elseberg, 2010.
A. Kleiner, 2009.
QUANTIFYING ROBOTIC MAPPING
• Mapping, in general, is spatial analysis of environmental features
of interest. Inherent to this process is its task dependency,
hence there is no 'optimal general mapping'.
• Mapping can be divided into two classes: topographic and
topological mapping.
• Topographic: concerned with detailed, correct geometry
• Topological: correct spatial relation between features only
• They are often referred to as 'global correctness' vs. 'local
accuracy‘ and can be related to GRID and POSE based approaches in
map evaluation.
• A toolkit developed for the 2008 Robocup Rescue Rescue
Competition, the ‘Jacobs Map Analysis Toolkit’ (A. Birk, Jacobs
University, Bremen)
• Purely Grid based (topographic evaluation)
• Depending on the application, this can introduce massive (fatal)
problems.
• Example: A is correct environment, B and C different maps. In a
rescue scenario, if B is favored, responders would try to approach
the victim through the wrong (left) hallway
A
B
C
• In contrast, pose based approaches would point out the global
error, but also the local correctness. Map C would be preferred.
• However, there are (of course) counter examples, showing
disadvantages of pose based approaches.
• Analytic research on
such examples can lead
to general approaches to
evaluation.
• In the example case, a
simple solution would
be a hybrid approach:
combined pose/grid
based evaluation,
emphasizing the
advantages of both
approaches.
a
b
c
d
The Temple Hybrid Map Evaluation Toolkit
• Compares a created map versus a groundtruth map
• Accounts for both pose-based and geometric
difference between maps
• Portable (Java implementation) and Flexible
(allows for segment and picture based maps)
Running the Toolkit
• Thanks to the Java webstart technologies, the
only download required is the ~1Kb jnlp file
• After this file is launched, the required java
files are acquired and the program is run
• No installations required! OS Independent!
Map Import
• Since map evaluation is not
a standardized procedure,
it follows that map formats
are not standardized either
• The program is set up to
handle segment-based
maps, and pictoral-based
maps (JPG, GIF, PNG, etc.)
• Pictoral based maps are
converted into segments
using a binarization (human
in the loop) and line finding
algorithm
Map Chopping
• For pose-based
evaluation the
ground truth map
can be “chopped”
into sections
• These sections can
be rotated,
translated, and
scaled to better fit
the target map
Map Alignment
• In the final step before evaluation, the ground-truth
map is fitted to the target map, keeping track of the
transforms of each map segment
• These translations can be performed either by the user,
or by an algorithm, or a combination of both
Map Evaluation
• The final quantitative value is a combination of the values of the
transformations in the Map Alignment step, with the geometric
measure given from the Jacobs Toolkit implementation
• M = αT+βA+γB
– T is the total translation of all the chops, and α is the Translational
weight factor
– A is the total absolute angular difference of all the chops, and β is the
Angular weight factor
– B is the geometrical difference between maps (derived from the
measure in Birk et al.) and γ is the Geometrical weight factor
• The weights in this equation can be modified to give more
emphasis to pose-based( increase γ) or geometric based (increase
α, β) applications
• For this measure, 0 is perfectly matched, and as the measure
increases the maps are less similar
Demo of Toolkit
Questions?