Overview of Amazon Picking Challenge 2015
The first Amazon Picking Challenge (APC) was held at the 2015 International Conference on Robotics
and Automation in Seattle Washington, May 26-27. The objective of the competition was to provide a
challenge problem to the robotics research community that involved integrating the state of the art in
object perception, motion planning, grasp planning, and task planning in order to manipulate realworld items in industrial settings. To that end, we posed a simplified version of the task that many
humans face in warehouses all over the world: pick items out of shelves and put them into containers.
In this case, the shelves were prototypical pods from Kiva Systems and the picker had to be a fully
autonomous robot.
The items were a pre-selected set of 24 products that were commonly sold on Amazon.com and which
we expected would pose varying degrees of difficulty for the contestants. On the easier end were simple
cuboids like a box of straws or a spark plug. Some items were chosen because they were easy to
damage, like the two soft-cover books, or the package of crushable Oreo cookies. Others were harder
to perceive and grasp, like the unpackaged dog toys or the black mesh pencil holder. The box of
Cheez-Its posed a challenge because it couldn’t be removed from the bin without twisting it sideways.
Each pod had 12 bins, and the 24 products were distributed among the bins in such a way that each
competitor had the same challenges. Each bin had one target item to be picked with a base score of 10,
15, or 20 points depending on how many other items were in the bin. In addition, some items that were
projected to be more difficult to pick were given 1 to 3 bonus points. Damaging an item incurred a 5
point penalty, while picking the wrong item incurred a 12 point penalty. Each competitor had 20
minutes to pick as many of the 12 target items as possible and could score as many as 190 points.
The competition was announced October 1, 2014. Through a series of video submissions, the
organizers selected 25 teams to receive equipment grants (sample pods and products) and travel grants
to help defray the costs of travel to the venue. In addition, Amazon provided $26,000 in prize money
for the winning teams.
26 teams from 11 countries made the trip to Seattle to try their robot’s hand at picking out of Kiva
pods. The success of the teams was mixed, but the enthusiasm and excitement was contagious. The
competition was won by RBO from the Technical University of Berlin. Their device, with a Barrett arm, a
Nomadic Technologies mobile platform, and a suction cup attached to a commercial vacuum cleaner
was able to successfully pick 10 of 12 correct items in under 20 minutes. Their score of 148 points put
them well into the lead. MIT placed second with seven items picked and 88 points. Their entry used an
industrial ABB arm and a scooper end-effector that could be flipped over to alternatively use a small
suction cup. Third place was Team Grizzly from Dataspeed Inc. & Oakland University with 35 points.
Their solution used a Baxter robot attached to their own custom mobile base.
The final scores are shown in Table 1. Many teams demonstrated successful picking in their warm ups
but for various reasons failed in their official 20-minute attempt. The reasons for failure varied widely,
and included last minute code changes, failure to model how a vacuum hose would twist around the
arm in certain poses, grippers that were so big that they couldn’t figure out how to get in the bin, etc.
However, even the systems that failed to pick any items demonstrated interesting robots, end effectors,
and technical approaches. Overall, 36 correct items were picked and 7 incorrect items were picked.
Team
Affiliation
Items Picked
Score
RBO
T.U. Berlin
10
148
Team MIT
M.I.T.
7
88
Grizzly
DataSpeed & Oakland U.
3
35
NUS Smart Hand
National University of Singapore
2
32
Zhejiang U., Univ. of Tech. Sydney & Nanjiang
1
Z.U.N.
Robotics Co.
Chubu U., Chukyo U., & Mitsubishi Electric
23
2
C^2M
Corporation
21
R U Pracsys
Rutgers
1
17
Team K
JSK, University of Tokyo
4
15
Nanyang
Nanyang Technological University
1
11
A.R.
Netherlands
1
11
Team Georgia
Georgia Tech
1
Tech
10
Team Duke
Duke University
1
10
CVAP
KTH (Sweden)
2
9
Other teams competing included Worcester Polytechnic, University of Texas at Austin, University of
Texas at Arlington, University of Washington, University of Alberta, Robological PTY LTD, Universitat
Jaume I, University of Colorado at Boulder, Colorado School of Mines, University of Pisa, University of
California at Berkeley, Dorabot and the University of Hong Kong, and St. Francis Institute of Technology
in India. The teams were supported by several hardware vendors, including Rethink Robotics, Barrett
Technologies, Yaskawa, Olympus Controls, and Clearpath Robotics.
The first APC was very successful, drawing a large number of competitors from around the world and
demonstrated the state of the art in both the software and the hardware required for robotic
manipulation. Despite being scattered over 16 testing bays in the ICRA competition area and spread
over two days, every team drew a large crowd of spectators eager to see how the robots would perform.
Contest Details
Prior to ICRA 2015 each team will be responsible for coordinating transportation of their system to
Seattle, WA. The contest organizers and ICRA challenge committee will assist with shipping details. A
setup bullpen will be arranged for teams to prepare their systems, practice, and demonstrate their work
to the wider ICRA audience. Prior to each official attempt, the team will be given time to set up their
system in front of the test shelf. At the start of the attempt the team will be given their work order and
shelf contents via the pre-defined Contest Interface. The work order and shelf content data will contain
all the items that must be picked, along with a list of the other contents of the shelf bins. The robot will
then be responsible for pulling as many of the items as possible out of the shelf bins and setting them
down inside the robot area within the allotted time. Points will be awarded for successfully picked items,
and deducted for dropped, damaged, or incorrectly picked items.
Shelving System
The shelving system will be a steel and cardboard structure seen in the CAD model below. A STL
and Gazebo model of the shelf is available for download on the picking challenge website, as well as a
physical copy sent to teams awarded practice equipment. Only a single face of the shelf will be
presented to the teams, but the stocking of the items will be changed pseudo-randomly between
attempts. Only a subset of 12 bins on the shelf face will be used. The subset is a patch of the shelf face
covering bins inside a roughly 1 meter x 1 meter area. The base of the first shelf from the floor is at a
height of approximately 0.78 meters. Each bin will contain one or more items. A single item in each bin
will be designated as a target item to be picked. Each bin is labeled (A-Z) and referred to in the Contest
Interface below. Teams will not be allowed to modify the shelf in any way prior to the competition (or
damage it during their attempt).
Items
The set of potential items that will be stocked inside the bins is shown in the graphic below. A list of all
the items can be found below. Note that the images on the Amazon shopping website are not
necessarily representative of the form the item will be in at the contest. The graphic below shows the
proper form (including things like packaging). The contest shelf may contain all the announced items,
or a partial subset of them. Teams awarded practice equipment will be provided a set of the contest
items. All items the teams are required to pick from the system will be a subset of this set. All items will
be sized and located such that they could be picked by a person of average height (170 cm) with one
hand.
http://www.amazon.com/gp/product/B0010XUO52/ref=ox_sc_act_title_1?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B00006JN7R/ref=ox_sc_act_title_2?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B00081J3N6/ref=ox_sc_act_title_3?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B00004YO15/ref=ox_sc_act_title_4?ie=UT
F8&psc=1&smid=A1T2QV8RUJMJWO
http://www.amazon.com/gp/product/B00BKL52YC/ref=ox_sc_act_title_5?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B002GJXSQ6/ref=ox_sc_act_title_6?ie=UT
F8&psc=1&smid=A3OUBK9YC1TQAT
http://www.amazon.com/gp/product/B00009OYGV/ref=ox_sc_act_title_7?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B00006IEES/ref=ox_sc_act_title_8?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B00006IEJC/ref=ox_sc_act_title_9?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B00176CGK8/ref=ox_sc_act_title_10?ie=U
TF8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B000GUZC2A/ref=ox_sc_act_title_1?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B008JBDMO8/ref=ox_sc_act_title_2?ie=UT
F8&psc=1&smid=A1G5V6RMJ146ZF
http://www.amazon.com/gp/product/B000CIQ47S/ref=ox_sc_act_title_3?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B0054YZDWC/ref=ox_sc_act_title_4?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/0486280616/ref=ox_sc_act_title_5?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B000BO6RWK/ref=ox_sc_act_title_6?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B000IVGBFY/ref=ox_sc_act_title_7?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B000E7AIA6/ref=ox_sc_act_title_8?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B000SHQ73Y/ref=ox_sc_act_title_9?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B0096XTJKO/ref=ox_sc_act_title_10?ie=U
TF8&psc=1&smid=A3G2RBEZBLAJ53
http://www.amazon.com/gp/product/B000MVYJ0O/ref=ox_sc_act_title_1?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B0006L0VMA/ref=ox_sc_act_title_2?ie=UT
F8&psc=1&smid=A2701FG002WN6Z
http://www.amazon.com/gp/product/B000N0SNHY/ref=ox_sc_act_title_3?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B008PNN8C6/ref=ox_sc_act_title_4?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/0800788036/ref=ox_sc_act_title_5?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B003YOON0C/ref=ox_sc_act_title_6?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B000Q3KHCM/ref=ox_sc_act_title_7?ie=UT
F8&psc=1&smid=ATVPDKIKX0DER
The Amazon Picking Challenge team at Berkeley has provided a wealth of 2D and 3D data on the
contest items by running them through the scanning system described in their ICRA 2014 work.
Detailed models and raw data can be downloaded at:
http://rll.berkeley.edu/amazon_picking_challenge/
Bin Stocking
For each attempt the bin location of the items will be changed. There will be bins that contain a single
item, and some that contain multiple items. Each bin will have only a single item that is a pick target.
The bin contents will be fully described by the contest Interface (below). The orientation of items within
the bin will be randomized (e.g. a book may be lying flat, standing vertically, or have its spine facing
inward or outward). Items will not occlude one another when looking from a vantage point directly in
front of the bin. Contestants can assume the following rough breakdown of bin contents:
•
Single-item bins: At least two bins will only contain one item (single-item bins). Both these
items will be picking targets.
•
Double-item bins: At least two bins will contain only two items (double-item bins). One item
from each of these bins will be a picking target.
•
Multi-item bins: At least two bins will contain three or more items (multi-item bins). One item
from each of these bins will be a picking target.
Duplicate items may be encountered. In the event that an item appears in the work order for a bin that
contains two of the same item in it either item can be picked (but not both).
Workcell Layout
The shelf will be placed in a fixed known stationary position prior to
the start of the attempt and the team allowed to perform any
calibration and alignment steps prior to the start to their timed
attempt (within the reasonable bounds of the contest setup time).
The precise alignment of the shelf with respect to the robot area will
be subject to slight deviations for natural reasons (and not
intentionally mis-aligned by the contest organizers). The team is not
allowed to move or permanently modify the shelf. If a team translates
the base of the shelf by more than 12 cm the attempt will be
stopped. An overhead view of the workcell layout can be seen in the
diagram below.
The team is allowed to place their robot anywhere within the
predefined robot workcell, but must have a starting gap of at least
10 cm from the shelf. The robot should be kept within the 2 meter X
2 meter square robot workcell for practical set-up & tear-down
purposes at the competition, but exceptions will be considered for
larger systems. Each robot must have an emergency-stop button that halts to motion of the robot. 110
Volt US standard power will be provided in the robot workcell.
Items need to be moved from the shelf into a container in the robot workcell referred to as the order
bin. Teams may put the order bin anywhere they prefer. The order bin is a plastic tote available from
the company U-Line (http://www.uline.com/Product/Detail/S-19473R/Totes-Plastic-StorageBoxes/Heavy-Duty-Stack-and-Nest-Containers-Red-24-x-15-x-8?model=S19473R&RootChecked=yes). Tables and other structures to support the bin can be brought with the
team equipment. Several table options will be provided on-site by the organizers. The robot must not
be supporting the order bin at the beginning or end of your challenge period, but it is allowed to pick
up and move the bin. If you end your run while still supporting the bin, your team will receive only half
the points of the items in the bin as your final score. If you drop the bin, it will count as dropping all of
the items in the bin. If a team has a question as to what might constitute 'supporting' the bin, or other
related questions, contact the mailing-list or committee and we can clarify for your situation.
Accidentally moving the order bin within the robot workcell will not be penalized. Moving the order bin
outside the workcell will result in no points being awarded for the objects in the bin.
Contest Interface
The contest Interface will consist of a single JSON file handed out to each time prior to the start of their
attempt that defines two things:
•
•
The item contents of all bins on the shelf face
The work order detailing what item should be picked from each bin
Rules
The first day of the contest will be used for practice attempts. Teams will be scored during practice, but
the outcome will not count toward prize evaluations. A schedule for scored attempts will be distributed
prior to the contest.
An attempt is defined as a single scored run. The attempt starts when the team is given the Contest
Interface file, and the attempt ends when one of the following conditions is met: the 20 minute time
limit expires, the leader of the team verbally declares the run is complete, or a human intervenes (either
remotely or physically) with the robot or shelf.
One hour prior to the start of an attempt the team will be given access to setup their system at the
contest shelf. Ten minutes prior to the start of the attempt the organizers will obscure the contestants'
view of the shelf and randomize the items on the shelf. Then, (2 minutes prior to the start of the
attempt) the organizers will give the team a JSON file containing the shelf layout and work order as
described in the Interface. The team will then be allowed to upload the data to their system, after which
the shelf will be unveiled and the attempt started. Items that appear in the work order can be picked in
any order.
No human interaction (remote or physical) is allowed with the robot after uploading the work order and
starting the robot. Note that this precludes any teleoperation or semi-autonomous user input to
the robot. Human intervention will end the attempt, and the score will be recorded as the score prior to
the intervention. Each team will have a maximum time limit of 20 minutes to complete the work order.
The team is free to declare the attempt over at any time prior to the end of the 20 minute period. The
number of attempts each team is allowed will be based on the number of participants (and decided
prior to the contest). In the event teams are allowed multiple attempts the highest score from each team
will be used to determine the contest outcome.
Robots that are designed to intentionally damage items or their packaging (such as piercing or
crushing) will be disqualified from the contest. Questionable designs should be cleared with the contest
committee prior to the competition.
Scoring
Points will be awarded for each target item removed from the shelf, and points will be subtracted for all
penalties listed below. The points awarded vary based on the difficulty of the pick.
Moving a target item from a multi-item shelf bin into the order bin
+20 points
Moving a target item from a double-item shelf bin into the order bin
+15 points
Moving a target item from a single-item shelf bin into the order bin
+10 points
+(0 to 3)
Target Object Bonus
points
Moving a non-target item out of a shelf bin (and not replacing it in the same
-12 points
bin)
Damaging any item or packaging
-5 points
Dropping a target item from a height above 0.3 meters
-3 points
Moving a target item from a multi-item shelf bin into the order bin: A target object picked from a
shelf bin that contains 3 or more total items. The item must be moved into the order bin.
Moving a target item from a double-item shelf bin into the order bin: A target object picked from a
shelf bin that contains 2 total items. The item must be moved into the order bin.
Moving a target item from a single-item shelf bin into the order bin: A target object picked from a
bin that contains only that item. The item must be moved into the order bin.
Target Object Bonus: An added point bonus uniquely specified for each different object. The bonus
points are added to the team's score for each target object that is successfully put in the order bin..
Moving a non-target item out of a bin (and not replacing it): A penalty is assigned for each item
removed (or knocked out of) the shelf that is not a target item. If the item is placed back on the shelf in
the same bin (by the robot) there is no penalty.
Damaging any item or packaging: A penalty is assigned for each item that is damaged (both target
and non-target items). Damaging will be assessed by the judges. An item can be considered damaged if
a customer would not be likely to accept the item if it was delivered in that condition.
Dropping a target item from a height above 0.3 meters: A penalty is assigned for an item dropped
from too high a height. Note that this only applies to target items.
For example: A team successfully picks two items out of single-item bins (+10 +10), and one item out
of a multi-item bin (+20). The first object was worth +0 bonus points, the second worth +3, and the
third worth +1. However, in the process of picking they knock another item out of its bin (-12) and it
falls to the floor. Also, when they go to place their first item down in the robot workspace it slips from
the hand 0.5 meters from the ground (-3). This would result in the team having a total score of 29
points.
In the event of a point tie the team that ended their run first (the team with the shortest time) will be
declared the winner. In the event that two teams are tied on points and time they will split the prize.
The omission of time as a critical challenge metric is intentional. In this first year of the picking
challenge we want to encourage participants to focus on picking as many items as possible, rather than
picking a select few items as fast as possible.
Prizes
To motivate contestants to focus on the full end-to-end task several prizes will be awarded to those
with the best performance (as judged by the contest Rules). In order to participate for the prize awards
teams must register their participation and sign the contest agreement by the contest enrollment
closure date. The top three teams will receive monetary prizes as listed below:
Prize (USD)
Minimum Score for Full Prize
1st Place
$20,000.00*
35 points
2nd Place
$5,000.00*
25 points
3rd Place
$1,000.00*
15 points
* All monetary prizes are conditional on eligible teams completing the minimum score criteria for that
prize as outlined below. In the event of a scoring and time draw the tied teams will split the prize for
their position. Prizes will not be awarded to teams with scores less than or equal to 0 points.
Minimum Score Criteria
To obtain the full 1st - 3rd place prizes a team must meet a minimum score criteria. If a team does place
in the prize pool but does not meet the minimum criteria for that prize they will be awarded half the
amount of the prize for that place. For example, if a team places 1st in the competition, but does not
meet the minimum 1st place score criteria, they will be awarded a $10,000 prize.