CSCI598A Robot Intelligence, Spring 2015
Page 1 of 4
Midterm Project
Robot Perception and Manipulation of Objects
Assigned: February 12, 2015
Multiple Due Dates (all materials must be submitted via BlackBoard)
Proposal Report Due: February 19, 23:59:59
Progress Report Due: March 5, 23:59:59
Code and Video Demo Due: March 19, 23:59:59
Final Report Due: March 23, 23:59:59
Demonstration/Competition: March 24, 15:30 – 16:45 in BBW325
Introduction
In this project, the goal is to implement all individual components on a Baxter robot that are required
for the Amazon Picking Challenge. This project is an opportunity for you to work as a team and play with
an advanced physical robot to solve real-world problems. This project must be done as a team.
Objectives
As a simple version of the Amazon Picking Challenge, the objective of the midterm project is to
recognize objects placed on a table top in front
of the Baxter robot (i.e., detecting and classifying
objects from 27 categories using both 2D and 3D
visual perception), estimate the contact point for
grasping, pick up these objects, and place them
into an order bin that is also placed on the table,
as illustrated in Fig. 1.
Accordingly, each team is required to implement
approaches to accomplish the following tasks:
1. Object recognition (including both detection
and classification) from 2D visual data collected
from color cameras on the Baxter robot’s wrists. Figure 1. Experiment setup..
2. Object recognition from 3D sensing data collected using a Kinect sensor that is installed on top of the Baxter’s head.
3. Contact point estimation for grasping.
4. Motion planning for object picking (from the table top) and placing (into the bin).
CSCI598A Robot Intelligence, Spring 2015
Page 2 of 4
Teams
Given the significant challenges in this project, the class is divided only into two teams:
Team 1
Team 2
Scoring
The midterm project will be evaluated using similar scoring rules in the Amazon Picking Challenge. The
table top will be divided into three areas: multi-item, double-item, and single-item areas. The scoring
rules are described as follows:
Moving a target item from a multi-item area into the order bin
+20 points
Moving a target item from a double-item area into the order bin
+15 points
Moving a target item from a single-item area into the order bin
+10 points
Moving a non-target item from the table top
-12 points
Damaging any item or packaging
-5 points
Dropping a target item from a height above 0.3 meters
-3 points
Important Note
Don’t forget to use the Berkeley Amazon Picking Challenge Object Dataset for learning and initial testing:
http://rll.berkeley.edu/amazon_picking_challenge/
Turning in your project
CSCI598A Robot Intelligence, Spring 2015
Page 3 of 4
The Midterm Project has multiple due dates. All materials must be submitted by the team leader via
Blackboard. The contents required to be included in each submission are listed as follows:
Proposal Report (due on February 19, 23:59:59):
(1) Provide a brief description of approaches you plan to use to address each objective;
(2) Include a timeline and workload breakdown, especially who will work on which tasks, to show
your plan of finishing the project.
Progress Report (due on March 5, 23:59:59):
(1) Describe the progress of your midterm project (e.g., preliminary results on Berkeley datasets,
working software and packages, etc.);
(2) Include a workload report (agreed upon by all members) that states what each team member
did on this project, along with a percentage breakdown (totaling 100%).
Note: It is not necessary to use Latex to prepare the Progress Report.
Code and Video Demo (due on March 19, 23:59:59):
(1) Submit a video demo of the midterm project (i.e., how the robot accomplishes the tasks);
(2) Submit the code that can produce the demo;
(3) Include a detailed instruction of how to compile, use and test the code;
(4) In general, your code must compile, be well-documented, and run without crashing.
Final Report (due on March 23, 23:59:59):
Please follow the detailed guidelines of final report in the next section.
In-lab Demo/Competition (on March 24, 15:30 – 16:45 in BBW325):
During that period, you will have a chance to show off your awesome implementation and compete with the other team. Good luck!
Final Report/Paper Guidelines
Similar to the Mini Project, as part of your completed midterm project, each team must prepare a paper (4-6 pages) describing your project. Your paper must be formatted using Latex and the standard 2column IEEE conference paper format (details of Latex and IEEE conference format was discussed in
Mini Project). Please do not exceed 6 pages for your paper.
The paper you turn in must be in pdf format and to BlackBoard by the team leader. Your paper must at
least include the following sections:
 Abstract: An abstract of 200 to 300 words summarizing your project and findings.
 Introduction: An introduction describing your project, including the task and formulation of the
problems, a brief description of your methods and structure of your paper.
CSCI598A Robot Intelligence, Spring 2015




Page 4 of 4
Approach: A detailed description of your approaches, with enough information to understand
and enable someone to recreate your system.
Experiments: Experimental results plus an explanation and discussion of the results, such as in
what situations your system can obtain the best performance, when it fails, the efficiency of
your program, etc.
Conclusion: A conclusion and future work section that summaries your midterm project, point
out future work you believe would improve your implementation for the Amazon Picking Challenge, and any other insightful observations you’d like to make.
Appendix: A workload report must be included in Appendix (agreed upon by all team members, which states what each team member did on this project in detail, along with a percentage breakdown (totaling 100%).
Grading
Your grade (totally 100 points) will be based on:
 5 points: Your proposal report,
 5 points: Your progress report,
 45 points: Your documented code and video demos,
 40 points: Your final paper,
 5 points: in-lab demo/competition (Grading won’t depend on the task scoring).
How grading will be done for teams: All members of the team will receive the same grade as a starting
point, based on the submission. Then, the Workload Report information (submitted with Progress Report and Final Report) either results in the scores staying the same, or it can result in one student’s
score moving down by some amount (maximum of 10 points). Be aware it is the team leader’s responsibility to coordinate all team members. If there is a potential conflict, the team leader should directly
contact the instructor.