Universität Bonn: Autonomous Intelligent Systems Group   Autonomous Intelligent Systems: NimbRo Picking

Amazon Research Award

Our proposal for the Amazon Research Award project "Learning Scene Parsing for Cluttered Bin Picking" was funded.

Amazon Research Awards

Amazon Robotics Challenge 2017, Nagoya, Japan

Amazon Robotics Challenge

Our team NimbRo Picking won the 2nd price in the Pick task and came in 2nd in the final Stow and Pick task.

Invited presentation at the ICRA 2018 Workshop on Advances in Robotic Warehouse Automation, Brisbane, Australia:
"Team NimbRo Picking at ARC 2017: Fast Learning Semantic Perception and Coordinating Two Arms"

The paper
Max Schwarz, Christian Lenz, German Martin Garcia, Seongyong Koo, Arul Selvam Periyasamy, Michael Schreiber, and Sven Behnke:
Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing
has been accepted for the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, to appear May 2018.

In the Finals, both stow and pick needed to be performed by the robotic system. Our robot stored 14 of the 16 items provided in a tote into our storage system, where additional 16 items were placed manually before. Afterwards, it picked eight of the possible nine requested items and placed them in the target card boxes. This resulted in 235 points, second only to ACRV from Australia, who reached 272 points. The missing ninth item – a mesh cup made from thin wires – was grasped by one of the manipulators, but then put back into the storage system due to a failed weight check.

ARC Final: Stow ARC Final: Stow ARC Final: Stow ARC Final: Stow ARC 2017: Award Ceremony Amazon Robotics Challenge 2017: Team NimbRo Picking

Our team had a great run in the Picking task evaluation. All requested items were picked and placed successfully in the three target card boxes. The score reached was 245, second only to team Nanyang (Singapore, score 257).

After recovering from a serious system breakdown, which occurred due to an electrical short cut, our system performed the Stow task evaluation. The task was to stow items provided chaotically in a tote into the storage system. Due to an issue in the weight file, the measured weights of the picked objects were inconsistent with the expected weights and the manipulators kept dropping the picked items back into the tote, resulting in a score of only 20 points.

Object capture on a turntableTeam NimbRo Picking Stow preparations ARC 2017 NimbRo Picking System Setup

In the official practice run, our team reached the top score (150), followed by team IFL PiRo (KIT, Germany, 140), and team Nanyang (Nanyang Technological University, Singapore, 120). Teams could chose the practice task and the scores do not count towards the competition results. The score is promising, but there is still room for improvement.

Our system performed some practice runs. Many items were sucessfully picked and placed, but there is still much room for improvement.

Our team arrived at the RoboCup venue in Nagoya, Japan, and started to assemble and set-up the developed robotic system. Our system consist of two Universal Robots UR5 manipulators which are equipped with a flexible suction finger and a second finger for pinch grasps, RGB-D gantry sensors consisting of a Nikon DSLR camera and a Photoneo® PhoXi® 3D Scanner XL, a GPU server with four NVidia Titan GPUs, and scales below totes and boxes.
This year, as final the Stow and Pick has been added, where the robotic system must pick the items that it stored before. Teams can design their own storage system, subject to space restrictions. The density of items in the tote and the storage systems has been increased. Maybe the most challenging aspect is that not all items are known in advance, but half of the items are introduced to the system only few minutes before the competition run.

NimbRo Picking system assembly NimbRo Picking ARC 2017 Robot System ARC 2017 NimbRo Picking Sensor Setup: Photoneo® PhoXi® 3D Scanner XL and DSLR camera

We gratefully acknowledge Photoneo s. r. o. who provided for our system a Photoneo® PhoXi® 3D Scanner XL free of charge and Amazon Robotics who provided travel support.

Amazon Picking Challenge 2016

Our Amazon Picking Challenge 2016 software was released.

The paper Max Schwarz, Anton Milan, Arul Selvam Periyasamy, and Sven Behnke:
RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter
has been accepted for International Journal of Robotics Research (IJRR), Sage Publications, to apprear 2017.

The abstract Max Schwarz and Sven Behnke:
Data-efficient Deep Learning for RGB-D Object Perception in Cluttered Bin Picking
has been accepted for the Warehouse Picking Automation Workshop (WPAW), IEEE International Conference on Robotics and Automation (ICRA), Singapore, May 2017.


The paper by Max Schwarz, Anton Milan, Christian Lenz, Aura Munoz, Arul Selvam Periyasamy, Michael Schreiber, Sebastian Schüller, and Sven Behnke: NimbRo Picking: Versatile Part Handling for Warehouse Automation has been accepted for the IEEE International Conference on Robotics and Automation (ICRA), Singapore, May 2017.
Finalist for Best Automation Paper Award.


We presented our approach to the APC tasks in the CASE Workshop on Automation for Warehouse Logistics, Fort Worth, Texas, USA.
[Presentation slides]



In the Picking task, our robot picked 10 of the 12 items from the shelf and placed them in the tote. It received 97 points, coming in third in the overall ranking, next to TU Delft and the Japanese Team PFN.

Chris NimbRo Team The most difficult object to suck on Results of the Picking task  Team NimbRo Picking 


Our team came in second in the Stowing task of the Amazon Picking Challenge, next only to the team of TU Delft. All but one object has been stowed. 

Stowing Results Max and Arul Anton and Aura


We used our practice run for the Stowing task. Our system was working quite well. NimbRo recived 123 points, followed by Duke (15 points) and MIT (10 points).


Our team made good progress in setting up the robot for the challenge. Here is a first video of picking from the tote.


Our team NimbRo Picking arrived at Leipziger Messe and is setting up the robot for the Amazon Picking Challenge.

NimbRo Picking Team NimbRo Picking robot


Our team NimbRo Picking is qualified for the Amazon Picking Challenge 2016, which will take place June 28th - July 3rd, 2016 at Leipziger Messe, colocated with RoboCup 2016. The challenge tasks consist of picking a large variety of objects from a shelf and placing them in a tote, and of picking unordered objects from a tote and stowing them in the shelf. 

Team NimbRo Picking     Amazon Picking Challenge 

Related Projects

Universität Bonn, Institute for Computer Science, Departments: I, II, III, IV, V, VI; Team NimbRo; Robotics in Bonn