Self-Supervised Learning for Precise Pick-and-Place Without Object Model

Lars Berscheid*, Pascal Meissner, Torsten Kröger

*Corresponding author for this work

Research output: Contribution to journalLetterpeer-review

38 Citations (Scopus)
6 Downloads (Pure)


Flexible pick-and-place is a fundamental yet challenging task within robotics, in particular due to the need of an object model for a simple target pose definition. In this work, the robot instead learns to pick-and-place objects using planar manipulation according to a single, demonstrated goal state. Our primary contribution lies within combining robot learning of primitives, commonly estimated by fully-convolutional neural networks, with one-shot imitation learning. Therefore, we define the place reward as a contrastive loss between real-world measurements and a task-specific noise distribution. Furthermore, we design our system to learn in a self-supervised manner, enabling real-world experiments with up to 25000 pick-and-place actions. Then, our robot is able to place trained objects with an average placement error of 2.7 (0.2) mm and 2.6 (0.8){\deg}. As our approach does not require an object model, the robot is able to generalize to unknown objects while keeping a precision of 5.9 (1.1) mm and 4.1 (1.2){\deg}. We further show a range of emerging behaviors: The robot naturally learns to select the correct object in the presence of multiple object types, precisely inserts objects within a peg game, picks screws out of dense clutter, and infers multiple pick-and-place actions from a single goal state.
Original languageEnglish
Pages (from-to)4828 - 4835
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number3
Early online date19 Jun 2020
Publication statusPublished - 1 Jul 2020

Bibliographical note

We would like to thank Tamim Asfour for his helpful suggestions and discussions.


  • reinforcement learning
  • Deep learning in grasping and manipulation
  • imitation learning


Dive into the research topics of 'Self-Supervised Learning for Precise Pick-and-Place Without Object Model'. Together they form a unique fingerprint.

Cite this