Learning How a Tool Affords by Simulating 3D Models from the Web

Paulo Abelha Ferreira, Frank Guerin

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

22 Citations (Scopus)
10 Downloads (Pure)


Robots performing everyday tasks such as cooking in a kitchen need to be able to deal with variations in the household tools that may be available. Given a particular task and a set of tools available, the robot needs to be able to
assess which would be the best tool for the task, and also where to grasp that tool and how to orient it. This requires an understanding of what is important in a tool for a given task, and how the grasping and orientation relate to performance in the task. A robot can learn this by trying out many examples.
This learning can be faster if these trials are done in simulation using tool models acquired from the Web. We provide a semi-automatic pipeline to process 3D models from the Web, allowing us to train from many different tools and their uses in simulation. We represent a tool object and its grasp and
orientation using 21 parameters which capture the shapes and sizes of principal parts and the relationships among them. We then learn a ‘task function’ that maps this 21 parameter vector to a value describing how effective it is for a particular task. Our trained system can then process the unsegmented point cloud of a new tool and output a score and a way of using the tool for a
particular task. We compare our approach with the closest one in the literature and show that we achieve significantly better results.
Original languageEnglish
Title of host publicationProceedings of IEEE International Conference on Intelligent Robots and Systems (IROS 2017)
PublisherIEEE Press
Number of pages7
ISBN (Electronic)978-1-5386-2681-8, 978-1-5386-2682-5
Publication statusPublished - 28 Sept 2017
Event2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) -
Duration: 24 Sept 201728 Sept 2017


Conference2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Bibliographical note

Thanks to: UoAs ABVenture Zone, N. Petkov, K. Georgiev, B. Nougier, S. Fichtl, S. Ramamoorthy, M. Beetz, A. Haidu, J. Alexander, M. Schoeler, N. Pugeault, D. Cruickshank, M. Chung and N. Khan.

Paulo Abelha is on a PhD studentship supported by the Brazilian agency
CAPES through the program Science without Borders. Frank Guerin received
no specific grant from any funding agency in the public, commercial,
or not-for-profit sectors.

Published in: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) DOI: 10.1109/IROS.2017.8206372
Date of Conference: 24-28 Sept. 2017
Conference Location: Vancouver, BC, Canada.


  • tools
  • three dimensional displays
  • solid modelling


Dive into the research topics of 'Learning How a Tool Affords by Simulating 3D Models from the Web'. Together they form a unique fingerprint.

Cite this