Source: PR2
Michigan, June 19th 2016, at R:SS

R:SS 2016 workshop on

Bootstrapping Manipulation Skills


When we talk about "learning a manipulation skill" in robotics, the robot is often learning only one component of the skill, e.g., the preconditions, the actions, or the effects. For example, "learning to grasp" often focuses on learning a suitable grasp frame, while assuming that the grasp controller and the success/failure conditions are predefined. In this manner, the predefined components serve to define the learning problem for the remaining component. However, relying on predefined components ultimately limits the autonomy of the robot. Instead, robots should bootstrap the skill acquisition process and learn multiple components together.

In recent years, researchers have proposed a number of paradigms for learning multiple skill components. These approaches include learning components sequentially using scaffolding, decomposing demonstrations using segmentation, and reusing components between skills with transfer learning. Researchers have also been exploring different information sources for bootstrapping skills, such as learning from demonstration, crowdsourcing, learning from planning, and self-exploration.

In this workshop, we want to bring together researchers to discuss different approaches for bootstrapping the skill learning process. Central to the discussion will be three key questions:

  1. What are the different components of manipulation skills that robots can learn?
  2. What (external) information sources can be used to initialize skill bootstrapping?
  3. Which learning frameworks can be used to acquire multiple components together?
In order to encourage discussion amongst participants, the workshop will include multiple short poster sessions spread across the day and invited talks that cover different aspects of skill bootstrapping. The workshop will also include a brainstorming session aimed at the workshop's three core questions. The goal of the workshop is to identify organizing principles/taxonomies of information sources and paradigms for bootstrapping manipulation skills. The workshop will thus help to identify the state of the art, as well as future trends and open problems, in bootstrapping manipulation skills.

Invited Speakers

  1. Maya Cakmak (University of Washington)
  2. Sonia Chernova (Georgia Institute of Technology)
  3. Kris Hauser (Duke University)
  4. Joseph Lim (Stanford)
  5. Ashutosh Saxena (Cornell University and Stanford University)
  6. Stefanie Tellex (Brown University)
  7. Emre Ugur (Boğaziçi University and Innsbruck University)


Invited Speaker Sessions

In order to provide a broad overview, we have invited an international group of renowned researchers to present their recent work and insights on this topic. Each presentation will be followed by 5-10 minutes of questions and answers.

Poster Sessions

In order to further encourage discussion amongst participants and to highlight some of the newest developments in the field, the workshop will also include two poster sessions. These sessions are a great way for researchers to present their latest results and discuss their current research with their peers.

Brainstorming and Discussion Sessions

The workshop will also include discussion and brainstorming sessions. The brainstorming session will focus on the three key questions at the core of the workshop. The discussion session will take place at the end of the day and focus on the current state of the art, and open problems in bootstrapping manipulation skills.


8:30 AM - Introduction
8:45 AM - Talk by Emre Ugur on
Bootstrapping Symbols from Continuous Manipulative Exploration
9:15 AM - Poster spotlight talks
9:30 AM - Poster Session I
10:00 AM - -- Coffee Break --
10:30 AM - Talk by Joseph Lim
11:00 AM - Talk by Scott Niekum
11:30 AM - Brainstorming Session
12:00 PM - -- Lunch Break --
2:00 PM - Talk by Ashutosh Saxena (slides) on
Bootstrapping Manipulation by Learning and Sharing Knowledge
2:30 PM - Talk by Stefanie Tellex (slides) on
Automatically Learning to Detect, Localize, and Manipulate Objects
3:00 PM - Poster Session II
3:30 PM - -- Coffee Break --
4:00 PM - Talk by Kris Hauser (slides) on
Reliable Physics Simulation of Underactuated Compliant Hands
for Manipulation Skill Learning
4:30 PM - Talk by Sonia Chernova on
Learning Manipulation Tasks from Web-Based User Demonstrations
5:00 PM - Final Discussion
5:30 PM Workshop Ends

Call for Contributions

We are soliciting submissions of extended abstracts of 1-2 pages on recent work that is relevant to the workshop. We welcome any work that explores learning for manipulation skills. We especially encourage submissions that investigate learning multiple skill components, or a novel source of information for bootstrapping the skill learning process. Abstracts should be submitted as PDF files using the same format as for the main R:SS conference submissions.

Submissions should be submitted by email to:

We will send you a confirmation once when we receive the submission. Abstracts will be single-blind reviewed on the basis of relevance, significance, and clarity. The accepted contributions will be presented as posters during the workshop. The authors may also be given the chance to additionally give a short spotlight presentation of their work.

Submission deadline: 26 May 2016

Notification of acceptance: 3 June 2016

Submitted abstracts will be reviewed as soon as possible. Authors that submit their abstract early may therefore also receive the notification of acceptance earlier.

Accepted Posters

  1. Learning Robot In-Hand Manipulation with Tactile Features; Herke van Hoof, Gerhard Neumann, Nutan Chen, Patrick van der Smagt, Maximilian Karl, Tucker Hermans, and Jan Peters abstract
  2. Manipulation Behaviors and Skill Learning; Juan Rojas abstract
  3. Learning to Switch between Sensorimotor Primitives using Multimodal Haptic Signals; Zhe Su, Oliver Kroemer, Gerald E. Loeb, Gaurav S. Sukhatme, and Stefan Schaal abstract
  4. Functional Object-Oriented Network for Manipulation Learning; David Paulius, Yongqiang Huang, Ahmad Babaeian, Roger Milton, William D. Buchanan, Jeanine Sam, and Yu Sun abstract
  5. The Manifold Particle Filter for State Estimation on High-dimensional Implicit Manifolds; Matthew Klingensmith, Michael C. Koval, Siddhartha S. Srinivasa, Nancy S. Pollard, and Michael Kaess
  6. Combining Grasp Pose Detection with Object Detection; Andreas ten Pas, Kate Saenko, and Robert Platt abstract
  7. HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards; Sanjay Krishnan, Animesh Garg, Richard Liaw, Lauren Miller, Florian T. Pokorny, and Ken Goldberg
  8. Exploration of Unknown Dynamic Environments: A Visual Saliency-Based Babbling Approach; Leni K. Le Goff, Pierre-Henri Le Fur, and Stephane Doncieux
  9. Learning Affordances in Environments with Incomplete Information; Carlos Maestre, Christophe Gonzales, and Stephane Doncieux

Workshop Organizers

Oliver Kroemer, University of Southern California
Scott Niekum, University of Texas at Austin
George Konidaris, Duke University
Manuel Lopes, INRIA

Should you have any questions, please contact the organizers through email.


Bootstrapping Manipulation Skills
Oliver Kroemer, Scott Niekum, George Konidaris, Manuel Lopes