This paper presents a novel approach to robot task learning from language-based instructions, which focuses on increasing the complexity of task representations that can be taught through verbal instruction. The major proposed contribution is the development of a framework for directly mapping a complex verbal instruction to an executable task representation, from a single training experience. The method can handle the following types of complexities: 1) instructions that use conjunctions to convey complex execution constraints (such as alternative paths of execution, sequential or nonordering constraints, as well as hierarchical representations) and 2) instructions that use prepositions and multiple adjectives to specify action/object parameters relevant for the task. Specific algorithms have been developed for handling conjunctions, adjectives and prepositions as well as for translating the parsed instructions into parameterized executable task representations. The paper describes validation experiments with a PR2 humanoid robot learning new tasks from verbal instruction, as well as an additional range of utterances that can be parsed into executable controllers by the proposed system.

REU Site: Collaborative Human-Robot Interaction, National Science Foundation PI: David Feil-Seifer, co-PI: Shamik Sengupta, Amount: $360,000, Feb. 1, 2018 - Jan. 31, 2022

Designing Collaborator Robots for Highly-Dynamic Multi-Human, Multi-Robot Teams, Office of Naval Research (ONR) PI: Monica Nicolescu, co-PI: David Feil-Seifer, Amount: $656,511, April 1, 2016 - March 31, 2019

DURIP: Humanoid Platforms for Human-Robot Collaboration, Office of Naval Research (ONR) PI: Monica Nicolescu, co-PI: David Feil-Seifer, Mircea Nicolescu, Amount: $312,000, May 1, 2014 - June 30, 2015