Goal tracking in a natural language interface: towards achieving adjustable autonomy (original) (raw)

Goal tracking and goal attainment: a natural language means of achieving adjustable autonomy

Intelligent mobile robots that interact with humans must be able to exhibit adjustable autonomy, that is the ability to dynamically adjust the level of autonomy of an agent depending on the situation. When intelligent robots require close interactions with humans, they will require modes of communication that enhance the ability for humans to communicate naturally and that allow greater interaction. Our previous work examined the use of multiple modes of communication, specifically natural language and gestures, to disambiguate the communication between a human and a robot. In this paper, we propose using context predicates to keep track of various goals during human-robot interactions. These context predicates allow the robot to maintain multiple goals, each with possibly different levels of required autonomy. They permit direct human interruption of the robot, while allowing the robot to smoothly return to a high level of autonomy.

Human-Robot Dialogue and Collaboration in Search and Navigation

2018

Collaboration with a remotely located robot in tasks such as disaster relief and search and rescue can be facilitated by grounding natural language task instructions into actions executable by the robot in its current physical context. The corpus we describe here provides insight into the translation and interpretation a natural language instruction undergoes starting from verbal human intent, to understanding and processing, and ultimately, to robot execution. We use a ‘Wizard-of-Oz’ methodology to elicit the corpus data in which a participant speaks freely to instruct a robot on what to do and where to move through a remote environment to accomplish collaborative search and navigation tasks. This data offers the potential for exploring and evaluating action models by connecting natural language instructions to execution by a physical robot (controlled by a human ‘wizard’). In this paper, a description of the corpus (soon to be openly available) and examples of actions in the dialo...

Towards Goal Inference for Human-Robot Collaboration

2020

Natural language instructions often leave a speaker’s intent under specified or unclear. We propose a goal inference procedure that extracts user intent using natural language processing techniques. This procedure uses semantic role labeling and synonym generation to extract utterance semantics, and then analyzes a task domain to infer the user’s underlying goal. This procedure is designed as an extension to the MIDCA cognitive architecture that enables human-robot collaboration. In this work, we describe a conceptual model of this procedure, lay out the steps a robot follows to make a goal inference, give an example use case, and describe the procedure’s implementation in a simulated environment. We close with a discussion of the benefits and limitations of this approach. We expect this procedure to improve user satisfaction with agent behavior when compared to planbased dialogue systems.

Speech and action: integration of action and language for mobile robots

2007

We describe the tight integration of incremental natural language understanding, goal management, and action processing in a complex robotic architecture, which is required for natural interactions between robots and humans. Specifically, the natural language components need to process utterances while they are still spoken to be able to initiate feedback actions in a timely fashion, while the action manager might need information at various points during action execution that must be obtained from humans. We argue that a finergrained integration provides much more natural human-robot interactions and much more reasonable multitasking.

A generic natural language interface for task planning — application to a mobile robot

Control Engineering Practice, 2000

This paper presents a generic natural language interface that can be applied to the teleoperation of di!erent kinds of complex interactive systems. Through this interface the operators can ask for simple actions or more complex tasks to be executed by the system. Complex tasks will be decomposed into simpler actions generating a network of actions whose execution will result in the accomplishment of the required task. As a practical application, the system has been applied to the teleoperation of a real mobile robot, allowing the operator to move the robot in a partially structured environment through natural language sentences.

Make It So: Continuous, Flexible Natural Language Interaction with an Autonomous Robot

2012

Abstract While highly constrained language can be used for robot control, robots that can operate as fully autonomous subordinate agents communicating via rich language remain an open challenge. Toward this end, we developed an autonomous system that supports natural, continuous interaction with the operator through language before, during, and after mission execution.

Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming

arXiv (Cornell University), 2023

In a rapidly evolving digital landscape autonomous tools and robots are becoming commonplace. Recognizing the significance of this development, this paper explores the integration of Large Language Models (LLMs) like Generative pre-trained transformer (GPT) into human-robot teaming environments to facilitate variable autonomy through the means of verbal human-robot communication. In this paper, we introduce a novel simulation framework for such a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with simulated robot agents through natural language, each powered by individual GPT cores. By means of OpenAI's function calling, we bridge the gap between unstructured natural language input and structured robot actions. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a simulated multi-robot environment. Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their simulated robot collaborators. Still, those users who did explore where able to benefit from a much more natural flow of communication and human-like back-and-forth. We provide a set of lessons learned for future research and technical implementations of similar systems.

A Semantic Approach to Enhance Human-Robot Interaction in AmI Environments

This paper presents a semantic approach for human-robot interaction in ambient intelligence environments (AmI). This approach is intended to provide a natural way for multimodal interactions between human and artificial agents embodied in cognitive companion robots or in any AmI smart device. It is applied for building cognitive assistance services based on the semantic observation and communication. The main contribution of this paper concerns the proposition of a semantic module, that allows, on the one hand, converting natural language dialogues to formal n-ary ontology knowledge, and on the other hand, making semantic inference on this knowledge to drive the dialogues and trigger actions. The target ontology language is NKRL (Narrative Knowledge Representation Language). This latter provides a semantic formal basis allowing narrative representation and reasoning on complex contexts by building spatio-temporal relationships between events. A scenario dedicated to the monitoring a...

Using Natural Language to Enable Mission Managers to Control Multiple Heterogeneous UAVs

Advances in Intelligent Systems and Computing, 2016

The availability of highly capable, yet relatively cheap, unmanned aerial vehicles (UAVs) is opening up new areas of use for hobbyists and for commercial activities. This research is developing methods beyond classical controlstick pilot inputs, to allow operators to manage complex missions without indepth vehicle expertise. These missions may entail several heterogeneous UAVs flying coordinated patterns or flying multiple trajectories deconflicted in time or space to predefined locations. This paper describes the functionality and preliminary usability measures of an interface that allows an operator to define a mission using speech inputs. With a defined and simple vocabulary, operators can input the vast majority of mission parameters using simple, intuitive voice commands. Although the operator interface is simple, it is based upon autonomous algorithms that allow the mission to proceed with minimal input from the operator. This paper also describes these underlying algorithms that allow an operator to manage several UAVs.