Integrating Pointing Gestures into Spanish-Spoken Dialog System for Conversational Service Robots (original) (raw)

Integrating Pointing Gestures into a Spanish-spoken Dialog System for Conversational Service Robots

International Conference on Agents and Artificial Intelligence, 2010

In this paper we present our work on the integration of human pointing gestures into a spoken dialog system in Spanish for conversational service robots. The dialog system is composed by a dialog manager, an interpreter that guides the spoken dialog and robot actions, in terms of user intentions and relevant environment stimuli associated to the current conversational situation. We demonstrate our approach by developing a tour-guide robot that is able to move around its environment, visually recognize informational posters, and explain sections of the poster selected by the user via pointing gestures. This robot also incorporates simple methods to qualify confidence in its visual outcomes, to inform about its internal state, and to start error-prevention dialogs whenever necessary. Our results show the reliability of the overall approach to model complex multimodal human-robot interactions.

Development of a Tour–Guide Robot Using Dialogue Models and a Cognitive Architecture

Lecture Notes in Computer Science, 2010

In this paper, we present the development of a tour–guide robot that conducts a poster session through spoken Spanish. The robot is able to navigate around its environment, visually identify informational posters, and explain sections of the posters that users request via pointing gestures. We specify the task by means of dialogue models. A dialogue model defines conversational situations, expectations and robot actions. Dialogue models are integrated into a novel cognitive architecture that allow us to coordinate both human–robot interaction and robot capabilities in a flexible and simple manner. Our robot also incorporates a confidence score on visual outcomes, the history of the conversation and error prevention strategies. Our initial evaluation of the dialogue structure shows the reliability of the overall approach, and the suitability of our dialogue model and architecture to represent complex human–robot interactions, with promising results.

A multi-modal dialog system for a mobile robot

Proc. Int. Conf. on Spoken Language …, 2004

A challenging domain for dialog systems is their use for the communication with robotic assistants. In contrast to the classical use of spoken language for information retrieval, on a mobile robot multimodal dialogs and the dynamic interaction of the robot system with its environment have to be considered. In this paper we will present the dialog system developed for BIRON -the Bielefeld Robot Companion. The system is able to handle multi-modal dialogs by augmenting semantic interpretation structures derived from speech with hypotheses for additional modalities as e.g. speech-accompanying gestures. The architecture of the system is modular with the dialog manager being the central component. In order to be aware of the dynamic behavior of the robot itself, the possible states of the robot control system are integrated into the dialog model. For flexible use and easy configuration the communication between the individual modules as well as the declarative specification of the dialog model are encoded in XML. We will present example interactions with BIRON from the "home-tour" scenario defined within the COGN-IRON project.

Interacting with a Robot: A Guide Robot Understanding Natural Language Instructions

Lecture Notes in Computer Science, 2012

A guide robot requires to accurately and robustly capture, process and understand the human order and to generate the correct behaviour for giving the correct answer to the asked task. To automatically perform that natural Human Robot interaction, we propose a solution combining natural language processing techniques, semantic technologies and autonomous navigation techniques able to guide people in a museum environment. We have evaluated the solution in a real scenario with different people and the results are promising.

Augmented Robotics Dialog System for Enhancing Human–Robot Interaction

Sensors, 2015

Augmented reality, augmented television and second screen are cutting edge technologies that provide end users extra and enhanced information related to certain events in real time. This enriched information helps users better understand such events, at the same time providing a more satisfactory experience. In the present paper, we apply this main idea to human-robot interaction (HRI), to how users and robots interchange information. The ultimate goal of this paper is to improve the quality of HRI, developing a new dialog manager system that incorporates enriched information from the semantic web. This work presents the augmented robotic dialog system (ARDS), which uses natural language understanding mechanisms to provide two features: (i) a non-grammar multimodal input (verbal and/or written) text; and (ii) a contextualization of the information conveyed in the interaction. This contextualization is achieved by information enrichment techniques that link the extracted information from the dialog with extra information about the world available in semantic knowledge bases. This enriched or contextualized information (information enrichment, semantic enhancement or contextualized information are used interchangeably in the rest of this paper) offers many possibilities in terms of HRI. For instance, it can enhance the robot's pro-activeness during a human-robot dialog (the enriched information can be used to propose new topics during the dialog, while ensuring a coherent interaction). Another possibility is to display additional multimedia content related to the enriched information on a visual device. This paper describes the ARDS and shows a proof of concept of its applications.

Human-robot interaction through spoken language dialogue

2000

Abstract The development of robots that are able to accept instructions, via a friendly interface, in terms of concepts that are familiar to a human user remains a challenge. It is argued that designing and building such intelligent robots can be seen as the problem of integrating four main dimensions: human-robot communication, sensory motor skills and perception, decision-making capabilities, and learning.

On developing a voice-enabled interface for interactive tour-guide robots

Advanced Robotics, 2003

This paper considers design methodologies in order to develop voice-enabled interfaces for tour-guide robots to be deployed at the Robotics Exposition of the Swiss National Exhibition (Expo.02). Human-robot voice communication presents new challenges for design of fully autonomous mobile robots, in that interactivity must be robot-initiated in conversation and within a dynamic adverse environment. We approached these general problems for a voice enabled interface, tailored to limited computational resources of one on-board processor, when integrating smart speech signal acquisition, automatic speech recognition and synthesis, as well as dialogue system into the multi-modal, multisensor interface for the expo tour-guide robot. We also focus on particular issues that need to be addressed in voice-based interaction when planning specific tasks and research experiments for Expo.02 where tour-guide robots will interact with hundred of thousands of visitors during six months, seven days a week, ten hours per day.

A dialogue system for multimodal human-robot interaction

Proceedings of the 15th ACM on International conference on multimodal interaction - ICMI '13, 2013

This paper presents a POMDP-based dialogue system for multimodal human-robot interaction (HRI). Our aim is to exploit a dialogical paradigm to allow a natural and robust interaction between the human and the robot. The proposed dialogue system should improve the robustness and the flexibility of the overall interactive system, including multimodal fusion, interpretation, and decision-making. The dialogue is represented as a Partially Observable Markov Decision Process (POMDPs) to cast the inherent communication ambiguity and noise into the dialogue model. POMDPs have been used in spoken dialogue systems, mainly for tourist information services, but their application to multimodal humanrobot interaction is novel. This paper presents the proposed model for dialogue representation and the methodology used to compute a dialogue strategy. The whole architecture has been integrated on a mobile robot platform and has been tested in a human-robot interaction scenario to assess the overall performances with respect to baseline controllers.

Intuitive multimodal interaction and predictable behavior for the museum tour guide robot Robotinho

2010

Deploying robots at public places exposes highly complex systems to a variety of potential interaction partners of all ages and with different technical backgrounds. Most of these individuals may have never interacted with a robot before. This raises the need for robots with an intuitive user interface, usable without prior training. Furthermore, predictable robot behavior is essential to allow for cooperative behavior on the human side. Humanoid robots are advantageous for this purpose, as they look familiar to persons without robotic experience. Moreover, they are able to resemble human motions and behaviors, allowing intuitive human-robot-interaction. In this paper, we present our communication robot Robotinho. Robotinho is an anthropomorphic robot equipped with an expressive communication head. Its multimodal dialog system incorporates body language, gestures, facial expressions, and speech. We describe the behaviors used to interact with inexperienced users in a museum tour guide scenario. In contrast to previous work, our robot interacts with the visitors not only at the exhibits, but also while it is navigating to the next exhibit. We evaluated our system in a science museum and report quantitative and qualitative feedback from the users.

Demonstration of a spoken dialogue interface for planning activities of a semi-autonomous robot

Proceedings of the second international conference on Human Language Technology Research -, 2002

Planning and scheduling in the face of uncertainty and change pushes the capabilities of both planning and dialogue technologies by requiring complex negotiation to arrive at a workable plan. Planning for use of semi-autonomous robots involves negotiation among multiple participants with competing scientific and engineering goals to co-construct a complex plan. In NASA applications this plan construction is done under severe time pressure so having a dialogue interface to the plan construction tools can aid rapid completion of the process. But, this will put significant demands on spoken dialogue technology, particularly in the areas of dialogue management and generation. The dialogue interface will need to be able to handle the complex dialogue strategies that occur in negotiation dialogues, including hypotheticals and revisions, and the generation component will require an ability to summarize complex plans.