To ask or to sense? Planning to integrate speech and sensorimotor acts (original) (raw)
Related papers
Combining Goal Inference and Natural-Language Dialogue for Human-Robot Joint Action
We demonstrate how combining the reasoning components from two existing systems designed for human-robot joint action produces an integrated system with greater capabilities than either of the individual systems. One of the systems supports primarily non-verbal interaction and uses dynamic neural fields to infer the user's goals and to suggest appropriate system responses; the other emphasises natural-language interaction and uses a dialogue manager to process user input and select appropriate system responses. Combining these two methods of reasoning results in a robot that is able to coordinate its actions with those of the user while employing a wide range of verbal and non-verbal communicative actions.
Planning Information Processing and Sensing Actions
2007
The goal of the CoSy project is to create cognitive robots to serve as a testbed of theories on how humans work [13], and to identify problems and techniques relevant to producing general-purpose humanlike domestic robots. Given the constraints on the resources available at the robot's disposal and the complexity of the tasks that the robot has to execute during cognitive interactions with other agents or humans, it is essential that the robot perform just those tasks that are necessary for it to achieve its goal. In this paper we ...
A communicative robot to learn about us and the world
2019
We describe a model for a robot that learns about the world and her com-panions through natural language communication. The model supports open-domain learning, where the robot has a drive to learn about new con-cepts, new friends, and new properties of friends and concept instances. The robot tries to fill gaps, resolve uncertainties and resolve conflicts. The absorbed knowledge consists of everything people tell her, the situations and objects she perceives and whatever she finds on the web. The results of her interactions and perceptions are kept in an RDF triple store to enable reasoning over her knowledge and experiences. The robot uses a theory of mind to keep track of who said what, when and where. Accumulating knowledge results in complex states to which the robot needs to respond. In this paper, we look into two specific aspects of such complex knowl-edge states: 1) reflecting on the status of the knowledge acquired through a new notion of thoughts and 2) defining the conte...
Towards Learning Human-Robot Dialogue Policies Combining Speech and Visual Beliefs
We describe an approach for multi-modal dialogue strategy learning combining two sources of uncertainty: speech and gestures. Our approach represents the state-action space of a reinforcement learning dialogue agent with relational representations for fast learning, and extends it with belief state variables for dialogue control under uncertainty. Our approach is evaluated, using simulation, on a robotic spoken dialogue system for an imitation game of arm movements. Preliminary experimental results show that the joint optimization of speech and visual beliefs results in better overall system performance than treating them in isolation.
Stepwise Acquisition of Dialogue Act Through Human-Robot Interaction
2019 International Joint Conference on Neural Networks (IJCNN), 2019
A dialogue act (DA) represents the meaning of an utterance at the illocutionary force level (Austin 1962) such as a question, a request, and a greeting. Since DAs take charge of the most fundamental part of communication, we believe that the elucidation of DA learning mechanism is important for cognitive science and artificial intelligence, especially for cognitive robotics. The purpose of this study is to verify that scaffolding takes place when a human teaches a robot, and to let a robot learn to estimate DAs and to make a response based on them step by step utilizing scaffolding provided by a human. To realize that, it is necessary for the robot to detect changes in utterance and rewards given by the partner and continue learning accordingly. Experimental results demonstrated that participants who continued interaction for a sufficiently long time often gave scaffolding for the robot. Although the number of experiments is still insufficient to obtain a definite conclusion, we observed that 1) the robot quickly learned to respond to DAs in most cases if the participants only spoke utterances that match the situation, 2) in the case of participants who built scaffolding differently from what we assumed, learning did not proceed quickly, and 3) the robot could learn to estimate DAs almost exactly if the participants kept interaction for a sufficiently long time even if the scaffolding was unexpected.
Speech and action: integration of action and language for mobile robots
2007
We describe the tight integration of incremental natural language understanding, goal management, and action processing in a complex robotic architecture, which is required for natural interactions between robots and humans. Specifically, the natural language components need to process utterances while they are still spoken to be able to initiate feedback actions in a timely fashion, while the action manager might need information at various points during action execution that must be obtained from humans. We argue that a finergrained integration provides much more natural human-robot interactions and much more reasonable multitasking.
arXiv Computer Science, 2019
Though great progress has been made for human-machine conversation, current dialogue system is still in its infancy: it usually converses passively and utters words more as a matter of response, rather than on its own initiatives. In this paper, we take a radical step towards building a human-like conversational agent: endowing it with the ability of proactively leading the conversation (introducing a new topic or maintaining the current topic). To facilitate the development of such conversation systems, we create a new dataset named DuConv where one acts as a conversation leader and the other acts as the follower. The leader is provided with a knowledge graph and asked to sequentially change the discussion topics, following the given conversation goal, and meanwhile keep the dialogue as natural and engaging as possible. DuConv enables a very challenging task as the model needs to both understand dialogue and plan over the given knowledge graph. We establish baseline results on this dataset (about 270K utterances and 30k dialogues) using several state-of-the-art models. Experimental results show that dialogue models that plan over the knowledge graph can make full use of related knowledge to generate more diverse multi-turn conversations. The baseline systems along with the dataset are publicly available 1 .
Emergent verbal behaviour in human-robot interaction
… (CogInfoCom), 2011 2nd International …, 2011
The paper describes emergent verbal behaviour that arises when speech components are added to a robotics simulator. In the existing simulator the robot performs its activities silently. When speech synthesis is added, the first level of emergent verbal behaviour is that the robot produces spoken monologues giving a stream of simple explanations of its movements. When speech recognition is added, human-robot interaction can be initiated by the human, using voice commands to direct the robot's movements. In addition, cooperative verbal behaviour emerges when the robot modifies its own verbal behaviour in response to being asked by the human to talk less or more. The robotics framework supports different behavioural paradigms, including finite state machines, reinforcement learning and fuzzy decisions. By combining finite state machines with the speech interface, spoken dialogue systems based on state transitions can be implemented. These dialogue systems exemplify emergent verbal behaviour that is robot-initiated: the robot asks appropriate questions in order to achieve the dialogue goal. The paper mentions current work on using Wikipedia as a knowledge base for open-domain dialogues, and suggests promising ideas for topic-tracking and robot-initiated conversational topics.
Enabling Robots to Understand Indirect Speech Acts in Task-Based Interactions
Journal of Human-Robot Interaction, 2017
An important open problem for enabling truly taskable robots is the lack of task-general natural language mechanisms within cognitive robot architectures that enable robots to understand typical forms of human directives and generate appropriate responses. In this paper, we first provide experimental evidence that humans tend to phrase their directives to robots indirectly, especially in socially conventionalized contexts. We then introduce pragmatic and dialogue-based mechanisms to infer intended meanings from such indirect speech acts and demonstrate that these mechanisms can handle all indirect speech acts found in our experiment as well as other common forms of requests.