Towards a Multimodal Interaction Space: categorisation and applications (original) (raw)
Related papers
Design and Development of Multimodal Applications: A Vision on Key Issues and Methods
Lecture Notes in Computer Science, 2015
Multimodal user interfaces provide users with different ways of interacting with applications. This has advantages both in providing interaction solutions with additional robustness in environments where a single modality might result in ambiguous input or output (e.g., speech in noisy environments), and for users with some kind of limitation (e.g., hearing difficulties resulting from ageing) by yielding alternative and more natural ways of interacting. The design and development of applications supporting multimodal interaction involves numerous challenges, particularly if the goals include the development of multimodal applications for a wide variety of scenarios, designing complex interaction and, at the same time, proposing and evolving interaction modalities. These require the choice of an architecture, development and evaluation methodologies and the adoption of principles that foster constant improvements at the interaction modalities level without disrupting existing applications. Based on previous and ongoing work, by our team, we present our approach to the design, development and evaluation of multimodal applications covering several devices and application scenarios.
Multimodal Interaction within Ambient Environments: An Exploratory Study
Lecture Notes in Computer Science, 2009
Inputs and outputs are not two independent phenomena in multimodal systems. This paper examines the relationship that exists between them. We present the results of a Wizard of Oz experiment which shows that output modalities used by the system have an influence on the users' input modalities for a large category of users. The experiment took place in a smart room. This kind of environment does not require any particular knowledge about computers and their use and thus allowed us to study the behavior of ordinary people including subjects who are not familiar with computers. The experiment also shows that speech is a favorite modality within smart room environments for a large part of users. We think that the results presented in this paper will be useful for the design of intelligent multimodal systems.
2009
The grand challenge of multimodal interface creation is to build reliable processing systems able to analyze and understand multiple communication means in real-time. This opens a number of associated issues covered by this chapter, such as heterogeneous data types fusion, architectures for real-time processing, dialog management, machine learning for multimodal interaction, modeling languages, frameworks, etc. This chapter does not intend to cover exhaustively all the issues related to multimodal interfaces creation and some hot topics, such as error handling, have been left aside. The chapter starts with the features and advantages associated with multimodal interaction, with a focus on particular findings and guidelines, as well as cognitive foundations underlying multimodal interaction. The chapter then focuses on the driving theoretical principles, time-sensitive software architectures and multimodal fusion and fission issues. Modeling of multimodal interaction as well as tools allowing rapid creation of multimodal interfaces are then presented. The article concludes with an outline of the current state of multimodal interaction research in Switzerland, and also summarizes the major future challenges in the field.
Software Design and Development of Mutimodal Interaction
IFIP International Federation for Information Processing, 2004
The multimodal dimension of a user interface raises numerous problems that are not present in more traditional interfaces. In this paper, we briefly review the current approaches in software design and modality integration techniques for multimodal interaction. We then propose a simple framework for describing multimodal interaction designs and for combining sets of user inputs of different modalities. We show that the proposed framework can help designers in reasoning about synchronization patterns problems and testing interaction robustness.
The Rise of Interactional Multimodality in Human-Computer Interaction
2021
The field of human-computer interaction (HCI) investigates the intersection between the design of devices and users. From an early focus on interaction modeling based on psychological experiments, the field has since experienced a shift towards the study of how actual humans interact with autonomous devices. The field became conductive to ethnographic, observational and videographic studies of human-device interaction. Conversation-analytic HCI became possible. That said, this new wave of researchers was never truly able to dethrone the psychological common sense of the field. With recent developments in both the technical-sensorial capabilities and outward actuational range of embodied virtual agents, the field of HCI has once again returned to the question of the sequential unfolding of the interaction between users and intelligent agents, and the multimodal interactional repertoire that is deployed throughout. This review will highlight the situational orientation of high-impact ...
Introduction: multimodal interaction
Semiotica, 2005
That human social interaction involves the intertwined cooperation of different modalities is uncontroversial. Researchers in several allied fields have, however, only recently begun to document the precise ways in which talk, gesture, gaze, and aspects of the material surround are brought together to form coherent courses of action. The papers in this volume are attempts to develop this line of inquiry. Although the authors draw on a range of analytic, theoretical, and methodological traditions (conversation analysis, ethnography, ...
Multimodal Systems: Taxonomy, Methods, and Challenges
ArXiv, 2020
Naturally, humans use multiple modalities to convey information. The modalities are processed both sequentially and in parallel for communication in the human brain, this changes when humans interact with computers. Empowering computers with the capability to process input multimodally is a major domain of investigation in Human-Computer Interaction (HCI). The advancement in technology (powerful mobile devices, advanced sensors, new ways of output, etc.) has opened up new gateways for researchers to design systems that allow multimodal interaction. It is a matter of time when the multimodal inputs will overtake the traditional ways of interactions. The paper provides an introduction to the domain of multimodal systems, explains a brief history, describes advantages of multimodal systems over unimodal systems, and discusses various modalities. The input modeling, fusion, and data collection were discussed. Finally, the challenges in the multimodal systems research were listed. The an...
2007
This paper reports on the Nordic Workshop and PhD course on multimodal interaction and interfaces, organised by the MUMIN network, in Helsinki and Tampere in November 202. We give an overview of the objectives of the two events, and describe the presentations, lectures and demonstrations during the inspiring week.