Anaphora and Gestures in Multimodal Communication (original) (raw)
Related papers
ACM Transactions on Computer-Human Interaction, 2002
Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-acc...
Analysing the Modifying Functions of Gesture in Multimodal Utterances
2015
Gestures may contribute to the meaning of an utterance not only by adding information but also by modifying the gestural or verbal content uttered in parallel. The phenomenon of modification is more common in natural interaction than it has been given attention. We created a corpus of natural communicative gestures and body movements and conducted a study to examine their modifying functions. Results show that index-finger-pointings are most prominent, which emphasise and affirm an uttered content and, thus, are not only used for referencing but also for modifying. Holds emphasise and colour the utterance by showing a stance towards something. Brushing gestures change the utterance in a discounting or downtoning way. A cluster analysis suggests four distinct categories: a focusing category for emphasising an aspect, an epistemic-attitudinal category to convey one’s own stance, an epistemic category for uncertainty, and a category where multiple viewpoints are discussed.
2017
Multimodality and multimodal communication is a rapidly evolving research field addressed by scientists working in various perspectives, from psycho-sociological fields, anthropology and linguistics, to communication and multimodal interfaces, companions, smart homes and ambient assisted living etc. Multimodality in human-machine interaction is not just an add-on or a style of information representation. It goes well beyond semantics and semiotic artefacts. It can significantly contribute to representation of the information as well as in interpersonal and textual function of communication. The study in this paper is a part of an ongoing effort in order to empirically investigate in detail relations between verbal and co-verbal behavior expressed during multi-speaker highly spontaneous and live conversations. It utilizes a highly multimodal approach for investigating into relations between the traditional linguistic (such as: paragraphs, sentences, sentence types, words, POS tags et...
Top-down Bottom-up Experiments on Detecting Co-speech Gesturing in Conversation
2016
Automatic analysis of conversational videos and detection of gesturing and body movement of the partners is one of the areas where technology development has been rapid. This paper deals with the application of video techniques to human communication studies, and focuses on detecting communicative gesturing in conversational videos. The paper sets to investigate the top-down-bottom-up methodology, which aims to combine the two approaches used in interaction studies: the human annotation of the data and the automatic analysis of the data.
Visnyk of Lviv University. Philology Series. Issue 52, 88-103. L'viv, 2011. (Вісник Львівського університету. Серія філологічна.Випуск 52. Львів, 2011). (ISSN 2078-5534)
The spontaneous gesticulation that accompanies speech is an integral part of the linguistic system. Movements of the body are made in conjunction with speech to produce meanings and perform a num- ber of essential discourse functions. Gesture is ‘gestural action’ and gesture symbolism is dynamic, schematic and imagistic. Gestures don’t just ‘depict’ but actually ‘do things’: they shape ideas and fuel thought; they describe or report scenes; they give directions; they expose, report, and sum up argu- ments; they achieve textual cohesion and regulate communicative interaction. Gestures are a window into the mind. As gestures are made, visible kinetic form is given to invisible mental representations and hidden cognitive mechanisms. As hands move within the gesture space, objets of conception are created, and cognitive processing is ‘acted out,’ using symbolic acts of pointing and manipulation. Key cognitive abilities are revealed in the process: the ability to construe ideas and events as objects and substances (conceptual reification); the ability to form image-schematic representations of ‘things’ and movements, and to use these iconically or metaphoricially; the ability to make symbolic uses of space. Gestures are also found to play a central role in the expression of grammatical meanings and mecha- nisms. Thus grammar and gesture are clearly integrated in the expression of temporal dimensions, aspectual notions and modal stances. Gesture activity is also shown to be involved in the expression of concession and comparison. Finally, technical and methodological dimensions of gesture-analysis are discussed. The case is made for a new, creative approach to gesture watching – the ‘language and gesture workshops’ – where students may observe and physically explore co-speech gestures, develop their own choreographic variations, and work on sound, gesture and meaning correspondence.