Multimodal feedback signaling in finnish (original) (raw)
Related papers
Journal on Multimodal User Interfaces, 2012
This article deals with multimodal feedback in two Danish multimodal corpora, i.e., a collection of map-task dialogues and a corpus of free conversations in first encounters between pairs of subjects. Machine learning techniques are applied to both sets of data to investigate various relations between the non-verbal behaviour-more specifically head movements and facial expressions-and speech with regard to the expression of feedback. In the map-task data, we study the extent to which the dialogue act type of linguistic feedback expressions can be classified automatically based on the non-verbal features. In the conversational data, on the other hand, non-verbal and speech features are used together to distinguish feedback from other multimodal behaviours. The results of the two sets of experiments indicate in general that head movements, and to a lesser extent facial expressions, are important indicators of feedback, and that gestures and speech disambiguate each other in the machine learning process.
Feedback and gestural behaviour in a conversational corpus of Danish
2011
This paper deals with the way in which feedback is expressed through speech and gestures in the Danish NOMCO corpus of dyadic first encounters. The annotation includes the speech transcription as well as attributes concerning shape and conversational function of head movements and facial expressions. Our analysis of the data shows that all communication modalities, i.e. head, face and eyebrows, contribute to the expressions of feedback, with repeated nods and smiles as the most frequent feedback gesture types. In general, the use of nods as feedback gestures in our data is comparable to what earlier studies have found for other languages, but feedback is also often expressed by other head movements and smiles.
Types of nods. the polysemy of a social signal
Proc. Int'l Conf. Language Resources …, 2010
The work analyses the head nod, a down-up movement of the head, as a polysemic social signal, that is, a signal with a number of different meanings which all share some common semantic element. Based on the analysis of 100 nods drawn from the SSPNet corpus of TV political debates, a typology of nods is presented that distinguishes Speaker's, Interlocutor's and Third Listener's nods, with their subtypes (confirmation, agreement, approval, submission and permission, greeting and thanks, backchannel giving and backchannel request, emphasis, ironic agreement, literal and rhetoric question, and others). For each nod the analysis specifies: 1. characteristic features of how it is produced, among which main direction, amplitude, velocity and number of repetitions; 2. cues in other modalities, like direction and duration of gaze; 3. conversational context in which the nod typically occurs. For the Interlocutor's or Third Listener's nod, the preceding speech act is relevant: yes/no answer or information for a nod of confirmation, expression of opinion for one of agreement, prosocial action for greetings and thanks; for the Speaker's nods, instead, their meanings are mainly distinguished by accompanying signals.
2015
This paper reports a study of the forms and functions of head movements produced in the dimension of depth in Finnish Sign Language (FinSL). Specifically, the paper describes and analyzes the phonetic forms and prosodic, grammatical, communicative, and textual functions of nods, head thrusts, nodding, and head pulls occurring in FinSL data consisting of a continuous dialogue recorded with motion capture technology. The analysis yields a novel classification of the kinematic characteristics and functional properties of the four types of head movement. However, it also reveals that there is no perfect correspondence between form and function in the head movements investigated. Keywords: non-manuals, head movements, prosody, Finnish Sign Language, Motion Capture
Analysis and measurement of head movements signalling feedback in face-to-face human dialogues
The aim of our study is twofold: explore how people use specific head movements to signal feedback and show evidence that it is possible to measure and quantify the extent of these movements with the Qualisys MacReflex motion tracking system. Naturally elicited dialogues between humans have been recorded and analysed with focus on those gestures that occur contemporarily with the production of short verbal feedback expressions. The results show that head nods and jerks are typical movements involved during the production and elicitation of feedback. These movements produce specific patterns and can be easily measured and quantified and eventually implemented in animated talking heads.
Sign Language & Linguistics, 2015
This paper reports a study of the forms and functions of head movements produced in the dimension of depth in Finnish Sign Language (FinSL). Specifically, the paper describes and analyzes the phonetic forms and prosodic, grammatical, communicative, and textual functions of nods, head thrusts, nodding, and head pulls occurring in FinSL data consisting of a continuous dialogue recorded with motion capture technology. The analysis yields a novel classification of the kinematic characteristics and functional properties of the four types of head movement. However, it also reveals that there is no perfect correspondence between form and function in the head movements investigated.
The emotional and communicative significance of head nods
2010
Head nods and shakes have been extracted from the SAL audiovisual database of spontaneous emotionally coloured dialogue. The dataset contains 154 nods and 104 shakes. Two trained observers rated them on multiple dimensions derived from linguistics on one hand, and the psychology of emotion on the other. One used audiovisual presentation, the other visual only. There was agreement on affective, but not linguistic significance – suggesting that the latter depends on speech context rather than the manner of movement per se. A few seem to form discrete types, but for the most part classical dimensional models of emotion captured the affective variation well.
Repeated head movements, their function and relation to speech
Proceedings of the Workshop on Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality. LREC, 2010
This paper presents a study of multimodal communication in spontaneous “getting to know each other conversations”. The study focuses on repeated head movements (head-nods and head-shakes) and the speech co-occurring with them. The main function of such repeated head movements is found to be communicative feedback. This is also the most frequent function of the speech co-occurring with the head movements. However, there is mostly no 1-1 relation between repetition in head movement and vocal words. Repeated ...
Investigating communicative feedback phenomena across languages and modalities
2007
This thesis deals with human communicative behaviour related to feedback, analysed across languages (Italian and Swedish), modalities (auditory versus visual) and different communicative situations (human-human versus human-machine dialogues). The aim of this study is to give more insight into how humans use communicative behaviour related to feedback and at the same time to suggest a method to collect valuable data that can be useful to control facial and head movements related to visual feedback in synthetic conversational agents. The study of human communicative behaviour necessitates the good quality of the materials under analysis, the support of reliable software packages for the audio-visual analysis and a specific coding scheme for the annotation of the phenomena under observation. The materials used for the investigations presented in this thesis span from spontaneous conversations video recorded in real communicative situations, and semi-spontaneous dialogues obtained with different eliciting techniques, such as map-task and information-seeking scenarios, to a specific corpus of controlled interactive speech collected by means of a motion capture system. When motion caption is used it is possible to register facial and head movements with a high degree of precision, so as to obtain valuable data useful for the implementation of facial displays in talking heads. A specific coding scheme has been developed, tested and used to annotate feedback. The annotation has been carried out with the support of different available software packages for audio-visual analysis. The procedure followed in this thesis involves initial analyses of communicative phenomena in spontaneous human-human dialogues and human-machine interaction, in order to learn about regularities in human communicative behaviour that could be transferred to talking heads, then, for the sake of reproduction in talking heads, the investigation includes more detailed analyses of data collected in a lab environment with a novel acquisition set-up that allows capturing the dynamics of facial and head movements. Finally the possibilities of transferring human communicative behaviour to a talking face are discussed and some evaluation paradigms are illustrated. The idea of reproducing human behaviour in talking heads is based on the assumption that the reproduction of facial displays related to communicative phenomena such as turn management, feedback production and expression of emotions in embodied conversational agents, might result in the design of advanced systems capable of effective multi-modal interactions with humans.
Multimodal Behaviour and Feedback in Different Types of Interaction
In this article, we compare feedback-related multimodal behaviours in two different types of interactions: first encounters between two participants who do not know each in advance, and naturally-occurring conversations between two and three participants recorded at their homes. All participants are Danish native speakers. The interactions are transcribed using the same methodology, and the multimodal behaviours are annotated according to the same annotation scheme. In the study we focus on the most frequently occurring feedback expressions in the interactions and on feedback-related head movements and facial expressions. The analysis of the corpora, while confirming general facts about feedback-related head movements and facial expressions previously reported in the literature, also shows that the physical setting, the number of participants, the topics discussed, and the degree of familiarity influence the use of gesture types and the frequency of feedback related expressions and ...