CONTACT: A Multimodal Corpus for Studying Expressive Styles and Informing the Design of Individualized Virtual Narrators (original) (raw)
Related papers
A Verbal and Gestural Corpus of Story Retellings to an Expressive Embodied Virtual Character
We present a corpus of 44 human-agent verbal and gestural story retellings designed to explore whether humans would gesturally entrain to an embodied intelligent virtual agent. We used a novel data collection method where an agent presented story components in installments, which the human would then retell to the agent. At the end of the installments, the human would then retell the embodied animated agent the story as a whole. This method was designed to allow us to observe whether changes in the agent's gestural behavior would result in human gestural changes. The agent modified its gestures over the course of the story, by starting out the first installment with gestural behaviors designed to manifest extraversion, and slowly modifying gestures to express introversion over time, or the reverse. The corpus contains the verbal and gestural transcripts of the human story retellings. The gestures were coded for type, handedness, temporal structure, spatial extent, and the degree to which the participants' gestures match those produced by the agent. The corpus illustrates the variation in expressive behaviors produced by users interacting with embodied virtual characters, and the degree to which their gestures were influenced by the agent's dynamic changes in personality-based expressive style.
Animated interactive fiction: Storytelling by a conversational virtual actor
Proceedings. International Conference on Virtual Systems and MultiMedia VSMM '97 (Cat. No.97TB100182)
This paper presents an architecture for interactive storytelling using state-of-the-art-technology in natural language processing, speech synthesis and 3Dcharacter animation. A conversational 3D-character is used to tell nonlinear stories interactively. Our research focuses on multimodal conversation that combines verbal-vocal with nonverbal-nonvocal natural language. We present a framework for synchronizing speech with facial movements, gesture and body posture by combining ndings from linguistics and psychology. The interaction process incorporates the parsing of a textual natural language utterance, the emotion-based triggering of a textual response and its vocal and visual performance-speech synthesis linked with lipsynchronous generation of facial expressions and gestures.
Virtual StoryTelling: A Methodology for Developing Believable Communication Skills in Virtual Actors
2003
Until recently virtual applications rarely incorporated any deep social, cultural or emotional capabilities exhibited in normal human interaction. In this paper, we present a methodology to develop virtual actors with believable communications skills who may provide a sufficient feeling of immersion to reproduce an efficient simulation of human activities. We focus on the relationship between gestures and verbal activities during communicative acts. We will show that an approach based on video analysis is valuable for the design of a system allowing realistic interactions between the avatars and/or the avatars and the environment. Although the analysis is still in progress, we present some preliminary results from our experiments.
Multimodal complex emotions: Gesture expressivity and blended facial expressions
2006
One of the challenges of designing virtual humans is the definition of appropriate models of the relation between realistic emotions and the coordination of behaviors in several modalities. In this paper, we present the annotation, representation and modeling of multimodal visual behaviors occurring during complex emotions. We illustrate our work using a corpus of TV interviews. This corpus has been annotated at several levels of information: communicative acts, emotion labels, and multimodal signs.
Introducing multimodal sequential emotional expressions for virtual characters
2010
ABSTRACT In this paper we present a system which allows embodied conversational agent to display multimodal sequential expressions. Recent studies show that several emotions are expressed by a set of different nonverbal behaviors which include different modalities: facial expressions, head and gaze movements, gestures, torso movements and posture.
Annotating Multimodal Behaviors Occurring During Non Basic Emotions
Lecture Notes in Computer Science, 2005
The design of affective interfaces such as credible expressive characters in story-telling applications requires the understanding and the modeling of relations between realistic emotions and behaviors in different modalities such as facial expressions, speech, hand gestures and body movements. Yet, research on emotional multimodal behaviors has focused on individual modalities during acted basic emotions. In this paper we describe the coding scheme that we have designed for annotating multimodal behaviors observed during mixed and non acted emotions. We explain how we used it for the annotation of videos from a corpus of emotionally rich TV interviews. We illustrate how the annotations can be used to compute expressive profiles of videos and relations between non basic emotions and multimodal behaviors.
Expressive Gestures Displayed by a Humanoid Robot during a Storytelling Application
New Frontiers in …, 2010
Our purpose is to have the humanoid robot, NAO, to read a story aloud through expressive verbal and nonverbal behaviors. These behaviors are linked to the story being told and the emotions to be conveyed. To this aim, we are gathering a repertoire of gestures and body-postures for the robot using a video corpus of story-tellers. The robot's behavior is controlled by the Greta platform that drives agents to communicate through verbal and nonverbal means. In this paper we are presenting our research methodology and overall framework.
Multimodal expressive embodied conversational agents
2005
Abstract In this paper we present our work toward the creation of a multimodal expressive Embodied Conversational Agent (ECA). Our agent, called Greta, exhibits nonverbal behaviors synchronized with speech. We are using the taxonomy of communicative functions developed by Isabella Poggi [22] to specify the behavior of the agent. Based on this taxonomy a representation language, Affective Presentation Markup Language, APML has been defined to drive the animation of the agent [4].
Based on an analysis accounting for the whole body as a possible articulator in the depiction of actions, this chapter argues for an expansion of the notion of ‘character viewpoint gestures’ to a notion of ‘multimodal action depiction from a character viewpoint’. Our study shows that speakers may deploy only single articulators, providing a semantically reduced depiction of the action, or they may deploy more bodily articulators and give a semantically rich picture of the event narrated. Our findings suggest a continuum of semiotic complexity, capturing the range of bodily involvement from less pantomimic (single articulators involved) to pantomimic depictions (more articulators involved) of actions. The paper closes by discussing our observations with respect to the notion of ‘constructed action’ and ‘role shift’ in sign languages and by giving some general remarks on the multimodal analysis of narrations.
This paper describes the implementation of an automatically generated virtual storyteller from fairy tale texts which were previously annotated for emotion. In order to gain insight into the effectiveness of our virtual story-teller we recorded face, body and voice of an amateur actor and created an actor animation video of one of the fairy tales. We also got the actor's annotation of the fairy tale text and used this to create a virtual storyteller video. With these two videos, the virtual storyteller and the actor animation, we conducted a user study to deter-mine the effectiveness of our virtual storyteller at conveying the intended emotions of the actor. Encouragingly, participants performed best (when compared to the intended emotions of the actor) when they marked the emo-tions of the virtual storyteller. Interestingly, the actor himself was not able to annotate the animated actor video with high accuracy as compared to his annotated text. This argues that for future work ...