Review: Svenja Adolphs and Ronald Carter. Spoken corpus linguistics. From monomodal to multimodal (original) (raw)

Multimodality I: Speech prosody and gesture

The Routledge Handbook of English Language and Digital Humanities, 2020

In the age of the Internet, trillions of bytes of media data are generated every day through telecommunications and social media. This surge of born-digital media data, for example, instant voice/video messages, conference calls, podcasts, video blogs and so on, offers researchers unprecedented opportunities to deepen their understanding of how human beings communicate and go about their social activities. However, such a large amount of data also brings a new problem: how may we plough through so much media data and extract meaningful information efficiently? This chapter explores opportunities and challenges at the interface between digital humanities and multimodality research which focuses on the use of prosody and gesture in spoken communication. Following an overview of key methods and frameworks in prosody and gestures research, it highlights selected projects which have showcased the ways in which today’s computer technology has revolutionised multimodality as an area of research. In recent years, many new computer tools have become available to aid media data acquisition, processing and analysis. These tools have (semi-)automatised many processes which were labour-intensive, expensive and tedious. Therefore, researchers can now afford to compile and process substantially larger multimodal datasets much faster and at a much lower cost. The chapter also introduces tools which open up new avenues for researchers to acquire new types of multimodal data (e.g. YouTube videos) and data streams (e.g. GPS, heartbeats). In the sample analysis, we demonstrate the typical workflow for using a range of these latest computer tools to generate a corpus of YouTube videos, automatically annotate prosodic patterns, align multiple data streams and perform a multimodal analysis on the use of the epistemic stance marker ‘I think’ in video blogs.

Multimodality in discussion sessions: corpus compilation and pedagogical use

Discussion sessions of conference paper presentations are spontaneous and unpredictable, in contrast to the prepared lecture that precedes them. These can be challenging, especially for novice presenters whose worst fear is to fail to understand the second meaning of a question or comment, and who know it is not only the quality of the research that is judged but also their prestige and worth. Additionally, spoken academic genres have traditionally been explored by focusing on the transcription of speech and disregarding the multimodal nature of spoken discourse. This study offers a comprehensive account of the design of a multimodal corpus of discussion sessions, where audio, video, transcriptions and annotations are time-synchronised. This multilayer analysis provides examples (not only of linguistic utterances of rhetorical moves and multimodal evaluation, but also of how they are actually expressed paralinguistically and kinetically), which can be used in the classroom and to design learning-teaching materials.

The FASiL Speech and Multimodal Corpora

… Conference on Speech …, 2005

In the context of the FASiL project, we have studied natural lan-guage interactions in a unimodal (speech only) and multimodal (speech and graphics) interface to a personal information man-agement database. We collected multilingual corpora to inves-tigate these ...

Multimodal Discourse: In Search of Units

2015

Human communication is inherently multimodal. In this study we focus on three channels of spoken discourse: the verbal component, prosody, and gesticulation. We address the question of units that can be identified within these components and in spoken multimodal discourse as a whole. The basic unit of the verbal channel is the clause, reporting an event or a state. A set of prosodic criteria help to define elementary discourse units, that is prosodic units serving as quanta of discourse production. The gestural channel consists of individual gestures, each defined by a set of features. Elementary discourse units are strongly coordinated with both clauses and gestures and can thus be considered basic units of multimodal discourse. Larger units can also be identified, such as prosodic sentences and series of gestures that again demonstrate coordination. By identifying units of natural discourse, coordinated across various channels, we make a step towards multimodal linguistics.

Spoken and Written Discourse in Online Interactions. A Multimodal Approach.

2013

Common patterns of interactions are altered in the digital world and new patterns of communication have emerged, challenging previous notions of what communication actually is in the contemporary age. Online configurations of interaction, such as video chats, blogging, and social networking practices demand profound rethinking of the categories of linguistic analysis, given the blurring of traditional distinctions between oral and written discourse in digital texts. This volume reconsiders underlying linguistic and semiotic frameworks of analysis of spoken and written discourse in the light of the new paradigms of online communication, in keeping with a multimodal corpus linguistics theoretical framework. Typical modes of online interaction encompass speech, writing, gesture, movement, gaze, and social distance. This is nothing new, but here Sindoni asserts that all these modes are integrated in unprecedented ways, enacting new interactional patterns and new systems of interpretation among web users. These "non verbal" modes have been sidelined by mainstream linguistics, whereas accounting for the complexity of new genres and making sense of their educational impact is high on this volume’ s agenda. Sindoni analyzes other new phenomena, ranging from the intimate sphere (i.e. video chats, personal blogs or journals on social networking websites) to the public arena (i.e. global-scale transmission of information and knowledge in public blogs or media-sharing communities), shedding light on the rapidly changing global web scenario.

Designing a Multimodal Spoken Component of the Australian National Corpus

2000

1 Cermák (2009) does not specify how he reached this estimation further, although it presumably applies to the averaged, typical adult's use of language over the course of a day. 2 Although see Rühlemann for an excellent study of interactional and pragmatic phenomena that draws from the British National Corpus, and Adophs (2008) for a similarly themed study drawing from the CANCODE corpus.

Editorial: multimodality, segmentation and prominence in speech

2020

Widening the lens: joining multimodality, segmentation and prominence in speech Speech and gestures meet at their departure point which is actionality. The same departing point keeps the two channels connected through their execution in the creation of meaning and interactivity. Both speech and gestures require segmentation in order to be studied and understood scientifically, as knowing what the units of analysis are is crucial to the scientific endeavor. Prominence is both a characteristic carried by prosody (be it defined functionally, physically or cognitively), as well as by several gestural acts, such as widening of the eyes, increased speed in hand motion, head tilting, among others. This link permits our joining multimodality, segmentation and prominence in speech as a topic for a scientific journal. As our knowledge about spoken language grows, thanks to empirically and experimentally based studies, the necessity for the never ending refining of methodologies is called into action, as well as the broadening of their boundaries. The understanding that gestuality actively interacts and partakes in communication is not a novel perception, as gesture forms a single system with speech and is an integral part of the communicative act (Kendon 1980; McNeil, 1992). However, the accurate pairing of how this interaction occurs is still not fully understood. Are gestures and speech additive, parallel, complementary? How are they linked in terms of the cognitive-neurological and motor routines involved? This issue of JoSS emerged from the topics proposed at the X LABLITA and XI LEEL International Workshop: Prosody and Gesture, an initiative of the Empirical and Experimental Language Research Lab (LEEL) that took place in 2019, at the Federal University of Minas Gerais (UFMG). The LEEL lab focuses a large part of its team efforts into the compilation and study of spoken corpora. By organizing this event, the lab team aimed at bringing the correlations between gesture and prosody to the front, as well as supporting discussions about multimodal corpora compilation.

Multimodality: A guide for linguists

This chapter provides a rationale for a multimodal perspective on meaning, communication and discourse. It draws attention to the range of different modes that people use beyond speech and writing and explores the theoretical and methodological implications of multimodality. The chapter addresses two key questions. First, what is multimodality? Why and how is it relevant to linguistics? Second, how has multimodality been taken up? What questions are addressed, what materials are collected, and what methods are used to analyze these materials? Taking one approach – Social Semiotics – and one area of research – online text making – as an example, the chapter defines and illustrates key concepts and steps in multimodal inquiry. It concludes with consideration of future directions.