Migrating Modes: Multimodality in Digital Poetics as Another Kind of Language. Mosaic: A Journal for the Interdisciplinary Study of Literature 47.4 (December 2014). (original) (raw)
Related papers
Locating the Semiotic Power of Multimodality
Written Communication, 2005
This article reports research that attempts to characterize what is powerful about digital multimodal texts. Building from recent theoretical work on understanding the workings and implications of multimodal communication, the authors call for a continuing empirical investigation into the roles that digital multimodal texts play in real-world contexts, and they offer one example of how such investigations might be approached. Drawing on data from the practice of multimedia digital storytelling, specifically a piece titled “Lyfe-N-Rhyme,” created by Oakland, California, artist Randy Young (accessible at https://www.youtube.com/watch?v=yfFg8zNkXZM), the authors detail the method and results of a fine-grained multimodal analysis, revealing semiotic relationships between and among different, copresent modes. It is in these relationships, the authors argue, that the expressive power of multimodality resides.
Text & Talk
This paper is a critical appreciation of some of Gunther Kress’s central (social) semiotic notions: i.e., motivation, materiality, rhetorical aptness and semiotic mode versus medium. These will be discussed in relation to four landmark models of sign-making and semiosis by Saussure, Peirce, Bühler and Jakobson. Based on these comments, the paper identifies the persistent difficulties current multimodality research faces in defining mode and in devising linguistically unbiased grammars of non-verbal modes. Finally, the argument is advanced that multimodal genre and discourse interpretation in particular deserve to be re-developed. The paper critiques Kress’s insistence on motivation as a universal principle of sign use and his overemphasis on materiality to the detriment of grammar, while praising his overall (social) semiotic legacy for multimodality research as far-sighted and lastingly influential.