(2020) Multimodality (original) (raw)
2020, in Mona Baker and Gabriela Saldanha (eds) The Routledge Encyclopedia of Translation Studies, 3rd edition, London and New York: Routledge, 346-351.
Multimodality, a term first used in the late 1990s (Jewitt et al. 2016:2), is the study of how we make meaning by combining multiple signifying means or modes-for example, image with writing, music and body movement, speech with gesture-into an integrated whole. While more established disciplines such as linguistics, semiotics and musicology engage primarily with those semiotic resources that fall within their respective conceptual remits, multimodality investigates the synergies between co-occurring semiotic resources. From a multimodal standpoint, each mode "is understood as realizing different communicative work" (Jewitt 2014/2017:16) and is moulded by its own semiotic limitations, potentialities and affordances. Multimodal ensembles-whether they adopt the form of a graphic novel, a film, an illustrated textbook, a museum exhibit or an everyday conversational encounterresult from the interplay between the relevant cooperating modes. Multimodality specialists have "not as yet focused on questions of translation" (Taylor 2016:222), while translation scholars have been slow to engage with multimodal concepts and methods. Theoretical and methodological developments in translation studies until the 1990s were driven by the conceptualization of translation as a process of written language transfer where the printed word is the only signifying means at play. The study of spoken texts as loci of interpreting activity has similarly tended to revolve around their verbal fabric, often glossing over the semiotic contribution of the orality and corporeality of interpreter-mediated speech. This disciplinary emphasis on language-centred meaning-making processes largely derives from the influence of linguistics during the formative period of translation (and interpreting) studies (Baker 2005). It is also consistent with the entrenched prevalence of monomodality, understood as the dominance of one signifying constituent, such as written language, over other types of meaning-making resources. Approaches to multimodality Translation studies has not yet managed to articulate clearly how semiotics and multimodality relate to one another. Stecconi (2009:261) argues that, when used in a lax sense, semiotics encompasses "research that goes beyond verbal language", including "multimedia/multimodal material", a term he associates with Gottlieb's (2005) work. But Gottlieb's (2017:46) own contribution to this debate identifies multimodality, described as an addition to the more established body of scholarship on multi-channel texts and paraverbal translation, exclusively with the work of multimodality specialists (Baldry and Thibault 2006; Kress 2010). Leaving aside the profusion of terms that translation scholars use to designate their theoretical explorations of meaning-making resources and practices, they would appear to agree that multimodality falls within the wider remit of semiotics.