Challenges in designing digital interfaces for the study of multimodal phenomena (original) (raw)

Analysing multimodality in an interactive digital environment: software as a meta-semiotic tool

Social Semiotics, 2011

The present paper discusses issues arising from the use of an interactive digital software tool to analyze multimodal communication. The focus is on the ways in which such technical resources and associated techniques enable the analyst of social semiosis to apply different types of analysis, and provide the site for critical reflection upon the results of such analyses. The aim is to present an argument for the use of an interactive digital software application as a meta-semiotic tool. Three major challenges for scholars engaged in multimodal social semiotics are addressed in pursuing this aim, specifically with reference to the development of digital interactive analytical resources. Firstly, scholars are faced with the task of not only accounting for an increasing range of semiotic resources, but also for the way different semiotic phenomena interact to produce meaning. Secondly, the multimodal semiotician has a broad range of analytical approaches potentially relevant to any task to draw upon. Thirdly, the nature of contemporary media forms offer challenges themselves in terms of access, analysis and presentation of analysis.

Analyzing Multimodality in an Interactive Digital Environment: Software as Metasemiotic Tool

Using a Social Semiotic Approach to Multimodality

2012

The aim of this paper is to show how a substantive area of social research-learning-can be investigated using a multimodal social semiotic approach. We apply the approach to three different institutions-a school, a museum and a hospital, illustrating key concepts and addressing issues around pedagogy and technology in contemporary society. A multimodal social semiotic approach focuses on meaning-making, in all modes. It is a theoretical perspective that brings all socially organized resources that people use to make meaning into one descriptive and analytical domain. These resources include modes such as image, writing, gesture, gaze, speech, posture; and media such as screens, 3 D forms of various kinds, books, notes and notebooks. All of these modes and media are also used in environments designed for learning. That makes a multimodal social semiotic approach particularly apt for studying learning.

Locating the Semiotic Power of Multimodality

Written Communication, 2005

This article reports research that attempts to characterize what is powerful about digital multimodal texts. Building from recent theoretical work on understanding the workings and implications of multimodal communication, the authors call for a continuing empirical investigation into the roles that digital multimodal texts play in real-world contexts, and they offer one example of how such investigations might be approached. Drawing on data from the practice of multimedia digital storytelling, specifically a piece titled “Lyfe-N-Rhyme,” created by Oakland, California, artist Randy Young (accessible at https://www.youtube.com/watch?v=yfFg8zNkXZM), the authors detail the method and results of a fine-grained multimodal analysis, revealing semiotic relationships between and among different, copresent modes. It is in these relationships, the authors argue, that the expressive power of multimodality resides.

NAVIGATING THE MODERN WORLD WITH SOCIAL SEMIOTIC MULTIMODAL APPROACH

Proceeding of SENABASTRA XI "Trends and Issues in Linguistics, Literature and Cultural Studies, 2019

In an era where messages are imparted through various semiotic resources, it is necessary that we learn social semiotic multimodal approach. This approach is concerned with the making of meanings in many walks of life. In constructing meanings, several semiotic modes are deployed, which include verbal elements, visual, sound, gesture, gaze, clothing, and even ceremonies. In a classroom, the participants exchange messages by linguistic elements, notes, pictures, screens, and books. The paper argues that such approach has an important bearing on the way we learn, know, and construct information. It starts with the Systemic Functional Theory as the foundation of the social semiotic multimodal approach, then briefly sketches the application to analysis of advertisement. It maintains that the increasingly multimodal world has changed the way we read and understand information. Next, it also addresses the application of multimodality in teaching-learning domain, arguing that vast opportunities lie ahead for researchers to identify the extent to which multimodal approach can be potentially helpful for teachers and learners.

Multimodal Methods for Researching Digital Technologies

The SAGE Handbook of Digital Technology Research

Chapter 17 Multimodal methods for researching digital technologies Carey Jewitt This chapter provides an introduction to the field of multimodality and discusses its potential application for researching digital data and environments. It begins by outlining what multimodality is, its theoretical origins in social semiotics, and its underlying assumptions. A number of concepts central to multimodality are introduced: these include mode, semiotic resource, materiality, modal affordance, multimodal ensemble and meaning functions. The scope and potential of multimodality for researching digital technologies are then discussed. The chapter sets out an illustrative example of multimodal research. It concludes with a discussion of the limitations and challenges of a multimodal approach for researching digital technologies. What is multimodality? Multimodality is an inter-disciplinary approach drawn from social semiotics that understands communication and representation as more than language and attends systematically to the social interpretation of a range of forms of making meaning. It provide concepts, methods and a framework for the collection and analysis of visual, aural, embodied, and spatial aspects of interaction and environments (Jewitt, 2009; Kress, 2010). While other modes of communication, such as gesture, have been recognized and studied extensively (e.g. McNeill, 1992), multimodality investigates the interaction between communicational means and challenges the prior predominance of This is a pre-print version of a chapter to be published in the SAGE Handbook of Digital Technology research 2013 Sara Price, Carey Jewitt & Barry Brown (eds.) spoken and written language in research (Scollon and Scollon, 2009). Speech and writing continue to be understood as significant but are seen as parts of a multimodal ensemble. Multimodality emphasizes situated action, that is, the importance of the social context and the resources available for meaning making, with attention to people's situated choice of resources, rather than emphasizing the system of available resources. Thus it opens up possibilities for recognizing, analyzing and theorizing the different ways in which people make meaning, and how those meanings are interrelated. Multimodality provides resources to support a complex fine grained analysis of artefacts and interactions in which meaning is understood as being realized in the iterative connection between the meaning potential of a material semiotic artefact, the meaning potential of the social and cultural environment it is encountered in, and the resources, intentions, and knowledge that people bring to that encounter. That is, it strives to connect the material semiotic resources available to people with what they mean to signify in social contexts. Changes to these resources and how they are configured are therefore understood as significant for communication. Digital technologies are of particular interest to multimodality because they make a wide range of modes available, often in new inter-semiotic relationships with one another, and unsettle and re-make genres, in ways that reshape practices and interaction. Digital technologies are thus a key site for multimodal investigation. Underlying this approach is the idea that language, and other systems or modes of communication (e.g. gesture, gaze), is shaped through the things that it has been used to accomplish socially in everyday instantiations, not because of a fixed set of rules and This is a pre-print version of a chapter to be published in the SAGE Handbook of Digital Technology research 2013 Sara Price, Carey Jewitt & Barry Brown (eds.) structures. This view of language as a situated resource encompasses the principle that modes of communication offer historically specific and socially/culturally shared options (or 'semiotic resources') for communicating. With this emphasis, a key question for multimodality is how people make meaning in context to achieve specific aims. Three interconnected theoretical assumptions underpin multimodality. These are briefly introduced and discussed below. The first assumption underlying multimodality is that while language is widely taken to be the most significant mode of communication, speech or writing are a part of a multimodal ensemble. Multimodality 'steps away from the notion that language always plays the central role in interaction, without denying that it often does' (Norris, 2004:3) and proceeds on the assumption that all modes have the potential to contribute equally to meaning. From a multimodal perspective, language is therefore only ever one mode nestled among a multimodal ensemble of modes. While others have analyzed 'nonverbal' modes, multimodality differs in that language is not its' starting point or provide a prototypical model of all modes of communication. The starting point is that all modes that are a part of a multimodal ensemble-a representation and/or an interactionneed to be studied with a view to the underlying choices available to communicators, the meaning potentials of resources and the purposes for which they are chosen. The second assumption central to multimodal research is all modes have, like language, been shaped through their cultural, historical and social uses to realize social functions as required by different communities. Therefore each mode is understood as having This is a pre-print version of a chapter to be published in the SAGE Handbook of Digital Technology research 2013 Sara Price, Carey Jewitt & Barry Brown (eds.) different meaning potentials or semiotic resources and to realize different kinds of communicative work. Multimodality takes all communicational acts to be constituted of and through the social. This also draws attention to the ways in which communication is constrained and produced in relation to social context and points to how modes come into spaces in particular ways. This connects with the third assumption underpinning multimodality-that people orchestrate meaning through their selection and configuration of modes. Thus the interaction between modes is significant for meaning making. Multimodal communication is not in and of itself, however, new digital media have foregrounded the need to consider the particular characteristics of modes, multimodal configurations, and their semiotic function in contemporary discourse worlds (Ventola, Charles and Kaltenbacher, 2004). The meanings in any mode are always interwoven with the meanings made with those of other modes cooperating in the communicative ensemble. The interaction between modes is itself a part of the production of meaning. A brief background Multimodality was developed in the early 2000s (see Kress and van Leeuwen, 2001;

Semiotic Multimodality Communication in The Age of New Media

Studies in Media and Communication

The age of new media is changing the social order of communication. The availability of the widest access for internet users to communicate without time and region limitations is a feature of this digital media era. The second distinctive feature is the communication of digital media by using semiotic multimodality (a combination of verbal and nonverbal signs, for example, emojis, images, videos, sounds, and music). The purpose of this paper is to identify how cultures communicate with the New Media Age's semiotic multimodality. The approach used is semiotics, that is, an approach that views the media as a medium of exchange for signs of multimodality (complex verbal and nonverbal signs used in real-time). This paper concludes that the new media produce three models of communication between social media users, namely verbal and verbal language (sign) communication, verbal and nonverbal communication, and nonverbal-nonverbal communication. This study adds to the importance of int...

Multimodality

The chapter reviews the growing field of multimodality in relation to the study of language, text and society. It introduces the concept of multimodality as an increasingly visible phenomenon of communication and it traces the developments of multimodality as a field of research, along with the extant theoretical approaches to multimodal analysis. The chapter further discusses and exemplifies key notions of a social semiotic perspective to multimodal analysis and mentions potentials and limitations, pointing to future directions of research in the field. Rather than a comprehensive review of extant studies in multimodality, the chapter discusses selected key assumptions, topics and analytical developments in multimodal research that are relevant to its relation with language and society. Keywords: multimodality, social semiotics, sign-making, text, multimodal analysis

Introduction: studies in multimodality

2013

Kay o'halloran delivered a plenary talk entitled "Multimodality around the world: Past, present and future directions for research". Prof. o'halloran argued, back in 2008, that multimodality was not just a fad. two months later in Florianópolis, brazil, during the iv latin american Systemic-Functional Conference (alSFal), Prof. Theo van leeuwen and Prof. Carmen r. Caldas-Coulthard gave their plenary talks based on studies in multimodality, drawing attention to the ubiquitous presence and use of different semiotic resources, or modes, in contemporary society. it was then only seven years since the publication of Kress and van leeuwen's seminal book Multimodal discourse: the modes and media of contemporary communication (2001) which, coupled with their previous book Reading images: the grammar of visual design (1996/2006), propelled a whole new area that had been pointed to by Gunther Kress and robert hodge (Social Semiotics, 1988) and Michael o'toole (The Language of Displayed Art, 1994), for instance.

Challenges in designing digital interfaces for the study of multimodal phenomena (original) (raw)

Related papers