Home conducting: Control the overall musical expression with gestures (original) (raw)

An Experimental Set of Hand Gestures for Expressive Control of Musical Parameters in Realtime

This paper describes the implementation of Time Delay Neural Networks (TDNN) to recognize gestures from video images. Video sources are used because they are non-invasive and do not inhibit performer's physical movement or require specialist devices to be attached to the performer which experience has shown to be a significant problem that impacts musicians performance and can focus musical rehearsals and performances upon technical rather than musical concerns (Myatt 2003). We describe a set of hand gestures learned by an artificial neural network to control musical parameters expressively in real time. The set is made up of different types of gestures in order to investigate: • aspects of the recognition process • expressive musical control • schemes of parameter mapping • generalization issues for an extended set for musical control The learning procedure of the Neural Network is described which is based on variations by affine transformations of image sequences of the hand gestures. The whole application including the gesture capturing is implemented in jMax to achieve real time conditions and easy integration into a musical environment to realize different mappings and routings of the control stream. The system represents a practice-based research using actual music models like compositions and processes of composition which will follow the work described in the paper.

Towards An Affective Gesture Interface For Expressive Music Performance

2008

This paper discusses the use of 'Pogany', an affective anthropomorphic interface, for expressive music performance. For this purpose the interface is equipped with a module for gesture analysis: a) in a direct level, in order to conceptualize measures capable of driving continuous musical parameters, b) in an indirect level, in order to capture high-level information arising from 'meaningful' gestures. The real-time recognition module for hand gestures and postures is based on Hidden Markov Models (HMMs). After an overview of the interface, we analyze the techniques used for gesture recognition and the decisions taken for mapping gestures with sound synthesis parameters. For the evaluation of the system as an interface for musical expression we made an experiment with real subjects. The results of this experiment are presented and analyzed.

Gesture interaction for electronic music performance

2007

This paper describes an approach for a system which analyses an orchestra conductor in real-time, with the purpose of using the extracted information of time pace and expression for an automatic play of a computer-controlled instrument (synthesizer). The system in its final stage will use non-intrusive computer vision methods to track the hands of the conductor. The main challenge is to interpret the motion of the hand/baton/mouse as beats for the timeline.

ZATLAB: A Gesture Analysis System to Music Interaction

ARTECH 2012 Conference, 2012

The human gesture is an important means of expression and interaction with the world, and has an important role on the perception and interpretation of human communication. Over the years, different approaches have been proposed to capture and study human gestures and movements by various fields of study, namely Human Computer Interaction or Kinesiology (the scientific study of the human motion properties). This paper proposes a new modular system, named Zatlab, that allows to control, in real-time, music generation through expressive gestures, allowing dancers and computer music performers (among others) to explore novel ways of interaction with music and sound creation computer tools. The system is based on realtime, non intrusive, human gesture recognition, which analyzes movement and gesture in terms of low level features (e.g. distance, velocity, acceleration) and high level features (e.g. quantity of movement), and uses machine learning algorithms to map them into various parameters in music generation algorithms, allowing a more semantically and artistically meaningful mapping of human gesture to sound.

A new control paradigm: software-based gesture analysis for music

2003

We present a flexible and inexpensive system for analyzing gesture data in a computer. Several basic data reductions are introduced, and the tradeoff between latency and reliability is discussed. An example of the application of this technology to musical performance is presented via the example of the radio drum, a musical controller that produces eight channels of gestural data. The advantages of analyzing this data stream in software are exposed, and future applications of the technology are presented.

From expressive gesture to sound

Journal on Multimodal User Interfaces, 2009

This paper contributes to the development of a multimodal, musical tool that extends the natural action range of the human body to communicate expressiveness into the virtual music domain. The core of this musical tool consists of a low cost, highly functional computational model developed upon the Max/MSP platform that (1) captures real-time movement of the human body into a 3D coordinate system on the basis of the orientation output of any type of inertial sensor system that is OSC-compatible, (2) extract low-level movement features that specify the amount of contraction/expansion as a measure of how a subject uses the surrounding space, (3) recognizes these movement features as being expressive gestures, and (4) creates a mapping trajectory between these expressive gestures and the sound synthesis process of adding harmonic related voices on an in origin monophonic voice. The concern for a user-oriented and intuitive mapping strategy was thereby of central importance. This was achieved by conducting an empirical experiment based on theoretical concepts from the embodied music cognition paradigm. Based on empirical evidence, this paper proposes a mapping trajectory that fa-P.-J. Maes ( ) · M. Leman · M. Lesaffre · M. Demey · cilitates the interaction between a musician and his instrument, the artistic collaboration between (multimedia) artists and the communication of expressiveness in a social, musical context.

Using an Expressive Performance Template in a Music Conducting Interface

Proceedings of the 2004 Conference on New Interfaces For Musical Expression, 2004

This paper describes an approach for playing expressive music, as it refers to a pianist's expressiveness, with a tapping-style interface. MIDI-formatted expressive performances played by pianists were first analyzed and transformed into performance templates, in which the deviations from a canonical description was separately described for each event. Using one of the templates as a skill complement, a player can play music expressively over and under the beat level. This paper presents a scheduler that allows a player to mix her/his own intension and the expressiveness in the performance template. The results of a forty-subject user study suggest that using the expression template contributes the subject's joy of playing music with the tapping-style performance interface. This result is also supported by a brain activation study that was done using a near-infrared spectroscopy (NIRS).

Gesture in performance with traditional musical instruments and electronics

Proceedings of the 2014 International Workshop on Movement and Computing - MOCO '14, 2014

This paper describes the implementation of gestural mapping strategies for performance with a traditional musical instrument and electronics. The approach adopted is informed by embodied music cognition and functional categories of musical gestures. Within this framework, gestures are not seen as means of control subordinated to the resulting musical sounds but rather as significant elements contributing to the formation of musical meaning similarly to auditory features. Moreover, the ecological knowledge of the gestural repertoire of the instrument is taken into account as it defines the action-sound relationships between the instrument and the performer and contributes to form expectations in the listeners. Subsequently, mapping strategies from a case study of electric guitar performance will be illustrated describing what motivated the choice of a multimodal motion capture system and how different solutions have been adopted considering both gestural meaning formation and technical constraints.