Gesture facilitates the syntactic analysis of speech - PubMed (original) (raw)

Gesture facilitates the syntactic analysis of speech

Henning Holle et al. Front Psychol. 2012.

Abstract

Recent research suggests that the brain routinely binds together information from gesture and speech. However, most of this research focused on the integration of representational gestures with the semantic content of speech. Much less is known about how other aspects of gesture, such as emphasis, influence the interpretation of the syntactic relations in a spoken message. Here, we investigated whether beat gestures alter which syntactic structure is assigned to ambiguous spoken German sentences. The P600 component of the Event Related Brain Potential indicated that the more complex syntactic structure is easier to process when the speaker emphasizes the subject of a sentence with a beat. Thus, a simple flick of the hand can change our interpretation of who has been doing what to whom in a spoken sentence. We conclude that gestures and speech are integrated systems. Unlike previous studies, which have shown that the brain effortlessly integrates semantic information from gesture and speech, our study is the first to demonstrate that this integration also occurs for syntactic information. Moreover, the effect appears to be gesture-specific and was not found for other stimuli that draw attention to certain parts of speech, including prosodic emphasis, or a moving visual stimulus with the same trajectory as the gesture. This suggests that only visual emphasis produced with a communicative intention in mind (that is, beat gestures) influences language comprehension, but not a simple visual movement lacking such an intention.

Keywords: P600; ambiguity; audiovisual; language; syntax.

PubMed Disclaimer

Figures

Figure 1

Figure 1

Materials and Results. Original German sentences, as uttered in our video stimuli, as well as literal English translation in italics. Full German SOV sentence: Peter sagt, dass die Frau die Männer gegrüß t hat. English gloss: Peter says that the woman has greeted the men. Full German OSV sentence: Peter sagt, dass die Frau die Männer gegrüß t haben. English gloss: Peter says that the men have greeted the woman. ERPs were time-locked to the critical sentence-final words (underlined). ERPs for the preferred Subject–Object–Verb order (SOV) are shown in blue and the ERPs for the less preferred Object–Subject–Verb order (OSV) are shown in yellow. Grand-average ERPs (Experiment 1: n = 24; Experiment 2: n = 19; Experiment 3: n = 23) were averaged across four regions of interest: Anterior-Left (AL), Anterior-Right (AR), Posterior-Left (PL), and Posterior-Right (PR). Text highlighted by red bars indicates those portions of speech emphasized either by a visual beat gesture (Experiment 1), a beat-induced pitch accent (Experiment 2) or a moving point (Experiment 3). Bar graphs show the amplitude of the P600 effect (±SEM). ROIs in which the P600 effect is significant at p < 0.05 are marked by an asterisk (*).

Figure 2

Figure 2

Outline of the cross-splicing procedure for Experiment 1. Literal translation of the sentence on the left (portions used for cross-splicing are highlighted in blue): Peter says, that the woman the men greeted has. Literal translation of sentence on the right (highlighted in orange): Peter says, that the men the woman greeted have.

Figure 3

Figure 3

Outline of the cross-splicing procedure for Experiment 2. Noun phrases in CAPITALS indicate those phrases during which the speaker produced a beat gesture during the recording.

References

    1. Alibali M. W., Heath D. C., Myers H. J. (2001). Effects of visibility between speaker and listener on gesture production: some gestures are meant to be seen. J. Mem. Lang. 44, 169–188 10.1006/jmla.2000.2752 -DOI
    1. Bader M., Meng M. (1999). Subject-object ambiguities in German embedded clauses: an across-the-board comparison. J. Psycholinguist. Res. 28, 121–143 10.1023/A:1023206208142 -DOI
    1. Bavelas J., Gerwing J., Sutton C., Prevost D. (2008). Gesturing on the telephone: independent effects of dialogue and visibility. J. Mem. Lang. 58, 495–520 10.1016/j.jml.2007.02.004 -DOI
    1. Boersma P., Weenink D. (2010). Praat: Doing Phonetics by Computer, Version 5.1.15 [Computer program]. Available at: http://www.praat.org/
    1. Crocker M. W., Knoeferle P., Mayberry M. R. (2010). Situated sentence processing: the coordinated interplay account and a neurobehavioral model. Brain Lang. 112, 189–201 10.1016/j.bandl.2009.03.004 -DOI -PubMed

LinkOut - more resources