Active Listener: Continuous Generation of Listener’s Head Motion Response in Dyadic Interactions (original) (raw)

Ghosh, Bishal, Li, Emma ORCID: https://orcid.org/0000-0003-4200-0669 and Guha, Tanaya ORCID: https://orcid.org/0000-0003-2167-4891(2024) Active Listener: Continuous Generation of Listener’s Head Motion Response in Dyadic Interactions. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2025), Hyderabad, India, 6-11 April 2025, ISBN 9798350368741(doi: 10.1109/ICASSP49660.2025.10889429)

Abstract

A key component of dyadic spoken interactions is the contextually relevant non-verbal gestures, such as head movements that reflect a listener’s response to the interlocutor’s speech. Although significant progress has been made in the context of generating co-speech gestures, generating listener’s response has remained a challenge. We introduce the task of generating continuous head motion response of a listener in response to the speaker’s speech in real time. To this end, we propose a graph-based end-to-end crossmodal model that takes interlocutor’s speech audio as input and directly generates head pose angles (roll, pitch, yaw) of the listener in real time. Different from previous work, our approach is completely data-driven, does not require manual annotations or oversimplify head motion to merely nods and shakes. Extensive evaluation on the dyadic interaction sessions on the IEMOCAP dataset shows that our model produces a low overall error (4.5 degrees) and a high frame rate, thereby indicating its deployability in real-world human-robot interaction systems. Our code is available at - https://github. com/bigzen/Active-Listener

University Staff: Request a correction | Enlighten Editors: Update this record

Funder and Project Information

DTP 2224 University of Glasgow

Christopher Pearce

EP/W524359/1

US - Office of the Vice Principals

Deposit and Record Details