One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (original) (raw)
Abstract:We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating face-to-face video conferencing experiences.
Submission history
From: Ting-Chun Wang [view email]
[v1] Mon, 30 Nov 2020 18:56:35 UTC (6,645 KB)
[v2] Tue, 30 Mar 2021 17:54:33 UTC (7,621 KB)
[v3] Fri, 2 Apr 2021 23:37:06 UTC (7,619 KB)