Speech-driven lip-synchronous face video synthesis algorithm based on concatenated convolution LSTM

Abstract

The invention discloses a speech-driven lip synchronous face video synthesis algorithm of cascaded convolution LSTM. The speech video of the target person is captured as the background video, and the3D face model of the target is obtained by reconstructing the 3D face of the image sequence, and the facial animation vector sequence of the background video is obtained. The audio signal extracts thespeech features of the filter bank; The speech features of the filter bank are used as the input of the concatenated convolution short-time memory network, and the facial animation vector sequence isused as the output for the training test. Facial animation vector sequences of audio signals are used to replace facial animation vector sequences of target 3D face models to generate new 3D face models and render face images to synthesize lip-shaped synchronous face videos. The invention retains more voiceprint information, innovates to obtain the speech characteristics of the filter bank through the two-dimensional convolution neural network, expands the receptive field of the convolution neural network, increases the network depth, and obtains accurate lip-shaped synchronous face video.

Description

technical field [0001] The present invention relates to the field of computer vision and related technologies of audio signal processing, in particular to a voice-driven lip sync human face video algorithm based on cascaded convolution long short-term memory network structure (cascaded convolution LSTM). Background technique [0002] After years of exploration and development, computer vision has been applied in many fields such as digital entertainment, medical health, and security monitoring. Synthesizing realistic visual content not only has great commercial value, but also has been expected by the industry. Many movie special effects would not be possible without the comprehensive visual effects of computer synthesis. At present, there are already a large number of artificially synthesized videos on the Internet. In addition, speech recognition and text-to-speech technologies have also been widely used in chatbots. The present invention hopes to make the online chat r...

AI Technical Summary

Problems solved by technology

Therefore, it is challenging to recover high-fidelity high-dimensional low-frequency video directly from low-dimensional high-frequency speech audio signals or text-to-speech audio signals, which is a severely underconstrained ill-conditioned problem.

Images