Method for generating mouth shape video based on voice

Abstract

The invention provides a method for generating a mouth shape video based on a voice. The method comprises the steps of firstly, intercepting a stationary picture when a video lag occurs, performing detection for obtaining a face image therein, and calculating an MFCC coefficient matrix; then, performing processing by means of a trained deep network Speech2Vid model; and finally, aligning the faceimage by means of similarity transformation, and performing replacement for obtaining a new static image, and using the obtained static image as a next frame of the video. Because of dependence to existing speech video fragments, the stationary image which comprises the corresponding mouth shape face is generated as the next image directly through studying the relation between mouth shape changesin an original audio and the video image. The method can effectively alleviate video lags and improves video calling experience of the people.

Description

technical field [0001] The invention belongs to the technical fields of computer vision and graphics processing, and in particular relates to a method for generating lip-shape videos based on speech. Background technique [0002] As video chat gradually replaces traditional voice communication methods, while people enjoy the fun of face-to-face conversations, some new problems have also arisen. Restricted by different network environments in different regions, the stability of video chat is poor under poor network conditions, which brings a very bad experience to users. Among them, the audio file is small, and a smooth communication experience can be obtained even in a poor network environment, but the video information may appear to be stuck due to relatively large files. To solve this problem, the audio information of the speaker can be analyzed to generate the corresponding mouth shape, so that the incoherent picture can be compensated, and the video chat experience unde...

AI Technical Summary

Problems solved by technology

LSTM selects a target mouth area from the dictionary of saved target frames instead of generating images, so a huge amount of video frames with unique target identities are required to choose from, which requires a large amount of storage space for video chat devices and strong performance, which is difficult to be satisfied in real life

Moreover, this method only compensates the picture after screening from the existing image information, and it is powerless if there is no image information to choose from.

Garrido et al. in the literature "P. Garrido, L. Valgaerts, H. Sarmadi, et al. "VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track", Computer Graphics Forum, pp.193-204, 2015 The above method was improved in the above method, and the mouth shape in the video of the dubber was transferred to the target video, so as to synthesize the face of the target speaker. However, this method requires the dubber in the video to speak the voice clip, so that the real-time performance cannot be guaranteed. , unable to meet the basic requirements of video chat

Images