Video synthesis model training method and device, video synthesis method and device, storage medium, program product and electronic equipment

Abstract

The invention provides a video synthesis model training method and device, a video synthesis method and device, a storage medium, a program product and electronic equipment, and the method comprises the steps: obtaining a sample text and a sample video which is a video that a real person reads the sample text; inputting the sample text into the speech synthesis sub-model to obtain a feature vector; inputting the feature vectors into a voice reconstruction human face sub-model to obtain human face feature parameters; inputting the face feature parameters and the sample video into the differentiable rendering sub-model to obtain a face feature map; and inputting the face feature map into the generative adversarial network sub-model to obtain a virtual real person video, and iteratively training the voice reconstruction face sub-model, the differentiable rendering sub-model and the generative adversarial network sub-model based on the virtual real person video and the sample video until a loss function value of the generative adversarial network sub-model meets a preset condition.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a training method for a video synthesis model, a training device for a video synthesis model, a video synthesis method, a video synthesis device, and a computer program for realizing a training method for a video synthesis model or a video synthesis method non-transitory computer-readable storage media and electronic devices. Background technique [0002] At present, with the rapid development of deep learning technology, text-driven video generation technology has gradually become a research hotspot, which can be applied to weather broadcasting, news broadcasting, online education and other fields. [0003] In related technologies, text-driven video generation technology usually uses multiple models for separate training. For example, text-to-speech synthesis uses a speech synthesis sub-model for separate training, and speech-to-face reconstruction also uses a separate...

AI Technical Summary

Problems solved by technology

[0004] The current multi-model separate training method needs to prepare multiple training sample data for each model, the training efficiency is low, and the video quality of the text-based video synthesis model based on the training obtained still needs to be improved.

Images