Method and system for driving virtual human in real time through voice and storage medium

Abstract

The invention provides a method and system for driving a virtual human in real time through voice and a storage medium. The method for driving the virtual human in real time through voice comprises the steps: acquiring a face RGB image; performing 3D face reconstruction on the face RGB image to obtain a 3D face parameter corresponding to the face RGB image; pre-acquiring a voice clip, denoising the voice clip and then storing the voice clip in a cache queue; acquiring a voice clip in real time, denoising the voice clip, and storing the voice clip in the cache queue; reading all the voice clips in the cache queue and splicing the voice clips to obtain spliced voice clips, and obtaining predicted 3D facial expression parameters by using the spliced voice clips and a pre-trained neural network; and obtaining a rendered RGB image according to the predicted 3D face expression parameters and 3D face parameters. According to the method, the real-time performance of the whole driving process can be achieved on the basis of not reducing the quality of the virtual human, so that the voice-driven virtual human technology can be applied to various real-time applications.

Description

technical field [0001] The present application belongs to the technical field of information security, and in particular relates to a method, system and storage medium for real-time voice driving of a virtual human. Background technique [0002] Voice-driven virtual human technology is a kind of virtual human-driven technology, which uses voice to drive a preset virtual human model to generate a dynamic virtual human image that conforms to the voice content. In recent years, with the development and maturity of voice-driven virtual human technology, voice-driven virtual human technology has derived quite a lot of applications, such as virtual anchors, virtual customer service and virtual idols. Since the avatar often needs to be presented directly to the user, the user has high requirements for the authenticity and accuracy of the voice-driven results. [0003] However, the existing voice-driven virtual human technology cannot be well extended to real-time applications such...

AI Technical Summary

Problems solved by technology

[0003] However, the existing voice-driven virtual human technology cannot be well extended to real-time applications such as virtual meetings. Voice acquisition, neural network computing speed, audio and video delays and other issues cannot meet the real-time requirements; on the other hand, the voice-driven virtual human technology in real-time scenarios requires the length of the input voice segment to be as short as possible, so as to meet the real-time requirements

In the process of voice feature calculation, due to the lack of timing information and necessary semantic information in short voice segments, it is easy to reduce the accuracy of the facial movement of the avatar during the driving process, thereby reducing the sense of reality.

Images