Method, system and storage medium for driving virtual human by voice in real time

Abstract

The present application provides a method, system and storage medium for driving a virtual person by voice in real time. The method for driving a virtual person by voice in real time includes: obtaining an RGB image of a human face; performing 3D face reconstruction on the RGB image of a human face to obtain an RGB image of a human face Corresponding 3D face parameters; pre-acquisition of a piece of voice and save it in the cache queue after denoising processing; real-time collection of voice clips and save them in the cache queue after denoising processing; read all voice clips in the cache queue and Splicing, to get the spliced ​​speech clips, and use the spliced ​​speech clips and the pre-trained neural network to get the predicted 3D facial expression parameters; get the rendered RGB image according to the predicted 3D facial expression parameters and 3D face parameters . This application can achieve the real-time performance of the entire driving process without reducing the quality of the virtual human, so that the voice-driven virtual human technology can be used in various real-time applications.

Description

technical field [0001] The present application belongs to the technical field of information security, and in particular relates to a method, system and storage medium for real-time voice driving of a virtual human. Background technique [0002] Voice-driven virtual human technology is a kind of virtual human-driven technology, which uses voice to drive a preset virtual human model to generate a dynamic virtual human image that conforms to the voice content. In recent years, with the development and maturity of voice-driven virtual human technology, voice-driven virtual human technology has derived quite a lot of applications, such as virtual anchors, virtual customer service and virtual idols. Since the avatar often needs to be presented directly to the user, the user has high requirements for the authenticity and accuracy of the voice-driven results. [0003] However, the existing voice-driven virtual human technology cannot be well extended to real-time applications such...

AI Technical Summary

Problems solved by technology

[0003] However, the existing voice-driven virtual human technology cannot be well extended to real-time applications such as virtual meetings. Voice acquisition, neural network computing speed, audio and video delays and other issues cannot meet the real-time requirements; on the other hand, the voice-driven virtual human technology in real-time scenarios requires the length of the input voice segment to be as short as possible, so as to meet the real-time requirements

In the process of voice feature calculation, due to the lack of timing information and necessary semantic information in short voice segments, it is easy to reduce the accuracy of the facial movement of the avatar during the driving process, thereby reducing the sense of reality.

Images