Method and system for driving human face animation through real-time voice

Abstract

The invention discloses a method and a system for driving human face animation by real-time voice. The method comprises the following steps: acquiring a neutral voice audio-visual data set from a first speaker; tracking and parameterizing face video data by using an active appearance model; converting the voice data into a phoneme label sequence; training a deep convolutional neural network modelbased on a sliding window; relocating the target of the reference face model to the target role model; and inputting the target phoneme label sequence from the second speaker into the deep convolutional neural network model of the target role model for prediction. The system provided by the invention comprises an acquisition module, a face conversion module, a phoneme conversion module, a trainingmodule, a target redetermination module and a target prediction module. According to the method and the system provided by the invention, the problems that the existing voice animation method dependson a specific speaker and a specific speaking style and cannot retarget the generated animation to any facial equipment are solved.

Description

technical field [0001] The invention relates to the fields of virtual reality and animation, in particular to a method and system for real-time voice-driven facial animation. Background technique [0002] Voice animation is an important and time-consuming aspect of generating photorealistic animation. Broadly speaking, voice animation refers to moving the facial features of a graphic (or robotic) model so that lip movements are synchronized with speech and give the impression of speech generation. As humans, we're all experts on faces, and poor voice animation can be distracting, unpleasant, and confusing. For example, audio-visual language mismatches can sometimes change what viewers think they hear, and high-fidelity voice animation is critical to effective character animation. [0003] However, existing machine learning-based speech animation methods are usually evaluated based on test samples distributed in the same distribution as the training set, and the results are...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the above-mentioned technical problems, the object of the present invention is to provide a method and system for real-time voice-driven facial animation that has nothing to do with the speaker and can be retargeted to any animation equipment, so as to solve the problem that existing voice animation methods rely on Issues with specific speakers, speaking styles, and not being able to retarget generated animations to any facial rig

Images