A real-time voice-driven face animation method and system

Abstract

The invention discloses a method and system for real-time voice-driven facial animation, the method comprising acquiring a neutral voice audio-visual data set from a first speaker; using an active appearance model to track and parameterize human face video data; Convert data to phoneme label sequences; train sliding window based deep convolutional neural network model; retarget reference face model to target character model; input retarget phoneme label sequence from second speaker to target role model Deep Convolutional Neural Network Models for Prediction. The system provided by the invention includes: an acquisition module, a face conversion module, a phoneme conversion module, a training module, a retargeting module and a target prediction module. The method and system provided by the present invention solve the problem that the existing voice animation method depends on a specific speaker and speaking style, and cannot retarget the generated animation to any facial equipment.

Description

technical field [0001] The invention relates to the fields of virtual reality and animation, in particular to a method and system for real-time voice-driven facial animation. Background technique [0002] Voice animation is an important and time-consuming aspect of generating photorealistic animation. Broadly speaking, voice animation refers to moving the facial features of a graphic (or robotic) model so that lip movements are synchronized with speech and give the impression of speech generation. As humans, we're all experts on faces, and poor voice animation can be distracting, unpleasant, and confusing. For example, audio-visual language mismatches can sometimes change what viewers think they hear, and high-fidelity voice animation is critical to effective character animation. [0003] However, existing machine learning-based speech animation methods are usually evaluated based on test samples distributed in the same distribution as the training set, and the results are...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the above-mentioned technical problems, the object of the present invention is to provide a method and system for real-time voice-driven facial animation that has nothing to do with the speaker and can be retargeted to any animation equipment, so as to solve the problem that existing voice animation methods rely on Issues with specific speakers, speaking styles, and not being able to retarget generated animations to any facial rig

Images