Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

AI synthetic anchor generation method, electronic device and readable storage medium

A generator and video frame technology, applied in biological neural network models, speech analysis, instruments, etc., can solve the problems of not having wide practicability, the unnatural combination of anchor voice and expression, and the inability to promote multiple speakers. Achieve the effect of small difference in details, high video quality, and easy to use

Active Publication Date: 2022-05-31
INNER MONGOLIA UNIVERSITY
View PDF14 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] At present, the performance of speech synthesis, mouth shape synthesis and expression synthesis technology has reached a relatively high level, but there are still many research difficulties, such as the inability to generate new anchor content with a small amount of speaker material, which is not widely applicable and cannot be extended to Multiple speakers can only target specific people, and the combination of the anchor's voice and expression is still unnatural

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • AI synthetic anchor generation method, electronic device and readable storage medium
  • AI synthetic anchor generation method, electronic device and readable storage medium
  • AI synthetic anchor generation method, electronic device and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0040] The lip-sync network consists of a face encoder and an audio encoder, both consisting of a stack of 2D convolutional layers

[0041]

[0042] P

[0045] The video generator and each discriminator are trained sequentially through the following two stages:

[0047]

[0049] The adversarial loss calculation of the frame discriminator is shown in formula (3):

[0050]

[0052] The feature matching loss calculation of the frame discriminator is shown in formula (4):

[0053]

[0058]

[0060] L

[0061]

[0062] In formula (6), L represents the length of the time interval for generating the adversarial loss, and J represents the video in the time length [t-L, t]

[0064]

[0067]

[0069] Through the face key point detection, the obtained eye aspect ratio is used to calculate the blinking loss, and the blinking loss is used to train the vision.

[0070]

[0071] L

[0073] Adversarial training using temporal adversarial loss, L1 reconstruction loss, lip sync loss, and blink loss ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses an AI synthetic anchor generating method, electronic equipment and a readable storage medium. The method includes the following steps: collecting data, preprocessing the data to obtain voice data and video frame data; training a mouth shape synchronization discriminator to judge Synchronization probability of voice data and video frame data; training video generator to generate synthetic video frames; the difference between the AI ​​synthetic anchor and real video generated by the present invention is small, the synchronization of voice and mouth shape in the video is good, the video transition is smooth, and there is no jump .

Description

AI synthetic anchor generation method, electronic device and readable storage medium technical field The invention belongs to artificial intelligence technical field, relate to a kind of AI synthetic anchor generation method, electronic equipment and readability storage medium. Background technique [0002] "AI (Artificial Intelligence) synthetic anchor" is to detect the key points of the face, and extract the features of the face. A number of cutting-edge technologies such as extraction, face reconstruction, etc., combined with multi-modal information such as voice and video frames for joint modeling training, generate and The AI ​​avatar model that is no different from a real person, this technology can automatically generate a video of the corresponding content from the input text, and ensure that the video The medium audio, facial expressions and lip movements are naturally consistent, showing the same information transmission effect as that of a live broadcaster. ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G10L21/10G10L25/57G10L25/24G06V40/16G06V10/82G06N3/04
CPCG10L21/10G10L25/57G10L25/24G10L2021/105G06V40/172G06V40/161G06N3/045
Inventor 王炜华董林坤张晖飞龙高光来
Owner INNER MONGOLIA UNIVERSITY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com