Cross-modal lip reading antagonism double-contrast self-supervised learning method

Abstract

The invention provides a cross-modal lip reading antagonistic double-contrast self-supervised learning method. The method comprises a visual encoder, an audio encoder, two multi-scale time convolution networks with average pools, an identity discriminator and a modal classifier. The method learns an effective visual representation by combining double contrast learning based on audio-visual synchronization, identity adversarial training and modal adversarial training. In double-contrast learning, noise contrast estimation is used as a training target to distinguish a real sample and a noise sample. In adversarial training, an identity discriminator and a modal classifier are provided for audio-visual representation, the identity discriminator is used for discriminating whether input visual features have a common identity, the modal classifier is used for predicting whether the input features belong to a visual mode or an audio mode, and then a momentum gradient inversion layer is used for realizing adversarial training.

Description

technical field [0001] The invention belongs to the field of image processing, and in particular relates to an adversarial double-contrast self-supervised learning method for cross-modal lip reading. Background technique [0002] Supervised deep learning has made revolutionary progress in many fields such as image classification, object detection and segmentation, speech recognition, machine translation, etc. Although supervised learning has made remarkable progress in the past few years, its success largely relies on large amounts of human-annotated training data. However, for some specific tasks, such as lip reading, the cost of annotation can be prohibitively expensive. In recent years, self-supervised learning has attracted increasing attention due to its high labeling efficiency and good generalization ability. Self-supervised learning methods have shown great potential in natural language processing, computer vision, and cross-modal representation learning. [0003]...

AI Technical Summary

Problems solved by technology

Previous studies in this area have attempted pairwise comparison strategies to bring visual embeddings closer to corresponding audio embeddings and further away from non-corresponding audio embeddings. Two-contrast learning requires manual selection of negative samples, and its effect largely depends on the effectiveness of negative samples; second, representation learning only relies on synchronized audio-video data pairs, and other self-supervised signals, such as speaker-related information and modality information, can also be used to optimize the quality of learned representations, but these self-monitoring signals are usually ignored in previous work

Images