Adversarial Cross-media Retrieval Method Based on Constrained Text Space

Abstract

The present invention discloses an adversarial cross-media retrieval method based on a restricted text space, and designs a feature extraction network, a feature mapping network, and a modality classifier, obtains a restricted text space through learning, and extracts images and images suitable for cross-media retrieval. Text features realize the mapping of image features from image space to text space; through the adversarial training mechanism, the difference in feature distribution between different modal data is continuously reduced during the learning process; thus cross-media retrieval is realized. The present invention can better fit human behavior in cross-media retrieval tasks; obtain image and text features that are more suitable for cross-media retrieval tasks, and make up for the lack of expression ability of pre-trained features; introduce adversarial learning Through the max-min game between the modality classifier and the feature mapping network, the retrieval accuracy is further improved.

Description

technical field [0001] The invention relates to the technical field of computer vision, in particular to an adversarial cross-media retrieval method based on a restricted text space. Background technique [0002] With the advent of the Web 2.0 era, a large amount of multimedia data (images, texts, videos, audios, etc.) began to accumulate and spread on the Internet. Different from traditional single-modal retrieval tasks, cross-media retrieval is used to achieve bidirectional retrieval between different modal data, such as text retrieval images and image retrieval texts. However, due to the inherently heterogeneous nature of multimedia data, their similarity cannot be directly measured. Therefore, the core problem of this type of task is how to find a homogeneous mapping space, so that the similarity between heterogeneous multimedia data can be directly measured. In the current field of cross-media retrieval, people have done a lot of research on the basis of this problem,...

AI Technical Summary

Problems solved by technology

However, they are also pre-trained on some datasets different from cross-media retrieval, so the extracted features are not suitable for cross-media retrieval

[0004] The second defect is reflected in the choice of isomorphic feature space

Therefore, this feature will also lose the rich action and interactive information contained in the image, which also shows that for cross-media retrieval, the Word2Vec space is not an effective text feature space.

[0005] The third defect is reflected in the difference in the characteristic distribution of different modal data

Although existing methods map data features of different modalities to an isomorphic feature space, there is still a modality gap between them, and there are obvious differences in feature distribution, which will lead to Media retrieval performance degradation

Images