Antagonistic cross-media search method based on limited text space

Abstract

The invention discloses an antagonistic cross-media search method based on limited text space. The method comprises the steps that a characteristic extraction network, a characteristic mapping networkand a modality classifier are designed, the limited text space is obtained by learning, image and text characteristics suitable for the cross-media search are extracted, and the mapping of the imagecharacteristic from image space to text space is achieved; the difference among characteristic distributions of different modality data is constantly reduced in the learning process by an antagonistictraining mechanism; and thereby the cross-media search is achieved. The antagonistic cross-media search method based on the limited text space has the advantages that behavioral expressions of peoplein cross-media search tasks can be better fitted; the image and text characteristics more suitable for the cross-media search tasks are obtained, and the defects of pre-training characteristics in anexpressive ability are made up; and the antagonistic learning mechanism is introduced, so that the search accuracy is further improved through the maximum minimum game between the modality classifierand the characteristic mapping network.

Description

technical field [0001] The invention relates to the technical field of computer vision, in particular to an adversarial cross-media retrieval method based on a restricted text space. Background technique [0002] With the advent of the Web 2.0 era, a large amount of multimedia data (images, texts, videos, audios, etc.) began to accumulate and spread on the Internet. Different from traditional single-modal retrieval tasks, cross-media retrieval is used to achieve bidirectional retrieval between different modal data, such as text retrieval images and image retrieval texts. However, due to the inherently heterogeneous nature of multimedia data, their similarity cannot be directly measured. Therefore, the core problem of this type of task is how to find a homogeneous mapping space, so that the similarity between heterogeneous multimedia data can be directly measured. In the current field of cross-media retrieval, people have done a lot of research on the basis of this problem,...

AI Technical Summary

Problems solved by technology

However, they are also pre-trained on some datasets different from cross-media retrieval, so the extracted features are not suitable for cross-media retrieval

[0004] The second defect is reflected in the choice of isomorphic feature space

Therefore, this feature will also lose the rich action and interactive information contained in the image, which also shows that for cross-media retrieval, the Word2Vec space is not an effective text feature space.

[0005] The third defect is reflected in the difference in the characteristic distribution of different modal data

Although existing methods map data features of different modalities to an isomorphic feature space, there is still a modality gap between them, and there are obvious differences in feature distribution, which will lead to Media retrieval performance degradation

Images