Image-text retrieval system and method based on multi-angle self-attention mechanism

Abstract

The invention belongs to the technical field of cross-modal retrieval, and particularly relates to an image-text retrieval system and method based on a multi-angle self-attention mechanism. The systemcomprises a deep convolutional network, a bidirectional recurrent neural network, an image, a text self-attention network, a multi-modal space mapping network and a multi-stage training module. The deep convolutional network is used for acquiring an embedding vector of an image region feature in an image embedding space. The bidirectional recurrent neural network is used for acquiring an embedding vector of a word feature in a text space, and the two vectors are respectively input to the image and the text self-attention network. The image and text self-attention network is used for acquiringan embedded representation of an image key area and an embedded representation of key words in sentences. The multi-modal space mapping network is used for acquiring the embedded representation of the image text in the multi-modal space. The multi-stage training module is used for learning parameters in the network. A good result is obtained on a common data set Flickr30k and an MSCOCO, and the performance is greatly improved.

Description

technical field [0001] The invention belongs to the technical field of cross-modal retrieval, and in particular relates to an image-text retrieval system and method based on a multi-angle self-attention mechanism. Background technique [0002] In various multimodal information processing tasks, the research on cross-modal analysis and processing between images and texts is a very important one among many research directions. Specifically, it includes tasks such as automatic generation of image descriptions and mutual search of images and texts. Here we focus on cross-modal retrieval, that is, image-text mutual search tasks. Image-text mutual search is to input an image and need to find K sentences with the most similar semantics. Or enter a sentence and find the K images most semantically related to it. Image-text mutual search is a very challenging task, because it involves two very important branch research fields of pattern recognition, namely computer vision and natur...

AI Technical Summary

Problems solved by technology

[0005] The present invention provides an image-text retrieval system based on multi-stage training and multi-angle self-attention mechanism in order to overcome the shortcomings of the features extracted by the existing CNN+RNN model in the image-text retrieval technology that are not detailed enough and the optimization method. method

Images