Image-text mixed retrieval method, device, storage medium, computer equipment

Abstract

The present invention relates to a method, device, storage medium, and computer equipment for a mixed image-text retrieval method. The image-text mixed retrieval method includes: acquiring information to be retrieved input by a user, if the information to be retrieved includes both text data and image data , then perform an inverted index query operation after the text data is segmented to obtain the first candidate set; input the image data into a pre-trained neural network model to obtain a feature vector corresponding to the image data, and After the feature vector encoding process, perform an inverted index query operation to obtain a second candidate set; merge the documents in the first candidate set and the second candidate set, and count the merged documents in the first candidate set set and / or the number of hits in the second candidate set, sort the merged documents according to the statistical results, and return the sorted results to the display page; while improving the retrieval accuracy, the system maintenance is effectively reduced cost.

Description

technical field [0001] The present invention relates to the technical field of information retrieval, in particular to a method, device, storage medium, and computer equipment for mixed image and text retrieval. Background technique [0002] The previous retrieval systems mainly include two modes: single-modal retrieval and cross-modal retrieval. Whether it is single-modal retrieval or cross-modal retrieval, the query data only contains data of one modality. However, in recent years, with the development of the mobile Internet, people's query requirements have also changed, and new retrieval requirements have gradually emerged, and mixed-modal retrieval is one of them. Mixed-modal retrieval refers to the retrieval of data that includes multiple modalities when inputting query data into the retrieval system. For example, in the field of education, when inputting query data, it generally includes a text description of a topic and related graphic images. At this time, Retrieva...

AI Technical Summary

Problems solved by technology

However, there are two problems with this method: one is that a separate index system needs to be maintained for text and image retrieval. When the amount of data is large, the storage cost and calculation cost of maintaining two index systems are very high ; Second, after obtaining the set of candidate image-text pairs through text index and image index retrieval, it is difficult to find a sorting strategy that can combine the two sets of retrieval results to sort and get the final query result

Images