Image-text hybrid retrieval method and device, storage medium and computer equipment

Abstract

The invention relates to an image-text hybrid retrieval method and device, a storage medium and computer equipment, and the method comprises the steps: obtaining to-be-retrieved information inputted by a user, carrying out the word segmentation of text data if the to-be-retrieved information contains the text data and image data at the same time, and then carrying out the inverted index query operation, obtaining a first candidate set; inputting the image data into a pre-trained neural network model to obtain a feature vector corresponding to the image data, encoding the feature vector, and performing inverted index query operation to obtain a second candidate set; merging the documents in the first candidate set and the second candidate set, counting the hit times of the merged documentsin the first candidate set and / or the second candidate set, sorting the merged documents according to a statistical result, and returning a sorting result to a display page; and the maintenance cost of the system is effectively reduced while the retrieval accuracy is improved.

Description

technical field [0001] The present invention relates to the technical field of information retrieval, in particular to a method, device, storage medium, and computer equipment for mixed image and text retrieval. Background technique [0002] The previous retrieval systems mainly include two modes: single-modal retrieval and cross-modal retrieval. Whether it is single-modal retrieval or cross-modal retrieval, the query data only contains data of one modality. However, in recent years, with the development of the mobile Internet, people's query requirements have also changed, and new retrieval requirements have gradually emerged, and mixed-modal retrieval is one of them. Mixed-modal retrieval refers to the retrieval of data that includes multiple modalities when inputting query data into the retrieval system. For example, in the field of education, when inputting query data, it generally includes a text description of a topic and related graphic images. At this time, Retrieva...

AI Technical Summary

Problems solved by technology

However, there are two problems with this method: one is that a separate index system needs to be maintained for text and image retrieval. When the amount of data is large, the storage cost and calculation cost of maintaining two index systems are very high ; Second, after obtaining the set of candidate image-text pairs through text index and image index retrieval, it is difficult to find a sorting strategy that can combine the two sets of retrieval results to sort and get the final query result

Images