A method and system for cross-modal retrieval

Abstract

The invention is applicable to the technical field of retrieval. A method for cross-modal retrieval is provided, which includes using a stacked restricted Boltzmann machine and a multi-modal depth confidence network to extract modal friendly features and modal mutual features of image and text respectively, wherein the modal friendly features can make the statistical characteristics between the obtained features more similar to the input, and modal mutual features can get the mutual information lost in the original input instances, and the two features can be fused to get the mixed features, and the final shared features can be obtained by multiple bimodal automatic coding. The embodiment of the invention utilizes a stacked restricted Boltzmann machine to extract the internal characteristics of each mode, and the mixed features suitable for cross-modal retrieval are constructed by fusing the lost mutual information between the two features in depth confidence network mining. The accuracy and retrieval speed of the cross-modal retrieval task are effectively improved by using the multi-layered and bi-modal automatic coding network mining the complex information of the cross-modal.

Description

technical field [0001] The invention belongs to the technical field of retrieval, and in particular relates to a cross-modal retrieval method and system. Background technique [0002] Cross-modal retrieval is a novel retrieval method that is capable of retrieving multimodal data. For example, input an image, retrieve the corresponding text in the text database; given the text, find the corresponding image in the image database. [0003] At present, the cross-modal retrieval method based on deep neural network mainly includes two steps: (1) extracting the internal features of each modality and the features between the modalities; (2) establishing the respective shared feature. However, in the first step, the mutual information between modalities is often lost; in the second step, current methods use relatively shallow networks, and it is difficult to mine complex information across modalities. Contents of the invention [0004] The technical problem to be solved by the p...

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is to provide a cross-modal retrieval method and system, aiming to solve the problem of missing the interrelationships between the modalities and making it difficult to mine cross-modal complex information in the prior art when performing cross-modal retrieval question

Images