Cross-modal correlation learning method based on multi-granularity hierarchical network

Abstract

The invention relates to a cross-modal correlation learning method based on multi-granularity hierarchical network. The method comprises the following steps: 1, building a cross-modal database containing multiple modal types, dividing the data in the database into a training set, a validation set and a testing set, processing the different modal data in the database in a partitioning way and extracting all modal original modal data and feature vector of data after partitioning; 2, training a multi-granularity hierarchical structure by utilizing the original data and partitioned data and learning unified representation for different modal data; 3, obtaining the unified representation of different modal data by utilizing the trained multi-granularity hierarchical structure, and calculating the similarity of different modal data; 4, taking any modal type in the testing set as a query modal, taking another modal type as a target modal, calculating the similarity between a query sample and a query target and obtaining the related result list of the target modal data according to the similarity. According to the method of the invention, the accuracy rate of cross-modal searching can be improved.

Description

technical field [0001] The invention relates to the field of multimedia retrieval, in particular to a cross-modal association learning method based on a multi-granularity hierarchical network. Background technique [0002] In recent years, with the rapid development of computer technology, the acquisition and processing of information has changed from a single mode of text, image, audio, video, etc. to a form of integration of multiple modes. Multimodal retrieval has become an important issue in the field of information retrieval, with wide applications in search engines and big data management. The traditional retrieval method is mainly in the form of a single modality, that is, the user submits a modality type of data as a query, and the retrieval system returns the retrieval results of the same modality, such as image retrieval, text retrieval, etc. This retrieval method cannot directly measure the similarity between data of different modalities, such as the similarity b...

AI Technical Summary

Problems solved by technology

However, there are three limitations in the existing methods. One is that in the first stage, the existing methods only model the association relationship within the modality, while ignoring the supplementary role of the inter-modal association for the separation of feature representation learning; the second is that in the In the second stage, the existing methods only use a single loss function to constrain, and cannot fully balance the association learning process between the modalities and the modalities; in addition, the existing methods only consider the original data of different modalities, while ignoring other modalities. The rich and fine-grained information provided by various internal parts cannot fully mine the cross-modal relationship

Images