Specific modal semantic space modeling-based cross-modal similarity learning method

Abstract

The invention relates to a specific modal semantic space modeling-based cross-modal similarity learning method. The method comprises the following steps of: 1, establishing a cross-modal database which comprises data of multiple modal types, and dividing data in the database into a training set, a test set and a validation set; 2, aiming at each modal type in the cross-modal database, constructinga semantic space aiming at the specific modal, and projecting data of the other modal types to the semantic space so as to obtain a cross-modal similarity aiming at the specific modal; 3, fusing cross-modal similarities, aiming at specific modals, obtained from different modal semantic spaces to obtain a final cross-modal similarity; and 4, calculating a similarity between a query example and a query target by taking any modal type in the test set as a query modal and taking another modal type as a target modal, and obtaining a related result list of data of the target modal according to thesimilarity. According to the method, the correctness of cross-modal retrieval can be improved.

Description

technical field [0001] The invention relates to the field of multimedia retrieval, in particular to a cross-modal similarity learning method based on specific modality semantic space modeling. Background technique [0002] Nowadays, multimodal data including images, videos, texts and audios widely exist on the Internet, and these multimodal data are the basis for helping artificial intelligence to recognize the real world. Some research work has been trying to break the heterogeneous gap between different modal data, and cross-modal retrieval, as one of the hot research issues, can realize information retrieval across different modal data, and has a wide range of practical application requirements. Such as search engines and digital libraries. Traditional single-modal retrieval, such as image retrieval, video retrieval, etc., is limited to a single modality, and can only return retrieval results of the same modality type as the query. The difference is that cross-modal ret...

AI Technical Summary

Problems solved by technology

Therefore, treating different modality data equally to mine the potential fine-grained alignment content and construct a unified space will lose the unique and useful information within the modality, instead of making full use of the rich intrinsic information provided by each modality.

Images