Cross-modal search method capable of directly measuring similarity of different modal data

Abstract

The invention discloses a cross-modal search method capable of directly measuring similarity of different modal data. The method includes the steps of firstly, feature extracting; secondly, model building and learning; thirdly, cross-media data search; fourthly, result evaluating. By the method compared with traditional cross-media search methods, similarity comparison of different modal data can be performed directly, for cross-modal search tasks, a user can submit texts, images, sounds and the like of optional modals so as to search required corresponding modal results, requirements of cross-media search are satisfied, and search intensions of a user can be achieved more directly. Compared with other cross-media search algorithms capable of directly measuring similarity of different modals, the method is high in noise interference resistance and expression capacity of loosely-related cross-modal data, and better search results can be achieved.

Description

technical field [0001] The invention relates to cross-modal retrieval, in particular to a cross-modal retrieval method that can directly measure the similarity between different modal data. Background technique [0002] Nowadays, the types of electronic data tend to be more and more colorful, and various types of data, such as text, image, sound, map, etc., exist widely on the Internet. Often the same semantic content can be described by data of one modality, or by data of other modalities. Sometimes, for the description of one type of data with a certain semantic, we hope to find the corresponding description of other types of data. For example, search for pictures with similar meanings to the text based on the text, or search for news reports related to the pictures based on the pictures. However, existing retrieval methods are generally aimed at unimodal data, such as text retrieval for text and image retrieval for images. There are also some multimodal or multimedia r...

AI Technical Summary

Problems solved by technology

However, existing retrieval methods are generally aimed at unimodal data, such as text retrieval for text, image retrieval for image

There are also some multimodal or multimedia retrieval methods, but most of these multimodal retrieval methods measure the similarity between the same modalities, and then calculate the similarity between cross-media data through function mapping, and rarely directly compare the differences. Modal Similarity Retrieval Method

The shortcoming of the cross-media retrieval method that measures the similarity between the same modalities is that it cannot learn the relationship between cross-modal data, and needs to rely on the pre-specified matching relationship in the database, and it is loose for multimedia data. The corresponding relationship, the query effect is not ideal

The difficulty of directly comparing the similarity of different modal data is that the characteristics of different modal data are quite different, and generally speaking, the dimension is high, and there is a problem of "semantic gap"

Images