Cross-modal retrieval method based on modal relation learning

Abstract

The invention provides a cross-modal retrieval method based on modal relation learning, which comprises the following steps of: inputting image text pairs with the same semantics in a data set and class labels to which the image text pairs belong into a cross-modal retrieval network model based on modal relation learning for training until the model converges, thereby obtaining a network model M; s2, respectively extracting feature vectors of an image / text to be queried and each text / image in the candidate library by utilizing the network model M obtained by training in S1, thereby calculating the similarity between the image / text to be queried and the text / image in the candidate library, carrying out descending sorting according to the similarity, and returning a retrieval result with the highest similarity; an inter-modal and intra-modal dual fusion mechanism is established for inter-modal relation learning, multi-scale features are fused in the modals, complementary relation learning is directly performed on the fused features by using label relation information between the modals, and in addition, an inter-modal attention mechanism is added for feature joint embedding, so that multi-scale multi-scale feature fusion is realized. And the cross-modal retrieval performance is further improved.

Description

technical field [0001] The invention relates to the fields of multimodal learning and information retrieval, in particular to a cross-modal retrieval method based on modal relationship learning. Background technique [0002] In recent years, different modal data such as images and texts have widely existed in people's Internet life. The traditional single-modal retrieval can no longer meet the increasing retrieval needs of users. It is useful for people to retrieve data between different modalities on the Internet Information brings inconvenience, so cross-modal retrieval becomes an important research problem. It aims to retrieve data between different modalities (image, text, voice, video, etc.), such as image retrieval text, text retrieval audio, audio retrieval video, etc., cross-modal retrieval in medical data analysis, big data management, It is widely used in public opinion detection and other fields. [0003] Modal data generally has the characteristics of low-level...

AI Technical Summary

Problems solved by technology

[0002] In recent years, different modal data such as images and texts widely exist in people's Internet life. The traditional single-modal retrieval can no longer meet the growing retrieval needs of users, and it is useful for people to retrieve massive Internet data between different modalities. Information is inconvenient, so cross-modal retrieval becomes an important research problem

Images