Cross-modal retrieval method and system based on dual coding and combination and storage medium

Abstract

The invention discloses a cross-modal retrieval method, system and device based on dual coding and combination. The method is a cross-modal retrieval algorithm based on dual coding and dual joint embedded learning. The method comprises the steps of extracting and encoding various features of a video through a neural network, performing multi-layer encoding on character features, learning and training two network models combined with video text embedding, and obtaining a text-to-video retrieval result or a video-to-text retrieval result through the two models. By using the method provided by the invention, the semantic difference between the video features and the text described by the natural language can be reduced, the potential information and the relationship between the video and thetext are captured, learned and optimized in a targeted and complementary manner, and finally, the retrieval accuracy between the video and the text is improved. The method can be widely applied to thetechnical field of video processing.

Description

technical field [0001] The present invention relates to the technical field of video processing, in particular to a cross-modal retrieval method, system, device and storage medium based on dual coding and association. Background technique [0002] Modality: Refers to a source or form of data, such as text, audio, image, video, etc. [0003] Cross-modality: Some data exist in different forms, but all describe the same thing or event. [0004] Cross-modal retrieval: Given a retrieval modal instance, retrieve another modal instance that is semantically similar or consistent with its instance. [0005] With the development of the Internet and information technology, there are more and more types of data. Common multimedia data include text data, image data, video data and audio data. The rapid growth of video on the Internet makes searching for video content using natural language queries a significant challenge. Compared with simple images, video is composed of consecutive m...

AI Technical Summary

Problems solved by technology

However, this type of video ignores the characteristics of the video and cannot use the complementary information contained in the video, such as spatio-temporal information and sound information, which limits the robustness of the retrieval system, and the accuracy of the retrieval results is not high enough to meet the actual needs.

Images