Video retrieval method based on multi-mode and self-supervised representation learning

Abstract

The invention discloses a video retrieval method based on multi-mode and self-supervised representation learning. The method is applied to the field of video retrieval. When a search video is given, videos with similar pictures or events can be found in a ten-million-level video library. The method can be used for solving the problems of news event aggregation, copyright protection infringement retrieval, multi-modal retrieval and the like of a short video platform. The method mainly comprises the following steps: 1, constructing a supervision data set through unlabeled picture data and picture-text pair data, and training a picture feature extraction network by using the supervision data set; 2, constructing a feature frequency library through a method of performing feature extraction on video frames and calculating domain density; and 3, extracting video representation and constructing a video library, and performing video retrieval by using a neighbor retrieval method. The video retrieval method based on multi-mode and self-supervised representation learning provided by the invention has relatively high accuracy and recall rate in a test data set, and has good robustness.

Description

technical field [0001] The invention belongs to the field of computer technology and image processing, in particular to a video retrieval method based on multimodal and self-supervised representation learning. Background technique [0002] Before 2015, image retrieval and image-text search were one of the most important technologies on the Internet. It is very important to search for pictures by text on search engines, search for pictures by pictures, and search for product pictures on e-commerce platforms. Search technology also urgently needs to quickly move from graphic demand to video demand. [0003] Video retrieval is a very important yet challenging problem, and in recent years, we have witnessed a dramatic increase in the amount of video generated over the Internet, exacerbated by the rapid development of social media applications and video sharing platforms. Due to the large number of videos released by users in a very short period of time on video platforms, thes...

AI Technical Summary

Problems solved by technology

[0004] At the same time, we have also observed that a large number of videos are actually stealing content from others, making secondary edits in violation of regulations, and obtaining huge illegal benefits at low cost

Images