Fine-grained cross-media retrieval method based on multi-model network

Abstract

The invention discloses a fine-grained cross-media retrieval method based on a multi-model network, and the method comprises the steps of obtaining a cross-media data set, and carrying out the preprocessing of the cross-media data set to obtain cross-media data; respectively extracting special features of each media data; extracting public features of the media data; performing weighted summationon the special features and the public features of the cross-media data to obtain final joint features; and measuring the similarity between different media features by using the cosine distance, andsorting the media features according to the similarity. According to the invention, a media private network and a public network are constructed, and the media private network comprises a feature extractor of each media and is used for extracting a private feature of each media; the public network comprises a unified network capable of learning four kinds of media at the same time and used for extracting public features of all the media, the two networks are combined to achieve the purpose that heterogeneous gaps among all the media are eliminated while the features of all the media are reserved to the maximum extent, and therefore effective cross-media retrieval is achieved, and the method has wide application prospects.

Description

technical field [0001] The invention belongs to the technical fields of computer vision, natural language processing, fine-grained recognition, multimedia retrieval, etc., and specifically relates to a fine-grained cross-media retrieval method based on a multi-model network. Background technique [0002] In recent years, with the rapid growth of multimedia data, multimedia data such as images, texts, audio and video has become the main form for people to understand the world. The research on multimedia data has been going on for several years. Past researches usually focus on a single media type, that is, the results of query and retrieval belong to the same media type. At present, the correlation between massive multimedia data is constantly improving, and at the same time, the user's demand for multimedia data retrieval has become very flexible, not only satisfied with a single media type retrieval, so how to realize cross-media retrieval is the key problem that needs to b...

AI Technical Summary

Problems solved by technology

[0003] At present, there are still some deficiencies in the research on fine-grained cross-media retrieval, the most important of which are two points, one is the heterogeneous gap between media, that is, the feature representation of data samples of different media types is very different, so directly Measuring the similarity between them is a very difficult problem

Another problem is that the existing research does not fully consider the fine-grained level caused by small inter-class differences (different fine-grained categories are very similar, such as gray-winged gulls, red birds), and large intra-class differences (objects of the same category are very similar). Because of obvious differences such as pose lighting), this problem makes fine-grained level retrieval more challenging than coarse-grained level

Images