Cross-modal semantic clustering method based on bidirectional CNN

Abstract

The invention discloses a cross-modal semantic clustering method based on a bidirectional CNN, and the method comprises the steps: firstly carrying out the preprocessing of data, and carrying out the pre-training of a text sample of a training set; then constructing a cross-modal retrieval network, training the cross-modal retrieval network through the training set, and calculating a loss function of the network; carrying out back propagation, and optimizing the connection weight through a selected optimizer and corresponding parameters; performing training for multiple rounds to obtain a final network model; and finally, testing the trained model through the test set, and calculating various evaluation indexes. According to the method disclosed by the invention, the accuracy and the efficiency of cross-modal retrieval are improved by utilizing clustering of semantic information. According to the method, the loss of the sample and the clustering center in the target space, the distribution difference loss of the categories in different modes and the discrimination loss are designed to help semantic clustering, so that the recognition capability among different categories is enhanced, and the correlation among different modes is enhanced.

Description

technical field [0001] The invention relates to the field of computer vision, in particular to a cross-modal retrieval method based on deep learning. Background technique [0002] In the era of the explosion of new media information, every new media user will publish a variety of multimedia information in different modalities anytime and anywhere, such as pictures, music, video or text. Due to the rapid development of multimedia information, with the increase in the number and types of multimedia information, it becomes difficult for users to accurately obtain the information they want. When obtaining information, it is always accompanied by other information with different degrees of relevance. These data are not only huge in number, but most of them are unlabeled data, and there is a "heterogeneous gap" between data in different modalities, so the main technical problem of cross-modal retrieval is to span these different modal data The "gap" between extraction precision a...

AI Technical Summary

Problems solved by technology

[0004] Although cross-modal retrieval is based on mapping to the same subspace, depending on the selection and setting of the loss function, there will be different efficiencies and accuracy

Images