Multi-modal Transformer image description method based on dynamic word embedding

Abstract

The invention discloses a multi-modal Transformer image description method based on dynamic word embedding, and belongs to the field of artificial intelligence. According to the invention, a model for simultaneously performing intra-modal attention and inter-modal attention is constructed, fusion of multi-modal information is realized, the convolutional neural network is bridged with the Transformer, image information and text information are fused in the same vector space, and the accuracy of language description of the model is improved; a semantic gap problem existing in the field of image description is reduced. Compared with a baseline model using Bottom-up and LSTM, the invention has the advantages that the BLEU-1, the BLEU-2, the BLEU-3, the BLEU-4, the ROUGE-L and the CIDEr-D are all improved.

Description

technical field [0001] The invention relates to an image description method of a multi-modal Transformer based on dynamic word embedding, and belongs to the technical field of artificial intelligence. Background technique [0002] Multimodal deep learning aims to achieve the ability to process and understand multi-source modal information through deep learning methods. With the rapid rise of society and economy, multimodal deep learning has been widely used in various aspects of social production, and has achieved remarkable results. At present, the popular research direction is multi-modal learning among image, video, audio and semantics. For example: In speech recognition, humans understand speech by combining speech-visual information. Visual modalities provide information on pronunciation locations and muscle movements, which can help disambiguate similar voices, as well as judge the speaker's emotions through body behavior and voice, and so on. [0003] Using natural...

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a multimodal Transformer image description method based on dynamic word embedding. This method uses a multimodal deep learning model and uses a joint modeling method of inter-modal and intra-modal attention to input The data is modeled to generate the corresponding description, which solves the problem that the traditional method only uses inter-modal attention, which leads to the model’s insufficient understanding of semantics and poor effect. The specific steps are as follows:

Images