Visual question and answer method based on multi-modal depth feature fusion and model thereof

Abstract

The invention discloses a visual question-answering method based on multi-modal deep feature fusion, and the method comprises the following steps: (1), obtaining two modal data features of an image and a text through employing a convolutional neural network and a long-short term memory network, then, attention modeling in the modals and between the modals is carried out by utilizing two obtained modality data characteristics of the image and the text; (2) constructing an attention network and stacking attention layers in series, wherein two modal features serve as reference of attention weight learning to perform deeper feature interaction; and (3) fusing the image information and the text semantics after attention weighting through a multi-modal fusion function, and transmitting fusion features into a classifier to combine with answer text data to predict a result. In addition, the invention also discloses a visual question and answer model based on multi-modal depth feature fusion. Compared with an existing method, the method has the advantages of being good in stability, higher in prediction accuracy, lower in experimental hardware environment requirement and the like.

Description

technical field [0001] The present invention relates to the field of visual question answering related to multimodal data fusion research, in particular to a visual question answering method and its model based on multimodal deep feature fusion, Background technique [0002] Visual question answering refers to: Given a picture and a question related to the picture, the goal of visual question answering is to combine the visual information and text content of the picture, and obtain the answer to the question by performing deep feature fusion processing on the image and text. [0003] The cross-modal interaction method adopted by the early research of visual question answering is based on simple feature combination. For example, the problem features represented by the bag-of-words model are directly concatenated and integrated with the convolutional features of the image, and input into the logistic regression classifier; another example is the combination of graphic and text...

AI Technical Summary

Problems solved by technology

[0006] In order to solve the problems that the current visual question answering model is prone to loss of key feature information and cross-modal features cannot interact closely, the present invention discloses a visual question answering method based on multi-modal deep feature fusion

Images