Visual context fused image description method

Abstract

The invention discloses an image description method fusing visual context. The method comprises the following steps: 1) preprocessing; 2) preprocessing of description labels of images; 3) feature extraction; 4) mean value pooling; 5) convolution and mean value sampling pooling; 6) acquisition of detection image entities; 7) acquisition of entity attributes; 8) convolution; 9) acquisition of entityattribute features; 10) convolution; 11) convolution; 12) convolution; 13) acquisition of relationships between the entities and the attributes; 14) matching of the relationships between entities andattributes; 15) LSTM training; 16) solving of exposure deviation; 17) dimension reduction; 18) normalization; 19) acquisition of a description statement, namely a model, of the current image; 20) acquisition of description sentences of all images; and 21) testing and verification of the training effect and performance of the model. With the method, the accuracy of image feature extraction can beguaranteed, visual errors are avoided, generated description is smoother so as to accord with grammatical rules of human beings, and higher scores are obtained for revaluation indexes.

Description

technical field [0001] The invention relates to the technical field of computer vision and the field of natural language processing, in particular to an image description method integrating visual context in a deep neural network and a reinforcement learning method. Background technique [0002] Image description can be understood as giving a picture and generating a text described in natural language. Image description and visual question answering belong to the intersection of computer vision and natural language processing, and are more effective than target detection, image classification and semantic segmentation. It is challenging because it extracts image entities and attributes while inferring the relationship between entities and attributes. Image description has broad application prospects in blind navigation, early childhood education, and image-text retrieval. [0003] Image description needs to use encoding network and decoding network. The proposal of residual ...

AI Technical Summary

Problems solved by technology

Therefore, sentences that have not appeared will seriously affect the performance of the test during the test

Images