Image description generation method based on text hierarchical structure

Abstract

The invention discloses an image description generation method based on a text hierarchical structure. According to the invention, a double-layer LSTM decoder is constructed, a visual and language information selection mechanism is introduced, effective selection is performed between image features and language information by utilizing image global features, word embedding and an attention guiding mechanism, and semantic information is generated through decoding to describe sentences more accurately. Aiming at the defect that a traditional language model is insufficient in semantic feature extraction capability, an ordered long-short term memory network improved by an FARIMA filter is introduced in a decoding stage. Semantic information of different text hierarchies is reserved through hierarchical structures of coded sentences, and semantic alignment is performed on the content by utilizing image space information. The cross-modal representation capability of a decoder in alignment of image features and semantic features is improved, and the long-time dependence of a network is increased. The method extracted according to the invention is richer in semantic relationship and more in line with natural language habits.

Description

technical field [0001] The invention relates to the fields of computer vision and natural language processing, in particular to an image description generation method based on a text hierarchy. Background technique [0002] Image description generation is known as one of the most important comprehensive topics in the intersection of computer vision and natural language processing. The main task of image description is to input an image to the machine, and then enable the machine to recognize and understand the objects, object attributes and their relationships in the image, and automatically generate a descriptive text with correct semantics and grammar in human natural language . [0003] Image description generation not only requires the model to recognize the target in the image, but also needs to recognize other visual elements, such as the actions and attributes of the target, understand the relationship between the targets, and generate human-readable description sent...

AI Technical Summary

Problems solved by technology

[0006] However, such methods do not distinguish non-visual words, the gradient from non-visual words may be misleading and reduce the overall effectiveness of visual signals in guiding the generation of text descriptions, the semantic information is not fully utilized, and there is semantic feature extraction ability Insufficient defects make the generated description not vivid enough to conform to natural language habits

And the image description generation model under the existing attention mechanism usually only pays attention to the local information of the image, and does not pay attention to the global information of the image.

Since it only describes the relationship between local objects in the image, it lacks a description of the relationship between the object and the scene, which will make it difficult for the sentences generated by the model to contain the overall information of the image.

The existing model generates sentences described by images, whose semantics are relatively one-sided and weak in integrity, and have a certain degree of deviation from visual intuitive cognition

Images