Image paragraph description method based on relation coding and hierarchical attention mechanism

Abstract

The invention discloses an image paragraph description method based on relation coding and a hierarchical attention mechanism. A method model is composed of a relation coding module and a hierarchical attention decoding module. The relation encoding module captures and encodes space relation information and semantic relation information through two encoders, wherein the prior knowledge of the semantic relation is learned by training a supervised semantic classifier when the semantic relation is encoded. The hierarchical attention of the hierarchical attention decoding module uses hierarchical attention with a relational gate and a visual gate to dynamically fuse relation information and object region features, the relational gate is used for switching between spatial relation information and semantic relation information, and the visual gate is used for determining whether to embed and use visual information; the model fuses visual information in a paragraph generation process by adopting a strategy of a spatial and semantic relationship from a coarse-grained region to a fine-grained region. A large number of experiments on a Steiner paragraph description data set show that the method is obviously superior to an existing method in multiple evaluation indexes in the field.

Description

technical field [0001] The invention relates to the technical field of image processing, in particular to an image paragraph description method based on relational coding and hierarchical attention mechanism. Background technique [0002] Image description is the task of automatically generating a descriptive sentence for a given image, also called image single-sentence description. This fundamental cross-modal task may have diverse applications such as image / video retrieval, early childhood education, and helping the visually impaired understand image content. Therefore, this task has attracted great attention in the artificial intelligence community. [0003] Over the past few years, many studies have made impressive progress on the task of generating one-sentence image descriptions. However, due to the limitation of describing an image in one sentence, it is usually not enough to summarize various details in an image in one sentence, because "a picture is worth a thousa...

AI Technical Summary

Problems solved by technology

However, there is a serious problem with this simple fusion method

Images