Image description generation method fusing visual common sense and enhancing multilayer global features

Abstract

The invention relates to the technical field of computer vision, and particularly discloses an image description generation method fusing visual common sense and enhancing multi-layer global features, and the method comprises the steps of fusing visual common sense features extracted by a VCR-CNN and local features extracted by a Faster R-CNN, and obtaining fusion features; mining a visual semantic relationship between the objects by adopting an X linear attention mechanism to obtain high-level local features and multi-level global features; enhancing the multi-layer global features by adopting an AoA mechanism, and performing linear mapping to obtain fused global features; screening the fusion global features by using long and short term memory of visual selection, weighing related information and adaptively selecting for high-level local features by using an X linear attention mechanism, and finally, using a semantic decoding gated linear unit to generate an output word sequence. The problems that an image description generation model of local features is insufficient in visual semantic relation mining, and redundant information exists in multi-layer global features extracted by an attention mechanism are solved.

Description

technical field [0001] The invention relates to the technical field of computer vision, in particular to an image description generation method that integrates common sense of vision and enhances multi-layer global features. Background technique [0002] Image description generation is one of the advanced tasks in the field of computer vision, and its purpose is to enable the computer to automatically generate a natural language description of a given image. Compared with low-level and mid-level tasks such as image classification and target detection, it not only needs to recognize the salient objects and their attributes in the image, understand the relationship between them, but also express them in accurate and fluent natural language. A very challenging task. When humans acquire information, the visual system will actively focus on the target area of ​​interest and extract relevant important information. Inspired by the human visual system, attention mechanisms have be...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to provide an image description generation method that integrates visual common sense and enhances multi-layer global features. There is a technical problem of redundant information in the extracted multi-layer global features

Images