Relation visual attention mechanism-based scene graph generation method

Abstract

The invention discloses a relation visual attention mechanism-based scene graph generation method and mainly aims to solve the problem that redundant relation prediction and interpretability are poorin the prior art. According to the embodiments of the invention, the method includes the following steps of: 1) obtaining the category and boundary frame of targets in images through target detection,and establishing a full-connection relation graph; 2) carrying out sparsification on the relational graph through analyzing a data set to obtain sparse relational graph representations; (3) learninga relation attention transfer function through alternate iteration, transferring subjects and objects to relation occurrence sites through union set features, and learning accurate relation representations; and 4) classifying the learned relation representations, and combining the relation representations into a final scene graph. According to the method, on the basis of the internal relationshipof the relation occurrence of two targets, a relation attention mechanism is established to accurately pay attention to a region where relations occur; a scene graph is accurately generated; the interpretability of a network is improved; and the method can be used for image description and visual question and answer tasks.

Description

technical field [0001] The invention belongs to the field of computer vision, and in particular relates to a method for generating a scene graph, which can be used for image description and visual question answering tasks. Background technique [0002] With the development of deep learning, the computer's understanding of images has reached a new level. From object detection to semantic segmentation to instance detection, computer vision has made great progress. But there are still many problems for deeper image understanding. Since the relationship between objects in the image does not exist independently, tasks such as object detection cannot grasp the relationship between instances, such as a person carrying a bag and a person holding a bag. Although the categories detected by the object may be the same, the relationship between them The relationship classes are different. In order to make computers understand images further like humans, Johnsn et al. proposed the scen...

AI Technical Summary

Problems solved by technology

[0007] Since neither of the above two methods has established a suitable attention mechanism, the features used by the network for the final classification relationship do not really focus on the area where the two targets are related, resulting in poor interpretability of the network.

Images