Scene graph generation method based on depth relation self-attention network

Abstract

The invention discloses a scene graph generation method based on a depth relation self-attention network. The method comprises the following steps: 1, data preprocessing and data set division; 2, feature extraction of an image by using a pre-trained target detection network; 3, construction of spatial features of a target; 4, construction of language features of the target; the method is used formodeling the RSAN network of the target context and the relation context at the same time, achieves a significant improvement effect on a task generated by a scene graph, and surpasses most mainstreammethods on the task. In addition, the RSAN network also has very important application value and huge potential in other cross-modal related fields such as image content questioning and answering andvisual relationship detection.

Description

technical field [0001] The present invention proposes a method for generating scene graphs (Scene Graph Generation) based on deep relational self-attention networks (Relational Self-Attention Networks). Background technique [0002] Scene graph generation is an emerging task in the field of multimedia, which aims to model contextual information of objects and relationships among objects, and generate corresponding scene graphs for images. Specifically, an image is input, and a scene graph abstracted from the image is generated after passing through the model. The scene graph contains node and edge information, and the nodes and edges represent the relationship between the target and the target respectively. For example, the content of the image is a person wearing a hat holding a barrel and feeding a horse, which may include "a person wearing glasses", "a person feeding a horse", "a person carrying a barrel", "a horse from a barrel" Semantic information such as "eating food...

AI Technical Summary

Problems solved by technology

[0009] (2) How to model the semantic information of the scene graph at a deeper level to better obtain rich context information: most of the methods are to separately model the target context information and the relationship context information, and there is no need to model the context information The network performs deep stacking processing. The problem that may arise in this method is that it cannot effectively model the target context information and the relationship context information, because the target context information and the relationship context information affect each other, so what we use here is Simultaneous Model target context information and relationship context information, and obtain richer context information by stacking network depth

Images