Multi-hop visual problem reasoning model and reasoning method thereof

Abstract

The invention discloses a multi-hop visual problem reasoning model and a reasoning method thereof, and the model comprises a multi-hop visual problem reasoning data set establishment unit which is used for fusing a scene graph and a knowledge base into a knowledge graph, and constructing a data set containing multi-hop knowledge reasoning question and answer pairs by using the knowledge graph; a convolutional neural network used for extracting image features of the input image; a long-short-term memory network used for extracting problem characteristics; and a knowledge routing modular networkused for analyzing the questions into query trees, the query trees being symbolic expressions of reasoning processes of the questions, extracting correct relationships or entities from the knowledgegraph by combining the query trees and the knowledge base, and carrying out multi-hop reasoning to give a final answer.

Description

technical field [0001] The invention relates to the fields of visual question answering, natural language processing, image recognition and deep learning, in particular to a multi-hop visual question reasoning model and reasoning method. Background technique [0002] The current visual question answering questions often only require the answer to the question, and the ultimate goal of visual question answering requires us to get a question that can understand any reasoning complexity (such as single-hop reasoning or multi-hop reasoning), while giving interpretable diagnosis. As a result, to improve the credibility of the model. An ideal model should be able to understand the internal entity relationship in the question when answering "What is the use of this girl's hand?" relationship or attribute to give an answer. [0003] Existing natural image scene visual question answering datasets usually contain relatively simple questions and only evaluate the accuracy of the fina...

AI Technical Summary

Problems solved by technology

There have been some recent works addressing this problem by balancing question-answer pairs. CLEVR proposes a synthetic dataset in which images and question-answer pairs are generated based on a given compositional layout. However, conventional neural networks that fuse between image feature maps and question encodings Still correctly answering these complex and combinatorial questions, it's still hard to diagnose what the neural network learned and how to get the correct answer

Images