Method for constructing deep visual Q&A system for visually impaired persons

Abstract

The present invention discloses a method for constructing a deep visual Q&A system for visually impaired persons. In the training phase, the method comprises: taking collected pictures and a corresponding Q&A text to constitute a training set; extracting picture features for the pictures by using the convolutional neural network; for a question text, converting questions into a word vector list by using the word vector technique, and taking the word vector list as input of the LSTM so as to extract question features; and finally, after carrying out element dot product on the pictures and the question features, carrying out classification on the pictures and the question features so as to obtain an answer prediction value, comparing the answer prediction value with an answer tag in the training set, calculating the loss, and using the back propagation algorithm to optimize the model. In the running phase, the method comprises that: a client obtains photos taken by the user and the question text, and uploads the photos and the question text to a server; the server inputs the uploaded photos and question text into a trained model, extracts question features by using the same manner, outputs a corresponding answer prediction value by using a classifier, and returns the answer prediction value to the client; and the client returns the answer prediction value to the user in a form of voice input.

Description

technical field [0001] The present invention relates to the field of visual question answering (Visual Question Answering), a cross field involving both Natural Language Processing (NLP) and Computer Vision (CV), in particular to a deep How to build a visual speech system. Background technique [0002] The visually impaired population accounts for a huge proportion of the world. In addition, according to the National Bureau of Statistics, in 2014 there were about 6-7 million blind people in China, and another 12 million patients with low vision in both eyes. These people will encounter many problems in their daily life. Take the blind people traveling as an example. Although there are blind roads, many blind people dare not walk on the blind roads at all, because the blind roads are often occupied, not to mention the danger of talking on the road. . So they really need some auxiliary equipment to help them "restore the light". The current voice assistants generally only ...

AI Technical Summary

Problems solved by technology

There is also a product called Third Eye, which is mainly able to take pictures to identify objects, and then voice input, which obviously skips the stage of interacting with users, and has very large limitations

Images