Hypothetical semi-supervised learning based open domain question answering method

Abstract

The invention discloses a hypothetical semi-supervised learning based open domain question answering method. The method includes the following steps: (1) extracting articles related to a question froma corpus by using an information retrieval technology; (2) assuming that articles that come with a given question answering training set are unique positive labels and the articles extracted for thecorpus are negative labels; (3) constructing a deep learning model, learning features of the positive label by training an article scorer, and training a reader to select correct answers from the articles; (4) carrying out article relevance ranking, and sending the first n articles with high relevance to the scorer and re-labeled the articles according to the scores; (5) repeating the step 3 and step 4 until the model converges; and (6) finishing the model training and applying the open domain question answering. The method can greatly improve the quality of article extraction and the accuracyof answers in a conventional open domain question answering system without relying on additional manual annotation and external knowledge.

Description

technical field [0001] The invention relates to the field of natural language processing, in particular to an open domain question answering method based on hypothetical semi-supervised learning. Background technique [0002] In recent years, open-domain question answering problems have become very popular and difficult problems in natural language processing. In this task, given a corpus and a question, the algorithmic system returns an answer from the corpus. The biggest difference between it and machine reading comprehension is that it adds the process of finding articles from the corpus in addition to answering questions based on the article. Open domain question answering systems are widely used, because traditional search engines can only meet the needs of a small number of people and most of the returned answers are just web page links rather than a specific answer. A question answering system that can extract articles from a large corpus and give ideal answers can ...

AI Technical Summary

Problems solved by technology

However, such an algorithm uses a simple information retrieval system in the step of extracting documents and then directly sends the results to the reading comprehension step indiscriminately, so it is too simple and has no semantic analysis, which will lead to the loss of many synonymous sentences. chances of being matched to an article containing the answer

Images