Method for long text retrieval in open domain question and answer task, and electronic equipment

Abstract

The invention discloses a method for long text retrieval in an open domain question and answer task, andelectronic equipment. The method comprises the following steps of: respectively encoding an opendomain document and a search request into a document dense vector and a request dense vector by using a pre-trained encoder, wherein the encoder adopts a historical search request, a positive sampleand a negative sample as sample data for training; calculating a similarity score of the search request and the open domain document according to the document dense vector and the request dense vector, and selecting the open domain document of which the similarity score meets the requirement as a candidate document; and selecting a target document corresponding to the search request from the candidate documents. According to the invention, the reusability is strong; words of the same anaphora are attached with approximate semantic expressions; the ambiguity problem brought to search by one word with multiple meanings is effectively improved; the model training effect is good; the method has relatively strong generalization capability on cross-domain documents; and the search effect, the search performance, the usability, the maintainability and the like are greatly improved, and the potential is improved.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a method and electronic equipment for long text retrieval in an open field question answering task. Background technique [0002] Open-domain question answering is an important task in the field of natural language processing. The process of open domain question answering can be simply described as: for a given factual question, first retrieve the document where the answer to the question is located from a large-scale multi-domain document library, and then extract or generate the answer from the document. Among them, the accuracy of document retrieval often determines the upper limit of the effect of the whole process. Therefore, document retrieval is the most important part of open domain question answering tasks. [0003] Currently, common methods in the document retrieval stage are based on sparse matrices, such as using TD-IDF or BM25. Specifically, suc...

AI Technical Summary

Problems solved by technology

[0004] This method has the following inevitable disadvantages in practical applications: (1) It requires complex manual feature engineering, which is cumbersome, time-consuming and error-prone

In addition, the code for manual feature engineering is only for a specific problem. When a new problem or new data set needs to be solved, the relevant code needs to be rewritten; (2) It is difficult to solve the problem of word ambiguity in the open field

For example, for the word "apple", if its contextual information is ignored, it is difficult for the system to identify whether it represents a fruit or a technology company; (3) lack of deep understanding of semantics

For example, for the words "Ministry of Industry and Information Technology" and "Ministry of Industry and Information Technology", the system cannot automatically find the correlation between them and needs to be normalized manually; (4) The space for effect optimization is limited

Due to the technical limitations of artificial feature engineering, when the retrieval effect reaches a certain level, it is difficult to continue to optimize; (5) Poor generalization

Due to the strong domain attributes of various index constructions in the system, when encountering search requests outside the text domain, the effect is often poor

Images