Multi-level long text vector retrieval method and device and electronic equipment

Abstract

The invention discloses a multi-level long text vector retrieval method and device and electronic equipment. The method comprises the following steps: segmenting a long text in the open field into text segments; respectively encoding the text fragment and the search request into dense vectors by using a trained encoder; querying to obtain a target text fragment similar to the search request based on vector retrieval by using the text fragment and the dense vector of the search request, wherein the encoder is obtained by training a training data set comprising multi-level text fragments. By considering the multi-level correlation between the text fragments in the training data set and the search request, the obtained model can easily select a proper fragment from a plurality of related fragments, and the recall efficiency is remarkably improved.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a multi-level long text vector retrieval method, device and electronic equipment. Background technique [0002] Open-domain question answering is an important task in the field of natural language processing. It can be simply described as: Given a fact-type question, the system needs to retrieve the document where the answer to the question is located from a large-scale multi-domain document library, and then extract or generate the answer from it. For open domain question answering tasks, document retrieval is often the most important part, and the accuracy of document retrieval determines the upper limit of the overall effect of the system. [0003] At present, the commonly used methods for document retrieval in open domain question answering tasks are based on sparse matrix or dense vector retrieval. Among them, retrieval methods based on sparse matrices ...

AI Technical Summary

Problems solved by technology

[0004] The retrieval method based on sparse matrix has the following disadvantages: (1) Manual feature engineering is required, which is a cumbersome, time-consuming and error-prone process, and the code for manual feature engineering each time is for a specific problem. When a new problem, new data set, we need to rewrite the relevant code; (2) it is difficult to solve the problem of word ambiguity in the open field

However, the training method using positive and negative binary classification labels has the following problems: After a long document containing answers is divided into multiple text segments when training the model, some text segments contain documents, and some text segments contain documents. Although it does not contain an answer, it is semantically related to the search request

For example, what are the material requirements for the application of the 2021 Beijing Natural Science Foundation of China? "For such a request, the corresponding document only has a hint of "2021 Beijing Natural Science Foundation Project" at the beginning, but because the long text is cut into text fragments, the first paragraph is used as a negative example for training, which reduces the effect of the model

Images