Intelligence type retrieval dialogue method based on pre-training and attention interaction network

Abstract

The invention discloses a knowledge-based retrieval dialogue method based on pre-training and an attention interaction network, and the method comprises the following steps: training a pre-training language model BERT on a target corpus through employing a domain adaptability pre-training method, and obtaining the domain adaptability BERT; using the field adaptability BERT as an encoder of the attention interaction network, and respectively encoding the dialogue context, the background knowledge and the plurality of candidate response texts to obtain corresponding representations; and finally, respectively inputting the dialogue context, the background knowledge and the representation of the plurality of candidate responses into the attention interaction network for matching, and training the attention interaction network to retrieve the optimal response from the plurality of candidate responses. According to the method, the powerful semantic characterization capability of the pre-training language model is utilized, the semantic characterization capability of the pre-training language model on a specific corpus is improved through two pre-training tasks, and the performance reduction caused by separation coding adopted for improving the retrieval speed is relieved by adopting the attention interaction network.

Description

technical field [0001] The invention relates to the fields of deep learning and natural language processing, in particular to a knowledge-based retrieval dialogue method based on pre-training and attention interaction network. Background technique [0002] Dialogue systems are an important topic in natural language processing, where the goal is to enable computers to understand human dialogue and build end-to-end dialogue devices. There are two kinds of mainstream dialogue systems at present, which are generative dialogue system and retrieval dialogue system. Generative dialogue systems understand dialogue and generate responses through an encoder-decoder structure; retrieval dialogue systems retrieve responses from a corpus. The knowledge-based dialogue response selection task is proposed by the role-based dialogue corpus (Persona-Chat) and the document-based dialogue corpus (CMUDoG), which requires the knowledge-based retrieval dialogue system to select from several candi...

AI Technical Summary

Problems solved by technology

This method is based on the pre-trained language model BERT, which can effectively use the powerful semantic capture ability learned by BERT pre-trained on a large general-purpose corpus, and by continuing to pre-train on the target corpus, use the domain that has learned the semantic representation of the target corpus Adaptive BERT as the encoder; the method also incorporates an interactive network based on cross-attention to alleviate the accuracy drop caused by separate encoding

Images