A method and system for semi-supervised question-answer induction based on deep generative models

Abstract

The invention discloses a semi-supervised question and answer pair induction method and system based on a deep generation model. The method comprises the following steps: sorting out candidate question and answer pairs from input dialogue data through a candidate question and answer pair generation method; scoring the candidate question and answer pairs by adopting a question and answer pair evaluation method based on a depth generation model; obtaining a high-quality question and answer pair by utilizing a question and answer pair screening method according to the candidate question and answer pair scoring result; and pre-training the deep generation model through a semi-supervised learning mode, and then applying the pre-trained deep generation model to the question and answer pair evaluation method. Through the mode, automatic question and answer pair induction can be carried out, manual participation is greatly reduced, and high-quality question and answer pairs are obtained.

Description

technical field [0001] The invention relates to a natural language processing technology, in particular to an automatic question-answer induction method and system. Background technique [0002] Chatbot (Chatbot) is one of the hot research directions in the field of artificial intelligence in recent years, and has received continuous attention from academia and industry. Information Retrieval Based Question Answering (IRQA) is the most commonly used question answering method for chat robots. As one of the methods, question-answer pairs are the most commonly used retrieval objects and reply sources in IRQA as an important data form. At the same time, question-answer pairs are also important supervised data in tasks such as sequence generation and end-to-end question answering based on deep learning. [0003] Question-answer pairs need to be generated by induction, and the most commonly used source of question-answer pair induction is session data. The present invention defin...

AI Technical Summary

Problems solved by technology

The quality of the question-answer pair is good enough to achieve good results in tasks such as information retrieval-based question answering, sequence generation, and deep learning-based end-to-end question answering. However, question-answer pair induction from dialogue data is a very challenging task. Task:

[0004] 1. There are one-to-many, many-to-one, and many-to-many situations between user questions and corresponding answers in the dialogue data, and the alignment is difficult;

[0005] 2. There are a lot of nonsense with low information content in the dialogue data, such as "okay", "um", etc., which makes it more difficult to summarize high-quality question and answer pairs;

[0006] 3. When summarizing domain-related question and answer pairs, there are also some unimportant dialogues that are irrelevant to the field (for example, chatting dialogues in e-commerce scenarios) in the question and answer pairs, which makes it more difficult to summarize high-quality question and answer pairs

[0007] However, the existing research results at this stage cannot completely solve the many challenging tasks brought about by question answering to induction, and there are still many defects, which are summarized as follows:

[0008] 1. Manual sorting mainly relies on manual reading of dialogue data, from which high-quality question-answer pairs are sorted out. This method requires a huge labor cost, time-consuming and labor-intensive;

[0009] 2. Use an unsupervised method based on high-frequency information to sort out question-answer pairs. This method cannot handle low-information content, such as "okay", "uh-huh" and other nonsense, nor can it handle some domain-independent dialogues , at the same time, this approach stays at the literal level, does not use semantic information, and cannot handle situations where the semantics are equivalent but the literals are different, so this approach does not work well

Images