Language Input Relevance Detection Method Based on Attention Model

Abstract

The invention discloses a language input association detection method based on attention model. The method includes the following steps: acquiring training linguistic data which is used for training a language model and pre-processing the linguistic data, labeling each word sequence data in the linguistic data; then training a recurrent neural network in a language model by using the labeled training sequence, training the updated language model by using all data sets in the training linguistic data, if the probability distribution of prediction words that are obtained is in convergence in a validation set, completing the training of the language model; and eventually scoring the input sentences by using the trained language model so as to obtain the relationship between the words. According to the invention, the method herein automatically extracts prediction words and relationships among the prediction words based on attention. The method herein can also introduce grammar and semantic information in the process of training word vectors, enables word vector to contain more information, and further increases the performances of the language model through technology expectation.

Description

technical field [0001] The invention relates to a technology in the field of voice input, in particular to a method for detecting language input relevance based on an attention model. Background technique [0002] In recent years, research on recurrent neural networks has become more and more popular. Based on the long short-term memory neural network (LSTM) of the gated memory unit, the gated recurrent unit neural network (GRU) is widely used in the field of natural language processing. The LSTM neural network adds memory units, input gates, output gates, and forget gates, and the GRU neural network adds reset gates and update gates. These gates and memory units greatly improve the modeling of long-distance dependencies between words. Effect. [0003] However, such models still have some limitations. The modeling of historical information in the gate-based neural network is encoded in the hidden layer. When the hidden layer needs to contain more information, the number o...

AI Technical Summary

Problems solved by technology

[0004] The present invention proposes a language based on the attention model in view of the defects that the existing technology cannot explicitly provide the correlation between words, adding hidden nodes will linearly increase the parameter amount of the entire model, and cannot fully utilize all historical information. Input correlation detection method, introduce additional control unit in the model, explicitly input historical sequence and additional information, use an attention-based method to automatically extract the correlation between predicted words and them

Images