Bidirectional coding target encoder construction method and device

Abstract

The invention relates to a bidirectional coding target encoder construction method and device. The method comprises the steps of obtaining a training text; encoding the training text into a first sequence according to the encoding relationship stored in the dictionary, wherein the encoding sequence of the first sequence is consistent with the text sequence of the training text; covering each element in the first sequence in sequence according to a character sequence of the training text to obtain a plurality of second sequences; rearranging the elements in each second sequence according to a sequence opposite to the current arrangement sequence to obtain a plurality of third sequences; and inputting the second sequence and the third sequence into a self-encoding language model, and outputting the model as a target encoder. When the target encoder is constructed, feature extraction training is performed by adopting forward encoding input and reverse encoding input, so that the feature representation capability of the encoder is improved through forward and reverse bidirectional encoding in a real spatial sense, and the technical problem that the translation accuracy at the end of asentence is continuously reduced is solved.

Description

technical field [0001] The present application relates to the technical field of pre-training models, and in particular to a method and device for constructing a bidirectional encoding target encoder. Background technique [0002] The language model is widely used in various natural language processing tasks, and its essence is equivalent to an encoder, which can effectively extract important feature information from the original text. In recent years, the research on language models is based on the connection of the frequently used Word2Vec to the neural network, and then the use of the bidirectional LSTM network and the Text-CNN network. The feature extractor with the best performance in recent years is the Transformer neural network. Therefore, transfer learning based on Transformer's pre-training-fine-tuning architecture has also become a trend, and its core is the pre-trained language model implemented by Transformer. This transfer learning method makes it more convenie...

AI Technical Summary

Problems solved by technology

Although such a transfer learning architecture based on self-encoded feature extraction acquires contextual features at the same time, its essence is one-way text input. Features are extracted from the context at the same time to reflect the "two-way" process. Therefore, when Transformer performs machine translation, along with As the text length increases, when translating from left to right, the translation accuracy at the end of the sentence continues to decrease

[0004] For the above problems, no effective solution has been proposed

Images