Multilayer neural network language model training method and device based on knowledge distillation

Abstract

The invention discloses a multilayer neural network language model training method and device based on knowledge distillation. The method comprises the steps that firstly, a BERT language model and amulti-layer BILSTM model are constructed to serve as a teacher model and a student model, the constructed BERT language model comprises six layers of transformers, and the multi-layer BILSTM model comprises three layers of BILSTM networks; then, after the text corpus set is preprocessed, the BERT language model is trained to obtain a trained teacher model; and the preprocessed text corpus set is input into a multilayer BILSTM model to train a student model based on a knowledge distillation technology, and different spatial representations are calculated through linear transformation when an embedding layer, a hiding layer and an output layer in a teacher model are learned. Based on the trained student model, the text can be subjected to vector conversion, and then a downstream network is trained to better classify the text. According to the method, the text pre-training efficiency and the accuracy of the text classification task can be effectively improved.

Description

technical field [0001] The invention relates to the fields of unsupervised text pre-training and deep learning, in particular to a multi-layer neural network language model training method and device based on knowledge distillation. Background technique [0002] With the rapid increase of online text information data on the Internet, the language model plays a vital role in information processing. It is a key technology for processing large-scale text information and promotes the development of information processing in the direction of automation. A language model is simply the probability distribution of a sequence of words. Building a reasonable pre-trained language model can solve many current text information problems, such as text classification, text similarity, reading comprehension, etc., and then can efficiently use a large amount of text corpus data on the Internet to better provide people with More convenient service. However, most of the current text pre-train...

AI Technical Summary

Problems solved by technology

However, most of the current text pre-trained language models have shortcomings such as large and complex network structures and slow training speeds.

Images