Language model compression method based on uncertainty estimation knowledge distillation

Abstract

The invention discloses a language model compression method for estimating knowledge distillation based on uncertainty, and mainly solves the problems of high training cost, low speed and noise interference in the knowledge distillation process in the existing network compression technology. According to the implementation scheme, the method comprises the steps of 1) performing half-and-half compression on an original language model to obtain a compressed neural network; 2) reasonably initializing parameters of the compressed neural network by using an original language model; 3) adding a parameter distillation loss function of a feedforward network structure, and designing an uncertainty estimation loss function and a cross entropy loss function of a natural language processing task; and 4) training the compressed neural network model by using the designed loss function. According to the method, the calculation amount in the network compression training process is reduced, the network compression rate is improved, the network reasoning speed is increased, the method can be widely applied to model deployment and model compression tasks, and a new model compression solution is provided for an application scene in shortage of hardware resources.

Description

technical field [0001] The invention belongs to the field of neural network compression, and in particular relates to a compression method for a pre-trained language model, which can be used for model deployment, model compression, and alleviation of model hardware burden. Background technique [0002] In recent years, the natural language processing research community has witnessed a revolution in pre-training and self-supervised models, with the research and application of large-scale pre-trained language models, first pre-trained on large-scale text data, and then on downstream tasks. Transfer learning, pre-training and fine-tuning have gradually become the basic paradigm for natural language processing solutions. The emergence of BERT has significantly improved the performance of many natural language processing tasks. However, pre-trained language models are often computationally expensive and memory intensive. These models typically have hundreds of millions of parame...

AI Technical Summary

Problems solved by technology

The scheme is verified on the Chinese named entity recognition NER task. Although the calculation cost of the compression method is not large, the accuracy of the compressed network model is reduced by 1-2 percentage points compared with the original network, and the network compression rate is low. , which reduces the efficiency of model operation and leads to a waste of a large amount of computing resources

[0006] The disadvantages of the above-mentioned existing network lightweight methods are: 1) lack of supervision on the intermediate reasoning process of the network, 2) insufficient utilization of the original network parameters, 3) lack of noise estimation in the knowledge distillation process

In turn, the calculation burden of the network compression process is too large, the efficiency of the compression process is too low, and the performance accuracy of the compressed network is low.

Images