Neural network model compression method based on sparse backward propagation training

Abstract

The invention discloses a sparse backward propagation compression method of a neural network model, belongs to the field of information technology, and relates to machine learning and deep learning technologies. In the process of backward propagation, each layer of the neural network model uses the output gradient of the previous layer as the input to calculate the gradient, and performs k large-value sparse processing to obtain the sparsely processed vector and the number of sparse return times, and record k The index corresponding to the value; use the sparse gradient to update the parameters of the neural network; according to the k-value subscript index, delete the neuron with a small number of return times, and compress the model. The present invention adopts a sparse method based on a large value of k in the backward propagation process, eliminates inactive neurons, compresses the size of the model, improves the training and reasoning speed of the deep neural network, and maintains good precision.

Description

technical field [0001] The invention belongs to the field of information technology, relates to machine learning and deep learning technology, and in particular to a compression algorithm based on a large k-value sparse (Sparsify) neural network model. Background technique [0002] The training process of the neural network model is divided into two steps: forward propagation (Forward Propagation) and backward propagation (Backward Propagation). The forward propagation calculates the output of each layer of the neural network, and the backward propagation calculates the gradient of the model parameters according to the gradient of the output result of the forward propagation, and finally the neural network model uses these gradients to update the parameters. However, in practical applications, the deployed model only performs inference operations, that is, forward propagation operations. [0003] In existing neural network model training methods, the size, or dimension, of ...

AI Technical Summary

Problems solved by technology

The disadvantage of the method is that the model is often too large. On the one hand, the training and reasoning of the neural network takes a long time, and the amount of calculation and power consumption is large; The accuracy of the network model is low, resulting in overfitting (Overfitting) phenomenon

In view of the problems existing in the above methods, some model compression algorithms have been proposed successively, such as the weight clipping algorithm for feedforward neural network, the weight progressive clipping algorithm for recurrent neural network, and the parameter quantization method for convolutional neural network, etc.; however, , because for any layer in the deep neural network model, the above algorithm needs to specify hyperparameters separately to control the last dimension of the layer, therefore, these existing methods are not flexible in practical applications, and it is difficult to apply to the deep neural network. train

Images