Neural network compression method and device, computer equipment and storage medium

Abstract

The invention relates to a neural network compression method and a device, computer equipment and a storage medium, and relates to the technical field of neural networks. The method comprises the following steps: respectively inputting training samples into a teacher network and a student network; obtaining first network data including a first model parameter of an ith layer and a first feature map in a teacher network, and performing cross calculation on the first network data and the second network data to obtain a loss function value, and updating the second model parameter of the ith layerin the student network according to the loss function value, wherein the second network data comprises the second model parameter of the ith layer and the second feature graph. According to the method, the accuracy of the compressed neural network can be improved under the scene that the trained neural network is compressed through a small amount of training data.

Description

technical field [0001] The embodiments of the present application relate to the technical field of neural networks, and in particular to a neural network compression method, device, computer equipment, and storage medium. Background technique [0002] In recent years, as data privacy issues in neural network compression have become increasingly prominent, the research on algorithms for compressing original models based on no training data or a small amount of training data has also attracted more and more attention from the industry. [0003] In related technologies, the compression algorithm of the neural network model without training data or based on a small amount of training data is mainly realized through the framework of knowledge distillation. For example, the trained neural network model to be compressed is used as the teacher network, and another neural network with the same structure and smaller model size than the teacher network is set as the student network, an...

AI Technical Summary

Problems solved by technology

[0004] However, in the scheme shown in the above-mentioned related technologies, due to too little training data, the student network is prone to overfitting, which leads to a low accuracy rate of the student network obtained by model compression.

Images