Quantification method of convolutional neural network based on knowledge distillation

Abstract

The invention provides a quantification method of a convolutional neural network based on knowledge distillation. The invention relates to the technical field of convolutional neural networks. The optimal cutting proportion and quantization digit of each network weight layer of the convolutional neural network is obtained by using a dynamic search mode; the trained convolutional neural network issequentially subjected to sparsification, knowledge distillation fine adjustment training by adopting a teacher network, layer-by-layer quantification and knowledge distillation fine adjustment training by using the teacher network again, so that the storage requirement and the calculation requirement of the convolutional neural network are greatly compressed on the premise of keeping the networkprecision. In the quantitative compression process of the convolutional neural network, the teacher network is used for knowledge distillation-based fine adjustment training, different quantization precisions can be adopted for different network weight layers of the convolutional neural network, and the network precision cannot be excessively lost, so that the compression degree of the convolutional neural network can be increased in the quantization process.

Description

technical field [0001] The invention relates to the technical field of convolutional neural networks, in particular to a quantification method of convolutional neural networks based on knowledge distillation. Background technique [0002] At present, a large number of visual application scenarios at home and abroad use convolutional neural network (Convolutional Neural Network) for feature extraction, including online image classification, recognition, detection services, face recognition, re-identification, security monitoring and other application scenarios. Compared with the traditional scheme, the solution based on convolutional neural network has the advantages of high precision and strong generalization ability, but it requires a large amount of calculation, high hardware requirements, and high storage requirements, which makes the popularization and use of the scheme subject to limit. The general convolutional neural network uses floating-point numbers for related op...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to provide a quantization method of convolutional neural network based on knowledge distillation, which solves the binarization problem of depth separable convolution, and then designs a more suitable model optimization for binarized convolutional neural network method, and applied to the ResNet series network, compared with the optimization method in the prior art, it can greatly improve the network accuracy of the convolutional neural network based on the ResNet series after binarization, and it is actually used for visual scenes of classification

Images