Convolutional neural network INT8 quantification method, system and device and storage medium

Abstract

The invention provides a convolutional neural network INT8 quantification method, system and device and a storage medium, and the method comprises the steps: obtaining a parameter quantification coefficient of a convolution layer, carrying out the parameter nonlinear mapping of a parameter of the convolution layer, and obtaining a quantified parameter; obtaining an input quantization coefficient of the convolution layer, and establishing an input quantization function of the convolution layer, the input quantization function being used for performing nonlinear mapping on input data of the convolution layer to obtain quantized input data; and obtaining an output quantization coefficient of the convolution layer, and establishing an output quantization function of the convolution layer according to the quantized parameter and the input quantization function, the output quantization function being used for performing nonlinear mapping on output data of the convolution layer to obtain quantized output data. By the adoption of the method and device, off-line nonlinear quantization is conducted on model parameters, input and output, pure integer operation of the whole model is achieved,and quantization precision is improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a convolutional neural network INT8 quantization method, system, device and storage medium. Background technique [0002] Today's neural network algorithms have great potential and amazing recognition rates in the visual field, and have huge applications in many fields. With the rise of mobile phones, the processing power of mobile phones has become stronger and stronger. The size of a few hundred megabytes is still insufficient. Most neural network model training is 32-bit floating-point numbers, and based on such models, in most cases, they cannot be well ported to mobile terminals, so model compression is particularly important. [0003] Although some convolutional neural network quantization methods have appeared in the prior art, such as the quantization methods of Google and Cambrian, these methods generally adopt linear quantization, and the expression abi...

AI Technical Summary

Problems solved by technology

[0003] Although some convolutional neural network quantization methods have appeared in the prior art, such as the quantization methods of Google and Cambrian, these methods generally adopt linear quantization, and the expression ability of smaller values ​​in parameters and input and output is not strong. prone to loss of accuracy

Moreover, there are often floating-point numbers in the model inference of the existing methods, which is not conducive to the underlying design of the hardware

In addition, the model correction in the existing scheme requires more pictures to determine the coefficients of the convolutional layer, and has higher requirements on the distribution of picture data. For example, the existing TensorRT quantization requires thousands of pictures for quantization.

Images