INT8 offline quantization and integer inference method based on Transform model

Abstract

The invention provides an INT8 offline quantization and integer inference method based on a Transform model. The INT8 offline quantization and integer inference method based on the Transform model comprises the following steps: converting an L2 norm of a normalization layer in an original Transform floating point model into an L1 norm; carrying out model training; performing forward inference through a small amount of data to obtain a quantization coefficient of input data of each layer of matrix operation, and extracting the quantization coefficient as general floating point data; obtaining a weight quantization coefficient of each linear layer in the floating point model, extracting the weight quantization coefficient as general floating point data, and determining an optimal weight quantization coefficient in each layer according to a mean square error calculation method; converting quantization coefficients related to quantization operation in the inference process into 2-n floating-point number forms, and adjusting the quantization coefficients through a joint coefficient adjustment method; and obtaining an integer inference model of INT8 based on the adjusted quantization coefficient in combination with a normalization layer of an L1 norm. According to the invention, errors caused by hardware resources required by model calculation and model quantification can be reduced, hardware resource consumption is reduced, and the deduction speed of the model is increased.

Description

technical field [0001] The present invention relates to the technical field of natural language processing, and more specifically, relates to an INT8 (8-bit integer) off-line quantization method and an integer inference method of a Transformer model-based natural language processing neural network model. Background technique [0002] With the emergence of deep learning algorithms, artificial intelligence has ushered in the third outbreak, and the increase in the number of parameters and computational complexity of deep learning algorithms puts forward higher performance requirements for hardware. Designing dedicated hardware accelerators in the field of deep learning is effective solution to this need. How to reduce the delay and storage of deep neural network calculations is an important research direction for the implementation of neural network algorithms and the design of neural network accelerators. [0003] Model quantization is an ideal technical approach to solve th...

AI Technical Summary

Problems solved by technology

[0004] As a new general-purpose model in the field of natural language processing, the Transformer model surpasses traditional neural networks such as LSTM in all aspects. The price paid is the multiplication of model complexity and network parameters, resulting in a sharp increase in computing power and power consumption. , making them difficult to run in edge devices

Directly using the INT8 offline quantization method of the existing convolutional neural network to the Transformer model will result in loss of accuracy

Images