Device and method for executing batch normalization operation

Abstract

The invention discloses a device for executing batch normalization operation. The device comprises an instruction storage unit, a controller unit, a data access unit and an operation unit. By use of the device, batch normalization operation in a multilayer artificial neural network can be realized. According to the batch normalization operation, in the forward process, a mean value is subtracted from input, and the result is divided by the square root of the sum of a variance and a minimum definite value; and then the result is multiplied by a learning parameter alpha plus a learning parameter beta, and output of the layer is obtained. In a reverse training process, a mean value vector of a gradient vector is subtracted from the input gradient vector, a mean value of the product of the gradient vector and output obtained forwards is subtracted from the result, the result is multiplied by the output, the obtained difference value is divided by the square root of the sum of a forward variance and a minimum definite value, and an output gradient vector of the layer is obtained. Through the device, support to batch normalization forward and reverse operation in the artificial neural network is effectively improved.

Description

technical field [0001] The present invention relates to artificial neural network technology, in particular to a device and method for performing forward and reverse operations of batch normalization in artificial neural network. Background technique [0002] Multi-layer artificial neural network is widely used in the fields of pattern recognition, image processing, function approximation and optimization calculation. In recent years, multi-layer artificial neural network has been favored by academic circles due to its high recognition accuracy and good parallelism. And the industry is getting more and more attention, and the batch normalization operation in the multi-layer artificial neural network is more and more used in the multi-layer neural network because it can accelerate the training speed of the neural network and improve the recognition accuracy. [0003] One known method to support batch normalization operations is to use a general-purpose processor. The method ...

AI Technical Summary

Problems solved by technology

Since the GPU is a device specially used to perform graphics and image calculations and scientific calculations, it does not have special support for multi-layer artificial neural network batch normalization operations, and still requires a lot of front-end decoding work to perform multi-layer artificial neural network operations, which brings a lot of overhead

In addition, the GPU has only a small on-chip cache, and the model data of the multi-layer artificial neural network batchnormalization needs to be repeatedly moved from off-chip, and off-chip bandwidth has become the main performance bottleneck.

And the batch normalization operation has a large number of normalization operations such as summation. The parallel architecture of the GPU is not suitable for such a large number of normalization operations.

Images