Mixed precision training of an artificial neural network

Abstract

The use of mixed precision values when training an artificial neural network (ANN) can increase performance while reducing cost. Certain portions and / or steps of an ANN may be selected to use higher or lower precision values when training. Additionally, or alternatively, early phases of training are accurate enough with lower levels of precision to quickly refine an ANN model, while higher levels of precision may be used to increase accuracy for later steps and epochs. Similarly, different gates of a long short-term memory (LSTM) may be supplied with values having different precisions.

Description

Background technique [0001] Artificial Neural Networks ("ANN" or "NN") is applied to many applications in artificial intelligence ("AI") and machine learning ("ml"), including image identification, speech recognition, search engine, and other suitable applications. Ann is usually trained in multiple "EPOCH). In each period, ANN training all training data in the training data in multiple steps. In each step, ANN first predicts an example of training data (in this article, as "sample"). This step is often referred to as "forward pass) (in this article, it can also be referred to as" forward training pass), but the steps may also include rear transmission. [0002] In order to perform prediction, the training data sample is fed to the first layer of ANN, which is often referred to as "input layer". Then, each layer of the ANN will then use a function to calculate a function on its input (Weights)) to generate inputs for the next layer. The output of the last layer of "Output Layer" i...

AI Technical Summary

Problems solved by technology

However, the use of quantization-precision floating-point formats can have certain negative effects on ANNs, such as but not limited to loss of accuracy

Images