Memory normalization network layer method based on double forward propagation algorithm

Abstract

The invention discloses a memory normalization network layer method based on a double forward propagation algorithm, so as to improve the training stability and network performance of a neural network. The method comprises the steps that statistical information of each patch is calculated, the patches refer to a set of data samples, and the statistical information comprises the mean value and variance of data; estimating an overall mean value and an overall variance of a plurality of continuous patches in the storage memory according to the statistical magnitude of each patch, and performing normalization operation on each generation of training data in the training process by using the statistical magnitude; after standard forward propagation and back propagation are executed once in eachiteration, forward propagation is additionally executed once to reduce distribution offset of network interlayer information caused by model updating. According to the method, more accurate and morestable statistical information can be obtained, consistent behaviors are always kept in training and testing, and finally, the generalization ability of the deep network can be greatly improved.

Description

technical field [0001] The invention relates to the technical field of deep learning and algorithm optimization, in particular to the design of network layers and the research on training algorithms of neural networks. Background technique [0002] Batch Normalization (BN) has become a standard component in deep networks. The most critical technology is that this method normalizes the data according to the statistics of the current batch data, so that it is distributed in a standard distribution, thus effectively solving the problem of distribution deviation in the training process. By introducing BN into the network structure, the training process of the model can converge faster and obtain better generalization ability. [0003] Although BN plays a vital role in the training of deep networks, it still has some limitations. On the one hand, during the training process, BN estimates the mean and variance of the overall distribution of samples based on the data of a single ...

AI Technical Summary

Problems solved by technology

In other words, the training process of BN is inconsistent with the testing process, resulting in model deviation

In particular, each stochastic gradient descent (SGD) during the training process will cause a change in the model parameters, which in turn will lead to deviations in the feature distribution before and after the model is updated.

Images