Training method and system of deep neural network based on critical damping momentum

Abstract

The invention discloses a training method and system of a deep neural network based on critical damping momentum. The method comprises the following steps: S1, starting a new round of iteration; S2, inputting a batch of new images, and calculating traces of a Hessel matrix of a loss function of the neural network, the Hessel matrix being a matrix formed by second derivatives of the loss function to each parameter of the neural network; S3, substituting the trace of the Hessel matrix into a critical damping solution of a second-order differential equation, and calculating to obtain a momentum coefficient of a neural network weight parameter; S4, updating parameters of the neural network in cooperation with a learning rate attenuation strategy; S5, judging whether all image batches are calculated or not, and if so, executing the step S6; if not, returning to the step S2; and S6, judging whether the iteration turns reach the maximum or not, if so, ending the training, and if not, returning to the step S1. According to the method, a stochastic gradient descent method containing momentum is improved, and an ideal training effect is achieved.

Description

technical field [0001] The invention relates to the technical fields of machine learning and numerical methods, in particular to a training method and system for a deep neural network based on critical damping momentum. Background technique [0002] Deep neural networks have demonstrated excellent performance in fields such as computer vision, natural language processing, and computer graphics. With the increasing complexity of the deep neural network in terms of network layers and topology, the number of parameters of the deep neural network is also increasing day by day. At present, the number of parameters of the latest neural network in the industry has reached 10. 7 ~10 8 Magnitude. Although the huge amount of parameters improves the scope and efficiency of the neural network, it also brings difficulties to the training and optimization of the network. The training of large networks often takes several days or more, and it also puts new demands on the performance of ...

AI Technical Summary

Problems solved by technology

However, the iteration speed of the SGD algorithm is relatively slow, and its learning rate is a hyperparameter that can only be set empirically

When the learning rate is set too small or too large, the iteration of the neural network parameters is prone to the problem of slow pace or violent oscillation.

Although the Adam algorithm can obtain faster speed than SGD under certain conditions, it is applicable to fewer network types than SGD, and may not converge in some special cases.

Images