Neural network optimization method for solving and lifting adjacent operator machine

Abstract

The invention discloses a neural network optimization method for solving and lifting an adjacent operator machine, and relates to the technical field of deep learning neural network optimization. In the training of a forward neural network, a block coordinate descent method is adopted to solve and improve an LPOM model of an adjacent operator, each sub-problem in the LPOM model has convergence, the weight and network activation of each layer of the neural network can be updated in parallel, and extra memory space is not occupied. By adopting the method of the invention, the parallelism, the applicability and the training effect of neural network training can be improved under the condition of using relatively less storage.

Description

technical field [0001] The present invention relates to the technical field of deep learning neural network optimization, in particular to a method for optimizing a neural network by solving a lifted proximal operator machine (LPOM). Background technique [0002] The feed-forward deep neural network is composed of hierarchical fully-connected layers, and there are no feedback connections. With recent developments in hardware and dataset size, feed-forward deep neural networks have become the standard for many tasks. For example, image recognition [16], speech recognition [12], natural language understanding [6] and as an important part of Go learning system [22]. [0003] In recent decades, the goal of optimizing feed-forward neural networks is usually a highly non-convex and nested function with respect to network weights. The main method for optimizing feed-forward neural networks is stochastic gradient descent (SGD) [21]. Its effectiveness is verified by its success in ...

AI Technical Summary

Problems solved by technology

It is very difficult to choose an appropriate learning efficiency when updating weights using gradient descent

Images