Distributed depth learning system based on momentum and pruning

Abstract

The invention provides a distributed depth learning system based on momentum and pruning, which relates to the field of cloud computing and depth learning. A spark distributed cluster is adopted, comprising a main control node and a plurality of working nodes. Any working node is connected with the main control node through a communication link, and the working node stores part of training data. According to the batch of training data stored locally, the work node propagates the training of the depth learning model forward and backward, obtains the update quantity of the weight parameter of the work node, and sends it to the main control node for interactive communication with the global weight parameter of the main control node. The main control node records the node information of the work node, balances the weight parameters of the obtained work node and transmits the obtained weight parameters to each work node. Repeating the interactive communication between the master control node and the working node until the iterative number or convergence condition is reached; The invention solves the problem that the model of the asynchronous algorithm converges slowly under the distributed environment, and at the same time, the operation speed of the asynchronous algorithm is improved.

Description

technical field [0001] The invention relates to the field of cloud computing and deep learning, in particular to a distributed deep learning system based on momentum and pruning. Background technique [0002] In the field of machine learning problems, deep learning has made remarkable progress. Relying on massive training data, deep learning methods can train more accurate models. However, the main disadvantage of the deep model is that there are many model parameters, and the calculation amount increases exponentially with the increase of the number of layers, so a distributed system is needed to manage such a huge calculation task and data. ApacheSpark is one of the mainstream distributed systems at present. It has high-performance memory data processing speed and fault tolerance. Therefore, the combination of deep learning and Spark framework has attracted extensive attention. Relevant researchers have proposed SparkNet, CaffeOnSpark, DeepSpark and Frameworks such as Te...

AI Technical Summary

Problems solved by technology

In 2012, some scholars proposed a distributed stochastic gradient descent algorithm. However, the batch synchronization implementation of stochastic optimization is usually slow in practice because it needs to wait for the slowest machine in each synchronization batch.

Based on the traditional distributed deep learning system, the weight dimension of the deep learning model is large, there are many cluster nodes, and the interactive communication load is relatively serious

Images