Cluster packet synchronization optimization method and system for distributed deep neutral network

Abstract

The invention discloses a cluster packet synchronization optimization method and system for a distributed deep neutral network. The method comprises the following steps of grouping nodes in a cluster according to the performance, allocating training data according to the node performance, utilizing a synchronous parallel mechanism in the same group, using an asynchronous parallel mechanism between the different groups and using different learning rates between the different groups. The nodes with the similar performance are divided into one group, so that the synchronization overhead can be reduced; more training data can be allocated to the nodes with the good performance, so that the resource utilization rate can be improved; the synchronous parallel mechanism is used in the groups with the small synchronization overhead, so that an advantage of good convergence effect of the synchronous parallel mechanism can be exerted; the asynchronous parallel mechanism is used between the groups with the large synchronization overhead, so that the synchronization overhead can be avoided; the different groups uses the different learning rates to facilitate model convergence. According to the method and the system, a packet synchronization method is used for a parameter synchronization process of the distributed deep neutral network in a heterogeneous cluster, so that the model convergence rate is greatly increased.

Description

technical field [0001] The invention belongs to the technical field of distributed optimization of deep neural networks, and more particularly relates to a method and system for grouping synchronization optimization of distributed deep neural network clusters. Background technique [0002] At present, Deep Neural Network (DNN) has been applied in many fields such as image, speech, and natural language processing, and has made many breakthroughs. Due to the large scale of its training data and trained model parameters, deep neural networks require sufficient computing resources and storage resources. Therefore, the traditional single-machine node training mode can no longer meet the requirements, and distributed computing modes such as clusters must be used. [0003] Distributed Deep Learning usually adopts data parallel mode for model training. Such as figure 1 As shown, data parallelism refers to splitting the training data, storing one or more split training data on eac...

AI Technical Summary

Problems solved by technology

However, the synchronization overhead between nodes is relatively high. While a node is waiting for other nodes to complete the current round of iterations, its own computing resources and network resources are idle. In heterogeneous clusters and large-scale homogeneous clusters, this The phenomenon is particularly serious

In a heterogeneous cluster, due to the large difference in the hardware configuration of the nodes, there are obvious performance differences among the nodes. Some nodes run fast, while others run slowly. Therefore, in each round of iteration, the fast All nodes need to wait for the slow nodes, resulting in idle resources of the fast nodes. The bottleneck of training lies in the slowest nodes; in a large-scale homogeneous cluster, although the performance of the nodes is the same, due to the large number of nodes, the overall The stability of the parameter server will be reduced, and some node performance fluctuations will inevitably occur. At the same time, the number of requests that the parameter server needs to process will also increase greatly, resulting in a relatively large synchronization overhead for each round of iteration.

The asynchronous parallel mechanism eliminates the time overhead of nodes waiting for each other because nodes do not need to consider the state of other nodes during each round of iteration, so the resource utilization of nodes is high, and its training speed is fast. However, because there is no parameter update synchronous operation , there will be old gradient problems. In the hyperdimensional space of model parameters, the update direction of parameters is more twists and turns. Therefore, under the same number of iterations, the convergence effect of the model will be worse than that of the synchronous parallel mechanism.

Images