Heterogeneous network perception model division and task placement method in pipelined distributed deep learning

Abstract

The invention provides a heterogeneous network perception model division and task placement method in assembly line distributed deep learning, which mainly comprises three parts, namely deep learningmodel description, model division and task placement and assembly line distributed training. According to the method, firstly, for resource requirements of deep learning application in the GPU training process, corresponding indexes such as calculation time, intermediate result communication quantity and parameter synchronization quantity in the training execution process of the deep learning application are described and serve as input of model division and task placement; then indexes and heterogeneous network connection topology of the GPU cluster are obtained according to model description, a dynamic programming algorithm based on min-max is designed to execute model division and task placement, and the purpose is to minimize the maximum value of task execution time of each stage afterdivision so as to ensure load balance. And finally, according to a division placement result, performing distributed training by using assembly line time-sharing injection data on the basis of modelparallelism, thereby realizing effective guarantee of training speed and precision.

Description

technical field [0001] The invention relates to a model division and task placement technology for heterogeneous network perception in pipeline distributed deep learning, and belongs to the technical field of distributed computing. Background technique [0002] Deep learning (deep learning) is a class of machine learning techniques that use multi-layer nonlinear information for supervised or unsupervised feature extraction and transformation, as well as techniques for pattern analysis and classification. Deep learning generally includes two processes, the training process and the inference process: the training process is to use the designed neural network to extract features from a large number of training sets (known labels) to perform predictions, and then calculate according to the error between the predicted value and the actual label value Gradient, using the method of gradient descent to perform parameter update, repeat iterations until convergence. The inference pro...

AI Technical Summary

Problems solved by technology

[0006] The present invention mainly aims at the fact that the model division and task placement of distributed deep learning in the current pipeline training mode cannot adapt to the network heterogeneity of the GPU cluster, and proposes a network-aware model division and task placement method

Images