Distributed operation coordination control method, device, computer equipment and storage medium

Abstract

The invention discloses a distributed job coordination control method, a device, a computer device and a storage medium, wherein the method may comprise: the first job_coordinator process in the container in which the parameter server process resides starts, spin synchronization is performed until all the parameter server Pods in the job are running state, a unique number is assigned to the parameter server process in the container, the parameter server process starts in the container, and the job is controlled to return to the state; the second job_coordinator process, located in the container in which the training process resides, starts, spin synchronization is performed until all the training process Pods in the job are running, unique numbers are assigned to the training process in the container, the training process in the container is started, links between the training process and all the parameter server processes in the job are established, and the job return status is controlled. The scheme of the invention can improve the resource utilization rate and simplify the realization logic, and the like.

Description

【Technical field】 [0001] The invention relates to computer application technology, in particular to a distributed operation coordination control method, device, computer equipment and storage medium. 【Background technique】 [0002] In distributed jobs such as distributed machine learning jobs and distributed deep learning jobs, the models used are getting larger and larger, and the amount of data is also increasing accordingly. Although this improves the accuracy of learning, it also increases the Training time, the most common method is to use a large-scale machine cluster for parallel training. [0003] A distributed machine learning job or a distributed deep learning job usually includes several parameter server processes and several training processes. The training process needs to communicate with all parameter server processes. The training process downloads model parameters from the parameter server process and trains The final model parameters are updated to the par...

AI Technical Summary

Problems solved by technology

[0005] However, in the former method, the machine learning algorithm and the deep learning framework need to rely on MPI internally, and implement the control logic of MPI within the algorithm or framework, which cannot effectively isolate jobs, resulting in the problem of low resource utilization.

[0006] In the latter method, since the Master needs to be implemented separately, and the communication process between the parameter server process and the training process and the Master needs to be implemented separately, the implementation logic is more complicated

Images