Model training request scheduling method and device

Abstract

The invention provides a model training request scheduling method and device, and aims at a model training system for training a model, the system is composed of a plurality of parameter nodes and a plurality of computing nodes, the parameter nodes are used for updating parameters of the model based on a gradient, and the computing nodes are used for computing the gradient based on the parameters of the model; the method comprises the following steps: acquiring to-be-processed requests sent by a plurality of computing nodes; sorting the to-be-processed requests according to node identifiers in the to-be-processed requests, the node identifiers being used for identifying computing nodes corresponding to the to-be-processed requests; and according to the sorting result, sequentially sending each to-be-processed request to the parameter node to obtain a processing result of each to-be-processed request. The requests are sequenced based on the identifiers of the computing nodes in the requests, so that the requests are responded in order, the time for the parameter nodes to respond to the requests of the single computing nodes is shortened, and the model training efficiency is improved.

Description

Technical field [0001] The present application relates to the field of deep learning technology, in particular to a model training request scheduling method and apparatus. Background [0002] In the field of deep learning, such as computer vision, natural language processing, personalized recommendation and other fields, in order to improve the quality or effect of the model, larger-scale model parameters or larger-scale training data are usually used in model training, such as click-through estimation models with trillion-level parameter scales, language models with hundreds of billions of parameter scales, etc. Distributed training has become a necessary means for efficient training of ultra-large-scale models. [0003] The commonly used distributed training architecture is the ps-worker architecture, which divides the nodes into ps (parameterserver, parameter server) and worker (worker or compute server) two roles, in asynchronous mode, each worker independently initiates pull...

AI Technical Summary

Problems solved by technology

Since different workers are asynchronous, the time when the request arrives at ps is random, and the method of processing each request in chronological order makes the request response inefficient, which reduces the speed of asynchronous training

Images