Sliding window sampling-based distributed machine learning training method and system thereof

Abstract

The invention provides a sliding window sampling-based distributed machine learning training method and system thereof. The method comprises the steps of initializing parameters of a machine learning model; obtaining a data fragment of all data and independently carrying out model training; collecting multiple rounds of historical gradient expiration degree samples, sampling the samples through sliding, calculating a gradient expiration degree context value, adjusting the learning rate and then initiating a gradient update request; asynchronously collecting the multiple gradient expiration degree samples, updating global model parameters by using the adjusted learning rates and pushing updated parameters; asynchronously obtaining pushed global parameters for updating, and further carrying out next training; checking the model convergence, if the model is not convergent, carrying out model training cycle; and if the model is convergent, obtaining model parameters. The learning rate of a learning device is controlled by using the expiration gradient, the stability and the convergence effect of distributed training are improved, the training fluctuation caused by the distributed system is reduced and the robustness of distributed training is improved.

Description

technical field [0001] The present invention relates to large-scale machine distributed training, in particular to a distributed machine learning training method and system based on sliding window sampling. Background technique [0002] Modern neural network architectures trained on large data sets can achieve impressive results across a wide variety of domains, ranging from speech and image recognition, natural language processing, to industry-focused applications such as fraud detection and recommendation systems and other aspects. However, training these neural network models has strict computational requirements. Although significant progress has been made in GPU hardware, network architecture, and training methods in recent years, the fact is that on a single machine, the time required for network training is still long. unrealistic. Fortunately, we are not limited to a single machine: a great deal of work and research has made efficient distributed training of neural...

AI Technical Summary

Problems solved by technology

[0004] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a distributed machine learning training method and system based on sliding window sampling, which is used to solve the problem that the gradient expiration degree of the learner in the prior art cannot be perceived. The context information of the gradient expiration degree of the controller is too simple to deal with the expired gradient, which leads to the problem of poor stability and convergence effect of distributed asynchronous training

Images