Model parameter training method and device, server and storage medium

Abstract

The invention discloses a model parameter training method and device, a server and a storage medium, which belongs to the technical field of information. The method comprises the steps that an initialparameter value and a sample set of a model parameter of a target model are acquired; the first gradient of the model parameter is calculated according to the initial parameter value and the sample set; iterative quantization processing is carried out on the first gradient to acquire a quantized second gradient, wherein the iterative quantization processing is quantization processing carried outbased on an error cumulative value corresponding to the t-1-th iteration round in the t-th iteration round, and the error cumulative value is a quantization error cumulative value calculated based ona preset time attenuation coefficient; and the quantized second gradient is transmitted to a primary computing node, wherein the quantized second gradient is used to instruct the primary computing node to update the initial parameter value according to the quantized second gradient to acquire an updated parameter value. According to the embodiment of the invention, a quantization error correctionmethod is used to quantize and compress the first gradient of the model parameter, which reduces the communication cost and network overhead of gradient transmission.

Description

technical field [0001] The present application relates to the field of information technology, in particular to a model parameter training method, device, server and storage medium. Background technique [0002] DistBelief is an artificial intelligence deep learning framework that can be used to train large-scale neural network models. It has been widely used in many fields such as text processing and image recognition. DistBelief provides a distributed training mode: Stochastic Gradient Descent (SGD), which defines a main computing node and N sub-computing nodes, each sub-computing node is used for training to generate a copy of the model, the main computing node It is used to share model parameters for N child computing nodes. [0003] Before the training starts, the main computing node sends the initial parameter values ​​of the model parameters to each sub-computing node, and divides the training data set into multiple sample sets, which are distributed to N sub-computi...

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present application provides a model parameter training method, device, server and storage medium, which can be used to solve the problem of relatively large communication cost and network overhead of gradient transmission in the process of training the model in the related art

Images