Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. The method comprises the steps: acquiring a data processing request, wherein the data processing request is used for requesting to update the weight of network parameters in a deep learning network; in response to the data processing request, determining matrix structure information corresponding to weight matrixes where the weights of the network parameters are located; according to the matrix structure information, distributing the weight matrix to each data processing process, wherein the object weight matrix distributed to each data processing process reaches a uniform distribution condition; and under the condition that each data processing process determines a reference gradient value based on the respective corresponding object weight matrix, performing updating processing on the object weight matrix allocated in the data processing process according to the reference gradient value. The technical problem that the zero technology cannot be compatible with the lambda optimizer for model training to save the video memory is solved.

Description

technical field [0001] The present invention relates to the field of computers, in particular to a data processing method and device, a storage medium and electronic equipment. Background technique [0002] With the increase in the complexity of deep learning models and the size of datasets, graphics processing unit (GPU) memory has always been considered to be the biggest bottleneck for training deep learning models, which means that large-scale training often encounters model parameters. If there are too many and cannot be accommodated by the video memory, the video memory of the GPU card is very limited, which is generally much lower than the system memory. Therefore, when the parameters of the deep learning model are large, the deep learning model may not be able to be trained in GPU mode. At present, the implementation of zero technology is usually used to completely "flatten" the split of the optimizer state and gradient of the deep learning model (that is, reduce the ...

AI Technical Summary

Problems solved by technology

[0004] Embodiments of the present invention provide a data processing method and device, a storage medium, and electronic equipment to at least solve the technical problem that the zero technology cannot be compatible with the lamb optimizer for model training to save video memory

Images