Model training method, device and equipment based on pipeline parallelism

Abstract

The embodiment of the invention discloses a model training method based on pipeline parallelism. The model training method comprises the steps of segmenting a to-be-trained model according to a normal training sequence to obtain a plurality of sub-slices; dividing a part of sub-slices with the normal training sequence in the front into a first group to form a first sub-model, and dividing the remaining sub-slices into a second group to form a second sub-model; distributing the sub-slices contained in the first sub-model to different working parties, and distributing the sub-slices contained in the second sub-model to different working parties; according to the training sequence obtained through adjustment, distributing training time blocks to the sub-slices on the working party to which the sub-slices are distributed, wherein the training sequence comprises the following steps: for a single-round training sample, sequentially executing the forward propagation of the first sub-model in the current period, the backward propagation of the first sub-model in the previous period, the forward propagation of the second sub-model in the current period and the backward propagation of the second sub-model in the current period; and according to the training time block, carrying out parallel training on the to-be-trained model on the plurality of working parties.

Description

technical field [0001] This specification relates to the field of machine learning technology, and in particular to a pipeline-based parallel model training method, device and equipment. Background technique [0002] In recent years, deep learning has greatly promoted the development of artificial intelligence technology, and has achieved good results in many fields. However, with the development of artificial intelligence technology, the parameter scale of the model is also increasing, and its growth rate has even far exceeded the growth rate of hardware processing performance. [0003] In order to solve this contradiction, model parallel technology has gradually attracted the attention of researchers. Model parallelism can divide a large-scale model into multiple model slices and distribute them to different working machines, so as to realize the training or reasoning of large-scale models. The traditional model parallel technology can only ensure that one working machin...

AI Technical Summary

Problems solved by technology

The traditional model parallel technology can only ensure that one working machine is running at the same time, and the others are vacant.

Images