Convolution neural network parallel processing method based on large-scale high-performance cluster

Abstract

The invention discloses a convolution neural network parallel processing method based on a large-scale high-performance cluster. The method comprises the steps that (1) a plurality of copies are constructed for a network model to be trained, model parameters of all the copies are identical, the number of the copies is identical with the number of nodes of the high-performance cluster, each node is provided with one model copy, one node is selected to serve as a main node, and the main node is responsible for broadcasting and collecting the model parameters; (2) a training set is divided into a plurality of subsets, the training subsets are issued to the rest of sub nodes except the main mode each time to conduct parameter gradient calculation together, gradient values are accumulated, the accumulated value is used for updating the model parameters of the main node, and the updated model parameters are broadcast to all the sub nodes until model training is ended. The convolution neural network parallel processing method has the advantages of being capable of achieving parallelization, improving the efficiency of model training, shortening the training time and the like.

Description

technical field [0001] The invention mainly relates to the design field of high-performance computing clusters, in particular to a convolutional neural network parallel processing method based on large-scale high-performance clusters. Background technique [0002] A high-performance computer is a computer cluster system, which connects multiple computer systems together through high-speed interconnection technology, and uses the comprehensive computing power of all connected systems to deal with large-scale computing problems, so it is often called a "high-performance computing cluster". Or "high-performance clusters". High-performance clusters are mainly used to deal with complex computing problems and are used in environments that require large-scale scientific computing, such as weather forecasting, oil exploration and reservoir simulation, molecular simulation, gene sequencing, etc. Applications running on high-performance clusters generally use parallel algorithms to d...

AI Technical Summary

Problems solved by technology

[0008] At present, the international cutting-edge research on the acceleration of neural networks mainly focuses on two directions: first, parallel acceleration based on multiple GPUs on a single server, a single server does not involve data transmission between multiple nodes, parallel acceleration is easy to achieve, but The scale of the network model is limited by the configuration of a single server; second, using a large-scale cluster to accelerate the training of the neural network, the DistBelief model was proposed, but it was not applied to the convolutional neural network. In the restricted Boltz Mann machine and deep belief network are widely used

Images