Deep learning framework Caffe system and algorithm based on MIC cluster

Abstract

The invention discloses a deep learning framework Caffe system and algorithm based on an MIC cluster. The deep learning framework Caffe algorithm runs on multiple nodes of the MIC cluster according to an MPI technology; the nodes share a task and data equally via MPI communication; different nodes parallelly execute subtasks and process subdata, perform ForwardBackward calculation in Caffe, and feed back execution results to a master node; the master node calculates and tabulates weight information fed back by each node, and distributes updated parameters to the nodes; and each slave node performs next iterative computation according to the new parameters. Therefore, multithread-based parallel computing in a same process is realized; and as a large quantity of matrix calculations are involved in a convolutional neural network, while multithread-based parallel computing is mainly applied to matrix calculations, the performance of an MIC processor can be played to the most extent by setting a quantity of parallel threads, and the runtime performance of the Caffe is improved.

Description

technical field [0001] The invention relates to the field of high-performance computing, and in particular to an optimization method for parallel processing of data and tasks of a deep learning framework based on a cluster system and ensuring load balancing between processes and nodes. Background technique [0002] The author of Caffe (Convolution Architecture For Feature Extraction) is Jia Yangqing, who graduated from UC Berkeley with a Ph.D. It is currently one of the most popular deep learning frameworks, and it is characterized by quick use, modularity, and openness. Caffe contains a variety of convolutional neural network implementation models, including googlenet, alexnet, etc. The training process of the entire convolutional neural network is realized by layer-by-layer calculations such as convolution and down-sampling. [0003] As the problems that need to be solved become more and more complex and the performance requirements of convolutional neural networks are get...

AI Technical Summary

Problems solved by technology

[0003] As the problems that need to be solved become more and more complex and the performance requirements of convolutional neural networks are getting higher and higher, more and more training data are required in the network, and distributed storage is in the network, correspondingly requiring more Trainable parameters and calculations, while the original version of Caffe is generally implemented serially based on a stand-alone system, which results in a considerable amount of time spent training a complex model that uses a large amount of data

[0004] The limitations of the original version of Caffe running on a single machine and single process lead to limited scalability and performance. At the same time, in the ForwardBackward calculation of the Caffe kernel part, complex operations such as matrices and equations are involved. The calculation of this part of the original version of Caffe is based on serial single-thread Execution, and when the size of the matrix is ​​large, the running time complexity on a single node will be quite large

Images