Data processing method and apparatus

Abstract

The invention provides a data processing method and apparatus. The method comprises the following steps that a control device obtains a data processing request and then allocates the data processing request to corresponding computing nodes according to load feature information of the computing nodes; and the computing nodes process the data processing request. According to the technical scheme, the data processing request is allocated according to the load feature information of the computing nodes, so that load balance among the computing nodes is realized; the utilization rate of system equipment is high; different computing nodes do not wait for one another; and the computing nodes in a system do not have idle states, so that the whole system can run efficiently.

Description

technical field [0001] The invention belongs to the field of high-performance computing, and in particular relates to a data processing method and device. Background technique [0002] With the continuous development of artificial intelligence technology, deep learning frameworks such as caffe, tensorflow, and cntk have emerged one after another. However, in terms of scalability, some do not support multiple machines, and some have poor scalability. However, the increasing amount of data and More and more complex deep learning networks put forward higher and higher requirements for the scalability of deep learning frameworks. [0003] The deep learning framework based on caffe has problems such as insufficient computing network bandwidth, small memory bandwidth and capacity of the current server computer system, and cannot process large-scale data. Contents of the invention [0004] The present invention provides a data processing method and device to solve the above prob...

AI Technical Summary

Problems solved by technology

[0002] With the continuous development of artificial intelligence technology, deep learning frameworks such as caffe, tensorflow, and cntk have emerged one after another. However, in terms of scalability, some do not support multiple machines, and some have poor scalability. However, the increasing amount of data and More and more complex deep learning networks put forward higher and higher requirements for the scalability of deep learning frameworks

[0003] The deep learning framework based on caffe has problems such as insufficient computing network bandwidth, small memory bandwidth and capacity of the current server computer system, and cannot process large-scale data.

Images