Data distribution method on basis of heterogeneous cluster

Abstract

The invention provides a data distribution method on the basis of a heterogeneous cluster, which is applied to a plurality of communicated equipment nodes. The method comprises the following steps of reading data block read information of each equipment node and determining required task data, wherein the data block read information comprises data blocks locally calculated when a local task is executed, data blocks read out by other equipment nodes and data blocks read in from the other equipment nodes; according to the read data block read information, predicting calculating ability of each equipment node; according to a predicting result, distributing the determined task data into the local task of each equipment node. According to the invention, by utilizing distribution of the task data on a bottom layer to guide dispatching of tasks on an upper layer and reasonably distributing the data, the calculating ability of each equipment node is matched with the distributed data; moreover, according to the invention, the determined task data is distributed into the local task of each equipment node, so that the problems of network expenses and aggravated network resource scramble which are caused by remote tasks and data movement are solved.

Description

technical field [0001] The invention relates to the technical field of distributed computing, in particular to a data distribution method based on heterogeneous clusters. Background technique [0002] Some recent research reports show that the next generation of data centers will have great heterogeneity due to reasons such as energy saving, unit price performance, and different performance prices. In data centers, in addition to these reasons, there is another very important Reasons for heterogeneity: A data center usually adds some new servers every once in a while to expand its computing capacity. Generally speaking, servers added one or two years apart will have several generations of hardware differences. At the same time, the data center is usually shared by multiple tasks, and the mutual influence between different tasks running at the same time will also cause the performance difference of each node. [0003] In the existing technology of heterogeneous clusters, in ...

AI Technical Summary

Problems solved by technology

It is the movement of these data that causes a lot of network overhead and worsens the network environment. At the same time, for slow nodes, because they also need to send data to fast nodes, this slows down the process of executing local tasks even more.

Images