Task processing method and system and nodes based on distributive type calculation system

Abstract

The invention discloses a task processing method and system and nodes based on a distributive type calculation system. The method comprises the steps that all nodes monitor self states and store state information, and a main node obtains the state information of all working nodes in a timing mode; when the main node determines an over-load working node, the main node informs non-overload working nodes of conducting load condition predication; the main node adjusts the task type of the overload working node according to predicated overload condition so as to reduce the load of the overload working node. Through the method, when overload occurs in a working node of the distributive type calculation system, and the main node adjusts the task type processed by all the working nodes in a dynamic mode according to the predicated load condition of all the working nodes in the future, so that dynamic load balance of all the working nodes of a cluster is realized, and the load balance of a system resource level is achieved. Consequently, the largest overall processing capacity of the cluster is improved.

Description

technical field [0001] The invention relates to distributed flow computing technology, in particular to a task processing method, system and node based on a distributed flow computing system. Background technique [0002] Stream computing is mainly used in functions such as real-time data processing and statistical learning. With the explosion of Internet big data, stream computing also adopts more advanced distributed computing methods to improve processing speed, which is called distributed stream computing system. The most representative of the distributed stream computing system is Storm, which uses a system similar to Hadoop Map / Reduce (Hadoop Map / Reduce is an easy-to-use software framework, based on which the application program can run on thousands of On a large cluster composed of commercial machines, and parallel processing of T-level data sets in a reliable and fault-tolerant manner), the data processing method adopts Zookeeper (ZooKeeper is an official sub-projec...

AI Technical Summary

Problems solved by technology

[0010] However, since Sprout and Bolt are logical units running on the port of the Supervisor work node, the relationship between Sprout and Bolt and workers is a many-to-one relationship. Therefore, the load of different workers is different, which leads to the load gap of the Supervisor where the workers are located. In addition, in the process of streaming computing, due to the different complexity of different data processing Bolts (processing N), the current method is to allocate ports sequentially, and Bolts of the same type may be allocated to the same Supervisor work node In this way, the loads of different working nodes are different; in addition, Shuffle Grouping can only work under specific conditions, and other Tuple (flow) methods cannot play a role in balancing the Supervisor. In the actual experiment, the inventor It is found that, for example, for the global grouping (Global Grouping) method, that is, all Tuples are sent to the same Task. At this time, the current Component concurrency number is generally set to 1, so that all Tuples will be concentrated on the same worker, that is, the same Supervisor. , resulting in a high load on the Supervisor

Images