Bottom-up distributed scheduling system and method

Abstract

The invention relates to a bottom-up distributed scheduling system and method, and the system comprises a global scheduler, which stores and identifies child nodes of enough resources with task request types in a distributed cluster, and receives scheduling task requests forwarded by local schedulers from all child nodes, and makes a scheduling decision according to the load of each child node andthe constraint of the task; local schedulers, used for arranging the tasks locally for processing, and if the local schedulers do not meet the scheduling requirements, the local schedulers can forward the tasks to the global scheduler; and a global control storage unit, used for storing the information of the tasks forwarded by the local scheduler. According to the system, a scheduling mode withlocal scheduling priority and global scheduling as assistance is set, the characteristic of task isomerism in reinforcement learning is met, and the scheduling speed of isomerism tasks is high; extension is very convenient, task assignment and task scheduling are completely separated, and delay minimization is achieved.

Description

technical field [0001] The present invention relates to the field of simulation computing, in particular to a bottom-up distributed scheduling system and method. Compared with traditional scheduling methods, the efficiency of processing tasks is higher, the number of tasks is larger, and the scalability is better. it is good. Background technique [0002] With the rapid development of artificial intelligence technology, a new generation of artificial intelligence applications, such as reinforcement learning, will continuously interact with the environment and learn from these interactions. These applications place new and demanding demands on the system, both in terms of performance and flexibility. This requires our scheduling method to be able to handle millions of heterogeneous tasks per second with millisecond latency. [0003] The current scheduling modes are roughly as follows: round-robin scheduling, weighted scheduling, hash scheduling, least task scheduling, lowes...

AI Technical Summary

Problems solved by technology

The disadvantage is: for distributed systems, the efficiency of this scheduling method is too low

The disadvantage is: this method can only be used in combination with other methods, and is an auxiliary supplement to other methods

The disadvantage is: for distributed systems, this method is inefficient and not easy to expand

The disadvantage is that some tasks will consume more node resources. Although the number of tasks processed in the cluster is balanced, the amount of processing may vary greatly, and the number of processed tasks cannot truly reflect the load of the cluster.

The disadvantage is similar to the least task scheduling method, judging by the length of the connection time, it cannot effectively balance the load

[0010] The current scheduling method is difficult to meet the needs of processing millions of heterogeneous tasks per second with millisecond-level latency

Images