Two-stage self-adaptive scheduling method suitable for large-scale parallel data processing tasks

Abstract

The invention discloses a two-stage self-adaptive scheduling method suitable for a large-scale parallel data processing tasks. Two-stage scheduling is performed on tasks from a task stage and a subtask stage. By the method, the problem of parallel scheduling difficulty caused by complex dependence relationship among the tasks is solved effectively, parallelism degree is increased, orderly and efficient parallel processing of large-scale data processing tasks is realized, task waiting or executing time is reduced, and overall executing time is shortened. In addition, by the method, executor operation statistical information can be fed back to a scheduler, self-adaptive adjusting of executor pool size and task type is realized, and scheduling is constantly optimized, so that system resourceusing efficiency is improved.

Description

technical field [0001] The invention relates to a two-level adaptive scheduling method suitable for large-scale parallel data processing tasks, belonging to the field of data processing task scheduling. Background technique [0002] With the continuous development of Internet technology, the demand for large-scale massive data storage and processing in various fields is increasing, and the requirements for its work efficiency and processing cost are also increasing. How to reasonably allocate large-scale data processing tasks to multi-processor systems, improve execution efficiency, and pursue the smallest overall execution time has become a problem that needs to be solved urgently. [0003] Traditional general task scheduling algorithms, such as first come first serve scheduling algorithm, high priority priority scheduling algorithm, time slice round robin scheduling algorithm, etc., have certain limitations and are not suitable for large-scale data processing task scheduli...

AI Technical Summary

Problems solved by technology

[0005] The problem solved by the technology of the present invention is: to overcome the deficiencies of the prior art, to provide a two-level adaptive scheduling method suitable for large-scale parallel data processing tasks, to process tasks based on two levels of task / subtask, and to improve the degree of parallelism , which effectively solves the difficult problem of parallel scheduling caused by complex dependencies among tasks, realizes orderly and efficient parallel processing of large-scale data processing tasks, and reduces the overall execution time

Images