Big data task scheduling system and method

Abstract

The invention discloses a big data task scheduling system and method. The big data task scheduling method comprises the following steps that: a task configuration module acquires a task configuration command, and configures a task parameter according to the task configuration command; a task scheduling center node generates an executable task according to the task parameter, performs statistics to obtain the number of tasks running on the task running node and a data source, and controls a concurrency degree of the tasks according to the task number and a preset threshold value, so that the task number is small than the preset threshold value; and task distribution is performed when the concurrency degree of the tasks is in a preset threshold value range and satisfies dependence, and the task running node runs the executable task allocated by the task scheduling center node. In the method, the concurrency degree of current tasks is obtained by statistics, and whether the number of run tasks exceeds the preset threshold value or not is judged to determine whether to continue issuing the executable task or not, so that over high system load caused by simultaneous running of excessive tasks is avoided. Through adoption of the system and the method, the concurrency degree of big data tasks is controlled, so that the running efficiency of the big data tasks is ensured.

Description

technical field [0001] The present invention relates to the field of big data, in particular to a system and method for scheduling big data tasks. Background technique [0002] With the development of Internet technology and information technology, a large amount of information has been digitized, resulting in massive data collections that cannot be captured, managed and processed by conventional tools. This data collection is also called big data. [0003] Big data has massive data scale and various data types. Big data systems are used to process big data and dig out the data value of massive data. In order to better process and analyze big data, the processing tasks of big data can be scheduled accordingly to ensure the efficient operation of big data tasks. [0004] The big data system includes a task scheduling system for scheduling big data tasks. Existing big data task scheduling systems generally manage tasks based on workflow. By setting the dependencies between t...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to provide a system for scheduling large data tasks, and the purpose is to solve the problem in the prior art that the system load is too high, resulting in low operating efficiency of large data tasks; another purpose of the present invention is to provide a large The method of data task scheduling, which controls the concurrency of big data tasks, makes the operation efficiency of big data tasks higher

Images