A big data task scheduling method and system

Abstract

The invention discloses a task scheduling method for big data. The method comprises the following steps of: creating a task template according to a task type; Selecting a task template and a task scheduling service number to create a task, forming a configuration file, enabling the created task to comprise a task name, task content and a task execution period, configuring the task content in a kvvalue mode, and establishing a dependency relationship among the tasks through a kv file format; The task is read, a task example is generated, and task execution process information is acquired. Theinvention further discloses a big data task scheduling system. According to the big data task scheduling management method, big data scene scheduling, distribution and execution and unique characteristics of different data types are combined, the characteristics of classification, multi-triggering, strategy calling, blood relationship dependence and the like are highlighted, big data internal ecology construction is carried out, and big data task scheduling management is carried out according to different task types and different strategies through big data task scheduling.

Description

technical field [0001] The present invention relates to the technical field of big data task scheduling management, and more specifically, to a big data task scheduling method and system. Background technique [0002] In the business application of big data, with the iteration of business indicators, making it more and more complicated, managing the related applications of big data will become a headache, such as: job dependency scheduling, task running status Monitoring, troubleshooting of abnormal problems, etc. These problems will complicate our daily work. [0003] In a big data analysis system, some scripts or execution units need to be started at a specific time, and some even need to be executed after certain conditions are met. In this case, it is difficult to achieve it manually, and some systems also provide some The configuration of scheduled tasks, but this kind of configuration is more troublesome to manage, and some of them must invade the execution machine sy...

AI Technical Summary

Problems solved by technology

[0005] The second purpose of the present invention is to provide a big data task scheduling system. In order to solve the problem of manual configuration management, it combines more with the unique characteristics of big data scene scheduling, allocation, execution and different data types, highlighting classification, With the characteristics of multiple triggers, policy invocation, and blood dependence, etc., the construction of the internal ecology of big data is carried out. Through big data task scheduling, big data task scheduling management is carried out according to different task types and according to different strategies.

Images