Data processing task dispatching system and method

Abstract

Provided is a data processing task dispatching system and a method thereof. The data processing task dispatching system comprises a task module, a task group module, a dispatching module, a log module, and a monitoring module; data processing requirements are firstly sent to the task module, the task module is interconnected with the task group module, the task group module is connected with the dispatching module and the monitoring module at the same time, the log module is connected with the dispatching module and the monitoring module at the same time. In a software application environment having a large batch background timed data processing tasks, the data processing task dispatching system and the method thereof achieve the unified dispatching, monitoring and controlling management of a series of data processing tasks by encapsulating tasks, designing the dispatching module and designing the monitoring module.

Description

technical field [0001] The invention belongs to the technical field of data processing, and in particular relates to a data processing task scheduling system and method. Background technique [0002] In many software application scenarios, especially in data warehouse and data analysis software applications, there are often a large number of data processing requirements. This type of data processing requirements generally has the following characteristics: (1) This type of requirement generally needs to process a large amount of data, which takes a relatively long time and is not suitable for real-time data processing during user interaction. , so it is generally used as a background batch processing task to be started regularly and pre-processed; (2) This type of data processing requirements is generally closely related to time elements, so this type of data processing requirements generally need to be processed at a certain time interval (such as every year , monthly, wee...

AI Technical Summary

Problems solved by technology

This type of data processing requirements generally has the following characteristics: (1) This type of requirement generally needs to process a large amount of data, which takes a relatively long time and is not suitable for real-time data processing during user interaction. , so it is generally used as a background batch processing task to be started regularly and pre-processed; (2) This type of data processing requirements is generally closely related to time elements, so this type of data processing requirements generally need to be processed at a certain time interval (such as every year , monthly, weekly, daily, etc.), and the business data processed often have different time attributes; (3) The normal operation of data processing tasks is often restricted by factors such as data quality and server operating status

Once the data processing program makes an error, it is difficult to find it in time, or it is difficult to locate the cause and location of the error, which is inconvenient to debug the program, and it is also inconvenient to save some data processing results before the error.

(4) There are often more than one data processing requirements of this type in the same application, and each requirement may be quite different, which is inconvenient for unified monitoring and management

Because the encapsulation of data processing tasks lacks a unified specification, these tasks are quite different, and the scheduling methods are varied, making it difficult to achieve overall design and unified management

[0005] (2) After the data processing task is executed, the simple execution result can only be viewed through the timing task management mechanism provided by the database or the operating system, and more detailed and easy-to-read status information cannot be obtained

Once a scheduled task fails, it is difficult to locate the cause of the error and the program location of the error

[0006] (3) Usually this type of data processing task is closely related to attributes such as the scope of business data and business time, and once the traditional scheduling method fails, because it is difficult to locate the status of various attributes when an error occurs, it often leads to the failure of the entire data processing. The previous efforts of the task are wasted, and it is difficult to save the phased results of data processing before the task error

[0007] (4) The number of such data processing tasks in an application system is generally large and the differences are large. In the past, the processing of such data processing tasks was relatively random. In a system application scenario, multiple timing task mechanisms are often used The completion of such tasks will bring greater difficulty to the normal operation and maintenance of the system

And there is a lack of unified planning, control, monitoring and fault handling for such timing data processing tasks

Images