Task processing system and method, computer equipment and storage medium

Abstract

The invention relates to the technical field of big data processing, and provides a task processing system and method, computer equipment and a storage medium. The task processing system comprises a task submitting module used for receiving created tasks, generating to-be-processed tasks and submitting the to-be-processed tasks; an application publishing module which is used for receiving each submitted to-be-processed task, applying execution resources of each to-be-processed task to the corresponding DataNode through the YARN resource scheduler, and sending each to-be-processed task to the corresponding execution queue; a task execution module which comprises a plurality of execution queues, and each execution queue executes each to-be-processed task in parallel according to the receivedto-be-processed task; and a monitoring module which communicates with the application release module and the task execution module, and the monitoring module collects and monitors the application release data and the task execution data to respond to the task submission module. According to the invention, automatic release, execution and real-time monitoring of tasks can be realized, and the taskprocessing efficiency and the operation stability are improved.

Description

technical field [0001] The present invention relates to the technical field of big data processing, in particular to a task processing system, method, computer equipment and storage medium. Background technique [0002] In recent years, with the rise of big data technology, more and more companies have established big data departments to manage their business through clusters. With the continuous growth of the company's business, there are some common problems in the clusters of various companies, such as: how non-departmental employees use cluster resources, how to start and stop a large number of tasks in the cluster, how to prevent tasks from being misoperated by irrelevant users, monopoly of application resources and Sharing issues, troubleshooting of task logs, etc. In order to avoid the above-mentioned problems, the industry generally integrates task processing into the big data architecture platform of the family bucket, and publishes it through the cloud platform or...

AI Technical Summary

Problems solved by technology

This processing method requires users to have a high degree of mastery of related commands, resulting in inflexible applications and poor usability.

In addition, due to reasons such as cluster authority management and security considerations, users cannot directly view the detailed logs of task execution, and cannot view the stages and results of task execution in a timely manner. Control

Images