Spark task processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention disclose a Spark task processing method and apparatus, an electronic device and a storage medium. The method comprises the steps of sequentially reading to-be-processed tasks according to priorities of the to-be-processed tasks; aiming at each currently read to-be-processed task, processing the to-be-processed task, and judging whether to execute the currently read to-be-processed task or not according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read to-be-processed task, if so, executing the currently read to-be-processed task, and judging the next to-be-processed task. The method comprises the steps that according to the running state of a Spark cluster system, current available resources and resources required by a current to-be-processed task, the current to-be-processed task is processed; whether the current to-be-processed task is executed or not is determined, so that the problem that the tasks sorted later but needing few resources cannot be executed due to the fact that the tasks sorted earlier consume too much time and occupy too many resources can be avoided, and then optimal allocation of the resources can be achieved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a Spark task processing method, device, electronic equipment and storage medium. Background technique [0002] Spark is a general-purpose engine that can be used to complete a variety of operations, including SQL queries, text processing, machine learning, and more. In big data systems, Spark is often used to fulfill various big data processing requirements. For example, to perform multiple SQL query statistics tasks through Spark, the general measures taken at this time are: when there are multiple tasks to be processed, spark-submit (Spark application deployment tool) will create sparkcontext through a unified main method , in the spark context, multiple tasks are submitted at the same time through multi-threading, and then after the spark context receives all these tasks, resources are allocated in turn through the central scheduling to execute each task, and finall...

AI Technical Summary

Problems solved by technology

Only in accordance with the order of tasks in the originally designed main method, one by one to apply for resources and queue up tasks, which is not flexible enough to meet actual needs

For example, when a task that is sorted earlier takes too long and takes up too many resources, it will cause the task that is sorted later but requires less resources to fail to execute, so that the optimal allocation of resources cannot be achieved

Another example is that when an urgent task needs to be performed temporarily, it cannot be dynamically scheduled and the task is executed first, so it is not flexible enough to meet actual needs

[0005] For the processing method of adding multiple main methods and dispersing multiple tasks in multiple main methods, since one task corresponds to one main method, there will be too many tasks and too detailed allocation, which will increase the maintenance cost of operation and maintenance personnel

Images