Hadoop cluster-based task scheduling method and computer device

Abstract

The invention provides a Hadoop cluster-based task scheduling method. The method comprises the following steps of 10, setting that a business has N priorities, enabling each priority to correspond toone scheduling queue, enabling each task to specify the corresponding priority according to business importance, entering the scheduling queue of the corresponding priority, and performing queuing according to a time sequence, wherein N is a positive integer; 20, setting a maximum task concurrency number of a system and a maximum task concurrency number of the queue corresponding to each priority,scheduling the tasks in the scheduling queues according to a priority sequence from high to low at a time interval; 30, automatically distributing the queued tasks to the task idle priority queue forscheduling in a balanced manner according to the priority sequence from high to low; and 40, enabling the tasks of running failure and the tasks exceeding a predetermined maximum running time to re-enter the original priority queue for queuing and scheduling. The invention furthermore provides a computer device. Task priority grouping, queuing scheduling, route strategies and fault transfer are realized, so that the scheduling efficiency of the cluster tasks is greatly improved.

Description

technical field [0001] The invention relates to the field of computer distributed system data processing, in particular to a Hadoop-based cluster task scheduling method and computer equipment. Background technique [0002] As the business continues to increase, the Hadoop-based big data platform handles more and more tasks. The existing Hadoop cluster-based big data platform can split tasks into cluster nodes for distributed processing, but does not prioritize tasks Distinguish between scheduling and concurrent quantity control. Since any task can be scheduled, limited by the bottleneck of cluster software and hardware resources, tasks cannot be scheduled due to clustering, resulting in delays or failures of some high-priority services, and some low-priority tasks are always scheduled, improving cluster processing efficiency Low. Contents of the invention [0003] One of the technical problems to be solved by the present invention is to provide a Hadoop-based cluster tas...

AI Technical Summary

Problems solved by technology

[0002] As the business continues to increase, the Hadoop-based big data platform handles more and more tasks. The existing Hadoop cluster-based big data platform can split tasks into cluster nodes for distributed processing, but does not prioritize tasks Distinguished scheduling and concurrent quantity control

Since any task can be scheduled, limited by the bottleneck of cluster software and hardware resources, tasks cannot be scheduled due to clustering of tasks, resulting in delays or failures of some high-priority services, and some low-priority tasks are always scheduled, improving cluster processing efficiency Low

Images