Method and device for cluster management task scheduling

Abstract

The invention relates to the field of node allocation, in particular to a cluster management task scheduling method, which can realize reasonable allocation of limited computing power resources between a user and a task as much as possible through reference and arrangement allocation of multiple dimensions such as task priority, user right number, running time length and user utilization rate. And the distribution balance of internal resources is realized. The method has the advantages that compared with traditional scheduling, the problem of butt joint coupling of a scheduling module and other modules in a traditional module is solved through independent scheduling module design, and a program is more flexible and easy to expand. A two-in-one mode of interruption and starting logic is adopted, task states, newly-built tasks, interrupted tasks or restarted tasks do not need to be concerned from the scheduling level, only results need to be calculated, the independence of a task scheduling module is improved from the bottom layer design, and the task scheduling efficiency is improved. Conditions are provided for more subsequent users, a larger cluster and a more complex platform service scene.

Description

technical field [0001] The invention relates to the field of computing clusters, in particular to a method and device for cluster management task scheduling. Background technique [0002] With the development of artificial intelligence technology, more and more companies, universities, and scientific research institutes regard artificial intelligence as a new direction and new driving force for research and exploration. At present, the main field of artificial intelligence technology development is deep learning. By combining low-level features to form more abstract high-level representation attribute categories or features, to discover the distributed feature representation of data, so as to solve many problems that traditional machine learning algorithms cannot solve in the past. The deep learning model relies on the training of large-scale samples and the support of computer hardware. With the increase of business volume and the continuous improvement of model complexity...

AI Technical Summary

Problems solved by technology

[0004] The traditional scheduling method has many defects. The static scheduling method means that the situation of the cluster, including changes in user quotas, task changes, and cluster resource changes, cannot be responded in time, and the situation of dynamically adjusting task scheduling

Simple first-in-first-out tends to lead to low flexibility of the scheduler, and some urgent tasks cannot be scheduled; priority scheduling will cause some tasks to occupy resources for a long time, which cannot improve the overall efficiency of the cluster

Moreover, the traditional scheduling method distinguishes between new and suspended tasks, and the scheduler is used once each time a new task is created, which logically leads to a high degree of coupling between the scheduling module and other modules, which is not conducive to the flexible expansion of the scheduling module, etc. problems to be solved

Images