Multi-heterogeneous cluster job unified scheduling method and API interface

Abstract

A multi-heterogeneous cluster job unified scheduling method comprises the steps that S1, job proxy services are deployed on multiple clusters, the job proxy services unify input and output data formats of different job scheduling systems, and a unified data interface view is provided; S2, a job service gateway is deployed at the upper layer of the cluster, a plurality of job proxy services are summarized and managed, and a plurality of clusters are uniformly managed; S3, the user obtains the authorization of the operation proxy service, and achieves the unified access to the cluster through the operation proxy service; S4, the user submits jobs, the job service gateway allocates the jobs to specific clusters according to rules, and job files are transmitted locally and among the clusters or among the clusters through job proxy service; and S5, the job proxy service converts the request into a command of a job scheduling system in the cluster for execution, converts a result into unified job management information and returns the unified job management information. Unified job scheduling on the multi-heterogeneous cluster is realized, the job scheduling operation of the multi-heterogeneous cluster is simplified, and computing resources are fully utilized.

Description

technical field [0001] The present invention relates to the field of computers, and more specifically, to a unified scheduling method and an API interface for multi-heterogeneous cluster operations. Background technique [0002] The traditional scheduling system in the field of high-performance computing mainly manages the jobs of a single cluster. Users submit jobs to designated computing resources through tools such as command lines, and scheduling systems such as SLURM, PBS, and LFS allocate computing resources for jobs and submit them. calculate. [0003] However, due to the increase in the abundance of computing resources and the degree of replication, there will be multiple clusters with different structures in a single computing center, and the computing center will continue to upgrade and purchase new clusters, and users will also use their own private computing clusters at the same time and public computing center resources. On the one hand, because these differen...

AI Technical Summary

Problems solved by technology

On the one hand, since these different computing clusters use different configurations or different types of job scheduling systems, that is, multi-heterogeneous clusters, it adds complexity to users in operation

And it is difficult to achieve targeted utilization of existing computing resources, resulting in a waste of cluster computing resources

On the other hand, the traditional command-line usage requires users to have a deep understanding of each computing cluster and job scheduling system in order to effectively submit computing tasks. In the process of using different computing clusters, the technical difficulty of operation is significant Increase, the computing center composed of multi-heterogeneous clusters is difficult to be widely used, and the full utilization of computing resources cannot be realized, which limits the development of computing resource utilization technology and hinders the development of computer processing, cluster computing, and computer communication technology

Images