Multi-tenant resource optimization scheduling method facing different types of loads

Abstract

The invention relates to a multi-tenant resource optimization scheduling method facing different types of loads. The multi-tenant resource optimization scheduling method comprises the following steps: 1, submitting jobs by system tenants and adding the jobs into a job queue; 2, collecting job load information and sending the job load information to a resource manager; 3, judging different load types of the jobs according to the job load information by the resource manager, and sending type information to a job scheduler; 4, carrying out job scheduling according to the different load types by the job scheduler; if the job is a calculation intensive type job, scheduling at the node; if the job is an I / O intensive type job, delaying and waiting; and 5, collecting job scheduling decision-making information to a scheduling reconstruction decision-making device, reconstructing a target calculation node, and carrying out the job scheduling according to a final decision-making result. According to the method provided by the invention, a multi-tenant shared cluster is realized, so that the cost of establishing an independent cluster is reduced, and meanwhile, a plurality of tenants can share more big data set resources. The better data locality is realized facing optimization of the different types of loads, and the balance between the equity and efficiency in a job scheduling process can be realized very well; and calculation performances of the whole cluster, such as throughput rate and job responding time, are improved.

Description

technical field [0001] The invention relates to a multi-tenant resource management technology, in particular to a multi-tenant resource optimization scheduling method for different types of loads. Background technique [0002] In recent years, network information technology has advanced by leaps and bounds, and the contradiction between massive digital information and people's ability to obtain the information they need has become increasingly prominent. On the other hand, the application of distributed computing frameworks such as MapReduce, Dryad, and Spark makes it possible for users to find correct information from massive databases within a tolerable time. Traditional web multi-tenant systems need to process complex business logic, and therefore need to ensure a high level of data consistency and support complex data queries. In contrast, a multi-tenant system based on a big data platform often has the characteristics of a very large amount of data, does not require st...

AI Technical Summary

Problems solved by technology

At the same time, most of the existing delay scheduling algorithms are based on fixed waiting time, and do not fully consider the different load types of jobs

Assuming that most of the data of the job is concentrated on a certain node or a small number of data nodes in the cluster, computing tasks may also be concentrated on a certain node or some nodes in the cluster, resulting in low parallel efficiency and long response time of the job time

Images