Data parallel job resource allocation method based on decision tree prediction

Abstract

The invention discloses a data parallel job resource allocation method based on decision tree prediction. The method comprises the steps that 1, acquiring a completed job data set; 2, training a decision tree model by using the completed job data set; 3, estimating an initial resource allocation estimated value of each job; 4, using an initial resource allocation algorithm to perform resource initial allocation on the jobs with the obtained initial resource allocation estimated values to obtain an initial resource allocation value of each job; and 5, submitting the jobs of which the initial resource allocation values are obtained to a computer cluster to start execution. The job completion time difference under different computing resource allocations is predicted through the decision treemodel, the computing resource allocation enabling the job completion time to be minimum is found, and a better computing resource allocation estimated value is obtained. According to the method, an underlying network, disk read-write operation and the like are used as prediction features to reflect network transmission and disk input and output overhead of a distributed computing framework, so that higher prediction accuracy than modeling prediction is realized.

Description

technical field [0001] The invention belongs to the field of parallel and distributed computing, in particular to a data parallel operation resource allocation method based on decision tree prediction. Background technique [0002] Resource allocation or resource management is one of the main research issues in computer science, involving issues such as network systems, distributed systems, and cloud environments. Its purpose is to allocate (allocation) and assign (assignment) specific computing resources (such as the number of CPU cores), network resources, storage resources, etc. for the jobs submitted by users. Since resource allocation can optimize the job completion efficiency and resource utilization of data centers and cloud platforms, it is an important functional component of the distributed computing framework. Especially in distributed systems like Hadoop and Spark, a data parallel job often involves the concurrent execution of multiple tasks, which includes the ...

AI Technical Summary

Problems solved by technology

When the number of computing resources exceeds a threshold, the completion time of the job cannot be reduced, but increased

This leads to the problem of over-allocation of computing resources

Images