Method for dynamic allocation of Map/Reduce data processing platform memory resources based on prediction

Abstract

Provided is a method for dynamic allocation of Map / Reduce data processing platform memory resources based on prediction. The allocation method comprises the steps of initialization, task memory resource use prediction, task memory resource release, task memory resource adding and backtrack. In the method, against the characteristic of obvious fluctuation of the memory resource use amount during the Map task and Reduce task running process, according to historical records of the memory use amount during the Map task and Reduce task running process, a linear regression and t test method is adopted, task memory use rules are calculated, the memory amount needing to be used in task follow-up operation process is predicted, the memory allocation amount of a Map task and a Reduce task which are in running is dynamically added or reduced according to the predicted task memory use amount, so that the use efficiency of the Map / Reduce data processing platform memory resources is effectively improved, and the execution efficiency of Map / Reduce operation is improved.

Description

technical field [0001] The invention belongs to the field of distributed computing, and in particular relates to a method for forecasting and dynamically allocating memory resources in a Map / Reduce massive data processing platform. Background technique [0002] Map / Reduce is a new type of parallel computing model, which has been widely used in the field of massive data processing. Memory is an important computing resource that supports the operation of Map / Reduce applications. In actual operation, a Map / Reduce application is composed of one or more Map / Reduce jobs. The execution of each Map / Reduce job usually includes a Map phase and a Reduce phase. Among them, the Map stage and the Reduce stage can be respectively mapped to multiple Map task processes and Reduce task processes to be executed in parallel. The operating platform of the Map / Reduce application (hereinafter referred to as the "Map / Reduce platform") allocates memory resources required for the Map / Reduce applic...

AI Technical Summary

Problems solved by technology

[0004] However, the application of the above method to the actual Map / Reduce production platform has the following problems: the usage of memory resources by Map tasks and Reduce tasks often has significant fluctuations during their operation, and the actual consumption requirements of users for task memory resources Difficult to accurately grasp

Therefore, it is an objective fact that the user-set-oriented memory allocation method is adopted in the Map / Reduce platform, which leads to excessive application of memory resources by users.

At the same time, in the existing user-setting-oriented method, the task cannot actively release the excessively occupied memory resources for use by the Map task and Reduce task to be scheduled.

This makes the task to be scheduled delay the start of execution due to the inability to obtain the initial memory allocation, thus greatly reducing the platform's task throughput and memory resource utilization

In addition, the user-setting-oriented method is difficult to prevent malicious users from excessively applying for memory resources, resulting in malicious competition for platform resources

Images