Resource allocation method, related equipment and system

Abstract

The embodiment of the invention discloses a resource allocation method, related equipment and a distributed system based on mapping / reduction MapReduce. The system includes a management node and a target calculation node, the management node is used for obtaining M calculation tasks, and a resource assessment model is built according to the data size of the M calculation tasks; information of onepart of calculation tasks in the M calculation tasks and the resource assessment model are sent to a target calculation node; the target calculation node is used for receiving the information, sent by the management node, of the part of calculation tasks and the resource assessment model; before each calculation state of the target task is executed, input data of the calculation stage is substituted into the resource assessment model so as to calculate the resource size of the calculation resource required in the calculation stage; through the calculation resource with the resource size in apreset resource pool, the input date is calculated. According to the resource allocation method, the related equipment and the distributed system, the utilization ratio of the calculation resource canbe improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a resource allocation method, related equipment and a system. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation, which can perform distributed processing on large amounts of data. Its core design is a distributed file system (English: Hadoop Distributed File System, referred to as: HDFS) and a mapping / reduction MapReduce programming model. HDFS provides storage for massive data, while MapReduce provides calculation for massive data. Normally, Hadoop's MapReduce processes a set of input key-value pairs (key / value pairs), and finally outputs a set of key-value pairs after being processed by a user-specified MapReduce function. MapReduce actually defines a Map function interface and a Reduce function interface. The Map function is used to convert input records to obtain intermediate results, and the Reduce func...

AI Technical Summary

Problems solved by technology

[0011] The defect of the existing technology is that the execution of some Reduce tasks is divided into multiple calculation phases, and the complexity of the algorithm used in each calculation phase is different. Correspondingly, the calculation required for each calculation phase will also result in Resources are also different. If the required computing resources are allocated for the entire execution phase of the Reduce task before the execution of the Reduce task, it will lead to waste of computing resources.

Images