MapReduce short job optimization system and method based on resource reuse

Abstract

The invention discloses a MapReduce short job optimization system and method based on resource reuse. The system comprises a master node, a primary slave node and a plurality of secondary slave nodes, wherein the master node is connected with the primary slave node; the primary slave node is connected with the plurality of secondary slave nodes; a resource manager and a primary scheduler are deployed on the master node; an application manager, a task performance estimator and a sub-scheduler are deployed on the primary slave node; the sub-scheduler is connected with the task performance estimator; the sub-scheduler is further connected with the master node; and node managers are deployed on the secondary slave nodes. Through adoption of the MapReduce short job optimization system and method, short job running performance is optimized from the aspect of increase of the effective resource utilization ratio; the resource allocation and recovery frequency is lowered; the resource allocation and recovery time is used for running short jobs; and the short job execution performance is improved in a way of shortening resource waiting time of jobs.

Description

technical field [0001] The invention relates to a resource reuse-based MapReduce short job optimization system and method. Background technique [0002] Industries such as the Internet, finance, and media are facing the challenge of processing large-scale data sets, but conventional data processing tools and computing models cannot meet their requirements. The MapReduce model proposed by Google provides an effective solution for it, and Hadoop is an open source implementation of MapReduce. Hadoop decomposes the submitted job into smaller-grained Map tasks and Reduce tasks. These tasks run in parallel on multiple nodes in the cluster, thus greatly reducing the running time of the job. Hadoop hides the details of parallel computing—distributing data to computing nodes, rerunning failed tasks, etc., allowing users to focus on specific business logic processing. Moreover, Hadoop provides good linear expansion, data redundancy, and high fault tolerance of computing, which make ...

AI Technical Summary

Problems solved by technology

Using the process pool reduces the cost of starting new jobs, but Tenzing has two disadvantages: one is that the number of reserved processes exceeds actual needs, which wastes cluster resources; the other is that the Tenzing scheduler can only use certain specific process, compromising the nature of task-local computing

However, the FAIR scheduling algorithm does not consider the heterogeneity of the cluster and the time constraints of the job

Images