Improved MapReduce data processing method under virtual machine cluster

Abstract

The invention discloses an improved MapReduce data processing method under a virtual machine cluster, which comprises the following steps: a task dispatching and managing center distributes tasks to virtual machines, the virtual machines which obtain the distributed tasks inquires a data storing and managing center for physical machines at which input data of the tasks is located, and each virtual machine selects the physical machine which is nearest to the topology of the virtual machine from the obtained virtual machines for carrying out data reading and data processing. By adopting the method, data storage and processing can be separate and independent, the physical machines are specially used for storing the data, the virtual machines are specially used for data processing, and a data reading strategy which is applicable to the virtual machine cluster is designed, thereby shortening the operation time, solving the performance hysteresis, reducing the resource consumption and improving the MapReduce data processing performance.

Description

Technical field [0001] The invention belongs to the field of distributed computing models, and specifically relates to the improvement of MapReduce performance under a virtual machine cluster. Background technique [0002] Cloud computing technology is developing rapidly and its applications are becoming more and more extensive. Cloud computing has two important components. One is virtualization technology: Virtualization technology is a decoupling technology that separates the underlying physical equipment from the upper layer operating system and software. It can realize the efficient and flexible use of computing resources. It can make fuller and reasonable use of computing resources to meet increasingly diverse computing needs, enabling people to use computing resources transparently, efficiently, and customizable, so as to truly realize the concept of flexible construction and on-demand computing; the second is large-scale data processing Middleware, a large-scale data proc...

AI Technical Summary

Problems solved by technology

However, the virtual machine cluster environment was not considered at the beginning of MapReduce design, so, so far, there has not been a considerable method in the research of combining the advantages of the two

Take EC2 (Elastic Cloud Computing) proposed by Amazon as an example. After users start MapReduce on a custom virtual machine cluster, MapReduce does not perceive changes in the underlying operating environment and still executes according to the original strategy. Serious performance lag problem: After the virtual cluster is enabled, the data needs to be imported from the physical cluster to the virtual cluster before it can be used by the MapReduce job, and after the calculation is completed, the data also needs to be exported. In this way, for the import and export of large-scale data, It takes a lot of time, for example, it takes a day to move 100TB of data on a 10Gb / s channel

Even if the data is not exported after the calculation is completed, the virtual machine cannot be shut down, affecting the performance of other users on the physical cluster, and will also bring additional power consumption

Images