A data-intensive cost optimization method based on mapreduce mechanism

Abstract

The invention relates to a data intensive computing cost optimization method based on a MapReduce mechanism. In the state that the popularity level and access situations of data change, execution time and execution cost for a MapReduce task are optimized. The method comprises the steps that a cost computing model based on the MapReduce mechanism in a cloud computing environment is provided; on the basis of the model, the number of copies of Map subtasks is dynamically changed according to the popularity level of the data in a load; as for hot data which have the large number of access times and are accessed frequently, file contention and bandwidth resource competition of networks are reduced by increasing the number of copies of the hot data, and therefore the computing cost and the computing time for the hot data are optimal; as for cold data having the small number of access times and are accessed seldom, by decreasing the number of copies of the cold data, creation and storage cost for the copies is reduced, and therefore the computing cost and the computing time for the cold data are optimal. By means of the data intensive computing cost optimization method, cloud computing users can effectively reduce resource using cost, and reliability and completeness of operation computed results can be guaranteed more efficiently.

Description

technical field [0001] The present invention relates to a cost-based optimization method, more specifically to a data-intensive computing cost optimization method based on a MapReduce mechanism. This method is suitable for the optimization of computing cost and computing time for cloud computing users when the popularity of data in the load and the access situation change dynamically. Background technique [0002] With the development of information technology, more and more industrial and academic organizations are facing the challenge of dealing with growing large-scale data, such as file analysis processing, scientific simulation and other applications. As the most mainstream programming model for processing large-scale data and data-intensive applications, MapReduce effectively decomposes large-scale input data sets into fixed-size data blocks, which are distributed and stored in different nodes in the cluster, and distributed parallel computing model , which greatly si...

AI Technical Summary

Problems solved by technology

[0003] The purpose of the present invention is to provide a data-intensive computing cost optimization method based on the MapReduce mechanism, which overcomes the above-mentioned shortages of file contention and limited resources of hot and busy data, and solves the problems caused by the dynamic access of data in the load. The problem of waste of computing time and cost; the data can be divided into hot data and cold data according to the data access situation in the load in real time, so as to dynamically change the number of data copies and optimize the computing time and computing cost of cloud computing users

Images