Automatic parameter tuning method for iterative MapReduce operation

Abstract

The invention discloses an automatic parameter tuning method for an iterative MapReduce operation. The method comprises the following steps of A) judging whether to meet an iterative ending condition or not, if so, ending an iterative calculation task, otherwise, executing step B; B) searching to obtain new MapReduce parameter configuration according to a given MapReduce parameter search space by utilization of a parameter searching algorithm; C) realizing the running of the current MapReduce operation by utilization of the searched new MapReduce parameter configuration, judging the running effect of the current MapReduce operation after the operation is finished, and performing corresponding data adjustment treatment according to a judging result; D) returning to execute the step A. By utilization of the method disclosed by the invention, the running efficiency of each iterative MapReduce operation can be improved, the convenience is brought to a user, and the waste of time resources is greatly reduced. The method disclosed by the invention can be widely applied to a Hadoop running parameter tuning field.

Description

technical field [0001] The invention relates to computer information processing technology, in particular to an automatic tuning method for Hadoop operating parameters for iterative MapReduce jobs. Background technique [0002] technical term explanation [0003] Parameter configuration: This refers to a set of parameter values ​​that are composed of MapReduce parameters provided by Hadoop and can be configured by MapReduce jobs to form a set of parameter configurations; in two parameter configurations, if and only if each The parameter values ​​of a MapReduce parameter are the same, the two parameter configurations are the same, otherwise they are different parameter configurations. [0004] Parameter search space: refers to the set of all parameter configurations that can be used by the parameter search algorithm, called the parameter search space. [0005] Neighbor parameter configuration: refers to the new parameter configuration obtained by changing only one of the Ma...

AI Technical Summary

Problems solved by technology

However, a large number of related research work and practical applications have proved that the MapReduce Job running with the default parameter configuration is often not in the optimal performance state. Performance

However, for Hadoop operation parameter tuning work, it requires relevant tuning personnel to have a deep understanding of the entire Hadoop platform, be familiar with the function of the operating parameters provided by the Hadoop platform, and have certain practical tuning experience. Therefore, for For most people, this parameter tuning work is not only very difficult but also requires a lot of time and effort

At present, in order to solve the problem of Hadoop manual experience tuning is difficult, experts have proposed a variety of solutions, for example, the automatic parameter tuning scheme of the Hadoop simulator based on discrete events, the automatic parameter tuning based on machine training Strategy, etc.

However, these solutions are more suitable for a Hadoop cluster environment that runs a wide variety of applications, and for a Hadoop cluster that only runs a single application, using these existing automatic tuning solutions is obviously not worth the candle , because a large amount of pre-machine training or data collection work will cause a huge waste of resources and time

Images