SPARK parameter automatic adjustment and optimization method based on cost model

Abstract

The invention provides a Spark parameter automatic adjustment and optimization method based on a cost model. The method comprises the following steps: 1, constructing a cost-based performance model byobtaining configuration of task execution and corresponding cost information, and obtaining optimal configuration in a given parameter space; 2, for tasks of unknown types, using default parameters for one-time operation, and giving a reference value of optimal configuration by judging the similarity of the tasks. According to the invention, a cost-based performance model is provided for solvingproblems possibly existing in current configuration parameter adjustment and optimization. A performance model is generated by analyzing Spark historical tasks, optimization parameters are obtained through a parameter space search algorithm. Meanwhile, the model is continuously upgraded and adjusted along with running of new tasks, the accuracy of the model is improved, and parameter reference values are provided after the model runs once for tasks of unknown types.

Description

technical field [0001] The invention relates to establishment of a Spark task performance model, space search of configuration parameters of a big data system, and task similarity judgment. Background technique [0002] With the continuous development of science and technology, as small as a mobile phone, a tablet, as large as an astronomical telescope, and the Large Hadron Collider, they are all data producers. massive data. How to store, process, and analyze these data has also become a realistic topic before everyone. Since Google published the Google File System paper in 2003, a number of distributed computing frameworks have emerged, such as hadoop and sparkSpark—cluster computing on working sets. It plays an important role in various application scenarios. In response to the low efficiency of hadoop in iterative processing, Spark came into being. Although spark has a good performance in processing iterative computing tasks, the performance optimization of spark tas...

AI Technical Summary

Problems solved by technology

[0009] For experienced operation and maintenance personnel, when there are few parameters to be adjusted, the task performance can be greatly increased by adjusting the configuration parameters, but this requires very familiarity with the system to achieve, and it is impossible to determine whether there is a better Configuration

But with the increase of parameters, even experienced operation and maintenance personnel, it is difficult to tune by manually adjusting the configuration parameters

[0010] To sum up, the current method is difficult to dynamically upgrade the performance model in the configuration parameter tuning; the generated performance model is greatly affected by the system operating state, and the model cannot be accurately established; manual tuning cannot adapt to the large parameter space

Images