Automatic optimization method for Apache Spark application based on historical task analysis

Abstract

The invention provides an automatic optimization method for an Apache Spark application based on historical task analysis. The method comprises the steps that a task submission interface of the Sparkapplication is encapsulated, whether or not a layered grey box time prediction model of the Spark application exists is judged, a database is accessed, the layered grey box time prediction model is read and updated, and a user selects whether or not optimization is conducted; if yes, an optimization parameter is generated, and if not, execution is conducted according to an original parameter, anda shell command running task is called. According to the automatic optimization method, known cluster hardware resource information is sufficiently used, an efficient search scheme for an optimal parameter on a high-dimension parameter space is selected and achieved, a configuration parameter with excellent performance is screened out under the established layered grey box time prediction model, and the aim of automatically optimizing a Spark parameter is achieved.

Description

technical field [0001] The invention belongs to the technical field of computer software, in particular to an Apache Spark application automation tuning method based on historical task analysis. Background technique [0002] Apache Spark is a widely used open source big data general processing platform. It has the advantages of simple programming and fast calculation speed, so it is widely used in the industry. However, since the Apache Spark system has about 150 configuration parameters, the values ​​of these parameters will affect the performance and performance of the system, so you need to be cautious about their configuration. Considering the complexity of the Spark system itself and the potential interaction between Spark system parameters, it is not very realistic to rely solely on manual parameter tuning. Moreover, manual tuning cannot make individual adjustments for each task according to its characteristics, which makes automatic parameter tuning technology of con...

AI Technical Summary

Problems solved by technology

However, the machine learning framework they proposed is relatively simple, and completely treats the performance of the system as a black-box model. There is room for improvement in the efficiency of experimenting with parameter selection and model accuracy, and it does not make full use of the resources of the Spark resource manager. Allocation patterns and some known cluster hardware information

Images