Method, device, medium, equipment and system for predicting operation performance

Abstract

The invention discloses a job performance prediction method, comprising: acquiring job-related parameters of a big data job; collecting relevant feature parameters of the instantaneous cloud host; establishing a Spark job basic performance prediction model to obtain a Spark job completion time; determining whether an instantaneous cloud host withdrawal event occurs, if not, predicting the Spark job completion time under the resource configuration based on the Spark job completion time and the resource configuration of the instantaneous cloud host; if so, checking the backup mechanism based onthe critical RDD data to evaluate the overhead, and predicting the Spark job completion time under the resource configuration based on the overhead, the Spark job completion time and the resource configuration of the instantaneous cloud host. The invention may perform job performance prediction regardless of whether the instantaneous cloud host has been recalled, and reduce the extra time overheadby checking the backup mechanism through the critical RDD data when the recall event occurs, thereby helping the user to reduce the cost budget. The invention also relates to a job performance predicting device, a medium, equipment and a system thereof.

Description

technical field [0001] The invention belongs to the technical field of job performance prediction for big data processing platform applications, and in particular relates to a big data analysis job performance prediction method, device, medium, equipment and system for instantaneous cloud hosts. Background technique [0002] With the advent of the big data era, big data technology has also been continuously developed and updated. Various big data processing platforms such as Apache Spark, MapReduce, Dryad, etc. have become the main application platforms for big data analysis and processing. In the application scenario of distributed big data analysis, DAG (Directed Acyclic Graph) is a very common computing structure. DAG graph is the abbreviation of directed acyclic graph. DAG type computing refers to the internal decomposition of computing jobs into several subtasks, and the computing logic relationship between subtasks is constructed into a DAG graph. A big data analysis ...

AI Technical Summary

Problems solved by technology

However, these models are often not considered comprehensively. Some models only establish a simple general linear function model on the amount of input data and the number of cloud hosts based on the calculation and data transmission structure, while some models do not consider the impact of task parallelism on data processing rate. impact, and others do not consider the time spent on intermediate data Shuffle

In addition, the existing performance prediction methods can only be used for big data analysis jobs without instantaneous cloud host withdrawal. A model method for predicting job completion time is not yet

Moreover, the huge overhead of recomputing will seriously degrade the job performance. Therefore, a suitable fault tolerance mechanism is also needed to reduce the performance loss caused by the withdrawal of the instantaneous cloud host, and try to ensure the job performance when the cloud host is withdrawn.

Images