Hadoop platform computing node load prediction method

Abstract

The invention provides a Hadoop platform computing node load prediction method. The Hadoop platform computing node load prediction method comprises: a data preprocessing method based on a sliding window secondary detection algorithm; a node load linear prediction method based on an ARIMA algorithm; a node load nonlinear residual prediction method based on an RNN algorithm; performing linear addition on results predicted by the ARIMA algorithm and the RNN algorithm to obtain a final prediction result. According to the method, through analyzing historical data of each settlement node, valuable information can be extracted, so that the load of the computing node in the next time period is reasonably predicted, and the accurate prediction of the load of the computing node can provide a basis for the resource manager to reasonably allocate resources to the AppMaster, so that the pressure of the high-load node is relieved, the computing resource utilization rate of the low-load node is improved, and the reliability and the performance of the Hadoop cluster are improved. According to the method, the ARIMA model and the RNN model are combined, so that the load can be predicted more accurately.

Description

technical field [0001] The invention relates to the fields of distributed, big data and cloud computing, and in particular to a Hadoop platform computing node load prediction method. Background technique [0002] In the Hadoop platform, as the amount of tasks submitted by users changes, the load of each computing node changes accordingly, and the load of computing nodes has significant differences in different time periods. Through the analysis of the historical data of each settlement node, valuable information can be extracted, and then the load of the computing node in the next period can be reasonably predicted. Accurate prediction of the load of the computing node can provide a basis for the resource manager to allocate resources to AppMaster reasonably , thereby alleviating the pressure of high-load nodes, improving the utilization of computing resources of low-load nodes, and improving the reliability and performance of Hadoop clusters. Load information can be regard...

AI Technical Summary

Problems solved by technology

The load information can be regarded as a time series. The time series is composed of linear features and nonlinear features. The linear features can be extracted by the traditional ARIMA model, but the nonlinear features are difficult to extract. Therefore, it is difficult for the traditional forecasting model to accurately fit the time series.

Images