Supercomputer job failure active prediction method based on application similarity

Abstract

The invention discloses a super computer operation failure active prediction method based on application similarity, and belongs to the field of super computers, and the method comprises the steps: S1, extracting feature data from an operation log, adding operation path data, preprocessing the feature data and the operation path data, and taking the preprocessed data as input features of a machine learning algorithm model; and S2, after the machine learning algorithm model processes the input feature data, the operation failure state is actively predicted. According to the method, the characteristics capable of accurately describing the job application attributes are mined, and a good prediction improvement effect is achieved; a machine learning algorithm is adopted to find an operation failure prediction method, the robustness of a prediction model is improved, and the method is especially suitable for nonlinear data; the clustering calculation overhead is remarkably reduced and the error is reduced by adopting a clustering method for the job application attributes; the prediction efficiency is high, and the method can be practically applied to large supercomputers.

Description

technical field [0001] The invention relates to the field of supercomputers, and more specifically, to a method for actively predicting job failures of supercomputers based on application similarity. Background technique [0002] A large number of jobs are submitted and waiting to be executed in the supercomputer, but during the execution of the job, the job may fail due to various reasons, such as system resources cannot meet the job requirements, memory errors, and software and hardware failures. At the same time, job failure will cause waste of system resources, prolong the waiting time of jobs in the queue and other adverse effects. The use of job failure prediction can be used to slow down the impact of these failures. Therefore, how to effectively predict job failure is crucial to improving system reliability and System resource utilization is critical. [0003] At present, there are many prediction methods for software and hardware failures of supercomputers (high-pe...

AI Technical Summary

Problems solved by technology

[0002] A large number of jobs are submitted and waiting to be executed in the supercomputer, but during the execution of the job, the job may fail due to various reasons, such as system resources cannot meet the job requirements, memory errors, and software and hardware failures

At the same time, job failure will cause waste of system resources, prolong the waiting time of jobs in the queue and other adverse effects. The use of job failure prediction can be used to slow down the impact of these failures. Therefore, how to effectively predict job failure is very important for improving system reliability and System resource utilization is critical

[0003] At present, there are many prediction methods for software and hardware failures of supercomputers (high-performance computing systems), but the research on prediction methods for job failures is relatively scarce, and some statistical methods, such as linear analysis and secondary discriminant analysis, are mainly used for prediction. Job failure, the core idea of ​​this type of method is to try to find the linearly separable relationship of job failure, but the effect is not ideal, because these methods require a large number of data samples and the calculation efficiency is not high

In addition, the characteristics used to predict failure are mostly resource and performance attributes, which are complex and changeable, and cannot accurately describe the application characteristics of the job, which is why the prediction method using linear analysis thinking is not ideal.

Images