An Active Prediction Method for Supercomputer Job Failure Based on Application Similarity

Abstract

The invention discloses an active prediction method for supercomputer job failure based on application similarity, which belongs to the field of supercomputers, and includes steps: S1, extracting feature data from job logs, adding job path data and preprocessing together, and then as The input features of the machine learning algorithm model; S2, after the machine learning algorithm model processes the input feature data, it realizes the active prediction of job failure status. The present invention excavates the characteristics that can accurately describe job application attributes, and has a good prediction improvement effect; uses a machine learning algorithm to find a job failure prediction method, improves the robustness of the prediction model, and is especially suitable for nonlinear data; the job application attributes The clustering method significantly reduces clustering calculation overhead and reduces errors; it achieves high prediction efficiency and can be practically applied to large supercomputers.

Description

technical field [0001] The invention relates to the field of supercomputers, and more specifically, to a method for actively predicting job failures of supercomputers based on application similarity. Background technique [0002] A large number of jobs are submitted and waiting to be executed in the supercomputer, but during the execution of the job, the job may fail due to various reasons, such as system resources cannot meet the job requirements, memory errors, and software and hardware failures. At the same time, job failure will cause waste of system resources, prolong the waiting time of jobs in the queue and other adverse effects. The use of job failure prediction can be used to slow down the impact of these failures. Therefore, how to effectively predict job failure is crucial to improving system reliability and System resource utilization is critical. [0003] At present, there are many prediction methods for software and hardware failures of supercomputers (high-pe...

AI Technical Summary

Problems solved by technology

[0002] A large number of jobs are submitted and waiting to be executed in the supercomputer, but during the execution of the job, the job may fail due to various reasons, such as system resources cannot meet the job requirements, memory errors, and software and hardware failures

At the same time, job failure will cause waste of system resources, prolong the waiting time of jobs in the queue and other adverse effects. The use of job failure prediction can be used to slow down the impact of these failures. Therefore, how to effectively predict job failure is very important for improving system reliability and System resource utilization is critical

[0003] At present, there are many prediction methods for software and hardware failures of supercomputers (high-performance computing systems), but the research on prediction methods for job failures is relatively scarce, and some statistical methods, such as linear analysis and secondary discriminant analysis, are mainly used for prediction. Job failure, the core idea of ​​this type of method is to try to find the linearly separable relationship of job failure, but the effect is not ideal, because these methods require a large number of data samples and the calculation efficiency is not high

In addition, the characteristics used to predict failure are mostly resource and performance attributes, which are complex and changeable, and cannot accurately describe the application characteristics of the job, which is why the prediction method using linear analysis thinking is not ideal.

Images