Method and system for predicting task resource waiting time

Abstract

The invention discloses a method and a system for forecasting task resource waiting time and relates to the large-scale calculating system resource management, optimization and distribution, in particular to the method and system for forecasting the task resource waiting time. The method for forecasting the task resource waiting time includes that acquiring historical task records, deleting task records with dependency relationships from the historical task records, and generating new historical task records; acquiring task records in the time period related to the time period to be forecast from the new historical task records through self-correlation functions to generate a task record set; setting a task resource waiting time threshold, acquiring the number of the task records of which the task resource waiting time is longer than the task resource waiting time threshold from the task record set, and forecasting the task resource waiting time in the time period to be forecast through a bayesian method according to the total amount of the task records in the task record set. The method and system for forecasting the task resource waiting time are capable of forecasting the usability of the calculating system resource and optimizing the task dispatching.

Description

technical field [0001] The invention relates to resource management, optimization and allocation of large-scale computing systems (including supercomputing and cloud computing), and in particular to a method and system for predicting the waiting time of task resources. Background technique [0002] Many existing approaches use model-based approaches to characterize, predict, optimize, and allocate resource usage in large-scale computing systems. Specifically, researchers first select one or several dimensions related to system resource usage for data tracking observation (such as the remaining running time of jobs, the queuing time of jobs in system queues, etc.), and then apply some correlation A probability model is used to describe the probability distribution of such dimensional data. Next, researchers use the probability distribution properties presented by this model to predict the future performance of the system, so as to achieve resource optimization and reasonable ...

AI Technical Summary

Problems solved by technology

[0003] The method of describing resource distribution based on a specific probability model will have the following practical problems in the practical application of large-scale computing system management and big data processing: the historical load records of massive data are difficult to obey a specific probability in practical applications In fact, by testing different actual data, the historical load records of big data not only do not obey a single probability distribution, but are even difficult to describe with a mixed probability distribution model; some commonly used probability models (such as the binomial distribution model ( binomial model) and its derived models), the assumption that the properties of jobs submitted by users in a short time interval are independent of each other is often not true in reality. In fact, through specific research on actual supercomputer historical load data, we found that most users Jobs with similar content and equivalent parameters will be submitted multiple times in a short period of time, so it is often inaccurate to use this type of probability model to evaluate and predict resource consumption

Images