Distributed resource elastic scheduling model and method, electronic equipment and storage medium

Abstract

The invention relates to the field of data processing application. The invention discloses a distributed resource elastic scheduling model and method, electronic equipment and a storage medium. The distributed resource elastic scheduling model and method realize cooperative work of task layer dynamic resource scheduling and system layer resource scheduling, give a response to the resource demand of the stream processing task in real time, allocate reasonable physical resources, achieve automatic elastic resource scheduling in the true sense, elastically increase and decrease physical resourcesoccupied by the distributed stream processing system, meet the resource requirement that stream processing tasks change at any time, guarantee the stable operation of the distributed stream processing system and the stable and efficient work of the stream processing system, and meanwhile, increase the resource utilization rate and reduce the cost.

Description

technical field [0001] The invention relates to the application field of data processing, in particular to a distributed resource flexible scheduling model, method, electronic equipment and storage medium. Background technique [0002] With the continuous integration of technologies such as the Internet of Things, social networks, and cloud computing into our lives and the rapid development of existing computing power, storage space, and network bandwidth, various changes have been brought to people's lives. , Finance, business, medical and many other fields continue to grow and accumulate. Since the accumulation of such data often exceeds the capacity of traditional data processing technologies, it is difficult for a single machine to satisfy such a large amount of calculation, and expensive equipment is not our first choice, and the distributed stream processing mode and distributed stream processing system are thus born. As an application that processes massive real-time...

AI Technical Summary

Problems solved by technology

However, the system-level resource scheduling of a distributed stream processing system cannot dynamically and flexibly schedule tasks according to the running status of tasks, and requires manual intervention to add or reclaim resources. That is, the system-level resource scheduling cannot perceive the running status of specific stream processing tasks, and can only be statically for resource allocation

Although task-level scheduling can be flexibly increased or decreased, it is limited by fixed physical resources

When the physical resources are insufficient, the decision made by the task-level scheduling cannot be executed, and the resource scheduling fails, which may cause the system to degrade or even crash

In addition, although system-level scheduling can accurately allocate resources at the initial stage, and manually schedule resources at runtime, it cannot achieve automatic elastic increase or decrease of physical resources. Such resource scheduling cannot meet the needs of real-time systems.

Images