Resource management method and device based on Hadoop cluster, equipment and storage medium

Abstract

The invention discloses a resource management method and device based on a Hadoop cluster, equipment and a storage medium, and belongs to the technical field of big data. The method comprises: obtaining resource use information data of all queues in the Hadoop cluster in a preset time period; inputting the resource use information data into a pre-trained resource use condition prediction model; judging the running state of the queue according to the prediction result; calculating the ratios of the queues in the congestion state in all the queues to obtain the proportion of the queues in the congestion state; and judging whether the proportion of the congestion state queue is greater than a preset threshold, and if so, adjusting the queue resources of the Hadoop cluster according to the ratio of the congestion state queue. According to the invention, the resource adjustment mode is determined by comparing the relationship between the congestion state queue proportion and the preset threshold, so that the scheme of queue resource adjustment is more flexible. The invention also relates to a blockchain technology, and the resource use information data can be stored in the blockchain node.

Description

technical field [0001] The present application belongs to the technical field of big data, and in particular relates to a Hadoop-based cluster resource management method, device, equipment and storage medium. Background technique [0002] With the rapid development of the Internet, the amount of network data is increasing exponentially. At present, more and more enterprises choose to build their own data warehouses based on Hadoop (Hadoop is a distributed system infrastructure developed by the Apache Foundation) to manage enterprises. Network data, in order to meet the data processing needs of enterprises in various application scenarios, it is often necessary to set up multiple queues on the same Hadoop cluster, but this can easily lead to a disordered state of Hadoop cluster resources, so it is necessary to establish a unified scheduling The platform realizes the unified management and scheduling of data resources. [0003] In practical applications, Hadoop generally uses...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the embodiment of the present application is to propose a method, device, computer equipment, and storage medium based on Hadoop cluster resource management, so as to solve the problem that existing Yarn scheduling can only perform fair scheduling when scheduling Hadoop cluster queue resources, resulting in Hadoop The problem that the flexibility of cluster queue resource adjustment is not high

Images