Hadoop Yarn (Yet another Resource Negotiator) network bandwidth resource scheduling and isolating method and device

Abstract

The invention discloses a Hadoop Yarn (Yet another Resource Negotiator) network bandwidth resource scheduling and isolating method and device. The scheduling method is performed at an RM (Resource Manager) side and comprises the steps of receiving work resource requests sent by an AM (Application Master), wherein the work resource requests comprise network bandwidth resource demand quantity of each task in work; and correspondingly allocating a container to each task through adoption of a preset fair scheduling algorithm according to the network bandwidth resource demand quantity of each task, wherein the network bandwidth resource demand quantity of the corresponding task is packaged in each container. The isolating method is performed at an NM (Node Manager) side and comprises the steps of obtaining a plurality of containers from the Am, wherein the network bandwidth resource demand quantity of the corresponding task is packaged in each container; and isolating a network bandwidth resource occupied by each task according to the network bandwidth resource demand quantity of each task. According to the methods and the devices, the scheduling for the network bandwidth resources is increased in a Yarn cluster, so the mutual interference among the tasks resulting from competition for the network bandwidth resources can be avoided.

Description

technical field [0001] The invention relates to the technical field of big data computing, in particular to a Hadoop Yarn network bandwidth resource scheduling and isolation method and device. Background technique [0002] Yarn (Yet Another Resource Negotiator, another resource coordinator) is a general resource management system responsible for resource allocation and task scheduling of Yarn clusters. Yarn mainly includes three components: RM (ResourceManager, resource manager), NM (Node Manager , Node Manager) and AM (Application Master, Application Manager). [0003] In Yarn, the unit of representation of resources is a container (Container), which is a part decomposed from MRv1. Container is an abstraction of resources in Yarn, which encapsulates a certain amount of computing resources and storage resources on a node, such as CPU and memory. Container is the division unit of dynamic resources. AM applies for resources from RM, and the scheduler in RM allocates Containe...

AI Technical Summary

Problems solved by technology

[0005] In the field of big data, Yarn needs to become a general resource management system, establish a comprehensive resource management and control network, and manage various resources in the cluster, including memory, CPU, network bandwidth, disk, etc., but currently, Yarn only supports The management of memory and CPU does not support the management of network bandwidth; moreover, since there are still many resources in the Yarn cluster that are not managed by Yarn, in the multi-tenant and multi-task shared Yarn cluster, the existence of these unmanaged resources will cause Affect the robustness of the cluster: non-shared occupation of shared resources; DoS (Denial of Service) attacks, such as malicious users running applications-occupying as few restricted resources as possible and as many unrestricted resources as possible, so that Overcommitted resources become the bottleneck of the cluster

Images