Resource management method and device based on distributed cluster system

A distributed cluster and resource management technology, applied in the field of resource management based on distributed cluster systems, can solve problems such as business impact, server downtime, server load imbalance in the cluster, etc., to ensure normal operation, avoid downtime, The effect of load balancing

Pending Publication Date: 2020-08-25
BEIJING SOGOU TECHNOLOGY DEVELOPMENT CO LTD
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Since the business party does not know the resource usage of its own business in most cases, most of them use the default resource value, so the actual resource consumption of the server may exceed the requested resource during actual operation, and the YARN platform itself cannot perceive the specific resource value. The real load information required for business operation
This resource scheduling method will not only lead to the problem of unbalanced server load in the cluster, but also cause a server to go down due to excessive load, which will have a great impact on the business of the cluster.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Resource management method and device based on distributed cluster system
  • Resource management method and device based on distributed cluster system
  • Resource management method and device based on distributed cluster system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0052] In order to enable those skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention will be further described in detail below in conjunction with the drawings and implementations.

[0053] Cluster resource management is to manage the resources (such as cpu, memory) of all machines in the cluster, so that the resources in the cluster can play a role for the same task, thereby achieving the purpose of speeding up task processing. When a large data task is to be processed, the data blocks are distributed on different servers, and each server is allocated the resources required to process its corresponding data block.

[0054]Aiming at the problems existing in server resource scheduling in the cluster in the prior art, the embodiment of the present invention provides a resource management method and device based on a distributed cluster system, which quantifies the load information of each serve...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a resource management method and device based on a distributed cluster system, and the method comprises the steps: calculating the real load value of each server in a cluster at the current moment; predicting a predicted load value of the server at the next moment according to the real load value of the server at the current moment and the recorded historical load information; determining the load state of the server according to the predicted load value of the server at the next moment; and performing task allocation according to the load state of each server in the cluster. By utilizing the method and the device, the load of each server in the cluster can be effectively balanced, the situation of downtime caused by overhigh load of the load server is avoided, andthe normal operation of services in the cluster is ensured.

Description

technical field [0001] The invention relates to the field of resource management of a distributed system, in particular to a resource management method and device based on a distributed cluster system. Background technique [0002] Hadoop is a distributed system architecture that implements a HDFS (Hadoop Distributed FileSystem, distributed file system). HDFS is designed as a distributed file system suitable for running on general-purpose hardware and can provide high-throughput data access. It is very suitable for applications on large-scale data sets. A general Hadoop cluster is composed of heterogeneous servers of various models, and the configuration of resources such as CPU and memory in each model is usually different. [0003] YARN is Hadoop's resource management and scheduling platform. It adopts a fair resource scheduling method, and performs resource scheduling based on the resource application submitted by the business party and the fixed resources of the Hadoop ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/182G06F9/48G06F9/50
CPCG06F9/4881G06F9/505G06F9/5083
Inventor 申贤强周杜凯
Owner BEIJING SOGOU TECHNOLOGY DEVELOPMENT CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap