Method and device used for realizing cluster dynamic expansion in distributed framework

A distributed and distributed computing technology, applied in the network field, can solve problems such as waste of cluster resources, high cost of cluster scaling, and low degree of intelligence, so as to achieve targeted adjustment, avoid cluster resource shortage, and improve flexibility Effect

Active Publication Date: 2017-03-08
BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF3 Cites 18 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The inventor found in the process of realizing the present invention that the existing mode of adjusting the cluster size and processing capacity needs to interrupt the service in each node of the Hadoop cluster, and the operations such as setting up the server need manual processing, so that the realization cost of the cluster scaling is relatively low. High, and less intelli

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device used for realizing cluster dynamic expansion in distributed framework
  • Method and device used for realizing cluster dynamic expansion in distributed framework
  • Method and device used for realizing cluster dynamic expansion in distributed framework

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0033] Embodiment 1. A method for implementing cluster dynamic scaling in a distributed framework.

[0034] figure 1 It is a flowchart of a method for realizing cluster dynamic scaling in the distributed framework of this embodiment, and figure 1 The shown method mainly includes: step S100 and step S110. The method described in this embodiment is usually executed in a computer device. Preferably, the method described in this embodiment can be executed in a server, a desktop computer, and other network devices. For example, the method is remotely connected to a cluster Executed in servers, desktop computers, and other network devices. next to figure 1 Each step is explained separately.

[0035] S100. Determine that there is a cluster dynamic scaling requirement.

[0036] The cluster in this embodiment may be a Hadoop cluster, or a cluster based on other distributed frameworks. The cluster in this embodiment includes: multiple slave nodes, that is, Slave nodes. The slave no...

Embodiment 2

[0074] Embodiment 2, a method for realizing cluster dynamic scaling in a Hadoop cluster.

[0075] The Hadoop cluster in this embodiment is built through a virtual machine. A specific process of using a virtual machine to build a Hadoop cluster through remote control is as follows: figure 2 shown, and figure 2 The illustrated method includes the following steps:

[0076] S200. Create a corresponding number of virtual machines based on the number of slave nodes included in the preset Hadoop cluster. In the process of creating each virtual machine, the basic information allocated for the virtual machine usually includes: LAN IP address (such as 192.168.0.62 ), the login account of the virtual machine (such as root) and the login password of the virtual machine;

[0077] S210. After successfully creating each virtual machine, continuously try to log in to each virtual machine through remote login to confirm that the network of each virtual machine is connected and available; ...

Embodiment 3

[0090] Embodiment 3, a method for realizing dynamic expansion and contraction of a cluster in a distributed framework.

[0091] The method for realizing cluster dynamic scaling in the distributed framework of this embodiment is executed by a remote control node that remotely communicates with the cluster, and the process of the method in this embodiment is as follows Figure 5 shown.

[0092] Figure 5 In S500, the remote control node receives the computing resource A1 occupied by all computing tasks in the current cluster and the storage space A2 occupied by the data stored by all the slave nodes in the current cluster reported by the master node of the cluster.

[0093] S510. The remote control node acquires the total computing resources Z1 that all slave nodes in the current cluster can provide and the total storage resources that all slave nodes in the current cluster can provide from the locally maintained computing resource group information and storage resource group i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a method and a device used for realizing cluster dynamic expansion in a distributed framework. The method comprises the steps of determining that a cluster dynamic expansion demand exists, wherein the cluster comprises multiple slave nodes, and the multiple slave nodes are divided into a calculation resource group and a storage resource group based on the provided resource services; and adjusting the number of the slave nodes in the calculation resource group and/or the storage resource group based on the cluster dynamic expansion demand. The technical scheme provided by the invention can better match a cluster size and processing capability with a demand for a storage resource and a calculation resource in an actual application scene, accordingly efficiently avoids the phenomenon of cluster resource shortage and cluster resource waste, and further improves the performance of the cluster while improving the flexibility of the cluster dynamic expansion.

Description

technical field [0001] The present invention relates to network technology, in particular to a method for realizing cluster dynamic expansion and contraction in a distributed framework and a device for realizing cluster dynamic expansion and contraction in a distributed framework. Background technique [0002] In the field of distributed technology, Hadoop has become a widely used distributed framework because of the transparency of the underlying details of the system. Hadoop-based cluster (that is, Hadoop cluster) scaling usually refers to the adjustment of cluster size and processing power. [0003] At present, the implementation method of Hadoop cluster scaling is usually: when a new slave node needs to be added in the Hadoop cluster, first configure the server where the slave node is located, and then interrupt the service in each node in the Hadoop cluster, and notify After each node in the Hadoop cluster adds a new slave node, start the service in each node of the Ha...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50H04L29/08
Inventor 周恺王倩肖远昊王家兴张发恩
Owner BEIJING BAIDU NETCOM SCI & TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap