Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Dynamic load balancing method for distributed file system under cloud environment

A distributed file and load balancing technology, applied in transmission systems, electrical components, etc., to achieve the effect of improving the degree of balance and task execution efficiency, improving efficiency, and reducing human subjectivity

Inactive Publication Date: 2018-06-22
NANJING UNIV OF POSTS & TELECOMM
View PDF2 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although this algorithm can be used as an effective algorithm for load balancing, among the parameters affecting the algorithm, the determination of the threshold λ is subjectively determined by the user according to the current disk usage status, which causes artificial errors in the balancing process. Subjectivity has a great impact on the balance effect, and the algorithm needs to be called by the user to realize it. In the process of operating the distributed file system, it is difficult for us to subjectively judge the load level of the cluster and how to determine the threshold

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Dynamic load balancing method for distributed file system under cloud environment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] The implementation of the technical solution of the present invention will be described in further detail below in conjunction with the accompanying drawings:

[0038] Aiming at the unbalanced load problem of the distributed file system in the cloud environment, the present invention proposes a balancing algorithm D-balancer that can dynamically obtain the threshold value. Use dynamic thresholds to re-adjust the load of the cluster to improve load balance and task execution efficiency.

[0039] Such as figure 1 As shown, the method includes the following steps:

[0040] Step 1: Obtain node information

[0041] The information of node i mainly includes disk space utilization, CPU utilization, memory utilization, disk I / O occupancy, and network bandwidth occupancy, respectively denoted as P diski ,P cpui ,P memi ,P ioi ,P bandwidthi . The steps to obtain are as follows:

[0042] 1.1. View the cluster status through hdfs dfsadmin–report to obtain the disk space us...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a dynamic load balancing method for a distributed file system under a cloud environment. The method mainly comprises the steps of acquiring information of all nodes of the distributed file system under the cloud environment, judging whether the file system is balanced, computing a threshold needed when the system is balanced according to a disk space utilization rate, a CPUutilization rate, a memory utilization rate, a disk I / O occupancy rate and a network bandwidth occupancy rate of each node, and performing unbalance adjustment on the load of the file system according to the threshold and the disk space utilization rate. The method provided by the invention balances the load of the file system while supporting the execution of a cloud computation task, and continuously adjusts the load via monitoring information of the nodes, and thus the execution efficiency of cloud computation to the file system is improved.

Description

technical field [0001] The invention relates to a method for dynamically balancing the load of a distributed file system in a cloud environment, and belongs to the field of big data cloud computing. Background technique [0002] With the rapid development and popularization of computer and information technology, the scale of industrial application systems is rapidly expanding, and the data generated by industrial applications is growing explosively. The amount of accumulated data is uploaded and stored in distributed file systems. Effectively utilizing and processing these data while ensuring the load balance of the distributed file system has become one of the research hotspots of the distributed file system. [0003] As a typical representative of distributed file system, HDFS is a scalable master-slave distributed file system. In the process of data storage in HDFS, the storage of data blocks is random. This storage method can easily cause unbalanced data storage among ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L29/08
CPCH04L67/1008H04L67/1029
Inventor 杨庚吴瑶瑶白双杰刘国秀马可
Owner NANJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com