Distributed storage-based dynamic metadata optimizing method and system

Abstract

The invention discloses a distributed storage-based dynamic metadata optimizing method and system. The method comprises the following steps: obtaining the load of mds (metadata server) nodes; obtaining the average load of mds clusters; calculating the processing power on the basis of the memory of the mds nodes and a CPU (Central Processing Unit) to obtain corresponding weighting coefficients; and judging whether the time when the load of the mds nodes is greater than the average load exceeds the threshold or not, and if yes, migrating data of the mds nodes of which the load is greater than the average load to the mds nodes of which the load is lower than the average load. The processing power is calculated on the basis of the memory of the mds nodes and the CPU to obtain the corresponding weighting coefficients, so that the data processing power of all the mds nodes is evaluated more accurately; regional division is carried out on the metadata server; when sub-trees are migrated, the load and the machine performance are comprehensively considered; the selection strategy of sub-tree migration targets is optimized on the basis of the data processing power and the current load of each mds node; and therefore, the network flow and the migration time are reduced, and the migration efficiency is increased.

Description

technical field [0001] The invention relates to the field of computer hardware, in particular to a method and system for optimizing dynamic metadata based on distributed storage. Background technique [0002] Distributed File System (Distributed File System) means that the physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network. The design of the distributed file system is based on the client / server model. A typical network may include multiple servers accessed by multiple users. In addition, the peer-to-peer feature allows some systems to perform the dual roles of client and server. For example, a user can "publish" a directory that other clients can access, and once accessed, the directory appears to the client as if it were using a local drive. Therefore, the distributed file system can be a storage platform for applications in high-performance computing, da...

AI Technical Summary

Problems solved by technology

However, when high-configuration, high-load nodes are migrated to low-configuration, low-load nodes, and the low-configuration mds nodes receive too much load, system performance will become a bottleneck, although the next round of balancing will reduce the low-configuration nodes Requests that cannot be processed are re-migrated to high-configuration nodes, but this process wastes network traffic and migration time; the goal of the subtree migration algorithm is global load balancing, and in the case of large network delays, this goal will cost a lot the price of

Images