Data archiving method for load sensing in erasure code storage

Abstract

The invention discloses a data archiving method for load sensing in erasure code storage. The data archiving method comprises the steps: acquiring a load weight value of each node of a production cluster in an erasure code storage cluster, storing the load weight values into an array and according to the load weight values of the nodes, determining archiving node sets of different bands in the production cluster; initializing the archiving node set of an ith band into an empty set, selecting a node with the highest load weight value from a data distribution set of the ith band, judging whether the load weight value of the node is greater than the number of data blocks of the node, and if yes, filling the data blocks corresponding to the node into the archiving node set of the band and updating the load weight values of the related nodes; if no, selecting a node with the second highest load weight value from the data distribution set of the band. The data archiving method solves the problem of performance bottleneck caused by the case that in an existing method, code nodes do not consider node weight values and all data blocks on bands are only acquired from fixed nodes.

Description

technical field [0001] The invention belongs to the technical field of computer storage, and more specifically relates to a load-aware data archiving method in erasure code storage. Background technique [0002] Distributed storage clusters use redundant data provided by unreliable single nodes to ensure data reliability. In distributed storage environments such as GFS, HDFS, and amazon S3, a three-copy redundancy method is adopted. In the case of three copies, data reliability and load balancing characteristics can be well obtained. In a large-scale cluster, the defect brought by three copies is that the storage efficiency is low, and the storage cost is too high. For the cluster environment where data is written once and read many times, erasure code data archiving can effectively improve storage efficiency. In a large-scale cluster environment, nodes in the cluster are heterogeneous due to different I / O service requests. During the archiving process, the node with too ...

AI Technical Summary

Problems solved by technology

[0007] Aiming at the above defects or improvement needs of the prior art, the present invention provides a load-aware data archiving method in erasure code storage. It becomes a performance bottleneck in the process of single-strip archiving, and all archiving encoding tasks are completed by the archiving management node. The management node becomes the performance bottleneck of the entire archiving process.

Images