Data redundancy processing method, device and distributed storage system

Abstract

A method and device for processing data redundancy and distributed storage system are provided. The method comprises: generating M pieces of slice data and N pieces of redundancy slice data of data to be written by using a redundancy algorithm (403), wherein when at most N pieces of data among said M pieces of slice data and N pieces of redundancy slice data are arbitrarily damaged, the damaged data can be recovered by using the undamaged data, with M being a natural number which is more than 1, and N being a natural number which is not less than 1; and respectively storing said M pieces of slice data and N pieces of redundancy slice data into objects located in at least two storage nodes (404), wherein each storage node contains at least one object set, and each object set contains s at least one object. The method can improve utilization rates of storage spaces and storage loads are reduced, moreover, different redundancy algorithms can be supported, thus different requirements for reliability of customers can be met and the flexibility is relatively higher.

Description

technical field [0001] Embodiments of the present invention relate to storage technologies, and in particular, to a data redundancy processing method, device, and distributed storage system. Background technique [0002] A distributed storage system includes multiple storage nodes interconnected through a network, and data can be distributed and stored on different storage nodes to achieve large-capacity, high-reliability, and easy-to-expand storage applications. [0003] Existing distributed storage systems can use object-based storage. For this storage method, the prior art adopts a multi-copy mechanism among storage nodes to ensure the reliability of data storage. Specifically, the multi-copy mechanism is to use one or several storage nodes in the distributed storage system as backup nodes of the main storage node. When writing data, it is necessary to write multiple copies of the same data on these storage nodes, so as to realize redundant backup of data. When a storag...

AI Technical Summary

Problems solved by technology

[0004] However, the above-mentioned existing multi-copy mechanism simply writes the same data to multiple storage nodes. If it is necessary to ensure that the data is still available after N storage nodes are damaged, at least N+1 copies of the data need to be repeatedly written, and the storage The space utilization rate of the node is at most 1 / (N+1), therefore, the utilization rate of the storage space is low, and the load of the storage node increases; moreover, this multi-copy mechanism cannot support various types of redundant backup methods, and is flexible lower sex

Images