Method, apparatus and system for data reconstruction in distributed storage system

Abstract

The invention relates to a method, apparatus and system for data reconstruction in a distributed storage system. A main storage node in the distributed storage system performs EC coding on the data to be written to generate an EC stripe; each EC block on the EC stripe is stored on each storage node; when part of the storage nodes cause failure of writing of EC blocks because of faults, the main storage node stores the EC blocks distributed to the storage nodes which fails to write in the local, and generates metadata information required by data reconstruction; and when the faults of the storage nodes recover, the main storage node retransmit the stored EC blocks distributed to the storage nodes and the metadata information corresponding to the EC blocks to the storage nodes so as to enable the storage nodes after recovering the faults to complete data reconstruction. The scheme for data reconstruction in the distributed storage system has the advantages that when part of the storage nodes malfunction, EC reversal coding is not required to recover the data on the fault nodes, but the buffer of main storage node is distributed to the EC blocks with fault nodes; after the fault nodes recover, the buffered EC blocks are retransmitted to the fault nodes to perform data reconstruction, thus avoiding computing resource consumption because of performing EC reversal coding when the storage nodes recover fault to perform data reconstruction, and avoiding consumption of network resource because large amount of data is transmitted during the EC reversal coding process at the same time.

Description

technical field [0001] The present invention relates to the field of IT technology, in particular to a data reconstruction method, device and system in a distributed storage system. Background technique [0002] In a storage system, in order to ensure data security, a multi-copy storage technology is usually used to implement data redundancy backup. Multi-copy redundancy technology is to store multiple identical copies of a piece of data at the same time. When one piece of data is lost, the lost data can be recovered through the data of other copies, thereby reducing the probability of data loss. The increase in the number of copies will greatly increase the consumption of system storage space and network bandwidth, thereby increasing the cost of data storage. For example, in the case of two copies, the real available space for users is 50% of the total storage space of the entire system, while in the case of three copies, the real available space for users is only 33%. ...

AI Technical Summary

Problems solved by technology

[0005] This application describes a method, device, and system for data reconstruction in a distributed storage system, which solves the problem of rebuilding data on a faulty node after the faulty node is restored, and does not need to use EC back-coding to restore data, reducing computing power. Resource and network resource consumption

Images