Distributed storage method, device and system

Abstract

Provided are a distributed data storage method, device and system. The method comprises: splitting a data file, generating K data slices, splitting each data slice of the K data slices, generating M data blocks for each data slice, performing check encoding on the M data blocks using a redundancy algorithm, and generating N check blocks (110); determining a first physical storage node corresponding to a block of the M data blocks and the N check blocks using a random algorithm, and determining at least M+1 different physical storage nodes according to a first ranking manner conforming to a rule, based on the first determined physical storage node (120); and respectively storing at least M+1 blocks of the M data blocks and the N check blocks on the at least M+1 different storage nodes, with K, M and N being integers (130). The method, device and system can improve the reliability of distributed storage systems.

Description

technical field [0001] The present invention relates to the field of data storage, in particular to a distributed storage method, device and system. Background technique [0002] In a distributed storage system, in order to achieve reliability, file-level multi-copy redundancy technology or data block-level redundancy coding technology, such as erasure coding (Erasure Coding) technology, is used. In multi-copy redundancy technology, the probability of data loss can be reduced by storing multiple identical copies of a data file, while in redundant coding technology, a check block can be added to any local data in the file to improve reliability. [0003] Usually, a distributed hash table (Distributed Hash Table, DHT) can be used to store data blocks and check blocks. However, due to the randomness of DHT, it is unavoidable to deploy multiple data blocks of the same data slice on the same physical storage node, so it is unavoidable that a single physical storage node (for ex...

AI Technical Summary

Problems solved by technology

However, due to the randomness of DHT, it is unavoidable to deploy multiple data blocks of the same data slice on the same physical storage node, so it is unavoidable that a single physical storage node (for example, when a rack, server, or hard disk) fails to cause data risk of loss

For example, using M+N erasure coding technology, where M is the number of data blocks and N is the number of check blocks, when more than N+1 data blocks or check blocks are deployed on the same hard disk, The hard disk failure will result in the loss of the above M data blocks, which may cause the entire file to be unavailable

Taking the 12+3 redundant storage mechanism as an example, when more than 4 data blocks are lost, the data slice will be lost and cannot be recovered

[0004] In other words, in existing distributed storage systems, failure of a single point (for example, hard disk, server, or rack) may result in data loss, especially when the scale of the distributed storage system is small, the risk and probability of failure are particularly Large, thus reducing the reliability of the distributed storage system

Images