Sharding storage method for data, device and system

Abstract

The invention relates to the field of distribution and especially relates to a technology of distributed sharding storage. The sharding storage method for data in a distributed storage system comprises the steps of determining M data nodes required for storing to-be-stored data; obtaining N copies of the to-be-stored data; carrying out sharding on each copy in the N copies according to the same sharding mode, thereby obtaining X data shards; storing the to-be-stored data in the M storage nodes, namely storing the N copies of each data shard in the X data shards in N storage nodes, wherein the number of the data shards of the data shard copies stored in the same N storage nodes is P or P+1, and the P is an integer quotient obtained by X / C [N ] [M ]. According to the distributed data sharding storage method provided by the invention, the data availability is improved, and the data recovery efficiency when faults occur in the nodes is improved.

Description

technical field [0001] Embodiments of the present invention relate to the field of distributed storage, and more specifically, to a method, device and system for fragmented storage of data in a distributed storage system. Background technique [0002] With the rapid development of information technology, the amount of data in the information system database is increasing. In order to meet the storage requirements of large amounts of data, distributed storage systems running on multiple servers have been widely used. In a distributed storage system, multiple database systems run on multiple servers. When data is stored, it is necessary to shard the data first, and then assign different data shards to different servers for storage. Fragmentation is a horizontal scaling method that distributes a large data set to multiple data nodes, and all data nodes will form a logical database to store this large data set. Sharding is transparent to the user (application layer), and the ...

AI Technical Summary

Problems solved by technology

However, when all the data nodes where the same data shard is located are faulty, it may happen that the primary and backup data of the two data shards are stored on the same multiple nodes. For example, in the example in Table 1, the data Fragment A and data fragment C are stored in nodes 1, 2, and 3. When the three data nodes fail, data fragments A and C will be lost

In addition, when a single node fails, a new node needs to be formed through data recovery, and the efficiency of concurrent recovery is not high during data recovery. For example, in the example in Table 1, when node 6 fails, at most through A node that stores F, B, and E data fragments, such as nodes 3, 4, and 5, implements data recovery concurrently, while other nodes cannot participate in the recovery

Images