Fast distributed-storage multi-replica consistency checking method and storage architecture

Abstract

The invention discloses a fast distributed-storage multi-replica consistency checking method and a storage architecture. The method adopts a control host-storage host processing architecture, and includes: evenly dividing a stored file into a plurality of data segments in advance, and respectively setting an individually-corresponding first hash value and a mark bit, which is used for indicating whether the corresponding first hash value expires, for each data segment; when a writing request is received, calculating the corresponding mark bit according to offset and length of the writing request, and setting the mark bit to expiry; and screening out the expiry mark bit, updating the first hash value corresponding to the mark bit, and then calculating a second hash value of the entire fileaccording to the first hash value of each data segment. According to the method, the large file is divided into the plurality of data segments, the hash values of the file are calculated in a segmented manner, then the hash value of the entire file is calculated through the hash value of each segment, reading data of the entire file is avoided, thus a consistency detection speed is increased, andbandwidth consumption of storage hosts is reduced.

Description

technical field [0001] The present invention relates to the technical field of information storage, in particular to a method and a storage architecture for quickly verifying the consistency of multiple copies of distributed storage. Background technique [0002] With the advent of the information age, the amount of global data is in an explosive growth trend. Improving the reliability of storage systems and ensuring data availability have become the research focus of enterprises. Most of the existing distributed storage systems use multi-copy technology to improve system reliability, availability, performance and scalability. However, the distributed storage system communicates through the network, and the instability of the network may easily cause inconsistency of the back-end data, and the distributed storage system generally includes a large number of server hosts and disks, and the probability of hardware damage is relatively high. [0003] If the consistency of the ...

AI Technical Summary

Problems solved by technology

[0004] But for large files, calculating the hash value will consume a lot of CPU and storage host bandwidth, seriously affecting the performance of the system

In addition, there are often few inconsistencies in the file, but calculating the hash value of the file requires reading the contents of the entire file, resulting in a huge waste of resources

Images