Self-adaptation data storage and reconstruction method for coding redundancy storage system

Abstract

The invention provides a self-adaptation data storage and reconstruction method for a coding redundancy storage system. The self-adaptation data storage and reconstruction method includes the following steps: (1) a client terminal calculates a Hash value of a file to be stored and uploads the Hash value to a server terminal; (2) the Hash value of the file to be stored is compared with Hash values of files already stored in the server terminal; (3) if the Hash value equal to the Hash value of the file exists, the server terminal does not accept uploading of the file to be stored, if the Hash value equal to the Hash value of the file does not exist, the server terminal accepts uploading of the file to be stored, the uploaded file is partitioned, Hash values of the file partitions are calculated and stored, and the file partitions are encoded to generate verification data partitions. Compared with the prior art, the self-adaptation data storage and reconstruction method has the advantages that the Hash value of the file to be stored and the Hash values of the file partitions are recorded, the corresponding file storage and reconstruction method is selected according to conditions of the system and the client terminal, and therefore network bandwidth pressure and calculation pressure, caused by data reconstruction, on a data center are reduced.

Description

technical field [0001] The invention relates to the field of information technology, in particular to an adaptive data storage and lost data reconstruction method of a data storage system using a coding redundancy strategy as a basic storage architecture. Background technique [0002] Compared with replication redundancy technology, reliability technology based on coding redundancy has lower data redundancy and storage overhead under the premise of the same fault tolerance. However, when a node is damaged or a data block is lost in the system, the storage strategy based on replication redundancy only needs to download the same amount of data as the lost data to realize the repair process, but based on coding redundancy, such as the redundancy of erasure code The strategy needs to download at least k times the amount of lost data to decode and reconstruct the lost data. Therefore, compared with the replication redundancy strategy, the erasure code redundancy strategy will oc...

AI Technical Summary

Problems solved by technology

Therefore, compared with the replication redundancy strategy, the erasure code redundancy strategy will occupy more network bandwidth resources when restoring files, which will put greater pressure on the already tight network bandwidth resources in the data center. In turn, it has a greater performance impact on data reading

It is precisely because the erasure code redundancy strategy requires greater maintenance bandwidth in the data repair process, coupled with the need for more complex management strategies, which greatly limits the application and promotion of coding-based fault-tolerant technology

Images