Redundant data marking and removing method

Abstract

The invention relates to a redundant data marking and removing method, and belongs to the technical field of data storage. The method comprises the following steps: when a file is written, performing dynamic variable-length segmentation on the file to form a plurality of data blocks with different lengths; grouping the plurality of data blocks to obtain a data block group, and calculating the Bloom value of each data block and the Bloom value of the data block group; processing the Bloom value of the data block to form a characteristic value of the data block; judging whether the characteristic value of the data block exists in a metadatabase or not; if the characteristic value of the data block exists in the metadatabase, calculating the Bloom value of the data block again, comparing the Bloom value of the data block with the Bloom value of each data block group in the metadatabase, positioning a similar group of the data block, and determining a redundant data block; and marking the redundant data blocks, and deleting or retaining the redundant data blocks according to a preset strategy. The method has the advantages of high redundancy recognition rate, high reliability, high robustness and less resource occupation.

Description

technical field [0001] The invention belongs to the technical field of data storage, and in particular relates to a method for marking and removing redundant data. Background technique [0002] Duplication-mark is a data reduction technique designed to reduce the storage capacity used in a storage system. It eliminates excessive redundant data by finding redundant variable-sized data blocks in different locations in different files, marking and deleting redundant data in the data storage system, and retaining only one or a few necessary copies. Redundant data marking technology can greatly reduce the consumption of physical storage space, improve the efficiency of services such as retrieval, save transmission bandwidth, and allow users to perform efficient and economical backup data replication between different sites. [0003] Redundant data marking and removal technology can be divided into pre-processing (Pre-Process), online processing (Online-Process) and post-processi...

AI Technical Summary

Problems solved by technology

The advantage of byte-level deduplication is that the deduplication rate is high, and the disadvantage is that the deduplication efficiency is low, the speed is slow, and the difference content accounts for a large proportion of the content, and it will also occupy a large amount of system storage resources. It is usually used in the post-redundancy method.

[0013] In addition, considering the widespread use of distributed storage systems, traditional redundant data identification methods are not applicable in terms of performance and accuracy, and there are certain defects.

Images