Concurrent hierarchy type replicated data eliminating method and system

Abstract

The invention discloses a concurrent hierarchy type replicated data eliminating method and a system. The method comprises the following steps: an input device receives data input externally and puts the data into a shared buffer queue; a plurality of blocking devices concurrently acquire the data from the buffer queue and carry out blocking, and inputs the blocks to a plurality of coarse grain duplication removers to carry out coarse grain duplication removal; and the coarse grain duplication removers carry out coarse grain duplication removal and judge whether the data blocks are duplicated, and if so, the index information of the duplicated data blocks is written into a memory by using a data read-write subsystem; and else, fine grain duplication removers carry out the fine grain duplication removal on the non-duplicated data blocks, and store the data blocks subject to duplication removal and the index information thereof into the memory through the data read-write subsystem.

Description

technical field [0001] The present invention relates to the field of information security, in particular to a concurrent hierarchical data deduplication method and system that can effectively utilize multi-core computing resources. Background technique [0002] With the continuous improvement of the degree of informatization, the amount of data continues to explode. According to statistics, in 2002 the world produced 5 EB of data, and it grew rapidly at a rate of 30% per year. It is estimated that by 2010, the total amount of global data will exceed 988 EB. At the same time, the importance of data continues to increase, and more and more data needs to be stored centrally through archiving and backup. According to the statistics of Enterprise Strategy Group (ESG), the amount of archiving and backup data is growing rapidly at a rate of 60% every year. , the scale has reached the PB level and will soon grow to hundreds of PB levels; the amount of data in the backup system is u...

AI Technical Summary

Problems solved by technology

[0005] By reducing the deduplication granularity, the deduplication rate can be significantly improved, but it will lead to a significant increase in computing overhead, especially when the byte-level deduplication mechanism is used, the computational overhead will increase significantly, which will seriously affect the deduplication rate; At the same time, in multi-core systems, none of the existing deduplication mechanisms can make full use of computing resources

Images