Data de-duplication method based on combination of similarity and locality

Abstract

The invention provides a data de-duplication method for combining the similarity and the locality of data, with less system memory expense and high data de-duplication efficiency. The method comprises the steps of: firstly partitioning and grouping files in a data stream, determining a similarity unit and a representative fingerprint of every data group, and storing the representative fingerprint in a memory; and traversing all the data groups and performing a similarity determination to determine which data in data groups are completely duplicate data and which data in data groups have non-duplicate data. If the non-duplicate data exists in the data groups, the locality determination can be continued to further determine which data in the data groups is the duplicate data. According to the method, the representative fingerprint is only stored in the memory, thus the memory expense is greatly reduced; supplement can be performed after the similarity determination by mining the locality of the data stream and caching the locality of the data stream in the memory, thus more duplicate data can be found, and simultaneously, frequent access to a disk index can be avoided and the utilization rate of the memory can be enhanced.

Description

technical field [0001] The invention belongs to the field of computer storage, in particular to a method for deleting duplicated data based on the combination of similarity and locality. Background technique [0002] In recent years, with the development and popularization of computer technology and networks, the amount of data information storage in the world has shown an explosive growth trend. Although the price of storage devices has been declining, it is far behind the speed of data expansion. Data deduplication (Data Deduplication), as a technology to effectively eliminate redundant data on a large scale, has become a hotspot in storage system research in recent years. Data deduplication can not only greatly save storage space and improve storage system performance, but also save network bandwidth by avoiding redundant data transmission. The rise of data deduplication originated from the demand for massive data backup and archiving in the storage market, and the dema...

AI Technical Summary

Problems solved by technology

That is to say, every time a data block is input, the entire disk fingerprint index needs to be traversed, which seriously affects the throughput of deduplication.

Images