Data deduplication method based on internet small computer system interface (iSCSI)

Abstract

The invention provides a data deduplication method based on internet small computer system interface (iSCSI), belongs to the technical field of computer information storage, and is suitable for an iSCSI-protocol-based internet protocol (IP) network remote mirroring system. In the invention, bandwidth simplification and synchronous time reduction can be realized through deleting repeated data of an iSCSI write data block on the premise of not changing the structure of the conventional IP network remote mirroring system. The data deduplication comprises two stages: in the first stage, a coarse-grained similarity data chunk detection technology is adopted, and a content-defined chunking (CDC) algorithm and a bloom filter algorithm are combined to search similarity chunks in a full range, so that the data deduplication can be more flexible and more accurate; and in the second stage, an improved fine-grained similarity data chunk detection technology is adopted, and a fixed-sized partition and sliding window method is combined, so that the deduplication is performed on chunks rather than files, and transparency of deduplication to users is realized.

Description

technical field [0001] The invention belongs to the technical field of computer information storage, and in particular relates to an iSCSI-based duplicate data deletion method, which is suitable for an IP network remote mirroring system based on the iSCSI protocol. Background technique [0002] IP network remote mirroring system has been widely used in disaster recovery system. The system is based on the iSCSI protocol, and transmits SCSI data and commands to the disaster recovery center through the IP network to achieve the consistency of local mirroring and remote mirroring. The system does not need to build a dedicated network, which greatly reduces the cost of building a disaster recovery system, and also makes the system have good scalability, as long as it can be connected to the IP network, the service can be used. [0003] With the explosive growth of digital information, the scale of data stored in the disaster recovery system is getting larger and larger. Studies...

AI Technical Summary

Problems solved by technology

The shingle algorithm for similar block detection needs to extract the feature set of the file first, and then calculate the similarity between the two files, but the calculation and storage costs are relatively large; while the bloom filter algorithm uses a set to represent the file features, and the calculation and storage costs are smaller than shingle There are many, but the objects to be compared must construct filter values ​​of the same length. For file groups with large file sizes, it is inconvenient to select an appropriate filter length for comparison. If it is too small, the misjudgment rate will be high, and if it is too large then the cost will be high

Images