Efficient duplicate removal method for repeated redundant data in cloud storage system

Abstract

The invention proposes an efficient duplicate removal method for repeated redundant data in a cloud storage system. The method comprises the following steps: multiple clients receive data uploaded by a user, take a data superblock as a data route unit, and extract route feature fingerprints for data route selection from the data; a metadata server and a data server cluster handle data route selection requests of the clients according to a route policy, wherein the data server cluster performs similar fingerprint matching on the route feature fingerprints to determine similar route nodes, and the metadata server determines a final data route address according to a load balancing policy; and the clients interact with corresponding data servers, and the data servers receiving similar redundant data perform efficient and quick duplicate removal on the redundant data. According to the method, the cloud storage system can achieve an efficient repeated redundant data removal effect while keeping high-performance, large-scale and high-throughput system properties, the disk utilization rate is increased, and the data management cost is reduced.

Description

technical field [0001] The invention relates to the technical field of computer information storage, in particular to an efficient deduplication method for redundant data in a cloud storage system. Background technique [0002] Data deduplication technology is a special data compression technology for coarse-grained lossless redundant data removal. The data deduplication technology divides the data at a coarse granularity, then extracts the Hash fingerprint of the data block through the fingerprint calculation technology, and judges whether the data is redundant through the index query of the fingerprint. If the content of the new data block is the same as that of the existing data in the storage system , the new data will not be stored, but the storage location of the data will be determined by saving the pointer to the original data block. The overhead for pointer storage is far less than the space occupied by the data. Therefore, the data deduplication technology can eff...

AI Technical Summary

Problems solved by technology

In the process of cluster deduplication, due to the consideration of the overall system performance overhead, global deduplication across nodes will seriously reduce system storage performance, so it is generally used to deduplicate data only on the internal data of the node

Therefore, the cluster deduplication system based on cluster deduplication technology will have a great impact on the overall deduplication effect of the system in terms of data routing.

Images