Data processing method and device

Abstract

The embodiment of the present invention provides a data processing method and device. Through the embodiment of the present invention, when it is found that the hash value of the data in the currently received data stream exceeds the preset first threshold, part or all of the data in the data stream will not be redone. delete, store directly, and avoid the data in the data stream from being scattered and stored in multiple storage areas, but will be stored in a centralized storage area, especially in scenarios with a large amount of data storage, which can effectively improve the deduplication overall. Rate.

Description

technical field [0001] Embodiments of the present invention relate to storage technology, and in particular, to a data processing method and device. Background technique [0002] Data deduplication (abbreviated as deduplication), also known as intelligent compression or single instance storage, is a method that can automatically search for duplicate data, keep only one copy of the same data, and replace other duplicate copies with pointers to the single copy , to achieve a storage technology that eliminates redundant data and reduces storage capacity requirements. [0003] In the prior art, in the data deduplication scheme, the received data is divided into blocks to obtain data blocks, and then the data blocks are formed into several data segments, and the characteristic value of each data segment is calculated using a certain method , using the computed eigenvalues ​​to represent data segments. Match the characteristic value of the data segment with the characteristic va...

AI Technical Summary

Problems solved by technology

[0004] The inventor found in the research that in the existing data deduplication, for example, the data received for the first time is stored as new data; when the data received for the second time is changed relative to the data received for the first time, The changed data will be stored separately as new data; and when the same data received for the third time is received for the second time, the data most similar to the data received for the third time is likely to be the data received for the first time , then compared to the data changed for the first time, the changed data will still be considered as new data and stored, but in fact, the changed data has already been stored, so it can be seen that the deduplication process of the prior art In , the more data is stored, the more storage areas the data will be distributed to, but the overall deduplication performance will decrease

Images