Data processing method and device

Abstract

Embodiments of the invention provide a data processing method and device. Through the data processing method and device, when a data hashed value in a currently received data flow exceeds a preset first threshold value, part or all of data in the data flow is directly stored without re-deletion, so that the data in the data flow is prevented from being dispersedly stored in a plurality of storage areas and is intensively stored in one storage area, and then the re-deletion rate is effectively improved on the whole under the scenes of large data storage amount.

Description

technical field [0001] Embodiments of the present invention relate to storage technology, and in particular, to a data processing method and device. Background technique [0002] Data deduplication (abbreviated as deduplication), also known as intelligent compression or single instance storage, is a method that can automatically search for duplicate data, keep only one copy of the same data, and replace other duplicate copies with pointers to the single copy , to achieve a storage technology that eliminates redundant data and reduces storage capacity requirements. [0003] In the prior art, in the data deduplication scheme, the received data is divided into blocks to obtain data blocks, and then the data blocks are formed into several data segments, and the characteristic value of each data segment is calculated using a certain method , using the computed eigenvalues ​​to represent data segments. Match the characteristic value of the data segment with the characteristic va...

AI Technical Summary

Problems solved by technology

[0004] The inventor found in the research that in the existing data deduplication, for example, the data received for the first time is stored as new data; when the data received for the second time is changed relative to the data received for the first time, The changed data will be stored separately as new data; and when the same data received for the third time is received for the second time, the data most similar to the data received for the third time is likely to be the data received for the first time , then compared to the data changed for the first time, the changed data will still be considered as new data and stored, but in fact, the changed data has already been stored, so it can be seen that the deduplication process of the prior art In , the more data is stored, the more storage areas the data will be distributed to, but the overall deduplication performance will decrease

Images