Duplication eliminating method based on multidimensional lattice data spatial model

Abstract

The invention discloses a duplication eliminating method based on a multidimensional lattice data spatial model. The method includes following steps: loading local cache data and building the multidimensional lattice data spatial model; transferring the data into customized data format and cutting the data into data points; searching the data one by one, positioning coordinates of each data point on dimensions corresponding to the data model, searching each data point from a first digit down digit by digit, characterizing each data point if the same does not exist in the data model, and marking the data as absence; and traversing the data points of the data, outputting the cache if the data is marked as absence, and searching next data until all the data is searched. The duplication eliminating method based on the multidimensional lattice data spatial model is suitable for filtration and duplication eliminating of various data, high in duplication eliminating efficiency and has fine application value in engineering. In addition, by the method, the problem of severe resource consumption caused by length difference of the data is solved.

Description

[technical field] [0001] The invention relates to data deduplication, in particular to a deduplication method based on a multidimensional lattice data space model. [Background technique] [0002] There are many deduplication technologies. The most commonly used deduplication technologies such as hash and Bloom process the content that needs to be deduplicated and then match them one by one. This collection processing method is feasible, but it is cumbersome in massive amounts of data, and when the quantity is to a certain extent, the data repetition rate will increase rapidly, so data deduplication is meaningless. In the process of deduplication, a series of issues such as how to save deduplicated data and cache loading also need to be considered. If the deduplicated data cannot be saved and cached, restarting the deduplication server will start a new deduplication work, and it will be a repeated work for the processed data, which will reduce the accuracy of the data virtua...

AI Technical Summary

Problems solved by technology

If the speed of the URL retrieval method is slow, it will form a bottleneck, seriously affecting the collection speed and scalability of the entire system

In the ever-increasing mass data, the retrieval will become slower and slower due to the increasing amount of data, and the system resource consumption will be more. Especially in the use of algorithms such as HASH, the growth of data volume will make the data loss Collisions (data duplication) increase, leading to increased data bit error rates

The length of data is a factor that cannot be ignored in efficiency and resource consumption. If every piece of data that needs to be deduplicated is original data, it will be fatal to the deduplication method, because there is no way to predict the length of the data.

At this time, it is necessary to convert the data into a unified data form, such as HASH. There are many deduplication methods using HASH, but because of the high rate of collision (data duplication), they will use a variety of different HASH algorithm combinations. The pen data is processed into multiples by HASH for de-duplication and filtering, so that the collision (data duplication) rate is reduced, but the efficiency is also reduced, and the resource consumption also shows a corresponding geometric increase, and at the same time, the cache that has been de-duplicated Saving and loading data increases the difficulty

Images