A sliding block repeating data deleting method based on edge calculation

Abstract

The invention belongs to the technical field of data deduplication, and particularly relates to a sliding block duplication data deletion method based on edge calculation, which comprises the following steps: calculating the summation of each overlapped block of a file object according to an Rsync summation check function and a sliding window to obtain the value of a check sum; Comparing, for eachblock, the value of the checksum to a previously stored value; If the value of the checksum is matched with the value stored in advance, comparing the calculated hash value of the SHA-1 with a previously stored value so as to carry out redundancy detection, and deleting a data block represented by the same data if the same data is detected; According to the sliding block repeated data deleting technology based on edge computing, the problems that traditional data de-duplication is poor in granularity and high in cost are solved, the defect that inserting and deleting of data are sensitive isovercome. A check method, a weak check detection method and a thought of manually setting the size of a window are added on the basis of an SHA algorithm 1, so that a better duplicate removal effect is achieved.

Description

technical field [0001] The invention belongs to the technical field of data deduplication, and in particular relates to a method for deduplicating data by sliding blocks based on edge computing. Background technique [0002] With the rapid development of the Internet of Things technology and 5G network architecture, new equipment and new technologies in the fields of equipment intelligence and wireless communication continue to emerge, the amount of digital information is growing rapidly, and the storage space required for data is increasing. Management data costs and The energy consumption of the central space is also becoming more and more serious. The traditional cloud computing model cannot effectively solve problems such as cloud center load and transmission bandwidth. Edge computing emerged as the times require. Edge computing refers to a new service model in which data or tasks can be calculated and executed on the edge of the network close to the source of the data....

AI Technical Summary

Problems solved by technology

The original complete file detection technology is a method of finding duplicate data at the granularity of files. The Single Instance Storage (SIS) of Windows 2000 uses this technology to delete. This method is simple to implement and fast in calculation speed, but the detection granularity is relatively coarse , the delete effect is poor

In order to improve the deduplication rate, a finer-grained fixed-length and variable-block block-level detection technology is proposed. Compared with the complete file detection and deletion rate, the deletion rate has been greatly improved, but both technologies have certain limitations. sex

Images