File data coverage method and device, and storage medium

Abstract

The invention provides a file data coverage method and device and a storage medium. The method comprises the steps of obtaining data of a file; determining a file unit corresponding to the data; adding the corresponding file unit to the tail of the used file unit linked list; covering the relationship between the file and the corresponding file unit in the file chain table; and storing the data in the corresponding file unit in a covering manner. According to the method and the device, the high efficiency and high timeliness of data coverage are improved by writing the file unit data and realizing efficient management of the data unit in a linked list form.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a file data overwriting method, device, and storage medium. Background technique [0002] With the mature application of big data applications in various industries, various industries are paying more and more attention to data collection and storage, and related data storage functions have been added to various products, but for equipment close to data collection, data The storage space of the cloud is always limited, and it cannot expand the data storage capacity unlimitedly like the data center. Therefore, for the cloud edge system, the data on the terminal needs to be deleted and overwritten after being transferred to the data center. [0003] Overwriting storage of file data is an important technology in high-speed storage devices. When overwriting file data, two important problems need to be solved, one is the performance of data overwriting, and the o...

AI Technical Summary

Problems solved by technology

The performance problem of data overwriting mainly means that when overwriting file data, the application layer needs to traverse all the files in the file system, sort the existing files according to time, find the oldest file, and replace the oldest file Delete to release the storage space, this method will cause the retrieval performance to decrease as the number of files increases, and the CPU (Central Processing Unit, central processing unit) frequently calls the file retrieval when the file data is overwritten, thus causing the file data to be overwritten The performance of writing is degraded, especially when overwriting high-speed data, the fluctuation of CPU scheduling causes the performance of overwriting and writing to degrade; the problem of data timeliness refers to the problem that is easy to cause when overwriting and writing large files. For example, if there is a 1TB file in the system, if it needs to be overwritten at this time, the file is usually deleted and new data is written at the same time. The new data that may be written in a short period of time may be less than 1GB or 1MB or even smaller , which will lead to a large proportion of data loss, which may have a greater impact on later data application analysis

[0004] In order to solve the problems of performance and timeliness in the file data overwriting method, different processing methods have been adopted in the data storage software in the industry. For example, in some devices, in order to reduce the performance problems caused by data retrieval during file data overwriting, in addition to the file system, an additional File time index table, or limit the number of files, this method can alleviate the performance problem of file overwriting, but when overwriting high-performance data, limiting the number of files will also cause the size of a single file to be too large, thus also It will aggravate the timeliness of data. Adding a file index table can solve the performance problem of retrieval. However, because the file system allocates storage space on the whole disk, it is easy to cause disk fragmentation after long-term storage, resulting in slower performance when data is overwritten and written.

For the timeliness of data, the usual mitigation method is to overwrite from the end of the file, and continuously change the size of the old file in the data application software to free up storage space. This method can solve the problem of deleting too large files at one time. It leads to the problem of data loss, but it does not completely solve the problem of timeliness, because the data deleted from the end of the file is newer than the data at the same head, which will cause data interruption during data analysis, and at the same time, changing the file size continuously will increase the file size. IO operations, increasing the IO burden, will also lead to file system performance degradation

[0005] Therefore, no matter which overwriting solution mentioned above can not perfectly solve the two problems of performance and timeliness in file data overwriting

Images