Index read-write method based on compression merging and asynchronization updating

Abstract

The invention relates to an index read-write method based on compression merging and asynchronization updating. The method includes the steps that indexes are used for generating index cache files according to storage region serial numbers; incomplete index cache files or writing index cache files are filtered and removed; when the total number of the index cache files meets the asynchronization updating initiation condition, data of storage regions corresponding to the same storage region serial numbers in all the index cache files are read respectively, and the data are merged and sorted; data of storage regions corresponding to the current storage region serial numbers in an index database are read and decompressed; the data are merged, sorted and compressed, the executed data are covered and updated to the storage regions corresponding to the current storage region serial numbers in the index database, and metadata are updated; whether the ends of files of the index cache files are read is judged, if yes, the process is completed, if not, the storage region serial numbers are added with one, and the process is executed continuously. Compared with the prior art, the index read-write method based on compression merging and asynchronization updating has the advantages of being efficient, asynchronous, capable of reducing the data size for reading a magnetic disk every time and reducing the index comparison frequency and the like.

Description

technical field [0001] The invention relates to a data storage technology, in particular to an index reading and writing method based on compression and asynchronous update. Background technique [0002] With the continuous development of computer systems, there are more and more effective data in computer systems. When reading some valid data, the user needs to find the data he needs in all the valid data, and as the valid data increases, the search time will become longer. If there is no effective way to quickly find the data the user needs, the increase in effective data can only increase the burden on the user. [0003] The introduction of indexing technology solves this problem well. During the data writing process, some important information will be recorded to uniquely identify the data and record the storage location and size information of the data. We call this information an index. Indexes can help us quickly find the data we need, but as the amount of data con...

AI Technical Summary

Problems solved by technology

This method effectively solves the IO bottleneck problem of reading the disk, but it also brings a new problem. Due to the large number of index categories, the data distributed to each storage area during batch writing will be very small, which will lead to batch writing. The number of times to write to the disk is very large, and the amount of data written to the disk each time is very small

Therefore, the performance is poor when writing in batches

Images