Easily compressed database index storage system using fragments and sparse bitmap, and corresponding construction, scheduling and query processing methods

Abstract

The invention relates to an easily compressed database index storage system using fragments and a sparse bitmap, and corresponding construction, scheduling and query processing methods. The index storage system specifically comprises a fragment type data storage structure and an index storage structure of a sparse Hash bitmap for fragment data, wherein the sparse Hash bitmap is summary knowledge for data required to be indexed and can serve as a method for extracting other summaries for the characteristics of the fragment data. By adoption of a filtering index system and the construction and query processing methods, the high-efficiency equivalent query of mass data can be realized, the disk read-write frequency of the data is greatly reduced, and occupation of an index space is extremely low. Compared with a B+ tree index, the invention is more applicable to query of historical mass data. Based on the system and the methods, a database and indexes can be conveniently compressed and stored, and a fragment compression packet can be filtered, so that the decoding overhead of compression of the database is greatly reduced.

Description

technical field [0001] The present invention relates to data information processing and database technology, in particular to a filter index system using fragmentation and sparse hash as database equivalent query and its construction and query processing methods. Background technique [0002] In many industries, a large amount of data is generated all the time, thus forming a storage requirement for massive historical data, and with the development of information systems in various industries, there is also a query requirement for these massive historical data, especially It is an equivalent query requirement, such as querying the records of a specific user, which requires us to index the user ID data to meet the fast query requirements. At the same time, due to the continuous generation of new data that needs to be imported into the storage system, the database system is also required to have a very high import speed. [0003] However, traditional indexes are not suitable ...

AI Technical Summary

Problems solved by technology

[0005] 2) The maintenance cost is high, and the indexing speed is slow; there are two ways to build the index, one is to build the index first, and the price is that the index needs to be maintained synchronously when the data is imported, and random index insertion may be very time-consuming. After the index is established, an independent compression process is still required; the second is post-establishment, which means that after the data loading is completed, a large amount of data needs to be sorted (relative to the B-tree, for the HASH index, it means random data insertion process) for index creation, if a single index maintains less data, it means that the number of indexes will increase, which reduces the performance of the index itself

Taken together, no matter which mode is used, maintaining traditional indexes will have a serious impact on data loading speed

Images