Column-based storage and research method and equipment based on hard disk and internal storage

Abstract

The invention provides a column-based storage and research method and equipment based on a hard disk and the internal storage. A data sheet structure on data source is created in the internal storage through creating metainformation of the data sheet corresponding to the data source, and a column-based data block is generated on the current data according to the metainformation and is stored into the hard disk; the internal storage can be used more effectively, the following query data performance in the hard disk reaching the similar performance with the query data in the internal storage is realized, and the strong data analysis ability based on high-speed query efficiency can be further supported in following; moreover, when the column is an index column, the following data query efficiency can be improved by establishing a reverse index for each index column and storing the index column to the document at corresponding position in a solid state disk with the adoption of RadixTree structure.

Description

technical field [0001] The present application relates to the fields of communications and computers, and in particular to a hard disk and memory-based columnar storage and query method and device. Background technique [0002] With the rapid development of traditional enterprise business, the processing requirements of big data has become an inevitable problem in all industries. Traditional databases are row-based storage, which stores complete data rows one by one in the file system. Row storage is suitable for scenarios where most of the data columns are used in queries, such as OLTP (On-Line Transaction Processing, online transaction processing system) Inquire. But for OLAP (On-Line Analytical Processing, online analytical processing), users only need to query a few data columns, and using row storage will load many useless data columns, resulting in performance degradation. In order to solve this problem, a columnar database was born. Columnar storage stores the same ...

AI Technical Summary

Problems solved by technology

However, in real life, the data volume of the production system often reaches the level of TB or PB, and the data cannot be completely stored in the memory.

With the development of hardware technology, the reading and writing performance of hard disks such as SSDs (Solid State Drives) has been continuously improved, and it has become a trend to use hard disks instead of memory as data cache. However, at this stage, the reading and writing of hard disks cannot Hard disk storage, especially designing efficient columnar storage is a meaningful and challenging problem

Images