Massive structured data storage and query methods and systems supporting high-speed loading

Abstract

The invention provides a massive structured data storage method, a massive structured data query method, a massive structured data storage system and a massive structured data query system, which all support high-speed loading. The distributed storage method for massive structured data comprises the following steps of: receiving the data which is loaded at high speed from a user; and caching the loaded data in a distributed way by utilizing a double-sliding window structure, and storing the cached data in the distributed way after a fixed period. By the distributed storage method for the massive structured data, newly-loaded data can be cached, so that the query efficiency of application such as streaming data which frequently uses recently-loaded data in post data query can be improved.

Description

technical field [0001] The present invention relates to a massive data management system and method in the field of information security, and more specifically relates to a complex query task-oriented query and distributed data management method and system, mainly used for landing storage and analysis of network messages in the field of information security And statistics, analysis and other applications of massive log data. Background technique [0002] In the field of contemporary information security, data management is no longer limited to simple data processing methods such as traditional data sampling and analysis, but uses efficient data storage systems to store data on the ground and support complex data statistics and analysis after the event. [0003] Since the currently widely used relational databases are subject to consistency constraints, the query methods and query systems based on relational databases have low loading efficiency and slow retrieval speed under...

AI Technical Summary

Problems solved by technology

As a result, HIVE's real-time query efficiency is low and the delay is large, and the purpose of online data loading and fast query cannot be realized, and the query application that does not require a large number of result sets cannot be satisfied.

[0006] (2) There is no index in HIVE, and all its query operations are performed by reading raw data files

Therefore, the query efficiency is low

[0007] (3) The query process of HIVE is that the user describes the query rules through the HQL language (a query expression similar to the SQL language). MapReduce-oriented task decomposition method (that is, multiple disk write and read operations are required when executing query tasks), so its execution efficiency is low, and it cannot be directly used for querying stream record data

[0008] (4) HIVE reads data from data files, so it does not support frequent loading of record streaming, nor does it support cache loading data or cache lookup data

Although the storage system in the prior art has the function of opening up the cache structure and improving the efficiency of data loading, it needs to wait until the cached data is written to the disk to support the query operation. The application of streaming record data is continuously loaded into the system. Moreover, recent data has a relatively higher frequency of use, so traditional methods cannot meet the needs of queries

Images