Massive label storage system and method

Abstract

The invention provides a massive label storage system and method. The system comprises the following modules: an interface module which uses a similar SQL grammar operation system and is used for metadata management, data modification and query; a data writing module used for writing column data which can be ignored during repetition; a data updating module used for updating column data and neglecting the column data when the column data does not exist; a data deleting module used for deleting data; a data query module used for querying and counting data; and a bottom layer data structure module used for persistently storing data. According to the system, column information is acquired as required, column-based updating and deleting logic is provided, and massive labels are stored; label updating and quick query are realized by using column-based storage and vectorization execution as well as a row-based small-amount updating strategy and a column-based batch updating strategy.

Description

Technical field [0001] The present invention relates to the field of massive data processing, and in particular, to a massive label storage system and method. Background technique [0002] Online Analysis (OLAP) is a major approach to rapidly solve multi-dimensional analysis issues in computer technology, with a wide range of business applications in sales business reports, marketing, business process management, budget, and forecasts, financial statements, and similar areas. In recent years, with the rise of artificial intelligence, OLAP has more practices and applications in large data and data mining, and is currently a technical means of raising a lightweight. [0003] Among the Chinese invention patent documents of CN107092529B, an OLAP service system, an OLAP service agency method, device and electronic device, and an OLAP service method, device and electronic device, and related to online analysis processing. . The OLAP service system includes an OLAP service proxy device ...

AI Technical Summary

Problems solved by technology

However, it is often difficult to update data in this way, and there is no good solution to the association problem of columnar storage.

In order to solve the update problem, some systems try to use two different storages to handle updates and queries, which increases the complexity of the architecture and reduces real-time performance.

Although there are some systems that try to solve the data update problem, such as the emergence of Kudu, these problems still exist

[0005] In addition, data association query is another significant problem of columnar storage. This problem is generally solved by defining wide tables, but when faced with multi-table associations (such as Data of different dimensions) is usually centralized for data association, which is inefficient, and this problem is more obvious in distributed scenarios

When the number of columns reaches a certain scale, the query and writing of data will also cause some troubles (some systems will obtain all table structures when querying, and different columns cannot be updated separately when writing)

Images