Real-time data warehouse platform

Abstract

The invention discloses a real-time data warehouse platform, which comprises a business data acquisition system, a log data acquisition system and an analysis system. The business data acquisition system comprises a candu module; the candu module is used for synchronously analyzing an update log of business data and storing the analyzed data into a kudu storage module of an analysis system; the log data acquisition system is used for acquiring log data, calculating the log data, and storing a calculation result to the kudu storage module; the kudu storage module is used for carrying out data analysis in real time according to the stored analyzed data and the stored calculation result. According to the real-time data warehouse platform provided by the invention, the update log of business data distributed on each business system is collected in real time through the candu module, so that real-time synchronization of the business data is realized.

Description

technical field [0001] The invention relates to the field of network technology, in particular to a real-time data warehouse platform. Background technique [0002] In the existing technical solutions for data warehouses, offline and non-updatable distributed hive data warehouses are used. It is difficult to achieve the level of real-time data warehouses, and it is impossible to synchronize business databases in real time. If the timeliness cannot be guaranteed, more improvements cannot be provided to the existing business data analysis. In addition, the existing data warehouse cannot be easily used by business personnel. [0003] Generally speaking, the existing log system has the following defects: 1) Most of the existing systems are offline distributed data warehouses of hive, which cannot meet the user's update and record-level insert functions. 2) Poor performance. The existing hive distributed data warehouse has extremely poor query performance for small data volume...

AI Technical Summary

Problems solved by technology

[0002] In the existing technical solutions for data warehouses, offline and non-updatable distributed hive data warehouses are used. It is difficult to achieve the level of real-time data warehouses, and it is impossible to synchronize business databases in real time.

If the timeliness cannot be guaranteed, more improvements cannot be provided to the existing business data analysis.

In addition, the existing data warehouse cannot be easily used by business personnel

[0003] Overall, the existing log system has the following defects: 1) Most of the existing systems are offline distributed data warehouses of hive, which cannot satisfy the user's update and record-level insertion functions

2) poor performance

The existing hive distributed data warehouse has extremely poor query performance for small data volumes, and even cannot reach the performance of traditional relational data warehouses

3) Fusion of log real-time data and historical data

Existing data warehouses are all offline data, which cannot be integrated with real-time log data, which indirectly hinders the analysis and mining of all business data

[0004] Aiming at the problem of up-stop in related technologies, no effective solution has been proposed yet

Images