Data stream architecture and data processing method

Abstract

The invention provides a data stream architecture and a data processing method. The data stream architecture sequentially comprises an ODS data layer, a DW data layer, a DM incremental data layer, a DM data layer and an analysis display data layer according to a data stream direction; the ODS data layer is used for acquiring newly added data; the DW data layer is used for storing the newly added data as historical data; the DM incremental data layer is used for acquiring required historical data from the DW data layer and determining the acquired historical data as to-be-displayed data; the DMdata layer is used for transmitting the data to be displayed to the analysis and display data layer; and the analysis display data layer is used for carrying out display analysis on the to-be-displayed historical data. According to the invention, the data flow pressure is effectively relieved, so that more memories can be saved to execute processing tasks such as association, statistics, calculation and analysis, and therefore, the data processing speed of a big data system can be increased, and risks such as data loss can be effectively reduced.

Description

technical field [0001] The present invention relates to the technical field of big data systems, and more specifically, to a data flow architecture and a data processing method. Background technique [0002] In the existing big data system, the data will directly enter the DM data layer after being processed at the DW data layer. Since the data enters the DM data layer from the DW data layer, a large number of processing tasks such as correlation, statistics, calculation, and analysis are required. To perform data transfer tasks, it is easy to cause full load and blockage of the memory execution of the big data system, or even downtime, which greatly affects the data processing speed of the big data system and causes risks such as data loss. Contents of the invention [0003] The purpose of the present invention is to provide a data flow architecture and a data processing method, which can improve the data processing speed of a big data system and effectively reduce risks ...

AI Technical Summary

Problems solved by technology

[0002] In the existing big data system, the data will directly enter the DM data layer after being processed at the DW data layer. Since the data enters the DM data layer from the DW data layer, a large number of processing tasks such as correlation, statistics, calculation, and analysis are required. To perform data transfer tasks, it is easy to cause full load, blockage, or even downtime of the memory execution of the big data system, which greatly affects the data processing speed of the big data system and causes risks such as data loss

Images