Data stream architecture and data processing method

A data processing and data flow technology, applied in the field of big data systems, can solve problems such as downtime, congestion, and full memory of big data systems

Pending Publication Date: 2021-02-26
HUNAN NEW CLOUDNET TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] In the existing big data system, the data will directly enter the DM data layer after being processed at the DW data layer. Since the data enters the DM data layer from the DW data layer, a large number of processing tasks such as correlation, statistics, calculation, and analysis are required. To perform data transfer tasks, it is easy to cause full load, blockage, or even downtime of the memory execution of the big data system, which greatly affects the data processing speed of the big data system and causes risks such as data loss

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data stream architecture and data processing method
  • Data stream architecture and data processing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0024] see figure 1 , which shows a schematic structural diagram of a data flow architecture provided by an embodiment of the present invention, which sequentially includes an ODS data layer 11, a DW data layer 12, a DM incremental data layer 13, a DM data layer 14, and analysis and presentation data according to the data flow direction Layer 15, where:

[0025] The ODS data layer 11 is used for: acquiring new data;

[0026] The DW data layer 12 is used for:...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data stream architecture and a data processing method. The data stream architecture sequentially comprises an ODS data layer, a DW data layer, a DM incremental data layer, a DM data layer and an analysis display data layer according to a data stream direction; the ODS data layer is used for acquiring newly added data; the DW data layer is used for storing the newly added data as historical data; the DM incremental data layer is used for acquiring required historical data from the DW data layer and determining the acquired historical data as to-be-displayed data; the DMdata layer is used for transmitting the data to be displayed to the analysis and display data layer; and the analysis display data layer is used for carrying out display analysis on the to-be-displayed historical data. According to the invention, the data flow pressure is effectively relieved, so that more memories can be saved to execute processing tasks such as association, statistics, calculation and analysis, and therefore, the data processing speed of a big data system can be increased, and risks such as data loss can be effectively reduced.

Description

technical field [0001] The present invention relates to the technical field of big data systems, and more specifically, to a data flow architecture and a data processing method. Background technique [0002] In the existing big data system, the data will directly enter the DM data layer after being processed at the DW data layer. Since the data enters the DM data layer from the DW data layer, a large number of processing tasks such as correlation, statistics, calculation, and analysis are required. To perform data transfer tasks, it is easy to cause full load and blockage of the memory execution of the big data system, or even downtime, which greatly affects the data processing speed of the big data system and causes risks such as data loss. Contents of the invention [0003] The purpose of the present invention is to provide a data flow architecture and a data processing method, which can improve the data processing speed of a big data system and effectively reduce risks ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/54
CPCG06F9/545
Inventor 李政军陈娅芳
Owner HUNAN NEW CLOUDNET TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap