Streaming data processing system based on metadata

Abstract

The invention provides a streaming data processing system based on metadata, and belongs to the technical field of data processing. According to the technical scheme, the streaming data processing system based on metadata comprises; a metadata management module, where the metadata management module can define the structure of a meta-model according to the form and storage mode of data; the process management module that comprises a process management component and a process arrangement component; the flow scheduling module that can obtain a flow execution graph, distributes execution tasks according to the flow execution graph and sends out corresponding execution instructions; and the process execution module that can receive the execution instruction, execute the corresponding execution task and start a corresponding operator to start calculation. The invention has the beneficial effects that metadata definition, flow definition and flow arrangement are carried out through a visualization method, and a streaming processing flow is created by using a dragging component and a model-driven logic.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a stream data processing system based on metadata. Background technique [0002] At present, with the rapid development of the Internet and Internet of Things technology, information browsing, retrieval, data reporting of Internet of Things devices, and many common life products such as e-commerce and the Internet have all been online. The requirements for real-time are further improved, and the interaction and communication of information is developing from point-to-point to information chain and even information network. This will inevitably bring about cross-correlation of data in various dimensions, and data explosion is inevitable. Therefore, stream processing comes into being. It solves the problem of real-time framework and large-scale calculation of data. [0003] At present, real-time processing requires mastering various big data technologies and stream process...

AI Technical Summary

Problems solved by technology

When there are many streaming data processing businesses, there is a lack of unified management, rapid business evolution, and rapid changes in requirements. Designing new data processing processes or modifying existing processes cannot be implemented quickly

[0004] In view of the rapid development of smart security, smart city, smart agriculture, smart transportation and other industries, a large amount of data has been accumulated. The aggregation and management of these data often needs to be completed by each platform, which creates a lot of repetitive work and lack of data management. , it is difficult to take full advantage of

Images