A method to realize production and consumption of real-time data flow across data sources

Abstract

The invention relates to the field of information technology, and provides a method for realizing production and consumption of real-time data flow across data sources. The purpose is to solve the existing problem that usually a consumer does not need all the data in the same message queue, but consumes part of the data in multiple message queues, because the data in the message queue can only be consumed sequentially, which will cause the consumer to consume Unnecessary data is generated, on the one hand, it causes a waste of resources, and on the other hand, it brings about data security issues. The main plan includes Step 1: Unify the data standards of the data producer and add data category identifiers; Step 2: Develop a message queue filter board, filter and classify the data in the message queue, and realize the data identification in the message column, Obtain the data source address, database name, and table name corresponding to each piece of data; Step 3: Develop the data distribution mechanism function, collect the data consumption needs of consumers, and perform data configuration distribution according to the needs.

Description

technical field [0001] The invention relates to the field of information technology, and provides a method for realizing production and consumption of real-time data flow across data sources. Background technique [0002] In order to better understand this proposal, it is necessary to understand the basic technologies: [0003] Message queuing is a communication method between processes. It has a remarkable feature: asynchronous processing, which allows users to put a message into the queue, but not process it immediately, and consume it in the queue when needed, and then to deal with them. Therefore, in order to ensure data consistency, the message queue adopts the FIFO first-in-first-out feature to ensure the order of messages. [0004] In daily use, usually a consumer does not need all the data in the same message queue, but consumes part of the data in multiple message queues, because the data in the message queue can only be consumed sequentially, which will cause the...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to solve the problem that usually a consumer does not need all the data in the same message queue, but needs to consume part of the data in multiple message queues, because the data in the message queue can only be consumed sequentially, which will lead to The consumer will consume unnecessary data, which on the one hand causes a waste of resources and on the other hand brings about data security issues

Images