Multi-source heterogeneous data real-time processing system and method based on Flink stream computing technology

Abstract

The invention discloses a multi-source heterogeneous data real-time processing system and method based on a Flink stream computing technology. The system comprises a data acquisition side which acquires heterogeneous data dispersed in a plurality of system components in a log mode and / or an SDK mode and / or an MQ mode, and sends the data to Kafka in a continuous flow mode after preliminary processing; a task management platform side which configures a data source type, configures cleaning and segmentation rules of heterogeneous data and dimensions and indexes of a configuration data set, starts a data real-time processing task based on an Flink stream computing technology after all configurations are completed, and defines and stores the data according to the data set after real-time computing of the data; and a data presentation and output side which obtains the result output of the data set. The system can analyze the data with multiple sources and different structures output in the existing service system to find the correlation between the log event and the service so as to help the operation and maintenance personnel improve the efficiency and provide supplementation for the existing service analysis system.

Description

technical field [0001] The present invention relates to a multi-source heterogeneous data real-time processing system and method, in particular to a multi-source heterogeneous data real-time processing system and method based on Flink stream computing technology. Background technique [0002] In the Internet+ era, in order to meet the needs of rapid business development and elastic scaling, the IT system architecture of enterprises is evolving towards Docker container clusters and microservices. This architecture improves resource utilization and brings greater flexibility. , Support high concurrency scenarios. [0003] However, with the expansion of business scale and the increase in the complexity of calling relationships between services, the amount of log output is increasing. When facing failures and performance problems, it is more difficult to analyze. Therefore, how to analyze the large amount of data output by the system It is an urgent problem to be solved to anal...

AI Technical Summary

Problems solved by technology

[0006] 1. It does not provide a visual graphical interface that can build data cleaning and segmentation rules, but implements it through configuration files

[0007] 2. The log data is not converted into structured time series data, so it is impossible to group and calculate index values ​​according to time and dimension

[0008] 3. There is no configurable data interaction chart display interface

However, at present, there is no real-time processing technology for multi-source heterogeneous data based on Flink stream computing technology

Images