Real-time collection method applied to MYSQL BINLog change data

Abstract

The invention discloses a real-time collection method applied to MYSQL BINLog change data, and belongs to the technical field of big data. The method comprises the following steps: 1, configuring database connection information and acquisition field information of an MYSQL database to be acquired in Excel, and configuring a configuration file for analyzing an Excel template position, Flink SQL script output file configuration information and execution environment configuration information; 2, executing an Flink mysql script tool, and automatically generating an Flink CDC acquisition script; and 3, uploading the Flink CDC acquisition script to each Flink client node to execute task release. The collection codes are automatically generated through simple configuration and tools, complex and frequent real-time collection research and development tasks are achieved, the collection tasks are integrated in an existing real-time application platform, cluster resources are fully utilized, unified operation and maintenance management is achieved, and the maintenance cost is reduced.

Description

technical field [0001] The invention belongs to the technical field of big data, and in particular relates to a real-time collection method applied to MYSQL BINLog change data. Background technique [0002] The value of data itself decreases over time. Only by promoting the real-time application of data can the value of data be fully utilized in a timely manner, so that data can better empower businesses and tap the value of business data. In the existing technology, the real-time architecture uses business data MYSQL and ORACLE as the main data sources. How to collect real-time change data of the source system in real time has become a difficult problem to be solved in the early stage of real-time architecture construction. [0003] The early architecture was limited by the technical environment at that time and the technical maturity of the market. Most of the real-time acquisition solutions were based on the BINLog API interface provided by MYSQL BINLog itself, and the re...

AI Technical Summary

Problems solved by technology

[0004] (1) Independent of the real-time computing platform, the maintenance cost is high;

[0005] (2) Based on the collection of configuration files, the collection database table is highly coupled, and the startup program will start together with the normal collection database table, resulting in the interruption of historical collection short-term business data collection, which will affect business continuity to a certain extent;

[0006] (3) Waste of resources, the current collection program is deployed on 6 collection nodes, the distribution of collection tasks is uneven, the utilization rate of resources is low, and the waste of resources is serious;

[0007] (4) The configuration file is long and complicated, and manual configuration is easy to be mistaken or mismatched, resulting in a short-term interruption of business collection when the collection task is started, which affects business continuity to a certain extent;

[0009] With the changes in architecture and technology iterations, the current collection scheme is stable and has prominent performance drawbacks, making it difficult to maintain the existing real-time architecture data real-time collection support

Images