Data push method and system

Abstract

The invention relates to a data push method and system. The data push method comprises the following steps of task creation extraction processing, dumping data from a source end database to a database increment log analysis server with a database embodiment being served as a boundary, obtaining incremental data through database incremental log analysis and temporarily storing the obtained incremental data into the memory of the database incremental log analysis server; information distribution processing, distributing the obtained incremental data to one or a plurality of corresponding message queue of a distributed message queue server with corresponding themes and marks according to a required rule, wherein the themes represent logical library names, the marks represent the logical table names, every distributed incremental data which is served as a message comprises a corresponding theme and mark; subscription push processing, enabling the distributed message queue server to push one or a plurality of message queues to a subscription end based on the subscripted themes of the subscription tasks.

Description

technical field [0001] The present invention relates to a large-scale data stream real-time consumption technology based on database incremental log analysis, and more specifically, relates to a data push method and system. Background technique [0002] With the development of the company's business, the demand for large-scale data extraction has become stronger. Data extraction refers to extracting the required data from the source database. In practical applications, relational databases are mostly used as source databases. Common data extraction methods include: (1) full data extraction, which is similar to data migration or data replication, which extracts the data of tables or views in the source from the database intact; (2) incremental data extraction Incremental data extraction, incremental data extraction only extracts new or modified data in the tables to be extracted in the database since the last extraction. Incremental data extraction is more widely used than...

AI Technical Summary

Problems solved by technology

[0004] The traditional incremental data extraction method is to regularly dump incremental data to generate corresponding data files, but this method has the following problems: it is necessary to add a timestamp field to the source-end business table, which pollutes the source-end business table to a certain extent; There is a certain degree of delay in the data; the dumped data files are too closely related to the business and are not easy to be reused; when the scheduled task is started, it often causes a large instantaneous pressure on the source database and network

[0011] This problem is also related to the above design, because the incremental data does not land, and the log storage time of the source database cannot be controlled by the data push platform, so the data push platform cannot promise the incremental data storage period to the downstream subscribers

This also reduces the ability of the data push platform to provide external services

[0012] (3). Subscribing end and extracting end depend too much

[0015] Business level: The extraction task is bounded by the database instance, and it is impossible to shield the subscriber from information that the subscriber should not care about, such as source sub-database sub-table, which leads to a stronger coupling between the extraction task and the subscription task

[0016] (4). The subscription end lacks support for common functions

Another open source project otter (the project's open source link address: https: / / github.com / alibaba / otter) supports some of the above functions, but still cannot meet the complex and flexible customization requirements

Images