Data processing system and method

Abstract

The invention discloses a data processing system. The data processing system comprises a communication interaction module, a receiving and processing module, a sending and processing module and a task scheduling module, wherein the communication interaction module is used for being coupled with a source access end and a target access end and receiving first data in a first format from the source access end; the receiving and processing module is used for receiving the first data in the first format from the communication interaction module, obtaining source data by verifying, decompressing and reducing the first data and storing the source data, passing the verification and subjected to decompression and reduction, into a data cache in a separate data segment manner according to the type of the source data; the sending and processing module is used for converting to-be-sent data into second data in a second format and sending the second data to the target access end through the communication interaction module; the task scheduling module is arranged between the receiving and processing module and the sending and processing module and is used for scheduling the to-be-sent data to the sending and processing module in a data segment manner. The invention further discloses a data releasing and extracting method.

Description

technical field [0001] The application relates to a data processing system and a method for publishing and extracting data between a source access terminal and a target access terminal. Background technique [0002] The current method of data extraction between the source database and the target database is basically based on the functions of the relational database, roughly as follows: ( 1 ) directly query the data on the source database and insert or update the query results to the target database; ( 2 ) Synchronize the changes on the source database to the target database through database logging, mirroring, archiving and other methods; ( 3 )pass ETL And other special tools to achieve. [0003] The above methods in the prior art cannot meet the current needs, especially in the case of frequent data extraction. The method in the prior art may easily cause a piece of data to be transmitted from the source database to the target database multiple times, req...

AI Technical Summary

Problems solved by technology

[0003] The methods of the above prior art cannot meet the current needs, especially in the case of frequent data extraction

The method in the prior art is likely to cause a piece of data to be transmitted from the source database to the target database multiple times, requiring multiple visits to the source database

Although this ensures the relative accuracy of the data, it needs to be at the expense of greatly reducing the processing performance

In addition, there are still problems in the existing technology such as large dependence on the database, large impact of the extraction method on the source database, scattered deployment and security exceptions.

Images