Method for parallel processing compare increment data extraction

Abstract

The invention relates to a method for parallel processing compare increment data extraction. In a system related by the invention, the core design ideal has the following points of: 1, change data capturing by adopting a data abstract technology; 2, modularization by adopting a multi-thread technology and a functional unit to realize the parallel processing; 3, large data volume sorting algorithm; and 4, a data buffer technology. The method greatly improves the handling capacity of the system under the condition of processing large volume of data, and has accurate data processing, time saving, high efficiency, resource saving and wide application range.

Description

technical field [0001] The invention relates to a data processing method, in particular to a parallel processing comparison type incremental data extraction method. Background technique [0002] In current data integration or application integration projects, especially data integration projects involving e-government, data collection is a prerequisite for the success of the entire project, and incremental data extraction technology is the key to realizing data collection; but currently , the main problem facing data collection is that the systems are often internal applications of professional departments or application software developed by third parties, especially government departments are fragmented. From the perspective of security or management mode, it is impossible for other systems to access these systems Internally, data can only be provided by importing files, using an intermediate database, or using manual entry; the problems brought about by this data provisio...

AI Technical Summary

Problems solved by technology

[0002] In current data integration or application integration projects, especially data integration projects involving e-government, data collection is a prerequisite for the success of the entire project, and incremental data extraction technology is the key to realizing data collection; but currently , the main problem facing data collection is that the systems are often internal applications of professional departments or application software developed by third parties, especially government departments are fragmented. From the perspective of security or management mode, it is impossible for other systems to access these systems Internally, data can only be provided by importing files, using an intermediate database, or using manual entry; the problems brought about by this data provision method are that the amount of data provided repeatedly is large, the processing complexity of the target system is large, and at the same time it causes The target system is overloaded, inefficient, and manual processing is prone to errors

In order to solve the above problems, other software manufacturers also use comparison methods to obtain incremental data. However, due to the simple algorithm used by the system or the unreasonable system architecture, the situation of large data volumes is not fully considered, and it is difficult to handle large data volumes.

Images