Method for performing data increment copying through hash value comparison

Abstract

The invention relates to a method for performing data increment copying through hash value comparison. The method includes the following steps that a primary key sequence and a non-primary key sequence of a source list are determined; the hash value of the currently recorded non-primary key sequence value is calculated; the currently recorded primary key sequence value and the harsh value are stored in a first file according to the currently recorded primary key value ascending sequence; the harsh value of the non-primary key sequence value recorded after the change is calculated according to the primary key sequence value ascending sequence recorded after the change; data comparison is performed, data increment is obtained, and the data increment is marked and classified into a database; the primary key sequence value recorded after the change and the harsh value are stored in a second file; the first file is covered with the second file, and data increment copying is completed. The method is not limited by database types and does not intrude into the database or affect the database, and comparison efficiency is improved.

Description

technical field [0001] The invention relates to a method for duplicating incremental data by using hash value comparison. Specifically, the calculated hash value is used to complete the capture of incremental data through comparison. Background technique [0002] In the process of data integration, the replication of incremental data is always a key point and a difficult point. To achieve incremental extraction, changing data needs to be captured quickly and accurately. A good incremental extraction mechanism requires that the integration process can accurately capture the changing data in the database at a certain frequency, and at the same time not cause too much pressure on the database. Affect existing business use. Compared with full extraction, the design of incremental extraction is more complicated. [0003] The existing incremental extraction mechanisms mainly include: trigger method, time stamp method, full table comparison method, database log analysis method, a...

AI Technical Summary

Problems solved by technology

Among them, the trigger method does not need to perform the comparison step in the extraction process, so the performance of incremental extraction is better, but the trigger method needs to directly create a trigger on the dataset table and write a temporary table at the same time. For frequent operations The business system may have a certain performance loss, especially when performing batch operations on the business table, row-level triggers will have a serious impact on performance; the timestamp method cannot capture delete operations, and needs to be used in conjunction with other methods. This method needs to modify the table structure of the data source system. For relational database products that do not support automatic update of the timestamp field, the function of the business system must also be modified so that it will display the update table when the source table performs each operation. Timestamp field; the full table comparison method requires a complicated comparison process to identify changed records, and the extraction performance is poor. In addition, some temporary tables need to be created, which is also rejected in some occasions; based on the database In the way of log analysis, because most database products only allow specific groups of users or even only DBAs to perform log analysis, this makes it impossible to implement the restrictions on database permissions open to the data integration process; specific database methods (OracleCDC methods) are limited The type of database is limited, it is difficult to apply widely

Images