Data synchronization method and data synchronization system based on log analysis

A data synchronization and log technology, applied in the direction of file system, database update, file system functions, etc., can solve the problems of wasting disk space, slowing down the restart process of log synchronization service, etc., to reduce the impact of performance and ensure accuracy Effect

Pending Publication Date: 2020-11-13
WUHAN DAMENG DATABASE
View PDF6 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Since the dictionary file retains the changes of each modification, in some extreme cases, some applications may frequently make adjustments to the table, resulting in multiple versions in the dictionary file, and some versions will be changed during the log analysis process. No need to visit again. If these obsolete versions are not cleaned up, it will cause a waste of disk space and slow down the restart process of the log synchronization service

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data synchronization method and data synchronization system based on log analysis
  • Data synchronization method and data synchronization system based on log analysis
  • Data synchronization method and data synchronization system based on log analysis

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0058] In the source database and destination database deployment synchronization system, the source database synchronization system reads the log from the source database, while the destination database synchronization system is responsible for applying the synchronization operation sent by the source to the destination database.

[0059] In a system based on log parsing synchronization, the source log analysis module stores the database object definition information to the local, the offline dictionary file has a significant advantage than using the connection online database to take the log, the former In addition to supporting DDL operation synchronization, it is also possible to implement log resolution during log resolution, and log resolution can still be performed in the case of abnormal source database. The method of data synchronization based on offline dictionary files is specifically described below, and managing the offline dictionary file in data synchronization.

[...

Embodiment 2

[0103] In the present embodiment, a specific example to perform the aforementioned method concise description of the process, for example as follows:

[0104] Source database sequentially perform the following operations:

[0105] CREATE TABLE T1 (C INT);

[0106] INSERT INTO T1 (C) VALUES (1);

[0107] ALTER TABLE T1 ADD C1 INT;

[0108] INSERT INTO T1 (C) VALUES (2);

[0109] ALTER TABLE T1 ADD C2 INT;

[0110] INSERT INTO T1 (C) VALUES (3);

[0111] DROP TABLE T1;

[0112] In the above-described operation will be log stream source produces the following transactions:

[0113] Transaction number operate TRX1 (LSN = 1) Create (T1), Object ID = 1000 TRX2 (LSN = 2) INSERT (c = 1) TRX3 (LSN = 3) ALTER (Add C1) TRX4 (LSN = 4) INSERT (c = 2) TRX5 (LSN = 5) ALTER (Add C2) TRX6 (LSN = 6) INSERT (c = 3) TRX7 (LSN = 7) Drop (t1)

[0114] The assumption is invalid version accounted for 50% it needs to be cleaned, dictionary maintenance proce...

Embodiment 3

[0127] See Figure 8 , Figure 8 It is a structural diagram of a data synchronization system according to an embodiment of the present invention. The data synchronization system of the present embodiment includes one or more processor 41 and a memory 42. in, Figure 8 Taking a processor 41 as an example.

[0128] Processor 41 and memory 42 can be connected by bus or other means, Figure 8 In connection with the bus connection.

[0129] The memory 42 can be used as a non-volatile computer readable storage medium based on a data synchronization method, and can be used to store non-volatile software programs, non-volatile computer executives, and modules, methods of the above embodiment, and corresponding procedures. instruction. The processor 41 performs various functional applications and data processing by running the nonvolatile software program, instructions and modules stored in memory 42, and implements a method of implementing the foregoing embodiments.

[0130] The memory 42 can...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data synchronization method and system based on log analysis. The method comprises steps of judging the operation type of a DDL operation after a log analysis thread analyzesthe DDL operation; when the DDL operation is a creation operation, establishing an offline dictionary file containing object definition information for a newly created target object; obtaining a target operation for the target object, and managing the offline dictionary file according to the target operation; and obtaining a log sequence number LSN1 of a check point from a destination end, and cleaning the offline dictionary file according to the log sequence number LSN1. According to the method, the offline dictionary file is managed and updated in time according to the actual operation of the source end database, so excessive versions of definition information in the dictionary file are avoided, the disk space is released in time, and the speed of slowing down the log synchronization service restart process is avoided.

Description

Technical field [0001] The present invention belongs to the field of data synchronization technology, and more particularly, log analysis data synchronization method and a data synchronization system based relates. Background technique [0002] Database synchronization technology is currently widely used in distributed systems, big data, cloud database and other new products, can effectively alleviate the enormous pressure of high concurrent access to large amounts of data through redundancy scheme to bring the system database. By extracting the source database transaction log, to obtain a data source database operations, including INSERT, DELETE, UPDATE, and then transmitted to the replication destination system via a network, to restore to the original destination of the SQL statement in the object database, a common database synchronization technology. It has little impact on the performance model data and the source database, it can be arranged very flexible various topologie...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/27G06F16/23G06F16/17G06F9/46
CPCG06F16/27G06F16/2365G06F16/1734G06F9/466
Inventor 孙峰付铨彭青松刘启春
Owner WUHAN DAMENG DATABASE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap