Data exchange adapter, data synchronization system and method between heterogeneous systems

Abstract

The invention discloses a data exchange adapter and a system and a method for synchronizing data among heterogeneous systems, and relates to the technical field of mass data processing. With a star-like mode design principle, a set of data exchange adapter device located between different types of databases or file systems is established, then the mass data are synchronized effectively among different heterogeneous systems, and the original net-shaped data synchronization mode formed by maintaining and managing a series of complicated data import and export tools is replaced. In addition, the embodiment of the invention provides a set of unified open framework to support secondary development, a DataIn plug-in and a DataOut plug-in provided by the scheme can be used or required DataIn plug-in and DataOut plug-in are self-defined, so that the task of synchronizing any mass data among the different types of databases or file systems is rapidly executed.

Description

Technical field [0001] The invention relates to the technical field of massive data processing, in particular to a data exchange adapter and a data synchronization system and method between heterogeneous systems. Background technique [0002] In large-scale business production system application scenarios, there are often many different types of databases or file systems, such as Oracle, MySQL, SQL Server, Sybase, DB2, Informix, PostgreSQL, MongoDB, Redis, HDFS (Hadoop Distributed File System, Hadoop distributed file system) and other databases or FS (File System, file system), etc., and it is often necessary to import or export massive amounts of data between them. At present, there are many mature data import and export tools for different types of databases or file systems, such as MySQLDumper, OracleSQLLoader, etc., but these tools generally only provide data export or import, and only support one or a few specific types of databases or file systems. [0003] With the subseque...

AI Technical Summary

Problems solved by technology

[0003] With the increase of business systems, the introduction and application of different types of libraries will lead to a linear increase in the demand for these data import and export tools.

In this way, we need to develop and maintain a series of tools to achieve data synchronization between different databases or file systems, resulting in a network structure for data synchronization between different systems, as follows figure 1 As shown, it greatly increases the difficulty of system maintenance and management

[0004] In addition, some of these tools exchange data in the form of files (File), and some transfer data in the form of pipelines (Pipe), all of which bring additional overhead to data synchronization to varying degrees, resulting in very large differences in efficiency

At the same time, many tools cannot effectively meet the common requirements in data warehouse system ETL (Extraction, Transformation&Loading, data extraction, transformation and loading) tasks, such as time format conversion, special character conversion, encoding conversion, etc.

Images