Data synchronism ETL (Extract Transform Load) system

Abstract

The invention belongs to the technical field of data synchronism, and particularly discloses a data synchronism ETL (Extract Transform Load) system. The data synchronism ETL system comprises the following function modules: a context parameter configuration module, a synchronous interface definition module, a synchronous script generation module, a workflow configuration module, a data synchronism core module and a log and prewarning module, wherein the context parameter configuration module, the synchronous interface definition module and the synchronous script generation module are basic modules and are operated according to the sequence of the modules to generate a script; the workflow configuration module mainly plays the role of a dispatching center, and configures the script in a workflow to realize parallel or serial execution, parallel progress amount, and what time or condition a task is triggered; when anomaly or an error task occurs in the execution steps, the log and prewarning module captures an error reason, and informs system management personnel of the error reason. The data synchronism ETL system is provided with a bidirectional synchronism mechanism, and is capable of supporting data sources configured with multiple context names and accessing multiple environments, the coverage of the synchronism method is wide, meanwhile, a complicated user-defined synchronism method is supported, and the expansibility is stronger.

Description

Technical field [0001] The invention belongs to the technical field of data synchronization, and specifically relates to a data synchronization ETL system. Background technique [0002] The existing data synchronization system mainly provides production environment business data for R&D troubleshooting and simulation of problem recurrence, etc., as well as for data analysts to analyze business data, and provide daily, weekly, and monthly business operation reports for management Use or provide data support for the sales department’s business. Due to the existence of a cross-network in the two environments, the servers in the two environments cannot communicate with each other, and dial in through VPN to access the production environment and synchronize data files to the local environment. [0003] Existing method one, initialization is to complete the restoration of historical data by fully backing up database A in the production environment, copying the backup file to server B, a...

AI Technical Summary

Problems solved by technology

[0007] 1. Data dispersion caused by "data islands" is not conducive to data integration analysis and value-added services;

[0008] 2. Data files need to be landed and then copied to the target end, which requires manual intervention in multiple steps to complete;

[0010] 4. The workflow scheduling dependency between steps cannot be resolved, and the triggering of tasks will be scattered in different processes;

[0011] 5. Lack of systematic tools to deal with, including compatibility with exception handling, alarm mechanism, etc.;

[0012] 6. Too much manual intervention, high error rate and low efficiency caused by manual operation

Images