Multi-data-source data synchronous monitoring method and system

Abstract

The invention discloses a multi-data-source data synchronization monitoring method and system. The method comprises the following steps: S110, generating a corresponding synchronization task according to channel distribution; s120, according to a data source set by the task, determining a processor corresponding to the data source; s130, the task is handed over to a data processor for slicing; s140, data reading, data assembling, data asynchronous batch writing and concurrency control are carried out; s150, checking data synchronization of each slice and whether synchronization is completed or not, when all slices are completed, completing a synchronization task, if one slice fails, determining that the whole task fails, notifying operation and maintenance personnel when the task fails, and notifying related personnel after the task is recovered; s160, triggering a synchronization task failure, and performing recovery notification; and S170, synchronously monitoring platform logs, displaying all synchronization tasks, checking slices to specifically see the condition of each slice, retrying a certain synchronization task and a certain slice, resetting an id segment of the slice, and then resynchronizing.

Description

technical field [0001] The invention relates to the technical field of computer networks and data processing, and in particular to a method and system for synchronizing and monitoring data from multiple data sources. Background technique [0002] With the continuous development of modern information technology, the speed of data generation becomes faster and faster, and a large amount of data is collected and stored. These massive data must be processed and analyzed systematically in order to tap the potential value contained in it. In this process, data synchronization plays a key role. It is necessary to integrate multiple data sources and then summarize the data for storage. Since the project of synchronizing data belongs to a process, it is also necessary to record synchronization tasks, synchronization results, and alarms. In the case of multiple data sources and changeable integrated data logic, data can be transmitted and summarized more flexibly, clearly, convenient...

AI Technical Summary

Problems solved by technology

At the same time, the new characteristics of big data itself also put forward more urgent needs and more severe challenges for the synchronous monitoring of multi-data source data

[0004] At present, there are two main methods for the development of multi-data source data synchronous monitoring: one is to invest a lot of manpower in customized development for each data source access to obtain more targeted data sources and data conversion logic, which requires repeated development each time. The commonality is not abstracted and cannot be reused; the second is to use DataX software to complete data synchronization through field mapping, data source configuration and simple SQL data conversion processing

Among them, the monitoring and retrying of tasks are relatively strong. For example, a task is divided into data units with a small amount of data for synchronization. It does not provide monitoring and retrying for smaller data units. Once it fails, it can only be retried. Executing the entire task requires high retry costs; through the template and data source configuration of DataX software to map and synchronize, there will be field mapping that is not enough to support the requirements, lack of on-demand synchronization according to business scenarios, low flexibility, and cannot meet the design requirements And other issues

Images