Data synchronization method and device

Abstract

The present invention provides a data synchronization method and apparatus, which can implement data unification inside a hadoop cluster or between hadoop clusters, and have the advantages of being simple and easy and the like. The data synchronization method comprises: judging a data synchronization type of a hadoop cluster, wherein the data synchronization type includes data copying inside the cluster, address sharing inside the cluster, and data copying among clusters; executing, according to a determining result, a pre-selected data quality checking task corresponding to the determining result; and when data inconsistency is checked during execution of the data quality checking task, executing the last data synchronization task.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a data synchronization method and device. Background technique [0002] At present, many large IT companies have launched services such as cloud platforms, big data platforms, cloud computing, cloud storage, and data marts, realizing data sharing and data transmission functions between different enterprises or between various business departments within the enterprise. figure 1 It shows the process of synchronizing data within and between clusters in environments such as big data platforms and cloud services. However, there is a lack of data quality checking after synchronization, and there may be data inconsistency problems. Specifically: when the data sent by the data sharing party changes, the data that has been synchronized by the data subscriber will be inconsistent with the data sharing party. It is difficult for the data subscriber to detect the change of the data shari...

AI Technical Summary

Problems solved by technology

However, there is a lack of data quality check after synchronization, and there may be data inconsistency problems. Specifically: when the data of the data sharing party changes, the data that has been synchronized by the data subscriber will be inconsistent with the data sharing party.

It is difficult for data subscribers to detect changes in the data of data sharing parties, and huge losses are often caused after discovery

After the data inconsistency problem occurred, the data subscriber did not obtain the latest data again in time, resulting in chain data errors of the data subscriber

To sum up, in the existing technology, after the data inconsistency problem occurs, there is a lack of a timely notification mechanism and a timely, automated, and intelligent data difference processing mechanism, which has brought huge losses

Images