Big data synchronization method and device, computer equipment and storage medium

Abstract

The invention provides a big data synchronization method and device, computer equipment and a storage medium, and the method comprises the steps: responding to a synchronization data obtaining instruction sent by a local terminal, and obtaining the synchronization completion state of a local historical database, wherein the historical database is used for executing batch synchronization operationon first historical data generated by a remote system in a first reference period; selecting a local first database or a local second database as a target incremental database according to the synchronization completion state of the historical database, wherein the first database is used for synchronizing the first historical data generated by the remote system in the first reference period in real time, and the second database is used for synchronizing the second historical data generated by the remote system in the second reference period in real time; taking all data in the historical database and the target incremental database as target synchronization data; and sending the target synchronization data to the local terminal.

Description

technical field [0001] The present invention relates to the technical field of data synchronization, in particular to a big data synchronization method, device, computer equipment and storage medium. Background technique [0002] From the perspective of a big data development platform, due to reasons such as component technical architecture or business security isolation, it is usually not possible to directly perform various calculations or retrieval processing on the data stored or generated by the online business system, so we need to These data are collected into various storage and computing components of the development platform for processing. In this process, data synchronization becomes a very important link. [0003] In the data synchronization business, a large number of data synchronization tasks are to synchronize the data of the previous day around the early morning to wait for batch analysis. In this scenario, the following problems may often be encountered:...

AI Technical Summary

Problems solved by technology

In this scenario, the following problems may often be encountered: the data may not be synchronized due to the large amount of information. For example, when the data synchronization task starts to execute on the second day, the data synchronization task on the first day has not been completed.

In this case, it is impossible to query and operate on the data of the previous day in time, such as querying the data of an insurance order generated the previous day in the insurance industry

[0004] Therefore, how to solve the problem that the new data cannot be obtained in time due to the long time-consuming synchronization process has become a technical problem to be solved urgently by those skilled in the art

Images