Data real-time synchronization method and device, equipment and storage medium

A real-time synchronization and data technology, applied in the field of data processing, can solve problems such as real-time business impact, high coupling, and inability to automatically detect data source differences, etc., to ensure integrity, improve data query efficiency, and meet massive data query requirements Effect

Pending Publication Date: 2022-04-05
ZHONGAN ONLINE P&C INSURANCE CO LTD
View PDF0 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the first implementation method has a high degree of coupling. Whether the interface caller application or the interface provider application has a system exception, it will affect the real-time business; what's more, it may cause systemic risks; the second implementation method, Although the decoupling between applications is realized; however, if the s

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data real-time synchronization method and device, equipment and storage medium
  • Data real-time synchronization method and device, equipment and storage medium
  • Data real-time synchronization method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0060] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0061] In order to facilitate those skilled in the art to understand the embodiment, some terms are explained:

[0062] (1) ES: Elasticsearch, search server.

[0063] (2) Mysql: a relational database management system.

[0064] (3) Kafka: A high-throughput distributed publish-subscribe message system that can handle all action flow data of consumers in the website; these data are usually resolved by processing logs and log aggregation due to throughput requirements .

[0065] (4) Binlog: Binary log file, used to record data upda...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a real-time data synchronization method and device, equipment and a storage medium. The method comprises the steps of creating a plurality of ES indexes based on a synchronization system; the synchronization system creates heterogeneous data based on the updated data; the synchronization system writes the heterogeneous data into the corresponding ES indexes; the synchronization system sends each piece of operation information of the ES to a compensation system; the compensation system determines a compensation target based on the operation information. According to the technical scheme, real-time synchronization of Mysql and ES data is achieved, the data can be directly queried from the ES index without being queried through Mysql full-library full-table scanning, the data query efficiency is improved, the mass data query requirement is met, and service processing and data synchronization are completely decoupled; in addition, data loss caused by various accidents is automatically compensated through timing comparison between heterogeneous data sources, and the data integrity is ensured under the condition that manual intervention is not needed.

Description

technical field [0001] The invention belongs to the technical field of data processing, and in particular relates to a method, device, equipment and storage medium for real-time data synchronization. Background technique [0002] The search engine Elasticsearch (ES for short) can be used as a heterogeneous data source of Mysql to make up for the shortcomings of Mysql sub-database and sub-table query. ES is an open source, distributed, high-performance search engine, each field can be searched as an index field. However, there will be corresponding problems when using ES. For example, how to store the data in Mysql into ES in near real-time synchronously, that is, the problem of data consistency. Due to network fluctuations and other reasons, some synchronous data may be lost. When this scenario occurs, how to compensate for the lost data and ensure data integrity. [0003] At present, the real-time synchronization of Mysql and ES data is realized based on the following two...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/22G06F9/54G06F16/23G06F16/2453G06F16/27
Inventor 全芸乔仁伟
Owner ZHONGAN ONLINE P&C INSURANCE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap