Data synchronization method and system based on XXL-JOB and FlinkSQL

A data synchronization and data source technology, applied in the fields of digital data protection, electronic digital data processing, structured data retrieval, etc., can solve the problems of inefficiency, single data synchronization scheme, etc. inefficient effects

Pending Publication Date: 2022-02-15
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] Aiming at the needs and deficiencies of current technological development, the present invention provides a data synchronization method and system based on XXL-JOB and FlinkSQL, which is used to solve the problem of efficient data synchronization between various heterogeneous data sources, thereby overcoming other data synchronization problems. Uniformity and inefficiency of the synchronization scheme

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data synchronization method and system based on XXL-JOB and FlinkSQL
  • Data synchronization method and system based on XXL-JOB and FlinkSQL
  • Data synchronization method and system based on XXL-JOB and FlinkSQL

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0044] combined with Figure 1-4 , this embodiment proposes a data synchronization method based on XXL-JOB and FlinkSQL, including the following steps:

[0045] (1) Based on the AES encryption algorithm, the accessed data source information is encrypted, transmitted and stored.

[0046] The AES encryption algorithm is a symmetric encryption algorithm. The specific process of encrypting and transmitting the accessed data source information based on the AES encryption algorithm is as follows:

[0047] The AES encryption function uses the key K to encrypt the plaintext P to obtain the ciphertext C;

[0048] Transfer the ciphertext C to the specified location;

[0049] The AES decryption function uses the key K to decrypt the ciphertext C to obtain the plaintext P.

[0050] (2) Build FlinkSQL tasks by selecting source database, target database, and configuring synchronization rules.

[0051] Flink SQL is a simplified computing model for Flink's real-time computing. The SQL con...

Embodiment 2

[0057] combined with Figure 2-5 , this embodiment proposes a data synchronization system based on XXL-JOB and FlinkSQL, its structure includes: AES encryption module 1, synchronization task construction module 2, timing task configuration module 3, timing task execution module 4.

[0058] AES encryption module 1 is used to encrypt, transmit and store the accessed data source information. The specific encryption process is as follows:

[0059] The AES encryption function uses the key K to encrypt the plaintext P to obtain the ciphertext C;

[0060] Transfer the ciphertext C to the specified location;

[0061]The AES decryption function uses the key K to decrypt the ciphertext C to obtain the plaintext P.

[0062] Synchronization task construction module 2 is used to construct FlinkSQL tasks by selecting source database, selecting target database, and configuring synchronization rules.

[0063] Timing task configuration module 3 is used to configure different timing rules fo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data synchronization method and system based on XXL-JOB and FlinkSQL, and relates to the technical field of data synchronization, and the method comprises the following steps: carrying out encryption transmission and storage on accessed data source information based on an AES encryption algorithm; constructing an FlinkSQL task by selecting a source database and a target database and configuring a synchronization rule; configuring different timing rules for the constructed FlinkSQL task based on XXL-JOB, and starting the timing task; after the timed task is started, submitting the Flink SQL task to the Flink cluster, and achieving data synchronization through the Flink cluster. According to the method and system, the problem of efficient data synchronization among various heterogeneous data sources can be solved, and the oneness and low efficiency of other data synchronization schemes are overcome.

Description

technical field [0001] The invention relates to the technical field of data synchronization, in particular to a data synchronization method and system based on XXL-JOB and FlinkSQL. Background technique [0002] With the continuous development of society, enterprises are building more and more supporting systems. Data is the core value of enterprise production and operation. Data sources include production data, sales data, market data and other data. Data storage also includes Relational database, NOSQL, NEWSQL, message queue, text file and other structures. Different data storages have different uses. How to synchronize data between different data storages and take advantage of different data storages is an important part of enterprise data management. Therefore, efficient synchronization of data between various heterogeneous data sources to important. However, the current data synchronization methods generally have problems such as supporting a single type of data sourc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F21/60G06F21/62G06F16/27
CPCG06F21/602G06F21/6218G06F16/27
Inventor 高鹏超张立勇孙向晖毛胜磊李宁
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap