Method for synchronously replicating data to Hadoop platform from MySQL database based on log analysis technology

A replication method and data synchronization technology, applied in database distribution/replication, electronic digital data processing, structured data retrieval, etc., can solve problems such as data exchange problems in business systems, reduce backup burden, improve efficiency, and reduce transmission volume Effect

Inactive Publication Date: 2018-06-29
CHINA REALTIME DATABASE +1
View PDF4 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This requires that the source and target databases must be MySQL databases to use the master-slave configuration scheme, which brings difficulties to data exchange between business systems
In particular, it is very difficult to replicate the data of the MySQL database to the Hadoop platform synchronously.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for synchronously replicating data to Hadoop platform from MySQL database based on log analysis technology
  • Method for synchronously replicating data to Hadoop platform from MySQL database based on log analysis technology
  • Method for synchronously replicating data to Hadoop platform from MySQL database based on log analysis technology

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0021] One embodiment of the present invention discloses a method for synchronously duplicating data from a MySQL database to a Hadoop platform based on log parsing technology, and its main architecture is as follows: figure 1 As shown, it mainly includes several stages of log parsing, message receiving, and SQL adaptation.

[0022] Before starting formal data synchronous replication, you must first enable the binary logging function of MySQL and modify it to a row-based replication mode.

[0023] see figure 2 , use the log parsing module to filter the logical logs of the MySQL database to be processed according to rules, and send complete data according to transaction integrity. Specifically, the log parsing module analyzes the format of the logical log of the MySQL database, and obtains the user's operation instructions and operation result sets for the database according to the fixed byte reading method and parsing rules, and adds transaction integrity during the parsing ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention belongs to the technical field of power system databases, and discloses a method for synchronously replicating data to a Hadoop platform from a MySQL database based on the log analysis technology. The method includes the steps that the binary log record function of a MySQL is started, and is modified to a replication mode based on a line; a log analysis module is used for filtering rules of logical logs of the MySQL database required to be processed and sending complete data according to the transaction completeness; an information receiving module is used for receiving the datafrom the log analysis module according to configured receiving information and writing the data into a local cache data file for data loading according to the local rule; a SQL adaption module is usedfor reading the cache data file, the cache data file is converted into a general standard SQL data statement format according to the type of the Hadoop platform, and the data is loaded and enters theHadoop platform. According to the method, database synchronous-replication efficiency is improved.

Description

technical field [0001] The invention belongs to the technical field of power system databases, and in particular relates to a method for synchronously duplicating data from a MySQL database to a Hadoop platform based on a log analysis technology. Background technique [0002] With the construction of the "State Grid Resource Planning Information System" (SG-ERP) project of the International Grid Corporation of China, the State Grid Corporation of China has built relevant application systems in terms of three sets of five majors, two centers, information platforms, and comprehensive analysis and decision-making. Information system architecture is more complex. In order to ensure data consistency between different business systems, the problem of data exchange between business systems must be solved, and real-time synchronization between business system databases is one of the feasible ways to solve this problem. [0003] However, there are many types of database synchronous ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/27
Inventor 张珂珩龚长平吴志勇黄伟金发秀
Owner CHINA REALTIME DATABASE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap