A kudu data import system and method based on a byte stream format

Abstract

The invention discloses a kudu data import system and method based on a byte stream format, belonging to the technical field of software service data synchronization. The kudu data import system basedon a byte stream format includes a source database, a source database extraction service module, a message-oriented middleware cluster module, a kudu warehousing service module and a kudu database. The source database extraction service module obtains the data stream of the source database, the message middleware cluster module forwards the data stream of the source database, and the kudu warehousing service module parses the table structure data of the source database, the total data of the source database, and the incremental data of the source database, and saves the data to the kudu database The kudu data import system based on the byte stream format of the invention supports distributed deployment, can make full use of machine performance, effectively improve data storage speed, andhas good popularization and application value.

Description

technical field [0001] The invention relates to the technical field of software service data synchronization, and specifically provides a kudu data import system and method based on a byte stream format. Background technique [0002] Apache Kudu is an open source storage engine by Cloudera, which can provide low-latency random read and write and efficient data analysis capabilities at the same time, and it has the advantages of both HBase and HDFS. Kudu database has a good application prospect. Most of the current data sources are stored in oracle, SqlServer, and MySQL. Although Kudu database provides efficient storage, batch scanning performance and powerful data analysis capabilities, if there is no data import method to import data When it comes to the Kudu database, it can't play its role. Now there is an urgent need for an efficient and stable method to import data into the Kudu database. [0003] The existing official method of importing Kudu database only supports im...

AI Technical Summary

Problems solved by technology

Kudu database has a good application prospect. Most of the current data sources are stored in oracle, SqlServer, and MySQL. Although Kudu database provides efficient storage, batch scanning performance and powerful data analysis capabilities, if there is no data import method to import data To the Kudu database, it can't play its role, and now there is an urgent need for an efficient and stable method to import data into the Kudu database

[0003] The existing official Kudu database storage method only supports the Kudu database storage through Impala. However, this method cannot support the import of commonly used OLTP (such as oracle, SqlServer, MySQL) business data, which has great limitations.

Images