A kudu data import system and method based on a byte stream format

A technology of data import and byte stream, which is applied in the directions of database indexing, structured data retrieval, database distribution/replication, etc. It can solve the problems of not being able to support business data importing and not being able to play, so as to achieve good promotion and application value and increase speed , the effect of easy scalability

Inactive Publication Date: 2019-01-08
INSPUR SOFTWARE CO LTD
View PDF5 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Kudu database has a good application prospect. Most of the current data sources are stored in oracle, SqlServer, and MySQL. Although Kudu database provides efficient storage, batch scanning performance and powerful data analysis capabilities, if there is no data import method to import data To the Kudu database, it can't play its role, and now there is an urgent need for an efficient and stable method to import data into the Kudu database
[0003] The existing official Kudu database storage method only supports the Kudu database storage through Impala. However, this method cannot support the import of commonly used OLTP (such as oracle, SqlServer, MySQL) business data, which has great limitations.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A kudu data import system and method based on a byte stream format
  • A kudu data import system and method based on a byte stream format

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0033] Such as figure 1 As shown, the kudu data import system based on the byte stream format of the present invention includes a source database, a source database extraction service module, a message middleware cluster module, a kudu storage service module and a kudu database, and the source database extraction service module obtains the source database The data flow of the source database is forwarded by the message middleware cluster module, and the kudu storage service module parses out the table structure data of the source database, the full amount of data of the source database, and the incremental data of the source database, and saves them in the kudu database.

[0034] By configuring the mapping relationship between the source data type and the kudu data type, the table structure of the source database is converted to the kudu database.

[0035] By receiving the byte stream containing the table structure data content of the source database, parsing the field content...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a kudu data import system and method based on a byte stream format, belonging to the technical field of software service data synchronization. The kudu data import system basedon a byte stream format includes a source database, a source database extraction service module, a message-oriented middleware cluster module, a kudu warehousing service module and a kudu database. The source database extraction service module obtains the data stream of the source database, the message middleware cluster module forwards the data stream of the source database, and the kudu warehousing service module parses the table structure data of the source database, the total data of the source database, and the incremental data of the source database, and saves the data to the kudu database The kudu data import system based on the byte stream format of the invention supports distributed deployment, can make full use of machine performance, effectively improve data storage speed, andhas good popularization and application value.

Description

technical field [0001] The invention relates to the technical field of software service data synchronization, and specifically provides a kudu data import system and method based on a byte stream format. Background technique [0002] Apache Kudu is an open source storage engine by Cloudera, which can provide low-latency random read and write and efficient data analysis capabilities at the same time, and it has the advantages of both HBase and HDFS. Kudu database has a good application prospect. Most of the current data sources are stored in oracle, SqlServer, and MySQL. Although Kudu database provides efficient storage, batch scanning performance and powerful data analysis capabilities, if there is no data import method to import data When it comes to the Kudu database, it can't play its role. Now there is an urgent need for an efficient and stable method to import data into the Kudu database. [0003] The existing official method of importing Kudu database only supports im...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/27
Inventor 许作亮邓光超李朝铭
Owner INSPUR SOFTWARE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap