Data persistence method and device

A persistence and data technology, applied in the computer field, can solve problems such as inability to load data

Inactive Publication Date: 2018-09-11
NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
View PDF5 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is to provide a data persistence method and device to solve the problem in the prior art that different storage engines cannot uniformly load data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data persistence method and device
  • Data persistence method and device
  • Data persistence method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] AVRO is a data serialization system that provides rich data structures, compressed, fast, binary data formats, and easy interaction with dynamic languages.

[0023] The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0024] This embodiment provides a data persistence method. Such as figure 1 Shown is a flowchart of a data persistence method according to an embodiment of the present invention.

[0025] Step S110, from the preset Topic (message topic) in Kafka (distributed message system), obtain data that has been converted into a unified data type in advance.

[0026] Before obtaining the data that is pre-converted into a unified data type, obtain the metadata information of the data to be persisted; determine the data type of the data accor...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data persistence method and device. The method comprises the following steps: obtaining data which is pre-conversed into a unified data type from a pre-set Topic in Kafka; selecting a target storage engine; and loading data in the Topic through Flume into the target storage engine in order to facilitate performing data persistence operation, wherein the unified data typeof data is converted into an original data type of the data before the target storage engine performs data persistence operation. The invention provides the data persistence method supporting multiplestorage engines based on Kafka and Flume. In the invention, unified loading of the multiple storage engines is achieved by adoption of a same data type through one-time data serialization (conversioninto the unified data type). Through one-time deserialization (coversion into an original data type), highly-efficient data persistence of the multiple storage engines is achieved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a data persistence method and device. Background technique [0002] With the rapid development and popularization of computer and information technology, the scale of industrial application systems has expanded rapidly, and the data generated by industrial applications has grown explosively. Industry / enterprise big data, which can easily reach hundreds of terabytes or even hundreds of petabytes, has far exceeded the processing capabilities of traditional computers and information technology systems. How to quickly and effectively summarize these massive data and quickly load them into the big data platform , so as to support the statistics and analysis of follow-up data, has become an urgent need of the current big data ecosystem. [0003] In a big data storage system, multiple storage engines are usually required for different application scenarios, because data metadata desc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 马秉楠吕雁飞白堃张鸿刘欣然惠榛
Owner NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap