Data processing method, system and device and storage medium

A data processing and data technology, applied in the field of data processing, can solve problems such as poor scalability, poor fault tolerance, and poor real-time performance, and achieve the effects of low latency, good throughput, and guaranteed data accuracy

Inactive Publication Date: 2019-10-15
PEOPLE'S INSURANCE COMPANY OF CHINA
View PDF4 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Various aspects of the present application provide a data processing method, system, device, and storage medium, using a distributed computing framework to solve the problems of poor real-time performance, poor scalability, and poor fault tolerance in the data synchronization process

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, system and device and storage medium
  • Data processing method, system and device and storage medium
  • Data processing method, system and device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0016] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and corresponding drawings. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0017] In view of the large amount of data, many data files, and inconsistent data formats in the current business data system, resulting in poor real-time performance, poor scalability, and poor fault tolerance in the data synchronization process, in some embodiments of the application, multiple computers The distributed computing framework is deploy...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a data processing method, system and device and a storage medium. In some embodiments of the invention, a distributed computing framework is deployed on a plurality of computer devices, and the distributed computing framework comprises a master control node, a data acquisition node, a message management node, a data flow engine node and a synchronization node. Under the control of the main control node, the data acquisition node, the message management node, the data flow engine node and the synchronization node process the data in real time, so that the data synchronization efficiency is improved, and the data processing expansibility and fault tolerance are improved. According to the method for processing the data stream by adopting the distributed computing framework, the data accuracy can be ensured under the condition of data disorder or delayed loading. Under the condition that a fault occurs, data recovery processing is carried out from alast completed check point through a check point mechanism, it is ensured that one-time state semantics are kept in an application program, large-scale parallel computing is supported, and good throughput and low delay are achieved.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a data processing method, system, device and storage medium. Background technique [0002] Data synchronization is to synchronize data between different systems. Depending on the specific business purpose and application scenarios, the functional focus of various data synchronization service frameworks is often different, such as data transmission services, data collection services, data exchange services, and so on. [0003] At present, the business data system has a large amount of data, many data files, and inconsistent data formats, resulting in poor real-time performance, poor scalability, and poor fault tolerance during the data synchronization process. Contents of the invention [0004] Various aspects of the present application provide a data processing method, system, device, and storage medium, using a distributed computing framework to solve th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/178G06F16/182G06F9/54G06F11/14H04L29/08
CPCG06F9/546G06F11/1438G06F11/1458G06F2209/548G06F16/178G06F16/182H04L67/1097
Inventor 郭翔邵利铎奎志钢帅玉廷杨亚刚楚彦凌周红娟闫冰王晗
Owner PEOPLE'S INSURANCE COMPANY OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap