Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data stream processing method and system

A processing method and processing system technology, applied in the field of data stream processing methods and systems, can solve the problems of high idle ratio of Flume collection threads, affect the timeliness of fault processing, and uneven business data volume, so as to simplify business online and offline operations , Solve the uneven use and waste of resources, and shorten the operation time

Inactive Publication Date: 2019-06-07
SUNING COM CO LTD
View PDF6 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In the process of realizing the present invention, the inventors found that: when a node device has a regular failure (such as deadlock, abnormal consumption, etc.), the system has no perception and requires manual processing, which affects the timeliness of failure handling; in addition, due to the same cluster Each node adopts the same configuration, but the amount of business data of each node is uneven, which may easily lead to a large idle ratio of Flume collection threads; in addition, because new business operations are frequently launched, and business configuration files need to be manually modified, a restart is required when modifying The entire cluster, thus affecting the normal execution of other businesses in the same cluster

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data stream processing method and system
  • Data stream processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Some, but not all, embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0052] In the description of the present application, it should be understood that the terms "first", "second" and so on are used for descriptive purposes only, and should not be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, "plurality" means two or more.

[0053] Befor...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data stream processing method and system, and belongs to the field of big data processing. The method comprises the following steps: determining one of a plurality of Masternodes as a main node through a Zookeeper cluster; the main node provides an external interface to receive an online request of a service and distributes a task to the service; the main node generatesconfiguration information of the task according to the current state information reported by the Worker nodes and writes the configuration information into a ZooKeeper cluster, and the configuration information comprises information indicating the Worker nodes scheduled to execute the task; and if the Worker node monitors that the task scheduled to the Worker node exists in the ZooKeeper cluster,it starts Flume service to execute. According to the invention, the high availability of the Master node and the Worker node can be realized, the availability of Flume service is improved, and the problems of uneven resource use and waste are avoided; moreover, the online and offline operations of the services can be greatly simplified, and the mutual influence between the services is reduced.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a data flow processing method and system. Background technique [0002] In the prior art, the Flume service is usually started on each node of the cluster to transfer data from the data source to the sink. [0003] In the process of realizing the present invention, the inventors found that: when a node device has a regular failure (such as deadlock, abnormal consumption, etc.), the system has no perception and requires manual processing, which affects the timeliness of failure handling; in addition, due to the same cluster Each node adopts the same configuration, but the amount of business data of each node is uneven, which may easily lead to a large idle ratio of Flume collection threads; in addition, because new business operations are frequently launched, and business configuration files need to be manually modified, a restart is required when modifying The entire cluster a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
CPCG06F9/50
Inventor 郭业俊李浩王志强孙迁
Owner SUNING COM CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com