Dynamic updating method and system for data stream distribution process configuration

A technology of dynamic update and data flow, applied in the field of data processing, can solve the problem of low efficiency of dynamic update of configuration, and achieve the effect of improving the efficiency of dynamic update

Active Publication Date: 2021-03-19
中科美络科技股份有限公司
View PDF13 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is how to provide a method and system for dynamically updating the configuration of the data stream distribution process to solve the technical problem in the prior art that the efficiency of dynamic update of the configuration of the data stream processing and distribution process is low

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Dynamic updating method and system for data stream distribution process configuration
  • Dynamic updating method and system for data stream distribution process configuration

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0050] First of all, it should be explained that the present invention figure 1 The illustrated embodiment is preferably applicable to scheduling processes.

[0051] figure 1 A schematic flowchart of a method for dynamically updating data stream distribution process configuration provided by an embodiment of the present invention, as shown in figure 1 As shown, the method includes:

[0052] S101: Obtain first configuration data of a target data stream distribution process at the current moment, and acquire second configuration data for the target data stream distribution process at the current moment, wherein the target data stream distribution process includes: kafkastream.

[0053] The data stream distribution process includes but is not limited to kafka stream, and can also include Storm Stream, SparkStreaming, Flink Stream, etc. when the first configuration data is updated, the data stream distribution process needs to be restarted. In the embodiment of the present inve...

Embodiment 2

[0069] In order to avoid the overload of the server caused by simultaneous dynamic updates of multiple kafka streams, Embodiment 2 of the present invention uses the following method on the basis of Embodiment 1 to generate a backup data stream distribution process corresponding to the target data stream distribution process in the cache :

[0070] The target data flow distribution processes kafka stream-1, kafka stream-2, kafka stream-3, and kafkastream-4 are all deployed on the same server A. And each kafka stream corresponds to a data processing process. At this point, kafka stream-1, kafka stream-2, kafka stream-3, and kafka stream-4 all need to perform dynamic update operations, and kafka stream-1, kafka stream-2, kafka stream-3, and kafka stream-4 first send The first zookeeper competes for the zookeeper lock. For example, if kafka stream-1 competes successfully first, then kafka stream-1 generates a backup data stream distribution process corresponding to the target dat...

Embodiment 3

[0074] In order to balance the load between servers, Embodiment 3 of the present invention uses the following method on the basis of Embodiment 1 to realize generating a backup data stream distribution process corresponding to the target data stream distribution process in the cache:

[0075]All target data flow distribution processes deployed on each server request to the second zookeeper, so that the second zookeeper generates coordination instructions according to the remaining computing power of each server, and the coordination instructions are used to allocate and generate backup data for each server The number of tasks of the flow distribution process; each server generates a backup data flow distribution process corresponding to the target data flow distribution process in the cache according to the coordination instruction sent by the second zookeeper after receiving the coordination instruction sent by each server.

[0076] Exemplarily, the target data stream distribu...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a dynamic updating method and system for data stream distribution process configuration, which are applied to a scheduling process, and the method comprises the following steps:A, obtaining first configuration data of a target data stream distribution process at the current moment, and obtaining second configuration data for the target data stream distribution process at the current moment, wherein the target data stream distribution process comprises a kafka stream; B, comparing whether the first configuration data is the same as the second configuration data or not; C, if not, generating a backup data stream distribution process corresponding to the target data stream distribution process in the cache; and taking the backup data stream distribution process as a target data stream distribution process to enable the target data stream distribution process to distribute data streams to a data processing process, the data streams including log data streams. By applying the embodiment of the invention, the dynamic updating efficiency of the configuration data can be improved.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method and system for dynamically updating configuration of a data stream distribution process. Background technique [0002] In the process of data flow processing, it is often encountered that the configuration of the process related to data flow needs to be updated. For example, the invention patent application with the application number of 201410314207.6 in the prior art discloses a logic of broadband access network The file dynamic loading method realizes the dynamic loading of FPGA logic files through the TCP protocol. After the loading is completed, the system does not need to be restarted, and during the operation of the board, a new version of the FPGA logic file can be loaded at any time to replace the currently running FPGA logic. file, which improves the flexibility of the system while ensuring the transmission efficiency and reliability; in the actual oper...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/24G06F11/14G06F9/48
CPCH04L41/082G06F9/4881G06F11/1464
Inventor 张洋吴同仁肖伟董俊庆杨元山付永庆
Owner 中科美络科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap