Kafka-based consumer-side load balancing method

A load balancing, consumer technology, applied in resource allocation, inter-program communication, instruments, etc., can solve problems such as increased energy consumption and reduced efficiency, avoid load balancing attempts, improve load balancing efficiency, and improve message processing capabilities and system stability

Pending Publication Date: 2022-01-18
BEIJING UNIV OF TECH
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In order to solve the above problems, the present invention proposes a Kafka-based consumer load balancing algorithm, which can solve the problem of increased energy consumption and reduced efficiency caused by each consumer independently completing load balancing tasks during the consumer load balancing process

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Kafka-based consumer-side load balancing method
  • Kafka-based consumer-side load balancing method
  • Kafka-based consumer-side load balancing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] Attached below Figure 1-4 The present invention is described further:

[0023] The present invention provides a method for load balancing based on Kafka, which is based on the Kafka message system, and the specific flow of the system is as follows figure 1 As shown, firstly, the producer cluster composed of producers publishes messages, and sends the information to be transmitted to the proxy server. The broker classifies the messages. According to the different types of messages, they are stored on the broker with different topics, and the topics will be It is divided into several partitions. Different partitions of the same topic may not be on one machine, and may be deployed on multiple machines. Consumer groups composed of consumers will subscribe to different partitions of Topic if they want to subscribe information. In the Kafka cluster One of the consumers will act as the coordinator of the entire consumer group, which is responsible for the consumption displac...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Kafka-based consumer side load balancing method, and belongs to the technical field of message processing. In order to solve the problem that consumers are isolated from one another in an existing consumer-side load balancing method, and the problem that the consumer message processing capacity is reduced due to consumer downtime cannot be efficiently solved, the invention provides an improved consumer-side load balancing method. In the whole process from producer cluster producing messages to broker storing messages to consumer cluster receiving messages in a Kafka message transmission system, coordinators selected at the beginning of load balancing are used for uniformly managing and distributing corresponding conditions of consumers and partitions, and different distribution schemes are formulated according to load conditions of the partitions, so that the problem that consumers are isolated from one another in an existing load balancing strategy is well solved, the Kafka system can be helped to efficiently and quickly process sudden load balancing events, and the applicability of the Kafka in a production environment is improved.

Description

technical field [0001] The invention specifically relates to a Kafka-based consumer load balancing method, which belongs to the technical field of message processing. Background technique [0002] Kafka is a leader in message middleware and an important means to solve the problem of message delivery in distributed systems. Kafka was developed by LinkedIn in Scala and later became an open source project of the Apache Software Foundation. It has the advantages of supporting multi-language development on the application side and supporting real-time large-scale message processing. There are mainly producers (Producer), consumers (Consumer) and brokers (Broker) in Kafka. Generally speaking, a Kafka cluster consists of multiple broker nodes. When a producer generates a new message, it will be divided and classified. In a certain topic, each topic is divided into multiple partitions, and these partitions are deployed on multiple Brokers. The producer continuously sends messages ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50G06F9/54
CPCG06F9/5083G06F9/546G06F9/542
Inventor 孙恩昌苏玉钊张延华杨睿哲司鹏搏李萌王菲
Owner BEIJING UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap