Message processing system and processing method based on kafka

Abstract

The invention discloses a message processing system and processing method based on kafka and belongs to the technical field of message processing. The message processing system and processing method based on kafka are aimed to solve the problems of the prior art that repeated messages appear in an order system frequently, the processing for each message consumes much system resource, the system resource consumption would greatly increase the database visitor volume at the consumer end, the message processing ability is lowered, and the system stability and service effectiveness are lowered. The message processing system based on kafka comprises a message producer cluster composed of one or more message producers, a kafka cluster, and a consumer cluster composed of one or more consumers, the kafka cluster is composed of one or more servers, and the consumer cluster comprises a filter module, a local message queue caching module and a business processing module. The message processing method realized through the system is used for the system especially the order system which needs to process a large number of messages.

Description

technical field [0001] The invention specifically relates to a kafka-based message processing system and processing method, and belongs to the technical field of message processing. Background technique [0002] Kafka is a distributed message queue system (Message Queue) developed by Linkedin. [0003] The main goal of kafka development is to build a data processing framework for processing massive logs, user behavior and website operation statistics. In the case of combining data mining, behavior analysis, operation monitoring and other requirements, it needs to be able to meet the requirements of various real-time online and batch offline processing applications for low latency and batch throughput performance. Fundamentally speaking, high throughput is the first requirement, followed by real-time and persistence. [0004] The Kafka cluster consists of multiple Broker servers. Each type of message is defined as a topic. Messages within the same topic are partitioned an...

AI Technical Summary

Problems solved by technology

[0006] Therefore, the present invention aims at the fact that repeated messages often appear in the order system in the prior art and processing each message needs to consume more system resources, which will cause a large increase in the number of database visits at the consumer end, a decline in message processing capabilities, and system stability and service. For the problem of reduced effectiveness, provide a kafka-based message processing system, including a message producer cluster composed of one or more message producers, a kafka cluster, a consumer cluster composed of one or more consumers, and a kafka cluster Composed of one or more servers, each server stores messages according to a certain topic, each message producer submits messages to one or more servers, and each consumer subscribes messages to one or more servers, characterized in that the The consumer includes a filter module, a local message queue cache module, and a business processing module. The filter module is a container that does not allow repeated data, and is used to receive messages subscribed by consumers from the server. The local message queue cache module uses For caching the message filtered by the filter module, the business processing module is used to obtain the message from the message queue cache module, and complete the business processing of the message

Images