Message processing method and device

Abstract

The invention discloses a message processing method and device, and relates to the technical field of computers. One specific embodiment of the method comprises the following steps: acquiring positioninformation of an unread message in a partition according to a message theme and the partition subscribed by a message user, and storing the position information of the unread message in a cache; counting the number of the unread messages, and allocating threads corresponding to the number of the unread messages, the threads being used for acquiring the unread messages; and obtaining an unread message in the partition and corresponding to the position information in the cache through the thread, so as to carry out service processing on the obtained message. According to the method, the position information of the unread message is cached, the unread message corresponding to the position information in the cache is acquired through the threads, and then service processing is performed on the unread message, so that the unread message can be acquired according to the specified position information in the cache, and the message processing real-time performance is improved.

Description

technical field [0001] The invention relates to the computer field, in particular to a message processing method and device. Background technique [0002] Kafka is a high-throughput distributed message publish-subscribe system. The existing Kafka-based message processing system includes a Kafka cluster composed of one or more message producers (producers, also referred to as producers in this article), and one or more servers (Kafka instances, also referred to as Brokers) As well as the message consumer (consumer, also referred to as the consumer in this article), the message generator and the message consumer are usually called the client, and the Kafka cluster is called the server. The message generator is responsible for generating and publishing messages to Broker, and the message consumer is responsible for reading messages from Broker and executing corresponding business logic processing. Each message published to the Kafka cluster has a category, which is called a t...

AI Technical Summary

Problems solved by technology

[0005] (1) Poor real-time performance of message processing

In the case of a large backlog of data on the server, the message consumer still reads the messages in the corresponding partition sequentially, which will cause slow data reading or network congestion, so that the latest data will be read and processed with a delay, and the message consumer will run Busy, reduced processing power

[0006] (2) If the message volume is large, the number of unprocessed messages will be large, which will occupy a large amount of local storage

[0007] (3) Deployment and maintenance costs are high, and multiple machines process messages repeatedly, wasting bandwidth

Images