Message caching system and message caching method based on Kafka

Abstract

The application discloses a message caching system and a message caching method based on Kafka. The system comprises: a service processor cluster, a main station application cluster and a data warehousing service cluster, which are all associated with a publisher's theme, wherein the main station application cluster comprises a first Kafka production for transmitting downlink request data to the service processor cluster to a downlink theme of the Kafka message queue; a first Kafka consumer of the service processor cluster sends the downlink data to the terminal, and a second Kafka producer sends the uplink data to an uplink theme of the Kafka message queue; and the data warehousing service cluster comprises a second Kafka consumer that receives the uplink theme and message theme of the Kafka message queue and bulks the warehousing. The message caching system and message caching method based on Kafka provided by the application can implement unified management of receiving / transmittinguplink and downlink data.

Description

technical field The present application relates to the technical field of message processing, and in particular to a Kafka-based message caching system and method. Background technique In recent years, electricity consumption data has grown rapidly, with 8 million new records added every day, and more than 20TB of new data per year. Since 2016, the number of entities participating in market transactions has continued to increase, and the number of large users participating in transactions has reached 390, an increase of 101% compared with the whole year of 2015. %; The trading mechanism has been continuously improved, and the frequency of transactions has been continuously increased, gradually transitioning from annual transactions, quarterly transactions, to monthly transactions, and even intraday transactions. Every day, the peak value of downlink request data and terminal uplink data generated by the main station and background applications will exceed hundreds of millio...

AI Technical Summary

Problems solved by technology

For some systems that need to process a large number of messages, such as an order system, due to the frequent occurrence of repeated messages and the consumption of more system resources (such as multiple database queries, access to file systems or network calls, etc.), it will cause consumers The amount of client database visits has increased significantly, message processing capabilities have declined, system stability and service effectiveness have declined

Images