Real-time data processing method

Abstract

The invention provides a real-time data processing method. The real-time data processing method comprises the following steps of: enabling a rocketmq server to push real-time data to a corresponding message queue; wherein the message queue comprises a plurality of message sub-queues, and averagely distributing the real-time data to the plurality of message sub-queues; scheduling the messages on the message sub-queues by adopting a load balancing algorithm to obtain scheduling information; distributing the scheduling information to a plurality of clients; and sending the scheduling information to a data processing module by the clients for data processing. The problem that messages cannot be received due to downtime of a single mq client is solved, data receiving performance is improved, real-time data processing pause and restart are achieved, the problem of updating loss under the condition of high concurrency is fundamentally solved, meanwhile, data loss caused by data overstock or data processing failure is avoided, and the reread data processing performance is greatly improved.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a real-time data processing method. Background technique [0002] Redis cache: Redis is a high-performance key-value memory database. In the official performance test, the set operation is 110,000 times per second, and the get operation is 81,000 times per second, which are often used for caching. [0003] rocketmq: Ali open source distributed message queue. Rocketmq uses topic for message management. Each topic contains multiple partition queues. Messages sent by a publisher to a topic will be evenly distributed to multiple partitions. Zookeeper coordinates and controls. When a consumer joins or leaves, it triggers a load balancing algorithm to balance the load of multiple consumers in a consumer group. [0004] LevelDB: Leveldb is a very efficient kv database implemented by Google, which can currently support billions of data volumes. LevelDB is a single-process service with ...

AI Technical Summary

Problems solved by technology

[0006] If the existing solution applies to the unexpected downtime of the RocketMQ client, resulting in a backlog of messages that cannot be processed, affecting the real-time performance of data

[0007] In the existing technology, data processing tasks can only be scheduled by the thread itself, and cannot be paused and restarted

In the existing technology, data will be updated by multiple threads at the same time, resulting in the loss of updates. If you want to avoid this situation, you cannot achieve high-concurrency and high-performance message processing

If the message processing fails, unprocessed messages need to be read from the database. If there are too many unprocessed messages, the performance of Broker will decrease and the processing efficiency will be low.

Images