A Distributed Environment Coordinated Consumption Queue Method and Device

Abstract

Embodiments of the present application provide a method and device for coordinating consumption queues in a distributed environment, which relate to the field of distributed technologies. The method includes: for a queue fragment to be consumed, based on the state data of the queue fragment acquired by the lease agreement; according to the state data, judging whether other clients are consuming the queue fragment; if determined If no other client is consuming the queue fragment, update the state data of the queue fragment, and obtain the current consumption progress of the queue fragment; continue to consume the queue fragment according to the current consumption progress, and set The new consumption progress of the queue fragments is recorded. This application can seamlessly transmit the consumption progress of the queue fragment to the current client when the current client A preempts the queue fragments that other clients are consuming under the circumstances of load balancing, a certain client crash, etc. Avoid repeated consumption of some data and make consumption results more accurate.

Description

technical field [0001] The present application relates to the field of distributed technologies, in particular to a distributed environment coordinated consumption queue method and a distributed environment coordinated consumption queue device. Background technique [0002] With the advent of the era of cloud computing and big data, the sources of data are getting wider and faster, and the quantity is getting bigger and bigger. For example, on the web server, various clients, sensors, etc., a large amount of data is generated in real time, and various user access requests, monitoring data, program running status and other information are recorded. In order to better mine the value of data, there are often multiple systems consuming these data, such as various real-time and offline systems for user behavior analysis, monitoring and alarming, program intrusion detection, etc. [0003] In order to reduce the coupling between data production and consumption, a queue system is u...

AI Technical Summary

Problems solved by technology

Although the Kinesis Client Library supports new clients to consume data, when performing operations such as load balancing, a client directly re-consumes the preempted queue fragments, which will lead to repeated consumption of some data, resulting in inconsistent consumption results. accurate

Images