Message system resource balancing device, method and equipment

Abstract

The embodiments of the invention provide a message system resource balancing device, method and equipment. The balancing device comprises a load monitoring engine, an analysis engine and an operationexecution engine, the load monitoring engine is used for collecting a Kafka index from the Kafka cluster and sending the Kafka index to the analysis engine; the analysis engine is used for, generatinga first equalization scheme corresponding to all subscribers according to the Kafka index, the hard target of the Kafka cluster and the soft target of the Kafka cluster, and sending the first equalization scheme to an operation execution engine, wherein the hard target is a target that must be realized by a plurality of Kafka clusters, and the soft target is a target that is preferably realized by a plurality of Kafka clusters; and the operation execution engine is used for generating a plurality of operation instructions according to the first equalization scheme and sending the plurality ofoperation instructions to the Kafka cluster so as to enable the Kafka cluster to perform resource equalization according to the operation instructions. According to the invention, the resource automatic equalization operation can be executed by taking hard and soft targets as a core basis, and the performance of a message system and the user experience are improved.

Description

technical field [0001] Embodiments of the present invention relate to the technical field of cloud computing, and more specifically, to a message system resource balancing device, method and device. Background technique [0002] With the in-depth application of cloud computing in all walks of life, the scale of application systems and tenants is getting larger and larger, and the requirements for real-time interaction between systems, real-time consumption of platform data and application performance data are also getting higher and higher, especially in telecommunications In the financial field, its core business system runs on the cloud platform, which puts forward higher requirements for the real-time performance of platform data and application business performance data of large-scale application systems. In this regard, the cloud management platform is also facing new challenges: the underlying technology is complex , There are many subsystems, frequent interactions bet...

AI Technical Summary

Problems solved by technology

[0003] With the all-inclusive management of resources such as IaaS, PaaS, and SaaS, the scale of managed platforms and business systems is getting larger and larger, and message system clusters need to process hundreds of thousands or even millions of messages every day. Such large-scale messages If the message processing of any message system service (Broker) fails, it will lead to unbalanced load of our overall cluster, and even cause an "avalanche" effect

As a result, the real-time data of the platform cannot be collected and updated; resources (containers, virtual machines, physical machines, databases, etc.) cannot be applied for and opened; historical data and business index data cannot be consumed in a timely manner, etc. These problems will directly affect the reliability of the platform and tenant usage perception

Therefore, in order to ensure the balance and availability of the message system cluster, it takes a lot of time and effort to redistribute partitions and rebalance replicas

[0004] These problems reflect that the processing capacity of the message system at this stage cannot meet the explosive growth of data requirements

[0005] To solve the above problems, the current common processing techniques are as follows: The first is to pre-set a sufficient number of message topic partitions, but this method cannot achieve partition balance

The second is to distinguish different types of message data through different topics to meet business needs; many topics need to be created, which will bring additional management costs

For real-time data and historical data consumption, it can only be satisfied by manually increasing the message consumption process, and reducing the message consumption process for a small amount of time. This method cannot realize automatic and dynamic expansion (zooming) of resources, and the resource utilization rate is low.

[0006] To sum up, the existing technology has the following defects: first, it is necessary to pre-set the partition size of each message topic and the resource utilization rate of each Broker for the message system, which cannot be adjusted in real time, which may easily cause unbalanced message copies

Moreover, real-time performance data cannot be obtained immediately

Images