Intelligent partitioning system under distributed scene

Abstract

The invention discloses an intelligent partitioning system under a distributed scene. The system bottom layer comprises a computing node and is accessed to a storage engine. Financial data flow arriving in real time is calculated and stored through a system bottom layer, and then data processing information is transmitted to the cluster sensing module. Each key value and the distribution condition of read-write pressure in the data is analyzed by using a cluster sensing module; the obtained distribution condition is input into an intelligent prediction module, and the read-write request distribution of each key value in the data is predicted in a future period of time in real time through an online learning algorithm; and when the skew degree exceeds a threshold value, the intelligent partition migration module receives a prediction result of the prediction unit, calculates a new partition table and executes partition migration operation, so that the purpose of balancing the future pressure of the cluster is achieved. The method is suitable for the field of financial scene business, the hotspot condition of the storage system can be obviously reduced, and the response speed of the system is improved.

Description

technical field [0001] The invention relates to the field of real-time processing of time-series big data, in particular to an intelligent partition system in a distributed scenario. Background technique [0002] In the financial field, big data analysis has the characteristics of large data traffic, low latency requirements, and strong data timeliness. Especially in some specific scenarios, such as the field of high-frequency quantitative trading, the distributed data analysis system needs to monitor hundreds of thousands of stocks and futures around the world, and these stocks and futures will generate millions or even tens of millions of transactions per second Flowing water, these data will be transmitted to the distributed analysis system, and after a series of calculations, the analysis results will be fed back. In the current high-frequency quantitative trading market, the opportunities to go long and short are fleeting. If the data analysis system cannot stably feed...

AI Technical Summary

Problems solved by technology

However, in actual financial scenarios, popular stocks may change over time. For example, with the emergence of new technologies, some technology stocks may suddenly turn from cold to hot. For example, news of financial fraud suddenly appears in some popular companies. Stocks fell by the limit, resulting in a sharp drop in the volume of transaction data

These accidental events may lead to drastic fluctuations in the volume of stock trading data. However, from the perspective of the entire macro stock market, these accidental events occur every day, which leads to the hotspot distribution of stock data changing every day.

In this case, the static, hard-coded form of partitioning is extremely unstable

Images