System and method for realizing real-time data association in big data environment

Abstract

The invention relates to a system and method for realizing real-time data association in the big data environment. The system comprises an association scheduling framework, a distributed storage framework, a relational database and Redis servers, wherein the association scheduling framework is used for conducting intra-class association, inter association and concurrent scheduling on business objects; the distributed storage framework is used for establishing a connection pool for a master server of each Redis server, storing data related to current data association, sending cold data surpassing updating time limit and excessive data surpassing memory capacity limit to the relational database, and inquiring key values of intra-class association and inter association transferred to the relational database; the relational database is used for storing key values of intra-class association, key values of inter association, intra-class contexts and monitoring contexts. Through the adoption of the system and method, real-time association supporting mass streaming data can be realized, the demand of complex association in various conditions can be met, the model generality is high, and the application range is relatively wide.

Description

technical field [0001] The invention relates to the field of computer applications, in particular to the field of real-time monitoring of high-throughput stream data, and specifically refers to a system and method for realizing real-time data association in a big data environment. Background technique [0002] The era of big data computing has arrived. This field is mainly divided into two directions: batch processing of static data and real-time processing of streaming data. Real-time data association is to receive data streams generated by multiple data sources in real time, and filter, deduplicate / change, generate association key values, intra-group association, inter-group association, multi-party splitting, and follow-up link triggering according to certain business correlations Wait for a series of processing. The associated data presents a group of data contexts, which can meet the needs of subsequent processing links such as process restoration, alarm judgment, tren...

AI Technical Summary

Problems solved by technology

The association itself is also hierarchical, including the generation of associated key values, intra-group association, inter-group association, parallel association, etc. This set of complex processing logic also needs to meet quality requirements such as high throughput, low latency, and good horizontal expansion capabilities , these are difficult to achieve with SQL-based technical solutions

Images