Dynamically configurable big data analysis system and method

Abstract

The invention discloses a dynamically configured big data analysis system and method. The system includes a real-time data storage management module, a real-time flow analysis and calculating module, an off-line analysis module, and a visualization module. At least one assembly capable of performing dynamical configuration management is arranged in each module, for example, a data management configuration assembly, a real-time flow analysis and calculating configuration assembly, an off-line analysis and calculating configuration assembly, and a dynamic configuration assembly. The invention further provides a dynamical configuration method of the big data analysis system. A data structure and an information structure of each module are designed, and a calculating method of warning redundancy and a dynamic configuration method are provided through the dynamic configuration of a state information driving system of a warning data structure in a dynamic configuration manager. According to the invention, the system can run at an efficient big data analysis and calculating level, and the optimization process of big data analysis platform management is effectively solved.

Description

technical field [0001] The invention relates to the application field of big data analysis, in particular to a dynamically configurable big data analysis system and method. Background technique [0002] Today's business intelligence systems, decision support systems, etc. are increasingly required to support big data integration and analysis. Due to the large amount of data, complex process, and long processing time of big data analysis and calculation, big data analysis and applications are also facing a new challenge. Challenges: The system must have high reliability, and the software system is required to be adaptive to changes. These systems need to have the ability to update the configuration without interrupting the system service. Fault tolerance management issues, how to handle exceptions in the case of update failure, make The system maintains normal and stable operation. That is, dynamic configuration technology is an important means to realize the adaptive reliab...

AI Technical Summary

Problems solved by technology

However, although the real-time performance and fault-tolerant performance of this new real-time stream computing framework have been greatly improved, the high reliability and high availability of the system are still a challenging problem.

Images