Stream computing system abnormity perception-oriented fault tolerance method and system

Abstract

The invention discloses a stream computing system abnormity perception-oriented fault tolerance method and system. The system comprises a computing module, a system monitoring module, a dynamic extension module, a fault recovery module and a computing resource pool module, wherein the system monitoring module periodically detects computing states of all working nodes in the system; when the detected state of a working node is in warning, the working node in the warning state is subjected to preventive fault recovery through interaction of the system monitoring module, the dynamic extension module, the fault recovery module and the computing resource pool module; the system monitoring module further monitors the working node in the warning state, and the fault recovery module performs different processing modes according to changes of the state of the working node from the warning state to a fault state, a normal state and an abnormal state. The fault tolerance method is a fault tolerance method for realizing transparency to a user in a stream computing system. The fault tolerance method disclosed by the invention has a complete fault recovery process, and the fault recovery efficiency of the stream computing system is improved.

Description

technical field [0001] The invention belongs to the field of big data distributed computing, and in particular relates to a fault-tolerant method in a stream computing system. Background technique [0002] Stream computing (Stream Computing) is a core technology that has received special attention in the field of big data processing in recent years. At the same time, stream computing service is also an important platform capability in the cloud computing system PaaS. Its main computing feature is that it can continuously process various A real-time dynamic data stream generated by a network entity. Different from big data batch processing systems such as Mapreduce and Pregel, it can enable public service systems, enterprise operation systems, and customer systems to obtain various key capabilities of big data applications such as online real-time high performance and massive throughput. Such systems have received high attention in various top international academic conferen...

AI Technical Summary

Problems solved by technology

However, the above methods have certain disadvantages

These methods perform error recovery only after system errors occur and these errors are detected, which makes the system error recovery time relatively long

In addition, when the system performs upstream backup of the state of the operator, in order to ensure the consistency of the state of the operator, the current operator must be frozen, which also has a certain impact on the computing efficiency of the system

Images