Big data platform running state monitoring method and system

Abstract

The invention relates to a big data platform running state monitoring method and system. The method comprises the steps that all task stages of a computational task of a big data platform are determined; task execution conditions of the computational task in all the task stages are obtained through monitoring nodes deployed on the big data platform in advance, wherein the monitoring nodes and nodes where all the task stages of the computational task of the big data platform are located are correspondingly deployed; according to the task execution conditions of the computational task in all thetask stages, the running state of the big data platform is monitored. In this way, the task execution conditions of the computational task of the big data platform in all the task stages are obtainedthrough the monitoring nodes deployed in advance, the running state of the big data platform is monitored according to the task execution conditions, the problem that in the traditional technology, by monitoring a service component of the big data platform, the accuracy is low is solved, the running state of the big data platform is accurately monitored, and the running efficiency of the big dataplatform is guaranteed.

Description

technical field [0001] The present invention relates to the field of big data technology, in particular to a method and system for monitoring the running state of a big data platform, computer equipment and a computer-readable storage medium. Background technique [0002] With the continuous improvement of social informatization, massive and real-time data have been generated in many business fields. At present, big data analysis is gradually dominated by unstructured, and the storage space and computing power of a single machine are difficult to meet the needs. Therefore, big data systems based on distributed hadoop clusters, spark clusters, and storm clusters are widely used. Big data cluster systems usually deploy hundreds of nodes. The expansion of node scale not only makes the cluster resource configuration and service deployment and maintenance of the big data platform more difficult, but also makes the submission, scheduling, Tasks such as retrying and canceling alar...

AI Technical Summary

Problems solved by technology

[0004] Traditional technologies usually obtain the running status of the big data platform by monitoring the service status of each service component of the big data platform. However, due to the limited service components of the big data platform and the difficulty of secondary development of the components, it is difficult to monitor the big data platform. For example, it is difficult to guarantee the computing quality of the computing tasks of the big data platform, which leads to the low accuracy of this technology in monitoring the running status of the big data platform.

Images