Chained processing-based data collecting system and method

Abstract

The invention discloses a chained processing-based data collecting system and method. The chained processing-based data collecting system mainly comprises a server cluster, and a system structure of the chained processing-based data collecting system comprises a data acquiring layer, i.e. a Client end, a data transfer layer, i.e. a Transfer end, and a data storage layer, wherein the data acquiring layer is completed by the Client end and is positioned on each server and used for completing specific data acquiring tasks and transmitting data to the Transfer end; the data transfer layer is completed by the Transfer end, and is in charge of receiving the data and carrying out chained processing on the received data through a series of Handlers; and the data storage layer is used for storing the data processed by the transfer layer or original data of the acquiring layer and is used as a storage engine of subsequent analysis display. By adopting the invention, through providing a general data format and a general processing structure, the generality and the extendibility of the data collecting system are improved, and a data processing node of access and dynamic extension of a multi-source data format is supported.

Description

technical field [0001] The invention relates to network communication and log collection technology, in particular to a data collection system and method based on chain processing, which is suitable for collection of performance data in a cluster system. Background technique [0002] With the growth of application scale, system administrators need to monitor the operation of the online system in real time, and also need to know the health status of each server. The monitoring items mainly include service logs and system load. The most primitive way is to log in to each system to check, but when there are many servers, the work becomes cumbersome and meaningless. [0003] Of course, there are many existing monitoring systems, such as nagios, cacti, ganglia, etc., which can realize real-time monitoring of the system, which can grasp the status of the application system to a large extent, but these monitoring systems have some limitations in data collection : It is highly de...

AI Technical Summary

Problems solved by technology

The most primitive way is to log in to each system to check, but when there are many servers, the work becomes cumbersome and meaningless

[0003] Of course, there are many existing monitoring systems, such as nagios, cacti, ganglia, etc., which can realize real-time monitoring of the system, which can grasp the status of the application system to a large extent, but these monitoring systems have some limitations in data collection : High dependence on raw data, difficult to integrate multi-source data; strong coupling of collection and storage, unable to flexibly customize data, etc.

Therefore, there is a disadvantage of large lag caused by offline

Images