A massive log statistical analysis system and method

Abstract

The invention discloses a massive log statistical analysis system, which includes a preprocessing module, a time sliding window module and a log result module; wherein, the preprocessing module is used to preprocess the logs according to a configured statistical analysis strategy, and The preprocessing result is sent to the time sliding window module; the time sliding window module is used to analyze the preprocessing result according to the configured statistical analysis strategy, and sends the analysis result to the log result module; the log result module is used to Assemble log retrieval statements according to the configured statistical analysis strategy and analysis results, and provide the retrieved log information to users. The invention also discloses a statistical analysis method for massive logs. By adopting the invention, the statistical analysis method for massive logs can be optimized, and the statistical analysis efficiency is improved.

Description

technical field [0001] The invention relates to log statistical analysis technology in a communication network, in particular to a massive log statistical analysis system and method. Background technique [0002] With the rapid development of modern communication technology and the increase of people's demand for information, the network structure of industry users has become more and more complex. The concentration of the network leads to the concentration of data and applications, and the application systems of data centers at all levels are getting larger and larger, and the management complexity is getting higher and higher. For security managers, it is necessary to regularly analyze the massive logs generated by a large number of network devices, security devices, application systems, databases, and hosts. Identify potential security risks through statistical analysis of massive logs. [0003] At present, there are generally three methods for the statistical analysis ...

AI Technical Summary

Problems solved by technology

[0003] At present, there are generally three methods for the statistical analysis of massive logs: the first is to store IT logs in the database, retrieve the log data through the application, and directly use the sliding window method for local statistical analysis. In some cases, the amount of data fetched locally is too large, the processing efficiency of the application is low, and it is difficult to bear the pressure of resource consumption; the second is to store the logs in the database and use the database for statistical analysis. However, since the start and end times of events are difficult to determine, Therefore, the efficiency of statistical analysis through the database is low, and the database resource consumption is large; the third method is to perform real-time statistical analysis on the obtained logs through the time sliding window. The advantage of this method is that it is more real-time. However, if the statistical time period is too long, For example, there are more than N operations related to the same account in one day, and the processing pressure of the time sliding window is too high, resulting in low analysis efficiency

Images