Method and system for analyzing large log

Abstract

The invention discloses a method for analyzing a large log, which comprises the steps that a log center divides a large log file into n modules to transmit to various nodes through encryption; a node host analyzes and processes the log, and sends an analysis result to a data center through the encryption; and log analysis results of various nodes are stored in a database by the data center after collated and merged. In addition, the invention further discloses a system for analyzing the large log. According to the method and the system, the resource consumption of the log center can be reduced; log collection is prevented from being influenced; the quality of the log collection is ensured; the log analysis efficiency and the real-time performance can be improved; and the analysis results in the database can be updated timely.

Description

technical field [0001] The invention relates to the field of computer information security, in particular to a method and system for analyzing large logs. Background technique [0002] With the development of information technology, the amount of data is increasing rapidly, and the accumulation of data is also increasing. Security is an important consideration in data transmission, exchange and processing. For this reason, many devices related to information processing (such as firewalls, intrusion detection systems, routers and servers, etc.) will generate logs, which record Various things happen on the device and in the network every day, and the status of each device and the entire network can be understood through the query and statistics of the logs. [0003] However, how to analyze such a large amount of log data better and faster is a difficult problem faced by administrators. Currently, there are log management products such as SIME\SOC\SIM\SEM in the industry, all ...

AI Technical Summary

Problems solved by technology

[0003] However, how to analyze such a large amount of log data better and faster is a difficult problem faced by administrators. Currently, there are log management products such as SIME\SOC\SIM\SEM in the industry, all of which involve log collection and analysis modules. Most of the log bottlenecks of these products are on the analysis server IO. The processing speed of massive logs is very slow, resulting in low overall efficiency of collection and analysis. This will undoubtedly affect the upstream system (front-end application) that uses logs, and the user experience is not good. to timeliness

At the same time, the long-term massive log processing and the IO peak for a long time may even lead to the collapse of the log collection and analysis system, and the crash of the log collection and analysis server, which seriously affects the use of the system.

Images