Log analysis method and system based on dynamic field template

Abstract

The invention discloses a log analysis method and system based on a dynamic field, and the method comprises the steps: carrying out the analysis of a log through a dynamic field template which records a log field name and a field position sequence, and automatically discovering the field change between a to-be-analyzed log format and an existing template through comparing the analyzed field content with the length of the field content in the template; and dynamically updating the template by activating the human-computer interaction interface to supplement a new field, so that the analysis method can automatically adapt to the change of the log format in the aspects of field sequence, field number, field content format change and the like, a regular expression does not need to be written, and the engineering implementation threshold of an SOC product is reduced for a front-end engineer.

Description

technical field [0001] The invention relates to the field of computer information technology, in particular to a security log analysis method and system in a large-scale network environment. Background technique [0002] Security Operations Center (SOC) is a system and device that manages security events and security alarms of the monitored network by monitoring and analyzing security equipment, system, and software logs. The core function of the SOC is to collect, analyze and process logs. Log parsing refers to field segmentation of the original log, and storing the field content extracted from the log into the corresponding predefined field of the SOC system. [0003] Since the logs collected by the SOC come from different hardware and software systems, and these hardware and software systems are provided by different manufacturers, and are constantly updated and upgraded with the rapid development of IT technology, the original log data collected by the SOC Often there ...

AI Technical Summary

Problems solved by technology

And because it is often impossible to predict in advance which logs in different formats need to be parsed in the real environment of the user, although the SOC will store all the parsed logs in a configuration file for direct parsing, it will still be encountered at the implementation site. Lots of unrecognized log formats

At this time, the front-end implementation engineer can only write regular expressions to analyze the logs based on the interface provided by the SOC. However, the front-end engineers are limited by the technical capabilities of the end engineers and the on-site debugging and development environment, so that the on-site regular expressions are manually written. The work has become difficult to be implemented smoothly. After the log samples are sent back to the R&D center, the back-end development engineers will complete the log analysis. This makes the SOC device often idle for a long time before the log analysis work is completed. Play its due role, so that the user's network environment cannot receive effective security operation and maintenance management services

[0006] It can be seen that the crux of the above-mentioned status quo of traditional log parsing is that the log identification mechanism based on static templates is too blunt and rigid: after using regular expressions to find the field content according to the field name, match it according to the character format of the field content, which cannot adapt to a slightly log format Changes, that is, when the log field name, field order, field content format (such as date display format), word segmentation method, etc. are different, the log recognition mechanism will fail, and it will be considered as a brand new log, and a new log needs to be rewritten Identify and field parse regular expressions to parse

The parsing mechanism of the field content of the log also completely relies on regular expressions to parse character by character. Although this can achieve parsing accuracy, it lacks adaptability to changes.

Images