Network safety log template extraction method and device

Abstract

The invention relates to a network safety log template extraction method and device. The method comprises steps: firstly, original safety logs are subjected to data cleaning, and log information after filtering of time and IP addresses is obtained; secondly, the log information which does not containing time and IP addresses are subjected to clustering, and logs with similar formats are classified to the same class; thirdly, for logs in the each class, template words describing formats in the logs are extracted, and the template of the log format is obtained. Concretely, a DBSCAN algorithm or an OPTICS algorithm is employed to cluster logs with similar formats, and template words describing formats in logs are extracted by utilization of an LDA Gibbs sampling algorithm. The device comprises a data cleaning unit, an information clustering unit and a template extraction unit. Priori knowledge is not needed, a template of a network safety log format can be obtained automatically, the system load can be reduced, and operational efficiency and accuracy are raised.

Description

technical field [0001] The invention relates to the field of information security, in particular to a method and device for extracting a network security log template. The solution does not depend on any prior knowledge and can automatically extract a template describing an unknown network security log format. Background technique [0002] Network security logs, including system logs generated by the operating system and alarm logs generated by network security devices, record various security events in the network environment and provide important clues for network abnormal diagnosis and network attack threat discovery. In the network security log analysis system, log format parsing is an essential step. Therefore, extracting network security log templates is of great significance for log analysis. At present, in some relatively mature network security log analysis products, such as OSSIM, Snort, OSSEC, etc., log parsing methods based on regular expressions are usually col...

AI Technical Summary

Problems solved by technology

Although many log formats can be obtained through relevant technical documents, the log format may change as the system is updated or upgraded

Moreover, the log formats of many security product manufacturers are not disclosed to the public. For the massive logs of unknown formats in the network, it is difficult for analysts to manually extract all the log templates.

Images