Distributed device log collection method

Abstract

The invention discloses a distributed device log collection method. According to the method, an integrated data intermediate layer is built through a distributed log processing framework in a mediator mode, integrated data intermediate management service is formed, the data intermediate management service collects device logs, the logs are stored on each distributed storage point in a distributed way, in addition, data collection is carried out, and if the distributed storage points need to be increased, the goal is achieved by adopting a distributed storage point dynamic expansion mechanism. The distributed device log collection method has the advantages that the integrated data intermediate layer is built by adopting the mediator mode, the logs are subjected to collection and formatting processing in a unified way, and the distributed storage points are uniformly managed and dispatched in a centralized way; the sub table structure is adopted, the multithreading processing advantages are better realized, a particular sub table indexing mechanism is built, and forms a super-volume data grading indexing system together with MariaDb database indexes of each data storage point, meanwhile, the performance advantage of a distributed server is utilized, and the storage and query performance of log data is greatly improved.

Description

technical field [0001] The invention relates to a distributed equipment log collection method, which belongs to the technical field of computer system integration and application. Background technique [0002] With the increasing scale of the network environment, the number of various devices in the network has increased sharply, and various security and attacks from the outside and inside have also increased sharply, threatening the security of network information. In order to continuously respond to new security challenges, enterprises and organizations have successively deployed antivirus systems, firewalls, intrusion detection systems, vulnerability scanning systems, UTMs, and so on. Under such a complex security system, security audit becomes extremely important. The data basis of security audit is anti-virus system, firewall, intrusion detection system, vulnerability scanning system, UTM, running host, switch, router, database system, middleware and other log events, ...

AI Technical Summary

Problems solved by technology

[0003] Due to the huge amount of logs at present, how to effectively process and store them has become extremely important. In a medium-sized enterprise, the amount of logs per day can reach dozens of gigabytes or even hundreds of gigabytes, and it is generally required to keep these logs at least 3 months, then in a security audit system, the storage of these logs becomes the most basic and important link, and the traditional single-point storage solution can no longer meet the demand

[0004] In addition, due to the large amount of logs, efficient log query is also a serious issue, especially in the single-point storage solution, querying a single database while fast storage is very inefficient.

[0005] The traditional log storage and query technology generally adopts a single-point storage solution, which is limited to disk I / O performance. When the server performance and database performance reach the limit, it is difficult to meet the current huge log volume requirements.

The existing distributed storage technologies often use decentralized storage and single-point query, which cannot balance the performance of each data storage point, nor can comprehensive audit query of global data

Images