Log processing system, log processing method, terminal and storage medium

Abstract

The embodiment of the invention provides a log processing system, which comprises a log acquisition module for acquiring log data, a Kafka log distribution cluster which is used for distributing the log data to obtain first distributed log data and second distributed log data, an Elasticsearch cluster which is used for carrying out first processing on the first shunt log data to obtain a first result, a HBase cluster which is used for carrying out second processing on the second shunt log data to obtain a second result, and a result display module which is used for displaying the first resultand / or the second result. The embodiment of the invention further provides a log processing method, a terminal and a computer readable storage medium. By utilizing the embodiment of the invention, theElasticsearch cluster stores short-term log data and performs log real-time processing, and the HBase cluster is mainly responsible for offline log data processing, so that the log processing efficiency is improved.

Description

technical field [0001] The present invention relates to the technical field of log generation process optimization, and in particular to a log processing system, a log processing method, a terminal, and a computer-readable storage medium. Background technique [0002] With the development of computers and networks, the data processing volume of log data is increasing, and the data volume of log data is usually more than one million, or even more than one trillion or one trillion. For such a huge log data system, first of all, higher requirements are put forward for the processing of log data. In the prior art, the log system generally adopts two schemes, one is to use the architecture of ELK, and ELK is a kind of Elasticsearch (real-time full-text search and analysis engine), Logstash (a tool for collecting, analyzing and filtering logs) and Kibana (a web-based graphical interface for searching, analyzing, and visualizing log data stored in Elasticsearch indicators) is the ...

AI Technical Summary

Problems solved by technology

In the prior art, the log system generally adopts two schemes, one is to use the architecture of ELK, and ELK is a kind of Elasticsearch (real-time full-text search and analysis engine), Logstash (a tool for collecting, analyzing and filtering logs) and Kibana (a web-based graphical interface for searching, analyzing, and visualizing log data stored in Elasticsearch indicators) is the basic architecture of the core suite. This method has good real-time performance and convenient query, but because Elasticsearch query is Http protocol, so it is not suitable for providing logs in large quantities; there is also a Hadoop-based architecture, which can aggregate logs and provide log files externally, but the real-time performance is poor, and the query is not convenient enough

Images