Quasi-real-time log data analysis and statistics method based on Elasticsearch

Abstract

The invention discloses a quasi-real-time log data analysis and statistics method based on Elasticsearch. The quasi-real-time log data analysis and statistics method based on Elasticsearch comprises the following steps: S1, storing Elasticsearch clusters in a distributed mode; s2, using an Elasticsearch cluster to carry out data writing; and S3, reading the data by adopting an Elasticsearch cluster. The invention has the beneficial effects that the invention can realize fragmented storage and efficient query, can support approximately real-time search, and the distributed deployment does not need other components and transverse extension to supports a plug-in development mode; the invention has rich open source plug-in libraries, cannot reduce the query efficiency along with the increase of the data volume, can be suitable for real-time data search, and realizes data statistics and query in scenes of real-time change updating and the like.

Description

technical field [0001] The present invention relates to the field of quasi-real-time log data analysis and statistical methods of Elasticsearch, in particular, to a quasi-real-time log data analysis and statistical method based on Elasticsearch. Background technique [0002] With the rapid development of emerging information and communication technologies such as mobile Internet, cloud computing, mobile communications, and the Internet of Things, data traffic has exploded and data structures have become increasingly complex. The information society has entered the era of networked big data. Social networks, Commercial applications such as e-commerce, mobile communications, and Internet of Things technology generate diverse data such as transaction data, interaction data, sensor data, and log data. These data are characterized by massive volume, various types, rapid growth, and huge value. At the same time, It is also highly complex and full of changes. To analyze and proces...

AI Technical Summary

Problems solved by technology

[0003] The present invention is a real-time log data analysis and statistics method based on Elasticsearch (elastic search, referred to as ES). Similar to this technology is the search engine technology based on solr. At present, when solr builds an index in real time, IO blockage will occur, and real-time query The efficiency is low, and as the amount of data increases, the query efficiency of solr will decrease, and it cannot be applied to data statistical analysis and query in scenarios such as real-time data search, real-time change update, etc.

Images