Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Log data compression processing method and device, electronic equipment and storage medium

A data compression and processing method technology, applied in the Internet field, can solve the problems of reducing search efficiency, destroying the text structure of log data, and the inability of full-text search engines to perform word segmentation, so as to reduce the amount of log data and improve search efficiency.

Pending Publication Date: 2021-12-07
BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF0 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] Commonly used data compression methods will make the compressed log text different from the original log text structure, thereby destroying the text structure of the log data, making it impossible for the full-text search engine to perform word segmentation and indexing, and thus unable to search the full text based on keywords. Search, reducing search efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log data compression processing method and device, electronic equipment and storage medium
  • Log data compression processing method and device, electronic equipment and storage medium
  • Log data compression processing method and device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0032] figure 1 It is a flow chart of a log data compression processing method provided by Embodiment 1 of the present invention. This embodiment is applicable to the case of performing compression processing on serialized log data with a character string structure. The method can be performed by a log data compression processing device, which can be realized by software and / or hardware, and integrated into the application end corresponding to the application program, wherein the application end can be a client with the application program installed, or It may refer to the server serving the application. Such as figure 1 As shown, the method specifically includes the following steps:

[0033] S110. Acquire each first attribute name in the first log data to be processed.

[0034] Wherein, the first log data may refer to serialized log data having a string structure, or may refer to log data to be serialized, that is, request response data and / or program internal data in the ...

Embodiment 2

[0053] image 3 It is a flow chart of the log data compression processing method provided by Embodiment 2 of the present invention. On the basis of the above-mentioned embodiments, this embodiment is concerned with "determining the first compression name corresponding to the first attribute name based on the preset mapping method" Further optimization was carried out. The explanations of terms that are the same as or corresponding to the above-mentioned embodiments will not be repeated here.

[0054] see image 3 The log data compression processing method provided in this embodiment specifically includes the following steps:

[0055] S310. Acquire each first attribute name in the first log data to be processed.

[0056] S320. Detect whether there is a compressed name corresponding to each first attribute name in the first log data in the mapping relationship between the currently stored attribute name and the compressed name, if yes, proceed to step S330, if not, proceed to...

Embodiment 3

[0086] Figure 4 It is a flow chart of the log data compression processing method provided by Embodiment 3 of the present invention. On the basis of the above embodiments, this embodiment describes the log search process in detail after obtaining the compressed second log data. The explanations of terms that are the same as or corresponding to the above-mentioned embodiments will not be repeated here.

[0087] see Figure 4 The log data compression processing method provided in this embodiment specifically includes the following steps:

[0088] S410. Acquire each first attribute name in the first log data to be processed.

[0089] specifically, Figure 5 An example of log data compression processing architecture is given. Such as Figure 5 As shown, the log serialization module in the application side can serialize the request response data object and / or the program internal data object, and obtain the serialized string structure request response data and / or program interna...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

Embodiments of the invention disclose a log data compression processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining each first attribute name in to-be-processed first log data; according to a mapping relation between currently stored attribute names and compression names and each first attribute name in the first log data, determining a first compression name corresponding to each first attribute name, wherein the character number of the first compression names is smaller than that of the first attribute names; and replacing each first attribute name in the first log data with the corresponding first compression name to obtain compressed second log data. Through the technical scheme of the embodiment of the invention, the log data volume can be greatly reduced under the condition that the text structure of the log data is not damaged.

Description

technical field [0001] The embodiments of the present invention relate to Internet technologies, and in particular to a log data compression processing method, device, electronic equipment and storage medium. Background technique [0002] With the rapid development of Internet technology, the system service architecture has developed from the original single system to distributed and the most popular micro-service system, which makes the request interaction process between services more complicated. [0003] Usually, the server can write the generated log files into a full-text search engine (for example, Elasticsearch search server), so that when a failure occurs, the log keywords can be used in the full-text search engine to search the logs, so that based on the search results, the search results can be quickly searched. Analyze the cause of the failure. As the number of application visits increases, the amount of generated log data also increases, so log data needs to be...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/174G06F16/18G06F16/14G06F16/16
CPCG06F16/1744G06F16/1815G06F16/148G06F16/166
Inventor 李汶良
Owner BEIJING WODONG TIANJUN INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com