Inheritable and extensible data processing method and application thereof to log analysis

A technology of data processing and processing methods, applied in special data processing applications, data exchange networks, digital transmission systems, etc., can solve problems such as relying on analysts, improve system performance, facilitate management and new additions, and reduce workload Effect

Active Publication Date: 2020-05-22
HANGZHOU ANHENG INFORMATION TECH CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But the values ​​of these tagged fields are too dependent on the parser compared to the values ​​directly extracted from the log

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Inheritable and extensible data processing method and application thereof to log analysis
  • Inheritable and extensible data processing method and application thereof to log analysis
  • Inheritable and extensible data processing method and application thereof to log analysis

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0048] This embodiment discloses an inheritable and extensible data processing method, including the following steps:

[0049] (1) Directly write the common parsing method in the log into a parent parsing rule;

[0050] (2) Build a correlation between the parent parsing rule and several sub-parsing rules, so that sub-parsing rules with different parsing expressions can directly obtain the parsing method in the parent parsing rule through inheritance;

[0051] (3) The parent parsing rules can be rewritten for the parsing methods in the child parsing rules that do not satisfy the parent parsing rules, so as to realize the expansion of the parent parsing rules.

[0052] Wherein, the common log parsing method in the step (1) includes the field assignment in the log and the field value in the log; the method in which the sub-parsing rule rewrites the parent parsing rule in the step (3) includes an analytic expression, field assignment.

Embodiment 2

[0054] This embodiment is based on the above-mentioned data processing method, which is applied to the analysis process of the diary, and the specific process is as follows figure 1 shown, including the following steps:

[0055] (1) Build the common parsing method in the log into a parent parsing rule class;

[0056] (2) input relevant sub-parsing rule expressions;

[0057] (3) Generate a corresponding parsing rule object according to the sub parsing rule expression;

[0058] (4) matching the parent parsing rule class according to the parsing rule object to generate corresponding sub-parsing rules;

[0059] (5) Load the generated sub-parsing rules into the system, then use them to parse the logs, and output the parsing results.

[0060] Wherein, in the step (2), the input parsing rule expression includes JSON format, XML format, YSML format, WDDX format.

Embodiment 3

[0062] On the basis of the above-mentioned embodiments, this embodiment further defines the specific process of generating corresponding sub-parsing rules, such as figure 2 as shown,

[0063] (3.1) According to the parsing rule object, it is judged whether there is a dependent parent parsing rule class in the generated rule;

[0064] (3.2) If there is a parent parsing rule class dependency, then determine whether the dependent parent parsing rule class has been generated;

[0065] (3.3) If the dependent parent parsing rule class has been generated, read the parent parsing rule class directly as the generated sub-parsing rule; if the dependent parent parsing rule class has not been generated, then generate by reading the parent parsing rule expression Sub-parsing rule class, and as a new parent parsing rule, add it to the original parent parsing rule class and save it;

[0066] (3.4) If there is no dependency on the parent parsing rule class, generate a new sub-parsing rule ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an inheritable and extensible data processing method. The method comprises the following steps that: 1) a general analysis mode is directly written in a log into a parent analysis rule; 2) correlation between the parent analysis rule and a plurality of child analysis rules is established, and the child analysis rules with different analysis expressions directly obtain an analysis mode in the parent analysis rule in an inheritance mode; and 3) the analysis mode which does not meet the requirement of the parent analysis rule in the child analysis rules can be used for rewriting the parent analysis rule, so that the extension of the parent analysis rule is realized. In addition, the data processing method is applied to the log analysis process, analysis rules are optimized, redundant parts in the analysis rules are reduced, management and addition of the analysis rules are facilitated, repeated operation content of analysis rule compilation personnel is reduced, the number of the analysis rules is reduced, system performance is improved, and the problem that values of fields with the same meaning are difficult to express in different analysis rules is solved inan inheritance mode.

Description

technical field [0001] The invention relates to the technical field of computer data processing, and specifically refers to an inheritable and extensible data processing method and a log analysis method thereof. Background technique [0002] Connected devices generate a large number of log files every day, and these log files are generally unstructured. In the field of big data technology, it is often necessary to parse logs, and parse unstructured logs into structured data to facilitate data statistics and analysis. [0003] The current log parsing rules are mainly composed of parsing expressions (such as regular expressions) and assignment fields. In this way, the same processing logic may exist in different parsing rules. Although there are already some ways of modularizing the parsing rules to improve the reusability of the parsing rules, thereby reducing the redundancy between parsing rules, and reducing the redundancy phenomenon. But for some similar logs (for examp...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/18H04L12/24
CPCG06F16/1815H04L41/069Y02D10/00
Inventor 施展范渊
Owner HANGZHOU ANHENG INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap