Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data analysis method based on nginx log

A data analysis and log technology, applied in the field of log files, can solve the problems that cannot meet the needs of complex businesses, cannot be generalized, and update services are uninterrupted and cannot be analyzed.

Inactive Publication Date: 2017-05-31
武汉卓尔云市集团有限公司
View PDF3 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] Existing products generally build a simple log center platform based on the ELK framework, but cannot meet the needs of complex businesses
In terms of log collection, it is often necessary to do customized collection for adding a new file collection, which cannot be generalized and the update service is uninterrupted; in terms of data report presentation, it cannot be analyzed based on complex dimensions

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data analysis method based on nginx log
  • Data analysis method based on nginx log

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0034] Such as figure 1 As shown, a data analysis method based on nginx logs is applicable to nginx logs. The data analysis methods include:

[0035] S1: Collect the log files in the nginx server through the flume module, and transfer and save the log files to the HDFS system;

[0036] S2: Use the Hive database tool to perform mapreduce operations on the data in the HDFS system, and save the processed data in the intermediate table;

[0037] S3: Migrate the data in the intermediate table to the result table of the MySQL database; when an error occurs during the migration process that exceeds the predetermined amount of data, perform a data rollback operation, otherwise continue the migration operation.

[0038] Fl...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a data analysis method based on nginx log. The data analysis method is suitable for the nginx log and includes: S1, collecting a log file in an nginx server through a flume module, transferring the log file, and storing the same into an HDFS (Hadloop distributed file system); S2, performing map reduce operation on data in the HDFS through an Hive database tool, and storing processed data into an intermediate table; S3, migrating the data in the intermediate table into a result table of a MySQL database. The data analysis method has the advantage that by the technical scheme, universalization, continual service updating and complex dimension analysis of data can be realized.

Description

technical field [0001] The invention relates to the technical field of log files, in particular to a data analysis method based on nginx logs. Background technique [0002] Existing products generally build a simple log center platform based on the ELK framework, but cannot meet the needs of complex businesses. In terms of log collection, custom collection is often required for adding a new file collection, which cannot be generalized and uninterrupted in updating services; in terms of data report presentation, it cannot be analyzed based on complex dimensions. Contents of the invention [0003] The technical problem to be solved by the present invention is to provide a data analysis method based on nginx logs. [0004] The technical scheme that the present invention solves the problems of the technologies described above is as follows: [0005] A kind of data analysis method based on nginx log, is applicable to nginx log, and described data analysis method comprises: ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/182G06F16/1815
Inventor 杨波
Owner 武汉卓尔云市集团有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com