Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Distributed collection and storage method of journal

A distributed and log technology, applied in the computer field, can solve problems such as frequent transaction arrivals, write operation blocking, and affecting system real-time performance and availability, and achieve the effects of improving real-time performance and availability, improving efficiency, and improving throughput

Inactive Publication Date: 2013-04-10
BLUEDON INFORMATION SECURITY TECH CO LTD
View PDF3 Cites 28 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But when syslog-ng writes a large amount of data to the named pipe, if it happens to be reading the data of the pipe, the write operation will block
Moreover, when syslog-ng is used as a log server to centrally write logs to the database, transactions arrive frequently, and a large number of database IO operations will become the bottleneck of system performance, affecting the real-time performance and availability of the system.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed collection and storage method of journal
  • Distributed collection and storage method of journal
  • Distributed collection and storage method of journal

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0011] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0012] The invention provides a log distributed collection and storage method, and the problem to be solved is that the client log can be quickly sent to the remote log collection server, and the log collection server can effectively write the log to the database. The entire invention flow chart is as follows figure 1 shown.

[0013] The logs collected by each client node are cached in an array container, and Protobuf serializes them and sends them to the remote l...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed collection and storage method of a journal. The method adopts transmission control protocol (TCP) / internet protocol (IP) to interchange data, a sending terminal serializes the data, and a receiving terminal deserializes the data. In order to maintain long connection, a client terminal sends heartbeat packet at regular intervals to determine whether a journal server terminal is on line, and when the journal server terminal corresponds the heartbeat packet when the journal server terminal is on line. A first speed regulator controls a connected FD (file descriptor) integrally and storages the processed data in a data-caching pool. A second speed regulator reads a certain amount of data from the data-caching pool and sends the data to a remote terminal. After the remote terminal of a computer receives the data, the remote terminal writes the data in a data base. By means of the distributed collection and storage method of the journal, the transmission efficiency of the client terminal and the journal server are improved, the handling capacity operation of the journal server and the data base is improved, and the instantaneity and the usability of the system are improved.

Description

technical field [0001] The invention relates to the technical field of computers, in particular to a method for distributed collection and storage of logs. Background technique [0002] In a distributed network and a high-performance system, there are a lot of data collected and processed from multiple nodes within a certain period of time, and the data processing capacity of a single database system is limited. How to improve the capacity of high-flow data collection, processing and storage is a problem. Problems to be solved. [0003] Currently, the syslog-ng log collection server is generally used to collect and process logs. When multiple clients send logs to the central log server, the server will uniformly write them into the database. But when syslog-ng writes a large amount of data to the named pipe, if it happens to be reading the data of the pipe, the write operation will block. Moreover, when syslog-ng is used as a log server to centrally write logs to the data...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L29/08
Inventor 柯宗贵柯宗庆杨育斌张道磊
Owner BLUEDON INFORMATION SECURITY TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com