Data transmission customization system and method

A technology for data transmission and customized systems, which is applied in transmission systems, electrical components, etc., can solve the problems of inconvenient development workload of the Flume log collection system, achieve a wide range of log collection scenarios, and improve efficiency

Active Publication Date: 2015-11-04
好活(重庆)网络科技有限公司
View PDF3 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this way, if the remote server changes, manually modifying the configuration in the FTP and Flume log collection system and redeveloping the client each time will bring a lot of inconvenience and development workload to the application of the Flume log collection system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data transmission customization system and method
  • Data transmission customization system and method
  • Data transmission customization system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0038] This embodiment discloses a data transmission customization system for the Flume log collection system, which combines the technical architecture of the Flume log collection system, and performs some general design and development for log transmission. Among them, such as figure 1 As shown, the core of the Flume log collection system is an agent (Agent) 100, including a data source (Source) 110, a data transmission 120 (Channel) and a data destination (Sink) 130. The Flume log collection system is used to collect log files of the remote server 200 .

[0039] Such as figure 2 As shown, the data transmission customization system 400 of this embodiment includes: an FTP client 410 , an FTP server 420 and an IP address input module 430 .

[0040] Wherein, the FTP server 420 is located at the remote server 200 and is used for saving the log files of the remote server 200 .

[0041] The IP address input module 430 is used to input the IP address of the remote server 200, t...

Embodiment 2

[0052] This embodiment discloses a method for customizing data transmission for the Flume log collection system, such as image 3 mentioned, including:

[0053] Step S10, save the log file of the remote server on the FTP server:

[0054] Generally, the remote server directly saves the log files on the FTP server.

[0055] Further, considering from the perspective of data transmission, the remote server compresses the log files and then saves them on the FTP server.

[0056] Moreover, for data security, the remote server also encrypts the log files, and then saves the encrypted log files on the FTP server.

[0057] Step S20, input the IP addresses of the remote server, the FTP client and the directory pool of the data source:

[0058] Since the data transmission customization method of this embodiment adopts FTP technology, the premise of its data transmission is to know the IP addresses of the client and the server; that is, it is necessary to obtain the IP address of the r...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data transmission customization system and a method for a Flume event collection system. The data transmission customization system comprises an FTP client, an FTP server and an IP address input module, wherein the FTP server is located at a remote server for storing event files of the remote server; the IP address input module is used for inputting IP addresses of the remote server, the FTP client and a spooling directory of a data source; the FTP client is located between the remote server and the data source for downloading the event file from the FTP server according to the IP address of the remote server; the event file is stored; and according to the IP address of the spooling directory of the data source, the event file is uploaded to the data source. The technical structure of the Flume event collection system is combined, event transmission generality design and development are completed, and the event collection and transmission efficiency is improved.

Description

technical field [0001] The invention relates to the field of log data transmission, in particular to a data transmission customization system and method based on a Flume log collection system. Background technique [0002] With the rapid development of big data log collection technology, there are more and more log collection and analysis systems based on big data cloud platform, but there has been a lack of universal design in the transmission of log data. Currently, the most used log collection system is the Flume log collection system. [0003] Flume is a log collection system provided by Cloudera. It has the characteristics of distributed, high reliability, and high availability. For massive log collection, aggregation, and transmission, Flume supports customizing various data senders in the log system. At the same time, Flume provides data processing Simple processing, and the ability to write to various data recipients. Traditional Flume architecture such as figure ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/06
Inventor 欧阳涛
Owner 好活(重庆)网络科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap