Data monitoring method and device based on streaming computing

A data monitoring method and streaming computing technology, which is applied in the computer field to achieve the effect of improving data processing efficiency, improving stability and reliability, and alleviating data pressure

Active Publication Date: 2020-07-17
BEIJING 58 INFORMATION TTECH CO LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The embodiment of the present invention provides a data monitoring method and device based on streaming computing, which can effectively overcome the problems existing in the prior art that would cause huge pressure on the MySQL database, and at the same time slow down the calculation speed and data display speed.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data monitoring method and device based on streaming computing
  • Data monitoring method and device based on streaming computing
  • Data monitoring method and device based on streaming computing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0042] The specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0043] figure 1 A schematic flow chart of a flow computing-based data monitoring method provided by an embodiment of the present invention; refer to the attached figure 1 It can be seen that this embodiment provides a stream computing-based data monitoring method, including:

[0044] S101: Obtain the log information of the service being called through the distributed publish-subscribe message system KAFKA;

[0045] KAFKA is a high-throughput distributed publish-subscribe messaging system that can handle all action streaming data in consumer-scale websites; such actions (web browsing, searches, and other user actions) are on the modern web A key factor in many social fun...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a streaming computing-based data monitoring method and apparatus. The data monitoring method comprises the steps of obtaining log information of service calling through a distributed message publishing / subscription system KAFKA; performing analysis processing on the log information by adopting a distributed streaming processing method Spark streaming to obtain log detail information and log summarization information; and writing the log detail information into a distributed file system HDFS, and storing the log summarization information in a relational database management system MYSQL. According to the streaming computing-based data monitoring method and apparatus provided by the invention, the problems of a huge pressure on a MySQL database and reduced computing speed and data display speed in the prior art are effectively solved, the data processing efficiency is improved, and the data pressure on the MySQL database is reduced, so that the stability and reliability of application of the data monitoring method are improved and the market popularization and application are facilitated.

Description

technical field [0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a data monitoring method and device based on streaming computing. Background technique [0002] With the rapid development of Internet technology, the network business volume continues to increase, and the number of business parties continues to increase, making the number of service calls significantly increased, which in turn puts forward higher requirements for big data processing on the network; in the prior art, through The storm streaming computing framework receives service request log information, and after simple processing by storm, writes the detailed data into MySQL. [0003] However, in the process of implementing this technical solution, it is found that the existing technology has the following defects: because the way Storm receives data only supports message queues in blocking mode, messages can only be received in real time. Storm's...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/17G06F16/182
CPCG06F16/1815G06F16/182
Inventor 李鹏于洋郭振强
Owner BEIJING 58 INFORMATION TTECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap