Stream data processing method, device and system

A processing method and streaming data technology, applied in the computer field, which can solve problems such as poor streaming computing throughput.

Active Publication Date: 2015-02-04
HUAWEI TECH CO LTD
View PDF3 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] In order to solve the problem of poor throughput of stream computing in the prior art, a stream data processing method provided by the embodiment of the present invention encapsulates the received business data into triplet stream data, and according to the initial time-sharing lower limit and triple Tuple stream data constructs small batch data sets, thereby improving the computing throughput of the stream computing system while ensuring the reliability of the stream computing system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Stream data processing method, device and system
  • Stream data processing method, device and system
  • Stream data processing method, device and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0078] The embodiment of the present invention provides a stream data processing method, which encapsulates the received business data to form triple stream data, and constructs a small batch data set according to the triple stream data and the initial time-sharing lower limit value, so that the While ensuring the reliability of the stream computing system, improve the computing throughput of the stream computing system. Embodiments of the present invention also provide corresponding devices and systems. Each will be described in detail below.

[0079] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. B...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a stream data processing method. The method is applied to a work node in a steam calculation system, and comprises the following steps of: obtaining the initial time-sharing lower limit value of a source operator; receiving the service data, and packaging the service data stream into triple stream data, wherein the triple stream data carries a time stamp and a data mark uniquely marking the service data; building a small-batch data set mini-batch according to the triple stream data and the initial time-sharing lower limit value of the source operator, wherein the time stamp is used for indicating the data order of the data in the building process, and the unique mark is used for indicating the data uniqueness in the building process; processing the small-batch data set mini-batch, and outputting processed data. The method provided by the embodiment of the invention has the advantages that the reliability of the stream calculation system is realized on the basis of the order and uniqueness of the triple stream data, and the calculation throughput of the stream calculation system can be improved on the basis of the initial time-sharing lower limit value, so the reliability of the stream calculation system is ensured, and meanwhile, the stream calculation throughput capability is also improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a method, device and system for stream data processing. Background technique [0002] In recent years, data-intensive services have been more and more widely used. Examples of these services include financial services, network monitoring, telecommunication data management, sensor detection and so on. The data generated by data-intensive services has the characteristics of large data volume, fast speed, and time-varying. The stream computing system starts to process the stream data when it receives it to ensure real-time performance. [0003] Such as figure 1 As shown, a stream computing system usually includes a master control node (Master) and multiple work nodes (worker). The master control node is mainly responsible for scheduling and managing each work node, and the work node is a logical entity that carries actual data processing operations. Specifically, the wor...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/38G06F9/46
Inventor 张松山
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap