Stream data processing method, device and system

Abstract

The embodiment of the invention provides a stream data processing method. The method is applied to a work node in a steam calculation system, and comprises the following steps of: obtaining the initial time-sharing lower limit value of a source operator; receiving the service data, and packaging the service data stream into triple stream data, wherein the triple stream data carries a time stamp and a data mark uniquely marking the service data; building a small-batch data set mini-batch according to the triple stream data and the initial time-sharing lower limit value of the source operator, wherein the time stamp is used for indicating the data order of the data in the building process, and the unique mark is used for indicating the data uniqueness in the building process; processing the small-batch data set mini-batch, and outputting processed data. The method provided by the embodiment of the invention has the advantages that the reliability of the stream calculation system is realized on the basis of the order and uniqueness of the triple stream data, and the calculation throughput of the stream calculation system can be improved on the basis of the initial time-sharing lower limit value, so the reliability of the stream calculation system is ensured, and meanwhile, the stream calculation throughput capability is also improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a method, device and system for stream data processing. Background technique [0002] In recent years, data-intensive services have been more and more widely used. Examples of these services include financial services, network monitoring, telecommunication data management, sensor detection and so on. The data generated by data-intensive services has the characteristics of large data volume, fast speed, and time-varying. The stream computing system starts to process the stream data when it receives it to ensure real-time performance. [0003] Such as figure 1 As shown, a stream computing system usually includes a master control node (Master) and multiple work nodes (worker). The master control node is mainly responsible for scheduling and managing each work node, and the work node is a logical entity that carries actual data processing operations. Specifically, the wor...

AI Technical Summary

Problems solved by technology

[0006] In order to solve the problem of poor throughput of stream computing in the prior art, a stream data processing method provided by the embodiment of the present invention encapsulates the received business data into triplet stream data, and according to the initial time-sharing lower limit and triple Tuple stream data constructs small batch data sets, thereby improving the computing throughput of the stream computing system while ensuring the reliability of the stream computing system

Images