Data out-of-order arrival processing method and system

Abstract

The invention discloses a data out-of-order arrival processing method and system. The method comprises the following steps of extracting a current time window field, and processing the time window field into date type data; judging whether user stream data has a specified time window field or not, and performing corresponding processing; marking a time window field where a time slicing field is located, and extracting a data set before the time window field from a Redis storage library; judging whether time slicing data of the marked time window field is in the extracted data set or not, and performing corresponding processing; and storing the user stream data in the Redis storage library, and updating the Redis storage library. The system comprises a data processing module, a first judgment module, a marking module, a second judgment module and a Redis storage library module. According to the method and the system, the problem of out-of-order arrival of real-time stream data is solved; the method and the system are especially suitable for solving the problem of a non-serialization scene of a data source; and the validity and time sequence of the data are improved.

Description

technical field [0001] The invention relates to the technical field of big data analysis and processing, in particular to a data out-of-order arrival processing method and system. Background technique [0002] In the context of the current big data industry, real-time streaming technology is a data processing technology that pushes batched, orderly, and neat serialized data to the analyzer in a fixed manner. Since the analyzer has strict requirements on the data format, this directly leads to the fact that in most cases, the data format is single and the serialization requirements are strict. However, in real-time streaming data sources, the data often does not come from highly serialized scenarios. Due to the out-of-order arrival of data, the data cleaning results are often inconsistent with the original data results, the data timing is poor, and the data quality is low. Contents of the invention [0003] The purpose of the present invention is to overcome the deficienci...

AI Technical Summary

Problems solved by technology

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art, and provide a method and system for processing data out of order. The present invention uses the principle of time slice and batch distribution of real-time streaming data to make the Spark processing node store Redis process Logical allocation is carried out to solve the problem of out-of-order arrival of real-time streaming data and improve the validity and timing of data

Images