Method for sorting out-of-order data streams

Abstract

The invention provides a method for sorting high-speed out-of-order data streams with high performance and high availability under the condition that a memory is limited. The method comprises the steps of sequentially inserting reached data into proper positions in a first buffer memory with fixed sizes, and sequentially inserting the reached data into proper positions in a second buffer memory if the proper positions cannot be found in the first buffer memory; sequentially reading data from the first buffer memory and transmitting the data to a downstream, and waiting if no data is read or data is invalid in some position; sequentially reloading the data in the second buffer memory to the first buffer memory. According to the method for sorting out-of-order data streams, and the size of the first buffer memory is fixed, so that the first buffer memory can work well in a system with limited primary cache capacity; meanwhile, the organizational form of the second cache is flexible, so that the cache capacity and expansibility can be improved, the wait after the data lacks number can be reduced, data obstruction can be decreased, and data loss can be avoided.

Description

technical field [0001] The invention relates to a method for sorting data streams, in particular to a method for sorting out-of-order data streams with high performance and high availability in a memory-constrained system. Background technique [0002] In the data processing system, the data source will generate a large amount of data, and the same type of data constitutes a series of data streams. The so-called out-of-order data stream means that the order in which the data in the data stream arrives is not the same as the order in which the data is generated. In the downstream of the data processing system, the input data is usually required to arrive in the order in which it was generated (serial number), so an intermediate sorting link is required to sort the data. Due to the innate characteristics of data disorder, data with a small sequence number (indicating that it was generated earlier) may arrive later. In the case where the order of data sent by the upstream syst...

AI Technical Summary

Problems solved by technology

[0005] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a method capable of sorting high-speed out-of-sequence data streams with high performance and high availability under the condition of limited memory, which is used to solve the problem of traditional sorting methods. The input size of a specified length is required, and it can only wait for the case of lack of numbers and broken numbers. In the high-performance and high-availability field, data blocking or even data discarding may be caused by waiting after data lack of numbers. The method of comprehensively caching data is in Problems such as inability to implement on memory-constrained systems

Images