Data processing method and device

Abstract

Provided are a data processing method and device, which relate to the field of computers, and can decrease the length time of the operation flow of merging data and reduce the probability and time of the occurrence of the stop-and-wait phenomena. The method comprises: dividing an operation flow of merging data into at least two phases to obtain 1st to mth phases, where m is greater than or equal to 2, and each phase comprises one or more sub-flows; respectively establishing a corresponding thread for each phase to obtain m threads; and executing a sub-flow in a phase corresponding to a kth thread on data input to the kth thread and outputting processed data to a k+1th thread, and in the process of the k+1th thread executing the sub-flow of the corresponding phase on the processed data, executing the sub-flow in the phase corresponding to the kth thread again on the data input to the kth thread again and outputting the processed data to the k+1th thread, until the processing of all of the data input to the kth thread is completed. The data processing method and device are applied to data processing.

Description

technical field [0001] The invention relates to the field of computers, in particular to a data processing method and device. Background technique [0002] With the development of the Internet, more and more network service storage systems use write-optimized data structures to reduce write latency. For example, LSM-Tree (Log-Structured Merge Tree) is a write-optimized data structure used to reduce write latency. LSM-Tree reduces write latency by aggregating small granularity, random updates to large granularity, and sequential updates. However, LSM-Tree requires background merge operations to maintain reasonable data structures and operational performance. Since the operation process of each data merger needs to read data from disk devices, decompress the data, verify and merge, compress and verify the merged data, and then write it back to the disk, the whole process takes a long time . While waiting for the completion of the operation process of data merging, the stor...

AI Technical Summary

Problems solved by technology

However, a data merging operation process is executed by one thread, and the data merging operation process takes a long time and is prone to stop and so on.

Images