Method for processing compression program parallelization based on computer clusters

Abstract

The invention relates to a method for processing compression program parallelization based on computer clusters. The method is characterized by comprising a provider-processor-producer frame, wherein the provider-processor-producer frame comprises a provider process, a processor process and a producer process; the provider process divides data to be processed into a plurality of fixed-length data blocks to be sent and then sends the data blocks to be sent to the processor process; idle processer process is gradually stopped after the data blocks to be sent are completely sent out; the processor process requests data blocks to be sent from the provider process and sends processed data blocks to a user process; after receiving the processed data blocks sent by the processor process, the user process outputs the processed data blocks sequentially; the processor comprises a plurality of serial compression algorithms and executes the serial compression algorithms in parallelization manner; and when the data blocks to be sent and the processed data blocks are transferred among the provider process, the processor process and the user process, unique data block serial numbers are respectively attached to the data blocks to be sent and the processed data blocks.

Description

technical field [0001] The invention relates to a computer parallel processing method, in particular to a computer cluster-based compression program parallel processing method. Background technique [0002] Today's computer technology has entered the era of parallelization and high data volume. The use of fast compression algorithms is very important for processing massive data. However, it is very time-consuming to use traditional compression software to process massive data. At the same time, a new compression algorithm is developed. It is quite difficult, so it is of great practical significance to parallelize the development of traditional compression software. At present, some people have rewritten the original serial compression algorithm in parallel, but because the rewritten core compression algorithm is very complicated, the process of completely parallelizing the original serial compression algorithm is very slow, and none of them is more general method to solve t...

AI Technical Summary

Problems solved by technology

However, the disadvantage of this method is that its algorithms for specific problems are not universal, and there is no general framework structure, which leads to the lack of universality in the redevelopment and expansion of these parallel algorithms.

Images