Data transmission method and device for Shuffle process

Abstract

The embodiment of the invention discloses a data transmission method and a device for a Shuffle process. According to one specific embodiment, the method comprises the steps that a data set before the Shuffle process and the number of partitions included in the data set after the Shuffle process are obtained, and the data set before the Shuffle process comprises at least one partition; obtaining the number of target service components; according to the number of the target service components, determining a corresponding relationship between each target service component and a partition of the data set after the Shuffle process; and respectively sending data of each partition in the data set before the Shuffle process to a target service component corresponding to the data set after the Shuffle process. According to the embodiment, decoupling of the Spark calculation process and the storage process is realized, re-execution of the Shuffle process caused by local faults of the node is effectively avoided, and the stability and performance of the Spark are improved.

Description

technical field [0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a data transmission method and device for a Shuffle process. Background technique [0002] With the rapid development of computer technology, general-purpose computing engines suitable for large-scale data processing have emerged as the times require. In the prior art, the Spark computing engine is usually used to complete the mutual conversion of RDDs (Resilient Distributed Datasets, elastic distributed data sets) according to dependencies. When the dependency relationship between the RDDs before and after the conversion is a wide dependency, it is necessary to complete the RDD conversion through Shuffle (shuffling). [0003] Since the partitions of Spark RDD are usually distributed on different computing nodes, when a node fails (including network failure, hard disk failure, etc.), it will trigger a very serious error (FetchFailed) in Spark, wh...

AI Technical Summary

Problems solved by technology

[0003] Since the partitions of Spark RDD are usually distributed on different computing nodes, when a node fails (including network failure, hard disk failure, etc.), it will trigger a very serious error (FetchFailed) in Spark, which will cause the Shuffle process to be re-executed. Consumes more computing resources and time

Moreover, since the number of random reads in the Shuffle Read process is related to the number of partitions of the two RDDs before and after the Shuffle process, when the number of partitions in the target RDD (the RDD formed after Shuffle) is too large, often due to the number of reads Too many will cause the reading time to be too long; if the number of partitions of the target RDD is too small, it is easy to cause OOM (OutOfMemoryError, memory overflow) due to the large amount of data processed by each partition of the target RDD

Images