Method and system for implementing partition load balancing in Spark environment

Abstract

The invention discloses a method for implementing partition load balancing in a Spark environment. The method comprises the steps of receiving a Spark application sent by a user, and analyzing the Spark application program to obtain an analysis result, wherein the RDD graph represents the relationship among the plurality of elastic distributed data sets RDDs, and the directed acyclic graph DAG isused in the scheduling stage; sequentially determining a dependency relationship between every two adjacent scheduling stages according to the DAG graph; numbering the wide dependency relationships inall the obtained dependency relationships; setting counter cnt = 1, and judging whether the cnt is greater than the total number of the wide dependency relationships or not; and if not, sampling datain all partitions in the last RDD in the Map end corresponding to the cnt-th wide dependency relationship to obtain hash tables which represent data key distribution and correspond to each partition,and merging the hash tables corresponding to all the obtained partitions. According to the method, the problem of data inclination in big data calculation can be solved, meanwhile, allocation of calculation resources is optimized, and the program running time is shortened.

Description

technical field [0001] The invention belongs to the field of big data and distributed parallel computing, and relates to a method and system for realizing partition load balancing in a Spark environment. Background technique [0002] With the rapid development of the Internet, people's daily behavior will generate a large amount of data, the total amount of data and the growth rate of data are increasing day by day. For the increasingly huge data, stand-alone computing can no longer meet the demand, and the MapReduce programming model came into being. MapReduce is a software framework for parallel processing of massive data in a reliable and fault-tolerant manner. Apache Spark is a fast and general-purpose processing engine for large-scale data based on the MapReduce model. Apache Spark can achieve high performance in both batch processing and stream processing. It is a physical execution engine that includes a stateful directed acyclic graph scheduler and a query optimizer....

AI Technical Summary

Problems solved by technology

[0005] Aiming at the above defects or improvement needs of the prior art, the present invention provides a method and system for realizing partition load balancing in the Spark environment, the purpose of which is to solve the existing hash-based partitioning method when data skew occurs , a technical problem with excessive processing time for the entire Spark application

Images