Cache value-based Spark cache elimination method and system

Abstract

The invention relates to the field of big data calculation, in particular to a cache value-based Spark cache elimination method and system. The method comprises the steps: obtaining an initial cache value of each RDD according to a cache value model based on RDD information; sorting the RDDs by using an improved quick sorting algorithm to obtain an RDD sequence; sequentially storing the calculation results of the RDDs in the RDD sequence into a cluster node memory according to the sequence of the initial cache values from high to low; updating the RDD information when each Stage is finished inthe dynamic task execution process; when the node memory is insufficient, calculating the cache value of the cached Block by utilizing a cache value model based on the Block information; and eliminating the Block with low cache value to release the memory space. According to the method, the RDD with the highest cache value is stored in the memory, unused Blocks are cleared in time, the calculation speed is increased, the RDD recalculation overhead is reduced, and the memory resource utilization rate is optimized.

Description

technical field [0001] The invention relates to the field of big data computing, in particular to a Spark caching method and system based on RDD (Resilent Distributed Datasets, elastic distributed data sets) and Block caching value. Background technique [0002] In today's big data era, the amount of data is increasing exponentially, and big data processing is increasingly valued by people. In order to quickly process these massive data information, more and more applications and scientific research projects will be based on huge data sets. For processing and analysis, some big data computing frameworks have emerged, such as: MapReduce parallel computing model for large-scale data processing, open source big data computing framework Hadoop, Spark framework, etc. The limitations of the Hadoop framework in multiple application domains and big data processing scenarios, such as large-scale structured data, graph data, and streaming data, currently Apache Spark has become a unif...

AI Technical Summary

Problems solved by technology

However, this Spark cache elimination method uses the time to generate RDDs to represent the calculation cost of RDDs is not accurate enough, there may be network delays in data transmission between multiple nodes in the cluster or the problem of weakened computing power caused by nodes running at full capacity. In addition, this This elimination mechanism ignores the future reuse times of the block, but instead increases the consumption of the system to re-extract and calculate the block

Images