Key value data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a key value data processing method and device, electronic equipment and a storage medium. The method comprises the steps that a to-be-processed task and meta information corresponding to the to-be-processed task are acquired; the to-be-processed task is processed through the mapping task in the calculation engine to obtain key value data, the key value data are transmitted to shuffling processing nodes, and the shuffling processing nodes are nodes independently packaged outside the calculation engine; and running sorting logic corresponding to the meta-information through the shuffling processing node so as to sort the key value data. According to the scheme provided by the invention, the key value data generated by various calculation engines are processed by adopting a general sorting mechanism, so that the sorting requirements of the key value data of the various calculation engines can be met at the same time.

Description

technical field [0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic equipment, computer-readable storage medium, and computer program product for processing key-value data. Background technique [0002] The current computing engine Mapreduce (a map-reduce model), Spark (a computing engine), etc. have a shuffle (shuffling) mechanism. The shuffling mechanism is responsible for the data transmission between MapTask (mapping task) and ReduceTask (reduction task) in the task. [0003] Currently, the shuffling mechanism is mainly divided into sort-based shuffle (sort-based shuffling) and hash-based shuffle (hash-based shuffling). Mapreduce mainly adopts a sorting-based shuffling method, which sorts key-value (key-value) data on the mapping task side according to partition (partition) and key (key), and outputs them to the local disk to form an orderly shuffling Wash files. In this way, the shuffled files o...

AI Technical Summary

Problems solved by technology

[0004] However, most of the current stream shuffling mechanisms within enterprises only support the Spark engine.

When there are both Spark and MapReduce jobs in the enterprise, the hash-based shuffling method of the Spark engine cannot meet the key-value data sorting requirements of the MapReduce job. Therefore, a situation that can work on multiple computing engines at the same time is urgently needed Next, the way to flexibly sort the key-value data of various computing engines

Images