Hybrid computing system and data processing method and device

Abstract

The embodiment of the invention provides a hybrid computing system and a data processing method and device. The hybrid computing system comprises a classification layer and a computing layer, and thecomputing layer comprises a first computing engine based on batch processing and a second computing engine based on stream processing. The classification layer is used for obtaining features of the calculation task according to the first code corresponding to the calculation task and determining a target calculation engine according to the features. The classification layer is further used for converting the first code into a second code corresponding to a target computing engine and sending the second code to the target computing engine, so that the target computing engine executes a computing task to process the to-be-processed data. The hybrid computing system comprises the first computing engine based on batch processing and the second computing engine based on stream processing, so that the hybrid computing system is suitable for executing both batch processing tasks and stream processing tasks, and the applicability of the hybrid computing system is improved.

Description

technical field [0001] Embodiments of the present invention relate to the field of big data technology, and in particular to a hybrid computing system, data processing method and device. Background technique [0002] Currently, big data processing frameworks include batch-based computing frameworks and stream-based computing frameworks. Among them, among the computing frameworks based on batch processing, MapReduce in the Apache Hadoop ecosystem is the most typical; among the computing frameworks based on stream processing, the most typical ones are Apache Storm and Samza. [0003] Batch processing is suitable for operating large-capacity historical data sets and returning results after the calculation process is completed. Its advantage is large-scale throughput, and its disadvantage is high latency; while stream processing does not need to perform operations on the entire data set, but on each data Item execution operations are suitable for processing real-time data. Stre...

AI Technical Summary

Problems solved by technology

[0003] Batch processing is suitable for operating large-capacity historical data sets and returning results after the calculation process is completed. Its advantage is large-scale throughput, and its disadvantage is high latency; while stream processing does not need to perform operations on the entire data set, but on each data Item execution operations, suitable for processing real-time data, stream processing can achieve low latency, but the data is not accurate enough

[0004] However, various computing tasks have different requirements for throughput, latency, and data accuracy. Currently, neither the batch-based computing framework nor the stream-based computing framework is suitable for various computing tasks.

Images