ETL system and method completely based on distributed type memory computation

Abstract

Provided is an ETL system completely based on distributed type memory computation. The system comprises five functional modules, namely the data extracting module, the data processing module, the dataintegrating module, the data outputting module and the metadata managing module, and one ETL operation procedure engine. The metadata managing module outputs a metadata control file to be used by theETL operation procedure engine. The ETL operation procedure engine reads the metadata control file, computes the number of the layer where each node is located, and conducts layering on all the operation nodes according to the layer numbers. The ETL operation procedure engine sequentially operates ETL operations of all the layers according to the executing route from the lower layer to the higherlayer and finally completes the execution of all the ETL operations in an ETL operation procedure diagram. By means of the system, based on the Spark and Alluxio technology, the elasticity of the system frame is improved, data management is simplified, and the data processing performance is improved.

Description

technical field [0001] The present application relates to an ETL system, in particular, it belongs to an ETL system completely based on distributed memory computing and a method thereof. Background technique [0002] The explosive growth of data and the development of big data applications have provided unprecedented development opportunities for ETL (data extraction, transformation and loading) software. Traditional ETL software is mostly based on a stand-alone architecture. When processing massive data, there are bottlenecks in IO throughput and system resources, and expansion is difficult and expensive. With the emergence of distributed technology, a new generation of ETL software based on Hadoop and Spark uses the Hadoop distributed file storage system HDFS as the storage layer, uses the distributed memory computing framework Spark as the computing layer, and utilizes advanced DAG execution engines and powerful performance The memory-based multi-round iterative computin...

AI Technical Summary

Problems solved by technology

However, there are many deficiencies in the architecture of Hadoop+Spark. 1. Spark has obvious defects in the logic of memory application, which is reflected in the fact that it cannot be released when the memory is insufficient, and it can only start suicide to end the task.

Memory consumers cannot ask other memory consumers to release when the memory is insufficient. They can only be released by writing data to disk. When the data scale increases, the performance drops sharply

2. The Hadoop + Spark architecture relies on Hadoop's HDFS as the storage system, and cannot support more distributed storage systems. The architecture is not flexible enough.

Images