Distributed ETL data exchange system and method based on micro-service architecture

Abstract

The invention belongs to the technical field of computers. The invention discloses a distributed ETL data exchange system and method based on a micro-service architecture. The method depends on a platform based on a micro-service architecture. A process dispatcher, a process instance registration center and an ETL process executor capable of dynamically generating instances in a stateless mode areadopted. Construction is carried out through a front-end and rear-end separation framework; According to the ETL data exchange platform capable of dynamically and elastically stretching, the functions of automatic fault transfer of the ETL process, automatic reduction of process instances and unified registration of process states are realized; when the server resources are excessive, the container arrangement platform can quickly recover the server resources, the server resources and memory occupied by the ETL system can be saved, the utilization rate of the enterprise server resources can be greatly improved, and the cost is saved; and memory occupation can be saved, the data processing capacity of the ETL process is improved, ETL process execution and scheduling efficiency is improved,meanwhile, management and monitoring efficiency is improved, and process scheduling reliability is improved.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to a distributed ETL data exchange system and method based on a microservice architecture. Background technique [0002] The scope of computer technology is becoming more and more extensive, and it has been widely used in industrial production, daily life, public management and other places. With the application of big data in all walks of life, there are more and more demands for data collection, cleaning, desensitization, etc. For example, many data in the enterprise business system are relatively scattered, and the data between each business system needs to realize fast interconnection and interoperability. The enterprise business system in the prior art mainly uses code writing or third-party open source tools for data For exchange and cleaning, the mainstream ETL tools or technologies are mainly developed or operated based on single-machine or C / S single-architect...

AI Technical Summary

Problems solved by technology

[0002] The scope of computer technology is becoming more and more extensive, and it has been widely used in industrial production, daily life, public management and other places. With the application of big data in all walks of life, there are more and more demands for data collection, cleaning, desensitization, etc. For example, many data in the enterprise business system are relatively scattered, and the data between each business system needs to realize fast interconnection and interoperability. The enterprise business system in the prior art mainly uses code writing or third-party open source tools for data For exchange and cleaning, the mainstream ETL tools or technologies are mainly developed or operated based on single-machine or C / S single-architecture models. There are problems with low data concurrency, low resource utilization, and inability to transfer to the server in time when a single node fails. Other nodes are running, and at the same time, it cannot implement orchestration and flexible scheduling based on containerization technologies such as Docker, and cannot perform distributed scheduling. As a result, when there are many ETL processes, it will cause too much server memory usage, resulting in memory overflow and other problems. At the same time, when a certain process has a problem It may affect other processes to stop scheduling; although there are some ETLs that separate the scheduler independently, the number of process execution nodes in this architecture is fixed, and the system cannot dynamically increase nodes in a timely manner according to the concurrency of processes and CPU usage When the utilization rate of server resources is not high, nodes cannot be recovered in time, resulting in waste of computing resources and low efficiency

Images