ETL (Extract-Transform-Load) realization method and system based on metadata

Abstract

The invention provides an ETL (Extract-Transform-Load) realization method based on metadata. The ETL realization method comprises the following steps: obtaining a pre-programmed ETL request; converting the ETL request to form ETL metadata according to a data conversion rule; leading the ETL metadata into a pre-established ETL metadata model to store the ETL metadata; generating an ETL configuration file according to the ETL metadata; and compiling the generated ETL configuration file into executable ETL JOB. According to the ETL realization method, the ETL management efficiency is greatly improved and convenience is provided for development and utilization of a data warehouse; the technical defects that faults can be checked only if the development is complete when an ETL tool is used are effectively solved; the standardized metadata model can very clearly show an ETL process and a developer can conveniently tease and check a current simulated ETL flow; the understanding on the ETL requirements by the developer are deepened and the developer is helped to find an optimal ETL design scheme; the finally obtained ETL JOB can be accurately and efficiently executed.

Description

technical field [0001] The invention belongs to the technical field of computer ETL, relates to an ETL implementation method and system, in particular to an ETL implementation method and system based on metadata. Background technique [0002] ETL (Extract Transform Load, data warehouse technology) is the process of data extraction (Extract), transformation (Transform), and loading (Load). It is an important part of building a data warehouse. Users extract the required data from the data source. After data cleaning and conversion, the data is finally loaded into the data warehouse according to the pre-defined data warehouse model. [0003] At present, the ETL data warehouses of various enterprises basically adopt two development methods: one is based on process language development, such as: SQL language (Structured Query Language, structured query language) and PERL language (Practical Extraction and Report Language, practical report extraction language) etc.; another graph...

AI Technical Summary

Problems solved by technology

[0005] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a method and system for implementing ETL based on metadata, which is used to solve the problem that a complete ETL JOB or process can be configured and verified when implementing ETL development in the prior art. , once an error occurs during the configuration verification process, it may be necessary to check the entire ETL JOB or reorganize the logic during troubleshooting, resulting in time-consuming and low-efficiency problems

Images