ETL scheduling method and device

Abstract

The invention provides an ETL scheduling method and device, and the ETL scheduling method comprises the steps: obtaining a target SQL statement list regularly, wherein the target SQL statement list comprises a plurality of SQL statements, and each SQL statement comprises a corresponding statement type identifier; if it is determined that the SQL statements belonging to the operation type contain the sub-queries according to the statement type identifiers of all the SQL statements, generating a blood relationship comparison relation table based on the identifiers of the source tables and the identifiers of the target tables of the SQL statements containing the sub-queries; and sending the blood relationship comparison relation table and the target SQL statement list to a target scheduling server, so that the target scheduling server executes target scheduling operation according to the mapping relationship table, the target data platform information, the blood relationship comparison relation table and the target SQL statement list. The ETL scheduling method can improve the ETL scheduling accuracy, flexibility and efficiency, can realize dynamic correction of the data model and visual display of the scheduling result, and can improve the data processing reliability.

Description

technical field [0001] The present application relates to the technical field of data processing, in particular to an ETL scheduling method and device. Background technique [0002] Traditional ETL (Extraction-Transformation-Loading) scheduling refers to cleaning and integrating scattered data and data from heterogeneous data sources and then loading them into big data platforms, data platforms, and data marts, which can be used for online analysis and processing. and data mining to provide decision support and data support. The ETL scheduling process is very complicated, not only the sequence and dependencies of each step need to be considered, but also the sequence and dependencies between different jobs need to be considered. [0003] In the fields of precision marketing, product innovation, risk control, market analysis, and resource allocation, raw data can be processed through ETL scheduling, which can provide data support for company management decision-makers, risk ...

AI Technical Summary

Problems solved by technology

[0006] The first method has high labor costs; the development cycle is long, and every time a requirement changes or a problem is corrected, a new version needs to be formulated; it is difficult to meet the business needs of dynamic and high-frequency changes Requirements; the second method, in order to ensure the integrity of the data, it is necessary to determine whether the T-1 or T data is due (T represents a natural day), and then process, count and visualize the data; in the case of frequent statistics, Additional repetitive workload is added, and the processing efficiency is low; manual triggering or timing triggering is required, and the timeliness of data processing is low

Images