A method and system for realizing ETL scheduling

Abstract

The application discloses a method for realizing ETL scheduling, which includes: determining at least one task required to perform based on the preset task configuration file including various tasks required to perform by completing the ETL scheduling and the dependency relationship between these tasks; acquiring the resource information of each application server, and comparing it with the restriction condition corresponding to the at least one task, and screening these application servers based on the comparative result, and selecting an application server to perform the at least one task; allocating the at least one task to the selected application server and indicating the application server to implement the at least one task based on the set program code. So that the centralization management of the application server resources is realized, to reasonable allocate the application server resources in the system, accordingly realizing the system load balancing. The application also discloses a management system.

Description

technical field [0001] The present application relates to the computer field, in particular to a method and system for realizing ETL scheduling. Background technique [0002] Data Warehouse (DW) is a subject-oriented, integrated, relatively stable data collection that reflects historical changes and is used to support management decisions. A data warehouse is an independent data environment, and data extraction, transformation and loading (Extraction-Transformation-Loading, ETL) is an important part of building a data warehouse. [0003] ETL is used to extract data from distributed and heterogeneous data sources (for example, relational data, flat data files, etc.) Loaded into the data warehouse, so that the built data warehouse becomes the basis of online analytical processing and data mining. Technically, ETL mainly involves several aspects such as association, transformation, increment, scheduling and monitoring. Usually, the data in the data warehouse does not require...

AI Technical Summary

Problems solved by technology

[0005] However, under the existing technology, program codes are scattered and configured on various application servers, and an application server can only execute a fixed task flow according to the content of the configured program code. Therefore, all tasks belonging to the same task flow must be assigned to the same application In this way, it is difficult for the application server resources in the system to be allocated reasonably, and it is easy to appear that some application servers are idle, while some application servers are overloaded, resulting in serious system load imbalance, and it is not conducive to application Further expansion of the number of servers; on the other hand, setting each application server as an independent operation unit will bring inconvenience to later maintenance, because when an application server has an operation error, maintenance personnel need to log in to each application server for maintenance, thereby It will consume a lot of manpower and material resources and increase the maintenance cost

Images