Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Processing method and system based on workflow ETL

A processing method and processing system technology, applied in the field of data processing in data warehouses, can solve problems such as weak process control ability and inability to combine manual approval processes, etc.

Pending Publication Date: 2021-12-07
COSCO SHIPPING TECH CO LTD
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention aims at the problems that the process control ability of the existing ETL process is weak and cannot be combined with the manual approval process. The present invention provides a processing method based on workflow ETL, which inserts the manual approval process into the automated ETL to ensure the sequential scheduling of tasks , and improved process control capabilities, solved ETL applications in some special scenarios, and improved ETL work efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Processing method and system based on workflow ETL
  • Processing method and system based on workflow ETL
  • Processing method and system based on workflow ETL

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] The present invention will be described below in conjunction with the accompanying drawings.

[0030] The present invention relates to a processing method based on workflow ETL, the flow chart of which is as follows figure 1 As shown, it includes: deploying a task scheduling platform, the task scheduling platform regularly schedules ETL task executors according to the set execution cycle, and provides several routing strategies to distribute tasks to different servers in the cluster by using a distribution algorithm; deploying ETL task execution The ETL task executor evenly distributes task scheduling to the servers in the cluster and executes the task through the configured ETL execution script, and also adds task types that require manual intervention in the task type, and configures the task status code To record the task execution status, update the corresponding task status code when entering the workflow or other task status changes; deploy the workflow platform, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a processing method and system based on workflow ETL, and the method comprises the steps: deploying a task scheduling platform, deploying an ETL task actuator, deploying a workflow platform, and deploying a metadata management platform, and deploying the task scheduling plate for timely regulate the ETL task actuator, a plurality of routing strategies are provided, and tasks are distributed to different servers in a cluster through a distribution algorithm; the ETL task executor is deployed to uniformly disperse task scheduling to servers in a cluster for execution, task types needing manual intervention are added in the task types, task execution states are recorded by configuring task state codes, in practical application, through the design mode, task ETL scheduling is achieved, the link of manual intervention is added, the ETL application of some special scenes is solved, and the ETL working efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of data processing in data warehouses, in particular to a processing method and system based on workflow ETL. Background technique [0002] The abbreviation of ETL (Extract-Transform-Load) is used to describe the process of how to extract, transform, and load from the source of heterogeneous data to the destination of the data warehouse. ETL is an important part of data warehouse construction. Users extract the required business data from various business systems, clean them according to certain rules, and finally load the data into the data warehouse according to the pre-designed data warehouse model. [0003] The current mainstream ETL extraction, loading and conversion jobs require a centralized scheduling platform to establish process execution flow control. The scheduling algorithm is used to ensure the extraction sequence of ETL, and to capture and process errors. Usually, the scheduling platform can o...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/28G06Q10/10
CPCG06F16/254G06F16/283G06Q10/103
Inventor 郭照军孙冠豪
Owner COSCO SHIPPING TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com