Eureka AIR delivers breakthrough ideas for toughest innovation challenges, trusted by R&D personnel around the world.

Method and system for monitoring ETL (extract-transform-load) data processing process

A data processing and data table technology, applied in the field of data processing, can solve problems such as poor accuracy and low efficiency of monitoring methods, and achieve the effects of improving accuracy, improving monitoring efficiency, and improving objectivity and accuracy

Active Publication Date: 2011-07-06
TAOBAO CHINA SOFTWARE
View PDF4 Cites 34 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The embodiment of the present application provides a monitoring method and system for the ETL data processing process to solve the problems of low efficiency and poor accuracy of the monitoring method of the ETL data processing process in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for monitoring ETL (extract-transform-load) data processing process
  • Method and system for monitoring ETL (extract-transform-load) data processing process
  • Method and system for monitoring ETL (extract-transform-load) data processing process

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] In view of the above-mentioned problems existing in the prior art, the embodiments of the present application provide a monitoring method and a system thereof for the ETL data processing process, through independent ETL data processing links (such as data extraction, conversion or Loading) for monitoring, and during the monitoring process, the monitoring indicators of each processing link are automatically generated according to the processing tasks of the processing link, so that on the one hand, the problem data can be found in time and the cause of the problem data can be further analyzed, thus improving the monitoring of ETL Efficiency and the accuracy of monitoring results are improved. On the other hand, it does not rely on manual configuration of monitoring assignments, which further improves efficiency and simplifies the implementation process of ETL monitoring, and reduces the influence of human factors. The accuracy of the ETL monitoring result and the quality ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The application discloses a method and system for monitoring the ETL (extract-transform-load) data processing process. The method comprises the following steps: determining the field type of output data of the ETL data processing process according to ETL data processing task information; generating the monitoring indexes of the ETL data processing process according to the field type of the outputdata; and counting or calculating the corresponding fields in the output data of the ETL data processing process according to the generated monitoring indexes to obtain the result values of the monitoring indexes; wherein the monitoring indexes indicate a process for counting or calculating the appointed fields in the output data of the ETL data processing process according to a specified manner.The method provided in the application can solve the problems of the ETL data processing process monitoring method in the prior art, such as low efficiency and poor accuracy.

Description

technical field [0001] The present application relates to data processing technology in the communication field, and in particular to a monitoring method and system for ETL data processing process. Background technique [0002] ETL is the abbreviation of Extraction-Transformation-Loading, and the Chinese name is data extraction, transformation and loading. ETL is responsible for extracting data from distributed and heterogeneous data sources, such as relational data, flat data files, etc., to the temporary middle layer for cleaning, conversion, integration, and finally loading them into data warehouses or data marts to become online analysis Fundamentals of data processing and data mining. ETL is a very important part of the data warehouse, it is a necessary step in the past. [0003] The quality problems of ETL are specifically manifested in several characteristics such as correctness, integrity, consistency, completeness, validity, timeliness and availability. There are...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
Inventor 杨柏刚
Owner TAOBAO CHINA SOFTWARE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com