Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A data quality control method and system based on an ETL process

A technology of data quality control and process data, applied in the field of data analysis, can solve the problems of large amount of original data and the inability of relational databases to support verification well, so as to achieve reliable data quality, reduce manpower verification costs, and improve efficiency

Pending Publication Date: 2019-06-28
BOCO INTER TELECOM
View PDF5 Cites 33 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In order to ensure the accuracy of key indicators, it is often checked whether the key fields meet the specification requirements, which requires a lot of technical foundation as support, especially for signaling XDR data, which has a large amount of original data, and traditional relational databases cannot well support verification. , requires specialized technical personnel to handle

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data quality control method and system based on an ETL process
  • A data quality control method and system based on an ETL process
  • A data quality control method and system based on an ETL process

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0085] In order to make the above objects, features and advantages of the present application more obvious and comprehensible, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0086] In the description of the present application, it should be understood that the terms "first" and "second" are used for description purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. "Plurality" means two or more, unless otherwise clearly and specifically defined. The terms "including", "comprising" and similar terms should be understood as open-ended terms, ie "including / comprising but not limited to". The term "based on" is "based at least in part on". The term "an embodimen...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data quality control method and system based on an ETL process. The method comprises the steps: importing metadata, and obtaining the ETL process according to the data of eachtable in the metadata; Setting a corresponding check rule for each check node, and defining an SQL needing to be executed; According to the SQL, setting a data scheduling task according to a preset execution period, and checking the collected data to obtain a checking result; Comparing the inspection result with a preset alarm threshold value, if the inspection result meets a threshold value range, generating an alarm detailed list, and inserting the alarm detailed list into a database; associating and summarizing the alarm detailed list data to the facts of the data warehouse layer to summarize the data; And presenting the inspection result in a manner of alarm order query, log query, process display and / or report display. Through the method and the device, the ETL process problem node can be quickly positioned, and the data quality is ensured.

Description

technical field [0001] The present application relates to the technical field of data analysis, in particular, to a data quality control method and system based on an ETL process. Background technique [0002] Data warehouse technology (Extract-Transform-Load, ETL) is used to describe the process of extracting, transforming, and loading data from the source to the destination. [0003] In the process of ETL data processing, many links will be managed, see figure 1 . Due to factors such as the filtering method, cleaning method, original data extraction rules, whether the conversion process is successfully executed, and whether the loading process type is correct, etc., in each link, data records are lost, data is inaccurate, conversion process fails, timeouts, etc. . When locating these links, due to the many links, the use of many technologies, and the many causes of the problems, maintenance personnel have no way to locate the problem, or spend a lot of time on data veri...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/25
Inventor 高宇周章雄陈少钦刘永江
Owner BOCO INTER TELECOM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com