Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

ETL data cleaning method and system

A data cleaning and data table technology, applied in database management system, database index, electronic digital data processing and other directions, can solve the problems of large amount of prisoner data, troublesome data reporting in prisons, error-prone, etc., to improve convenience and accuracy The effect of stability, reliable design principle and simple structure

Pending Publication Date: 2020-05-01
SHANDONG ZHONGCI ROMOTE VIDEO TECH
View PDF5 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In the existing technology, the data of each provincial prison is copied to the Ministry of Justice through the USB flash drive sent by the company and the respective prisons, and then the data is transferred manually. This method is relatively slow and inconvenient, and because there are many systems for prisoner data statistics , and the data volume of prison inmates is relatively large, the above method is prone to errors, which has caused great trouble for the prison to report data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • ETL data cleaning method and system
  • ETL data cleaning method and system
  • ETL data cleaning method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0064] figure 1 It is a schematic flowchart of an ETL data cleaning method according to an embodiment of the present invention.

[0065] Such as figure 1 As shown, the method 100 includes:

[0066] Step 110, select the type of source database;

[0067] Step 120, select a source data table belonging to the selected type;

[0068] Step 130, select the target data table to be matched;

[0069] Step 140: Read the selected source data table and target data table, and according to the set field mapping relationship of the source data table and its corresponding target data table, convert the target field and the field value in the corresponding source data table to The Json format is stored in the target Json file; the target field is a field of the target data table involved in the field mapping relationship;

[0070] Step 150: Parse the above-mentioned target Json file to obtain each target field and the field value in the corresponding source data table;

[0071] Step 160: Generate a corres...

Embodiment 2

[0111] See figure 2 Compared with Embodiment 1, this embodiment is different in that the method 100 described in this embodiment further includes step 180: customizing the start time of ETL data cleaning.

[0112] When in use, the user can customize the start time of ETL data cleaning; when the start time set by the user is reached, step 140 is executed.

Embodiment 3

[0114] Such as image 3 As shown, the difference between this embodiment and Embodiment 2 is that the method 100 described in this embodiment further includes step 190: customizing the field mapping relationship between the source data table and its corresponding target data table .

[0115] When in use, after the user customizes the field mapping relationship between the source data table and its corresponding target data table in step 190, in step 140, it is directly based on the source data table and its corresponding target data table customized by the user. The field mapping relationship of the target field and its corresponding field value in the source data table are stored in the corresponding target Json file in Json format.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an ETL data cleaning method and system. The ETL data cleaning method and system can both select the type of a source database; selecting a source data table belonging to the selected type; selecting a target data table to be matched; reading the selected source data table and target data table, and storing a target field and a field value in the corresponding source data table into a target Json file in a Json format according to a set field mapping relationship between the source data table and the corresponding target data table; wherein the target field is a field ofa target data table involved in the field mapping relationship; analyzing the target Json file to obtain each target field and a field value corresponding to the target field in the source data table;generating a corresponding SQL statement according to the analyzed target field and field value; and writing data in the target Json file into a corresponding target data table by adopting the generated SQL statement. The ETL data cleaning method and device are used for improving the ETL data cleaning accuracy and convenience.

Description

Technical field [0001] The invention relates to the field of database data conversion, in particular to an ETL data cleaning method and system. Background technique [0002] ETL, short for Extract Transform Load, is the process of data extraction (Extract), transformation (Transform), and loading (Load). It is an important part of building a data warehouse. The user extracts the required data from the data source, after data cleaning, and finally loads the data into the target data warehouse according to the pre-defined data warehouse model. [0003] In the prior art, the prison data of each province is copied to the Ministry of Justice through the USB flash drive sent by the company and the respective prison, and then the data is transferred manually. This method is slow and inconvenient, and because there are more systems for collecting prisoner data in prisons In addition, the amount of prison data is relatively large, and the above methods are prone to errors, which causes gr...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/22G06F16/25
CPCG06F16/215G06F16/2282G06F16/254
Inventor 贾伟光牟骏李咸明王兴郭梅子
Owner SHANDONG ZHONGCI ROMOTE VIDEO TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com