Database conversion and cleaning information processing method

Abstract

The invention relates to a database conversion and cleaning information processing method, which comprises the following steps of: 1) connecting a target database to a data source; 2) selecting a target data table to be cleaned in the target database; 3) selecting an update mode, executing the fourth step if the incremental update is adopted, and executing the tenth step if the total update is adopted; 4) obtaining the maximum update time last_update in the target data table, and defaulting the last_update as the set time if the target data table is null; 5) screening all records with the update time greater than the last_update in the data source to a temporary table temp_table; 6) deleting repeated records in the temp_table by restraining fields in the target data table; 7) comparing the target data table and the temp_table and obtaining the records of the temp_table in the target data table, and the like. Compared with the prior art, the method has the advantages that the problems of data repetitiveness and omission in the data cleaning process are effectively avoided, the data consistency and the completeness are ensured, and the like.

Description

technical field [0001] The invention relates to a database-related technology, in particular to a database conversion and cleaning information processing method. Background technique [0002] Data cleaning and transformation, also known as ETL (Extract, Transform, Load), is a problem that often needs to be solved in the field of databases, especially in the field of data warehouses. ETL is responsible for extracting data from distributed and heterogeneous data sources such as relational data, flat data files, etc. to the temporary middle layer for cleaning, conversion, integration, and finally loading into the target database (data warehouse, data mart, etc.), Become the foundation of online analytical processing and data mining. [0003] Although there are many professional tools for data cleaning on the market, such as Datastage of Ascential, Powercenter of Informatica, ETL Automation of NCR Teradata, etc., most of these tools are powerful, but their use is also relativel...

AI Technical Summary

Problems solved by technology

[0003] Although there are many professional tools for data cleaning on the market, such as Ascential’s Datastage, Informatica’s Powercenter, NCR Teradata’s ETL Automation, etc., most of these tools are powerful, but at the same time their use is relatively complicated

However, as a general small and medium-sized application, the cost of using these professional tools is too high, and generally turn to some lighter tools, such as SSIS or directly use stored procedure programming to achieve

Images