Data importing processing method and data processing device

Abstract

The invention discloses a data importing processing method and a data processing device. The data importing processing method includes acquiring source data to be imported, a source data model and a target data model; establishing a mapping relation between the source data model and the target data model and converting the acquired source data according to the established mapping relation; acquiring preset business rules of the target data model and filtering the converted source data; and computing global similarity of the filtered source data and target data, deleting the source data with global similarity values higher than a preset similarity threshold value and importing the source data which meet similarity conditions into a data table of a target database after similarity processing is performed on the source data. The data importing processing method and the data processing device have the advantages that by the data importing processing method and the data processing device, the consistency of stored data can be improved, the storage efficiency is improved, and data redundancy is reduced.

Description

technical field [0001] The invention relates to data processing technology, in particular to a data import processing method and a data processing device. Background technique [0002] With the rapid development of the Internet / Intranet (Internet / Intranet) and the wide application of database technology, the amount of data generated by applications in various industries is also increasing. Among them, through the database technology, the massive data generated can be stored, the data can be backed up, and the data analysis and maintenance of the stored data can be facilitated later; and through the Internet / Intranet, the resource sharing of application data in various industries can be realized. In the prior art, databases can use different data structures to store data. Therefore, the data structures of the stored data are different among the databases, which is called heterogeneous data. How to make the heterogeneous data between the databases undergo data processing , ca...

AI Technical Summary

Problems solved by technology

[0005] However, in practical applications, since data can be imported from multiple source databases, the data in the imported target database will have duplicate data, garbage data, null values, wrong data, etc. Data table attributes are incompatible, resulting in various technical problems such as source data import failures

[0006] It can be seen from the above that in the existing data processing method based on data import, when the properties of the source data table are compatible with the properties of the data table in the target database, the source data table containing the source data is copied to the data table of the target database ; and when the attributes of the source data table are incompatible with the attributes of the data table in the target database, the source data table containing the source data is inserted into the target database as a new data table. In this way, data processing of the source data is not considered. Make the data imported into the target database redundant, and import garbage data and duplicate data that do not meet the data table attribute requirements in the target database into the target database, resulting in poor consistency of data stored in the target database and poor storage efficiency. Low; further, the target data contains data with different data table attributes, which affects the performance of the target database, and even causes the target database to fail to operate normally

Images