An Unsupervised Automatic Data Cleaning Method

Abstract

The invention discloses an unsupervised data automatic cleaning method, which includes the following steps: A. Data model learning, learning the dependency relationship between attributes from the original data that may contain invalid data, by finding out the implicit non-absolute Or relatively weak dependencies, to obtain a data model expressed in the form of a Bayesian network; B. Generation of data cleaning rules; after obtaining the complete data model of the original data or original data sampling, the data cleaning rules are generated , and specifically generate predicates and first-order predicate rules; C. Generate a Markov logic network based on the predicates and first-order predicate rules generated in step B; D. Generate inference rules based on the Markov logic network generated in step C And data cleaning based on inference results. The method of the invention can effectively improve the data quality of various business systems of the company without consuming a lot of manpower and material resources, and helps the management to make correct decisions.

Description

technical field [0001] The invention relates to the technical field of data management, in particular to an unsupervised automatic data cleaning method. Background technique [0002] Data in the real world usually needs to be cleaned (the data that needs to be cleaned is defined as dirty data below), such as may contain inconsistent, noisy, incomplete or repeated values. In business, erroneous data can cause great financial losses. For example, wrong customer information may lead to wrong delivery of the goods purchased by the company, which not only increases the delivery cost of the company, but also has a relatively large negative impact on the image of the company for a long time. [0003] Among the existing data cleaning methods, some methods require heavy manual participation in the data cleaning process, such as providing suggestions for cleaning or confirming repairs, etc.; although some methods do not require manual participation in the cleaning process, they need ...

AI Technical Summary

Problems solved by technology

In business, erroneous data can cause large financial losses

For example, wrong customer information may lead to wrong delivery of the goods purchased by the company, which not only increases the delivery cost of the company, but also has a relatively large negative impact on the image of the company for a long time

[0003] Among the existing data cleaning methods, some methods require heavy manual participation in the data cleaning process, such as providing suggestions for cleaning or confirming repairs, etc.; although some methods do not require manual participation in the cleaning process, they need to be formulated in advance. rule

Existing data cleaning methods are not applicable when data rules are unknown or labor costs are unaffordable

Images