Data cleaning method based on functional dependency

Abstract

The present invention discloses a data cleaning method based on functional dependency. The method is characterized by comprising: performing data conversion on raw data so as to totally convert different types of attributes into numerical attributes; extracting self-dependency function characteristics of the attributes of the raw data after data conversion; and extracting an inter-dependency function among the attributes of the raw data after data conversion; and determining the attributes which need to be cleaned and are to be cleaned as well as an sample according to the self-dependency function characteristics and the inter-dependency function, forming a relevant cleaning policy basis according to the attributes and the sample, determining whether an to-be-cleaned attribute object is cleaned by adopting the self-dependency function or the inter-dependency function, and if the self-dependency function is adopted to clean the to-be-cleaned attribute object, performing calibration and recovery on the sample which does not accord with the condition according to a polynomial determined by the self-dependency function, and using white noise as random perturbation. According to the data cleaning method provided by the present invention, the dirty data problem in mass data can be solved and high quality data is provided for subsequent analysis and mining of the mass data.

Description

technical field [0001] The invention belongs to the field of big data processing, and more specifically, relates to a data cleaning method based on functional dependence Background technique [0002] With the rapid development of mobile Internet and information technology, the data of the government, enterprises and various industries is growing at a rate of TB / s. People's life, business decision-making and precision services are increasingly dependent on data, and the requirements for data quality It is also getting higher and higher, and any "dirty data" will have an impact on data analysis and target decision-making. As the awareness of "data-driven operations" has been widely recognized and popularized in all walks of life, people are spending more and more energy on data analysis and mining. More than 80% of the time is spent on "dirty data" processing. [0003] The causes of "dirty data" include system failures, cross-system, multi-source data, changes in data standa...

AI Technical Summary

Problems solved by technology

[0004] However, the above-mentioned studies and methods have certain limitations in application scenarios, either constrained by the spatiotemporal characteristics and importance of the data itself, or more influenced by expert experience

Images