Data cleaning algorithm based on Internet trading information

Abstract

The invention provides a method for cleaning data based on different data resources, i.e., different Internet trading platforms. According to the method, firstly, tuples in a database are classified; correctness-confirmed tuple data in the tuples are subjected to mode interaction with an expert knowledge base; fuzzy matching based on retrieval contents of the knowledge base is used as a tool to obtain corresponding mode knowledge; then, the found mode knowledge is used for cleaning applicable data with quality problems. Meanwhile, a proper efficient detection scheme is also provided for quality errors of different types of mass data. A BP (Back Propagation) neural network method is adopted for realizing the self-learning expert knowledge base, thereby providing an efficient and safe cleaning mode for the Internet trading information data cleaning.

Description

technical field [0001] The invention relates to the field of computer applications, in particular to a data cleaning algorithm for Internet transaction information. Background technique [0002] In recent years, my country's Internet transactions have continued to maintain rapid development, with an average growth rate of 80% in the past five years. In 2013, the total transaction volume of e-commerce exceeded 10 trillion yuan, and the scale of the online retail market has surpassed that of the United States to become the world's largest online retail market. With the development of e-commerce, some problems that are difficult to solve in the market itself have emerged, including false promotion of products, proliferation of counterfeit goods, popular online fraud and phishing websites, irregular logistics and distribution services, difficulties in returning goods and unsmooth reverse logistics, and Internet users. Personal information leakage and other issues. The main rea...

AI Technical Summary

Problems solved by technology

[0005] 1. Data cleaning through the functional dependencies between keys in relational data is a relatively straightforward method, but it is not sufficient for rule mining of massive data such as transaction information on the Internet

[0006] 2. The data method based on conditional functional dependence uses functional dependence as the basis and adds semantic constraints, which can effectively clean data tuples with functional dependence. However, Internet transaction information comes from different e-commerce platforms, and many The functional dependence of the data is not clear, and some data cannot obtain the functional relationship before cleaning

The advantage of this method is that the accuracy of human participation will be greatly improved, but the time consumption of processing is relatively large; at the same time, different people cannot guarantee that the criteria for judging the rules of dependency are completely consistent, and the subjective dependence is too strong.

[0008] 4. Adopt the feedback method of machine learning, that is, replace the human feedback process with the machine learning method, let the machine learn the correct cleaning operation before the cleaning process, and then accumulate and learn continuously during the cleaning process, which can increase the time efficiency of the algorithm , but the accuracy has declined, and the learning process will increase the additional overhead of the system, and at the same time, the dependency between the data is still relatively high during the cleaning process

[0009] To sum up, the current data cleaning methods have certain limitations for the processing of Internet transaction information.

Images