A Large-Scale Data Mining Method Guaranteeing Quality Monotonicity

Abstract

The invention provides a data mining method capable of guaranteeing quality monotony. The method comprises the following steps: after an original big data set is compressed by a PCA (principal components analysis) technology, mapping the original big data set onto an R-tree data structure; then, carrying out mining processing on the data set by an improved K-nearest neighbor classification algorithm. The method mainly comprises the following two parts including a coding part and a mining part, wherein the coding part utilizes R-tree to present data, data with high similarity in the data is combined to serve as one node of the R-tree so as to achieve a purpose of mass data compression and improve the efficiency of the mining part; the mining part utilizes the thought of the improved K-nearest neighbor classification algorithm to process the data node and predict the classification of an input test point. According to the large-scale data mining method, the problem that the quality of a mining result and resource restriction cannot be balanced and the quality monotony of an approximate result cannot be guaranteed when big data is mined by a traditional algorithm under the restriction of limited time and resource restriction can be solved.

Description

technical field [0001] The invention relates to a method for efficiently processing data, through which the monotonicity of the quality of large-scale data mining results is guaranteed, and belongs to the cross-technical application field of data mining, big data and computer software. Background technique [0002] The data capacity and type diversity of big data make us willing to use algorithms to generate approximate results to mine big data. Traditional algorithms are difficult to achieve mining results when mining big data under limited time and resource constraints. The balance of quality against resource constraints and the problem of ensuring quality monotonicity of approximated results. To solve this problem, based on Shannon entropy, we design a big data mining method that guarantees quality monotonicity. The mining method is divided into two parts: the coding part and the mining part. By ensuring the monotonicity of the entropy of the coding part of the algorithm ...

AI Technical Summary

Problems solved by technology

[0002] The data capacity and type diversity of big data make us willing to use algorithms to generate approximate results to mine big data. Traditional algorithms are difficult to achieve mining results when mining big data under limited time and resource constraints. The balance between the quality of , and the resource constraints and the problem of guaranteeing the monotonicity of the quality of approximate results

Images