An xml big data clustering integration method for parallel AP propagation

Abstract

The invention discloses a parallel AP propagating XML big data clustering integration method. The method includes the steps that preprocessing such as cleaning, dividing and extracting is conducted on each piece of XML big data; all keywords in an extracted subtree are regarded as the feature description of a data point; a clustering integration basic idea is adopted; a large similarity matrix decomposition idea is also related; ultimate clustering integration is achieved. According to the parallel AP propagating XML big data clustering integration method, a random subspace classifier is established, and parallel random selection of the subtree is conducted to enlarge the difference of clustering members and improve the clustering performance; disambiguation processing is introduced, the ambiguity problem caused by the inconformity of semantic related environments and content in each subtree is solved, meanwhile, semantic similarity and path similarity are integrated, and the influence of inaccurate XML document similarity calculation on an initial clustering result is eliminated; a system capacity theory is used, the iterative approach of an attribution matrix and an absorption matrix in an AP algorithm is improved, so that clustering integration of the XML big data is realized, and the clustering integration method efficiency is improved.

Description

technical field [0001] The invention belongs to the application field of big data integration methods, in particular to an XML big data clustering and integration method for parallel AP propagation. Background technique [0002] At present, XML big data, like other types of big data, has the characteristics of large volume, complex structure, fast generation speed, huge value but low density, and the data volume ranges from MB to GB, TB, PB to ZB. In addition, it The data presents non-convex characteristics and is very unevenly distributed, with many data noises and outliers, and many data appear on the Web in the form of data streams. Therefore, for these fast-changing and highly time-sensitive XML big data , if traditional algorithms are used for clustering integration, these integration methods have obvious deficiencies in solving large XML data sets, which are mainly manifested in: (1) large storage space occupied, slow prediction speed, and poor prediction effect; (2) ...

AI Technical Summary

Problems solved by technology

[0002] At present, XML big data, like other types of big data, has the characteristics of large volume, complex structure, fast generation speed, huge value but low density, and the data volume ranges from MB to GB, TB, PB to ZB. In addition, it The data presents non-convex characteristics and is very unevenly distributed, with many data noises and outliers, and many data appear on the Web in the form of data streams. Therefore, for these fast-changing and highly time-sensitive XML big data , if traditional algorithms are used for clustering integration, these integration methods have obvious deficiencies in solving large XML data sets, which are mainly manifested in: (1) large storage space occupied, slow prediction speed, and poor prediction effect; (2) Online machine learning is difficult, effective for small-scale data, but poor for large-scale data; (3) poor dynamics and real-time performance, unable to process streaming data; (4) due to lack of prior knowledge, the algorithm cannot grasp the global characteristics of XML data distribution Inaccurate, eventually leading to unsatisfactory requirements for clustering accuracy and clustering results

Images