Clustering method and system based on big data parallel computation

Abstract

The invention discloses a clustering method based on big data parallel computation. The clustering method comprises following steps of receiving data to be aggregated which is acquired by multiple threads of a larger cluster in parallel; saving the data to be aggregated in a first database; extracting the data characteristic of the data to be aggregated, calling cluster models in parallel during the same period by multiple threads, calculating and analyzing the aggregation class of the data to be aggregated independently in a distributed manner, and carrying out same class aggregation; saving the data which is subjected to same class aggregation in a second database; and storing the data which is subjected to same class aggregation in a memory, and establishing a cluster data index. The invention also discloses a clustering system based on big data parallel computation, the Text fingerprint is accurately positioned, dimension reduction is simple, and the clustering topic accuracy is improved.

Description

technical field [0001] The invention relates to the field of text mining and automatic clustering, in particular to a clustering method and system based on big data parallel computing. Background technique [0002] Existing article clustering technologies such as kmeans, hierarchical clustering, SOM, and FCM are all based on word frequency and probability to classify and integrate articles, and there are uncontrollable errors. Among them, the initial point of k-means clustering algorithm The selection is unstable and is randomly selected, which causes the instability of the clustering results; although hierarchical clustering does not need to determine the number of categories, once a split or merge is performed, it cannot be corrected, and the clustering quality is limited; FCM clustering The class algorithm is sensitive to the initial cluster center, needs to manually determine the number of clusters, and is easy to fall into a local optimal solution; the SOM clustering al...

AI Technical Summary

Problems solved by technology

[0002] Existing article clustering technologies such as kmeans, hierarchical clustering, SOM, and FCM are all based on word frequency and probability to classify and integrate articles, and there are uncontrollable errors. Among them, the initial point of k-means clustering algorithm The selection is unstable and is randomly selected, which causes the instability of the clustering results; although hierarchical clustering does not need to determine the number of categories, once a split or merge is performed, it cannot be corrected, and the clustering quality is limited; FCM clustering The class algorithm is sensitive to the initial clustering center, needs to manually determine the number of clusters, and is easy to fall into a local optimal solution; the SOM clustering algorithm has a strong theoretical connection with the actual brain processing

But the processing time is longer, further research is needed to adapt it to large databases

Moreover, the existing article clustering technology is not a parallel computing version. The general probability model is used to obtain weights, and the error is relatively high. The global feature vector is used for association, and the amount of calculation is huge.

Images