Spark-based multi-feature combined efficient Chinese text clustering method

Abstract

The invention discloses a Spark-based multi-feature combined efficient Chinese text clustering method. the method comprises the following steps of: uploading mass data sets into an HDFS file system byutilizing high fault tolerance and high data access throughput of the HDFS file system, carrying out data preprocessing and submitting the data sets to a Spark cluster; and after text set preprocessing is completed, respectively calculating a semantic similarity and a word frequency statistics-based cosine similarity of a dimensionality-reduced text, combining the two similarities to obtain a final text similarity, and carrying out text clustering by utilizing the obtained text similarity and combining a maximum distance method. According to the method, semantic information and word frequencystatistics information are combined to ensure that the text similarity calculation is more correct and the number of iterations is greatly decreased at the same time.

Description

technical field [0001] The invention belongs to the field of machine learning, in particular to a Spark-based multi-feature combined Chinese text efficient clustering method. Background technique [0002] Clustering technology is a kind of machine learning. It mainly divides the original sample data set into several different data categories based on the differences between sample data and different parameters. Therefore, the ultimate goal of clustering is to make the difference between different samples divided into the same data cluster smaller, while the difference between samples divided into different data clusters is larger. [0003] Text clustering technology is a kind of clustering, which is mainly based on the following principle: the differences between texts belonging to the same cluster are small, while the differences between texts belonging to different clusters are relatively large. Different from classification, clustering technology belongs to a class of un...

AI Technical Summary

Problems solved by technology

[0008] (1) High-dimensional sparseness: The current text clustering algorithms are all calculated based on the VSM model. Although this model is very simple, because this model expresses text as vectors, it leads to the generation of high-dimensional vectors. This increases the computational complexity and time complexity

[0009] (2) Loss of semantic information and simplification of clustering features: In the calculation of text similarity based on TF-IDF weights, because the semantic similarity between words is not considered, the effect of clustering is not good

[0010] (3) It takes a long time and takes up a lot of space: most of the current algorithms are based on a single machine, which takes a long time to process data and has low computational efficiency.

Images