Text data clustering method, device and equipment based on non-parametric VMF hybrid model

Abstract

The invention discloses a text data clustering method, device and equipment based on a non-parametric VMF hybrid model. The method comprises the steps of S101 obtaining a to-be-clustered text data set, wherein the text data set comprises a plurality of texts, and each text is expressed as a D-dimensional text vector feature by using a word frequency-inverse text frequency index standardization method; S102 performing modeling on each text by using a non-parametric VMF hybrid model based on a Pitman-Yor process; S103 estimating model parameters of the non-parametric VMF hybrid model through a variational Bayesian inference algorithm; S104 judging whether the non-parametric VMF hybrid model is converged or not according to the deduced model parameters; if not, returning to the step S103, andif yes, executing the step S105; and S105 judging the category of each text according to the posterior probability of the indication factor so as to cluster the texts according to the categories. According to the invention, algorithm convergence can be ensured and the convergence state can be effectively detected.

Description

technical field [0001] The invention relates to the field of text mining, in particular to a text data clustering method, device and equipment based on a non-parametric VMF mixed model. Background technique [0002] With the rapid development of the Internet and the widespread use of news documents, text data clustering, as one of the most useful tasks in text mining, has received increasing attention in recent years. [0003] In prior art 1, Zhong Wenliang et al. proposed a method for clustering unbalanced text data based on the Pitman-Yor process. In this method, each text is represented by a TF (term frequency, term frequency) vector, each attribute of the vector represents the frequency of a specific term (term) appearing in the document, and all terms in each category obey The same multinomial distribution (Multinomial Distribution). This method uses the Polya urn model to build a Pitman-Yor process model based on multinomial distribution, and uses the Gibbs sampling ...

AI Technical Summary

Problems solved by technology

The discrete nature of the Pitman-Yor process model cannot be represented intuitively

[0006] 3. The Gibbs sampling algorithm used to solve the model parameters cannot obtain an analytical solution, and it is not easy to converge and it is difficult to determine the convergence state

The concentration parameter in the prior art 2 uses an asymptotic approximation method to obtain an estimated value, but this estimation method cannot effectively deal with high-dimensional data

Images