Text data clustering method, device and equipment based on non-parametric vmf mixed model

Abstract

The invention discloses a text data clustering method, device and device based on a non-parametric VMF hybrid model. The method includes: S101, acquiring a text data set to be clustered; wherein, the text data set includes a plurality of texts, and each text data set includes a plurality of texts. The texts are represented as D-dimensional text vector features by the word frequency-inverse text frequency index normalization method; S102, each text is modeled using a non-parametric VMF mixture model based on the Pitman-Yor process; S103, through variational Bayesian The inference algorithm estimates the model parameters of the non-parametric VMF hybrid model; S104, according to the inferred model parameters, determine whether the non-parametric VMF hybrid model converges; if not, return to step S103, and if so, execute step S105; S105: Determine the category to which each text belongs according to the posterior probability of the indicator factor, so as to cluster the text according to the category. The present invention can ensure the algorithm convergence and can effectively detect the convergence state.

Description

technical field [0001] The invention relates to the field of text mining, in particular to a text data clustering method, device and equipment based on a non-parametric VMF mixed model. Background technique [0002] With the rapid development of the Internet and the widespread use of news documents, text data clustering, as one of the most useful tasks in text mining, has received increasing attention in recent years. [0003] In prior art 1, Zhong Wenliang et al. proposed a method for clustering unbalanced text data based on the Pitman-Yor process. In this method, each text is represented by a TF (term frequency, term frequency) vector, each attribute of the vector represents the frequency of a specific term (term) appearing in the document, and all terms in each category obey The same multinomial distribution (Multinomial Distribution). This method uses the Polya urn model to build a Pitman-Yor process model based on multinomial distribution, and uses the Gibbs sampling ...

AI Technical Summary

Problems solved by technology

The discrete nature of the Pitman-Yor process model cannot be represented intuitively

[0006] 3. The Gibbs sampling algorithm used to solve the model parameters cannot obtain an analytical solution, and it is not easy to converge and it is difficult to determine the convergence state

The concentration parameter in the prior art 2 uses an asymptotic approximation method to obtain an estimated value, but this estimation method cannot effectively deal with high-dimensional data

Images