Rapid text clustering method on large corpus

Abstract

The invention belongs to the technical field of relational databases, and particularly relates to a rapid text clustering method on a large corpus. Due to the fact that the text data usually has the advantages of high dimension and sparsity, a good effect is difficult to achieve merely through a clustering method based on data similarity, but for a method based on a generation model, such as a Dirichlet multiple hybrid model, expression is more prominent. Accordingly, optimization is conducted through symmetric prior and construction index of Dirichlet distribution, the total time only dependson the number of different words in a document, and efficient running can be achieved in long documents.

Description

technical field [0001] The invention belongs to the technical field of relational databases, and in particular relates to a fast text clustering method on a large corpus. Background technique [0002] Text clustering is a common problem in data mining. It is an important means to organize text information effectively and plays an important role in the research of natural language processing and other aspects. [0003] Since the text data is only composed of words, compared with other extracted feature data, it usually has a higher dimension and is more sparse. The clustering method based solely on data similarity is difficult to obtain better results, and the method based on generative models such as Di Likelier multinomial mixture models are more prominent in performance. [0004] However, the time spent by the Dirichlet multinomial mixture model is proportional to the length of the document. For a large corpus, the documents in it are often large, resulting in unsatisfact...

AI Technical Summary

Problems solved by technology

[0003] Since the text data is only composed of words, compared with other extracted feature data, it usually has a higher dimension and is more sparse. The clustering method based solely on data similarity is difficult to obtain better results, and the method based on generative models such as Di Likelier multinomial mixture models are more prominent in performance

[0004] However, the time spent by the Dirichlet multinomial mixture model is proportional to the length of the document. For a large corpus, the documents in it are often large, resulting in unsatisfactory convergence speed and affecting the overall data processing efficiency.

Images