Microblog topic detection method based on improved Single-pass clustering algorithm

Abstract

The invention discloses a microblog topic detection method based on an improved Single-pass clustering algorithm. The method includes microblog text content collection, text preprocessing, text vectormodel establishment based on LDA (Latent Dirichlet Allocation), text clustering based on the improved Single-pass clustering algorithm, and result evaluation. The improved Single-pass clustering algorithm includes: adding a time parameter, calculating clustering center points for category data, and inputting the data in a batch manner. According to the method and the algorithm, identity of topicsis guaranteed through adding the time parameter; the new data are compared with the clustering center points through calculating the clustering center points for the category data, thus reducing of frequency of comparing the new data with each piece of data is facilitated, and efficiency of calculation is improved; and newly input clustering center points are compared with the center points afterclustering through inputting the data in the batch manner, namely firstly clustering the data and then inputting the same, operation efficiency is improved, and operation space is saved.

Description

technical field [0001] The invention relates to the technical field of topic detection, in particular to a microblog topic detection method based on an improved Single-pass clustering algorithm. Background technique [0002] LDA (Latent Dirichlet Allocation) is a document topic generation model, which contains three layers: words, topics and documents. The generative model is that each word of an article is obtained through a process of "selecting a certain topic with a certain probability, and selecting a certain word from this topic with a certain probability". Express this process with formula (1): [0003] P(word|document)=∑ 主题 P(word|topic)*P(topic|document) (1) [0004] In the LDA model, it is necessary to solve the two model parameters of "word-topic" and "topic-document". The probability map of the probabilistic topic model is as follows: figure 1 shown. [0005] exist figure 1 middle, Denotes a "topic-term" probability model with a multinomial probability di...

AI Technical Summary

Problems solved by technology

At the same time, the existing single-pass topic detection can only be classified one by one, and each new data must be calculated with each piece of data that has been clustered, which is very poor in efficiency and accuracy.

Images