Microblog hot topic discovery algorithm based on BTM and GloVe similarity linear fusion

Abstract

The invention relates to a microblog hot topic discovery algorithm based on BTM and GloVe similarity linear fusion, which is characterized by comprising three stages of data acquisition and preprocessing, modeling and clustering, and comprises the following steps of: performing data acquisition and preprocessing, modeling obtained data, and clustering the modeled data; the invention provides a microblog hot topic discovery algorithm based on BTM and GloVe similarity linear fusion in order to solve the problem that a distance function of a K-means algorithm can influence a microblog hot topic clustering result. The GloVe model only trains non-zero elements in the word and word co-occurrence matrix rather than the whole sparse matrix to utilize statistical information, and the sparsity problem faced by the TF-IDF algorithm in the document-word vector matrix construction process is effectively relieved. The GloVe model is combined with a global matrix decomposition method and a local context window method at the same time, the trained word vector can carry more semantic information, and the problem that one word has multiple meanings and cannot be well solved by a BTM topic model canbe relieved to a certain extent.

Description

technical field [0001] The invention relates to the technical field of topic discovery and tracking in natural language processing, in particular to a microblog hot topic discovery algorithm based on linear fusion of BTM and GloVe similarity. Background technique [0002] With the rapid development of traditional Internet and mobile Internet, Weibo has flourished. Weibo allows users to publish messages through web pages, external programs, and mobile phone clients to achieve message sharing. The advantages of short text, timeliness and interactivity of Weibo have been recognized by the public, and it has gradually become an important tool for people to obtain and publish information. How to mine hot topics from massive and disorderly microblog data has become an urgent problem to be solved. [0003] In order to solve the above problems, there are many methods nowadays, mainly including "use TF-IDF to vectorize the microblog text set, and then use clustering algorithm to fi...

AI Technical Summary

Problems solved by technology

[0029] (2) JS divergence

[0067] 1. TF-IDF is an algorithm based on statistical information such as word frequency, which ignores the semantic information of words and affects the accuracy of topic clustering

[0068] 2. The research goal of this paper is a large number of short microblog texts. The document-word vector matrix constructed by the TF-IDF algorithm will have high sparsity, which will affect the efficiency of the algorithm

Images