LDA fusion model and multilayer clustering-based news topic detection method

Abstract

The invention belongs to the field of data mining, natural language processing and information retrieval, and provides a news topic detection method. For the defect of a TF-IDF-based vector space algorithm in semantics and the defects of time complexity and accuracy of textual level clustering, feature extraction, representation modeling, similarity calculation and quick and accurate text clustering methods for a large amount of news texts are improved. The LDA fusion model and multilayer clustering-based news topic detection method comprises the following steps of 1: building a similarity model by using a vector space model (VSM); 2: finally obtaining accurate parameter settings; 3: organically fusing two text models; 4: judging whether a topic is a new topic or not; 5: calculating the similarity until all documents are clustered; and 6: adding an ISP&AH clustering algorithm of AHC based on the step 5. The method is mainly applied to the design and manufacturing occasions.

Description

technical field [0001] The invention belongs to the fields of data mining, natural language processing and information retrieval, and relates to monitoring technology and network information filtering technology, especially text analysis and topic detection methods. Specifically, it involves a news topic detection method based on latent Dirichlet Allocation (LDA) fusion model and multi-layer clustering. Background technique [0002] Topic Detection and Tracking (TDT) evolved from Event Detection and Tracking (EDT) in the early years. It is an automatic content analysis of news reports without human intervention. Techniques for identifying, mining, and organizing taxonomies. The vector space model (Vector SpaceModel, VSM) based on Term Frequency–Inverse Document Frequency (TF-IDF) has shown powerful capabilities in text representation. The vector space model is an algebraic model used to represent text files. It applies to information filtering, information retrieval, inde...

AI Technical Summary

Problems solved by technology

[0006] In order to overcome the deficiencies in the prior art, the present invention aims to propose a news topic detection method based on the LDA fusion model and multi-layer clustering, aiming at the defects in the semantics of the TF-IDF vector space algorithm, and the time complexity and complexity of the text hierarchical clustering Defects in accuracy, improve feature extraction, representation modeling, similarity calculation and fast and accurate text clustering methods for a large number of news texts

Images