News automatic labeling method based on LDA model

Abstract

The invention relates to a news automatic labeling method based on an LDA model. The news automatic labeling method includes the steps: extracting text data features at a semantic level, and having better effect in practical application; proposing improvements for an LDA model, utilizing point mutual information for quantizing the subject term relation, obtaining the co-occurrence relation betweensubject terms by calculating the weights of the subject terms, and setting a threshold value to select the optimal subject term. For the news automatic labeling method, keywords with high accuracy are selected according to the strength of the representation ability of vocabularies corresponding to different topics, and mutual information can be introduced to improve a topic-lexical item matrix, so that the accuracy of an LDA model in news document automatic label application is improved, and the correlation between subject terms is better described.

Description

technical field [0001] The invention relates to the technical field of text processing, and more specifically, relates to an LDA model-based automatic labeling method for news. Background technique [0002] With the development of the information network, information overload, news texts exploded, and most of the texts are long. If you can get a general understanding of what the article is about before reading it carefully, you can save time and quickly find the news content that you care about. Select Read a certain piece of news carefully. The task of automatic news labeling is to characterize the text content, and then filter out useful information. How to more accurately extract the information to be expressed in the text is one of the important topics of current research and has been widely It is applied to natural language processing tasks such as text classification, clustering, news recommendation, machine translation, and paper indexing. The LDA topic model is a co...

AI Technical Summary

Problems solved by technology

[0003] Commonly used methods for automatic labeling of keywords are statistical methods, such as TFIDF, which are fast and simple, but only based on word frequency, which is not comprehensive enough and ignores semantic information. From the semantic aspect, there are topic-based methods, such as LDA The model is very effective for the extraction of semantic information and dimensionality reduction of feature space. There is also TextRank, which does not require training data and is fast, but it ignores the correlation between semantics and does not consider the relationship between contexts. relation

Although the LDA model is widely used, it still has some shortcomings. The LDA model will carry out topic labels on all terms, which cannot represent the original information of the data well.

Images