Topic feature text keyword extraction method

Abstract

The invention discloses a topic feature text keyword extraction method. Through the method, text keyword extraction results better than those of a traditional TF-IDF method can be obtained. Accordingto the technical scheme, at a training stage, word segmentation, stop word removal, part-of-speech filtering and other preprocessing are performed on a training text, statistical analysis is performedon inverse document frequency of words, meanwhile a topic model method is utilized to learn and obtain a topic probability matrix of the words, normalization processing is performed, topic distribution entropy of the words is calculated according to the topic probability matrix of the words, global weights of the words are calculated in combination with the inverse document frequency and the topic distribution entropy, and global weight calculation results are output to a test stage; and after a test text is preprocessed, statistical analysis is performed on normalized term frequency of wordsin the test text, the normalized term frequency is combined with the global weight calculation results obtained at the training stage, comprehensive scores of the words are calculated are ordered, and a plurality of words with the highest scores in the score order are used as automatic keyword extraction results of the current test text.

Description

technical field [0001] The invention belongs to the technical field of natural language processing, and in particular relates to a method for extracting text keywords based on topic distribution characteristics of words. Background technique [0002] Keyword extraction is the key to technologies such as information retrieval, text classification and clustering, and automatic abstract generation, and is an important means to quickly obtain document topics. Keywords are traditionally defined as a set of words or phrases that summarize the subject matter of a document. Keywords characterize the topic and key content of a document, and are the smallest unit to express the core content of a text. Keywords have very important applications in many fields, such as automatic summarization of documents, web page information extraction, classification and clustering of documents, search engines, etc. However, in most cases, the text does not directly provide keywords, so it is necess...

AI Technical Summary

Problems solved by technology

The disadvantage of the statistical method is that the amount of calculation is large; the extraction results will be meaningful and incomplete strings, resulting in low accuracy; low-frequency words cannot be extracted; a large amount of original text is required

IDF cannot effectively reflect the importance of words and the distribution of feature words, making it unable to complete the function of weight adjustment well, so the accuracy of TF-IDF method is not very high

Images