Topic phrase extraction method

Abstract

The invention relates to a topic phrase extraction method. The topic phrase extraction method includes preprocessing documents, seeking a document-topic set, a full text lexical chain set and a noun phrase set, seeking a central word set, seeking a candidate topic phrase set, and seeking a topic phrase set. The topic phrase extraction method has the advantages that topic phrases are extracted through combination between an LDA (latent Dirichlet allocation) model and a lexical chain, a knowledge base WordNet with complete semantic information outside a corpus can be utilized, a strong lexical chain can be acquired through semantic relevance calculation and strong chain rule filtration, and accordingly, the ambiguity of topic words is reduced greatly; the topic phrases are extracted according a central word extraction method and by N-P rule combination and deduplication steps, and topics are expressed by the topic phrases with rich semantic information, so that the problems such as low granularity and recognition degree of the topic words are solved, topic extraction accuracy and recall rate can be guaranteed, topic drifting is reduced, and needs of practical applications can be wellmet.

Description

technical field [0001] The invention belongs to the technical field of text mining, and in particular relates to a topic phrase extraction method. Background technique [0002] Literature topic extraction technology can not only improve the quality of document retrieval, but also effectively deal with the high-dimensional sparsity of the document vector space representation model. It is widely used in NLP tasks such as text classification, clustering, and information recommendation. Therefore, topic extraction is It is one of the research focuses in the field of text mining today. [0003] The LDA model is a probabilistic topic model commonly used in the field of document topic research. It can identify potential topic information in large-scale document collections and corpora without relying on knowledge bases. However, the traditional LDA model is based on the assumption of the "bag of words" model, ignoring the order of the words in the document, sampling the topics of...

AI Technical Summary

Problems solved by technology

However, the traditional LDA model is based on the assumption of the "bag of words" model, ignoring the order of the words in the document, sampling the topics of the corpus itself, and only using the semantic information inside the corpus, which is prone to more ambiguity problems of topic words

Therefore, in many cases, the effect of using the LDA model for topic extraction is not ideal, and there are problems such as too small information granularity of topic words, low topic recognition, and ambiguity of topic words.

Images