A document keyword extraction method and device based on LDA and word vectors

Abstract

The invention relates to the technical field of natural language processing and deep learning, in particular to a document keyword extraction method based on LDA and word vectors, which comprises thefollowing steps of (A) judging whether a document title and content are consistent or not by using a title discriminator, and executing the next step if the document title and the content are consistent; (B) calculating the weight of the theme in the document and the weight of the vocabularies in the document on the theme; (C) calculating the weight of the vocabularies in the document, and sortingaccording to the weight values to generate a candidate keyword set of the document; (E) mapping the vocabularies into a word vector space; (F) calculating the distance between the word vectors in theword vector space, sorting the word vectors according to the distance, and selecting the first M sorted vocabularies as keywords of the document. The invention further discloses an extraction device.Compared with a traditional method, the extracted document keywords are high in precision and high in reliability, the titles and the Chinese characters are filtered out, the interference of noise data is avoided, and accuracy is further improved.

Description

technical field [0001] The present invention relates to the technical fields of natural language processing and deep learning, in particular to a document keyword extraction method and device based on LDA and word vectors. Background technique [0002] Keywords can concisely and accurately describe the content of the text, and generally consist of several words and phrases. Keyword extraction, also known as keyword tagging, refers to extracting a number of representative words or phrases from text or text collections to reflect the main semantic information of the text. An important channel for information of interest. The advent of the Internet era puts forward new requirements for keyword extraction. The extracted keywords should have the following three characteristics: significance, readability and comprehensiveness. Significance means that the extracted keywords should reflect the core content of the document. For example, "machine translation" is extracted from the d...

AI Technical Summary

Problems solved by technology

There are some deficiencies in these studies: first, the keywords recommended at the topic level tend to be common words in documents, and cannot highlight the core content of each document; second, the extracted topic words may contain other irrelevant words, resulting in The offset of keywords; third, the appearance of synonyms or near synonyms with the greatest topic relevance leads to redundant recommended keywords, and it is impossible to realize the comprehensiveness of keywords to document topics

Images