Graph model text abstract generation method based on word frequency and semantics

Abstract

The invention discloses a graph model text abstract generation method based on word frequency and semanteme. The method comprises the following steps of 1) performing word segmentation on sentences ina text, and performing part-of-speech tagging; 2) filtering the lexical items, and only reserving the lexical items with specific part-of-speech; and 3) training word vectors by using a Word2Vec model and a BM25 algorithm to form a feature word vector set, further representing sentences, and constructing a sentence-word text matrix; 4) constructing a text undirected graph model through the text matrix; and 5) performing iterative computation of sentence node weights by using a TextRank algorithm until convergence, and selecting TOP-K sentences to generate text abstracts. 6) experimental results show that the method is suitable for industrial production, compared with a traditional text automatic abstracting method considering a single word frequency characteristic of a text and based on atext semantic characteristic, according to the method, under the optimal adjustment factor combination, a higher Rouge value is obtained, it is proved that the method effectively integrates text wordfrequency and semantic features, and then the abstract generation accuracy is improved through a TextRank algorithm based on contextual information.

Description

technical field [0001] The invention relates to a method for generating text summaries, in particular to a method for generating text summaries based on word frequency and semantic graph model. Background technique [0002] The BM25 algorithm or the traditional neural network algorithm is commonly used in text summarization to evaluate the correlation between search terms and documents. However, the following problems will be highlighted in the selection stage of the text similarity used in text summarization generation, so that the text similarity has a significant impact on performance. lacking. [0003] 1. Traditional neural network algorithms require a large corpus, long training time, slow generation of summaries, and poor applicability. Using a shallow neural network to compress the dimension of words can shorten the training time and generate summaries faster. [0004] 2. Previously supervised algorithms required training corpus and manual labeling of corpus, which ...

AI Technical Summary

Problems solved by technology

[0004] 2. Previously supervised algorithms required training corpus, manual labeling of corpus, etc., which is not practical; compared with supervised learning, unsupervised learning uses unlabeled data

[0005] 3. The abstract generation method based on machine learning can fully obtain the semantic information of words and sentences through the training corpus, but this kind of method relies too much on the corpus containing multiple target words, and is only suitable for processing short texts. Too long text input sequences will As a result, the learning framework cannot accurately obtain the semantic information of the sentence, and the encoder cannot accurately extract the semantic information of the text, resulting in a long-distance dependency problem, which leads to the inability of the model to converge, which in turn affects the accuracy of summary generation

Images