Automatic microblog text abstracting method based on unsupervised key bigram extraction

Abstract

The invention discloses an automatic microblog text abstracting method based on unsupervised key binary word extraction. The automatic microblog text abstracting method comprises the steps of preprocessing a microblog; standardizing a binary word; extracting a key binary word based on a mixed TF-IDF (term frequency-inverse document frequency), TexRank and an LDA (local data area); sequencing sentences based on the intersection similarity and a mutual information strategy; extracting abstract sentences based on a similarity threshold value; generating abstract by reasonably combining the abstract sentences. According to the automatic microblog text abstracting method, the binary word is used as a minimum vocabulary unit, and the binary word has richer text information than words, so that the sentences based on the key binary word is higher in noise immunity and accuracy than the sentences based on key word extraction; meanwhile, when the abstract sentences are extracted, the similarity threshold value is introduced to control redundancy, so that the abstract is higher in recall rate. The abstract generated by the method is accurate, simple and comprehensive; the efficiency and the quality that a user acquires knowledge are obviously improved, and the time of the user is greatly saved.

Description

technical field [0001] The invention relates to a method for automatically summarizing short texts in social media such as microblogs, in particular to a method for automatically summarizing microblog texts based on non-supervised key word string (bigram) extraction. Background technique [0002] At present, there are not many methods for automatic summarization of a large amount of Weibo text generated by social media platforms such as Twitter and Sina Weibo. Most of the existing summarization methods based on microblog text features score or rank sentences directly based on the bag-of-words model, and then extract the top-ranked sentences to combine into a summary output (for example, refer to the article Inouye, D., Kalita, J.K. "Comparing twitter summarization algorithms for multiple post summaries", Social Computing, 2011, 298-306). However, for Weibo, which is an extremely non-standard dialogue-like short text, it is easy to introduce a lot of noise, resulting in a lo...

AI Technical Summary

Problems solved by technology

[0003] In order to overcome the deficiency that the existing microblog text automatic summarization method is not robust to noise, which leads to the low accuracy rate of the extracted summaries, the present invention provides a microblog text automatic summarization method based on unsupervised key binary word string extraction, In order to automatically generate concise, accurate and comprehensive summaries for thousands of Weibo texts under the same hot topic, thereby improving the user's browsing experience

Images