Short text keyword extraction method and device

Abstract

The invention provides a short text keyword extraction method and device.The method comprises the steps that a to-be-extracted keyword text and a text related to the source of the to-be-extracted keyword text are spliced, and a long text is obtained; performing theme classification on the long text by utilizing a set theme model to obtain theme classification data; and calculating a word importance score considering category statistical information based on the subject classification data to obtain keywords of the text to be subjected to keyword extraction. By means of the scheme, the problems that the short text is short and the data set deflects due to the fact that text differences in all fields are large can be solved, and therefore the accuracy of keyword extraction on the short text can be improved.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a short text keyword extraction method and device. Background technique [0002] In recent years, Weibo has developed rapidly due to its platform's openness and concise content, and is becoming an important channel and carrier for social relationship maintenance and information dissemination in human society. People can use Weibo to share data in real time. At the same time, because Weibo content can be released in real time through various communication means (such as mobile phones, etc.), it is easy to generate a large amount of data in a short time. However, these data are usually messy, and it is difficult to obtain interesting information from them in a timely and accurate manner. Therefore, it is particularly important to extract keywords of microblog text. Accurately extracting keywords can effectively identify hot words and hot topics of the day. ...

AI Technical Summary

Problems solved by technology

Secondly, some common background words (such as "tomorrow", "hehe", etc.) appear very frequently in Weibo, which will also have a certain impact on text mining tasks

Furthermore, because microblogs limit the number of characters of blog posts published by users, for example, they cannot exceed 140 characters, so most of the microblog texts are short texts published or forwarded and commented on other people's microblogs, and short texts are very important for extracting keywords. have a great impact

In addition, the texts forwarded and commented on other people's microblogs are not only short, but often lack important information, which leads to the inability to effectively identify keywords

In addition, Weibo texts cover a wide range of fields, and the proportion of each field varies greatly. This feature also seriously affects the accuracy of keywords extracted based on statistical information.

[0004] Therefore, the accuracy of keyword extraction needs to be improved urgently for short texts similar to Weibo that have the characteristics of high noise, lack of important information, and rich coverage but large differences in proportion.

Images