Method for automatically filtering stop words

Abstract

The invention discloses a method for automatically filtering stop words, which is used for filtering out stop words in texts. The method comprises the following steps of: carrying out preprocessing on a text, i.e. carrying out decomposition and generalization on the text so as to compress the size of a word stock; searching absolute stop words and filtering absolute stop words, wherein the absolute stop words include words unassociated with the specific properties of a corpus; searching relative stop words and filtering relative stop words, wherein the relative stop words are expressed by using a natural language but not the combinations of discrete keywords; and dynamically recognizing stop words, i.e. based on the text size of a context associated with a word and the position of the word, calculating the conditional probability that the word is a stop word, recognizing a word (with a conditional probability greater than a conditional probability threshold) as a stop word, and filtering the stop word. By using the method for automatically filtering stop words, disclosed by the invention, the judgment on stop words is implemented based on the analysis on texts, and for different texts, stop words in each text are filtered, so that the filtering accuracy of stop words can be improved.

Description

technical field [0001] The invention relates to the technical field of data retrieval, in particular to a method for automatically filtering stop words. Background technique [0002] Keywords are used as the main basis for retrieval in data retrieval. There are many words in a piece of text, some can be used as keywords, and some obviously cannot be used as keywords. Words that obviously cannot be used as keywords are stop words. Strictly speaking, stop words (Stop Words) are "virtual words and non-retrieval words in computer retrieval". It is mainly used to save storage space and improve efficiency when extracting keywords. It is widely used in search engines and classification technologies. In actual operation, the algorithm will automatically ignore certain words or words, which are called stop words. (Stop Words). [0003] Stop words are equivalent to filter words to a certain extent, but the scope of filter words is larger. Keywords containing sensitive information ...

AI Technical Summary

Problems solved by technology

Similarly, each language is also advancing with the times, and the intercommunication between languages ​​will also change. Once an article is mixed with several languages, it becomes very complicated to refer to the stop word list up

For example, an article that introduces the relationship between Chinese and English is mixed with Chinese and English, and keywords are extracted from it. If this article is aimed at Chinese users, then the keywords obtained must be in Chinese. How to deal with the English words in the article, this is a relatively complicated process

[0009] (3) Lack of semantic level analysis

These words are very likely to be keywords in some articles, but after the word segmentation algorithm is completed, it is very likely to filter them out through stop words, so the accuracy rate will definitely be affected

Images