Text classification based on tfidf algorithm and related word weight correction

Abstract

The invention relates to a text classification method based on a tfidf algorithm and related word weight correction. The text classification method comprising the steps of S1, extracting category keywords; S2, forming a sliding text window, setting a word weight and modifying the position thereof in the sliding text window; S3, calculating the word frequency of words according to a word frequencystatistical correction function; S4, performing weighting calculation according to a TFIDF algorithm to realize vectorization of the words in a text; and S5, classifying the text by a SVM classifier.In the process of text classification, the weight of the category keywords is increased, so that the result of the text vectorization better reflects the text information. The method of the inventionintroduces the text sliding window and takes full account of the position information of the words in the text. The category keywords come from partially training data and users, the category keywordsare extracted by using the tfidf algorithm, the characteristics of the keywords can be extracted efficiently and accurately, meanwhile, the case of a few category keywords of an actual application scene is balanced, and the category keywords are extracted comprehensively and accurately.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a text classification method based on tfidf algorithm and related word weight correction. Background technique [0002] In the prior art, the commonly used technology for text classification is to use the tfidf algorithm to calculate the weights of related words and vectorize the related words. [0003] The tfidf algorithm was proposed by Salton in 1988. The core of the algorithm is: words that appear more frequently in the same text and appear less frequently in different texts should be given higher weights. Among them, words that appear in the text Frequency (TF), used to describe the ability of the word to reflect the content of the document; Inverse Document Frequency (IDF), used to calculate the ability of the word to distinguish different documents, the calculation formula is as follows: [0004] [0005] [0006] TF_IDF (i，j) =TF (i，j) *IDF i [0007] no (i，...

AI Technical Summary

Problems solved by technology

[0011] The disadvantage of using the tfidf algorithm for text classification is: the tfidf algorithm assumes that the smaller the text frequency, the greater the ability to distinguish different types of text, so the concept of inverse document frequency IDF is introduced, and the weight TF is completed. Adjustment, the purpose of adjusting weights is to highlight important words and suppress secondary words. The simple structure of the algorithm cannot effectively reflect the importance of words and the position distribution of feature words, and cannot well complete the function of weight adjustment. The accuracy of text classification is not very high

Images