Improved feature evaluation function based Bayesian spam filtering method

Abstract

Disclosed is an improved feature evaluation function based Bayesian spam filtering method. The method includes the steps of 1), preprocessing a training mail set into a mail head part and a text part; 2), respectively deleting prepositions, pronouns, adverbs, auxiliary words, conjunctions and words with the work frequency lower than the given threshold P in two feature sets T1 and T2; 3), respectively calculating a mutual information value MI (tk)' in the feature sets T1 and T2 by adopting the improved feature evaluation function; 4), in the training set, sorting the MI (tk)' according to the order from big to small, and selecting feature items corresponding to first n values as representation of the training set; 5), performing spam filtering on to-be-tested samples by adopting a Bayes classifier at the sorting phase. With the method, mails can be classified highly accurately, and spasm can be filtered out.

Description

Technical field [0001] The invention relates to a Bayesian spam filtering method based on an improved feature evaluation function. Background technique [0002] The most common feature selection method in Bayesian spam filtering is the "mutual information" method. This method can effectively express the degree of dependence between words in text classification, but it will be prominent when used in the feature selection stage of spam filtering. The following problems cause the performance of the entire filtering method to be lacking: 1 Positive and negative correlation problem: The correlation between the feature item and the text category is divided into two types: positive correlation and negative correlation. Both cases indicate that the feature item has a definition of the category. The effect is that the positive correlation has a stronger performance ability on the category, and the negative correlation performance ability is weak, but the meaning expressed from the formula...

AI Technical Summary

Problems solved by technology

[0002] The most common feature selection method in Bayesian spam filtering is the "mutual information" method. This method can effectively express the degree of dependence between words in text classification. However, it will be highlighted in the feature selection stage of spam filtering. The following problems make the entire filtering method lack in performance: 1. Positive and negative correlation issues: the correlation between feature items and text categories is divided into two types: positive correlation and negative correlation. The effect is that the positive correlation has a strong ability to express the category, and the negative correlation has a weak ability to express, but the meaning expressed from the formula is that the positive and negative offset each other, that is, the negative correlation has the opposite effect on the performance, which is contrary to the original intention; 2 Ignoring word frequency and tending to low-frequency words: the mutual information feature selection method is based on the assumption that the amount of text in each category is roughly equal

In addition, only the occurrence and non-appearance of the term is considered, regardless of the number of times the term appears in the document, but usually we think that the feature words with more occurrences (that is, higher word frequency) are more related to the category and more representative This category, so this has an impact on the feature items that appear frequently in an email; 3. The feature items in different positions have different contribution to the category definition: the feature items extracted from the two different positions of the email title and the body The ability to contribute to classification will be very different. In actual spam filtering, users can often judge whether an email is normal email or spam from its main image.

However, for the above problems, there is currently no improvement method for these unsuitable problems.

Images