Text classification method

Abstract

A text classification method comprises following steps: dividing the initial training text collection into a plurality of subsets including the text in the same category based on the category, extracting the corresponding probability topic model from each subset; generating new text to balance the categories of the subsets by the corresponding probability topic model; constructing a classifier based on the balance training text collection corresponding to plural subsets; and processing text classification by the classifier. The invention can improve the classification effect of the text classification method under the condition of data skew.

Description

technical field [0001] The invention relates to data preprocessing technology, in particular to a text classification method. Background technique [0002] With the rapid development of the Internet, electronic texts such as webpages, e-mails, databases, and digital libraries on the network are growing exponentially. How to effectively process and classify these texts is a very important topic. Text classification refers to constructing a classification model based on existing data, that is, a classifier. It determines a category for each document in the test document collection according to a predefined classification system, so that users can browse documents conveniently, and can also make document search easier by limiting the search scope. Automatic text classification is to use a large number of texts with class marks to train the classification criteria or model parameters, and then use the training results to identify texts of unknown categories. [0003] However,...

AI Technical Summary

Problems solved by technology

[0003]However, in actual classification applications, data skew is often encountered, also known as data imbalance or category imbalance, which is one of the important factors affecting classification performance , which poses a challenge to traditional classification methods

Most classification algorithms are proposed for uniformly distributed data. For the case of data skew, only traditional classification methods cannot achieve ideal text classification results.

Images