Active learning short text classification method and system based on sampling frequency optimization

Abstract

The invention discloses an active learning short text classification method and system based on sampling frequency optimization, broadens the active learning optimization direction, and provides a simple and effective optimization framework widely used in the industry. According to the technical scheme, the method includes enabling a text classifier to learn labeled data; sampling and evaluating the unlabeled data based on the learning result of the text classifier and selecting the most valuable data; and manually labeling the selected data and adding the selected data into the labeled data,and repeating the steps until the number of iterations reaches the upper limit or the accuracy reaches the standard. In the sampling evaluation process, the labeled data is classified according to thecategory to which the labeled data belong, and the labeled data volume of each category is counted to obtain respective sampling frequency data; for the unlabeled data, the invention includes evaluating the unlabeled data to obtain an initial evaluation score and a prediction result category of the initial evaluation score, then obtaining corresponding sampling frequency data according to the prediction result category, and obtaining a final evaluation score based on the initial evaluation score and the sampling frequency data of the corresponding category.

Description

technical field [0001] The invention relates to a technique for using active learning to process short text classification, in particular to an active learning short text classification method and system based on sampling frequency optimization. Background technique [0002] With the prevalence of e-commerce and online communication, in many application fields, such as instant messaging, online chat logs, bulletin board system headlines, Internet news comments, Twitter, etc., short text content is flooding people's daily life. In the face of many needs such as topic recommendations and e-commerce chat robots derived from this, short text classification becomes very important. However, due to the non-standard expression of short text, labeling the required content will consume a lot of manpower, and the effect is relatively limited. [0003] At present, the industry uses active learning to deal with short text classification problems, but when the mainstream active learning ...

AI Technical Summary

Problems solved by technology

However, due to the non-standard expression of short text, labeling the required content will consume a lot of manpower, and the effect is relatively limited

[0003] At present, the industry uses active learning to deal with short text classification problems, but when the mainstream active learning methods are directly applied to this, we find that the performance in short text classification is not good, and with the increase of the number of categories in the data set, the existing The performance of the sampling method will also decrease accordingly

Images