Short-text data stream classification method based on short-text expansion and concept drift detection

Abstract

The invention discloses a short-text data stream classification method based on topic models and concept drift detection. The method includes: 1, acquiring an external corpus from a knowledge libraryto construct the LDA topic model; 2, dividing a short-text data stream into data blocks according to a sliding window mechanism, and using the LDA topic model to expand short text in the data blocks to obtain an expanded data stream; 3, constructing the online BTM topic model for each data block in the expanded short-text data stream, and obtaining a topic representation of each piece of short text; 4, selecting data blocks of Q topic representations to construct a classifier to use the same to predict a class label of a newly arrived data block; 5, dividing the data blocks of the Q topic representations into category clusters according to class label distribution, and calculating semantic distances between the category clusters and the newly arrived data block to judge whether concept drift occurs; and 6, updating the classifier according to a concept drift situation. The method can be used for the short-text data stream classification problem of unceasingly changed class label distribution.

Description

technical field [0001] The invention belongs to the field of mining text data streams in practical applications, and in particular relates to the classification problem of ever-changing short text data streams. Background technique [0002] With the rapid development of instant messaging and Internet technology, network users and network servers generate a large number of short text data streams, including Sina Weibo, online comments and instant messages. These short text data contain rich value for scientific research institutes, government departments and Internet service providers. The short text data stream has the following three characteristics: 1. Each short text is short in length and does not have enough information, resulting in serious data sparsity; 2. The huge amount of data generated in a short period of time is likely to cause serious dimension disasters; 3. , The potential drift of text topics over time. Based on these three characteristics, traditional sho...

AI Technical Summary

Problems solved by technology

Due to the phenomenon of concept drift often occurring in short text data streams, existing data stream classifications rarely address this problem, so it is difficult to achieve good classification results

Images