Dynamic streaming data clustering method

Abstract

The invention discloses a dynamic streaming data clustering method, which comprises the steps of converting structured data into time field streaming data, sorting the time field streaming data according to time fields so as to acquire time slices, and solving a union set; building a training model, and building HMM prediction for the missing data; checking the data validity, and adding time slices for repeated data points; eliminating abnormal data, checking whether data with abnormal fluctuations exists or not according to all of the time slices; and performing mass center data clustering. According to the invention, special optimization is performed in allusion to characteristics of the data, an HMM is adopted to perform prediction in allusion to the missing data, and processing is performed in allusion to repeated data with the same identification in the same time slice, so that time-varying characteristics of the data can be reflected more accurately, abnormal data can be distinguished, the number of clustering categories is optimized automatically, and a high-quality clustering result is acquired.

Description

technical field [0001] The invention relates to a clustering method, in particular to a clustering method for dynamic stream data. Background technique [0002] Clustering algorithm is one of several major methods in the field of data mining, such as classification, clustering, regression, and factor analysis. In the era of big data, clustering algorithms are used to analyze massive data to obtain better decision-making capabilities. The advantages of clustering algorithms can handle unsupervised machine learning and actively classify unlabeled data. With the deepening of research on clustering algorithms, researchers have proposed more and more different clustering algorithms, including partition-based clustering, grid-based clustering, and hierarchical-based clustering. These algorithms are proposed for data sets of different dimensions, scales, and types. For the same data set, different clustering algorithms are used, and the results obtained may vary greatly. [0003]...

AI Technical Summary

Problems solved by technology

[0005] However, these methods use time as a separate field or dimension and integrate it into the original data. In fact, they only upgrade the original data to a dimension for clustering calculations.

This will cause a problem. Some business scenarios are originally changing according to time, and time is only a common dimension. The clustering effect is not good, and it cannot better reflect the classification characteristics of the data.

[0006] In summary, when the number of data changes over time, or is added, or decreased, or the data changes in the middle, using traditional K-means, X-means and other methods cannot effectively deal with the above complex effective data clustering in different situations, and currently there is no better solution for real-time streaming data in the industry

Images